Month: April 2015

VMware vRealize Automation Part 8 – Adding a VMware vCloud Air Endpoint & Publishing a Cloud VM Blueprint

Chan Leave a comment

 

So now we have a fully functioning vRA 6.2.1 deployment, fully integrated to the on-premise vCenter instance, the vRO appliance for workflow orchestration and NSx for network orchestration (via vRO). Now lets look at how to set up a cloud endpoint so that you (or the users) can request VM’s to be provisioned on the cloud rather than their local vSphere cluster. We are looking at adding VMware’s own vCloud Air platform in this article (if I managed to gain access to an Amazon AWS instance, I’d publish a future post on that too as each cloud platform integration is different to one another.

VMware vCloud Air (formally known as vCHS) is VMware’s own managed and operated cloud platform, that runs on the same vSphere technology as your on-premise environment. They have a vCloud Director instance in front, which manages the multi tenancy aspect of a collection of vSphere clusters which you can either buy a subscription as an on demand basis (similar to AWS) or monthly / annual subscription basis (with no usage charges which is real handy). vCloud Air has been around a while now and is quite popular given that you don’t have to change the architecture of your on-premise applications or servers (VMs) that they are installed on to move them to the cloud (which is the case with both Amazon and Azure and could be painful and expensive). With vCloud Air, you just move the whole VM as is with the application already deployed on it and it will work fine on vCloud Air platform just like it did on your own vSphere cluster (You also have the option to do a “Stretched deployment” which is a way of  moving the VM to the cloud but establishing a Layer 2 network between your vSphere cluster and vCloud platform over a VPN so no IP’s need changing which is awesome).

Just like AWS, vCloud Air (as well as any other 3rd party cloud provider who runs their cloud platform behind vCloud Director basically) can be integrated to your on-premise vRA instance as an endpoint. Imaging that you have a number of developers who, as a part of an application development cycle, would require multiple copies of your production environment (System Integration Testing, User Acceptance Testing…etc.) can easily be offloaded on to a vCloud Air platform without having to buy expensive kit locally to host multiple copies of your prod environment (we are talking additional SAN, Compute, Hypervisor & Networking costs here). Lets also imaging that they want to be able to use vRA so that they can self provision clones / copies of the production environment using pre-defined blueprints defined & published on the vRA IaaS catalog portal? You can quite easily make this happen and attach a vCloud Air endpoint, create a resource reservation on that endpoint and associate that with the business group that the developers belong to and create vCloud (vApp) type blueprints on vRA so that everytime a developer want to create a copy of that SQL server with 2 x App and 2 x Web servers to test a new application, they go to the vRA catalog, request those be provisioned and the servers will automatically be created on the mapped vCloud Air platform. (You can create a single Multi-Machine blueprint to group all of those individual server blueprints too which we’ll cover later)

Ok, enough of what we can do with vRA and vCloud Air and how cool that is… Lets look at what it takes to integrate the vCloud Air subscription you have to vRA and create and publish a vCloud blueprint & provision a VM on cloud that way.

Given below are the steps involved

  1. Create a vCloud Air (vCloud Director) endpoint
    1. Note: If you can remember what we covered in a previous post here, Infrastructure Admins usually create the endpoints within vRA. So login to the vRA portal using as the infrastructure admin (if you are using the default tenant, the URL is “https://<FQDN of the vRA Appliance>/shell-ui-app”. If you have a tenant specified, it’ll be https://<FQDN of the vRA Appliance>/shell-ui-app/org/<TenantName>”. I’m using a tenant called Tenant1 in my example within vRA)
    2. Go to Infrastructure->Endpoints->Credentials and set up credentials to access the vCloud Air endpoint – this is the same username & password you use to login to the vCloud Air online portal that you should have been given / created during the vCloud Air onboarding process (first thing that happens once you’ve signed up)  01
    3. Go to Infrastructure->Endpoints-> and create a new vApp (vCloud) type endpoint (this is the same as if you were creating an endpoint to a local vCloud Director instance)   02
    4. Once the endpoint is created, hover the mouse over the endpoint name and select the data collection and start the collection. You need to wait for this to complete first.
  2. Create a new Fabric group (Infrastructure Admin)
    1. Go to Infrastructure->Groups->Fabric groups and create a new Fabric Group (or you can use an existing fabric group and map the vCloud Air endpoint to it. 1
  3. Create a reservation for the vCloud Air endpoint (Fabric Admin)
    1. Note: Creating a reservation maps a logical portion of the vCloud Air endpoint to the business groups you have. I’m using an existing business group but if you need to create a new business group, do that first and select that business group during the reservation creation here.
    2. Go to Infrastructure->Reservation and create a new cloud reservation of type vApp (vCloud), as Fabric Admin user, selecting the mapped endpoint and the business group 2.1
    3. Go to the Resources tab and select a memory portion and storage portion to be used for this reservation 2.2Reservation-Resources
    4. Go to the Network tab and select the network you want to map to the reservation. Networks available here depends on the networks you’ve created within your vCloud Air portal. By default, you’ll have 2 networks, the default-isolated (private network) and default-routed (network with external connectivity) – Note here that at some point in the future, VMware will roll out NSX on the vCloud Air platform and once that’s complete, you’d also be able to create the logical networking via the same vRA / vCO blueprint too. This is going to be really cool and I don’t think any other public cloud vendor will have this capability for a while. If you have a network profile with static IP’s configured, select that network profile here which will allocate an IP to the VM from the network profile (which we covered in a previous post of the series). I’m not using a one here. 2.3
  4. Create & Publish vApp Component Blueprint (Tenant Admin)
    1. Note: When creating vCloud Air blueprints, its a 2 step process whereby you need to create a vApp Component blueprint first for each VM and then create a higher level master (group) vApp blueprint which will contain 1 or more of the lower level vApp Component blueprints. This is because on vCD (vCloud Director), every VM is placed inside a vApp so you need to create both through the vRA. But when you ultimately create the service & publish it with entitlements to the users, you only need to publish the master vApp blueprint.
    2. Login as tenant admin & go to Infrastructure->Blueprints and create a new cloud blueprint of type vApp Component (vCloud). Provide a name and select the Machine prefix 3.1vApp Component blueprint - info
    3. Go to build information tab and select the cloning action and select the template (you can select from a list of VM templates available within vCloud Air here provided that the data collection from the endpoint has been successful. You have a default set of global templates VMware provides (include CentOS, Ubuntu, Major Windows flavours with SQL) or if you’ve migrated some of your local templates you’ve created, that is specific to your environment (i.e. a Standard server build template from your local vSphere cluster which you can do using vCloud Connector to the vCloud Air portal), they too would appear here. And select the machine resources appropriate. 3.2vApp Component blueprint - Build info
    4. Add any custom properties in the next tab  and click OK.  3.3 vApp Component Blueprint - Properties
    5. Once the vApp Component blueprint is created, don’t forget to publish it (hover the mouse over the blueprint and click publish).  3.4vApp Component blueprint - Publish
  5. Create & Publish a vApp Blueprint (Tenant Admin)
    1. Note: now its the time to create the master vApp blueprint (which, as I explained above, is going to include the component blueprint and which will be published to users)
    2. Create a new cloud blueprint of type vApp (vCloud) and provide the information. Select the same reservation as used for the vApp component blueprint. 4.1vApp blueprint - Build info
    3. Go to the build information tab and select the clone action, and the clone from template should be the same as what you’ve chosen for the component blueprint. Then, nder the components, select the previously created component blueprint to link the child to the parent. 4.2vApp blueprint - Build info
    4. Once completed, don’t forget to publish this one too. 4.3vApp blueprint - Publish
    5. Create a Service to list the blueprint within the catalog (Tenant Admin)
      1. Go to Administration->Catalog Management->Services and add a service and provide all the information required including an icon, owner & support group details. 5.1 Service
      2. Select the service create and click on manage Catalog Items and add the vApp blueprint. Make sure you don’t add the vApp component blueprint here. 5.2 Service Catalog items
    6. Create Entitlements (Tenant Admin)
      1. Go to Entitlements and add a new entitlement and set the status to active. Also select the users / groups (from the business group) that this blueprint is entitled to. 6.1 Entitlements
      2. Go to the Items & Approvals tab and select the created service under entitled services & the same vApp blueprint under the catalog items and all relevant user actions. 6.2 Entitlements - items & Approvals

 

That’s it. You’ve now successfully created a public cloud endpoint within your on-premise vRA, and created and published a VM blueprint that can be used to deploy VM’s on the cloud automatically by your users.

If you now login to the same vRA URL as a valid user who were given the appropriate entitlements above, you’ll see the new blueprint item being available.

7. Catalog items

If you go ahead and request a VM using this cloud blueprint, the request status would be shown under the requests tab 8 Provisioning request on vCloud Air

If you now look directly at the vCloud Air online management portal, you’ll see the VM is being provisioned automatically. Once its complete, you’ll notice the owners name changes.9. Being provisioned in vCloud Air portal automatically 10 Provisoning complete

Once the VM is successfully provisioned in the cloud, the user will also see the status of that within the on-premise vRA portal which they can either access through vRA (console access) or though the vCloud Air online management portal directly (provided that they have a valid user account to login with – note that this account is separate. 11 Item now available on vRA

There you have it. VMware vRA can be a single point of automation and orchestration engine to automate and orchestrate various tasks, machine / VM provisioning on-premise as well as VM provisioning on the cloud. And this shows how vRA can be a key part of what I believe to be the true hybrid cloud infrastructure where you can place workloads on-premise or off premise based on your needs.

If your on-premise vRO is integrated with vCloud Air also, you can create further customisation workflows within vRO and publish them on vRA as an advanced service blueprint too (I will cover that in a future post)

Cheers

Chan

Next: (Optional) – vRA Part 9 – Extensibility – Custom Properties & Build Profiles & Property Dictionary –>

VMware vRealize Automation Part 7 – Tenant Administrator & Basic Blueprints

Chan 1 Comment

 

Now that we’ve got all the system-wide roles (System / Infrastructure & Fabric Admin) fully configured and each user has done their configuration tasks, next in the to-do-list is the configuration items to be performed by the Tenant Administrator. We wont be looking at everything a Tenant Admin can do here, but only focus on key items relating to blueprints.

First of all, lets briefly look at the Tenant Administrator role (note that the System Administrator assigned the Tenant Administrator role during a previous stage mentioned here)

Tenant Administrator is a key user that is typically a business manager or an IT administrator who is responsible for a tenant who configure vRA according to the needs of the company. He / She is responsible for the followings primarily,

  • Manages and configures the tenant
  • Manages the users and groups (within the tenant)
  • Manages catalog services
  • Creates approval policies & entitlements
  • Manage tenant brandings
  • Manage business groups within the tenant
  • Tracking resource usage by all tenant users
  • Create & manage global (shared) blueprints

Key Tenant Admin tasks

  1. Tenant Configuration Tasks
    1. Configure Identity Stores – This should have already been done at an earlier stage by the System Admin
    2. Configure custom groups
      1. Configure Custom groups (approval Administrator / Release Dashboard user / Release engineer / Release manager & Service architect roles
        and map them to AD users / groups   121
    3. Configure Identity store users such as business group admins, business group users and support users and configure additional privileges to these users (Administration->Users&Groups->Identity Store Users & Groups)
    4. Configure Tenant branding if this is required (Administration->Branding)
    5. Configure notification providers such as email servers to be used within the Tenant for the approval notifications via Administration->Notification->Email servers (I’m not doing this here)
    6. Create Business groups (We’ve already created a business group earlier as Tenant Admin user)
  2. Create & Publish IaaS blueprints
    1. Create an IaaS blueprint
      1. Notes:
        1. We’ll create a simple VM provisioning blueprint here using a CentOS 6.6 template & a customization specification (both need to have been pre-created and be registered in the compute / production cluster vCenter server)
        2. We will be using a basic cloning as the provisioning method (linked cloning and NetApp FlexClone options are also available)
      2. Go to Infrastructure->Blueprints and use new blueprint button to create a new virtual vSphere blueprint and fill out the basic Blueprint information 212
      3. Go to the build information tab and select the appropriate options (as below) 213
        1. You can use custom properties to achieve various functions. A full list of available all custom properties are available here and a list of specific custom properties applicable for a cloning blueprint (similar to what we are creating here) are available here.
        2. Some key, useful custom properties are as follows
          1.  Snapshot.Policy.Limit = <Depth of Snapshot limit. Default is 1, max is 31>
          2. Snapshot.Policy.AgeLimit = <Snapshot age limit in days. Default is no limit>
          3. VirtualMachine.Admin.ThinProvision = True/False (Applicable for VMware & Hyper-V using local or iSCSI storage)
          4. VMware.VirtualCenter.Folder = <Folder to place the VMs within vCenter>
          5. VirtualMachine.Admin.Owner=
          6. VirtualMachine.Admin.AddOwnerToAdmins=True
          7. VirtualMachine.Admin.AllowLogin=True
          8. VirtualMachine.Network0.Address=<IP address for Network 0>
          9. VirtualMachine.NetworkN.MacAddressType=generated/static
          10. VirtualMachine.NetworkN.MacAddress=<MAC Address>
          11. VirtualMachine.NetworkN.Name=<VMNetworkName>
          12. Machine.SSH=True
      4. We will cover the Build profiles and custom properties section separately 214
      5. Select the Actions that would be made available within the Blueprint definition (so they’d be available as entitlement actions to assigned users)   215
      6. Now we have the first basic blueprint definition created successfully.
    2. Publish the IaaS blueprint
      1. Hover the mouse over the created blueprint and select publish & click Ok to confirm. 221
    3. Create a service
      1. In order for a published blueprint to be made available on a catalog to the users, a service need to be created and the blueprint must be associated with that service
      2. Go to Administration->Catalog management -> Services and add a new service, populate the information required and add.   232
      3. Associate the blueprint with the service by selecting the service created and click on “manage catalog items” 233
      4. Add a catalog item and select the previously created blueprint 234
    4. You can create approval policies if required next(Administration->Approval Policies). I’m opting to not create approval policies and instead opt for no approvals.
    5. Create Entitlements
      1. Once a blueprint is created, and has been associated within a service, it (the service) need to be given entitlements. In other words, the service needs to be mapped to a user / group to be given access so that they can request a VM to be provisioned from this service / blueprint.
      2. Go to Administration->Catalog management->Entitlements and add a new entitlement. Provide a name for the entitlement (note: To avoid confusion, if the relationship between a blueprint:service:entilement is a 1:1:1, I keep the name identical for all 3 to keep thins simple and related) and select the AD groups / users that this blueprint / service is made available for within the online catalog. Click next 252
      3. Add the entitled services (created above), Catalog item (blueprint created above) and the appropriate entitled actions.
        1. Note: if the VM actions are not fully populated within the entitlements section, on the IaaS server host, navigate to “C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\Café” folder from command line and run “Vcac-Config.exe” registercatalogtypes –v”253
      4. Once added, the entitlements are complete 254
  3. User request through the catalog
    1. All tasks required to make the blueprint listed on an IaaS catalog are now complete and the entitled users can now request a machine provisioning based on this blueprint through the portal
    2. Login to the vRA portal (URL “https://<FQDN of the vRA Appliance>/shell-ui-app” if using the default tenant) using the appropriate business group user (bg-user in our example based on the entitlements we set above) and you’ll see the CentOS 6.6 server is now available in the users catalog.    32
    3. Request a VM to be provisioned off of this catalog item.    33
    4. All submitted requests can be tracked within the requests tab. 34
    5. You can see the cloning task initiated by the vRA service account within vSphere client / web client. Note that there may be a slight delay before this is executed on the vCenter after being submitted via vRA. Note that the default custom properties set for the business group (explained in the previous post of this series) dictated where the VM is created within the vCenter folder hierarchy & the name of the VM is automatically based on the default machine prefix attached to the business group (also created earlier) 35
    6. Once successfully created, the item will appear under the “Items” tab within the vRA IaaS portal for the user to access based on the entitled actions previously defined.   36

That’s basically it. This effectively now concludes the basic deployment tasks I’ve intended to cover as the typical deployment & configuration tasks you’d have to do in the field to deploy vRA 6.2.1 along with vRO 6.0.1 and NSX 6.1.x all fully integrated for extensibility. Hopefully this series of posts gave you a summarized view of all the initial tasks that you’d need to do, in the exact order they need to be done, including any undocumented little quarks I’ve come across that potentially could have cost you some troubleshooting time in the field. Hope this was of use to some. I know I will refer back to this in the future for my own benefit.

I would appreciate any feedback

Cheers

Chan

 

P.S. While this post concludes the series of posts formally, I will continue to provide additional vRA articles for further extensibility where we will look at more extensibility options to leverage vCO workflows during the provisioning stage to achieve various results and creating custom actions, advanced service blueprints, using vRA for publishing non IaaS functions such as AD password resets…etc. However they are not really mandatory for a typical vRA deployment and would only be required depending on the end user’s needs.

 

Next: (Optional Fix) Missing Catalog / Entitlement Actions on vRA 6.2.x –>

VMware NSX Upgrade from 6.1.2 to 6.1.3

Chan Leave a comment

OK, I’ve had NSX 6.1.2 deployed in my lab, simulating a complex enterprise environment and the DLR (Distributed Logical Routers) kept on causing numerous issues (one of which I’ve blogged about in a previous article here). Anyways, VMware support has requested that I upgrade to the latest NSX version 6.1.3 which is supposed to have fixed most of the issues I’ve been encountering so I’ve decided to upgrade my environment and thought I’d write a blog about it too, especially since there aren’t very many precise NSX related documentation out there.

Note that despite the title of this article (upgrading 6.1.2 to 6.1.3), these steps are also applicable to upgrading from any NSX 6.0.x version to 6.1.x version.

High level steps involved

  1. First thing to do is to check for compatibility and I have vSphere 5.5 which is fully supported with NSX 6.1.3 as per the compatibility Matrix here.
  2. Then have a look at the 6.1.3 release notes to make sure you are up to date with what’s fixed (not all the fixes are listed here btw), what are the known issues…etc. That is available right here.
  3. Next thing is to download the upgrade media from the NSX / Nicira portal. If you have an active NSX subscription, you may download it from My VMware portal or else, (if you have taken the course and were deemed to be suitable to have access to the eval, you can download it from Nicira portal here. Once you’ve downloaded the Nicira upgrade media (VMware-NSX-Manager-upgrade-bundle-6.1.3-2591148.tar.gz file), you may also download the install & upgrade guide here.(Note that there don’t appear to be a separate set of documentation for 6.1.3 as it appears to be for the whole 6.1.x platform).
  4. If you are upgrading vSphere to v6.0 also at the same time, follow the steps below
    1. Upgrade NSX Manager and all NSX components to 6.1.3 in the following order.
      1. NSX Manager
      2. NSX controller(s)
      3. Clusters and Logical Switches
      4. NSX Edge
      5. Guest Introspection and Data Security
    2. Upgrade vCenter Server to 6.0.
    3. Upgrade ESXi to 6.0. – When the hosts boot up, NSX Manager pushes the ESXi 6.0 VIBs to the hosts. When the hosts show reboot required on the Hosts and Clusters tab in the left hand side of the vSphere Web Client, reboot the hosts a second time.  NSX VIBs for ESXi 6.0 are now enabled.
  5. If you are not upgrading vSphere at the same time (this is the case for me), follow the step below
    1. Upgrade NSX Manager and all NSX components to 6.1.3 in the following order.
      1. NSX Manager
      2. NSX controller(s)
      3. Clusters and Logical Switches
      4. NSX Edge
      5. Guest Introspection and Data Security

Now lets look at the actual upgrade process of NSX

NSX Upgrade steps

  1. NSX Manager Upgrade
    1. I don’t have Data security deployed. If you did, uninstall it first
    2. Snapshot the NSX Manager virtual appliance (as a precaution)
    3. Login to the NSX manager interface and go to the upgrade tab. Click on the upgrade button on top right and select the previously downloaded VMware-NSX-Manager-upgrade-bundle-6.1.3-2591148.tar.gz file & continue.          1.3
    4. Select to enable SSH and click upgrade.                                                           1.4
    5. Upgrade of the NSX Manager appliance will now start.       1.5
    6. Once the upgrade completes, the appliance will automatically reboot (this is not so obvious unless you look at the console of the appliance). Once rebooted, login to the NSX manager instance and confirm the version details (Top right).    1.6
  2. NSX controller(s) Upgrade
    1. Notes
      1. Upgrades are done at cluster level and would appear with an upgrade link in NSX manager.
      2. It’s recommended that no new VM’s are provisioned, no VMs are moved (manually or via DRS) during the controller upgrade
    2. Backup controller data by taking a snapshot
      1. If the existing NSX version is < 6.1, This has to be done using a REST API call
      2. If the existing NSX version is => 6.1, you can do this using the GUI of the web client as follows 2.2.2
    3. Login to the Web client, and go to the Networking & Security -> Installation and click on upgrade available link as shown below 2.3
    4. You’ll now see the upgrade being in progress (may ask you to reload the web client). Controller(s) will reboot during the upgrade 2.4
    5. The upgrade should complete within 30 mins normally. If not completed by then, upgrade may need to be launched using the upgrade available link again. Once successful, the upgrade status should state upgraded and the software version should be updated with the new build number. Note that if the upgrade status says failed, be sure to refresh the web client prior to assuming that its actually failed. It may state its failed due to the 30 standard timeout even though the controller upgrade has completed successfully (happened to me couple of times) 2.5
  3. Clusters and Logical Switches
    1. Go to the host preparation tab and click update                           3.1
    2. Installation and upgrade guide states that each host will automatically be put in to the maintenance mode & rebooted. If the upgrade failed with a Not ready status for each node, you may need to click resolve each time to retry (happened o me once) and it will proceed. Some host may require manual evacuation of VM’s too. Also worth noting that if you keep getting the task “DRS recommends hosts to evacuate” logged in vSphere client, you may want to temporarily disable HA so that the ESXi host can be put in maintenance mode by the NSX cluster installer (Arguably, in a well managed cluster, this should not happen).    3.2
    3. When the cluster is updated, the Installation Status column displays the software version that you have updated to.    3.3
  4. NSX Edge (Including the Distributed Logic Router instances)
    1. Go to the NSX Edges section and for each instance, select the Upgrade Versions option from the Actions menu.   4.1
    2. once the upgrade is complete, it will show the new version of the Edge device, weather that’s a DLR or an Edge gateway device.   4.2
  5. Guest Introspection and Data Security
    1. Upgrading these components are relatively straight forward and I’d encourage you to look at the install & upgrade guide…
  6. Now, create a NSX backup (You may already have routine backup scheduled on the NSX manager)
  7. If all is well, remove the snapshot created earlier, on the NSX manager appliance.

 

All in all, it was a fairly straight forward upgrade but the time it took due to somewhat finicky nature of some of the components (especially the cluster upgrade where a number of manual retries were required) unnecessarily added time. Other than that, all appears to be well in my NSX deployment.

Cheers

Chan

VMware vRealize Automation Part 6 – System / Infrastructure & Fabric Administrators

Chan 1 Comment

There are various types of Access Roles within vRA, each with different permissions that exist within vRA (more roles introduced through the IaaS server). Some are System-Wide, some are Tenant-Wide and some are Business Group wide roles. First part of this post, we’ll briefly cover all the different roles available within vRA, the order they are created and each roles remit followed up by a brief look at different architectural concepts available within vRA IaaS service to understand how the resources are mapped to users. The next part of the post will then look at the 3 system wide roles (System Admin / Infrastructure Admin / Fabric Admin) and what need to be set up under each user role during the initial deployment and in what order. Lets take a look at the roles at first. There are multiple roles each with a certain set of privileges within vRA. The number and the types of roles can get very confusing and often, I’ve not been able to remember every role and all of their abilities properly with the order of when and how they need to be created. So, I thought I’d document it all in one place as follows. Roles

Now the various user roles available within vRA and what they do and what order you create them is clear, one may wonder what the heck are those endpoints, business groups, fabric groups…etc. A fair  question, and the user roles discussed above may not fully be understood until these essential vRA components & concepts are also understood. A typical vRA environment provides a complete multi – tenancy deployment capability so that an ISP for example, can deploy vRA to automate IaaS service provisioning to a number of its customers, each of which may have a dedicated, separate tenant, associated with each client’s own directory service system, all hosted securely on the same vRA platform. And then, vRA uses a number of specific internal constructs to map external resources (such as vCenter servers with compute clusters, vCloud Director instances with Org VDCs…etc) to a set of business users in order to grant them role based access to those resources. This require understanding of those constructs and in what order they are created which, I’ve found to be pretty damn convoluted when I first started off (to be honest, I still tends to forget the order if I stay away from working with it for a while). So, I’ve thought to summarise all the key constructs used by vRA, and how they are used to map external resources and the order of setting those up, using a simple example below. Architecture Example

System-Wide Roles & Initial Configuration

Ok, now that we’ve established a reasonable understanding of the vRA IaaS platform architecture with regards to Multi tenancy and the types of users & groups involved, lets look at how to setup things, in the exact order they need to be done, following on from the previous posts. (We configured up to creating the Tenants and specified the Tenant Administrators within the post 3) Once you’ve completed the IaaS server components specified within the post 4,  you can carry on with the following tasks, in the specified order to complete the IaaS configuration. For the purpose of the IaaS configuration from here on, we are assuming that we are configuring the Default Tenant (vSphere.local). If you decide to create additional tenants, they can also be configured the same way, however the URL you need to login to non default Tenants would be “https://<FQDN of the vRA Appliance>/shell-ui-app/org/<Tenant Name>”.

    1. System Administrator – Setup Infrastructure Administrators for the IaaS platform
      1. Login with the System Administrator privileges (Administrator@vsphere.local) to vRA UI for the default tenant (URL “https://<FQDN of the vRA Appliance>/shell-ui-app”). Note that the Infrastructure Administrators section inside the Default Tenant is only enabled because we’ve deployed the IaaS server component on top of the vRA appliance.5.2
      2. Specify the Active Directory Group to be given the Infrastructure Administrator privileges here.  (I have an Active Directory group I will be using here named “VMware vCAC Inf Admins – Default” & a specific user as a member of that group called “<Domain>\inf-admin” which will inherit this permission through this group. 6.1.2
    2. Infrastructure Administrator
      1. Login as the newly setup Infrastructure Admin to the default tenant on vRA (URL “https://<FQDN of the vRA Appliance>/shell-ui-app”) and follow the Goal Navigator (Click on the >> on the top left). Else, the steps are as follows
      2. License the vRA IaaS component (Administration->Licensing) – same vRA license code as used in initial vRA appliance configuration as described here.
      3. Setup Endpoints & Credentials
        1. Note: Its best practise to always create the endpoint first here before installing the agents (in this example, the agent has already been installed during the IaaS server deployment)
        2. vCenter Endpoint
          1. Create the vCenter credentials. I’m using the vRA service account (domain\svc_vRA) which I’ve permissioned as a vCenter Administrator here. 6.2.3.1.1 new
          2. Add a vCenter endpoint using Infrastructure->Endpoints->New Endpoint->Virtual->vSphere (vCenter URL is https://vCenter FQWDN/sdk). Note that the endpoint name here for vCenter should match the endpoint name defined during the IaaS server vSphere agent component deployment (covered in the post 4 of this series. Screenshot is here). If the 2 names are different, vCenter endpoint will not connect.   6.2.3.1.2 new
        3. vRO Endpoint Setup for IaaS functions
          1. Create the vRO credentials. I’m using the vRA service account (domain\svc_vRA) which I’ve also permissioned as a vRO Administrator user (when setting up the vRO in post 5, we mapped an AD administrative group as a vRO admin group of which the svc_vRA user is made a member of) 6.2.3.2.1
          2. Add a vRO endpoint using Infrastructure->Endpoints->New Endpoint->Orchestration->vCenter Orchestrator
            1. Use the vRO URL as https://vRO FQWDN:8281/vco
            2. Ensure to add a custom property VMware.VCenterOrchestrator.Priority with a value of 1 6.2.3.2.2.2
            3. Then go to the Infrastructure->Endpoints and hover the mouse over the vCO endpoint and select Data Collection. Start collection and verify success 6.2.3.2.2.3
        4. NSX Endpoint
          1. Note: before adding the NSX endpoint, the NSX plugin for vRO must have been configured correctly (we did this in the previous post of the series – step 9.3.1)
          2. Set the vRA service account (Domain\svc_vRA) with appropriate NSX privileges within the NSX manager via  the web client.   6.2.3.3.2
          3. Create NSX credentials by specifying the Domain\svc_vRA account (mapped as NSX Enterprise Administrator above)   6.2.3.3.3
          4. Go the configured vCenter endpoint, edit and select “Specify manager for network and security platform” checkbox and specify the NSX manager FQDN as the address and the NSX credentials specified above                         6.2.3.3.4
          5. We need to start a NSX data collection here and verify which we cannot do until the fabric administrator account is created and logged in which we will do later on below. (another weird design within vRA where Infrastructure admins can initiate data collection of vCloud Air and vRO endpoints but only Fabric Admin can do the same for vCenter & NSX endpoints….).
        5. Additional endpoints such as vCloud Air Endpoint will be covered in a separate post in the future.
      4. Create Fabric Groups & Fabric Administrators
        1. Go to Infrastructure->Groups->Fabric Groups and create a new Fabric Group
        2. Select an AD group as Fabric Administrators and the compute cluster to be mapped to this Fabric Group. (refer to the diagram above for resource mappings) – note that with previous version of vRA / vCAC, you were NOT able to add an AD group here (had to be a single AD user). Appears fixed in this version.6.2.4.1
        3. That’s it. You now have a fabric group with a fabric administrator group defined. Next step is to login as the Fabric Admin and complete his tasks.
    3. Fabric Administrators – Part 1
      1. Note: I have an AD user named Domain\fab-admin who is a member of the Fabric Administrators AD group used above and will be using this user account to login to vRA to perform with for all Fabric Administrators tasks.
      2. Verify data collection for the vSphere endpoint and the NSX endpoint
        1. Login to vRA as the Fabric Admin. (URL “https://<FQDN of the vRA Appliance>/shell-ui-app”)
        2. Go to Infrastructure->Compute resources and hover the mouse over the compute cluster and select data collection and ensure Compute/Inventory/State/performance / Networking & Security (NSX) data collections have succeeded. (I normally change the interval to 1 hour)   6.3.2.2
        3. If the NSX data collection is not complete, invoke a collection manually via request now.
        4. Once the NSX collection is successful, login to the vRO client (you can get the client using https://vRO FQDN:8281/vCO) and go to the Inventory tab
        5. Reload by pressing F5 and expand the NSX field on the left to confirm vRO is populated fully with the NSX configuration of your environment   6.3.2.5
        6. If you are planning to use NSX security policy features from vRealize Automation, an administrator must run the Enable security policy support for overlapping subnets workflow in vRealize Orchestrator. The security policy support for overlapping subnets workflow is applicable to an VMware NSX 6.1 and later endpoint. For 6.1, 6.1.1, and 6.1.2, you must run this workflow only once to enable the AppliedTo flag in the service composer. For VMware NSX 6.1.3 and later, you do not need to run the workflow because this support is enabled by default. – As my NSX version is 6.1.2, I’ve run this workflow as follows.   6.3.2.6
      3. Create the Machine prefixes
        1. Some points on Machine Prefixes
          1. Machine prefixes are mandatory  and are used to generate names of vRA provisioned machines.
          2. They can be shared across all Tenants
          3. Consist of a base name followed by a counter (i.e. Prod-XXX where XXX is a number will produce machine names such as Prod-001, prod-002…etc.
          4. When business groups are created by Tenant admins, you need to assign a default machine prefix to each.
          5. Every blueprint must have a machine prefix or share the default machine prefix of the business group
          6. Must conform to DNS naming restrictions (ASCII letters A-Z and digits 0-9 with no special characters).
          7. If used for a Windows machine, need to be < 15 characters in length.
        2. Go to Infrastructure->Blueprints->Machine prefixes and create a machine prefix.    6.3.3.2
    4. Tenant Administrators – Part 1
      1. Note: Next up in the list of to do things is for the Fabric Admin to create the reservations, reservations policies & network profiles. But thanks to somewhat less than ideal role allocation available within vRA (VMware: Hope you are listening??), you cannot do those steps without having the business groups created first which Fabric Admins cannot do. So we have to login as a Tenant Admin briefly here and create the business groups before we can resume with the rest of the fabric admin’s duties.
      2. Create Business Groups
        1. Important points on Business Groups
          1. A business group associates a set of services and resources to a set of users such as a line of business, department or other organizational unit.
          2. Reside within a Tenant
          3. Created by a Tenant Admin of that particular Tenant
          4. To request machines, a user MUST belong to an appropriate business group with the appropriate privileges (Business Group User)
        2. Login as Tenant Administrator (I have a dedicated tenant admin domain account named Domain\Tenant-Admin which is a member of the Tenant Admin AD group which we had earlier permissioned with the tenant Admin privileges). – The login URL is “https://<FQDN of the vRA Appliance>/shell-ui-app”
        3. Go to Infrastructure->Groups->Business Groups and create a new business group.
          1. AD container can be the location where machine objects would be created
          2. Group Manager / Support Role / User Role can all be AD groups (this wasn’t possible with earlier versions of vCAC) or an AD user account. I’ve used AD groups here as its neater and allows RBAC. I’ve then created individual user accounts (Domain\bg-admin, Domain\bg-support, Domain\bg-user) that have been nested in the relevant AD group to inherit permissions.
          3. Create Custom properties (if applicable)
            1. Custom properties are quote useful in manipulating where machine accounts gets created within vCenter, relevant snapshot policies…etc. We will look at Custom properties in detail a little later on but for now, will create few basic custom properties as shown below.   6.4.3.3
        4. That’s it. You now have a Business Group created that can be used within a reservation so we will now go back to finishing off fabric admins duties.   6.4.2.4
    5. Fabric Administrators – Part 2
      1. Create Reservations
        1. Key Points
          1. Reservation is a share of provisioning resources allocated by the Fabric Admin (from a Fabric group) to a particular business group to use.
          2. Each reservation is for a single business group
          3. Each machine or blade can only belong to a single reservation
        2. Login as the fabric administrator (URL is “https://<FQDN of the vRA Appliance>/shell-ui-app”)
        3. Go to Infrastructure->Reservations and create a new Reservation from the previously created Fabric Group to be used by the previously created Business Group.    6.5.1.3
        4. Select the Resources tab and allocated a portion of the available storage & Memory to be included within this reservation   6.5.1.4
        5. Select a VM Network by selecting the appropriate VM Network Port group Name within the Network Tab. (Note that NSX specific networking will be setup later)   6.5.1.5
      2. Create Reservation policies
        1. Key points
          1. Reservation Policy restricts a blueprint to a subset of the available resource & storage reservations (a Blueprint which is created at Tenant level can otherwise deploy to any reservation within that template)
        2. Go to Infrastructure->Reservations-> Reservation Policies and create a reservation policy. 6.5.2.2
        3. Now amend the existing reservation “prodclust01-Res-1-UAT” (created above) to include this reservation policy 6.5.2.3
      3. Create Network Profiles
        1. Key points
          1. Network profiles allow machines to be assigned static IP addresses
        2. Create a network profile by going to Infrastructure->Reservations->Network Profile and selecting a new External network profile 6.5.3.2
        3. Define the IP range so that each machine provisioned by vRA will have an IP assigned from this range 6.5.3.3
      4. Modify the reservation profile to map the Network profile
        1. Go the reservation created above and under the network tab, select the Network Profile created. 6.5.4.1
        2. Now, all VMs created in to this reservation will have an IP allocated using this IP range defined within the network profile.

 

There you go, we now have all the 3 main System-wide roles setup and their subsequent duties completed. Next up is the Tenant Administrators tasks which we’ll cover in a separate, dedicated post as the next logical step in our vRA deployment.

 

 Next: vRA Part 7 – Tenant Administrator & Basic Blueprints –>

VMware vRealize Automation Part 5 – vRO Deployment & Configuration & Integration

Chan 1 Comment

Next: vRA Part 6 – System / Infrastructure & Fabric Administrators –>

vRealize Orchestrator (was known as vCenter Orchestrator before) is an integral part of a vRA deployment as most of the vRA extensibility features relies on vRO being available and correctly integrated to vRA appliance & IaaS components. vRO in tern, connects to a number of other 3rd party systems through plugins, such as Microsoft Active Directory, Microsoft SQL, VMware vSphere, NSX….etc., where you can create a range of vRO workflows to carry out / automate tasks within these 3rd party systems. Due to the integration between vRA and vRO, these vRO workflows can be published within the vRA catalog and be made available via the built in self service portal (for example, you can publish a blueprint for resetting the passwords for an AD user through a workflow created on vRO).

I personally haven’t seen much documentation or clear guidelines with sufficient details on how to set up vRA with vRO for this level of extensibility which was one of the main reasons why I decided to publish this series of posts (which, hopefully will be addressed by VMware in the future).

So, now its the time to deploy the vRO and have it fully configured in preparation for the integration with vRA in due course. This article therefore looks at the deployment and the initial configuration of vRO component. (This should be considered a must have part of your vRA deployment)

vRO is available in many different forms

  • Available as a standalone appliance to be downloaded free of charge for vSphere customers
  • Automatically deployed if you install Windows version of vCenter (vRO Windows services are disabled by default though) – Note that this is now deprecated with the vRO 6.0.1 and hence SHOULD NOT be used going forward.
  • Available built in to the vRA itself (used for PaaS functions only, NOT used for IaaS functions for which you must register an external vRO)

In the interest of performance, and separation (and with the added ability of clustering should you need that in future), I prefer to deploy the standalone appliance (which is also the VMware’s recommended approach) so this article will rely on that. However, note that the post deployment configuration is still the same between the standalone appliance and the default version installed with vCenter.

The latest version of vRO released as of today, is the version 6.0.1, which was released in March 2015, in line with the announcement of vSphere 6.0. The release notes for the new version can be found here. It is compatible with vCenter server 4.1 and above.

Few highlights are below,

  • The stability of the vCenter Server plug-in has been improved by resolving major issues based on customer feedback (this was a real issue with earlier versions)
  • vRealize Orchestrator 6.0.1 supports the vSphere Web Client integration and context execution of vRealize Orchestrator workflows as part of vSphere Web Client 6.0
  • Switch Case introduced to handling conditional and branching operations within an automated workflow (If a value matches a switch case, specific workflow branch is executed)
  • Global error handling introduced
  • vRO control center beta introduced (Centralized admin interface for runtime execution, workflow monitoring & correlation with system resources, unified log access and configuration)

I will not be covering details about the Orchestrator itself, but only how to deploy & configure it within this post. If you need information about the vRO itself, including its architecture…etc. please refer to the official documentation found here.

Orchestrator Appliance Deployment

  1. Download the vRO appliance from My VMware portal as an ova and deploy on to the Management Cluster (NOT the compute / production cluster. Refer to the proposed architecture diagram in the previous post of this series here)
  2. During the deployment, you’d need to provide the information required and click next (ensure SSH is enabled) O3
  3. Once the appliance is deployed and powered on, SSH on to the appliance as root and change the root password expiry date.
    1. The password for the root account of the Orchestrator Appliance expires after 365 days. You can increase the expiry time for an account by logging in to the Orchestrator Appliance as root, and running passwd -x number_of_days name_of_account. o3.1
  4. Configure the appliance
    1. Go to the https://<vRO FQDN>:5480 and login using root.
    2. Change the Time Zone & Setup NTP (Admin->Time Settings)
    3. Update the Proxy settings should it be required and verify all other networking settings.

That’s it. the basic vRO appliance is now deployed and ready to be configured for vRO functionality which we’ll look at next.

 

vRO Configuration

  1. Login to the vRO appliance Configuration page 
    1. https://<vRO FQDN>:8283/vco-config (username: vmware)
  2. Bind the IP & DNS
    1. Click on Network on the left and select the IP address from the drop down menu to bind the vRO to the correct IP address.vRO2
  3. Import vCenter SSL Certificate
    1. Under Network, click on SSL Trust Manager tab and type import the vCenter server SSL certificate using the URL “https://<FQDN of the vCenter Server>” vRO3
    2. Note that this vCenter server should be the one managing the compute / production ESXi cluster, NOT the one managing the Management cluster.
  4. Setup Authentication Type
    1. Best practise recommendation is to configure the authentication source to be the vCenter SSO as this allows the user of vRO through vSphere web client. Note that the built in OpenLDAP is NOT recommended for production deployments.
    2. Import SSO server SSL certificate using the SSL Trust Manager
      1. Host: https://<FQDN of the SSO server>:7444 vRO4.2.1
    3. Setup vCenter SSO for authentication
      1. SSO Registration
        1. Select the Authentication mode as SSO Authentication within the Authentication section.
        2. Host: https://<FQDN of the SSO server>:7444
        3. Admin user name: Administrator@vsphere.local   vRO4.3.3
      2. SSO Configuration (Setup vRO admin group)
        1. Specify the SSO group that gets rights to the vRO server admin role here
        2. Select the SSO domain (I’m selecting the Active Directory attached to the SSO server)
        3. Select the dedicated Active Directory group (I’ve got a dedicated AD group created called “VMware vRO Administrators” which I’m going to add here.)   vRO 4.3.2.3
        4. Make sure the following accounts are members of this group on the AD
          1. vRA service account (<domain>\svc_vRA)
          2. vRO service account (<domain>\svc_vRO)
          3. Any domain user account requiring access to vRO workflows through vRO client.
        5. Note: One thing I’ve noticed here is that when changing this SSO Admin group (SSO or AD) always unregister the orchestrator and re-register with SSO. In the old version of vCO, it really didn’t like it when you just change this vRO admin group
  5. Assign vCenter license (default vRO license is only valid for 90 days)
    1. Under Licenses, select vCenter Server license
    2. Provide the vCenter address (<FQDN of the vCenter Server>) and an administrative user name to the vCenter (I’d create a new vRO service account, such as  <domain name>\svc_vRO, and assign vSphere administrative privileges to it and use it here as that’s neat and follow the RBAC guidelines) vRO5.2
    3. restart the vRO server service using the startup options
  6. Assign an external Microsoft SQL database
    1. The vRO 6.0.1 built in database is sufficient for small to medium scale deployments only and we will be using a separate SQL server as per VMware best practise.
    2. External database is also a pre-requisite for future clustering of the vRO.
    3. Prepare the SQL database
      1. Note: I’m going to use an external SQL 2008R2 server instance here.
      2. Verify that the protocol and port access is setup correctly on the SQL server vRO6.3.2
      3. Create an empty database on the SQL server (vRO601) vRO6.3.3.1
      4. Create a SQL user account (svc_vRO) and map this to the vRO601 database with dbo permission
        1. Note: With using the vRO Linux based appliance, I’ve had weird errors here and trying to get both Windows authenticationand SQL authentication work here. To cut a long story short, I only managed to get it to work after adding a Windows domain account (froot\svc_vRO) and a SQL account (svc_vRO), both with the same name added as DBO to the database. (makes no sense). This account will only be used by vRO to connect to its database. Ensure that SQL password expiry is ticked off when creating the sql account to avoid future problems. vRO6.3.4
    4. Import the SQL server SSL certificate to vRO using Trust Manager (Only applicable if the DB server uses SSL certs. Mine doesn’t so I will skip this)
    5. Setup the SQL server database connectivity
      1. Provide the SQL database connectivity information required and click apply changes vRO6.5.1
      2. Once prompted, create the database tables using the link provided. Once completed successfully, the status will be displayed. Click Apply Changes vRO6.5.2
  7. Assign a SSL Certificate to vRO
    1. Either create a self signed server certificate or import a certificate database (using an existing CA) vRO7.1.1
    2. Export the certificate DB for DR                                        vRO7.1.2
  8. Configure Orchestrator Plugins (Using vRO configuration interface)
    1. Configuration of Active Directory, SSH & vCenter Server plugins (all come installed by default in vRO) are now done through pre-defined workflows available within vCO rather than through the configuration interface
    2. Type the vRO admin username and password within the plugin section (When the Orchestrator server starts, the system uses these credentials to set up the plug-ins. The system checks the enabled plug-ins and performs any necessary internal installations such as package import, policy run, script launch, and so on) – I’m using the vRO domain service account (<domain>\svc_vRO) which is also a member of the vRO administrators group I’ve mapped under Authentication.
  9. Install new Orchestrator plugins
    1. Various addition vRO plugins can be downloaded from the vRO official web page (here) and installed on to the vRO via the configuration page so that vRO workflows can be created to integrate and manipulate external systems.
    2. Use the Install Application tab within the General section of the vRO configurator to install plugins that have a .vmoapp extension.
    3. There are a number of additional plugins available here but I will be installing the followings, for the purpose of this article
      1. Install the plugin VMware vCenter Orchestrator Plugin for NSX 1.0.1 – Required for vRO workflow creation for NSX in order to orchestrate some NSX tasks
        1. Compatibility: Based on the VMware product Interoperability Matrix as of now, for vRO 6.0.1, NSX version 6.1, 6.1.1 & 6.1.2 are the only versions of NSX supported and that requires the vRO NSX plugin version 1.0.1. But the vcoteam.info (official site for vRO team) site’s plugin list only contains the NSX 1.0.0 plugin and not version 1.0.1. So be sure to download the correct version via the link I’ve provided above.
        2. Note: This plugin is required for vRO workflow creation for NSX in order to orchestrate some NSX tasks. The current workflows included within this package are limited however and the installation of the NSX-v Dynamic Types plugin V2 (below) will add additional NSX related workflows. Once these workflows are available, they can be leveraged by vRA.
        3. Install the NSX plugin (using o11nplugin-nsx-1.0.1.vmoapp file) vRO9.2.1.2
        4. Accept license & Apply changes                                            vRO9.2.1.3
        5. Restart vRO server to take effect (note that the plugin status may not change to “installed” immediately after the vCO service restart. This can take some time)   vRO9.2.1.4
        6. Login to the vRO client and verify the NSX workflows are now available within Library->NSX folder vRO9.2.1.5
      2. Install the plugin NSX-v Dynamic Types plug-in V2
        1. Note: A handy NSX Dynamic Types plugin to use with vRA Advanced Service Designer. This plug-in can be customized and extended using the plug-in generator V2 package.
        2.  Install instructions are all available in the link above (summary below)
          1. Rename the downloaded package com.vmware.coe.plugin.dynamictypes.nsx.v2.package
          2. Import the package (Using vRO client, NOT the configuration page)
          3. Verify that the package and the additional NSX workflows now appear within the vRO client. vRO 9.2.3
          4. More steps are required to be carried out to complete this which will be done under step 10 below
      3. Install the plugin VMware vRealize Orchestrator vRA Plug-in 6.2.0
        1. Note: This is required for integration with vRA 6.2.
        2. Install the vRA 6.2.x plugin using the o11nplugin-vcac-6.2.0-2287231.vmoapp on the vRO configuration page vRO9.3.1
        3. Verify the installation and restart the vCO service (note that the plugin status may not change to “installed” immediately after the vCO service restart. This can take some time)                                                                                                                                                            vRO9.3.2
        4. Note: Login to the vRO client and ensure all default workflows are visible within the client. (I’ve had repeated issues whereby, after the vRA 6.2 plugin has been installed, all default plugins and their workflows disappeared within vCO. If this happens, within the vRO configuration page, go to Troubleshooting section and click “Reset current version” under “Reinstall the plug-ins when the server starts and reboot the vRO appliance) – This can potentially save you lots of time….!!
  10. Configure Orchestrator Plugins (using vRO workflows) – We will configure the key plugins here and leave some to be configured later
    1. Configure vCenter plugin
      1. Note: The way to integrate vCenter server may differ depending on whether you have vCenter 5.5 or 6.0 as well as whether its a windows install or the vCSA. In my case, its vCenter 5.5 on good old Windows so the below steps are based on that configuration.
      2. Login to the vRO client (you can download the client installable from https://<vRO FQDN>:8281 or launch the client directly from there)  and run the workflow Library-> vCenter->Configuration->Add a vCenter server instance   10.1.2
      3. Now, select whether you are happy to use a shared session for vRO to connect to vCenter server. If you using a shared session (meaning, you’ve answered “No”), provide the credentials that the vRO uses to connect to the vCenter with (shared credentials for all users). I’m using the vRO service account (svc_vRO) which I’ve made an Administrators on the vCenter instance. This is usually good enough as I can see all vRO driven activities on the vCenter logs / events clearly thanks to the service account name.  Also, since the ultimate goal is to call these workflows via vRA portal, and when I specify blueprints based on these workflows, I’m already controlling who has access to access those blueprints, so the shared credentials between vRO and vCenter is the best option. If however you decides to not use shared credentials, you need ensure each user who access vCO workflows though vRA catalog, will have sufficient permission on vCenter to execute that specific workload and I don’t think that is a good way to go about it, architecturally.     10.1.3
      4. Once the workflow has been successfully run, you’ll see the message on the screen as follows. 10.1.4
    2. Register vCenter Orchestrator as a vCenter Server Extension
      1. Note: This important so that vRO workflows will automatically be available within the vSphere Web Client to execute.
      2. Login to vRO client and rung the workflow: Library -> vCenter -> Configuration -> Register vCenter Orchestrator as a vCenter Server Extension
      3. Select the vCenter server (managing compute & edge cluster that the above workflow was run against) here. 10.2.2
      4. Verify it ran successfully
      5. Verify on the Managed Object Browser for that vCenter that the vRO extension is listed correctly. Instructions are as follows
        1. In a Web browser navigate to the managed object browser of your vCenter Server instance. https://<vcenter_server_ip?/mob
        2. Log in with vCenter Server credentials.
        3. Under Properties, click content.
        4. On the Data Object Type: ServiceContent page, under Properties, click ExtensionManager.
        5. On the Managed Object Type page, under Properties, click the Orchestrator extension string.extensionList[“com.vmware.vco”]. The extension has a server property which contains an array of type ExtensionServerInfo. The array should contain an instance of the ExtensionServerInfo type with a url property which contains the URL of the registered Orchestrator server.
        6. On the Data Object Type: Extension page, under Properties, click server. You can see information about the Orchestrator server registered as an extension, such as serverThumbprint and url. The serverThumbprint property is the SHA-1 thumbprint of the Orchestrator server certificate, which is a unique identifier of the Orchestrator server. The url property is the service URL of the Orchestrator server. There is one record per IP address. If the Orchestrator server has two IP addresses, both of them are displayed as service URLs.        10.2.4.6
    3. Configure the Active Directory Plugin
      1. Login to the vRO client and run the workflow: Library->Microsoft->Configuration->Configure Active Directory Server
      2. Provide the answers including the AD dc name (I’d providing the root of the domain address which is automatically directed to any of the DCs in the domain rather than pointing at a single DC.                                      AD 10.3.1
      3.      Now select the authentication method to the AD as a shared session. If a shared session is not used, the user who runs the vRO workflow is expected to have sufficient rights in the AD to run the particular workflow which WILL NOT be practical once the vRO is integrated with vRA and users start requesting machine provisioning from blueprints which in turn, calls for vRO workflows. AD-1
    4. Configure VMware vRealize Orchestrator vRA Plug-in 6.2.0 – This step is key….!!
      1. Add vRA Host
        1. Import the vRA server SSL certificate
          1. Run the workflow “Import a certification from URL” under library->configuration->SSL Trust Manager
          2. Provide the vRA appliance details in the URL & Submit. Verify Its run correctly. vRA 10.4.1.1.2
        2. Add the vRA host
          1. Run “Add a CAC host” workflow under vCAC->Configuration and select the vRA server FQDN and the URL (with https://) vRA 10.4.1.2.1
          2. Use the shared session option using the Administrator@vsphere.local account to connect to the vRA server vRA 10.4.1.2.2
          3. Verify its run successfully at the end.
      2. Add the vRA IaaS host
        1. Setup Infrastructure Administrators Group on the default tenant
          1. Login to the vRA UI using https://<vRA Appliance FQDN>/shell-ui-app using the Administrator@vsphere.local account IaaS 0
          2. Go to the Administratprs tab within the vSphere.local tenant (default) and add an appropriate AD group for vRA Infrastructure Administrators (I have a group created here called <domain>\VMware vCAC Inf Admins and a user account named <domain>\inf-admin as a member of that group) IaaS 0 1
        2. Run “Add a vCAC IAAS host” workflow
          1. Run the workflow “Add a vCAC IAAS host” under vCAC->Configuration and select the vRA server (already added to vRO) IaaS 1
          2. Host name is are auto populated. Change this to be the name of your IaaS windows server (as shown below) and Click next.                        IaaS 2
          3. My preference is to use the shared session option here. The shared user account used to connect to the IaaS windows server needs to be a administrative account that has local administrative rights to the IaaS server where the model manager database is installed. As such, I’m going to use the vRA service account (domain service account that was created and used to install the IaaS services with during an earlier step here which had admin rights to the IaaS server).  DO NOT type the domain name here. Just type the account name as the domain is provided in the next step.      IaaS 3
          4. Change the domain name to the appropriate domain name and submit.   IaaS 2.5
          5. Verify it runs successfully.
      3. Install vRO Stub Workflow Customization
        1. Note: This workflow will modify the default Workflow Stubs used by vRA so that vRO workflows can be called during normal vRA driven machine provisioning & disposing to achieve e significant level of customisability.
        2. Run the vCloud Automation Center->Infrastructure Administration->Extensibility->Installation->Install VCO Customisation  IaaS 4
        3. Verify that the defaults Stub Workflows are now modified using vRA Designer
          1. Download the vRA Designer from using the URL https://<vRA appliance FQDN>:5480 and under vRA Settings tabIaaS 5
          2. Install the vRA Designer (model manager runs on the IaaS server. In my case, its the first IaaS server deployed. Ensure you use the FQDN for the server name here)                              IaaS 6
          3. Launch the vCAC Designer application installed above               IaaS 7
          4. Click on Load button at the top to load the internal vRA workflow configuration to see the updated version numbers of the built in workflows IaaS 8
      4. Due to this customization, it provides the ability to call vRO workflows during provisioning process using vRA IaaS Blueprints. To assign these vRO workflows to each Blueprint (to be invoked during provisioning stage), we’ll need to later run the Library->vCAC->Inf Administration->Extensibility->Assign state change workflow and select the required Blueprint. – We will be using this extensibility feature later on, but so far we’ve just had all the preparation work completed.

There you have it. vRO 6.0.1 is now deployed, its integrated with AD, vCenter and vRA and the vRA’s default provisioning stub workflows have also been modified so that we can call vRO workflows during vRA IaaS machine provisioning process and use custom properties to populate the input parameters being required by those vRO workflows.

Now that we have all the pre-requisites for a fairly complete vRA deployment, we’ll next look at the initial vRA setup and all the different user accounts involved.

Cheers

Chan

 Next: vRA Part 6 – System / Infrastructure & Fabric Administrators –>

NSX 6.1.2 Bug – DLR interface communication issues & How to troubleshoot using net-vdr command

Chan 5 Comments

I have NSX 6.1.2 deployed on vSphere 5.5 and wanted to share the details of a specific bug applicable to this version of NSX that I came across, especially since it doesn’t have a relevant KB article published for it.

After setting up the NSX manager and all the host preparations including VXLAN prep, I had deployed a number of DLR instances, each with multiple internal interfaces and an uplink interface which in tern, was connected to an Edge Gateway instance for external connectivity. And I kept on getting communication issues between certain interfaces of the DLR, where by for example, the internal interfaces connected to vNIC 1 & 2 would communicate with one another (I can ping from a VM on VXLAN 1 attached to the internal interface on vNIC 1 to a VM on VXLAN 2 attached to the internal interface on vNIC 2) but none of them would talk to the internal interface 3, attached to the DLR vNic 3 or even uplink interface on vNic 0 (Cannot ping the Edge gateway). The interfaces that cannot communicate were completely random however and was persistent on multiple DLR instances deployed. All of them had one thing in common which was that no internal interface would talk to the uplink interface IP (of the Edge gateway attached to the other end of the uplink interface).

One symptom of the issue was what was described in this blog post I posed on the VMware communities page, at https://communities.vmware.com/thread/505542

Finally I had to log a call with NSX support at VMware GSS and according to their diagnosis, it turned out to be an inconsistency issue with the netcpa daemon running on the ESXi hosts and its communication with the NSX controllers. (FYI – netcpa gets deployed during the NSX host preparation stage, as a part of the User World Agents and is responsible for communication between DLR and the NSX controllers as well as the VXLAN and NSX controllers- see the diagram here)

During the troubleshooting, it transpired that some details (such as VXLAN details) were out of sync between the hosts and the controllers (different from what was shown as the VXLAN & VNI configuration in the GUI) and the temporary fix was to stop and start the netcpa daemon on each of the hosts in the compute & edge cluster ESXi nodes (commands “/etc/init.d/netcpad stop” followed up by “/etc/init.d/netcpad start” on the ESXi shell as root).

Having analysed the logs thereafter, VMware GSS confirmed that this was indeed an internally known issue with NSX 6.1.2. Their message was “This happens due to a problem in the tcp connection handshake between netcpa and the manager once the last one is rebooted (one of the ACKs is not received by netcpa, and it does not retry the connection). This fix added re-connect for every 5 seconds until the ACK is received“. Unfortunately, there’s no published KB article out (as of now) for this issue which doesn’t help much. expecially if you deploying this in a lab…etc.

This issue has (allegedly) been resolved in NSX version 6.1.3 even thought its not explicitly stated within the release notes as of yet (the support engineer I dealt with mentioned that he’s requested this to be added)

So, if you have similar issues with NSX 6.1.2 (or lower possibly), this may well be the way to go about it.

One thing I did lean during the troubleshooting process (which was probably the most important thing that came out of this whole issue for me, personally) was the understanding of the importance of the net-vdr command, which I should emphasize here though. Currently, there are no other ways to check if the agents running on the ESXi hosts have the correct configuration settings to do with NSX other than looking at it on command line…. I mean, you can force a resync of the appliance configuration or redeploy the appliances themselves using NSX GUI but that doesn’t necessarily update the ESXi agents and was of no help in my case mentioned above.

net-vdr command lets you perform a number of useful operations relating to DLR, from basic operations such as adding / deleting a new VDR (Distributed Router) instance, dumpi9ng the instance info, configure DRl settings including changing the controller details, adding / deleting DLR route details, listing all DLR instances, ARP operations such as show, add & delete ARP entries in the DLR & DLR Bridge operations and has turned out to be real handy for me to do various troubleshooting & verification operations regarding DLR settings. Unfortunately, there don’t appear to be much documentation on this command and its use, not on the VMware NSX documentation NOR within the vSphere documentation, at least as of yet, hence why I thought I’d mention it here.

Given below is an output of the commands usage,

net-vdr

So, couple of examples….

If you want to list all the DLR instances deployed and their internal names, net-vdr –instance -l will list out all the DLR instances as how ESXi sees them.

net-vdr --instance -l

If you NSX GUI says that you have 4 VNI’s 5000, 5001, 5002 & 5003 defined and you need to check whether if all 4 VNI configurations are also present on the ESXi hosts, net-vdr -L -l default+edge-XX (where XX is the unique number assigned to each of your DLR) will show you all the DLR interface configuration, as how the ESXi host sees it.

net-vdr -L l

If you want to see the ARP information within each DLR,  net-vdr –nbr -l default+edge-XX (where XX is the unique number assigned to each DLR) will show you the ARP info.

net-vdr --nbr -l default+edge-XX

 

Hope this would be of some use to those early adaptors of NSX out there….

Cheers

Chan

 

VMware vRealize Automation Part 4 – IaaS Server Deployment

Chan Leave a comment

Next: vRA Part 5 – vRO Deployment & Configuration & Integration –>

IaaS server is installed separately on a Windows server and is a key part of a vRealize Automation deployment. IaaS is the part within vRA that enables the rapid modelling and provisioning of servers & desktops across virtual and physical, private and public or Hybrid cloud infrastructures. Without the IaaS components, a vRA deployment is pretty none existent and in my view, pretty useless.

IaaS Component Architecture

IaaS components included within the vRA are as follows

  • IaaS Web site:
    • The IaaS Web site component provides the infrastructure administration and service authoring capabilities to the vCloud Automation Center console
    • Gets the updates from the Model manager for DEM, Proxy agents and the SQL database
  • Model Manager:
    • Provides updates from the DEM, proxy agents, and database to the IaaS web site.
    • The Model Manager holds the core of the business logic for vRA.
    • This business logic contains all the information required for connecting to external systems like VMware vSphere, Microsoft System Center Virtual Machine Manager, and Cisco UCS Manager…etc.
    • The Model Manager Web service component can have multiple instances and communicates with a Microsoft SQL database.
  • Manager Service
    • The Manager service coordinates communication between DEMS, agents, and the database.
    • The Manager Service communicates with the console Web site through the Model Manager. This service requires administrative privileges to run on the IaaS server
  • Distributed Execution Managers (DEMs)
    • DEMs execute the business logic of a vCloud Automation Center model, interacts with external systems, and manages virtual, cloud, and physical machine resources
    • DEMs are used for provisioning and managing machines on vCloud Director, vCloud Air Service, Red Hat Enterprise Virtualization Manager, Microsoft System Center Virtual Machine Manager, Amazon Web Services, Physical Server Management Interfaces for Dell, HP, and Cisco.
    • Runs as a Windows Service (1 service for DEM Orchestrator and another for DEM Worker
  • SQL Database
    • The IaaS component of vRealize Automation uses a Microsoft SQL Server database to maintain information about the machines it manages and its own elements and policies
    • A system administrator need to manually create the database during installation
  • Agents
    • Hypervisor proxy agents: Provisioning and managing machines and services on vSphere, Citrix, XenServer, Hyper-V. They send commands to and collect data from vSphere ESX Server, XenServer, and Hyper-V virtualization hosts and the virtual machines provisioned on them
    • EPI Agents: External provisioning infrastructure PowerShell agents
    • VDI Agents: Virtual Desktop Infrastructures PowerShell agents for XenDesktop Delivery Controller and VMware View Horizon enabling the XenDesktop web interface access through vRA
    • WMI Agents: Windows Management Instrumentation agents enhances the ability to monitor and control system information and allows you to manage remote servers from a central location. It enables the collection of data from Windows machines that vRealize Automation manages.
    • Management Agents: Management Agents collect support and telemetry information and registers IaaS nodes. A Management Agent is installed automatically on each IaaS node.
    • Each agent runs as a Windows service

One thing I’ve noticed on most of the VMware documentation is a lack of clarity in the form of a diagram of how these components interact. I’ve therefore attempted to document this as below.

1. Components

 

IaaS Server Deployment

The deployment of the IaaS component is not the easiest of the tasks and is somewhat unlike the typical user friendly deployment style we are used to with other VMware products (guessing this was because it came from the Dynamic Ops acquisition rather than being developed in house by VMware). It could be a quite a tedious task to ensure that all the various pre-requisites are in place on the IaaS Windows server manually and then having to run the setup (which is also a little cumbersome). Fortunately, a VMware TME (Brian Graf) has put together a really handy PowerShell script to automate the pre-requisite setup which I’ve used few times in the past and it has saved me lots of time. We’d be using that here.

Here are the steps involved in deploying the IaaS server components, starting from the Pre-requisites

  1. Ensure the Pre-requisites are in place on the IaaS server
    1. Build a Windows 2008 R2 / Windows 2012 / Windows 2012 R2 server
    2. Create a Domain account as the vRA service account. Lets call it <DomainName>\svc_vRA.
    3. Login to the Windows server (VM) and ensure that the vRA service account is a member of the local Administrators group 1.3
    4. Download the vRA 6.2 pre-req checker PowerShell script from GitHub and copy locally.
    5. Login as the vRA service account Run the “vRA 6.2 PreReq Automation Script.PS1” and follow the guided install wizard to add / download the additional components as required and install them automatically.   1.5
  2. Ensure the pre-requisites are in place on the SQL server
    1. Grant the vRA service account (svc_vRA) sysadmin rights on the SQL server instance (This is only temporary and is required during the installation period only for the installer to automatically create the database required which can later be revoked.2.1
  3. Verify the Pre-Requisites are correctly installed & configured on the IaaS server
    1. login to the IaaS server as the vRA service account.
    2. Download the IaaS installer specific to your vRA deployment by logging in to the https://<FQDN of the vRA Appliance>:5480 (login using root) 3.2
    3. Login to the installer using root and password specified during the deployment of the vRA appliance 3.3
    4. As this is the first IaaS server, I will be installing all the roles on this server (Will add a secondary DEM Orchestrator and DEM worker to another server later). Therefore select Complete Install and click next 3.4
    5. The built-in Pre-Requisite checker will now verify that you’ve got all the pre-requisites and confirm. 3.5
    6. If there are warnings against the Windows Firewall (even if its disabled), ensure that the Distributed Transaction Coordinator is allowed through the firewall and once verified, select the Firewall related warnings and click bypass. 3.6
    7. Move on to the next step
  4. Install the IaaS components
    1. From the Step 3.7 above, click next to proceed with the installation
    2. Provide the followings to the installer
      1. vRA service account username
      2. vRA service account password
      3. Passphrase (is a series of words that generates the encryption key used to secure database data and would required if the DB is to be restored)
      4. SQL server name (DO NOT type the instance name if there’s only a singly instance on the server. Just use the SQL server FQDN)
      5. vRA Database name & Click Next 4.2
    3. Provide the DEM and Agent names and click next 4.3
    4. Under the Component Registry,
      1. Provide the FQDN of the vRA appliance
      2. Load the default Tenant
      3. Download the certificate using the button & accept using the check box
      4. Provide the default SSO Administrator credentials (Administrator@sphere.local if using the vCenter SSO) & click test to verify. Verify the IaaS server name & Click next 4.4
    5. Click Install to begin the installation. Install log would be at “C:\Program Files (x86)\VMware\vCAC\InstallLogs\” folder.    4.5
    6. Once the installation complete (can take around 20 mins), click next & finish
  5. Verify the IaaS installation & Service registration
    1. Now login to the https://<FQDN of the vRA Appliance>:5480 as root and ensure the IaaS-service has a status as REGISTERED5.1
    2. Also verify that you can see the Infrastructure Administrators section being enabled within the Login to vRA UI for the default tenant (URL “https://<FQDN of the vRA Appliance>/shell-ui-app) when you login with the default SSO administrator credentials (Administrator@vsphere.local). Note that this was previously disabled pending the installation of the IaaS components. 5.2
  6. Revoke the temporary SQL permissions
    1. SysAdmin privileges assigned to the vRA service account on the SQL server instance is no longer required, (verify that the account has automatically been given the DBO permission to the vRA database) so you can now revoke this permission on the SQL server.6

 

That is it. The vRA IaaS server components have now been set up successfully. Next, we’ll look at deploying & configuring the latest version of vRO (vRealize Orchestrator – 6.0.1) which is a critical part of a useable vRA deployment.

Cheers

Chan

Next: vRA Part 5 – vRO Deployment & Configuration & Integration –>

VMware vRealize Automation Part 3 – vRA Appliance Deployment

Chan Leave a comment

Next: vRA Part 4 – IaaS Server Deployment –>

Ok, now that we’ve established the need for a dedicated management cluster to host the vRA management components, lets look at the deployement highlights of the vRA components within the management cluster.

  1. vRA Identity Appliance:
    1. If you want to use the vRA’s own identity management appliance, that should be the first component to be deployed. Deployment of this appliance is pretty straight forward and is self explanatory, hence I will not be covering here. However I will instead be using the vSphere SSO as the identity management source for vRA environment also, in order to keep all authentication for the my virtual infrastructure centralised (and simple).
    2. But if you are NOT planning on using vSphere SSO, make sure you download the vRA identity appliance from VMware and deploy on to the management cluster.
    3. Once deployed, ensure to configure the time zone and the NTP server settings using the management IP (specified during the Appliance deployment)
  2. vRA Appliance deployment
    1. Download the vRA appliance from VMware. Documentation for the latest release can be found here
    2. Deploy the appliance on to the Management Cluster using the vCenter server that manages it. You would need the following information during the deployment 1
  3. Configure the vRA Appliance
    1. Once the appliance deployment is complete, use the URL https://<FQDN/IP of the vRA appliance>:5480 to access the management interface
    2. Login with username root and password as specified during the deployment
    3. Set up the Time Zone (System->Time Zone)
    4. Set up the Host name (Network->Address) & Proxy server (Network->Proxy) if applicable
    5. Set up NTP to sync time (Admin->Time Settings)
    6. Set up the vRA specific settings
      1. Setup SSL (vRA Settings->Host Settings) – You can either self generate a Certificate or import a certificate obtained from a CA    2. SSL
      2. Set up SSO (vRA Settings->SSO) – You can connect to default vRA identity appliance or the vSphere SSO (>5.5 1b) as below using default SSO admin account. Note the following key points regarding the SSO host name
        1. Important Note: If you have an existing vRA / vCAC deployment already that is using a vSphere SSO server, note that you CANNOT use the same SSO server for another vRA server. I had number of issues when I attempted this, and the most notable one 1 where the non of the group / roles created within the default tenant (such as infrastructure admin & tenant Admin) would work and may come up with “401 – Unauthorized: Access is denied due to invalid credentials” error when logged in. This doesn’t appear to be correctly documented so watch out, but shouldn’t apply to most as its unlikely that you’d have multiple vRA deployments in the same organisation. the only way around this (if you have to use the same SSO source for multiple vRA’s) is to create a tenant rather than using the default tenant. Note the tenant name should be unique within the whole SSO too (if you have Tenant-A in vRA-A, you cannot add another Tenant-A on vRA-B using the same SSO)
        2. Host Name: When using the vSphere SSO server, host name should have the same case as what’s been registered in the vCenter SSO (if unsure, browse to https://ssoserver:7444/websso/SAML2/Metadata/vsphere.local and save the vsphere.download file when prompted.  Open the vsphere.download file in notepad or some text editor.  Locate the entityID attribute of the EntityDescriptor element.  Use the SSO server name in the way its specified here paying attention to the case)
        3. Port: 7444 in the host name for the vCenter SSO is NOT required with the vRA6.2.1 (this was required to be specifically specified in the host name field with the earlier versions of vCAC)
          3. SSO
      3. Add the appropriate license.(vRA Settings->License). It should be noted here that the license key added here should be the vRA standard or vRA Adavnced and not the vCloud suite license.
      4. Database connectivity (vRA Settings->Database) can be ignored in most cases unless you want to connect to an external Postgre SQL server / cluster
      5. Messaging (vRA Settings->Messaging) can also be ignored as this should have been automatically configured.
      6. Cluster configuration (vRA Settings->Cluster) can be bypassed unless you are creating a vRA appliance cluster in which case you can join an existing cluster here.
      7. Once all above are configured, allow couple of minutes and ensure all vRA services are now registered within the “Services” tab. 4. Services
    7. Configure the Identity stores
      1. Here, you can create new tenants (for a multi tenant deployment) or use the default tenant (automatically created). I’m going to use the default tenant here.
      2. Login to vRA UI using the URL “https://<FQDN of the vRA Appliance>/shell-ui-app” with the default SSO administrator credentials (Administrator@vsphere.local). The default vSphere.local tenant should be available. 5. Login
      3. Click on the vSphere.local, go to Identity stores and verify the default domain name listed (by default, the native ad domain would have been added here)
      4. If you need a separate identity / authentication realm (AD or open LDAP supported), you add it here
    8. Setup Tenant Administrators for the tenants
      1. Login with the SSO Administrator account and click on the tenant and then go to Administrators (using the default tenant in the example below)
      2. Add the AD user account or group to be used as the Tenant Administrator 7. Tenant admin 2
    9. Add the inbound and outbound email servers within the Email Servers tab on the left
    10. (Optional) Set up branding for the vRA user interface if required using the Branding tab

     

That’s it, the vRA appliance is not set up and the Tenant Admin account is also setup. Next up would be the IAAS server installation.

 

Next: vRA Part 4 – IaaS Server Deployment –>