VMware NSX Upgrade from 6.1.2 to 6.1.3

OK, I’ve had NSX 6.1.2 deployed in my lab, simulating a complex enterprise environment and the DLR (Distributed Logical Routers) kept on causing numerous issues (one of which I’ve blogged about in a previous article here). Anyways, VMware support has requested that I upgrade to the latest NSX version 6.1.3 which is supposed to have fixed most of the issues I’ve been encountering so I’ve decided to upgrade my environment and thought I’d write a blog about it too, especially since there aren’t very many precise NSX related documentation out there.

Note that despite the title of this article (upgrading 6.1.2 to 6.1.3), these steps are also applicable to upgrading from any NSX 6.0.x version to 6.1.x version.

High level steps involved

  1. First thing to do is to check for compatibility and I have vSphere 5.5 which is fully supported with NSX 6.1.3 as per the compatibility Matrix here.
  2. Then have a look at the 6.1.3 release notes to make sure you are up to date with what’s fixed (not all the fixes are listed here btw), what are the known issues…etc. That is available right here.
  3. Next thing is to download the upgrade media from the NSX / Nicira portal. If you have an active NSX subscription, you may download it from My VMware portal or else, (if you have taken the course and were deemed to be suitable to have access to the eval, you can download it from Nicira portal here. Once you’ve downloaded the Nicira upgrade media (VMware-NSX-Manager-upgrade-bundle-6.1.3-2591148.tar.gz file), you may also download the install & upgrade guide here.(Note that there don’t appear to be a separate set of documentation for 6.1.3 as it appears to be for the whole 6.1.x platform).
  4. If you are upgrading vSphere to v6.0 also at the same time, follow the steps below
    1. Upgrade NSX Manager and all NSX components to 6.1.3 in the following order.
      1. NSX Manager
      2. NSX controller(s)
      3. Clusters and Logical Switches
      4. NSX Edge
      5. Guest Introspection and Data Security
    2. Upgrade vCenter Server to 6.0.
    3. Upgrade ESXi to 6.0. – When the hosts boot up, NSX Manager pushes the ESXi 6.0 VIBs to the hosts. When the hosts show reboot required on the Hosts and Clusters tab in the left hand side of the vSphere Web Client, reboot the hosts a second time.  NSX VIBs for ESXi 6.0 are now enabled.
  5. If you are not upgrading vSphere at the same time (this is the case for me), follow the step below
    1. Upgrade NSX Manager and all NSX components to 6.1.3 in the following order.
      1. NSX Manager
      2. NSX controller(s)
      3. Clusters and Logical Switches
      4. NSX Edge
      5. Guest Introspection and Data Security

Now lets look at the actual upgrade process of NSX

NSX Upgrade steps

  1. NSX Manager Upgrade
    1. I don’t have Data security deployed. If you did, uninstall it first
    2. Snapshot the NSX Manager virtual appliance (as a precaution)
    3. Login to the NSX manager interface and go to the upgrade tab. Click on the upgrade button on top right and select the previously downloaded VMware-NSX-Manager-upgrade-bundle-6.1.3-2591148.tar.gz file & continue.          1.3
    4. Select to enable SSH and click upgrade.                                                           1.4
    5. Upgrade of the NSX Manager appliance will now start.       1.5
    6. Once the upgrade completes, the appliance will automatically reboot (this is not so obvious unless you look at the console of the appliance). Once rebooted, login to the NSX manager instance and confirm the version details (Top right).    1.6
  2. NSX controller(s) Upgrade
    1. Notes
      1. Upgrades are done at cluster level and would appear with an upgrade link in NSX manager.
      2. It’s recommended that no new VM’s are provisioned, no VMs are moved (manually or via DRS) during the controller upgrade
    2. Backup controller data by taking a snapshot
      1. If the existing NSX version is < 6.1, This has to be done using a REST API call
      2. If the existing NSX version is => 6.1, you can do this using the GUI of the web client as follows 2.2.2
    3. Login to the Web client, and go to the Networking & Security -> Installation and click on upgrade available link as shown below 2.3
    4. You’ll now see the upgrade being in progress (may ask you to reload the web client). Controller(s) will reboot during the upgrade 2.4
    5. The upgrade should complete within 30 mins normally. If not completed by then, upgrade may need to be launched using the upgrade available link again. Once successful, the upgrade status should state upgraded and the software version should be updated with the new build number. Note that if the upgrade status says failed, be sure to refresh the web client prior to assuming that its actually failed. It may state its failed due to the 30 standard timeout even though the controller upgrade has completed successfully (happened to me couple of times) 2.5
  3. Clusters and Logical Switches
    1. Go to the host preparation tab and click update                           3.1
    2. Installation and upgrade guide states that each host will automatically be put in to the maintenance mode & rebooted. If the upgrade failed with a Not ready status for each node, you may need to click resolve each time to retry (happened o me once) and it will proceed. Some host may require manual evacuation of VM’s too. Also worth noting that if you keep getting the task “DRS recommends hosts to evacuate” logged in vSphere client, you may want to temporarily disable HA so that the ESXi host can be put in maintenance mode by the NSX cluster installer (Arguably, in a well managed cluster, this should not happen).    3.2
    3. When the cluster is updated, the Installation Status column displays the software version that you have updated to.    3.3
  4. NSX Edge (Including the Distributed Logic Router instances)
    1. Go to the NSX Edges section and for each instance, select the Upgrade Versions option from the Actions menu.   4.1
    2. once the upgrade is complete, it will show the new version of the Edge device, weather that’s a DLR or an Edge gateway device.   4.2
  5. Guest Introspection and Data Security
    1. Upgrading these components are relatively straight forward and I’d encourage you to look at the install & upgrade guide…
  6. Now, create a NSX backup (You may already have routine backup scheduled on the NSX manager)
  7. If all is well, remove the snapshot created earlier, on the NSX manager appliance.

 

All in all, it was a fairly straight forward upgrade but the time it took due to somewhat finicky nature of some of the components (especially the cluster upgrade where a number of manual retries were required) unnecessarily added time. Other than that, all appears to be well in my NSX deployment.

Cheers

Chan

VMware vRealize Automation Part 5 – vRO Deployment & Configuration & Integration

Next: vRA Part 6 – System / Infrastructure & Fabric Administrators –>

vRealize Orchestrator (was known as vCenter Orchestrator before) is an integral part of a vRA deployment as most of the vRA extensibility features relies on vRO being available and correctly integrated to vRA appliance & IaaS components. vRO in tern, connects to a number of other 3rd party systems through plugins, such as Microsoft Active Directory, Microsoft SQL, VMware vSphere, NSX….etc., where you can create a range of vRO workflows to carry out / automate tasks within these 3rd party systems. Due to the integration between vRA and vRO, these vRO workflows can be published within the vRA catalog and be made available via the built in self service portal (for example, you can publish a blueprint for resetting the passwords for an AD user through a workflow created on vRO).

I personally haven’t seen much documentation or clear guidelines with sufficient details on how to set up vRA with vRO for this level of extensibility which was one of the main reasons why I decided to publish this series of posts (which, hopefully will be addressed by VMware in the future).

So, now its the time to deploy the vRO and have it fully configured in preparation for the integration with vRA in due course. This article therefore looks at the deployment and the initial configuration of vRO component. (This should be considered a must have part of your vRA deployment)

vRO is available in many different forms

  • Available as a standalone appliance to be downloaded free of charge for vSphere customers
  • Automatically deployed if you install Windows version of vCenter (vRO Windows services are disabled by default though) – Note that this is now deprecated with the vRO 6.0.1 and hence SHOULD NOT be used going forward.
  • Available built in to the vRA itself (used for PaaS functions only, NOT used for IaaS functions for which you must register an external vRO)

In the interest of performance, and separation (and with the added ability of clustering should you need that in future), I prefer to deploy the standalone appliance (which is also the VMware’s recommended approach) so this article will rely on that. However, note that the post deployment configuration is still the same between the standalone appliance and the default version installed with vCenter.

The latest version of vRO released as of today, is the version 6.0.1, which was released in March 2015, in line with the announcement of vSphere 6.0. The release notes for the new version can be found here. It is compatible with vCenter server 4.1 and above.

Few highlights are below,

  • The stability of the vCenter Server plug-in has been improved by resolving major issues based on customer feedback (this was a real issue with earlier versions)
  • vRealize Orchestrator 6.0.1 supports the vSphere Web Client integration and context execution of vRealize Orchestrator workflows as part of vSphere Web Client 6.0
  • Switch Case introduced to handling conditional and branching operations within an automated workflow (If a value matches a switch case, specific workflow branch is executed)
  • Global error handling introduced
  • vRO control center beta introduced (Centralized admin interface for runtime execution, workflow monitoring & correlation with system resources, unified log access and configuration)

I will not be covering details about the Orchestrator itself, but only how to deploy & configure it within this post. If you need information about the vRO itself, including its architecture…etc. please refer to the official documentation found here.

Orchestrator Appliance Deployment

  1. Download the vRO appliance from My VMware portal as an ova and deploy on to the Management Cluster (NOT the compute / production cluster. Refer to the proposed architecture diagram in the previous post of this series here)
  2. During the deployment, you’d need to provide the information required and click next (ensure SSH is enabled) O3
  3. Once the appliance is deployed and powered on, SSH on to the appliance as root and change the root password expiry date.
    1. The password for the root account of the Orchestrator Appliance expires after 365 days. You can increase the expiry time for an account by logging in to the Orchestrator Appliance as root, and running passwd -x number_of_days name_of_account. o3.1
  4. Configure the appliance
    1. Go to the https://<vRO FQDN>:5480 and login using root.
    2. Change the Time Zone & Setup NTP (Admin->Time Settings)
    3. Update the Proxy settings should it be required and verify all other networking settings.

That’s it. the basic vRO appliance is now deployed and ready to be configured for vRO functionality which we’ll look at next.

 

vRO Configuration

  1. Login to the vRO appliance Configuration page 
    1. https://<vRO FQDN>:8283/vco-config (username: vmware)
  2. Bind the IP & DNS
    1. Click on Network on the left and select the IP address from the drop down menu to bind the vRO to the correct IP address.vRO2
  3. Import vCenter SSL Certificate
    1. Under Network, click on SSL Trust Manager tab and type import the vCenter server SSL certificate using the URL “https://<FQDN of the vCenter Server>” vRO3
    2. Note that this vCenter server should be the one managing the compute / production ESXi cluster, NOT the one managing the Management cluster.
  4. Setup Authentication Type
    1. Best practise recommendation is to configure the authentication source to be the vCenter SSO as this allows the user of vRO through vSphere web client. Note that the built in OpenLDAP is NOT recommended for production deployments.
    2. Import SSO server SSL certificate using the SSL Trust Manager
      1. Host: https://<FQDN of the SSO server>:7444 vRO4.2.1
    3. Setup vCenter SSO for authentication
      1. SSO Registration
        1. Select the Authentication mode as SSO Authentication within the Authentication section.
        2. Host: https://<FQDN of the SSO server>:7444
        3. Admin user name: Administrator@vsphere.local   vRO4.3.3
      2. SSO Configuration (Setup vRO admin group)
        1. Specify the SSO group that gets rights to the vRO server admin role here
        2. Select the SSO domain (I’m selecting the Active Directory attached to the SSO server)
        3. Select the dedicated Active Directory group (I’ve got a dedicated AD group created called “VMware vRO Administrators” which I’m going to add here.)   vRO 4.3.2.3
        4. Make sure the following accounts are members of this group on the AD
          1. vRA service account (<domain>\svc_vRA)
          2. vRO service account (<domain>\svc_vRO)
          3. Any domain user account requiring access to vRO workflows through vRO client.
        5. Note: One thing I’ve noticed here is that when changing this SSO Admin group (SSO or AD) always unregister the orchestrator and re-register with SSO. In the old version of vCO, it really didn’t like it when you just change this vRO admin group
  5. Assign vCenter license (default vRO license is only valid for 90 days)
    1. Under Licenses, select vCenter Server license
    2. Provide the vCenter address (<FQDN of the vCenter Server>) and an administrative user name to the vCenter (I’d create a new vRO service account, such as  <domain name>\svc_vRO, and assign vSphere administrative privileges to it and use it here as that’s neat and follow the RBAC guidelines) vRO5.2
    3. restart the vRO server service using the startup options
  6. Assign an external Microsoft SQL database
    1. The vRO 6.0.1 built in database is sufficient for small to medium scale deployments only and we will be using a separate SQL server as per VMware best practise.
    2. External database is also a pre-requisite for future clustering of the vRO.
    3. Prepare the SQL database
      1. Note: I’m going to use an external SQL 2008R2 server instance here.
      2. Verify that the protocol and port access is setup correctly on the SQL server vRO6.3.2
      3. Create an empty database on the SQL server (vRO601) vRO6.3.3.1
      4. Create a SQL user account (svc_vRO) and map this to the vRO601 database with dbo permission
        1. Note: With using the vRO Linux based appliance, I’ve had weird errors here and trying to get both Windows authenticationand SQL authentication work here. To cut a long story short, I only managed to get it to work after adding a Windows domain account (froot\svc_vRO) and a SQL account (svc_vRO), both with the same name added as DBO to the database. (makes no sense). This account will only be used by vRO to connect to its database. Ensure that SQL password expiry is ticked off when creating the sql account to avoid future problems. vRO6.3.4
    4. Import the SQL server SSL certificate to vRO using Trust Manager (Only applicable if the DB server uses SSL certs. Mine doesn’t so I will skip this)
    5. Setup the SQL server database connectivity
      1. Provide the SQL database connectivity information required and click apply changes vRO6.5.1
      2. Once prompted, create the database tables using the link provided. Once completed successfully, the status will be displayed. Click Apply Changes vRO6.5.2
  7. Assign a SSL Certificate to vRO
    1. Either create a self signed server certificate or import a certificate database (using an existing CA) vRO7.1.1
    2. Export the certificate DB for DR                                        vRO7.1.2
  8. Configure Orchestrator Plugins (Using vRO configuration interface)
    1. Configuration of Active Directory, SSH & vCenter Server plugins (all come installed by default in vRO) are now done through pre-defined workflows available within vCO rather than through the configuration interface
    2. Type the vRO admin username and password within the plugin section (When the Orchestrator server starts, the system uses these credentials to set up the plug-ins. The system checks the enabled plug-ins and performs any necessary internal installations such as package import, policy run, script launch, and so on) – I’m using the vRO domain service account (<domain>\svc_vRO) which is also a member of the vRO administrators group I’ve mapped under Authentication.
  9. Install new Orchestrator plugins
    1. Various addition vRO plugins can be downloaded from the vRO official web page (here) and installed on to the vRO via the configuration page so that vRO workflows can be created to integrate and manipulate external systems.
    2. Use the Install Application tab within the General section of the vRO configurator to install plugins that have a .vmoapp extension.
    3. There are a number of additional plugins available here but I will be installing the followings, for the purpose of this article
      1. Install the plugin VMware vCenter Orchestrator Plugin for NSX 1.0.1 – Required for vRO workflow creation for NSX in order to orchestrate some NSX tasks
        1. Compatibility: Based on the VMware product Interoperability Matrix as of now, for vRO 6.0.1, NSX version 6.1, 6.1.1 & 6.1.2 are the only versions of NSX supported and that requires the vRO NSX plugin version 1.0.1. But the vcoteam.info (official site for vRO team) site’s plugin list only contains the NSX 1.0.0 plugin and not version 1.0.1. So be sure to download the correct version via the link I’ve provided above.
        2. Note: This plugin is required for vRO workflow creation for NSX in order to orchestrate some NSX tasks. The current workflows included within this package are limited however and the installation of the NSX-v Dynamic Types plugin V2 (below) will add additional NSX related workflows. Once these workflows are available, they can be leveraged by vRA.
        3. Install the NSX plugin (using o11nplugin-nsx-1.0.1.vmoapp file) vRO9.2.1.2
        4. Accept license & Apply changes                                            vRO9.2.1.3
        5. Restart vRO server to take effect (note that the plugin status may not change to “installed” immediately after the vCO service restart. This can take some time)   vRO9.2.1.4
        6. Login to the vRO client and verify the NSX workflows are now available within Library->NSX folder vRO9.2.1.5
      2. Install the plugin NSX-v Dynamic Types plug-in V2
        1. Note: A handy NSX Dynamic Types plugin to use with vRA Advanced Service Designer. This plug-in can be customized and extended using the plug-in generator V2 package.
        2.  Install instructions are all available in the link above (summary below)
          1. Rename the downloaded package com.vmware.coe.plugin.dynamictypes.nsx.v2.package
          2. Import the package (Using vRO client, NOT the configuration page)
          3. Verify that the package and the additional NSX workflows now appear within the vRO client. vRO 9.2.3
          4. More steps are required to be carried out to complete this which will be done under step 10 below
      3. Install the plugin VMware vRealize Orchestrator vRA Plug-in 6.2.0
        1. Note: This is required for integration with vRA 6.2.
        2. Install the vRA 6.2.x plugin using the o11nplugin-vcac-6.2.0-2287231.vmoapp on the vRO configuration page vRO9.3.1
        3. Verify the installation and restart the vCO service (note that the plugin status may not change to “installed” immediately after the vCO service restart. This can take some time)                                                                                                                                                            vRO9.3.2
        4. Note: Login to the vRO client and ensure all default workflows are visible within the client. (I’ve had repeated issues whereby, after the vRA 6.2 plugin has been installed, all default plugins and their workflows disappeared within vCO. If this happens, within the vRO configuration page, go to Troubleshooting section and click “Reset current version” under “Reinstall the plug-ins when the server starts and reboot the vRO appliance) – This can potentially save you lots of time….!!
  10. Configure Orchestrator Plugins (using vRO workflows) – We will configure the key plugins here and leave some to be configured later
    1. Configure vCenter plugin
      1. Note: The way to integrate vCenter server may differ depending on whether you have vCenter 5.5 or 6.0 as well as whether its a windows install or the vCSA. In my case, its vCenter 5.5 on good old Windows so the below steps are based on that configuration.
      2. Login to the vRO client (you can download the client installable from https://<vRO FQDN>:8281 or launch the client directly from there)  and run the workflow Library-> vCenter->Configuration->Add a vCenter server instance   10.1.2
      3. Now, select whether you are happy to use a shared session for vRO to connect to vCenter server. If you using a shared session (meaning, you’ve answered “No”), provide the credentials that the vRO uses to connect to the vCenter with (shared credentials for all users). I’m using the vRO service account (svc_vRO) which I’ve made an Administrators on the vCenter instance. This is usually good enough as I can see all vRO driven activities on the vCenter logs / events clearly thanks to the service account name.  Also, since the ultimate goal is to call these workflows via vRA portal, and when I specify blueprints based on these workflows, I’m already controlling who has access to access those blueprints, so the shared credentials between vRO and vCenter is the best option. If however you decides to not use shared credentials, you need ensure each user who access vCO workflows though vRA catalog, will have sufficient permission on vCenter to execute that specific workload and I don’t think that is a good way to go about it, architecturally.     10.1.3
      4. Once the workflow has been successfully run, you’ll see the message on the screen as follows. 10.1.4
    2. Register vCenter Orchestrator as a vCenter Server Extension
      1. Note: This important so that vRO workflows will automatically be available within the vSphere Web Client to execute.
      2. Login to vRO client and rung the workflow: Library -> vCenter -> Configuration -> Register vCenter Orchestrator as a vCenter Server Extension
      3. Select the vCenter server (managing compute & edge cluster that the above workflow was run against) here. 10.2.2
      4. Verify it ran successfully
      5. Verify on the Managed Object Browser for that vCenter that the vRO extension is listed correctly. Instructions are as follows
        1. In a Web browser navigate to the managed object browser of your vCenter Server instance. https://<vcenter_server_ip?/mob
        2. Log in with vCenter Server credentials.
        3. Under Properties, click content.
        4. On the Data Object Type: ServiceContent page, under Properties, click ExtensionManager.
        5. On the Managed Object Type page, under Properties, click the Orchestrator extension string.extensionList[“com.vmware.vco”]. The extension has a server property which contains an array of type ExtensionServerInfo. The array should contain an instance of the ExtensionServerInfo type with a url property which contains the URL of the registered Orchestrator server.
        6. On the Data Object Type: Extension page, under Properties, click server. You can see information about the Orchestrator server registered as an extension, such as serverThumbprint and url. The serverThumbprint property is the SHA-1 thumbprint of the Orchestrator server certificate, which is a unique identifier of the Orchestrator server. The url property is the service URL of the Orchestrator server. There is one record per IP address. If the Orchestrator server has two IP addresses, both of them are displayed as service URLs.        10.2.4.6
    3. Configure the Active Directory Plugin
      1. Login to the vRO client and run the workflow: Library->Microsoft->Configuration->Configure Active Directory Server
      2. Provide the answers including the AD dc name (I’d providing the root of the domain address which is automatically directed to any of the DCs in the domain rather than pointing at a single DC.                                      AD 10.3.1
      3.      Now select the authentication method to the AD as a shared session. If a shared session is not used, the user who runs the vRO workflow is expected to have sufficient rights in the AD to run the particular workflow which WILL NOT be practical once the vRO is integrated with vRA and users start requesting machine provisioning from blueprints which in turn, calls for vRO workflows. AD-1
    4. Configure VMware vRealize Orchestrator vRA Plug-in 6.2.0 – This step is key….!!
      1. Add vRA Host
        1. Import the vRA server SSL certificate
          1. Run the workflow “Import a certification from URL” under library->configuration->SSL Trust Manager
          2. Provide the vRA appliance details in the URL & Submit. Verify Its run correctly. vRA 10.4.1.1.2
        2. Add the vRA host
          1. Run “Add a CAC host” workflow under vCAC->Configuration and select the vRA server FQDN and the URL (with https://) vRA 10.4.1.2.1
          2. Use the shared session option using the Administrator@vsphere.local account to connect to the vRA server vRA 10.4.1.2.2
          3. Verify its run successfully at the end.
      2. Add the vRA IaaS host
        1. Setup Infrastructure Administrators Group on the default tenant
          1. Login to the vRA UI using https://<vRA Appliance FQDN>/shell-ui-app using the Administrator@vsphere.local account IaaS 0
          2. Go to the Administratprs tab within the vSphere.local tenant (default) and add an appropriate AD group for vRA Infrastructure Administrators (I have a group created here called <domain>\VMware vCAC Inf Admins and a user account named <domain>\inf-admin as a member of that group) IaaS 0 1
        2. Run “Add a vCAC IAAS host” workflow
          1. Run the workflow “Add a vCAC IAAS host” under vCAC->Configuration and select the vRA server (already added to vRO) IaaS 1
          2. Host name is are auto populated. Change this to be the name of your IaaS windows server (as shown below) and Click next.                        IaaS 2
          3. My preference is to use the shared session option here. The shared user account used to connect to the IaaS windows server needs to be a administrative account that has local administrative rights to the IaaS server where the model manager database is installed. As such, I’m going to use the vRA service account (domain service account that was created and used to install the IaaS services with during an earlier step here which had admin rights to the IaaS server).  DO NOT type the domain name here. Just type the account name as the domain is provided in the next step.      IaaS 3
          4. Change the domain name to the appropriate domain name and submit.   IaaS 2.5
          5. Verify it runs successfully.
      3. Install vRO Stub Workflow Customization
        1. Note: This workflow will modify the default Workflow Stubs used by vRA so that vRO workflows can be called during normal vRA driven machine provisioning & disposing to achieve e significant level of customisability.
        2. Run the vCloud Automation Center->Infrastructure Administration->Extensibility->Installation->Install VCO Customisation  IaaS 4
        3. Verify that the defaults Stub Workflows are now modified using vRA Designer
          1. Download the vRA Designer from using the URL https://<vRA appliance FQDN>:5480 and under vRA Settings tabIaaS 5
          2. Install the vRA Designer (model manager runs on the IaaS server. In my case, its the first IaaS server deployed. Ensure you use the FQDN for the server name here)                              IaaS 6
          3. Launch the vCAC Designer application installed above               IaaS 7
          4. Click on Load button at the top to load the internal vRA workflow configuration to see the updated version numbers of the built in workflows IaaS 8
      4. Due to this customization, it provides the ability to call vRO workflows during provisioning process using vRA IaaS Blueprints. To assign these vRO workflows to each Blueprint (to be invoked during provisioning stage), we’ll need to later run the Library->vCAC->Inf Administration->Extensibility->Assign state change workflow and select the required Blueprint. – We will be using this extensibility feature later on, but so far we’ve just had all the preparation work completed.

There you have it. vRO 6.0.1 is now deployed, its integrated with AD, vCenter and vRA and the vRA’s default provisioning stub workflows have also been modified so that we can call vRO workflows during vRA IaaS machine provisioning process and use custom properties to populate the input parameters being required by those vRO workflows.

Now that we have all the pre-requisites for a fairly complete vRA deployment, we’ll next look at the initial vRA setup and all the different user accounts involved.

Cheers

Chan

 Next: vRA Part 6 – System / Infrastructure & Fabric Administrators –>