NSX for vSphere 6.0.5 Release Notes

The NSX for vSphere 6.0.5 release replaces the 6.0, 6.0.1, 6.0.2, 6.0.3, and 6.0.4 releases.

What's in the Release Notes

• What's New
• System Requirements and Installation
• Known Issues
• Resolved Issues

What's New

NSX vSphere 6.0.5 includes a fix for the OpenSSL known issues CVE-2014-0224, CVE-2014-0198, CVE-2010-5298, and CVE-2014-3470 as well as other bug fixes documented in the Resolved Issues section.
Customers using NSX vSphere version 6.0, 6.0.1, 6.0.2, 6.0.3, or 6.0.4 must immediately upgrade to 6.0.5.

System Requirements and Installation

For information about system requirements and installation instructions, see the NSX Installation and Upgrade Guide.

To upgrade to this release, follow the steps below.

1. Back up your NSX Manager data and take a snapshot of NSX Manager.


2. Upgrade NSX Manager to the 6.0.5 release. For instructions, see Upgrading NSX in the NSX Installation and Upgrade Guide.


3. Retrieve the controller IDs and IP addresses by logging to the vSphere Web Client. Navigate to Networking & Security > Installation. The NSX Controller Nodes table lists the controller IDs (Name column) and IP addresses (Node column) of each controller. You will need these when you take a backup of the controllers and upgrade them.
   
   


4. Take a snapshot of each controller using the following REST API call.
   GET https://NSXManagerIPAddress/api/2.0/vdn/controller/controllerID/snapshot
   The output of the GET call is an octet stream containing the controller snapshot. To download the snapshot, use the following REST API call.
   curl -u admin:default -H "Accept: application/octet-stream" -X GET -k https://NSXManagerIPAddress/api/2.0/vdn/controller/controllerID/snapshot > controller_backup.snapshot
   Do not restore the controller snapshot by yourself. If the upgrade process to NSX vSphere 6.0.5 is not successful (steps 5-7 below), call VMware Support for help.


5. Upgrade the controllers in your environment one at a time by following the steps below.
   1. Download the upgrade bundle file VMware-NSX-Controller-upgrade-bundle-6.0.5-36402.nub and save it to a Linux or ESXi server such that the file is accessible through an SSH server.
   2. On the server where you copied the file in step 5a, set file permissions to ensure that the upgrade bundle file is readable and writeable:
      chmod 644 VMware-NSX-Controller-upgrade-bundle-6.0.5-36402.nub
   3. Log in to the first controller using SSH by typing the following command in a terminal window:
      ssh -l admin controllerIPAddress
      See step 3 for information on retrieving controller IP addresses.
   4. Copy the file to the first controller by typing the following in the terminal window:
      copy file username@server:/Path_to_upgrade_bundle_file targetFileName
   5. Type the following command to complete the upgrade:
      install software-update targetFileName
      Type y at the prompt asking whether you want to continue.
      After installation is complete, you are logged out of the SSH session. Wait for a few minutes and log back in to the controller. Verify that the welcome banner shows the new build number (36402).
      
   6. Ensure that the upgraded controller is active by typing the following CLI command in a terminal window.
      show control-cluster status
      The output should be similar to the sample below and Majority status should say Connected to cluster majority.
      
   7. Repeat steps c through f for each controller in your environment.
      On the UI, the Software Version column for the controller says 6.0 even after the upgrade.
   
   
6. Perform this step only if you are upgrading from NSX vSphere 6.0.3 or earlier.
   Navigate to the Controller UI to regenerate the certificate for each controller. For this, you need to delete each controller one at a time and add a replacement controller. Follow the steps below carefully.
   If you have a single controller in your environment, do not follow the steps below and call VMware Support.
   Ensure that you have the list of controller IDs and IP addresses from step 3.
1. Go back to the Installation tab of the vSphere Web Client.
2. Delete a controller by selecting it and clicking the Delete icon.
3. Wait for a few minutes.
4. Add a controller to replace the first controller by clicking the Add icon and completing the fields in the Add Controller dialog box. The Status column for the new controller shows Deploying.
   
   Proceed to the next step only when the Status column for the newly added controller says Normal with a green check mark.
   
   Also, note that the Software Version column for the new controller says 6.0 instead of 6.0.5.
5. Repeat steps b, c, and d for each controller in your environment - one controller at a time.


7. Update the host clusters in your environment and upgrade NSX Edge virtual machines to NSX vSphere 6.0.5. For instructions, see Upgrading NSX in the NSX Installation and Upgrade Guide
   Note the following:
   
   • If you upgrade an NSX Logical Router which has High Availability enabled on it, you must re-synchronize it with NSX Manager by selecting More Actions > Force Sync.
   • If you use SSL VPN, you must uninstall the SSL VPN client after upgrading to 6.0.5 and then reinstall it. To install the new client, go to https://ssl-vpn-ip-address where ssl-vpn-ip-address is the uplink IP address assigned to the Edge interface with which SSL VPN service is configured to listen on.
8. Perform this step only if you are upgrading from NSX vSphere 6.0.3 or earlier.
   If you are using SSL VPN, change the certificates and keys used by SSL VPN by following the steps below.
1. Add a new server certificate by following the steps below.
1. In the Networking & Security tab of the vSphere Web Client, click NSX Edges.
2. Double-click an NSX Edge.
3. Click the Manage tab and then click Settings.
4. Click the Certificates link and click Add Certificate.
5. Paste the certificate contents and private key.
6. Click OK.
2. Delete the old server certificate.
1. Select the old certificate and click the Delete icon.
2. Click OK.
3. Remove trust to the deleted certificates from your browser and operating system. Also, ensure that revocation checking is enabled for your browser and its operating system.
4. Contact your certificate provider to get the deleted certificate revoked.


9. Perform this step only if you are upgrading from NSX vSphere 6.0.3 or earlier.
   If you are using SSL VPN, configure SSL VPN to work with the new certificate.
1. In the SSL VPN-Plus tab, click Server Settings.
2. Click Change.
3. Ensure that Use Default Certificate is not selected.
4. Select the new certificate that you added in step 8a and click OK.


10. Perform this step only if you are upgrading from NSX vSphere 6.0.3 or earlier.
    If you are using an HTTPS load balancer profile, configure the Load Balancer to use the certificate you added in step 8a.
1. In the Load Balancer tab, select Application Profiles.
2. Select an Application profile and click Edit.
3. In Type, select HTTPS and select the new certificate.
4. Click OK.


11. If you are using SSL VPN, change the SSL VPN passwords. For instructions, see Virtual Private Networks in the NSX Administration Guide.

For information on upgrading vCenter Server 5.5 and ESX, see VMware Security Advisories.

Known Issues

The known issues are grouped as follows:

• Installation and Upgrade Issues
• General Issues
• NSX Manager Issues
• NSX Edge Issues
• Logical Switch Issues
• vShield Endpoint Issues

Installation and Upgrade Issues

NSX Edge upgrade fails but NSX Manager does not rollback Edge appliances to older version
If NSX Edge upgrade fails and the system does not rollback, you may experience network disruption and may be unable to manage the Edge. Workaround: Redeploy NSX Edge and then upgrade the Edge again.

SSL VPN client must be uninstalled and re-installed after upgrade
After upgrading to vShield 6.0.5, you must uninstall the SSL VPN client and then reinstall it. To install the latest client, go to https://ssl-vpn-ip-address where ssl-vpn-ip-address is the uplink IP address assigned to the Edge interface with which SSL VPN service is configured to listen on.

vSphere Distributed Switch MTU does not get updated
If you specify an MTU value lower than the MTU of the vSphere distributed switch when preparing a cluster, the vSphere Distributed Switch does not get updated to this value. This is to ensure that existing traffic with the higher frame size isn't unintentionally dropped.
Workaround: Ensure that the MTU you specify when preparing the cluster is higher than or matches the current MTU of the vSphere distributed switch. The minimum required MTU for VXLAN is 1550.

If all clusters in your environment are not prepared, the Upgrade message for Distributed Firewall does not appear on the Host Preparation tab of Installation page
When you prepare clusters for network virtualization, Distributed Firewall is enabled on those clusters. If all clusters in your environment are not prepared, the upgrade message for Distributed Firewall does not appear on the Host Preparation tab.
Workaround: Use the following REST call to upgrade Distributed Firewall:
PUT https://vsm-ip/api/4.0/firewall/globalroot-0/state

Service virtual machine deployed using the Service Deployments tab on the Installation page does not get powered on
Workaround: Follow the steps below.

1. Manually remove the service virtual machine from the ESX Agents resource pool in the cluster.
2. Click Networking and Security and then click Installation.
3. Click the Service Deployments tab.
4. Select the appropriate service and click the Resolve icon.
   The service virtual machine is redeployed.

After upgrading NSX from version 6.0 to 6.0.x, NSX Edges are not listed on the UI
When you upgrade from NSX 6.0 to NSX 6.0.x, the vSphere Web Client plug-in may not upgrade correctly. This may result in UI display issues such as missing NSX Edges.
This issue is not seen if you are upgrading from NSX 6.0.1 or later.
Workaround: Follow the steps below.

1. On the vSphere Server, navigate to the following location:
   /var/lib/vmware/vsphere-client/vc-packages/vsphere-client-serenity
2. Delete the following folders:
   com.vmware.vShieldManager-6.0.x.1546773
com.vmware.vShieldManager-6.0.1378053
3. Restart the vSphere Web Client service.

Service groups are expanded in Edge Firewall table during upgrade from vCloud Networking and Security 5.5 to NSX 6.0.3
User created service groups are expanded in the Edge Firewall table during upgrade - i.e., the Service column in the firewall table displays all services within the service group. If the service group is modified after the upgrade to add or remove services, these changes are not reflected in the firewall table.
Workaround: Follow the steps below:

1. Re-create the service groups in the Grouping Objects tab after upgrade.
2. Edit the Service column for the affected firewall rules and point to the appropriate service groups.

General Issues

NSX vSphere CPU licenses as displayed as VM licenses
NSX vSphere CPU entitlements are displayed as VM entitlements in the vSphere Licensing tab. For example, if a customer has licenses for 100 CPUs, the UI displays 100 VMs.
Workaround: None.

ESX CLI reports a MAC address of FF:FF:FF:FF:FF:FF
When a MAC address is unknown to the host, ESX CLI output shows FF:FF:FF:FF:FF:FF as its value instead of explicitly printing unknown.

vSphere Web Client 5.5 server restarts and UI becomes inaccessible
If you are using the vCenter Server appliance or running vCenter Server on a virtual machine, you may run into issues if the appliance or virtual machine does not meet certain requirements.
Workaround: Ensure that the appliance or virtual machine has a minimum of 12 GB of memory and an Intel or AMD x64 processor with two or more logical cores, each with a speed of 2 GHz.

When navigating between NSX Edge devices, vSphere Web Client hangs or displays a blank page
Workaround: Restart your browser.

vSphere Web Client displays error: Cannot complete the operation, see the event log for details
When you install services such as vShield Endpoint or a partner appliance through the Service Deployments tab, the vSphere Web Client may display the above error. You can ignore this error.

Cannot remove and re-add a host to a cluster protected by vShield Endpoint and third-party security solutions
If you remove a host from a cluster protected by vShield Endpoint and third-party security solutions by disconnecting it and then removing it from vCenter Server, you may experience problems if you try to re-add the same host to the same cluster.
Workaround: To removed a host from a protected cluster, first put the host in maintenance mode. Next, move the host into an unprotected cluster or outside all clusters and then disconnect and remove the host.

NSX Manager Issues

NSX Manager does not restore correctly from backup
After NSX Manager is restored from backup, communication channels with Logical Router control virtual machine do not recover correctly. Because of this, logical switches and portgroups cannot be connected to and disconnected from the Logical Router.
Workaround: After restoring the NSX Manager backup, reboot all Logical Router control virtual machines.

vMotion of NSX Manager may display error Virtual ethernet card Network adapter 1 is not supported
You can ignore this error. Networking will work correctly after vMotion.

Cannot delete 3rd party services after restoring NSX Manager backup
Deployments of 3rd party service(s) can be deleted from vSphere Web Client only if the restored state of NSX Manager contains the 3rd party service registration.
Workaround: Take a backup of NSX Manager database after all 3rd party services have been registered.

NSX Edge Issues

Enabling HA on a deployed Logical Router causes the router to lose its distributed routes on ESXi hosts
The Logical Router instance is deleted and re-created on ESXi hosts as part of the process of enabling HA. After the instance is re-created, routing information from the router's control virtual machine is not re-synced correctly. This makes the router lose its distributed routes on ESXi hosts.
Workaround: Reboot the Logical Router control virtual machine after enabling HA to restore the routes.

HA enabled NSX Logical Router does not redistribute routes after upgrade or redeploy
When you upgrade or redeploy an NSX Logical Router which has High Availability enabled on it, the router does not redistribute routes.
Workaround: After upgrading the NSX Logical Router, re-synchronize it with NSX Manager by selecting More Actions > Force Sync.

Static routes do not get pushed to hosts when NextHop is not specified
The UI allows you to create a static route on an NSX Edge device without specifying a NextHop. If you do not specify a NextHop, the static route does not get pushed to hosts.
Workaround: Always specify a NextHop for static routes.

Cannot configure NSX firewall using security groups or other grouping objects defined at global scope
Administrator users defined at the NSX Edge scope cannot access objects defined at the global scope. For example, if user abc is defined at Edge scope and security group sg-1 is defined at global scope, then abc will not be able to use sg-1 in firewall configuration of the NSX Edge.
Workaround: The administrator must use grouping objects defined at NSX Edge scope only, or must create a copy of the global scope objects at the NSX Edge scope.

Load balancer pool member displays WARNING message
Even though a load balancer pool member shows a WARNING message, it can still process traffic. You can ignore this message.

Cannot configure untagged interfaces for a logical router
The VLAN ID for the vSphere distributed switch to which a logical (distributed) router connects cannot be 0.
Workaround: Create tagged interfaces only.

VDR LIF routes are advertised by upstream ESG even if VDR OSPF is disabled
Upstream Edge Services Gateway (ESG) will continue to advertise OSPF external LSAs learned from VDR connected interfaces even when VDR OSPF is disabled.
Workaround: Disable redistribution of connected routes into OSPF manually and publish before disabling OSPF protocol. This ensures that routes are properly withdrawn.

When HA is enabled on gateway, OSPF hello and dead interval configured to values other than 30 seconds and 120 seconds respectively can cause some traffic loss during failover
When the primary NSX Edge fails with OSPF running and HA enabled, the time required for standby to take over exceeds the graceful restart timeout and results in OSPF neighbors removing learned routes from their Forwarding Information Base (FIB) table. This results in dataplane outage until OSPF reconverges.
Workaround: Set the default hello/dead interval timeouts on all neighboring routers to 30 seconds for hello interval and 120 seconds for dead interval. This enables graceful failover without traffic loss.

The UI allows you to add multiple IPs to a logical router interface though it is not supported
This release does not support multiple IPs for a logical router interface, even though the UI allows you to add multiple IPs.
Workaround: Do not add multiple IPs.

Cannot configure OSPF on more than one NSX Edge uplink
It is not possible to configure OSPF on more than one of the NSX Edge uplinks.

HA configuration fails if you choose the same interface for HA management and L2 VPN configuration
If an NSX Edge has L2 VPN enabled and you try to enable HA on the same NSX Edge, the configuration may fail. There are two cases in which this can happen:

1. If you manually select the same interface for HA management and L2 VPN.
2. If you select automatic HA configuration, in which case HA may end up using the same interface as L2 VPN.

L2 VPN enabled edges may give inconsistent results if high availability is enabled.
Workaround: Do not use L2 VPN with high availability.

SSL VPN does not support Certificate Revocation Lists (CRL)
You can add a CRL to an NSX Edge, but this CRL is not consumed by SSL VPN.
Workaround: CRL is not supported, but you can enable user authentication with client certificate authentication.

Cannot add an external authentication server to SSL VPN-Plus
You cannot use the FQDN or hostname of the external authentication server.
Workaround: You must use the IP address of the external authentication server.

SSL VPN client with proxy enabled in Use Safari Settings does not work on MAC computers
If you select Proxy with Safari setting in the SSL VPN client on a Mac computer, it is automatically unselected. Due to this, you will not be able to connect to that MAC computer through SSL VPN client.
Workaround: Use SOCKS version 4/5 or HTTP instead of Safari setting on MAC computers.

Cannot download NSX Edge tech support logs with Google Chrome 29
Workaround: Use Google Chrome version 30 or later.

Logical Switch Issues

Microsoft Clustering Services failover does not work correctly with Logical Switches
When virtual machines send ARP probes as part of the duplicate address detection (DAD) process, the VXLAN ARP suppression layer responds to the ARP request. This causes the IP address acquisition to fail, which results in failure of the DAD process.
Workaround: None.

Problems deleting EAM agencies
In order to successfully remove EAM Agencies from ESX Agent Manager (EAM), the NSX Manager that deployed the services corresponding the EAM Agencies must be available.
Workaround: Make sure the NSX Manager is available.

No warning displayed when a logical switch used by a firewall rule is being deleted
You can delete a logical switch even if it is in use by a firewall rule. The firewall rule is marked as invalid, but the logical switch is deleted without any warning that the switch is used by the firewall rule.

vShield Endpoint Issues

After deployment, the vShield Endpoint service virtual machine fails to communicate with NSX Manager
Workaround: Follow the steps below.

1. Manually remove the service virtual machine from the ESX Agents resource pool in the cluster.
2. Click Networking and Security and then click Installation.
3. Click the Service Deployments tab.
4. Select the appropriate service and click the Resolve icon.
   The service virtual machine is redeployed.

Resolved Issues

The following issues have been resolved in the 6.0.5 release.

• Where required, OpenSSL 1.0.1 is updated to 1.0.1h to address CVE-2014-0224, CVE-2014-0198, CVE-2010-5298, and CVE-2014-3470.
• Where required, OpenSSL 0.9.8 is updated to version openssl-0.9.8za to address CVE-2014-0224, CVE-2014-0198, CVE-2010-5298, and CVE-2014-3470.
• Virtual machine loses network connectivity during migration between resource pools, clusters, or vApps.
  For details, see A virtual machine loses network connectivity during migration between resource pools, clusters or vApps in vCloud Networking and Security 5.1.4, 5.5.2 and NSX for vSphere 6.0.4.
• NSX Edge upgrade fails with an error message saying that port 22 is in use
  If you are upgrading an NSX Edge with a load balancer and SSH is enabled, and the load balancer virtual server is set up to listen on a port that includes "22" (for example, port 22 or port 8228), the upgrade fails.
  
• NSX/PAN service profile is not applied for logical switches
  When you select a logical switch as an applied object for the Palo Alto NGFW service, the service profile is not applied.
• Stack overflow on NSX Manager may lead NSX Manager to restart
  Generating tech support logs for vShield Edge may result in a stack overflow that restarts NSX Manager.
• When a virtual machine or host moves from one resource pool to another, or from one cluster to another, the virtual machine may lose connectivity
  When a virtual machine is moved between resource pools or clusters, the path variable of the virtual machine may not be set correctly. This may result in the virtual machine losing connectivity.
• If the virtual machine vNIC connection is toggled frequently between on and off, the host may crash
• If you configure VSphere Web Client on port 443, you cannot access the Networking & Security tab in the vSphere Web Client
  If vSphere Web Client is configured on port 443, proxy calls from the vSphere Web Client to NSX Manager do not go through as AMF channel is not configured.
• vMotion fails for guest virtual machine when partner services are deployed
• VXLAN ARP suppression can fail for certain traffic patterns
  When doing dataplane learning of ARP entries, VXLAN updates any existing ARP cache entry. This leads to a situation where an ARP entry awaiting controller response can be populated from dataplane traffic instead, even if the entry ends up not existing in the controller.

The following issue has been resolved in the 6.0.4 release.