VMware Horizon and vCenter Server Certificate Issue

After upgrading the vCenter server for Horizon you may find that the vCenter Server tab on the dashboard of VMware Horizon Administrator would turn red and would present with a certificate issue. Upon checking the vCenter Server option under View Configuration tab you may get the message that the certificate is untrusted and will present you with an option to either accept or reject the certificate. Once we accept the certificate, we may get the message, “There was an error identifying the validity of the server”.

To fix the issue you can follow the below set of steps:

  1. Use a web browser that uses the operating certificate store on Windows (Such as Internet Explorer or Google Chrome).
  2. From the browser go to the base URL of the vCenter Server system or the vCenter Server Virtual Appliance without appending the port numbers or ‘vsphere-client’ extension. For example: https://vcenter.domain.com/
  3. Direct URL to download the vCenter server certificate can also be used which is https://vCenter_FQDN/certs/download.zip
  4. Right Click the download trusted root CA certificate link at the bottom of the grey box on the right and select save link as. This will prompt you to download a file called ‘download.zip’.
  • The file is a ZIP file of all root certificates and all CRLs in the VMware Endpoint Certificate Store (VECS).
  • Extract the contents of the ZIP file.
  • The result is a folder that contains two types of files. Files with a number as the extension (.0,.1 and so on) are root certificates. Files with an extension that starts with an r ( .r0,.r1 and so on) are CRL files associated with a certificate.

Next is to add the certificates on all the Connection Servers in VMware Horizon environment.

  1. RDP to the Connection server.
  2. Open the MMC Certificates Snap-in for Local Computer Account.
  3. Click Trusted Root Certificate Authority, Certificates folder and then Right Click > All Tasks >Import.
  4. In the Welcome Screen to the Certificate Import Wizard, Click Next.
  5. Browse and select the vCenter Certificates which you downloaded in the steps mentioned above.
  6. Select automatically select the certificate store based on the type of the certificate and click next.
  7. Complete the Certificate Import Wizard.
  8. The certificates should be imported to the root cert store successfully.
  9. After the process has been completed on all the connection servers, the servers need to be rebooted.
  10. Reboot the connection servers as per the KB: https://kb.vmware.com/s/article/2068381.
  11. Once the Connection server have been rebooted and services have been restarted verify the vCenter certificate in VMware Horizon Administrator Console.

VMware Horizon Virtual Desktops in Error State

I had a customer with VMware Horizon 7.0.2 desktop pools with desktops that were stuck with errors while recomposing. We were seeing the error message : “View Composer Error: Failed to delete VM – null” on the Desktop Machines in VMware Horizon View Administrator.

To fix the issue I had to manually delete the Virtual machines from vCenter inventory and then clean up the VMware Horizon database. Once that was achieved, VMware Horizon re-provisioned the machines. The re-provisioned machines were available for users to access.

The below command can be used to clean up machines in error state in VMware Horizon View Administrator.

The tool I used was viewdbchk which was introduced with Horizon View 6.2. The viewdbchk file is located in the installation of Horizon View directory in the VMware Horizon View Connection servers. The location is : “Installation Drive”\Program Files\VMware\VMware View\Server\tools\bin directory.

  • viewdbchk.cmd –scanMachines –limit 10

When the command was run, It scanned the desktops pools one by one for any machines in “Error” state and prompted if I wanted to delete the machines. A simple “yes” helped and we were able to delete the machines from VMware Horizon Admin console.

There are other ways also to fix the same issue but I found this to be the simplest one as it scanned the machines for errors in the environment.

The command would ask you to disable the desktop pool, scan for the machines in error state and will ask you if you want to enable the desktop pool again. If you do not want to scan a particular pool you can always type “no” when message appears to disable the desktop pool. If you type “no” it will skip that pool and move on to the next pool.

I hope this was helpful.

VMware vSAN Troubleshooting Commands

vSAN achieves high availability and performance through distribution of data across multiple hosts in the vSAN cluster. Data is transmitted over the vSAN network. There are cases where a large amount of data must be copied over the vSAN network. During the Resync operation there are chances that the VMs on the vSAN cluster might become inaccessible if there is not enough raw capacity available on the vSAN datastore.

By default, there should be 20% of capacity available for vSAN to function optimally. When the disks have less than 20% of free space available, vSAN automatically attempts to balance the capacity utilization by moving data from the disk to other disks in the vSAN cluster.

vSAN waits for 60 minutes by default before starting any repair and rebuild operation. vSAN has this delay of 60 minutes as many issues are transient.

Changing the default time for vSAN:

You can change the default time to a longer time frame by using the below command and restart the  Cluster Level Object Manager(CLOM) clomd service. These set of commands need to be run on all ESXi hosts in the vSAN cluster:

esxcli system settings advanced set -o /VSAN/ClomRepairDelay -i <value in minutes>

/etc/init.d/clomd restart

Note: The default 60 minutes is designed to cover a multitude of different configurations, setting the above option too aggressively can cause unnecessary resync operations to occur, when changing this advanced option consider these factors:

  • Installation of ESXi updates (if performing updates)
  • ESXi host boot time (Including Power On Self-Test)
  • SSD Log recovery for vSAN

Changing the disk threshold for rebalancing of vSAN objects:

You can change the default rebalance threshold of 20% free capacity available on the data disks of the vSAN cluster by using the below command.

esxcfg-advcfg -s 85 /VSAN/ClomRebalanceThreshold && /etc/init.d/clomd restart

The above command will change the free capacity of vSAN disks to be 15% free. Hence vSAN will start the rebalance of objects once the overall utilisation of disks is 85% or above.

Preparing for VMware Cloud on AWS

The process of deploying VMware SDDC in AWS is quite simple and should be complete within a matter of minutes, however one needs to be prepared for the deployment.

Following blog will cover the most basic set of pre-requisites which are required before starting with the deployment on VMware SDDC.

Cost Assessment

VMware provides an easy and simple to use VMware Cloud on AWS assessment tool which can help you with the number of hosts which will be required, an estimated cost of using VMware Cloud on AWS and a comparison to your private cloud environment.

The assessment tool supports two ways you would like to move your workload to the cloud, that are:

  1. Choosing specific Apps/VMs to move.
  2. Retiring old HW and moving a complete cluster to cloud.

The Tools used for Cost Assessment is vRealize Business for Cloud 7.3.1 (or later) . Once the tool is deployed, you can add your applications, VMs, datacenters, or Clusters to the assessment tool and create a scenario. Run the created scenario to determine the approximate cost and number of hosts needed.

You can go through this blog with detailed instructions on How to run a VMware Cloud on AWS Assessment.

You can watch video as well on VMware Cloud on AWS Assessment here.  

Planning for SDDC Administration

VMware Cloud on AWS accounts are based on an Organization, subscribed to VMware Cloud on AWS services. Your MyVMware account, is used to create the Organization and will make you an Organization Owner.

Organizational Owners are assigned the Organization Owner role which allows them to invite new users. New users can be assigned the following roles:

  1. Organization Owner role  
  2. Organization Member role.

Both types of users can manage the SDDC cloud, but only those with the Organizational Owner role can invite more users. Both users will have access to all the resources and services of the Organization and can create, manage, and access SDDCs belonging to the Organization.

The major tasks performed by organization users include, but are not limited to:

  • Adding and removing hosts to the SDDC
  • Configuring the management network for vCenter access/administration: VPN, DNS, Firewall rules
  • Configuring and maintaining the compute network for workloads: logical networks, firewall rules, NAT, VPN, DNS, Public IPs

For additional information you can view the Account Creation and Management section in this VMware document.

Planning for vCenter Administration

The Cloud administrator in VMware SDDC requires fewer privileges than an Administrator role in on-premises datacenter because several administration tasks in cloud SDDC are taken care by VMware themselves. This includes, but is not limited to, managing the lifecycle of Cloud SDDC software stack like deployment, configuration, patching etc., configuring the AWS infrastructure, adding/removing hosts and networks during failures or cluster scaling operations.

VMware Cloud on AWS introduces two new roles to traditional vCenter user model, that are:

  1. CloudAdmin

The CloudAdmin role has the necessary privileges for you to create and manage workloads on your SDDC. However, you cannot access or configure certain management components that are supported and managed by VMware, such as hosts, clusters, and management virtual machines.

  • GlobalCloudAdmin

The CloudGlobalAdmin role is associated with global privileges and allows you to perform only certain global tasks like create and manage Content Library objects.

A new vCenter user group called CloudAdminGroup will also be created and given the privileges associated with both roles.

For more details please review Privileges Reference for CloudAdmin and CloudGlobalAdmin on VMware docs.

  • Discrete vCenter Administrators

Some customers may wish to manage their VMware Cloud on AWS separately from their on-premise environment and will not want to connect them together. Although they will be missing out many of the benefits of a true hybrid cloud, this is possible with VMware Cloud on AWS. You will need to manage vCenter users directly in the SDDC vCenter console and simply create new users you want to give cloud administrator rights to and add them to the CloudAdminGroup described above.

  • Unified Administration

Unified administration refers to a unified management platform that spans across on-premises and cloud SDDC environment. To achieve this, VMware has developed a brand new feature called Hybrid Linked mode (HLM) which is like Enhanced Linked Mode which is used in on-premises environment.  Obviously, there are differences between Hybrid Linked Mode and Enhanced Linked Mode in their requirements and what is the need to use each one.

Hybrid Linked Mode is a flexible solution that allows us to jointly manage both the VMware Cloud on AWS and On-Premises SSO domains. HLM provides a one way trust from on-premises to VMware Cloud on AWS. It retains the separation between on-prem and VMware Cloud on AWS permissions in case we need to break the trust between the environments.

Once HLM is setup, On-Prem workloads can be migrated to VMware Cloud on AWS, best part being, the migration works both ways and workloads can be migrated from VMware Cloud on AWS to On-Prem infrastructure.

HLM supports both embedded vCenter servers and external platform deployment models for on-prem. Management of both the environments can be done by logging into VMware Cloud on AWS vSphere client with an on-prem user account.

More on HLM will be added in my later blogs.

Points to remember for Unified Administration:

  1. vSphere 6.0 U3 or above is supported. Make sure your current vSphere environment is upgraded to atleast vSphere 6.0 U3 or above.
  2. Determine your Identity Source. Supported Identity Source types are Active Directory over LDAP(s) or Open LDAP.
  3. Determine the following information to link on-prem to Cloud environment.
    1. Platform Services Controller FQDN.
      1. SSO Domain Name
        1. SSO Username and Password
  4. Determine the on-prem DNS server(s) that can resolve the Platform Services Controller and Identity Sources.
  5. Create or use an existing cloud administrator group from On-prem identity source which will have access to login to VMware Cloud on AWS vCenter.
  6. Add the users to the newly created SSO group. Remember, members of this group will be added to the CloudAdminGroup and become admins of the SDDC. They will be able to manage the On-Prem and Cloud vCenter in a single view.

Planning Migration of Applications

One of the main use cases of using VMWare Cloud on AWS is the ability to extend the on-premises environment and easily migrate the workloads between environments without making any changes to applications/workloads. One of the biggest problems is identifying which applications should be migrated to cloud.

Deploying and connecting to VMWare Cloud on AWS is a very structured process and an easy one if all pre-requisites are in place but identifying which applications to migrate is much more of a subjective decision which needs to be thought over and discussed.

Following points need to be discussed when planning migration of applications to Cloud:

  1. Collecting the data required for migration.
  2. Tools which can be leveraged to speed up the collection process.
  3. Analysing the data.
  4. What options are available for migrating the application.
  5. What VMWare services are available that can help with the migration process.

One of the new features with VMWare Cloud on AWS is HCX or Hybrid Cloud Extension which can abstract the on-prem and cloud resources and present to the apps as a single continuous cloud environment. HCX facilitates secure and seamless migration of apps from on-prem to cloud.

More on HCX will be added in my later blogs.

Preparing for Cloud Network and Connectivity

VMware Cloud on AWS helps in creating and rapidly provisioned software defined datacenter with just a few clicks and have the power of a public cloud environment with the on-prem infrastructure. To leverage the flexibility which VMWare Cloud on AWS provides, you need to ensure connectivity exists between the on-prem infrastructure, the AWS environment, the Internet and the deployed SDDC.

In this section we will discuss the connectivity options which are available to connect everything together. Some basic outlines will be drawn to help understand what is needed to successfully connect and operate the hybrid Cloud environment like:

  1. VPNs used by the SDDC.
  2. Configuring the Gateways.
  3. Firewall Rules.
  4. Network that will be used throughout the SDDC infra.

The SDDC is bifurcated into 2 components:

  1. Management Component
    1. Compute Component (Workloads)

The Management Component such as  vCenter, vSAN, NSX and ESXi hosts are accessed over Management Gateway(MGW)which is a NSX Edge Security Gateway that provides connectivity to vCenter and NSX manager running in the SDDC.

The Compute Gateway connects over Compute Gateway(CGW). The Compute Gateway utilizes a separate NSX Edge Security Gateway server and Distributed Logical Router(DLR) to manage the ingress and egress traffic of the VMs running the SDDC.

A secure connection must be established between the On-Prem environment and the two gateways. This is done using VPN connectivity.

Following considerations should be looked at while planning the connectivity:

  1. Review IPSEC VPN Requirements.
  2. Internet VPN or AWS Direct Connect.
  3. Management CIDR Block.
  4. DNS
  5. Management Firewall Settings.
  6. IPSEC or Layer 2 VPN for connectivity to Compute Gateway.
  7. AWS VPC Subnet.
  8. Logical Network for Workloads.
  9. Public IPs and NAT settings for Workloads.
  10. Firewall settings for Compute Workloads.

Hybrid Content Management

When we have our SDDC setup completed and connectivity options also tested, first thing we need to do is spin some VMs on the SDDC infrastructure. To achieve this, we will need the VM templates, OVF files, ISOs, and Scripts that we use in the datacenter today.  One of the best way to onboard or share these objects is using Content Library.

  • Content Library:

Content Library is the fastest and the easiest way to onboard content to the VMWare Cloud on AWS. Content Library organizes and automatically shares the ISOs, OVF Templates, Scripts across vCenters.

The first step will be to create a content library in your on-prem environment and add the desired files to it. Next step will be to simply publish the library with other vCenters. When you create a SDDC, you will have to create a content library as a Subscriber Library. This will allow us to automatically synchronize all the files immediately or we can choose to synchronize on demand.

  • Content Onboarding Assistant(COA):

If you are not using Content Library in your current on-prem environment then you can leverage a simple tool called Content Onboarding Assistant. The COA is designed to simplify bulk onboarding of content by letting us specify which ISOs, OVF Templates, Scripts needs to be published and automatically transfers the files.

Preparing for Disaster Recovery Services

VMware has brought together Site Recovery technology and VMware Cloud on AWS to provide Disaster Recovery as a Service (DRaaS).This new add-on feature to VMware Cloud on AWS enables customers to protect and recover applications without the requirement of a dedicated secondary site. It is a service which is managed and sold by VMware as a service.

Disaster Recovery as a Service with VMware Site Recovery can easily help you:

  • Accelerate time-to-protection: Remove the need to build a secondary DR site and implement DR in a day with familiar tools and the same operating environment from on-premises to the public cloud.
  • Simplify DR operations: Streamline operations with automated failover and failback and simplify ongoing maintenance and non-disruptive testing.
  • Reduce secondary site management costs with cloud-managed infrastructure and only pay for what you use with granular, on-demand cloud pricing.

More on DR as a service in later blogs.

Others things to remember:

Update PowerCli Scripts,vRO and Other Scripts.

Right Sizing of Workloads.

What is VMware Cloud on AWS?

VMware cloud on AWS in an enterprise-class Software defined datacenter software deployed on AWS infrastructure with optimized access to AWS services. It is powered by VMware Cloud Foundation, VMware Cloud on AWS integrates VMware’s compute (ESXi), Storage (vSAN) and Network (NSX-T) virtualization products along with VMware vCenter Management and is optimized to run on dedicated, elastic and bare-metal AWS infrastructure.

VMware Cloud on AWS is fully managed by VMware and is provided as a service to customers to enable hybrid cloud environment.

VMware Cloud on AWS SDDC is running directly on AWS elastic bare metal infrastructure, which provides high bandwidth, low latency connectivity to AWS services. Virtual machine workloads can access public API endpoints for AWS services such as AWS Lambda, Amazon Simple Queue Service (SQS), Amazon S3 and Elastic Load Balancing, as well as private resources in the customer’s Amazon VPC, such as Amazon EC2, and data and analytics services such as Amazon RDS, Amazon DynamoDB, Amazon Kinesis and Amazon Redshift. You can also now enjoy Amazon Elastic File System (EFS) for fully managed file service to scale the file-based storage automatically to petabyte scale with high availability and durability across multiple availability zones and the newest generation of VPC Endpoints designed to access AWS services while keeping all the traffic within the AWS network.

You can connect to your VMware Cloud on AWS using the following methods:

  • AWS Direct Connect.
  • VPN connectivity between on-prem infrastructure and VMware Cloud on AWS.

Why Use VMware Cloud on AWS?

  • Rapid and easy migration of VMware vSphere based workloads to the cloud. Less risk and cost of cloud migrations compared to alternatives that require conversions or re-architecture. Leverage familiar VMware tools and skillsets to accelerate cloud migrations. Once in the cloud you can leverage VMware and AWS services to modernize your applications at your pace.
  • Easily extend your footprint into the cloud and get VMware-consistent, enterprise-grade environments in the AWS cloud in a fast and cost-effective way.
  • Easily deliver business continuity with VMware Site Recovery, on-demand disaster recovery as a service. Reduce secondary site costs by providing a secondary site that is operationally consistent with your VMware datacenter.
  • Deliver a robust, feature-rich cloud platform for virtual desktops and applications. Optimize infrastructure costs with flexible, consumption-based billing while paying only for what you actually use.
  • Extend on-premises datacenters and easily migrate targeted application workloads to VMware Cloud on AWS without conversions. Obtain bi-directional workload portability between on-premises and VMware Cloud on AWS.
  • Access native AWS services seamlessly for new app development.
  • Get VMware SDDC capacity in the AWS Cloud in minutes whenever your business needs. Take advantage of elastic capacity and usage-based costing, by seamlessly moving live applications into an environment that is operationally consistent with your VMware datacenter.