Setup

Instructions for setting up a Kubernetes cluster.

Edit This Page

VMware vSphere

This page covers how to get started with deploying Kubernetes on vSphere and details for how to configure the vSphere Cloud Provider.

Getting started with the vSphere Cloud Provider

Kubernetes comes with a cloud provider for vSphere. A quick and easy way to try out the cloud provider is to deploy Kubernetes using Kubernetes-Anywhere.

This page also describes how to configure and get started with the cloud provider if deploying using custom install scripts.

Deploy Kubernetes on vSphere

To start using Kubernetes on top of vSphere and use the vSphere Cloud Provider use Kubernetes-Anywhere. Kubernetes-Anywhere will deploy and configure a cluster from scratch.

Detailed steps can be found at the getting started with Kubernetes-Anywhere on vSphere page

vSphere Cloud Provider

vSphere Cloud Provider allows using vSphere managed storage within Kubernetes. It supports:

  1. Volumes
  2. Persistent Volumes
  3. Storage Classes and provisioning of volumes.
  4. vSphere Storage Policy Based Management for Containers orchestrated by Kubernetes.

Documentation for how to use vSphere managed storage can be found in the persistent volumes user guide and the volumes user guide

Examples can be found here

Enable vSphere Cloud Provider

If a Kubernetes cluster has not been deployed using Kubernetes-Anywhere, follow the instructions below to enable the vSphere Cloud Provider. These steps are not needed when using Kubernetes-Anywhere, they will be done as part of the deployment.

Step-1 Create a VM folder and move Kubernetes Node VMs to this folder.

Step-2 Make sure Node VM names must comply with the regex [a-z](([-0-9a-z]+)?[0-9a-z])?(\.[a-z0-9](([-0-9a-z]+)?[0-9a-z])?)* If Node VMs does not comply with this regex, rename them and make it compliant to this regex.

Node VM names constraints:

Step-3 Enable disk UUID on Node virtual machines

The disk.EnableUUID parameter must be set to “TRUE” for each Node VM. This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly.

For each of the virtual machine nodes that will be participating in the cluster, follow the steps below using GOVC tool

Note: If Kubernetes Node VMs are created from template VM then disk.EnableUUID=1 can be set on the template VM. VMs cloned from this template, will automatically inherit this property.

Step-4 Create and assign Roles to the vSphere Cloud Provider user and vSphere entities.

Note: if you want to use Administrator account then this step can be skipped.

vSphere Cloud Provider requires the following minimal set of privileges to interact with vCenter. Please refer vSphere Documentation Center to know about steps for creating a Custom Role, User and Role Assignment.

Roles Privileges Entities Propagate to Children
manage-k8s-node-vms Resource.AssignVMToPool
System.Anonymous
System.Read
System.View
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddNewDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.RemoveDisk
VirtualMachine.Inventory.Create
VirtualMachine.Inventory.Delete
Cluster,
Hosts,
VM Folder
Yes
manage-k8s-volumes Datastore.AllocateSpace
Datastore.FileManagement
System.Anonymous
System.Read
System.View
Datastore No
k8s-system-read-and-spbm-profile-view StorageProfile.View
System.Anonymous
System.Read
System.View
vCenter No
ReadOnly System.Anonymous
System.Read
System.View
Datacenter,
Datastore Cluster,
Datastore Storage Folder
No

Step-5 Create the vSphere cloud config file (vsphere.conf). Cloud config template can be found here

This config file needs to be placed in the shared directory which should be accessible from kubelet container, controller-manager pod, and API server pod.

vsphere.conf for Master Node:

[Global]
        user = "vCenter username for cloud provider"
        password = "password"
        server = "IP/FQDN for vCenter"
        port = "443" #Optional
        insecure-flag = "1" #set to 1 if the vCenter uses a self-signed cert
        datacenter = "Datacenter name" 
        datastore = "Datastore name" #Datastore to use for provisioning volumes using storage classes/dynamic provisioning
        working-dir = "vCenter VM folder path in which node VMs are located"
        vm-name = "VM name of the Master Node" #Optional
        vm-uuid = "UUID of the Node VM" # Optional        
[Disk]
    scsicontrollertype = pvscsi

Note: vm-name parameter is introduced in 1.6.4 release. Both vm-uuid and vm-name are optional parameters. If vm-name is specified then vm-uuid is not used. If both are not specified then kubelet will get vm-uuid from /sys/class/dmi/id/product_serial and query vCenter to find the Node VM’s name.

vsphere.conf for Worker Nodes: (Only Applicable to 1.6.4 release and above. For older releases this file should have all the parameters specified in Master node’s vSphere.conf file)

[Global]
        vm-name = "VM name of the Worker Node"

Below is summary of supported parameters in the vsphere.conf file

Step-6 Add flags to controller-manager, API server and Kubelet to enable vSphere Cloud Provider. * Add following flags to kubelet running on every node and to the controller-manager and API server pods manifest files.

--cloud-provider=vsphere
--cloud-config=<Path of the vsphere.conf file>

Manifest files for API server and controller-manager are generally located at /etc/kubernetes

Step-7 Restart Kubelet on all nodes. * Reload kubelet systemd unit file using systemctl daemon-reload * Restart kubelet service using systemctl restart kubelet.service

Note: After enabling the vSphere Cloud Provider, Node names will be set to the VM names from the vCenter Inventory.

Known issues

vmware#220 : vSphere Cloud Provider can not be used on the Kubernetes Cluster when vCenter port is configured other than the default port 443. Fix for this issue is already out (Kubernetes PR# 49689). We will make sure that, PR 49689 is cherry picked to 1.7, 1.6 and 1.5 branches.

Support Level

IaaS Provider Config. Mgmt OS Networking Docs Conforms Support Level
Vmware vSphere Kube-anywhere Photon OS Flannel docs   Community (@abrarshivani), (@kerneltime), (@BaluDontu), (@luomiao), (@divyenpatel)

If you identify any issues/problems using the vSphere cloud provider, you can create an issue in our repo - VMware Kubernetes.

For support level information on all solutions, see the Table of solutions chart.

Analytics

Create an Issue Edit this Page