This guide explains how to use Kubernetes Federated Services to deploy a common Service across multiple Kubernetes clusters. This makes it easy to achieve cross-cluster service discovery and availability zone fault tolerance for your Kubernetes applications.
This guide assumes that you have a running Kubernetes Cluster Federation installation. If not, then head over to the federation admin guide to learn how to bring up a cluster federation (or have your cluster administrator do this for you). Other tutorials, for example this one by Kelsey Hightower, are also available to help you.
Federated Services are created in much that same way as traditional Kubernetes Services by making an API call which specifies the desired properties of your service. In the case of Federated Services, this API call is directed to the Federation API endpoint, rather than a Kubernetes cluster API endpoint. The API for Federated Services is 100% compatible with the API for traditional Kubernetes Services.
Once created, the Federated Service automatically:
Clients inside your federated Kubernetes clusters (i.e. Pods) will automatically find the local shard of the Federated Service in their cluster if it exists and is healthy, or the closest healthy shard in a different cluster if it does not.
Federations of Kubernetes Clusters can include clusters running in different cloud providers (e.g. Google Cloud, AWS), and on-premises (e.g. on OpenStack). Simply create all of the clusters that you require, in the appropriate cloud providers and/or locations, and register each cluster’s API endpoint and credentials with your Federation API Server (See the federation admin guide for details).
Thereafter, your applications and services can span different clusters and cloud providers as described in more detail below.
This is done in the usual way, for example:
kubectl --context=federation-cluster create -f services/nginx.yaml
The ‘–context=federation-cluster’ flag tells kubectl to submit the request to the Federation API endpoint, with the appropriate credentials. If you have not yet configured such a context, visit the federation admin guide or one of the administration tutorials to find out how to do so.
As described above, the Federated Service will automatically create and maintain matching Kubernetes services in all of the clusters underlying your federation.
You can verify this by checking in each of the underlying clusters, for example:
kubectl --context=gce-asia-east1a get services nginx NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx 10.63.250.98 220.127.116.11 80/TCP 9m
The above assumes that you have a context named ‘gce-asia-east1a’ configured in your client for your cluster in that zone. The name and namespace of the underlying services will automatically match those of the Federated Service that you created above (and if you happen to have had services of the same name and namespace already existing in any of those clusters, they will be automatically adopted by the Federation and updated to conform with the specification of your Federated Service - either way, the end result will be the same).
The status of your Federated Service will automatically reflect the real-time status of the underlying Kubernetes services, for example:
$kubectl --context=federation-cluster describe services nginx Name: nginx Namespace: default Labels: run=nginx Selector: run=nginx Type: LoadBalancer IP: LoadBalancer Ingress: 18.104.22.168, 22.214.171.124, 126.96.36.199, 188.8.131.52, ... Port: http 80/TCP Endpoints: <none> Session Affinity: None No events.
Note the ‘LoadBalancer Ingress’ addresses of your Federated Service correspond with the ‘LoadBalancer Ingress’ addresses of all of the underlying Kubernetes services (once these have been allocated - this may take a few seconds). For inter-cluster and inter-cloud-provider networking between service shards to work correctly, your services need to have an externally visible IP address. Service Type: Loadbalancer is typically used for this, although other options (e.g. External IP’s) exist.
Note also that we have not yet provisioned any backend Pods to receive the network traffic directed to these addresses (i.e. ‘Service Endpoints’), so the Federated Service does not yet consider these to be healthy service shards, and has accordingly not yet added their addresses to the DNS records for this Federated Service (more on this aspect later).
To render the underlying service shards healthy, we need to add backend Pods behind them. This is currently done directly against the API endpoints of the underlying clusters (although in future the Federation server will be able to do all this for you with a single command, to save you the trouble). For example, to create backend Pods in 13 underlying clusters:
for CLUSTER in asia-east1-c asia-east1-a asia-east1-b \ europe-west1-d europe-west1-c europe-west1-b \ us-central1-f us-central1-a us-central1-b us-central1-c \ us-east1-d us-east1-c us-east1-b do kubectl --context=$CLUSTER run nginx --image=nginx:1.11.1-alpine --port=80 done
kubectl run automatically adds the
run=nginx labels required to associate the backend pods with their services.
Once the above Pods have successfully started and have begun listening for connections, Kubernetes will report them as healthy endpoints of the service in that cluster (via automatic health checks). The Cluster Federation will in turn consider each of these service ‘shards’ to be healthy, and place them in serving by automatically configuring corresponding public DNS records. You can use your preferred interface to your configured DNS provider to verify this. For example, if your Federation is configured to use Google Cloud DNS, and a managed DNS domain ‘example.com’:
$ gcloud dns managed-zones describe example-dot-com creationTime: '2016-06-26T18:18:39.229Z' description: Example domain for Kubernetes Cluster Federation dnsName: example.com. id: '3229332181334243121' kind: dns#managedZone name: example-dot-com nameServers: - ns-cloud-a1.googledomains.com. - ns-cloud-a2.googledomains.com. - ns-cloud-a3.googledomains.com. - ns-cloud-a4.googledomains.com.
$ gcloud dns record-sets list --zone example-dot-com NAME TYPE TTL DATA example.com. NS 21600 ns-cloud-e1.googledomains.com., ns-cloud-e2.googledomains.com. example.com. SOA 21600 ns-cloud-e1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 1209600 300 nginx.mynamespace.myfederation.svc.example.com. A 180 184.108.40.206, 220.127.116.11, 18.104.22.168, 22.214.171.124,... nginx.mynamespace.myfederation.svc.us-central1-a.example.com. A 180 126.96.36.199 nginx.mynamespace.myfederation.svc.us-central1-b.example.com. A 180 188.8.131.52 nginx.mynamespace.myfederation.svc.us-central1-c.example.com. A 180 184.108.40.206 nginx.mynamespace.myfederation.svc.us-central1-f.example.com. CNAME 180 nginx.mynamespace.myfederation.svc.us-central1.example.com. nginx.mynamespace.myfederation.svc.us-central1.example.com. A 180 220.127.116.11, 18.104.22.168, 22.214.171.124 nginx.mynamespace.myfederation.svc.asia-east1-a.example.com. A 180 126.96.36.199 nginx.mynamespace.myfederation.svc.asia-east1-b.example.com. CNAME 180 nginx.mynamespace.myfederation.svc.asia-east1.example.com. nginx.mynamespace.myfederation.svc.asia-east1-c.example.com. A 180 188.8.131.52 nginx.mynamespace.myfederation.svc.asia-east1.example.com. A 180 184.108.40.206, 220.127.116.11 nginx.mynamespace.myfederation.svc.europe-west1.example.com. CNAME 180 nginx.mynamespace.myfederation.svc.example.com. nginx.mynamespace.myfederation.svc.europe-west1-d.example.com. CNAME 180 nginx.mynamespace.myfederation.svc.europe-west1.example.com. ... etc.
Note: If your Federation is configured to use AWS Route53, you can use one of the equivalent AWS tools, for example:
$aws route53 list-hosted-zones
$aws route53 list-resource-record-sets --hosted-zone-id Z3ECL0L9QLOVBX
Whatever DNS provider you use, any DNS query tool (for example ‘dig’
or ‘nslookup’) will of course also allow you to see the records
created by the Federation for you. Note that you should either point
these tools directly at your DNS provider (e.g.
@ns-cloud-e1.googledomains.com...) or expect delays in the order of
your configured TTL (180 seconds, by default) before seeing updates,
due to caching by intermediate DNS servers.
The above set of DNS records is automatically kept in sync with the current state of health of all service shards globally by the Federated Service system. DNS resolver libraries (which are invoked by all clients) automatically traverse the hiearchy of ‘CNAME’ and ‘A’ records to return the correct set of healthy IP addresses. Clients can then select any one of the returned addresses to initiate a network connection (and fail over automatically to one of the other equivalent addresses if required).
By default, Kubernetes clusters come pre-configured with a cluster-local DNS server (‘KubeDNS’), as well as an intelligently constructed DNS search path which together ensure that DNS queries like “myservice”, “myservice.mynamespace”, “bobsservice.othernamespace” etc issued by your software running inside Pods are automatically expanded and resolved correctly to the appropriate service IP of services running in the local cluster.
With the introduction of Federated Services and Cross-Cluster Service
Discovery, this concept is extended to cover Kubernetes services
running in any other cluster across your Cluster Federation, globally.
To take advantage of this extended range, you use a slightly different
DNS name (of the form “
So, using our NGINX example service above, and the Federated Service
DNS name form just described, let’s consider an example: A Pod in a
cluster in the
us-central1-f availability zone needs to contact our
NGINX service. Rather than use the service’s traditional cluster-local
DNS name (
"nginx.mynamespace", which is automatically expanded
"nginx.mynamespace.svc.cluster.local") it can now use the
service’s Federated DNS name, which is
"nginx.mynamespace.myfederation". This will be automatically
expanded and resolved to the closest healthy shard of my NGINX
service, wherever in the world that may be. If a healthy shard exists
in the local cluster, that service’s cluster-local (typically
10.x.y.z) IP address will be returned (by the cluster-local KubeDNS).
This is almost exactly equivalent to non-federated service resolution
(almost because KubeDNS actually returns both a CNAME and an A record
for local federated services, but applications will be oblivious
to this minor technical difference).
But if the service does not exist in the local cluster (or it exists
but has no healthy backend pods), the DNS query is automatically
(i.e. logically “find the external IP of one of the shards closest to
my availability zone”). This expansion is performed automatically by
KubeDNS, which returns the associated CNAME record. This results in
automatic traversal of the hierarchy of DNS records in the above
example, and ends up at one of the external IP’s of the Federated
Service in the local us-central1 region (i.e. 18.104.22.168,
22.214.171.124 or 126.96.36.199).
It is of course possible to explicitly target service shards in availability zones and regions other than the ones local to a Pod by specifying the appropriate DNS names explicitly, and not relying on automatic DNS expansion. For example, “nginx.mynamespace.myfederation.svc.europe-west1.example.com” will resolve to all of the currently healthy service shards in europe, even if the Pod issuing the lookup is located in the U.S., and irrespective of whether or not there are healthy shards of the service in the U.S. This is useful for remote monitoring and other similar applications.
Much of the above discussion applies equally to external clients, except that the automatic DNS expansion described is no longer possible. So external clients need to specify one of the fully qualified DNS names of the Federated Service, be that a zonal, regional or global name. For convenience reasons, it is often a good idea to manually configure additional static CNAME records in your service, for example:
eu.nginx.acme.com CNAME nginx.mynamespace.myfederation.svc.europe-west1.example.com. us.nginx.acme.com CNAME nginx.mynamespace.myfederation.svc.us-central1.example.com. nginx.acme.com CNAME nginx.mynamespace.myfederation.svc.example.com.
That way your clients can always use the short form on the left, and always be automatcally routed to the closest healthy shard on their home continent. All of the required failover is handled for you automatically by Kubernetes Cluster Federation. Future releases will improve upon this even further.
Standard Kubernetes service cluster-IP’s already ensure that non-responsive individual Pod endpoints are automatically taken out of service with low latency (a few seconds). In addition, as alluded above, the Kubernetes Cluster Federation system automatically monitors the health of clusters and the endpoints behind all of the shards of your Federated Service, taking shards in and out of service as required (e.g. when all of the endpoints behind a service, or perhaps the entire cluster or availability zone go down, or conversely recover from an outage). Due to the latency inherent in DNS caching (the cache timeout, or TTL for Federated Service DNS records is configured to 3 minutes, by default, but can be adjusted), it may take up to that long for all clients to completely fail over to an alternative cluster in the case of catastrophic failure. However, given the number of discrete IP addresses which can be returned for each regional service endpoint (see e.g. us-central1 above, which has three alternatives) many clients will fail over automatically to one of the alternative IP’s in less time than that given appropriate configuration.
Check that your
See the federation admin guide to learn how to bring up a cluster federation correctly (or have your cluster administrator do this for you), and how to correctly configure your client.
kubectl describe clusters)
kubectl --namespace=federation logs $(kubectl get pods --namespace=federation -l module=federation-controller-manager -oname)
service-controllererrors in the output of
kubectl logs federation-controller-manager --namespace federation).
service-controllererrors or successes in the output of
kubectl logs federation-controller-manager --namespace federation)