An open source system for automating deployment, scaling, and operations of applications.

Monday, October 31, 2016

Introducing Kubernetes Service Partners program and a redesigned Partners page

Kubernetes has become a leading container orchestration system by being a powerful and flexible way to run distributed systems at scale. Through our very active open source community, equating to hundreds of person years of work, Kubernetes achieved four major releases in just one year to become a critical part of thousands of companies infrastructures. However, even with all that momentum, adopting cloud native computing is a significant transition for many organizations. It can be challenging to adopt a new methodology, and many teams are looking for advice and support through that journey.

Today, we’re excited to launch the Kubernetes Service Partners program. A Service Partner is a company that provides support and consulting for customers building applications on Kubernetes. This program is an addition to our existing Kubernetes Technology Partners who provide software and offer support services for their software. 

The Service Partners provide hands-on best practice guidance for running your apps on Kubernetes, and are available to work with companies of all sizes to get started; the first batch of participants includes: Apprenda, Container Solutions, Deis, Livewyer, ReactiveOps and Samsung SDS. You’ll find their listings along with our existing Technology Partners on the newly redesigned Partners Page, giving you a single view into the Kubernetes ecosystem. 

The list of partners will grow weekly, and we look forward to collaborating with the community to build a vibrant Kubernetes ecosystem.


--Allan Naim, Product Manager, Google, on behalf of the Kubernetes team.


Tail Kubernetes with Stern

Editor’s note: today’s post is by Antti Kupila, Software Engineer, at Wercker, about building a tool to tail multiple pods and containers on Kubernetes.

We love Kubernetes here at Wercker and build all our infrastructure on top of it. When deploying anything you need to have good visibility to what's going on and logs are a first view into the inner workings of your application. Good old tail -f has been around for a long time and Kubernetes has this too, built right into kubectl.

I should say that tail is by no means the tool to use for debugging issues but instead you should feed the logs into a more persistent place, such as Elasticsearch. However, there's still a place for tail where you need to quickly debug something or perhaps you don't have persistent logging set up yet (such as when developing an app in Minikube).

Multiple Pods

Kubernetes has the concept of Replication Controllers which ensure that n pods are running at the same time. This allows rolling updates and redundancy. Considering they're quite easy to set up there's really no reason not to do so.

However now there are multiple pods running and they all have a unique id. One issue here is that you'll need to know the exact pod id (kubectl get pods) but that changes every time a pod is created so you'll need to do this every time. Another consideration is the fact that Kubernetes load balances the traffic so you won't know at which pod the request ends up at. If you're tailing pod A but the traffic ends up at pod B you'll miss what happened.

Let's say we have a pod called service with 3 replicas. Here's what that would look like:

$ kubectl get pods                         # get pods to find pod ids
$ kubectl log -f service-1786497219-2rbt1  # pod 1
$ kubectl log -f service-1786497219-8kfbp  # pod 2
$ kubectl log -f service-1786497219-lttxd  # pod 3

Multiple containers

We're heavy users gRPC for internal services and expose the gRPC endpoints over REST using gRPC Gateway. Typically we have server and gateway living as two containers in the same pod (same binary that sets the mode by a cli flag). The gateway talks to the server in the same pod and both ports are exposed to Kubernetes. For internal services we can talk directly to the gRPC endpoint while our website communicates using standard REST to the gateway.

This poses a problem though; not only do we now have multiple pods but we also have multiple containers within the pod. When this is the case the built-in logging of kubectl requires you to specify which containers you want logs from.

If we have 3 replicas of a pod and 2 containers in the pod you'll need 6 kubectl log -f <pod id> <container id>. We work with big monitors but this quickly gets out of hand…
If our service pod has a server and gateway container we'd be looking at something like this:

$ kubectl get pods                                 # get pods to find pod ids
$ kubectl describe pod service-1786497219-2rbt1    # get containers in pod
$ kubectl log -f service-1786497219-2rbt1 server   # pod 1
$ kubectl log -f service-1786497219-2rbt1 gateway  # pod 1
$ kubectl log -f service-1786497219-8kfbp server   # pod 2
$ kubectl log -f service-1786497219-8kfbp gateway  # pod 2
$ kubectl log -f service-1786497219-lttxd server   # pod 3
$ kubectl log -f service-1786497219-lttxd gateway  # pod 3

Stern

To get around this we built Stern. It's a super simple utility that allows you to specify both the pod id and the container id as regular expressions. Any match will be followed and the output is multiplexed together, prefixed with the pod and container id, and color-coded for human consumption (colors are stripped if piping to a file).

Here's how the service example would look:

$ stern service
This will match any pod containing the word service and listen to all containers within it. If you only want to see traffic to the server container you could do stern --container server service and it'll stream the logs of all the server containers from the 3 pods.
The output would look something like this:
$ stern service
+ service-1786497219-2rbt1 › server
+ service-1786497219-2rbt1 › gateway
+ service-1786497219-8kfbp › server
+ service-1786497219-8kfbp › gateway
+ service-1786497219-lttxd › server
+ service-1786497219-lttxd › gateway
service-1786497219-8kfbp server Log message from server
service-1786497219-2rbt1 gateway Log message from gateway
service-1786497219-8kfbp gateway Log message from gateway
service-1786497219-lttxd gateway Log message from gateway
service-1786497219-lttxd server Log message from server
service-1786497219-2rbt1 server Log message from server

In addition, if a pod is killed and recreated during a deployment Stern will stop listening to the old pod and automatically hook into the new one. There's no more need to figure out what the id of that newly created pod is.

Configuration options

Stern was deliberately designed to be minimal so there's not much to it. However, there are still a couple configuration options we can highlight here. They're very similar to the ones built into kubectl so if you're familiar with that you should feel right at home.
  • --timestamps adds the timestamp to each line
  • --since shows log entries since a certain time (for instance --since 15min)
  • --kube-config allows you to specify another Kubernetes config. Defaults to ~/.kube/config
  • --namespace allows you to only limit the search to a certain namespaceRun stern --help for all options.
Examples

Tail the gateway container running inside of the envvars pod on staging
     stern --context staging --container gateway envvars
Show auth activity from 15min ago with timestamps
     stern -t --since 15m auth
Follow the development of some-new-feature in minikube
     stern --context minikube some-new-feature
View pods from another namespace
     stern --namespace kube-system kubernetes-dashboard

Get Stern

Stern is open source and available on GitHub, we'd love your contributions or ideas. If you don't want to build from source you can also download a precompiled binary from GitHub releases






Monday, October 24, 2016

How We Architected and Run Kubernetes on OpenStack at Scale at Yahoo! JAPAN


Editor’s note: today’s post is by the Infrastructure Engineering team at Yahoo! JAPAN, talking about how they run OpenStack on Kubernetes. This post has been translated and edited for context with permission -- originally published on the Yahoo! JAPAN engineering blog


Intro
This post outlines how Yahoo! JAPAN, with help from Google and Solinea, built an automation tool chain for “one-click” code deployment to Kubernetes running on OpenStack. 

We’ll also cover the basic security, networking, storage, and performance needs to ensure production readiness. 

Finally, we will discuss the ecosystem tools used to build the CI/CD pipeline, Kubernetes as a deployment platform on VMs/bare metal, and an overview of Kubernetes architecture to help you architect and deploy your own clusters. 

Preface
Since our company started using OpenStack in 2012, our internal environment has changed quickly. Our initial goal of virtualizing hardware was achieved with OpenStack. However, due to the progress of cloud and container technology, we needed the capability to launch services on various platforms. This post will provide our example of taking applications running on OpenStack and porting them to Kubernetes.

Coding Lifecycle
The goal of this project is to create images for all required platforms from one application code, and deploy those images onto each platform. For example, when code is changed at the code registry, bare metal images, Docker containers and VM images are created by CI (continuous integration) tools, pushed into our image registry, then deployed to each infrastructure platform.



We use following products in our CICD pipeline:


Function
Product
Code registry
GitHub Enterprise
CI tools
Jenkins
Image registry
Artifactory
Bug tracking system
JIRA
deploying Bare metal platform
OpenStack Ironic
deploying VM platform
OpenStack
deploying container platform
Kubernetes

Image Creation. Each image creation workflow is shown in the next diagram.

VM Image Creation:


  1. push code to GitHub
  2. hook to Jenkins master
  3. Launch job at Jenkins slave 
  4. checkout Packer repository
  5. Run Service Job
  6. Execute Packer by build script
  7. Packer start VM for OpenStack Glance 
  8. Configure VM and install required applications
  9. create snapshot and register to glance
  10. Download the new created image from Glance
  11. Upload the image to Artifactory
Bare Metal Image Creation:

  1. push code to GitHub
  2. hook to Jenkins master
  3. Launch job at Jenkins slave 
  4. checkout Packer repository
  5. Run Service Job
  6. Download base bare metal image by build script
  7. build script execute diskimage-builder with Packer to create bare metal image
  8. Upload new created image to Glance
  9. Upload the image to Artifactory
Container Image Creation:

  1. push code to GitHub
  2. hook to Jenkins master
  3. Launch job at Jenkins slave 
  4. checkout Dockerfile repository
  5. Run Service Job
  6. Download base docker image from Artifactory
  7. If no docker image found at Artifactory, download from Docker Hub
  8. Execute docker build and create image 
  9. Upload the image to Artifactory
Platform Architecture.

Let’s focus on the container workflow to walk through how we use Kubernetes as a deployment platform. This platform architecture is as below.






Function
Product
Infrastructure Services
OpenStack
Container Host
CentOS
Container Cluster Manager
Kubernetes
Container Networking
Project Calico
Container Engine
Docker
Container Registry
Artifactory
Service Registry
etcd
Source Code Management
GitHub Enterprise
CI tool
Jenkins
Infrastructure Provisioning
Terraform
Logging
Fluentd, Elasticsearch, Kibana
Metrics
Heapster, Influxdb, Grafana
Service Monitoring
Prometheus

We use CentOS for Container Host (OpenStack instances) and install Docker, Kubernetes, Calico, etcd and so on. Of course, it is possible to run various container applications on Kubernetes. In fact, we run OpenStack as one of those applications. That's right, OpenStack on Kubernetes on OpenStack. We currently have more than 30 OpenStack clusters, that quickly become hard to manage and operate. As such, we wanted to create a simple, base OpenStack cluster to provide the basic functionality needed for Kubernetes and make our OpenStack environment easier to manage.

Kubernetes Architecture

Let me explain Kubernetes architecture in some more detail. The architecture diagram is below.




Product
Description
OpenStack Keystone
Kubernetes Authentication and Authorization
OpenStack Cinder
External volume used from Pod (grouping of multiple containers)
kube-apiserver
Configure and validate objects like Pod or Services (definition of access to services in container) through REST API
kube-scheduler
Allocate Pods to each node
kube-controller-manager
Execute Status management, manage replication controller
kubelet
Run on each node as agent and manage Pod
calico
Enable inter-Pod connection using BGP
kube-proxy
Configure iptable NAT tables to configure IP and load balance (ClusterIP)
etcd
Distribute KVS to store Kubernetes and Calico information
etcd-proxy
Run on each node and transfer client request to etcd clusters
Tenant Isolation To enable multi-tenant usage like OpenStack, we utilize OpenStack Keystone for authentication and authorization.

Authentication With a Kubernetes plugin, OpenStack Keystone can be used for Authentication. By Adding authURL of Keystone at startup Kubernetes API server, we can use OpenStack OS_USERNAME and OS_PASSWORD for Authentication. Authorization We currently use the ABAC (Attribute-Based Access Control) mode of Kubernetes Authorization. We worked with a consulting company, Solinea, who helped create a utility to convert OpenStack Keystone user and tenant information to Kubernetes JSON policy file that maps Kubernetes ABAC user and namespace information to OpenStack tenants. We then specify that policy file when launching Kubernetes API Server. This utility also creates namespaces from tenant information. These configurations enable Kubernetes to authenticate with OpenStack Keystone and operate in authorized namespaces. Volumes and Data Persistence Kubernetes provides “Persistent Volumes” subsystem which works as persistent storage for Pods. “Persistent Volumes” is capable to support cloud-provider storage, it is possible to utilize OpenStack cinder-volume by using OpenStack as cloud provider. Networking Flannel and various networking exists as networking model for Kubernetes, we used Project Calico for this project. Yahoo! JAPAN recommends to build data center with pure L3 networking like redistribute ARP validation or IP CLOS networking, Project Calico matches this direction. When we apply overlay model like Flannel, we cannot access to Pod IP from outside of Kubernetes clusters. But Project Calico makes it possible. We also use Project Calico for Load Balancing we describe later.

In Project Calico, broadcast production IP by BGP working on BIRD containers (OSS routing software) launched on each nodes of Kubernetes. By default, it broadcast in cluster only. By setting peering routers outside of clusters, it makes it possible to access a Pod from outside of the clusters. External Service Load Balancing
There are multiple choices of external service load balancers (access to services from outside of clusters) for Kubernetes such as NodePort, LoadBalancer and Ingress. We could not find solution which exactly matches our requirements. However, we found a solution that almost matches our requirements by broadcasting Cluster IP used for Internal Service Load Balancing (access to services from inside of clusters) with Project Calico BGP which enable External Load Balancing at Layer 4 from outside of clusters.

Service Discovery
Service Discovery is possible at Kubernetes by using SkyDNS addon. This is provided as cluster internal service, it is accessible in cluster like ClusterIP. By broadcasting ClusterIP by BGP, name resolution works from outside of clusters. By combination of Image creation workflow and Kubernetes, we built the following tool chain which makes it easy from code push to deployment.
Summary
In summary, by combining Image creation workflows and Kubernetes, Yahoo! JAPAN, with help from Google and Solinea, successfully built an automated tool chain which makes it easy to go from code push to deployment, while taking multi-tenancy, authn/authz, storage, networking, service discovery and other necessary factors for production deployment. We hope you found the discussion of ecosystem tools used to build the CI/CD pipeline, Kubernetes as a deployment platform on VMs/bare-metal, and the overview of Kubernetes architecture to help you architect and deploy your own clusters. Thank you to all of the people who helped with this project. --Norifumi Matsuya, Hirotaka Ichikawa, Masaharu Miyamoto and Yuta Kinoshita. This post has been translated and edited for context with permission -- originally published on the Yahoo! JAPAN engineer blog where this was one in a series of posts focused on Kubernetes.

Friday, October 14, 2016

Building Globally Distributed Services using Kubernetes Cluster Federation

Editor's note: Today’s post is by Allan Naim, Product Manager, and Quinton Hoole, Staff Engineer at Google, showing how to deploy a multi-homed service behind a global load balancer and have requests sent to the closest cluster.

In Kubernetes 1.3, we announced Kubernetes Cluster Federation and introduced the concept of Cross Cluster Service Discovery, enabling developers to deploy a service that was sharded across a federation of clusters spanning different zones, regions or cloud providers. This enables developers to achieve higher availability for their applications, without sacrificing quality of service, as detailed in our previous blog post. 

In the latest release, Kubernetes 1.4, we've extended Cluster Federation to support Replica Sets, Secrets, Namespaces and Ingress objects. This means that you no longer need to deploy and manage these objects individually in each of your federated clusters. Just create them once in the federation, and have its built-in controllers automatically handle that for you.

Federated Replica Sets leverage the same configuration as non-federated Kubernetes Replica Sets and automatically distribute Pods across one or more federated clusters. By default, replicas are evenly distributed across all clusters, but for cases where that is not the desired behavior, we've introduced Replica Set preferences, which allow replicas to be distributed across only some clusters, or in non-equal proportions (define annotations). 

Starting with Google Cloud Platform (GCP), we’ve introduced Federated Ingress as a Kubernetes 1.4 alpha feature which enables external clients point to a single IP address and have requests sent to the closest cluster with usable capacity in any region, zone of the Federation. 

Federated Secrets automatically create and manage secrets across all clusters in a Federation, automatically ensuring that these are kept globally consistent and up-to-date, even if some clusters are offline when the original updates are applied.

Federated Namespaces are similar to the traditional Kubernetes Namespaces providing the same functionality. Creating them in the Federation control plane ensures that they are synchronized across all the clusters in Federation.

Federated Events are similar to the traditional Kubernetes Events providing the same functionality. Federation Events are stored only in Federation control plane and are not passed on to the underlying kubernetes clusters.

Let’s walk through how all this stuff works. We’re going to provision 3 clusters per region, spanning 3 continents (Europe, North America and Asia). 



The next step is to federate these clusters. Kelsey Hightower developed a tutorial for setting up a Kubernetes Cluster Federation. Follow the tutorial to configure a Cluster Federation with clusters in 3 zones in each of the 3 GCP regions, us-central1, europe-west1 and asia-east1. For the purpose of this blog post, we’ll provision the Federation Control Plane in the us-central1-b zone. Note that more highly available, multi-cluster deployments are also available, but not used here in the interests of simplicity.

The rest of the blog post assumes that you have a running Kubernetes Cluster Federation provisioned. 

Let’s verify that we have 9 clusters in 3 regions running.

$ kubectl --context=federation-cluster get clusters

NAME              STATUS    AGE
gce-asia-east1-a     Ready     17m
gce-asia-east1-b     Ready     15m
gce-asia-east1-c     Ready     10m
gce-europe-west1-b   Ready     7m
gce-europe-west1-c   Ready     7m
gce-europe-west1-d   Ready     4m
gce-us-central1-a    Ready     1m
gce-us-central1-b    Ready     53s
gce-us-central1-c    Ready     39s

You can download the source used in this blog post here. The source consists of the following files:

  • configmaps/zonefetch.yaml - retrieves the zone from the instance metadata server and concatenates into volume mount path
  • replicasets/nginx-rs.yaml - deploys a Pod consisting of an nginx and busybox container
  • ingress/ingress.yaml - creates a load balancer with a global VIP  that distributes requests to the closest nginx backend
  • services/nginx.yaml - exposes the nginx backend as an external service

In our example, we’ll be deploying the service and ingress object using the federated control plane. The ConfigMap object isn’t currently supported by Federation, so we’ll be deploying it manually in each of the underlying Federation clusters. Our cluster deployment will look as follows:

We’re going to deploy a Service that is sharded across our 9 clusters. The backend deployment will consist of a Pod with 2 containers:
  • busybox container that fetches the zone and outputs an HTML with the zone embedded in it into a Pod volume mount path
  • nginx container that reads from that Pod volume mount path and serves an HTML containing the zone it’s running in

Let’s start by creating a federated service object in the federation-cluster context.

$ kubectl --context=federation-cluster create -f services/nginx.yaml

It will take a few minutes for the service to propagate across the 9 clusters. 


$ kubectl --context=federation-cluster describe services nginx

Name:                   nginx
Namespace:              default
Labels:                 app=nginx
Selector:               app=nginx
Type:                   LoadBalancer
IP:
LoadBalancer Ingress:   108.59.xx.xxx, 104.199.xxx.xxx, ...
Port:                   http    80/TCP
NodePort:               http    30061/TCP
Endpoints:              <none>
Session Affinity:       None

Let’s now create a Federated Ingress. Federated Ingresses are created in much that same way as traditional Kubernetes Ingresses: by making an API call which specifies the desired properties of your logical ingress point. In the case of Federated Ingress, this API call is directed to the Federation API endpoint, rather than a Kubernetes cluster API endpoint. The API for Federated Ingress is 100% compatible with the API for traditional Kubernetes Services.


$ cat ingress/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: nginx
spec:
 backend:
   serviceName: nginx
   servicePort: 80


$ kubectl --context=federation-cluster create -f ingress/ingress.yaml
ingress "nginx" created

Once created, the Federated Ingress controller automatically:
  1. creates matching Kubernetes Ingress objects in every cluster underlying your Cluster Federation
  2. ensures that all of these in-cluster ingress objects share the same logical global L7 (i.e. HTTP(S)) load balancer and IP address
  3. monitors the health and capacity of the service “shards” (i.e. your Pods) behind this ingress in each cluster
  4. ensures that all client connections are routed to an appropriate healthy backend service endpoint at all times, even in the event of Pod, cluster, availability zone or regional outages
We can verify the ingress objects are matching in the underlying clusters. Notice the ingress IP addresses for all 9 clusters is the same.


$ for c in $(kubectl config view -o jsonpath='{.contexts[*].name}'); do kubectl --context=$c get ingress; done
NAME      HOSTS     ADDRESS   PORTS     AGE
nginx     *                   80        1h
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        40m
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        1h
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        26m
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        1h
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        25m
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        38m
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        3m
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        57m
NAME      HOSTS     ADDRESS          PORTS     AGE
nginx     *         130.211.40.xxx   80        56m

Note that in the case of Google Cloud Platform, the logical L7 load balancer is not a single physical device (which would present both a single point of failure, and a single global network routing choke point), but rather a truly global, highly available load balancing managed service, globally reachable via a single, static IP address.

Clients inside your federated Kubernetes clusters (i.e. Pods) will be automatically routed to the cluster-local shard of the Federated Service backing the Ingress in their cluster if it exists and is healthy, or the closest healthy shard in a different cluster if it does not. Note that this involves a network trip to the HTTP(S) load balancer, which resides outside your local Kubernetes cluster but inside the same GCP region.

The next step is to schedule the service backends. Let’s first create the ConfigMap in each cluster in the Federation. 

We do this by submitting the ConfigMap to each cluster in the Federation.


$ for c in $(kubectl config view -o jsonpath='{.contexts[*].name}'); do kubectl --context=$c create -f configmaps/zonefetch.yaml; done

Let’s have a quick peek at our Replica Set:


$ cat replicasets/nginx-rs.yaml

apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
 name: nginx
 labels:
   app: nginx
   type: demo
spec:
 replicas: 9
 template:
   metadata:
     labels:
       app: nginx
   spec:
     containers:
     - image: nginx
       name: frontend
       ports:
         - containerPort: 80
       volumeMounts:
       - name: html-dir
         mountPath: /usr/share/nginx/html
     - image: busybox
       name: zone-fetcher
       command:
         - "/bin/sh"
         - "-c"
         - "/zonefetch/zonefetch.sh"
       volumeMounts:
       - name: zone-fetch
         mountPath: /zonefetch
       - name: html-dir
         mountPath: /usr/share/nginx/html
     volumes:
       - name: zone-fetch
         configMap:
           defaultMode: 0777
           name: zone-fetch
       - name: html-dir
         emptyDir:
           medium: ""

The Replica Set consists of 9 replicas, spread evenly across 9 clusters within the Cluster Federation. Annotations can also be used to control which clusters Pods are scheduled to. This is accomplished by adding annotations to the Replica Set spec, as follows:


apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
 name: nginx-us
 annotations:
   federation.kubernetes.io/replica-set-preferences: |
       {
           "rebalance": true,
           "clusters": {
               "gce-us-central1-a": {
                   "minReplicas": 2,
                   "maxReplicas": 4,
                   "weight": 1
               },
               "gce-us-central10b": {
                   "minReplicas": 2,
                   "maxReplicas": 4,
                   "weight": 1
               }
           }
       }

For the purpose of our demo, we’ll keep things simple and spread our Pods evenly across the Cluster Federation.

Let’s create the federated Replica Set:


$ kubectl --context=federation-cluster create -f replicasets/nginx-rs.yaml

Verify the Replica Sets and Pods were created in each cluster:


$ for c in $(kubectl config view -o jsonpath='{.contexts[*].name}'); do kubectl --context=$c get rs; done
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         42s
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         14m
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         45s
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         46s
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         47s
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         48s
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         49s
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         49s
NAME      DESIRED   CURRENT   READY     AGE
nginx     1         1         1         49s

$ for c in $(kubectl config view -o jsonpath='{.contexts[*].name}'); do kubectl --context=$c get po; done
NAME          READY     STATUS    RESTARTS   AGE
nginx-ph8zx   2/2       Running   0          25s
NAME          READY     STATUS    RESTARTS   AGE
nginx-sbi5b   2/2       Running   0          27s
NAME          READY     STATUS    RESTARTS   AGE
nginx-pf2dr   2/2       Running   0          28s
NAME          READY     STATUS    RESTARTS   AGE
nginx-imymt   2/2       Running   0          30s
NAME          READY     STATUS    RESTARTS   AGE
nginx-9cd5m   2/2       Running   0          31s
NAME          READY     STATUS    RESTARTS   AGE
nginx-vxlx4   2/2       Running   0          33s
NAME          READY     STATUS    RESTARTS   AGE
nginx-itagl   2/2       Running   0          33s
NAME          READY     STATUS    RESTARTS   AGE
nginx-u7uyn   2/2       Running   0          33s
NAME          READY     STATUS    RESTARTS   AGE
nginx-i0jh6   2/2       Running   0          34s

Below is an illustration of how the nginx service and associated ingress deployed. To summarize, we have a global VIP (130.211.23.176) exposed using a Global L7 load balancer that forwards requests to the closest cluster with available capacity.

To test this out, we’re going to spin up 2 Google Cloud Engine (GCE) instances, one in us-west1-b and the other in asia-east1-a. All client requests are automatically routed, via the shortest network path, to a healthy Pod in the closest cluster to the origin of the request. So for example, HTTP(S) requests from Asia will be routed directly to the closest cluster in Asia that has available capacity. If there are no such clusters in Asia, the request will be routed to the next closest cluster (in this case the U.S.). This works irrespective of whether the requests originate from a GCE instance or anywhere else on the internet. We only use a GCE instance for simplicity in the demo.


 

We can SSH directly into the VMs using the Cloud Console or by issuing a gcloud SSH command. 


$ gcloud compute ssh test-instance-asia --zone asia-east1-a
-----
user@test-instance-asia:~$ curl 130.211.40.186
<!DOCTYPE html>
<html>
<head>
<title>Welcome to the global site!</title>
</head>
<body>
<h1>Welcome to the global site! You are being served from asia-east1-b</h1>
<p>Congratulations!</p>

user@test-instance-asia:~$ exit
----

$ gcloud compute ssh test-instance-us --zone us-west1-b
----
user@test-instance-us:~$ curl 130.211.40.186
<!DOCTYPE html>
<html>
<head>
<title>Welcome to the global site!</title>
</head>
<body>
<h1>Welcome to the global site! You are being served from us-central1-b</h1>
<p>Congratulations!</p>

----

Federations of Kubernetes Clusters can include clusters running in different cloud providers (e.g. GCP, AWS), and on-premises (e.g. on OpenStack). However, in Kubernetes 1.4, Federated Ingress is only supported across Google Cloud Platform clusters. In future versions we intend to support hybrid cloud Ingress-based deployments.

To summarize, we walked through leveraging the Kubernetes 1.4 Federated Ingress alpha feature to deploy a multi-homed service behind a global load balancer. External clients point to a single IP address and are sent to the closest cluster with usable capacity in any region, zone of the Federation, providing higher levels of availability without sacrificing latency or ease of operation.

We'd love to hear feedback on Kubernetes Cross Cluster Services. To join the community: