How Does The Kubernetes Networking Work? : Part 3

Level Up How Does The Kubernetes Networking Work? : Part 2 Article Image
How Does The Kubernetes Networking Work? : Part 2
August 19, 2018
Show all

How Does The Kubernetes Networking Work? : Part 3

Level Up How Does The Kubernetes Networking Work? : Part 3 Article Image

Hi there, welcome to the third and last part of the Kubernetes Networking article series. The first and second part are already uploaded. Newcomers, please go through the first and second part for a better understanding of Kubernetes Networking. Trust me, it won’t take much time, but will make a big difference!

As always, let us recap some important things covered in our previous articles. So in the first part, we discussed that the network enables pods for connectivity. The pods connect with each other as well with nodes in a Kubernetes cluster.  In the second part, we discussed service networking. How a service network facilitates load balancing for pods. Due to this, the clients within the cluster are able to communicate with the service network in an efficient and a reliable manner. In this article, we will use the concepts from the first and second articles. I will show you how the clients that reside outside of the cluster connect to the pods within, using the same network. This article uses most of the concepts from the first and second parts. So once again, I request to refer it once.

Before we begin the actual content, allow me to say some good words regarding Kubernetes.

It is a rapidly maturing platform. Kubernetes is improving and is also being used variedly. Why am I telling this? Because I am proud of Kubernetes. Follow the Kubecon events and you will come to know how valuable is Kubernetes! Most of the Kubernetes architecture is plug-able, including networking! For example, Google’s Kubernetes Engine is simply amazing. I haven’t got a chance to work with Amazon’s Elastic Kubernetes Service, but I believe it is good as well.

Kubernetes has its own standard way of dealing with networking issues. It is unique. Keep these concepts in mind while you think about the alternatives like unified service meshes, etc. Alright, folks, it is now time for some action.

So let us begin with Ingress.

Level Up How Does The Kubernetes Networking Work? : Part 3 Article Image

Firstly, tell me, “Are routing and load balancing equal?”

Of course not, they aren’t equal. In our previous post, we created a deployment with some pods. There was a service that was assigned an IP address. It was called the “Cluster IP”. To the Cluster IP, requests intended for the pods were sent. Can you recollect? I hope so! We will continue discussing the same example. The cluster IP address 10.3.241.152 is in such an IP address range that is different from the pod network. Also, it is different from the network that the nodes are on. This address space is known as the services network.

I had mentioned this in the previous article. It doesn’t require a name as such because it purely works as per the routing rules. It doesn’t have connected devices. Please have a look at the image shared. We see how a network is implemented by a Kubernetes component, known as the Kube-proxy. It collaborates with a Linux Kernel module known as the Netfilter for receiving and rerouting the traffic sent to the cluster IP. So why this rerouting?? The traffic should be sent to a healthy pod.

Level Up How Does The Kubernetes Networking Work? : Part 3 Article Image

Image Source: https://cdn-images-1.medium.com/max/800/1*Ow5A6_zjjwdKkf2Yi1KgYw.png

Till now, we have talked about connections, requests, and traffic. To understand how and more importantly why Ingress works in this way, we need to be more specific.

Connections and requests are managed by OSI layer 4 (TCP) or by layer 7 (Http, RPC, etc). The Netfilter rules are basically routing rules. They operate on the IP packets at layer 3. How are the routing decisions made? All the routers including Netfilter make routing decisions as per the information in the packet. The main information is, where the packet is coming from and where is it going? Now, let us focus on the behavior based on the packet information. We can divide it into three layers. First of all, each packet is destined for the service at 10.3.241.152:80. It arrives at a node’s eth0 interface and is processed through the Netfilter. The rules are matched with the ones established for the service and is then forwarded to some IP address.

So which is this IP address? It is the address of some healthy pod.

The same routing infrastructure is used by external clients to call into our pods. This means that the external clients call the cluster IP and port. Why? It is the front end for all the machinery. This machinery makes it possible for us not to care where a pod is running at any given time. The cluster IP is only reachable from a node’s ethernet interface. Outside the cluster, no one knows what to do with addresses in that range. So the question is, how can we forward the traffic in such a scenario? From a publicly visible IP endpoint to an IP that is only reachable once the packet is already on a node? What is the solution?

Maybe, you can examine the Netfilter rules using the IP Tables utility. Let me tell you that by doing this, you will discover something surprising. The rules for the example service are not limited to a particular origin network! So now if there is a packet from anywhere arriving on the node’s ethernet interface with a destination address 10.3.241.152:80, it is going to get matched and routed to a pod. So is it legitimate to give clients that cluster IP? Can we assign it some friendly domain name? Maybe then, add a route to get those packets to one of the nodes?

Please have a look at the image below:

Level Up How Does The Kubernetes Networking Work? : Part 3 Article Image

Image Source: https://cdn-images-1.medium.com/max/800/1*skurXk737KHhzbXPbV-Xcg.png

Let us trace the procedure. First of all, clients call the cluster IP. Then, the packets follow a route to the node. The packets are then forwarded to a pod. At this instance, you might be tempted for completing the process. But this isn’t a good idea.

The consequences are serious for this.

Why? The nodes are ephemeral like pods. But they aren’t as ephemeral as the pods. They can be migrated to some new VM (Virtual Machine). There, the clusters can scale up and down, etc. However, remember that the routers operate on the layer 3 packets. They don’t know the difference between health and unhealthy services. Yes, but they do expect that the next hop in the route is stable and available. Unfortunately, if the node becomes unreachable, the route will break and stay broken for quite a good time. This happens a lot. Even if the route is durable, all the external traffic will pass through a single node. This is not an optimum solution.

But we also need to bring client traffic. So we do so in a manner that doesn’t depend on the health of a particular node in the cluster. Without an active management, routing is of no use in this case. It won’t lead us to a reliable solution. So what is active management? For example, the role of Kube-proxy in managing Netfilter is an active management. Extending the Kubernetes responsibilities to the management of some external router doesn’t make sense to the designers. On top of that, we already have proven tools for distributing client traffic to a set of machines. They are known as load balancers and they are also our solution for the Kubernetes Ingress. It works in a durable manner.

So we need a public IP to use a load balancer. It will help distribute the client traffic to the nodes in a cluster. The clients can connect to the public IP. We also need addresses on the nodes to which the load balancer forwards the requests. The service network (Cluster IP) alone is not enough to easily create a stable static route between the gateway router and the nodes. The other available addresses are on the network to which the ethernet addresses are connected. In our example, the network address is 10.100.0.0/24. The gateway router is smart enough and knows how to get packets to these interfaces. The connections sent from the load balancer to the router will get to the right place. But suppose if the client wants to connect to our service on port 80, we can’t simply send packets to that port on the nodes’ interfaces.

Here is the next image:

Level Up How Does The Kubernetes Networking Work? : Part 3 Article Image

Image Source: https://cdn-images-1.medium.com/max/800/1*I03T4FnCjlPmHNXhao4arg.png

But why does this fail? If you notice, there is no process listening on 10.100.0.3:80. The Netfilter rules intercept our request and direct it to a pod that doesn’t match the destination address. The only thing that matches is the cluster IP on the service network at 10.3.241.152:80. Due to this, the packets aren’t delivered when they arrive on the interface. The Kernel responds with “ECONNREFUSED”.  So the situation is now complicated and confusing.

Firstly, the network that Netfilter has set up for forwarding the packets is not easily routable from the gateway to the nodes. Secondly, the network that is easily routable is not the one to which Netfilter forwards the packets. So everything is a bit difficult at the moment. But there is a solution! What? How about a bridge between these networks? Oh yes, Kubernetes can help us with something known as the NodePort. So what is it? Let us find out!

The savior NodePort Services

The example service created by us didn’t specify a type. So we consider the default type ClusterIP. But there are other types of services that create additional capabilities. The one that is important to us is type NodePort. Have a look at this example

kind: Service
apiVersion: v1
metadata:
  name: service-demo
spec:
  type: NodePort
  selector:
    app: service_demo_pod
  ports:
  - port: 80
    targetPort: http

In this code, a service of type NodePort is a ClusterIP service. The name of the service is service-demo. This service has an additional capability of reaching the IP address of the node as well as the one at the assigned cluster IP. How is this accomplished? Kubernetes creates a NodePort service known as the Kube-proxy. It allocates a port in the range 30000-32767 and opens the port on the eth0 interface of every node. This is the reason why it is known as NodePort. The connections from this pod are directed to the service’s cluster. Let us do something, we create the above service and then run Kubectl get svc service-demo. It will help us see the allocated NodePort.

So here is the command:

$ kubectl get svc service-demo

NAME:                   CLUSTER-IP       EXTERNAL-IP      PORT(S)

service-demo      10.3.241.152         <none>                    80:32213/TCP

In the example, our service was allocated to the NodePort 32213. So we can connect to the service on either node in the cluster, at 10.100.0.2:32213 or 10.100.0.3:32213. The traffic will get forwarded to the service. So now we have a complete pipeline for load balancing the external client requests to all the nodes in the cluster.

Level Up How Does The Kubernetes Networking Work? : Part 3 Article Image

Image Source: https://cdn-images-1.medium.com/max/800/1*Uo_wGCIlFopJZbf6THu0OQ.png

From the image shared above, we can see that the client connects to the load balancer through a public IP address. The load balancer selects a node and connects to it at the address 10.100.0.3:32213. The Kube-proxy receives this connection and forwards it to the service at the cluster IP 10.3.241.152:80. At this point, the request matches the Netfilter rules and gets redirected to the server pod on 10.0.2.2:8080. It appears to be a bit complicated, but there aren’t straightforward solutions.  

This mechanism also has some issues.

When you use the NodePorts, it indirectly exposes your service to the clients on some non-standard port. Although, the load balancer can expose the usual port and mask the NodePort. But under some scenarios like internal load balancing on Google cloud, the NodePort is propagated upstream. Also, NodePorts are a limited resource. There are also some restrictions regarding the preservation of source IPs in requests.

So NodePorts aren’t the complete solution. They need some kind of a load balancer in front of the cluster. So this need gave rise to two different ways in which the load balancer configuration is specified from within the Kubernetes. So let us switch over to it.

The advent of LoadBalancer Services and Ingress Resources

The external traffic ends up at the cluster through a NodePort. We just discussed it in the previous section. The platform designers had the choice to stop there itself. But this will create problems for you! What about the public IPs and load balancers? So in environments that support API-driven configuration of networking resources, Kubernetes makes it possible to define everything in one place. LoadBalancer is a simple service. It has all the capabilities of a NodePort service and can build out good ingress path. I assume that you are running the environment like GCP or AWS, the one that supports API-driven configuration.

Have a look at the code:

kind: Service
apiVersion: v1
metadata:
name: service-demo
spec:
type: LoadBalancer
selector:
app: service_demo_pod
ports:
- port: 80
targetPort: http

Suppose we delete and recreate the example service on Google Kubernetes Engine. What will the Kubectl command result in?

Let us try it out.

$ kubectl get svc service-demo

NAME           CLUSTER-IP        EXTERNAL-IP PORT(S)                AGE

openvpn       10.3.241.52           35.184.97.156 80:32213/TCP        5m

The allocation of an external IP can take some time, I mean minutes. It depends on the number of resources. On GCP, the requirements are different. First of all, create an external IP, a forwarding rule, a target proxy, a backend service, and possibly an instance group. After IP allocation, the service can be connected using all of the above. The domain name is also assigned and distributed to the clients.

But LoadBalancer type services also have some limitations. One can’t configure the lb to terminate the https traffic. Virtual hosts and path-based routing isn’t possible. So a single load balancer isn’t enough to proxy multiple services. All this led to the addition in version 1.2 of a separate Kubernetes resource for configuring load balancers. It is known as the Ingress. While LoadBalancer extends a single service to support external clients, an Ingress offers a separate resource to configure a load balancer flexibly. The Ingress API supports TLS termination, virtual hosts, and path-based routing. Due to this, it can set up a load balancer for handling multiple backend services.

The implementation follows a basic Kubernetes pattern. There are a resource type and a controller for managing that type. Here, the resource is Ingress. It includes a request for the networking resources.

Let us have a look at an Ingress resource for our test service.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-ingress
annotations:
kubernetes.io/ingress.class: "gce"
spec:
tls:
- secretName: my-ssl-secret
rules:
- host: testhost.com
http:
paths:
- path: /*
backend:
serviceName: service-demo
servicePort: 80

The Ingress controller satisfies the requests by driving resources in the necessary state environment. With Ingress, you can create your services as type NodePort. Once you create, the Ingress controller is responsible for receiving and channeling traffic to the nodes. But please understand that mixing Ingress resources and LoadBalancer service can cause subtle issues. It depends on the environment.

The Final Touch with HostPort and Host Network

HostPort is a property of a container. It is declared in the ContainerPort structure. If it is set to a given port number, it causes the port to be opened on the node. It is forwarded directly to the container. There is no proxying. The port is only open to the nodes where the container is running.

The next property is known as the HostNetwork of a pod. If it is set to true, it has similar effects as the network=host argument to docker run. All the containers in the pod use the node’s network namespace. They have access to eth0 and open ports directly. Most probably, you are never going to need this. If you already have a use case for this, then I salute you! It means that you are already a Kubernetes contributor.

The Conclusion

So here we are at the end of the Kubernetes Article series. This Kubernetes platform is amazing and I enjoy every bit of it. I have learned a lot by practicing, working, and teaching Kubernetes. Personally, I think that Kubernetes is a revolution. It is making things possible to reliably manage and interconnect huge container fleets.

Kubernetes has a great future and is certainly maturing every day. With this article series, I hope that everyone gets to know more about Kubernetes. Trust me, the more you know, the more you will fall in love with Kubernetes! Thank you for supporting and reading these articles.

Here is a link to the Kubernetes Bible, start learning from the basics now :

The Kubernetes Bible for Beginners & Developers

And here is the link to the Kubernetes Video Tutorial

Learn Kubernetes from a DevOps guru Video Course on Kubernetes

James Lee
James Lee
James Lee is a passionate software wizard working at one of the top Silicon Valley-based startups specializing in big data analysis. In the past, he has worked on big companies such as Google and Amazon In his day job, he works with big data technologies such as Cassandra and ElasticSearch, and he is an absolute Docker technology geek and IntelliJ IDEA lover with strong focus on efficiency and simplicity.

2 Comments

  1. David S says:

    Good one. I was waiting for the third part. Thank you for the Networking Series James.

  2. Daniel says:

    Thanks James! That was excellent.

Leave a Reply

Your email address will not be published.

LEARN HOW TO GET STARTED WITH DEVOPS

get free access to this free guide, downloaded over 200,00 times !

You have Successfully Subscribed!

Level Up Big Data Pdf Book

LEARN HOW TO GET STARTED WITH BIG DATA

get free access to this free guide, downloaded over 200,00 times !

You have Successfully Subscribed!