Hi there, welcome to the third and last part of the Kubernetes Networking article series. The first and second part are already uploaded. Newcomers, please go through the first and second part for a better understanding of Kubernetes Networking. Trust me, it won’t take much time, but will make a big difference!
As always, let us recap some important things covered in our previous articles. So in the first part, we discussed that the network enables pods for connectivity. The pods connect with each other as well with nodes in a Kubernetes cluster. In the second part, we discussed service networking. How a service network facilitates load balancing for pods. Due to this, the clients within the cluster are able to communicate with the service network in an efficient and a reliable manner. In this article, we will use the concepts from the first and second articles. I will show you how the clients that reside outside of the cluster connect to the pods within, using the same network. This article uses most of the concepts from the first and second parts. So once again, I request to refer it once.
It is a rapidly maturing platform. Kubernetes is improving and is also being used variedly. Why am I telling this? Because I am proud of Kubernetes. Follow the Kubecon events and you will come to know how valuable is Kubernetes! Most of the Kubernetes architecture is plug-able, including networking! For example, Google’s Kubernetes Engine is simply amazing. I haven’t got a chance to work with Amazon’s Elastic Kubernetes Service, but I believe it is good as well.
Kubernetes has its own standard way of dealing with networking issues. It is unique. Keep these concepts in mind while you think about the alternatives like unified service meshes, etc. Alright, folks, it is now time for some action.
Of course not, they aren’t equal. In our previous post, we created a deployment with some pods. There was a service that was assigned an IP address. It was called the “Cluster IP”. To the Cluster IP, requests intended for the pods were sent. Can you recollect? I hope so! We will continue discussing the same example. The cluster IP address 10.3.241.152 is in such an IP address range that is different from the pod network. Also, it is different from the network that the nodes are on. This address space is known as the services network.
I had mentioned this in the previous article. It doesn’t require a name as such because it purely works as per the routing rules. It doesn’t have connected devices. Please have a look at the image shared. We see how a network is implemented by a Kubernetes component, known as the Kube-proxy. It collaborates with a Linux Kernel module known as the Netfilter for receiving and rerouting the traffic sent to the cluster IP. So why this rerouting?? The traffic should be sent to a healthy pod.
Connections and requests are managed by OSI layer 4 (TCP) or by layer 7 (Http, RPC, etc). The Netfilter rules are basically routing rules. They operate on the IP packets at layer 3. How are the routing decisions made? All the routers including Netfilter make routing decisions as per the information in the packet. The main information is, where the packet is coming from and where is it going? Now, let us focus on the behavior based on the packet information. We can divide it into three layers. First of all, each packet is destined for the service at 10.3.241.152:80. It arrives at a node’s eth0 interface and is processed through the Netfilter. The rules are matched with the ones established for the service and is then forwarded to some IP address.
The same routing infrastructure is used by external clients to call into our pods. This means that the external clients call the cluster IP and port. Why? It is the front end for all the machinery. This machinery makes it possible for us not to care where a pod is running at any given time. The cluster IP is only reachable from a node’s ethernet interface. Outside the cluster, no one knows what to do with addresses in that range. So the question is, how can we forward the traffic in such a scenario? From a publicly visible IP endpoint to an IP that is only reachable once the packet is already on a node? What is the solution?
Maybe, you can examine the Netfilter rules using the IP Tables utility. Let me tell you that by doing this, you will discover something surprising. The rules for the example service are not limited to a particular origin network! So now if there is a packet from anywhere arriving on the node’s ethernet interface with a destination address 10.3.241.152:80, it is going to get matched and routed to a pod. So is it legitimate to give clients that cluster IP? Can we assign it some friendly domain name? Maybe then, add a route to get those packets to one of the nodes?
Let us trace the procedure. First of all, clients call the cluster IP. Then, the packets follow a route to the node. The packets are then forwarded to a pod. At this instance, you might be tempted for completing the process. But this isn’t a good idea.
Why? The nodes are ephemeral like pods. But they aren’t as ephemeral as the pods. They can be migrated to some new VM (Virtual Machine). There, the clusters can scale up and down, etc. However, remember that the routers operate on the layer 3 packets. They don’t know the difference between health and unhealthy services. Yes, but they do expect that the next hop in the route is stable and available. Unfortunately, if the node becomes unreachable, the route will break and stay broken for quite a good time. This happens a lot. Even if the route is durable, all the external traffic will pass through a single node. This is not an optimum solution.
But we also need to bring client traffic. So we do so in a manner that doesn’t depend on the health of a particular node in the cluster. Without an active management, routing is of no use in this case. It won’t lead us to a reliable solution. So what is active management? For example, the role of Kube-proxy in managing Netfilter is an active management. Extending the Kubernetes responsibilities to the management of some external router doesn’t make sense to the designers. On top of that, we already have proven tools for distributing client traffic to a set of machines. They are known as load balancers and they are also our solution for the Kubernetes Ingress. It works in a durable manner.
So we need a public IP to use a load balancer. It will help distribute the client traffic to the nodes in a cluster. The clients can connect to the public IP. We also need addresses on the nodes to which the load balancer forwards the requests. The service network (Cluster IP) alone is not enough to easily create a stable static route between the gateway router and the nodes. The other available addresses are on the network to which the ethernet addresses are connected. In our example, the network address is 10.100.0.0/24. The gateway router is smart enough and knows how to get packets to these interfaces. The connections sent from the load balancer to the router will get to the right place. But suppose if the client wants to connect to our service on port 80, we can’t simply send packets to that port on the nodes’ interfaces.
But why does this fail? If you notice, there is no process listening on 10.100.0.3:80. The Netfilter rules intercept our request and direct it to a pod that doesn’t match the destination address. The only thing that matches is the cluster IP on the service network at 10.3.241.152:80. Due to this, the packets aren’t delivered when they arrive on the interface. The Kernel responds with “ECONNREFUSED”. So the situation is now complicated and confusing.
Firstly, the network that Netfilter has set up for forwarding the packets is not easily routable from the gateway to the nodes. Secondly, the network that is easily routable is not the one to which Netfilter forwards the packets. So everything is a bit difficult at the moment. But there is a solution! What? How about a bridge between these networks? Oh yes, Kubernetes can help us with something known as the NodePort. So what is it? Let us find out!
The example service created by us didn’t specify a type. So we consider the default type ClusterIP. But there are other types of services that create additional capabilities. The one that is important to us is type NodePort. Have a look at this example
kind: Service apiVersion: v1 metadata: name: service-demo spec: type: NodePort selector: app: service_demo_pod ports: - port: 80 targetPort: http
In this code, a service of type NodePort is a ClusterIP service. The name of the service is service-demo. This service has an additional capability of reaching the IP address of the node as well as the one at the assigned cluster IP. How is this accomplished? Kubernetes creates a NodePort service known as the Kube-proxy. It allocates a port in the range 30000-32767 and opens the port on the eth0 interface of every node. This is the reason why it is known as NodePort. The connections from this pod are directed to the service’s cluster. Let us do something, we create the above service and then run Kubectl get svc service-demo. It will help us see the allocated NodePort.
$ kubectl get svc service-demo
NAME: CLUSTER-IP EXTERNAL-IP PORT(S)
service-demo 10.3.241.152 <none> 80:32213/TCP
From the image shared above, we can see that the client connects to the load balancer through a public IP address. The load balancer selects a node and connects to it at the address 10.100.0.3:32213. The Kube-proxy receives this connection and forwards it to the service at the cluster IP 10.3.241.152:80. At this point, the request matches the Netfilter rules and gets redirected to the server pod on 10.0.2.2:8080. It appears to be a bit complicated, but there aren’t straightforward solutions.
When you use the NodePorts, it indirectly exposes your service to the clients on some non-standard port. Although, the load balancer can expose the usual port and mask the NodePort. But under some scenarios like internal load balancing on Google cloud, the NodePort is propagated upstream. Also, NodePorts are a limited resource. There are also some restrictions regarding the preservation of source IPs in requests.
So NodePorts aren’t the complete solution. They need some kind of a load balancer in front of the cluster. So this need gave rise to two different ways in which the load balancer configuration is specified from within the Kubernetes. So let us switch over to it.
The external traffic ends up at the cluster through a NodePort. We just discussed it in the previous section. The platform designers had the choice to stop there itself. But this will create problems for you! What about the public IPs and load balancers? So in environments that support API-driven configuration of networking resources, Kubernetes makes it possible to define everything in one place. LoadBalancer is a simple service. It has all the capabilities of a NodePort service and can build out good ingress path. I assume that you are running the environment like GCP or AWS, the one that supports API-driven configuration.
kind: Service apiVersion: v1 metadata: name: service-demo spec: type: LoadBalancer selector: app: service_demo_pod ports: - port: 80 targetPort: http
Suppose we delete and recreate the example service on Google Kubernetes Engine. What will the Kubectl command result in?
$ kubectl get svc service-demo
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
openvpn 10.3.241.52 18.104.22.168 80:32213/TCP 5m
The allocation of an external IP can take some time, I mean minutes. It depends on the number of resources. On GCP, the requirements are different. First of all, create an external IP, a forwarding rule, a target proxy, a backend service, and possibly an instance group. After IP allocation, the service can be connected using all of the above. The domain name is also assigned and distributed to the clients.
But LoadBalancer type services also have some limitations. One can’t configure the lb to terminate the https traffic. Virtual hosts and path-based routing isn’t possible. So a single load balancer isn’t enough to proxy multiple services. All this led to the addition in version 1.2 of a separate Kubernetes resource for configuring load balancers. It is known as the Ingress. While LoadBalancer extends a single service to support external clients, an Ingress offers a separate resource to configure a load balancer flexibly. The Ingress API supports TLS termination, virtual hosts, and path-based routing. Due to this, it can set up a load balancer for handling multiple backend services.
The implementation follows a basic Kubernetes pattern. There are a resource type and a controller for managing that type. Here, the resource is Ingress. It includes a request for the networking resources.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: demo-ingress annotations: kubernetes.io/ingress.class: "gce" spec: tls: - secretName: my-ssl-secret rules: - host: testhost.com http: paths: - path: /* backend: serviceName: service-demo servicePort: 80
The Ingress controller satisfies the requests by driving resources in the necessary state environment. With Ingress, you can create your services as type NodePort. Once you create, the Ingress controller is responsible for receiving and channeling traffic to the nodes. But please understand that mixing Ingress resources and LoadBalancer service can cause subtle issues. It depends on the environment.
HostPort is a property of a container. It is declared in the ContainerPort structure. If it is set to a given port number, it causes the port to be opened on the node. It is forwarded directly to the container. There is no proxying. The port is only open to the nodes where the container is running.
The next property is known as the HostNetwork of a pod. If it is set to true, it has similar effects as the network=host argument to docker run. All the containers in the pod use the node’s network namespace. They have access to eth0 and open ports directly. Most probably, you are never going to need this. If you already have a use case for this, then I salute you! It means that you are already a Kubernetes contributor.
So here we are at the end of the Kubernetes Article series. This Kubernetes platform is amazing and I enjoy every bit of it. I have learned a lot by practicing, working, and teaching Kubernetes. Personally, I think that Kubernetes is a revolution. It is making things possible to reliably manage and interconnect huge container fleets.
Kubernetes has a great future and is certainly maturing every day. With this article series, I hope that everyone gets to know more about Kubernetes. Trust me, the more you know, the more you will fall in love with Kubernetes! Thank you for supporting and reading these articles.
Here is a link to the Kubernetes Bible, start learning from the basics now :
And here is the link to the Kubernetes Video Tutorial