Kubernetes Networking Best Practices (Gilad David Maayan)

What Is Kubernetes Networking?

Kubernetes is an open source platform for managing and automating the deployment, scheduling, monitoring, maintenance, and operation of application containers across clusters of machines. It makes it possible to move workloads between private, public and hybrid clouds, packaging software applications with all the necessary infrastructure, and enabling rapid deployment of new versions.

Kubernetes networking enables communication between cluster components such as containers, pods, nodes, and the applications running within them. Kubernetes uses a flat network architecture—it does not require mapping of host ports to container ports. The Kubernetes platform provides a way to run distributed systems, by sharing physical machines between applications without dynamically allocating ports.

How Kubernetes Networking Works

Kubernetes reduces costs using a flat networking structure that eliminates port mapping between hosts and containers.

Kubernetes networks connect containers, pods, and nodes. A container works as a lightweight VM sharing network resources. A pod is a small deployment unit that groups containers that share networking and storage resources (all containers in a pod share the same IP address). A node is a virtual or physical machine that groups multiple pods (a combination of nodes is a cluster).

Kubernetes deployments typically have networking variations to consider, including the following.

Container-to-Container

The containers residing on a pod share the same network namespace, stored in the “Pause” container alongside the shared resources. Different containers within a pod should have separate ports to enable communication through port and localhost numbers.

Pod-to-Pod

Pod-to-pod networking is possible across and within nodes. Individual nodes have classless inter-domain routing (CIDR) blocks that define the IP addresses for their pods. Pods can communicate using a veth pair or virtual ethernet device (VED) to connect. A veth pair is a coupled network interface spread across two namespaces.

Pod-to-Service

Kubernetes enables the dynamic replacement of pods, meaning pods don’t have durable IP addresses by default. Kubernetes maintains inter-pod communication using services. A service manages pod states and abstracts pod addresses using a cluster IP, enabling pod IP address tracking. It prevents the creation and deletion of pods from impacting communications and can act as a load balancer.

Internet-to-Service

Most deployments require networking between Kubernetes services and the Internet (for external and internal applications). Two traffic control techniques use allowlist or denylist policies to enable external access:

● Egress—routes traffic from nodes to outside connections, often via a gateway attached to the VPC. The gateway maps IP addresses using NAT (network address translation) but cannot map individual pods on a node. Kubernetes finalizes communications with cluster IPs and IP tables.

● Ingress—affects communications to services from an external client. It involves rules that define the permitted and denied connections with internal services.

Kubernetes Networking Best Practice

Prioritize VPC Design

Designing the virtual private cloud (VPC) should be an early step in the Kubernetes networking setup. Some design choices at the organizational level are not easily reversible later. The VPC network topology’s design should be simple to help ensure the architecture is easily understood, manageable, and reliable.

If the naming conventions are intuitive and consistent, end-users and administrators will more easily understand the location and purpose of network resources, helping them differentiate similar resources.

A conventional enterprise network typically contains many separate address ranges. Reasons include identifying or isolating applications or keeping the broadcast domain small. However, applications should be grouped into a smaller number of subnets with large address ranges, making collections of similar applications more manageable.

A shared VPC allows businesses with multiple development teams to collaborate. It extends the simplicity of a unified VPC network architecture across multiple groups. One simple approach is deploying the shared VPC network with the host project and attaching a service project for each host project.

Use GitOps

A Git-based workflow helps ensure successful Kubernetes deployments by streaming the team’s workflow processes. This workflow uses continuous integration and delivery (CI/CD) pipelines to enable automation and increase the speed and efficiency of application deployments.

CI/CD methods also help provide a deployment audit trail. The Git repository should be the single truth source to automate all pipeline steps and enable unified Kubernetes cluster management.

Use Liveness and Readiness Probes

These probes are deployment health checks. Readiness probes ensure that pods are functional before letting Kubernetes direct a load to a given pod. When a pod is not ready, Kubernetes will redirect requests away from the service until a probe confirms that the pod is running.

Liveness probes verify if an application is running. They attempt to trigger a response from a pod to verify its health. If the pod doesn’t respond, it indicates the application is not running on that pod. Liveness probes can launch new pods and start applications on them when a check fails.

Implement Network Security

Kubernetes clusters require protection from external attacks. The infrastructure outside a cluster must also be protected from potentially compromised elements within the cluster (i.e., internal threats).

Here are some best practices to ensure Kubernetes network security and securely integrate clusters with the surrounding infrastructure:

● Restrict Internet access to clusters—in some cases, clusters must be Internet-accessible, either indirectly through a load balancer or directly to a public node IP address. However, many situations don’t require Internet access to clusters. Where possible, it is best to block Internet access to reduce the risk of attack. When clusters require Internet accessibility, it is best to expose the smallest possible number of pods and services, for example, using an ingress controller.

● Restrict inter-cluster host and workload communication—internal network policies can limit communications between clusters and their surrounding infrastructure. These policies are platform-agnostic and workload-aware.

● Restrict inter-cluster traffic—a perimeter firewall or cloud equivalent, such as a security group, can help secure traffic to and from clusters. However, these solutions are not workload-aware and may have limited granularity. These controls remain an important part of security-in-depth strategies.

● Restrict internal cluster traffic—it is important to implement zero trust network policies to block all traffic within a cluster that is not explicitly permitted. This approach helps reduce the impact of cluster breaches by preventing attackers from moving laterally within the cluster and compromising other sensitive components.

Conclusion

In this article, I explained the basics of Kubernetes networking and provided several best practices that can help you efficiently manage communication in your Kubernetes clusters:

● Prioritize VPC design - use cloud VPCs to manage segmentation of your Kubernetes networks.

● Use GitOps - streamline deployments and create a solid audit trail by storing environments as code in a Git repository.

● Use liveness and readiness probes - leverage automated Kubernetes health checks to ensure reliable communication.

● Implement network security - ensure ingress/egress communication as well as communication between pods and containers is secured.

I hope this will be useful as you level-up your Kubernetes networking skills.