Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes.
Kilo connects nodes in a cluster by providing an encrypted layer 3 network that can span across data centers and public clouds. By allowing pools of nodes in different locations to communicate securely, Kilo enables the operation of multi-cloud clusters. Kilo's design allows clients to VPN to a cluster in order to securely access services running on the cluster. In addition to creating multi-cloud clusters, Kilo enables the creation of multi-cluster services, i.e. services that span across different Kubernetes clusters.
How it works
Kilo uses WireGuard, a performant and secure VPN, to create a mesh between the different nodes in a cluster.
The Kilo agent,
kg, runs on every node in the cluster, setting up the public and private keys for the VPN as well as the necessary rules to route packets between locations.
Kilo can operate both as a complete, independent networking provider as well as an add-on complimenting the cluster-networking solution currently installed on a cluster. This means that if a cluster uses, for example, Flannel for networking, Kilo can be installed on top to enable pools of nodes in different locations to join; Kilo will take care of the network between locations, while Flannel will take care of the network within locations.
Installing on Kubernetes
Kilo can be installed on any Kubernetes cluster either pre- or post-bring-up.
Step 1: install WireGuard
Kilo requires the WireGuard kernel module on all nodes in the cluster. For most Linux distributions, this can be installed using the system package manager. For Container Linux, WireGuard can be easily installed using a DaemonSet:
Step 2: open WireGuard port
The nodes in the mesh will require an open UDP port in order to communicate. By default, Kilo uses UDP port 51820.
Step 3: specify topology
By default, Kilo creates a mesh between the different logical locations in the cluster, e.g. data-centers, cloud providers, etc.
For this, Kilo needs to know which groups of nodes are in each location.
If the cluster does not automatically set the topology.kubernetes.io/region node label, then the kilo.squat.ai/location annotation can be used.
For example, the following snippet could be used to annotate all nodes with
GCP in the name:
Kilo allows the topology of the encrypted network to be completely customized. See the topology docs for more details.
Step 4: ensure nodes have public IP
At least one node in each location must have an IP address that is routable from the other locations. If the locations are in different clouds or private networks, then this must be a public IP address. If this IP address is not automatically configured on the node's Ethernet device, it can be manually specified using the kilo.squat.ai/force-endpoint annotation.
Step 5: install Kilo!
Kilo can be installed by deploying a DaemonSet to the cluster.
To run Kilo on kubeadm:
To run Kilo on bootkube:
To run Kilo on Typhoon:
To run Kilo on k3s:
Administrators of existing clusters who do not want to swap out the existing networking solution can run Kilo in add-on mode. In this mode, Kilo will add advanced features to the cluster, such as VPN and multi-cluster services, while delegating CNI management and local networking to the cluster's current networking provider. Kilo currently supports running on top of Flannel.
For example, to run Kilo on a Typhoon cluster running Flannel:
Kilo also enables peers outside of a Kubernetes cluster to connect to the VPN, allowing cluster applications to securely access external services and permitting developers and support to securely debug cluster resources. In order to declare a peer, start by defining a Kilo peer resource:
This configuration can then be applied to a local WireGuard interface, e.g.
wg0, to give it access to the cluster with the help of the
A logical application of Kilo's VPN is to connect two different Kubernetes clusters.
This allows workloads running in one cluster to access services running in another.
For example, if
cluster1 is running a Kubernetes Service that we need to access from Pods running in
cluster2, we could do the following:
important-service can be used on
cluster2 just like any other Kubernetes Service.
The topology and configuration of a Kilo network can be analyzed using the
kgctl command line tool.
For example, the
graph command can be used to generate a graph of the network in Graphviz format: