design.md

Design documentation

Goals

Hepto is yet another kubernetes distribution. It is driven by the following goals:

• working out of the box over WAN without compromising security
• offering a much opinionated kubernetes experience
• hiding much of the complexity of setting up kubernetes
• running with the minimum resource footprint (mostly memory)
• bundling vanilla upstream with as few to no fork if possible
• compiling to a (true) single binary on both amd64 and arm64

Rationale

Instead of optimizing for disk size or embedding all kinds of batteries, hepto focuses on the minimal feature set for a kubernetes distribution. Here are so dos and donts:

• do optimize for lower resident memory, this implies:
  • do not unpack multiple static binaries at runtime, which do not map well in the page cache
  • do not optimize for size by compressing part of the binary and inflating at runtime
  • do not embed external binaries and assume Internet access to load further resources
• do work out dependency issues, this implies:
  • do update packages as much as possible
  • do depend on Go modules from vanilla upstreams
  • do not depend on our own forks, if possible not on any fork
  • do not pin dependencies unless required for compatibility
  • do write down the rationale for every quirk
  • do upstream as many fixes as possible to upstreams
• do offer an opinionated experience:
  • do explicit choices and their rationale
  • do hide complexity any time a user choice is not required
  • do not offer too many choices to the user
  • do not embed optional batteries

General architecture

Components

Hepto is distributed as a single-binary, which requires that we compile a single Go program, and do not embed any large resource. Details are provided under "Software architecture".

Hepto includes the apiserver and related controllers, backend etcd storage, kubelet, containerd and all related components (shim, runc, etc.), all components are enabled based on the node role.

Hepto is bootstrapped based on a SWIM gossip cluster (using Hashicorp memberlist), protected by a secret cluster-wide key that encrypts and authicates every bootstrapping message. Every node is a SWIM node, which means that any node can be used to discover the entire cluster. Nodes with stable IP address are preferred and called anchors.

Hepto nodes are connected by a mesh Wireguard VPN that protects everything from the underlying network. Wireguard configuration (both routes and public keys) are distributed using the gossip mesh.

A PKI is embedded, which is managed on the master node and over the gossip mesh. The shared gossip state contains certificates authorities published by the master node, and a map of nodes to certificates (CSR, then signed certificate).

Hepto only supports IPv6, which requires all components, bootstrapping, VPN, kubernetes API, etc. to support IPv6.

Containerization

Before running its components, a node always self-containerizes. It creates a set of namespaces (pid, mount, ipc, networking) and associated cgroups, then runs itself inside the container.

The container is configured with an ipvlan interface based on the host public interface.

The container bind mounts storage for certificates, etcd data, containerd pod storage, and any additional storage for local physical volues,

Software architecture

Components

The project is based on homemade libraries:

• go.acides.og/selfcontain for container management, based on runc for running the current binary itself inside a container
• go.acides.org/sml for bootstrapping, that wraps Hashicorp memberlist as a high-level library
• go.acides.org/daeman for component management, which provides dependency management with goroutines
• go.acides.org/pekahi for certificate management, based on the standard library for general PKI use

It is then divided into 3 packages:

• cmd for command-line management, which takes care of basic argument parsing, self-containerization, and single-binary behavior
• services which defines daeman units and their dependencies, which constitute the hepto node, each is documented as godoc
• wg for Wireguard management, which should later by exposed as a reusable library
• utils with various utilities, probably not generic enough to be exposed as a dedicated library

Go dependency management

Building a single binary requires some fine-tuning of dependency versions for compatibility. This was hardly possible before kubernetes 1.26 without explicitely forking some packages.

All pinned dependencies are declared as replace in go.mod with associated explaination. Their number should be reduced to the bare minimum, and all other dependencies should be updated based on go get -u.