Hepto is a kubernetes distribution for geodistributed desployments

This is currently highly experimental and unusable as is.

Goals

Hepto is yet another kubernetes distribution. It is driven by the following goals:

• working out of the box over WAN without compromising security
• offering a much opinionated kubernetes experience
• hiding much of the complexity of setting up kubernetes
• running with the minimum resource footprint (mostly memory)
• bundling vanilla upstream with as few to no fork if possible
• compiling to a (true) single binary on both amd64 and arm64

Rationale

Instead of optimizing for disk size or embedding all kinds of batteries, hepto focuses on the minimal feature set for a kubernetes distribution. Here are so dos and donts:

• do optimize for lower resident memory, this implies:
  • do not unpack multiple static binaries at runtime, which do not map well in the page cache
  • do not optimize for size by compressing part of the binary and inflating at runtime
  • do not embed external binaries and assume Internet access to load further resources
• do work out dependency issues, this implies:
  • do update packages as much as possible
  • do depend on Go modules from vanilla upstreams
  • do not depend on our own forks, if possible not on any fork
  • do not pin dependencies unless required for compatibility
  • do write down the rationale for every quirk
  • do upstream as many fixes as possible to upstreams
• do offer an opinionated experience:
  • do explicit choices and their rationale
  • do hide complexity any time a user choice is not required
  • do not offer too many choices to the user
  • do not embed optional batteries

Architecture

hepto's architecture is still a moving target, here are some hints of our current thinking:

• do not rely on service mesh or complex CNI features for securing traffic, simply run a mesh VPN underneath kubernetes
• do not try and work out Etcd at scale over geographic WAN, keep a central apiserver and make sure it can be backed up and restored with ease
• embed everything in a single binary for optimized resident memory usage
• run everything in namespace, with cgroup configuration prepared during startup
• the namespace root should only contain the binary itself (for shims and CNI), plus
• if possible, drop privileges and run usermod containerd (this probably requires that shims are run separately, plus that kubeproxy and maybe volume managers are bundled instead of running in-cluster)