Lab Cluster¶
This repository is my lab Kubernetes cluster in a declarative state. Flux watches my cluster folder and makes the changes to my cluster based on the YAML manifests.
Feel free to open a Github issue or join the k8s@home Discord if you have any questions.
This repository is built off the k8s-at-home/template-cluster-k3s repository.
Cluster setup¶
My cluster is k3s provisioned overtop Ubuntu 21.04 using the Ansible galaxy role ansible-role-k3s. This is a semi hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes while I have a separate server for (NFS) file storage.
See my ansible directory for my playbooks and roles.
Cluster components¶
- calico: For internal cluster networking using BGP configured on Opnsense.
- rook-ceph: Provides persistent volumes, allowing any application to consume RBD block storage.
- Mozilla SOPS: Encrypts secrets which is safe to store - even to a public repository.
- external-dns: Creates DNS entries in a separate coredns deployment which is backed by my clusters etcd deployment.
- cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.
- kube-vip: HA solution for Kubernetes control plane
- Kasten: Data backup and recovery
Repository structure¶
The Git repository contains the following directories under cluster
and are ordered below by how Flux will apply them.
- base directory is the entrypoint to Flux
- init directory contains SOPS and other secrets inittialization.
- crds directory contains custom resource definitions (CRDs) that need to exist globally in your cluster before anything else exists
- core directory (depends on crds) are important infrastructure applications (grouped by namespace) that should never be pruned by Flux
- apps directory (depends on core) is where your common applications (grouped by namespace) could be placed, Flux will prune resources here if they are not tracked by Git anymore
./cluster
├── ./apps
├── ./base
├── ./core
├── ./init
└── ./crds
Automate all the things!¶
- Github Actions for checking code formatting
- Rancher System Upgrade Controller to apply updates to k3s
- Renovate with the help of the k8s-at-home/renovate-helm-releases Github action keeps my application charts and container images up-to-date
Hardware¶
Device | Count | OS Disk Size | Data Disk Size | Ram | Purpose |
---|---|---|---|---|---|
HP DL580 | 3 | 256GB SSD | N/A | 196GB | k3s Masters (embedded etcd) |
HP DL360 | 3 | 240GB SSD | N/A | 196GB | k3s Workers |
HP DL380 | 3 | 240GB SSD | 1TB NVMe (rook-ceph) | 196GB | k3s Workers (label: storage=yes) |
TrueNAS Core | 1 | 120GB SSD | 8x2TB RAIDz2 | 32GB | Shared file storage |
Tools¶
Tool | Purpose |
---|---|
direnv | Sets environment variable based on present working directory |
go-task | Alternative to makefiles, who honestly likes that? |
pre-commit | Enforce code consistency and verifies no secrets are pushed |
stern | Tail logs in Kubernetes |
Thanks¶
A lot of inspiration for my cluster came from the people that have shared their clusters over at awesome-home-kubernetes