/dev/hack Core Infrastructure

This repo is used to manage all of the /dev/hack core infrastructure.

Common Tasks

• HTTP Ingress to Out Of Cluster Services: To route a subdomain under devhack.net to your service inside the space, edit k8s/http-ingress/traefik/out-of-cluster-services.yaml and add your service to the list at the top. Submit your changes as a PR to this repo, and when it's merged the changes will be deployed to Traefik, which will pick them up within a few minutes.
• Add or Update Secret: To provide a static secret, visit bao.devhack.net, select "OIDC" for sign in method, and click the blue "Sign in with OIDC Provider" button. Once authenticated, select static-secrets, then "Create secret +" to create your secret. The secret name must start with <namespace>/<service account>/. The service account name is usually default. The OpenBao UI also allows you to update existing secrets, but will not allow you to see the secret data.
• Create an OIDC Client: If you want your app to be able to authenticate users via our IDP (Keycloak), add your app to tf/keycloak.tf and submit a PR. Your client credentials will be stored in OpenBao in a kv secret at static-secrets/<namespace>/default/oidc-client-credentials-<client_id> with keys client_id and client_secret. An example of how to read this in Kubernetes is available in k8s/README.md, and we are working to get options for other platforms.

Setting Up

To simply get started with developing the components you need, see each relevant section.

Note

If you don't have the OpenBao secrets, then you won't need to (or be able to) proceed past the Kubernetes section.

Direnv

1. If not using Nix flakes, manually install kubectl and direnv.
2. Follow direnv setup to set up direnv for your shell.
3. cd into this repo. Run direnv allow to allow direnv to load the environment variables in .envrc

This may cause some OpenBao errors to be logged if you haven't authenticated it locally yet, which is expected.

Kubernetes

1. Follow the Direnv instructions above.
2. If not using Nix flakes, manually install kubelogin.
3. Reload direnv with direnv reload or restart your shell.
4. Run something like kubectl get all to trigger the login flow.
5. Apply all the Kubernetes manifests: kubectl apply -k k8s/operators --server-side && kubectl apply -k k8s.

If orchestrating Kubernetes from scratch, see the section at the bottom.

OpenBao

1. If OpenBao is freshly started, then it will be sealed. You may check this on the Web UI. Unseal OpenBao by having enough key holders visit the web UI and enter their keys (3 keys required by default).
   • This requires that you have access to the unseal keys, which should be securely held by key holders (system administrators).
   • If doing this alone, you can port-forward each OpenBao pod individually and sequentially unseal each one.
2. Login to OpenBao with your admin/root token via bao login.
   • This requires that you have access to the initial root token, which should also be securely held by system administrators.
3. Provision OpenBao (along with everything): cd tf && tofu apply.
   • NOTE: this assumes you are restoring from a backup and will likely have essentially no effect. For a fresh install, see below.

Forgejo Runners

1. Use ansible/ to provision the Forgejo runners: ansible-playbook playbook-forgejo-runner.yaml
2. Ensure this repository is pushed to Forgejo, so the containers get built. Some Kubernetes components will be unable to start until the containers are available.

NixOS

1. Install Nix if you haven't already: Nix installation instructions. Make sure it has Flakes enabled.
2. Ensure you're logged into OpenBao as above.
3. Provision NixOS: cd nixos && tofu apply
   • This builds all machines on your local machine, so ensure you have sufficient resources (CPU, RAM, disk space).
   • TODO: consider NixOS shared cache and remote builders to speed this up.

Repository Components

These components interoperate to form the full system. They're contained in their own directories.

containers/

Custom builds of container images used by other components, when the software isn't distributed as a container image or the official container image is somehow insufficient.

See containers/README.md for more information.

helm/

Uses Kustomize and Helm to render helm charts into static YAMLs in k8s. For things that don't distribute a kustomize-compatible yaml.

k8s/

Kustomize yaml based configuration for the core k8s cluster

tf/

OpenTofu (fka Hashicorp Terraform) configuration for certain components, currently only used to configure OpenBao (fka Hashicorp Vault), which doesnt seem to have it's own method of declarative configuration.

nixos/

NixOS configuration for the VMs and physical machines used in the system that are considered core infrastructure. These machines are provisioned with OpenTofu and somewhat hook into OpenBao for secrets management.

To set them up from scratch, it is recommended to prototype a NixOS machine configuration using disko and nixos-hardware, with OpenSSH ready and a default root password. Once installed, you can onboard the configuration into this repo via OpenTofu and turn off the default root password.

ansible/

Ansible is used to provision VMs. Currently just used to provision Forgejo runners.

Initializing OpenBao from scratch

OpenBao is tricky to initialize from scratch. We should never actually need to do this, as we would simply restore the database backup.

1. Deploy OpenBao to Kubernetes with the rest of the components.
2. Follow the OpenBao documentation to initialize and unseal Bao. This will approximately consist of running the initialization command: kubectl -n openbao exec -ti openbao-0 -- bao operator init, followed by unseal several times: kubectl -n openbao exec -ti openbao-0 -- bao operator unseal.
3. Using the root token from the output of bao operator init, login to bao on your local machine: bao login (will prompt for token)
4. Now on to the provisioning with OpenTofu: cd tf
5. Do the first part, which will provision parts of OpenBao but ultimately fail: tofu apply
6. Now go into Keycloak and create an OIDC client for OpenBao. Currently the client_id is expected to be set to the string openbao
7. Copy the client_secret (from the credentials tab) and write it to bao: bao kv put static-secrets/openbao/default/oidc-client-credentials-tofu client_id=openbao client_secret=$CLIENT_SECRET

Initializing Kubernetes from scratch

1. Manually provision a Debian machine or cluster of machines and deploy k3s to it
2. Copy the K3s kubeconfig to .kubeconfig in this repo (it will be gitignored)
3. Deploy the Kubernetes operators: kubectl apply -k k8s/operators --server-side
4. Deploy the rest of the Kubernetes system: kubectl apply -k k8s