/dev/hack Core Infrastructure

This repo is used to manage all of the /dev/hack core infrastructure.

Common Tasks

• HTTP Ingress to Out Of Cluster Services: To route a subdomain under devhack.net to your service inside the space, edit k8s/http-ingress/traefik/out-of-cluster-services.yaml and add your service to the list at the top. Submit your changes as a PR to this repo, and when it's merged the changes will be deployed to Traefik, which will pick them up within a few minutes.
• Add or Update Secret: To provide a static secret, visit bao.devhack.net, select "OIDC" for sign in method, and click the blue "Sign in with OIDC Provider" button. Once authenticated, select static-secrets, then "Create secret +" to create your secret. The secret name must start with <namespace>/<service account>/. The service account name is usually default. The OpenBao UI also allows you to update existing secrets, but will not allow you to see the secret data.
• Create an OIDC Client: If you want your app to be able to authenticate users via our IDP (Keycloak), add your app to tf/keycloak.tf and submit a PR. Your client credentials will be stored in OpenBao in a kv secret at static-secrets/<namespace>/default/oidc-client-credentials-<client_id> with keys client_id and client_secret. An example of how to read this in Kubernetes is available in k8s/README.md, and we are working to get options for other platforms.

Repository Components

These components interoperate to form the full system. For a full install:

1. Manually provision a Debian machine or cluster of machines and deploy k3s to it
2. Copy the k3s kubeconfig to .kubeconfig in this repo (it will be gitignored)
3. Run direnv allow or manually export environment variable KUBECONFIG=/path/to/this/repo/.kubeconfig to your shell
4. Deploy the Kubernetes operators: kubectl apply -k k8s/operators --server-side
5. Deploy the rest of the Kubernetes system: kubectl apply -k k8s
6. Unseal OpenBao by having enough key holders visit the web UI and enter their keys (3 keys required by default).
7. Provision OpenBao: cd tf && tofu apply NOTE: this assumes you are restoring from a backup and will likely have essentially no effect. For a fresh install, see below.
8. Use ansible/ to provision the Forgejo runners: ansible-playbook playbook-forgejo-runner.yaml
9. Ensure this repository is pushed to Forgejo, so the containers get built. Some Kubernetes components will be unable to start until the containers are available.

ansible

Used to provision VMs. Currently just used to provision Forgejo runners.

containers

Custom builds of container images used by other components, when the software isn't distributed as a container image or the official container image is somehow insufficient.

helm

Uses kustomize and helm to render helm charts into static yamls in k8s. For things that don't distribute a kustomize-compatible yaml.

k8s

Kustomize yaml based configuration for the core k8s cluster

tf

OpenTofu (fka Hashicorp Terraform) configuration for certain components, currently only used to configure OpenBao (fka Hashicorp Vault), which doesnt seem to have it's own method of declarative configuration.

Initializing OpenBao from scratch

OpenBao is tricky to initialize from scratch. We should never actually need to do this, as we would simply restore the database backup.

1. Deploy OpenBao to Kubernetes with the rest of the components.
2. Follow the OpenBao documentation to initialize and unseal Bao. This will approximately consist of running the initialization command: kubectl -n openbao exec -ti openbao-0 -- bao operator init, followed by unseal several times: kubectl -n openbao exec -ti openbao-0 -- bao operator unseal.
3. Using the root token from the output of bao operator init, login to bao on your local machine: bao login (will prompt for token)
4. Now on to the provisioning with OpenTofu: cd tf
5. Do the first part, which will provision parts of OpenBao but ultimately fail: tofu apply
6. Now go into Keycloak and create an OIDC client for OpenBao. Currently the client_id is expected to be set to the string openbao
7. Copy the client_secret (from the credentials tab) and write it to bao: bao kv put static-secrets/openbao/default/oidc-client-credentials-tofu client_id=openbao client_secret=$CLIENT_SECRET