Opened 6 weeks ago
Last modified 6 weeks ago
#1181 new defect
get log aggregation going
| Reported by: | m | Owned by: | |
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | Hackery | Keywords: | |
| Cc: |
Description
some services (cough synapse cough) log enough that only the last 30 minutes is accessible via kubectl logs. lets get a log server going to handle retention, searching, all that fun stuff.
the zigbee hub supports syslog server only.
it seems like k8s doesn't do anything special for logging backends and just expects a container to watch it's log directory: https://kubernetes.io/docs/concepts/cluster-administration/logging/
docker supports journald, fluentd, syslog, awslogs, splunk, etwlogs, gcplogs among others: https://docs.docker.com/engine/logging/configure/#supported-logging-drivers
dunno what else we should be targetting? Raw machines running systemd? it seems journald/systemd doesn't like offloading it's logs elsewhere, but it does have an offloadtosyslog: https://www.loggly.com/ultimate-guide/centralizing-with-syslog/
Change History (4)
comment:1 by , 6 weeks ago
follow-up: 4 comment:2 by , 6 weeks ago
diamond got started on a victoriametrics VM inside proxmox, but dropped it when it left (schism). i wouldn't expect someone to pick up that work, but we did spend a long time deliberating on logging software and settled there previously.
comment:4 by , 6 weeks ago
Replying to io@…:
diamond got started on a victoriametrics VM inside proxmox, but dropped it when it left (schism). i wouldn't expect someone to pick up that work, but we did spend a long time deliberating on logging software and settled there previously.
for context the work is here: https://git.devhack.net/devhack/core-infra/src/commit/main/k8s/vm &
i suggest this be run on a lower power bare metal machine to insulate against e.g. proxmox going down.