Taking Over Production and Fixing a Six-Month Silent Failure

The Challenge

The client sells auto parts online across multiple regions on a Magento 2 platform. Nexora took over operation of a production system that another team had built, without a complete handover. The environments are two production servers in two regions and one staging environment, and Nexora now operates and maintains them and develops features through a ticket flow. Taking over a system like this means producing trustworthy recommendations quickly while the store stays up.

Two Servers That Were Not Alike

Both production servers run the same codebase, and the natural assumption was that what you learn on one transfers to the other. It did not. An operational rule that held in Region A was being used as a general rule, and in Region B it was false. Direct measurement showed the preprocessed-template copy in Region B was a month older than source while pages still rendered correctly. A piece of work on Region B was nearly postponed over a risk that existed only in Region A. The two machines differ at enough configuration points that the same operation produces opposite results, and when it goes wrong the system raises no error.

Survey Method and Findings

Nexora engineers surveyed both environments between 3 July and 11 August 2026 and recorded eleven measured differences. All eleven are silent failure modes: exit code 0, green checks and HTTP 200 while the result is wrong. The deliverable was a living document plus a process. Before applying a conclusion from one machine to another, three questions must be answered. What configuration does the rule depend on, was it measured or deduced, and if it is wrong, does it fail loudly or silently?

• On-demand static content generation is off in Region A and on in Region B, so staging passing does not prove Region A passes
• Opcode cache timestamp validation is disabled in Region B, so skipping a PHP-FPM reload makes a deploy look green while old code still runs
• A backup script written for Region A fails silently on Region B, where the backup directory is root-owned
• Choosing a deploy window by Vietnam local time lands on Region A’s peak hours

Monitoring Built From a Silent Incident

An application indexing process stopped around October 2025 and nobody noticed until April 2026. Data shown to users drifted from the real data and the system never reported an error. The temp directory looked full, yet the root filesystem had around 800 GB free. The actual cause was inode exhaustion: the file count hit its ceiling while byte capacity stayed abundant, so a capacity graph looked normal for six months. The monitoring Nexora built covers inodes, temp-directory file counts and the indexing process itself, next to disk, web, static content, MySQL, Redis, TLS certificates and backups. Checks run every 5 minutes on both production servers with a one-hour cooldown per alert, automatic recovery notices and a daily digest that doubles as a heartbeat. Every check was verified by deliberately breaking it and confirming the alert fired.

• Inodes: warning at 85% used, critical at 92%
• Temp-directory file count: warning at 80,000, critical at 100,000
• Indexing process: warning when newly invalidated, critical when invalid for more than 2 hours

Production Change Rules

A team that runs production needs written rules for the things that must not go wrong. Four are enforced. Every write to production needs explicit approval of the exact command and target, and approval for one task is not approval for the next. A recovery point is created first and verified as usable. Results are confirmed by reading state back from the system, never by trusting an exit code. And a credential’s permissions are established by reading its grants, never by trying an operation to see what happens. One person on the Nexora side sends anything that leaves the team, because a wrong command can be restored from backup and a wrong sentence to a client cannot.

Three Quality Gates

Six bugs reported by the client in a single month shared a root cause: the fix was not anchored somewhere durable, did not detect its own failure and did not account for its blast radius. Three gates now address each. A fix counts as done only when it lives in a commit, and any fix that exists only on the production filesystem goes into a debt register with a task to pay it off. Fixes that touch search, checkout, nginx or static content deploy get an automated canary check that runs every 5 minutes. A change to a shared layer triggers smoke tests on every downstream subsystem, and a buyer-path smoke test runs after every production deploy: search, product page, add to cart, checkout and blog. Search is verified by checking the result grid actually renders, since an empty-results incident once went unnoticed for 5 days because the page returned HTTP 200.

Results

The client’s uptime, revenue and traffic figures stay with the client. What Nexora can show is process, which predicts how a partner behaves in an incident better than a good uptime number does. The client now has an eleven-row difference table with method and date on every row, monitoring that fires on the failure that went unseen for six months, and a written set of rules for changing production.