One of our own systems runs a backup every night. One night it stopped working. The job started, hit a problem, and failed in about a twentieth of a second.
Then it tried again. And again. By the time anyone looked, it had failed roughly twelve and a half thousand times and written about twenty five thousand lines into the log. The error message, incidentally, was perfectly clear and perfectly accurate the very first time it appeared. And the screen we look at to check on things said, calmly, that the job was running.
Nothing here was hidden. The information was all there, in enormous quantity. That is exactly what made it useless, and it is the reason this post is not really about backups.
How most businesses find out something is broken
A customer tells them.
Somebody rings up because the order they placed on Tuesday has not arrived and they never got a confirmation. Or the sales team notice that the website enquiry form has not produced a single lead since Thursday, which is unusual, and it turns out it stopped emailing anyone on Wednesday.
By that point two things have already happened. The damage is done, and it was done over days rather than minutes. And the first person to notice a fault in your business was somebody you were trying to sell to. That is an expensive way to run a quality check.
"Is the website up?" is the wrong question
Most monitoring, where it exists at all, answers one question: does the website respond? Something checks the homepage every few minutes, and if it fails to load, an email goes out.
That is worth having, and it catches the most dramatic failure. It also misses most of the ones that actually cost money, because a website can respond perfectly while the thing that matters has quietly stopped. All of the following leave your homepage loading in under a second:
- The nightly backup has not completed successfully for three weeks.
- Orders are being taken but have stopped flowing through to the system that fulfils them.
- The overnight price and stock update ran, found nothing, and updated nothing.
- Payment notifications from your provider stopped arriving, so orders sit unpaid and nobody chases them.
- The enquiry form still says thank you, and the email it should send goes nowhere.
Every one of those is invisible to an uptime check, and every one of them is the kind of thing you find out about weeks later.
The better question: did the thing that should have happened, happen?
The shift that makes the difference is to stop watching for errors and start watching for absence.
Instead of waiting for a system to report a problem, you have each important piece of work check in when it finishes successfully. The backup checks in. The order feed checks in. The overnight update checks in. If a check-in that should arrive every night does not arrive, that silence is itself the alert.
This is a small idea with a large consequence. A system that has crashed badly enough cannot send you an error, because sending you an error is one of the things it is no longer able to do. It can, however, fail to check in, and you can detect that from the outside without its cooperation.
This is how our own monitoring works. Every site we run reports in on a short cycle, and a separate watcher marks anything that has gone quiet. We built a tool called blu::monitor to put all of it on one wall, so the question "is everything all right this morning" has an actual answer rather than a shrug. The principle matters more than the tool, and you can apply it with whatever you already use.
Who gets told, and when
Detection is only half of it. An alert that arrives at three in the morning, into an inbox nobody opens until nine, has bought you nothing except an earlier timestamp on the bad news.
Worth deciding in advance, once, in plain terms:
- Which failures genuinely justify waking a human, and which can wait until the morning. Most can wait. Taking payments cannot.
- Who is the second person, when the first does not respond.
- What the person receiving it is actually supposed to do. An alert with no obvious action attached gets ignored within a fortnight.
The trap: alerts nobody reads
This is where most monitoring quietly dies, and it is worth being blunt about it because it is so common.
A system that sends a warning every day trains everybody to ignore warnings. Once that habit forms, the one alert that mattered arrives into a folder that people delete unread, and you are back to finding out from a customer. You now have monitoring, a monthly bill for it, and the same problem.
Our own backup story is the same failure in a different costume. Twenty five thousand log lines is not a louder version of one log line. It is camouflage. The first message was clear, correct and completely sufficient, and repeating it twelve thousand times is what buried it. A system that fails once, loudly, and then stops, is far more useful than one that fails continuously and quietly.
What monitoring will not do for you
Being honest about the limits, because this gets oversold:
- It fixes nothing. It tells you sooner, which is the whole value, but somebody still has to act.
- It cannot watch something you never told it to watch. New system, new feed, new integration: if nobody adds it, it is invisible, and it will be invisible for exactly as long as it takes for something to go wrong.
- It does not know what a failure costs you. It can tell you the feed stopped. It cannot tell you that means forty unfulfilled orders.
- It will occasionally cry wolf. A brief network problem is not an outage, and tuning that out takes a few weeks of living with it.
Where to start
You do not need to monitor everything, and trying to is how these projects stall.
Write down the three things in your business that would cost you the most if they silently stopped for a week. For most companies it is some combination of taking money, receiving orders, and having a backup that actually works. Then ask one question about each of them: if that stopped tonight, how would I find out, and when?
If the honest answer for any of them is "a customer would tell us", that is the thing to fix first, and it is usually a small piece of work.
We build this into the systems we run for clients as standard, and we are happy to look at systems we did not build. Get in touch, or read more about our hosting and support.