On the night of July 11, gablvm.org went down and stayed down for twelve hours and forty-one minutes. That is the longest outage in this site's history, and you deserve a straight account of what happened, why our own monitoring took ten hours to notice, and what has already changed so it cannot repeat. No visitor data was ever at risk. This was an availability failure, not a security event.
What visitors experienced
From 11:05 pm UTC on July 11 until 11:46 am UTC on July 12, anyone visiting gablvm.org saw our "back shortly" maintenance page instead of the website. Some visitors were spared for part of that window because our content delivery network kept serving saved copies of pages, but as those copies expired, more and more people hit the maintenance page. Prayer times, the Quran player, and every other tool on the site were unreachable for most of that period.
What maintenance mode is, and why it exists
Maintenance mode is a deliberate safety feature, not the failure itself. When a website applies updates to its database, visitors arriving mid-update could see half-changed pages, broken forms, or errors, and their actions could even corrupt data that is being restructured at that moment. So the platform puts up a polite "back shortly" barrier, finishes its work behind it, and then takes the barrier down. The entire cycle normally lasts under two minutes and most visitors never notice it. The failure in this incident was not that maintenance mode turned on; it was that the process responsible for turning it back off died before reaching that step, leaving a two-minute barrier standing for nearly thirteen hours.
The timeline
- July 11, 10:59 pm UTC. A routine database update was run on the server, the same kind of update the site has absorbed dozens of times.
- July 11, around 11:00 pm UTC. The update tool crashed partway through. The updates themselves had already completed safely, but the tool never reached its final step: switching off the maintenance mode it had turned on to protect the site while it worked.
- July 11, 11:05 pm UTC. The server began answering every visit with the maintenance page. This is the first failed request in our logs.
- July 12, 8:55 am UTC. Our automated monitoring saw its first failure, nearly ten hours later. More on that embarrassing gap below.
- July 12, 9:05 am UTC. Second consecutive failed check; the monitoring raised its alarm.
- July 12, 11:41 am UTC. An engineer picked up the alert and began investigating.
- July 12, 11:43 am UTC. Investigation confirmed the stuck maintenance switch as the cause.
- July 12, 11:46 am UTC. Maintenance mode switched off, caches refreshed, site verified healthy. Full details were published as incident 76 on our status page as the fix went out.
The detection timeline, in plain numbers
Break the twelve hours and forty-one minutes apart and the story becomes uncomfortable in a useful way. The site failed at 11:05 pm. Our monitoring first observed a failure at 8:55 am, nine hours and fifty minutes later. The alarm went out at 9:05 am. An engineer began investigating at 11:41 am, and the site was restored at 11:46 am, five minutes after investigation began.
In other words: diagnosis and repair took five minutes. Detection took nearly ten hours, and the alert then waited another two and a half hours for human attention overnight. Almost the entire cost of this outage was time-to-detect, not time-to-fix. That framing drove every improvement we made, because an outage this simple should have been a fifteen-minute blip at three minutes past eleven.
Why it happened
Two separate weaknesses had to line up, and both were ours.
First, the crash itself. The command-line tool that applies database updates was running with a far smaller memory allowance than the site actually needs. To cope, it would hand its work to a fresh copy of itself partway through every update. On this night the handoff failed, the tool stopped without cleaning up, and because switching maintenance mode back off was scheduled as the very last step of the job, that step simply never ran. The site sat behind its maintenance page with nobody the wiser.
Second, the detection gap, which deserves its own section.
Why the cache delayed detection for ten hours
Like most modern websites, gablvm.org sits behind a content delivery network, a layer of servers around the world that keeps copies of recently served pages and answers repeat requests from those copies instead of asking our origin server every time. This is standard practice and it is why the site feels fast: most visits are served from a nearby cache in milliseconds.
Every cached copy carries a lifetime, and the delivery network may also continue serving a previously known-good copy while it retries a failing origin in the background, a resilience feature that keeps sites readable through brief origin hiccups. That resilience became the problem. Our uptime monitor checked the homepage through the same delivery network our visitors use, from the same vantage point. When the origin began refusing every request at 11:05 pm, the edge kept answering the monitor with a cheerful, healthy cached copy of the homepage, exactly as designed. The monitor was therefore measuring cache freshness, not origin health. Our own logs show the monitor's requests reaching the origin and failing all night while the monitor itself recorded success after success from the cache in front of it. Our own caching blindfolded our own watchman for ten hours.
The general lesson is well known in the industry and we relearned it the hard way: a health check that shares its vantage point with the caching layer can only tell you what the cache is serving, never whether the system behind the cache is alive. You need both measurements, because they answer different questions.
What has already changed
- The monitor now checks the site twice on every run: once through the delivery network the way visitors see it, and once with a cache-defeating request that is guaranteed to reach the origin server. A server failure now raises an alarm within twenty minutes at most, no matter how healthy any cached copy looks.
- The monitor also watches the maintenance switch itself. If maintenance mode stays on longer than ten minutes, a dedicated alarm goes out naming the exact problem, because a normal update needs it for under two.
- The update tool now runs with eight times the memory headroom, which removes the fragile mid-job handoff that crashed. The updates that triggered all this were verified complete and intact.
What was not affected
No data was lost or exposed. Newsletter subscriptions, contact submissions, and every piece of site content came through untouched. The server itself, and the company that hosts it, performed flawlessly throughout; the failure was entirely in our own update tooling and monitoring design.
What we learned
- Monitor the origin, not just the edge. A cached healthy response is an answer about the cache. From now on, anything we call a health check must observe the layer it claims to be checking.
- Any process that flips a site-wide switch must be accountable for flipping it back. Cleanup that is scheduled as a final step dies with the process; the safety net has to live outside the process, which is why the maintenance switch is now independently watched.
- Time-to-detect deserves as much engineering as time-to-fix. The repair here took five minutes; the noticing took ten hours. We had invested in the wrong half of the problem.
- Verify repairs and updates through a path that cannot lie. The update that caused this was declared healthy that night based on a check that the cache answered. Post-change verification now uses the same cache-defeating path as the new monitoring.
- Boring failures are the dangerous ones. This outage involved no attack, no hardware fault, and no data problem, just a crashed cleanup step and a blind spot, and it outlasted every dramatic incident we have ever had.
Why we publish these
GABLVM asks blind and low vision Muslims to rely on this site for daily worship tools. Reliance has to be earned, and part of earning it is telling you plainly when we fall short and what we fixed. The status page at status.gablvm.org carries every incident as it unfolds, and for anything this large we will publish an analysis like this one.
Thank you
Behind every request in those logs is a person. Someone checked prayer times before Fajr that night and found a wall instead. Someone sat down to listen to the Quran with their morning tea and could not. If that was you, we are sorry, and we are grateful for the patience you showed us. And to everyone who never noticed a thing but trusts this site day after day: that trust is why the monitoring now watches twice, why the analysis you just read exists, and why we will keep holding ourselves to this standard.
One more thing. If the site ever seems wrong to you, tell us through the contact page. This outage taught us that a single visitor speaking up can outrun the best automation. You are not bothering us. You are the reason the site exists.