In March 2026, GABLVM services experienced two unplanned outages caused by infrastructure issues at our hosting provider. This post explains what happened, what caused both incidents, and what we have done to permanently resolve them.
Incident 1: March 22 to 23
- March 22, 4:09 PM CT: All services stopped responding. No alerts were generated because the server itself was unresponsive.
- March 23, 3:15 AM CT: Outage discovered during a routine check. Server manually restarted.
- March 23, 3:19 AM CT: All services restored and verified.
- Duration: Approximately 11 hours.
Incident 2: March 29
- March 29, 2:30 PM CT: All services stopped responding. External health monitoring detected the disruption within 10 minutes.
- March 29, 4:15 PM CT: Server restarted by hosting provider after manual intervention.
- March 29, 4:20 PM CT: All services restored and verified.
- Duration: Approximately 1 hour 50 minutes.
Root Cause
Our hosting provider conducted a thorough investigation and confirmed both incidents were caused by the same underlying issue on the physical host server:
- Orphaned virtual disk images: Two virtual disk files from other customers on the same physical server had been deleted but were still referenced by the operating system. These orphaned files consumed 480 GB of invisible disk space and produced thousands of input/output errors that cascaded to all servers on the host.
- Host disk capacity: The physical server was operating at 92 to 93% disk capacity, leaving minimal room for normal operations.
- Server density: The physical host was running 52 virtual servers, creating significant resource contention. Six servers were affected during the March 29 incident, not just ours.
Our server was healthy in both incidents: 8% disk usage, minimal memory consumption (5 GB of 32 GB), no software errors, and no kernel issues. The freezes were caused by input/output stalls at the host level that made our server completely unresponsive.
Impact
All GABLVM services were unavailable during both incidents, including the website, Quran Player, Prayer Times, Islamic Calendar, and all other resources. Total unplanned downtime: approximately 13 hours across two incidents in 7 days.
No data was lost in either incident. All content, user data, and configurations were intact after each restart.
Resolution: Server Migration
On March 30, our hosting provider migrated our server to a completely different physical host in a new data center. The new host:
- Has only 6 other servers (compared to 52 on the previous host).
- Has significantly more available disk and memory resources.
- Is located in a different geographic region for improved reliability.
All services, data, configurations, and functionality transferred successfully. The migration was verified with comprehensive testing across all pages and services.
What We Did to Prevent Recurrence
- Server migration: Moved to a low-density host with healthy resource levels.
- Persistent crash logging: System logs now survive restarts, giving us diagnostic data even after a crash.
- External health monitoring: Our status page checks all services every 10 minutes from outside the server, so outages are detected even when the server is completely frozen.
- Service watchdog: An automated script checks critical services every 2 minutes and restarts them if they crash.
- Automated backups: Daily backups to an external cloud storage service ensure no data loss.
- Server audit logging: All system activity is logged for security and diagnostic review.
- DNS and email infrastructure: Updated all DNS records and email authentication for the new server.
Accountability
Our hosting provider took full responsibility for both incidents. The root cause was infrastructure-level issues on their physical host that were outside our control. They responded to our investigation with transparency, provided detailed technical findings, and completed the migration within 48 hours of escalation.
We hold ourselves to a high standard of reliability because our community depends on these tools daily for prayer times, Quran access, and staying connected. We are committed to transparency. You can always check the current status of our services at status.gablvm.org.
If you experienced any issues during these outages or have questions, please contact us at [email protected].