On August 5, around 15:00, our database experienced an unexpected spike in size and reached 90% of its capacity. Our team responded quickly by initiating a cleanup process to reduce usage. Shortly after that, we also increased the database storage by 30% as a preventive measure.
Unfortunately, the proximity of these two operations triggered a rare and previously unknown bug in the database engine. As a result, the database became stuck in a deadlock state—unable to finish the cleanup due to insufficient space, and unable to restart because the storage was still near full. The system was effectively blocked from recovering automatically.
To resolve the issue, we performed a point-in-time recovery, restoring the database to its last known healthy state—just seconds before the failure. This allowed us to bring the app back online approximately one hour after the incident was discovered.
We’re currently investigating the root cause of the rapid database growth and have implemented new safeguards to better predict and respond to similar scenarios. We’ve also adjusted our internal procedures to avoid time proximity between cleanup operations and storage adjustments in the future.
We sincerely apologize for the disruption and thank you for your continued trust.
— Your Swiss Post Label Printer App Team
