Mementomori.social experienced 57 minutes of downtime earlier today. Here's what happened and what we've done about it.

What happened: A large volume of connections from external crawler/bot IP ranges accumulated in a stuck state (CLOSEWAIT) inside our web server (nginx). These zombie connections exhausted nginx's connection pool and connections per worker, which caused it to stop accepting new connections entirely resulting in ERRCONNECTION_CLOSED for everyone trying to reach the site.

Why it took 57 minutes: The sole admin available (me) was in a meeting when the issue began, and other admins were unavailable. There was no automated recovery in place to detect and resolve this type of failure.

How it was resolved: Nginx was restarted, which cleared all stale connections and immediately restored service.

What we've done to prevent this:
- Installed and configured Monit, a process monitoring tool that automatically checks all critical services (web, background workers, streaming, database, cache, and nginx) every 2 minutes and restarts them if they become unresponsive
- This means similar outages should now self-heal within minutes, even when no admin is available

I'm really sorry for the disruption. We take the reliability of this instance seriously, and this change should make a real difference moving forward. I'm also planning to apply other mitigations, as some parts of our instance may have been underconfigured.

Talk about this: https://mementomori.social/@rolle/116724816543799638