My latest fix narrowed the problem space down, but it looks like there's still a problem.

Looking into it...

Edit: Looks it's a different issue. Nothing showed up in the logs this time except for the fact that requests were timing out.

Gonna deploy a hack or two to rule out some possibilities...

This weekend I'd like to upgrade my database software so that I can rewrite a few queries to take advantage of some new features which will reduce the amount of in-flight database requests. Not quite a fix but will possibly reduce the catastrophe of whatever is causing this downtime event.