OLH’s Recent Downtime

The Open Library of Humanities main site, and Orbit and IJWWE journal sites are all on Janeway and are hosted on Reclaim Hosting’s great Shared Hosting service. Reclaim leverage Digital Ocean’s VPS and Block Storage services to provide this service, so when DO’s Frankfurt data center fell to pieces on the 1st of April (Apr 01, 2018–07:31 UTC) we were among the unfortunate victims.

Our sites were offline from 0731 on 01/04 to 1937 02/04. This length of downtime, we feel, isn’t okay so we are setting out to ensure it doesn’t happen again. We plan to:

1. Setup a second Reclaim account in a different data center/region (London)
2. Register a VPS and setup a Haproxy in TCP mode to forward traffic to the live server (Frankfurt)
3. Setup MySQL Replication between Frankfurt and London
4. Sync the home folders of Frankfurt and London, bidirectionally, using unison
5. Sync SSL certs from Frankfurt to London
6. Implement simple switch to change the backend on the VPS from Frankfurt to London
7. Add heartbeat and automatic failover when Frankfurt fails