Jump to content

What Happened to Head-Case?


Recommended Posts

What The Hell Happened to Head-Case?!?!?

TLDR version:  The datacenter that the head-case server is located in sustained a power outage on April 4.  When the backup generator kicked on it caught fire and that lead to the fire suppression system (a fucking water based one!) kicking on and causing water damage to equipment.  The head-case server was in the 'more affected' area of the datacenter and as such was offline until early in the morning on April 24.  For further details on the incident you can see here:  https://webnx.com/outage-updates/ 

You may be wondering why during such a long time the site wasn't simply migrated elsewhere, this is a fair question and the answer is simple.  As this is a hobby site the backup process hadn't been properly verified as working in quite a while and the latest verified backup was nearly a year old. 

Migrating the site and loading it from that last verified backup would have meant losing a years worth of content.  This was not a valid option and until I could confirm that the drives in the server were dead I wasn't willing to restore that old of a backup. 

From what I can tell there has been no data loss but I don't currently have a full report as to what if any hardware needed replacing to get the server back online.  Due to that lack of info, I don't trust the server and I wouldn't be surprised if there's some additional downtime due to hardware issues.

The Future

A fresh backup at the time of this post has been taken and moved off-site.  Over the next week or so a new backup process will be put in place and verified.  After that is done it's very likely that the site will be moved to a new hosting provider.  Further details about any move will be provided at the proper time.  
 

  • Like 10
  • Thanks 33
  • Haha 1
Link to post
Share on other sites

Thanks for the update Todd!  :)  I've received dozens and dozens of emails from some very worried people so I'm glad it is back. 

  • Like 2
Link to post
Share on other sites

Thanks Bro, just happy to be up and running.

Who puts a water based surpression system in a data center?  That's common sense 101.  I mean anyone who's watched Terminator 2 should know about Halon.  I don't think they use it in new installs but still.

Link to post
Share on other sites
18 minutes ago, sbelyo said:

Who puts a water based surpression system in a data center?  That's common sense 101. 

Believe it or not, it's more common than you think and I know of several cases where it's done solely to meet local municipality requirements.  Most people hear 'data center' and think of these super fancy expensive facilities.  95% of data centers do not match that and are built to the lowest possible cost.  Building nice, enterprise grade facilities is very, very expensive.

It could have been worse, OVH had an entire one of their data centers burn to the ground because they simply didn't have any fire suppression system:  https://www.reuters.com/article/us-france-ovh-fire/millions-of-websites-offline-after-fire-at-french-cloud-services-firm-idUSKBN2B20NU

  • Like 1
Link to post
Share on other sites
1 hour ago, sbelyo said:

Thanks Bro, just happy to be up and running.

Who puts a water based surpression system in a data center?  That's common sense 101.  I mean anyone who's watched Terminator 2 should know about Halon.  I don't think they use it in new installs but still.

water cooled data centre means you can overclock the cpu(s) more....

Link to post
Share on other sites
29 minutes ago, luvdunhill said:

Ok, who logged back into Huddler-Fi? Come clean.

Not I. I've been good so I wouldn't punish myself with something like that.

  • Like 1
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...