Offline time

You will note I don't use the term "down" as none of our servers or services experienced any sort of technical fault or outage, in fact, with the right Internet connection you will have been able to access all of our services normally.

We first noticed the problem at around 3:50pm and tried to contact our Data Center in Northampton, only to find the entire data center was off-line. Apparently this was a general Internet based fault blocking traffic to the entire site.

A little more research indicated the fault was primarily down to a fault at Kingston Telecom (who provide transit for the DC) and not something we could influence in terms of fix times. On contacting the Kingston helpdesk we were informed that there was a major outage and the entire KAROO network was out. (I understand KAROO to be their 'Internet Platform') I was informed that their engineers were working on the problem but no further details were available.

Anyway, it looks like they rectified the problem a little after 10pm.

Judging by the fix timescale, it does look rather like someone stuck a screwdriver into the back of something he shouldn't and as a result something fairly major needed to be replaced. I'm guessing that on Monday morning there'll be a red-faced engineer facing some questions he'd rather not have to answer.

Rather frustrating to look at our monitoring graphs which of course show everything up over the period, but with nothing going on. Here's the activity graph from our front-line apache Hydra service;

If we can get more details from Kingston with regards to the nature of the fault, we'll post it here, however this is the first outage we've experienced this year, which in hosting terms isn't bad going. (we were averaging 5 outages per year with our previous DC)