Some air-con contractors accidently cut power to the entire datacentre! :mad:

today is NOT a fun day!

When you say "datacentre" do you mean "server cupboard"?

Come on Tom we all know it's just a pc under his desk..... and the aircon is a swivel fan :P:D

When you say "datacentre" do you mean "server cupboard"?

I wish! lol. :rofl:

Call CTU and ask for Mr Bauer.

And you don't have backup power Col?

Depends... usually there is a big mains in and the generators live on the outside of this so it can feed in as if it was from the mains.

The UPS will only last so long and in a big data centre that usually isn't long enough to get in a lot of new cable.

Yes we have a generator and yes we have UPS too.. not entirely sure what happened, but i think the aircon people were trying to isolate one unit but accidently isolated the entire datacentre

I was on a site today in a 'just in case' role. Basically it was a portacabin setup and the main contractor has some clients on site in their offices. As long as the client was happy that all of their terms had been fulfilled regarding phones, data, welfare facilities they would release One Million quid.

My bit was fine, but BT had cocked the DSL order up, when the line had been installed they hadn't marked it as such on the system so the DSL order hadn't gone through, I was drinking coffee and lounging about whilst the IT and site manager rang everyone they could think of to get it sorted.

Didn't hang around to see what happened, i'm guessing the million quid didn't get transferred though.

Then I got an early finish!

We had a planned power down last weekend in our datacentre -one of the supplies was taken down for a battery check. Everything is dual powered in there -or so we thought. It turned out that quite a lot of stuff wasn't.

I've just finished restoring the last server today, so I feel your pain, believe me!

Phil

Sounds like fun.

We have a super dooper datacenter that they say can withstand a plane crash....but not squires lmao

At the place my dad used to work at (big multinational with a large computer system), some workers in the data centre dropped a floor tile which dropped through the hole and smashed a valve off the A/C system and flooded one of the computing halls with rather mucky water. This is quite a few years ago and while cleaning up they were apparently opening the doors on the big tape drives to let the water pour out. Needless to say caused absolute havoc.

I feel your pain - I remember walking past our ground floor server/comms room and seeing a grimy puddle leaking out from under the door. The aircon unit, which we'd argued against on the grounds of being underpowered, had popped a pipe and dribbled all over the main rack. I have no idea how the router (which was at the top) didn't go foom. What I did know is that the unit appeared to have been clogged and not circulating properly for some time, and somebody appeared to have skimped on teh service record.

I was also surprised when our health and safety guy told me off for telling my team not to go in until it had been made safe - apparently water drenched electrics and legionnaires disease aren't problems. That was a fun argument, and subsequent swift retreat on his part, I can assure you

The last place I worked at had their customers datacentre based a very close distance from a river and when that river burst its banks in Sheffield last year the water level was less than 1inch away from the top of the bricks and flooding the datacentre :eek::eek::eek: and on top of that there was a swipe card entry/exit system with a push button to release the doors from the inside too, some bright spark decided to put the DataCentre Emergency shut off switch right next to it and NOT label it or put a box next to it, the amount of contractors that pressed that button instead of the door release was shocking, it's hardly any wonder they lost thier biggest IT customer and are now not doing very well

It's nearly as bad as where I work now puting a backup Datacentre right next to a main runway at the UK's largest airport :rofl: at least the main Datacentre is a quite a few miles away although close'ish to another extremly large UK Airport :doh:

At least you know what went wrong Col....

Exchange servers went awol last night about 4:30.....probably wrong type of snow or something....

called up central IT this morning at 9am. The response was a priceless:

"we know there is...er...something wrong....but...ahh....we don't know what it is yet"

Arh the classic ups was interrupted today when... cheer up col, least it's not summer and the ac' is on low.

Oh and farcehosts ran out of fuel last year for there gens...think they last two or three days then poof out went a lot of websites and zeros appeared on daily income sheets :(

Lesson learnt, seperate dns hosting from web hosting and seperate email from web.

Those generators will drink fuel. On a good week running 10 diesel engine test cells with a mix of 3.9 to 11 litre engines we would use around 110,000 litres of diesel :eek:

Oh and farcehosts ran out of fuel last year for there gens...think they last two or three days then poof out went a lot of websites and zeros appeared on daily income sheets :(

I have a friend who used to work at a London datacentre, where they had a couple of huge generators proudly on display.

The one time they had a power outage, the UPS kicked in, the gennies fired up, lasted 10 minutes then died.

Subsequent investigation showed that some of the night shift staff had been helping themselves from the diesel tanks and had jammed the level gauges. The backup level sight glass was so hard to see that nobody bothered, and why would the diesel level drop when they only ran them on Saturdays for a 10 minute test? . Anyone on the shift rotation with a diesel car was suspect, but they never proved anything.

Phil

Depends... usually there is a big mains in and the generators live on the outside of this so it can feed in as if it was from the mains.

The UPS will only last so long and in a big data centre that usually isn't long enough to get in a lot of new cable.

And of course, if they cut it on the output side of the UPS .

At least you know what went wrong Col....

Exchange servers went awol last night about 4:30.....probably wrong type of snow or something....

called up central IT this morning at 9am. The response was a priceless:

"we know there is...er...something wrong....but...ahh....we don't know what it is yet"

That's exchange for you.

I worked in a Datacentre but the firm we worked for are closing the Northern operations to move them down south and to Ireland.

We never had any major floods. the only real c*ck up was when we had to put new encryption keys on the servers and they ended up really screwing with the system so we had to roll back to the old encryption disks.

....put new encryption keys on the servers and they ended up really screwing with the system so we had to roll back to the old encryption disks.

Sounds like me.

Staff member here was having an odd response from IE - it shut down everytime they tried to add data items to one of our web-interface based systems.

So, rang central IT, they said "update to latest IE", which was duly done.

This single act then destroyed their access to the Student registration systems, resulting in the need for a complete system rebuild. All because the registration system only works with IE 6, but no one mentions that..........

We recently had a datacentre outage where one of our major clients is hosted - they somehow lost power to the whole place, and this is a five storey dedicated datacentre hosting some very large companies and government departments

Not only that, it happened a second time, and this time managed to deliver a power surge to lots of servers and SANs!

The hosting company had to replace over AU$1m worth of hard disks throughout the place :eek:

Its times like these when I realise why I dont normally post in the tech shed.

Whooosh over my head.