When datacenter cooling goes wrong

chrismid259    says:

Oh dear. Are your datacenters in the UK by any chance?
Posted 7 months ago. ( permalink )

willajd says:

D:

o noez where r my scrobbles gonna go!?
Posted 7 months ago. ( permalink )

igap says:

Hot news
Posted 7 months ago. ( permalink )

Master Bob    says:

Sounds like the redundancy of that datacenter is lacking ....

Seems to me that 1 chiller failing shouldn't lead to all other 5 chillers failing. Though, I wouldn't really know.
Posted 7 months ago. ( permalink )

erKURITA    says:

We're in for some cooked surprises :<
Posted 7 months ago. ( permalink )

Master Bob    says:

@willkvlt: Scrobbles are up. Check the status.
Posted 7 months ago. ( permalink )

uniCorn 27 [deleted] says:

Ouch! But then it was very hot here today!
Posted 7 months ago. ( permalink )

willajd says:

@Master Bob

I know, i was being facetious. ;D
Posted 7 months ago. ( permalink )

alexmuller says:

Oh dear. Are your datacenters in the UK by any chance?

divine_infekt says:

i don't know nothing about that stuff... but it really doesn't look very good. :/
hope you can fix it soon.
Posted 7 months ago. ( permalink )

chrismid259    says:

@alexmuller Actually, looking at status.last.fm, they are in London.

:)
Posted 7 months ago. ( permalink )

Damianoo says:

so hot
Posted 7 months ago. ( permalink )

AnaYorke says:

Oh god...
Posted 7 months ago. ( permalink )

garwaymatt says:

Someone at Level3 is getting their nads trampled right about now...
Posted 7 months ago. ( permalink )

aritzb says:

The Radiohead's shoutbox. That's the problem.
Posted 7 months ago. ( permalink )

vovchychko    says:

Yowch!

And +1 to Master Bob, this is indicative of a problem that goes beyond chillers failing.
Posted 7 months ago. ( permalink )

Richard aka Dave    says:

Toasty!
Posted 7 months ago. ( permalink )

alasdairlumsden    says:

Yeah, Last.fm are located inside the Level 3 datacenter on Braham Street (at a guess - the graph has "Braham Street" in it!)

www.level3.com/userimages/dotcom/network/EU/l ondon_buildi...
Posted 7 months ago. ( permalink )

Musica Eternal says:

the servers are in London, as that is where Braham Street is. It has been hot today here, my pc froze a few times too haha
Posted 7 months ago. ( permalink )

JTWilcox    says:

I was trying to add a new venue to an event listing right when this happened. This is all my fault. I'm so sorry...
Posted 7 months ago. ( permalink )

drop_dead_red says:

Epic fail! D:
Posted 7 months ago. ( permalink )

lifeforms says:

ohshi-
Posted 7 months ago. ( permalink )

empoprises says:

JTWilcox, sounds like a hot new venue. :)
Posted 7 months ago. ( permalink )

Ernest Le Vampire says:

Burning ....... haaaaaa
Posted 7 months ago. ( permalink )

Tannyz says:

You should move the datacentres to Antarctica!!! (^^.)
Posted 7 months ago. ( permalink )

Ernest Le Vampire says:

Today Menu: Last fm broiled in their juices.
Posted 7 months ago. ( permalink )

TheMu says:

Aw man, when I clicked on the link I expected to see a picture of your servers on fire or something. That would at least make up for the downtime.
Posted 7 months ago. ( permalink )

vibjuan says:

Hawt servers D:
Posted 7 months ago. ( permalink )

Tannyz says:

^
lol ;)
Posted 7 months ago. ( permalink )

Bayou16 says:

good luck!
Posted 7 months ago. ( permalink )

Alessio Borrello says:

ooh!! damn!!
Posted 7 months ago. ( permalink )

Tonie    says:

@JTWilcox : no it was my fault. i'll admit it: that new profile picture i uploaded on last fm was just too hot...
Posted 7 months ago. ( permalink )

banditdee says:

that's unfortunate
Posted 7 months ago. ( permalink )

indiemixer says:

As long as my plays are scrobbled, I'm happy!
Posted 7 months ago. ( permalink )

Tannyz says:

The last post wins! ;)
Posted 7 months ago. ( permalink )

Aaron_frumkin says:

I'm glad that this is not the work of Amerika banning musics..
Posted 7 months ago. ( permalink )

- .:.    says:

shite : (
Posted 7 months ago. ( permalink )

Amanda Moksha says:

Aww, and it's absolutely crucial I leave someone a shout RIGHT NOW.
:P
Posted 7 months ago. ( permalink )

ViiperGuy says:

=-O
Posted 7 months ago. ( permalink )

Dahminator says:

as long as it keeps my scrobbles, I won't get too angry. Good luck fixing it!
Posted 7 months ago. ( permalink )

Tannyz says:

(^^.)
Posted 7 months ago. ( permalink )

Danzor says:

pewp. :/
Posted 7 months ago. ( permalink )

Tannyz says:

Let's play taste o meter ;)
Posted 7 months ago. ( permalink )

gray_um    says:

Level3 :(

Probably gonna be a long night for you guys, good luck.
Posted 7 months ago. ( permalink )

newlyfoundnoise says:

My recently played is still showing up in the scrobbler, so it's all okay :)
Hope everything gets fixed soon, though
Posted 7 months ago. ( permalink )

nucumnon says:

hope the database recovers soon... i'm addicted to checking my stats :P
Posted 7 months ago. ( permalink )

KatKMeanders    says:

I hope it isn't catastrophic!
Posted 7 months ago. ( permalink )

jtfearnley says:

Should I block the scobbler from connecting so it caches everything?

It seems to be submitting fine... and the submissions server seems to be operational:
status.last.fm/
Posted 7 months ago. ( permalink )

Amarinder Cheema [deleted] says:

i think staff forgot that summer time is here.
Posted 7 months ago. ( permalink )

Amanda Moksha says:

How hot does it have to get in London for this to occur??
Posted 7 months ago. ( permalink )

GabyrelIa says:

I think it's getting better! If you check the last.fm scrobbler button on the button right of the screen(Windows), while your playing music, it used to not have info, but now it does! :D
Posted 7 months ago. ( permalink )

Tannyz says:

Genius ;)
Posted 7 months ago. ( permalink )

Aaron_frumkin says:

Mark of the Legion...
Posted 7 months ago. ( permalink )

ViiperGuy says:

Viewed 11,486 times in 1 hour
Posted 7 months ago. ( permalink )

HD Gallery    says:

Good old England... a bit of snow and the country grinds to a halt, a bit of sun and the country also grinds to a halt! :)
Posted 7 months ago. ( permalink )

jtfearnley says:

I don't think a datacentre constitues a whole country
:L
But it is a pretty poor show! It was hardly 'that' warm today compared with other places...
Posted 7 months ago. ( permalink )

francescograssotti    says:

it's like the netherlands: a bit of snow and all transport stops, no train leaves a station anywhere, and on all the highways, and even on the mainroads of cities, there will be jams!
Posted 7 months ago. ( permalink )

Topener    says:

and we just wait for the site to be back up! Good luck guys! I know bringing servers back online is stresstime
Posted 7 months ago. ( permalink )

greensky.sk    says:

oh sh.., the global warming has the first issue!
Posted 7 months ago. ( permalink )

kimada says:

that's hot
Posted 7 months ago. ( permalink )

blueishgreykitty says:

@Master Bob actually the chillers can only handle so much. If one fails, then the temperature could potentially rise to a point to frying the motors (or other things) of the other units because they're compensating =) Hope the site's back soon!
Posted 7 months ago. ( permalink )

Deschanel..    says:

whew
Posted 7 months ago. ( permalink )

Danielus Matiatko says:

pfuff..
Posted 7 months ago. ( permalink )

ViiperGuy says:

whats the longest time Last.fm has been down?
Posted 7 months ago. ( permalink )

Danny_G says:

yep, move to North Pole
Posted 7 months ago. ( permalink )

KhaLduNTR says:

Now I understand I can't live without last.fm :) . When everything is ok I'll subscribe my account.. But I can't see my listened tracks in the diagnostics section of my scrobbler. Were they scrobbled??
Posted 7 months ago. ( permalink )

Philip Vasquez    says:

ROFL, a few days earlier you were bragging about uptime....


"The server with the longest uptime at Last.fm: 1244 days (3.4 years!) Can anyone beat that? I3M7B"

Big ups to the epic fail guys!
Posted 7 months ago. ( permalink )

Aaron_frumkin says:

^ Yeah, they must feel very silly about this whole ordeal after that...
Posted 7 months ago. ( permalink )

adaptivelabs    says:

That's hot....
Posted 7 months ago. ( permalink )

rjmancuso says:

That sucks. Hopefully the site is back up soon.
Posted 7 months ago. ( permalink )

dameemerica    says:

Yes, the testing moments always seem to come just when you think things are going well (like after 3.4 years of up time!) Good luck guys, we've all been there, though I am surprised that the backup plans don't seem to be in place for this type of fault. Lessons learned for the next time around.
Posted 7 months ago. ( permalink )

Master Bob    says:

@blueishgreykitty: Of course. My point was that it doesn't seem too far fetched for one chiller to fail, as such a backup should be in place or something along those lines.

Perhaps monkeys carrying fans ....
Posted 7 months ago. ( permalink )

World In My Eyes * says:

@http://twitter.com/lastfm
Apologies again for the outage. Your scrobbles should be cached by your clients, so none should be lost.

uhuhhu no perdi mis scrobbles =)
Posted 7 months ago. ( permalink )

lenehan    says:

time to cool off in the meantime
Posted 7 months ago. ( permalink )

gerxrdo says:

outch! my scrobbles! noew! i'm going to die xD
Posted 7 months ago. ( permalink )

andronicusmax says:

Haha brilliant.
Posted 7 months ago. ( permalink )

Nick Caron says:

wow jsut throw some wate r on them to cooll of, not that hardguys wtf is the problem
Posted 7 months ago. ( permalink )

hiddenspark42    says:

We've had the chiller pump fail and the backup pump not come online even after extensive testing with our data center. No one is immune.

Best of luck!
Posted 7 months ago. ( permalink )

mritz_p    says:

oh boy!
Posted 7 months ago. ( permalink )

sp0rk4prez says:

My Core2Quad CPU is running @ 100% load right now @ 50C. Barely breaking a sweat. :)
Posted 7 months ago. ( permalink )

Jonty Wareing    says:

@sp0rk4prez: That's your CPU temperature on-die, the graph shows the ambient temperature of the room the servers are in. The CPU safety shutoff is about ~70C and it easily exceeded that in our servers.
Posted 7 months ago. ( permalink )

drplokta    says:

Could have been worse. Three years ago, Level 3's other London data centre at Goswell Road had a complete power outage for five hours.
Posted 7 months ago. ( permalink )

lozzd says:

@Drplokta
Umm, in April last year Braham Street lost power for 8 hours. :) As well as being broken into..
Posted 6 months ago. ( permalink )

WatkinsHire says:

hire chillers in such situations....
Posted 6 months ago. ( permalink )