(Why) is FastMail down? I want better answers.

[Onno]

Just thought I'd mention there is some sort of precedence of this happening before. There was arguably even more bad luck involved there - GMX.net (back in the day) already had several data centres. One got knocked offline by a massive lightning strike, the other developed such problems dealing with the resulting load that it died completely also. They were completely off the air for 2-3 days.

Similarly, sometime last year on the main Internet Exchange in Amsterdam all power was lost, and for some reason the backup power didn't kick in. A very large part of the Dutch Internet connection to the world (and a portal to the rest of Europe) was offline for the night while people frantically got their routers back up when the power was restored. Routers don't like having their power cut. At all.

What I'm trying to say here is that some things you just cannot prepare for. We do not know what cascade of events have led to the current situation, but of course lessons will be learned once we find out more information.

[davidbstanley]

Is it possible for FM to have a backup on a different continent? Or is this out of FM's reach at present?

[pintsighs]

I think Jeremy has already answered this:

Originally posted by Jeremy Howard
When we looked into this we found we would have to, at a minimum, triple our prices to maintain such an environment (due to bandwidth, maintenance, sysadmin, and admin costs). We felt that most of our customers would not be prepared to pay that amount extra to avoid occassional outages due to major events like this.

[Kyle Babich]

IMAP Access Restored @ 6:44:41 EST

[funso]

Web Access restored

[elvey]

Originally posted by FMRocks
One request to the mods: Can someone please merge all the threads concerning this together (EXCEPT Jeremy's notification message)? Thanks.

Please DON'T do that. I started this thread because I wanted to focus on the specifics of what happened.

Again, why:
#1: The european backup servers were down. They were not accepting mail, that is. Why?

#2: The primary servers went poof. (to use the formal technical term) Connectivity? Power?

Major backbones and colo centers are supposed to have generator backup power, not just battery backup power.

I think these are what need to be answered (when the dust settles) so that people can have confidence in the system.

Presumably FM will be getting a refund from NYI, as their uptime guarantee hasn't been met. (but that's getting off topic...)

[edit: Jeremy has already begun to answer #1 - part of the problem was that the server was just overloaded.:http://www.emaildiscussions.com/...392#post128392
But, it was also refusing mail to virtual domains at the RCPT TO stage.
And I agree with what he said here ]

[kander]

Originally posted by Edwin
They had contingency plans in place. They host at a data center with backup power. The largest electrical outage IN HISTORY (i.e. by definition a unique event, and so not one that could have been adequately planned for) has knocked out some vital link between Fastmail's servers (which I'm 99% sure are working just fine - probably better than usual because there's no load!) and the outside world.

Good thing you didn't place a bet on that 99%-sure... As Rob pointed out in the other thread in fact their servers WERE down, backup power failed to kick in due to a damaged pump...

--K

[Jeremy Howard]

Thanks for your important questions, Elvey. Rob has tried to answer them, plus various other frequently asked questions, here: http://www.emaildiscussions.com/...threadid=14963

[Jeremy Howard]

Originally posted by elvey
But, it was also refusing mail to virtual domains at the RCPT TO stage.

Actually, it was simply deferring connections when sender verification failed. See Rob's post I linked to above.

[oysterquartz]

Originally posted by jwyfchan
Oh please, FastMail doesn't charge anywhere near a premium. Sorry. I think US$40 a year for an annual membership for their top level account is simply a 'steal'. If you want your type of reliability, I would only **think** about complaining that loudly if they start charging me US$40 a month....just some reality check here old chap.

But Yahoo! or Hotmail offer superior reliability for free

I don't think that $40 per year is a "steal". It may be fair, but people would expect that the European data center would be able to kick in and take over in such eventualities.

This issue isn't why fastmail.fm per se is down, but

(1) is it a weakness in their chosen provider (NYI) which is located in a "vulnerable" area after all given the events of 9/11, and,

(2) what exactly does the European fastmail data center do?

Personally, I don't get that upset about outages, as I make sure my business email isn't handled by fastmail or any other small provider. Personal email can wait after all

[kander]

I think I can answer 2... correct me if wrong.

The European server serves as a backup SMTP server, making sure that mail can still get queued there even when the 2 US servers have been abducted and held for ransom. When the US servers come back the EU server will begin sending it's mailqueue there.

The Euroserver also hosts the scripts that check if everything is running smoothly, and an alternative location for the documentation.

--K

[pcguy]

Originally posted by oysterquartz
But Yahoo! or Hotmail offer superior reliability for free

When was the area that houses the Yahoo and Hotmail servers hit with a area wide blackout?

[kander]

Erm, I think that in order to take Hotmail down there'd need to be a global takedown of the entire world-wide grid.. I am sure they have servers in just about every corner of the world :-/

--K

[Edwin]

Originally posted by kander
Good thing you didn't place a bet on that 99%-sure... As Rob pointed out in the other thread in fact their servers WERE down, backup power failed to kick in due to a damaged pump...

Sorry, my 99% didn't include datacenter incompetence in having a single (seemingly untested) point of failure. I've read tales of datacenters surviving exploding transformers, failed primary generators - even floods - so to be stuck because of a pump is something so mundane I didn't even factor it in.

Sounds like NYI didn't have enough redundancy - they should have had a backup generator totally independent of the main generator (different circuits, fuel source, make and model)

[kander]

I think nobody expected that, Edwin... my comment wasn't really serious either, I mean, who could possible consider that a datacenter makes such an inexcusable mistake? What good is a backup generator if it's not properly maintained?

--K