ForumsNewsRackspace failed us again


Rackspace failed us again
Author Message
Jake

Toodledo Founder
Posted: Jul 07, 2009
Score: 0 Reference
This is getting really really frustrating. Rackspace (who runs our datacenter) had another power outage today (their 4th). This affected us for about 45 minutes. We are currently back up and running without any data loss. However, we are 1 server short right now, so everything is going to run slowly until we get that other server back online.

I know that 4 outages in a few weeks makes Toodledo look totally amateurish and ridiculous and I assure you that we are working like crazy to migrate everything to a more reliable datacenter. Migrating everything is a complex coordination issue that takes some time to test and complete, but it can't happen fast enough.
karin.rosner

Posted: Jul 07, 2009
Score: 1 Reference
I still love you, Toodledoo. I hope we get this sorted soon. I saw the outage and went to lunch.
MFA

Posted: Jul 07, 2009
Score: 0 Reference
biting my fingernails there for a sec
jalex508

Posted: Jul 07, 2009
Score: 0 Reference
My company uses Rackspace and we have no issues. It's that Dallas Datacenter.
Anders

Posted: Jul 07, 2009
Score: 1 Reference
Yes, it was at least nicely timed for lunchtime on the east coast. I wouldn't be surprised if Rackspace winds up with a lot of free rackspace before long :)
gadgetude

Posted: Jul 07, 2009
Score: 0 Reference
Do you have a proposed time line for the data center move?
jbeckham

Posted: Jul 07, 2009
Score: 0 Reference
Thanks for keeping us posted on what's going on. Me not having my todo list isn't quite as bad as you having to deal with all this crap... good luck.
Jake

Toodledo Founder
Posted: Jul 07, 2009
Score: 0 Reference
Our proposed timeline for the datacenter move is ASAP.
Andrew

Posted: Jul 07, 2009
Score: 0 Reference
Dare I ask where you are moving to?
Linden

Posted: Jul 07, 2009
Score: 0 Reference
Yes, I too lucked out and only experienced the outage over lunch.

Sorry to hear that this power issue is STILL ongoing. Best of luck in shopping and migrating hosts.

I'm not especially conversant in hosting issues, so forgive me if this is an impossible/expensive/whatever idea, but:
Might it be possible to have a server with one host and another with a different host?

Rackspace, from what I've been hearing, is generally one of the top hosting solutions available. So I'm gathering that others might have similar problems. If it's possible/affordable, it could be nice to have servers with two companies so that an outage with one still leaves the site operating with the other -- regardless of who it is.

In any case, I'm definitely sticking around while you sort it out. The system has been priceless for me identifying and acting on my goals!
bsglaser

Posted: Jul 07, 2009
Score: 0 Reference
I'm suddenly missing my Gmail Gadget...is this related to the outage?
Jake

Toodledo Founder
Posted: Jul 07, 2009
Score: 0 Reference
The other server is back online, so we are 100% back up now.

To answer Linden's question, while it is possible to have servers at different datacenters, the challenged would be keeping the data 100% in sync between the two datacenters. This would require a high-bandwidth private connection between the two, which might not be possible or might be cost prohibitive. The other issue is that switching traffic from the failed datacenter to the working datacenter would take time due to the way DNS works. So it would really only be protection from a major catastrophe, like a fire that took our server offline for multiple days. But we are exploring this option to see if it is affordable.
Proximo

Posted: Jul 07, 2009
Score: 0 Reference
I understand this is frustrating, but I believe many of us can appreciate the complexity of migrating a to a new data center, even if we never experienced doing so at such a large scale.

I just switched hosting services for my personal business and website on a much smaller scale. I did not plan as well and lost some data.

So please, take your time and do it right. We trust the backup system is working because it's proven itself recently.

I am not sure where Toodledo is located, but some suggest you are here in Dallas. If that's the case, I would love to visit one day. :-)
frossie

Posted: Jul 07, 2009
Score: 0 Reference
Toodledo must have been unlucky - this time some of their Dallas cluster managed to stay up on UPS. Aren't you sure you don't want to stay at Dallas and keep getting the service credits? :-)

Seriously, do you really want to spend the effort on moving datacenters? Rackspace are bound to fix this, especially since they are now becoming a bit of a joke over this center.
frossie

Posted: Jul 07, 2009
Score: 0 Reference
And if it makes you feel any better, check out these poor guys:

http://smugmug.wordpress.com/2009/07/07/7709-site-offline/

(they are a great service too, as you can tell from the honesty of their announcement)
Linden

Posted: Jul 07, 2009
Score: 0 Reference
Posted by Toodledo:
To answer Linden's question, while it is possible to have servers at different datacenters, the challenged would be keeping the data 100% in sync between the two datacenters. This would require a high-bandwidth private connection between the two, which might not be possible or might be cost prohibitive. The other issue is that switching traffic from the failed datacenter to the working datacenter would take time due to the way DNS works. So it would really only be protection from a major catastrophe, like a fire that took our server offline for multiple days. But we are exploring this option to see if it is affordable.


Thanks for the answer. Sounds to me like the cons/difficulties outweigh the potential pros -- especially since the pros appear to be negligible.
Anders

Posted: Jul 07, 2009
Score: 0 Reference
@Frossie Wow, that's terrible, and unlike Toodledo, sadly self-induced. They are one of the biggest photo sharing sites out there, aren't they?
frossie

Posted: Jul 07, 2009
Score: 0 Reference
Anders - yes, they are a photo hosting site and actually they are really great. Not only is the interface really well designed, but they have lots of different levels of privacy control, which is really nice for those of us who are not quite as exhibitionist as the youngsters these days :-)

On the overall issue, I think customers are willing to tolerate some level of service interruption if it is infrequent and quickly and honestly dealt with. The thing people really can't tolerate is data loss. I don't think a business could recover from that.
Anders

Posted: Jul 07, 2009
Score: 0 Reference
I agree. If you can't live with occasional interuptions, then you basically can't use the internet, but people do like to be kept in the loop. Both websites have handled that part quite well.
kaipaniolo

Posted: Jul 07, 2009
Score: 0 Reference
Our radio station corporate email is out of the same Rackspace server farm. When Toodledo is down, our email usually is too. Though, we are in Hawaii, so when this happens, we can just take the Cell phone to the beach and wait it out. :-)
You cannot reply yet

U Back to topic home

R Post a reply

Skip to Page:  1   2      Next

To participate in these forums, you must be signed in.