ForumsNewsToodledo migrating to new servers


Toodledo migrating to new servers
Author Message
Jake

Toodledo Founder
Posted: Jul 29, 2009
Score: 4 Reference
Many of you may be aware that we have experienced some unscheduled periods of downtime in the last month. The root cause of the problems are that our data center, run by Rackspace.com, is old and doesn't have enough electrical redundancy. When their utility feed went down, they attempted to switch to generators, but the generators failed. Rackspace has nine data centers around the world, and it turns out that Toodledo's servers were in the oldest and least redundant one.

It became apparent after the second power failure that we needed to migrate to something more reliable. We investigated a lot of different options and ultimately decided to stick with Rackspace, but to migrate to their newer data center in Virginia.

You may say that we are silly to stick with Rackspace after all the power failures and grief that their issues have caused us, but we don't see it that way. Hardware issues happen to everyone, and it is how the people at Rackspace responded to these issues that set them apart. Throughout the entire ordeal, Rackspace was amazingingly helpful and responsive to our requests and spent an enormous amount of time with us to get everything working again. I seriously doubt that we would have received that amount of dedicated support from another company.

The data center that we are moving to is state of the art. I have been personally told by both the CTO and VP of Operations that this datacenter is the best one that they have ever built and that the power issues that occurred at our old datacenter could never happen in the new one due to complete redundancy in the power infrastructure.

The new upgraded servers from Rackspace are also much better. They have their own built in redundancy and greatly increased capacity that should give us a lot of additional room to grow in the future. The following diagram illustrates our old configuration vs our new configuration.




You'll see that most of the single points of failures have been replaced with redundant systems. On the whole, this is a substantial upgrade for us that should greatly increase our reliability, scalability and performance.

We have been extensively testing our new servers over the last few weeks and we are almost ready to switch over. When we flip the switch, it will require a short window of scheduled downtime. We don't know exactly when that will be yet, but we will do our best to give as much prior notice as possible and schedule it for an off-peak time.

We have done a lot of testing on the new environment, and we are fairly certain that the switch will be seamless, but with a migration as complex as this one, we would not be surprised if there were some small bugs introduced along the way. We will do our best to respond to these as quickly as possible.

We are taking multiple precautions with data integrity (currently all of your data lives securely on 6 separate servers in 3 different geographic locations). If you want, you can also make your own backup by using the XML export tool.

We'll update this post as we have more information to share.

Thanks!


This message was edited Jan 08, 2010.
Vin Thomas

Posted: Jul 29, 2009
Score: 1 Reference
Great news! I am glad to see Rackspace acting so responsibly with you!
Joshua

Posted: Jul 29, 2009
Score: 1 Reference
Excellent news. I also appreciate the explanation.
todoing

Posted: Jul 29, 2009
Score: 0 Reference
Wow, great setup. Are those redundant database servers or are you splitting users across servers?
gui

Posted: Jul 29, 2009
Score: 2 Reference
You guys are great! In a field where some developers (Cultured Code/Things) remove their forum because they do not wish to read complaints or when RTM does not even bother to make releases or answer forum requests, you share something like this.

I am with Toodledo for all the right reasons.

Now, let's talk about that UI......
Jake

Toodledo Founder
Posted: Jul 29, 2009
Score: 0 Reference
To answer todoing: They are redundant databases, so that if one fails, it instantly switches over to the other without any interruption.
Andrew

Posted: Jul 29, 2009
Score: 0 Reference
That is really cool. Is the redundancy easy to set up, or did you have to spend hours and hours coding it yourself? It would be great if it was easy to load-balance any site you wanted to easily.
Jake

Toodledo Founder
Posted: Jul 29, 2009
Score: 1 Reference
It took 2 solid weeks to setup and test, so I wouldn't say that it was easy. If you have a little site, the best bet is to use something like the Rackspace Cloud which is automatically redundant and scalable, but Toodledo is too big for that.
Claudio

Posted: Jul 30, 2009
Score: 0 Reference
Great news!

It's good to know that my vital task management data is safer than anything I could hope to do with my own systems.
Lance

Posted: Jul 30, 2009
Score: 2 Reference
You guys ROCK! It's not always about the $$ from a customers perspective, but the service. Speaking for myself, but I think many out there feel the same, I'm willing to pay a few dollars more for consistently good, quality service. And Toodledo clearly exemplifies this. It's a shame, and perhaps a sad indicator of our society, that there are so few businesses out there that do the same. Thanks for keeping us in the loop.

p.s. I'm a RTM convert because they just didn't "get it."
Proximo

Posted: Jul 30, 2009
Score: 0 Reference
Greatness.

Thanks Jake
Linden

Posted: Aug 01, 2009
Score: 0 Reference
Wow! I didn't check the news for a couple of days and came back to find this!

That is an amazing upgrade, and I'm happy to hear that you're able to do it with the same company. It sounds like Rackspace has given you the same top-notch customer service that you provide us.
MFA

Posted: Aug 13, 2009
Score: 0 Reference
Well done! TD rocks.
Allen

Posted: Aug 13, 2009
Score: 0 Reference
Congratulations on the quick and painless move!
Proximo

Posted: Aug 13, 2009
Score: 0 Reference
Mission accomplished. Everything is good with the world.
Allen

Posted: Aug 13, 2009
Score: 0 Reference
Posted by Proximo:
Mission accomplished. Everything is good with the world.


Well, almost . . . took me 10 minutes to get past the "Toodledo is Offline" screen just now. D'oh!
Jake

Toodledo Founder
Posted: Aug 13, 2009
Score: 1 Reference
It was just offline for about 2 minutes (not 10). I was trying to make a backup of the database and mistakenly took it from the online server instead of the offline server. As soon as I noticed, I stopped the backup. Sorry about that. I'll be more careful in the future.

I'm still learning how the new environment works a little bit :)


This message was edited Aug 13, 2009.
Allen

Posted: Aug 13, 2009
Score: 0 Reference
Only 2 minutes? Something must have gone wrong on my end, too, then . . . I went out to my car and came back between tries. Can't do that in 2 minutes :)
Jake

Toodledo Founder
Posted: Aug 13, 2009
Score: 0 Reference
Your browser may have cached the error page. The backup only takes about 10 minutes to finish, and I canceled it way before that.

This message was edited Aug 13, 2009.
pepe0612

Posted: Aug 17, 2009
Score: 0 Reference
The data in one of my filders is gone and was replavced with the below message. I have waited for many days but still the original data is not showing. Please help as this was important information:

<DIV style="MARGIN: auto; WIDTH: 450px">
<P style="TEXT-ALIGN: center">

Toodledo is currently undergoing scheduled maintenance.

We are switching over to newer, better, faster, stronger servers. We'll be back online shortly. </P></DIV>
You cannot reply yet

U Back to topic home

R Post a reply

Skip to Page:  1   2      Next

To participate in these forums, you must be signed in.