ForumsNewsAnd we're back! (from a very prolonged outage)
And we're back! (from a very prolonged outage)
| Author | Message |
|---|---|
|
Kevin Christy |
Stuff does happen, but let me just say that task management is too mission critical not to have a desktop client. During the rare times I don't have internet I don't sweat it, but not having access to my task list while at my desk needing to reference it and make changes is pretty painful. Desktop app: I'll pay for it! Just make it happen!
|
|
Ann |
I am a pro-subscriber, and my subscription was going to end in august. I could do without the pro-features, so the previous days I was wondering if I was going to pay for a pro-subscription again.
Because of the way you handled things today, it was clear for me, that maybe I do not NEED the pro-features, but that you deserve that I'll be a pro-customer at least for the next year. I paid for my second year already... just to discover that you gave me an extra month. I really, really, really do appreciate the way you communicated with us today. And go sleeping now, please... |
|
morkemo |
Thanks Jake! I hope other companies can learn something from your excellent work today!
Tore |
|
jamfac |
Is there a way to do automated scheduled exports? this would have really helped me today. Even better would be a desktop client (like evernote).
|
|
ThinkingSerious |
I'm glad you got things squared away with relatively minimal damage. I was a bit sick also, not having my Toodledo, fortunately, I did a sync yesterday morning with your iPhone app.
SUGGESTION It would be great to have Toodledo work with Google Gears like RTM. |
|
Peter Scott |
Your honesty and forthrightness are a refreshing break from the CYA scripted blandness of most companies. Knowing that there is a real person up there who can exercise judgment and free will instead of being a corporate robot, I intend to keep my pro account and recommend Toodledo to my clients.
|
|
sanders |
I greatly appreciate your candor. You handled the problem like a true professional. Transparency fosters customer loyalty. I appreciate all your efforts. Things like this happen.
Sidenote: Toodledo.com & iPhone Toodledo are in need of a cosmetic makeover. The apps look dated. Any chance this is forthcoming? Thank you, Loyal customer |
|
cesugden |
Man one day without you at work and I was lost!
Great way of handling things, really appreciated the updated messages. Think you did a great job. It's fantastic that you are so open about what happened, in fact am so impressed I'm signing up for your top account! Bad luck, but great job and great website you've produced. |
|
kmf |
Awesome job Jake. I think you need to take a long weekend :)
For a moment there I thought I had to get a RTM importer. You have just proven, yet again, that honesty is the best policy. Thanks for that. Karl |
|
TheGriff_2 |
Have to admit the others who have subscribed due to the handling of the situation inspired me. I had been a Pro subscriber but am now a Pro Plus subscriber. Not sure if I'll use the file space but you deserve the support!
|
|
DavidSzp |
I came here to find I've been beaten to the punch by a bunch of people already! Most have said what I was going to. Basically, downtime happens, it was handled very well (I saw Amazon EC2 went down too, can happen to anyone!) and the fact that there was no data loss is excellent. I'll echo someone else who said (as a Pro user) that I won't be asking for any refunds, and I do appreciate the month credit but I'd have stayed without it.
This is the first outage I've seen for Toodledo since I signed up sometime last year, outside of planned maintenance and a few times that connections were flaky for a few minutes (more network than server issues I'm sure) late at night here and there. That's an excellent track record, and if anything, the things you've changed and learned from this downtime makes a repeat performance that much more unlikely to happen again! The only thing I can think that would be more disaster-proof is having a real-time slave replicated to an entirely different datacenter in another part of the country, but that would increase costs in bandwidth and other areas. Thanks again and good job! (Forgot to mention so I'm editing: Great job on communication, the updates on the homepage were great and detailed, and your explanation here on the forums was excellent. I love working with people who are honest about their mistakes and provide the details--details mainly out of my own technical curiosity :-) This message was edited Jun 11, 2009. |
|
Rietze |
Outages really suck don't they? I've been doing IT a long time and more than once had to sleep on a data center floor worrying about exchange or backups. I feel for you man.
Regardless of the outage, I still love your service. |
|
Leo.Eumelen |
I'm trying Toodledo for some days now and was thinking about upgrading my account. Now I know for sure - my upgrade is on its way. Great job and you learned some things to ;-)
|
|
nick |
Jake,
Great job. I just dealt with this same thing 2 weeks ago. I was literally sick to the point where I wanted to vomit! I had two raid 5 machines in a cluster with a NAS device attached for backups. The first node of the cluster went down and then while we were working on the first node the second one went down. Your thinking is with the cluster that you should be good. If one machine goes down the other just steps in and does it's thing. What are the odds that BOTH would go down. (ROFL) We had no bare metal restore disk (big problem!) so we had to build the a new server from scratch and then transfer the files from the backup NAS (SO SLOW!!!) Fortunately in our case having a lag time in data was no big deal, since all the realtime data is loaded via web services from a remote database, the data on the machine itself is batched nightly. We did eventually get one of the machines from the cluster back up and running, however it took over 16 hours to rebuild the raid. I wasn't about to push that back into production though. Then the PHP version on the new server was the latest stable 5 release and the site was still using php 4.x All of our web services code was using an old XML parser that wouldn't run on php 5.x... This was on our "to do" list to update, however I didn't expect to do this rewrite that code at 2am after working for 16 hours straight! My brain was so fried, it was insane. 20 hours later we got the server back online on a brand new machine while we sent the cluster back to the hardware vendor to troubleshoot/rebuild. It was a hardware issue with the raid cards on those machines, basically they were a ticking time bomb from the moment we purchased them. All of this gets sorted out and I go to check on our other clustered server to make sure it wasn't going to have the same thing happen. I started looking at it when the thing went down, both machines with corrupted raids again! Talk about murphy's law. We were able to get that one back up in a new machine in about 14 hours. Reconfigured another brand new machine and copied the files. This time the raid was much smaller so it rebuilt in not too much time. So we were able to just copy over live data after we setup the new machine. We could have had it back up sooner, but my brain was so fried I couldn't type or think very clearly. The only bright side of the situation is I killed a bunch of tasks for upgrading scripts, out of necessity, when we preformed these "under the gun" updates. Needless to say we re-evaluated our backup solutions. We now have a full configured hot spares just waiting for latest database/file snapshots, bare metal restore disks etc. At least you had a solution, one of my earliest experiences with hosting companies, I got a shared hosting account and did a bunch of work, probably 8 to 14 hours. I went to show it to my client and the hosting account was down. When I wrote the hosting company they said "a fire burnt your server and your backup tapes" I would post the name of the company but i would be surprised if they are still around. This was 1998 after all. The point of this story is nothing sucks worse than server outages! The worst! -Nick |
|
Dan Stratton |
Thanks for the news, Jake. Those of us in IT have been caught by the Disaster Bug once (or even twice). This outage may cost me though. I have an iPod Touch and still had access to all my tasks. My wife, however, didn't and she was looking over my shoulder, extremely jealous that I could still function. So, I will probably be having to buy her one soon. :-)
Thanks for Toodledo. We're not going anywhere. |
|
Sheila_1313615210 |
Congrats on the way you've handled this and the ability to keep from losing data! I have been so far behind I've been considered not even looking at my account because it SO bums me out. Today when I got your explanation and NOT my list I realized I may not like how long my list is but I sure need it!!!
Thanks for doing such a great job!! Now go rest..... :) |
|
peter.george.sherman |
Posted: Jun 11, 2009
Score: 0
Reference
Thanks so much...it's heartening to see how much Toodledo means to it's admins.
I will most certainly be continuing with my pro-subscription in 2010. This message was edited Jun 11, 2009. |
|
weisspatti |
I think Toodledo is the best deal for the money around. I am impressed by how you explained "skeletons and all." While I was missing my Toodledo while it was down, things happen. Anyone who cancels their subscription or is mad has never been in a similar situation and just does not understand. You did a great job in the face of enormous stress and your humble responsibility for issues tells a big story about your great integrity. Thanks for all you do.
|
|
nick |
Oh I forgot to mention, I was happily using the toodleto app on my iphone during the outage. Just couldn't sync.. Love that app!!
|
|
MacBernac |
I just upgraded to Pro today, not yet knowing of this event. To my delight, I discovered I had a 13-month subscription instead of the 12-month I was expecting. Then perused the forum and learned of the outage and your gracious extension of a free month. Thanks much!!
|
You cannot reply yet
U Back to topic home
R Post a reply
To participate in these forums, you must be signed in.