Sunday Downtime

And I wish I meant that as in “sat around in my pajamas reading the Sunday paper and having a cup of tea” downtime – but, well, here’s what happened.

Our Open Diary servers (there are several) were scheduled to be moved from their current (up until yesterday) server room into a new and improved data center (as of today). This move was supposed to begin at midnight last night and be accomplished within a couple of hours. It was supposed to happen in a phased move that would bring down the servers in stages and back up at the new location so that service interruption would be at a minimum. It was extremely well-planned and timed to the minute, as most things that go totally wrong are.

I was preparing to post a warning message about the upcoming move last night at about 11:00 – I knew that the site would be on and off for a couple hours, but at those hours traffic is pretty light – so I figured a simple “hey you might see some interruptions in service over the next couple hours” would do the job. As I was preparing to make that post, the phone rang – one of my children, calling from the side of the highway – the family car he was driving had experienced a service interruption of its own. So, when I should have been posting and monitoring the beginning of this move, I was standing on the side of the road with him, waiting for a tow truck. It was a case of too little coolant + too much internally-combusted heat = a car with no go.

By the time the car was towed and everyone was safely returned home, our sysadmins (the guys who make sure all of the pulleys and gears of OD stay in alignment) had already begun the move. I returned to the same “page cannot be displayed” page that all of you have been seeing for most of today – instead of putting up a nice, calming “we’re moving to make life better” screen.

To make an extremely long story short, the physical move of the servers took longer than it should, and then the sysadmins ran into some router problems in the new data center when they tried to bring OD back online. It’s really amazing how much traffic hits our servers in a few seconds after they have been down for awhile. It was a classic man vs. machine battle, but in the end mankind won out (and no sentinel-fighting or hovercraft-flying required). In any case, with some tuning and tweaking, everything is now running as it should. I spent my Sunday on the phone with the sysadmins and pressing the Refresh button over and over and over, so I hope you all had a better day.

The good news is that the new data center provides a much more robust environment for our servers – there is better physical security, better environmental control, better (and faster) communications equipment, and an improved disaster-recovery infrastructure. All (okay, not all but many) of the things that make us geeks happy, and a much better place for the machines of OD to live.

I do apologize for the hours of downtime – it was mostly unforeseen, and (unfortunately) unheralded. Next time I’ll post my warnings about technical stuff before one of my kids borrows the car keys. I will be crediting the accounts of all OD Plus subscribers with an extra day to make up for the lost time.

Enjoy what’s left of your weekend,

The DiaryMaster

Thanks for the info! Your kids do come first, so all’s well that ends well!

i was sad to see it was down but i’m so grateful for the update. that’s all i need to keep me happy. and that extra day doesn’t hurt either 🙂 thanks so much for all of your hard work.

kids these days!

Thanks for the explanation. You’re a good Dad, and that’s the most important thing.

Thanks much for all you do. Have a feeling you two are just as fabulous as parents as you are fabulous here : ) Hope you get some of that preferred sort of down time soon.

Thank you very much for doing this for us, it must have been a pain. Your dedication is very much apprecited. At least from me. =D Mel

personally DM, I have to say thanks, because with OD being down, I actually got some homework done 😉

Do you know if anyone else has been having problems with getting blank pages? Periodically (usually when I need to read something the most) all I can see are the menus and nothing else. I thought maybe it was my browser, but it’s set at its default settings. Any idea?

Kids will do that to you everytime. You are doing a great job, I have no complaints. Enjoy your Sunday, or whats left of it.

poor guy. I know that diarists often give you a Hard Time whenever the site goes down… (I always find that So curious.) Just in case, I want to help balance any bratty stuff with a, “Not to Worry,” and a, “Thanks for this Great Place.” Hope next sunday is a better one. regards,

No worries!!!! Thanks for helpin us out?! haha and keepin this site free!!

hope the kid is ok even if the car didnt make it.

Tanti grazie! I was a bit alarmed but figured it was something temporary.

Thank you SO much for clearing that up! I was worried that you crashed or something. Can this site be viewed at opendiary.com too, by the way? I have been told that it now can.

Thank you.

I just wanted to leave you a quick note to tell you what a better environment OD has taken on. I like that it feels more like a community now, with you to watch over everyone and update us all on what’s going on – whereas before, it was very much a man vs. machine, fingers crossed for even a tiny explanation to what was wrong this time.Thanks for the turnaround 🙂

Awesome job! (As always!) I’ve never left you, Sire, The Almighty DiaryMaster, a note…but I had a strange feeling about the site being down. Thank Goodness that this was planned downtime, and not something unknown and disasterous! (knock on wood) thanks again! Good luck with the family vehicle! PS – the site is running MUCH faster! Woo hoo!

Why were we not told at least a week in advance, with reminders as it approached?

Well, I about went into withdrawal, but I’m glad sonny-boy is safe. 🙂

Thank you! 🙂

Thanks for all of your effort!

i’m glad your son is ok and thanks for the x-tra day. i know some people voiced whether you’d be doing that or not, including myself. so again, thank you!

I’m sorry about your car. And You shouldn’t have had to explain. OD is pretty reliable about being online 🙂

And we thank you for it!

Whew! Thanks for letting us know what was going on. I was messaging my fellow ODers on AIM, asking if they could get on, blah blah blah… fun stuff. I was glad to know that I wasn’t the only one, but still, a tad frustrated. I’m glad you and yours got home safely, I’ve run into overheating problems myself. Take care,

Well! Haven’t YOU got some nerve having a family and a life that doesn’t focus on OUR needs! *huff* *harumph* (That stuff dripping down the inside of your monitor is sarcasm!) Anyone who can’t understand that your child came before this site has no soul!

For some reason, I never pictured you old enough to have a kid that could drive. Eh.