Incident Description:
Hello all,
Server iis409 is currently down. All of the content is in the process of being copied and the server will be fully recreated and back up in, what Anton calls an “optimistic estimate” of 4.5 hours. We know that this is long and we apologize. We’re all scrambling to make this server right again, because it being down is all wrong.
We will keep you posted as we know more. Please follow our status Twitter, IXStatus as it will be frequently updated as well. Thank you for your patience/subdued fury!
Update @May15, 1.43PM EST: Hello everybody. The server is up, and has been up since 7.27am this morning. I didn’t update this status blog because I was sleeping (long hours last night), and my coleagues didn’t update it because they didn’t have permission (at the time) to update it. No databases were lost in the process, because we host databases on different servers. All data should be present on the server as it was at the moment the server went down.
RJS says:
Is this the same server that has been down for over 24 hours ? I hope you replaced faulty hardware and didn’t just reinstall software. We can’t affort to have this slow down or crash again !
Link | Posted at 3:28 pm EST
Tiberiu says:
Of course we replaced the hardware. All of it. And we actually added a little bit more ram too, and a new generation of CPU. So when it comes back, it should be better, faster… awesome-er
Link | Posted at 3:33 pm EST
Corey says:
How are we looking with getting the server back in the action?
Got people ringing me that my sites been out of action for over a day now.
Link | Posted at 6:56 pm EST
Tiberiu says:
@Corey: the rough ETA is 24 hours. We’ve had rare problems with windows servers, but when a server gets fubar-ed, it usually takes a huge amount of time to restore it.
Link | Posted at 7:08 pm EST
Jeff says:
So is the server going to be down for another 24 hours, or just another couple more according to the last ETA?
Link | Posted at 7:13 pm EST
Tiberiu says:
Jeff: While hiding behind a big AC unit (so you can’t hit me) I have to say the ETA is 24 hours from now. (according to the guys that are actually working on the server right now, the optimistic ETA is 18 hours, the pessimistic one is 26-28 hours). All this time is needed to restore content of the websites on IIS409 from offsite backup to the server itself. It’s a little over 1TB of data, spread across a few billion files.
Link | Posted at 7:18 pm EST
Jeff says:
No need to hide, just wanted to know. Thanks for doing your job.
Jeff
Link | Posted at 7:30 pm EST
Allan says:
Wow! 24-26 hour is over. Anything?
Link | Posted at 1:55 am EST
RJS says:
I think they are saying it will be over 48 hours, not 24. It was another 24 hours at 7:08pm EST. This outage is killing my business. Please add lots and lots of Ram and smoking fast hard drives to make up for this outage.
By the way, are you runing Raid 5, 10 or 0 ?
We know you guys are working hard to get this back up. Thanks
Link | Posted at 2:59 am EST
rmpal says:
Recent past i am facing this service issues with ixweb, which i was not having last year.
There should be some fundamental issue in sever outage very often, the infrastructure team should re-engineer their processes in server farm maintenance.
Hope it will improve. get the server up fast….
Link | Posted at 4:07 am EST
Allan says:
Oh my god, this is getting ridiculous!
Link | Posted at 4:44 am EST
Ken says:
I’ve been told several times for past couple of days it will be repaired 3-5 hrs, do you have a realistic time?
Link | Posted at 10:28 am EST
RJS says:
IX What is the latest – Please update us !!
Link | Posted at 12:32 pm EST
SH says:
hey guys, could you give us an estimated time? I believe you are that much experienced to give us at least an estimated time when the sites gonna be up. Can u please confirm if my files, directories especially my DATABASES ARE OK? I would really appreciate if u confirm us.
Link | Posted at 12:52 pm EST
Tiberiu says:
@Ken, RJS and SH: your updates are a few hours after the server has actually been back online. Are your sites still down? They shouldn’t be.
Link | Posted at 1:45 pm EST
RJS says:
I had to move my sites to another server I have with you while repairs were being made as it was costing me a forturne to be down. You guys should send email out when a server is back online as we are not able to check every hour. It looks like it is up again. Thanks for getting things back to normal. Time to move over my sites again.
Link | Posted at 1:56 pm EST
Joan says:
My site is still down and I cannot ftp to it. 15 minutes ago, I could ftp, but my content was not there — only the webalizer folder and another one. I am hopeful that you are restoring content now, and that is why I cannot ftp.
Link | Posted at 2:25 pm EST
Tiberiu says:
@Joan: is it IIS409 related or WEB405 related?
Link | Posted at 2:28 pm EST
Ken says:
My site is up and running. Everything seems to be back to normal.
Link | Posted at 8:42 pm EST
RJS says:
Hey, just turned in a trouble ticket. iis409 Server appears to be down again or at least my FTP and Web sites are all dead, have been down for hours looks like by my we site counter. What’s up ? Hope the “new” hardware didn’t fail again.
Link | Posted at 12:04 am EST