Current Time: 15:26:04 EST
 
09 Feb 2013 11:26 pm

MAIL308 Down for Maintenance- Resolved

Posted In: Outage — Feb 09th, 2013 at 11:26 pm EST by IX: Omari J.

Incident Description:

This server has gone down due to a file systems error check.

UPDATES BELOW

Which Customers are Impacted?

All customers on mail308

How are Customers Impacted?

All emails will be inaccessible

How often will we be updated?

4 hours

Time to Resolution (ETA)

4 hours

Incident Updates

  • 2/10/2013 12:59 A.M. - The initial FSCK has failed and restarted. ETA will be 4 from now. Another update will be provided when more information becomes available.
  • 2/10/2013 3:51 A.M. - The FSCK is currently at 54% and the ETA has been extended another 4 hours
  • 2/10/2013 9:07 A.M. - There was an error with FSCK and it has had to be restarted. It is currently at 17%. Current ETA 4 hours.
  • 2/10/2013 12:07 P.M. - We have had to take the server offline to verify the raid. ETA is an additional 4 hours.  Current messages are being held in queue to be delivered once this is finished.
  • 2/10/2013  7:13 P.M. - The mail server is currently in the "Verifying" stage and does not give a progress bar so we do not have a percentage or ETA at this time.
  • 2/11/2013  12:27 A.M. - The mail server is currently in the "Verifying" stage and does not give a progress bar so we do not have a percentage or ETA at this time.
  • 2/11/2013  04:40 A.M. - The mail server is now able to deliver.  Mail delivery may be delayed while the server works through the email queue.  
  • 2/11/13 12:53 P.M. - Mail server has been rebooted and access has been restored.
  • 2/11/13 8:11 P.M. - There were still issues with customer logins on the server. These issues should now be resolved. Please contact our support if you have further problems.
  • 2/11/2013 9:30 PM EST - While rebooting, a large number of requests to the server to retrieve and send email caused a portion of the file system to go into read only. The server has been restarted, however taken offline so that it can fully reboot safely. Once the server is online, SA will reconnect it to our network and customers will be able to log in without the issues experienced before.
  • 2/12/13 01:32 A.M. The Raid has completed and the server has been added back to the network.  We will continue to monitor the server.
  • 2/12/13 06:55 A.M This server continues to function normally.  Email should be delivered as normal.
  • 2/12/13 08:55 A.M This server has gone back offline. We are waiting for more information from our administrators and will update again shortly.
  • 2/12/13 09:36 A.M This server has been restarted and is back online at this time.

Resolution Description

n/a

posted in Outage by IX: Omari J.

98 Comments to "MAIL308 Down for Maintenance- Resolved"

Sun, February 10th, 2013

Umamaheswra Raju says:

bartronicsindia.com mails not working

Link | Posted at 1:53 am EST

Sun, February 10th, 2013

IX: Omari J. says:

Hello Umamaheswra,

Mail on this server will not be working until the server comes back online. Please be patient before sending out any emails. Thank you

Link | Posted at 2:24 am EST

Sun, February 10th, 2013

anonima says:

the emails are still off line!!!!
after at least 6 hours!!!

Link | Posted at 5:41 am EST

Sun, February 10th, 2013

Ahmed Zaki says:

my incoming emails what it send in the time it not working
it will appear in mailbox after it is work or not?

Link | Posted at 7:31 am EST

Sun, February 10th, 2013

IX: Robyn M. says:

Hi Ahmed,

Once the server is back up your incoming mail will start to come in.

Link | Posted at 8:25 am EST

Sun, February 10th, 2013

Carmelo says:

What about incoming emails? Will these be lost? Is there a backup or fall back email server? What about redundancy?

Link | Posted at 8:30 am EST

Sun, February 10th, 2013

IX: Robyn M. says:

Hello Carmelo,

Current emails are being queued and will be delivered once the server is back online.

Link | Posted at 9:11 am EST

Sun, February 10th, 2013

Mike says:

Server down for over 7 hours….

Link | Posted at 10:52 am EST

Sun, February 10th, 2013

Mike says:

When is the server expected to be working again ?

Link | Posted at 11:39 am EST

Sun, February 10th, 2013

IX: Robyn M. says:

Hi Mike,

It should hopefully be in up a few more hours. We will update the blogs in about another hour with more information.

Thanks for your patience!

Link | Posted at 11:59 am EST

Sun, February 10th, 2013

Mike says:

Server down for more than 11 hours now. Hurting biz badly. We will move next week to another web hosting company.

Link | Posted at 3:40 pm EST

Sun, February 10th, 2013

JD says:

Agree with Mike, issues like this should be transparent to the end user, backup/redundant servers etc. I’m sure a lot of people will be searching for a new hosting company over this.

Link | Posted at 4:23 pm EST

Sun, February 10th, 2013

Carmelo says:

Yeah, in today’s world even 2 hours without email service may mean lost business (no matter what day it is). Never expected such downtime without moving on to a backup server.

It’s a shame this had to happen.

Link | Posted at 5:11 pm EST

Sun, February 10th, 2013

Jeff says:

What happened to the noon update. How long does it take to change the settings in a backup server?

Does this mean there is no backup server?!?

Link | Posted at 7:06 pm EST

Sun, February 10th, 2013

jh says:

any updates???????

Link | Posted at 7:07 pm EST

Sun, February 10th, 2013

jr says:

How about an update – it’s been 7 hours since the last update.

Link | Posted at 7:07 pm EST

Sun, February 10th, 2013

erin says:

What IS going on IX??? Have been w/out email now since last night approx 6:30 pm PST. Please give us all some clue as to when we will be back online! Where ARE the BU servers?

Link | Posted at 7:12 pm EST

Sun, February 10th, 2013

IX: Chris B says:

Hello jh,
We are sorry for the delay on this. WE have just updated the status. The sever is currently verifying the rebuild.
Thank You.

Link | Posted at 7:16 pm EST

Sun, February 10th, 2013

IX: Chris B says:

Hello jr,

We are sorry for the delay on this update. We have just updated the status and the server is currently in the verifying stage of the rebuild.

Thank You.

Link | Posted at 7:17 pm EST

Sun, February 10th, 2013

IX: Chris B says:

Hello erin,

We do apologize greatly for this downtime and any/all issues it has/may have caused. We are trying to get this back up as soon as we can. The server is currently verifying the raid rebuild and unfortunately gives no status bar to let us know the exact progress of verifying the information. Once we have a more exact ETA we will make sure to let you know.

Thank You.

Link | Posted at 7:19 pm EST

Sun, February 10th, 2013

Bradshaw says:

Wow!
I appreciate how responsive and nice everyone has been on the phone but this outage is truly affecting all of our credibility. I manage over 1000 domains across the U.S. and I am afraid that this extended outage may become a deal breaker.

Please try to keep us posted on any progress so that we can calm our customer base.

Thank you for all of your help and support!

Link | Posted at 7:23 pm EST

Sun, February 10th, 2013

IX: Chris B says:

Hello Bradshaw,

We definitely understand that every outage is going to cause issues, especially when it comes to email. We are trying to get this backup as soon as we can and soon as we have an update on this we will make sure to let you know.

Thank You.

Link | Posted at 7:32 pm EST

Sun, February 10th, 2013

Bradshaw says:

Thank you Chris!
I truly understand and appreciate your position.

We look forward to your next update and hopefully final resolution very soon.

Link | Posted at 7:38 pm EST

Sun, February 10th, 2013

Srinivas GV says:

Mail server is still not up since the last >20 hrs and I am unable to send out quotes to my customers or business critical email.

This is affecting business very badly. It is ridiculous that you guys don’t have a backup server / cluster to move the affected mailboxes to in case of a down time.

Unscheduled outages of this duration are unacceptable.

Link | Posted at 7:44 pm EST

Sun, February 10th, 2013

IX: Chris B says:

Hello Srinivas GV,

We definitely understand that outages such as this are unacceptable. This was not planned in any way and we are working as fast as we can to get this back up and running as soon as possible. Once we have more information on this we will make sure to let you know.

Thank You.

Link | Posted at 8:58 pm EST

Sun, February 10th, 2013

jh says:

what is happening now? This is getting really worrisome.

Link | Posted at 9:48 pm EST

Sun, February 10th, 2013

Greg Gaunt says:

Good thing this was fixed in the 4 hours I was quoted this morning.

Link | Posted at 9:53 pm EST

Sun, February 10th, 2013

JohnGraz says:

Seriously? This outage is unacceptable and unprofessional. My email is critical to my business and has been down for almost a day now. How can there be no bu servers or some other plan? Without a major account credit we will be looking elsewhere, it is embarrassing and frustrating to have to start calling clients and having email sent to a different working server and address. Makes us look like amateur hour. Too bad. Sorry for the vent.

Link | Posted at 10:20 pm EST

Sun, February 10th, 2013

Frank says:

It has now been over 24 hours. I was always under the impression there was a backup server with the importance of email communication in todays world. I never for a momment thought this could happen. I don’t intend on leaving over it but I may change my mind if it’s not corrected my the time I start my day tomorrow.

Link | Posted at 10:42 pm EST

Sun, February 10th, 2013

jr says:

It’s been 4 hours since the last update

Link | Posted at 11:13 pm EST

Mon, February 11th, 2013

Lach says:

This kind of outage is incredibly damaging to businesses that rely on email. I understand that the issue is being rectified as quick as possible now (and I am grateful), but I’m sure a lot of other customers like myself would like to know how IX are going to prevent this from happening again.

A detailed release explaining what happened, why it happened and what new architecture IX will be putting in place to prevent this from happening again in the future, with deployment timelines is needed here.

If I don’t see a release like this within 48 hours of this problem being rectified, I will be moving my domains and my client’s domains away from IX to a provider I know that does have suitable redundancy solutions in place. Apologies for the outage can only do so much, and does not recuperate lost income.

Link | Posted at 12:04 am EST

Mon, February 11th, 2013

jh says:

update, please!

Link | Posted at 12:24 am EST

Mon, February 11th, 2013

Muath says:

I never hear of a mail server is down for more than 48 hours . even though they are using a Linux Hosting server . a backup solution should always be there for the clients that should not take more than 4 hours . not 48 hours , Let us see whats gona happen and how will IX Webhosting compensate us.

Link | Posted at 12:46 am EST

Mon, February 11th, 2013

Monchai says:

Pls let me know when you fix already

Link | Posted at 12:51 am EST

Mon, February 11th, 2013

Srinivas GV says:

Its already 11AM MONDAY in India, peak time for my business and the mail server is still down.

This is really hurting business. Downtime is now more than 30 hrs. This is getting ridiculous by the hour.

If this continues, you will have to make good my losses.

Link | Posted at 12:57 am EST

Mon, February 11th, 2013

Carmelo says:

This is unbelievable. All this time without a mail server. You should have implemented a backup server at once. Who is taking decisions there? It is in times like these that you show the quality of your service. You should update us more often. At first you started every four hours and now you are not even keeping up to that. We need to know.

Link | Posted at 1:09 am EST

Mon, February 11th, 2013

Carmelo says:

Please note that you guarantee a 99.9% uptime. That’s 8hrs and 45 minutes of downtime in a year.

II. 99.9% Uptime Guarantee

IX Web Hosting guarantees that your website will be up and running at least 99.9% of the time during any 12-month period. This guarantee includes network uptime, server uptime, web server and service uptime. It does not cover any areas where IX Web Hosting has no direct influence, such as backbone provider failures, fibre-optic main line cuts, DNS or Registrar issues with subscriber’s domain name, routing issues between subscriber location and IX Web Hosting data center. The uptime guarantee is also not applicable if the service interruption was caused by external issues such as Acts of God, Wars or any other natural or unnatural events that IX Web Hosting cannot directly influence.

To request a service credit due to an uptime concern, please submit a request for this guarantee via IX Web Hosting’s “Contact Us” form located at this URL:
http://www.ixwebhosting.com/support.

If requesting a “service credit” we ask that subscriber briefly explains the reasons for doing so.

Taken from your Terms of Service on your site. You should have opted for a backup server after the first 4 hours.

Link | Posted at 1:23 am EST

Mon, February 11th, 2013

Alex says:

Please at least update status. 6+ hours since last update.

Link | Posted at 1:32 am EST

Mon, February 11th, 2013

Monchai says:

Now it long time to fix it and make many company to lost important contact .

Link | Posted at 2:09 am EST

Mon, February 11th, 2013

Monchai says:

Pls update status

Link | Posted at 2:10 am EST

Mon, February 11th, 2013

Srinivas GV says:

Hey, What’s happening on reviving the mail server??????

What happened to your 24/7 support??? Have everybody died along with the mailserver ?

Link | Posted at 2:12 am EST

Mon, February 11th, 2013

Roy Tal says:

Do as we are already done :

1. Download FileZilla and download all of your domains to a local computer.

2. Get a better service from a different Hosting company.

3. Upload your website and start using the new and much better service from a different company.

4. Say goodbye to IX

Link | Posted at 2:45 am EST

Mon, February 11th, 2013

Mike says:

Now without e-mail communication for over 24 hours. On the IX main webpage it says “Trusted. Safe. Reliable”. Well, now we find out what is the true meaning of those words in the perception of IX. It also says that the 24/7 helpdesk is “A truly unique support experience”. Really ? No updates here while customers are desperate and live chat Russians don’t know anything when the server will come back on ?

Link | Posted at 2:46 am EST

Mon, February 11th, 2013

jh says:

I am on the phone with support right now – your customer service representative doesn’t even seem to be aware of this massive outage – he is checking my domain and my account to see what is going on. Seriously, please update us on what is going on.

Link | Posted at 2:47 am EST

Mon, February 11th, 2013

Kaveshan says:

Hi there..

You guys have not updated in 2 hours.. Please advise on situation as business is being lost at a very fast pace…

Link | Posted at 3:35 am EST

Mon, February 11th, 2013

Kaveshan says:

Its 11am in South Africa… Peak hour business and no emails… Can someone please advise what is going on!!!

Link | Posted at 3:48 am EST

Mon, February 11th, 2013

Mike says:

It is working again but all e-mails received over the last 22 hours are lost. Thank you very much IX.

Link | Posted at 4:47 am EST

Mon, February 11th, 2013

Srinivas GV says:

Will emails sent to us during the outage get delivered now?

Link | Posted at 4:56 am EST

Mon, February 11th, 2013

Carmelo says:

Please let us know when the email queue is finished. I am still missing test emails I sent myself from a different account.

Link | Posted at 6:19 am EST

Mon, February 11th, 2013

Jh says:

I got few emails several hours ago. Now it says that my password for the account is incorrect. I aven’t change a thing anywhere. So basically, I sm back to having no email service again.

Link | Posted at 11:33 am EST

Mon, February 11th, 2013

Srinivas GV says:

When I try ti send and receive email, I get a pop up which says “Password was rejected”

I have not changed the password. Infact I received a few emails after the server came up and suddenly this has started now….Am back to not being able send or receive emails.

Link | Posted at 11:40 am EST

Mon, February 11th, 2013

Jh says:

I am having the exact same thing happening, Srinivas GV.

Link | Posted at 11:43 am EST

Mon, February 11th, 2013

DR says:

Worked for awhile but email is not working again. Please fix this ASAP. Most of us are trying to run a business.

Link | Posted at 11:44 am EST

Mon, February 11th, 2013

JohnGraz says:

Is it back down again…. I can’t get e-mail. Time to implement a plan B. Sorry IX this is just not acceptable. E-mail is mission critical to my business and now I’d be better off with g-mail or even an AOL address. Who would have thought that could be true. I hope a MASSIVE account credit is coming our way.

Link | Posted at 11:50 am EST

Mon, February 11th, 2013

erin says:

Still not getting email. Rec’d some, but then it went down again. Contacted support and seems there are bugs. When will this be finished. This is the longest server outage I have encountered, yet! Please address this post haste!

Link | Posted at 12:21 pm EST

Mon, February 11th, 2013

Bradshaw says:

Oh Boy! What a way to start a Monday!
Unfortunately our customers e-mail is back down as well!

IX Please provide updates as this is getting harder and harder to explain.

Thank You!

Link | Posted at 12:30 pm EST

Mon, February 11th, 2013

Roy Tal says:

And once again, the mail server is down.
Your service SUCKS!!!

Link | Posted at 12:37 pm EST

Mon, February 11th, 2013

jh says:

when are the emails from the 28 hour window when the server was diwn going to get to us? I assume these are the ones in the queue. I believe I haven’t got any of those yet.

Link | Posted at 3:44 pm EST

Mon, February 11th, 2013

IX: Jared says:

Hello jh,

If the mails were not stuck in the queue for any reason, the sender would receive a “bounce back” to their inbox usually within a few minutes of sending. There is the potential that some messages may have bounced back while the server was unavailable.

If you have further questions or concerns, feel free to comment here or contact our support team.

Link | Posted at 4:34 pm EST

Mon, February 11th, 2013

jh says:

Thank you, Jared.
Well, now AGAIN my password has been rejected.

Link | Posted at 6:08 pm EST

Mon, February 11th, 2013

DR says:

This sucks!

Link | Posted at 6:26 pm EST

Mon, February 11th, 2013

Mahmoud says:

my mailbox is not accepting my password ??????

Link | Posted at 6:34 pm EST

Mon, February 11th, 2013

IX: Chris B says:

Hello jh, DR, and Mahmoud,

We are aware that this is happening and have already notified our administrators to look into this for you.

Thank You.

Link | Posted at 6:39 pm EST

Mon, February 11th, 2013

DR says:

The problem is with mail customers on mail308 right? Can’t you move us all to a new mail###?

Link | Posted at 6:45 pm EST

Mon, February 11th, 2013

jr says:

Incoming and outgoing mail are not going through. We are dead in the water again. This is 2 days now.

Link | Posted at 6:55 pm EST

Mon, February 11th, 2013

Alex says:

Come on guys. We know you are better than this. Please solved or move us to other server. We need that service stable.

Link | Posted at 8:07 pm EST

Mon, February 11th, 2013

IX: Jared says:

@JR: I have tested it, and it was not working. After I contacted our administrators they believe it to be resolved. It should be working now. Please retry from your end – it is working for me.

@DR: It should be working now, please retry. Also, in the future please be aware that moving to another mail server is a lengthy process, not possible while the server is down, and afterwards can bring your mail down for up to 72 hours. Generally, waiting for the server issue to be resolved is much quicker.

To everyone – I apologize for the inconvenience, and I hope you all have a great rest of your day.

Link | Posted at 8:15 pm EST

Mon, February 11th, 2013

jr says:

Sporadically receiving messages; still cannot send.

Link | Posted at 9:15 pm EST

Mon, February 11th, 2013

Peter says:

It is now 9:10 PM EST and it has stopped working again. It was fine most of the afternoon and I received many emails from the weekend. Now I cannot log on to receive mail again.

Link | Posted at 9:15 pm EST

Mon, February 11th, 2013

jr says:

Test message send from another e-mail 10 minutes ago has not been received.

Link | Posted at 9:16 pm EST

Mon, February 11th, 2013

Monchai S. says:

Can’t send and receive again!!!

Link | Posted at 9:19 pm EST

Mon, February 11th, 2013

jr says:

Attempts to send email gets the following error message:

Sending of message failed.
The message could not be sent because the connection to SMTP server mail.sdgenweb.com timed out. Try again or contact your network administrator

Link | Posted at 9:21 pm EST

Mon, February 11th, 2013

Srinivas GV says:

Emails not working again…..whats going on?

Link | Posted at 9:26 pm EST

Mon, February 11th, 2013

Rick says:

Love the canned response “we are working as fast as we can to get this back up and running as soon as possible.” Like anyone can make a file system check or verify run faster. Don’t BS us, we were frustrated enough.

Link | Posted at 9:31 pm EST

Mon, February 11th, 2013

jr says:

Move us to a different server, please!!!

Link | Posted at 9:31 pm EST

Mon, February 11th, 2013

jh says:

it works, then it doesn’t, then it does, then it doesn’t and so on.

Link | Posted at 9:35 pm EST

Mon, February 11th, 2013

IX: Clay says:

Hello All!

I’ve updated the blog, but wanted to take a moment to apologize for the confusion. The server did come back online, however as many of you have experienced, portions of the disk automatically went into a read only state so that the server could keep up with all the incoming requests after the downtime. Because of this, you may have noticed that incoming or outgoing mail may not have worked, or in many cases, you could not log in to your account.

Once the server has had a chance to fully restart without requests, we will bring it into our network for mail access.

To those of you asking if we can move you to a new server, I do understand the want for that, however doing so would force your domain into propagation, and mail would be unavailable for even longer.

Once again, I am very sorry for the downtime, all of our admins are working hard to make sure the server will remain online for all customers before reinstating access to it. Thank you for your patience!

Link | Posted at 9:52 pm EST

Mon, February 11th, 2013

Jeff says:

Three days and counting. There is no way that in this day and age it should take 3 DAYS TO RESOLVE AN EMAIL ISSUE.

Customers are getting VERY upset. If they don’t pay me, I can’t pay you.

About to lose a cloud deployment because of this.

Not happy at all…

Link | Posted at 10:09 pm EST

Mon, February 11th, 2013

JohnGraz says:

Honestly, again no e-mail. This is truly insane. You have crippled my business. Clients think my e-mail is a joke. 2 big accounts are at risk and I can’t communicate with them and their e-mails to me are being bounced back to them for the 3rd time is 2days? They think my business is a joke.

Link | Posted at 11:17 pm EST

Mon, February 11th, 2013

ALex says:

so sad, I thought that ix was a good service, I’m disappointed. what kind of problem requiere more than 3 days to resolve?

Link | Posted at 11:35 pm EST

Tue, February 12th, 2013

anonima says:

EMAIL!!!

Link | Posted at 12:17 am EST

Tue, February 12th, 2013

jh says:

again not working.

Link | Posted at 12:29 am EST

Tue, February 12th, 2013

Muath says:

We hope that it get fixed today and forever , otherwise when the hosting expires I’ll definitely move away from her.

Link | Posted at 12:45 am EST

Tue, February 12th, 2013

Roy Tal says:

Look at their status blog – They have a server (iis557) which is down more than a week!
Can you imagin your business working without any website for more than a week???

Link | Posted at 12:49 am EST

Tue, February 12th, 2013

kamal says:

Again not working, also no update from IX, very frustrating.

Link | Posted at 12:56 am EST

Tue, February 12th, 2013

video magnifiers says:

Sale The Looky 3.5 Inch Color Hand Held Video Magnifier 2 Hrs. of Battery Use! Cheap Price, Best Discount The Looky 3.5 Inch Color Hand Held Video Magnifier 2 Hrs. of Battery Use! Reviews, Buy The Looky 3.5 Inch Color Hand Held Video Magnifier 2 Hrs. of Battery Use! Price

Link | Posted at 1:54 am EST

Tue, February 12th, 2013

Peter says:

Working now. Thanks!

Link | Posted at 1:56 am EST

Tue, February 12th, 2013

jh says:

Not working for me yet!

Link | Posted at 2:03 am EST

Tue, February 12th, 2013

Monchai S. says:

Working thank god!

Link | Posted at 2:32 am EST

Tue, February 12th, 2013

kamal says:

Again causing problem, when it will be working properly and permanently?

Link | Posted at 8:00 am EST

Tue, February 12th, 2013

IX: Robyn M. says:

Hi Kamal,

We are looking into the issue again. I am extremely sorry for all of the issues.

Link | Posted at 8:10 am EST

Tue, February 12th, 2013

anonima says:

email down again!!!
PLEASE!!

Link | Posted at 8:15 am EST

Tue, February 12th, 2013

DR says:

Seriously this is terrible service. I’ll be switching hosts and expect a full-refund on my 2 year service. And change the blog post to NOT resolved!

Link | Posted at 8:25 am EST

Tue, February 12th, 2013

Jeff says:

Well, as long as you’re extremely sorry, that’s ok.

Does anyone on here have a recommendation for another hosting company? Maybe one that can resolve an email issue in 5 days?

Link | Posted at 8:28 am EST

Tue, February 12th, 2013

Srinivas GV says:

Email is down again.

This is unacceptable. Give us a refund and we will move our domains over. You need to get sued over this poor service.

I am unable to send an receive emails and I get the message password rejected and I get a pop up asking me to enter the password.

Link | Posted at 8:32 am EST

Tue, February 12th, 2013

Srinivas GV says:

Jeff, I am looking at hostgator.com.

Link | Posted at 8:34 am EST

Tue, February 12th, 2013

JohnGraz says:

I guess maybe the 4th time will be a charm. Is Mickey Mouse running your IT department….

Link | Posted at 8:36 am EST

Wed, February 13th, 2013

JD says:

How will we be compensated for this extended outage?

Link | Posted at 2:17 am EST

 
© 2011 IX Web Hosting.