Current Time: 15:26:04 EST
 

Mail300 Maintenance August 02 9PM to 11PM

Posted In: Maintenance — Jul 28th, 2015 at 4:46 am EST by IX: Antonio S.
Shared services are affected

Incident Description:

Our System Administrators will be moving the Mail300 server to a new virtual environment.  Affected customers will be informed of the maintenance via a ticket in their account as well as this status blog.  No mail is expected to be lost during maintenance but it will be unavailable.  Mail delivered to the server during maintenance will enter a queue to be delivered once the server is back up.

Which Customers are Impacted?

All customers on 300

How are Customers Impacted?

Mail will be inaccessible.

How often will we be updated?

As required

Time to Resolution (ETA)

7 hours after initiation

Incident Updates

N/A

Resolution Description

N/A

Server Maintenance – Wednesday, July 29, 2015 & Thursday, July 30, 2015

Posted In: upgrades to products — Jul 29th, 2015 at 6:08 pm EST by IX: Greg Cook
Cloud services are affected

Incident Description:

Starting July 29, 2015, IX Web Hosting will be adding new hardware to the existing servers that currently house your data.  These upgrades will provide additional capacity, as well as improved disk performance to our existing cloud infrastructure.
The upgrades to the server pool will take place July 29, 2015 and July 30, 2015 at our Data Center located in Columbus Ohio.

These upgrades will be applied so as not to impact any customer VMs.  This upgrade will be seamless, and the only noticeable change will be an increase in overall disk performance.

 

We do not anticipate any customer to experience downtime during this maintenance.  Additionally, as a precaution, you can also perform a local backup of your content. Refer to our Knowledge Base for assistance or via web chat, or please call us at anytime.

To follow the progress of the update, please watch this status blog for updates.

Thank you,
Your IX Web Hosting Team

Which Customers are Impacted?

All CloudbyIX product customers.  Some customers on IX and CloudByIX brands also may be impacted.

How are Customers Impacted?

No impact to any customers VMs.

How often will we be updated?

As required.

Time to Resolution (ETA)

Maintenance is expected to be complete by August 3rd.

Incident Updates

Additional storage has been added and as data spreads across the new storage, we are seeing even better server performance for all cloud customers.  The system is moving data slowly, as resources allow, so that it never interferes with customer site performance.  This method is safest for customer performance but it is taking longer to migrate this data than anticipated.  Right now, System Admins are expecting this maintenance to be completed by August 3rd.

Resolution Description

N/A

DDoS (Distributed Denial of Service) attack – Resolved

Posted In: Outage — Jul 29th, 2015 at 5:58 pm EST by IX: Greg Cook
Shared services are affected

Incident Description:

We are currently experiencing a DDoS (Distributed Denial of Service) attack against shared IP 98.130.148.2 on web415. In order to stop the attack on it will be blocked on 2 hours.

We will update this status blog post as we get additional information from our systems administrators.

Which Customers are Impacted?

Anyone on shared IP 98.130.148.2 on web415.

How are Customers Impacted?

Domains on shared IP 98.130.148.2  will not be available.

How often will we be updated?

2 hours

Time to Resolution (ETA)

2 hours

Incident Updates

  • 2015/07/29 09:25 AM EDT - Block has been released and IP is online right now.

Resolution Description

The DDoS is resolved.

Mail310 RAID Failure – Resolved

Posted In: Maintenance,Outage — Jul 28th, 2015 at 2:06 pm EST by IX: Alan M.
Shared services are affected

Incident Description:

Mail310 encountered a catastrophic RAID error and had to be recreated. Unfortunately, any mail stored on the server was lost.

Updated description of the issue:

Mail 310 is a mailserver made up of several hard drives working together in a RAID array.  The RAID array spreads data across several drives allowing for faster access speed and helps protect the server from hardware failure.  If a drive in the array fails, the server uses the data spread across the rest of the drives to recreate the data on a spare drive that remains unused in the server until needed.  Yesterday, a drive failed on this mailserver but the RAID array failed to recreate the data on the spare.  That happens sometimes if the spare encounters an error and is usually easily corrected by taking the server down for maintenance to replace the failed or spare drive.

At 11:58AM EDT yesterday, the mail server was taken off-line to replace the failed drive and upon booting the server, the RAID failed to rebuild again.  Next, we replaced the RAID card that controls the array for the server but it still failed to rebuild.

Upon further investigation, we found that the remaining RAID data had become corrupted which was the root cause of the initial array failure.  This meant we had to completely recreate the mail server and all of the data on the failed drives was unrecoverable.

Any mail sent to a domain using this mail server after the mail server was taken offline on July 28th 11:58AM EDT should be delivered to your mailbox, though it may be delayed, as some servers will delay resend attempts up to 16 hours or more.  If you have your mail client set-up to store a copy of your mail to a local computer, then anything received after the last time you connected to your mail and before the server was taken offline is lost.  If you do not have your mail client set-up to store a copy of your mail to a local computer, then any mail stored on the server has been lost.

We keep regular backups of most server types, but are unable to do so on mailservers due to the size of mailservers and the changing nature of the data on them. Shared hosting mail servers are large, serving a great number of customers, and backups of this amount of data, without affecting server performance, requires several hours.  This wouldn’t be a problem, except that the data on mail servers is different from the data on a web or database server because it’s constantly changing.  Over a several hour period, the majority of data on a mail server changes completely as mail arrives, is sent, is deleted, etc, so backup systems are not able to keep up with the constantly changing data.

If email is essential to you, we highly recommend that you set-up your email client to store a local copy of all email received, or investigate a permanent mailserver solution that services just your domains and can be backed up, such as a Cloud hosting server.

We are in the process of moving all of our shared services to a virtualized environment which will provide higher availability and further protection against problems like this in the future.

Which Customers are Impacted?

All Customers on mail310

How are Customers Impacted?

mail is inaccessible and mail stored on the server before recreation is lost

How often will we be updated?

as needed

Time to Resolution (ETA)

24 hours

Incident Updates

n/a

Resolution Description

Maintenance is now complete and services are now returned to normal.

Inbound Phone Support Issues – Resolved

Posted In: Outage — Jul 28th, 2015 at 3:47 am EST by IX: Antonio S.
Phone support is affected

Incident Description:

Currently at this time we are experiencing issues with inbound calls to our support lines. We have contacted our service provider and currently working on resolving this issue.

We apologize for the long wait and will be calling you back once we see your number come in to the queue.

Thank you for your patience.

Which Customers are Impacted?

All customers calling in to support.

How are Customers Impacted?

May not be able to pick up your call.

How often will we be updated?

N/A

Time to Resolution (ETA)

N/A

Incident Updates

  • 2015/07/28 03:46 AM EDT - We are still experiencing issues with our phone services. We have escalated the issue to the development team to further investigate.
  • 2015/07/28 07:15 AM EDT - Resolved

Resolution Description

N/A

Mail504 Maintenance – July 25th 12:00AM to 07:00AM – Resolved

Posted In: Maintenance — Jul 22nd, 2015 at 3:29 pm EST by IX: Greg Cook
Shared services are affected

Incident Description:

Our System Administrators  will be moving the Mail504 server to a new virtual environment.  Affected customers will be informed of the maintenance via a ticket in their account as well as this status blog.  No mail is expected to be lost during maintenance but it will be unavailable.  Mail delivered to the server during maintenance will enter a queue to be delivered once the server is back up.

Which Customers are Impacted?

All customers on Mail504

How are Customers Impacted?

Mail will be inaccessible.

How often will we be updated?

As required

Time to Resolution (ETA)

7 hours after initiation

Incident Updates

n/a

Resolution Description

Mail services have been resumed. No issues were experienced during maintenance. Thank you.

Inbound Shared and Cloud Support Phone Lines Down – Resolved

Posted In: Outage — Jul 18th, 2015 at 5:17 am EST by IX: Kristopher G.
Phone support is affected

Incident Description:

Currently at this time we are experiencing issues with receiving calls to our Shared and Cloud Support phone lines. We apologize for the inconvenience at this time. We have contacted our service provider and will update once we know more information.

Please feel free to contact our Live Chat representatives for further assistance.

Which Customers are Impacted?

All customers contacting support by phone.

How are Customers Impacted?

Inbound support is unavailable at this time.

How often will we be updated?

As soon as further information is received.

Time to Resolution (ETA)

N/A

Incident Updates

N/A

Resolution Description

We are now able to receive inbound phone calls for any support needs. Thank you for your patience.

Limited Number of Virtual Machines Cannot Access Storage – Resolved

Posted In: Outage — Jul 16th, 2015 at 4:06 pm EST by IX: Greg Cook
Cloud services are affected

Incident Description:

Our System Administrators have identified an issue where one of our blades has lost communication with the storage subsystem.  This affects a limited number of cloud customers and we are in the process of investigating the cause so that we can get the affected severs back online as soon as possible.

Which Customers are Impacted?

Very limited number of customers who are being contacted.

How are Customers Impacted?

Cannot access storage

How often will we be updated?

1 hr

Time to Resolution (ETA)

Unknown at this time.

Incident Updates

  • 2015/7/16 04:24 PM EDT  - 80% of the affected customers have been moved and are back up.
  • 2015/7/16 04:35 PM EDT  - All vitrual machines were brought online and our System Administrators still continue to investigate cause of the issue.
  • 2015/7/16 04:35 PM EDT  - All  affected customers are back up now.

Resolution Description

All affected customers are back up now.

Incoming Shared and Cloud Phone Support Down – Resolved

Posted In: Outage — Jul 16th, 2015 at 4:45 am EST by IX: Antonio S.
Phone support is affected

Incident Description:

Currently at this time we are having trouble with our inbound phone lines for Cloud and shared support. We apologize for any delays. Please feel free to contact us by live chat or ticket.

Which Customers are Impacted?

Customers calling the Shared and Cloud Support line.

How are Customers Impacted?

Customers will be unable to call in.

How often will we be updated?

N/A

Time to Resolution (ETA)

N/A

Incident Updates

N/A

Resolution Description

Phone support lines are now available. Thank you for your patience.

Webmail Unavailable – Resolved

Posted In: Maintenance — Jul 16th, 2015 at 2:12 am EST by IX: Kristopher G.
Shared services are affected

Incident Description:

Our administrators are currently investigating issues with accessing mail via the Horde Web Mail portal. At this time customers may not be able to access their mail via Horde.

Which Customers are Impacted?

All customers using web mail to access their mail.

How are Customers Impacted?

All shared customers

How often will we be updated?

1 hour

Time to Resolution (ETA)

1 hour

Incident Updates

N/A

Resolution Description

Web mail is now accessible. Thank you for your patience.

Incoming Cloud Phone Support Down – Resolved

Posted In: Outage — Jul 15th, 2015 at 5:48 am EST by IX: Antonio S.
Phone support is affected

Incident Description:

Currently at this time we are having trouble with our inbound phone lines with our service provider. We apologize for any delays. Please feel free to contact us by live chat or ticket.

Which Customers are Impacted?

Customers calling the Advance Support line.

How are Customers Impacted?

Customers will be unable to call in.

How often will we be updated?

N/A

Time to Resolution (ETA)

N/A

Incident Updates

N/A

Resolution Description

Phone line is back up.

Server Connectivity Issues – Resolved

Posted In: Outage — Jul 10th, 2015 at 9:46 am EST by IX: Andrew Y.
Cloud services are affected

Incident Description:

Our System Administrators have detected an issue with a portion of the backend systems of our Cloud services this morning. As a result, some cloud servers may be unable to write data to their storage volumes.   We are in the process of investigating this issue and we will be getting the affected severs back online as soon as possible.

Which Customers are Impacted?

Customers are unable to write data to their server.

How are Customers Impacted?

As often as possible.

How often will we be updated?

As often as possible.

Time to Resolution (ETA)

We will provide an ETA as soon as possible.

Incident Updates

N/A

Resolution Description

The temporary outage was caused when a core router on our storage network restarted due to a bug in the running version of the software.  Some virtual machines placed themselves in read-only mode in order to protect their data.  We updated the software on the storage router to address the bug and restarted machines that had partitions in a read-only state so that the partitions could be remounted in read-write mode.

Mail 400 – FSCK – Resolved

Posted In: Outage — Jul 09th, 2015 at 4:46 pm EST by IX: Greg Cook
Shared services are affected

Incident Description:

Our administrators have found an issue with the server and will be performing maintenance. Currently the server is under File System Check(FSCK).

Which Customers are Impacted?

All customers on Mail400

How are Customers Impacted?

Mail will be inaccessible.

How often will we be updated?

1.5 hours

Time to Resolution (ETA)

1.5 hours

Incident Updates

  • 2015/7/9 07:24 PM EDT  - Mail400 is back online now.

Resolution Description

Mail400 is back online now.

Webmail Maintenance – Resolved

Posted In: Outage — Jul 08th, 2015 at 11:57 am EST by IX: Angela A.
Shared services are affected

Incident Description:

Our webmail is currently experiencing issues and Horde is not available, at this time. Our system administrators are working on maintenance of one of the webmail nodes, in order to get webmail back up and running.

Which Customers are Impacted?

Customers trying to access Horde, in order to check their mail.

How are Customers Impacted?

Customers are able to log into Horde but are immediately logged back out.

How often will we be updated?

N/A

Time to Resolution (ETA)

N/A

Incident Updates

N/A

Resolution Description

Webmail is back up and running now.

MCP Maintenance – Wednesday, July 8, 2015 5:00 AM EDT – Resolved

Posted In: Maintenance — Jul 07th, 2015 at 5:18 pm EST by IX: Greg Cook
Cloud services are affected
my.ixwebhosting.com affected

Incident Description:

We are upgrading production management servers for CloudbyIX product to increase the servers power.  During the maintenance we will bring down the database servers to increase RAM memory.

We have two(2) management servers that need to be upgraded.  Therefore maintenance and testing will take around 1 hour, scheduled between 5:00 AM and 6:00 AM EDT.

Which Customers are Impacted?

All CloudbyIX product customers.  Some customers on IX and CloudByIX brands also may be impacted.

How are Customers Impacted?

During MCP maintenance  control panel will not be available.

How often will we be updated?

Hourly

Time to Resolution (ETA)

One hour

Incident Updates

n/a

Resolution Description

MCP Maintenance is complete.

 
© 2011 IX Web Hosting.