Our system administrators have detected an issue with the RAID array on one of the main servers that hosts client email and MySql databases. RAID refers to “redundant array of independent disks”, a technology that allows us to achieve high levels of storage reliability from our server drives. It does this by arranging the devices into an array. Simplified, this means they act like one large hard drive, but if one drive dies, there is enough data stored on the rest to recreate the lost data once the broken hard drive is replaced with a new one.
If a RAID fails, or becomes corrupted, it must be rebuilt. This means the architecture that allows for RAID redundancy must be repaired or completely rebuilt.
The following mail servers will be affected: mail1, mail34, mail44, mail45, mail901, mail902, mail903, mail905, mail906, mail908, mail909, mail910, mail914, mail915, mail916, mail919, mail920, mail1001, mail1002, mail1004, mail1005, mail1006, mail1007, mail1008, mail1009, mail1010, mail1011, mail1012, mail1013, mail1014, mail1015, mail1017, mail1018, mail1101, mail1103, mail1104, mail1105, mail1108, mysql917, mysql1103, mysql1106, mysql1107, mysql1109. There is no downtime expected, but mail services may appear slow which can result in time outs and give the appearance that the service is unavailable. We would like to ensure you that no data loss is expected through this process.
UPDATE: 2016/04/14 010:00 AM EDT
As of now, the RAID rebuild has completed. Unfortunately, we didn’t see the load on the SAN improve over time as we expected it would. As the RAID rebuild finished we expected the SAN to process its workload and over the course of a few hours the load would slowly decrease until we got to a regular operating level. That did not happen and we are currently investigating the cause and solutions. We have also escalated with our hardware provider and are troubleshooting with them.
At this point, our System administration team is pausing email delivery on some mail servers under maintenance. We expect that this will help reduce the load on the SAN. Once the load is stabilized, they will re-enable it again. When delivery is paused, you will not be able to send email until delivery is resumed. Incoming mail will not be delivered to mailboxes right away, but will be pending while delivery is paused.
UPDATE: 2016/04/14 12:15 PM EDT
We are returning mail services to normal on the array. At this time you should be able to login and send email without issues. Please note that incoming email will be delayed for the next several hours while the SAN works through the queue of incoming mail that has built up over the last 24 hours.