Our system administrators have detected an issue with the RAID array on one of the main servers that hosts client email and MySQL databases. RAID refers to “redundant array of independent disks”, a technology that allows us to achieve high levels of storage reliability from our server drives. It does this by arranging the devices into an array. Simplified, this means they act like one large hard drive, but if one drive dies, there is enough data stored on the rest to recreate the lost data once the broken hard drive is replaced with a new one.
If a RAID fails, or becomes corrupted, it must be rebuilt. This means the architecture that allows for RAID redundancy must be repaired or completely rebuilt.
There is no downtime expected, but mail services may appear slow which can result in time outs and give the appearance that the service is unavailable. We would like to ensure you that no data loss is expected through this process.
UPDATE: Please see most recent updates below. RAID rebuild has been stopped and manual evacuations are in progress.
Update 2016/08/22 1:30 PM EDT:
The following mail servers are affected: mail1201, mail1202, mail1203, mail1204, mail1205, mail1206, mail1207, mail1208, mail1209, mail1210, mail1211, mail1212, mail1213, mail1214, mail1215, mail1216, mail1217, mail1218, mail1301, mail1302, mail1303, mail1304, mail1305, mail1306, mail1307, mail1308, mail1309, mail1310, mail1311, mail1312, mail1313, mail1314, mail1315, mail1316, mail1317, mail1318, mail1319, mail1320, mail1401, mail1402, mail1403, mail1404, mail1405, mail1406, mail1407, mail1408, mail1409, mail1410, mail1411, mail1412, mail1413, mail1414, mail1415, mail1416, mail1417, mail1418, mail1419, mail1420, mail1421, mail1422, mail1423, mail1424, mail1425
The following mysql servers are affected: mail310, mysql1201, mysql1202, mysql1203, mysql1204, mysql1205, mysql1206, mysql1207, mysql1208, mysql1209, mysql1210, mysql1211, mysql1212, mysql1213, mysql1214, mysql1215, mysql1216, mysql1217, mysql1218, mysql1219, mysql1220, mysql1301, mysql1302, mysql1303, mysql1304, mysql1305, mysql1306, mysql1307, mysql1308, mysql1309, mysql1310, mysql1401, mysql1402, mysql1403, mysql1404, mysql1405, mysql1406, mysql1407, mysql1408, mysql1409, mysql1410, mysql1411, mysql1412, mysql1413, mysql1414, mysql1415, mysql1416, mysql1417, mysql1418, mysql1419, mysql1420, mysql1421, mysql1422, mysql1423, mysql1424, mysql1425, mysql1426, mysql1427, mysql1428, mysql1429, mysql1430
The following pgsql servers are affected: pgsql1201, pgsql1202, pgsql1301, pgsql1302, pgsql1401