Current Time: 15:26:04 EST
 

CP 9 – 14 System Maintenance – Resolved

Posted In: Maintenance — Sep 30th, 2014 at 3:12 am EST by IX: Antonio S.

Incident Description:

Due to the possibility of degraded communication between virtual machines and the storage network, our system administrators will be performing urgent system maintenance within the virtual environments that CP9 -CP14 servers reside on. Administrators are performing this maintenance with the utmost care to prevent downtime to any customer facing servers but some servers may require a reboot and file system check during this process.

We will be providing updates on a server by server basis as they go under maintenance and as maintenance is completed.

Please feel free to contact support via live chat or phone should there be any concerns or questions regarding the maintenance.

List of servers to be worked on by our administrators will be as follows:

WEB1001 – Maintenance Completed
MYSQL1009 – Maintenance Completed
WEB1007 – Maintenance Completed
MAIL1011 – Maintenance Completed
MYSQL1003 – Maintenance Completed
MYSQL1012 – Maintenance Completed
WEB1021 – Maintenance Completed

WEB1004 – Maintenance Completed
MAIL1013 – Maintenance Completed
MAIL1002 – Maintenance Completed
MAIL1009 – Maintenance Completed
WEB1011 – Maintenance Completed
MAIL1015 – Maintenance Completed
MAIL1001 – Maintenance Completed
MAIL1018 – Maintenance Completed

WEB1409 – Maintenance Completed
MYSQL1406 – Maintenance Completed
MYSQL1407 – Maintenance Completed
MYSQL1421 – Maintenance Completed
MAIL1405 – Maintenance Completed
WEB1420 – Maintenance Completed
MYSQL1409 – Maintenance Completed
WEB1416 – Maintenance Completed
MAIL1423 – Maintenance Completed
WEB1422 – Maintenance Completed
MYSQL1416 – Maintenance Completed

WEB1029 – Maintenance Completed
WEB1014 – Maintenance Completed
MYSQL1010 – Maintenance Completed
WEB1018 – Maintenance Completed
MAIL1012 – Maintenance Completed
MAIL1014 – Maintenance Completed
PGSQL1002 – Maintenance Completed

MYSQL1011 – Maintenance Completed
WEB1028 – Maintenance Completed
MAIL1016 – Maintenance Completed
MAIL1017 – Maintenance Completed
MYSQL1014 – Maintenance Completed
WEB1002 – Maintenance Completed
WEB1003 – Maintenance Completed

MAIL1313 – Maintenance Completed
MYSQL1310 – Maintenance Completed
WEB1309 – Maintenance Completed
MYSQL1306 – Maintenance Completed
MAIL1311 – Maintenance Completed
WEB1317 – Maintenance Completed
DB2.WEBMAIL – Maintenance Completed
WEB1308 – Maintenance Completed
MAIL1314 – Maintenance Completed
MAIL1305 – Maintenance Completed
WEB1326 – Maintenance Completed
MAIL1422 – Maintenance Completed
MAIL1417 – Maintenance Completed
MYSQL1428 – Maintenance Completed
MAIL1411 – Maintenance Completed
MAIL1408 – Maintenance Completed
WEB1432 – Maintenance Completed
MAIL1419 – Maintenance Completed
MAIL1424 – Maintenance Completed
WEB1415 – Maintenance Completed
WEB1402 – Maintenance Completed
MYSQL1413 – Maintenance Completed
MYSQL1424 – Maintenance Completed
MAIL1412 – Maintenance Completed
MAIL1403 – Maintenance Completed
WEB1401 – Maintenance Completed
MAIL1404 – Maintenance Completed

Which Customers are Impacted?

All customers on CP9 - CP14.

How are Customers Impacted?

There may be some slowness while the server is under maintenance.  A few servers may also require reboot during this maintenance.  If maintenance requires a server reboot, we expect a file system check to run and services will be unavailable until this completes.

How often will we be updated?

Servers will be added as they come under maintenance.

Time to Resolution (ETA)

N/A

Incident Updates

  • 2014/09/30 03:08:14 AM EST - These VMs were moved without downtime MAIL1313, MYSQL1310, WEB1309 ,MYSQL1306, MAIL1311 ,WEB1317WEBMAIL ,WEB1308 ,MAIL1314, MAIL1305, WEB1326,MAIL1422, MAIL1417, MYSQL1428 ,MAIL1411, MAIL1408, WEB1432,MAIL1419, MAIL1424 WEB1415,WEB1402, MYSQL1413, MYSQL1424,MAIL1412, MAIL1403, WEB1401,MAIL1404
  • 2014/09/30 03:08:14 AM EST - The next  set for maintenance is as follows: WEB1409, MYSQL1406, MYSQL1407, MYSQL1421, MAIL1405 WEB1420, MYSQL1409, WEB1416, MAIL1423,WEB1422, MYSQL1416
  • 2014/09/30 03:37 AM EST - These VMs were moved without downtime: MYSQL1011, WEB1028, MAIL1016, MAIL1017 MYSQL1014, WEB1002,WEB100
  • 2014/09/30 03:37 AM EST - The next  set for maintenance is as follows: WEB1029, WEB1014, MYSQL1010WEB1018, MAIL1012, MAIL1014,PGSQL100
  • 2014/09/30 04:05 AM EST - The  VMs  were moved with no downtime.
  • 2014/09/30 04:05 AM EST - The next  set for maintenance is as follows: WEB1004, MAIL1013, MAIL1002, MAIL1009,WEB1011, MAIL1015, MAIL1001,MAIL1018
  • 2014/09/30 05:05 AM EST - Previous VMs were successfully migrated. Blade was rebooted.
  • 2014/09/30 05:05 AM EST -  The next  set for maintenance is as follows: WEB1001, MYSQL1009, WEB1007, MAIL1011 MYSQL1003 MYSQL1012 WEB1021

Resolution Description

VMs were moved without downtime. Maintenance has been finished.

Web520 – Down – Resolved

Posted In: Outage — Sep 29th, 2014 at 5:59 pm EST by Alan Marion

Incident Description:

Our administrators have identified errors with Web520 and is causing the server to be unreachable. We are currently investigating the cause and will update once further information is obtained.

Which Customers are Impacted?

All customers on Web520

How are Customers Impacted?

All services on Web520 are unreachable

How often will we be updated?

N/A

Time to Resolution (ETA)

N/A

Incident Updates

N/A

Resolution Description

Web 520 is up and running now.

CP 9 – 14 System Maintenance – Resolved

Posted In: Maintenance — Sep 29th, 2014 at 2:08 am EST by Kevin Mb.

Incident Description:

Our system administrators will be performing system maintenance for servers within CP9 – CP14. Administrators are performing this maintenance with the utmost care to prevent downtime to any customer, but some servers may require a reboot and file system check during this process.

We will be providing updates on a server by server basis as they go under maintenance and as maintenance is completed. Below is a list of the servers that will be receiving this maintenance:

WEB929 - Maintenance finished
WEB916 - Maintenance finished
WEB904 - Maintenance finished
WEB918 - Maintenance finished
mail-13 - Maintenance finished
MAIL914 - Maintenance finished
WEB933 - Maintenance finished
WEB1013 - Maintenance finished
MYSQL1006 - Maintenance finished
WEB1009 - Maintenance finished
MAIL1004 - Maintenance finished
MYSQL1008 - Maintenance finished
WEB1010 - Maintenance finished
WEB1025 - Maintenance finished
mail-22 - Maintenance finished
MYSQL1206 - Maintenance finished
MYSQL1218 - Maintenance finished
WEB410 - Maintenance finished
WEB1201 - Maintenance finished
WEB1228 - Maintenance finished
MAIL1208 - Maintenance finished
MYSQL1209 - Maintenance finished
MAIL1212 - Maintenance finished
WEB1213 - Maintenance finished
WEB1232 - Maintenance finished
MYSQL1215 - Maintenance finished
MAIL1215 - Maintenance finished
WEB1223 - Maintenance finished
WEB305 - Maintenance finished
MYSQL1302 - Maintenance finished
WEB1312 - Maintenance finished
WEB1301 - Maintenance finished
WEB1333 - Maintenance finished
WEB1330 - Maintenance finished
MAIL1315 - Maintenance finished
WEB1318 - Maintenance finished
WEB1310 - Maintenance finished
MAIL1319 - Maintenance finished
MAIL1420 - Maintenance finished
WEB1439 - Maintenance finished
WEB1413 - Maintenance finished
WEB1403 - Maintenance finished
WEB1426 - Maintenance finished
MAIL1402 - Maintenance finished
MYSQL1414 - Maintenance finished
WEB1406 - Maintenance finished
WEB1429 - Maintenance finished
WEB1433 - Maintenance finished
MYSQL1430 - Maintenance finished
MAIL1409 - Maintenance finished
WEB927 - Maintenance finished
mail-44 - Maintenance finished
MAIL912 - Maintenance finished
WEB908 - Maintenance finished
MAIL911 - Maintenance finished
WEB939 - Maintenance finished
WEB943 - Maintenance finished
MYSQL907 - Maintenance finished
MAIL908 - Maintenance finished
MAIL916 - Maintenance finished
WEB946 - Maintenance finished
WEB1024 - Maintenance finished
MYSQL1015 - Maintenance finished
MAIL1008 - Maintenance finished
MYSQL1017 - Maintenance finished
WEB1006 - Maintenance finished
MAIL1005 - Maintenance finished
MYSQL1016 - Maintenance finished
WEB1026 - Maintenance finished
MAIL1010 - Maintenance finished
MYSQL1005 - Maintenance finished
WEB1012  - Maintenance finished
MYSQL913 - Maintenance finished
WEB937 - Maintenance finished
mail-23  - Maintenance finished
WEB1030 - Maintenance finished
MAIL1006 - Maintenance finished
MYSQL1013  - Maintenance finished
WEB1019  - Maintenance finished
WEB1027 - Maintenance finished
MYSQL1001  - Maintenance finished
WEB1208  - Maintenance finished
WEB1212  - Maintenance finished
MYSQL1208 - Maintenance finished
MYSQL1205  - Maintenance finished
WEB1214  - Maintenance finished
WEB1229 - Maintenance finished
WEB1215  - Maintenance finished
MAIL1216  - Maintenance finished
MYSQL1207 - Maintenance finished
WEB12125  - Maintenance finished
MYSQL1201  - Maintenance finished
MAIL51 - Maintenance finished
WEB1202  - Maintenance finished
WEB1313  - Maintenance finished
WEB1305 - Maintenance finished
MAIL1304 - Maintenance finished
MAIL1318  - Maintenance finished
WEB1323 - Maintenance finished
WEB1336  - Maintenance finished
WEB1315  - Maintenance finished
MYSQL1303 - Maintenance finished
WEB1306  - Maintenance finished
WEB1304  - Maintenance finished
MAIL1421  - Maintenance finished
WEB1412  - Maintenance finished
MYSQL1403 - Maintenance finished
WEB1427  - Maintenance finished
MAIL1401  - Maintenance finished
WEB1421 - Maintenance finished
WEB1437  - Maintenance finished
WEB1425  - Maintenance finished
MYSQL1420 - Maintenance finished
MAIL1425  - Maintenance finished
MAIL1414  - Maintenance finished
MYSQL901 - Maintenance finished
WEB925  - Maintenance finished
MAIL917 - Maintenance finished
WEB913  - Maintenance finished
MYSQL918  - Maintenance finished
MAIL906 - Maintenance finished
MYSQL909  - Maintenance finished
MYSQL906  - Maintenance finished
WEB931 - Maintenance finished
WEB935  - Maintenance finished
MYSQL1004  - Maintenance finished
MAIL1003  - Maintenance finished
WEB1017 - Maintenance finished
WEB1016  - Maintenance finished
MAIL1007  - Maintenance finished
PGSQL1001 - Maintenance finished
WEB1015  - Maintenance finished
WEB1005  - Maintenance finished
WEB1238  - Maintenance finished
WEB1230  - Maintenance finished
WEB1206 - Maintenance finished
mail-42  - Maintenance finished
WEB1210  - Maintenance finished
WEB1221 - Maintenance finished
PGSQK1201  - Maintenance finished
WEB1219  - Maintenance finished
WEB1234 - Maintenance finished
MYSQL1202  - Maintenance finished
WEB1233  - Maintenance finished
mail-52 - Maintenance finished
MYSQL1203 - Maintenance finished
MAIL1302  - Maintenance finished
WEB1324  - Maintenance finished
MYSQL1308  - Maintenance finished
MAIL1309  - Maintenance finished
MAIL1320  - Maintenance finished
WEB1322 - Maintenance finished
MYSQL1304  - Maintenance finished
MYSQL1307  - Maintenance finished
WEB1334 - Maintenance finished
MAIL1307  - Maintenance finished
MYSQL1301  - Maintenance finished
MYSQL1404  - Maintenance finished
MYSQL1423  - Maintenance finished
MYSQL1415  - Maintenance finished
MYSQL1412  - Maintenance finished
WEB1436  - Maintenance finished
WEB1411 - Maintenance finished
MAIL1407  - Maintenance finished
WEB1423  - Maintenance finished
WEB1408 - Maintenance finished
WEB1430  - Maintenance finished
MAIL904 - Maintenance finished
WEB917  - Maintenance finished
WEB948 - Maintenance finished
WEB903  - Maintenance finished
WEB938  - Maintenance finished
WEB911 - Maintenance finished
MAIL903  - Maintenance finished
PGSQL902 - Maintenance finished
WEB945  - Maintenance finished
WEB941  - Maintenance finished
MAIL1217  - Maintenance finished
WEB1236  - Maintenance finished
MYSQL1210  - Maintenance finished
WEB1226  - Maintenance finished
WEB312  - Maintenance finished
WEB1204 - Maintenance finished
WEB1220  - Maintenance finished
MAIL37  - Maintenance finished
mail-11 -  Maintenance finished
MAIL1206  -  Maintenance finished
WEB1237  -  Maintenance finished
WEB1222 - Maintenance finished
mail-21  - Maintenance finished
MAIL1203  - Maintenance finished
MYSQL1419  - Maintenance finished
MYSQL1402  - Maintenance finished
MYSQL1410  -  Maintenance finished
WEB1414  - Maintenance finished
MAIL1418  - Maintenance finished
WEB1435 - Maintenance finished
WEB1410  - Maintenance finished
MYSQL1427  - Maintenance finished
WEB1405 - Maintenance finished
MAIL1406  - Maintenance finished
MYSQL1408  - Maintenance finished
MAIL1416 - Maintenance finished
MYSQL1426  - Maintenance finished
WEB1434 - Maintenance finished
PGSQL901 - Maintenance finished
MAIL910 - Maintenance finished
WEB924 - Maintenance finished
MYSQL910 - Maintenance finished
WEB923 - Maintenance finished
WEB928 - Maintenance finished
WEB932 - Maintenance finished
MYSQL915 - Maintenance finished
MAIL913 - Maintenance finished
WEB914 - Maintenance finished
WEB942 - Maintenance finished
MYSQL920 - Maintenance finished
WEB922 - Maintenance finished
WEB905 - Maintenance finished
WEB934 - Maintenance finished
mail-12 - Maintenance finished
MYSQL1212 - Maintenance finished
WEB1224 - Maintenance finished
MAIL1209 - Maintenance finished
WEB1231 - Maintenance finished
mail-31 - Maintenance finished
WEB1211 - Maintenance finished
WEB1235 - Maintenance finished
MYSQL1219 - Maintenance finished
WEB1205 - Maintenance finished
MYSQL1211 - Maintenance finished
MAIL1207 - Maintenance finished
MYSQL1204 - Maintenance finished
MAIL1201 - Maintenance finished
MYSQL1216 - Maintenance finished
WEB1424 - Maintenance finished
MYSQL1411 - Maintenance finished
WEB1431 - Maintenance finished
WEB1404 - Maintenance finished
WEB1428 - Maintenance finished
MYSQL1405 - Maintenance finished
MAIL1410 - Maintenance finished
MYSQL1401 - Maintenance finished
WEB1419 - Maintenance finished
MYSQL1422 - Maintenance finished
PGSQL1401 - Maintenance finished
MYSQL1429 - Maintenance finished

Which Customers are Impacted?

All customers on CP9 - CP14.

How are Customers Impacted?

There may be some slowness while the server is under maintenance.  A few servers may also require reboot during this maintenance.  If maintenance requires a server reboot, we expect a file system check to run and services will be unavailable until this completes.

How often will we be updated?

Servers will be added as they come under maintenance.

Time to Resolution (ETA)

N/A

Incident Updates

  • 2014/09/29 02:45:14 AM EST - These VMs were moved without downtime: WEB929, WEB916, WEB904, WEB918, mail-13, MAIL914, WEB933, WEB1013, MYSQL1006, WEB1009, MAIL1004, MYSQL1008, WEB1010, & WEB1025.
  • 2014/09/29  03:15:03 AM EST - Maintenance is complete on following servers: mail-22, MYSQL1206, MYSQL1218, WEB410, WEB1201, WEB1228, MAIL1208, MYSQL1209, MAIL1212, WEB1213, WEB1232, MYSQL1215, MAIL1215, WEB1223, WEB305, MYSQL1302, WEB1312, WEB1301, WEB1333, WEB1330 , MAIL1315, WEB1318, WEB1310, MAIL1319, MAIL1420, WEB1439, WEB1413, WEB1403, WEB1426, MAIL1402, MYSQL1414, WEB1406, WEB1429, WEB1433, MYSQL1430, and MAIL1409. No down time experienced.
  • 2014/09/29 4:00AM EST - Maintenance is complete on the following servers: WEB927, mail-44, MAIL912, WEB908, MAIL911, WEB939, WEB943, MYSQL907, MAIL908, MAIL916, WEB946, WEB1024, MYSQL1015, MAIL1008, MYSQL1017, WEB1006, MAIL1005, MYSQL1016, WEB1026, MAIL1010, MYSQL1005, and WEB1012. No downtime.
  • 2014/09/29  04:43:26 AM EST - These VMs were moved without downtime: WEB1030, MAIL1006, MYSQL1013, WEB1019, WEB1027, MYSQL1001, WEB1208, WEB1212, MYSQL1208, MYSQL1205, WEB1214, WEB1229, WEB1215, MAIL1216, MYSQL1207, WEB12125, MYSQL1201, MAIL51, WEB1202, MYSQL913, WEB937, mail-23, WEB1313, WEB1305, MAIL1304, MAIL1318, WEB1323, WEB1336, WEB1315, MYSQL1303, WEB1306, WEB1304, MAIL1421, WEB1412, MYSQL1403, WEB1427, MAIL1401, WEB1421, WEB1437, WEB1425, MYSQL1420, MAIL1425, and MAIL1414.
  • 2014/09/29   07:06:18 AM EST - The following list of VMs have been moved without downtime: MYSQL901, WEB925, MAIL917, WEB913, MYSQL918, MAIL906, MYSQL909, MYSQL906, WEB931, WEB935, MYSQL1004, MAIL1003, WEB1017, WEB1016, MAIL1007, PGSQL1001, WEB1015, WEB1005, WEB1238, WEB1230, WEB1206, mail-42, WEB1210, WEB1221, PGSQL1201, WEB1219, WEB1234, MYSQL1202, WEB1233, mail-52, MYSQL1203, MAIL1302, WEB1324, MYSQL1308, MAIL1309, MAIL1320, WEB1322, MYSQL1304, MYSQL1307, WEB1334, MAIL1307, MYSQL1301, MYSQL1404, MYSQL1423, MYSQL1415, MYSQL1412, WEB1436, WEB1411, MAIL1407, WEB1423, WEB1408, and WEB1430.
  • 2014/09/29   07:20:19 AM EST - Maintenance for the following VMs has completed: MAIL1217, WEB1236, MYSQL1210, WEB1226, WEB312, WEB1204, WEB1220, MAIL37, mail-11, MAIL1206, WEB1237, WEB1222, mail-21, MAIL1203, MYSQL1419, MYSQL1402, MYSQL1410, WEB1414, MAIL1418, WEB1435, WEB1410, MYSQL1427, WEB1405, MAIL1406, MYSQL1408, MAIL1416, MYSQL1426, and WEB1434.
  • 2014/09/29   09:42 AM EST - Maintenance on all servers has finished

Resolution Description

Maintenance has been completed successfully.

DNS Name Servers Under DDoS Attack – Resolved

Posted In: Outage — Sep 27th, 2014 at 1:55 am EST by IX: Kevin N.

Incident Description:

Our Name Servers are currently experiencing a large Denial of Service attack. This is causing DNS queries to become slow or fail for some domains on our DNS servers. Our system administration team is working to identify the source and block this attack. We are working to resolve this issue as quickly as possible.

Which Customers are Impacted?

Customers using any of our web, email or database services.

How are Customers Impacted?

Services may be slow or down.

How often will we be updated?

As information becomes available.

Time to Resolution (ETA)

N/A

Incident Updates

  • 2014/09/27 02:33 AM EDT: Our administrators have mitigated a portion of the attack and accessibility of impacted services is continuing to improve.  Some users might still experience intermittent service unavailability until this issue is completely resolved.
  • 2014/09/27 10:59 AM EDT: Our administrators have narrowed the impact of this attack to a small subset of customers using the n23.ixwebhositng.com and ns24.ixwebhosting.com nameservers.
  • 2014/09/27 1:23 PM EDT: ns23 and ns24.ixwebhosting.com are still being attacked by this massive DDoS which continues to cause intermittent connection issues on some cp12 customer sites.

Resolution Description

Our administrators have verified that this DDoS attack has been completely resolved. All services should be back to normal.Thank you for your patience and cooperation during this time.

web255 – FSCK – Resolved

Posted In: Maintenance,Outage — Sep 26th, 2014 at 10:06 pm EST by IX: Antonio S.

Incident Description:

web255 is currently unresponsive, and our system admins are currently investigating.  We will update with more info as soon as it becomes available.

Which Customers are Impacted?

All customers on web255

How are Customers Impacted?

All websites will be unreachable during the downtime.

How often will we be updated?

N/A

Time to Resolution (ETA)

N/A

Incident Updates

9/26/2014 10:00PM EST - The server had to be restarted, and is currently undergoing a mandatory file system check (FSCK).

Resolution Description

FSCK finished. Server is online.

CP 9 – 14 Urgent System Maintenance – Resolved

Posted In: Maintenance — Sep 25th, 2014 at 10:31 pm EST by IX: Antonio S.

Incident Description:

Due to the possibility of degraded communication between virtual machines and the storage network, our system administrators will be performing urgent system maintenance within the virtual environments that CP9 -CP14 servers reside on. Administrators are performing this maintenance with the utmost care to prevent downtime to any customer facing servers but some servers may require a reboot and file system check during this process.

We will be providing updates on a server by server basis as they go under maintenance and as maintenance is completed.

Please feel free to contact support via live chat or phone should there be any concerns or questions regarding the maintenance.

 

List of servers to be worked on by our administrators will be as follows:

WEB1125 - Maintenance Completed
MYSQL1113 - Maintenance Completed
MAIL1114 - Maintenance Completed
WEB1124 - Maintenance Completed
MYSQL1117 - Maintenance Completed
MAIL1122 - Maintenance Completed
WEB1115 - Maintenance Completed
MAIL1123 - Maintenance Completed
WEB1122 - Maintenance Completed
MYSQL1105 - Maintenance Completed
WEB1104 - Maintenance Completed
CP11 - Maintenance Completed

WEB930 – Maintenance Completed
MYSQL911 – Maintenance Completed
MAIL902 – Maintenance Completed
MYSQL908 – Maintenance Completed
WEB901 – Maintenance Completed
WEB919 – Maintenance Completed
WEB907 – Maintenance Completed
MYSQL905 – Maintenance Completed
WEB947 – Maintenance Completed
MAIL901 – Maintenance Completed
WEB936 – Maintenance Completed
WEB909 – Maintenance Completed
WEB912 – Maintenance Completed
MYSQL912 – Maintenance Completed
MYSQL1116 – Maintenance Completed
MYSQL1127 – Maintenance Completed
WEB1101 – Maintenance Completed
WEB1135 – Maintenance Completed
MAIL1118 – Maintenance Completed
WEB1110 – Maintenance Completed
WEB1138 – Maintenance Completed
WEB1139 – Maintenance Completed
MAIL1108 – Maintenance Completed
MYSQL1126 – Maintenance Completed
MAIL1110 – Maintenance Completed
WEB1140 – Maintenance Completed
MYSQL1112 – Maintenance Completed
WEB1109 – Maintenance Completed
WEB1208 – Maintenance Completed
WEB1212 – Maintenance Completed
MAIL51 – Maintenance Completed
MYSQL1205 – Maintenance Completed
WEB1214 – Maintenance Completed
MYSQL1208 – Maintenance Completed
WEB1215 – Maintenance Completed
MAIL1216 – Maintenance Completed
WEB1229 – Maintenance Completed
WEB1225 – Maintenance Completed
MYSQL1201 – Maintenance Completed
MYSQL1207 – Maintenance Completed
WEB1202 – Maintenance Completed

MAIL907 – Maintenance Completed
WEB921 – Maintenance Completed
MAIL918 – Maintenance Completed
WEB906 – Maintenance Completed
WEB944 – Maintenance Completed
WEB920 – Maintenance Completed
MAIL909 – Maintenance Completed
MYSQL902 – Maintenance Completed
WEB1227 – Maintenance Completed
PGSQL1202 – Maintenance Completed
MAIL1211 – Maintenance Completed
WEB1216 – Maintenance Completed
mail-32 – Maintenance Completed
MAIL1218 – Maintenance Completed
WEB1207 – Maintenance Completed
MYSQL1214 – Maintenance Completed
MAIL1202 – Maintenance Completed
MYSQL1220 – Maintenance Completed
WEB1218 – Maintenance Completed
WEB1217 – Maintenance Completed
mail-41 – Maintenance Completed
MAIL1213 – Maintenance Completed
MAIL1205 – Maintenance Completed
WEBMAIL – Maintenance Completed
YSQL1305 – Maintenance Completed
WEB1320 – Maintenance Completed
WEB1314 – Maintenance Completed
PGSQL1302 – Maintenance Completed
WEB1307 – Maintenance Completed
WEB1325 – Maintenance Completed
PGSQL1301 – Maintenance Completed
MAIL1317 – Maintenance Completed
MAIL1310 – Maintenance Completed
WEB1440 – Maintenance Completed
MAIL1415 – Maintenance Completed
MAIL1413 – Maintenance Completed
MYSQL1417 – Maintenance Completed
MYSQL1418 – Maintenance Completed
WEB1417 – Maintenance Completed
MYSQL1425 – Maintenance Completed
WEB1407 – Maintenance Completed
WEB1418 – Maintenance Completed
WEB1438 – Maintenance Completed
MAIL1404- Maintenance Completed

WEB926- Maintenance Completed
WEB910- Maintenance Completed
WEB940- Maintenance Completed
MYSQL914- Maintenance Completed
WEB902- Maintenance Completed
WEB915- Maintenance Completed
MAIL1101- Maintenance Completed
WEB1129- Maintenance Completed
MAIL1115- Maintenance Completed
MAIL1109- Maintenance Completed
WEB1136- Maintenance Completed
WEB1113- Maintenance Completed
WEB1102- Maintenance Completed
WEB1116- Maintenance Completed
WEB1130- Maintenance Completed
WEB1105- Maintenance Completed
WEB1108- Maintenance Completed
WEB1227- Maintenance Completed
PGSQL1202- Maintenance Completed
MAIL1211- Maintenance Completed
WEB1216- Maintenance Completed
mail32- Maintenance Completed
MAIL1218- Maintenance Completed
WEB1207- Maintenance Completed
MYSQL1214- Maintenance Completed
MAIL1202- Maintenance Completed
MYSQL1220- Maintenance Completed
WEB1218- Maintenance Completed
WEB1217- Maintenance Completed
mail41- Maintenance Completed
MAIL1213- Maintenance Completed
MAIL1205- Maintenance Completed
WEB1337- Maintenance Completed
WEB1311- Maintenance Completed
WEB1329- Maintenance Completed
MYSQL1309- Maintenance Completed
WEB1303- Maintenance Completed
WEB1302- Maintenance Completed
MAIL1303- Maintenance Completed
MAIL1316- Maintenance Completed
WEB1327- Maintenance Completed
WEB1332- Maintenance Completed
MAIL1417- Maintenance Completed
MAIL1420- Maintenance Completed
WEB1439- Maintenance Completed
WEB1413- Maintenance Completed
WEB1403- Maintenance Completed
WEB1426- Maintenance Completed
MAIL1402- Maintenance Completed
MYSQL1414- Maintenance Completed
WEB1406- Maintenance Completed
WEB1429- Maintenance Completed
WEB1433- Maintenance Completed
MYSQL1430- Maintenance Completed
MAIL1409- Maintenance Completed

Which Customers are Impacted?

All customers on CP9 - CP14

How are Customers Impacted?

There may be some slowness while the server is under maintenance.  A few servers may also require reboot during this maintenance.  If maintenance requires a server reboot, we expect a file system check to run and services will be unavailable until this completes.

How often will we be updated?

Servers will be added as they come under maintenance.

Time to Resolution (ETA)

N/A

Incident Updates

  • 2014/09/26 03:27 AM EST - Maintenance was started and following VMs are being moved -  WEB1125,  MYSQL1113, MAIL1114, WEB1124,  CP11,  MYSQL1117, MAIL1122,  WEB1115,  MAIL1123, WEB1122,  MYSQL1105,  WEB1104.
  • 2014/09/26 04:14 AM EST  - Second set for maintenance is as follows: WEB930, MYSQL911, MAIL902, MYSQL908,  WEB901, WEB919, WEB907,  MYSQL905,  WEB947, MAIL901,  WEB936,  WEB909,WEB912, MYSQL912, MYSQL1116, MYSQL1127, WEB1101, WEB1135,  MAIL1118,  WEB1110,WEB1138, WEB1139,  MAIL1108,, MYSQL1126,  MAIL1110, WEB1140, MYSQL1112 WEB1109 , WEB1208,  WEB1212,  MAIL51, MYSQL1205, WEB1214 , MYSQL1208,WEB1215, MAIL1216, WEB1229, WEB1225, MYSQL1201, MYSQL1207,WEB1202, webxxx2_server.
  • 2014/09/26 05:40 AM EST -  Second set of servers have been moved and VM is booting up.
  • 2014/09/26 05:52 AM EST - Third  set for maintenance is as follows: MAIL907, WEB921, MAIL918, WEB906, WEB944, WEB920 ,MAIL909,  MYSQL902 ,WEB1227, PGSQL1202, MAIL1211, WEB1216, mail-32, MAIL1218, WEB1207, MYSQL1214, MAIL1202, MYSQL1220, WEB1218 , WEB1217, mail-41, MAIL1213,  MAIL1205, WEBMAIL, MYSQL1305, WEB1320, WEB1314, PGSQL1302, WEB1307, WEB1325,  PGSQL1301, MAIL1317, MAIL1310, WEB1440,  MAIL1415 ,MAIL1413, MYSQL1417, MYSQL1418, WEB1417, MYSQL1425. WEB1407, WEB1418, WEB1438, MAIL1404
  • 2014/09/26 06:50 AM EST - Third set of servers have been moved and VM is booting up.
  • 2014/09/26 06:50 AM EST -  Fourth set for maintenance is as follows: WEB926, WEB910, WEB940, MYSQL914 ,WEB902, WEB915, MAIL1101 WEB1129,  MAIL1115, MAIL1109, WEB1136, WEB1113, WEB1102, WEB1116, WEB1130, WEB1105, WEB1108, WEB1227, PGSQL1202, MAIL1211, WEB1216, mail-32, MAIL1218, WEB1207, MYSQL1214, MAIL1202,  MYSQL1220, WEB1218, WEB1217, mail-41, MAIL1213, MAIL1205, WEB1337 WEB1311, WEB1329,MYSQL1309, WEB1303, WEB1302, MAIL1303, MAIL1316, WEB1327, WEB1332,  MAIL1417, MAIL1420, WEB1439, WEB1413 WEB1403, WEB1426, MAIL1402, MYSQL1414, WEB1406, WEB1429 ,WEB1433, MYSQL1430, MAIL1409

Resolution Description

The blade has been rebooted and maintenance is now over.

CP 9 – 14 Urgent System Maintenance – Resolved

Posted In: Maintenance — Sep 25th, 2014 at 2:24 am EST by IX: Kristopher G.

Incident Description:

Due to the possibility of degraded communication between virtual machines and the storage network, our system administrators will be performing urgent system maintenance within the virtual environments that CP9 -CP14 servers reside on. Administrators are performing this maintenance with the utmost care to prevent downtime to any customer facing servers but some servers may require a reboot and file system check during this process.

We will be providing updates on a server by server basis as they go under maintenance and as maintenance is completed.

Please feel free to contact support via live chat or phone should there be any concerns or questions regarding the maintenance.

List of servers to be worked on by our administrators will be as follows:

MYSQL1119 – Maintenance Completed
PGSQL1101- Maintenance Completed
MYSQL1115- Maintenance Completed
MYSQL1106- Maintenance Completed
MAIL915 – Maintenance Completed
MYSQL903- Maintenance Completed
MAIL920 – Maintenance Completed
MAIL1106 – Maintenance Completed
MYSQL1102 – Maintenance Complete
WEB1137 – Maintenance Complete
MAIL1111 – Maintenance Complete
WEB1117 – Maintenance Complete
MYSQL1123 – Maintenance Complete
MAIL1113 – Maintenance Complete
MAIL1105 – Maintenance Complete
MYSQL1104 – Maintenance Complete
MAIL1119 – Maintenance Complete
WEB1118 – Maintenance Complete
WEB1126 – Maintenance Complete
MAIL1117 – Maintenance Complete
MYSQL1108 – Maintenance Complete
MAIL1120 – Maintenance Complete
WEB1132 – Maintenance Complete
WEB1120 – Maintenance Complete
MYSQL1124 – Maintenance Complete
WEB1127 – Maintenance Complete
MYSQL1128 – Maintenance Complete
MAIL1121 – Maintenance Complete
MYSQL1121 – Maintenance Complete
WEB1131 – Maintenance Complete
MAIL1112 – Maintenance Complete
MYSQL1118 – Maintenance Complete
WEB1114 – Maintenance Complete
MYSQL1119 – Maintenance Complete
PGSQL1101 – Maintenance Complete
MYSQL1115 – Maintenance Complete
MYSQL1101 – Maintenance Complete
MAIL1104 – Maintenance Complete
MAIL1102 – Maintenance Complete
WEB1133 – Maintenance Complete
MYSQL1110 – Maintenance Complete
WEB1121 – Maintenance Complete
MYSQL1106 – Maintenance Complete
MAIL1103 – Maintenance Complete
MAIL1107 – Maintenance Complete
MYSQL1103 – Maintenance Complete
MYSQL1114 – Maintenance Complete
WEB1128 – Maintenance Complete
WEB1107 – Maintenance Complete
WEB1119 – Maintenance Complete
WEB1111 – Maintenance Complete
MYSQL1120 – Maintenance Complete
MYSQL1109 – Maintenance Complete
MYSQL1122 – Maintenance Complete
MAIL1125 – Maintenance Complete
WEB1134 – Maintenance Complete
MYSQL1125 - Maintenance Complete
WEB1123 - Maintenance Complete
MAIL1116 - Maintenance Complete
MYSQL1107 - Maintenance Complete
WEB1106 - Maintenance Complete
WEB1103 - Maintenance Complete
MAIL1124 - Maintenance Complete
WEB112 - Maintenance Complete
MYSQL1111 - Maintenance Complete
SS11 - Maintenance Complete

Which Customers are Impacted?

All customers on CP9 - CP14

How are Customers Impacted?

There may be some slowness while the server is under maintenance.  A few servers may also require reboot during this maintenance.  If maintenance requires a server reboot, we expect a file system check to run and services will be unavailable until this completes.

How often will we be updated?

Servers will be added as they come under maintenance.

Time to Resolution (ETA)

The maintenance window will be completed by 9AM.

Incident Updates

  • 2014/09/25 02:10 AM EDT - Maintenance was started and following VMs are being moved - MYSQL1119, PGSQL1101, MYSQL1115, MYSQL1106
  • 2014/09/25 03:00 AM EDT - First set of servers have been moved and VM is booting up.
  • 2014/09/25 03:20 AM EDT - First set of servers completed maintenance with no problems.  Second set for maintenance is as follows: MAIL915, MYSQL903, MAIL920
  • 2014/09/25 03:42 AM EDT - Maintenance is complete on the following servers: MAIL915, MYSQL903, MAIL920. No downtime was experienced.
  • 2014/09/25 05:02 AM EDT - Maintenance is complete on following servers: MAIL1106 ,MYSQL1102, WEB1137, MAIL1111, WEB1117, MYSQL1123, MAIL1113, MAIL1105, MYSQL1104, MAIL1119, WEB1118, WEB1126, MAIL1117. No down time experienced.
  • 2014/09/25 06:50 AM EDT - Maintenance is complete on MYSQL1108, MAIL1120, WEB1132, WEB1120, MYSQL1124, WEB1127, MYSQL1128, MAIL1121, MYSQL1121, WEB1131, MAIL1112, and MYSQL1118.  No down time experienced.
  • 2014/09/25 07:30 AM EDT - Maintenance is complete on WEB1114, MYSQL1119, PGSQL1101, MYSQL1115, MYSQL1101, MAIL1104, MAIL1102, WEB1133, MYSQL1110, WEB1121, MYSQL1106, MAIL1103, MAIL1107. No downtime experienced.
  • 2014/09/25 8:35 AM EDT - Maintenance is complete on MYSQL1103, MYSQL1114, WEB1128, WEB1107, WEB1119, WEB1111, MYSQL1120, MYSQL1109, MYSQL1122, MAIL1125, WEB1134. No downtime experienced.
  • 2014/09/25 8:53 AM EDT - Maintenance is currently processing on MYSQL1125, WEB1123, MAIL1116, MYSQL1107, SS11, WEB1106, WEB1103, MAIL1124, WEB1112, MYSQL1111
  • 2014/09/25 9:42 AM EDT - Maintenance is complete on MYSQL1125, WEB1123, MAIL1116, MYSQL1107, SS11, WEB1106, WEB1103, MAIL1124, WEB1112, MYSQL1111

Resolution Description

Maintenance has been completed, all servers are up and running.

CP9 and CP12 – Server Downtime – Resolved

Posted In: Outage — Sep 24th, 2014 at 12:04 pm EST by IX: Angela A.

Incident Description:

Some servers behind cp9 and cp12 are experiencing downtime right now. Our administrators are working on resolving this issue.

Affected Servers:

mail-42 – Up
mail-51 – Up
mail-52 – Up
mail914 – UP
mail1204 – Up
mail1210 – Up
mail1215 – Up
mail83 – Up
mail902 – Up
mysql1202 – Up
mysql1203 – Up
mysql1204 – Up
pgsql1201 – Up
web305 – Up
web904 – Up
web914 - Up
web916 - Up
web918 - Up
web929 - Up
web933 - Up
web1203 - Up
web1206 - Up
web1210 - Up
web1219 - Up
web1221 - Up
web1230 – Up
web1233 - Up
web1234 - Up
web1237 – Up
web1238 – Up

Which Customers are Impacted?

All customers using any of the servers listed.

How are Customers Impacted?

Websites, email or databases are affected.

How often will we be updated?

As often as possible

Time to Resolution (ETA)

~1 hour

Incident Updates

  • 2014/09/24 01:14 PM EDT - The following servers are back up and running:
mail1204 mail1210 mail902 mysql1204 web1237 web1230 The following servers are in FSCK status: mail-51 mail1215 mail914 mysql1202 mysql1203 pgsql1201 web1238
  • 2014/09/24 01:43 PM EDT -  All servers are up and running except the following, which are in FSCK status:  mail914, web1238, mail-42, mail-52
  • 2014/09/24 02:16 PM EDT -  All servers are up and running except the following, which are in FSCK status:  mail914, web1238
  • 2014/09/24 02:36 PM EDT -  mail914 and web1238 are still under FSCK status.
  • 2014/09/24 03:30 PM EDT -  web1238 is back up and running, mail914 is still under FSCK status.
  • 2014/09/24 4:31 PM EDT - mail914 is still under FSCK status.
  • 2014/09/24 5:44 PM EDT - mail914 still under FSCK
  • 2014/09/24 6:40 PM EDT - mail914 is still under FSCK

Resolution Description

All servers are back up and running.  If you are still experiencing any problems, please don't hesitate to let us know.

mysql1215 – Maintenance – Resolved

Posted In: Maintenance — Sep 20th, 2014 at 12:23 pm EST by IX: Angela A.

Incident Description:

Server is going offline for maintenance.

Which Customers are Impacted?

All customers using mysql1215 as a database for their websites.

How are Customers Impacted?

All sites using mysql1215 will be unavailable during maintenance.

How often will we be updated?

N/A

Time to Resolution (ETA)

~30 mins

Incident Updates

N/A

Resolution Description

Server is back online.

VDC1 A11-2-01 urgent reboot – Resolved

Posted In: Maintenance — Sep 20th, 2014 at 5:38 am EST by IX: Antonio S.

Incident Description:

Due to stuck process A few host needs to be rebooted affected vms will be online in time from 10 to 60 min.

Which Customers are Impacted?

Customer on VDC1 A11-2-01

How are Customers Impacted?

N/A

How often will we be updated?

N/A

Time to Resolution (ETA)

1hour

Incident Updates

N/A

Resolution Description

affected vms UP and RUNNING

Mail1401 – Maintenance – Resolved

Posted In: Outage — Sep 19th, 2014 at 5:14 pm EST by IX: John B.

Incident Description:

Mail1401 has ran into an expected error.  We are currently copying all content from Mail1401 to a new drive, and the server will be in read-only mode during the copy.

Which Customers are Impacted?

All customers on Mail1401

How are Customers Impacted?

Mail1401 will be in read-only mode: customers will be able to check current email, but no new email can be sent or received.

How often will we be updated?

4 hours

Time to Resolution (ETA)

~8 hours

Incident Updates

n/a

Resolution Description

mail1401 content copying has been finished successfully. New partition passed disc fsck without any problems.

IIS1007 – Down – Resolved

Posted In: Outage — Sep 19th, 2014 at 4:38 am EST by IX: Kristopher G.

Incident Description:

Our administrators have identified errors with IIS1007 and is causing the server to be unreachable. We are currently investigating the cause and will update once further information is obtained.

Which Customers are Impacted?

All customers on IIS1007

How are Customers Impacted?

All services on IIS1007 are unreachable

How often will we be updated?

3 hours

Time to Resolution (ETA)

3 hours

Incident Updates

N/A

Resolution Description

iis1007 is up and running

DDoS (Distributed Denial of Service) attack – Resolved

Posted In: Outage — Sep 19th, 2014 at 12:37 am EST by Kevin Mb.

Incident Description:

We are currently experiencing a DDoS (Distributed Denial of Service) attack against our services. Our system administration team is working to identify the source IPs and block this attack.

 

We will update this status blog post as we get additional information from our systems administrators.

UPDATE:

Currently at this time the DDOS attack has been identified to be affecting all customers on CP14 and anyone utilizing the name servers for NS27 and NS28. Our administrators are working to find the source of the attack and block it. We apologize for the continued delay in services.

Which Customers are Impacted?

All sites, databases, and email on CP14 and using NS27 and NS28 for domain name servers.

How are Customers Impacted?

Services unavailable on CP14 and using NS27 and NS28 for domain name servers.

How often will we be updated?

As information is provided

Time to Resolution (ETA)

N/A

Incident Updates

  • 2014/09/19 02:40 AM EDT - Our administrators have narrowed down the source of the DDOS and issue is now resolved at this time. Thank you for your continued patience and cooperation.
  • 2014/09/19 03:28 AM EDT - Our administrators have identified that CP14 is still experiencing a DDOS attack on it. At this time all customers on CP14 and using NS27 and NS28 for domain name servers are affected.
  • 2014/09/19 05:00 AM EDT -  Our administrators have placed a filter on the mentioned NS records. This means that any DNS requests to NS27/28 are currently being filtered. We are checking periodically in hopes that the DOS ceases, however are still working for a permanent resolution.

Resolution Description

Our administrators have identified the source of the attack and have now resolved this issue. All services are returning back to normal. Thank you for your patience and cooperation during this time.

DNS Name Servers Under DDoS attack – Resolved

Posted In: Outage — Sep 17th, 2014 at 12:36 pm EST by IX: Kevin N.

Incident Description:

Our Name Servers are currently experiencing a large Denial of Service attack. This is causing DNS queries to become slow or fail for some domains on our DNS servers. Our system administration team is working to identify the source and block this attack. We are working to resolve this issue as quickly as possible.

Which Customers are Impacted?

Customers using any of our web, email or database services.

How are Customers Impacted?

Services may be slow or down.

How often will we be updated?

As information becomes available.

Time to Resolution (ETA)

N/A

Incident Updates

  • 2014/09/17 02:53 PM EDT - The DDoS has been resolved, and all services are back up and running.  We will continue monitoring the servers to make sure that no other issues arise.

Resolution Description

The DDoS has been resolved, and all services are back up and running.  We will continue monitoring the servers to make sure that no other issues arise.

web412 – Maintenance

Posted In: Outage — Sep 16th, 2014 at 11:37 am EST by IX: Angela A.

Incident Description:

web412 is currently undergoing maintenance

Which Customers are Impacted?

All customers on web412

How are Customers Impacted?

All sites will be unavailable during maintenance, including Webshell and FTP access.

How often will we be updated?

N/A

Time to Resolution (ETA)

1 hour

Incident Updates

2014/09/16 10:30 AM EST - Servers are undergoing maintenance. ETA 1 hour.

Resolution Description

N/A

 
© 2011 IX Web Hosting.