Current Time: 15:26:04 EST
 

Emergency Maintenance – mysql1102 – Resolved

Posted In: Outage — Jan 17th, 2017 at 6:50 am EDT by IX: Jason H.
Shared services are affected

Incident Description:

The systems administrators have brought the mysql1002 down for emergency maintenance.

Which Customers are Impacted?

All customer's sites using the mysql1102 server.

How are Customers Impacted?

Sites utilizing the mysql1102 server will show errors.

How often will we be updated?

As provided.

Time to Resolution (ETA)

3 hour.

Incident Updates

  • 01-17-2017 10:15 AM EDT -  All major maintenance works were finished and we need 1 more hour to finalize the process.

Resolution Description

Database server is back online and working correctly.

Cloud Storage Maintenance – Resolved

Posted In: Outage — Jan 09th, 2017 at 11:58 pm EDT by IX: Jason H.
Cloud services are affected

Incident Description:

System Engineers identified an issue with cloud storage connectivity.  Working with the hardware vendor, they determined that a reboot of the storage controller is necessary to resolve this issue.  This controller reboot will also result in some VM’s being rebooted to return to normal operation.  Once the controller is back online, all servers will be checked and addressed by our Admin team.

Which Customers are Impacted?

Some Cloud customers may be affected

How are Customers Impacted?

Some Cloud VMs will require a reboot

How often will we be updated?

30 mins

Time to Resolution (ETA)

To be announced

Incident Updates

  • 1-10-2017 12:40 AM EDT: 24% of effected servers have been confirmed online.
  • 1-10-2017 1:10 AM EDT: 64% of effected servers have been confirmed online.
  • 1-10-2017 1:30 AM EDT: 98% of effected servers have been confirmed online.
  • 1-10-2017 2:00 AM EDT:  All servers have been confirmed online.

Resolution Description

All servers have been confirmed online.

Connectivity Issues – Resolved

Posted In: Outage — Jan 07th, 2017 at 9:23 am EDT by IX: Daren H.
Shared services are affected

Incident Description:

Our engineers have discovered an issue with connectivity to servers and that it affects some servers across cp9 through CP11.

Which Customers are Impacted?

Customers on CP9 through CP11

How are Customers Impacted?

Customers on CP9 through CP11

How often will we be updated?

Update as soon as we have more info.

Time to Resolution (ETA)

N/A

Incident Updates

N/A

Resolution Description

The issue was resolved by a reboot of an internal networking switch.

mail1105 – Emergency maintenance – Resolved

Posted In: Outage — Jan 05th, 2017 at 6:29 am EDT by IX: Victoria Witten
Shared services are affected

Incident Description:

We are currently experiencing issues with the mail1105  server.  The server is being restarted and is now doing a mandatory file system check (FSCK).   No data loss is expected.

Which Customers are Impacted?

All customers using the mail1105 server.

How are Customers Impacted?

Customers will not be able to send or receive email messages from the server.

How often will we be updated?

As provided.

Time to Resolution (ETA)

1 hour.

Incident Updates

  • 01/05/2017 7:15 EDT: 59% of FSCK has been completed.

Resolution Description

All services are now back to normal.

Mail server emergency maintenance – Resolved

Posted In: Maintenance — Jan 04th, 2017 at 10:43 am EDT by IX-Toi Lynn S
Shared services are affected

Incident Description:

Mail servers listed below are currently under heavy load causing them to be inaccessible, our system admins are working to reduce the load. All mail will still be received.

Which Customers are Impacted?

All customers on listed mail servers will be affected.

How are Customers Impacted?

The mail servers will be inaccessible but all mail will still be received.

How often will we be updated?

When available.

Time to Resolution (ETA)

30 Minute.s

Incident Updates

      • 01/04/2017 15:06 EDT: Administrators solved the issue with the mail servers indicated above and are working to further stabilize the mail environment. While they complete this work, other servers may experience intermittent slowness.  We are currently seeing a load on mail1105, mail1104, mail920, mail1103 and mail903 and administrators are addressing those servers now.
      • 01/04/2017 16:06 EDT: Administrators are monitoring the system as it stabilizes and customers may see intermittent slowness over the next two hours
      • 01/04/2017 16:36 EDT: mail1104 and mail920 load is back to normal

Resolution Description

Loads have returned to normal on all mail servers.

VDC Environment Issues – Resolved

Posted In: Outage — Jan 04th, 2017 at 2:52 am EDT by IX: Jason H.
Cloud services are affected

Incident Description:

We are currently experiencing issues with our VDC environment that is causing some VMs to become temporarily unavailable.  Our engineers are working on this issue.

Which Customers are Impacted?

All clients on the VDC hosting environment.

How are Customers Impacted?

Servers may not be accessible.

How often will we be updated?

As provided.

Time to Resolution (ETA)

Will be provided when available.

Incident Updates

  • 01/04/2017 2:56 AM (EST): Engineers have discovered an issue causing excessive write latency on some volumes. We are working with the storage vendor to troubleshoot the cause of the issue. We will provide updates as the SA investigation continues.
  •  01/04/2017 3:09 AM (EST): The issue causing write latency has been resolved but the latency resulted in some VMs changing to ReadOnly status in order to protect the data. We are currently rebooting VMs to put them back into a normal running state.
  • 01/04/2017 4:23 AM (EST):  Administrators continue to reboot and manually confirm normal operation of all affected servers.  39% of effected servers have been confirmed online.
  • 01/04/2017 4:46 AM (EST): 57.6% of effected servers have been confirmed online.
  • 01/04/2017 5:15 AM (EST): 72.3% of effected servers have been confirmed online.
  • 01/04/2017 5:43 AM (EST): 84.2% of effected servers have been confirmed online.
  • 01/04/2017 6:07 AM (EST): 98.4% of effected servers have been confirmed online.

Resolution Description

Service has restored.

VDC Environment – storage connection issues- Resolved

Posted In: Other Issues — Dec 29th, 2016 at 12:05 am EDT by IX: Victoria Witten
Cloud services are affected

Incident Description:

We are currently experiencing storage connectivity issues with our VDC environment that is causing some Linux VMs to move to read only status or need to be rebooted.  Our engineers are working on this issue.

Which Customers are Impacted?

All clients on the VDC hosting environment.

How are Customers Impacted?

Servers may not be accessible.

How often will we be updated?

As provided.

Time to Resolution (ETA)

Will be provided when available.

Incident Updates

  • 2016-12-29 1:00 AM EST: Engineers have discovered an array issue that caused a connectivity loss which put the VMs into ReadOnly status.   We are currently rebooting VMs to put them back into full running state and working with the storage vendor to troubleshoot the cause the issue.
  • 2016-12-29 1:40 AM EST: 14% of the effected servers has been brought back online.
  • 2016-12-29 3:00 AM EST: 39% of the effected servers has been brought back online.
  • 2016-12-29 3:30 AM EST: 50% of the effected servers has been brought back online.
  • 2016-12-29 4:30 AM EST: 67% of the effected servers has been brought back online.
  • 2016-12-29 5:00 AM EST: 76% of the effected servers has been brought back online.
  • 2016-12-29 5:30 AM EST: 90% of the effected servers has been brought back online.
  • 2016-12-29 6:00 AM EST:  93% of the effected servers has been brought back online.
  • 2016-12-29 8:04 AM EST: 96% of the effected servers has been brought back online.

Resolution Description

n/a

Mail Server Emergency Maintenance – Resolved

Posted In: Maintenance — Dec 21st, 2016 at 4:00 pm EDT by IX: John Richards
Shared services are affected

Incident Description:

At this moment mail server mail507 is under an emergency maintenance. Our admin team is having to replace the network card on the server. No email is expected to be lost by replacing the card.

Which Customers are Impacted?

all customers on mail507

How are Customers Impacted?

Customers will not be able to send or receive email messages from the server.

How often will we be updated?

n/a

Time to Resolution (ETA)

1 hour

Incident Updates

n/a

Resolution Description

The mail server is back online and working as normal.

Emergency Maintenance – mysql1004 – Resolved

Posted In: Maintenance,Outage — Dec 20th, 2016 at 3:23 am EDT by IX: Victoria Witten
Shared services are affected

Incident Description:

The systems administrators have brought the mysql1004 down for emergency maintenance.

Which Customers are Impacted?

All customer's sites using the mysql1004 server.

How are Customers Impacted?

Sites utilizing the mysql1004 server will show errors.

How often will we be updated?

As provided.

Time to Resolution (ETA)

1 hour

Incident Updates

  • 2016/12/20 3:55 AM EST: Maintenance has been extended for one hour.
  • 2016/12/20 4:50 AM EST:  Maintenance has been extended for two additional hour.
  • 2016/12/20 6:50 AM EST: Maintenance has been extended for an additional 30 minutes.

Resolution Description

The database server is back online and working as normal.

Urgent Network Maintenance – Resolved

Posted In: Outage — Dec 12th, 2016 at 7:31 am EDT by IX: John Richards
Shared services are affected

Incident Description:

Our networking team reports issues with a switch providing network connectivity to some web and MySQL servers. They have attempted to reset the switch but determined it has to be replaced. They are performing maintenance to replace it and expect services to return normally, with no data loss, soon.

Which Customers are Impacted?

All customers who utilize these servers.

How are Customers Impacted?

Services may become unavailable or slow.

How often will we be updated?

n/a

Time to Resolution (ETA)

n/a

Incident Updates

n/a

Resolution Description

The new switch has been installed and all servers are back online.

web419 – Scheduled Maintenance-Complete

Posted In: Maintenance — Nov 21st, 2016 at 11:22 am EDT by IX: Alena S.
Shared services are affected

Incident Description:

On 26th of November at 3am EST system administrators will convert the web419 shared server to a virtual machine environment.  The server will be offline for up 20 minutes during the switch over.

Which Customers are Impacted?

All customers on the web419 server.

How are Customers Impacted?

Server will be inaccessible during the switch over.

How often will we be updated?

As provided.

Time to Resolution (ETA)

20 minutes.

Incident Updates

N/A

Resolution Description

Maintenance has completed and the server is now up in the new virtual environment. No issues were experienced during the change over. Please contact support if any issues are experienced. Thank you.

web419 – FSCK(File System Check) – Resolved

Posted In: Outage — Nov 18th, 2016 at 5:10 am EDT by IX: Customer Relations
Shared services are affected

Incident Description:

Our system administrators have detected an issue with this server. The server is being restarted and is now doing a mandatory file system check (FSCK). During this time the web server will be unavailable, meaning that websites hosted on the server will not be working.

Which Customers are Impacted?

All clients using this web server.

How are Customers Impacted?

All websites/FTP/control panel will not be reachable.

How often will we be updated?

3 hours

Time to Resolution (ETA)

3 hours

Incident Updates

N/A

Resolution Description

FSCK has completed and all services resumed back to normal. Thank you.

web418 – Scheduled Maintenance – Resolved

Posted In: Maintenance — Nov 14th, 2016 at 10:21 am EDT by IX: Victoria Witten
Shared services are affected

Incident Description:

On 19th of November system administrators will convert the web418 shared server to a virtual machine environment.  The server will be offline for up 20 minutes during the switch over.

Which Customers are Impacted?

All customers on the web418 server.

How are Customers Impacted?

Server will be inaccessible during the switch over.

How often will we be updated?

As provided.

Time to Resolution (ETA)

20 minutes.

Incident Updates

  • 2016/11/19 03:00 AM EDT - Maintenance has begun.
  • 2016/11/19 03:10 AM EDT- Maintenance has been completed.

Resolution Description

Maintenance has been completed.  The services are running.

iis1001 – ASP.Net – Down – Resolved

Posted In: Outage — Nov 14th, 2016 at 1:49 am EDT by IX: Jason H.
Shared services are affected

Incident Description:

iis1001 – ASP.Net is down. Server became stuck and our System Administrators have to restart it by force. No data loss is expected to occur from this.

Which Customers are Impacted?

All clients using this web server will be affected.

How are Customers Impacted?

All clients using this web server.

How often will we be updated?

30 Minutes

Time to Resolution (ETA)

30 Minutes

Incident Updates

  • 2016/11/14 1:59 am EST : Server has come back up

Resolution Description

Services have been restored.

Live Chat – Maintenance – RESOLVED

Posted In: Outage — Nov 13th, 2016 at 5:42 am EDT by IX: Jason H.
Chat support is affected

Incident Description:

Our live chat provider is experiencing issues which have caused our live chat support to be temporarily unavailable. We are in communication with our provider and they are currently working on resolving this issue. It is expected that service will be normal throughout the remainder of the maintenance window, there may be some momentary disconnections that can last up to 2 minutes.

We apologize for the inconvenience, but our ticket and phone support are fully operational.

Which Customers are Impacted?

All customers using Live Chat.

How are Customers Impacted?

May not be able to reach us via Live Chat.

How often will we be updated?

When updates become available.

Time to Resolution (ETA)

There is currently no ETA.

Incident Updates

Maintenance has been completed and live chat is available.

Resolution Description

Maintenance has been completed and live chat is available.

 
© 2011 IX Web Hosting.