Current Time: 15:26:04 EST
 

web1419 Urgent Maintenance – Resolved

Posted In: Maintenance — Aug 28th, 2015 at 5:09 am EST by IX: Antonio S.
Shared services are affected

Incident Description:

Our administrators are performing emergency maintenance on our Web1419  server.  We are working to get this back up and running as soon as possible.

Which Customers are Impacted?

All customers on web1419

How are Customers Impacted?

Web and FTP will be unavailable during this time.

How often will we be updated?

1 hour

Time to Resolution (ETA)

1 hour

Incident Updates

  • 8/28/2015 05:08AM EDT - Our administrators are performing emergency maintenance on our Web1419 server.

Resolution Description

All services have been resumed back to normal now. Thank you for your patience.

Web300 – FSCK (File System Check) – Resolved

Posted In: Maintenance — Aug 26th, 2015 at 11:33 pm EST by IX: Antonio S.
Shared services are affected

Incident Description:

Our administrators have found an issue with the server and will be performing maintenance. Currently the server is under File System Check(FSCK).

Which Customers are Impacted?

All customers on web300

How are Customers Impacted?

Services may be slow or inaccessible.

How often will we be updated?

N/A

Time to Resolution (ETA)

1 hour

Incident Updates

  • 8/26/15 11:31PM EST - Current FSCK progress: 67.3%

Resolution Description

Server is online.

Network Maintenance Wednesday, Aug 26th, 10:00 PM EDT – Resolved

Posted In: Maintenance — Aug 26th, 2015 at 3:12 pm EST by IX: Greg Cook
Cloud services are affected
Shared services are affected

Incident Description:

Our administrators will be performing maintenance on our network to expand its capacity Wednesday, Aug 26th, 10:00 PM EDT. This maintenance will take about 2 hours to complete. There is no expected downtime or interference in customer services. All servers will remain online during maintenance and we do not anticipate any effect on services.

Which Customers are Impacted?

All customers

How are Customers Impacted?

No impact expected.

How often will we be updated?

1 hour

Time to Resolution (ETA)

30 min

Incident Updates

  • 2015/8/26 11:55 PM EDT - We are extending maintenance window by 1 hour, till 1AM EDT 2015/8/27. All servers will remain online during maintenance and we do not anticipate any effect on services.
  • 2015/8/27 12:55 AM EDT - We are extending the maintenance window by 1 hour, until 2AM EDT 2015/8/27. All services will remain online during maintenance and expect no down time.

Resolution Description

Our administrators have finished the maintenance on our network and experienced no down time. Thank you for your patience and cooperation.

Web519 Maintenance 8/28/2015 2AM EDT – Resolved

Posted In: Maintenance,Server recreation — Aug 26th, 2015 at 11:17 am EST by IX: Alan M.
Shared services are affected

Incident Description:

System administrators have detected that there’s a risk of hard drive failures soon on Web519. As a precautionary measure, to ensure server stability and data integrity, we decided to migrate this server to virtualized environment. Now we performing background data sync from hardware server to virtual machine. No websites are affected by this operation. 8/28/2015 2AM EDT, Web519 will be brought down for a short period of time to sync all the changes happened during data copy and prevent any bit of data from being lost. After that downtime, new virtual server will be brought up as Web519.

Which Customers are Impacted?

Any hosted on Web519

How are Customers Impacted?

Websites on Web 519 will be down for approximately 1h

How often will we be updated?

1 hour

Time to Resolution (ETA)

1 hour

Incident Updates

  • 2015/8/28 02:10 AM EDT - Server recreation has started.
  • 2015/8/28 03:00 AM EDT - We are extending the maintenance window another 30 minutes to finalize the server recreation.
  • 2015/8/28 03:30 AM EDT - We are extending the maintenance window another 30 minutes as we have run in to some issues with changing over to new virtual environment.
  • 2015/8/28 04:06 AM EDT - ETA has been extended 1 hour due to issues with re-adding IP's to the new server.

Resolution Description

Web519 has now been changed over to the new environment. All services are now resumed back to normal. Thank you for your patience.

Web520 – Emergency Maintenace – Resolved

Posted In: Maintenance — Aug 25th, 2015 at 3:21 pm EST by IX: Alan M.
Shared services are affected

Incident Description:

System administrators were alerted of multiple drive failures on Web520. To preserve the data on that server, admins have to take it offline for emergency maintenance to replace the failed drives. Once replaced, the server is expected to come back online with no data loss but customers may experience slowness while the file system rebuilds.

Critical Update

The server is up. During the maintenance the server failed to recognize the 2 new drives. We are now in the process of copying the data to a virtual machine. Approximate ETA for data copy is 12-48 hours. During data copy the server may perform slowly. After the data is copied the server will be switched over to the virtual machine which will bring the server down for approximately 30 minutes.

To help protect this server from issues like this, it is being recreated in our new virtualized environment, which will provide a more stable environment.

Which Customers are Impacted?

All customers on Web520. To determine if your site uses this webserver:

  • login at my.ixwebhosting.com
  • click 'My Products'
  • click the 'Manage' button next to your hosting account
  • in the hosting control panel, click 'FTP Manager'
  • you will find your server name next to 'Host name'

How are Customers Impacted?

All websites on Web 520 will be unreachable.

How often will we be updated?

As available.

Time to Resolution (ETA)

1hr 30m

Incident Updates

  • 2015/8/25 4:25 PM EDT:  The server is up. During the maintenance the server failed to recognize the 2 new drives. We are now in the process of copying the data to a virtual machine. Approximate ETA for data copy is 12-48 hours. During data copy the server may perform slowly. After the data is copied the server will be switched over to the virtual machine which will bring the server down for approximately 30 minutes.
  • 2015/8/25 5:35 PM EDT:  The virtual machine has been created. Hsphere packages are being installed at the moment.
  • 2015/8/25 8:05 PM EDT:  We have started resource creation on the virtual machine. If everything goes fine the next step is content resync.
  • 2015/8/25 10:45 PM EDT: Resources have been successfully created. We have started the resync process from backup server. We will make hourly updates on resync progress.
  • 2015/8/25 11:02 PM EDT: Current resync progress: 11G of 252G copied.
  • 2015/8/26 12:00 AM EDT: Current progress: 37G of 252G copied.
  • 2015/8/26 1:00 AM EDT: Current progress: 65G of 252G copied.
  • 2015/8/26 2:00 AM EDT: Current progress: 81G of 252G copied.
  • 2015/8/26 3:00 AM EDT: Current progress: 99G of 252G copied.
  • 2015/8/26 4:00 AM EDT: Current progress: 116G of 252G copied.
  • 2015/8/26 5:00 AM EDT: Current progress: 132G of 252G copied. We expect the server to complete the copy process within 7 hours. Our next step will be synchronizing content that was changed while copy process was in progress.
  • 2015/8/26 6:00 AM EDT: Current progress: 151G of 252G copied.
  • 2015/8/26 7:00 AM EDT: Current progress: 165G of 252G copied.
  • 2015/8/26 8:00 AM EDT: Current progress: 188G of 252G copied.
  • 2015/8/26 9:20 AM EDT: Current progress: 206G of 252G copied.
  • 2015/8/26 11:30 AM EDT: Copy Complete. We are now copying data from the hardware server to the virtual machine. Web520 will remain online and web services will not be affected during this period.
  • 2015/8/26 3:25 PM EDT: We are done copying data from hardware to the virtual machine. Tonight at 8:00 PM EDT, we will bring Web 520 offline, perform final sync of the latest changes and bring up new virtual machine as Web 520 in 1 hr, at 9:00  PM EDT.
  • 2015/8/26 7:55 PM EDT: The system administration team is now taking Web 520 offline for a final sync. During this operation, websites at this server will be unreachable. After that, the new virtual machine will be brought up as Web 520. We estimate downtime required for this operation as up to 1 hour and a half.
  • 2015/8/26 9:40 PM EDT: The switch over to the new server is now complete and all services have been resumed. Our administrators will be monitoring this server going forward to make sure everything is resolved. ETA 30 minutes.

Resolution Description

Web520 has resumed all services at this time in the new environment. Thank you for your patience and cooperation during this time.

Webmail maintenance – Resolved

Posted In: Maintenance — Aug 23rd, 2015 at 3:46 am EST by IX: Charles Smith
Shared services are affected

Incident Description:

this is related to removal of small portion of unused accounts from Webmail backend DB. Webmail could be slightly slow during this maintenance but it is unlikely.

Which Customers are Impacted?

All customers using web mail to access their mail.

How are Customers Impacted?

Webmail could be slightly slow during this maintenance but it is unlikely

How often will we be updated?

n/a

Time to Resolution (ETA)

n/a

Incident Updates

2015/8/23 2:07 AM EDT: ETA is 70 minutes 2015/8/23 3:00 AM EDT: Maintenance is over.

Resolution Description

resolved

Scheduled Maintenance – Dashboard Control Panel – Resolved

Posted In: Outage — Aug 17th, 2015 at 5:13 pm EST by IX: Andrew Y.
Cloud services are affected
VPS services are affected
Shared services are affected

Incident Description:

On Wednesday morning August 19th from 4AM to 5AM EDT, our administrators will perform maintenance on our Dashboard Control Panel database array to increase redundancy and improve stability. During this maintenance period, the Dashboard Control Panel will be unavailable. All user servers, websites, and email are unaffected by this maintenance and will operate normally.

Which Customers are Impacted?

All Customers attempting to login to their Dashboard Control Panel

How are Customers Impacted?

The Dashboard Control Panel will be unavailable during maintenance.

How often will we be updated?

Upon resolution

Time to Resolution (ETA)

1 hour

Incident Updates

N/A

Resolution Description

Maintenance is complete.

Web317 Maintenance – Resolved

Posted In: Maintenance — Aug 15th, 2015 at 6:45 pm EST by IX: Eric Lutton
Shared services are affected

Incident Description:

A physical hard drive failed on Web317. After replacing the drive, the RAID failed to rebuild and the file system was not recoverable. Our system administration team is recreating the server which is estimated to be completed within 16 hours. During this time the sites hosted on the server will be unavailable.

To help protect this server from issues like this, it is being recreated in our new virtualized environment, which will provide a much stabler environment. After the recreation, all web content will be restored from our most recent backup dated Aug-09.

Which Customers are Impacted?

All customers on web317.

How are Customers Impacted?

Some customer sites or a portion of some customer sites may appear down until everything is updated by system administrators.

How often will we be updated?

Customers will be updated as often as possible.

Time to Resolution (ETA)

30 minutes.

Incident Updates

  • 2015/8/15 08:16 PM EDT - Updated description with more detailed information
  • 2015/8/15 09:42 PM EDT - VM creation is complete. OS and software installation is in progress.
  • 2015/8/15 10:36 PM EDT - System Administrators are now creating user home directories. This is the final step before the full data copy from backup begins.
  • 2015/8/16 12:28 AM EDT - Physical Creator has finished account creation. We have started the full data copy from backup server.
  • 2015/8/16 1:36 AM EDT - Full data copy from backup server is in progress: 17GB out of 642GB has been copied.
  • 2015/8/16 2:30 AM EDT - Full data copy from backup server: 35GB out of 642GB has been copied.
  • 2015/8/16 3:30 AM EDT - Full data copy from backup server: 55GB out of 642GB has been copied.
  • 2015/8/16 4:30 AM EDT - Full data copy from backup server: 75GB out of 642GB has been copied.
  • 2015/8/16 5:30 AM EDT - Full data copy from backup server: 94GB out of 642GB has been copied.
  • 2015/8/16 6:30 AM EDT - Full data copy from backup server: 110GB out of 642GB has been copied.
  • 2015/8/16 7:50 AM EDT - Full data copy from backup server: 128GB out of 642GB has been copied.
  • 2015/8/16 8:35 AM EDT - Full data copy from backup server: 151GB out of 642GB has been copied.
  • 2015/8/16 10:55 AM EDT - Full data copy from backup server: 198GB out of 642GB has been copied.
  • 2015/8/16 12:05 PM EDT - Full data copy from backup server: 220GB out of 642GB has been copied.
  • 2015/8/16 1:20 PM EDT - Full data copy from backup server: 241GB out of 642GB has been copied.
  • 2015/8/16 2:24 PM EDT - Full data copy from backup server: 257GB out of 642GB has been copied.
  • 2015/8/16 2:44 PM EDT - Full data copy from backup server: 283GB out of 642GB has been copied.
  • 2015/8/16 3:33 PM EDT - Full data copy from backup server: 302GB out of 642GB has been copied.
  • 2015/8/16 4:37 PM EDT - Full data copy from backup server: 325GB out of 642GB has been copied.
  • 2015/8/16 5:37 PM EDT - Full data copy from backup server: 357GB out of 642GB has been copied.
  • 2015/8/16 6:45 PM EDT - Full data copy from backup server: 381GB out of 642GB has been copied.
  • 2015/8/16 7:25 PM EDT - Full data copy from backup server: 402GB out of 642GB has been copied.
  • 2015/8/16 8:42 PM EDT - Full data copy from backup server: 431GB out of 642GB has been copied.
  • 2015/8/16 9:32 PM EDT - Full data copy from backup server: 449GB out of 642GB has been copied.
  • 2015/8/16 10:32 PM EDT - Full data copy from backup server: 472GB out of 642GB has been copied.
  • 2015/8/16 11:44 PM EDT - Full data copy from backup server: 501GB out of 642GB has been copied.
  • 2015/8/17 12:38 AM EDT - Full data copy from backup server: 521GB out of 642GB has been copied. We expect the server to complete the copy process and to be up within 7 hours.
  • 2015/8/17 1:32 AM EDT - Full data copy from backup server: 543GB out of 642GB has been copied.
  • 2015/8/17 2:42 AM EDT - Full data copy from backup server: 566GB out of 642GB has been copied.
  • 2015/8/17 3:38 AM EDT - Full data copy from backup server: 589GB out of 642GB has been copied.
  • 2015/8/17 4:36 AM EDT - Full data copy from backup server: 612GB out of 642GB has been copied.
  • 2015/8/17 6:54 AM EDT - We have finished the copy process and the server is up and running. Our Administrators are doing the final check on all services to ensure that everything works smoothly. The new virtual machine runs an updated operating system so this check includes a check of any binary files that need updating. As a result, a portion of affected sites' content may be inaccessible until this update process is complete.
  • 2015/8/17 9:25 AM EDT - Updated ETA
  • 2015/8/17 11:28 AM EDT - Issue Resolved

Resolution Description

N/A

Web Server Maintenance – Resolved

Posted In: Maintenance — Aug 12th, 2015 at 8:32 pm EST by IX: Greg Cook
Shared services are affected

Incident Description:

Tonight, Wednesday August 12th at 11:30PM EDT, we will be relocating 7 servers within our data center.  During maintenance, the servers will be powered down and powered back up.  We will do this one server at a time and we do not expect more than 10 minutes of downtime per server.
Servers affected:
web241 — Server is now UP
web242 — Server is UP
web243 — Server is UP
web245 — Server is UP
web247 –Server is UP
web248 –Server is UP
web249 — Server is UP

 

Which Customers are Impacted?

All customers on

web241
web242
web243
web245
web247
web248
web249
 

How are Customers Impacted?

We do not expect more than 10 minutes of downtime per server.

How often will we be updated?

As necessary.

Time to Resolution (ETA)

Approximately 70 minutes.

Incident Updates

  • 2015/8/12 11:30 PM EST - Maintenance is started and Web241 is the first server.
  • 2015/8/13 12:38 AM EST - web243 has been brought back up in it's original place and the move will be reattempted at a later time.
  • 2015/8/13 12:46 AM EST - Web243 is now being taken down and relocated.
  • 2015/8/13 12:58 AM EST - Web243 is now up in it's new location. Web245 is next

Resolution Description

Maintenance is now complete and all servers are up and running.

Webmail Services Offline – Resolved

Posted In: Outage — Aug 12th, 2015 at 12:02 pm EST by IX: John Richards
Shared services are affected

Incident Description:

Currently we are seeing issues with customers trying to access their email services through our webmail or through mail.domain.com and/or mail.opentransfer.com URLs. Right now these services are down but local mail clients (Outlook, Mac Mail, Thunderbird) are working correctly.

Which Customers are Impacted?

Any customer trying to access their email services through our webmail portal.

How are Customers Impacted?

Unable to check email online using our webmail portal.

How often will we be updated?

n/a

Time to Resolution (ETA)

n/a

Incident Updates

n/a

Resolution Description

Our system admin team was able to bring the webmail services back online.

Web407 – FSCK – Resolved

Posted In: Outage — Aug 11th, 2015 at 1:29 am EST by IX: Charles Smith
Shared services are affected

Incident Description:

The OS on Web 407 reported errors with a partition and forced the server into read only mode in order to preserve data. System Administrators have determined that the error was caused due to bad blocks on one of the server’s hard drives. This can cause instability of data so the FSCK was stopped and the server was brought online to update backups with the most current server data. In order to preserve data and increase performance, stability, and reliability on the server, admins will be using this updated backup to recreate the server as a virtual machine on new hardware. The server is up and running while the backup information is being copied to the new machine. Once the copy is completed, we will have to bring the server down for up to 30 minutes to perform a final sync of recently changed data then the server will be back online in its new stable environment.

Which Customers are Impacted?

All customers on Web407

How are Customers Impacted?

All services have been resumed.

How often will we be updated?

Where updates become available.

Time to Resolution (ETA)

30 Minutes

Incident Updates

  • 2015/08/11 03:52 AM EDT - System Administrators started updating data in backup vault to have immediate copy of the server. No services are affected, web server is up and running while data is being copied in a background.
  • 2015/08/11 06:47 AM EDT - Backup copy  has finished and the backup vault now contains the latest copy. We are proceeding with copying the data to the new server. Currently Web407 is online and all web services are running fine.
  • 2015/08/11 07:36 AM EDT - Our system admin team are in the process of copying the data to the new server. Due to the volume of data on the server, we extending the ETA to 30h. Please remember that Web407 is online and will remain online during copying of the data to the new server. Web services and server performance are not affected.
  • 2015/08/11 11:25 PM EDT - 50% of the content has been copied to the new server. Note that web407 is still online and will remain online while content is copied. No web services are affected by this operation.
  • 2015/08/12 5:35 PM EDT - Data copy to the new server is finished and it now contains the latest backup copy. We are proceeding with syncing most recent changes from web 407 to the new server.  Note that web407 is still online and will remain online while content is copied. No web services are affected by this operation
  • 2015/08/12 11:21 PM EDT - Syncing is taking longer than expected, new ETA is 2-3 hours
  • 2015/08/12 2:11 AM EDT - The background data sync between web 407 and the new virtual server is finished. The server must be taken offline to perform a final sync of any content that has changed since  the process began and to switch over to the new virtual hardware. This could take up to 4h, but we will do whatever possible to reduce downtime.
  • 2015/8/13 06:14 AM EDT - The servers data sync has been extended another 3 hours at this time.
  • 2015/8/13 06:47 AM EDT - Sync is now complete and our administrators are working to change over to the new server at this time. New ETA 30 minutes.
  • 2015/8/13 07:18 AM EDT - The switch over to the new server is now complete and all services have been resumed. Our administrators will be monitoring this server going forward to make sure everything is resolved. ETA 30 minutes.

Resolution Description

Our system admin team was able to create a new server and move the content from Web407.

Problem making changes to some domains – Resolved

Posted In: Outage — Aug 10th, 2015 at 9:49 pm EST by IX: Greg Cook
my.ixwebhosting.com affected

Incident Description:

There is problem affecting the ability to make changes and updates to some existing domains.  From the My.IX interface, when trying to open “Settings” for domain registration the following notification pops up:

“We are unable to communicate with the domain registrar at this time. Please try again later.”

This problem is affecting domains registered via Tucows.  (The registrar is visible in the whois details.)

This issue is limited to the ability to edit domain settings and does not impact the function of the domain.

Which Customers are Impacted?

All customers with domains registered via Tucows.

How are Customers Impacted?

Customers will be unable to update domain product details.

How often will we be updated?

1 hour

Time to Resolution (ETA)

Unknown

Incident Updates

  • 10:15 PM EDT - We have escalated to senior levels with our vendor.
  • 10:54 PM EDT - Vendor engineers have identified the cause of the issue and are working on resolution.  No ETA at this time.
  • 11:46 PM EDT - Our registrar has resolved the issue and customers should be able to update their domain.

Resolution Description

Resolved

Live Chat Support Issues – Resolved

Posted In: Outage — Aug 03rd, 2015 at 2:39 am EST by IX: Charles Smith

Incident Description:

Our live chat provider is experiencing issues which have caused our live chat support to be temporarily unavailable. We are in communication with our provider and they are currently working on resolving this issue.

We apologize for the inconvenience, but our ticket and phone support are fully operational.

Which Customers are Impacted?

All customers using Live Chat.

How are Customers Impacted?

May not be able to reach us via Live Chat.

How often will we be updated?

When updates become available.

Time to Resolution (ETA)

There is currently no ETA.

Incident Updates

na

Resolution Description

Our Live Chat services have resumed back to normal. Thank you for your patience.

Server Maintenance – Wednesday, July 29, 2015 & Thursday, July 30, 2015 – Resolved

Posted In: upgrades to products — Jul 29th, 2015 at 6:08 pm EST by IX: Greg Cook
Cloud services are affected

Incident Description:

Starting July 29, 2015, IX Web Hosting will be adding new hardware to the existing servers that currently house your data.  These upgrades will provide additional capacity, as well as improved disk performance to our existing cloud infrastructure.
The upgrades to the server pool will take place July 29, 2015 and July 30, 2015 at our Data Center located in Columbus Ohio.

These upgrades will be applied so as not to impact any customer VMs.  This upgrade will be seamless, and the only noticeable change will be an increase in overall disk performance.

 

We do not anticipate any customer to experience downtime during this maintenance.  Additionally, as a precaution, you can also perform a local backup of your content. Refer to our Knowledge Base for assistance or via web chat, or please call us at anytime.

To follow the progress of the update, please watch this status blog for updates.

Thank you,
Your IX Web Hosting Team

Which Customers are Impacted?

All CloudbyIX product customers.  Some customers on IX and CloudByIX brands also may be impacted.

How are Customers Impacted?

No impact to any customers VMs.

How often will we be updated?

As required.

Time to Resolution (ETA)

Maintenance is expected to be complete by August 3rd.

Incident Updates

Additional storage has been added and as data spreads across the new storage, we are seeing even better server performance for all cloud customers.  The system is moving data slowly, as resources allow, so that it never interferes with customer site performance.  This method is safest for customer performance but it is taking longer to migrate this data than anticipated.  Right now, System Admins are expecting this maintenance to be completed by August 3rd.

Resolution Description

The upgrades are completed.

DDoS (Distributed Denial of Service) attack – Resolved

Posted In: Outage — Jul 29th, 2015 at 5:58 pm EST by IX: Greg Cook
Shared services are affected

Incident Description:

We are currently experiencing a DDoS (Distributed Denial of Service) attack against shared IP 98.130.148.2 on web415. In order to stop the attack on it will be blocked on 2 hours.

We will update this status blog post as we get additional information from our systems administrators.

Which Customers are Impacted?

Anyone on shared IP 98.130.148.2 on web415.

How are Customers Impacted?

Domains on shared IP 98.130.148.2  will not be available.

How often will we be updated?

2 hours

Time to Resolution (ETA)

2 hours

Incident Updates

  • 2015/07/29 09:25 AM EDT - Block has been released and IP is online right now.

Resolution Description

The DDoS is resolved.

 
© 2011 IX Web Hosting.