Current Time: 15:26:04 EST
 
20 May 2013 3:15 pm

Smart Host Mail Queues – Resolved

Posted In: Outage — May 20th, 2013 at 3:15 pm EST by IX: Omari J.

Incident Description:

The mail queues, on the smart host servers, have backed up and are currently being worked on to continue the flow of mail normally.

Which Customers are Impacted?

All customers will be affected when sending mail.

How are Customers Impacted?

Mail may be not be delivered instantaneously

How often will we be updated?

3 hours

Time to Resolution (ETA)

4-6 hours

Incident Updates

N/A

Resolution Description

The SMH queues have been cleared and mail should be functioning correctly now.

19 May 2013 4:23 am

Web1436 shared IP: 76.162.106.2 – Filtered – Resolved

Posted In: Security Issues — May 19th, 2013 at 4:23 am EST by Kris G

Incident Description:

We are experiencing irregular traffic on this shared IP. Our system administrators have initiated an IP filter. We will recheck incoming traffic when the filter expires.

Which Customers are Impacted?

clients hosting websites using this shared IP.

How are Customers Impacted?

Websites will be unavailable while IP is filtered.

How often will we be updated?

When the current filter expires.

Time to Resolution (ETA)

N/A

Incident Updates

  • 2013/05/19 03:59 AM EDT - The IP filter has been initiated for 4 hours

Resolution Description

The current filter has expired and traffic seems to be back to normal.

18 May 2013 4:25 pm

web1424 Shared IP: 50.118.40.2 – Filtered – Resolved

Posted In: Security Issues — May 18th, 2013 at 4:25 pm EST by IX: Angela

Incident Description:

We are experiencing irregular traffic on this shared IP. Our system administrators have initiated an IP filter. We will recheck incoming traffic when the filter expires.

Which Customers are Impacted?

Clients hosting websites using this shared IP.

How are Customers Impacted?

Websites will be unavailable while IP is filtered.

How often will we be updated?

When the current filter expires.

Time to Resolution (ETA)

n/a

Incident Updates

  • 2013/05/18 3:02 PM EST - The IP filter has been initiated for 80 minutes
  • 2013/05/18 11:30 PM EST - The IP filter has been initiated for 12 hours

Resolution Description

The current filter has expired and all traffic seems to be back to normal.

18 May 2013 4:22 pm

web1432 shared IP: 76.162.83.2 – Filtered – Resolved

Posted In: Outage — May 18th, 2013 at 4:22 pm EST by IX: Angela

Incident Description:

We are experiencing irregular traffic on this shared IP. Our system administrators have initiated an IP filter. We will recheck incoming traffic when the filter expires.

Which Customers are Impacted?

clients hosting websites using this shared IP.

How are Customers Impacted?

Websites will be unavailable while IP is filtered.

How often will we be updated?

When the current filter expires.

Time to Resolution (ETA)

n/a

Incident Updates

  • 2013/05/18 2:52 PM EDT - The IP filter has been initiated for 80 minutes
  • 2013/05/18 11:04 PM EDT - The IP filter has been initiated for 4 hours
  • 2013/05/19 03:59 AM EDT - The IP filter has been initiated for 12 hours

Resolution Description

The current filter has expired and traffic seems to be back to normal.

17 May 2013 11:53 pm

mail511 Down for maintenance – FSCK – Resolved

Posted In: Outage — May 17th, 2013 at 11:53 pm EST by IX: Omari J.

Incident Description:

Mail511 has gone down due to a forced file systems check (FSCK).

Which Customers are Impacted?

All customers on mail511

How are Customers Impacted?

Email will not be available

How often will we be updated?

3 hours

Time to Resolution (ETA)

2-3 hours

Incident Updates

  • 5/18/2013 1:28 AM EST - The FSCK is currently at 84% completed. The ETA is currently 1 hour.

Resolution Description

Server is back online.

16 May 2013 9:41 am

Data Center Five Year Maintenance – May 17, 2013 – Resolved

Posted In: Outage — May 16th, 2013 at 9:41 am EST by IX: Brian Silhanek

Incident Description:

Starting on Friday, May 17th, 2013 at 08:00PM EDT, we will be conducting maintenance on our data center power systems.  This will be the first of two maintenance periods, with the second one starting on May 24th at the same time.

The purpose of this maintenance is to replace the batteries in each of our uninterruptible power supplies (UPS) and is conducted every five years.  During the maintenance, a portion of our UPS’ will be put into bypass mode and their compliment of 80 batteries will be replaced.  During this time, all servers powered by the applicable UPS’ will be running directly on generator power to prevent downtime in the unexpected case of a commercial power failure.

The maintenance is scheduled to be completed within 8 hours.

There is no expected downtime or customer impact for either of these maintenance windows.

Which Customers are Impacted?

All customers

How are Customers Impacted?

n/a

How often will we be updated?

We will update when we start the maintenance, when new information is available and when the maintenance is finished.

Time to Resolution (ETA)

6-8 hours

Incident Updates

  • 5/17/2013 8:47 PM EST - The data center maintenance has started and another update will be posted when available.

Resolution Description

The maintenance has completed successfully and all servers are back off of the UPS.

16 May 2013 2:13 am

iis503 Maintenance – Resolved

Posted In: Maintenance — May 16th, 2013 at 2:13 am EST by Antonio S

Incident Description:

On iis503 RAID needs to be rebuilt. The server is online and websites will be accessible, but may experience slowness.

Which Customers are Impacted?

customers on iis503

How are Customers Impacted?

All websites/FTP/control panel will experience slowness.

How often will we be updated?

8 hours

Time to Resolution (ETA)

3 days

Incident Updates

  • 2013/05/16 4:26 AM EST - RAID rebuild progress 8%
  • 2013/05/16 9:32 AM EST - RAID rebuild progress 40%
  • 2013/05/16 10:02 PM EST - RAID rebuild progress 99%.
  • 2013/05/16 10:57 PM EST - Rebuild is completed. Server is working normally.

Resolution Description

Rebuild is completed. Server is working normally.

15 May 2013 4:10 pm

Data Center Maintenance – May 16, 2013 at 07:00AM EDT – Resolved

Posted In: Maintenance — May 15th, 2013 at 4:10 pm EST by IX: Brian Silhanek

Incident Description:

We will be performing system maintenance on Thursday, May 16th at 7:00AM EDT.  Maintenance is expected to last two hours.  During this maintenance period, we will be rerouting power to our servers and Storage Area Networks (SAN) to verify that the servers and SAN are properly connected to both of our backup power supplies.

There is no expected downtime or impact to customers during this maintenance window.  

Which Customers are Impacted?

All customers

How are Customers Impacted?

None

How often will we be updated?

We will update when we start the maintenance, when new information is available and when the maintenance is finished.

Time to Resolution (ETA)

2 hours

Incident Updates

  • 2013/05/16 7:26 AM EDT - Maintenance has begun.  ETR is still 2 hours and no interruption of services is expected.
  • 2013/05/16 8:28 AM EDT - Maintenance has been completed

Resolution Description

Maintenance has been completed

15 May 2013 12:16 am

web1134 shared IP: 71.18.174.2 – Filtered – Resolved

Posted In: Outage — May 15th, 2013 at 12:16 am EST by IX: Luna F.

Incident Description:

We are experiencing irregular traffic on this shared IP. Our system administrators have initiated an IP filter.  We will recheck incoming traffic when the filter expires.

Which Customers are Impacted?

All clients hosting websites using this shared IP.

How are Customers Impacted?

Websites will be unavailable while IP is filtered.

How often will we be updated?

Once the current filter has expired.

Time to Resolution (ETA)

Once the current filter has expired.

Incident Updates

  • 2013/05/14 11:52 PM EDT - The IP Filter has been initiated for 4 hours.
  • 2013/05/15 04:06 AM EDT - We are still experiencing irregular traffic on this IP. The IP Filter has been extended for an additional 12 hours.
  • 2013/05/15 04:10 PM EDT - Traffic to this IP appears normal. The IP Filter has expired.

Resolution Description

The IP Filter has expired.

14 May 2013 4:48 am

iis319 – Maintenance – Resloved

Posted In: Maintenance — May 14th, 2013 at 4:48 am EST by IX: Luna F.

Incident Description:

Our system administrators have detected an issue with this server. The server’s RAID is being rebuilt. During this time the web server will be slow.

Which Customers are Impacted?

All clients using this web server.

How are Customers Impacted?

All websites/FTP/control panel will experience slowness.

How often will we be updated?

8 hours

Time to Resolution (ETA)

24-48 hours

Incident Updates

  • 2013/05/14 04:43 AM EDT - Maintenance has been started.  ETA is 15 hours.
  • 2013/05/14 09:52 AM EDT - Server is currently is rebuilding and is 6% complete
  • 2013/05/14 1:13 PM EDT - The server rebuild is currently at 10% completed.
  • 2013/05/14 5:16 PM EDT - The server rebuild is currently at 16% completed.
  • 2013/05/14 8:59 PM EDT - The server rebuild is now at 21% complete.
  • 2013/05/15 2:52 AM EDT - The server rebuild is now 28% complete.
  • 2013/05/15 7:46 AM EDT - The server rebuild is now 35% complete.
  • 2013/05/15 11:00 AM EDT - The server rebuild is now 39% complete.
  • 2013/05/15 1:39 PM EDT - The server rebuild is now at 42% complete.
  • 2013/05/15 4:09 PM EDT - The server rebuild is now at 46% complete.
  • 2013/05/15 9:23 PM EDT - The server rebuild is now at 52% complete.
  • 2013/05/16 1:35 AM EDT - The server rebuild is now at 58% complete.
  • 2013/05/16 4:53 AM EDT - The server rebuild is now at 61% complete.
  • 2013/05/16 9:36 AM EDT - The server rebuild is now at 67% complete.
  • 2013/05/16 2:29 PM EDT - The server rebuild is now at 74% complete.
  • 2013/05/16 6:07 PM EDT - The server rebuild is now at 79% complete.
  • 2013/05/16 10:01 PM EDT - The server rebuild is now at 83% complete.
  • 2013/05/17 01:26 AM EDT - The server rebuild is now at 88% complete.
  • 2013/05/17 04:59 AM EDT - The server rebuild is now at 92% complete.
  • 2013/05/17 03:00 PM EDT - The server rebuild is now complete.

Resolution Description

Raid rebuild is finished, all sites are up and running now.

14 May 2013 3:37 am

iis406 – Maintenance – Resolved

Posted In: Maintenance — May 14th, 2013 at 3:37 am EST by IX: Luna F.

Incident Description:

Our system administrators have detected an issue with this server. RAID on this server is undergoing a mandatory test. The server will be offline during this time.  Websites hosted on the server will be unavailable.

Which Customers are Impacted?

All clients using this web server.

How are Customers Impacted?

All websites/FTP/control panel will not be reachable.

How often will we be updated?

8 hours

Time to Resolution (ETA)

24-48 hours

Incident Updates

  • 2013/05/14 03:21 AM EDT - Maintenance has been started.  ETA is 1 hour.
  • 2013/05/14 04:12 AM EDT - Server is online and is now going through a RAID rebuild.  This will cause slowness for websites, FTP, and control panel.   ETA is 10 hours.
  • 2013/05/14 09:52 AM EDT - Server is currently is rebuilding and is 17% complete
  • 2013/05/14 1:14 PM EDT - The server rebuild is currently at 22% completed.
  • 2013/05/14 5:20 PM EDT - The server rebuild is currently at 31% completed.
  • 2013/05/14 9:00 PM EDT - The server rebuild is now at 46% complete.
  • 2013/05/15 2:53 AM EDT - The server rebuild is now 65% complete.
  • 2013/05/15 7:46 AM EDT - The server rebuild is now 72% complete.
  • 2013/05/15 11:00 AM EDT - The server rebuild is now 78% complete.
  • 2013/05/15 1:40 PM EDT - The server rebuild is now at 81% complete.
  • 2013/05/15 4:10 PM EDT - The server rebuild is now at 85% complete.
  • 2013/05/15 9:24 PM EDT - The server rebuild is now at 93% complete.
  • 2013/05/16 4:58 AM EDT - The server is now verifying 3%
  • 2013/05/16 9:34 AM EDT - The server is now verifying 8%
  • 2013/05/16 2:38 PM EDT - The server is now verifying 16%
  • 2013/05/16 6:03 PM EDT - The server is now verifying 21%
  • 2013/05/16 10:01 PM EDT - The server is now verifying 30%.
  • 2013/05/17 01:27 AM EDT - The server is now verifying 37%.
  • 2013/05/17 04:59 AM EDT - The server is now verifying 40%.
  • 2013/05/17 05:16 PM EDT - The server is now verifying 53%
  • 2013/05/17 09:54 PM EDT - The server is now verifying 65%
  • 2013/05/18 06:43 AM EDT - The server is now verifying 78%
  • 2013/05/18 3:39 PM EDT - The server is now verifying 89%
  • 2013/05/18 11:20 PM EDT - The server is now verifying 89%

Resolution Description

Rebuilding has been completed successfully.

13 May 2013 6:07 am

Web255 – Hardware Issues – Resolved

Posted In: Outage — May 13th, 2013 at 6:07 am EST by IX: Steven A.

Incident Description:

web255 was experiencing hardware and is unable to boot. The boot partition and the operating system partition are both corrupted, but we do not expect any data loss. We have started the recreation process of this server

Which Customers are Impacted?

All customers on web255

How are Customers Impacted?

All websites/FTP/control panel will not be reachable.

How often will we be updated?

4 hours

Time to Resolution (ETA)

12 hours

Incident Updates

  • 2013/05/13 9:19AM EST:  Server being rebooted.  Original ETA is unchanged
  • 2013/05/13 10:39 AM EST:  The Server is currently unable to be rebooted and the the recreation process has restarted.  New ETA is 12 hours.  No data loss is expected at this time.
  • 2013/05/13 3:32 PM EST: New base OS has been installed, and the content is currently copying to the new drive. Due to drive corruption this process has to take place offline with Linux rescue disc. New ETA is 4 hours for the data transfer process.
  • 2013/05/13 9:09 PM EST - Content has been completely copied and is preparing for the next stage of full recreation. Current ETA is now about 12 hours.
  • 2013/05/14 02:21 AM EDT - Our system administrators are still working on recreation.  No ETA at this time.
  • 2013/05/14 07:46 AM EDT - Server is back online

Resolution Description

Server is back online

11 May 2013 11:52 am

web602 shared IP: 173.83.144.130 – Filtered- Resolved

Posted In: Outage — May 11th, 2013 at 11:52 am EST by IX: Robyn M.

Incident Description:

We are experiencing irregular traffic on this shared IP. Our system administrators have initiated an IP filter. We will recheck incoming traffic when the filter expires.

Which Customers are Impacted?

ll clients hosting websites using this shared IP.

How are Customers Impacted?

Websites will be unavailable while IP is filtered.

How often will we be updated?

When the current filter expires.

Time to Resolution (ETA)

n/a

Incident Updates

2013/05/11 11:52 AM EST - The IP Filter has been initiated for 1 hour.

Resolution Description

The filter has expired. Sites are back up.

11 May 2013 5:02 am

Mail907 – File system check- Resolved

Posted In: Maintenance — May 11th, 2013 at 5:02 am EST by Antonio S

Incident Description:

Our system administrators have detected an issue with this server. The server is being restarted and is now doing a mandatory filesystem check (FSCK). During this time the mail server will be unavailable, meaning that email hosted on the server will be unavailable as well.

Which Customers are Impacted?

Any customer on mail907

How are Customers Impacted?

Email will not be working.

How often will we be updated?

4 hours

Time to Resolution (ETA)

n/a

Incident Updates

  • 5/11/2013 4:43 AM EST -  The server is running a FSCK. ETA 3-4 hours.

Resolution Description

The FSCK has completed and the server is now back online.

11 May 2013 3:47 am

Mail505 maintenance – Resolved

Posted In: Maintenance — May 11th, 2013 at 3:47 am EST by Antonio S

Incident Description:

Our system administrators have detected an issue with this server. The server’s RAID is being rebuilt. During this time the mail server will be slow.

Which Customers are Impacted?

All customers on mail505 will be affected

How are Customers Impacted?

Email will be down.

How often will we be updated?

n/a

Time to Resolution (ETA)

n/a

Incident Updates

  • 5/11/2013 3:02 AM EST - We are performing a migration to new hardware. ETA 2 hours.
  • 5/11/2013 4:25 AM EST - The server is up and running with new hardware. We will start the final re-sync from the old server to new the server. ETA 3 hours.

Resolution Description

Mail505 is up and running on the new hardware.

 
© 2011 IX Web Hosting.