|
| |
 |
|
6/16/08: Problem with ATT blocking resolved.
Recent Event 6/15/08: ATT, SBCGLOBAL, BELLSOUTH has begun blocking all mail being forwarded to them via @yourdomain.com. This is across multiple servers, multiple data centers. Even our billing system located with a completely different provider is having their mail blocked by att.net/blocked RBL. This is affecting a lot of people. Please be very aware, if you forward your email to your ISP, you have "zero" guarantee that your email will ever get to your inbox due to blocks by your ISP. We are currently working with ATT to get the blocks removed. As a work around, you should either forward your mail to a non blocking account, ie gmail, or get your email directly from the server. If you need help doing so, please submit a ticket.
Recent event: 5/31 @ 6:35pm EST - 6/2 @ 2:45am EST Affected one dedicated server.
Affected Server up at 2:45am Mon EST 6/2
Direct updates from Data Center:
8:36pm EST Sat
Today at approximately 6:45 p.m.EST, a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down. We are working with the fire department, with our facilities staff on site, to assess the situation.
8:51pm EST Sat
We have determined that no servers in the data center have been damaged. Nonetheless, they are down because power is out. Teams across the board are working to take appropriate action.
9:17pm EST Sat
In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center.
11:17pm EST Sat
We have talked to the DC at this time, the ETA to be back online is tomorrow morning. If this changes, we will keep you updated. There is really nothing we can do but wait, sorry folks, I know this is frustrating for some. Thanks for your patience everyone.
12:39am EST Sun
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
7:54am EST Sun
We are continuing to work through various issues this morning. We will have additional contractors on-site this morning starting at approx. 7 AM. Some will hand-off from contractors who worked overnight and others will start the recovery/installation of new electrical gear to power the data center. In addition to the above, the network engineering group worked overnight to prepare the network for the recovery of H1. We expect the reconvergence of the network to go smooth once H1 comes back online. We do not have an Estimated Time to Repair at present; we should have a better estimate this morning.
8:47pm EST Sun
We continue to work to restore power to the data center and bring all affected customer servers online.
Note from Stormer: 11:33pm EST Sun
At this point, I do not have an ETA. The "rumor" is by tomorrow morning. I am becoming less confident of this major situation and am willing to setup new accounts for all affected clients to at least bring up email and some sort of splash page for client websites. If you would like to go this course, please submit a ticket and it will be done for you. I am hopeful it will be brought back online by tomorrow morning, but have zero garuntee this will happen. Aproximately 9000 servers are currently down at this data center. Of which only 1 is being used by Stormer Hosting. No doubt the datacenter is going to feel the "pinch" of time as Monday morning rolls around.
1:55am EST Mon
After the fire marshall inspected the H1 location, we were given the green light to bring power back to the facility. The generators have been turned on, and we are receiving power on the second floor. The generator power restoration is the first step in the full restoration of service to the data center. From here, we will begin the process of cooling the DC floor, which could take a few hours. As soon as the power integrity is confirmed and the DC floor is ready for operation, we will be restoring power and checking server hardware on a rack-by-rack basis.
Note from Stormer: 2:45am EST Mon
Server is back online and responding to request. Evidently, this server is on the second floor and must be next to the power swtich!
2:55am EST Mon
Following the restoration of power to the second floor of the data center, we've cooled the data center floor and are now in the process of systematically restoring power to racks. We've got a full staff in the data center to power up racks in sections and verify that the server hardware starts up successfully. This process may take a few hours to restore service to all customer servers on the second floor.
|
|
|
