02-04-2003, 03:09 AM
|
|
Administrator
|
|
Join Date: Jan 2003
Posts: 1,350
|
|
Ok, here is the official word from the hosting company. For those of you wondering, we are an East Coast server, so remember that while reading the email.
Quote:
1) First it was spamming
2) then last night a Dos Attack
My Weekend in a nutshell:::::::
East Coast Noc was upgrading the switch for our server(s) (and many many others). In advance of this certain preparatory work needed to be finished to ensure a smooth switch and the promised 30 minutes of downtime. This preparatory work was not done correctly and when the new switch was brought online it corrupted everything in its path and the tables had to be totally rebuilt....and so we were down for 3-4 hours.
Saturday AM, In the location of the NOC there was a massive power failure, lasting from roughly 7:00am till noon. The NOC switched over to generators but the connection.line to the area of the NOC we are on failed. In doing so, it also cause the servers to hard crash. a number of servers had memory failures and a small amount had drives go bad. To restart the machines, all the machines had to be manually rebooted with root passwords. It was then the hardware issues were also found....I know I said this earlier, but these servers do not have keyboards and monitors hooked up to them normally so this was a massive undertaking with hundred and hundreds of boxes.
Saturday night, again there was a small downtime as the NOC finally switched over fully to the GiGe switch and we seemed to be doing OK.
Sunday again a small outage (don't know why) and Sunday night a 20 minute or so DOS Attack.
Monday at noon, we experienced a temporary network outage, today at 12PM EST lasting approximately 20 minutes. The recent upgrade to a new GIGE switch, after rigorously testing the connection for 24 hours, continuously stable pushing 300MBPS, dropped the connection. As a temporary solution, they switched back to a failover connection of a 100MBPS FE switch, which prior to the upgrade was nearing max capacity. They are currently replacing the Fiber running from the core router to our main router, as we believe this to be the cause of the problem. That was completed and we were switched over at 5:08pm
Once the stability of the GiGe switch is again assured, the 100MBPS FE switch is being upgraded to a 200MBPS switch so if the need arises to switch back there will be adequate capacity.
OK, now all of this has been a bit of a nightmare, but for those out there that have servers on the west coast too, they can tell you about 5-6 hours of downtime due to that little internet worm that damn near crashed the entire internet 2 weekend ago. On the East Coast, they had packet loss for all of 20 minutes. We all have our days in the sun folks and some days..well we don't
The combination of events would be almost amusing if they didn't effect all of us and I am the last to make light of it. I have done my best to keep you informed but quite honestly when we are getting hundreds of e-mails and IM's it is impossible to give each of you individual attention on these matters. We try, but this one just overwhelmed us.
So as of now we are back on the GiGe switch and all seems well...Let just hope for a couple of days and months of quite stability.
|
|
Linear Mode
