Information

[Mark]

Information

At approximately 8PM EST, the exchange for our server into the datacenter dropped unexplicably. For approximately 2 hours and 18 minutes, the server was unavailable to the outside world. When the link came back up, I found the server restarted (also unexplicably), and promptly logged in and started the shard. After this, everything seemed to be okay, but after 30 or so connections or so, 'Fire Slayer', EV1s anti-DDoS mechanism, began throttling connections and causing problems with connections to the shard. This has (and continues) to cause problems on and off, allowing very few people to connect for long periods of time.

Throughout this course of events, I've had trouble tickets open with EV1 and I am working dilligently to have them resolve this as soon as possible. There will be no data loss, we have nightly backups, and although many of you cannot connect, the shard is still running and saving correctly. The only issue here is connectivity and that is something which is mostly beyond my control. We have contingency plans if this problem persists.

Thank you for your patience. This thread will remained locked and will be updated as I receive information regarding our situation.

 

[Mark]

http://bandwidth.ev1servers.net/RSbw8.cgi?cfg=15_20.cfg&log=67.15.20.247_111

The above traffic reports clearly show the 2 hour 'black out' with total loss of connectivity, followed by heavy throttling. Still waiting on resolution.

Hang in there.

 

[Mark]

Still no resolution from our current ticket in their support system. I am working with everyone I possibly can to get this fixed. As normal business hours approach, I will have more to work with.

Thank you for your patience.

 

[Mark]

Currently working with the CTO of EV1 directly to have our issue resolved. He is investigating the situation as I write this.

 

[Mark]

Connectivity has been restored. We are now optimizing and working to ensure this does not happen again.

 

[Mark]

All connectivity issues have been resolved. Once again, thank you all for being especially patient with us during these last few hours. Suprisingly, I received many kindly worded private messages of encouragement throughout the night. Maybe you guys arent so bad after all.

As always, a big thank you to Randy Williams (CTO of EV1) for his fast responses and dedication to providing excellent service to us.

And with that, I'm going to sleep. Enjoy.

 

[Ryan]

I personally wanted to take the time to thank Mark for everything he did. I was not available to work with Randy last night and Mark took the ball and ran with it. This is the kind of dedicated staff members that run this shard.