Publieke Forums/Skylords Forum

Bericht Titel: Server Downtime ()

Poster: System Administrator

Bericht gestart:

Berichten: 10 Laatste bericht: door EL

System Administrator Gepost:

Berichten: 48

Berichten: 50

Locatie: United States



Has passed. I'm exhausted. Been sleeping far too little these past few days - a rundown of what happened therefore.

I have SkyLords set to replicate to my desktop computer, just incase anything should fail on the hard disk, I have backups to the very moment it failed, if I am booted into Linux. If I am booted into Windows mode, those updates are saved. I have to reboot weekly into Windows mode on my desktop computer however, what was the weekly downtime during the (fortunately briefish) time SL was hosted from my desktop - nobody was very fond of that solution, and it's permanent home is much nicer anyway, without pesky things like graphics cards and speakers, monitors and mice.

Well, as is amazingly common, we were under an attack from the internet on Thursday, some dumb cracker trying to guess our passwords, 10 per second or the like. Relatively boring stuff - but it was filling up our log files with failed login attempts. What ended up happening was a combination of factors which should never occur again. On Thursday morning, the disk filled up. This caused the white screen that many of you saw for a few days while I tried to free up disk space.

However, there was a lot of things, cron jobs mostly, waiting to happen. These sucked up disk space as fast as I could reclaim it, until finally I had most of the jobs pushed through - however, there were three jobs, written during my absence, I'll leave it to you to imagine how I feel about the author, that didn't fail gracefully - they continued to suck up disk space in an infinite loop. I found and fixed these final points on Friday night(?).

I wasn't content however to just leave it at that - I wanted to ensure that this triple disk hogging couldn't ever use up all of the 7 gb allocated... Far, far, too little, but I didn't think about that when I set it up back when. So, I tried to resize the disk. Again and again, I tried various tools. Finally giving up, I opened the case, pulled the disk, and resized it from my desktop machine :P.

When I put it back in however... It didn't boot. This was Saturday night around 3 am, and I needed to hit church the next day (was excellent however). Since I decided to let it rest for a day, I got back to it this evening after work (I was late to work already this morning, not able to work on the server). Now, past midnight, with less than 24 hours till the end of the round, I have fixed the booting problem (it was a strange kernel update that broke compatibility with SATA for some motherboards - I was unlucky to have the update from the brief time it wasn't fixed).

SkyLords now is running off of a ... much larger partition on a borrowed disk, and should be fine there for the foreseeable future. Just a note - before the downtime, the SkyLords server was running fine for almost a full 70 days without a single interruption. No reboots, hiccups, or needing to restart apache. SkyLords is running on some pretty awesome hardware, with a very wide Internet connection, and I dare you to NOT try to disrupt it please thank you very much :]. We are safe from any more accidental DOS attacks now however :P..

Thank you for your patience.

~Iiridayn


Omnibus Gepost:

Berichten: 30

Berichten: 10

Geslacht: Man



Thanks Iiri!!!


TallDude3rd Gepost:

Berichten: 1850

Berichten: 37

Locatie: Netherlands

Geslacht: Man



Bruteforce attack on our server? That is just stupid




__________________________
UNOFFICIAL BIG KAHUNA OF THE UNIVERSE

EL Gepost:

Berichten: 1705

Berichten: 182

Locatie: United Kingdom

Geslacht: Man



So glad you got it fixed Iiri thank you :)

Yes it it pretty stupid attacking the server but at least we are now prepared for any future attacks


Tom Bomadial Gepost:

Berichten: 423

Berichten: 76

Locatie: United States

Geslacht: Man



Thank you.
Not only for fixing the system.
But also for going to church.




__________________________
"Is not easily provoked"
Translated:
It is not good to wake a sleeping bear

TallDude3rd Gepost:

Berichten: 1850

Berichten: 37

Locatie: Netherlands

Geslacht: Man



He is just going to church to program a text-speech so the preacher-man can take a break when reading the bible... lol




__________________________
UNOFFICIAL BIG KAHUNA OF THE UNIVERSE

Zerohours Gepost:

Berichten: 2443

Berichten: 287

Locatie: Canada

Geslacht: Man



TY Iiri




__________________________
I am here to help in anyway I can! Let's have fun!

Pray Gepost:

Berichten: 2255

Berichten: 147

Locatie: United States

Geslacht: Man



LOL I-Flight for sermons. Extreme & Deep.




__________________________
A storm is coming, a storm is coming in..

20:48:19 | Professor planet 0 from paladin
20:20:21 | Arcanix planet 14 Arcanix 01

Diplomacy & Trading Forum Moderator.



me naam is m Gepost:

Berichten: 985

Berichten: 37

Locatie: Netherlands

Geslacht: Man



*lol @ TD+Pray*

Anyway, I am wondering what or who was trying to ddos?




__________________________
Where has the rum gone!?

FORTRANshadow Gepost:

Berichten: 383

Berichten: 17

Locatie: United States

Geslacht: Man



Thanks, Iiri! I know how hard it is to keep things going in the big bad internet. Please know that we appreciate your efforts.

You may want to look at installing denyhosts on your server. Although it is normally configured just for ssh, I think the scripts can be set to review all login attempts in your auth.log.


EL Gepost:

Berichten: 1705

Berichten: 182

Locatie: United Kingdom

Geslacht: Man



It would work.


Vorige 1  Volgende

| | |
Publieke forums
TICKETS
Taal Forums
Skylords Chat