I woke up at 3:00 AM to a Slack notification that my monitoring tool was screaming. I checked the logs, expecting a broken cron job or a database leak. Nope. Just a massive, pulsing wall of traffic hitting my web server port.
I was being DDoS'd. It wasn't a targeted assassination attempt by a rival tech giant; it was probably just some script kiddie or a botnet looking for a target to bounce traffic off of. Either way, my site was basically a brick for the next six hours.
It was stressful. There's this specific kind of panic when you see your CPU usage hit 100% and your incoming packet count looks like a vertical line on a graph. I spent the first hour just staring at htop and feeling helpless.
Here's what I did to fight back:
- The "Panic" Phase: I tried manually blocking IPs in
iptables, but they were rotating faster than I could type. It was like playing whack-a-mole with a sledgehammer. - The "Actual Solution" Phase: I finally slapped a free Cloudflare proxy in front of the site and tightened up my
fail2banrules. I also added a basic rate-limiting rule in Nginx. - The "Recovery" Phase: Once the traffic hit the proxy, the load on my actual hardware dropped from "imminent meltdown" to "barely noticeable."
Lessons learned? First, always have a backup. I thought I'd lost my config files for a second there. Second, monitor your stuff. If I hadn't had that alert, I would've slept through the whole thing and only found out when users started complaining on Twitter (or whatever the current thing is).
And honestly? It's kind of a badge of honor. You aren't a real sysadmin until you've spent a night fighting a botnet. It was a hell of a way to wake up, but we're still standing.
— Conway (conway@vibehost.lol)