I have just recovered from a DDOS/Brute force attack on my server from a botnet running on Amazon EC2 instances. (Technical)

[Gromett]

Before I start this is a technical post. I am posting so that the few who may be interested can read it, but I doubt it will be many.

It has been many, many years since one of my own servers has been attacked in a serious way. I am too secure and too small for it to be worth doing.
Most of the attacks and hacks I deal with are for clients and I cannot discuss those due to confidentiality reasons. This one was against me! so I am free to talk about it

Last weekend starting on Friday I got a monitoring system warning that something was amiss on my server. Memory was being used up at a bit of a rate and CPU loading was increasing.

What started off as a minor attack from multiple servers around the world which I quickly got under control became what appeared to be a revenge attack using Amazons huge processing power.
This attack peaked with over 8,000 of Amazon's EC2 servers hitting my little dedicated server and 15K attempts per minute. I appear to have pi%$ed off the attacker and he brought his big guns to the game.

You can read the full story here of what happened and how I dealt with it.

Attacked by an Amazon EC2 hosted botnet. – Gray.me.uk – Tech and Linux Blog

 

[Badknee]

Glad you got it sorted Gromett

 

[Jimbohorlicks]

I read most of your story but don't understand a lot of it.
Why would someone want to attack or hack your server ,what purpose would it make for the attacker? Why would you have pi££ed them off?

 

[phillipsheila]

I would have turned it off and back on again

 

[Richard n Ann]

I read most of your story but don't understand a lot of it.
Why would someone want to attack or hack your server ,what purpose would it make for the attacker? Why would you have pi££ed them off?

That's what I thought.. Denial of service just bombards the server and slows it right down as far as I know. Perhaps just mischievousness?

^{_{Subscribers  do not see these advertisements}}

 

[Googlebot]

Very interesting reading, cyber warfare is well and truly here.

States are spending lots of money on it as well.

 

[olley]

I reckon the attacker is a petrol head.

 

[Gromett]

I read most of your story but don't understand a lot of it.
Why would someone want to attack or hack your server ,what purpose would it make for the attacker? Why would you have pi££ed them off?

It was a Brute force attack on one of the sites I was hosting. They were hitting the login page with POST containing their username/password guesses.
Each login attempt runs a program which accesses the database to check the credentials. Each access takes a little bit of memory, cpu time and db access resources.
If you have enough of them running concurrently you end up running out of resources and the server slows down.

Initially they just wanted to get access to my clients database of clients. Probably targetting email addresses for spamming. email/passwords of customers of my clients to see if they use the same ones on other services. plus they may have been hoping for Card data as it is an ecommerce store.
However. The master password is 12+ characters and random ones at that which is never re-used. The db is encrypted, and there is no card data stored or even passing through the server. They wouldn't succeed statistically in 1,000 years. But that doesn't stop them trying.

I pi££ed them off by successfully blocking them after they had been working for a week to build up their attempt rates. They then escalated. I don't know if they bought their ec2 instances off other hackers, paid other hackers to try for them, or had hacked ec2 instances in reserve.
But once they got to 15K attempts the server had slowed to a crawl and was barely usable. That was the DDOS component. They were using a distributed network of computers (AWS EC2 instances) to Deny Access to my server by overloading it. Like trying to fit a queue of 15,000 people through a doorway designed for 100 people at a time.

 

[phillipsheila]

I love this program

 

[Gromett]

Very interesting reading, cyber warfare is well and truly here.

States are spending lots of money on it as well.

I have been dealing with attacks like this since 1997 Nothing new here. Just the scale that has increased and the methods. Defences have improved also.
You can think of it like the history of war. people started with daggers, added shield, moved to spears, moved to armour etc etc etc. it is a leapfrogging situation.

^{_{Subscribers  do not see these advertisements}}

 

[mitzimad]

never sell you old computer to dodgy london geezers

 

[kevenh]

never sell you old computer to dodgy london geezers

If you sell it, format & clean the hard drive/ssd or just remove it!

Help on what EC2 instances are: these are cloud/virtual computers running in AWS datacentres. You can scale up (& down) the number of computers pretty rapidly. But running each instance has it's own cost. Typically the scaling is done on a schedule when you can anticipate demand, or by rules that track metrics like CPU load, etc.
You could have a scale up rule for when CPU load is over 80% for a minute. But, remember to have a scale down rule to turn off the extra computes when the load is stable below 60% (for e.g.)
Spot instances are the cheapest - when Amazon let users pick up idle EC2 instances on the cheap. But those can be pulled back as needed by AWS' full price customers.
Still, even running hundreds of virtual PC at the cheapest rate seems an extravagant attack {cost wise}.
edit: clarity

 

[Gromett]

If you sell it, format & clean the hard drive/ssd or just remove it!

To clarify, he was having an in joke with me. I sold him one of my old computers and I guess he was trying to infer something humorously
To be clear, I sold it without a hard disk, he sent me two hard disks which I installed windows and linux on for him before he collected.

Help on what EC2 instances are: these are cloud/virtual computers running in AWS datacentres. You can scale up (& down) the number of computers pretty rapidly. But running each instance has it's own cost. Typically the scaling is done on a schedule when you can anticipate demand, or by rules that track metrics like CPU load, etc.
You could have a scale up rule for when CPU load is over 80% for a minute. But, remember to have a scale down rule to turn off the extra computes when the load is stable below 60% (for e.g.)
Spot instances are the cheapest - when Amazon let users pick up idle EC2 instances on the cheap. But those can be pulled back as needed by AWS' full price customers.
Still, even running hundreds of virtual PC at the cheapest rate seems an extravagant attack {cost wise}.
edit: clarity

extravagant attack if he was paying for the instances. My guess is if he bought them himself he paid with stolen card details. But looking at the blacklists for some of the IPs they are all blacklisted so my guess is these are other peoples EC2 instances that have been exploited/hacked.
I may have been the first person to put this together and provided Amazon with a full list of EC2 instances operated by this hacker. It appears Amazon is doing something as they are starting to go down now. There are only 238 currently attacking me. I have been checking to see which ones are still accessible and pretty much all of them I have tried so far have not responded to pings or connections on common ports 25,80,443... I think Amazon "may" be doing something now? But I can't be certain just yet.

 

[SpeedyDux]

Good work Gromett!

 

[Quackers]

Good fight Gromett, I have turned a few routers off in the past the twats then move on to the next target and stop throwing bots at you.
What ever happened to all the bank robbers, how times have changed.

^{_{Subscribers  do not see these advertisements}}

 

[Gromett]

Good fight Gromett, I have turned a few routers off in the past the twats then move on to the next target and stop throwing bots at you.
What ever happened to all the bank robbers, how times have changed.

They don't move on. I have firewalled them with -j DROP. This simply drops the connection as though my server is not switched on.
However, I can still see the connections via the firewall rules.

 

[Gromett]

There are only 238 currently attacking me

Oops. I misread the stat. There are still 1,000's hitting me but I am no longer blocking by individual ip addresses but by IP ranges. So one range for example may have 255 ip addresses within it, of which 3 server may be attacking me. But I only count 1 for each range.

That said my load average is extremely low, so I don't really care. The hacker is wasting their time and money (lol on the money)...

load average: 0.21, 0.15, 0.14

 

[L' Hobo]

I have been dealing with attacks like this since 1997 Nothing new here. Just the scale that has increased and the methods. Defences have improved also.
You can think of it like the history of war. people started with daggers, added shield, moved to spears, moved to armour etc etc etc. it is a leapfrogging situation.

I am now a retired but I still use my old company email address on AOL, all of last night my internet, which is normally very secure and reliable, went from very slow to non-existent, could there be any connection?

It now appear to be back to normal.

 

[Quackers]

Can be entertaining watching and defeating them Gromett but it is always a worry, what next. Your ISP can be your friend as long as you are not with one of the domestic type.

 

[cmcardle75]

You'd think Amazon would get on top of EC2 instances being used for simple DDOS attacks against auth servers. It must be pretty obvious at the network layer, especially if they keep metrics and show sudden increases in activity, either generally or directed in a single direction. What's the point in expensively recruiting loads of pensioner's PCs by scam phone calls if you can just get Amazon to do the hard work for you?

^{_{Subscribers  do not see these advertisements}}

 

[Gromett]

I am now a retired but I still use my old company email address on AOL, all of last night my internet, which is normally very secure and reliable, went from very slow to non-existent, could there be any connection?

It now appear to be back to normal.

No connection sorry. The internet is MASSIVE the chances of one event being related to another especially at this small scale is non existent...

 

[Gromett]

Can be entertaining watching and defeating them Gromett but it is always a worry, what next. Your ISP can be your friend as long as you are not with one of the domestic type.

I enjoy fighting them when I don't have clients on my back. Some people find chess a challenge, or jigsaw puzzle or sudoku etc. For me it is either programming or fixing issues with a server.
With hackers the problems tend to be more dynamic, and occasionally I do get into fights with them. In one instance they were trying to spam 100's of different forms across a clients network of servers.
I wrote a spam filtering script. The spammer was pretty dumb initially in that they were using 8 random letters for the 1st name and the last name. After looking at the entire list of names that had been used across the entire history of the company it was apparent that no one had submitted a request with 2 names that had 8 letters, so just checked for that and blocked his ip at the firewall to prevent further submission. He then went away and came back with a random number of random letters for the 1st and 2nd name. These were all lower case though, so I simply prevented people submitting names in all lower case, if they came back and corrected the lower case letter at the beginning I let them through. If the same IP came back with a different string of random letters they got blocked. He figured that one out and started adding capitals to the beginning of first name and last name. So I downloaded a list of first names from the internet, checked them against the historical data and found they were all present in the dictionary. I started checking all names against this dictionary. He started using a dictionary to generate the names.
So now we were getting proper names in each submission so I went onto the emails. I checked if the email address had been used for a previous name as the 1st check as he regularly re-used email addresses. As this was a canadian company that only served Canadian businesses we dropped all emails with country code domain names except the canadian one. Anyway you get the idea. This went on for months with me beating him each time and blocking each of his IP addresses he used. I think he was getting short on spare IPs as he started swearing at me in the messages and telling me I was destroying his business. The guy was a moron, When I detected one of his attempts after this I started automatically returning errors with whacky and rude (not nasty) error messages before blocking that IP at the firewall. He was seriously pi££ed by the end. And I now have a really good anti spam system for forms which makes me a few $ now and again

 

[Gromett]

You'd think Amazon would get on top of EC2 instances being used for simple DDOS attacks against auth servers.

You would think they would get on top of ones like this because they got a fully detailed report of what, when and how's of the attack. But it wasn't against an auth server it was against a web application on port 443.

It must be pretty obvious at the network layer, especially if they keep metrics and show sudden increases in activity, either generally or directed in a single direction. What's the point in expensively recruiting loads of pensioner's PCs by scam phone calls if you can just get Amazon to do the hard work for you?

Not really. They won't have just been attacking me, and keeping all the connection data from a server is just not practical.

 

[Gromett]

Back up to 35,000 attempts per minute. Glad I have firewalls the entire EC2 network.

Tomorrow, I have a new plan. I figured out how to log/drop on subnets and have a script to add additional data.

This will quickly generate a report from the log file which I can regularly email to Amazon until they fix it. In the meantime it has no effect on my server.
As per their request, as new IP's are added they want a fresh report. So I will start sending them, as often and as regularly as I can to be helpful.

1 Minute of attacks. Already up to 7.7M hits since I last reset a few hours ago.
$ /root/time-firewall.sh
Tue 25 Jan 06:24:30 GMT 2022
7707K
Tue 25 Jan 06:25:30 GMT 2022
7742K

 

[Silver-Fox]

Do you think it’s Neo or Mr Smith that’s behind the attacks

^{_{Subscribers  do not see these advertisements}}

 

[Quackers]

You have some patience Gromett. You need to give them a prize. Leave them an easy access server with a payload, I sure some one like Sophos would oblige if they knew what you have achieved so far.

 

[Jamesh]

You have some patience Gromett. You need to give them a prize. Leave them an easy access server with a payload, I sure some one like Sophos would oblige if they knew what you have achieved so far.

Would that be 3.5t or above? Not sure which licence Grommet has..!

 

[Millcourt]

Fascinating. Thanks for posting this Gromett . I could just about follow the concepts to which you refer. It is a long time since I worked in IT (remember COBOL, punched cards, Mag Tape Drives……!!!).

There must be not insignificant costs for the perpetrators of this mayhem. Do the the financial rewards make this worthwhile rather than just the “amusement” of trying to cause chaos? To what extent do you reckon this activity is “State Sponsored”? I came to the view some time ago that we are already in “WW 3“ with so much aggressive behaviour seemingly intended to undermine our democratic institutions and destroy our trust in political leaders of all political views…….. or, is my thinking just fantasy?

 

[Carpmart]

Why not just subscribe to a cloud edge traffic protection services? Stops all DOS attacks and cleans traffic, so improving the service time for real users. They have the scale for any size of attack.

Also DDOS attacks are ever increasingly being used for very specifically targeted distraction purposes, rather than just disruption purposes. The real breach ‘stuff’ going on in the background whilst the distraction is going on and the attacked party focusing resources and time because of the business impact.

Finally, it’s always about not being the slowest wilderbeast in the herd, which creates protection in itself as the ‘bad’ people are not going to try very hard if the return is poor.

 

[kevenh]

Just FYI:
330 open AWS tickets for the period Friday to now with "DDoS" as a keyword {for cases CREATED in that time window}
And I'm not sure how deep I can access AWS IT support tickets

edit: clarity & so ~142 other cases were resolved in that period

^{_{Subscribers  do not see these advertisements}}