Golang works well for this application because it can easily cope with very large numbers of idle goroutines.
What the author may be missing is that golang also works well for bots and scanners, for exactly the same reason. Attackers' time isn't being "wasted" by this, their goroutines are just sitting idle for longer.
I think it still works. You have two scenarios where the attacker is efficiently using goroutines; (1) you also use goroutines or (2) you do not. In the latter, the attack is more expensive for you.
Another detail is that an attacker with many idle connections to your host might not instantiate any new ones.
Of course, in the scenarios where the attacker is not using goroutines then you have the upper hand as well.
This isn't a real ssh server, so the "cost" to you of the attack isn't really relevant. You can choose not to run this software at all, and the additional cost to you is zero.
Though for a large amount of HTTP bots, the authors don't even bother changing the default Python User-Agent. I'd assume a large proportion of these bots still can't run concurrently.
The original endlessh hints at this, but doesn't go further into details, and the endlessh-go's README doesn't mention it at all. Am I suppose to have endlessh run on port 22 and then have my real SSH server run on an obscure port? In none of the examples does it run on port 22. I feel like I'm missing something obvious, that the READMEs simply take for granted I know.
I run endlessh on the port 2222 and I configured fail2ban to redirect the source ip addresses who did X failed attempts from the dest port 22 to the dest port 2222 transparently.
I use the table NAT and prerouting to achieve that, you can use ipset to match the source ip addresses.
Isn't the point of a honeypot that it's not a real server? What guarantees are there that there won't be an exploit that allows escaping the honeypot into the real data? Personally, I do not believe anything is 100% secure. So inviting the vampire into your facade home, and then getting upset when the vampire sees the charade and walks into your real home is just one of those "well of course that happened" situations.
if you use port knocking, the first hit on your honeypot, can be the trigger to lockdown or redirect, a lot of other ports to somewhere away from your actual.
But they don't scan every port. I've been running my SSH server on a non-standard port for a long time, it took four years until I had the first bot with login attempts. About a year ago I changed the port and haven't seen any bots since then.
You can surround your custom port by a couple of ports on which a simple server listens for connection attempts. Any connection attempt is considered hostile and the ip will then be blacklisted in iptables. This prevents portscans from reaching your port.
Different quality of locks in the ever-escalating arms race. Probably there are many many more sequential scanners out there. For the persistent actors who are doing random ordering or shuffle then you could add port-knocking for the real sshd... but then they just have to find a working client and sniff the connection requests... to which you add a TOTP step for determining which ports to use, and so on...
Excuse the old school metaphor - you put a lock on your door so your house is harder to break into, not to prevent anyone from breaking into your house.
Absolutely agree, when I wrote this I was thinking more of defending against the low hanging fruit - mass scanners.
Once someone has deemed you a worthwhile target and is carefully proving all ports, these more nuanced approaches become more worthwhile. Even then, a sophisticated adversary may have many unique src IPs at their disposal.
The more targeted/sophisticated ones will, but there's a crapload of bots that just scan all publicly addressable IPs for port 22 and attempt to connect. If your goal is to trap as many bots as possible in the tarpit, you'll get a lot more if you run on port 22.
Following the SSH hardening guide stops 99% of bots and scanners because they can't negotiate a cipher using whatever ancient ones their SSH implementation is set up to use.
This, and a handful of simple firewall rules in the raw table can block about 90%+ of that remaining 1% just looking at the spoofable banner that none of the bots seem to spoof I assume due to being lazy like me.
In the raw table:
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "SSH-2.0-libssh" --algo bm --from 10 --to 60 -j DROP
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "SSH-2.0-Go" --algo bm --from 10 --to 60 -j DROP
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "SSH-2.0-JSCH" --algo bm --from 10 --to 60 -j DROP
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "SSH-2.0-Gany" --algo bm --from 10 --to 60 -j DROP
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "ZGrab" --algo bm --from 10 --to 60 -j DROP
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "MGLNDD" --algo bm --from 10 --to 60 -j DROP
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "amiko" --algo bm --from 10 --to 60 -j DROP
Adding the server IP minimizes risks of also blocking outbound connections as raw is stateless
I rarely do this any more given they rotate through so many LTE IP's. Instead I get the bot operators to block me by leaving SSH on port 22 and then giving them a really long VersionAdendum that seems to get the bots feeling broken, sticky and confused. There are far fewer SSH bot operators than it appears. They will still show up in the logs but that can be filtered out using drop patterns in rsyslog.
VersionAddendum " just put in a really long sentence in sshd_config that is at least 320 characters or more"
Try it out on a test box that you have console access to just in case your client is old enough to choke on it. Optionally use offensive words for the bots that log things to public websites. Only do this on your hobby nodes, not corporate owned nodes unless legal is cool with it, in writing.
I don't know if this is still the case, but -m string used to be resource intensive, because it has to parse each packet for the string before passing it on to other rules.
It can be. This this case however it is limited to eth0, tcp, port 22. If any of those don't match there will be no parsing and thus no impact. Another mitigating factor is that we are only looking at specific byte regions of the packet so parsing is minimized. On busy SFTP servers I would probably avoid using such rules if CPU load is becoming a problem. For most people this will not even register in htop or vmstat. There are also ways to use this string check in combination with ipset and/or xt_recent to minimize the times we see a packet from a bot. Here is an example using an IPSet called "bots" that we drop early on in the raw table and also use in the filter outbound rules to reset openssh trying to respond the first time we see the bad string so we close the socket earlier.
Anything I explicitly drop I do so in the raw table to keep them out of the state table. The state table is more CPU expensive especially at high packet rates and runs the risk of depleting the default state table limits especially for anything that now has a broken state on purpose like these poor lil bots. Since I brought it up, here is how to increase the state table limits.
# from /etc/sysctl.conf: increase state table limits.
# Requires 1/4 mem to hash table plus 400 overhead because I am the cargo culting king:
# cat /etc/modprobe.d/nf_conntrack.conf
# options nf_conntrack expect_hashsize=256400 hashsize=256400
net.nf_conntrack_max = 1024000
Should people use default state table memory allocations on a busy node, everyone can be locked out of it regardless of how many TB of RAM are free. The node can appear "down".
Endlessh periodically sends data so the read timeout won't trigger. Specifically, it draws out the crypto negotiation stage indefinitely by exploiting a feature of the SSH protocol.
(Of course, the bot author could detect that behaviour too.)
You're understanding perfectly. The way this works is that it sends a slow drip of junk before the SSH version banner string. A scanner running at any real scale is going to have an overall timeout beyond which it doesn't bother waiting any longer for the banner string.
This is going to very slightly irritate some of the extremely low-level actors. Is setting up a tool to do that a good use of time?
If you want to effectively deter attackers using a sand-trap approach, you need to find some kind of task with asymmetric cost in your favour. This isn't that.
You could probably achieve better here by providing fake weak credentials and then getting an actual human to connect and check out the honeypot on as many IPs as possible.
At the same time it's much easier to write code that just died the bare minimum. Imagine you're a bot herder, if your bot net consists of stolen CPU cycles what difference does it make if your bots are slowed down. It doesn't cost you money.
> if your bot net consists of stolen CPU cycles what difference does it make if your bots are slowed down. It doesn't cost you money.
This is wrong. It does cost you money - either directly, because you paid money to use someone else's botnet, or as an opportunity cost, in that you can't use your bots on as many targets.
Funny but my first thought wasn't wasting their time at all, that's easily fixed with a few code adjustments on their client end. My thought was to harvest their IPs and publish them in blocklists.
I think you could employ the same tactics that advanced fuzzers do with these tarpits: then mutate the responses randomly, to try get "new" responses from the attackers, instead of new coverage in the code as in the fuzzer. Unless they are using static scripts, which would be boring.
I have understood that most attacks are super-simple sort of, so probably not much to learn there. But an interesting project!
Depends on how well programed the bot is I guess. Personaly I use the encrypted packet port knocking package fwknop on my home server to hide the ssh port until I need it.
The point of this isn't to hide your actual SSH service, but to tie up resources for those who are somewhat blindly scanning/connecting to any open SSH port.
> Unfortunately the wonderful original C implementation of endlessh only provides text based log, but I do not like the solution that writes extra scripts to parse the log outputs, then exports the results to a dashboard, because it would introduce extra layers in my current setup and it would depend on the format of the text log file rather than some structured data. Thus I create this golang implementation of endlessh to export Prometheus metrics and a Grafana dashboard to visualize them.
" I didn't like the logging, so I re-implemented the entire thing."
I'm not mocking, I just see this often (and have done it myself!). It's interesting the things we do to get around the little things we don't like.
The thing about coders is that they're often creators at heart.
The stated reason is likely only the excuse they told themselves to justify the project. But the real reason was likely that they wanted to create something, and this was a good justification.
Might just be me projecting though, because I do that all the time
I've stopped lying to myself, I'm making cool stuff just for the sake of it. It's like playing a video game, it doesn't have to be productive if I enjoy it.
People build in languages they are comfortable with. Golang is fairly easy to ship with and the performance is fine for something like this. Seems like a good tool for the job
I was more pointing out the fact that people reinvent existing stuff in a new language, then a few years go by, the 'existing stuff' is still 'existing', the 'new language reinvention' is dead, and someone new then reinvents the 'existing stuff' in some new language :)
What the author may be missing is that golang also works well for bots and scanners, for exactly the same reason. Attackers' time isn't being "wasted" by this, their goroutines are just sitting idle for longer.
Another detail is that an attacker with many idle connections to your host might not instantiate any new ones.
Of course, in the scenarios where the attacker is not using goroutines then you have the upper hand as well.
In over 10 years I've never had a single probe on that port with ssh.
Once someone has deemed you a worthwhile target and is carefully proving all ports, these more nuanced approaches become more worthwhile. Even then, a sophisticated adversary may have many unique src IPs at their disposal.
In the raw table:
Adding the server IP minimizes risks of also blocking outbound connections as raw is statelessI rarely do this any more given they rotate through so many LTE IP's. Instead I get the bot operators to block me by leaving SSH on port 22 and then giving them a really long VersionAdendum that seems to get the bots feeling broken, sticky and confused. There are far fewer SSH bot operators than it appears. They will still show up in the logs but that can be filtered out using drop patterns in rsyslog.
Try it out on a test box that you have console access to just in case your client is old enough to choke on it. Optionally use offensive words for the bots that log things to public websites. Only do this on your hobby nodes, not corporate owned nodes unless legal is cool with it, in writing.In a startup / init script / systemd unit file:
In this example I am using a bigger netmask much in the way name servers rrl rate limit.In the raw table, drop bots we saw for a week:
In the filter table outbound rules: This should only be performed on servers that one has console / out-of-band access to, after exhaustive testing.Create /etc/modprobe.d/nf_conntrack.conf
And then in /etc/sysctl.conf: Should people use default state table memory allocations on a busy node, everyone can be locked out of it regardless of how many TB of RAM are free. The node can appear "down".Or am I misunderstanding this?
(Of course, the bot author could detect that behaviour too.)
There's more info from the author of Endlessh: https://nullprogram.com/blog/2019/03/22/
This is going to very slightly irritate some of the extremely low-level actors. Is setting up a tool to do that a good use of time?
If you want to effectively deter attackers using a sand-trap approach, you need to find some kind of task with asymmetric cost in your favour. This isn't that.
At the same time it's much easier to write code that just died the bare minimum. Imagine you're a bot herder, if your bot net consists of stolen CPU cycles what difference does it make if your bots are slowed down. It doesn't cost you money.
This is wrong. It does cost you money - either directly, because you paid money to use someone else's botnet, or as an opportunity cost, in that you can't use your bots on as many targets.
Please do, it would mean good karma.
I have understood that most attacks are super-simple sort of, so probably not much to learn there. But an interesting project!
" I didn't like the logging, so I re-implemented the entire thing."
I'm not mocking, I just see this often (and have done it myself!). It's interesting the things we do to get around the little things we don't like.
The stated reason is likely only the excuse they told themselves to justify the project. But the real reason was likely that they wanted to create something, and this was a good justification.
Might just be me projecting though, because I do that all the time
And did that in the "language of the week" :)
The stuff that was reinvented in eg. ruby a few years ago is now reinvented in go and rust.
eg: https://news.ycombinator.com/item?id=19276751
https://github.com/remacs/remacs
last commit, 3 years ago.
Yeah and perhaps pick up valuable skills, that might help us down the road in ways that are hard to quantify.