This is not unique to Cloudflare, Level3 or any of the other networks I've seen mentioned in this thread. The entire internet scans all the common ports and this should be expected to continue indefinitely. If this is causing alerts from your logging, there are a few options:
- Implement a firewall on your instances.
- Implement port knocking for sshd.
- Move sshd to a non standard port to avoid the nmap/bot noise.
- Only log successful logins.
- Any combination of the above.
There are pros and cons to each item so you would have to decide which combination is the most appropriate and least friction for your organization. If leaving the port exposed consider enforcing key based authentication and disabling password authentication. For high risk accounts such as monitoring accounts that use passwordless unrestricted sudo, you can even restrict what networks the keys are valid from.
I created gofwd, a cross-platform TCP port forwarder with Duo 2FA and Geographic IP integration. Its use case is to help protect services when using a VPN is not possible. While it's use cases are limited, it works great for me when I need to connect to my home network.
The overall elegance of this solution is that no additional software is needed. As long as you are within your predefined geo-ip location, have your phone, and know your hostname/ip address (and port number), then you will be able to access your system remotely.
I've been happy with having my firewall count SSH connection attempts, and if you make too many attempts in too short a period of time, it just blacklists that IP for 24 hours.
My first 'defense' was just moving off of port 22. I had planned other things, but to be honest, the attempts went from hundreds a day to zero, so I never bothered doing more.
I do the same, but only 10 minutes after 5 failed logins. That's enough for bots while not really getting in your way if it accidentally blocks the hotel you're in.
Combined with only allowing key based login, password is disabled.
My experience was similar. I opened a public SSH once and there were many unsuccessful login attempts. The way we resolved it was by installing VPN and closing the IP for public access.
Does that mean that you automatically block IPs that try to login on port 22 (since they are obviously malicious/port scannners)? If yes, is there any specialized daemon for that or are you using firewall rules? Or are you running a SSH honeypot on port 22?
A tarpit is a technique to waste your opponents time and network sockets. Usually by accepting the connection but only sending or receiving data at like 1byte/s.
Personally, I just whitelist SSH access on the firewall (UFW and/or AWS SecGrps). Everyone on the admin team has a static residential address. On the road, we use ZeroTier.
Shodan scans a handful of common ports [1] but you are right, a targeted attack will not be stopped by moving ports. Along the lines of what jms703 mentioned, moving ports just removes the noise from the bots so you can have better alerting on real attacks and not suffer from alert fatigue.
Or Fail2Ban. I have mine setup so that it does an API call to my OPNSense firewall. A few wrong logins to my self hosted things (SSH included) and that IP is permanently blocked from everything.
Update: they apparently also have a paid product called "Cloudflare for Teams", which also uses the same infrastructure and are targeted for companies. Probably the reason they don't block out SSH is that companies do want SSH access - but I'm pretty sure that RDP access would be higher on the list.
My friend working in an ISP has cleared the mystery for me - it's Cloudflare's Warp product (a semi-VPN (in the "teleport to other countries" sense, not the original definition)), which can be used for free, and I'm not shocked that it is being abused. Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.
So I don't know how to respond to this specifically except that I didn't know Cloudflare offers a VPN (apparently a limited knocked-down one though) and I'm pretty sure that other people in other ASes scan the default SSH port - in fact, I see more and more scans from ASes of cloud providers than residential ASes, whereas five years ago I mainly saw Chinese and Brazilian residential ISPs. This is why I often move the SSH port - it's not that it improves my security posture, just so that the noise in the logs are minimised.
> Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.
Hi, I'm the tech lead of Workers, so I'll clarify.
At present, Workers can only generate HTTP traffic, because the only API we've given Workers for network communications is `fetch()`, which is HTTP-specific. This similar to how in-browser JavaScript today cannot open arbitrary TCP connections, because there's no API for it. It's likely we'll add an API for arbitrary TCP eventually, but at the moment it's not possible that this traffic comes from Workers.
(As far as where it did come from, I don't have any internal knowledge, but WARP traffic (that's our VPN-like service) seems like an obvious explanation to me.)
His dump of packets shows an MSS of 1380 versus the more common default 1460, which sounds like it's coming from something with tunneled traffic. MSS 1380 would correspond to a source MTU of 1420, which is a typical default for a GRE tunnel in IPv4sec Tunnel mode. Pretty sure that's what WARP is.
WARP is Wireguard but yeah, otherwise this is pretty much on point. Traffic is tunneled, likely just a random WARP user abusing Cloudflares free VPN service to do some scanning.
FYI the WARP VPN is _not anonymizing_ and doesn’t claim to be. If you connect through WARP to a website behind Cloudflare, they will include your real IP in a header.
The main point of WARP is to circumvent ISP traffic shaping (which works quite well btw).
I mean, I just learned about Warp because of this, so if there are misunderstandings that my friend didn't think to clarify (probably because she thought that I already knew Warp), then apologies. The HN edit clock has since been long-expired, so I cannot add corrections about this.
Maybe that's a case, but their abuse team hasn't replied anything in a 2 weeks about that after I gave them all timestamps and both source/destination IP addresses
VPS and VPN providers should be very mindful of their reputation in this regard. If they get a reputation as a "black hole" where complaints vanish and nothing ever happens, the effect may be that other customers start to find themselves blocked or throttled on a subnet level.
I report them from time to time. I'm not sure why people don't take them seriously. If 500 people a day came up to your front door and tried the knob, hell maybe even tried a couple of keys in the lock... I'm pretty sure you'd be calling the cops.
An ssh server isn’t really like a front door though. I don’t necessarily think attempting a random ssh server should be a crime.
I used to spend time on custom iptables scripts but came to the conclusion it’s much better to just architect things in a way where the bots and scanners can’t plausibly create a problem and then ignoring them.
I can understand your frustration with background internet noise, but please note Cloudflare is not known for broadcasting their customers' names to the first abuse report with a pcap of a TCP handshake.
There may be more realistic ways to go about protecting people's SSH servers that trying to dox Cloudflare VPN users.
I can‘t understand it. There doesn‘t appear to be any downside or even abuse happening. The fact OP expects a company to explain who and why a customer of theirs did a legal non abusive act is just an outstanding level of entitlement.
I assumed OP wants to know the identity of the Cloudflare users scanning their SSH ports.
I think OP guessed it was probably not Cloudflare themselves scanning their ports, so I think that's what they meant by "hear who and why".
Maybe dox is too strong a word. My point is, from what I've heard, the general sentiment is that you're unlikely to get any information about customers just by sending abuse reports to Cloudflare.
Not to disappoint you, but except for logging SSH honeypots are becoming useless (most bots automatically disconnect when they detect a long login banner).
I don't think that demanding that every ISP inspect and censor outgoing packets is ever going to work. Every network operator has their own definition of good and bad, and it's probably up to that operator to enforce it themselves. Regarding SSH, you can basically assume that 4.2 billion IPs will concurrently try every password database that exists against your server 24/7. Passwords are a failed experiment. People can't remember them, and attackers can easily guess them. The solution is: don't accept password logins, and ban particular IPs after a certain number of failed attempts. (You should do the same for any unauthenticated resource -- if someone gets 1000 pages from your website in a second, you should probably cut off access to them for a while. Some script has gone awry, and it's costing you money, not the author of the script.)
The topics that HN discusses can show you the breadth of opinions on this issue. One day it's "I hate cloud provider X for blocking my abusive service" with 3000 comments agreeing. The next day it's "I hate cloud provider Y for not blocking someone's abusive service" with a different 3000 commeters agreeing. No consensus will ever be reached, and there is no Internet police force you can bully into agreeing with you. I would just take precautions and move on, rather than appealing to a mob or administrative body for some meaningless justice.
Probably yes, but hey! Why do Cloudflare Worker would need SSH connection establishment? I do not asking Cloudflare to block 22/tcp entirely, but all this situation is very odd - I am seeing anomaly, reported abuse to them, but no explanation why this is happening.
About password authentication I totally agreeing with you, but this is a bit out of scope of this thread.
The only purpose of this article is to know the truth what is really happening. I have never seen such many connections to the SSH even from researchers.
I think you need to spend more time looking at logs and network traffic in general, this is standard. I bet your public home IP will be scanned at least a couple times just today.
To the best of my knowledge, the IP blocks in the 8.0.0.0/8 range announced by cloudflare are their VPN exit blocks.
I had similar issues with them in the past spewing weird traffic. Didn't really ever reach a conclusion on it, other than just reducing logging for that error condition a scanner was triggering.
Enabling Warp via the "1.1.1.1" Android app gets me an 8.x.x.x VPN address, at least. This /24 appears to be routed to my city's Cloudflare node, so presumably there's a /24 per city they run this service in.
Running a quick port scan from my phone against one of my machines works, so it doesn't look like they are restricting this too heavily.
And I'm not logged into this app and haven't granted it additional permissions, so I'm not sure they have any idea who I am here.
Just use something like fail2ban to drop these at the packet filter level, either courtesy of your cloud provider or on the host itself. Make sure your sshd config is watertight.
What is the issue exactly? Is there any kind of exploitation attempts or is someone fishing for banners to do statistics upon? The second case is research, and legitimate...
I have asked Cloudflare about this traffic, but support says "Cloudflare helps protect sites, and accelerate them. We do not attack sites, and our network can't be used to generate attack traffic."
They deny any malicious outgoing traffic, but after I have provided pcap dump they just ignoring me and do not reply anything. Very odd behaviour for researchers.
Cloudflare likes to use the "We're just a proxy" defense when it suits them, but with the rapid release of other products they are very capable of generating arbitrary traffic and hosting content without an origin.
Just curious/not furious: is there a convincing argument for why one should enable this particular kind of research at the cost of potentially diminished security? I recognize that this question relies on an assumption that is itself arguable, and that it might vary with context.
(Aside: I really wish English had a better way to distinguish between an earnest question and a rhetorical challenge.)
Port scanning was always considered abuse until researchers decided they needed additional blog post material. I worked for a “business ISP” in the early 90s and we had no qualms kicking someone off the network for it, although usually the originator of the scan had themselves been compromised… because… as I say it was considered at least a breach of terms of service.
I ran engineering for what was at the time the most popular ISP in Chicago from 1995-1997 (the company later sold to RCN) and port scanning has never been considered abuse, though there have always been people who wished it was and loudly proclaimed it so.
I'd say that port scanning is attack prep. When you port scan me I take an adversarial stance. Certainly volume plays a role in determining whether scanning is abusive or not.
There are differences between walking by and noticing a door, knocking on a door, trying a key in a door, trying to defeat the lock. Before you can break the lock you have to notice the door. I haven't met any friendly door knockers on the internet.
Cloudflare warp is essentially a free vpn, this is going to happen. Iirc they use completely separate subnets for this so you can filter it/block it. Ip starts with 8. Something
Cloudflare is all over HN and their CTO, jgrahamc, will frequently appear to answer questions. California is just waking up so I expect a few ho it a before we get some solid answers to this issue.
I'm beginning to suspect that in addition to Cloudflare fanbois who'll downvote anything that paints Cloudflare in a bad light, Cloudflare may even have bots of their own. There're just too many downvoters to be explained by simple fanbois.
I get that some companies have fans, and I get that some of those fans become fanbois - fans regardless of the truth, with a ridiculous us-versus-them mentality. But I seriously doubt there are enough to explain what we see here on HN.
It'd be nice if Cloudflare weren't such assholes. They refuse to answer questions directly, they intentionally use form responses to be vague and uncommunicative when it suits them, and they clearly don't care about doing the right thing. They COULD be a much better company if they just communicated without being smug assholes.
Even a straightforward answer that isn't what we want to hear, like, "These are our VPN endpoints, and we have no plans to do anything about VPN customers who are attempting to brute force ssh." would still make it easy for us to respect Cloudflare. Instead, they facilitate abuse, ignore abuse complaints, and refuse to communicate, which, at least in my opinion, makes them assholes.
- Implement a firewall on your instances.
- Implement port knocking for sshd.
- Move sshd to a non standard port to avoid the nmap/bot noise.
- Only log successful logins.
- Any combination of the above.
There are pros and cons to each item so you would have to decide which combination is the most appropriate and least friction for your organization. If leaving the port exposed consider enforcing key based authentication and disabling password authentication. For high risk accounts such as monitoring accounts that use passwordless unrestricted sudo, you can even restrict what networks the keys are valid from.
I created gofwd, a cross-platform TCP port forwarder with Duo 2FA and Geographic IP integration. Its use case is to help protect services when using a VPN is not possible. While it's use cases are limited, it works great for me when I need to connect to my home network.
The overall elegance of this solution is that no additional software is needed. As long as you are within your predefined geo-ip location, have your phone, and know your hostname/ip address (and port number), then you will be able to access your system remotely.
After starting it, everything becomes much calmer. Nice thing is, fail2ban can protect much more than SSH.
Combined with only allowing key based login, password is disabled.
It's for the Pi as that's the target audience but it should apply generally for OpenSSH.
There's also a sister post about improving your "first factor" for those still using passwords: https://2byt.es/post/totp2/
For those who, like me, are unfamiliar with the concept of a SSH tarpit, it's a technique consisting of inserting delays into SSH connection attempts.
Old HN discussion on Endlessh:
https://news.ycombinator.com/item?id=24491453
I'm guessing that the person you replied to is referring to the same.
There are scanners like shodan that will scan every single port you have now so moving it to a non-standard port doesn't stop all the attackers.
[1] - https://en.wikipedia.org/wiki/Shodan_(website)
My friend working in an ISP has cleared the mystery for me - it's Cloudflare's Warp product (a semi-VPN (in the "teleport to other countries" sense, not the original definition)), which can be used for free, and I'm not shocked that it is being abused. Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.
So I don't know how to respond to this specifically except that I didn't know Cloudflare offers a VPN (apparently a limited knocked-down one though) and I'm pretty sure that other people in other ASes scan the default SSH port - in fact, I see more and more scans from ASes of cloud providers than residential ASes, whereas five years ago I mainly saw Chinese and Brazilian residential ISPs. This is why I often move the SSH port - it's not that it improves my security posture, just so that the noise in the logs are minimised.
Hi, I'm the tech lead of Workers, so I'll clarify.
At present, Workers can only generate HTTP traffic, because the only API we've given Workers for network communications is `fetch()`, which is HTTP-specific. This similar to how in-browser JavaScript today cannot open arbitrary TCP connections, because there's no API for it. It's likely we'll add an API for arbitrary TCP eventually, but at the moment it's not possible that this traffic comes from Workers.
(As far as where it did come from, I don't have any internal knowledge, but WARP traffic (that's our VPN-like service) seems like an obvious explanation to me.)
It's weird how the "VPN providers" have taken that acronym and turned it into what is basically another synonym for "proxy".
The main point of WARP is to circumvent ISP traffic shaping (which works quite well btw).
In any case I wouldn't think much of it. If you obsess over any garbage traffic you get you will go insane.
I used to spend time on custom iptables scripts but came to the conclusion it’s much better to just architect things in a way where the bots and scanners can’t plausibly create a problem and then ignoring them.
There may be more realistic ways to go about protecting people's SSH servers that trying to dox Cloudflare VPN users.
I think OP guessed it was probably not Cloudflare themselves scanning their ports, so I think that's what they meant by "hear who and why".
Maybe dox is too strong a word. My point is, from what I've heard, the general sentiment is that you're unlikely to get any information about customers just by sending abuse reports to Cloudflare.
Some of my servers don't even have any IPv4 connectivity and there haven't been any failed SSH logins over IPv6.
*tarpit
A honeypot lets people "in" to see/research malware that's in the wild:
* https://en.wikipedia.org/wiki/Honeypot_(computing)
A tarpit just takes up the attacker's resources:
* https://en.wikipedia.org/wiki/Tarpit_(networking)
https://lwn.net/Articles/848291/
I don't think that demanding that every ISP inspect and censor outgoing packets is ever going to work. Every network operator has their own definition of good and bad, and it's probably up to that operator to enforce it themselves. Regarding SSH, you can basically assume that 4.2 billion IPs will concurrently try every password database that exists against your server 24/7. Passwords are a failed experiment. People can't remember them, and attackers can easily guess them. The solution is: don't accept password logins, and ban particular IPs after a certain number of failed attempts. (You should do the same for any unauthenticated resource -- if someone gets 1000 pages from your website in a second, you should probably cut off access to them for a while. Some script has gone awry, and it's costing you money, not the author of the script.)
The topics that HN discusses can show you the breadth of opinions on this issue. One day it's "I hate cloud provider X for blocking my abusive service" with 3000 comments agreeing. The next day it's "I hate cloud provider Y for not blocking someone's abusive service" with a different 3000 commeters agreeing. No consensus will ever be reached, and there is no Internet police force you can bully into agreeing with you. I would just take precautions and move on, rather than appealing to a mob or administrative body for some meaningless justice.
About password authentication I totally agreeing with you, but this is a bit out of scope of this thread.
The only purpose of this article is to know the truth what is really happening. I have never seen such many connections to the SSH even from researchers.
For some reason, this is 6× more than a server I have on an university network.
> When unauthorized port scanning is detected by AWS, it is stopped and blocked.
Page 13
https://web.archive.org/web/20160331233541/https://d0.awssta...
It's not clear, now.
You can also go further if you like: Put the sshd service behind WireGuard (or spiped). Then malicious actors will never find your SSH ports.
I had similar issues with them in the past spewing weird traffic. Didn't really ever reach a conclusion on it, other than just reducing logging for that error condition a scanner was triggering.
Running a quick port scan from my phone against one of my machines works, so it doesn't look like they are restricting this too heavily.
And I'm not logged into this app and haven't granted it additional permissions, so I'm not sure they have any idea who I am here.
Remedy: Don't let fail2ban send mail, or at least remove the whois part.
Pretty nifty feature.
The OP has too much time on their hands. :)
You can play this whack-a-mole game indefinitely. If you have SSH open to the Internet, expect that others will try to connect 24/7.
(Aside: I really wish English had a better way to distinguish between an earnest question and a rhetorical challenge.)
There are differences between walking by and noticing a door, knocking on a door, trying a key in a door, trying to defeat the lock. Before you can break the lock you have to notice the door. I haven't met any friendly door knockers on the internet.
I get that some companies have fans, and I get that some of those fans become fanbois - fans regardless of the truth, with a ridiculous us-versus-them mentality. But I seriously doubt there are enough to explain what we see here on HN.
It'd be nice if Cloudflare weren't such assholes. They refuse to answer questions directly, they intentionally use form responses to be vague and uncommunicative when it suits them, and they clearly don't care about doing the right thing. They COULD be a much better company if they just communicated without being smug assholes.
Even a straightforward answer that isn't what we want to hear, like, "These are our VPN endpoints, and we have no plans to do anything about VPN customers who are attempting to brute force ssh." would still make it easy for us to respect Cloudflare. Instead, they facilitate abuse, ignore abuse complaints, and refuse to communicate, which, at least in my opinion, makes them assholes.