This is not unique to Cloudflare, Level3 or any of the other networks I've seen mentioned in this thread. The entire internet scans all the common ports and this should be expected to continue indefinitely. If this is causing alerts from your logging, there are a few options:
- Implement a firewall on your instances.
- Implement port knocking for sshd.
- Move sshd to a non standard port to avoid the nmap/bot noise.
- Only log successful logins.
- Any combination of the above.
There are pros and cons to each item so you would have to decide which combination is the most appropriate and least friction for your organization. If leaving the port exposed consider enforcing key based authentication and disabling password authentication. For high risk accounts such as monitoring accounts that use passwordless unrestricted sudo, you can even restrict what networks the keys are valid from.
I created gofwd, a cross-platform TCP port forwarder with Duo 2FA and Geographic IP integration. Its use case is to help protect services when using a VPN is not possible. While it's use cases are limited, it works great for me when I need to connect to my home network.
The overall elegance of this solution is that no additional software is needed. As long as you are within your predefined geo-ip location, have your phone, and know your hostname/ip address (and port number), then you will be able to access your system remotely.
Does that mean that you automatically block IPs that try to login on port 22 (since they are obviously malicious/port scannners)? If yes, is there any specialized daemon for that or are you using firewall rules? Or are you running a SSH honeypot on port 22?
Shodan scans a handful of common ports  but you are right, a targeted attack will not be stopped by moving ports. Along the lines of what jms703 mentioned, moving ports just removes the noise from the bots so you can have better alerting on real attacks and not suffer from alert fatigue.
Update: they apparently also have a paid product called "Cloudflare for Teams", which also uses the same infrastructure and are targeted for companies. Probably the reason they don't block out SSH is that companies do want SSH access - but I'm pretty sure that RDP access would be higher on the list.
My friend working in an ISP has cleared the mystery for me - it's Cloudflare's Warp product (a semi-VPN (in the "teleport to other countries" sense, not the original definition)), which can be used for free, and I'm not shocked that it is being abused. Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.
So I don't know how to respond to this specifically except that I didn't know Cloudflare offers a VPN (apparently a limited knocked-down one though) and I'm pretty sure that other people in other ASes scan the default SSH port - in fact, I see more and more scans from ASes of cloud providers than residential ASes, whereas five years ago I mainly saw Chinese and Brazilian residential ISPs. This is why I often move the SSH port - it's not that it improves my security posture, just so that the noise in the logs are minimised.
> Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.
Hi, I'm the tech lead of Workers, so I'll clarify.
(As far as where it did come from, I don't have any internal knowledge, but WARP traffic (that's our VPN-like service) seems like an obvious explanation to me.)
His dump of packets shows an MSS of 1380 versus the more common default 1460, which sounds like it's coming from something with tunneled traffic. MSS 1380 would correspond to a source MTU of 1420, which is a typical default for a GRE tunnel in IPv4sec Tunnel mode. Pretty sure that's what WARP is.
I mean, I just learned about Warp because of this, so if there are misunderstandings that my friend didn't think to clarify (probably because she thought that I already knew Warp), then apologies. The HN edit clock has since been long-expired, so I cannot add corrections about this.
VPS and VPN providers should be very mindful of their reputation in this regard. If they get a reputation as a "black hole" where complaints vanish and nothing ever happens, the effect may be that other customers start to find themselves blocked or throttled on a subnet level.
I report them from time to time. I'm not sure why people don't take them seriously. If 500 people a day came up to your front door and tried the knob, hell maybe even tried a couple of keys in the lock... I'm pretty sure you'd be calling the cops.
An ssh server isn’t really like a front door though. I don’t necessarily think attempting a random ssh server should be a crime.
I used to spend time on custom iptables scripts but came to the conclusion it’s much better to just architect things in a way where the bots and scanners can’t plausibly create a problem and then ignoring them.
I can understand your frustration with background internet noise, but please note Cloudflare is not known for broadcasting their customers' names to the first abuse report with a pcap of a TCP handshake.
There may be more realistic ways to go about protecting people's SSH servers that trying to dox Cloudflare VPN users.
I can‘t understand it. There doesn‘t appear to be any downside or even abuse happening. The fact OP expects a company to explain who and why a customer of theirs did a legal non abusive act is just an outstanding level of entitlement.
I don't think that demanding that every ISP inspect and censor outgoing packets is ever going to work. Every network operator has their own definition of good and bad, and it's probably up to that operator to enforce it themselves. Regarding SSH, you can basically assume that 4.2 billion IPs will concurrently try every password database that exists against your server 24/7. Passwords are a failed experiment. People can't remember them, and attackers can easily guess them. The solution is: don't accept password logins, and ban particular IPs after a certain number of failed attempts. (You should do the same for any unauthenticated resource -- if someone gets 1000 pages from your website in a second, you should probably cut off access to them for a while. Some script has gone awry, and it's costing you money, not the author of the script.)
The topics that HN discusses can show you the breadth of opinions on this issue. One day it's "I hate cloud provider X for blocking my abusive service" with 3000 comments agreeing. The next day it's "I hate cloud provider Y for not blocking someone's abusive service" with a different 3000 commeters agreeing. No consensus will ever be reached, and there is no Internet police force you can bully into agreeing with you. I would just take precautions and move on, rather than appealing to a mob or administrative body for some meaningless justice.
Probably yes, but hey! Why do Cloudflare Worker would need SSH connection establishment? I do not asking Cloudflare to block 22/tcp entirely, but all this situation is very odd - I am seeing anomaly, reported abuse to them, but no explanation why this is happening.
About password authentication I totally agreeing with you, but this is a bit out of scope of this thread.
The only purpose of this article is to know the truth what is really happening. I have never seen such many connections to the SSH even from researchers.
Enabling Warp via the "18.104.22.168" Android app gets me an 8.x.x.x VPN address, at least. This /24 appears to be routed to my city's Cloudflare node, so presumably there's a /24 per city they run this service in.
Running a quick port scan from my phone against one of my machines works, so it doesn't look like they are restricting this too heavily.
And I'm not logged into this app and haven't granted it additional permissions, so I'm not sure they have any idea who I am here.
I have asked Cloudflare about this traffic, but support says "Cloudflare helps protect sites, and accelerate them. We do not attack sites, and our network can't be used to generate attack traffic."
They deny any malicious outgoing traffic, but after I have provided pcap dump they just ignoring me and do not reply anything. Very odd behaviour for researchers.
Cloudflare likes to use the "We're just a proxy" defense when it suits them, but with the rapid release of other products they are very capable of generating arbitrary traffic and hosting content without an origin.
Just curious/not furious: is there a convincing argument for why one should enable this particular kind of research at the cost of potentially diminished security? I recognize that this question relies on an assumption that is itself arguable, and that it might vary with context.
(Aside: I really wish English had a better way to distinguish between an earnest question and a rhetorical challenge.)
Port scanning was always considered abuse until researchers decided they needed additional blog post material. I worked for a “business ISP” in the early 90s and we had no qualms kicking someone off the network for it, although usually the originator of the scan had themselves been compromised… because… as I say it was considered at least a breach of terms of service.
I ran engineering for what was at the time the most popular ISP in Chicago from 1995-1997 (the company later sold to RCN) and port scanning has never been considered abuse, though there have always been people who wished it was and loudly proclaimed it so.
I'd say that port scanning is attack prep. When you port scan me I take an adversarial stance. Certainly volume plays a role in determining whether scanning is abusive or not.
There are differences between walking by and noticing a door, knocking on a door, trying a key in a door, trying to defeat the lock. Before you can break the lock you have to notice the door. I haven't met any friendly door knockers on the internet.
I'm beginning to suspect that in addition to Cloudflare fanbois who'll downvote anything that paints Cloudflare in a bad light, Cloudflare may even have bots of their own. There're just too many downvoters to be explained by simple fanbois.
I get that some companies have fans, and I get that some of those fans become fanbois - fans regardless of the truth, with a ridiculous us-versus-them mentality. But I seriously doubt there are enough to explain what we see here on HN.
It'd be nice if Cloudflare weren't such assholes. They refuse to answer questions directly, they intentionally use form responses to be vague and uncommunicative when it suits them, and they clearly don't care about doing the right thing. They COULD be a much better company if they just communicated without being smug assholes.
Even a straightforward answer that isn't what we want to hear, like, "These are our VPN endpoints, and we have no plans to do anything about VPN customers who are attempting to brute force ssh." would still make it easy for us to respect Cloudflare. Instead, they facilitate abuse, ignore abuse complaints, and refuse to communicate, which, at least in my opinion, makes them assholes.