AS13335 doing SSH scanning

(encryp.ch)

166 points | by jimsi 25 days ago

18 comments

  • LinuxBender 25 days ago
    This is not unique to Cloudflare, Level3 or any of the other networks I've seen mentioned in this thread. The entire internet scans all the common ports and this should be expected to continue indefinitely. If this is causing alerts from your logging, there are a few options:

    - Implement a firewall on your instances.

    - Implement port knocking for sshd.

    - Move sshd to a non standard port to avoid the nmap/bot noise.

    - Only log successful logins.

    - Any combination of the above.

    There are pros and cons to each item so you would have to decide which combination is the most appropriate and least friction for your organization. If leaving the port exposed consider enforcing key based authentication and disabling password authentication. For high risk accounts such as monitoring accounts that use passwordless unrestricted sudo, you can even restrict what networks the keys are valid from.

    • jftuga 25 days ago
      https://github.com/jftuga/gofwd

      I created gofwd, a cross-platform TCP port forwarder with Duo 2FA and Geographic IP integration. Its use case is to help protect services when using a VPN is not possible. While it's use cases are limited, it works great for me when I need to connect to my home network.

      The overall elegance of this solution is that no additional software is needed. As long as you are within your predefined geo-ip location, have your phone, and know your hostname/ip address (and port number), then you will be able to access your system remotely.

      • akira2501 25 days ago
        I've been happy with having my firewall count SSH connection attempts, and if you make too many attempts in too short a period of time, it just blacklists that IP for 24 hours.
        • silisili 25 days ago
          My first 'defense' was just moving off of port 22. I had planned other things, but to be honest, the attempts went from hundreds a day to zero, so I never bothered doing more.
          • bayindirh 25 days ago
            The first package I install on public facing servers is fail2ban.

            After starting it, everything becomes much calmer. Nice thing is, fail2ban can protect much more than SSH.

            • t0mas88 25 days ago
              I do the same, but only 10 minutes after 5 failed logins. That's enough for bots while not really getting in your way if it accidentally blocks the hotel you're in.

              Combined with only allowing key based login, password is disabled.

              • _moof 25 days ago
                I've recently added 2FA as well. Super easy to do.
                • alias_neo 25 days ago
                  I wrote a blog on how to do it a couple years back if anyone is interested in a "tutorial": https://2byt.es/post/totp/

                  It's for the Pi as that's the target audience but it should apply generally for OpenSSH.

                  There's also a sister post about improving your "first factor" for those still using passwords: https://2byt.es/post/totp2/

            • airocker 25 days ago
              My experience was similar. I opened a public SSH once and there were many unsuccessful login attempts. The way we resolved it was by installing VPN and closing the IP for public access.
              • twodai 25 days ago
                My favorite is to implement a tarpit while moving the ssh port.
                • rualca 25 days ago
                  > My favorite is to implement a tarpit while moving the ssh port.

                  For those who, like me, are unfamiliar with the concept of a SSH tarpit, it's a technique consisting of inserting delays into SSH connection attempts.

                  Old HN discussion on Endlessh:

                  https://news.ycombinator.com/item?id=24491453

                  • hda111 25 days ago
                    Isn’t this easily detectable for any malicious script? Just a small timeout and try the next port.
                  • themulticaster 25 days ago
                    Does that mean that you automatically block IPs that try to login on port 22 (since they are obviously malicious/port scannners)? If yes, is there any specialized daemon for that or are you using firewall rules? Or are you running a SSH honeypot on port 22?
                    • c0wb0yc0d3r 25 days ago
                      The post is referring to this: https://github.com/skeeto/endlessh

                      I'm guessing that the person you replied to is referring to the same.

                      • ben_bai 25 days ago
                        A tarpit is a technique to waste your opponents time and network sockets. Usually by accepting the connection but only sending or receiving data at like 1byte/s.
                    • rollcat 25 days ago
                      Personally, I just whitelist SSH access on the firewall (UFW and/or AWS SecGrps). Everyone on the admin team has a static residential address. On the road, we use ZeroTier.
                      • vgb2k18 25 days ago
                        Adding: [-] whitelist allowed ip's for port 22. Requires static ip's client-side obviously [-] fail2ban if static ip's are not available.
                        • codexon 25 days ago
                          > - Move sshd to a non standard port to avoid the nmap/bot noise.

                          There are scanners like shodan that will scan every single port you have now so moving it to a non-standard port doesn't stop all the attackers.

                          • LinuxBender 25 days ago
                            Shodan scans a handful of common ports [1] but you are right, a targeted attack will not be stopped by moving ports. Along the lines of what jms703 mentioned, moving ports just removes the noise from the bots so you can have better alerting on real attacks and not suffer from alert fatigue.

                            [1] - https://en.wikipedia.org/wiki/Shodan_(website)

                            • jms703 25 days ago
                              The parent post didn’t say to do this to stop attackers. They said to do this to reduce alerts in the logs.
                            • seized 25 days ago
                              Or Fail2Ban. I have mine setup so that it does an API call to my OPNSense firewall. A few wrong logins to my self hosted things (SSH included) and that IP is permanently blocked from everything.
                              • hacker_newz 25 days ago
                                If you expose sshd to the internet in the first place you are doing something wrong.
                              • zinekeller 25 days ago
                                Update: they apparently also have a paid product called "Cloudflare for Teams", which also uses the same infrastructure and are targeted for companies. Probably the reason they don't block out SSH is that companies do want SSH access - but I'm pretty sure that RDP access would be higher on the list.

                                My friend working in an ISP has cleared the mystery for me - it's Cloudflare's Warp product (a semi-VPN (in the "teleport to other countries" sense, not the original definition)), which can be used for free, and I'm not shocked that it is being abused. Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.

                                So I don't know how to respond to this specifically except that I didn't know Cloudflare offers a VPN (apparently a limited knocked-down one though) and I'm pretty sure that other people in other ASes scan the default SSH port - in fact, I see more and more scans from ASes of cloud providers than residential ASes, whereas five years ago I mainly saw Chinese and Brazilian residential ISPs. This is why I often move the SSH port - it's not that it improves my security posture, just so that the noise in the logs are minimised.

                                • kentonv 25 days ago
                                  > Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.

                                  Hi, I'm the tech lead of Workers, so I'll clarify.

                                  At present, Workers can only generate HTTP traffic, because the only API we've given Workers for network communications is `fetch()`, which is HTTP-specific. This similar to how in-browser JavaScript today cannot open arbitrary TCP connections, because there's no API for it. It's likely we'll add an API for arbitrary TCP eventually, but at the moment it's not possible that this traffic comes from Workers.

                                  (As far as where it did come from, I don't have any internal knowledge, but WARP traffic (that's our VPN-like service) seems like an obvious explanation to me.)

                                  • tyingq 25 days ago
                                    His dump of packets shows an MSS of 1380 versus the more common default 1460, which sounds like it's coming from something with tunneled traffic. MSS 1380 would correspond to a source MTU of 1420, which is a typical default for a GRE tunnel in IPv4sec Tunnel mode. Pretty sure that's what WARP is.
                                    • jpgvm 25 days ago
                                      WARP is Wireguard but yeah, otherwise this is pretty much on point. Traffic is tunneled, likely just a random WARP user abusing Cloudflares free VPN service to do some scanning.
                                      • tyingq 25 days ago
                                        Ah, okay. Google searches suggests the default MTU on Wireguard is also 1420, which would produce an MSS of 1380. So that makes sense.
                                  • userbinator 25 days ago
                                    a semi-VPN (in the "teleport to other countries" sense, not the original definition)

                                    It's weird how the "VPN providers" have taken that acronym and turned it into what is basically another synonym for "proxy".

                                    • TechBro8615 25 days ago
                                      FYI the WARP VPN is _not anonymizing_ and doesn’t claim to be. If you connect through WARP to a website behind Cloudflare, they will include your real IP in a header.

                                      The main point of WARP is to circumvent ISP traffic shaping (which works quite well btw).

                                      • zinekeller 24 days ago
                                        I mean, I just learned about Warp because of this, so if there are misunderstandings that my friend didn't think to clarify (probably because she thought that I already knew Warp), then apologies. The HN edit clock has since been long-expired, so I cannot add corrections about this.
                                    • vadfa 25 days ago
                                      Cloudflare has a VPN product, WARP. It's possible that they are clients of theirs doing the scanning.

                                      In any case I wouldn't think much of it. If you obsess over any garbage traffic you get you will go insane.

                                      • jimsi 25 days ago
                                        Maybe that's a case, but their abuse team hasn't replied anything in a 2 weeks about that after I gave them all timestamps and both source/destination IP addresses
                                        • brohee 25 days ago
                                          What kind of answer would you expect, in all seriousness? The thing you are reporting is not illegal or even dodgy.
                                          • marginalia_nu 25 days ago
                                            VPS and VPN providers should be very mindful of their reputation in this regard. If they get a reputation as a "black hole" where complaints vanish and nothing ever happens, the effect may be that other customers start to find themselves blocked or throttled on a subnet level.
                                            • eli 25 days ago
                                              I don’t know of any provider that would take action on reports of ssh scanning.
                                              • sleepybrett 24 days ago
                                                I report them from time to time. I'm not sure why people don't take them seriously. If 500 people a day came up to your front door and tried the knob, hell maybe even tried a couple of keys in the lock... I'm pretty sure you'd be calling the cops.
                                                • eli 24 days ago
                                                  An ssh server isn’t really like a front door though. I don’t necessarily think attempting a random ssh server should be a crime.

                                                  I used to spend time on custom iptables scripts but came to the conclusion it’s much better to just architect things in a way where the bots and scanners can’t plausibly create a problem and then ignoring them.

                                                • TechBro8615 25 days ago
                                                  A little bird told me a story that AWS will forward abuse reports to customers performing outbound nmap scans.
                                                • sleepybrett 24 days ago
                                                  • TechBro8615 25 days ago
                                                    I don’t think anyone is about to block or throttle traffic from Cloudflare IP ranges.
                                                    • marginalia_nu 25 days ago
                                                      Incoming connections? Why not. I have at times, it's pretty great.
                                                  • jimsi 25 days ago
                                                    I expect to hear who (and why) generates that kind of traffic from cloudflare owned subnets.
                                                    • tux3 25 days ago
                                                      I can understand your frustration with background internet noise, but please note Cloudflare is not known for broadcasting their customers' names to the first abuse report with a pcap of a TCP handshake.

                                                      There may be more realistic ways to go about protecting people's SSH servers that trying to dox Cloudflare VPN users.

                                                      • that_guy_iain 25 days ago
                                                        I can‘t understand it. There doesn‘t appear to be any downside or even abuse happening. The fact OP expects a company to explain who and why a customer of theirs did a legal non abusive act is just an outstanding level of entitlement.
                                                        • Hrundi 25 days ago
                                                          Who's trying to dox those users?
                                                          • tux3 25 days ago
                                                            I assumed OP wants to know the identity of the Cloudflare users scanning their SSH ports.

                                                            I think OP guessed it was probably not Cloudflare themselves scanning their ports, so I think that's what they meant by "hear who and why".

                                                            Maybe dox is too strong a word. My point is, from what I've heard, the general sentiment is that you're unlikely to get any information about customers just by sending abuse reports to Cloudflare.

                                                            • chmike 25 days ago
                                                              OP obviously simply expected at least an explanation on the cause of these ssh connection probes. He got the explanation here (VPN).
                                                    • miyuru 25 days ago
                                                      If your ISP and the server support IPv6, just disable SSH on IPv4.

                                                      Some of my servers don't even have any IPv4 connectivity and there haven't been any failed SSH logins over IPv6.

                                                    • vadfa 25 days ago
                                                      They won't disconnect clients over SSH scanning. That's a ridiculous expectation. It's 2021. Nobody cares.
                                                    • mcwhy 25 days ago
                                                      It could also be used for IP reputation, running ssh can be a sign of a proxy or other server/service that website owners might want to block.
                                                      • jimsi 25 days ago
                                                        I think single SYN packet would be enough to know whether this port open or not, but they trying to gather full banner.
                                                        • sowhatw 25 days ago
                                                          Who cares? Get a better hobby.
                                                    • jrockway 25 days ago
                                                      This is probably Cloudflare Workers?

                                                      I don't think that demanding that every ISP inspect and censor outgoing packets is ever going to work. Every network operator has their own definition of good and bad, and it's probably up to that operator to enforce it themselves. Regarding SSH, you can basically assume that 4.2 billion IPs will concurrently try every password database that exists against your server 24/7. Passwords are a failed experiment. People can't remember them, and attackers can easily guess them. The solution is: don't accept password logins, and ban particular IPs after a certain number of failed attempts. (You should do the same for any unauthenticated resource -- if someone gets 1000 pages from your website in a second, you should probably cut off access to them for a while. Some script has gone awry, and it's costing you money, not the author of the script.)

                                                      The topics that HN discusses can show you the breadth of opinions on this issue. One day it's "I hate cloud provider X for blocking my abusive service" with 3000 comments agreeing. The next day it's "I hate cloud provider Y for not blocking someone's abusive service" with a different 3000 commeters agreeing. No consensus will ever be reached, and there is no Internet police force you can bully into agreeing with you. I would just take precautions and move on, rather than appealing to a mob or administrative body for some meaningless justice.

                                                      • jimsi 25 days ago
                                                        Probably yes, but hey! Why do Cloudflare Worker would need SSH connection establishment? I do not asking Cloudflare to block 22/tcp entirely, but all this situation is very odd - I am seeing anomaly, reported abuse to them, but no explanation why this is happening.

                                                        About password authentication I totally agreeing with you, but this is a bit out of scope of this thread.

                                                        The only purpose of this article is to know the truth what is really happening. I have never seen such many connections to the SSH even from researchers.

                                                        • dapids 25 days ago
                                                          I think you need to spend more time looking at logs and network traffic in general, this is standard. I bet your public home IP will be scanned at least a couple times just today.
                                                          • Symbiote 25 days ago
                                                            Judging my my home network, they can probably expect an SSH connection attempt on a home IP about every 5 seconds or so, or 18,000 times a day.

                                                            For some reason, this is 6× more than a server I have on an university network.

                                                        • stefan_ 25 days ago
                                                          Please name the cloud providers that have a policy of allowing malicious actors to do SSH scans? Do you mean Cloudflare?
                                                          • judge2020 25 days ago
                                                            Let me know if you get banned for doing nmap -p 22 on AWS or GCP.
                                                      • cpach 25 days ago
                                                        As others have pointed out, disabling password-based logins for sshd is a must.

                                                        You can also go further if you like: Put the sshd service behind WireGuard (or spiped). Then malicious actors will never find your SSH ports.

                                                        • benjojo12 25 days ago
                                                          To the best of my knowledge, the IP blocks in the 8.0.0.0/8 range announced by cloudflare are their VPN exit blocks.

                                                          I had similar issues with them in the past spewing weird traffic. Didn't really ever reach a conclusion on it, other than just reducing logging for that error condition a scanner was triggering.

                                                          • jimsi 25 days ago
                                                            How did you discover this? Anyway thanks for additional information
                                                            • aaron42net 25 days ago
                                                              Enabling Warp via the "1.1.1.1" Android app gets me an 8.x.x.x VPN address, at least. This /24 appears to be routed to my city's Cloudflare node, so presumably there's a /24 per city they run this service in.

                                                              Running a quick port scan from my phone against one of my machines works, so it doesn't look like they are restricting this too heavily.

                                                              And I'm not logged into this app and haven't granted it additional permissions, so I'm not sure they have any idea who I am here.

                                                              • benjojo12 25 days ago
                                                                via support ticket with cloudflare. They didn't explicitly confirm it, but my experience is that they would have corrected me if that was wrong :)
                                                                • joshenders 25 days ago
                                                                  jimsi, employees are also extremely active on Twitter. It’s not the most scalable but that’s usually my goto for eyebrow raising Cf related questions.
                                                              • nickdothutton 25 days ago
                                                                Just use something like fail2ban to drop these at the packet filter level, either courtesy of your cloud provider or on the host itself. Make sure your sshd config is watertight.
                                                                • loeg 25 days ago
                                                                  You know, or just don’t use fail2ban: https://research.securitum.com/fail2ban-remote-code-executio... . It’s adding extra attack surface for a cosmetic benefit.
                                                                  • tempodox 25 days ago
                                                                    Good catch, thanks!

                                                                    Remedy: Don't let fail2ban send mail, or at least remove the whois part.

                                                                  • nousermane 25 days ago
                                                                    Are there reputable places where people can share the IPs tripping fail2ban? Like spamhaus.org, but for scanners?
                                                                    • cpach 24 days ago
                                                                      Honest question: Why bother?
                                                                    • megous 25 days ago
                                                                      nftables allows to automatize blocking of IP addresses (even with a automatic timeout) without any userspace involvement.

                                                                      Pretty nifty feature.

                                                                    • brohee 25 days ago
                                                                      What is the issue exactly? Is there any kind of exploitation attempts or is someone fishing for banners to do statistics upon? The second case is research, and legitimate...
                                                                      • Bluecobra 25 days ago
                                                                        > What is the issue exactly?

                                                                        The OP has too much time on their hands. :)

                                                                        You can play this whack-a-mole game indefinitely. If you have SSH open to the Internet, expect that others will try to connect 24/7.

                                                                        • jimsi 25 days ago
                                                                          I have asked Cloudflare about this traffic, but support says "Cloudflare helps protect sites, and accelerate them. We do not attack sites, and our network can't be used to generate attack traffic." They deny any malicious outgoing traffic, but after I have provided pcap dump they just ignoring me and do not reply anything. Very odd behaviour for researchers.
                                                                          • r1ch 25 days ago
                                                                            Cloudflare likes to use the "We're just a proxy" defense when it suits them, but with the rapid release of other products they are very capable of generating arbitrary traffic and hosting content without an origin.
                                                                          • _moof 25 days ago
                                                                            Just curious/not furious: is there a convincing argument for why one should enable this particular kind of research at the cost of potentially diminished security? I recognize that this question relies on an assumption that is itself arguable, and that it might vary with context.

                                                                            (Aside: I really wish English had a better way to distinguish between an earnest question and a rhetorical challenge.)

                                                                          • gtirloni 25 days ago
                                                                            Is SSH scanning considered "abuse" now?
                                                                            • nickdothutton 25 days ago
                                                                              Port scanning was always considered abuse until researchers decided they needed additional blog post material. I worked for a “business ISP” in the early 90s and we had no qualms kicking someone off the network for it, although usually the originator of the scan had themselves been compromised… because… as I say it was considered at least a breach of terms of service.
                                                                              • tptacek 25 days ago
                                                                                I ran engineering for what was at the time the most popular ISP in Chicago from 1995-1997 (the company later sold to RCN) and port scanning has never been considered abuse, though there have always been people who wished it was and loudly proclaimed it so.
                                                                              • annoyingnoob 25 days ago
                                                                                I'd say that port scanning is attack prep. When you port scan me I take an adversarial stance. Certainly volume plays a role in determining whether scanning is abusive or not.

                                                                                There are differences between walking by and noticing a door, knocking on a door, trying a key in a door, trying to defeat the lock. Before you can break the lock you have to notice the door. I haven't met any friendly door knockers on the internet.

                                                                                • tptacek 25 days ago
                                                                                  Yes you have. Censys is a good example (you have assuredly met Censys, whether you know it or not.)
                                                                                • chmike 25 days ago
                                                                                  Some ssh scanning is obviously abuse. Is it hard to make the difference ?
                                                                                • sowhatw 25 days ago
                                                                                  Why do you think this matters in any way whatsoever? Ports are there to be connected to. I don't see the problem.
                                                                                  • giantrobot 25 days ago
                                                                                    So...drop 8.0.0.0/8 traffic and move on? Port scans happen all the time, some are malicious and others not.
                                                                                    • ronsor 25 days ago
                                                                                      I don't think dropping traffic from roughly 1/200th of public IP space is a real solution.
                                                                                      • giantrobot 24 days ago
                                                                                        Isn't it? If you don't need to accept SSH traffic from that block you can just drop it.
                                                                                    • loloquwowndueo 25 days ago
                                                                                      Just install sshguard or some such and move on with your life. Or - block the entire IP range, and see who complains.
                                                                                      • specto 25 days ago
                                                                                        Yeah lots of baddies using workers. I blocked cloudflare a long time ago because of workers.
                                                                                        • kristianpaul 25 days ago
                                                                                          I run OSSEC that comes with automatic responses for common SSH authentication events
                                                                                          • sammy2244 25 days ago
                                                                                            Cloudflare warp is essentially a free vpn, this is going to happen. Iirc they use completely separate subnets for this so you can filter it/block it. Ip starts with 8. Something
                                                                                            • sm_javaarchive 24 days ago
                                                                                              Yep, if the site you're accessing is not on the cloudflare network, they'll see an ip starting with 8.
                                                                                            • cuu508 25 days ago
                                                                                              In case Cloudflare monitors the keyword "Cloudflare" in HN comments, I'll mention it here – Cloudflare.
                                                                                              • evgen 25 days ago
                                                                                                Cloudflare is all over HN and their CTO, jgrahamc, will frequently appear to answer questions. California is just waking up so I expect a few ho it a before we get some solid answers to this issue.
                                                                                                • jimsi 25 days ago
                                                                                                  Previously this post was named in such way, but post get flagged... So I removed its mentioning and replaced with ASN
                                                                                                  • johnklos 25 days ago
                                                                                                    I'm beginning to suspect that in addition to Cloudflare fanbois who'll downvote anything that paints Cloudflare in a bad light, Cloudflare may even have bots of their own. There're just too many downvoters to be explained by simple fanbois.

                                                                                                    I get that some companies have fans, and I get that some of those fans become fanbois - fans regardless of the truth, with a ridiculous us-versus-them mentality. But I seriously doubt there are enough to explain what we see here on HN.

                                                                                                    It'd be nice if Cloudflare weren't such assholes. They refuse to answer questions directly, they intentionally use form responses to be vague and uncommunicative when it suits them, and they clearly don't care about doing the right thing. They COULD be a much better company if they just communicated without being smug assholes.

                                                                                                    Even a straightforward answer that isn't what we want to hear, like, "These are our VPN endpoints, and we have no plans to do anything about VPN customers who are attempting to brute force ssh." would still make it easy for us to respect Cloudflare. Instead, they facilitate abuse, ignore abuse complaints, and refuse to communicate, which, at least in my opinion, makes them assholes.