I love Tailscale, it’s by far the best VPN I’ve used, and the easiest wireguard implementation to get up and running I’ve used.
I can certainly see the value of this feature for some orgs, but it seems little scary to me. With this setup, if an attacker is able to compromise Tailscale and add a key to your tailnet, that person will immediately have access to your network AND shell access to all of your boxes, rather than just network access if you use Tailscale with vanilla ssh.
I feel the same way. I wish they offered a second factor for SSH auth — when I last looked, they didn’t.
I also send myself notifications any time a failed or successful SSH login attempt occurs by tailing the ssh service with journalctl. When I last tried Tailscale SSH, it didn’t log anything to journalctl and so my self-notification via journalctl method did not work.
Tailscale lock looks interesting -- I had not seen that before.
ACLs still come down to "just trust tailscale is working as advertised." And while I generally do (I'm a happy user), if that last few years have shown anything, for the vast majority of companies/products, it's not if you get compromised, it's when. Given that I'd prefer to have multiple distinct layers between the internet and my boxes.
Though definitely see the value in terms of being able to easily tie SSH access to ACLs and your SSO provider -- as with all things security it's a trade off between ease of use and locking everything down.
Don’t need to add a key, if you have access to the network interface on a laptop or server you are effectively that entity. ACLs are just one level of defense, and iirc you can force re-auth on SSH connections but it certainly becomes a single point of compromise.
We use Tailscale quite heavily and the SSH feature. Along with many other features, it is great. However, the article doesn't mention pricing, which for me personally seems quite high at $18/month/user. [1]
It is quite clearly stated in both this announcement and the docs [1] that it is available for the personal (aka free) plan. Not sure why their pricing plan does not include any mention of it.
What is their incentive to offer such a service for free?
Following the adage "if it's for free, you are the product", what is going on behind the scenes? Are they providing their services as a giant honey-pot to sniff on traffic?
Their response in the past about their free tiers has been that it's low cost to offer and it results in larger sales (devs use it themselves, like it and bring it to places they work).
> TL;DR: Tailscale’s free plan is free because we keep our scaling costs low relative to typical SaaS companies. We care about privacy, so unlike some other freemium models, you and your data are not the product. Rather, increased word-of-mouth from free plans sells the more valuable corporate plans. I know, it sounds too good to be true. Let’s see some details.
So it's a weighed choice between "if something seems like it's too good to be true, it often is", and "the explanations they give make good sense, and it's a way of doing business that some ethical company could choose to take".
We probably won't know in the short-to-medium term, so we'll have to take their word for it..
But I must admit, their products look pretty impressive. I'll have to have a closer look at them.
For what it's worth the scaling costs for their service are quite low. Tailscale connections are almost entirely peer to peer after an initial NAT busting operation. They can afford to do a loss leader like this and the product is actually so good that I've recommended it to a number of places. It's literally the first VPN that I think is worth paying for. I wouldn't have known that if the free tier didn't exist. Using is believing in their case. It's not uncommon to be literally angry at how easy it is to set up/manage/deploy given how much of a trash fire most vpn software is.
> Tailscale connections are almost entirely peer to peer after an initial NAT busting operation.
Ah, interesting, thanks. That would indeed make it a lot less costly. I would need to dive into it to get a better understanding how their service works.
Would you happen to have some good resources you found useful?
Well, this addresses the sniffing concern. From the link:
Note that the private key never, ever leaves its node. This is important because the private key is the only thing that could potentially be used to impersonate that node when negotiating a WireGuard session. As a result, only that node can encrypt packets addressed from itself, or decrypt packets addressed to itself. It’s important to keep that in mind: Tailscale node connections are end-to-end encrypted (a concept called “zero trust networking”).
Which other vendors did you get quotes from? Genuinely curios. Ime they are very competitive on price compared to some others.
One of my former gigs used a competitor (also a startup) that was 2x the price and offered way less flexible setup but our “architect” rushed it bc of the way he was. Other more established competitors were even more expensive
I've been impressed with ZeroTier.
The free tier is pretty generous. Plus the desktop client allows you to access more than one network/account at the same time. IIRC, Tailscale requires you to switch between accounts.
Tailscale SSH is quite interesting because it'd require Tailscale authentication. So it would segment SSH access off, and makes SSH access also generally available to all clients utilizing Tailscale, regardless of host OS.
I checked, and it doesn't seem like Headscale supports SSH access.
I dont know about the pricing. Over the last couple of months i have been investigating different alternatives for better access control to our internal resources. And Tailscale seems to be on the cheaper end compared to other offerings.
How do you "untrust" a single person's key under this scheme? You would have to visit all of the machines and remove them from the authorized keys file.
Not really, with an SSH CA you’re trusting the CA and not installing individual keys into authorized_keys files.
Anything signed by the SSH CA will work for logins.
To deal with the “untrust” issue it’s normal for operations with an SSH CA to rely on (very) short-lived certificates, meaning often issued and valid for < 24 hours (it’s configurable, I’ve seen this be as short as 30 minutes).
Smallstep wrote a summary here which is pretty good —
> To deal with the “untrust” issue it’s normal for operations with an SSH CA to rely on (very) short-lived certificates, meaning often issued and valid for < 24 hours (it’s configurable, I’ve seen this be as short as 30 minutes).
So you want a way to get rid of long-lived SSH certificates, instead authenticating users with your corporate single-sign-on system then issuing them a temporary credential?
And presumably you've got some audit logs, so you know who connected to what, when and why. Perhaps a familiar command line tool, that makes temporary credential rotation easy for users? Perhaps some paperwork to hand to your SOC2 compliance auditors?
I mean, this is sounding a lot like tailscale ssh, teleport, and suchlike...
Tailscale doesn't need special handling for SSH. You can run your plain old sshd just fine.
Tailscale comes with an additional SSH server, which runs in parallel with your other SSH server. It does use Wireguard keys directly, so effectively you don't need to manage keys.
Additionally, this SSH server is implemented in userspace, so it won't (can't) interfere with anything else on your system (like your other sshd).
The 'tailscale ssh' command is an optional wrapper around the system 'ssh'
command that's useful in some cases. Tailscale SSH does not require its use;
most users running the Tailscale SSH server will prefer to just use the normal
'ssh' command or their normal SSH client.
The 'tailscale ssh' wrapper adds a few things:
* It resolves the destination server name in its arguments using MagicDNS,
even if --accept-dns=false.
* It works in userspace-networking mode, by supplying a ProxyCommand to the
system 'ssh' command that connects via a pipe through tailscaled.
* It automatically checks the destination server's SSH host key against the
node's SSH host key as advertised via the Tailscale coordination server.
It is about using SSO and tagging to auth the users.
So for example you could have instances tagged with {users} that cannot ssh to certain boxes, but other users tagged with {admin} that can. All these users can be part of your tailnet.
Bravo to Alex in the embedded video for clearly explaining the benefits in an interesting way without being overly salesy. I get a genuine sense of enthusiasm from him.
It's very different, CF tunnel is a reverse proxy to your service, Tailscale SSH manages authentication to machines inside your VPN. I can't see any resemblance, personally.
First of all, for CF, it manages the authentication part of ssh with SSO support (personally I used github)
Secondly, it has clients (on iOS it is called Cloudflare One) which can act as a VPN service as well. You can access any IP addresses (if you setup the vpc correctly [0]) directly accessible to cloudflared daemons.
My understanding is that only Tailscale is end-to-end-encrypted, though both ensure that all traffic is encrypted on the wire. I don’t claim this as fact because, unlike the Cloudflare docs, Tailscale’s claim (yell) that there is no way for them to decrypt.
I’d be pleased to be proven wrong (and jgrahamc is def ITT) as I use a bunch of CF services already and it would be great to have one less PaaS in my life.
How secure is Tailscale in general, and their new SSH offering?
I assume the crypto is unbreakable for outside parties that just sniff the traffic along the way.
But what if Tailscale gets hacked? Are my keys available there for someone else to connect into my network? How hard would it be for the hacker to add their own machine to my network?
About the question about Tailscale being hacked or injecting hosts in your network. With the default configuration yes. There is Tailscale lock https://tailscale.com/kb/1226/tailnet-lock In which you need to sign the devices participating in your network. The signing keys are in your device.
Not directly related to this, but I'm trying to migrate to Tailscale now from OpenVPN, and it doesn't seem like there's a way to use one "account" (for example, Google auth) for multiple tailnets. Our use case is to have the user be able to select whether they want to connect to staging or prod.
Are people just doing this with time-based ACLs within the same tailnet? Curious if there's something more obvious.
I think you could achieve this if you used GitHub for auth, create tailnets on two GitHub Orgs and it’ll ask you which one you want to login to with the net being org-name.github.
Hi! Tailscalar here. This is very topical for me! Over the past 3 weeks I've been working with internal stakeholders to remove our SSO tax - the sso tax is a pet hate of mine. A couple of weeks ago we removed it from our pricing plan after my proposal was approved, and today I released a blog on our website to announce it more widely: https://tailscale.com/blog/sso-tax-cut
I knew of https://sso.tax (which we are not listed on but I did include in my blog), but didn't know there was another website too!
I don't get what's wrong with charging more for SSO? They're in the business of making money, and if you need SSO and you need their service you're more likely to have money. It's nothing to do with the feature itself.
The problem is we in the software industry are so used to getting handed everything for free that a lot of software developers don't even know how to get their employer to buy something.
Every engineer in every other field of engineering knows exactly how to order things, and has access to a properly organised budget for doing so. You need electronic components? Custom-machined parts? Raw material like metal sheets? Nuts and bolts? Aluminium extrusion? Of course you pay for it - and the organisation's purchasing procedure is something you learn in your first week.
We in the software industry, on the other hand, get free operating systems, free compilers, free IDEs, free databases, free libraries, free documentation and training, free support - pretty much free everything. So you can get surprisingly far in the industry without ever learning the difference between a quote and a proforma invoice - or how to explain to the boss why we should give JetBrains $50k per year in terms she'll understand and approve of.
As such, when the people gifting us and our employers free stuff add a paid tier with features we want, but they charge money for it, it's an almighty inconvenience.
If you're storing passwords for the non-SSO option then you're hurting yourself as well as your customers, because storing passwords is a massive security and operational headache. If your limited SSO options, before charging the SSO tax, are "social" (e.g. Google) logins, then you're letting BigTech in between you and your customers.
I'm all for price discrimination, but SSO is the exception, because the alternative is too expensive.
1. SaaS providers need it so they don't store your creds. If they don't store them, they can't leak them.
2. You need it because 1.
3. Nobody needs SSO any more :-) Actually you only need OpenID Connect which shows up as "Sign in with" or "Continue with", which -- if coupled with a domain name validation on your canonical user ID -- amounts to most of the same value as the complicated SSO / SAML dance without per customer config. It is less work for a SaaS provider to support sign in with than to make an entire auth chain.
My current recommendation to new SaaS offerings is OIDC plus magic links as a fallback. (Many SaaS go a very long way with just magic links, those plus a domain name in email address check can also tie employees to a company, regardless of the company's IdP.)
All that said, SCIM and group-to-role sync, etc., should be EXTRA. You need extra moving parts, and enterprises with information barrier or other compliance or regulatory obligations are thrilled to pay for this.
The argument for it is that SSO should be treated as a basic security feature. It'd be like if you had to cough up enterprise pricing for the company to keep your passwords hashed instead of in plain text.
You don't have to use the service though! Can't afford it, don't use it. The price to get x service with the features you want is y no matter how you slice it by feature.
The problem is the cost of data breaches is shared. "You" might always pay for security but the number of data breaches that happen indicate that not very one does yet they affect you all the same.
I don't think there is anything wrong about it, but I think it is a good thing to be informed about it, especially for those involved with smaller businesses.
If you dont remember sso was sold as a panacea, and there was a lot of lip service to it. Whether you believe it is or is not matters less than what the industry promised.
So selling this fundamental, foundational security service as a top tier premium value add is disingenuous, and either deceitful or greedy.
And frankly if you're overly cynical about it; it's a lot like streaming in the sense of why am i forced to pay 10 vendors for the same library? There's a data ownership issue, why am i forced to let them own my user data on their cloud unless i pay for the privilege of owning my own data?
And it also seems like its wrong (or outdated?), Given they offer "SSO with any IdP" on ALL plans, including the free (personal) one: https://tailscale.com/pricing
Hi! Tailscalar here. I posted to the OP, but just to flag this is correct. We removed our 'sso tax' a couple weeks ago from our pricing plans, and I released a blog today about it being removed entirely. Extremely happy it's gone!
I can certainly see the value of this feature for some orgs, but it seems little scary to me. With this setup, if an attacker is able to compromise Tailscale and add a key to your tailnet, that person will immediately have access to your network AND shell access to all of your boxes, rather than just network access if you use Tailscale with vanilla ssh.
I also send myself notifications any time a failed or successful SSH login attempt occurs by tailing the ssh service with journalctl. When I last tried Tailscale SSH, it didn’t log anything to journalctl and so my self-notification via journalctl method did not work.
I use journalctl to follow the log. My NixOS module for it is here: https://github.com/heywoodlh/nixos-configs/blob/master/nixos...
If that isn’t clear, let me know and I can send my Ansible example :)
EDIT: To be more precise I set up the monitoring service as a systemd service
that's why tailscale has https://tailscale.com/kb/1226/tailnet-lock
> that person will immediately have access to your network AND shell access to all of your boxes
it would be extremely weird and negligent to deploy Tailscale at a company and not have any ACLs.
ACLs still come down to "just trust tailscale is working as advertised." And while I generally do (I'm a happy user), if that last few years have shown anything, for the vast majority of companies/products, it's not if you get compromised, it's when. Given that I'd prefer to have multiple distinct layers between the internet and my boxes.
Though definitely see the value in terms of being able to easily tie SSH access to ACLs and your SSO provider -- as with all things security it's a trade off between ease of use and locking everything down.
[1] https://tailscale.com/pricing
[1] https://tailscale.com/kb/1193/tailscale-ssh
Following the adage "if it's for free, you are the product", what is going on behind the scenes? Are they providing their services as a giant honey-pot to sniff on traffic?
> TL;DR: Tailscale’s free plan is free because we keep our scaling costs low relative to typical SaaS companies. We care about privacy, so unlike some other freemium models, you and your data are not the product. Rather, increased word-of-mouth from free plans sells the more valuable corporate plans. I know, it sounds too good to be true. Let’s see some details.
So it's a weighed choice between "if something seems like it's too good to be true, it often is", and "the explanations they give make good sense, and it's a way of doing business that some ethical company could choose to take".
We probably won't know in the short-to-medium term, so we'll have to take their word for it..
But I must admit, their products look pretty impressive. I'll have to have a closer look at them.
Ah, interesting, thanks. That would indeed make it a lot less costly. I would need to dive into it to get a better understanding how their service works.
Would you happen to have some good resources you found useful?
One of my former gigs used a competitor (also a startup) that was 2x the price and offered way less flexible setup but our “architect” rushed it bc of the way he was. Other more established competitors were even more expensive
Tailscale SSH is quite interesting because it'd require Tailscale authentication. So it would segment SSH access off, and makes SSH access also generally available to all clients utilizing Tailscale, regardless of host OS.
I checked, and it doesn't seem like Headscale supports SSH access.
Tailscale gives you authorized connectivity between hosts, and DNS; won't it be sufficient to run plain sshd?
(If wireguard-key-level auth were sufficient, even rlogin or netcat would be enough, because the transport is encrypted already.)
It means you can configure/activate/deactivate peoples access centrally too.
But now I need to trust these random host keys, instead of a key signed by my SSH CA...
Anything signed by the SSH CA will work for logins.
To deal with the “untrust” issue it’s normal for operations with an SSH CA to rely on (very) short-lived certificates, meaning often issued and valid for < 24 hours (it’s configurable, I’ve seen this be as short as 30 minutes).
Smallstep wrote a summary here which is pretty good —
https://smallstep.com/blog/use-ssh-certificates/
So you want a way to get rid of long-lived SSH certificates, instead authenticating users with your corporate single-sign-on system then issuing them a temporary credential?
And presumably you've got some audit logs, so you know who connected to what, when and why. Perhaps a familiar command line tool, that makes temporary credential rotation easy for users? Perhaps some paperwork to hand to your SOC2 compliance auditors?
I mean, this is sounding a lot like tailscale ssh, teleport, and suchlike...
Tailscale comes with an additional SSH server, which runs in parallel with your other SSH server. It does use Wireguard keys directly, so effectively you don't need to manage keys.
Additionally, this SSH server is implemented in userspace, so it won't (can't) interfere with anything else on your system (like your other sshd).
SSH to a Tailscale machine
USAGE tailscale ssh [user@]<host> [args...]
The 'tailscale ssh' command is an optional wrapper around the system 'ssh' command that's useful in some cases. Tailscale SSH does not require its use; most users running the Tailscale SSH server will prefer to just use the normal 'ssh' command or their normal SSH client.
The 'tailscale ssh' wrapper adds a few things:
* It resolves the destination server name in its arguments using MagicDNS, even if --accept-dns=false.
* It works in userspace-networking mode, by supplying a ProxyCommand to the system 'ssh' command that connects via a pipe through tailscaled.
* It automatically checks the destination server's SSH host key against the node's SSH host key as advertised via the Tailscale coordination server.
So for example you could have instances tagged with {users} that cannot ssh to certain boxes, but other users tagged with {admin} that can. All these users can be part of your tailnet.
- Wireguard keys for authentication and encryption.
- Tags for authorization to run a remote shell.
Highly recommended.
I have been using Cloudflare's cloudflared tunnels. It was great for tunneling ssh traffic behind firewalls. And it starts free.
[0] https://developers.cloudflare.com/cloudflare-one/connections...
The equivalent http(s) side of things from Tailscale would be Tailscale Funnel [0], although it's incomplete since you can't BYO domain to TS Funnel.
In essence, CF Tunnel = Tailscale Funnel w/ BYOD + Tailscale SSH
[0] https://tailscale.com/kb/1223/funnel
> In essence, CF Tunnel = Tailscale Funnel w/ BYOD + _Tailscale SSH_
> CF Tunnel = ... Tailscale SSH
Secondly, it has clients (on iOS it is called Cloudflare One) which can act as a VPN service as well. You can access any IP addresses (if you setup the vpc correctly [0]) directly accessible to cloudflared daemons.
[0] https://developers.cloudflare.com/cloudflare-one/connections...
I’d be pleased to be proven wrong (and jgrahamc is def ITT) as I use a bunch of CF services already and it would be great to have one less PaaS in my life.
I assume the crypto is unbreakable for outside parties that just sniff the traffic along the way.
But what if Tailscale gets hacked? Are my keys available there for someone else to connect into my network? How hard would it be for the hacker to add their own machine to my network?
About the question about Tailscale being hacked or injecting hosts in your network. With the default configuration yes. There is Tailscale lock https://tailscale.com/kb/1226/tailnet-lock In which you need to sign the devices participating in your network. The signing keys are in your device.
If you use Mullvad VPN you need to sign the Mullvad exit nodes.
Just make sure to have more than one signing device in case something happens to your main computer.
if tailscale got completely owned then obviously an attacker can fuck with the keys, which is why they have this feature: https://tailscale.com/kb/1226/tailnet-lock
Are people just doing this with time-based ACLs within the same tailnet? Curious if there's something more obvious.
> This traffic is rerouted to an SSH service inside the Tailscale daemon instead of to your standard SSH server.
Was their sshd code audited? This is a lot of trust to use different sshd imho.
I knew of https://sso.tax (which we are not listed on but I did include in my blog), but didn't know there was another website too!
Respect for pushing that through the org!
Every engineer in every other field of engineering knows exactly how to order things, and has access to a properly organised budget for doing so. You need electronic components? Custom-machined parts? Raw material like metal sheets? Nuts and bolts? Aluminium extrusion? Of course you pay for it - and the organisation's purchasing procedure is something you learn in your first week.
We in the software industry, on the other hand, get free operating systems, free compilers, free IDEs, free databases, free libraries, free documentation and training, free support - pretty much free everything. So you can get surprisingly far in the industry without ever learning the difference between a quote and a proforma invoice - or how to explain to the boss why we should give JetBrains $50k per year in terms she'll understand and approve of.
As such, when the people gifting us and our employers free stuff add a paid tier with features we want, but they charge money for it, it's an almighty inconvenience.
I'm all for price discrimination, but SSO is the exception, because the alternative is too expensive.
1. SaaS providers need it so they don't store your creds. If they don't store them, they can't leak them.
2. You need it because 1.
3. Nobody needs SSO any more :-) Actually you only need OpenID Connect which shows up as "Sign in with" or "Continue with", which -- if coupled with a domain name validation on your canonical user ID -- amounts to most of the same value as the complicated SSO / SAML dance without per customer config. It is less work for a SaaS provider to support sign in with than to make an entire auth chain.
My current recommendation to new SaaS offerings is OIDC plus magic links as a fallback. (Many SaaS go a very long way with just magic links, those plus a domain name in email address check can also tie employees to a company, regardless of the company's IdP.)
All that said, SCIM and group-to-role sync, etc., should be EXTRA. You need extra moving parts, and enterprises with information barrier or other compliance or regulatory obligations are thrilled to pay for this.
So selling this fundamental, foundational security service as a top tier premium value add is disingenuous, and either deceitful or greedy.
And frankly if you're overly cynical about it; it's a lot like streaming in the sense of why am i forced to pay 10 vendors for the same library? There's a data ownership issue, why am i forced to let them own my user data on their cloud unless i pay for the privilege of owning my own data?
Are they talking about something different?