I used this years ago to implement SSL for a bunch of customer vanity domains, and as far as I know it's still operating.
The best thing is it's declarative: you provide a text file with domains you want in each certificate, and it makes it happen.
Certbot has discrete commands you have to run to add, remove, or modify certificates, so you have to keep track of state. We had a management UI to control these domains, so it didn't work well with something like terraform (which I think otherwise solves the state problem for certbot). Instead we just wrote out the domain file based on db config changing + daily, and ran dehydrated. If there were no certificate changes or renewals needed, nothing would happen.
Also the hooks are great, iirc we had a pre-check in place for our own http-01 validation, so as not to cause failures on let's encrypt. (Mostly customers would cname domains to us, but lots could go wrong). We were also using s3 to store the validation files (again, super easy with bash hooks) to make it work across a pool of load balancers.
This is a great example of a tool where a focus on simplicity (config, hooks, and the way it stores everything) creates an incredibly easy to use, flexible (and thus powerful) system.
ZeroSSL are the only ones providing certificates for IP addresses and free year-long certificates. Cudos to them for disrupting this market, almost monopolized by letsencrypt. Don't have high hopes, though, big players probably will kill them as they killed other free certificate issuers. For some reason letsencrypt status-quo as the only free certificate issuer benefits big players.
In my experience Python software is not really nice when it comes to distribution. Hate the venv stuff and pip install hassles. Besides, it is slow. Honestly, if runtime interpretation is a requirement, PHP would be better then in these respects.
You can do letsencrypt stuff using python standard library, without touching pip and virtualenv. This is my own weekly cron script, unchanged since 2020, with openssl(which is usually preinstalled) and acme_tiny.py with system python. You can easily check what the script is doing by adding set -x and acme_tiny.py is quite readable too. Unlike the monstrosity above.
The point of using '#!/usr/bin/env python3' is that you don't have to mess with virtual environments.
Using this exact shebang line signals "This script is intended to be executed directly (not a python module) and it only needs the python3 standard library."
I would not trust that script.
If Python is such a hassle then there is also Lego: https://github.com/go-acme/lego which supports tons of DNS providers and is very well documented.
Interesting how that was achieved. Most popular Linux distributions are very hard to install without python3 an perl5. LSB https://refspecs.linuxfoundation.org/lsb.shtml defines them as standard.
Maybe not so hard as you might think. On my openSUSE desktop, uninstalling python3 would remove only calibre and virt-manager. Most of my server/container/vm installs don't have any python version. Perl isn't much harder to get rid of depending on the use case, for example some development stuff uses it.
I'm also not sure distros generally care about LSB. Debian dropped it years ago and did any distro even bother to implement it correctly?
I don't get all these arguments. You would really prefer to use hard-to-check shell script to simply installing system python package if it isn't installed already?
Because it allows for scripting infrastructure with far fewer obfuscated hacks than bash/awk. And if you use system python, vulnerabilities are more likely to get patched by distribution vendor. Your homebrewn json parsing regexes won't.
I don't think you really have any idea what you're talking about, to be honest - there are millions of RHEL 7 and CentOS 7 boxes in production right now, and many of them are serving websites and could benefit from SSL certificates.
I know of a couple large organizations that have tens of thousands of RHEL 7 machines that use letsencrypt for customer websites, and I sent this project to sysadmins on their teams, so it's very realistic to expect that businesses are still using supported software.
Not everyone can just upgrade to the latest version of Ubuntu every 6 months for their production workloads like you might expect.
systemd socket activation opens extra fds starting from fd 3 (SD_LISTEN_FDS_START). Though I can't think of a scenario where this may be relevant for dehydrated. Maybe some weird systemd unit somewhere that invoke dehydrated does this?
The only joke here is Certbot, a tool intended for use on servers but distributed on a channel intended for consumer use - Snap. The only package manager that will update on its own, when it feels like it, without a killswitch, runs as a daemon intertwined with the system.
Icing on the cake is that the Certbot team advertises alternative install methods, of which none work and all of the lengthy guides for them recommend to use Snap instead. It’s an insult for professionals.
I'm a simple man, I see a comment complaining about snap and I automatically upvote.
It's 2024. If you're still distributing snaps as the preferred method of install, you're alienating your users. AppImage, Flatpak, and regular containers are all far better deployment options.
> If you're still distributing snaps as the preferred method of install, you're alienating your users.
Dropped lxd because of it. Now I hear they're a proper debian package, but that decision was so weird (especially the auto-update part) I don't trust them anymore.
So ridiculous to have to install yet another package manager, snapd, to get certbot installed "the easy way".
The alternative to get around snapd works fine (IMHO, for my situation, a lightsail instance running amazon linux 2023 where installing snapd is a pain in the ass), but you still have to jump through some hoops.
> Partial support
> The Certbot team supports this installation method on a best effort basis. If you are on a more obscure or heavily customized system, these instructions may not work and the Certbot team may be unable to help you resolve the problem.
This is an instant no-go for any professional environment I ever worked in.
I wouldn’t call it a problem, but let’s hear what the Certbot docs have to say:
> this mode of operation is unable to install certificates or configure your webserver, because our installer plugins cannot reach your webserver from inside the Docker container.
>
> Most users should use the instructions at certbot.eff.org. You should only use Docker if you are sure you know what you are doing and have a good reason to do so.
These problems are solvable if you know what you do, but the whole premise of ACME was making it easier to obtain certificates; plus, I shouldn’t need to decide between an autonomous and hostile package manager or keeping a container environment running, secure, and configured - to set up bloody TLS certificates for a Webserver.
That said, good for you if it works :)
I mount webserver docroot and /etc/letsencrypt directories into the certbot container, /docroot and /etc/letsencrypt mount points respectively, it is totally prepared for this.
it's a two line script running periodically. the first line is docker the second is nginx -s reload. As /etc/letsencrypt dir is mounted from nginx into certbot, certbot upgrades that, nginx reload picks them up. Easy as pie.
Certbot is a massive pile of dependencies that you need to care and feed and update. You should not need 100s of megs of nonsense for something that can be done with a shell script.
It doesn't even work well in my experience. ACME.sh is a lot more hassle free, I've always had issues with Certbot's DNS verification.
Basically I'm not avoiding Certbot to make a point, I just think it's inferior for my specific use cases. I don't know about Dehydrated but I also expect it to be BS free.
Yup, I gave up on certbot after trying to understand why I couldn't get the Digital Ocean DNS plugin to work, which is maintained as part of the Certbot repo. (This was a few years ago, so details may have changed.)
I discovered the issue was that the plugin does some pretty broad-brush guesswork about which domain in your DNS hosting it should actually populate with the response value. If you own a bunch of similar domain names (as many orgs do), the plugin may guess wrong.
Much happier to be using dehydrated, and I don't regard it as a major impediment that I had to spend 10 minutes hand writing the necessary API call to the DNS provider.
Brainfart from my side. When I first installed dehydrated.sh it was originally named letsencrypt.sh (not acme.sh) and later renamed to dehydrated. That was quite a while ago.
I am, yes. Alpine is not full-source-bootstrapped, often imports and trusts external binaries blindly, has no signed commits, no signed reviews, no signed packages, and is not reproducible. It is one phished git account away from a major supply chain attack any day now.
Alpine chooses low security for low contribution friction. It is the Wikipedia of Linux distros, which granted it a huge package repository fantastic for experimental use and reference, but it is not something sane to blindly trust the latest packages of in production.
It is one of the reasons why I made stagex, which in most cases is a near drop-in replacement.
People are always talking like shell scripts are without dependencies. Shell scripting ist mostly piping program outputs to different programs, that have dependencies on libraries themselves. Each of them have to be kept up to date to stay secure. Just because they are mostly preinstalled on the system doesn't make them not a dependency.
This, together with the fact that most shell scripting is bash based which is, in my opinion, not a very good language, makes it less secure to me than a python tool.
If it is just a few curl and openssl commands, why make a user install hundreds of megs of python deps just to ultimately call mostly openssl commands anyway
One of the biggest risks today is supply chain attacks. The more dependencies you have, the more people you are giving the ability to tamper with your critical code paths.
The only things that are a bit heavy in the list are openssl and curl. Still, they are relatively self-contained tools, nothing to do with the dependency hell of certbot.
Wouldn't this bash script be just as susceptible to supply chain attacks? What dependencies does openssl, cURL, sed, grep, awk, mktemp pull in, and could it be vulnerable to attack like we saw with xz?
I think the point is to minimize the total attack surface on a system. If the bash script only depends on programs like those you list, which Linux distros package as default anyway, the shell script has not introduced any new dependencies on your system.
> Wouldn't this bash script be just as susceptible to supply chain attacks? What dependencies does openssl, cURL, sed, grep, awk, mktemp pull in, and could it be vulnerable to attack like we saw with xz?
All the packages listed are already probably installed on your system, so you have to worry about their integrity already (your system package manager (RPM, Deb) probably leverages them).
Something like Certbot pulls in dependencies on top of what your system already has, whereas Dehydrated or Acme.sh use tools that you already have to worry about anyway because they're part of the base OS.
I’m pretty sure every open source repository ever created is vulnerable to “an attack like we saw with xz,” and by that I mean a labyrinthine, multi-year effort involving highly sophisticated exploits combined with psychological manipulation, coordinated sock-puppets, global actors, and deep intrigue—to the extent that it’s absurd to cite as a threat that “dependency management” can meaningfully prevent.
I actually full source bootstrapped a Linux distribution from zero recently. Most distros have a lot of dependencies you do not need.
In this case you only actually need curl, openssl, and busybox. Those depend on at least a small libc implementation and a kernel but those are certainly already present.
Coreutils or busybox is probably already installed too. Still, I will grant the requirement of musl and a linux kernel since we are being pedantic or maybe talking about an embedded linux use case, so 5 deps total to boot from metal and get a cert.
To be fair I would never actually ship openssl or busybox in a real embedded project. Would probably write a simple standalone binary using the standard library of Go or something.
How is that in any way realistic? Most people don't bootstrap Linux systems. And just because it is pre-installed doesn't make it safe. Most software on a modern Linux distribution is dynamically linked to several libraries which might depend on other libraries.
One of the biggest benefits of dehydrated is that it doesn't try to integrate with a DNS provider on its own. It just connects to ACME server and calls a hook, which can be implemented with a simple shell script[1]. The most popular third-party integration is lexicon[2], though you're not required to use Lexicon. (e.g. you're free to use awscli, gcloud, linode-cli, etc. to do the actual DNS record manipulation)[3]
This means its dependencies footprint is much smaller, and allows you to do things that can be a nightmare to configure with Certbot or other alternatives. For example, at one of the scenarios I had to set up was that we had to query a credential via HashiCorp Vault, which is then used to cURL into an API endpoint. The shell script in total was pretty short (~200 LOC) and it worked extremely well. The fact the shell script is so simple that I could test adding/removing records without ever invoking ACME process is also a huge benefit.
I created a k8s deployment which spins up nginx in a job to update TLS secrets with dehydrated because I couldn’t get an admin to install cert-manager. It works great, especially because I had scheduled rebuilds of the base container off of latest images. I left the job 3 years ago, and it’s still running fine and staying ahead of the vuln scanners.
I had also used a pre-shared private key, which I put on a F5 BigIp, and just scheduled a job on the F5 to pull the updated certs daily.
I think the amount of alternative ACME clients is a testament to the absolute mess that certbot is.
Constant breakage unless you're using snaps on the most popular distributions. The CLI is absolutely idiotic and convoluted and the plugins do a lot of guess work without informing you, which results in some fun debugging.
It's also a large pile of dependencies that take up space.
How does dehydrated compare to nginx-proxy with acme-companion from an operational perspective? I've been looking at other options and the best so far seems to be Caddy.
Was my default way of updating my certs until I switched to using an Nginx proxy docker image that handled it for me automatically. Still highly recommend this tool.
I am not familiar with zsh, but is it really interpreted by zsh? Because the script has #!/usr/bin/env bash in its shebang, isn't it executed by bash on your system, even if launched from zsh?
This script is explicitly a Bash script and it is not executable by every other shell present on modern unix-like systems. Examples are Korn shell, Almquist shell. Hence the distinction: if one states that the script is a shell-script, it implies that it can be interpreted by any modern shell, for which there is only one common denominator, POSIX. This script is explicitly only Bash shell compatible, not any-shell compatible.
Sure. Now look at the dependencies that are installed compared to dependencies that are installed for dehydrated (or acme.sh, etc) which generally are: bash, OpenSSL, cURL. This is very handy for more appliance-like system (I ran dehydrated on (RH-based) F5s for years before ACME was put into the GUI; also ran it on (FreeBSD-based) Isilons.)
Also, if you want to do an audit of the code, how many lines of Python need to be examined (including dependencies) compared to how many lines of Bash? (Both would have common dependencies like (Open)SSL and HTTP/cURL libraries.) As we saw with the recent XZ kerfuffle, 'software supply chains' are becoming important.
Personally I find it much easier to understand / configure dehydrated:
The best thing is it's declarative: you provide a text file with domains you want in each certificate, and it makes it happen.
Certbot has discrete commands you have to run to add, remove, or modify certificates, so you have to keep track of state. We had a management UI to control these domains, so it didn't work well with something like terraform (which I think otherwise solves the state problem for certbot). Instead we just wrote out the domain file based on db config changing + daily, and ran dehydrated. If there were no certificate changes or renewals needed, nothing would happen.
Also the hooks are great, iirc we had a pre-check in place for our own http-01 validation, so as not to cause failures on let's encrypt. (Mostly customers would cname domains to us, but lots could go wrong). We were also using s3 to store the validation files (again, super easy with bash hooks) to make it work across a pool of load balancers.
This is a great example of a tool where a focus on simplicity (config, hooks, and the way it stores everything) creates an incredibly easy to use, flexible (and thus powerful) system.
They plan to keep the project open source and employ Lukas to continue maintaining it.
https://crt.sh/?q=zerossl.com
filter="$(printf 's/.\[%s\][[:space:]]\([^"]*\)/\\1/p' "$(json_path "${1:-}" "${2:-}")")"
Please, no! How can you know this is safe?
Why not just use Python? It is installed pretty much everywhere, supports json without such hacks.
https://github.com/acmesh-official/acme.sh/issues/4668
#!/bin/sh
set -eu
set -o pipefail
cd /home/letsencrypt
openssl verify -CAfile lets-encrypt-x3-cross-signed.pem -attime `date -d 'next month' '+%s'` mah_domain.crt | grep expired || exit 0
DT=`date "+%Y%m%d"`
python3 acme-tiny/acme_tiny.py --account-key account.key --csr mah_domain.csr --acme-dir /srv/www/mah_domain/htdocs/.well-known/acme-challenge/ > signed_${DT}.crt
curl -sO https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.p...
cat signed_${DT}.crt lets-encrypt-x3-cross-signed.pem > mah_domain_chained_${DT}.pem
mv mah_domain_chained_${DT}.pem mah_domain_chained.pem
mv signed_${DT}.crt mah_domain.crt
set +e
nginx -c /srv/nginx/conf/nginx.conf -s reload
Using this exact shebang line signals "This script is intended to be executed directly (not a python module) and it only needs the python3 standard library."
https://jqlang.github.io/jq/
maybe for dev environments, but certainly not for servers
I'm also not sure distros generally care about LSB. Debian dropped it years ago and did any distro even bother to implement it correctly?
Yes, but which version?
MacOS Sonoma (the latest version) doesn't include Python at all; older versions of MacOS which are still supported ship with Python 2.x
I know of a couple large organizations that have tens of thousands of RHEL 7 machines that use letsencrypt for customer websites, and I sent this project to sysadmins on their teams, so it's very realistic to expect that businesses are still using supported software.
Not everyone can just upgrade to the latest version of Ubuntu every 6 months for their production workloads like you might expect.
> I don't think
> target audience
You != people
Then address this to the owners of those machines. And get a load of cash ready to give it to them, so they could migrate to somehitng newer.
Not the modules, but Python itself.
https://github.com/dehydrated-io/dehydrated/commit/b116e6bc2...
Icing on the cake is that the Certbot team advertises alternative install methods, of which none work and all of the lengthy guides for them recommend to use Snap instead. It’s an insult for professionals.
It's 2024. If you're still distributing snaps as the preferred method of install, you're alienating your users. AppImage, Flatpak, and regular containers are all far better deployment options.
Dropped lxd because of it. Now I hear they're a proper debian package, but that decision was so weird (especially the auto-update part) I don't trust them anymore.
So ridiculous to have to install yet another package manager, snapd, to get certbot installed "the easy way".
The alternative to get around snapd works fine (IMHO, for my situation, a lightsail instance running amazon linux 2023 where installing snapd is a pain in the ass), but you still have to jump through some hoops.
I've only been using it for a little while, but the pip + venv method seems to work decently well: https://certbot.eff.org/instructions?ws=other&os=pip
> Partial support > The Certbot team supports this installation method on a best effort basis. If you are on a more obscure or heavily customized system, these instructions may not work and the Certbot team may be unable to help you resolve the problem.
This is an instant no-go for any professional environment I ever worked in.
Is there a problem with that?
> this mode of operation is unable to install certificates or configure your webserver, because our installer plugins cannot reach your webserver from inside the Docker container. > > Most users should use the instructions at certbot.eff.org. You should only use Docker if you are sure you know what you are doing and have a good reason to do so.
These problems are solvable if you know what you do, but the whole premise of ACME was making it easier to obtain certificates; plus, I shouldn’t need to decide between an autonomous and hostile package manager or keeping a container environment running, secure, and configured - to set up bloody TLS certificates for a Webserver. That said, good for you if it works :)
Basically I'm not avoiding Certbot to make a point, I just think it's inferior for my specific use cases. I don't know about Dehydrated but I also expect it to be BS free.
I discovered the issue was that the plugin does some pretty broad-brush guesswork about which domain in your DNS hosting it should actually populate with the response value. If you own a bunch of similar domain names (as many orgs do), the plugin may guess wrong.
Much happier to be using dehydrated, and I don't regard it as a major impediment that I had to spend 10 minutes hand writing the necessary API call to the DNS provider.
https://github.com/acmesh-official/acme.sh
https://github.com/dehydrated-io/dehydrated
Also their official builds are built with Alpine which is a hobby distro that does not even do signed code or packages.
Alpine chooses low security for low contribution friction. It is the Wikipedia of Linux distros, which granted it a huge package repository fantastic for experimental use and reference, but it is not something sane to blindly trust the latest packages of in production.
It is one of the reasons why I made stagex, which in most cases is a near drop-in replacement.
https://codeberg.org/stagex/stagex
EDIT: Also, stagex looks pretty compelling; I hope it catches on!
This, together with the fact that most shell scripting is bash based which is, in my opinion, not a very good language, makes it less secure to me than a python tool.
One of the biggest risks today is supply chain attacks. The more dependencies you have, the more people you are giving the ability to tamper with your critical code paths.
awk, grep, curl, getent, sudo, mkdir, mv, ln, cat, rm, openssl, touch
Most of these have around 10 dependencies on libs.
Don't misunderstand me. I have nothing against the script. I just don't like the argument that bash is better because it has no dependencies.
The only things that are a bit heavy in the list are openssl and curl. Still, they are relatively self-contained tools, nothing to do with the dependency hell of certbot.
All the packages listed are already probably installed on your system, so you have to worry about their integrity already (your system package manager (RPM, Deb) probably leverages them).
Something like Certbot pulls in dependencies on top of what your system already has, whereas Dehydrated or Acme.sh use tools that you already have to worry about anyway because they're part of the base OS.
In this case you only actually need curl, openssl, and busybox. Those depend on at least a small libc implementation and a kernel but those are certainly already present.
Coreutils or busybox is probably already installed too. Still, I will grant the requirement of musl and a linux kernel since we are being pedantic or maybe talking about an embedded linux use case, so 5 deps total to boot from metal and get a cert.
To be fair I would never actually ship openssl or busybox in a real embedded project. Would probably write a simple standalone binary using the standard library of Go or something.
This means its dependencies footprint is much smaller, and allows you to do things that can be a nightmare to configure with Certbot or other alternatives. For example, at one of the scenarios I had to set up was that we had to query a credential via HashiCorp Vault, which is then used to cURL into an API endpoint. The shell script in total was pretty short (~200 LOC) and it worked extremely well. The fact the shell script is so simple that I could test adding/removing records without ever invoking ACME process is also a huge benefit.
[1]: https://github.com/dehydrated-io/dehydrated/blob/master/docs...
[2]: https://github.com/AnalogJ/lexicon
[3]: https://github.com/dehydrated-io/dehydrated/wiki
I had also used a pre-shared private key, which I put on a F5 BigIp, and just scheduled a job on the F5 to pull the updated certs daily.
Constant breakage unless you're using snaps on the most popular distributions. The CLI is absolutely idiotic and convoluted and the plugins do a lot of guess work without informing you, which results in some fun debugging. It's also a large pile of dependencies that take up space.
Why would POSIX be implied in any way? I mainly use windows, should I have been upset because I thought cmd or powershell was implied?
They seem to have gone through some effort for this to be true.
Sure. Now look at the dependencies that are installed compared to dependencies that are installed for dehydrated (or acme.sh, etc) which generally are: bash, OpenSSL, cURL. This is very handy for more appliance-like system (I ran dehydrated on (RH-based) F5s for years before ACME was put into the GUI; also ran it on (FreeBSD-based) Isilons.)
Also, if you want to do an audit of the code, how many lines of Python need to be examined (including dependencies) compared to how many lines of Bash? (Both would have common dependencies like (Open)SSL and HTTP/cURL libraries.) As we saw with the recent XZ kerfuffle, 'software supply chains' are becoming important.
Personally I find it much easier to understand / configure dehydrated:
(Ubuntu/Debian has packages for it as well.)Also not everyone wants to install few dozens of packages just for this little thing.
For the rest, I'll indeed look into lighter alternatives some day. But the setup works on its own so I'm a bit lazy.
There’s this misconception that you need extra packages to get certificates working with different software.