You know, when Linux admins say "don't run as root!" we don't only mean that users should not run as root. Nothing that doesn't absolutely have to should run as root.
And before someone says "but only root can read those files!", please take this opportunity to learn about filesystem ACLs. https://linux.die.net/man/1/setfacl
edit: Also, yes, this would not have fully solved the problem, but it is very likely that the amount of potential harm that could have been caused would have been significantly reduced.
Tangentially, the "disable selinux" as a first step for installing software is incredibly lazy in 2024. There are tools that help you analyze in permissive mode and easily convert the output to the contexts you need.
I agree with this. I'm kind of appalled by the apparent lack of SELinux controls on a firewall that runs Linux under the hood.
If I can run SELinux in enforcing mode on a Gentoo Desktop, any Linux administrator worthy of their job title can with a more enterprise/user friendly Linux distro too.
I just wrote this up elsewhere, but wow, is that Python code awful. It uses string interpolation to build a curl command line. Then it passes that string to a subprocess.Popen(cmd,shell=True). For those who don’t speak Python, that’s saying “run the string passed in here as a shell command exactly as if someone typed it into a bash prompt”. The right thing to to there would have been to create a list of arguments and pass that list in instead of a string, and set shell=False so that the list gets sent straight to the underlying syscall without being parsed and interpreted as a command line, with all the edge cases that brings.
As worryingly, the free, open source Bandit Python static analyzer immediately flagged this code when I tested it locally. If Palo Alto had any kind of static analysis in their build pipeline, this would have failed. If they’re not doing that, what else are they not doing?
When I left PANW in the late 2010s the platform team was still on Python 2.6, I think, and the snippets in this write-up are valid 2.x code. It's very possible that this hasn't been touched in a decade beyond whatever was necessary to get to 3.6, which is itself 2+ years past its shelf date. That's a bigger and slightly more abstract problem than not running a mediocre linter.
Oof. That sounds in dire need of some dev resources.
I disagree with that last bit though. First, I think bandit's more than a "mediocre linter". More importantly, it's a free, open source, CI-friendly app that could be plugged into any reasonably normal build pipeline. It would have caught that. If bandit would have, I'm sure any number of other more advanced tools would too. The equally free ruff checker reports: "S604 Function call with `shell=True` parameter identified, security issue". What I take from that is that there's no reasonable amount of code analysis at all. If they don't even run linters, I'm sure something like mypy is out of the question. Are there unit tests? I don't have confidence of it based on this code alone.
I've tried Bandit on a few codebases and each time it's yielded thousands of false positives and no actual findings. In the best cases it's something like hashlib.sha1, where you should set usedforsecurity=False anyway. That's at least an actionable, if ineffectual, alert. You can make the code clearer, and the warning goes away. But in the worst cases it's something like having a variable named "token" (You idiot! Is that a password in your source?), or constructing SQL queries, which by design may never accept user input but, you know, Bandit can't determine that statically, so it goes ahead and throws an absolute fit anyway.
Conversely, I apply Pylint and mypy aggressively, because what they find almost always should be changed. But Bandit's findings are overwhelmingly wrong in context, so all you can do is litter your code with ignore directives. For reference, in my current codebase of 175k lines, there are 52 Pylint and 66 mypy ignores (with mypy at max strictness). Bandit would need 1,823.
Palo Alto absolutely should do better, but Bandit, or probably any "security" linter, isn't the answer.
I've had a vastly different experience. Filtering only high level and high confidence findings can get rid of lots of chaff.
But regardless, even ruff looks for that specific setting. And even if there were only 2 real findings out of 1,823, if it's in a product that you're selling to enterprise security customers, there's not an excuse for not having had someone look at the results of bandit or something like it and ruling out the false positives. They have $89B worth of reputation to lose with things like this.
I think you're making a base rate error here. To reconfigure your example, if Palo Alto themselves blocked 1,821 good files for every 2 bad files they caught, we wouldn't be having this conversation. Nobody would buy that product. The company wouldn't exist.
Enterprise security is a game that balances protecting the business against letting people get work done. IT giveth, and SECURITY taketh away. If all you care about is the security part, then you can just turn all the computers off and send everyone home. But nobody actually wants that.
To be clear, the very specific case I'm making is:
1) Against Bandit, at a pretty fundamental level, given the kind of tool it is. On the rare occasions I have to silence mypy, it's because there's a bug in mypy. On the frequent occasions I'd have to silence Bandit, it's because that's the nature of Bandit. Ruff is different.
2) That Palo Alto needs a pretty serious shakeup of its engineering culture. I think we agree on this part.
Couldnt have put it better myself. The sheer incompetence is staggering and frankly it's depressing that these giant corporations are so good at selling their steaming piles of shit to companies who should really know better.
If PAN-OS was open source then no one would want to be within 10,000 miles of it!
If it were open source, I could imagine sending some PRs there way, wishing them well, then running away to a safe distance. That this is the output of an $89B security company is mind boggling.
The combination of multiple bad behavior strung together is really interesting. I imagined the author of each bad behavior was unaware of the other bad behaviors and was basically like "what's the worst they can do?"
I feel a lot of good exploits work this way where each small bad behavior can be leverages to magnify another.
The funny thing is that this isn't the first time PAN-OS is susceptible to RCE because of "arbitrary empty file creation" (directory in this case): https://seclists.org/fulldisclosure/2017/Dec/38
Really? I'd think truly arbitrary — root access to / — writes would be an RCE. Could you not just write out a cronjob, and wait for it to execute? Or change any common binary that would easily get invoked to do some "extra" work, etc.
like most products connected to the internet, VPNs and firewalls can have vulnerabilities too. As long as its not super low hanging fruit that should have been caught or is a bad default configuration, its the response that matters the most to me.
> like most products connected to the internet, VPNs and firewalls can have vulnerabilities too.
Except being exposed without protection to the hostile internet is their very reason to exist. I expect such devices to be developed with the awareness that they are going to be attacked by capable adversaries. Thus secure coding practices are to be followed and state of the art exploit mitigation techniques employed on the OS level. There's no excuse for high severity vulnerabilities to be found in these kinds of products.
take a look at the history of pwn2own. It could be said that google chrome is one of the most actively developed projects period and people still find crazy 0 interaction exploits for this product.
You just need to threat model around that and limit exposure in case something like this happens. Most orgs dont get hit with 0 days. Therefore the best thing you can do is patch often, when possible.
You ought to adjust your expectations because they don't line up with reality. There is nothing particularly surprising or noteworthy about this vulnerability.
But it's the exploit of the week! How can you not be excited about shaming the company for this amazing fuck up!
Positive feedback for is completely missing from the field. Instead security professionals mock developers and celebrate their human errors. This blog post is a good example of that. Everyone has vulnerabilities and security oopsies in their processes. Instead of cataloging individual vulnerabilities it would make sense to catalog the types of responses and claimed process improvements vendors and projects have.
A few days ago, a colleague saw a port/service scan:
> User-Agent: Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com
I could maybe see that for license enforcement, or for aggressively alerting users to the vulnerability even in absence of active service contracts.
Could also be someone other than PAN, looking for vulnerable PAN boxes.
Even if they honor that exactly as promised, I can't help imagining that email address being an alias for `audit@pan`. "Hey, who owns 12.34.56.78 that wouldn't want us seeing if they had our software running?"
I would totally believe this is driven mainly by enterprise marketing, to get a foot in the occasional door where they haven't managed to otherwise, especially when they have the tool just sitting around.
And before someone says "but only root can read those files!", please take this opportunity to learn about filesystem ACLs. https://linux.die.net/man/1/setfacl
edit: Also, yes, this would not have fully solved the problem, but it is very likely that the amount of potential harm that could have been caused would have been significantly reduced.
If I can run SELinux in enforcing mode on a Gentoo Desktop, any Linux administrator worthy of their job title can with a more enterprise/user friendly Linux distro too.
As worryingly, the free, open source Bandit Python static analyzer immediately flagged this code when I tested it locally. If Palo Alto had any kind of static analysis in their build pipeline, this would have failed. If they’re not doing that, what else are they not doing?
I disagree with that last bit though. First, I think bandit's more than a "mediocre linter". More importantly, it's a free, open source, CI-friendly app that could be plugged into any reasonably normal build pipeline. It would have caught that. If bandit would have, I'm sure any number of other more advanced tools would too. The equally free ruff checker reports: "S604 Function call with `shell=True` parameter identified, security issue". What I take from that is that there's no reasonable amount of code analysis at all. If they don't even run linters, I'm sure something like mypy is out of the question. Are there unit tests? I don't have confidence of it based on this code alone.
Conversely, I apply Pylint and mypy aggressively, because what they find almost always should be changed. But Bandit's findings are overwhelmingly wrong in context, so all you can do is litter your code with ignore directives. For reference, in my current codebase of 175k lines, there are 52 Pylint and 66 mypy ignores (with mypy at max strictness). Bandit would need 1,823.
Palo Alto absolutely should do better, but Bandit, or probably any "security" linter, isn't the answer.
But regardless, even ruff looks for that specific setting. And even if there were only 2 real findings out of 1,823, if it's in a product that you're selling to enterprise security customers, there's not an excuse for not having had someone look at the results of bandit or something like it and ruling out the false positives. They have $89B worth of reputation to lose with things like this.
Enterprise security is a game that balances protecting the business against letting people get work done. IT giveth, and SECURITY taketh away. If all you care about is the security part, then you can just turn all the computers off and send everyone home. But nobody actually wants that.
To be clear, the very specific case I'm making is:
1) Against Bandit, at a pretty fundamental level, given the kind of tool it is. On the rare occasions I have to silence mypy, it's because there's a bug in mypy. On the frequent occasions I'd have to silence Bandit, it's because that's the nature of Bandit. Ruff is different.
2) That Palo Alto needs a pretty serious shakeup of its engineering culture. I think we agree on this part.
If PAN-OS was open source then no one would want to be within 10,000 miles of it!
I feel a lot of good exploits work this way where each small bad behavior can be leverages to magnify another.
Usually arb file create bugs with no control of content are at best a DoS unless there’s another issue at play :)
This is about turning an empty file creation (0 bytes) or a directory creation into code execution, via buggy cron scripts etc that process filenames.
Except being exposed without protection to the hostile internet is their very reason to exist. I expect such devices to be developed with the awareness that they are going to be attacked by capable adversaries. Thus secure coding practices are to be followed and state of the art exploit mitigation techniques employed on the OS level. There's no excuse for high severity vulnerabilities to be found in these kinds of products.
You just need to threat model around that and limit exposure in case something like this happens. Most orgs dont get hit with 0 days. Therefore the best thing you can do is patch often, when possible.
Positive feedback for is completely missing from the field. Instead security professionals mock developers and celebrate their human errors. This blog post is a good example of that. Everyone has vulnerabilities and security oopsies in their processes. Instead of cataloging individual vulnerabilities it would make sense to catalog the types of responses and claimed process improvements vendors and projects have.
> User-Agent: Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com
I could maybe see that for license enforcement, or for aggressively alerting users to the vulnerability even in absence of active service contracts.
Could also be someone other than PAN, looking for vulnerable PAN boxes.
https://www.paloaltonetworks.com/cortex/cortex-xpanse
Palo Alto Networks PAN-OS Zero-Day Exploitation - https://news.ycombinator.com/item?id=40016985 - April 2024 (59 comments)