JavaScript for Shell Scripting

(github.com)

385 points | by gigel82 1056 days ago

38 comments

  • pwdisswordfish8 1056 days ago
    I hope the $ interpolator at least performs escaping instead of blindly concatenating shell scripts?

    Ah, who am I kidding.

    https://github.com/google/zx/blob/5ba6b775c4c589ecf81a41dfc9...

        function substitute(arg) {
          if (arg instanceof ProcessOutput) {
            return arg.stdout.replace(/\n$/, '')
          }
          return arg
        }
    
        // […]
    
        let cmd = pieces[0], i = 0
        for (; i < args.length; i++) cmd += substitute(args[i]) + pieces[i + 1]
        for (++i; i < pieces.length; i++) cmd += pieces[i]
    
    Sigh… yet another code injection vector. And to think the whole template literal syntax was specifically designed with this in mind. Have people not learned from 20 years of SQL injection?
    • akst 1056 days ago
      This feels like a pretty dramatic response when this is already a problem with child_process.exec and most shell scripting languages in the first place.

      I’m not sure what you’re expecting if you’re taking arbitrary IO output to build up a process with arguments separated by spaces. A lot of things had to go wrong before you get to this point.

      If we’re being realistic here, this library’s likely intended use is for this is smallish scripts anyways… not large pieces of software that are creating commands on the fly to from arbitrary IO

      • pwdisswordfish8 1056 days ago
        Ordinary variable substitution in shells splits on spaces, which is still bad, but at least doesn’t immediately lead to arbitrary code execution. I’m expecting at the very least an equivalent of Python’s shlex.quote. This is supposed to be an improvement on the status quo, not a regression.
      • goshx 1056 days ago
        > If we’re being realistic here, this library’s likely intended use is for this is smallish scripts anyways… not large pieces of software that are creating commands on the fly to from arbitrary IO

        If we're actually being realistic here, we know users will use this for whatever scenario, regardless of the author's intent.

      • seniorsassycat 1056 days ago
        You should use child_process.execFile or the execve equiv in your language. Shell has expansion issues but rigorous quoting helped by shellcheck make it safe.

        And zx's `$` could make better use of tagged template literals.

        Something like this, tho it isn't correct

            function $(strings, ...args) {
              const cmd = [];
              for (const part of strings) {
                cmd.push(...part.split(/\s+/));
                if (args.length > 0) {
                  cmd.push(args.shift());
                }
              }
              return child_process.execFile(cmd[0], cmd.slice(1));
            }
        
        
        https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
      • lhorie 1056 days ago
        But with child_process.exec you can at least pass values via env and have the shell script come from a file (which you can throw shellcheck at)

        Also, spawning node from shell to spawn shell to spawn something like ls is madness. Node has fs.readdir already and there are util packs on NPM like fs-extra and friends.

    • lobstrosity420 1056 days ago
      Escaping in the $ interpolator is a planned feature. You definitely have a point, but the project was made public 4 days ago and deserves a little bit of slack don't you think?
      • pwdisswordfish8 1056 days ago
        Injection vulnerabilities are one of the most pervasive and well-known kinds of bugs, especially in shell scripting. This should have been considered on day 1 and implemented before anything else started depending on it being otherwise. Fixing bad designs after the fact is long and painful. Just look at PHP.
        • lobstrosity420 1056 days ago
          We are talking pre alpha stages of development here, nothing that depends on this can reasonably expect for the API to not break several times over in the near future. This software is still figuratively on it's day 1 of development. Weather you or someone else would have implemented this feature faster matters very little to me.
      • cutemonster 1056 days ago
        Looks nice I think :-)

        I wonder if it can work with Deno, and maybe even with Deno's security features: https://deno.land/manual/getting_started/permissions

      • pvtmert 1055 days ago
        These stuff generally called as "0-day".
      • maxgee 1056 days ago
        no?
    • jiofih 1056 days ago
      are you bringing in user input from a web form into your shell scripts?
      • jerf 1056 days ago
        "are you bringing in user input from a web form into your shell scripts?"

        This is shell scripting. We have literally decades of experience with these. We know for a positive fact that simple concatenation is dangerous. The contents of files, the names of files on the file system, and all the other things that shell scripts normally encounter are perfectly sufficient to wreck your day if you are too casual about them. Should you reply with something incredulous about this claim, be prepared for dozens of people to jump in with their war stories about how a single space in a filename in an unexpected place cost their business millions of dollars because of their shell script screwup. (Obviously, they are not that consequential on average, but the outliers are pretty rough in this case.)

        Shell scripting is frankly dangerous enough just with what is on your system already; actually hooking up user input to it is borderline insane. Shell scripting would have to level up quite a bit to only be something to be concerned about when "user input" was being fed into it.

        Learning to do half-decent shell scripting in most shells consists about half of learning the correct way to do things like pass through arguments from a parent to a child script, because for backwards compatibility reasons, all the easy ways are the wrong ways from the 1970s. It's nice when a more modern take on shell scripting is nicer than that.

        I will also say when I'm evaluating libraries for things like shell scripting, I look for things like this, and it definitely doesn't score any points when I see stuff like this.

        • chrisfinazzo 1056 days ago
          Just using `set -euo pipefail` will prevent many stupid things, but then again, conventional wisdom these days just seems to be to not use a shell if you can help it.

          https://sipb.mit.edu/doc/safe-shell/

          • jerf 1056 days ago
            That's another example of what I mean, learning the magic invocations that amount to "Oh, please shell, act halfway like this is the 21st century, please?" It's like how I still have "#!/usr/bin/bash NL use strict; NL use warnings;" still burned into my fingers for Perl. (AIUI that's obsolete now but I never got to upgrade to the versions where that became obsolete, and now I'm just out of it.)
        • lhorie 1056 days ago
          > We have literally decades of experience with these. We know for a positive fact that simple concatenation is dangerous.

          Yes and no. I agree with the overall premise that the footguns are well documented, but at the same time, projects like this show that there are still large segments of developers who will gleefully shoot themselves in the foot because they never took the time to learn shell, or they just never had the opportunity to earn the battle scars.

          At least Google has a bug bounty program.

      • gnfargbl 1056 days ago
        The developers of a not-insignificant portion of IoT firmware absolutely are bringing user input in from web forms and chucking it into shell scripts. And unfortunately, it's that same class of developers that are disproportionately likely to pick up zx and run with it.

        The point is, at this stage in the evolution of internet security, we pretty much know where the bugs come from. Injection attacks are still a huge practical problem. It would be nice if new scripting languages reduced that attack surface rather than increasing it.

      • pwdisswordfish8 1056 days ago
        Who knows, maybe? Or maybe I just want to process file names with spaces in them? Maybe I don’t want to worry about apostrophes in people’s surnames?
        • dotancohen 1055 days ago

            > apostrophes in people’s surnames
          
          In English, my daughter's given name has an optional apostrophe (Ma'ayan). I've seen systems that escape last-name apostrophes but not in the first name.
        • tutfbhuf 1056 days ago
          How do you deal with it, when you are writing bash scripts?
          • pwdisswordfish8 1056 days ago
            By not using eval (and sh -c, and everything equivalent) unless it’s absolutely unavoidable and always quoting variables. The $ construct acts basically like (the shell’s) eval. Proper escaping is an absolute must here.
            • 40four 1056 days ago
              Indeed, I’m not a bash expert, but I’ve heard multiple times that using eval is a bad idea. Your point is a good reminder of why.
              • TechBro8615 1056 days ago
                I'm pretty sure that "eval is a bad idea" was mentioned in the "thought terminating cliches" topic on Ask HN a few days ago :)

                In bash, `eval` is a footgun like any other. You can use it, but you just need to be aware of where your toes are when it shoots.

                It's usually a "bad idea" in the sense that if you think you need to use it, you probably don't, and 90% of the time, there is an easier way to accomplish what you want to do. The next 5% of the time, using `eval` might be easier but will also create maintenance debt with its overgeneralization. And the final 5% of the time might be actual legit use cases for `eval`.

                I just grepped my codebase for `eval` and I almost never use it. One example of the "overgeneralized" 5% might be when I realized I could use `eval` to set "variable variables" (i.e. the name of the variable is itself a variable, taken from a function argument). It was cool, but I ended up deleting it in favor of a more concrete solution.

                Personally, if I'm hesitating to use `eval`, it's usually not for any security reasons. In general, my bash scripts only exist in dev machines and CI runners, and I don't copy them into the application containers that are exposed to a live runtime environment with untrusted users. So for CI/dev scripts, I can safely assume the code will only run in CI/dev, and therefore I can trust arbitrary user input (which I can of course still validate).

                • 40four 1052 days ago
                  Thanks for the detailed response! I’ve been trying to level up my bash lately, and I definitely have seen a lot of ‘poor’ examples where they bailed out & use eval plus something else, for something native bash can in fact handle just fine, if you dig deep enough!
          • selfhoster11 1056 days ago
            This is usually answered with "poorly", or "with great difficulty".
          • DonHopkins 1056 days ago
            By immediately invoking Python and getting the hell out of bash.
            • Spivak 1056 days ago
              Python is my absolute favorite language but it's not suitable for the kinds of things you would use bash for.

              This is the real code that a Ansible uses to run a shell command correctly and is 350 lines and is still a small subset of the features of a single line of bash. https://github.com/ansible/ansible/blob/a2776443017718f6bbd8...

              The Python code to do what a single mv invocation does is 120 lines https://github.com/ansible/ansible/blob/a2776443017718f6bbd8...

              People always focus on the footguns that exist in Bash the language but ignore how much systems programming is abstracted away from you in the shell environment.

              In Bash you can enter a Linux namespace with a single nsenter invocation. If you want to do the same in Python you have use ctypes and call libc.clone manually.

              • nonameiguess 1056 days ago
                How is that a remotely real comparison? The ansible mv function deals with preserving SELinux context, which mv doesn't do, and it automatically deals with a whole lot of common error conditions, whereas mv just fails. If you just want to replicate mv, Python has shutil.move, one line of code. Ansible is trying to do a lot more.

                By the way, I don't know if this is the canonical implementation, but FreeBSD mv is 481 lines of C: https://github.com/freebsd/freebsd-src/blob/master/bin/mv/mv...

              • BiteCode_dev 1056 days ago
                The code you are showing does a lot more than MV:

                - it's portable accross OSes (and keep flags if the OS supports it, deal with encoding, etc)

                - it ensures selinux context is saved if there is such a thing

                - it has proper and rich error communication with the calling code

                - it's includes documentation and comments

                - it outputs json par parsing and storage

                No to say "mv" is not awesome, because it is. There is much more boiler plate in python, and is why I'll often do subprocess.check_call(['mv', 'src', 'dst']) if my script is linux only.

                But you are pushing it

              • overtomanu 1056 days ago
                i think its not fair comparison. mv command implementation in 'C' might have more lines of code. Maybe we should complain that there are no OOTB library functions in python to move the file.
                • Spivak 1056 days ago
                  I don't really care how many lines of code there are in an implementation. I care how many lines of code I actually have to write.

                  Python has shutil.move and os.rename but the Ansible example is to illustrate that there's a lot of code that needs to surround those calls to make them useful and they're not 1-1.

            • FranchuFranchu 1056 days ago
              Or xonsh
      • judofyr 1056 days ago
        It also means that accidentally adding a space somewhere (“$HOME/go” -> “$HOME /go”) can have catastrophic effect. I wouldn’t dare write a single “rm” if I’m not 100% sure the argument is being quoted.
      • mirekrusin 1056 days ago
        It's called shell microservice, you don't need b/e developers anymore.
    • Siira 1053 days ago
      I wrote essentially the same thing (with proper interpolation, using zsh’s builtin quoting system) for [Python](https://github.com/NightMachinary/brish). I have been using it extensively for months, and I’m very happy with it.

      I have also used a REST API based on the Python version to implement the same thing easily in other languages, including a nifty [elisp macro](https://github.com/NightMachinary/doom.d/blob/master/night-b...) that lets you do:

      (z du -h (split-string (z ls -a) "\n" t))

    • Too 1054 days ago
      Looks like they fixed it now.
  • rattray 1056 days ago
    Honestly the more appealing aspect than the "await $`cmd`" syntax is that it imports a bunch of handy libs by default.

    For some reason there's just something that rubs me the wrong way about having a bunch of requires at the top of a bash script (or cluttering my package.json with handy tools I use once in a script). But you're gonna want things like (promisified) fs, os, chalk, readline, node-fetch, etc quite a lot of the time.

    Definitely wish they'd included shelljs though; almost strange not to.

    I hope they add (or already have) syntax highlighting as .mjs without the file extension (just the shebang) for GitHub, VSCode, etc.

    • tln 1056 days ago
      Currently, your scripts have to use the .mjs syntax to work because zx uses import() to actually execute the script.
  • alpaca128 1056 days ago
    Neat idea, but what's the point of async-await when you just put await before 100% of calls like in the first example? Now you've got more to type for no gain.

    I don't get this popularity of async-await, especially in JS where I find its combination of syntax and absence of pre-run checks overly confusing and error-prone.

    And this, seriously?

        await $`exit 1`
    • eyelidlessness 1056 days ago
      All IO in JS is async, you either have a Promise-based API (which can be sugared with await) or callbacks. The only exceptions are Node APIs which are explicitly synchronous in their names and CommonJS require. All of which are either discouraged or being generationally phased out (CJS->ESM).

      That it extends to the $ function (anything preceding a backtick in JS is a “tagged literal” but also a function call) is just API consistency because it can’t know whether you’re calling exit versus something which actually performs IO. So it always returns a Promise.

      • megous 1056 days ago
        > All IO in JS is async, you either have a Promise-based API (which can be sugared with await) or callbacks.

        No it's not. There are all kind of *Sync functions even in nodejs and in browsers (there was sync variant of XHR), and JS engine generally doesn't care if you block in your functions or not. JS engines don't even have an event loop, that's just a construct within the app that uses the engine, like nodejs or whatever.

        • shepherdjerred 1056 days ago
          In this case it's probably fine to block, but in most others it's absolutely not.
    • CGamesPlay 1056 days ago
      I'm really unclear on what repository you're commenting on. The first example shows an example of executing commands in parallel, and does 3 jobs with 1 await.

        await Promise.all([
          $`sleep 1; echo 1`,
          $`sleep 2; echo 2`,
          $`sleep 3; echo 3`,
        ])
      
      The only `exit 1` in the README is an example on how to handle cases where your job fails, so don't really understand what your complaint is.
      • alpaca128 1056 days ago
        Okay, but why would I do that instead of just

            sleep 1 && echo 1 &
            sleep 2 && echo 2 &
            sleep 3 && echo 3 &
            wait
        
        and be done with it? Now I don't need to prepend 'await $`...`' in front of every other command.
        • Normal_gaussian 1056 days ago
          when I write scripts in js or python over bash, its to leverage their access to data structures, error handling, and libraries. Often I use TypeScript, because types are good and concurrent i/o is a first class feature.

          This means that promise.all probably looks something more like

            const lastWeek = DateTime.now().minus({ days: 7 }).toISOString()
            const pods = JSON.parse(await $`kubectl get pods -o json`)
            useValidate(pods, isKubeCtlPodList)
            await Promise.all(
              pods.filter(pod => !protectedPods.includes(pod.name))
                  .filter(pod => target.test(pod.name))
                  .filter(pod => pod.created < lastWeek)
                  .map(pod => $`kubectl delete pod ${pod.name}`)
            )
          
          Id certainly encapsulte my calls, parsing, and validation differently in real code, but you get the gist.

          In shell I can wrangle jq, date, xargs and get the same result; only this is waay easier to write, gives me validation and usable error messages, and can be altered much more easily than shell.

          I write complex bash scripts somewhat often. They can be quicker to write, they are great for one-offs. But if I'm coming back to something non-trivial or if its getting deployed into production I want a language like js or python (or nearly any other language. C++, OCaml, Java all help more than they hinder).

        • mirekrusin 1056 days ago
          If process in the middle exists with non zero exit code, your whole script won't exit with non zero exit code. You have to collect exit codes and check them.
          • curryst 1055 days ago
            The whole thing will exit if any of those fail because of the double ampersands. Semicolons are the ones that ignore exit codes.

            Things like this are why I don't love bash. Whether a script will fail if a step fails is too important to be hidden behind a single character change in a place most people ignore.

        • jiofih 1056 days ago
          that’s just ignoring all the other benefits, or pitfalls of bash that you have to worry about. This project is not about enabling better parallelism.

          (I’ve written massive bash scripts, and now strongly prefer Ruby / python / whatever is available in the system)

          • blablabla123 1056 days ago
            Also I think you need to add a trap to make sure the processes exit on ^C.
          • alpaca128 1056 days ago
            I agree that for complex scripts bash isn't ideal. But my criticism was about unnecessarily verbose syntax. Bash is synchronous by default, this project is the exact opposite.
        • confiq 1056 days ago
          I totally get your point! As someone who is not coming from JS, I agree with you!

          But you must understand that I/O in JS is async, it's how the language is build! People who are advanced in async programming find this very comfortable.

    • hutrdvnj 1056 days ago
      It depends on how sequential your script is. It's not very uncommon to start a background task, do something in the meantime and continue when the bg task has finished. These types of async control flows are very easy to model in JS.

      And as far as I can see you don't have to await every single statement, because you can do multiple statements await $`echo 1; mkdir test; exit 0`

    • devit 1056 days ago
      Yeah, feels like they should use the Haskell IO monad "do" notation style where the await is implicit unless you use let.

      This would require the scripts to no longer be vanilla JavaScript, but it seems easy to extend sucrase/babel to do that transpiling.

      • efortis 1056 days ago
        In plain JS: pipe(a, b, c) is equivalent to a().then(b).then(c)

          function pipe(...functions) {
            (async () => {
              let accum;
              for (const fn of functions)
                accum = await fn(accum);
            }());
          }
    • Raed667 1056 days ago
      I'll give you 2 examples and tell me which is more readable:

        console.log('start')
        doA()
          .then(() => {
            doB()
             .then(() => {
                doC()
                 .then(() => {
                   console.log('finish')
              })
            })
          })
          .catch((error) => {
            console.log(error)
          })
      
      
      Or

        console.log('start')
        try {
          await doA()
          await doB()
          await doC()
          console.log('finish')
        } catch (error) {
          console.log(error)
        }
      • leipert 1056 days ago
        No need to nest promises:

          console.log('start')
          doA()
            .then(() => doB())
            .then(() => doC())
            .then(() => console.log('finish'))
            .catch((error) => {
              console.log(error)
            })
        • BeefWellington 1056 days ago
          Is this not less typing and as clear?

            console.log('start')
            doA()
              .then(doB)
              .then(doC)
              .then(() => console.log('finish'))
              .catch((error) => {
                console.log(error)
              })
          • mdaniel 1056 days ago
            It's been my experience that doing it that way can fiddle with the `this` value of those B and C functions, although I do in general agree there's usually no need for the outer wrapping arrow function if the interior one doesn't care about `this`

            I believe your .catch can similarly be `.catch(console.log)` for the same reason

        • Raed667 1056 days ago
          i was being a bit facetious. "Real-life" code is usually more complex and nesting is needed when you have conditional logic (call doC or doD depending on the return value of the previous call etc..)
      • bryanrasmussen 1056 days ago
        The complaint really is that if everything is sequential it is a fault in the language to make you explicitly say it every time, not that await is somehow worse than promises.
        • andresgottlieb 1056 days ago
          Is everything sequential?
          • alpaca128 1056 days ago
            When I need to run something async in Bash I can just add a `&`. Done. The majority of my scripts are purely sequential, though, as the language lends itself well to sequential data processing/piping.

            I'm not seeing a benefit by wrapping almost every single command in 'await $`...`'. I get why you'd want to wrap Bash in a different language, especially when handling numbers. But I'd rather use something like Python than this verbose trickery.

            • vagrantJin 1056 days ago
              I agree. My scripts are also rigidly sequential and verbose which is easier for me and other devs after me to have a simple mental model of what is going on at a glance. Which I think saves time in the long run because any dev can easily understand and make intelligent additions to make it better. For complex scripts, Python is the goto and isn't terribly difficult to grasp either.
          • bryanrasmussen 1056 days ago
            I didn't say it was, but in the example given basically everything was, hence the complaint the parent made was that in JS we are often writing a lot of await, or promises or what have you because we have a lot of things that need to be done sequentially in a particular part of the program, or, in a small script, where everything needs to be sequential.
            • Raed667 1056 days ago
              It's not really a JS issue here, just the ZS project decided to make `$` function async (for some good reason I'd think) but they could have gone the other way and make it synchronous by default.

              I don't believe this is a language specific choice, they could have done the same with C#

          • _ZeD_ 1056 days ago
            well, almost?
      • trog 1056 days ago
        Here's a lazy third option:

          console.log('start')
            try {
              doA()
              doB()
              doC()
              console.log('finish')
            } catch (error) {
              console.log(error)
          }
        
        It seems to me async/await is a desperate attempt to try to make JavaScript less painful to deal with because of its asynchronous execution nature.

        I am not a JS expert and have not had many opportunities to work with it extensively, but I confess I have always struggled with this when dealing with JS; I have often wondered if the (alleged?) performance gains from executing JavaScript like this outweighs [what I have always perceived as] the significant extra verbosity and complexity required to manage simply executing things in order.

        • jmull 1055 days ago
          I agree with this syntax, but it’s too late now (for JavaScript).

          To clarify, JavaScript could have made await the default for async functions, so that if you called an asynchronous function without putting any keyword in front of it, execution would block until the operation completed. In that design, there would be some keyword you would use when you did not want to wait.

          What they did: you have to opt-out of async execution by adding the await keyword.

          What they could have done: you opt-in to async execution by adding a keyword.

          They just chose the wrong default (IMO). It’s not a big deal — it’s easy enough to type “await” here and there. It’s bad, though, that you don’t really see the basic flow of control from examining the code making function calls. You also need to consult the function definitions to see which ones are async.

          Another option would be to have no default, and instead require that async functions be called with an explicit indication of whether they are to execute sync or async. That too heavy-handed, IMO. Fine for a statically checked language but not a dynamic one.

          • trog 1055 days ago
            Thanks, this is basically what I was trying to say but is written so it actually makes sense.
        • yoz-y 1056 days ago
          I don't understand your example, if would only work if your doA,B,C functions were synchronous, which completely changes the paradigm and use case. It's simpler yes, but very different. (simple example: if that function were to be run when you click on a button, now you have a frozen UI.)
          • trog 1056 days ago
            Yep. I am just wondering if the paradigm of JS being asynchronous is just not worth the added effort in code complexity and all the workarounds that have had to be added in over the years to try to make it manageable.

            I personally find it a chore to work in JS compared to other languages that work the opposite way - everything is synchronous and things become a chore when you want to deal with async stuff. Again I have limited JS experience and have never enjoyed working with it (I just am not interested in front end stuff) so I'm sure it's stuff people get used to.

            • noahtallen 1055 days ago
              Could you clarify what the paradigm of JS being asynchronous means? Usually, async calls are actually “async” in that you’re doing a network request or something actually asynchronous. The reason they have to be async is that you cannot just block all execution or else other things can’t happen while the network request is happening. Which means your entire application grinds to a halt. Other languages solve this in a similar way IMO, with async/await and actual threads. JS doesn’t have threads, so you can’t fork a process to handle network stuff apart from UI. I personally think that async/await is a lot more straightforward to use than forming a process.

              Additionally, async/await allows you to wait when you need to. You don’t have to await a promise — you can just call it, and then it will work in the background. You’ll just not be able to respond when it finishes work. (Because you aren’t “wait”ing for it.)

              What are some other examples with easier async in other languages? I’m just confused what you’re trying to get at. Everything in JS is synchronous. You only have async when you have async. Which is a lot, because a lot of stuff in the web is async.

              • trog 1055 days ago
                Yep, sorry & thanks for your polite post in response to my basically unhinged ramblings (late night after a tough week, always a bad time to try to do anything).

                The poster jmull above basically wrote[1] what I was trying to say much more succinctly; this other post[2] is also more clear.

                Really I was just being snarky about the idea of shell scripting where you have to type 'await' in front of every command you want to run in a three-line file, like in the examples presented for this tool.

                1. https://news.ycombinator.com/item?id=27080734

                2. https://news.ycombinator.com/item?id=27077752

        • Raed667 1056 days ago
          I wasn't attempting to defend JavaScript design choices. I'm just stating that working with blocking vs non-blocking code is now MUCH easier (and less error prone) since we have async/await.

          Callback-hell and Promise-hell were real issues that plagued any project of significant size.

          • trog 1056 days ago
            Oh yeh sorry I didn't mean to give the impression I thought you were wrong. I 100% agree with you.

            I remember the first time I experienced "callback hell" (2016, for me, but I'm sure it was a huge problem for others before then) when I was doing some JavaScript stuff implementing Keybase's library for GPG support - I learned they'd built a whole separate JavaScript thing called IcedCoffeeScript[1] specifically to add await/defer support to get rid of those huge callback pyramids.

            1. http://maxtaco.github.io/coffee-script/

    • jiofih 1056 days ago
      If you want to have the same code structure as this, but without await, you’re gonna implement your own queuing system (every call adds an operation to a queue). Many old tools operated this way, before ES6.

      The problem with that it doesn’t meld together with most third party libs, since they will have no knowledge of your queue and will just execute immediately or out of order.

    • intergalplan 1056 days ago
      IMO it was always a fundamental mistake to force the programmer to deal with the event loop by default. Run async in the background, but present as sync to the developer unless otherwise requested, would have been a much saner design. Unless you're developing a framework or library, odds are good your ratio of leveraging async versus contorting yourself to make it (from your script's perspective) go away will be 1:10 at best.

      JS keeps coming up with new ways to make this less painful, but it's ridiculous every time because it's a fundamental problem with how Node presents itself. A comical sea of "await"s in front of damn near every call is the modern version of this, and is utterly typical in real JS codebases, but before it's been callback hell, or screwing around with promises all over your codebase when 90+% of the time you just wanted things to execute in order (from your perspective), and so on.

      • alpaca128 1056 days ago
        I think it's also made much worse by how JS doesn't care about whole classes of bugs. Forgot an `await` somewhere or called an async function assuming it's sync? Now you've got a bug and in some cases no idea where to look for it. TypeScript is also blind to it (although now that I think of it a linter might flag it?).
        • intergalplan 1056 days ago
          A really good point, and all the more reason to make developer-visible async behavior something the developer has to to ask for, even if the call is in fact async under the hood and might let, say, code handling another request run while it's waiting on I/O.

          I think a pattern where there are one or two great places at the lowest level of a Node program for program flow to act async, and then a bunch of business logic where it rarely is (probably running "under" the part where async makes sense, if you take my meaning) is far more common than those where async-friendly flow is what you want for over 50% of calls. "Call this chunk of code async, but run everything in it exactly in order" is super-common, and the interface to achieve that is exactly backwards in Node.

        • deckard1 1056 days ago
          I can't even count the number of times I've seen a unit test giving a false positive (or, perhaps more accurately, not even run) because the developer forgot to properly use async/await or use the done callback.
        • dariosalvi78 1055 days ago
          Linter flags when you put an await in front of a-non async function, but, alas, the opposite is not true (call of async without await). This has always been source of bugs in my code.
    • tusharsadhwani 1055 days ago
      Yeah, I saw this yesterday and didn't like it so I made my own cleaner version: https://github.com/tusharsadhwani/zxpy
  • cltsang 1056 days ago
    Atwood's law is still accurate after more than a decade:

    "Any application that can be written in JavaScript, will eventually be written in JavaScript." - Jeff Atwood

    source: https://blog.codinghorror.com/the-principle-of-least-power/

  • c0l0 1056 days ago
    This embodies so much of what is wrong with some JavaScript programmers' (that I've known personally) mindset that I find it hard to distinguish from satire...
    • classified 1056 days ago
      Yep, the quest continues to rewrite every software there ever was in JS.
      • the_cat_kittles 1056 days ago
        nothing wrong with wanting to avoid the insane syntax of bash imo.
        • c0l0 1056 days ago
          But you don't get to have that! You get the "insane" syntax oh bash, encapsulated in a ridiculously verbose and frankly quite unintuitive JavaScript contraption that seems to provide little, if any, benefit.

          Also, Shell scripting is mostly about correctly and safely interacting with other CLI-based tools - which this new thing wrapping a shell (and badly at that, see other comments in this thread) won't help you get right, either.

          • kube-system 1056 days ago
            There are some projects which I've migrated from bash to js, in the middle of the migration, we had a mix of both. This project would have been really helpful to make that migration quicker. Of course, it wasn't ideal, but there's always a journey before the destination.
          • the_cat_kittles 1055 days ago
            i feel like you could avoid some of the kind of annoying syntax like array iteration and conditional syntax. would it be better to know that stuff 100%, and write it in pure bash? yeah. is it one of those todos that people like me never seem to get around to, and this is kind of a stop gap? maybe. it doesnt seem entirely useless
    • christiansakai 1056 days ago
      Welcome to GME/Doge era.

      Very bullish on JavaScript.

  • chetangoti 1056 days ago
    This is good, enabling people with Javascript proficiency write complex scripts IMO.

    There is also https://github.com/shelljs/shelljs which is implemented on top of nodejs APIs.

    • niffydroid 1056 days ago
      I pretty much got rid of my bash scripts and just replaced them using shelljs. Makes it a lot easier and quicker to maintain, it means any developer can jump in fix it and add to it.
    • jeswin 1056 days ago
      If you like shelljs, then check out https://bashojs.org (mine).

        # Example, list all dev dependencies:
        cat package.json | basho --json 'Object.keys(x.devDependencies)'
      • Too 1054 days ago
        I think use case of shelljs and this zx is more for developers who need to perform file system and subprocess operations but rather stay away from bash or other footgun-ridden shells and instead write things in safe code with access to richer data types and more libraries.

        bashojs seem to start in the other end, add js statement-execution to shells and and yet another pipeline-language in the mix. More like AWK with js-syntax.

        Looks good as a "jq" on steroids but i wouldn't compare it to shelljs.

  • franciscop 1056 days ago
    I was hoping for an article on how to use Node.js for normal scripting, since it's already pretty close to what it's shown in this library. I've written two libraries to help with scripting in Node.js:

    `files`: https://github.com/franciscop/files/

        import { read, walk } from 'files';
    
        // Find all of the readmes
        const readmes = await walk('demo')
          .filter(/\/readme\.md$/)          // Works as expected!
          .map(read);
    
        console.log(readmes);
        // ['# files', '# sub-dir', ...]
    
    `atocha`: simplest cli runner (no escaping though!) https://github.com/franciscop/atocha/

        import cmd from 'atocha';
    
        // Any basic command will work
        console.log(await cmd('ls'));
    
        // Using a better Promise interface, see the lib `swear`
        console.log(await cmd('ls').split('\n'));
    
        // Can pipe commands as normal
        console.log(await cmd('sort record.txt | uniq'));
    
    Both of them are wrapped with Swear, a "promise extension" (totally compatible with native promises!) so that's why the first example works. You can build operations on top of the return of the promise, so that these two are equivalent:

        // Without `swear`:
        const list = await walk('demo');
        const readmes = list.filter(item => /\/readme\.md$/.test(file));
        const content = await Promise.all(readmes.map(read);
    
        // With `swear`:
        constn content = await walk('demo').filter(/\/readme\.md$/).map(read);
  • szhu 1056 days ago
    Compare to Deno, which also makes JS friendlier for shell scripting! https://deno.land/

    Coming from Deno, the single biggest advantage I see here is the handy tag function $:

        await $`cat package.json | grep name`
    
    Hoping someone will write a Deno port of this.
  • justsomeuser 1056 days ago
    As soon as I hit an “if” branch in a shell script I move to JS, Python or Ruby
    • dfinninger 1056 days ago
      I think if's are generally alright. My personal bar is arrays. Anything beyond

          for x in a b c; ...
      
      is getting written in Python.
      • justsomeuser 1055 days ago
        It's not the "if" in particular, it's more of a signal that I am moving from a simple set of commands to something more complex with logic that will need objects, arrays and possibly a package manager.
    • dahfizz 1056 days ago
      Do you have node installed on all Linux boxes you interact with?
      • justsomeuser 1055 days ago
        I typically deploy my code in docker containers. Often bash-like scripts are for my dev machine.

        But if portability was an issue I would still probably move to something that outputs a single binary (Go, Rust).

        I just think bash is not a very good language compared to the alternatives.

    • joncp 1056 days ago
      Well, the whole reason to choose bash in the first place is that it just works on every single machine I have access to.. Ruby and js aren't installed everywhere. As for python, I have to be willing deal with versioning hell and the whole virtualenv vs venv vs conda mess.

      Maybe C could work, but I have to make sure it's statically linked.

      • pwdisswordfish8 1053 days ago
        > Ruby and js aren't installed everywhere.

        Neither is Bash, despite belief to the contrary.

  • christiansakai 1056 days ago
    Lol. Anything that can be written in JavaScript will be written in JavaScript.
    • Bombthecat 1056 days ago
      It's getting really painful..
    • chaostheory 1056 days ago
      It seems like Steve Yegge predicted it 10 years before it happened. Spot on.
    • devilduck 1056 days ago
      Except for Javascript itself, since it can't do that
  • zmix 1056 days ago
    Nashorn (Oracle's, now discontinued ECMAScript implementation for the JVM and Rhino's successor) can be used for Shell scripting, too, when started with the `-scripting` parameter. Here is a little overview[1]. I think, GraalVM's new ECMAScript implementation is compatible with both NodeJS and Nashorn, including the scripting features.

    [1]: https://docs.oracle.com/javase/8/docs/technotes/guides/scrip...

  • Diti 1056 days ago
    Why is Javascript a « perfect choose » for shell scripts?
    • Shacklz 1056 days ago
      I had very strong anti-javascript opinions for the longest of times, and still feel that writing any non-weekend-project in JS is a mistake.

      That being said, I have grown very fond of writing "plumbing"-scripts in nodeJS. The syntax is so much more sane than bash, the documentation of nodeJS is actually pretty good in my opinion, and once nodeJS is installed and available, its super easy to mash something together to get working. For small scripts, the uglier parts of JS do not really matter, and while nodeJS does have some APIs that feel a bit odd to use at times, it's at least well-documented with examples/guides all over the web.

      Some folks use python for that exact purpose, but I always felt that getting something going takes me more effort than nodeJS (the fact that I'm just not fond of Python as a language might also have something to do with this).

      All in all... for folks still using bash, I really recommend to give JS a try.

      • dehrmann 1056 days ago
        I prefer Python for this because more OSes ship with it, it has a fairly rich standard library, and SREs are more comfortable with it, so you're more likely to find others using it as glue.
        • deergomoo 1056 days ago
          This will probably not be particularly popular with the HN crowd, but PHP is also a surprisingly capable general purpose scripting language.

          It's pre-installed on a lot of systems, backwards compatibility is good, and despite the inconsistencies, the standard lib can do a lot of stuff.

          • tored 1056 days ago
            Agree, but it would be nice to have access to the entire composer community without setting up a composer environment, so not to toot my own horn, but I wrote proof of concept script to download PHP composer dependencies by parsing comments from the PHP script itself.

            https://gist.github.com/tored/b500eb7c10fbabbe2043126e51caf2...

            It would be nice to have something like this more integrated within the shell or maybe composer itself.

        • throw_m239339 1056 days ago
          > I prefer Python for this because more OSes ship with it, it has a fairly rich standard library,

          I mean nodejs =/= javascript, the problem is nodejs and some of the people who designed it who basically made it so that it would rely heavily on NPM, a commercial solution, thanks to the virtually non existing standard library. Now Microsoft virtually owns Nodejs since it owns NPM. Nodejs creator himself regretted this decision, publicly.

          • Shacklz 1056 days ago
            For what I called "plumbing"-scripts above, I rarely ever need any dependencies via npm. What nodeJS delivers out of the box suffices completely.

            Sure, doing things like http directly with node is a bit cumbersome and having a library that does some abstraction comes in handy if you're working on something that goes way beyond what you'd ever consider doing in bash. But especially for that very purpose of things where I used nodeJS instead of using bash, nodeJS always had everything I needed out-of-the-box, almost never required any dependencies.

            That being said, if you re-use code, plan to share scripts etc. with other people and all these things - using something like typescript (and possibly some testing-libraries) comes in very handy for maintainability, and there you can't get around npm. But for one-off-"plubming-scripts", nodeJS had all I ever needed.

      • golergka 1056 days ago
        Have you tried Typescript? It has one of the best type systems among mainstream (not FP) languages, IMO.
        • Shacklz 1056 days ago
          TS is probably my favorite language out there, so yes, I definitely did :)

          But just for "one-off"-scripts, JS usually does the job pretty well. For bigger scripts or scripts where I want to reuse/share parts, typescript is an option - but it comes with a few hurdles: Most of the scripts I'm talking about work with zero npm-dependencies, while typescript adds at least one.

          Furthermore, I either have to take care of compiling it, or use ts-node - one more dependency; and in both cases, I can't avoid the usual es6 vs. commonJS shenanigans. It's a bit of overhead to get going; always worth it if I spend more than a few hours on a script, but I can do without if it's just a few lines that I will hardly ever see again.

          • dmux 1056 days ago
            I think Deno may be a good choice in this use case: single binary that has a good standard-lib makes it seem like a good candidate for simple shell and pipeline scripting.
      • _ZeD_ 1056 days ago
        have a look at perl :)
    • throw_m239339 1056 days ago
      > Why is Javascript a « perfect choose » for shell scripts?

      It isn't, that project looks like perl re-invented but worse, since it mixes Javascript via nodejs and shell script syntax.

      Shell scripts already support loops, functions, conditions, variables, ... it makes more sense to write a nodejs script that works as a regular shell process and include it in a shell script than the other way around...

    • giorgioz 1056 days ago
      JS is the most used language for web development both client and server side. It's easy to find fullstack repos with only JS code and a bunch of shell scripts to launch the tooling like webpack bundler and cypress e2e tester. By wrapping the tooling also in JS it's possible to take advantage of shared configuration written in JS and write reusable functions to share across scripts. Currently I have all my tooling scripts in bash (not yet in JS) and it's becoming hard to manage. Sharing functions and constant values across folders in shell bash script is very cumbersome.
      • EvilEy3 1056 days ago
        > and server side.

        Cool story.

    • abiro 1056 days ago
      I thought that was weird too. I much prefer Python to JS for scripting.
      • Mauricebranagh 1056 days ago
        Or even Perl - I have only done some scripts in Python to use beautiful soup, or for scripts that may be used by less experienced developers.
    • kristopolous 1056 days ago
      This is purely performative and has no other benefit than to bow to a set of aesthetics.

      It's merely doctrinal orthodoxy full of needless and ceremonious frivolities for the adherents to chin stroke in approval.

      It is not, in any way, a reasonable way to do things, to exercise engineering, write maintainable or reliable code or otherwise accomplish tasks.

      It is just spectacle.

    • goatlover 1056 days ago
      Javascript all the things.
      • artificial 1056 days ago
        When the only tool you have is a hammer :)
        • bryanrasmussen 1056 days ago
          You look like Thor!
          • Mauricebranagh 1056 days ago
            More like Jeremy Clarkson hitting thing with a hammer instead of the right tool for the job.
  • stunt 1056 days ago
    Javascript is just the new utility knife that all devs have in their pocket now mostly because you don't need to master it to build something with it, but also because it's acceptable to write improper JS code.

    PHP used to be the utility knife for a while for the same reasons. It was easy to learn, and run it. And it was ok to write bad PHP code. I remember most of the exploits and remote shells were written with PHP because it was the easiest language for hackers to learn.

    • nicbou 1056 days ago
      Python is a much better utility knife in my opinion, because it's used in more domains than just web development, and comes bundled with most operating systems.
      • devilduck 1056 days ago
        python is Way better than js but js people can't cope with other languages for some reason
    • Toutouxc 1056 days ago
      JS as the utility knife? No thanks, I'll write my quick scripts in a language with an actual useful standard library.

      That would be Ruby for me, but I also accept Python and maybe dozens of others.

  • exdsq 1056 days ago
    Honestly I’d rather use Bash than JavaScript. I’ve been using Go for scripting recently which has been nice.
    • hbn 1055 days ago
      Do you compile binary(s) or just execute a `go run` and rely on the fast build times?
  • gitgud 1056 days ago
    Would be good to have a function which outputs all stdout from a sub command in real time, rather than after it's finished.

    I generally use "spawn" from child_process instead of "exec" (which this tool uses), which can pipe output to the terminal as it happens. Great for creating build scripts.

    • lobstrosity420 1056 days ago
      Yes, you could use node's streams to implement this. It would be really cool.
  • ilaksh 1056 days ago
    Maybe there could be a shorthand for the await keyword. Such as instead of

        await mycall()
    
    Maybe

        mycall()!
    
    Or something. Or perhaps a flag you could set at the top of the script that would transform every call into an await call.
  • tusharsadhwani 1055 days ago
    Honestly, i saw the syntax and thought I can do better, so I made it: https://github.com/tusharsadhwani/zxpy
    • cdaringe 1055 days ago
      he doesn't demonstrate it, but his allows for simple parallelism, of which the python version will be more arduous
  • maxandersen 1056 days ago
    Proper type safe Java scripts are nice too :)

    https://jbang.dev

  • lucasmullens 1056 days ago
    > JavaScript is a perfect choice

    I mean I love JS as much as the next person, but perfect? No way.

  • austinshea 1056 days ago
    Do people actually find this more convenient than bash?

    Their own readme isn’t trying to suggest that it’s an improvement, just some sort of convenience.

    • mewpmewp2 1056 days ago
      More productive for folks who are comfortable with JS, but not bash, which would be very common for front end folks.
      • ttt0 1056 days ago
        Will people crucify me for suggesting that maybe you should also learn a language that's not JavaScript?
        • mewpmewp2 1056 days ago
          I should, but not sure if learning bash specifically is best for productivity. Especially if I don't use it daily, I will forget syntax etc, will still have issues understanding what is going on quickly, more likely to introduce bugs when trying to improve something etc.
          • gen220 1056 days ago
            The shell is a long-term investment, and decades of programmers are happy to quietly testify to its continuing dividends.

            POSIX shells (and OS) are the common thread underlying all *nix-based software.

            Bash prioritizes terseness and backwards compatibility; which is occasionally contradictory with UX, but are valid prioritizations nonetheless.

            Also, learning something new gives you a more nuanced appreciation of what you already knew. It’s a mind-expanding experience, even if it’s occasionally frustrating.

            TLDR. I’d advocate for it. It’s OK if things break or fall apart on the road, this is the price of knowledge acquisition.

            • mewpmewp2 1050 days ago
              Makes sense, but I'm still afraid if I don't use it daily I will tend to forget it. I have learned everything I know by doing it in practice, so personally unless I would take some huger type of side project to learn up bash I don't see how I could do it. And in the end if I'm working with other frontend devs, I'd also have to convince them to learn bash as opposed to just using JavaScript/TypeScript for scripting.

              Do you have any suggestions on how one should go about learning it?

              If I do some course right now, then in a year's time if I use it maybe on few occasions in a year, I won't remember it, and it will stop being practical again.

  • Ajnasz 1053 days ago
    What is it good for? Why would you want to execute shell commands from JavaScript. If you already know the command, then why don't you just use the shell? Or why don't you just use JavaScript libraries/frameworks/whatevers if you can't make a shell script work but you know JavaScript?
  • moocowtruck 1056 days ago
    looks cool, but just want to drop this here https://book.babashka.org/ https://github.com/babashka/babashka
    • iLemming 1055 days ago
      What's the point? Most people choose to suffer, even when you offer them something that requires learning the thing just once. No matter how nice the thing is.
  • sneak 1056 days ago
    Why would you want a shell script to be async? All this means is that you're going to type await 47 times. This isn't a network server, I'm not sure why this is the right tool for this job.

    Edit: that parallel rsync example answers my question nicely. I should have read more carefully.

  • ilyash 1056 days ago
    What if I told you there is a modern programming language designed specifically for DevOps?

    https://ngs-lang.org/

    Born out of frustration with bash and Python.

    and ... nope, never considered JS for that type of scripting.

  • parhamn 1056 days ago
    Curious. When will JS tooling ecosystem support executing typescript by stripping the type information? Are there any plans to add generic soft type annotations to the spec? It’d be great to write these in TS without changing the runtime tooling.
    • jeswin 1056 days ago
      You can already do that with deno.
    • rockwotj 1056 days ago
      I've found esbuild to be great for this: https://esbuild.github.io/content-types/#typescript
    • campers 1056 days ago
      What about using ts-node? You can run typescript files directly with that. I use it for my scripting purposes.
      • parhamn 1056 days ago
        It’s a great tool. It wouldn’t work in this case would it? Also, IIRC it checks types, doesn’t just strip (sometimes I prefer to just run the thing and do my type checking at the IDE/build level). Also still leaves the multiple tools problem.
        • CGamesPlay 1056 days ago
          Well `tsc` is the program that "strips" types after checking them. To be clear what you're asking for is a JavaScript runtime that supports parsing types but ignoring any errors it finds? Is there any computer programing language that has this behavior?
          • goodside 1056 days ago
            Python, pretty much exactly.
          • demurgos 1056 days ago
            Babel can do this if you wish to only strip TS extensions without typechecking.
        • campers 1056 days ago
          You can use the --transpile-only option to skip type checking. I'm going start using that myself for scripts I know are already good! Just tried it and makes a good difference to startup time.
  • juddlyon 1056 days ago
    Better than bash but nothing beats Python for scripting IMO. I’m a mediocre programmer and search results are rich with Python examples for anything under the sun.
    • louis-lau 1055 days ago
      To be fair, the same goes for javascript.
  • astrostl 1055 days ago
    > Bash is great, but when it comes to writing scripts, people usually choose a more convenient programming language. JavaScript is a perfect choice

    Awaiting citations on usually, convenient, and perfect.

    > await $`cat package.json | grep name`

    I mean, in the first line they're well into "I don't understand shells" territory with a superfluous cat pipe (you can just grep name package.json). This is seriously published under the google banner?

  • cafard 1056 days ago
    I occasionally have used JScript on Windows, mostly because I don't much like VBScript, and PowerShell want you to sign its scripts.
    • vips7L 1055 days ago
      Just set your powershell execution policy to unrestricted? Thats the default on non-windows machines.
  • vedantroy 1055 days ago
    This reminds me of my `jsmv` tool, which I use for manipulating the file-system in place of Bash / find:

    https://github.com/vedantroy/jsmv

  • vedantroy 1055 days ago
    Does anyone know why Bash has low usability?

    It seems like Bash has tons of footguns and unintuitive syntax. Is this just because the language grew organically?

  • ape4 1056 days ago
    Would the $ conflict with jQuery.

    (Of course jQuery's main job is DOM manipulation which isn't needed here)

    • LocalPCGuy 1056 days ago
      This isn't a browser-based script, so no real fear of conflict. Plus, it's basically past time to retire jQuery anyways, outside of maintaining legacy apps.
    • lobstrosity420 1056 days ago
      You can load JQuery into a different global variable if you want. There are cases where you might want to parse a DOM in the context of shell scripts, such as scrapping some web page. Although JQuery wouldn't be the tool I'd personally reach for in this case.
  • johndoe42377 1056 days ago
    This is getting beyond rediculous .

    It is already crossed into embarrassment.

  • bronlund 1056 days ago
    Jesus Christ. As if Node wasn't bad enough.
  • graderjs 1056 days ago
    This is cool. To the front page you go...
    • graderjs 1056 days ago
      But then it happened. Don't hate. I was upvoter no. 2 and I just knew this thing was gonna be on PAGE1. I know it's an "non substantial comment", but still! :P ;) xx