Homoiconicity isn’t the point (2012)


80 points | by tosh 6 days ago


  • Syzygies 5 days ago
    Processing parentheses is still parsing. It's kindergarten-level parsing, but it's still parsing.

    I've written thousands of lines of scheme code preprocessed to eliminate most parenthesis. I'm convinced Lisp self-asphyxiated by fighting a last stand on parentheses, which is so far from the point of Lisp. With modern editor "language server" support and syntax coloring, languages can shape-shift to any appearance they want, separate from meaning.

    While I love the poetic look of parenthesis-spare Lisp, Bill Joy got it right when he said that code density on a screen affects programmer productivity. I love Haskell, both for this and because one reasons about the activity of programming as if one is doing algebra.

    While Lisp more easily modifies itself than any other language, macros are bolted on. No one can tell me with a straight face that with a thousand runs of the civilization simulation, our Lisp macros would be the best run. I'd love to see a language where manipulating the language is the core strength, and all ease with data structures follows as a consequence. Lisp is not that language; peek into some of the other simulation runs.

    • blacktriangle 5 days ago
      The parenthesis issue is one that I simply cannot grasp, but have to acknowledge as true simply because so many programmers have voiced it. As far as I can tell there is no correlation between who has an issue with them and who does not. New programmers vs experienced, young vs old, bad vs good, it does not matter. Some programmers love them and for others they are a show stopper.

      For me, until you demonstrate the same level of structural editing support and powerful macro system without S-expressions, you can pry my parenthesis from my cold dead hands.

      • zelphirkalt 1 day ago
        Some say that with a "proper IDE" or with "proper editing support" this is possible for other languages too, but I ultimately doubt it, for all languages, which do not mark the beginning and end of their expressions and all languages, which consist of a mix of statements and expressions. The point is, that the parentheses make it unmistakably clear, where something starts and ends, which means, that an editor has no problem figuring it out. If a language has statements and blocks and expressions and whatever else there is, but not delimited by tokens like parentheses, then there will be ambiguity. That ambiguity cannot be removed and hinders how good an editor can be at selecting exactly that portion of the code, that you want to select and possibly move or wrap in something or do whatever with.

        Some languages replace such parentheses tokens with a "begin" and "end" keyword for example, but imagine what it would look like, if you had to use "begin" (5 characters) and "end" (3 characters) for really everything, for which parentheses in lispy languages are used. Now that would be a syntax nightmare.

        • taeric 5 days ago
          I think it is only true because so many insist it is true. Same reason python is "easy for beginners.". So much propaganda insists it that folks believe it must be so.

          Note, my hypothesis is essentially any language that folks insisted is easy for beginners could be easier. There is a certain difficulty that comes from thinking something must be difficult.

          • pierrebai 5 days ago
            Portraying widely held opinions as propaganda is quite counter-productive, bordering on denial. Given that you provide no reasoning one way or another, I cannot say more about your personal views, but...

            Denying Python's simplicity is surprising. Clarity of syntax, avoiding superfluous tokens, and ordering of code in speak-order are a few of the reason for it's simplicity. Interestingly, Python fails at syntax clarity exactly where it adopts functional-style construct: list-comprehension for example.

            Like it or not, a language, like Lisp, which insist on polish notation is just unnatural, at least for most western speakers. (I don't know any spoken language that would map to placing verbs first in a sentence.)

            • anaerobicover 5 days ago
              > (I don't know any spoken language that would map to placing verbs first in a sentence.)

              There are some less spoken, and some more widely spoken languages have this as an optional possibility. Verb-Subject-Object order: https://en.wikipedia.org/wiki/Verb%E2%80%93subject%E2%80%93o...

              • taeric 4 days ago
                This is silly. All programming is already in the same order for anything not math. And even math has plenty of prefix usage. Sigma, gamma, etc.

                So, I should use another term than propaganda, as that is negatively loaded. Consider it, rather, effective marketing.

                Edit: and apologies on dodging the point on python being simple. I really don't know how to respond on that. I still fight heavily to read most python I am exposed to, as it would be way easier in sql for most of the etl I see.

                • oblio 4 days ago
                  Many (most?) programmers don't like (formal) math.
              • bsder 4 days ago
                > Same reason python is "easy for beginners.". So much propaganda insists it that folks believe it must be so.

                Python's grammar really does seem to be significantly easier that just about everything else.

                Someone on HN mentioned an article in which there were a bunch of graphs of the grammars of various languages, and Python was just obviously smaller and cleaner than everybody else.

                Hopefully someone with a better memory than I will relink it.

                • taeric 4 days ago
                  This feels like a somewhat dubious metric, all told. For one, complexity of language says little about complexity of programs made in language. And, that is usually more influenced by age of the code base with how many are contributing to the code base.

                  Still, if you find the article, is be delighted to see the argument being made.

                  • bsder 3 days ago
                    > This feels like a somewhat dubious metric, all told.

                    The claim was: Python was easy for beginners. Certainly a simpler, more consistent grammar is easier for beginners.

                    For me, I left Perl for exactly that reason. Python just seemed to "fit in my head" much more easily than Perl. Scalar vs List context is a PITA.

                    > For one, complexity of language says little about complexity of programs made in language.

                    True. But I can also tell you that I could come back to my Python code a couple months later and still understand it. That never happened with my Perl code.

                    And, this really got put to the test when Python code I wrote 15 years ago came back home to roost. Yeah, I cringed at some of the ways I did things in the code (although probably I should be kinder to my younger self--it was Python 1 code chewing through nasty binary formats), but I could still understand it and debug it.

                    • taeric 3 days ago
                      I disagree that a simpler grammar makes it easier.

                      And comparing to perl is amusing. That is famously a write only language. Easy to build quick text processing in, but about it.

                      For older code, this is a point I give to lisp. Not only can I still read most older code, I can still run it.

            • lmilcin 5 days ago
              > I'm convinced Lisp self-asphyxiated by fighting a last stand on parentheses, which is so far from the point of Lisp.

              I think the reason Lisp self-asphyxiated is because people don't get it.

              Most try to spend a little bit time programming it as if it was C/Java/Python/whetever else they have used on their last project. They just see a language that can do what they need but it is all wonky and so why bother?

              Compared to other languages, to get benefits from using Lisp you actually need to understand what the benefits are and what causes the benefits.

              You can switch from C to Python and reap benefits of Python without having some kind of deep understanding.

              That's because programming in Python is basically same thing as programming in C, it is just more efficient. There is less stuff to set up (unless you run into compatibility problems), less stuff take care for (begone pesky pointers and memory allocation!) and better API to do repeatable tasks.

              On the other hand programming in Lisp is different than every other language. But you also can program as if you were programming any other language. So you get confused about what makes Lisp different and if you don't stick it out you might think you just learned Lisp and "what is all teh fuss about?"

              I think getting from regular programming to Lisp should be presented as same step change as going from not programming to programming.

              Person who has never programmed and never heard about programming might have a lot of trouble understanding why you would even want to program (in the end it is just adding numbers -- why would that accomplish anything, I am not an accountant?)

              • Guthur 5 days ago
                For me it's that lisp just sets up the language to be so easily used at multiple levels of abstraction, which is all at once a truism by also not nearly enough to communicate what it is like to use.

                At this stage there are likely many languages that will allow you to work at similar number of abstractions including syntax, but I'd be surprised if many feel so much like idiomatic code. Implementing macros has some constraints but it's quite possible to internalise and start writing s-exp manipulating code as if it was just any standard problem.

                • lmilcin 5 days ago
                  That is a good point.

                  To be able to utilize Lisp well you need to be able to structure your application, but you would need already a lot of experience and exposure to well structured applications.

                  Whereas if you are a Java developer you get structure from people who designed Spring and now this is how you fill in controller, this is how you make service method, this is how you make your database layer... all these decisions were already made for you and you just need to follow to get a passable result.

                  Freedom is a double-edged sword. If you don't know how to use it it may very well be worse than not having it in the first place.

                • AnimalMuppet 4 days ago
                  > That's because programming in Python is basically same thing as programming in C, it is just more efficient.

                  Depends on what you're doing. If you're writing an operating system, programming in Python is not at all the same thing as programming in C. Ditto if you're writing an embedded system, especially if it uses memory-mapped IO.

                  • lmilcin 4 days ago
                    Well, I meant from the point of view of the act of programming.

                    It is understood that programming languages have other strengths and weaknesses. For example, JavaScript will be more suitable to programming in browser app than Python, even though Python is better general purpose language.

                    Those strengths and weaknesses frequently come down to some pretty arbitrary choices the author made and have nothing to do with the language.

                    I can imagine situation where JavaScript never happened and instead we would be programming browsers in Python.

                  • mattyw 5 days ago
                    > I think the reason Lisp self-asphyxiated is because people don't get it.

                    As programmer who has spent most time in Python/Java/C but has dabbled in a few lisp dialects but isn’t sure if I “get it”: how can I know if I “get it” or not?

                    • tluyben2 5 days ago
                      It clicked for me only after getting significantly far in SICP back in the day, doing all the excersizes and creating things inspired by the book. Before that I just thought lispys were just some weird emacs thing. While this is a long time ago for me, I think it still will work for you to feel that 'click'.
                      • lmilcin 5 days ago
                        For me it clicked after I red Practical Common Lisp, ANSI Common Lisp and On Lisp, Let over Lambda and after I did couple projects in Common Lisp.

                        One day I looked at various REST API clients generated by JHipster. The clients generated for all languages had huge amount of boilerplate.

                        "Well, that's fair for ability to generate it from DSL", I thought to myself.

                        Then I looked at the code generated for Clojure client and it was many, many times smaller and looked as if somebody wrote it by hand. It was nice and neat.

                        Basically, the calls to APIs were macros and rather than generate a stack of code to be put in repository, the macros generated everything at runtime. The DSL was translated 1:1 to calls to macros and all the complexity of the generated code was completely hidden.

                        This caused me to spend considerable time thinking about the nature of difference between lisp and all those other languages and at some point it just clicked.

                      • rscho 5 days ago
                        Are you in love with Lisp? Then yes, you get it ;)
                      • q-big 5 days ago
                        > That's because programming in Python is basically same thing as programming in C, it is just more efficient.

                        You likely got this illusion because from beginning on, you actually programmed in C in a very "Pythonic" way.

                        • lmilcin 5 days ago
                          Actually, I learned by programming assembly for a long gone platform, 25 years ago. I have learned Python already after many years programming in C, Perl, Java and Common Lisp, professionally.

                          Experience with Lisp gave me a certain way of looking at programming languages. Rather than "oh, how fun!", I am more like "let's see which subset of Lisp it implements".

                          The only really interesting language in recent years that I have learned is Rust which truly does something new and it recalibrated my thinking about utility of making further subsets of Lisps. It seems there is still a lot of possibility for improvement from the language standpoint, and that it is possible that a lot of improvement may come from restricting the developer from doing certain things rather than giving unrestricted freedom.

                      • amw-zero 5 days ago
                        With a straight face - how can you do anything better than macros in Lisp? I’ve always thought of them as the Platonic ideal of macros. What are your problems with them?
                        • zelphirkalt 1 day ago
                          Perhaps not better, but different and yet on the same "power level": Smalltalk metaprogramming. Other than that, I am not sure anything more powerful is possible inside a classical computer.
                          • didibus 4 days ago
                            The only thing I've seen compete is Rebol like languages and their concept of dialects.
                          • bsder 4 days ago
                            > I'm convinced Lisp self-asphyxiated by fighting a last stand on parentheses

                            Lisp self-asphyxiated because the teaching materials were garbage^W poorly focused for normal programmers from 1985-1995.

                            Stupidly useful things like "Hash Tables" and "Imperative Linear Loops" were always sort of an afterthought/footnote in Lisp books as opposed to "Recursion and Y-combinator" which were 70%+ of the books and 90+% useless to daily programming.

                            Perl stood up and said "Regexs and Hashes are important, useful on a daily basis, and easy to use." And consequently wiped the floor of the non-systems programming community until Python.

                            • minhm 5 days ago
                              > macros are bolted on

                              As I understand it, macro is at a metalanguage level above the current code (eg. C macros vs C "normal" codes). In lisp, this metalanguage is in lisp forms, and thus it is parsed by ... the very same lisp parser for the code. Therefore the whole language is available to you at this meta level, and for all the meta levels above this meta level (ie. for macros that build other macros), so it's lisp all the way up.

                              I am not sure why you see it as "bolted-on".

                              • Izkata 4 days ago
                                I only know Lisp at a really basic introductory level, but knowing a different homoiconic language my understanding was that "macros" aren't even actually a meta-level thing in these - it's just the language being able to manipulate itself. Using it like this just gets called "meta-level" or "macros" to keep a cleaner separation for developers to reason about.
                                • dragonwriter 4 days ago
                                  It feels more bolted on than in, say, REBOL, where there aren’t separate macros because functions can selectively take any or all of their arguments unevaluated, as a value but not fully evaluated, or fully evaluated.

                                  (Which isn’t to say the Lisp way is worse; there’s strengths and weaknesses of both.)

                                • rscho 5 days ago
                                  Macros are bolted on in Lisp!? Ok, can you expand a bit on what a language with (syntactic) macros at its core would be like, please?
                                  • snicker7 5 days ago
                                    > I'd love to see a language where manipulating the language is the core strength,

                                    This is the appeal of Julia, where the community uses macros to do things like auto-differentiation (which itself feeds into modeling engines), create domain-specific languages, and to customize compiler behavior (auto-vectorization, target GPU compute shaders).

                                    • taeric 5 days ago
                                      I think the point is that you don't have to parse from strings. Instead, you can parse the atoms of the lists.

                                      I'm large, it comes down to what data type eval takes in. In most languages that have an eval, they take in strings. In lisp, it takes in a list of code.

                                      • bsder 4 days ago
                                        > Processing parentheses is still parsing.

                                        Processing parentheses is straightforward. Parsing a DOT is a nightmare.

                                        Anybody who says Lisp doesn't have syntax has never parsed CONS-pairs properly.

                                        • didibus 4 days ago
                                          I'm not sure I follow your point? Do you mean Lisp should evolve its syntax and try to find a representation that doesn't need so many parenthesis?
                                          • fzzzy 4 days ago
                                            The amazing thing about this is that M Expressions were proposed in 1960, but never implemented.
                                          • imode 5 days ago
                                            What do you mean by "manipulating the language"?
                                          • k__ 5 days ago
                                            "What makes Lisp’s syntax powerful is not the fact that it can be represented as a data structure, it’s that it’s possible to read it without </em>parsing</em>."

                                            For me that always was exactly was homoiconicity was all about.

                                            • rscho 5 days ago
                                              Homoiconicity is a start. But the killer feature is simplicity. In Lisp, only symbols and parentheses. In prolog, only predicate head, body and atoms. This is what makes those two languages so suitable to syntactic macros, while other languages claiming homoiconicity with more syntactic irregularity make syntactic macros impractical (although certainly possible). IMO, semantic AST-level macros are certainly a better choice in non homoiconic and syntactically irregular languages.
                                            • taeric 6 days ago
                                              This was what turned out to be more of a subtle point than I thought it would be in https://taeric.github.io/CodeAsData.html

                                              For me, there is a mindset of how to read the code that is unlocked by not having to specially parse so much of it in my head. It really can be seen as a list of code. And I can more easily see ways of transforming the code.

                                              Granted... I actually like the LOOP and FORMAT forms a lot. They somewhat break the easy to parse, but often they work as a coherent unit of code by themselves.

                                              • chriswarbo 5 days ago
                                                Even without macros, it's useful for languages to have a format that's "readable without parsing", since new language versions can introduce keywords that break existing tooling.

                                                The worst case I experienced was a project to perform static analysis on Haskell code. Some optional features (e.g. LambdaCase) alter the way files must be parsed, and I recall at least one causing an ambiguity. This made parser libraries pretty useless, so I had to use the GHC compiler directly. This was around 2015, when GHC's API was much harder to use programatically; in particular it would often crash (saying "the impossible happened") if it wasn't given exactly the right "DynFlags". Those were hard to guess, so had to be passed through as commandline arguments, essentially re-creating the GHC command!

                                                Sometimes those features and flags are toggled in a "pragma" comment at the top of a file, but it is also very common for projects to specify these in a separate config file for the Cabal build system. Those files are a non-standard format, so we have to parse them using the Cabal library. Hence I ended up essentially re-implementing the Cabal command, as a way to pass the right arguments to my re-implemented GHC command.

                                                That still struggled to parse Haskell code, due to widespread use of the C preprocessor (CPP).

                                                I ended up using the vanilla GHC and Cabal commands, but added my own optimisation pass as a GHC plugin. This pass is a function accepting and returning "GHC Core" (one of GHC's intermediate representations); the only side-effects it permits are for error reporting. Hence my pass just returned its input unmodified, whilst also printing it to stderr (as s-expressions, for simplicity).

                                                This "GHC Core" representation was usable for my project, but I still find it incredible that I wasn't able to reliably parse Haskell code (especially given how well-suited Haskell is for parsing!).

                                                • didibus 4 days ago
                                                  The reason you don't need to parse text is very much because of homoiconicity.

                                                  Homoiconicity = same representation

                                                  It means that the way you represent code in a language is the same way you represent some data-structure in that language. This applies both syntactically and semantically.

                                                  (+ 1 2)

                                                  This is the Lisp syntax for a list of three elements. Parenthesis denote a list, and each word inside it denote an element of that list. The above list has elements +, 1 and 2.

                                                  So the above code is the syntax for a list, and when you deserialize this (read in Lisp parlens), you get a list of three elements. This is the semantic part. Now you have a List object of three elements. You can now loop over this list, access elements in it, change the order of the elements, you can append or prepend more elements to it, remove elements, and whatever else you can do with a list data-structure.

                                                  Finally, it happens that code is represented this way as well. So syntactically if you want to add 1 and 2 together, you'd also write:

                                                  (+ 1 2)

                                                  And semantically, the interpreter/compiler will not take text as input, it will take a list of three elements as input, because code is assumed to be modeled as a list of elements. Not using some special CompilerList, but using the standard list data-structure you use for any other list.

                                                  Which means to execute code, you take text which models code using the syntax used to represent lists, and you parse/deserialize it (read it) into a list data-structure, and then you feed this list to the evaluator which will execute it as code.

                                                  Thus the textual representation for code and for a list is the same, and the in-memory representation of code and of a list is the same, and finally this list can be executed as code or simply processed as a list. This is homoiconicity.

                                                  • cout 4 days ago
                                                    I would love to see a language with exactly what you described, but with dictionaries instead of lists as the fundamental data structure.
                                                    • didibus 4 days ago
                                                      Ya that'd be interesting. How do you imagine the representation and evaluation semantics working? Something like:

                                                          {call: function,
                                                           arg1: value1,
                                                           argN: valueN}
                                                      Instead of (function value1 valueN)

                                                      So maybe you'd have:

                                                            call: defn,
                                                            name: hello,
                                                            args: {first: String, last: 
                                                            ret: IO,
                                                             call: print,
                                                              call: concat,
                                                              strings: ["Hello ", first, " ", last]
                                                            call: hello,
                                                            first: "John",
                                                            second: "Doe"
                                                      Where the number after "do" is the order in which things should execute, or maybe we can simplify it by using insertion order preserving maps...

                                                      Looking at it, if people thought nested parens made for an eyesore, I'm not sure this would be any better received haha.

                                                      • didibus 4 days ago
                                                        Replying to myself here, I'm curious to try it if we used order insertion preserving maps.

                                                        What if then we had:

                                                            {function: {arg1: value, argN: value}}
                                                        So we'd get a hello world as such:

                                                            {defn: {
                                                              name: hello,
                                                              args: {first: String, last: 
                                                              ret: IO,
                                                              impl: {print: {
                                                                      msg: {concat: {
                                                                             strings: ["Hello ", first, " ", last]}}}}},
                                                             hello: {first: "John", second: "Doe"}
                                                  • skybrian 5 days ago
                                                    What you get from the flexibility of having macros, it seems like you lose by being unable to do static analysis at any level other parentheses matching?

                                                    In particular, consider trying to figure out all usages of a function without executing code. I don't think you can even resolve imports without possibly getting fooled by macros that you didn't hard-code an understanding of?

                                                    (But on the other hand, it's not going to be 100% in languages that support reflection either. We just assume that's rare.)

                                                    • Firadeoclus 4 days ago
                                                      If you have access to the whole codebase, and the ability to ask the system for code after macro expansion, you can do any kind of analysis you want.
                                                      • skybrian 4 days ago
                                                        One problem is that macro expansion is arbitrary code execution, which means your static analysis is potentially insecure. But I suppose it could run in a sandbox.

                                                        A second issue is that the macro-expanded code is a lowered representation compared to to what the user sees. This tends to make user-friendly error reporting harder.

                                                        Another problem might arise when macro expansion is intermixed with normal execution, making it harder to just expand macros with executing code. Then you can still do a lot in a debugger, but that's quite different than static code analysis.

                                                    • airstrike 5 days ago
                                                      That example at the very end seems very contrived... It's not really valid JS, as far as my noob eyes can tell
                                                      • draw_down 5 days ago
                                                        If the value of Lisp is how wonderfully readable it is... then I guess it was never meant to be with me and Lisp.