The idea that a certain programming language could be proven to be more productive than another is laughable. There are so many factors involved in that, I can pretty much guarantee you it will never be studied to an acceptable level of rigor.
The author presents some flimsy "study" about coding string processing algorithms with a straight face, and basically loses all credibility. What about lifetime productivity of a codebase? Speed of refactoring, onboarding, testing etc..
Why don't you just be honest and say you don't care Python is slow because you like using it?
Performance doesn't matter? Of course it does. On the frontend, bad performance has been proven by many eCommerce companies to lead to lost business. On backend data pipelines, bad performance kills your flexibility. The longer it takes you to process N, the less times you can do it. The more times you can do it, the more times you can get it wrong.
Also who says that time-to-market is everything? Tech history has shown that product quality/brand marketing win out in the long term. There are tons of forgotten shitty products that were first. Being first is over-rated.
”The idea that a certain programming language could be proven to be more productive than another is laughable.”
I disagree. In the extreme, take python on the one hand and a variant of brainfuck that requires the hash (could be as simple as CRC-32 or as hard as SHA-256) of every source file to be zero on the other.
I don’t think you have to go that far to get provable differences, though. Some features that, I think, can fairly easily be proven to make programmers more productive in large parts of the space of programming tasks:
- assembler macros
- variable names that can have more than two characters
- direct support for floating point
- support for complex numbers
- local variables
- structures
- support for serialization and deserialization. Compare the way one can make simple structures serializable using Swift’s Codable protocol to whatever is ‘best’ for C (probably something macro-based)
- generics
- support for Unicode strings
- iterators
Of course, “python vs Java” or “python vs Perl” are much closer races where I expect measuring differences is way harder and results will depend on problem domain, experience of the subjects of the study, number of good and bad answers on StackOverflow, etc, but even for ‘normal’ languages, I expect some conclusions to be possible, given enough effort is spent. Python 1.0 didn’t have complex numbers, for example, so if your program needed them, python 1.4 was a huge improvement.
I doubt evaluating productivity of languages is worthwhile, though.
> The idea that a certain programming language could be proven to be more productive than another is laughable.
The idea that one chemical could be proven to be more effective than another is an equivalent problem. History, application, duration, there's a permutation of issues under that umbrella, as well. The problem is that nobody wants to argue about it very much. The Quorum project tried, to a small degree.
> Why don't you just be honest and say you don't care Python is slow because you like using it?
It's that and that you have an existing toolchain/codebase you don't want to rewrite in a potentially SLOWER way, because you don't know another language well enough. Python devs rather "solve real problems" than worry about a class of problems that their tool is inadequate for. There are always new trivial problems, where performance doesn't matter.
It is interesting to note that the listed libraries are not written in Python (Cython at best). The moment what you want to do with your data is not covered by the library, you either hack it or you're screwed.
That is my experience as well - the only redeeming feature of the language is the vast selection of easy-to-use libraries. The speed that the language sacrificed isn't some inherent trade-off, but completely orthogonal to what makes the language useful.
Exactly, Lisp, Scheme, Dylan, Smalltalk, JavaScript, Prolog are all examples of languages as dynamic as Python, while having good JIT/AOT toolchains available by default.
Python has to be one of the worst languages used today. It literally provides no value except for an abundance of libraries. Maybe you could argue it’s easy to learn, but so is Racket or Scheme. It’s dog slow, doesn’t support multi-threading, and lacks any sort of cohesive design.
Lambdas suck, collection types are totally inconsistent (list, dict, etc), too many magic keywords and concepts. Trying to program using a FP style is almost impossible. Also no macros, no traits, no real classes (duck typing is garbage and lazy).
You know a language is garbage when the most popular libraries aren’t even written in the language!
You do realize that "vast selection of easy-to-use libraries" is the most important feature of a programming language. Right? It's up there with, "it's not hard to hire programmers for the language", which Python also benefits from.
Like, I'm partial to Haskell, but if I'm actually trying to get work done, I reach for a language with a robust ecosystem. These days I reach for Typescript, but before then, python was my go-to since it's way nicer to program with than java/c++/c/javascript.
>You do realize that "vast selection of easy-to-use libraries" is the most important feature of a programming language.
No, it's not. When I switched from Python to OCaml it felt like blessing, since the quality of the libraries felt much higher in OCaml. There are also the projects that you have to build from the ground up without libraries (any mission critical software), and in such cases you don't care about libraries at all, you care about the merits of the language solely.
Python is the BASIC of the XXI century, and it's fine in that role, but not as a general purpose language.
One big factor in performance is whether a process can be parallelized. It can be an interesting computational problem, and these techniques has really gained relevance lately now the the barrier to clusters has dropped (cost and technical complexity). A lot of algorithms (say for solving a massive optimization problem) that were 10x slower in total operations but can be split up and solved in arbitrarily small pieces are suddenly immensely valuable. And in this context, the actual speed of the programming language probably isn’t terribly important.
I'm not sure if people using functional languages are more productive. One of the points Uncle Bob makes in his talk supporting functional programming is that it is harder and slower to write:
That said, I have to wonder if these dynamic scripting languages are truly enabling productivity or if it just feels like that because you can write it and try many things quickly. It reminds me of being such in traffic on the highway - sometimes I get off and take back roads even if my GPS tells me it will take longer, because it is more pleasant to be moving than feeling stuck.
That said, I think the biggest benefit of dynamic languages is that they are so easy to refactor. You don't have as rigid a structure, so you can try easily hack together some idea that, done "properly", would require a big refactor. Despite being functional, Haskell's strong type system makes refactoring a huge pain. If course the same things enabling this quick refactoring in the dynamic languages works against you when it comes to speed and reliability.
I sit with an FP expert (he might deny it, but he is). What I find, is that he takes a lot longer to get to the solution that I hack out in a day in Python3 (with his assistance most of the time) but his solution has three interesting properties.
1) its usually founded on a very good understanding of the problem, and the abstractions inherent in the problem. If he can re-use the abstractions he has used before he does, to great effect in speed-to-solution
2) it is very usually correct. it is remarkably bug free because the Haskell typing removes a huge amount of ambiguity about the problem going in
3) he has absolutely no problem explaining what it does 3 months or more later.
I value these behaviours immensely. My own Python is usually buggy, often re-produces techniques I did a week ago, badly, and I can't explain why I used that lambda/map function.
Typically processing data from logs, analyses of SQL state changes in the absence of good knowledge of the prior states. Some stuff which is aggregation over disparate inputs where a DSL can be useful. A lot of the time radix trees and other prefix/length matching logic is being used. Sometimes we're cross correlating different sources against disparate common keys.
Some stuff connecting type: value record structures into JSON.
We work a lot on ill-formed requests to refine data previously worked on with new questions. We work a lot on small distributed systems exploring eventual consistency and event sourcing.
We work with quite large logfiles collected asynchronously, with poor clock alignment. So data buffering and sorting is not uncommon.
Quick 'n' dirty might be fine for a move-fast-and-break-things web startup, but the world is much larger than that. I sincerely hope the people writing the software in, let's say an airliner to choose a topical example, choose option 2. Dealing with bugs that have killed people is not a "nice problem to have".
(It depresses me that I have to write a defense of good engineering on Hacker news. The value of doing things correctly should be a tacit core principle of any engineer, to be broken only under circumstances of exceptional duress.)
Most domains are not of the "killing people" type.
In those places, they also have other kinds of compliance requirements for the code -- which makes sense: I wouldn't leave the airline software PMs to chose between the above two options.
Most domains are also not the "finish the job by next Thursday or the company folds" type either. You're going to have to pay back that technical debt someday, with interest - why not save time and do it right the first time?
(Typical SV answer: you're only thinking of the short term because you're trying to get investment and you plan to exit stage left with a fat packet long before the technical debt rears its ugly head, leaving somebody else to clean up the mess. But this is not how most engineering is done in the real world.)
>Most domains are also not the "finish the job by next Thursday or the company folds" type either.
Even in those that aren't, the managerial "finish the job by next Thursday or you're fired" is equally good motivator to not "save time and do it right the first time".
You'd be surprised. There are tons of stories of incompetent self-important consultants, coming in with big ideas, and getting paid mucho dinero per day for what amounts to nothing (if not actively bad advice). I've work with some.
And don't get me started on the "specialist" consultants, whose job is to push some monstrosity, like IBM's stack, or SAP, or whatever, until it kinda fits the problem domain, and in the process sell even more crap from their vendor.
Some fun stuff (haven't checked them all, but most stuff I've read from WTF across the years was hilarious, if not true to life):
(And inversely, there are companies whose culture is all about good code and craftsmanship. But they usually don't get very far: if they don't fold, they're still niche shops. Perhaps Viaweb was one such).
I am pretty aware of that and was already expecting such a reply from you.
The point was that while for consultancy projects, customers do refuse to pay for what is considered a bad piece of work, for other kinds of software delivery, with exception of app stores, that never happens.
It is very interesting experience to take part in a customer meeting, which isn't very pleased how things are going, regardless of the outcome.
I also have my share of being part of such teams, similar to your examples, yet that doesn't mean one should stop pushing for quality as much quixotic endeavour as it might seem.
Most products are simple components put together and made to do a simple thing under controlled conditions.
Software is infinitely malleable and can be used under all kinds of environments (e.g. hardware configurations, OS versions, etc).
It's more like a novel than a microwave. When you buy a novel you don't get your money back when it doesn't perform well, or puts you to sleep in the 10th page.
i don't think that's true for multiparadigm functional languages like f#, racket, and others. these languages allow you to solve problems in the best way and don't necessarily bind you to some draconian perspective like haskell. functional doesn't necessarily mean static typing. scheme and racket are functional languages.
I have found that much of the benefit of functional programming is in the consistent guarantees it makes, not only about your own code but also others' that you depend on. Languages that try to do it all often leave me feeling disjointed. You end up with some libraries written in a strong OOP style, but then others are written with a functional flavor, and then your own code necessarily has to take on a strange amalgamation of the two.
Great observation, 100% on point. "Microservices are inherently slow. The very concept of a microservice is to break up a boundary by a network call. This means you are taking what was a function call (a couple cpu cycles) and turning it into a network call. There isn’t much you could do that is worse in terms of performance. Network calls are really slow compared to the CPU. But these big companies still choose to use microservices. There really isn’t an architecture slower than microservices that I know of. Microservices’ biggest con is performance, but greatest pro is time-to-market. By building teams around smaller projects and code bases, a company is able to iterate and innovate at a much faster pace. This just goes to show that very large companies also care about time-to-market, not just startups."
Except it doesn't scale. Amazon, Google and Netflix (the companies referenced before this quote) can't run their applications on a single machine. Distributed programming is unavoidable at scale.
Also there's a big difference between throughput and latency. Networking might introduce a longer delay than a CPU call, but if each one of those calls does significantly more work, the overall performance of the system can increase.
Whether slow code matters or not depends on the scale, because executing code requires power.
If it's a one off or otherwise rarely executed, yeah, it doesn't matter much. But it starts to matter when it's constantly running on a significant number of systems.
Widely distributed frequently running slow code can waste a lot of energy. Cell phones and laptops heat up and have a shorter battery life. Even desktops require more power.
On the global scale, a large number of such systems can waste several nuclear reactors worth of electricity.
10 times slower code might require 10 times the number of servers. Maybe it doesn't matter much when it's just the difference between one and 10 servers. Or perhaps you don't even consume 10% of a single server. But there's a point when the hardware and power costs start to dominate.
The author presents some flimsy "study" about coding string processing algorithms with a straight face, and basically loses all credibility. What about lifetime productivity of a codebase? Speed of refactoring, onboarding, testing etc..
Why don't you just be honest and say you don't care Python is slow because you like using it?
Performance doesn't matter? Of course it does. On the frontend, bad performance has been proven by many eCommerce companies to lead to lost business. On backend data pipelines, bad performance kills your flexibility. The longer it takes you to process N, the less times you can do it. The more times you can do it, the more times you can get it wrong.
Also who says that time-to-market is everything? Tech history has shown that product quality/brand marketing win out in the long term. There are tons of forgotten shitty products that were first. Being first is over-rated.
I disagree. In the extreme, take python on the one hand and a variant of brainfuck that requires the hash (could be as simple as CRC-32 or as hard as SHA-256) of every source file to be zero on the other.
I don’t think you have to go that far to get provable differences, though. Some features that, I think, can fairly easily be proven to make programmers more productive in large parts of the space of programming tasks:
- assembler macros
- variable names that can have more than two characters
- direct support for floating point
- support for complex numbers
- local variables
- structures
- support for serialization and deserialization. Compare the way one can make simple structures serializable using Swift’s Codable protocol to whatever is ‘best’ for C (probably something macro-based)
- generics
- support for Unicode strings
- iterators
Of course, “python vs Java” or “python vs Perl” are much closer races where I expect measuring differences is way harder and results will depend on problem domain, experience of the subjects of the study, number of good and bad answers on StackOverflow, etc, but even for ‘normal’ languages, I expect some conclusions to be possible, given enough effort is spent. Python 1.0 didn’t have complex numbers, for example, so if your program needed them, python 1.4 was a huge improvement.
I doubt evaluating productivity of languages is worthwhile, though.
The idea that one chemical could be proven to be more effective than another is an equivalent problem. History, application, duration, there's a permutation of issues under that umbrella, as well. The problem is that nobody wants to argue about it very much. The Quorum project tried, to a small degree.
> Why don't you just be honest and say you don't care Python is slow because you like using it?
It's that and that you have an existing toolchain/codebase you don't want to rewrite in a potentially SLOWER way, because you don't know another language well enough. Python devs rather "solve real problems" than worry about a class of problems that their tool is inadequate for. There are always new trivial problems, where performance doesn't matter.
I’m practically forced to use python for quant finance at my job because of the network effect: numpy, pandas, cvxpy, etc.
But the language itself is total garbage compared to, say, Scala or OCaml. And both of those languages are also much faster.
Lambdas suck, collection types are totally inconsistent (list, dict, etc), too many magic keywords and concepts. Trying to program using a FP style is almost impossible. Also no macros, no traits, no real classes (duck typing is garbage and lazy).
You know a language is garbage when the most popular libraries aren’t even written in the language!
Like, I'm partial to Haskell, but if I'm actually trying to get work done, I reach for a language with a robust ecosystem. These days I reach for Typescript, but before then, python was my go-to since it's way nicer to program with than java/c++/c/javascript.
No, it's not. When I switched from Python to OCaml it felt like blessing, since the quality of the libraries felt much higher in OCaml. There are also the projects that you have to build from the ground up without libraries (any mission critical software), and in such cases you don't care about libraries at all, you care about the merits of the language solely.
Python is the BASIC of the XXI century, and it's fine in that role, but not as a general purpose language.
Unfortunately for me, those are all things I do at my job. Outside of these domains, JVM libraries are much better.
How about compared to a functional language like Haskell, OCaml, or F#?
I agree that we should choose productivity over performance when possible. However, Java, for example, is extremely verbose.
My impression is that people who use functional languages are quite productive.
Plus, they need fewer unit tests.
https://youtu.be/7Zlp9rKHGD4
That said, I have to wonder if these dynamic scripting languages are truly enabling productivity or if it just feels like that because you can write it and try many things quickly. It reminds me of being such in traffic on the highway - sometimes I get off and take back roads even if my GPS tells me it will take longer, because it is more pleasant to be moving than feeling stuck.
That said, I think the biggest benefit of dynamic languages is that they are so easy to refactor. You don't have as rigid a structure, so you can try easily hack together some idea that, done "properly", would require a big refactor. Despite being functional, Haskell's strong type system makes refactoring a huge pain. If course the same things enabling this quick refactoring in the dynamic languages works against you when it comes to speed and reliability.
1) its usually founded on a very good understanding of the problem, and the abstractions inherent in the problem. If he can re-use the abstractions he has used before he does, to great effect in speed-to-solution
2) it is very usually correct. it is remarkably bug free because the Haskell typing removes a huge amount of ambiguity about the problem going in
3) he has absolutely no problem explaining what it does 3 months or more later.
I value these behaviours immensely. My own Python is usually buggy, often re-produces techniques I did a week ago, badly, and I can't explain why I used that lambda/map function.
Some stuff connecting type: value record structures into JSON.
We work a lot on ill-formed requests to refine data previously worked on with new questions. We work a lot on small distributed systems exploring eventual consistency and event sourcing.
We work with quite large logfiles collected asynchronously, with poor clock alignment. So data buffering and sorting is not uncommon.
Slower to write (and get right) -> fewer bugs to deal with later
Pick your poison
Having your company survive (because you got something faster to market), and having bugs to deal with later, is a so-called "nice problem to have".
The bug-less software of a dead company does nobody good.
(It depresses me that I have to write a defense of good engineering on Hacker news. The value of doing things correctly should be a tacit core principle of any engineer, to be broken only under circumstances of exceptional duress.)
In those places, they also have other kinds of compliance requirements for the code -- which makes sense: I wouldn't leave the airline software PMs to chose between the above two options.
(Typical SV answer: you're only thinking of the short term because you're trying to get investment and you plan to exit stage left with a fat packet long before the technical debt rears its ugly head, leaving somebody else to clean up the mess. But this is not how most engineering is done in the real world.)
Even in those that aren't, the managerial "finish the job by next Thursday or you're fired" is equally good motivator to not "save time and do it right the first time".
And don't get me started on the "specialist" consultants, whose job is to push some monstrosity, like IBM's stack, or SAP, or whatever, until it kinda fits the problem domain, and in the process sell even more crap from their vendor.
Some fun stuff (haven't checked them all, but most stuff I've read from WTF across the years was hilarious, if not true to life):
https://thedailywtf.com/articles/Consulting_The_Consultants
https://thedailywtf.com/articles/Invasion-of-the-Consultants
https://thedailywtf.com/articles/consultant-designed-success
(And inversely, there are companies whose culture is all about good code and craftsmanship. But they usually don't get very far: if they don't fold, they're still niche shops. Perhaps Viaweb was one such).
The point was that while for consultancy projects, customers do refuse to pay for what is considered a bad piece of work, for other kinds of software delivery, with exception of app stores, that never happens.
It is very interesting experience to take part in a customer meeting, which isn't very pleased how things are going, regardless of the outcome.
I also have my share of being part of such teams, similar to your examples, yet that doesn't mean one should stop pushing for quality as much quixotic endeavour as it might seem.
Software is infinitely malleable and can be used under all kinds of environments (e.g. hardware configurations, OS versions, etc).
It's more like a novel than a microwave. When you buy a novel you don't get your money back when it doesn't perform well, or puts you to sleep in the 10th page.
It is just that a large majority of population isn't aware of their consumer rights, or just doesn't want to bother.
Does it surprise you, though, on Hacker news?
The same researcher must write a study of the median temperature at the hospital.
No need to sacrifice performance for productivity.
[1] - Yeah, I know it is dead now, just wanted to mention it anyway.
Also there's a big difference between throughput and latency. Networking might introduce a longer delay than a CPU call, but if each one of those calls does significantly more work, the overall performance of the system can increase.
If it's a one off or otherwise rarely executed, yeah, it doesn't matter much. But it starts to matter when it's constantly running on a significant number of systems.
Widely distributed frequently running slow code can waste a lot of energy. Cell phones and laptops heat up and have a shorter battery life. Even desktops require more power.
On the global scale, a large number of such systems can waste several nuclear reactors worth of electricity.
10 times slower code might require 10 times the number of servers. Maybe it doesn't matter much when it's just the difference between one and 10 servers. Or perhaps you don't even consume 10% of a single server. But there's a point when the hardware and power costs start to dominate.
The scale matters.
Python isn't slow. Programming languages aren't slow or fast but interpreters are.
cpython is slow. PyPy interpreter is reasonably faster, upto 8 times.
Recently pypy 7.0 released which brought more hope.
http://speed.pypy.org