It's astonishing they managed to debug and fix this, given how old and how far away Voyager is. I wonder if currently created designs are so robust in terms of repairability. Obviously, everything is much higher-level now; is it possible to fall back to something like low-level, direct memory access communication if something in higher layers fails?
I'm glad to see so many senior engineers on the team; it must be an exciting lifelong journey to work on the mission.
> Obviously, everything is much higher-level now; is it possible to fall back to something like low-level, direct memory access communication if something in higher layers fails?
I have a feeling that code for NASA spacecrafts is not so high-level, for a reason.
From what I understand current flight control systems are still written in relatively low level languages like C++, with extremely careful memory management strategies.
Let's not be overly pessimistic. Even today's phones have a low-level debug backdoor. I imagine Voyager3 with today's tech would provide much more interesting data, but it's a matter of time, who wants to wait another 50 years
Something very wrong is going on with voyager, your terrestrial network or the universe!!! The spread of that ping is 0.4 hrs which I take to be 24 minutes! During that time a radio signal would travel 3 times the distance between the sun and the earth (AU). The only thing I can think of that could possibly be causing that delay is having to wait for a deep space network dish to aim properly.
I totally lack the patience to be able to deal with sending a command and having to wait 2 days to see the result. Everything about this is awe-inspiring for me.
Imagine opening Vim and gradually typing the commands in to edit a file with a 2-day time lag. Only at the end to notice the fateful words at the bottom of the editor: [read-only].
I wonder if you could teach millions of terminal sessions to an LLM and then have mosh run the inference algorithm to hide the latency (and eventually correct when it gets the real data)..
This is basically why I quit cancer research to go into software. It was like trying to debug via the postal service. You do something to your cell cultures, and you only find out 6 weeks later if something went wrong.
And because a million different things can happen to cells over the course of 6 weeks, you don't get a line number, or a stack trace, or very much debugging information at all.
So you quit potentially life-saving research because you couldn't open a terminal session to a cell culture? Why did you even enter that field in the first place? Genuine question.
Because I legitimately thought it would be a field where I could maximize how much of my brain I used. Instead, it turned out to be 99% dishwashing.
I very much loved undergraduate studies. I legitimately love studying what has been done and what has been discovered already. While we often say that we know nothing (it's somewhat true), the amount of human knowledge that already exists, especially in the life sciences, is unfathomable. And that's genuinely challenging and exciting.
But as soon as you start actually doing the research work, it's a lot less exciting, not much happens, and there is not a lot of problem solving (at least not in comparison to software). It's just a slow methodical grind, with pretty much no pivoting opportunities. You just stick to the plan, and you must try the same thing over and over again. I was surprised by that. Your experiences may differ depending on where you worked, on your funding obligations, and on your specific research area.
And after you finish your PhD, you might get lucky and get one of these fancy $ 40k post-doctoral fellow salaries. /s
(The way science works at a large scale is pretty fantastic, but how it feels for an individual is a very different thing.)
This just sounds like academia needs drastic reforms.
In Engineering we have 22 year olds directing technicians to do that kind of menial work.
The fact that the work is so unvaluable yet years were dedicated to it, seems like a failing somewhere. Not everything needs immediate economic value, but to see supposedly smart people not make 6 figures means:
>They arent that smart, smart engineers get picked up by irrelevant industries simply for being smart
>Someone is directing the smart people poorly.
Of course its both, the leadership of academia has failed us. Those who stick with Academia into PhDs and beyond are essentially second class performers who couldn't make it in Industry.
There are of course a lot of things I would like to change or improve, but I don't know if I share your broad pessimism on academia or research.
Science and engineering are very different endeavours, and while both should take lessons from each other, you just can't approach science in the same way that you approach engineering.
It sounds like my old company (not NASA) when we submitted tickets to offshore tech support - 2 days response time was the norm, and sometimes we never got a response back at all...
I told my dad about the Voyager fix, and he said that back in the days of punched cards, latency of 2 days was quite normal! He was still impressed at the high stakes of the repair though, with no option to go over and turn it off and on again.
My computer course in high school has a one week turnaround time. Send your punched card deck off Thursday of one week, get the printout back the next Thursday.
I'm in awe that it's maintainable even though it launched almost 50 years ago, while there is software deployed right now that won't be maintainable in the next 5 years.
Because it makes economic sense to apply extremely rigorous standards when writing software for a space probe, but not for a web app for expense reclaims?
When you inherit some random security update to an obscure library and it causes things to break, you were never expected to understand that edge-case.
However, if you spin up your own safety critical embedded system, you should be able to follow the logic completely.
Different costs. One is in the billions of dollars, one is in the hundreds of thousands of dollars.
True, yet I feel the expectation regarding software has shifted from expecting much better quality even 10-15 years ago to things we have now, where nobody is surprised about web apps crashing your browser etc.
> Through their investigation, Voyager's ground team discovered a single chip responsible for storing a portion of the FDS memory stopped working, probably due to either a cosmic ray hit or a failure of aging hardware. This affected some of the computer's software code.
IIRC, the Voyagers have redundant FDS computers, but I believe the non-active ones have been considered bad for decades. Now that they're seeing hardware failures on the active one, I wonder if they're evaluating the problems in the inactive ones to determine if they might actually be in better shape.
Would anyone be able to hack Voyager? Just wondering - if the software is that old, as long as you can get a signal to it anyone should be able to hack it / wipe it right?
Is the biggest barrier the enormous satellite dish you'd need to contact it or do the commands have a auth header with some key you'd need to brute force?
I think the big radio telescope is the main barrier. At this point there's only one in the world with enough transmit power to talk to it, and a handful that can receive the signal.
So they test in production, and they see what happens.
It's a really cool job they have (I'm not sarcastic), life-balance at NASA is known to be great ( = low to no pressure ) + since it's publicly funded, no clients, no profit to make, and if politics ask for feedback, you can claim anything and they can't verify it, and you can play with tech that others don't have the chance to see.
> It's a really cool job they have (I'm not sarcastic), life-balance at NASA is known to be great ( = low to no pressure ) + since it's publicly funded, no clients, no profit to make [...]
I've known some NASA folks and they would not describe it as "low to no pressure". It's highly bureaucratic, constantly under budget pressure, and contractors get much of the cool work.
Don't know about voyager, but I expect that many of those folks have been there decades-- like a closed society. I am sure it's awesome for them, more power to them, but it's certainly not the norm.
This sounds like a miscommunication. They probably don't have a simulator for the entire device, but attempting to fix the software without having an emulator of the computer would be downright irresponsible.
Not knowing that IF it would indeed show up! Although the benefit here is that after the 45 hours they would know if it's ever coming back, as opposed to having to keep waiting.
Something I always wonder about when Voyager has these issues is this: how long would it take for a probe launched today to catch up to Voyager? I assume there are better engines now, better capability for payload launch, better fuels, etc. I then wonder how long it would take to get that probe ready to be launched assuming that it was given the green light.
Most of the Voyager acceleration was done by using gravity assist ("slingshot flybys" -- effectively falling toward a large mass like Jupiter and missing). As far as I know gravity has not improved or become better since the 1970s.
It's a bit more complicated than falling and missing. Getting out of the gravity well takes the same amount of energy as you gain from falling into it, so those two things alone would be pointless.
I'm not an astrophysicist but I think you have to do it in such a way that you "steal" some of the planet's orbital momentum.
Barely related, but on the same spirit, I recommend Kafka's short story "The city coat of arms" for his perspective on the "there are better engines" line of reasoning:
I'm glad to see so many senior engineers on the team; it must be an exciting lifelong journey to work on the mission.
I have a feeling that code for NASA spacecrafts is not so high-level, for a reason.
PING voyager.1 (###.###.###.###) 56(84) bytes of data.
64 bytes from voyager.1 (###.###.###.###): icmp_seq=1 ttl=inf time=44.8 hrs
64 bytes from voyager.1 (###.###.###.###): icmp_seq=2 ttl=inf time=45.1 hrs
64 bytes from voyager.1 (###.###.###.###): icmp_seq=3 ttl=inf time=44.9 hrs
64 bytes from voyager.1 (###.###.###.###): icmp_seq=4 ttl=inf time=45.2 hrs
64 bytes from voyager.1 (###.###.###.###): icmp_seq=5 ttl=inf time=45.0 hrs
Imagine being able to close Vim.
And yes, I’m hopeless. Nano please.
And because a million different things can happen to cells over the course of 6 weeks, you don't get a line number, or a stack trace, or very much debugging information at all.
sudo make cell
I very much loved undergraduate studies. I legitimately love studying what has been done and what has been discovered already. While we often say that we know nothing (it's somewhat true), the amount of human knowledge that already exists, especially in the life sciences, is unfathomable. And that's genuinely challenging and exciting.
But as soon as you start actually doing the research work, it's a lot less exciting, not much happens, and there is not a lot of problem solving (at least not in comparison to software). It's just a slow methodical grind, with pretty much no pivoting opportunities. You just stick to the plan, and you must try the same thing over and over again. I was surprised by that. Your experiences may differ depending on where you worked, on your funding obligations, and on your specific research area.
And after you finish your PhD, you might get lucky and get one of these fancy $ 40k post-doctoral fellow salaries. /s
(The way science works at a large scale is pretty fantastic, but how it feels for an individual is a very different thing.)
In Engineering we have 22 year olds directing technicians to do that kind of menial work.
The fact that the work is so unvaluable yet years were dedicated to it, seems like a failing somewhere. Not everything needs immediate economic value, but to see supposedly smart people not make 6 figures means:
>They arent that smart, smart engineers get picked up by irrelevant industries simply for being smart
>Someone is directing the smart people poorly.
Of course its both, the leadership of academia has failed us. Those who stick with Academia into PhDs and beyond are essentially second class performers who couldn't make it in Industry.
Science and engineering are very different endeavours, and while both should take lessons from each other, you just can't approach science in the same way that you approach engineering.
Two days would have been luxury.
If not, I’d quite like to try a course on punchcard usage.
When you inherit some random security update to an obscure library and it causes things to break, you were never expected to understand that edge-case.
However, if you spin up your own safety critical embedded system, you should be able to follow the logic completely.
Different costs. One is in the billions of dollars, one is in the hundreds of thousands of dollars.
IIRC, the Voyagers have redundant FDS computers, but I believe the non-active ones have been considered bad for decades. Now that they're seeing hardware failures on the active one, I wonder if they're evaluating the problems in the inactive ones to determine if they might actually be in better shape.
Is the biggest barrier the enormous satellite dish you'd need to contact it or do the commands have a auth header with some key you'd need to brute force?
(pdf) https://deepspace.jpl.nasa.gov/files/820-100-H.pdf
70 meter antenna, 80 kW X-band transmit power (122 dBW EIRP)
HN thread [1]
[0]: https://old.reddit.com/r/explainlikeimfive/comments/1cbhvwg/...
[1]: https://news.ycombinator.com/item?id=40141836
Has anyone attempted to build this? Assuming the documentation is available
So they test in production, and they see what happens.
It's a really cool job they have (I'm not sarcastic), life-balance at NASA is known to be great ( = low to no pressure ) + since it's publicly funded, no clients, no profit to make, and if politics ask for feedback, you can claim anything and they can't verify it, and you can play with tech that others don't have the chance to see.
Don't know about voyager, but I expect that many of those folks have been there decades-- like a closed society. I am sure it's awesome for them, more power to them, but it's certainly not the norm.
I didn't know about the contractors.
It still seems like a great place to work for, even in terms of social recognition.
More discussion on official post:
https://news.ycombinator.com/item?id=40117599
111001111011110010000011000111101111110111111101100 ok
CONSTANT X ok
HEX X U. 18F7EFEC okDECIMAL X U. 418901996 ok
8 BASE ! X U. 3075767754 ok ok
I give up. What does this number represent?
I'm not an astrophysicist but I think you have to do it in such a way that you "steal" some of the planet's orbital momentum.
https://what-if.xkcd.com/38/
Barely related, but on the same spirit, I recommend Kafka's short story "The city coat of arms" for his perspective on the "there are better engines" line of reasoning:
https://www.babelmatrix.org/works/de/Kafka%2C_Franz-1883/Das...