Defense attorneys in a DUI case got their hands on the source code for the breathalyzer. It turned out to have terrible programming, e.g. calculating new averages by averaging a new value with the previous average. The case went all the way to the New Jersey Supreme Court, which still found the device to be acceptable.
Well, of course, because that statement is deeply incorrect—the described mistake would cause the most recent reading to have more weight.
If you have a set of readings, say, [0.1, 0.02, 0.3, 0.05, 0.08], normally when you average them you would get 0.55—the mean of the set.
Calculating the average by "averaging the new reading with the previous average" would mean new + old / 2 every time. That means that for each reading after the first, your "averages" would be: [0.06, 0.18, 0.115, 0.195].
If we add a new reading of 0.01 to each of these, in the first case, we would get an average of 0.46, and in the second case, 0.1025. As you can see, even taking into account the already-very-skewed numbers, the second case biases it much further in favor of the new reading (which, in this case, is very low compared to the existing readings).
yeah the language is slightly ambiguous enough where you can't for certain know. first vs newest.
the issue is that taken as a whole the quantization of the samples into 8 bins is a much bigger issue along with the problem of the no hardware watchdog or hardware malfunction alarms. along with the poor testing methodologies too.
Idk, hard to guess based on the layman description, but maybe what they are describing is a rolling average? That would be quite a standard low-pass filter for applications like this.
Hard to get upset over that. What matters is not the signal processing but the validation. You take a bunch of people with various blood alcohol levels measured by some already accepted lab technique and you verify that your new measurement technique is measuring within some acceptable error bound of that.
An exponential moving average is a form of weighted rolling average that would match the plain-text description in the complaint and is often used in signal processing as an IIR low-pass filter (an unweighted rolling average over a window is an FIR filter)
There's an excellent Radiolab podcast episode about how often cosmic rays cause computer errors in practice. It's engaging and educational: https://radiolab.org/podcast/bit-flip
I always knew about the theoretical cosmic ray bit flips. Before listening to this episode, I did not stop to think how often they actually cause problems.
I tried to observe bit flips and failed. I'm not sure if there was a problem with my methodology or my apartment was well shielded or what.
My methodology was simple. On a Linux home server that had plenty of spare memory (non ECC RAM) I ran a process that simply alloced a large buffer and filled it with a pattern. It would then periodically scan through the buffer looking for changes to the pattern.
I ran this for over a year which should have been long enough given the amount of RAM I was using and the rates that I found in the literature for cosmic ray induced bit flips resulted in several flips.
My method would have missed a flip if the page it happened on had been paged out sometime in the past and that paged out copy still existed, and the bit flip happens between the time of the last scan and the time the kernel decides to discard that page. On the next scan it would page fault and load the good page.
But the system was very lightly loaded and almost never actually had to page things out, so most of the time if a bit flipped it should have still been there by the next scan and so I don't think this explains why I saw no flips.
A few years later I got a Mac Pro, which had ECC memory. I used that mostly at work from 2008-2017. I got another Mac Pro in 2009 which I used at home from 2009-2017. I'd occasionally look at the memory status in System Report which should say "ECC Errors" if the ECC had to fix any errors and only ever saw "OK". I'm not sure if that resets on boot and I only looked occasionally so if it does reset than it is quite likely I would have missed an error statuses.
My experience from a ~ 2000 server job with all ECC RAM is that errors cluster a lot. Easily 90%+ of the systems never had any ECC errors (correctable or no) in their whole life. Some systems would get one error a day. A small number got 100s per hour until we shut them down. Even less got thousands per hour. One in my whole career got like 50,000 in the time between when it started running very slowly (from processing ECC interrupts) and the next hourly reporting interval. You might get close to the literature numbers by taking that one instance and dividing it over the total ram byte hours. Although that was probably hardware failure and not comsic rays, but then you don't get a different ECC code for comsic rays.
Also, there was an expectation that comsic ray induced bit errors would grow as ram circuit features shrunk, but it ended up not happening; reasons unknown or at least I never saw anything suggesting a reason.
Getting errors in a small sample of RAM is unlikely, unless you specifically induce them by using debug features or misconfiguring your system (but some systems conspire to misconfigure themselves, making it easier to observe! a couple years ago, retail motherboards really liked setting the ram voltage too low)
Were some of your servers in lead-lined coffins while others weren’t?
Jokes aside, that it seems really unlikely that cosmic rays would be clustered past, like, a couple hours, right? It isn’t like some neutron star is, like, tracking your server as the world spins (well, I hope not, I mean who’d you piss off for that to happen?).
Anyway, this fits my totally unscientific expectation that cosmic rays are just sort of like an informal description of hardware bugs that nobody can reasonably find beforehand. A server that seems to be hit by lots of cosmic rays probably has a dodgy connection somewhere inside it, but I mean maybe it’s the RAM, swap that out or replace it… but maybe inside the chip SOC, so what are we going to do, bust out the electron microscope to check all those connections?
Yes, these kinds of error rates almost certainly indicate either hardware faults of some kind, unshielded electrical noise, or possibly poor cooling pushing temps past the stable operating range.
You'd have to ask our host; these were all rental dedicated servers, none of us ever got to see them at all. Pictures of their racks never included coffins though. I don't think we pissed anybody off enough for them to get out the cosmic ray gun, but I'm also thinking the people we did piss off didn't have cosmic ray guns anyway. ;)
In terms of diagnostics, we pretty much just asked for ram replacement, if that didn't work, cpu replacement, if that didn't work, motherboard replacement. If that didn't work, the chassis / rack position is clearly cursed, don't give us anything there again, please. :D I don't know what they did with the hardware we didn't like, maybe send it to the manufacturer, maybe give it to customers they don't like, maybe surplus it.
Joking aside that's incredibly fascinating, I never thought that ECC memory has that much of a performance impact. Might be more optimal to just get a large jerry can of water and put that over the server as radiation shielding lol.
ECC interrupts were never a problem at reasonable counts. It's just this machine where the memory was falling apart where it was a problem. Our system was robust to a machine halting, but not so great at dealing with a machine running very very slow. Plus, it wasn't easy to connect and shut down the service (maybe we should have just killed it from IPMI, but this was the only time it happened, so learning experience).
I had a similar issue one time where a Pentium III era server rebooted and came up with a comically small amount of memory, maybe 2-4 mb instead of 128 mb. That wasn't too bad, because it was a very lightly service; important to be on its own machine for reasons, but didn't need much. Just ran a little slow when it was running from swap. I think it did trigger a swap usage alert, and then it was like why is it swapping, why is it so slow, wait why doesn't it have any memory!?
I think this study https://www.cs.toronto.edu/~bianca/papers/ASPLOS2012.pdf (full disclosure: authors were my labmates) supports your observations, and supports the notion that cosmic rays are not the leading cause of random bit-flips in RAM.
This particular server wasn't mine as a product manager. But we had what passed as a distributed server in the mid-80s and our biggest company was a retail insurance/finance company of some sort. No ECC.
So, at scale, they were getting failures constantly. (It didn't help that the QIC tape backup was basically write-only.)
I think it's possible that we've historically mis-attributed memory errors to cosmic rays when in fact they were caused by heat.
If you ran that machine in a hotbox at 85°C for a year I think you probably would have experienced higher error rates. Bonus points if you also force the data to swap out a lot so it gets transferred through as many data paths as possible.
The incidence of cosmic rays is biased vertically, especially close to sea level where you get more muons than electrons, so that is one thing you would want to think about - you’d want to maximize your surface area toward that because your detection volume is so low.
As a rule of thumb, you get 2 muons through your head a minute - but of course your head has a huge volume compared to memory chips.
I recall seeing system log messages about ECC error correction on Sun Sparc machines back in the 90's. I don't know if it was radiation or flaky memory though.
I always assumed computers were "perfect"- at least, always executing their instructions according to a spec. FDIV and other bugs convinced me otherwise. Then, I worked on a system (a large processing accelerator) where some small fraction of machines had chips that would fail on a limited set of operations (producing the wrong result for a known input, vaidated by offline computation).
Now I treat computers as statistically correct- you have to assume there is some level of corruption/error occurring due to hardware bugs or physical events.
you would enjoy Dave Ackley's perspective on 'robust computing' vs 'efficient computing'. But his one of his ideas is allowing for error: which can speed up calculations if you can also reduce the error as the calculation goes on.
Well... there's several things going wrong here and his perspective only seems to cover some of them.
In the case of a scientific simulation, we can find algorithmic speedups that are "lossy"- an example would be approximations to n-body systems, where you need to calculate n-squared interactions (between all pairs). You can calculate all n-squared interactions which produces teh completely correct result. But since atoms that are far away don't interact strongly (falls off as 1 over r squared or more). So you can maintain a neighborlist- all atoms within a distance R- cheaper than you can calculate n-squared interactions, with some tiny error that is unknown. It's assumed in many cases the errror is neglible and the speedup is huge.
Or you can switch to using a particle-mesh method which involves taking a fourier transform, doing some work in fourier space, then an inverse transform. The results are nlogn, and the error is small (and known). Speedup is signfiicant but takes much, much more computational skill and infrastructure.
Next, the case I'm referring to of an accelerator running a tensorflow job, it's a totally different scenario- here, some random subset of machines will repeatedly return the wrong result- say, for a matrix-vector operation. Maybe garbage numbers, maybe all zero, maybe some infs. When that gets summed into your gradient, it often causes blow-up and the entire job terminates. It's not clear whether it makes sense to make high-performance jobs have to be robust- I see them as special cases where you're working hard to make sure the computing substrate is effectively 100% reliable (by sending/fixing those machines).
Other folks have observed that some amount of small noise injection to the gradient can help training, but the sorts of errors I've seen almost immediately terminate the training job. I don't mind intentional noise addition, but noise due to hardware that is provably, reliably, and repeatedly miscalculating results? Not so much.
At work we make a hardware product, and among other things, it has a small microcontroller on it. We keep several hundred instances of our product to run automated integration tests. At that scale, we find SRAM bit-flips one of the microcontrollers every few weeks. We have a handful of software measures that are able to stop most instances of bit-flip from having a user-visible effect.
The rate of cosmic rays impacting hardware should be extremely low, unless there's some kind of environmental factor (like being at a very high altitude). Or maybe there was a special train going by as in: https://www.jakepoz.com/debugging-behind-the-iron-curtain/
As we went from zero to 10K+ embedded systems (full PCs with significant RAM) the issues got weirder.
The best was a one-off error log along the lines of "unknown type System.DateTime". Huh? That's a system defined type that just went missing. Never saw it again.
Another at a different employer was a crash that occurred after a check condition that absolutely should have gated the crash from being reached. Single threaded. Simple microcontroller. Had to reflash it to flip the bit back. After doing the math on how much RAM we had in the wild vs. cosmic bit flip rates reported in super computers, we had to expect one flip per year.
If it's a safety critical system, server or not, use ECC RAM!!
I am of the opinion that far more than safety critical systems should use ECC. You should use ECC anytime bit flips might cost you more money then the ECC does, which is why I insist on ECC for my desktop computers.
It's interesting to see the negative reactions to this while an awful lot of us are employed specifically and payed exorbitantly because computers regularly are not operating correctly in weird subtle ways that are hard to figure out. Especially as society leans into ML models to solve computationally hard problems, the legal notion that the computer is "correct" by default absolutely needs to go out the window.
We are paid exorbitantly to work hard to maintain the illusion that the computers are operating correctly.
Any reasonably competent software developer or engineer should know that just because the customer dashboard is green and everything is working as expected doesn't mean that there isn't an absolute dumpster fire raging in the background (at least, having worked in tech for 20 years across many different verticals, that has been my experience).
And that's even with things operating to some semblance of regularity and conformance to some established baseline.
Not even covering the "kick it down the road" issues that sometimes (and sometimes don't) evolve into "features that aren't bugs" that have workarounds on workarounds on workarounds that will eventually have their own bugs. The stuff we know we shouldn't do / that isn't good, but that happens anyway.
I subscribe to the philosophy that computers cannot be held accountable, because they're only operating as intended, or at least as implemented. CentOS used to bill itself as a "bug for bug" compatible RHEL clone. I have always thought that to be a good perspective.
Humans are ultimately responsible for the results of computing. In my life that means I review the reports output by my ERP and sanity check numbers before I sign tax returns or remit quarterly tax payments. I trust that the numbers going in are correct, and I trust that (as was said up-thread) statistically my ERP is reliable, but at the end of the day someone has to be accountable for how the output is used.
My dad and I were watching a TV show where someone received a time traveling fax. I remarked that the software on the Fax machine may have just had a bug.
He immediately remarked "they're scientists (physicists who sent the fax) and it was impossible that they wouldn't have accounted for that".
I've been a software engineer for 10 years. He's a well-read hard-working blue-collar guy, working as a taxi driver and behind a deli most of his career. I just nodded and moved past it.
People want to anthromorphize AI. People want to yield divine knowledge to computers. Any sufficiently advanced technology is indistinguishable from magic indeed.
Heh. Ask him if the people at the deli he works at are generally competent.
I have a somewhat awful belief about humanity: that about 70% of people work jobs they’re not very good at. Most of the time this is ok, because so many jobs are bullshit anyway. Bad real estate agents still sell houses. Grumpy sales people still sell product. Social media managers - well, let’s not go there. But it has some strange consequences - like how most therapists are (in one study) less useful for the patient than journaling. Or in my opinion, the prevalence of crap software.
Ask about his work. Ask if his coworkers made a fax machine, if it would work reliably. If he gets it, you’ll know. You’ll see it in his eyes.
I think there's a huge and fundamental difference between the assertion that a computer program has malfunctioned and the assertion that a computer program does not accomplish what it intended to do at all, that the algorithm is incorrect, or that there simply is no known reliable way for any computer program to do the thing.
The latter problem is more important, but by lumping this together with "malfunction" and giving technologists basically a complete pass on the entire hard part, this kind of rule is a loophole wide enough to pass a jetliner through
The authors seem to be torching a semantic straw man here. The same abuse of terms affords their spicy title.
I'm not a UK lawyer, but the law they quote says nothing about the logic machines are programmed to follow presumptively creating reliable evidence. It could be read to say that computers should be presumed to be executing the instructions they're given reliably, unless evidence shows otherwise. It's about malfunction, not misapplication.
Perhaps some of the Horizon case decisions showed judges improperly presuming that Horizon calculated correctly, and not just that the computers were running Horizon correctly. But the article doesn't show they did, or even explicitly say they did. Conflating two separable issues, it fails to address whether or why different presumption rules for each might be desirable.
"
The Law Commission failed to address the strongest arguments against repeal without replacement, …. It ignored the advice of the experts they cited who all argued that the focus of courts should be on the reliability of computer evidence, …. The Law Commission’s comments and conclusions revealed that they had not understood the nature of computers and complex software systems as described in the sources upon which they relied.’
"
https://www.computerweekly.com/opinion/The-cause-of-the-Post...
I would like to get those commission members to sit down and use a computer program which asks them their name, does a fancy "analyzing" animation, and concludes that that they owe $1 million in fees and are 99% likely to be the perpetrator of an unsolved murder.
I mean, hey, the computer is operating correctly, it can't make a mistake, so give me my money and turn yourself in...
"It could be read to say that computers should be presumed to be executing the instructions they're given reliably, unless evidence shows otherwise. It's about malfunction, not misapplication."
You are making a technical distinction the law does not make. The law relates to the output - to the evidence generated from the system as a whole. The presumption is that the logic is correct too - which is why it is such a terrible assumption.
In the specific case of the Post Office miscarriages of justice, the system as a whole included Horizon central office staff who made manual and sometimes incorrect adjustments to the individual Post Office ledgers, and the discrepancies were later blamed on the postmasters.
Lots of people here saying judges and lawyers are incredibly stupid. For decades there was a giant mystique surrounding software and computers, the belief that computers didn't make mistakes. And who promoted that belief? Computer and software companies. So they could make absurd amounts of money. Credit where its' due. Courts should have been much more critical, but they were getting hit with a cultural tidal wave generated by us.
The products were promoted using practices approved by the legal system. If those practices aren't appropriate to use when targeting judges and lawyers, then they're not appropriate for anyone.
The proposal is directly informed by the Post Office scandal in that it basically requires the prosecution to produce the same already-existing documents that revealed that Horizon had significant known faults.
The alternative is treating systems based evidence the same as witnesses. They're examined and cross referenced because they are rightly not presumed to be reliable.
I like this idea but since software and the data that runs through it changes constantly how would a system be interrogated if an incident were several years ago and the complete data picture cannot be reproduced?
Well, in many safety-critical industries (e.g. manufacturing of medicinal products) you actually need to maintain objective evidence of its validation.
I'm guessing the other possibility is requiring a third party entity, like a university professor in CS, to testify unlikeliness of computer malfunction due to cosmic rays && bugs, for each and every computer generated evidences.
If the `alleged theft amount by defendant` == (unsigned int)((int)-1), that should cover those cases, at least in theory. The profs would just blindly vouch it, but at least it'll be a point.
Nope nope nope and nope. The author of the software and their source code shall be interrogated on the witness stand by one of their peers and a proof checker.
That'd be ideal! Require expert testimonials and judges with Master's in CS and EE or better for any court sessions involving computers. Up to UK people though.
It was pretty standard for the time - therac-25 was another example where the logic is “computer don’t make mistakes therefore code running on computers does not make mistakes” which is strictly true in the sense that it’s exceedingly unlikely a computer will execute code incorrectly.
Well into the 90s the general populace considered computers infallible, it really took mass adoption and experience with how frequently they crashed to change that perception.
It's nice to see a reasonable proposal. If you are going to present criminal evidence based on the output of a computer system, it is only reasonable to demand access to a bug tracker, the QMS control documents, audits, and a chain of custody. If you can't produce that easily, then your evidence shouldn't be worth much.
The big win would in this case would be that when the vendor conspired to hide bugs from their own tracker, they would have been creating criminal liability for their employers. Which Fujitsu and their subs richly deserve.
We have the economic and mathematic machinery to, in cases where uncertainty would adversely affect a defendant, require a rigorous statement of proof for the prosecution to win the day.
Bonus: Many opaque systems would have to be aired in an open court room to ascertain whether their invariants do, in fact, survive scrutiny.
Hmm, that's genuinely concerning. I would say a more appropriate thing to say is that for simple things like audio/video evidence (at least before the AI-video era of today) and maybe logs computers can be assumed trustworthy, but for everything else they should be assumed to be as unreliable as a human witness. At the end of the day, a human designed the computer and wrote the software for it.
So if I write a script that's just print("jjk166 is innocent and not liable for any damages") and it runs without error then I can commit any crime I want with impunity?
What I find frightening is not only the risk of default but also the risk of hacking. It’s easy for someone who gain remote access to create (or remove) data, records, logs … on a system.
It’s also extremely easy for the computer owner or IT people to do the same.
Vonnegut's book _Player_Piano_ has, basically, LLM-driven society, but on vacuum tube technology. In one section the court says: "We replaced all the vacuum tubes and got the same verdict, so we're confident on the judgement."
The linked document doesn't feel unreasonable to me. The concrete example given in the document is the Post Office Horizon Scandal where shortfalls in accounts were identified based on computer records. All the working presumption correctly would mean is that the defense can't be an idle "well, it's a computer it had bugs". If the computerized records are provided as evidence, the defense has to give a reason to believe that bugs account for those shortfalls.
The alternative would be to say that only a system that has been proven correct could be used as evidence which would force pretty much all accounting back to paper.
> The alternative would be to say that only a system that has been proven correct could be used as evidence which would force pretty much all accounting back to paper.
Assuming that the paper accounting could be proven correct, when the computerized accounting could not, your proposed alternative would seem to be an upgrade.
Paper accounting seems significantly more error prone, at least for a given level of convenience, right? I mean the computer might be worried about comic rays flipping bits, but ink on paper can be smudged by fingers, and they don’t even have to be traveling at an appreciable percentage of the speed of light.
You can make paper backups of course, but you can also make electronic ones, the electronic ones can be shipped around the world basically for free.
The legal system already has checks and balances to account for errors in paper accounting. They don't require an assumption that the paper accounting is correct, but rather rely on the jury to make judgements, because juries are familiar with the limitations of paper accounting.
We're still a long way away from pulling that off with computers, but the idea that someone could have made a mistake shouldn't be that hard to grasp.
> The alternative would be to say that only a system that has been proven correct could be used as evidence which would force pretty much all accounting back to paper.
No, there are many alternatives. Including, most critically, the one laid out in the paper under discussion. Which is – spoilers – not “only a system that has been proven correct could be used as evidence”.
That's exactly what this proposal addresses. Consider the paper they cite, in part, for the proposal:
"It is a matter of surprise that important documentary records, such as the Fujitsu Known Error Log (KEL), were disclosed only in response to a direction from the court and in the face of opposition by the Post Office." (https://journals.sas.ac.uk/deeslr/article/view/5240/5083)
The proposal in this paper would require prosecution to disclose extant documents related to the reliability of the system. In the Horizon case, such documents existed, but were never disclosed, as current practices did not require the prosecution to do so.
I believe so, but they have a notorious track record of not doing so.
But from other articles it sounds like these prosecutions involved some kind of parallel legal system just for the mail?
I think though that the issue that this saga hit is that the law was such that the computer “evidence” was presumptively correct so they tried to pretend any “bugs” they were aware of didn’t change that fact and so the existence of bugs was not “exculpatory” - obviously BS, but it seems like that was the core behaviour/belief of the post office. That put the victims in the position of having to disprove the accusation but the only “evidence” in the case was the presumptively true report from the buggy system.
Ie the only evidence presented was a system that was buggy, but the victim could not get evidence the system was broken without first proving it was broken.
Would that be a bad thing? If you want the convenience you either pay for something that is proven correct or you take the inefficiency of paper (under your implicit assumption that it's somehow invulnerable to error, burning down, etc), or you accept that you take the convenience but will not be able to use it in court as evidence. If you suspect somebody is stealing, you could always take further measures like switching to magical paper that can't go wrong, or set regular inspections, or new procedures, etc.
Don't pretend that fucking over innocent lives in the pursuit of profits is a law of nature.
>The alternative would be to say that only a system that has been proven correct could be used as evidence which would force pretty much all accounting back to paper.
No. It'd actually just force authorship and disclosure of test cases.
This is just an assumption that generally, computers are doing what they're supposed to. It doesn't mean that you can't challenge it with evidence. With the amount of evidence being submitted being increasingly digital, I'm sure the intention behind this is to stop people from jamming up the courts by calling every single digital artefact into question.
The article makes a strong case otherwise. Proving a system built and maintained by a large, well heeled entity is unlikely to be possible for all except other large, well heeled entities.
> This is just an assumption that generally, computers are doing what they're supposed to.
It's more than that. You don't have to instruct a jury that they have to accept that a person does what they're supposed to do unless proven otherwise... unless that person wrote code for a computer.
I can understand how they got here: Most computer-provided evidence will be something like a printout of an email thread or a saved Excel sheet. Proving that the hard drive, printer, CPU, drivers, application software, etc. are all untampered and working properly is a huge amount of work that will very rarely end up finding a problem.
The original rule-makers probably never considered the idea of a giant, bespoke enterprise software system; it's just a particularly terrible edge case that got caught up in what looks like a reasonable efficiency measure.
> I can understand how they got here: Most computer-provided evidence will be something like a printout of an email thread or a saved Excel sheet. Proving that the hard drive, printer, CPU, drivers, application software, etc. are all untampered and working properly is a huge amount of work that will very rarely end up finding a problem.
Eye witnesses, expert witnesses, etc. can be unreliable. It's very difficult to prove that they have not made any errors. However, we don't ask juries to presume they are infallible.
> Eye witnesses, expert witnesses, etc. can be unreliable. It's very difficult to prove that they have not made any errors. However, we don't ask juries to presume they are infallible.
There is a difference between being infallible and being correct. No one assumes that computers cannot err, just that they have not erred unless there is reason to believe they have. Likewise, if a witness gives coherent evidence and no one has any reason to assume they are wrong or lying, that evidence will not generally be disregarded simply because humans are fallible and therefore the evidence is presumed to be flawed.
With an eyewitness, simply saying that you think they are mistaken, with no evidence as to how they are mistaken, is sufficient. In this case, it was easier for the courts to believe hundreds of people were lying and committing fraud, then that there might have been a problem in the software. The only way to change the court's perspective was to provide evidence that was not even accessible to the defendants.
There's no reason to think that programmers are less fallible than other people.
> required the prosecution to prove that a computer was operating properly at the relevant time before a document produced by such a computer could be admitted as evidence.
But I'm sure you actually mean to agree, and that your comment was just garbled in transit. :)
That's fair. But as the article shows, that presumption ends up weighing heavily in favor of computers working correctly. The nuance is in the difference between plumbing, which generally works well, and the bespoke business logic of individual applications, especially when said logic was responsible for the existence of critical evidence in a criminal case. The difference matters enormously and is not observed at all in the law.
Defense attorneys in a DUI case got their hands on the source code for the breathalyzer. It turned out to have terrible programming, e.g. calculating new averages by averaging a new value with the previous average. The case went all the way to the New Jersey Supreme Court, which still found the device to be acceptable.
If you have a set of readings, say, [0.1, 0.02, 0.3, 0.05, 0.08], normally when you average them you would get 0.55—the mean of the set.
Calculating the average by "averaging the new reading with the previous average" would mean new + old / 2 every time. That means that for each reading after the first, your "averages" would be: [0.06, 0.18, 0.115, 0.195].
If we add a new reading of 0.01 to each of these, in the first case, we would get an average of 0.46, and in the second case, 0.1025. As you can see, even taking into account the already-very-skewed numbers, the second case biases it much further in favor of the new reading (which, in this case, is very low compared to the existing readings).
1-Yes
2-No
3-Unspecified
Of course the average was around 2.011.
With software that bad, who can say
the issue is that taken as a whole the quantization of the samples into 8 bins is a much bigger issue along with the problem of the no hardware watchdog or hardware malfunction alarms. along with the poor testing methodologies too.
Hard to get upset over that. What matters is not the signal processing but the validation. You take a bunch of people with various blood alcohol levels measured by some already accepted lab technique and you verify that your new measurement technique is measuring within some acceptable error bound of that.
Which is not appropriate here...
I always knew about the theoretical cosmic ray bit flips. Before listening to this episode, I did not stop to think how often they actually cause problems.
My methodology was simple. On a Linux home server that had plenty of spare memory (non ECC RAM) I ran a process that simply alloced a large buffer and filled it with a pattern. It would then periodically scan through the buffer looking for changes to the pattern.
I ran this for over a year which should have been long enough given the amount of RAM I was using and the rates that I found in the literature for cosmic ray induced bit flips resulted in several flips.
My method would have missed a flip if the page it happened on had been paged out sometime in the past and that paged out copy still existed, and the bit flip happens between the time of the last scan and the time the kernel decides to discard that page. On the next scan it would page fault and load the good page.
But the system was very lightly loaded and almost never actually had to page things out, so most of the time if a bit flipped it should have still been there by the next scan and so I don't think this explains why I saw no flips.
A few years later I got a Mac Pro, which had ECC memory. I used that mostly at work from 2008-2017. I got another Mac Pro in 2009 which I used at home from 2009-2017. I'd occasionally look at the memory status in System Report which should say "ECC Errors" if the ECC had to fix any errors and only ever saw "OK". I'm not sure if that resets on boot and I only looked occasionally so if it does reset than it is quite likely I would have missed an error statuses.
Also, there was an expectation that comsic ray induced bit errors would grow as ram circuit features shrunk, but it ended up not happening; reasons unknown or at least I never saw anything suggesting a reason.
Getting errors in a small sample of RAM is unlikely, unless you specifically induce them by using debug features or misconfiguring your system (but some systems conspire to misconfigure themselves, making it easier to observe! a couple years ago, retail motherboards really liked setting the ram voltage too low)
Jokes aside, that it seems really unlikely that cosmic rays would be clustered past, like, a couple hours, right? It isn’t like some neutron star is, like, tracking your server as the world spins (well, I hope not, I mean who’d you piss off for that to happen?).
Anyway, this fits my totally unscientific expectation that cosmic rays are just sort of like an informal description of hardware bugs that nobody can reasonably find beforehand. A server that seems to be hit by lots of cosmic rays probably has a dodgy connection somewhere inside it, but I mean maybe it’s the RAM, swap that out or replace it… but maybe inside the chip SOC, so what are we going to do, bust out the electron microscope to check all those connections?
In terms of diagnostics, we pretty much just asked for ram replacement, if that didn't work, cpu replacement, if that didn't work, motherboard replacement. If that didn't work, the chassis / rack position is clearly cursed, don't give us anything there again, please. :D I don't know what they did with the hardware we didn't like, maybe send it to the manufacturer, maybe give it to customers they don't like, maybe surplus it.
It's not 3.6 roentgen...
Joking aside that's incredibly fascinating, I never thought that ECC memory has that much of a performance impact. Might be more optimal to just get a large jerry can of water and put that over the server as radiation shielding lol.
I had a similar issue one time where a Pentium III era server rebooted and came up with a comically small amount of memory, maybe 2-4 mb instead of 128 mb. That wasn't too bad, because it was a very lightly service; important to be on its own machine for reasons, but didn't need much. Just ran a little slow when it was running from swap. I think it did trigger a swap usage alert, and then it was like why is it swapping, why is it so slow, wait why doesn't it have any memory!?
I'm told it's the equivalent of a chest x-ray...
I see you, brother! :D
So, at scale, they were getting failures constantly. (It didn't help that the QIC tape backup was basically write-only.)
If you ran that machine in a hotbox at 85°C for a year I think you probably would have experienced higher error rates. Bonus points if you also force the data to swap out a lot so it gets transferred through as many data paths as possible.
As a rule of thumb, you get 2 muons through your head a minute - but of course your head has a huge volume compared to memory chips.
In the case of a scientific simulation, we can find algorithmic speedups that are "lossy"- an example would be approximations to n-body systems, where you need to calculate n-squared interactions (between all pairs). You can calculate all n-squared interactions which produces teh completely correct result. But since atoms that are far away don't interact strongly (falls off as 1 over r squared or more). So you can maintain a neighborlist- all atoms within a distance R- cheaper than you can calculate n-squared interactions, with some tiny error that is unknown. It's assumed in many cases the errror is neglible and the speedup is huge.
Or you can switch to using a particle-mesh method which involves taking a fourier transform, doing some work in fourier space, then an inverse transform. The results are nlogn, and the error is small (and known). Speedup is signfiicant but takes much, much more computational skill and infrastructure.
Next, the case I'm referring to of an accelerator running a tensorflow job, it's a totally different scenario- here, some random subset of machines will repeatedly return the wrong result- say, for a matrix-vector operation. Maybe garbage numbers, maybe all zero, maybe some infs. When that gets summed into your gradient, it often causes blow-up and the entire job terminates. It's not clear whether it makes sense to make high-performance jobs have to be robust- I see them as special cases where you're working hard to make sure the computing substrate is effectively 100% reliable (by sending/fixing those machines).
Other folks have observed that some amount of small noise injection to the gradient can help training, but the sorts of errors I've seen almost immediately terminate the training job. I don't mind intentional noise addition, but noise due to hardware that is provably, reliably, and repeatedly miscalculating results? Not so much.
The best was a one-off error log along the lines of "unknown type System.DateTime". Huh? That's a system defined type that just went missing. Never saw it again.
Another at a different employer was a crash that occurred after a check condition that absolutely should have gated the crash from being reached. Single threaded. Simple microcontroller. Had to reflash it to flip the bit back. After doing the math on how much RAM we had in the wild vs. cosmic bit flip rates reported in super computers, we had to expect one flip per year.
If it's a safety critical system, server or not, use ECC RAM!!
[0] - https://podcastaddict.com/wissenschaft-auf-die-ohren/episode...
Any reasonably competent software developer or engineer should know that just because the customer dashboard is green and everything is working as expected doesn't mean that there isn't an absolute dumpster fire raging in the background (at least, having worked in tech for 20 years across many different verticals, that has been my experience).
Not even covering the "kick it down the road" issues that sometimes (and sometimes don't) evolve into "features that aren't bugs" that have workarounds on workarounds on workarounds that will eventually have their own bugs. The stuff we know we shouldn't do / that isn't good, but that happens anyway.
I subscribe to the philosophy that computers cannot be held accountable, because they're only operating as intended, or at least as implemented. CentOS used to bill itself as a "bug for bug" compatible RHEL clone. I have always thought that to be a good perspective.
Humans are ultimately responsible for the results of computing. In my life that means I review the reports output by my ERP and sanity check numbers before I sign tax returns or remit quarterly tax payments. I trust that the numbers going in are correct, and I trust that (as was said up-thread) statistically my ERP is reliable, but at the end of the day someone has to be accountable for how the output is used.
He immediately remarked "they're scientists (physicists who sent the fax) and it was impossible that they wouldn't have accounted for that".
I've been a software engineer for 10 years. He's a well-read hard-working blue-collar guy, working as a taxi driver and behind a deli most of his career. I just nodded and moved past it.
People want to anthromorphize AI. People want to yield divine knowledge to computers. Any sufficiently advanced technology is indistinguishable from magic indeed.
I have a somewhat awful belief about humanity: that about 70% of people work jobs they’re not very good at. Most of the time this is ok, because so many jobs are bullshit anyway. Bad real estate agents still sell houses. Grumpy sales people still sell product. Social media managers - well, let’s not go there. But it has some strange consequences - like how most therapists are (in one study) less useful for the patient than journaling. Or in my opinion, the prevalence of crap software.
Ask about his work. Ask if his coworkers made a fax machine, if it would work reliably. If he gets it, you’ll know. You’ll see it in his eyes.
The latter problem is more important, but by lumping this together with "malfunction" and giving technologists basically a complete pass on the entire hard part, this kind of rule is a loophole wide enough to pass a jetliner through
I'm not a UK lawyer, but the law they quote says nothing about the logic machines are programmed to follow presumptively creating reliable evidence. It could be read to say that computers should be presumed to be executing the instructions they're given reliably, unless evidence shows otherwise. It's about malfunction, not misapplication.
Perhaps some of the Horizon case decisions showed judges improperly presuming that Horizon calculated correctly, and not just that the computers were running Horizon correctly. But the article doesn't show they did, or even explicitly say they did. Conflating two separable issues, it fails to address whether or why different presumption rules for each might be desirable.
" The Law Commission failed to address the strongest arguments against repeal without replacement, …. It ignored the advice of the experts they cited who all argued that the focus of courts should be on the reliability of computer evidence, …. The Law Commission’s comments and conclusions revealed that they had not understood the nature of computers and complex software systems as described in the sources upon which they relied.’ " https://www.computerweekly.com/opinion/The-cause-of-the-Post...
I mean, hey, the computer is operating correctly, it can't make a mistake, so give me my money and turn yourself in...
You are making a technical distinction the law does not make. The law relates to the output - to the evidence generated from the system as a whole. The presumption is that the logic is correct too - which is why it is such a terrible assumption.
In the specific case of the Post Office miscarriages of justice, the system as a whole included Horizon central office staff who made manual and sometimes incorrect adjustments to the individual Post Office ledgers, and the discrepancies were later blamed on the postmasters.
Extending this rule to LLM’s would clearly be disastrous. At a minimum, a record-keeping system needs to be written the old-fashioned way.
The proposal is directly informed by the Post Office scandal in that it basically requires the prosecution to produce the same already-existing documents that revealed that Horizon had significant known faults.
If the `alleged theft amount by defendant` == (unsigned int)((int)-1), that should cover those cases, at least in theory. The profs would just blindly vouch it, but at least it'll be a point.
edit: gender neutral language
Well into the 90s the general populace considered computers infallible, it really took mass adoption and experience with how frequently they crashed to change that perception.
The big win would in this case would be that when the vendor conspired to hide bugs from their own tracker, they would have been creating criminal liability for their employers. Which Fujitsu and their subs richly deserve.
Bonus: Many opaque systems would have to be aired in an open court room to ascertain whether their invariants do, in fact, survive scrutiny.
It’s also extremely easy for the computer owner or IT people to do the same.
But this is so very dumb.
The alternative would be to say that only a system that has been proven correct could be used as evidence which would force pretty much all accounting back to paper.
Assuming that the paper accounting could be proven correct, when the computerized accounting could not, your proposed alternative would seem to be an upgrade.
You can make paper backups of course, but you can also make electronic ones, the electronic ones can be shipped around the world basically for free.
We're still a long way away from pulling that off with computers, but the idea that someone could have made a mistake shouldn't be that hard to grasp.
No, there are many alternatives. Including, most critically, the one laid out in the paper under discussion. Which is – spoilers – not “only a system that has been proven correct could be used as evidence”.
"It is a matter of surprise that important documentary records, such as the Fujitsu Known Error Log (KEL), were disclosed only in response to a direction from the court and in the face of opposition by the Post Office." (https://journals.sas.ac.uk/deeslr/article/view/5240/5083)
The proposal in this paper would require prosecution to disclose extant documents related to the reliability of the system. In the Horizon case, such documents existed, but were never disclosed, as current practices did not require the prosecution to do so.
Are prosecutors in the UK required to disclose both incriminating and exculpatory evidence to the defendant?
But from other articles it sounds like these prosecutions involved some kind of parallel legal system just for the mail?
I think though that the issue that this saga hit is that the law was such that the computer “evidence” was presumptively correct so they tried to pretend any “bugs” they were aware of didn’t change that fact and so the existence of bugs was not “exculpatory” - obviously BS, but it seems like that was the core behaviour/belief of the post office. That put the victims in the position of having to disprove the accusation but the only “evidence” in the case was the presumptively true report from the buggy system.
Ie the only evidence presented was a system that was buggy, but the victim could not get evidence the system was broken without first proving it was broken.
Yay!
Don't pretend that fucking over innocent lives in the pursuit of profits is a law of nature.
No. It'd actually just force authorship and disclosure of test cases.
It's not like they're going to give you the keys to the server room.
It's more than that. You don't have to instruct a jury that they have to accept that a person does what they're supposed to do unless proven otherwise... unless that person wrote code for a computer.
The original rule-makers probably never considered the idea of a giant, bespoke enterprise software system; it's just a particularly terrible edge case that got caught up in what looks like a reasonable efficiency measure.
Eye witnesses, expert witnesses, etc. can be unreliable. It's very difficult to prove that they have not made any errors. However, we don't ask juries to presume they are infallible.
There is a difference between being infallible and being correct. No one assumes that computers cannot err, just that they have not erred unless there is reason to believe they have. Likewise, if a witness gives coherent evidence and no one has any reason to assume they are wrong or lying, that evidence will not generally be disregarded simply because humans are fallible and therefore the evidence is presumed to be flawed.
There's no reason to think that programmers are less fallible than other people.
> required the prosecution to prove that a computer was operating properly at the relevant time before a document produced by such a computer could be admitted as evidence.
But I'm sure you actually mean to agree, and that your comment was just garbled in transit. :)