The submitted article was "EFF questions methodology behind Dartmouth cheating allegations [pdf]" (https://www.eff.org/files/2021/04/14/fire_and_eff_letter_to_...). It is worth reading, but since this story hadn't had a significant discussion on HN yet, we changed the URL to an article that gives more background.
Horrifying. The Dartmouth med school faculty identifies 17 students they suspect as cheating. Nearly half of these they eliminate as false positives. Rather than grappling with the obvious unreliability of a methodology with at least a 50% false positive rate (from the NYT: “the use of Canvas in the cheating investigation was unusual because the software was not designed as a forensic tool”), they decide the remaining students must all be guilty.
To become a doctor in the U.S. requires an incredible amount of intelligence, hard work and personal sacrifice. To destroy the lives of these med students based on such an unreliable and unfair process is, in my opinion, a true injustice.
Anyone who has ever done analysis of raw data knows that the process consists of iteratively excising bad data until you can actually precisely and accurately explain your analysis results.
Tools like this, where the data consumers (campus admins) have no possible way of understanding the data analysis methodology, let alone the raw data themselves, when the _responsibility_ for the consequences lives with the consumers, have no business operating like they do today.
It is really bad, but to compare the the standard screenning for HIV ends up with an about 50% false positive rate in the general population because the background rate is so low. I wouldn't throw out the result based on the tool being inaccurate, I would dig deeper.
Yes, I didn’t mean to make that implication (although I see now I could have phrased it better). My main point was that this injustice seems especially egregious given the sacrifices made by med students/residents/etc. during their medical education.
> Tensions flared in early April when an anonymous student account on Instagram posted about the cheating charges. Soon after, Dartmouth issued a social media policy warning that students’ anonymous posts “may still be traced back” to them.
It's totally normal and proper for a university to tell students who speak out about being expelled by a kangaroo court of administrators "we will find you".
If they didn't want students looking at the material they should have just removed it, not relied on this insane post-hoc attempt to try and identify cheaters. Using this to try and destroy these students futures in the medical profession is unconscionable.
The whole thing smells like bullshit. The idea that under a timed medical school exam a student would spend 5 minutes rooting through Canvas to find an answer rather than just Googling it in 10 seconds is absurd.
Likely the faculty was too lazy to remove the material, too tech-unsavvy to understand the possibility that Canvas could have the ability to generate automated traffic, and too paranoid to give students the benefit of the doubt.
They can’t even properly identify who’s cheating and who isn’t but they want students to believe they’re capable of figuring out who’s behind an anonymous instagram account?
"Almost immediately, questions emerged over whether the committee had mistaken automated activity on Canvas for human activity, based on a limited subset of exam data.
Geisel students said they often had dozens of course pages open on Canvas, which they rarely logged out of. Those pages can automatically generate activity data even when no one is looking at them, according to The Times’s analysis and technology experts."
It’s exciting that the EFF is getting a new sense of relevancy due to digitization of all the things through 2020.
They seem to be in a strong place to launch clear counterpoints to admin teams who signed up for whatever tech without really understanding it but are now trying to make admin findings with the tech.
Auditing an entire year's log seems very excessive. However, timestamps are recorded, focus is recorded, and I've heard of cheaters being caught from this information. It's not the main feature, but Canvas is partly a forensic tool.
And from my own observation, having a file open in the background with no focus does send occasional requests for other random file metadata that aren't open at all (edit: only for four minutes after load, and then a ping indefinitely), so that's probably a source of the false accusations. Also, it's open source if any hackers would like to take a look.
> Dr. Compton said the committee’s dismissal of cases over time validated its methodology.
What? This makes no sense. That some of the accusations have been proven incorrect doesn’t mean the remaining accusations must not also be incorrect.
If a doctor has a documented history of falsely diagnosing a patient with X and that doctor diagnoses you with X should you get a second opinion? Or take it at face value because the doctor has “validated their (clearly flawed) methodology”?
I’m just glad I got through medical school without having to deal with most of this, but we too had examsoft and canvas - just a whole lot less software supervision and always proctored in person.
I don’t understand why the policy wasn’t to just close all other browser windows/tabs before starting an exam. Maybe it was?
Or better yet - if medical schools really want to do this kind of testing remotely, provide your own devices strictly for testing instead of requiring a device with a backup! We didn’t pay 40-50k a year to deal with this nonsense.
I think the problem was that students need to also have a smartphone or something available during the test for questions and announcements or something, since the software locks down the testing computer. And there was no way to prevent people from looking stuff up there. Anyway, once they do impose this requirement to not access canvas, a cheater would just make it available offline. Ultimately the way professors spot cheaters is in their patterns of answers.
Another possible solution to this problem -- one some gamers are starting to use to fend off accusations of cheating -- is to set up a separate camera recording oneself and the screen. It's not a perfect solution but it could potentially rule out some false positives.
This is standard procedure for many medical school exams during COVID. Examsoft locks down your primary device, and you have a secondary device (smartphone/other laptop) pointed at you throughout the exam with a live video stream to the proctor. Each student does a 360 of their workspace prior to beginning the exam, and they are recorded throughout.
From the article: "To hinder online cheating, Geisel requires students to turn on ExamSoft — a separate tool that prevents them from looking up study materials during tests — on the laptop or tablet on which they take exams. The school also requires students to keep a backup device nearby. The faculty member’s report made administrators concerned that some students may have used their backup device to look at course material on Canvas while taking tests on their primary device."
I'm not sure what's going on here. The med school I'm affiliated with has students running the live video stream to the proctor through the secondary device. I guess Dartmouth wasn't doing that, but was having students keep it nearby only as a communication device in case WiFi went down or something?
Either way, the fact that they've taken students in one of the most competitive and stressful post-graduate programs in the country, told them to sit at home for a year-and-a-half and teach themselves medicine, and subjected them to a scored STEP 1 the year before it goes pass/fail is beyond fucked.
The 360 always seemed like pointless theatre to me sense its done at the beginning and someone else can easily come in and assist after the sweep around
Some proctoring software (like Proctorio) watches you through your camera and flags you if your eye movements are suspicious or suggest you're looking at a second screen.
These schools want people who are good at cheating without getting caught. Those people become very rich and powerful. They don’t want hamfisted cheaters who get busted. Those people generate bad press, especially when they go to jail.
The enforcement against naive cheaters while leaving avenues open to sophisticated cheaters is a feature, not a bug.
Were the MCATs used for admission sufficiently reliable? Most people think so, and since future test scores are most strongly predicted by prior test scores, medicine is trending away from ranking (ie non instructive) exams once you’re “in.” Net increase in pass fail medical programs. Step 1 just became pass fail. Things like ABSITE becoming optional (ie diagnostic or instructional). Inbound students are so high quality despite there being so much more of them that maybe the testing really is counterintuitively too intense.
Even if it were true that the students cheated, it isn’t material anymore, it maybe never has been, after you’ve been admitted.
Could it work differently? I feel like your average medical dropout (no offense, essentially the average programmer on Hacker News) probably cheated more in school than an admitted medical student does. Remote leetcode has exacerbated that.
On the other hand, a 23 year old who has made it just past the 1st year into their BigCo job probably faces more soul crushing false positive bureaucracy like watching your every digital move than a 23 year old 1 year into a top ranked medical program. The EFF would get much more mileage out of advocacy about 360° reviews, PIPs, stack rank, meetings about meetings, “mental health days,” etc. than this fortunately isolated and rapidly ameliorating incident.
Because if there’s one thing big tech people are actually really bad at, it’s reducing soul crushing bureaucracy.
To become a doctor in the U.S. requires an incredible amount of intelligence, hard work and personal sacrifice. To destroy the lives of these med students based on such an unreliable and unfair process is, in my opinion, a true injustice.
Tools like this, where the data consumers (campus admins) have no possible way of understanding the data analysis methodology, let alone the raw data themselves, when the _responsibility_ for the consequences lives with the consumers, have no business operating like they do today.
It's totally normal and proper for a university to tell students who speak out about being expelled by a kangaroo court of administrators "we will find you".
If they didn't want students looking at the material they should have just removed it, not relied on this insane post-hoc attempt to try and identify cheaters. Using this to try and destroy these students futures in the medical profession is unconscionable.
Likely the faculty was too lazy to remove the material, too tech-unsavvy to understand the possibility that Canvas could have the ability to generate automated traffic, and too paranoid to give students the benefit of the doubt.
"Almost immediately, questions emerged over whether the committee had mistaken automated activity on Canvas for human activity, based on a limited subset of exam data.
Geisel students said they often had dozens of course pages open on Canvas, which they rarely logged out of. Those pages can automatically generate activity data even when no one is looking at them, according to The Times’s analysis and technology experts."
They seem to be in a strong place to launch clear counterpoints to admin teams who signed up for whatever tech without really understanding it but are now trying to make admin findings with the tech.
A few weeks ago I donated for the first time and became a member.
https://supporters.eff.org/donate/30for30--D
https://en.wikipedia.org/wiki/Cindy_Cohn
And from my own observation, having a file open in the background with no focus does send occasional requests for other random file metadata that aren't open at all (edit: only for four minutes after load, and then a ping indefinitely), so that's probably a source of the false accusations. Also, it's open source if any hackers would like to take a look.
https://github.com/instructure/canvas-lms
What? This makes no sense. That some of the accusations have been proven incorrect doesn’t mean the remaining accusations must not also be incorrect.
If a doctor has a documented history of falsely diagnosing a patient with X and that doctor diagnoses you with X should you get a second opinion? Or take it at face value because the doctor has “validated their (clearly flawed) methodology”?
I don’t understand why the policy wasn’t to just close all other browser windows/tabs before starting an exam. Maybe it was?
Or better yet - if medical schools really want to do this kind of testing remotely, provide your own devices strictly for testing instead of requiring a device with a backup! We didn’t pay 40-50k a year to deal with this nonsense.
From the article: "To hinder online cheating, Geisel requires students to turn on ExamSoft — a separate tool that prevents them from looking up study materials during tests — on the laptop or tablet on which they take exams. The school also requires students to keep a backup device nearby. The faculty member’s report made administrators concerned that some students may have used their backup device to look at course material on Canvas while taking tests on their primary device."
I'm not sure what's going on here. The med school I'm affiliated with has students running the live video stream to the proctor through the secondary device. I guess Dartmouth wasn't doing that, but was having students keep it nearby only as a communication device in case WiFi went down or something?
Either way, the fact that they've taken students in one of the most competitive and stressful post-graduate programs in the country, told them to sit at home for a year-and-a-half and teach themselves medicine, and subjected them to a scored STEP 1 the year before it goes pass/fail is beyond fucked.
Wouldn’t be surprised if there’s a class action lawsuit against the colleges and companies that create this shitty spyware.
More reading: https://www.vice.com/en/article/n7wxvd/students-are-rebellin...
Download the files before the exam and use them to cheat --OK. Browse to the class site during the test -- cheating.
The enforcement against naive cheaters while leaving avenues open to sophisticated cheaters is a feature, not a bug.
Were the MCATs used for admission sufficiently reliable? Most people think so, and since future test scores are most strongly predicted by prior test scores, medicine is trending away from ranking (ie non instructive) exams once you’re “in.” Net increase in pass fail medical programs. Step 1 just became pass fail. Things like ABSITE becoming optional (ie diagnostic or instructional). Inbound students are so high quality despite there being so much more of them that maybe the testing really is counterintuitively too intense.
Even if it were true that the students cheated, it isn’t material anymore, it maybe never has been, after you’ve been admitted.
Could it work differently? I feel like your average medical dropout (no offense, essentially the average programmer on Hacker News) probably cheated more in school than an admitted medical student does. Remote leetcode has exacerbated that.
On the other hand, a 23 year old who has made it just past the 1st year into their BigCo job probably faces more soul crushing false positive bureaucracy like watching your every digital move than a 23 year old 1 year into a top ranked medical program. The EFF would get much more mileage out of advocacy about 360° reviews, PIPs, stack rank, meetings about meetings, “mental health days,” etc. than this fortunately isolated and rapidly ameliorating incident.
Because if there’s one thing big tech people are actually really bad at, it’s reducing soul crushing bureaucracy.