Assuming this unverified version of the story is true, the danger of accidentally leaking credentials in code is enormous and one of the reasons I continue to maintain and develop gitleaks. Those credentials would have been caught by the gitleaks' generic rule 
When I was looking into the streaming side of things I set up an overlay image which could be toggled with a hotkey to hide my screen (it actually also hid my desktop scene too in case the image didn't load or whatever)
My main precaution though was separating dev/prod and never looking at prod stuff online. Worst case someone could spin up some guff in my dev/test account until I can cycle the credentials
In my case the separation also included a different system user on my computer for stream work. Possibly overkill but why risk it when the costs are so low?
I can't see myself trusting a key blurring app if I'm honest. Rather fix the issue earlier in the process than rely on something that would probably break on edge cases (word wrap enabled? Here's the key but it's in two parts, that sort of thing)
I think it would be a good tool to have, I had to contact a conference organizer once who switched tabs while sharing her screen in a recording and revealed a note in Google Keep that read "LastPass master password" xD
It doesn't help that so many tools are like "give me your secret key in plain text in the config file" without at least offering a link to a webpage on the github of how you could secure your keys and use this software
Vault is not just a drop in and go system - setting up a vault instance is an ordeal in and of itself, and the pricing for vault on hashicorp cloud is incredibly expensive. The problem with the other options is that you have to get the secrets into environment variables, or out of github/lab secrets and into your application. To use most of the services like AWS secrets manager, vault, etc, it will cost you more to manage secrets than it will to host the app on a small DO droplet for example.
Self-hosted Vault within a minimum Kubernetes cluster in GCP costs us roughly $35 a month. Maintenance effort can be neglected if not scaling. Vault has its learning curve there but I think it's totally worth it, given its secret management and API-first features integrated with many other DevOps tools.
`pass` by itself might introduce false positives. `passwd` and `password` are common and more likely to be in the ROI of a secret. That said, I'm not opposed to `pass` by itself. I'll have to think about this one...
> but I assume they were chosen based on the statistics?
Nope, not statistics. Identifiers and keywords are chosen based on what I see out in the wild being a software engineer.
I don't know this guy, but how can he confirm this? does he possess any inner information? why I got the feeling that he is so eager to put a conclusion on this when it is still open for debate at this stage.
Binance is the largest cryptocurrency trading platform globally.
According to this tweet  they have a "threat intelligence" department that continually monitors for potential issues. It makes sense that they would be on the lookout for leaks of this nature, as they are highly dependent on correctly verifying and identifying their customers.
One of the major benefits of ephemeral tokens is that they become less attractive to put into the code, and more attractive to put in a config file/vault that's easier to update and keep secret. This in itself is useful because it makes it less likely that it will be in some source file someone shows, or pushed to some remote repo that at some point has permissions allowed so people can see it.
We got rid of all IAM users used by applications and moved to role-based access. Nowhere in the application do you need to enter AWS credentials. AWS SDK will attempt to discover short-lived credentials for you and will assume the role specified at the infrastructure layer, e.g. in a task definition.
You add the long lived IAM user API key/secret to it and it stores it in a password protected storage (MacOS keychain or similar).
Then you invoke aws-vault with an IAM role and command, and it will handle obtaining short-lived credentials scoped to that role (including TOTP 2-factor code auth), and then run the command with those temporary credentials as env vars.
With the right AWS permissions on your user, it can also automatically rotate the IAM user API keys for you.
It can either use a secret injected into an env var to bootstrap rotating ephemeral/refresh tokens or use a role provided by the environment (which can also provide short lived tokens), depending on your runtime environment and use case (on prem, cloud, k8s, etc).
Static, long lived secrets with limited governance that have no conditional access guards are weapons of mass self destruction.
Keeping secrets in environmental variables has always seemed dodgy to me. Unless specifically cleared, they get inherited by all child processes. Maybe there are never any child processes in your application, or that could be desired behavior in some circumstances, but generally it seems like asking for trouble.
Yes, but credentials should either be long lived with (very) limited scope _or_ short lived with required scope.
For example, for AWS you can create long lived credentials for users which are scoped to only allow one operation, namely obtaining a short lived token (with the aid of a hardware token such as a Yubikey) with scope to perform other operations.
This is not at all the takeaway from this. It's "this shitty developer should not have had access to this data in the first place". With a nuance of "this database probably shouldn't exist in this form in one place to begin with".
How you come up with a name is up to you and how you use it. Personally I would go with "July 2022 Shanghai National Police database leak" because I'm not having any conversation where a cute codename would be less confusing.
At work we codename security issues we are working on for Slack channels, etc. We use unrelated names that you could get from a name generator.
There is something deeply wrong with the authoritarian politics of the right and its casual use of racism to further political control.
> it's senselessly adding fuel to our political division.
This comment, whether you realize it or not, is coming from a place of extreme social privilege.
Remember that for the majority of people, politics is not a game. It is serious. People lose their rights to live the life they want all the time. Sometimes those politics turn violent and people lose everything.
I wonder if you could make a luhn-like check that would require an additional approval step to post if it comes back positive. Something like "It looks like you may be posting a secret *****. Do you wish to continue?
More and more providers have been adding unique prefixes to their tokens and access keys which makes detection much easier. Ex, GitLab adds `glpat-` to their PAT.
A project I maintain, Gitleaks, can easily detect "unique" secrets and does a pretty good job at detecting "generic" secrets too. In this case, the generic gitleaks rule would have caught the secrets . You can see the full rule definition here  and how the rule is constructed here .
Unfortunately, it's not as simple as that.
Lots of secrets are "generic" (think of a DB user/password combination), meaning that you need to take into account the surrounding source code context to be able to determine if they are a "real" secret.
I was thinking about that too, but it's actually tricky, even the example given, they use the var `accessId` but you could filter for all that, even the standard ones, but you couldn't have enough confidence in it so that if someone did post with a typo or even a random var name, they would think "Okay, no warning so must be okay".
Something like giving false confidence to the user. Not the best idea.
Wouldn't matter. Tons of bots are scraping every inch of the internet all the time, and if something been online for five seconds, it has been cached/stored somewhere. Always assume that anything you've put up on the internet, can forever be accessed by someone.
The only thing you can do is rotating the token/secret.
The consensus in Chinese community is while this is likely how the token got leaked, this alone isn't enough. To visit private Alibaba Cloud instance you can't just use some random IP. It's isolated from the Internet in certain way.
Remember when we still used password Windows Authentication and a private shielded network you could only get into with VPN instead of public cloud services with generic access credentials. It still didn't make leaking credentials right, but it was one extra layer of protection.
I made a webapp home icon from my Firefox and picked out the app-bait popover with uBlock.
Basically just about every app (YouTube, Reddit, Facebook, ...) is better this way. I.e., no ads, erase-able elements, less spyware, defaults to no notification and sometimes even gets better functionality. For instance, it (browsers) gets rid of "hearts" in Duolingo for whatever damn reason, so you can practice however much you'd like in a day.
The downsides I've found is that you seemingly can't Chrome-cast from it, and it often creates new tabs instead of reusing existing ones or making it's own app-instance, so you gotta close all tabs every so often.
Same with reddit on a mobile browser... it actually shuts you out, and says (after a couple of clicks) that they have locked you out "for your protection" as the content is "unverified", and that you need to use their app..
There's just a lot of randomness in what gets attention/traction off /newest. That's why HN doesn't try to prevent reposts of stories that haven't had significant attention yet.
It sucks when you're earlier and don't 'win', but it evens out in the long run if you post lots of good stories, since sometimes the lottery works in your favor. One of these years we'll get around to implementing karma-sharing to spread credit across multiple submitters.
What's the point of "winning" if everything is made up and the points don't matter? I get there's satisfaction in posting content that was useful, and HN isn't Fb/Twitter/Reddit and awash in ad $, but I feel fake internet points kinda manipulative since there's $ for the platform in your work.
I’m not interesting in “winning” points but do hoped this important story gets revealed and discussed earlier in this community I enjoyed participating. Thanks for taking time explaining this and appreciate all the efforts you put into HN community.
It seems the majority of people on the planet now have had some of their data leaked. Or are becoming ever more entangled with government and corporate systems which control and peddle their information as they see fit.
Is it ultimately a big nothing burger, or is this some singularity we are passing through?
At this point I've basically accepted that all my info will be found on sites like fastpeoplesearch.com and that anything I tell any company (or I guess in this case, govt too) will eventually be leaked, correlated, and used against me.
names and relations to family members and all their phones and addresses
a lot of it is collected from voter registration data (so your party affilition can be gathered as well)
I was royally pissed when I moved into a new home and literally a day after I signed up for internet service with Spectrum cable I got spam calls that know what state I'm in and my new home address is up online before I ever get around to updating my ID etc so I assume my data was sold immediately by them
I was able to connect from France, it's for people living in the US, look like you can search for people and there'd be aggregated information scrapped from god knows where. I checked a few (not really famous) people I knew of and it seems they have some accurate information.
I was thinking - if I had this, what could I do with the personal
records of a billion Chinese people?
And I must conclude - absolutely nothing. It's of no interest to me.
Now, I probably lack sufficient criminal imagination, but the point is
stuff like this is hard to fence because there's a very small market
of buyers. In an article I wrote for Routledge about the markets for
stolen digital data (specifically movie and album releases) I
suggested that the underlying problem is there's symbiosis between
leakers and buyers.
If you want to do anything, target the buyers. There's less of
them. Don't try to secure inherently insecure massively centralised
systems (Blotto + Dolev Yeo problem) . Or chase leakers. Or blame
users. Or fire the CIO. Find out who wants this stuff and take down
the show from the demand-side.
But hold on! Guess who the buyers are. And guess what sincere will
exists within "law enforcement" to tackle this sort of "cybercrime".
This type of information is used all of the time to discover and compromise web accounts of the victims in bulk. There are scripts that take in this data as input and will do a lot of the work for you to take over their accounts (or at least find their active accounts across web). Any additional data you are able to trawl can be sold itself, leaving the next steps to more advanced or motivated threat actors.
It’s also useful for more targeted social engineering attacks.
I suppose you could go the other direction. You could be an international human rights organization, and treat the database like a billion claim checks.
Having a definitive record of people's existence would make it more difficult for the authorities to skimp on natural disaster rescue efforts then lie about casualty numbers, treat citizens as canon fodder for military purposes, or simply wipe out individuals who have grievances with the government or powerful functionaries.
After my data was leaked, now scammers periodically call my phone to let me know that "I'm from bank security and someone's recently tried to change phone number for your bank account" or "I'm from police and we're opening a criminal case against you". It was fun first few times, but now I'm considering changing my phone number because I could miss an actual bank security call.
And I'm sure that plenty of gullible people were scammed and lost their money because of those leaks. When someone calls you, knows your full name and talks with enough confidence, it causes some trust.
My voicemail now indicates that I am no longer answering unknown numbers and to please leave a message or text me if you can.
It’s a little lengthy, but it’s cut down on the number of spam calls I actually answer specifically and I’m reasonably sure that anyone who actually needs to get a hold of me has an easy path to do so.
You have email addresses I believe. You could spam billions of chinese people with some scheme. It doesn't have to be believable, but you can bet out of billions of people you will capture some naive or mentally ill people who will fall for your scam and potentially allow you to clear them out of everything they have.
They were "protected". That is, they didn't leak out of the government into private hands. But that still turned out pretty badly.
In fact, information in the government's hands is the most dangerous, because they have more power than anyone else to use it against you.
(On the other hand, as others have said about Denmark and Netherlands, data that was not in government hands became in government hands, and was used against people. So it's not "safer" if it's in private hands, except to the degree that the government has to go through the extra step of getting it.)
There were also the "pink lists" tracking gay men  (link to German Wiki sorry) and which the nazis also greatly appreciated. Although to be fair^blunt they were collected exactly for reasons of prosecution, so not that far off from their use by the nazis.
IIRC there was a central registry of religion in the Netherlands that had the same effect. Can't find anything on that now, though (it's mentioned in Wikipedia in an unsourced paragraph; I think I first read about it on HN, actually).
Tangent: the info pages on the Anne Frank House site have sections cycling through different pastel background colours. I've wondered before whether something like that would the brain acquire context in a long page, making comprehension more like that of a physical book. Seeing it implemented, it doesn't seem to help. I think being able to easily flip to a previous page and back was one of the advantages of printed paper, so maybe a sticky TOC with the same colours or a minimap scrollbar would allow that? Actually, why not have that standard in browsers?
Hmm, the concept of coloured sections was known in 2013 already.
"Fun" fact: It was IBM who helped tabulate data from the 1933 national census, which was then used to identify hundreds of thousands more Jews than would have been found by the Nazi party without their efforts.
"Machine-tabulated census data greatly expanded the estimated number of Jews in Germany by identifying individuals with only one or a few Jewish ancestors. Previous estimates of 400,000 to 600,000 were abandoned for a new estimate of 2 million Jews."
Ahh...well there is the famous saying, "I decide who is a Jew." It was used on the head of the German Manhattan Project and a Jewish head (like a headmaster some shit) of a concentration camp, forget which one. And that's why we say "German Manhattan Project" stedda "Americaner Atomwaffenunternehmen" (I made that word up, it is correct in German to make words up, that means atom weapon undertaking), because German antisemitism amounted to forfeiting the bomb.
That was the price, the defeat of their last hope against the Allies. All of the Great Jews that slapped those firecrackers together were exiled due to antisemitism: Fermi, Szílard, Einstein (to get the president to read the letter to get the Los Alamos show on the road in the first place, get Roosevelt to read top to bottom left to right, no easy task), von Neumann (spesh because of his schizophrenia, no concentration camp for him, he would have been experimented on to then do that same sin to everybody in the camps, Schizophrenic Jews were at the absolute bottom o the Nazi world order).
I would say, impossible to compare. Digital changes the cost of acting upon this information, for good or bad purposes.
Obvious comparisons to e.g. the Netherlands' famous over-registering of religion and how the Nazis abused that. But I feel this is long term potentially worse than that. Not in the level of horribleness, but in the effect on society moving forward.
All you can do (in the USA) is freeze your credit and sign up for one of the free (or paid) credit monitoring services. That only protects you from financial ruin though. Not sure about people using your credentials to commit fraud, fake birth certificates, etc.
Well, if you look at (global) society as a dynamical system it seems to me that there are two stable basins or attractors, call them "Star Trek" and "North Korea".
In the "Star Trek" future the people in charge are themselves also subject to the panopticon, and the world is ruled fairly and humanely. (The other name I use for this is the "Tyranny of Mrs. Grundy".)
In the "North Korea" future there are (human or AI or hybrid) masters and brain-chipped cyborg slaves, and rule is absolute and enforced with digital precision.
(Of course, this is all predicated on the idea that we can't put the genie back in the bottle in re: ubiquitous surveillance. I think that's likely the case (although I do not like it) but I'm not going to make the argument here unless someone asks.)
Given the above the thing to do is work to make politicians subject to 24/7 total surveillance (ASAP, before everybody else) so we can keep an eye on them. This policy would also presumably weed out the crazies and corrupt, eh?
> Do you really believe, that us being on an utopian trajectory is realistic?
Oh yes. Very much so. In fact, by many measures we have been on an utopian trajectory for several centuries. Today even our failures are the result of unimaginable power. We have to learn to wield our power with wisdom.
We have all of the physical technology we need already. We can practice regenerative agriculture that increases topsoil fertility and volume; we have methods of construction that can build housing for everyone; machines and factories that churn out the physical necessities of life; etc. We need only deploy our resources and technology efficiently. It's down to logistics now, and we have more than enough computer power to sort that out.
The only thing holding us back is that most people still don't realize this yet.
I'm with you, in that a Star Trek utopia is indeed possible, technology and resource wise. And i would love to see the world society bending the trajectory, before it's too late. I just have seen too much greediness, to see a chance, that this will happen. Some poles would have to be moved, to bend the trajectory ;)
I would counter that, although it could, some groups will be able to evade it, effectively maintaining their advantage/power. Effectively averaging out the position of middle and lower classes, and lowering their chances of moving up the social ladder?
The standard "leak" of names and addresses of people is totally meaningless, though HN "privacy" obsessives blow it out of the water all the time. It's basically public information, we used to have everyone in phone books in the US and almost no one cared.
Cell phone number is a riskier one because of the opportunity for 2FA hacks. It's not hard to get people's cell phone numbers as it is (you can buy direct marketing lists for pennies per person in the US) but its not good to make it easy for hackers.
However this leak in particular appears to go much deeper so it is insidious. Police records are named and who knows what else. That is a genuine privacy issue and sucks for those involved.
Names and addresses can absolutely be used to stalk and harass people, and there are password reset flows that involve physically mailing secrets to people. Perhaps almost no one cared about phone books, but if you thought about the differences between phone books and a website for a moment, you'd see that these are different technologies that have different implications, and that it is entirely reasonable for people to have a different reaction.
You've chosen some arbitrary amount of information where you begin to care and become interested, and decided everyone with a different cutoff is an absolutist you don't need to listen to. But it's really just that your situation permits you to leak that information without fear, and you haven't deigned to imagine that other people are in a different situation.
Someone posted a comment explaining a little more about Shanghai's special relationship with the CCP/PLA:
>Shanghai is a city with a unique role in the progression of the CCP and its global efforts. Also PLA Unit 61398 is in Pudong, the shanghai district mentioned in the article. Overall there's a lot of CCP/PLA-adjacent tech talent in the area, and of course the local police still ultimately report to the CCP.
Surprise, it's 2022, and XP is still a de-facto standard Windows version, with hacked Win7 slowly gaining.
Why? Tons of Software was written for XP, and then abandoned without any support. Many of that stuff in the government sector. A lot of online banking clients outright say "only works on XP," and copyright years reads 2006.
This is similar how Android 7+ support was almost nuked in China for nearly a year because Tencent didn't want to port Wechat to newer APIs cuz "nobody uses Android newer than 4.X in China"
That was not why they refused to port it to newer APIs though. It was because Google changed the permissions API to be more granular and request permissions at runtime, which would have meant Tencent would have to request tons of permissions to gather user data (presumably users would not be inclined to grant so many permissions).
The leaked screenshot of the data's metadata looks like the output of Elasticsearch's /_cat command. Someone probably left the port 9200 open to the public, or stored the index on a public cloud but somehow leaked its keys either on github-like service or in some discussion forum -- a typical mistake that engineers make.
Another reason why not everyone should use the cloud. Sure, the cloud can be as secure as on-prem or even safer in many cases. But it's just so easy to keep on-prem data safe by just not connecting it to the outside world. If no server can be accessed from anywhere but the premise, leaks like these just can't happen. A key won't help you unless you can break into the police building.
Access just based on credentials seems so wrong anyway. There should always be whitelisted IPs for sensitive stuff like that.
The Shanghai police thinks like you, so they purchased a very expensive "private deployment of Alibaba Cloud", which in China usually works like this:
1. The customer build a data center.
2. Alibaba Cloud purchases servers, deploys them in the customer's data center along with all Alibaba Cloud software (same as in the public cloud).
3. Customers do whatever they want to the thing.
Basically by "private cloud" they really mean it, something AWS won't ever do.
In this case, the system is technically "not connected to the Internet", but we all know what this mean: it certainly will be occasionally.
Most cases I know, the customer cite "data security" as the reason why they would like to do this, because on-prem are always more secure right? But I hope we could agree on why this does not work:
- It is now very difficult for Alibaba Cloud to do ops work on these private deployments, so ... there will be maybe 2 releases per year, or in some cases never, including security patches. It's not rare to find a 5-years-old struts2 vuln in the control plane of such private deployments, and in the coming years it would be log4j2 I guess.
- Alibaba Cloud put serious effort into securing their public cloud, and even covering the ass for the customer. For example similar to GitHub+AWS secret scanning, they also proactively revoke access keys once the key appears on the Internet. The customers, on the other hand, usually do none of these.
In short, security is largely an Ops work and economies of scale also work here.
In the end these on-prem systems depend solely on network isolation for their security, and... air-gap does not always work.
If the records are digital and non-air-gapped in any system of any country, you can assume that the US government has access to those records already. The exceptions to this assumption are exceedingly rare.
As a US citizen I want to believe bravado like this but I’m guessing this is just your fantasy world talking not actual knowledge of the government being competent, which in my personal experience seems extremely unlikely.
The government isn't competent as a whole.. but the intelligence agencies are rather powerful. I've worked for DARPA and IARPA and you wouldn't believe half the stuff I could say publicly and none of the stuff I can't disclose.
We're always told that grand "conspiracy theories" are impossible because if hundreds or thousands of people are involved someone will always leak something about it. Interesting there's all these people out there who seem to know much bigger things than Snowden revealed.
I don't know how it works in China but where I am a person's criminal record is not public but not exactly private either. In the sense that an employer can ask for your criminal record and you have the choice giving a printout of it or not having your job. Making it kind of hard to see how the knowledge of a criminal record could be used to blackmail someeone.
As for "data brokers. Advertisement, financial credibility, trustworthines of buisness partners etc.". Maybe. But these companies would turn themselves into criminals by using or purchasing this information.
It is likely, that this DB contains more information, than what a formal printout gives.
"But these companies would turn themselves into criminals by using or purchasing this information."
Which is why they probably would not deal with the information gathering directly, but use a service of a data analyst company. When they do something illegal, nobody who contracted then did ever know anything. I think this game is played in china as well.
> While the Shanghai government and police department have largely been silent over the leak,
Someone/some team in the police department is probably in serious panic right now. Not only because the data is leaked, but also because the leak has displayed an example of what they are actually recording.
For example, according to the posts that other people has posted online (probably rumor and speculations), the `address_merge_with_mobile_data.json` file is a collection of external data submitted to the police database. In the file, there are data source types such as:
I don't think Chinese people actually cares, most people don't even know what "Data" actually is, let alone how "Data" effects them.
The altitude of most people here (I'm a Chinese BTW) is "I did nothing wrong, so why should I care?".
But, it's one thing to hear the humor, it's another thing when you can actually see it. People will have different opinions about privacy when their wives asks them why their business trip to another city showed up as a hotel night 500 meters away from home on the government database.
>People will have different opinions about privacy when their wives asks them why their business trip to another city showed up as a hotel night 500 meters away from home on the government database.
Had a good chuckle. But then it got me thinking about other situations in which government-leak-induced friction that might lead to cause for legal action. The government demands and collates data presumably supported by legal under-pinnings that citizens must comply with. When these are leaked, shouldn't the government bear legal, fiduciary, etc. responsibility?
According to the Network Security/Safety (it's the same word in Chinese) Law, the network operator is responsible to ensure the security of their system, which includes 1) setting up security and auditing rules, 2) prevent virus, hacking and other attacks, 3) monitor and record the status of the network and perform backups etc.
The law has also stated that if an institution violated the law, and civil damages is generated as result, the institution is also responsible for the damage.
There is another law which might also apply to this case, it's called Data Security/Safety Law. This law also stated that the institution is responsible for civil damages if the institution has violated the law.
But, I'm not a lawyer and certainly not a Chinese lawyer. This situation is fairly complex already, it's hard for me to tell which direction this case would go.
I guess they'll investigate it first, but the company/team which worked on the project has probably already read the laws before, thus I assume they're well-prepared for this. Another twist is the programmer who accidentally published the secret database API key to the public network, this needs to be investigated as well to determine whether or not the key has actually been used in the attack.
So I guess in the meantime, the husbands must wait.
- 10 BTC sounds a lot but it's peanuts for such large data sets.
- 750k row of sample data is large enough for a leak by itself, many on reddit/twitter/fediverse have already started to explore the data set for gender ratio, age composition and frequency of raping cases, etc.
Take these threads with giant grain of salt though, they're far from thorough and some of them lack basic understanding of statistics. And I personally don't think the dataset (at least the sample) is actually random so not really a good representation of China's demographics.
Kinda interesting that The Register does not even speculate about steps which China's higher-level security services might take in response, to "memorably demonstrate their displeasure" at the theft. (A certain cynical attitude is usually part of The Register's stock-in-trade.)
Governments have been collecting (and poorly securing) this sort of information and more for most of recorded history. It's not to say that I like it, or would work for somewhere like Meta or the like, but plenty of these major data leaks have been from places that used to collect and store physical data bases of this stuff since before most of us were alive.
I'm talking calmly about this because people have been screaming in my ear about it for 20 years, and I listened. And then I lived my life around the fact that this was going to be happening whether you scream yourself hoarse or not, at least for now.
I am guessing he means that it highlights the incompetence or even just the consequences of centralizing power.
Personally I don't expect this to bear true. Historically in China, government failures have been cited as evidence for further centralizing the power of the federal government. And this argument is bought hook-line-and-sinker by the people. I don't think that will change until there is serious economic hardship.