The weird thing about this is that the only company I've seen doing problematic user-agent handling in recent years is Google themselves. They have released several products as Chrome-only, which then turned out to work fine in every other browser if they just pretended to be Chrome through the user agent. Same with their search pages, which on mobile were very bad in every non-Chrome browser purely based on user agent sniffing.
If you have the new Chromium-based Edge ("Edgium") installed: the compatibility list at edge://compat/useragent is really interesting.
Edgium pretends to be Chrome towards Gmail, Google Play, YouTube, and lots of non-Google services; on the other hand, it pretends to be Classic Edge towards many streaming services (HBO Now, DAZN, etc.) because it supports PlayReady DRM, which Chrome doesn't.
Other browsers support DRM too, but with different tradeoffs.
Chrome uses Widevine, but one of Chrome's philosophies is that you should be able to wipe a Chrome install, reinstall Chrome, and have no trace that before/after are the same person. That means no leveraging machine-specific hardware details that would persist across installs. "Software-only DRM", essentially.
Edge on Windows (and Safari on OSX) are able to leverage more hardware-specific functionality --- which from a DRM perspective are considered "more secure", but the tradeoff is a reduction of end-user anonymity (i.e. if private keys baked into a hardware TPM are involved).
Last I checked, Chrome/Firefox were capped at 720p content, Safari/Edge at 1080p, though it looks like Edge is now able to stream 4k.
Last time I used piratebay, I saw a lot of porn and malware/scam ads. I had to find and install a torrent client. Then I had to make sure I was downloading a movie that had enough seeders. And then I couldn't watch the movie until (and if) the download finished.
When I use netflix, I have a much better experience.
I know this is all anecdotal, but last time I used a torrent site, I found the movie immediately and it pulled the whole thing down in under 3 minutes. Could be that it was a newer movie and pretty popular. I do see a lot of older stuff that's not being seeded much anymore.
They may not have the same DRM restrictions as actual content? It's not that Firefox is not technically capable of rendering the videos, it's that it doesn't give the DRM the control that Netflix and rightholders want to require in exchange.
You can have a similar experience with a private tracker and a seedbox. The content is curated (no malware), there's a larger selection, the quality is sometimes higher, and peers generally have better connections.
After you pick your torrent, it takes the seedbox a few seconds to download the content. Then you can stream your download using emby, vlc over http, or whatever you prefer.
With torrents you can get a film in minutes. With popcorn you are exposed the entire time you watch the film.
In Germany they monitor peer connections and send a payment demanding an out of court settlement. After two years they escalate to a court appearance in a remote town. If you don't show, you lose and they turn it over to debt collection.
I don't know this technology, but I'd really recommend using a VPN that provides a SOCKS proxy for your Bittorrent connections. Otherwise you're just announcing your IP address related to your torrent activities to the whole world.
I don't live in Europe but I canceled my Netflix the minute I learned that Reed Hastings is taking advantage of the coronavirus situation to increase Netflix's bottom line by limiting streaming quality to everyone in Europe after one phone call with a French politician.
There are plenty of torrent streaming and download clients that work just as well and are just as convenient as Netflix, without needing to rely on a central authority.
You need: 1. Usenet provider (this allows you to download the links) and Usenet indexer (this allows you to search for content you want because files like movies usually have obfuscated filenames). Example of the former is Frugal Usenet, example of the latter is Drunken Slug (free-ish) or DogNZB (paid). One of the many things you need to watch out for with providers is retention, the length of time they keep the files so you can download them. In general, the older content you find on the indexer, the less likely it's still going to be available. The downloading itself can be done in an automated way (for example with Sonarr and the like) or kind-of manual with tools like SABnzbd.
I am not sure about Edge specifically, but as someone who tries to use mostly open source software: Digital Rights Management (DRM) requirements often directly conflict with licensing related to open source software.
It allows websites to run arbitrary code blobs which interact with a hardware backdoor in your CPU with a higher permission level than the OS. With the multiple exploits that have been found in intel cpus and the Management Engine itself there is no way you should be letting any website do that.
There are different kinds of DRM. Streaming websites allow different quality for different kinds of DRM. E.g. they allow best quality only for best protected DRM (which should use encryption all the way from Netflix webserver to your display). There's software DRM (decrypting stream inside proprietary blob) which is considered weaker, so you'll receive acceptable quality in Chrome. I don't know why Chrome did not implement the most secure DRM. Hopefully Microsoft will contribute their patches back.
I had been thinking recently as I've been using Firefox more that Google maps had got clunky. With a little fiddling prompted by your comment, it turns out Maps sniffs specifically to reduce fluid animations on Firefox (and probably some other browsers).
Unfortunately there doesn't seem to be any workaround for ReCaptcha on FF. I generally end up opening the website in the GNOME or KDE (Falkon) browser which use something like WebKit/Blink - there it works on the first try every time.
Some advice about ReCaptcha, the audio test is way easier and usually only makes you do it once or twice (as opposed to the 5-10 times you usually have to do when you have disabled tracking). Sometimes it will say you aren't eligible or something, just refresh the page and it will let you try again.
I wonder what will be the point of these captchas if most people started avoiding them by installing extensions. If the end goal is to put significant resource constraint on the scrapper or bots, you can do it quite easily by running something to short the usage up for a few seconds if you think someone is suspicious.
I just use the contact site to send them a note to cut it out. There are better ways to prevent spam. Like my password that I just entered in the latest case. I'll go elsewhere in the next, they have competitors that won't be that different in price and easier to use.
I was hitting that too! I wondered what I was doing. I kept getting these weird fields. That fucking sucks. I'd like someone to explain google's point of view about why this is happening? I do override user agent on some systems so random websites work.
I wouldn’t really shame them on this one. Google docs and sheets uses contenteditable under the hood. Contenteditable isn’t specified or standardized anywhere and varies wildly between browsers, and sometimes between browser versions.
Oh wait...THAT might be the problem?? I've been having that issue too! I have to cut the cell content, delete the cell, and then paste the cell content back in in order for the row height to appear the same. It never even occurred to me to switch browsers, I thought it was an issue with Google Sheets.
Maybe. Still not switching away from Firefox though. I have Chrome installed because WebRTC is a lot better and teleconferencing needs have shot up recently, and because I lock my phone away in a Kitchen Safe and use 2FA with Authy the Chrome app, but Firefox is my daily driver :)
Every single Google product is slower on Firefox and it’s hard to not call this malice and artificial. Many people check out Gmail and GMaps on Firefox and go back to Chrome because of their clunkiness on Firefox.
Probably a lot of the other way around as well. Google employees likely use Chrome and Docs/Sheets heavily, so they get a rapid high-quality feedback loop with any bugs or performance issues affecting the Chrome devs and people around them.
Huh, I guess "hard not to call" could be interpreted as a disclaimer or qualifier meaning "I can't prove this, and it's only my opinion".
However, I interpreted it differently, to basically mean that because of objective fact, of all the explanations you can think of, only one of them is defensible. In other words, it's similar to saying "admit it, this is the only reasonable conclusion".
I prefer mu4e (mu for email) although ultimately I'm still using my gmail account just with another interface. I like the idea of ultimately just setting up a mail server but realistically its really really hard to switch everything at once when you have a lot of existing accounts so I'll probably be maintaining a gmail account forever.
I'm trying maps.me because unlike google it does offline walking directions but I haven't used it enough.
I really want to like duck duck go but it feels like google still provides better results.
In fact, just this week I thought about why Google doesn't make its mobile maps website fast again. It is such a pain to use it on older phones and I totally don't get, why it has to be that slow (doesn't matter if chrome or Firefox).
Here in Safari, Gmail is not only 10x buggier than it used to be before the redesign, it also uses at least 10x more client-side resources (CPU, network, ...). A handful of open Gmail tabs single-handedly use more CPU over here than hundreds of other web pages open simultaneously, including plenty of heavyweight app-style pages.
It’s hard to escape the conclusion that Google’s front-end development process is completely incompetent and has no respect for customers’ battery or bandwidth.
> The weird thing about this is that the only company I've seen doing problematic user-agent handling in recent years is Google themselves.
I frequently consume web articles with a combination of newsboat + Lynx, and it's astounding how many websites throw up HTTP 403 messages when I try to open a link. They're obviously sniffing my user agent because if I blank out the string (more accurately, just the 'libwww-FM' part, then the site will show me the correct page.
I'm pretty sure that the webmasters responsible for this are using user agent string blocking as a naive attempt to block bots from scraping their site, but that assumes that the bots that they want to block actually send an accurate user agent string the first place.
> I'm pretty sure that the webmasters responsible for this are using user agent string blocking as a naive attempt to block bots from scraping their site, but that assumes that the bots that they want to block actually send an accurate user agent string the first place.
That is exactly what they are doing, and it works really well.
We blocked user agents with lib in them at reddit for a long time.
Any legit person building a legit bot would know to fake the agent string.
The script kiddies would just go away. It drastically reduced bot traffic when we did that. Obviously some of the malicious bot writers know to fake their agent string too, and we had other mitigations for that.
But sometimes the simplest solutions solve the majority of issues.
It actually is encouraging people to have useful user agents. By default most people end up with a user agent that's something like "libcurl version foo.bar.baz", which isn't actually a description of who or what they are; given the prevalence of curl, it really just tells you that it's a program that uses http.
Yes perhaps. But it caused problems for regular users like this fellow. I also have tried various 'download via script' for web pages for offline use. I thought I had a problem on my end, I never realized I could have been getting blocked.
Hard to argue with the economics of that mitigation though. The abuse:legitimate use ratio is probably pretty high. Getting rid of user agent strings will bring back the scaling problems, as they should probably be addressed directly.
It's the exact opposite. Without User-Agents, sites need to depend on feature detection, and closing the feature discrepancy between Chrome and other browsers is more complicated than just spoofing your UA to get Google to serve you functioning versions of their products.
1. Our Chrome detection relies on the User-Agent header.
2. But people can just lie in the User-Agent header.
3. Let's get rid of it and use something that's harder to lie about.
Closing any feature discrepancy isn't a goal here, as far as I can see. The whole point is to lie to the user that a feature discrepancy exists when it doesn't.
You can make the argument that Google is free to do their browser detection however they want (and therefore doesn't need to solve this problem by eliminating User-Agents), but this is still an obvious example of the User-Agent header causing problems for Google.
I interpreted your parent's comment differently; namely, if Google's developers can't do User-Agent detection, then internally even they will have to improve how they develop (eg. via feature detection), making their products more compatible with other browsers.
Many people assume Google, as an upper-level business decision, purposely makes products work better on Chrome in order to vendor-lock users to the browser. Maybe that's true; or maybe it's developers being lazy and using User-Agent detection. Removing their ability to do so might actually improve cross-browser compatibility of Google products.
You can see Chrome devrels on Twitter expressing disappointment with Chrome-only web sites, saying that they raise the issue internally. Of course we have no visibility into what happens after that, but it's an indicator that you're right.
I’m guessing all of Google’s internal apps are only tested on Chrome, with plenty of Chrome extensions, which means that all of the developers have to use Chrome to make the tools work, and at that point, switching back and forth between different browsers is a pain so none of the other browsers get the love they deserve.
The attitude of “it works on Chrome, I don’t care about anything else” is fairly widespread anyway. Just to stem the tide a little bit I’ve been developing on Firefox and Safari first, and then checking Chrome last.
I got bitten before when I made a browser game, and then noticed that it was all sorts of broken on Edge, even though Edge supposedly had all the features I needed. It turns out that Edge did have all the features I needed, but I had accidentally used a bunch of Chrome features I didn’t need. The easy way out is to turn things off when I detect Edge. The hard way is to find all the broken parts and fix them. So nowadays, I don’t do any web development in Chrome.
At least in my part of the googleverse, we have automated tests running in all the browsers (even ie11).
But I'll admit I will also poke around outside of the tests, and I'll usually only be doing that in chrome, unless I've had a bug report about firefox in particular. And I'll only really open up Safari when I'm testing VoiceOver. ChromeVox just isn't good enough.
Considering that there's been an internal and external bug filed about the US states not being in alphabetical order on contacts.google.com for years, making it impossible to type 'new y' to get New York, I don't think raising it as an issue will help much.
I’ve started seeing alphabet employees use this as an excuse: “oh that happened on team x there’s nothing I could’ve done”. On small technical issues the excuse is fine - on large moral issues it does not work.
In addition to these, I've also noticed outlook365 (use it for accessing work email) gives a very minimal interface to the point of being nearly unusable with a FreeBSD user agent. Switch to a more popular one and I get full functionality.
It's usually just that the company doesn't want to have to deal with potential support calls that come in about their website not working on x os in x browser, so they do the bare minimum to disable it - then if a user complains about it not working, they can plausibly deny supporting the os/browser configuration.
> I would guess they have built something into chrome that gets even more data that isn't user-agent based.
Chrome includes a unique installation id in requests to Google owned domains. They don't need any cookies or user agents to guess who you are and best of all they don't have to share that information with their competition.
Not for a technical reason, but they had a reason: they provided no support or guarantee that Netflix would ever work on Linux + FF (Ubuntu + Chrome was guaranteed) and they didn't want any support calls for something that they wouldn't help people with anyway.
A lot of stuff gets blocked for this reason. The company doesn't want you calling them because HD video doesn't work on Firefox even though you pay for HD quality, they do not test or guarantee Firefox compatibility in the slightest and yet they have to talk to an angry customer now. It makes business sense to redirect people to supported use cases when you know your product probably won't work as intended otherwise.
You don't have to agree with the decision (and you can always cancel your membership if you do) but they had their reasons.
Facebook also uses the user-agent string to determine which version of a site to send to someone. I installed a user-agent spoofer a while back and messenger.com would fail due to it every few refreshes (as evidenced by JS console).
To get Microsoft Teams to run in Chromium there used to be a user-agent hack to make it pretend to be Chrome. This was superseded by someone packaging up using Electron. And finally this has been superseded by Micrsoft themselves supporting Linux using something that looks and feels like Electron again.
So, basically, Microsoft using user-agent to detect Chrome....
I don't really understand how this will result in any real difference in privacy or homogeneity of the web. Realistically every browser that implements this is gonna offer up all the info the server asks for because asking the user each time is terrible UX.
Additionally this will allow google to further segment out any browser that doesn't implement this because they'll ask for it, get `null` back and respond with sorry we don't support your browser, only now you can't just change your UAS and keep going, now you actually need to change your browser.
And if other browsers do decide to implement it, they'll just lie and claim to be chrome to make sure sites give the best exp... so we're back to where we started.
> I don't really understand how this will result in any real difference in privacy or homogeneity of the web.
It does a little: sites don't passively receive this information all the time, instead they have to actively ask for it. And browsers can say no, much like they can with blocking third party cookies.
In any case I'm not sure privacy is the ultimate goal here: it's intended to replace the awful user agent sniffing people currently have to do with a sensible system where you query for what you actually want, rather than infer it from what's available.
Switching it from passive to active means you can count it towards https://github.com/bslassey/privacy-budget . Yes, sites can ask for all sorts of things, but if they ask for enough that they could plausibly be fingerprinting you then they start seeing their requests denied.
(Disclosure: I work at Google, speaking only for myself)
The problem is that without User Agent sniffing in some circumstances there is no other way of working round a browser bug e.g. There are cases where browsers will report that it supports such feature using one of the feature checks but the implementation is garbage. The only way is to have a work around based on user-agent sniffing.
Sure a lot of developers abuse the feature but I fear this might create another set of problems.
> It does a little: sites don't passively receive this information all the time, instead they have to actively ask for it. And browsers can say no, much like they can with blocking third party cookies.
Lets run through that scenario:
sites that don't need this info still aren't gonna ask for it or use it. sites that want it will get it this way and even if you respond with "no" that's useful to them as well for fingerprinting and as a way to fragment features to chrome only. So, what's changed?
> sites that want it will get it this way and even if you respond with "no" that's useful to them as well for fingerprinting
To an extent, sure. But to follow the model of third party cookies, let's say client hints are used extensively instead of user agent and all cross-domain iframes are blocked from client hint sniffing. All the third party iframe is going to be able to detect is whether user has a client hint capable browser or not. That's a big difference from the whole user agent they get today.
The idea is that this won't be a Chrome-specific API. It's been submitted to standards bodies, but Chrome is the first to implement. For example, Firefox have said they "look forward to learning from other vendors who implement the "GREASE-like UA Strings" proposal and its effects on site compatibility" so they're not dismissing the idea, they're just saying "you first".
Also, last year, Vivaldi switched to using a user-agent string identical to Chrome’s because websites refused to work for Vivaldi, but worked fine with a spoofed user-agent string.
If companies like Google wouldn't abuse the user agent string to block functionality, serve ads, force their users to specific browser then companies like Google wouldn't have to use fake UA strings and then maybe companies like Google wouldn't have to drop their support.
Very little trouble? That user agent says it’s Mozilla, then says it’s Windows 10, then says it’s Apple, then says it’s Gecko, then says it’s Chrome, then even though it said it was Windows it goes on to say it’s Safari.
The individual headers, on the other hand, tell you EXACTLY what the system and browser is.
That's because you have no experience. Any web developer worth their salt knows what the string means and what browser it represents. Which is okay because user agents are to be parsed by computers and rarely developers, not end users.
Well that's quite rude, you don't know me at all so you have no idea how much experience I have. And it's also quite contradictory... if the string is only rarely to be parsed by developers, why would "any web developer worth their salt" bother memorizing every browser's user agent? Experienced web developers don't have anything better to do than memorize a useless string they rarely have to work with?
This is a good idea, and is something I've thought of for a while; the user agent header was a mistake from both a privacy and a UX perspective.
Ideally, web browsers should attempt to treat the content the same no matter what device you are on. There shouldn't be an iOS-web, and a Chrome-web, and a Firefox-web, and an Edge-web; there should just be the web. In which case, a user-agent string that contains the browser and even the OS only encourages differences between browsers. Adding differences to your browser engine shouldn't be considered safe.
Which brings me to privacy. It's not as if there aren't other ways to try and fingerprint a browser, but the user agent is a big mistake for privacy. It'd be one thing if the user-agent just said "Safari" or "Firefox", but there's a lot more information in it beyond that.
If the web should be the same web everywhere, then the privacy trade-off doesn't make much sense.
I agree, but this also is incredibly dependent on the major players (e.g. Google) not going off on their own making changes without agreement from other browsers...
Thinking cynically, it could be a power-move by Google to strengthen their hold on the ecosystem.
Right now when they go out and make their own API changes without consensus (which already happens), it's possible to distinguish the "for Chrome" case and still support the standard. But if there were no User-Agent, and Google wanted to strongarm the whole group into something, and 90% of browsers are Chromium-based, devs will likely just support the Chromium version and everyone else will have no choice but to fall in line.
As far as I can tell, HTTP/2 is such a major improvement that no strong-arming is necessary. Speaking as a consumer of the web, as an individual who runs their own website, and at a developer working at a company with a major web presence.
The web suffers a ton from the “red queen” rule in so many different ways anyway—you have to do a lot of work just to stay in the same place.
Yes, unequivocally. It’s amazing, even without push. The websites that use it are faster, and the development process for making apps or sites that load quickly is much more sane. You don’t have to resort to the kind of weird trickery that pervades HTTP/1 apps.
> Or is it just an improvement for Cloud provider that keep pushing the Kool-Aid ?
I don’t see how that makes any sense at all. Could you explain that?
> But maybe you did put and leverage HTTP2 on your own website ? At your company ? Did you use push ? Do you use it with CDN ?
From my parent comment,
> Speaking as a consumer of the web, as an individual who runs their own website, and at a developer working at a company with a major web presence.
My personal web site uses HTTP/2. It serves a combination of static pages and web apps. No push. HTTP/2 was almost zero effort to set up, and instantly improved performance. With HTTP/2, I’ve changed the way I develop web apps, for the better.
My employer’s website uses every technique under the sun, including push and CDNs.
>> Or is it just an improvement for Cloud provider that keep pushing the Kool-Aid ?
> I don’t see how that makes any sense at all. Could you explain that?
I've seen a few CDN having a page loading a grid of image in HTTP/1 at page load, and then load the same stuff with HTTP/2 on a button click. It indeed shows you a nice speed up.
Except, when you block the first HTTP/1 load and start with loading with HTTP/2 first and flush cache between loads, the speedup vanishes. The test is disingenuous, it is not testing HTTP/2 but DNS cache velocity.
So, those type of website makes me rather cautious. And the test, for the small scale workloads I work with, have not been very conclusive.
Do you have serious articles on the matter to recommend ? Preferably not CDN provider trying to sell me there stuff.
> Except, when you block the first HTTP/1 load and start with loading with HTTP/2 first and flush cache between loads, the speedup vanishes. The test is disingenuous, it is not testing HTTP/2 but DNS cache velocity.
The demos I’ve seen use different domain names for the HTTP/1 and HTTP/2 tests. This makes sense, because how else would you make one set of resources load with HTTP/1 and the other with HTTP/2? This deflates your DNS caching theory.
I didn’t rely on tests by CDNs, though. I measured my own website! Accept no substitute! The differences are most dramatic over poor network connections and increase with the number of assets. I had the “privilege” of using a high-RTT, high congestion (high packet loss) satellite connection earlier this year and difference is bigger.
What I like about it is that I feel like I have more freedom from CDNs and complicated tooling. Instead of using a complicated JS/CSS bundling pipeline, I can just use a bunch of <script>/<link>/"@import/import". Instead of relying on a CDN for large assets like JS libraries or fonts, I can just host them on the same server, because it’s less hassle with HTTP/2. If anything, I feel like HTTP/2 makes it easier to make a self-sufficient site.
Finally, HTTP/2 is so dead-simple to set up on your own server, most of the time. It’s a simple config setting.
I, too, remember how this opinion was repeated as nauseam about a decade ago. It didn’t make much sense then, and even less now.
Nobody today expects identical rendering: people are used to responsive websites, native widgets etc. The problem people are actually experiencing (far less now than in the past) were more serious, such as z-axis ordering differences resulting in backgrounds obscuring content.
If I'm connecting to a site with Lynx, I sure as heck don't want them to try to serve me some skeleton HTML that will be filled in with JS. Because my browser doesn't support JS, or only supports a subset of it.
User Agent being a completely free form field is the real mistake IMO. Having something more structured, like Perl's "use" directive, might have been better.
One problem with this is that browsers don't behave the same. For example, iOS Safari prevents multiple pop-up windows to be opened by a single user interaction. Each one requires clicking back to the original page and allowing the popup. Now you might say, "Why would you ever want to do that?" But there are always going to be edge cases—in this case it's an integral part of one of the features of autotempest.com. But that's just one example. And the only way we can detect whether that behaviour is going to be blocked is by checking the UA.
I can understand why this is a good thing for privacy. Like many things to do with security on the web though, it's just a shame that bad actors have to ruin so many things for legitimate uses. (The recent story on Safari local storage being another example of that...)
There are a variety of scenarios where this comes up (e.g. we ship a site that is rendered, by another vendor, within an iframe; so we have to set SameSite: None on our application's session cookie so that it's valid within the iframe, thus allowing AJAX calls originating from within the iframe to work based on our current auth scheme.. BUT only within Chrome 70+, Firefox but NOT IE, Safari, etc).
We are in the same boat. Certain browser/OS combinations don't handle Same-Site correctly, so we are using UA sniffing to work around their limitations by altering Same-Site cookie directives for those browsers. We will likely have to look at some other mechanism for dealing with nonconforming Same-Site behavior.
These days, it feels like the sole use of User-Agent is as a weak defence against web scraping. I've written a couple of scrapers (legitimate ones, for site owners that requested machine-readable versions of their own data!) where the site would reject me if I did a plain `curl`, but as soon as I hit it with -H "User-Agent: [my chrome browser's UA string]", it'd work fine. Kind of silly, when it's such a small deterrent to actually-malicious actors.
(Also kind of silly in that even real browser-fingerprinting setups can be defeated by a sufficiently-motivated attacker using e.g. https://www.npmjs.com/package/puppeteer-extra-plugin-stealth, but I guess sometimes a corporate mandate to block scraping comes down, and you just can't convince them that it's untenable.)
Preventing scraping is an entirely futile effort. I've lost count of the number of times I've had to tell a project manager that if a user can see it in their browser, there is a way to scrape it.
Best I've ever been able to do is implement server-side throttling to force the scrapers to slow down. But I manage some public web applications with data that is very valuable to certain other players in the industry, so they will invest the time and effort to bypass any measures I throw at them.
As a person who scrapes sites (ethically), I think it's impossible or pretty damn near impossible to prevent a motivated actor from scraping your website. However, I've avoided scraping websites because their anti scraping measures made it not worth the effort of figuring out their site. I think it's still worth for do minimal things like minify/obfuscate your client side JS and use some type of one time use request token to restrict replay-ability. The difference between knowing that I can figure it in 30 minutes vs 4 hours vs a few days is going to filter out a lot of people.
Of course, sometimes obfuscating how your website works can make it needlessly more complicated, so it's a trade off.
Checking the user-agent string for scrapers doesn't work anyway. In addition to using dozens of proxies in different IP address blocks, archive.is spoofs its user agents to be the latest Chrome release and updates it often.
Meanwhile, you can still use youtube.com/tv to control playback on your PC from your phone—but only if you spoof your User-Agent to that of the Nintendo Switch . Sounds like they are more interested in phasing out user control than ignoring the header entirely.
Yes. I firmly believe this is an attack on user control.
For example, I believe they REALLY want us to use the youtube app:
- Viewing youtube.com on a new iPad pro, Goolag lies and says "your browser doesn't support 1080p."
- Ok, change to desktop version in app. Goolag once again lies and says "your browser doesn't support full screen." They also lie and say they've redirected you to the "desktop version", and nag you with a persistent banner that you should return to the safety of the mobile website.
If I were to use the app, they would have FULL CONTROL.
The same is true for server side applications of user-agent. There are plenty of non-privacy-invading reasons to need an accurate picture of what user agent is visiting.
And a lot of those applications that need it are legacy. Updating them to support these 6 new headers will be a pain.
Chrome will support the legacy apps by maintaining a static-user agent. It just won't be updated when chrome updates. If you want to build NEW functionality that where you need to test support for new browsers, you do that via feature detection.
Interesting. We don't use UA to track customers, but it has been invaluable information for trying specific bugs. Eg, twice in the past 2 months, I've had to fix weird bugs that didn't make sense. The only way I was able to solve them was to look for patterns in which browsers and versions those who reported the bugs were using. Both turned out to be to do different iOS Safari cookie related bugs that only occurred in specific versions. Without logging the UA there would have been no way I would have been able to discover those bugs and create workarounds for those iphone users.
I'm all for preventing tracking, but I can't imagine a time where all browsers behavior so similarly that we won't have to write workarounds for browser bugs and differences. As a developer I can't imagine caring about Edgium vs Chrome, but it's important to know what the underlying engines are.
Why not getting rid of the `User-Agent` completely?
Try browsing the web without any UA header for a week or two, and you'll understand. You get blank pages, strange server errors, and other weird behaviour --- almost always on very old sites, but then again, those also tend to be the sites with the content you want. Using a UA header, even if it's a dummy one, will at least not have that problem.
(I did the above experiment a long time ago - around 2008-2009. I'm not sure whether sites which expect a UA have increased or decreased since then.)
I agree with getting rid of all that new noise, however.
I mean at first this change will cause all of these errors too until the servers migrate from user agent string to device hints. Getting rid of UAs would actually force meaningful update rather than migration.
The browser isn't for "regular" websites, it's for all websites.
I remember when I was writing a library around the audio API, and the ways it behaved on Chrome were different across Macs, Windows and Android. Detecting the OS with the user-agent string was literally the only way to build code that would work.
Yeah, aren't we just making things a lot more messy? Especially if we're not planning on removing the User-Agent header?
This pattern keeps repeating itself, freeze "Mozilla/5.0", start changing "Chrome/71.1.2222.33", freeze that, start changing "Sec-CH-UA", etc. Browsers will start needing to fake "Sec-CH-UA" to get websites to work properly, etc.
> Blocking known bots and crawlers
Currently, the User-Agent string is often used as a brute-force way to block known bots and crawlers. There's a concern that moving "normal" traffic to expose less entropy by default will also make it easier for bots to hide in the crowd. While there's some truth to that, that's not enough reason for making the crowd be more personally identifiable.
This means that consumers of the Google Ad stream have one less tool to identify bots, and will pay Google for more synthetic traffic, impressions and clicks; this could be a huge revenue boost for Google. A considerable amount of their traffic is synthetic. I doubt this was overlooked.
does this mean there will no longer be a way of determining if the device is primarily touch (basically all of "android", "iphone" and "ipad") or guesstimating screen size ("mobile" is typical for phones in the UA) on the server?
Presumably you'll grab the dimensions (could cache after first load) and then render dynamically based on that. If you're doing some sort of if statement on the server to deliver content based on screen size you're probably doing it wrong. Obviously I can't speak for every mobile user, but for myself, it's infuriating to have a completely different set of functionality on mobile.
> If you're doing some sort of if statement on the server to deliver content based on screen size you're probably doing it wrong. Obviously I can't speak for every mobile user, but for myself, it's infuriating to have a completely different set of functionality on mobile.
there's not a "right" and a "wrong" here; it's about trade-offs.
you're either stripping things down to the lowest common denominator (and leaving nothing but empty space on desktop) or you're wasting a ton of mobile bandwidth by serving both versions on initial load (the most critical first impression).
you frequently cannot simply squeeze all desktop functionality from a 1920px+ screen onto a 320px screen - unless you have very little functionality to begin with. Amazon (or any e-commerce/marketplace site) is a great example where client-side responsiveness alone is far from sufficient.
https://www.walmart.com/ does it okay, but you can see how much their desktop site strips down to use the same codebase for desktop and mobile.
browser feature detection is the way grown up developers have been doing this for several years now. user agent sniffing is dumb because it bundles a ton of assumptions with a high upkeep requirement, all wrapped up in an unreadable regex. It's been bad practice for ages; I'd be surprised if that's how Amazon is doing it still.
Can't happen soon enough. As a frequent user of various non-mainstream browsers i'm sick and tired of seeing "your browser isn't supported" messages with download links to chrome/etc. At least in the case of Falkon it has a built in user agent manager, and I can't remember the last time flipping the UA to firefox/whatever actually caused any problems. Although, i've also gotten annoyed at the sanctimoniousness web sites that tell me my browser is to old because the FF version I've got the UA set to isn't the latest.
Google wants to create a walled-garden net. Goog-net. All ads, shopping, videos, documents, email, locations, articles flowing through THEIR protocols and THEIR servers and THEIR fiber. No possibility of blocking ads they don't want blocked. No URLs. No Agent strings. No user control. Only user consumption.
If they have to allow some small chump players to have a piece of this cake (a la: "Oh trust us, AMP is an 'open' protocol and anyone can host it. it's not just for our own benefit in the end. Trust us.") in order for the entire population to accept their changes bit by bit, so be it in their eyes. They know we would reject an outright takeover.
User-agent is super useful to human people. But corporate people don't have a use for it. They will get that information via running arbitrary code on your insecure browser anyway. So, because mega-corps now define the web (instead of the w3c) this is life.
But it doesn't have to be. We don't have to follow Google/Apple-Web standards. Anyone that makes and runs websites has a choice. And every person can simply choice not to run unethical browsers.
For what? Honest question. You have to be like a 5th-level user agent wizard to make any sense of user agent strings, since every browser now names every other browser. How do you do anything useful with this in a way that's forward-compatible?
Not sure why you are being downvoted since your statements are correct.
Few advertisers rely on user agent for ad targeting since it can be easily mocked with each HTTP request. It is used for fingerprinting, sure, but from my experience, mostly as a way to identify bot traffic.
It is also true that the advertisers that fingerprint people rely on JS that executes WebGL code in order to get data from the machine.
Finally, you are right that it doesn't make sense that a company like Google dictates these standards since they have a conflict of interests worth almost a trillion dollars.
This is insane. You know, no HN post to a google blogspot site works for me because these jerks are the only ones that discrinate on UA?
Google engoneering is Sooooooo disconnected from the rest of the world, I think we need legal regulation to stop them from doing stuipd things like this. Do they have any idea how many things need it?
HNers with a position of power at work, I plead with you: please advocate banning of Chrome at work and replacing it with any one of the webkit based alternatives or firefox. These people are insane. Every month I hear of some ridiculous thing. They took out navbar url parameters, add links to in page words, now this!
For those who think this is good for privacy...it is not! This is the same old sneaky ass evil thing they do. UA can be used to finger print you but it's very easy to set a generic user agent. Actually, if you look at user agents most of them have the latest string for Chrome, IE or firefox so it isn't useful without a whole lot of other details correlated with it. You know what the exception is? Android and iPhone browsers that incluse your device make and model in the UA and apps that includes whole lot more like facebook's apps.
Do you know what a "flexible" api like the one they're talking about allows? More fingerprintable data points! The fact that you even use that api is a privacy issue. Let's sat clienthints allows for 10 different variations of responses from clients,your specific client details might have just 3 things different from the mean and bam, now they can track your specific device. With UA,all versions of a client have the same exact detail and most people need an extension to change it, so it makes it much less easy to finger print.
This is the same ol sneaky bait and switch Google pulls. The content of your UA is not the privacy concern (although it contains too much at times) , it is the fact that it can be correlated with timing info,IP (especially v6),and if they already know your UA they will also use client default http header options to identify and track you without consent.
gp is likely asking about how servers decide to redirect to the m.* version instead of the desktop version (or in some cases serve a different mobile version under the same domain), in which case, yes, it’s usually user agent sniffing.
UA strings have never been an accurate indication. If you're not using JS, then you probably have no reason to be sniffing the UA string to detect browser features, since most of those features are JS-related anyway.
It's an upgrade for the people who actually need to get an indication of the supported features and APIs of the user's browser. Otherwise, you should be using media queries.
One exception: you might want to user sniff IE and serve a completely different version due to all the CSS problems. (I know you can use IE-only comments too, but I’ve been in the situation where making a modern version simultaneously IE9-compatible was just too frigging maddening.)
If you want to just change the styling and layout of the page depending on the user's device, then you can use css's media queries. But if you want to serve two totally different pages (one for mobile and another for desktop), then I don't see how it can be done without JS or reading the user agent.
> While removing the User-Agent completely was deemed problematic, as many sites still rely on them, Chrome will no longer update the browser version and will only include a unified version of the OS data.
So, nearly all of the information that makes User-Agent strings problematic will remain. They're just phasing out precise version information.
Annoying, as I just added a user agent based workaround for another Chrome compatibility problem (the increased security on same-site cookies, which can't be handled in a compatible way with all browsers).
How will things be broken? Google is not removing the user agent, they're just freezing it. So all sites that currently depend on the user agent will continue to do just fine. New sites can use client hints instead, which are a much more effective replacement for user agent sniffing.
This solution very specifically places the burden on "techy nerds" and not users, so I'm not sure where you're coming from.
Right, using user agent on the client side has been unsalvageably broken for a long time. Other things, like checking the existence of window.safari or window.chrome are more reliable.
For the server side, I’m not too aware of too many cases it’s useful other than analytics, and there is too much info leakage and fingerprinting happening anyway.
So killing user agent doesn’t really seem user-hostile, save for the fact that the company doing it has near monopoly market share and doesn’t need to provide a user agent, as it’s assumed that everyone is writing code to run on Google’s browser. In that sense it’s a flex.
I use Safari with no plugins. Even Disney World has a broken website for buying tickets for me. The web is breaking because it's gotten way too complex and the fight against trackers is leading to random failures of things that used to work.
Which is a shame, but I would lay the blame squarely at the feet of the team who built a checkout that throws errors when their analytics events don't fire. QA should really include a manual run w/ an adblocker...
Thats true, but its quite likely that simply testing on a couple available browsers and avoiding browser specific checks means that safari (and other non mainstream browser) users will be fine.
Seems they considered this issue and created a work-around:
>While removing the User-Agent completely was deemed problematic, as many sites still rely on them, Chrome will no longer update the browser version and will only include a unified version of the OS data.