Show HN: I built a FOSS service to protect against link shorteners tracking us

(unshort.link)

57 points | by l1am0 1608 days ago

11 comments

myself248 1608 days ago
Related, the Archive Team realized a while ago, that when a link shortening service goes dark, all the links that went through it instantly break. This has bad implications for future historians trying to figure out how the present-day web was interlinked.
So there's a project called URLteam to index and save them. I just started running a Warrior instance on the URLteam project, and it's really straightforward. I figure folks reading this may be interested. More info:
https://www.archiveteam.org/index.php?title=URLTeam
https://www.archiveteam.org/index.php?title=ArchiveTeam_Warr...
Wowfunhappy 1608 days ago
Where and why are you guys still seeing/using shortened URLs? I don't think I've personally made a shortened URL since Twitter built one into their service, and I stopped seeing shortened URLs around the same time.
There are two exceptions:
• I still see service-specific short URLs, e.g. goo.gl, nyti.ms, etc. Because these can only be created by the service in question, they don't introduce security/privacy concerns.
• When a URL shortener is used to create a sort of vanity link to e.g. a Google doc. (OP's service could be useful for these!)
Outside of those times... why would you use a URL shortener? When are you limited by number of characters?
[-]
- Terretta 1608 days ago
  If you use an tracker blocker, you’ll find a slew of app to app and site to site links and callbacks fail because they went through a tracker that sometimes is a short URL service, but often just a long URL redirector.
  These are definitely not curated by the service in question.
  On iOS, examples include:
```
    [business].app.link
    [business].onelink.me
    [hash].ulink.adjust.com
    [business].bttn.io
    branch.io
    etc...
```
  Link shorteners are a subclass of this, that do something useful (shorten link) while doing something unhelpful (obscuring the link). But it’s all the same problem.
- tyingq 1608 days ago
  I use one inside of a big corp so I can put memorable urls like myco.co/news on posters or email signatures. Especially when the real url is some terrible long thing I cannot control, like some internal wiki.
- l1am0 1608 days ago
  Link shorteners are also used to ubfuscate tracking urls. E.g you can hide https://fb.com/?utm_source=hackernews&other_tracking_paramet... behind a nice looking bit.ly/fb url.
  In my research I also found that it is a quite nice thing for phishing to fool you into opening a shady website.
- kick 1608 days ago
  • I still see service-specific short URLs, e.g. goo.gl, nyti.ms, etc. Because these can only be created by the service in question, they don't introduce security/privacy concerns.
  This isn't necessarily true; I've seen companies that offer to do the 'custom shortlink' thing for various companies. It's safe to assume that there are privacy concerns with at least some of them.
  [-]
  - tyingq 1608 days ago
    Bitly runs nyti.ms, so yes, you aren't trusting just the NYT.
    [-]
    - edoceo 1608 days ago
      Yes, and nyti.ms only ever pints to New York Times, a well known entity (well, that's what's intended, right?)
      [-]
      - tyingq 1608 days ago
        Sure. Noting that you have to trust that bitly isn't fingerprinting your browser and tracking you across their domains.
- iudqnolq 1608 days ago
  I use them all the time to transfer URLs between devices that enxode a lot of state in a long url. For example, Firefox Send pits the password in the URL. Of course, I'd never do this with something I didn't watch associated with me.
- type0 1607 days ago
  On youtube every so often complementary materials are linked via shortened urls, but youtube also tracks those in turn via their redirect urls.
- syrrim 1608 days ago
  Some subreddits use them to track which links people click on in their sidebar. /r/listentothis in particular.
- BruiseLee 1605 days ago
  SMS spam.
iudqnolq 1608 days ago
Reminds me when security researchers used data Bit.ly exposes to go from one url shortened by Russian military intelligence to the hundreds of other short URLs they made for phishing emails, many of which had state embedded in query parameters that gave away who they were targeting. All because the GRU forgot to set their accounts to private.
https://www.vice.com/en_us/article/mg7xjb/how-hackers-broke-...
l1am0 1608 days ago
Oh wow thanks for all the feedback. Should not post a link on HN and than sleep directly after.
I came to work on this as I wantend to not get bothered with the short link services (and the links behind them) tracking me. So unshort.link also tries to "learn" (by trying different url parameters) which paramters on the long url are required and which are not and are only tracking you (with a HTML diff).
The service learns new shortlink services the moment you enter them on unshort.link and they than are also automatically used in the extension.
OJFord 1608 days ago
Yes yes yes! I hate URL shorteners. It's not just privacy - I can't decide if I want to click it or jot before I do.
I suppose that issue could also be solved with a browser extension.
It'd be nice to do this with DNS rather than an extension though, above point aside, e.g. if already running Pi-hole or similar, send shortener URLs to unshort.link, and redirect immediately.
[-]
- tyingq 1608 days ago
  Pi-hole would only see the hostname, and not the "slug", so that wouldn't work.
  [-]
  - OJFord 1608 days ago
    What I mean is, you'd send `t.co` and similar to your unshort.link server, which then has the `Host` header `t.co` and the path; so it can do the redirect.
    I know it's not how it works today, but I wasn't describing how it works, I was describing how I'd like it to.
  - edoceo 1608 days ago
    Pi-Hole sees the bit.ly DNS lookup,that responds with HTTP/30x to full.domain.tls, which is a second DNS lookup the Pi could catch/filter.
    [-]
    - tyingq 1608 days ago
      But bitly.com/xyz can be mapped to full.domain/abcdef
      The pi-hole doesn't see the /xyz, or any of the http redirects either.
- l1am0 1608 days ago
  unshort.link has a browser extension for chrome & firefox, you find it on the site.
  If you directly want to get redirected without showing the url you can activate that feature in the plugins config.
  Or you can send the shortlinks to http://unshort.link/[http://short.link/sd] and it will directly redirect you
  [-]
  - OJFord 1608 days ago
    Yes, that's the browser extension that I was referring to when I said I'd rather do it (for all devices) via DNS.
```
    1. Resolve shorterners to unshort.link server
    2. Use `Host` header and path to redirect to unshortened page
```
    [-]
    - l1am0 1608 days ago
      Would be awesome if you build something like that! You can get the currently to unshort.link known short link providers in a nice json format from https://unshort.link/providers
      Maybe you can run a cronjob setting that urls in some firewall file or so.
      You can get the redirect info from unshort link also via its api without redirecting, just replace /d/ with /api/ in the GET request: e.g. https://unshort.link/api/https://tinyurl.com/unshortchromeex...
      If you build an DNS solution, please do not forget to add the documentation on how to do it via a pull request to the unshort.link repo, so everyone could profit from it
tyingq 1608 days ago
Pretty cool. You might consider adding some of the more popular vanity names that bitly runs for big sites. Like nyti.ms (New York Times), wapo.st (Washington Post), etc.
[-]
- l1am0 1608 days ago
  You can add them yourself :D Just unshort them on unshort.link (not the browser extensions) and the service learns that it found a new shortlink service and it will be deployed right away.
  After the next browser restart also the extension knows about that new service. (No updated required)
  You can see if your updated worked if e.g. wapo.st is listed in https://unshort.link/providers
pmoriarty 1608 days ago
How do we know this site is not tracking us?
[-]
- l1am0 1608 days ago
  TBO you would never know, that's why the project is open source and if you want to be really sure what happens here, run it yourself.
  The browser extension can be easily (in the settings) tweaked to use your own server instead of unshort.link
- notduncansmith 1608 days ago
  You can run it yourself, and audit the source if you like: https://github.com/simonfrey/unshort.link/
  [-]
  - eat_veggies 1608 days ago
    If you run it yourself, do you not lose some of the privacy benefits?
    [-]
    - notduncansmith 1608 days ago
      I believe you gain privacy by running it yourself, since you're no longer submitting a portion of your browsing history to a third party. The README for the server claims "You can build & run it yourself for even better privacy".
      [-]
      - OJFord 1608 days ago
        Shorteners are frequently used to hide/prettify query parameters used for tracking; so if you run your own you're losing the aggregate anonymity-ish, and gaining that OP can't see what URLs you're unshortening.
        I suppose it depends on your use, but IMO if you run it yourself as the only user, you lose more privacy than you gain.
        [-]
        notduncansmith 1607 days ago
        If the website you're ultimately visiting can see your IP address anyways, I fail to see the lost privacy. Their traffic will be sliced and diced into cohorts along many axes, and this is only one axis on which you are slightly anomalous. If anything, you'll be targeted less as a result since you won't show up in their funnels.
zzo38computer 1606 days ago
Is it possible to make such a thing working if using command-line programs such as curl or wget to download a file rather than opening it in the web browser?
[-]
- l1am0 1606 days ago
  It should totaly be possible. Same as with the DNS solution you can get the data via the api of unshort.link and than build a small bash script for it. If I have time I may try to do such a script, but not sure if and when I will be able to make it.
  Would be awesome if you build something like that! You can get the currently to unshort.link known short link providers in a nice json format from https://unshort.link/providers
  You can get the redirect info from unshort link also via its api without redirecting, just replace /d/ with /api/ in the GET request: e.g. https://unshort.link/api/https://tinyurl.com/unshortchromeex....
  If you build a solution, please do not forget to add the documentation on how to do it via a pull request to the unshort.link repo, so everyone could profit from it
- moehm 1605 days ago
  This is what I use which also removes utm_* tracking parameters.
```
  getloc () {
     curl -sI "$1" | grep -i "Location" | awk '/http.*/{print $2}' | sed 's/[%+]*//g' | sed 's/&*utm_[a-zA-Z0-9]*=[a-zA-Z0-9]*//g' | sed 's/?\s*$//g'
  }
```
zelon88 1608 days ago
This is really cool. And it's all written in Go? Interesting.
Do you have any plans on releasing the static JS/HTML? Or is there a way this could be run on standard Apache/Nginx/IIS web servers?
[-]
- l1am0 1608 days ago
  No I do not play to make it runnable on Apache/Nginx/ISS as there is more logic in it, than just the redirect.
  The js/html assets are in the subfolder server/static/ for your use
gesman 1605 days ago
Add: Replacement of affiliate or referral links with your user' affiliate link and you can charge for this service.
gesman 1605 days ago
I wanted to make something like this with my C.GG domain but couldn't refuse an offer to sell the name for $10k