Mostly in the US, where filters (if any) are set up by mailing eachother deadtree LOAs about party A allowing party B to announce their prefix. It's basically the checkbook way of doing things, but for BGP.
Meanwhile in EMEA we've been using IRR (and recently RPKI) for automating filters for years now. I couldn't do a BGP hijack through my EU upstreams even if I wanted to.
The world's financial systems depend heavily on trust too.
Take credit cards. As an end user you perceive some fairly heavyweight albeit imperfect security, but in reality there are two parallel systems and only one of them is using those security features.
Authorisation is the part that ties your good name to your transactions. The bank doesn't trust you, so considerable technological effort is used in Authorisation. Chip cards, PINs, extra passwords and codes for web transactions, that's all Authorization.
Settlement moves money between your account and a merchant account. This isn't secured at all, it's just based on trust. Any merchant can assert that you used your card to pay them any amount and by default the bank takes it on trust.
Authorisation is abuse resistant. If you try to replay the same authorization multiple times it doesn't work.
Settlement doesn't care. If a worker messes up and puts a big merchant's daily settlement transactions through twice, every single one of those transactions just happens again. Paid $80 for groceries leaving $38 in the account until pay day? Sorry, now you're $42 overdrawn. This really happens.
If you complain they'll fix it. If hundreds of customers complain they'll realize the error and fix it for everyone. But this is why you have to check your credit card bill. Sooner or later there will be errors, and if you don't check you'll never see them which can get expensive fast.
On the other hand see how double spending works in less regulated financial markets like cryptocurrency. What keeps that honor system working in financial systems is that it's backed by strong regulation and strong supervisory authorities. There are so many controls built into this that mistakes happen less often than one might think, considering the volume of non-bank transactions (think billions++, yearly).
ISPs have noting of the kind. One routing mistake (or "mistake") and all of your customers' data suddenly went through China, Russia, US (you name it), for better or worse. What's worse, in a manner of speaking that data is now devalued. There's value date equivalent in the ISP's world. As a customer you have no avenues, the thing isn't even between you and any ISP you may work with. The bank will never get away with telling you "well we accidentally made this extra payment, it got routed thorough this extra bank, so now you lost 25% of the money".
Financial institutions are much more heavily regulated by governments than ISPs. The honor system in finance only works because regulators ensure (today more so than a few decades ago) that all players fulfill minimum requirements.
Cogent and Level3 weren't filtering their own customer. This is what happens when large companies say prefix list updates / validation is too much of a burden and allow customers to run bareback BGP. That said, Cogent is going to be doing RPKI on customers soon so that's good.
Rostelecom (AS12389) is directly connected to Level3 (AS3356) and announced the hijacked routes to them but Level3 filtered them out.
But...
AS12389 announced them to Rascom (AS20764), who accepted them and sent them on to Cogent (AS174), who accepted them and propogated them on to Level3 (AS3356), who, unfortunately, then accepted them from Cogent.
The AS path would've been much longer than if Level3 had accepted them directly from AS12389 so, at the least, the actual effects of the hijacking weren't as bad as they could have been.
When the internet really started to become a big "thing" in the mid-90's and I started learning how it all worked I thought, "well, as soon as hackers realize how insecure this whole thing is, that'll be the end of that..."
Border Gateway Protocol is just one routing protocol, calling it "the fundamental protocol" is stretching a point. That award probably goes to TCP. Maybe IPv4.
The ability to do this is effectively what enables DNS traffic amplification attacks (and others).
Trust has been broadly effective, but also quite damaging at times. It's part of why I'm glad https is nigh universal - http's trust allowed stuff like Firesheep and all the hotel shit-injection that used to be so prevalent.
>The route leak was distributed quite well through Rascom (AS20764) , then Cogent (AS174) and in a couple of minutes through Level3 (AS3356) to the world.
Is there a reason why these can't be filtered at the ISP level? eg. Cogent knows that Rascom shouldn't be announcing those prefixes, and refusing to route traffic to them unless there's manual verification?
> Cogent knows that Rascom shouldn't be announcing those prefixes, and refusing to route traffic to them unless there's manual verification
I run a large production BGP network. With two exceptions every provider needed a Letter of Authorization from me to send to their upstream that authorized the announcement of my IP space (the exceptions being India where they wanted to charge extra for filtering announcements, and Russia where they offered to not do filtering for an extra charge).
This "manual verification" already takes place. It just doesn't apply to large transit providers interconnecting like what happened here
The challenge is with scale. If I'm a network who has many networks downstream of me I might have 50 to 100 prefix changes in a month. None of my upstream providers are going to like manually vetting all of those - and none of my customers don't want to wait days before their announcements propagate. So there is this sort of state you can land yourself as a "you're a big customer, we seem to trust you".
The reality is that RPKI Origin Validation solves this (but not as path spoofing, we need ASPA for that) and people need to publish valid ROAs.
Back in the 90's, nobody did LOAs. I remember reading prefixes over the phone to a guy at InternetMCI when I was moving an ISP from a 56K line to a T1. Fun times!
IRR already contains all the data. Cryptographically signing it can't increase the rate at which people will use it (one could argue throwing crypto into the mix decreases adoption). RPKI is just crypto nerds trying to math their way in to a human and financial problem.
Until routing vendors start shipping sane IRR based route filtering in default configs and make it hard to turn off, this is something we have to deal with.
> RPKI is just crypto nerds trying to math their way in to a human and financial problem.
Well-designed cryptography lets us tilt the odds violently on these human problems so as to give us greater confidence to take action when appropriate.
If you say maybe your technician typed 185.42.16.0 when they meant to type 158.42.61.0, that they uploaded last month's data instead of this month's, they wired a patch incorrectly - these are not implausible accidents. But cryptographically signed records don't happen by accident. You can't "typo" a 1-in-billions-of-billions chance.
IRR contains data - question is can you trust it? RIPE has cleaned up their stuff, sure, but the rest of the IRRs out there haven't. Next question is which IRR once you get past the RIRs? AltDB? There's a few out there. Lots of crap objects and proxy registration, that said there are a lot more things in IRR than ROAs out there.
I do agree with you that routing vendors have made this difficult to easily ingest prefix list data for easy import into inbound filters. There have been NANOG presos from last decade showing the performance hit when you try to do this at scale. It is much easier to enable RPKI OV than it is to build a robust IRR filtering system.
Some of them, where origin didn't match ROA. The rest can be filtered by bgp filters based on IRR data. IRR filtering would cut more that 80% of that leak.
The real question is how many networks doing RPKI Origin Validation (OV) weren't accepting the ARIN TAL (ignoring ROAs from ARIN) due to their license to provide them indemnity. Remember that RPKI validators require a network to manually add ARINs TAL.
This is very similar to the caller-id problem that plagues telephones. Our networks were built for trust that no longer exists. Our protocols need to catch up with the times.
Oh look, another submission of a tweet. Probably won't be able to read the tweet in the future when it gets deleted, no real context or other information in the tweet.
It would literally be more useful if you just screenshot it and posted that, as at least it would last longer, not to mention add detail.
> Probably won't be able to read the tweet in the future when it gets deleted, no real context or other information in the tweet.
I find the tweet helpful. I didn't know about BGPmon, for instance.
Screenshots, on the other hand... They're not accessible, and they don't provide easy links back to the author, nor access to the tweet's reply context. I find all of those valuable.
Why do you say Facebook has the means to mitigate this quite easily? That's not really true as I understand it.
The problem is that these announcements are made by third parties, and accepted by fourth parties, both of which Facebook has no control over.
While it's possible for Facebook to implement some controls over what routes THEY accept and therefor what routes Facebook sends traffic TO - it cannot control where others send traffic. If a third party advertises Facebooks routes to a fourth party, and they accept them, Facebook cannot do anything directly about that. And then those fourth parties accepting the routes will send traffic TO Facebook via the third party.
I didn't look to see how much of Facebook's ranges got hijacked. Assuming some got hijacked and others didn't (which is usually what happened), they could direct more traffic to ranges that weren't hijacked (assuming their DNS ranges didn't get hijacked). For the ranges where the hijacking wasn't at the /24 level, Facebook also has plenty of staff who could adjust to advertise more specific routes. All the big internet companies with ASNs should have that though; but the smaller places that probably got hijacked too may not.
Disclosure: I used to work at WhatsApp, including while it was part of Facebook. Everything in this message is armchair routing policy though.
Meanwhile in EMEA we've been using IRR (and recently RPKI) for automating filters for years now. I couldn't do a BGP hijack through my EU upstreams even if I wanted to.
As a bonus, my US upstream doesn't care if I have RPKI/ROA/IRR - even if I do, they ask for dead tree LOA anyway.
When I sent a dead tree to one of my EU upstreams out of habit they were like "what do I need this for?"
Take credit cards. As an end user you perceive some fairly heavyweight albeit imperfect security, but in reality there are two parallel systems and only one of them is using those security features.
Authorisation is the part that ties your good name to your transactions. The bank doesn't trust you, so considerable technological effort is used in Authorisation. Chip cards, PINs, extra passwords and codes for web transactions, that's all Authorization.
Settlement moves money between your account and a merchant account. This isn't secured at all, it's just based on trust. Any merchant can assert that you used your card to pay them any amount and by default the bank takes it on trust.
Authorisation is abuse resistant. If you try to replay the same authorization multiple times it doesn't work.
Settlement doesn't care. If a worker messes up and puts a big merchant's daily settlement transactions through twice, every single one of those transactions just happens again. Paid $80 for groceries leaving $38 in the account until pay day? Sorry, now you're $42 overdrawn. This really happens.
If you complain they'll fix it. If hundreds of customers complain they'll realize the error and fix it for everyone. But this is why you have to check your credit card bill. Sooner or later there will be errors, and if you don't check you'll never see them which can get expensive fast.
ISPs have noting of the kind. One routing mistake (or "mistake") and all of your customers' data suddenly went through China, Russia, US (you name it), for better or worse. What's worse, in a manner of speaking that data is now devalued. There's value date equivalent in the ISP's world. As a customer you have no avenues, the thing isn't even between you and any ISP you may work with. The bank will never get away with telling you "well we accidentally made this extra payment, it got routed thorough this extra bank, so now you lost 25% of the money".
As for Level 3 the world wonders.
But...
AS12389 announced them to Rascom (AS20764), who accepted them and sent them on to Cogent (AS174), who accepted them and propogated them on to Level3 (AS3356), who, unfortunately, then accepted them from Cogent.
The AS path would've been much longer than if Level3 had accepted them directly from AS12389 so, at the least, the actual effects of the hijacking weren't as bad as they could have been.
RIP can't cope with the scale of the internet today, but to say BGP is essential to interconnecting networks is not right
If you take away BGP, there is no Internet!
No other protocol in existence/use today can scale the way BGP has.
TCP connection setup is the lowest layer that makes it nontrivial to spoof a sender.
Trust has been broadly effective, but also quite damaging at times. It's part of why I'm glad https is nigh universal - http's trust allowed stuff like Firesheep and all the hotel shit-injection that used to be so prevalent.
Is there a reason why these can't be filtered at the ISP level? eg. Cogent knows that Rascom shouldn't be announcing those prefixes, and refusing to route traffic to them unless there's manual verification?
I run a large production BGP network. With two exceptions every provider needed a Letter of Authorization from me to send to their upstream that authorized the announcement of my IP space (the exceptions being India where they wanted to charge extra for filtering announcements, and Russia where they offered to not do filtering for an extra charge).
This "manual verification" already takes place. It just doesn't apply to large transit providers interconnecting like what happened here
The reality is that RPKI Origin Validation solves this (but not as path spoofing, we need ASPA for that) and people need to publish valid ROAs.
Until routing vendors start shipping sane IRR based route filtering in default configs and make it hard to turn off, this is something we have to deal with.
Well-designed cryptography lets us tilt the odds violently on these human problems so as to give us greater confidence to take action when appropriate.
If you say maybe your technician typed 185.42.16.0 when they meant to type 158.42.61.0, that they uploaded last month's data instead of this month's, they wired a patch incorrectly - these are not implausible accidents. But cryptographically signed records don't happen by accident. You can't "typo" a 1-in-billions-of-billions chance.
I do agree with you that routing vendors have made this difficult to easily ingest prefix list data for easy import into inbound filters. There have been NANOG presos from last decade showing the performance hit when you try to do this at scale. It is much easier to enable RPKI OV than it is to build a robust IRR filtering system.
This is an interesting perspective. Do you have the same opinion of certificate authorities?
The real question is how many networks doing RPKI Origin Validation (OV) weren't accepting the ARIN TAL (ignoring ROAs from ARIN) due to their license to provide them indemnity. Remember that RPKI validators require a network to manually add ARINs TAL.
https://www.ripe.net/support/service-announcements/accidenta...
Last week a large BGP routing leak (>20k prefixes) occurred in Russia briefly impacted internet traffic around the world.
See the interactive 3D visualization of the incident here: https://map.internetintel.oracle.com/leaks#/id/20764_12389_1...
It would literally be more useful if you just screenshot it and posted that, as at least it would last longer, not to mention add detail.
> Probably won't be able to read the tweet in the future when it gets deleted, no real context or other information in the tweet. I find the tweet helpful. I didn't know about BGPmon, for instance.
Screenshots, on the other hand... They're not accessible, and they don't provide easy links back to the author, nor access to the tweet's reply context. I find all of those valuable.
The problem is that these announcements are made by third parties, and accepted by fourth parties, both of which Facebook has no control over.
While it's possible for Facebook to implement some controls over what routes THEY accept and therefor what routes Facebook sends traffic TO - it cannot control where others send traffic. If a third party advertises Facebooks routes to a fourth party, and they accept them, Facebook cannot do anything directly about that. And then those fourth parties accepting the routes will send traffic TO Facebook via the third party.
Disclosure: I used to work at WhatsApp, including while it was part of Facebook. Everything in this message is armchair routing policy though.
When are people going to wake the f* up....