I'm not able to ssh into any of my boxes or access any of the sites, yet my status monitor isn't showing downtime. Spotify is also down for me, which is another GCP customer.
I'm in Los Angeles but the servers are hosted on us-central1
Hi all - Seth from Google here. Our team is aware and we are working on mitigation. In short, a third party telco provider is advertising on one of our IP blocks. Unfortunately that's all the information I can share at this time.
EDIT: This is a general statement, I am not complaining to google here.
This kind of thing should not be possible. Are there any protocol proposals or other kind of upgrades to the routing protocols that would prevent these kind of mistakes/attacks?
Check out BGPSec and RPKI - they should prevent issues like this one. They're not widely implemented/enforced. Maybe it's going to change though now that it looks like we've got a "misconfiguration" somewhere every month or so.
On the other hand it's not that fragile everywhere and for everyone. When ISP markets are not monopolized and the service doesn't rely on a big cloud - much fewer users will get rerouted through random countries and the service itself can failover to properly working datacenters, tolerating all those BGP misdesigns.
It's if the internet doesn't like all that centralization with all that market domination. It's naturally resilient only when there is a lot of competition.
There are a lot of proposals but the problem is quite difficult to begin with and also involves centralizing policy. And then the chosen protocol has to be implemented by parties that tend to move at a glacial pace.
Yes there is an approach out there, solving many problems of the internet at once. It‘s called SCION and is being used in production at large swiss banks today.
Thanks for the info. We have reports showing it started a bit earlier than that, but every piece of information is helping in managing an incident. I'll make sure the team is aware.
Google IPs seem to be being routed to China for us.
We have servers in San Jose that cannot access Google services. Trace route shows everything going to China when leaving the San Jose data center. We can access the same services from Vancouver just fine.
How many times does this have to happen before China's privileges to do things like this get revoked? At this point, it can't be just a mistake and must be some state-sponsored hacking. Seems like a great way to find out where a particular Spotify user's IP address is.
> China's privileges to do things like this get revoked
By revoking China's privileges, you reinventing the Great Firewall. Or at least part of how it works, by "revoking routing privilege of selected IP ranges."
You know what, that's the narrative of China's wall building proposal in the place. To end USA's "Internet supremacy" and to advocate "Internet sovereignty".
IIRC, Google can try to dictate a BGP policy that says not to accept any routes that goes through China. However, without any verification checks (via cryptography), an entity can lie about the path that they are advertising.
Despite the subdomain, the IP for ChinaTelecom-gw.transtelecom.net (217.150.59.249) seems to be based in Russia, as does the carrier: https://en.wikipedia.org/wiki/TransTelekom
Seems likely to be TT's gateway to CT. New theory: TransTelecom brought up a new gateway to ChinaTelecom, which incorrectly gossiped all advertisements from ChinaTelecom. This caused a leak, since CT has bgp highjacking of Google IP ranges for the GFW within China, but ordinarily doesn't leak them outside the country. TransTelecom misconfigured the gateway to broadcast everything advertised by ChinaTelecom, bringing external traffic into the GFW.
Reading through the comments here I'm recognizing "China Telecom" from an article on a BGP hijack that was published about a week ago, I still had the article open in my browser:
I am on the East Coast, in Florida and seeing the same thing with traffic heading to China, lots of "chinatelecom-gw.transtelecom.net" in traceroutes I have never seen prior.
Definitely something interesting going on, and I am sure no shortage of some frantic research and effort to resolve this all at Google and such right now.
I'm not familiar with BGP routing attacks; the article above seems to imply the attacker needs to compromise certs in order to glean useful data from the attack.
If that's accurate, is this Google-oriented traffic vulnerable to this type of attack?
for Google traffic, assuming certificate pinning is in place, I can't see this being that successful.
However for more general traffic, well look at the trusted root list in your browser/OS. Realise that every single one of those trusted routes can issue certificates for a given domain...
This could just be a mistake of course, malicious intent isn't needed :)
Of the top of my head, assuming malicious intent, well not all browser (especially older ones) do certificate pinning, so perhaps then Chinese users of Google services using old browsers would find their traffic being intercepted?
Past that the leakage would seem fairly minor, a list of source IP addresses and destination hosts.
we manage services deployed in every GCE region, and our monitoring in London is reporting every GCE region having intermittent connectivity. no problems with our services in the other major clouds (we use basically all of them)
I hate to break it to everyone, but the technology to filter this sorta thing has existed for a very long time, but people often don't use it.
Most of the time this sort of thing is accidental (IE: operator error)so a lot of operators kinda ignore it.
Check out "IRR Power Tools" if you're interested.
Same thing here in San Diego. Traceroute to spotify.com going through LA, San Jose, NY, London, Amsterdam, Frankfurk, "mskn17ra-lo1.transtelecom.net", then ChinaTelecom-gw.transtelecom.net.
Our teams are continuing to work with upstream and downstream service providers to remedy the issue.
This kind of thing should not be possible. Are there any protocol proposals or other kind of upgrades to the routing protocols that would prevent these kind of mistakes/attacks?
It sounds like you're asking google to solve https://en.wikipedia.org/wiki/BGP_hijacking ?
It's if the internet doesn't like all that centralization with all that market domination. It's naturally resilient only when there is a lot of competition.
Getting the big ISPs/Telcos to adopt them... that's another matter
https://www.scion-architecture.net/
When reality conflicts with what you believe to be possible, it's time to reexamine your assumptions.
Edit: Didn't someone recently share a tool to monitor BGP hijack attempts?
[0]: https://www.cnet.com/news/how-pakistan-knocked-youtube-offli...
We have servers in San Jose that cannot access Google services. Trace route shows everything going to China when leaving the San Jose data center. We can access the same services from Vancouver just fine.
By revoking China's privileges, you reinventing the Great Firewall. Or at least part of how it works, by "revoking routing privilege of selected IP ranges."
You know what, that's the narrative of China's wall building proposal in the place. To end USA's "Internet supremacy" and to advocate "Internet sovereignty".
It needs to filter traffic to any address, and wouldn't have specific google ranges configured.
https://arstechnica.com/information-technology/2018/11/stran...
In another comment in this thread I read:
> Seems like its time to start or accelerate a working group on secure BGP.
Indeed things can't go on like this for much longer...
Definitely something interesting going on, and I am sure no shortage of some frantic research and effort to resolve this all at Google and such right now.
I'm not familiar with BGP routing attacks; the article above seems to imply the attacker needs to compromise certs in order to glean useful data from the attack.
If that's accurate, is this Google-oriented traffic vulnerable to this type of attack?
However for more general traffic, well look at the trusted root list in your browser/OS. Realise that every single one of those trusted routes can issue certificates for a given domain...
Of the top of my head, assuming malicious intent, well not all browser (especially older ones) do certificate pinning, so perhaps then Chinese users of Google services using old browsers would find their traffic being intercepted?
Past that the leakage would seem fairly minor, a list of source IP addresses and destination hosts.
https://news.ycombinator.com/item?id=18429099
Is our first time actually rolling over the entire stack to AWS - and it worked!
GCP outage currently is massive, can't even use other regions.
Edit: This also affected AWS Oregon region earlier. I do not know how yet, but they too were unreachable briefly. Seems to be okay now.
17 195.219.156.146 (195.219.156.146) 152.490 ms 152.423 ms *
18 * * mskn17ra-lo1.transtelecom.net (217.150.55.21) 198.658 ms
19 * * Google-gw.transtelecom.net (217.150.44.9) 192.230 ms
20 * * 108.170.250.111 (108.170.250.111) 172.086 ms
Edit: We're also in Los Angeles, connecting to us-central1. Seems to be a pattern?
EDIT: Some services are intermittently responsive. I had ~5 minutes of no access to anything. Some are slowly coming back.
Also in LA, had intermittent issues with google.com and Spotify all morning.
edit: linked to wrong issue