I'm not sure that I've seen like half the internet down before
21-Jun-22 3:17am
#1 metsfan718
At this very second like half of the internet is down. Sites like discord, Ring home security, door dash , MLB.com and much more.
I'm seeing that it's a cloud fare issue or something?
Kind of makes you think how fragile the internet is. Especially when the larger websites are hosted by so few companies.
But don't worry. The legendary GTZ will live forever!
Edit: starting to come back up. Took like 15-20 minutes. Went down a little past 3AM.
#1 metsfan718
At this very second like half of the internet is down. Sites like discord, Ring home security, door dash , MLB.com and much more.
I'm seeing that it's a cloud fare issue or something?
Kind of makes you think how fragile the internet is. Especially when the larger websites are hosted by so few companies.
But don't worry. The legendary GTZ will live forever!
Edit: starting to come back up. Took like 15-20 minutes. Went down a little past 3AM.
21-Jun-22 8:53am
#3 MrBean
Didn't know this happened until just now ... trying to find root cause, and cloudflare clearly isn't sharing what actually happened. Appears to have been related to some DNS issues.
#3 MrBean
Didn't know this happened until just now ... trying to find root cause, and cloudflare clearly isn't sharing what actually happened. Appears to have been related to some DNS issues.
21-Jun-22 9:14am
#4 longhornsk57
Probably someone pushed a bad BGP route into some core routers. Same thing that happened with Facebook a while back.
#4 longhornsk57
Probably someone pushed a bad BGP route into some core routers. Same thing that happened with Facebook a while back.
21-Jun-22 10:24am
#5 John
Cloudflare really needs to change their error message. People not using NGINX are like "What the heck is NGINX??" and people that happen to also use NGINX themselves are like "Is my NGINX failing??" Just change the message to say Cloudflare. Grrr...
#5 John
Cloudflare really needs to change their error message. People not using NGINX are like "What the heck is NGINX??" and people that happen to also use NGINX themselves are like "Is my NGINX failing??" Just change the message to say Cloudflare. Grrr...
21-Jun-22 10:38am
#6 longhornsk57
It's an automatic message from their system. Changing an error message on the fly for a whole set of servers is probably not a good use of time.
Anyone who even knows what NGINX is and is getting that error knows it's not on their end.
#6 longhornsk57
It's an automatic message from their system. Changing an error message on the fly for a whole set of servers is probably not a good use of time.
Anyone who even knows what NGINX is and is getting that error knows it's not on their end.
21-Jun-22 11:30am
#7 Anxiouz
I'm reading up on this, this morning. This sounds awfully similar to what happened a few months ago (during US business hours) which is concerning.
#7 Anxiouz
I'm reading up on this, this morning. This sounds awfully similar to what happened a few months ago (during US business hours) which is concerning.
21-Jun-22 12:26pm
#8 sinnie
Yeah it's happened before but this one most of us didn't notice as it was in the middle of the night.
#8 sinnie
Yeah it's happened before but this one most of us didn't notice as it was in the middle of the night.
21-Jun-22 1:24pm
#9 longhornsk57
At the end of the day the entire internet is just a fiber optic cables, some core routers, and BGP routes (a protocol that tells the routers where to send packets). It really is pretty frail.
#9 longhornsk57
At the end of the day the entire internet is just a fiber optic cables, some core routers, and BGP routes (a protocol that tells the routers where to send packets). It really is pretty frail.
21-Jun-22 1:56pm
#11 John
longhornsk57 wrote:
It's an automatic message from their system. Changing an error message on the fly for a whole set of servers is probably not a good use of time.
Yeah, the hundreds of CloudFlare servers are not individually maintained, I'm sure. They would push it out as a new update to all of their servers at once. And we can agree to disagree on if it is a good use of time. (Note that, on the Reddit SysAdmin sub, it is full of other IT people like me wishing that they would change this stupid message.)
Anyone who even knows what NGINX is and is getting that error knows it's not on their end.
That's simply not true. It looks just like the standard NGINX message -- which makes it difficult to tell, at first look, if the problem is a local NGINX issue or CloudFlare. That's literally why it came up on Reddit recently -- because an admin thought it was his local proxy at first.
#11 John
longhornsk57 wrote:
It's an automatic message from their system. Changing an error message on the fly for a whole set of servers is probably not a good use of time.
Yeah, the hundreds of CloudFlare servers are not individually maintained, I'm sure. They would push it out as a new update to all of their servers at once. And we can agree to disagree on if it is a good use of time. (Note that, on the Reddit SysAdmin sub, it is full of other IT people like me wishing that they would change this stupid message.)
Anyone who even knows what NGINX is and is getting that error knows it's not on their end.
That's simply not true. It looks just like the standard NGINX message -- which makes it difficult to tell, at first look, if the problem is a local NGINX issue or CloudFlare. That's literally why it came up on Reddit recently -- because an admin thought it was his local proxy at first.
21-Jun-22 2:03pm
#12 longhornsk57
Ya I mean my thing is if there's still an issue with routing or something preventing connectivity they can't push a global config update to change an error message that by now any sysadmin probably knows isn't a local issue
#12 longhornsk57
Ya I mean my thing is if there's still an issue with routing or something preventing connectivity they can't push a global config update to change an error message that by now any sysadmin probably knows isn't a local issue
21-Jun-22 10:47pm
#13 willyum
Didn't something like this happen before? I think it was an Amazon Web Sever at that time...
#13 willyum
Didn't something like this happen before? I think it was an Amazon Web Sever at that time...
22-Jun-22 2:36pm
#15 John
longhornsk57 wrote:
Ya I mean my thing is if there's still an issue with routing or something preventing connectivity they can't push a global config update to change an error message that by now any sysadmin probably knows isn't a local issue
I think the point is that -- when things are working -- they don't seem to update the "failure" message. It's not that people want them to change the message DURING the event. It is that they are tired of it coming up that way every time they have an event and think that they should change it when things are working -- like right now. So that next time it happens, SysAdmins don't have to go "Is this a local issue?"
#15 John
longhornsk57 wrote:
Ya I mean my thing is if there's still an issue with routing or something preventing connectivity they can't push a global config update to change an error message that by now any sysadmin probably knows isn't a local issue
I think the point is that -- when things are working -- they don't seem to update the "failure" message. It's not that people want them to change the message DURING the event. It is that they are tired of it coming up that way every time they have an event and think that they should change it when things are working -- like right now. So that next time it happens, SysAdmins don't have to go "Is this a local issue?"
22-Jun-22 2:37pm
#16 John
Reed wrote:
This is why we need a decentralized internet built on the blockchain.
When you speak, I'm always like "Is Reed making a joke? Or does he believe that..."
#16 John
Reed wrote:
This is why we need a decentralized internet built on the blockchain.
When you speak, I'm always like "Is Reed making a joke? Or does he believe that..."
22-Jun-22 8:36pm
#17 longhornsk57
John wrote:
longhornsk57 wrote:> Ya I mean my thing is if there's still an issue with routing or something preventing> connectivity they can't push a global config update to change an error message that> by now any sysadmin probably knows isn't a local issue I think the point is that -- when things are working -- they don't seem to update the "failure" message. It's not that people want them to change the message DURING the event. It is that they are tired of it coming up that way every time they have an event and think that they should change it when things are working -- like right now. So that next time it happens, SysAdmins don't have to go "Is this a local issue?"
Ya I can agree with that point, they should just change the error message altogether when they can.
#17 longhornsk57
John wrote:
longhornsk57 wrote:> Ya I mean my thing is if there's still an issue with routing or something preventing> connectivity they can't push a global config update to change an error message that> by now any sysadmin probably knows isn't a local issue I think the point is that -- when things are working -- they don't seem to update the "failure" message. It's not that people want them to change the message DURING the event. It is that they are tired of it coming up that way every time they have an event and think that they should change it when things are working -- like right now. So that next time it happens, SysAdmins don't have to go "Is this a local issue?"
Ya I can agree with that point, they should just change the error message altogether when they can.
23-Jun-22 7:50pm
#18 metsfan718
Verizon FIOS had an East Coast internet outage during the overnight and into the morning. That happened last night. Not sure what was up with that. They're usually super reliable.
#18 metsfan718
Verizon FIOS had an East Coast internet outage during the overnight and into the morning. That happened last night. Not sure what was up with that. They're usually super reliable.
I'm not sure that I've seen like half the internet down before