It’s wild that these cloud providers were seen as a one-way stop to ensure reliability, only to make them a universal single point of failure.
It’s mostly a skill issue for services that go down when USE-1 has issues in AWS - if you actually know your shit, then you don’t get these kinds of issues.
Case in point: Netflix runs on AWS and experienced no issues during this thing.
And yes, it’s scary that so many high-profile companies are this bad at the thing they spend all day doing
Yeah, if you’re a major business and don’t have geographic redundancy for your service, you need to rework your BCDR plan.
But if everyone else is down too, you don’t look so bad 🧠
No one ever got fired for buying IBM.
I wouldn’t be so sure about that. The state government of Queensland, Australia just lifted a 12 year ban on IBM getting government contracts after a colossal fuck up.
It is still a logical argument, especially for smaller shops. I mean, you can (as self-hosters know) set up automatic backups, failover systems, and all that, but it takes significant time & resources. Redundant internet connectivity? Redundant power delivery? Spare capacity to handle a 10x demand spike? Those are big expenses for small, even mid-sized business. No one really cares if your dentist’s office is offline for a day, even if they have to cancel appointments because they can’t process payments or records.
Meanwhile, theoretically, reliability is such a core function of cloud providers that they should pay for experts’ experts and platinum standard infrastructure. It makes any problem they do have newsworthy.
I mean,it seems silly for orgs as big and internet-centric as Fortnite, Zoom, or forturne-500 bank to outsource their internet, and maybe this will be a lesson for them.
It’s also silly for the orgs to not have geographic redundancy.
They zigged when we all zagged.
Decentralisation has always been the answer.
sidekicks in '09. had so many users here affected.
never again.
yeah, so many things now use AWS in some way. So when AWS has a cold, the internet shivers
A single point of failure you pay them for.
That explains why my Matrix <-> Signal bridge was complaining about being disconnected.
The one that hits us in self hosted is https://auth.docker.io/
Yeah I ran into this as well. Wondered why it needs a call to auth for public container images in the first place.
You guys don’t selfhost a registry?
I hadn’t actually considered that before. What’s your preferred way to do that?
Oh god, that just 404s for me
Who wants to bet Amazon gave AI full access to their prod config and it screwed it up.
That’s a good theory haha
A bad day for Jeff Bezos is a good day for all of us
Yeah, was reading about it here too
Ring doorbells, Alexa, ahh… the joys of selfhosting.
Is there no way to check the doorbell video locally?
An Amazon employee misconfigures something and now your doorbell doesn’t work
I don’t have one (because of that point), so I don’t know…
Presumably the app and doorbell are hardcoded to go to an AWS URL (so it’s “easier” for consumers), but in theory the data’s all on your wifi.
For some reason I hear Gilfoyle pontificating about what he does
It makes me wish I was selfhosting more services, music & chat in particular. It wasn’t important enough to set up yet
Can recommend Jellyfin, I use it for both music and tv/movies. Not sure on the chat bit, there are so many option it could get a long list
I have Jellyfin, but I haven’t tried it with music. How does it compare to Navidrome?
For chat, I was thinking something super simple for the weird situations like this. Alternatively, Briar if you’re near the person you want to contact
Finamp as a music specialized client is really awesome. Just get the beta version as they are reworking it deeply and the stable one is not really updated (also app password make it easier to use OIDC sso plugin on jellyfin)
I moved from subsonic to jellyfin years ago, cuz subsonic didnt do video very well.
Jellyfin looks to do all the stuff Navidrome does, plus video in the same way
It takes 5-10 reloads to get an page from IMDB lol
OMG, IMDB too
They are an Amazon company, so it makes sense they’d be using AWS.
A fun game to play right now is to try to hit any of your regularly visited sites and see which ones are down. 😂
