Big systems worth their salt already do this as weighted uptime, considering request successes / total requests rather than wall clock uptime as internal SLOs. But these numbers aren't really ever published because it gives away information about your customer base.
"Failed customer interactions" - if you have a way to actually see requests before they hit your datacenter, e.g. some async third party client libraries.
On this topic, as a service operator, it's really nice when you also own your SDKs, and have client-side telemetry about failed requests. Gives you a much clearer picture of end-to-end reliability (at least for the subset of customers who opt-in to telemetry)
Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or that unofficial github status page repeatedly posted here that counts all github services together into one number.
While not a home-run server, the NTP system is a distributed service that receives 100 billion to trillions of requests per day, and it's running pretty smoothly - it's never gone down completely since it started in 1985. It's also very simple. The reason it has so many 9's uptime is because it is simple. Given a low amount of complexity, it's not unreasonable to think that an individual could run a >1B requests per day service.
Salesforce is not simple. It's wildly, overly complex. It's amazing it has any 9's at all and not 8's or 7's. Salesforce offers three 9's, which allows for 43 minutes downtime per month. The current outage is at 8 hours (and counting) so Salesforce is now at 98.9% uptime for the month - there's an "8" in there now. Not good, but considering the complexity of Salesforce, it's still kind of amazing.
It's the scaling nature of enterprise software. If you have a mature B2C app, you have millions of users. What each user wants isn't so important, so it's more of a take it or leave it experience. If you have an enterprise app, one big company can and does push you around to get their features in. And as you grow, you get a few hundred big companies that push you around. The result is this huge mess of features, and now you have to maintain this mess.
I remember Cisco before iOS used to have hundreds of branches for their router, one branch for each major customer that was demanding specific features. It was unmanageable, but that's what you needed to do to win those "enterprise customers".
It also turns out customers aren't very good at articulating their needs and putting them into a cohesive vision of the product. But they sure have specific demands to get stuff in. I'm not blaming the customer, this is just how this world works -- All of the "enterprise software" apps are extremely complex with hidden knobs and weird behavior that was pushed in by a customer twenty years ago all over the place.
> I remember Cisco before iOS used to have hundreds of branches for their router, one branch for each major customer that was demanding specific features. It was unmanageable
Ouch. I heard of a company in my home town (small B2B service provider) doing something similar - they paid well but I didn't think it was worth it.
I think it’s like advertising - 50% of my code is wildly over complicated - I just don’t know which 50%
But the GP is essentially correct - there is a 2% of salesforce that could be built run and keep 80% of salesforce users happy. Except that you could not charge enough to be able to advertise on F1 cars and take SVPs out to dinner.
So you could not actually make 80% of them happy - they would ever buy it.
> showstopping technical debt is unavoidable is very new
No it's not. The push and pull between shipping and paying down technical debt is as old as there's been software to sell. Sales has been selling features that don't exist quite yet ever since they've been talking to customers, and engineering has been pushing back on implementing them yesterday since there's been features to implement. Showstopping technical debt is merely a side effect of who wins that argument in a given org.
I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach.
Smaller is, in fact, better, because it's easier to operate reliably.
Is it? When your internet is out for five days because your ISP takes a few days to get to you, do you acknowledge that you're now at 98.5% availability for the year, far worse than any SaaS email service?
I think people forget that those large environments are there for a reason. To make sure the service stays up in the face of problems outside your own control.
In my entire adult lifetime (mid 40s), my ISP has never been out for five days. Compare to Github, Microsoft, Salesforce, and AWS outages that are always occurring in some fashion. Reddit is down constantly in various ways and still continues to operate as a business, public no less, so I disagree about the need to chase five nines and broadly speaking, large distributed systems that are potentially unnecessary for the use case and target outcome.
Consider yourself lucky that you’ve never been the victim of a fiber cut. But what about if the power to your house goes out? Or what if your server blows the power supply?
My entire point is that you have no redundancy in your system and you also aren’t big enough to have any pull with the vendors who can fix these types of outages so you’re basically at the mercy of your providers with no recourse.
That’s why these systems are built the way they are.
And generally four nines is considered the gold standard these days. I can tell you for sure that both Netflix and Ebay would lose money anytime they drop below four nines because I have at some point been responsible for both. You’re correct that Reddit has a lot more leeway and outage time before they start losing money but not that much leeway.
I've contributed to building out data centers, as well as managed colos for others, primarily in downtown Chicago at Level3 and at 350 E Cermak. I am familiar with architecture required for reliability and diversity, from power and fiber in all the way up the stack to the Kubernetes cluster and software defined networking. If you participate in the capital markets, your data traverses systems I've participated in designing and implementing. There is a time and place for complexity (in this context, large/global distributed systems), but too often, complexity exists where it need not (imho).
"What are you optimizing for?" is always an important question, as is "The Five Whys."
> That’s why these systems are built the way they are.
Built how? Because I can state with confidence that I have cleaned up a ton of failed upgrades/zombie terraform deploys of these serverless kubernetes wonders that followed every best practice under the sun, and these things are not really considered even moderately reliable (as designed by imperfect mortals under real world conditions), meanwhile professionally, people who stand to lose a lot of money should their systems go down generally operate systems whose architectures were designed decades ago, are generally horizontally scaled monoliths, and are extremely conservative in software choice.
Also downtime often is no biggie, as long as it's planned and or don't lose (too much) customer critical data.
Like nodobody cares if your test db cluster goes down for the weekend. We even shut down our db instances to save money.
None of that is an argument against small being more reliable. Rather it's an argument that the smaller you are the more important being distributed becomes when managing mundane day to day failures.
Even if a solar flare takes out an entire continent or two I think it's safe to say that the bittorrent network will still be running in some form. Can you be so certain about any given SaaS product?
I've dealt with a fiber cut, it wasn't nearly that bad. Fiber cuts impacting my SaaS providers were worse because there was nothing I could do about it.
I mean, if you really need redundancy, isn’t a second instance on a VPS somewhere that you manually switch over to, enough?
Over multiple ISPs, so far internet outages for more then a few minutes is very rare (though the few minutes would make me not want to host something requiring high availability; and a cut cable is really annoying because there simply is no quick fix), power outages even rares, I experienced 3 in 40 years, and the longest was 6 hours.
Ignoring for now how you are synchronizing the database and filesystem, and how doing so may well result in your duplicate experiencing the same failure as the original, you can maybe recover from a small class of availability issues that could knock you out of an SLA.
But that assumes you can get online and can fully orchestrate the transition within less than 53 minutes of it starting. Including the time you took to become aware of it. And including the time to diagnose and decide that a switchover would resolve the problem. Including the time it takes for DNS caches to expire and point to the new host. Including the DNS caches which may ignore your TTL. And including all these things again when you switch back.
And assuming, of course, that it doesn't happen again for a whole year.
Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale).
"Simplicity is the ultimate sophistication." -- Da Vinci
Hidden in this discussion around self-hosting reliability are other options as well.
Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted.
Keeping a database in sync between home and the instance might be simple or more complex depending on needs, but would it really be that hard to have Claude help you setup a replicating Postgres server? If your database (or data files) are 1 gigabyte and don't update that often... maybe just rsync it every night or something
There's a thread you and others are pulling on here, and we need to pull it. Hosting doesn't have to be the domain of the big vendors anymore.
The other thing people miss is change velocity. NTP server doesn't change much. Salesforce teams are under enormous pressure to constantly deliver new features and ship changes every day. 99.95+% with constant change, millions of customers, 1e10-11 requests a day is challenging.
Well ackchually.. I get that large scale systems pose their own challenges on their own, but it also matters what's the smallest isolable unit.
What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale.
In contrast, something like a bank or social media isn't really reducible - every user needs to be able to interact with every other user in a consistent manner.
So running a midsize bank's backend which processes 10m transactions per day, might be as if not more complex (all consistent, repeatable, and must never fail), that having a product which is a 10-10k org's IT infra replicated a thousand times.
And yes, lots of people have worked at banks and other fintech companies of this scale, including me.
I am not an expert, as I never worked on the 'core' systems but I know folks who did, and everyone told me there's an arcane database monolith that sits at the heart of these, very expensive and exotic big box SW & HW (at least for us unwashed rubes used to EC2 instances)
> What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale.
This is only true if you exclude problems like “finding a CDN node from the device,” “managing congestion,” etc. as part of the problem statement
> What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale.
They do though! They mostly try to avoid it since hitting the network to serve any kind of latency would unacceptably increase latency, but you wildly underestimated the amount of complexity there is to running a CDN.
I probably underestimated the complexity and I didn't mean to knock on CDNs - I just wanted to say that not all distributed systems have equal complexity, and some require essentially almost serializable transactions, while others are fine with small channels of eventual consistency
Before the ad, you had no brand awareness at all of Liberty Mutual. Now, you know their name enough to call them out specifically in a Hacker News comment. Your propensity to buy has improved a lot just based on that fact, regardless of whether you consciously want to avoid them now.
Literally every single HN thread these days mentioning big companies the top comment is some cynical conspiratorial nonsense. Say Big Tech is out to get you in any way possible no matter how outlandish and people eat it up. Nothing can just happen for good or neutral reasons it's always tied to some master plan.
Just as a curiosity, have you ever worked in big tech yourself, spoken "off-the-record" with managers or executives at these places or know closely people who do/have worked there?
Yes, except these kinds of fears related to big tech/surveillance end up being true more often than not. Unsurprising given the profit incentives and the total lack of ethics in the industry.
10+ years ago you were considered a lunatic if you told people that putting internet-connected cameras and microphones everywhere is a bad idea because their TV would be used to spy on them, that their cameras would be accessed by the police without their consent for surveillance, or that car company employees would watch them walk around naked through the cameras on their car.
Is anyone tracking every prediction ever made in a spreadsheet that you could use to verify that the rate is literally >50%? No, not that I'm aware of.
The only anecdata I can give is "look around you", how all major (and many smaller) companies are implementing invasive spying tech into their products, harvesting our personal information, selling it, or abusing it any way they can. They sell the products as one thing then shaft their customers and change the terms overnight at their earliest convenience.
What you call "a conspiracy theory" is just "following clear market incentives". There's a clear motive for doing everything with a clear track record of doing so in the past.
If someone made a derivative financial product that roughly equated to the prediction "Big tech will use any means to expand their access, to invade and abuse our privacy, and generally be willing to do anything in pursuit of profit" I would be willing to invest my life savings into it.
Maybe we travel in different circles. I still hear about Covid vaccines having 5g chips in them and all sorts of other crazy stuff. Rough guess, maybe 10% of what I hear is plausible, and maybe a quarter of that is likely.
It is not conspiracy theory. Optimizing for profit (which includes not crossing regulators/governments) is a good enough explanation usually. Big (and most small) companies put profit above anything else, so this is usually the right angle to approach anything they do.
Hah, if anything does happen outside of expected conspiracy, it happens because two conspiracies just happenened to nullify each other's outcomes. Some of the people here even made some of the ridiculous choices that shape our lives today:D Just sayin'
This is probably why they are launching now. There's no real other competitor right now in the nicely made managed openclaw space for personal use (or at least, very very early. Maybe Grokbot can be considered the lead competitor but afaik it did not gain mindshare). Instinct is supposed to come out soon and by not being associated with Meta it'd be really appealing just on that front vs this. But being first to market is a real reason to use it and chance to gain lasting market share.
The best selling point is an end-to-end purchase flow that you don't have think about at all, it does the price comparisons for you. However, if you want to use it to just get the list of top candidates and compare and go make the final decision yourself I'm sure it can do that for you too. Either way I see a future where these claw-like agents can fit into everyday use cases.
Maybe... if run locally with full transparency into the prompts (including system etc) and independent benchmarks for the models (i.e. tests on how eager they are about selling me stuff)
But even in this hypothetical world I would still have a hard time teaching an LLM what I find important in a product. Not everything can be captured in easy to compare numbers. "Does the device creak under pressure?", "What's the one weirdly unergonomic thing about it you only notice after weeks of using?"... Stuff like that. Things which are easy to miss even if the model ingests all relevant reviews about the product, because it might not stand out, but might be important for me.
HN isn't really a good place to have an informed discussion on ads efficacy unfortunately. In this thread we have many many people talking about how online ads in general are completely worthless. All of the top comments I see are harping on things like Google Ads having bad targeting for reasons like you are getting recommendations on products you just bought (having experience in the ads industry, yes this is actually a good signal that you might buy another one, even if you don't want to believe it).
Meanwhile, the article is actually discussing ChatGPT ads specifically compared to Google Ads, which they do have success with.
Anyway, my guess is that it's a combination of 1. OpenAI being new and not having the massive work needed yet to set up a fully successful ads network, but they're recruiting a lot of talent from the area + recsys so it's probably not too far off. And 2. we don't really know in general yet how best to do in context ad recommendations. I think this will also improve pretty rapidly with how much talent is working in the space. The best ads will probably look a lot more like Google Search per-request targeted ads rather than Instagram style per-user targeted ads, but I believe we'll see lots of gains fast.
Regarding showing ads for something you just bought, I’ve always assumed the principle here is that because conversion is extremely unlikely for a given ad, the super unlikely case you’ll buy something twice is actually more likely than a random other thing.
Nah, I think this is a common misunderstanding of how LLMs work, where people think that they mimic the pre-training data. Stylistically everything you see is an artifact of post-training, which is from reinforcement learning not from absorbing mass amounts of text. At some point a person or more recently a bot gave a thumbs up to an A/B tested response including em-dashes and claudisms galore.
> Stylistically everything you see is an artifact of post-training,
It is still not exactly clear if it is true or not. Unless we have base "pt" snaphot of Claude we can't say one way or another. I've played a bit with base models of Nemo, Gemma etc and they all had tics, not much different from RLHFed instruct versions.
So question then, why is it so hard to make an ai that doesn’t do these things? And why do Claude and ChatGPT have the same -isms? They’re both doing the same a/b post training with the same decisions?
Yeah, but I understand that fingerprinting is essentially a pseudorandom overlay onto a pseudorandom base signal. And unless you have access to both the random number generators and the weights, I don't think you can detect it?
So "fingerprinting" operates on a totally different and basically invisible level, as opposed to the obvious stylistic patterns that the average programmer can identify in about 2 sentences.
Nothing is priced according to what they really cost. Things are priced at what the market will bear based on supply and demand. You might as well ask this question for oil, iphones, bitcoin, restaurants, or housing the answer is all the same.
That is the point though, if labs are maximizing svg image generation capabilities it is a very good thing. That's a general skill that is useful. So assuming they aren't specifically maximizing pelican bicycle svgs (and it doesn't look like they are) then incentives are aligned that the "benchmark" is measuring a general desirable capability.
https://cloud.google.com/blog/products/gcp/available-or-not-...
reply