What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements.
> On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customary step of writing to the Board through the corporate secretary.
Why is that customary? I have not come across it, and though I have seen situations of some concern in the past, I previously had little experience with US corporate norms. What is normal here for such a level of concern?
More, why is this public not a court case for wrongful termination?
Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?
>Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?
IME, yes.
I'm currently working as an SRE supporting a large environment across AWS, Azure, and GCP. In terms of issues or incidents we deal with that are directly caused by cloud provider problems, I'd estimate that 80-90% come from Azure. And we're _really_ not doing anything that complicated in terms of cloud infrastructure; just VMs, load balancers, some blob storage, some k8s clusters.
Stuff on Azure just breaks constantly, and when it does break it's very obvious that Azure:
1. Does not know when they're having problems (it can take weeks/months for Azure to admit they had an outage that impacted us)
2. Does not know why they had problems (RCAs we're given are basically just "something broke")
3. Does not care that they had problems
Everyone I work with who interacts with Azure at all absolutely loathes it.
But doesn’t this experience contradict what OP is saying in a way. If azure is always breaking wouldn’t that imply that changes like “adding smart pointers” are being introduced into the codebase?
I don't think it contradicts the OP. OP says the system is unreliable. Memory leaks that lead to out of memory failures for example. Smart pointers would stabilize things. (Also note that OP says their smart pointers PR was rejected).
That's a generalized statement. Smart pointers can stabilize things, if used wrongly they can cause just as many issues. Sprinkling in smart pointers such that there is now mixed use with smart and raw pointers can cause double frees, and huge maintenance issues. So, creating a single PR to introduce smart pointers in my opinion is not necessarily "stability". He should have created an architecture plan and got upstream and downstream aligned.
As a former MSFTy it does sound weird to me too. I didn’t see what Axels level was but a lot of people work for Microsoft and not many of them can expect to email the CEO and get a response. It seems a bit like a crash out, not the first I’ve seen levied at Azure, won’t be the last. They probably think it’s a mental health episode, if you’re an important CEO crazy people will email you all the time and the staff probably filter them out before they see it. Also this is a lot of internal gossip, I would be worried that airing this publicly would impinge on future career opportunities, even healthy orgs would appreciate some discretion.
I’m sure everything he said is completely true, Azure is one of the few tech stacks I refuse to work with and the predominant reason I left.
If you’ve joined an org and nothing works the reason is usually that the org is dysfunctional and there is often very little you can do about it, and you’re probably not the first person who’s tried and failed at it.
While Microsoft is hierarchical - but it did encourage reaching out in a "flat" manner internally.
In my experience - a loooong time ago ago now - executive leadership would participate in high-level escalations/critsits for large/key customers on calls. I was just a lowly field-engineer - but over the course of nearly 4-years, was on calls about 5 times with some of the big-names from that era that everyone knows about... And they seemed to emit enough empathy with the specific customer situation to move things forward.
However - being on the "other-side-of-the-fence" (i.e. external, consulting with Microsoft customers - some of them who even spend $1.5billion/year in M365/Azure licensing) and assisting clients with issues and remediations for the last 10-years, things are no longer the same. No amount of escalation gets further than occasionally reaching some level of the product team - and it can take 8-12 months before that even occurs. Troubleshooting and deep-engineering support skills for cloud customers are typically non-existent, and the assigned resources seem to just wait until the issue resolves itself...
Never worked at a FAANG, but from what I read from their cultures I don't think a letter to the CEO from a senior engineer would go entirely unnoticed there. CEO's might receive crazy letters, but hopefully not regularly from their senior engineering staff..
putting aside that MS is too huge to even just know about the names of your senior engineers across the globe and that the mail might have gone directly to spam
there is still the issue that this might have been classified as "a crazy letter"
a lot of the article reminds me of people which might (or might not) have competency but insists they know better and are very stubborn and very bad and compromising on solutions. The subtext of the articles is not that far afar from "everyone does everything wrong, I know better, but no one listens to me". If you frame it like that it very much sounds like a "crazy" letter.
Strictly speaking it reminds me a lot about how Pirate Software spoke about various EA related topics. (Context: Pirate Software was a streamer and confidence man who got complemented up due to family connections and "confidently knew" everything better while having little skill or contributions and didn't know when to stop having a "confidently bad" opinion. Kinda sad ending given that he did motivate people to peruse their dream in game design and engage themself for animal protection.).
Or how I did do so in the past. Appearing very confident in your know-how ironically isn't always good.
And in case it's not clear: The writing reminding me of it and having patters of someone trying to create a maximally believable writing to make MS look bad doesn't mean that he behaves like that or that the writing is intended to be seen that way.
It's more about how we have a lot of "information" which all look very believable, but in the end miss means to both: Verify many of the named "facts". And, more importantly, judge the sentiment/implicit conveyed information.
Especially if we just take the mentioned "facts" without the implicit messages and ignore the him<->management communication issues I would guess a lot of that is true.
Microsoft has a large PR department to put out such false impressions. The culture has changed, AFAIK you used to be able to email Bill Gates and be fairly confident he would read it, but you better be sure it was worth reading or he would fire you. Now they’re unlikely to fire you but they’re unlikely to read it either.
Senior leadership seems to be more far sequestered now, a bit like Trump, surrounded by lackeys giving them an entirely false impression of the world. That’s how they could legitimately believe they were going to bury the IPhone.
A "Senior Software Engineer" at Microsoft is someone with a pulse and 3 years of experience (due to title inflation); so despite the "senior" in the title definitely not "senior engineering staff".
I like how caring about fiduciary responsibility is a mental health episode or personality disorder to enough people in the comments. Simply being employed gives you a vested interest in keeping an operation above board and healthy. If you have a stock plan, you have equal rights to comment on issues as some low IQ private equity chief that does an end run to manipulate a company for their own benefit. The cattle psychology of most IT workers and mid level managers never ceases to amaze me.
It is emblematic of a crash out, I’ve seen a lot of them, and I’ve thought about doing it myself. I understand the impulse first hand. I did quit over my misgivings but I did not write a blog post about it, that would have been a career ended for me. I am expected to keep secrets as part of my job.
I work for myself now, for less money, but I do get to build things to the quality level that I want.
Sure, it is a "crash out" in the sense that you have an infinitesimal chance of changing things unless the founder is still involved because the situation didn't happen in a vacuum. And yep, you are narrowing your path significantly... sycophants have no tolerance for hardliners. But the mental heath episode is the opposite, to keep reporting for duty once you know something is deleterious. A lot of white collar people are on psychotropic drugs or otherwise self-medicate with alcohol and addictions and it's not difficult to diagnose why. Congrats on going your own way.
In my experience Azure is full of consistency issues and race conditions. It's enough of an issue that I was talking about new OpenAI models becoming available via Bedrock on AWS and how convenient that was since I wouldn't have to deal with Azure and my colleague in enterprise architecture went on an unprompted rant about these exact issues. It's not the first time something like this has happened and I've experienced these issues first hand, so yes. I'd say reliability is a critical issue for Azure and it hasn't gotten better each time I've gone back to check.
I recall seeing some pretty damning reports from a security pentester that was able to escape from a container on Azure and found the management controller for the service was years old with known critical unpatched vulnerabilities. Always been a bit sceptical of them since then
Large orgs make decisions that prioritize short-term metrics over long-term quality all the time and nobody tracks whether those tradeoffs actually paid off. The decision to ship fast and fix later sounds reasonable in a meeting setting until articles like this surface and the reality comes through clearly.
Wild guess, touching this with a 10-foot pole risks validating his claims. If they sue for breach of NDA, it means his claims are factually correct, and if they sue for libel and it goes to court, they may be forced to submit documents they don't want to.
Thanks for the direct reply! I wasn’t aware it was ever customary to write to a board.
But I do see you have very clear concerns.
One thing I don’t fully follow is: how did it get from such a nicely designed system, built by Dave Cutler, to this — simply moving fast and building tech debt?
> What are we reading here? These are extraordinary statements. Also with apparent credibility.
I left Microsoft in 2014. Already back then I could see this sort of stuff starting to happen.
The Office Org was mostly immune from it because they had a lot of lifers, people who had been working on the same code for decades and who thought through changes slowly.
But even by 2014 there were problems hiring developers who knew C++, or who wanted to learn it. COM? No way. One one team we literally had to draw straws once to determine who was going to learn how to write native code for Windows.
It wasn't even a talent thing, Windows development skills are a career dead end outside of Microsoft. They used to be a hot commodity, and Microsoft was able to hire the best of the best from industry. Now they have to train people up, and Microsoft doesn't offer any of the employment perks that they used to use to attract top talent (Seattle used to be a low CoL area, everyone had private offices, job stability).
When I started at Microsoft in 2007, the interview bar included deep knowledge of how computers worked. It wasn't unusual to have meetings drop down to talking about assembly code. Your first day after orientation was a bunch of computer parts and you were told to "figure out how to setup your box".
Antivirus wasn't mandatory. The logic was if you got a virus, they made a mistake hiring you and you deserved to be fired.
When your average developer can go that deep on any topic, you can generally leave engineers well enough alone and get good software.
> But even by 2014 there were problems hiring developers who knew C++, or who wanted to learn it. COM? No way.
It doesn't help that there are some teams that are hardcore in keeping things as they are and don't want any tooling that might improve COM development experience.
To this day Microsoft is yet to have any COM related tooling for C++ as easy to use as C++ Builder does it.
MFC, ATL, WRL, WIL,.... you name it.
The only time it seemed they finally got it, with C++/CX, there was a group that managed to kill this product, replace it with C++/WinRT, with no tooling other than the command line IDL compiler, now also abandoned as they refocused into windows-rs.
On the other hand there was e.g. CVE-2021-1647 where Microsoft's antivirus would compromise the PC with no user action.
(At least I think that's the one I'm thinking of. It's marked as a high-severity RCE with no user interaction but they don't give any details. There was definitely at least one CVE where Windows Defender compromised the system by unsafely scanning files with excessive privileges.)
“One team we literally had to draw straws once to determine who was going to learn how to write native code for Windows.”
Jesus, you have tons of people who are willing to do that, even now. Microsoft just don’t care to hire from non-target schools, or ordinary professionals and train them —- sure the reason is, people believe that you cannot improve mediocrity, which I don’t believe so.
On a completely different page, most of the generals and advisors and high level bureaucrats of the first Emperor of the Han dynasty came from exactly one county — the county of Pei. But in peaceful time they are just “ordinary people”.
Yeah I thought that was extreme. An engineer going to the board of any corporation let alone Microsoft is not normal or customary IME. That could explain why they got no response.
When you see significant risks to the org and its value, and they go completely unaddressed by management, the board is the final step before going to the public. It is the board’s duty to the public owners to make sure management isn’t driving the company into the ground.
It would be interesting to see this raised in the next shareholders meeting as a question of whether the board and exec team are actually competent and doing their work.
A man can dream anyway. When there is this much money on the line, sometimes people actually get held somewhat accountable.
It's a baffling flaw in human nature. The board should have cared about these issues, but in practice communications to and from the board are tightly controlled, and communications outside of those constraints are discarded.
This occurs whether or not it makes sense. Machiavelli actually warns about the specifically: if someone else controls access to you and communication with you, they have real leverage over you.
The CEO is accountable to the board. If they are derelict in their obligations to the company, that's where you need to raise a stink so they can fix it.
Well, yeah, that’s what a board does, but I think the issue is whether it is customary to go to the board directly in this situation. The answer is a resounding NO. Very odd, but cool idea and approach.
Maybe naive, but why not? If it's a serious enough issue, and you're not getting anywhere through your management chain all the way up to the CEO, why is it novel to contact the people the CEO reports to? They're not royalty, they're other human beings who also eat, piss and fart like everyone else.
Before 6 years of Google I’d co-sign what you said, but it never ever plays out that way.
The law of the jungle is an iron law, make people around you feel bad, be a tattletale, and you’re choosing to be ostracized.
That said yr interlocutor disturbs me a bit because yes, they certainly will make it out to be a mental health episode. But the implicit deal there is “STFU. You can even take paid health leave.” It’s not healthy either. BigCo is insane I’ll never work for one again without outrageous comp.
You’d be stunned by even the simplest story. Ex. a year in some crazy shit was going down and my manager asked for my thoughts on a topic, I was honest and basically said “I don’t think it’s a good idea, but in my experience, raising issues involving people only raises more issues.” He swore up and down it wouldn’t be a problem, eventually made a deal I could email it to him privately. Next 1:1 with my area lead was horrible, them seeing red, hearing a mistranslated version of what I said, and I had 0 warning.
I really hope this isn't an April Fool's. Skype was an amazing platform.
Then:
* Removal of peer to peer
* Windows UI redesign, away from classic usable Windows
* Mac UI also changed to web-style (which (obviously personal opinion) was a real kick in the face to people who use Macs and valued Mac UI. The UI wasn't flashy. It worked and it fit the system.)
* Emojis everywhere. I used to use it for work. I recall an eternally bouncing emoji for a while during every work call. Incredibly distracting.
* Covid's opportunity completely missed. Zoom and the rest took over.
* Oh, no-one's using Skype, I wonder why.
If they brought back Skype as a well designed, native UI-first, peer to peer, secure, EU-governed platform it stands a chance of outstanding success.
Don't forget about live speech translation. You could speak one language, the other guy a different language, yet you both heard your language all the time.
I've run into this, and I highly doubt I am one of the more extraordinary users. I have delays between working with it, don't have many running at once, am running on smaller codebases, etc. Yet just a few minutes ago I hit a quota. In the past I did far more work with it without running into the quota.
I emailed their support a few days ago with details, concerns, a link to the twitter thread from one of their employees, and a concrete support request, which had an AI agent ('Fin') tell me:
> While our Support team is unable to manually reset or work around usage limits, you can learn about best practices here. If you’ve hit a message limit, you’ll need to wait until the reset time, or you can consider purchasing an upgraded plan (if applicable).
I replied saying that was not an appropriate answer.
You're absolutely right re the lack of transparency and accountability. On one hand, Anthropic generates good will by appearing to have a more ethical stance then OpenAI, and a better product. On the other hand, they kill it fast through extremely poor treatment of their customers.
If they have a bug, they need to resolve it: and in the meantime refund quotas. 'Unable to' - that's shocking. This is simple and reasonable. It's basic customer service. I don't know if they realise the damage their attitude is doing.
Fin is the most useless thing ever. There's no obvious way to get reports in front of a human in a timely manner, and there's no clue to believe fin interactions are retained.
This does mean ultimately no loyalty. I can't stay loyal to a brand that doesn't actually respond to inquiries, bug reports or down reports at all.
I do understand that Anthropic is operating at a tremendous scale and can't have enough humans in the loop. This sounds like a good use for ai classification and triage, really!
> I can't stay loyal to a brand that doesn't actually respond to inquiries, bug reports or down reports at all.
Amen to this.
Being in business means having to respond to customer enquiries at some point.
Given the amount of billions being pumped into Anthropic's pockets and given the millions their senior-leadership no doubt pay themselves, I'm sure they could spare a bit of cash to get off their backsides and sort out the Customer Service.
I simply do not buy the "poor Antropic, they are operating at scale, they are too busy winning to deal with customer service" argument that comes up time and time again.
The fact is there are many large businesses, many large governments that are able to deal with customers "at scale".
Scale means you respond a bit slower, maybe a few days or at most a couple of weeks AT MOST. But complete silence for months or years is inexcusable.
All of my experiences with "Fin" matches that of my friends and colleagues .... namely that "Fin" is a synonym for "black hole". I've got "tickets" opened with "Fin" months ago that have not had a modicum of reply.
This is really cool, even better than the post, IMO (sorry OP!)
One question: when the second hand resets from 60->0, it visually jerks as the triangle moves. After the smooth movement, gradients, the cool multicolour fill, it feels very odd. Any way of smoothing that one out? Animating the flip back to zero? I do understand it's a one-way line not a circle...
> regulation actually does work when you have a competent government
This is the free market. Free as in, regulated to allow and encourage market entry and competition (as with replacement keyboards), not free as in unregulated. When you look back at when 'free market' was first strongly mentioned as a term, this is what it meant.
Late nineties is approaching thirty decades ago; if the C++ committee has now been working on this for nearly a decade, that's fifteen to twenty years of them not working on it. It's quite plausible that contracts simply weren't valued at the time.
Also, in my view the committee has been entertaining wider and wider language extensions. In 2016 there was a serious proposal for a graphics API based on (I think) Cairo. My own sense is that it's out of control and the language is just getting stuff added on because it can.
Contracts are great as a concept, and it's hard to separate the wild expanse of C++ from the truly useful subset of features.
There are several things proposed in the early days of C++ that arguably should be added.
I am not sure what the "truly useful features are" if you take into account that C++ goes from games to servers to embedded, audio, heterogeneous programming, some GUI frameworks, real-time systems (hard real-time) and some more.
I would say some of the features that are truly useful in some niches are les s imoortant in others and viceversa.
I run into bugs every day. It wakes, and has a black screen not wallpaper. Change spaces and the focus is wrong for half a second. Login screen is a pain because it collapses all users together. Notifications don’t scroll if they stop scrolling when the cursor is over a gap between them. Something on the system constantly eats disk space, and I think it’s the system updates. If I dock two apps in one space, sometimes one is black. If I zoom out to the Spaces overview it shows fine in the preview though. In the Terminal if I close a tab it can focus an entirely different window.
I could go on for hours. It’s a buggy mess these days and I miss Lion and Snow Leopard desperately.
Yes, these got a lot worse after Tahoe. The past few versions have all had issues on multiple machines.
None of this sounds like a hardware error. Something like notification scrolling is simple bad programming and bad QA. You scroll the list of them, but when the mouse cursor ends up on a gap between them, the new scroll event doesn't apply. They're all individual even though shown together.
Or a black screen on wake - that has the mouse cursor and login prompt, it just sometimes doesn't load the wallpaper or does it slowly. Not hardware - just something buggy. It's unbelievable when I compare to Leopard or whichever version it was introduced the rotating 'cube' of login screens, which always had wallpaper and loaded fast. Here we are fifteen years later with incredibly better hardware and the thing lags.
Nothing mentioned in the previous comment is indicative of a hardware problem. If you think I'm wrong, please describe a plausible mechanism to cause any of the problems described above. They all are plausibly software bugs. I mean, Apple hardware is not really any better than any other piece of fallible hardware, and their OS has been a buggy mess since Apple DOS. Most pieces of software as large as an OS are buggy in many ways, and Apple has not been proven to be the exception.
I'm curious how the starless versions are created. From the steps at the end, I couldn't see a 'this is how stars are removed' step. Maybe it's part of stacking (but most stars would remain present?) or the calibration process treating stars as noise?
Traditionally (pre-ai) you would use another image of the same part of the sky and negate the items that you want to remove from the image
As an example terrestrial telescope mirrors get dusty. You're not going to break down the scope just to clean up the dust as this is a many days operation in most cases. So instead you would take "flats" that were of a pure white background and thus showed the dust in its full, dusty, glory. When you take your actual images, you negate (subtract from the original image) the flat and thus any noise generated by the dust. You can use this same method for removing brighter stars from an image that would otherwise saturate the ccd and wash out the background. Turns out it doesn't work for planes. Ask me how I know!
> Traditionally (pre-ai) you would use another image of the same part of the sky and negate the items that you want to remove from the image.
I'm not an astrophotographer, so I'm interested about why that method would work for stars. Are not stars fixed in relation to the images taken? I could see how the technique would work with planets, maybe, but not stars.
Why does the technique not work with aircraft? Because they generally fly on fixed routes?
As the Earth rotates over the course of the night, the background stars and nebulae move as a single unit, no?
Maybe for some close stars parallax might work to remove them over the course of half a year. But no way could the Earth's rotation during a single night move background stars out of a nebulae.
Sure, but the nebulae also move along with the stars. The questions is how one can subtract the stars without also subtracting the nebulae. (I'm assuming different filters and/or a database of known star positions)
The ESA catalog is not precise enough to remove a star from an image of the structure of a nebulae - never mind Hipparcos. Filters while photographing and image processing in post are the way to go.
Don't forget that not only does the star need to be removed, but also the diffraction spikes. Those are internal reflections in the lens assembly - not mapped by any star catalog ))
So this part of the blog post is essentially false: "no generative AI of any kind"
I have yet to see a precise technical definition of what "generative AI" means, but StarXTerminator uses a neural network that generates new data to fill in the gaps where non-stellar objects are obscured by stars. And it advertises itself as "AI powered".
I don't consider photos I take on iPhone to be "AI generated" or even "AI augmented" even though iPhone uses neural networks and "AI" to do basic stuff like low light photography, blurring backgrounds, etc.
I agree that I wouldn't call these photos "AI generated", because the majority of what you're seeing is real.
But that's very different to saying that no generative AI was used at all in their production. "AI augmented" sounds pretty accurate to me.
Likewise, if someone posted a photo taken with their iPhone where they had used the built-in AI features to (for instance) remove people or objects, and then they claimed that no AI was involved, I would consider that misleading, even if the photo accurately depicts a real scene in other respects.
As a photographer and machine learning guy, I would call a lot of modern phone photos AI augmented. AI to stack photos or figure out what counts as the background is a little bit of a gray area, but an img-to-img CNN is about as close as you can get to full AI generation without a full GAN or diffusion model.
“StarNet is a neural network that can remove stars from images in one simple step leaving only the background. More technically, it is a convolutional residual net with encoder-decoder architecture and with L1, Adversarial and Perceptual losses.”
> So these are more artistic photo works than real science photos...
I disagree. If there are many flies around a statue, and I photograph the statue but remove the flies in the photo (via AI or any other technique), then I'm still producing an image of something that exists in the world - exactly as it appears in the world.
I agree that the claim "no generative AI used" is technically incorrect, but I do feel that the image does not contain any AI-hallucinated content and therefore is an accurate representation of reality. These structures appear in the image exactly as they exist in nature.
AI-related definitions aside, if it's a strictly subtractive/destructive tool that only removes light, it's hard to characterise as "generative" and arguably not much different to filtering frequencies!
It's not just "removing light", because if you removed all the light from stars, you would be left with black spots instead of white spots. The stars are bright enough to completely saturate a region of the image sensor. So there was actually no data recorded about what was in that particular part of the nebula or whatever.
The "generative" part is that the algorithm is filling in a plausible guess as to what would have been observed if there was no star "in the way".
I feel like the stars are probably pretty easy to mask out since they’re very bright relative to the rest of the image. Once you have the mask, each one is small enough that you could probably fill it with the values from adjacent pixels. Kinda like sensor mapping to hide dead pixels. That’s just a guess though, I’m sure there’s more to it than that.
Bright stars are so bright they literally mask areas of the sky. You'll probably need deconvolution algorithms (CLEAN being the standard some time ago, don't know whether some AI/deep-inv approach works nowadays...) to remove them.
There are several “AI” deconvolution tools to remove stars which work exceptionally well: two of the most popular ones being StarNet and RC-Astro’s StarXTerminator. I’m willing to bet that the author used the latter for star removal as it’s become something of a standard in the astrophotography world.
Hey thanks! I don't mean to hijack this great wine news with my own project, but since you asked, the top of the post has links to more. I will fix the link.
> On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customary step of writing to the Board through the corporate secretary.
Why is that customary? I have not come across it, and though I have seen situations of some concern in the past, I previously had little experience with US corporate norms. What is normal here for such a level of concern?
More, why is this public not a court case for wrongful termination?
Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?
reply