The blog post in general was good/informative, but I gotta say, this quote does reduce my confidence in Fly quite a lot:
> To get the merit badge, we also had to document an approval process that ensured changes that hit production were reviewed by another developer.
> This isn’t something we were doing prior to SOC2. We have components that are effectively teams-of-one; getting reviews prior to merging changes for those components would be a drag. But our auditors cared a lot about unsupervised PRs hitting production.
> We asked peers who had done their own SOC2 and stole their answer: post-facto reviews. We do regular reviews on large components, like the Rust fly-proxy that powers our Anycast network and the Go flyd that drives Fly machines. But smaller projects like our private DNS server, and out-of-process changes like urgent bug fixes, can get merged unreviewed (by a subset of authorized developers). We run a Github bot that flags these PRs automatically and hold a weekly meeting where we review the PRs.
Letting code go straight to prod without a review is just IMO a really bad practice. It sounds like they've improved significantly here, but still have plenty of gaps where people can ship to prod with no code review until it's been running in prod for up to a week. This isn't just about stopping bad actors, it's 99% about preventing all sorts of bugs/mistakes/bad ideas from hitting prod. Obviously automated tests are the main line of defence there, but code reviews are very important too. I'm kind of shocked that they want to skip out on this. I personally wouldn't want to rely on a core piece of infrastructure from a team practicing this level of cowboy coding.
In this context, prod means all the things you'd expect, plus: internal admin app, blog, static marketing site, old Rails app no one has touched in 2 years that one customer still needs, bash scripts to diagnose host issues, etc. There's a reasonable scope for "PR reviews are good", and it does not extend across everything SOC2 covers.
That's because SOC2 is only concerned about vectors for exploiting code, and gives very few shits about how well the platform actually works. The policy had to cover the full scope, though.
This is the difference between a "policy" and a "practice". We've long been doing code reviews on critical code, even last year when there were only 7 people here. And we've long had a release process meant to minimize the risk of bugs harming users.
We don't let most code go straight to prod without review:
* Post-facto review is a norm only on a couple oddball projects
* Out-of-process PR merges are a privilege for only a subset of developers
* We do the same in-depth code review most teams I've worked with in the last 10 years do. If you're rolling out a feature, teammates are reviewing your code.
Take a typical, clueful engineering team and SOC2 it sometime, and you'll see the difference between "not allowing cowboy coding culture" and "satisfying the letter of the SOC2 code review controls".
> To get the merit badge, we also had to document an approval process that ensured changes that hit production were reviewed by another developer.
> This isn’t something we were doing prior to SOC2. We have components that are effectively teams-of-one; getting reviews prior to merging changes for those components would be a drag. But our auditors cared a lot about unsupervised PRs hitting production.
> We asked peers who had done their own SOC2 and stole their answer: post-facto reviews. We do regular reviews on large components, like the Rust fly-proxy that powers our Anycast network and the Go flyd that drives Fly machines. But smaller projects like our private DNS server, and out-of-process changes like urgent bug fixes, can get merged unreviewed (by a subset of authorized developers). We run a Github bot that flags these PRs automatically and hold a weekly meeting where we review the PRs.
Letting code go straight to prod without a review is just IMO a really bad practice. It sounds like they've improved significantly here, but still have plenty of gaps where people can ship to prod with no code review until it's been running in prod for up to a week. This isn't just about stopping bad actors, it's 99% about preventing all sorts of bugs/mistakes/bad ideas from hitting prod. Obviously automated tests are the main line of defence there, but code reviews are very important too. I'm kind of shocked that they want to skip out on this. I personally wouldn't want to rely on a core piece of infrastructure from a team practicing this level of cowboy coding.