Part of taking on SOC 2 is also choosing whether you want your attitude to be "Let's do the minimum to get past the audit" and "Let's take this framework, and figure out where we can learn from it."
The post mentions background checks. On the one hand, I understand there's a real issue with these. On the other, if my PAAS isn't ensuring repeat offender fraudsters don't have access to sensitive data, that's possibly an area of concern. Hopefully the things they took from the other mentioned company do increase due diligence in vetting employees who have access to sensitive/regulated information.
Use it as a framework to actually think about BCP, DRP, etc, etc, and it won't be a total waste of time.
Edit:
Also adding I bring up background checks as an example of learning from vetted practices, rather than trying to attack the decisions of fly. I respect this article, especially where it's easy for people on the internet to criticize decisions, when the reality is security is a series of tradeoffs, and to function as a business means having imperfect processes.
> On the one hand, I understand there's a real issue with these.
My personal hate of these is how much information they require you to hand over to some random organisation.
Some make you take photos or videos of yourself holding your ID, and sure they say that they delete the information, but we all know that rarely happens.
I just don't trust their infosec policies, and it's only a matter of time before one of these companies gets reported as having a public S3 bucket, or employee laptop, or USB drive stuffed with video and ID scans and background check data.
Bingo. Most background check companies seem like literally the scummiest things in the world. I have no problem with a background check, but I don't trust these companies at all.
It would be a different story if people at companies of our ballpark size had stories about how background check results were informative, or mitigated some risk. But nobody has those stories.
It's funny how much fraud could have been prevented with simple background checks however. There's an entire podcast about financial fraud called "Oh My Fraud" about many instances where a simple background check might have prevented hundreds of thousands of dollars of fraud at charities. Ah well.
I understand the negative impact upon recidivism that background checks and stigma may have, but also we have to balance the interest of financial and information controls of organizations performing essential functions in society. Ideally speaking.
We definitely don't do the minimum! Our goal was to keep the scope manageable so the Type 2 certification goes smoothly. The side effect of this is that the things we _did_ put into scope are things we expect to do really well.
Preventing access to sensitive data is important. None of the top ten ways we try to solve that include "background checks", though.
You can run a SOC2 compliance program earnestly or as a check-the-box exercise.
If you're running earnestly, I would argue that the hardest thing about a SOC2 is ensuring that you stick to your guns on approaches that work for you and not adding cruft that you don't care about. If you let the latter happen, you will invariably end up a box-checker, and being a box-checker eventually contaminates a robust engineering / security culture.
And it's hard to walk back more restrictive / cumbersome policies; if you delegate your SOC2 to a person who doesn't deeply care, they'll eventually agree to put ClamAV on all the hosts or something (to make the auditors go away) and then you're going to be stuck with that for a while.
(So you need to find someone who has enough business context and good judgement to run the process, which is super painful from an opportunity cost perspective at a startup, and hard to locate at all at a larger company.)
> If you're running earnestly, I would argue that the hardest thing about a SOC2 is ensuring that you stick to your guns on approaches that work for you and not adding cruft that you don't care about. If you let the latter happen, you will invariably end up a box-checker, and being a box-checker eventually contaminates a robust engineering / security culture.
That's spot on, not only for SOC2 but for many, if not most, relevant certifications. The most important part is "not adding cruft". Nothing sucks like being stuck in a ISO9xxx certified process because you over-specified your processes even though you'd get the "ISO9xxx-certified" label for 10% of what you did. Suddenly you cannot react with common sense anymore because doing so would violate contracts you made with exactly those big customers you got the certification for in first place.
> if you delegate your SOC2 to a person who doesn't deeply care, they'll eventually agree to put ClamAV on all the hosts
Bingo. Just spell out the consequence: you no longer can optimize your compute costs by switching to a managed Kubernetes, because there is no ClamAV there.
I will have more to yell about this in 3 hours because I have a thing I want to yell about this. Free stickers if someone yells it for me before I’m done with this appointment.
I mean, they're pretty effective for determining whether people have convictions (caveats apply, depending on your jurisdiction) for that kind of stuff, right?
One of the problems with not doing background checks is ex post facto if you do have problems with an employee, and it turns out that you could have discovered them with a background check, then that can figure into your liability.
The main thing that makes me uncomfortable is that they will raise all sorts of unrelated things that can bias you or others in your company against a candidate, or even if you choose to ignore them, be brought up against you if it turns out you hired someone with a drug conviction or something unrelated to their work at your company.
I don't really give a shit if someone was arrested for a drug offense (or many other offenses), and knowing that information brings up all sorts of complications, to the degree that it outweighs the value of knowing relevant stuff (largely because genuinely relevant things from a background check are rare).
What you do is decide what you care about from a conviction perspective (say, crimes of dishonesty, serious violence etc). Then you write up a policy that says these are exclusionary. Then you only let HR run background checks immediately pre-contract, and you don't let anyone outside of the immediate background check team see any of that stuff. They're empowered to give you a yes/no to hire, but that is all.
Hiring managers should not be looking at BGC material.
Are they? How much will that differ between an employee from Peru, one from Canada, and one from Romania? How comparable will the data be, how much data will you even get in the first place?
And what kind of employee will you lose if you set such restrictions?
In some countries (e.g. US) people are convicted all the times for things they haven't done. Or have done but which is in the past and really irrelevant for the job.
On the other time people are not convicted for a lot of kinds of white collar fraud all the time, too.
Slightly different, but in a past life working in a field the company had extreme access to people's homes, we definitely weeded out some people who should not be given access to a strangers home using background checks.
Again, they're not ideal, and there's a large social concern of a permanent record like that, but you have a duty of care if your customers are trusting you.
You are arguing that background checks should be a requirement in order to have access to certain protected resources. While this is a fair argument, the counterargument is that most people in this particular software business shouldn't have access to user data/protected branches anyway. If someone needs elevated access, they would likely already have a significant pedigree at the company, and a background check may not add much value. In reality, most companies don't do background checks for security purposes; they do it to screen out candidates who aren't agreeable people, which raises ethical questions. I don't have an opinion on whether this is fair or unethical, but if security was the sole purpose, it would make more sense to background-check employees as a precondition to privileged access, not candidates as a precondition to employment.
I think they're intrusive and mostly unuseful, but I can at least see the rationale behind it at the kinds of companies that hire people en masse, or the kinds of jobs that people get bonded to do.
I would think it would be fairly obvious that a candidate could be “bought or bribed,” simply from the fact they’re asking for a job in the first place. They’re willing to exchange their time for money, i.e. “be bought.”
So why do people not commit corporate espionage? Well, it might have more to do with character traits than financial stability. In fact, most spies probably have their life fairly well together, and will have perfect credit. As for any asset they might compromise, what’s the difference between someone with poor credit applying to a job because they need the money, vs. applying to a job because they need more money from your enemy bribing them? I’d argue the difference comes down to character.
So for that reason, I’m skeptical of the effectiveness of a credit report as a proxy for likelihood to commit corporate espionage. A good credit report doesn’t seem to offer meaningful signal in either case of a malicious attacker or a desperate contributor. A bad credit report produces as many false positives as a good one.
This is what Schneier calls a "movie-plot threat". Instead of imagining a complicated narrative that connects a poor credit score with episodes of control fraud, improve internal controls so that individual contributors don't have the ability to steal from customers. This would be safer, and more considerate of your colleagues.
I'm not sure I follow what you're trying to say here, so don't take any of this personally, because maybe I'm misreading you.
Telling any startup engineering team that they should maximize their SOC2 Type I audit is borderline malpractice. Every engineering decision you make to support a Type I is something you'll need to live with in your Type II. That might just be an irritating own-goal if it's you doing the Type II, and you have to waste a couple hours on the phone explaining to your auditors why you've decided to remove ClamAV from all your servers. But it's could actually be destructive if it's someone other than you running the Type II, and gets cornered by your original Type I claims into supporting that bad decision.
Telling any startup engineering team that they should build a security practice informed by a SOC2 audit --- that they should take the COSO framework and figure out what they can learn from it --- might also approach malpractice. There are probably reasonable corpsec process controls you could build based on the AICPA Security TSCs. But the TSCs are not based on modern software engineering best practices and they aren't informed by modern software security. They are heavily influenced by the security concerns of medium-to-large sized enterprises with sprawling legacy IT footprints, and you can easily lose security by rolling out controls that aren't relevant to your environment but require you to add attack surface and operational overhead to mitigate vulnerabilities that you don't have because you don't share printers or run Windows Server 2008.
SOC2 is not security. Security is its own thing. Good compliance work is a byproduct of sound security engineering. It does not work the other way around.
Naturally, it's easy and gratifying to point out that a SOC2 report doesn't make a company secure. I don't think we could have come up with a clearer way to have said that, or, for that matter, to explain that we did our best to minimize the impact SOC2 had on our engineering practices.
As for background checks: once again, we can't background check a decent-sized fraction of our team, because they're in jurisdictions that (very reasonably) forbid employers from running intrusive background checks. We considered just background checking the unfortunate US team members we have that can be made subject to them. I had a fairly long conversation in a Slack channel full of secops people from about a dozen security companies, and none of them told me a single story about how background checks (which are, as it turns out, performative, superficial, and error-prone) was a win for them. I did get stories about how they were problematic: for instance, I did not make up the thing about high school transcripts.
So, long story short: because of our workforce, our platform needs to be resilient against bad hires. You don't get that from background checks; you get it from security engineering, tight access control, tightly designed roles informed by those access control decisions, regularly reviewed internal audits, detailed audit logs, and sound hiring practices. SOC2 covers most of this stuff only superficially. Security engineering is the real work, at least for technology startups.
Later
This is way angrier than I want this to come across. I promise, it's not personal, and if I'm caricaturing any point you made, I apologize in advance. I had a 6-hour tattoo session that ended with an hour of ditch work and I am in a fucking _mood_.
Also, since I'm predictably being pulled in the direction of repeatedly dunking on SOC2 in this thread, I want to say very clearly that I had a fantastic experience with our auditors, who were more clueful than any other auditor I've worked with. Our auditors are great. Don't flunk us!
Thanks for the detailed response. I definitely don't take it angrily. One additional point of context is I managed a Series B SAAS companies first SOC 2 (and its renewal) in the recent past, so I definitely understand what you're saying, and I think it lines up with the point I was trying to make.
My main point is you can either treat the SOC 2 as an adversary to overcome, or actually try and leverage it to be better. No matter what it's going to suck and be annoying, though Vanta/Drata can help. But one can leverage it to be a better company.
A less controversial example than background checks is infra cost monitoring. Where a lot of SOC 2 is focused on business continuity, in addition to security, one of the things required is that you're actually paying attention to your costs. A lot of cash flushed VC backed startups don't. So, once SOC 2 hits, the company that's treating it adversarially will just rubber stamp some quarterly meeting where they "look" at infra costs. Or, you can actually take that moment to level up the company to have macro review of the cost of goods sold, an ensure the business is on a healthy path.
Again, not a comment on your article, one of the big takeaways for me in running a security program for years was a general anxiety around being transparent on the program externally, because there's a certain type of "security" person who gets off on picking apart policies, without understanding tradeoffs that we were careful to make sure kept the company safe, while letting it function smoothly.
Coming out with an article like this is a great thing to do, where a lot of content out there is just "we got our SOC 2, and now we're prefect."
I won't comment on the background check thing again, not the least because I don't want to argue more for something I don't like, just think may be a needed evil.
> you can easily lose security by rolling out controls that aren't relevant to your environment but require you to add attack surface and operational overhead to mitigate vulnerabilities that you don't have because you don't share printers or run Windows Server 2008.
Can you talk about that more concretely? I'm sure there are many cases where compliance requires you to consider a risk that's not relevant, but it's hard to imagine that thinking about each risk and taking appropriate action (which, sure, in many cases will just be documenting why it doesn't apply) would damage one's security.
At the end of the day any list of potential risks is going to be imperfect, and some will be more imperfect than others, but I'd still think that the vast majority of the time you'll get a better outcome by engaging seriously with a given list of risk factors than by treating it as an exercise in doing the minimum. If you were trying to do security from the ground up without any regard to compliance you'd ultimately end up doing something quite similar - coming up with a big set of risks and then figuring out what you're doing about each of them - and sure, you could probably start with a better-targetted one than what SOC2 has, but that sounds like a matter of degree rather than being so different that you can't get any shared use out of doing those things together.
I'm sure they'll add their own responses, but I've seen some tremendous issues caused by legacies like this. I spent a solid week including a case opened with Microsoft attempting to determine why Outlooks "report as phishing" button didn't work. That's definitely harming security, people stopped reporting phishing. The root cause was a Windows XP era Internet Explorer hardening policy that served no purpose on a modern desktop.
In 2019, Microsoft removed a recommendation for an old font related security setting[0].
From a management level, a wide variety of modern best practices are already the default from Windows 2019 or so. The cognitive effort of looking at 50 security settings and convincing yourself they are all reasonable and won't break things is substantively better than the 400 or so we used to have. It's one thing to inherit this sort of legacy but it's a much worse thing to be implementing all these policies in a greenfield 2022 environment because they are all on some checklist.
So combine that with “post-facto” code reviews on a weekly cadence; there is potentially 7 day window during which a bad faith employee could act unrestrained?
Certainly this is giving me pause on using your platform for anything other hobby projects
I agree. You're a fintech startup deployed on Fly.io right now?
The best way to get detailed information about how our security practice works at Fly.io is to ask us about it directly. We're trying to be up-front about how weak SOC2, for everything else it might be good for, is with respect to security. Unfortunately, in the process of speaking plainly about SOC2, we have apparently sent the message that we think most of security is performative, which is not remotely true; the point is that we don't think SOC2 is an especially meaningful representation of the work.
I personally dont consider soc2 or similar certifications a good framework bc of its checklist nature. A lot of those items end up being orthogonal or sometimes even detrimental to actual security.
Using your example of background checks it’s probably more valuable to have proper acls and audit trail internally than doing background checks which is a really low signal compared to the level of hassle
It's fine for what it claims to be. It's an actual audit, in the accounting sense, not a detailed investigation of your security engineering practice. People are very hung up on this, and I get the urge to jump into SOC2 conversations to point out that SOC2 isn't a passing grade on security engineering. But your SOC2 auditors are up-front about what they're doing. There are management practices that can be verified by retrospective paperwork audits: from a random sample of the people you off-boarded in the last 12 months, did you reliably terminate access within N hours of severing their employment? SOC2 is fine for that. Do you have a security policy that puts employees on notice of their personal obligations with respect to data security, and did a random sample of your employees sign it? SOC2 FTW.
There's real value in being forced through this stuff, because these kinds of management processes are a real weak point at a lot of shops with otherwise strong security engineering. I'm glad that our policies and processes are clarified, and that there's an external process that keeps us honest and forces us to do the routine scheduled meetings, rather than keeping stuff in our heads. We started doing SOC2 prep work a year ago, and even before the audit, we were better than we were before we started.
But it is what it is. The thing that drives me nuts is when people suggest that good teams will maximize SOC2 so their security engineering can be informed by it. Yikes. No.
> Using your example of background checks it’s probably more valuable to have proper acls and audit trail internally than doing background checks which is a really low signal compared to the level of hassle
I agree with your idea, but background checks are a poor example. They're negligable cost, always outsourced, and trivial to perform. They're worth doing if only to validate that your candidate said the same things as the background check says (if they say they're not a felon and they are, that's a red flag -- if they admit to it and explain why, you're not being lied to). In contrast, you actually need to spend time working on audit trails and stuff. One is hiring a vendor and checking a box, one is probably engineering work.
What's actual security? Looking for zero days? Malware research? Continuous red team?
I think at the end of the day, SOC 2 aims to instill a basic level of organizational security so the company doesn't shoot itself in the foot. If a company can't genuinely follow a basic set of SOC 2 controls, can I trust them to do actual security?
Also, badly written checklists might be bad, but not all checklist are bad. Pilots use them. Doctors use them. Mechanics use them. In fact, most fields that involve critical life or death operations use them. Why? Because humans have a limited memory and tends to miss critical tasks all the time.
Genuine question: is consulting for compliance exclusively the possibility for big four type firms,or can't we do this at (traditional not scale up) start-up style? I'd buy your pre money common stock off your comment alone and the similar applies to numerous others I keep reading here.
A little delayed, but there are plenty of companies doing compliance consulting. Eden Data is a small shop I worked with briefly. If you wanted to talk more, my emails in my bio.
Glad to see this at the top. While a lot of folks will groan when it comes to the minutia that are necessary to do a compliance audit, there can be real value in them if you take them seriously. I think of it like this:
1. There are a set of things you need to do for "real" security
2. There are a set of boxes you need to check to pass a compliance audit
I think SOC 2 is pretty reasonable in that, if you're taking it with the right mindset, there is a large, large amount of overlap between #1 and #2.
The post mentions background checks. On the one hand, I understand there's a real issue with these. On the other, if my PAAS isn't ensuring repeat offender fraudsters don't have access to sensitive data, that's possibly an area of concern. Hopefully the things they took from the other mentioned company do increase due diligence in vetting employees who have access to sensitive/regulated information.
Use it as a framework to actually think about BCP, DRP, etc, etc, and it won't be a total waste of time.
Edit: Also adding I bring up background checks as an example of learning from vetted practices, rather than trying to attack the decisions of fly. I respect this article, especially where it's easy for people on the internet to criticize decisions, when the reality is security is a series of tradeoffs, and to function as a business means having imperfect processes.