I've just shifted my SWE infrastructure from AWS to Hetzner (literally in the last month). My current analysis looks like it will be about 15-20% of the cost - £240 vs 40-50 euros.
Expect a significant exit expense, though, especially if you are shifting large volumes of S3 data. That's been our biggest expense. I've moved this to Wasabi at about 8 euros a month (vs about $70-80 a month on S3), but I've paid transit fees of about $180 - and it was more expensive because I used DataSync.
Retrospectively, I should have just DIYed the transfer, but maybe others can benefit from my error...
Extremely useful information - unfortunately I just assumed this didn't apply to me because I am in the UK and not the EU. Another mistake, though given it's not huge amounts of money I will chalk it up to experience.
Hopefully someone else will benefit from this helpful advice.
I think this is partly true. Raising the necessary funds, hiring enough of the right people and become sufficiently visible to get "mindshare" are all important factors in building a successful business. It is a lot harder to do these things if your ideology is out of step with what is considered mainstream.
It's not uncommon. My cousin sent out a Christmas card announcing her divorce - I think it stops a lot of 1-1 conversations with people which can be quite draining when you're already pretty raw.
This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen.
Also, consider using fck-nat (https://fck-nat.dev/v1.3.0/) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges.
(Or, just run your own Debian nano instance that does the masquerading for you, which every old-school Linuxer should be able to do in their sleep.)
The reason to not include the endpoint by default is because VPCs should be secure by default. Everything is denied and unless you explicitly configure access to the Internet, it's unreachable. An attacker who manages to compromise a system in that VPC now has a means of data exfiltration in an otherwise air gapped set up.
It's annoying because this is by far the more uncommon case for a VPC, but I think it's the right way to structure, permissions and access in general. S3, the actual service, went the other way on this and has desperately been trying to reel it back for years.
Right, I can appreciate that argument - but then the right thing to do is to block S3 access from AWS VPCs until you have explicitly confirmed that you want to pay the big $$$$ to do so, or turn on the VPC endpoint.
A parallel to this is how SES handles permission to send emails. There are checks and hoops to jump through to ensure you can't send out spam. But somehow, letting DevOps folk shoot themselves in the foot (credit card) is ok.
What has been done is the monetary equivalent of "fail unsafe" => "succeed expensively"
s3 access is blocked from an EC2 by default unless you give the attached IAM role access to S3.
Then it is still blocked unless you add a NAT gateway or Internet gateway to the VPC and at a route to them.
If you are doing all of this via IAC, you have to take a lot of steps to make this happen. On the other hand, if I’m using an EC2 instance to run an ETL job from data stored on S3, I’m not putting that EC2 instance in a subnet with internet access in the first place. Why would I?
And no you don’t need internet access to access the EC2 instance ftom your computer even without a VPN. You use System Manager Session Manager.
I do the same with lambda - attach then to a VPC without internet access with the appropriate endpoints. Even if they are serving an API, they are still using an API gateway
There’s zero reason why AWS can’t pop up a warning if it detects this behavior though. It should clearly explain the implications to the end user. I mean EKS has all sorts of these warning flags it pops up on cluster health there’s really no reason why they can’t do the same here.
To be fair, while EKS warnings are useful, I've grown a habit to ignore them completely, since I've seen every single RDS cluster littered with "create a read replica please" and "enable performance insights" bs warnings.
The second someone doesn’t pay attention to that warning and suffers an exfiltration, like the cap1 s3 incident, it’s aws’ fault as far as the media is concerned.
I don't get your argument. If an ec2 needs access to an s3 resource, doesn't it need that role? Or otherwise, couldn't there be some global s3 URL filter that automagically routes same-region traffic appropriately if it is permitted?
My point is that, architecturally, is there ever in the history of AWS an example where a customer wants to pay for the transit of same-region traffic when a check box exists to say "do this for free"? Authorization and transit/path are separate concepts.
The EC2 needs credentials, but not necessarily a role. If someone is able to compromise an EC2 instance that has unrestricted S3 connectivity (no endpoint policies), they could use their own credentials to exfiltrate data to a bucket not associated with the account.
I'll have to dive in and take a look. I'm not arguing, but here is how I naively see it:
It seems there is a gap between "how things are" and "how things should be".
"Transiting the internet" vs. "Cost-free intra-region transit" is an entirely different question than "This EC2 has access to S3 bucket X" or "This EC2 does not have access to S3 bucket X".
Somewhere, somehow, that fact should be exposed in the design of the configuration of roles/permissions/etc. so that enabling cost-free intra-region S3 access does not implicitly affect security controls.
I agree. The real question is why do I need an "VPC endpoint" to save money in the first place?! us-east-1 EC2 isn't actually going over the internet to connect to us-east-1 S3, regardless or whether it's using a NAT gateway or VPC endpoint. AWS knows what routes are on its own network.
Changing defaults doesn't have to mean changing existing configurations. It can be the new default for newly created VPCs after a certain date, or for newly created accounts after a certain date.
And if there are any interoperability concerns, you offer an ability to opt-out with that (instead of opting in).
> Changing defaults doesn't have to mean changing existing configurations. It can be the new default for newly created VPCs after a certain date, or for newly created accounts after a certain date.
This is breaking existing IAAC configurations because they rely on the default. You will never see the change you're describing except in security-related scenarios
> There is precedent for all of this at AWS.
Any non-security IAAC default changes you can point to?
AWS is not going to enable S3 endpoints by default, and most of the thread is downvoting the correct explanations like thinking in terms of a small hobby VPC, not the architectures AWS actually has to support.
Why it should not be done:
1. It mutates routing.
Gateway Endpoints inject prefix-list routes into selected route tables. Many VPCs have dozens of RTs for segmentation, TGW attachments, inspection subnets, EKS-managed RTs, shared services, etc. Auto-editing them risks breaking zero-trust boundaries and traffic-inspection paths.
2. It breaks IAM / S3 policies.
Enterprises commonly rely on aws:sourceVpce, aws:SourceIp, Private Access Points, SCP conditions, and restrictive bucket policies. Auto-creating a VPCE would silently bypass or invalidate these controls.
3. It bypasses security boundaries.
A Gateway Endpoint forces S3 traffic to bypass NAT, firewalls, IDS/IPS, egress proxies, VPC Lattice policies, and other mandatory inspection layers. This is a hard violation for regulated workloads.
4. Many VPCs must not access S3 at all.
Air-gapped, regulated, OEM, partner-isolated, and inspection-only VPCs intentionally block S3. Auto-adding an endpoint would break designed isolation.
5. Private DNS changes behavior.
With Private DNS enabled, S3 hostname resolution is overridden to use the VPCE instead of the public S3 endpoint. This can break debugging assumptions, routing analysis, and certain cross-account access patterns.
6. AWS does not assume intent.
The VPC model is intentionally minimal. AWS does not auto-create IGWs, NATs, Interface Endpoints, or egress paths. Defaults must never rewrite user security boundaries.
These are all good arguments. Then do the opposite and block S3 access from VPCs by default. That would violate none of those.
“We have no idea what your intent is, so we’ll default to routing AWS-AWS traffic expensively” is way, way worse than forcing users to be explicit about their intent.
Minimal is a laudable goal - but if a footgun is the result then you violate the principle of least surprise.
I rather suspect the problem with issues like this is that they mainly catch the less experienced, who aren’t an AWS priority because they aren’t where the Big Money is.
Yes, this. You lock it into Terraform or some equivalent.
And ok, this is a mistake you will probably only make once - I know, because I too have made it on a much smaller scale, and thankfully in a cost-insensitive customer's account - but surely if you're an infrastructure provider you want to try to ensure that you are vigilantly removing footguns.
Especially true now with Claude generating decent terraform code. I was shocked how good it is at knowing AWS gotchas. It also debug connectivity issues almost automagically. While I hate how it writes code I love how it writes terraform.
AI is surprising good at boilerplate IaC stuff. It’s a great argument for configuration as code, or really just being able to represent things in plain text formats
If you are creating a VPC from the console that might be a reasonable default. But any serious implementation is going to be using IAC - like they were - and I would expect to spell out everything explicitly.
I personally prefer to just memorize the data and recite it really quickly on-demand.
Only half-joking. When something grossly underperforms, I do often legitimately just pull up calc.exe and compare the throughput to the number of employees we have × 8 kbit/sec [0], see who would win. It is uniquely depressing yet entertaining to see this outperform some applications.
[0] spherical cow type back of the envelope estimate, don't take it too seriously; assumes a very fast 200 wpm speech, 5 bytes per word, and everyone being able to independently progress
OPs experience working their way up the keyboard stack is very similar to my own. I settled on the Dygma Raise. I now own one of the Raise and Raise 2.
Yes, they are a lot of money. I don’t have time to game any more, and they clearly focus mainly on gamers. But if you’re a software / IT person your wrists are your livelihood, so for goodness sakes invest in them. There is no silver bullet and you will probably have to try a number of possible solutions if you suffer from wrist and forearm pain when working, but do not ignore it and take your workplace ergonomics seriously.
For younger engineers, learn to minimise your “travel” and learning editor shortcuts, terminal shortcuts and similar so that you can be smoothly productive with constantly shifting from mouse to keyboard and back again. And take regular breaks! Get up and walk around. If you are WFH get out for walk at lunch.
In general, care for your body so that you may write code into your 80s.
Place modifiers on the thumb keys or - if you don't have any of those - use home row mods!
My ranking of measures from most effective to least effective:
1) Do everything you can to minimize workload of weak fingers (pinky & ring fingers). Just flipping control and caps lock is often not enough.
2) Split keyboard; halves roughly shoulder-width apart. Optimize for straight wrists both at rest and "in action". This usually results in zero tilting or slightly negative tilting.
The truth is that if you have an intelligent child, independent school is a complete waste of money. In the UK you will be spending in the vicinity of £200k over a child’s education to finish a levels, and although they will get better a levels on average, their results at university do not reflect their a level achievements. This is why independent schools find themselves downgraded in university offers.
This isn’t a surprise, because independent schools hothouse children to ensure they peak at a levels, whereas what universities want is students who will continue to improve at university.
I have two children (3xA*, 1A for one and 3As for the other) who were not interested in Oxford or Cambridge. My experience of Cambridge students (I live in Cambridge) is that I have seen many burn out. You also end up with a very narrow program of study which for children with broader interests forces them into a box very early. It’s also a 3 year undergrad program with 24 contact weeks a year, which is insanely short.
My children have gone to Scotland (Edinburgh and St Andrews) which allows significantly more flexibility than English universities offer in choosing subjects outside your chosen degree pattern. St Andrews even lets you change degree completely if you find something else you like.
If you really really want to be a mathematician at 18 then I can see why Cambridge or Oxford might appeal; for kids with more breadth, I think it’s a poor choice.
>You also end up with a very narrow program of study which for children with broader interests forces them into a box very early
To some extent, but one of the things about it that I liked was the course I was on was more general than most other English universities. But still, it's not as broad as e.g. a US university, so it's pretty relative. (Basically, for engineering the curriculum is basically 'all engineering' until the second year, where you then can pick specific modules to go into specific areas. Natural Science and Mathematics are similar. But, relevant to your point about burnout, they didn't really cut anything from each area compared to other, more focused courses, so the workload was definitely intense). For me it was a perfect fit because I knew I wanted to go into engineering but I didn't really have a strong preference for which type (still haven't really given up being a generalist).
> My experience of Cambridge students (I live in Cambridge) is that I have seen many burn out.
100%. I "burnt out" (actually, I think I discovered there was more to life than the academic slog I'd spent my entire schooling immersed in) and despite 6 A levels came 94/97 in my third year.
It happens a lot, and my suspicion is that the burnout is caused by the whiplash of going from a high intensity/pressure school environment (where you're likely told you're the smartest person in the room), to a more adult, self-driven one (where it's clear you're not).
> You also end up with a very narrow program of study which for children with broader interests forces them into a box very early.
This depends on the course I think. I did natural sciences which is extremely broad, and allows much later specialisation. Other courses are far narrower d think.
> You also end up with a very narrow program of study
This just isn't true in my area (Physics), the courses at Oxbridge are just as broad but go much deeper than you'd study in another University.
I don't think it's true of written subjects either, from friends that studied there it sounds like the cranking out of essays is weekly or more at Oxbridge whereas my housemates at University were doing termly stuff.
By narrow program of study I don’t mean “breadth within the discipline” I mean “credit for work outside the discipline”. My son is studying maths but has taken two semesters of biblical Greek, previously did a music subject and this term is doing Hebrew. Can’t do that at Oxbridge.
Yes I'm really not sure obsessive patterns of behaviour will be helped by an unqualified stochastic parrot giving you advice. Unfortunately, when treatment for conditions in some countries costs you an arm and a leg people will resort to quackery. But even in countries where it doesn't, the waiting list for therapy can be so long that you could be waiting years if your case isn't somehow life threatening or profoundly limiting you won't be a high priority.
I really don’t understand why people are downvoting these remarks. We can feel desperately sad and sorry for his wife and child and family while also recognising that he has literally espoused wide availability of guns AND the inevitability of gun deaths as a result.
He talked the talk, and now he has walked the walk.
Gun deaths are inevitable because there are bad people that we can't expunge from civilized society that will kill regardless, not because he wanted to be killed by one.
Expect a significant exit expense, though, especially if you are shifting large volumes of S3 data. That's been our biggest expense. I've moved this to Wasabi at about 8 euros a month (vs about $70-80 a month on S3), but I've paid transit fees of about $180 - and it was more expensive because I used DataSync.
Retrospectively, I should have just DIYed the transfer, but maybe others can benefit from my error...
reply