LOL that paragraph hides all of it. On prem means that someone is administering your application that does not understand it. Which means YOU are administering it but via slacking with your customers infra engineers rather than through your own control plane.

The article is right that there are customers who require this and if you want them you'll have to figure this out, but it is a huge pain in the ass and should be avoided if at all possible.

I think B2B software should be able to create manageable installations for technically competent IT professionals..

If your install has that long of a manual task list, your cloud SaaS infra is probably junk too.

Back then, learning how to manage your own PC and install software was tantamount to earning an IT certificate. Doing so for a few years in high school would be enough to qualify you for a career in IT.

So I politely disagree. The industry was never able to ship software that a non-technical person could install. Especially not B2B, but barely B2C as well.

In the very early 1990s, something like "Insert disk 1 and click the disk icon, then click <name of application>". Installation to the hard disk was optional, and might be one or two steps.

Later in the 1990s, installation was pressing "Next", "Next", "Next" after inserting the CD.

Software has to handle much more hostile environments.

Perhaps not thinking of a "time when", perhaps thinking of an OS that required an IT certificate.

Because even early Macintosh System Software (classic Mac OS before OS X) was bewildering to Windows users, since any user could "drag" a program from a floppy disk onto their hard disk, leave it anywhere, and it worked.

Arguably, OS X introduced the Applications folder in 2001 partly to give people a place to drag things to and feel more comfortable that's where they could find their "Program Files" (but mostly so multi-user system users would get to add and re-use apps in a common clear place they "should" be).

But you still didn't need an IT admin.

Installing software means clicking the button, waiting, clicking yes on some permission prompts, and logging in with Google.

The only thing about any other software I can think of, that would be inherently any harder, is if it involves domain names and certificates. Which is a problem we should be working to solve, and I'm still annoyed at Mozilla for cancelling FlyWeb.

Unless you're such a big company you can definitely afford IT staff, you're probably not doing anything that needs separate services or a database or anything beyond one executable with SQLite, same as consumer apps.

Mac OS X Server?

The vast vast majority of SaaS products talked about on HN are probably not even 1/10th as complex.

"But we want it to integrate with our single-sign on product."

"Our security team scanned it with our chosen tools and you have to fix these things before we will deploy it."

"We aren't willing to make those network changes to allow it to run."

"We won't allow it to connect to our <foobar> server but it is a requirement to connect to our <foobar> server if it is going to be hosted internally."

This is the stuff that makes "enterprise" deployments difficult. Oh and they want you to hold their hand through it but they aren't willing to pay for consulting.

These sound like tire kickers and un-serious customers, why focus on their expectations?

Serious customers, almost by definition, are willing to pay for custom work they want done.

When I worked in shrink-wrapped software back in dark ages the documentation writing team and a very extensive manual QA department where each the same size as the development department. Think people trying for DAYS to find out why out of 100s of thousands of active users, a few dozen reported being able to launch 2 instances of the main window when that should not be allowed. (Fix: Race condition in the "double click" handling code with a window of a few milliseconds)

While this is somewhat true, it's not unreasonable to keep the matrix very small to start with and gradually expand it. People think that matrix testing is some insurmountable task when it mainly comes down to some reasonable, thoughtful engineering. If for some reason you've got some part of the matrix that's particularly expensive (looking at you AIX) then charge a heavy premium for that platform.

> dedicated staging environments

Even if you're a SaaS company you should have dedicated staging environments. Without this you're using your customers as QA, and that's just rude.

> sensible logging schemes

Please have sensible logging schemes. Your on-call/ops folks deserve to be happy too.

Having lived on both sides of the on-prem / SaaS flow, the things that make an on-prem offering successful also make the SaaS offering easier to develop, maintain, and troubleshoot. On-prem can be more expensive but it also means bigger deals, more predictable income, and higher margins. Not every company can fully adopt the mechanisms to generate an on-prem offering but the purported costs of an on-prem offering frequently come down to "ship higher quality software" rather than "shipping somewhat sketchy code at a breakneck pace."

This attitude is why I get paged. Perhaps we shouldn’t be encouraging lightning-fast growth above all else.

There has to be a name for this argument-type: A leads to B, and those who seem to know only B start arguing for A because they don't remember why A lead to B in the first place. I see this all over the place.

I know as a consumer I'm much more reluctant to buy a new subscription product than I am to make a single purchase, but the options get less and less every year.

The problem was that the maintenance revenue was valued at a much higher multiple by investors than their Perpetual Licensing revenue because it was recurring. Cue the thought... "What if the whole thing was recurring?"..."Hmm how could we justify that?"..."We make it a SERVICE that WE host!"... and then here we are now where if you want to actually own software, chances are you're out of luck.

I ran a SaaS and PaaS for fintechs and we had annual events where we had to hard negotiate with customers who didn’t want to incur the employee training burden for upgrading to newer releases with new features etc. Instead of accepting the rolling releases they would do one or two updates/year and we would stand up monolithic support programs to accommodate them. Moving away from this customer model was about headaches and release/support debt.

There are a lot of challenges with complex on-perm solutions that hosting-for-customers can alleviate. Staffing for every customer hosting variation gets unduly complicated — and impossible — very quickly.

Don't get me wrong, it's also hard, but providing on-premise software is less difficult than it used to be in absolute terms. The main things that changed are SaaS driving expectations up, and needing to upgrade faster.

I actually remember why one company I worked for moved from on-premise to SaaS. They told us that the market valued recurring software revenue at a higher multiple than it did one-time software revenue.

On-prem was a large one-time sale and a small recurring support contract. SaaS was almost all recurring revenue.

On the customer side, it is capital expenses vs. operating expenses.

At some companies, this is clearly demarcated from the SWE role - at others, stereotypically the "IBM mainframes, SAP and Oracle DBs" category the OP mentions, they kinda shade into each other. Which is, of course, why those technologies have such negative reputations. (There's also the Palantir route, where you have a clear demarcation but lie about it to new grads who don't fully understand the conflict between "forward deployed" and "software engineer".)

Basically, if you want to build an on-premise offering, look at the way Nginx, Postgres, Redis and similar projects are packaged and do the same.

Nope. You should look at how an app that uses all of those is published. That’s what on-premise software truly is. A combination of dependencies and your companies secret sauce.

Your own app code is just one cog in that engine.

These are orthogonal. Resourcing is no guarantee of ops competency, or a culture where ops is enabled to deliver, hence the concept of shadow IT. On-prem software is bought by orgs with low risk appetite or other compliance objectives that skew procurement towards running on prem. I have seen exceptional ops teams on a shoestring ramen budget, and I have seen dumpster fire ops teams in companies with thousands of workers and billions of dollars a year in revenue who could not get you a single virtual machine in less than 90 days even if preventing the end of the world depended on it.

There's no reason other products couldn't do this as well, even those with a complicated stack that requires provisioning cloud resources (or equivalently, SSH'ing into servers on the local network or calling private cloud API endpoints). If you can deploy a commit from a pipeline, then you can make a self-hosted bootstrappable installer for your users.

The installer could be a tiny shim running on the user's laptop that's responsible for bootstrapping the rest of the infrastructure and then deleting itself after redirecting the user to the cluster admin panel. The trick is to make it smooth, so even non-technical users can provision and eventually administer a secure, self-updating cluster as easily as they can keep their iPhone up to date. Walk them through the process of getting AWS credentials. Heck, scrape and script the login flows if that makes it smoother (it's their computer running the code so who cares?). Ask the user the minimal questions needed to fill the config. Once all that's in place, trigger your deployment scripts on the cluster, to pull images from a container registry or whatever. Wait for it to be up and running.

When the cluster's up, redirect them to its admin panel that's more sophisticated and controlled by you. Let them check for updates, install plugins, run common operations (like "restart," etc).

Then just don't ship broken code, make sure your scaling policies work (or at least that they work well enough to avoid phone calls from customers that don't make you any money). And boom you've got a nice product.

I'm not saying it's easy. But if you already have automated deployments (you do have automated deployments, right?), then it's not that many extra steps. Productizing it is hard but can be done.

I got a first-hand view into how this model falls apart - and it wasn't the fault of the customer's infra engineers; it was entirely our fault and we walked away from piles of money because we couldn't deal with the issue.

Here's my rather salty take on the issue.

The killer issue with SaaS software that attempts to migrate to on-prem offerings is that SaaS companies tend to build up cultures that are locked into the "move fast and break things" model. Simply creating an on-prem offering from a SaaS product does not generate the cultural shift needed to successfully ship on-prem software.

If it's extremely easy to ship then the development cycle can be fast and sloppy. You can ship buggy software, bypass meaningful testing, lean on your ops team/on call rotation to limp the software along, skip documentation and use humans to glue the thing together. SaaS is popular because of the accelerated iteration cycle because shipping a bug is much less difficult.

Things fall apart extremely fast when moving from a SaaS offering to an on-prem offering because _all of the developers are still conditioned for the fast iteration cycle_. Developers are still going to be targeting SaaS first because they can interact with it, debug it, and evaluate telemetry from that running system. They won't be developing in the model where they're truly freezing features, polishing implementations, and shaking bugs out of the system. This ultimately means that regular releases are more of snapshots of main rather than something that can be shipped to a customer and run unattended.

In addition, having an ops team or on-call rotation for a SaaS offering means that troubleshooting/operational guides can be largely informal or loosely written runbooks. For an on-call/ops person, if given the option between scratching together a loose note or using Slack for documentation, or taking the time to write well polished documentation that can be consumed by outsiders, operational velocity is going to drive people to follow the easier, less labor intensive. The end result of this is that on-prem offerings are not going to have a lot of critical operational documentation.

I was on the team responsible for building, shipping, and supporting an on-prem version of our software; it was one of the most grindingly miserable work experiences I've had to date. In addition to dealing with the on-prem version my team was also on the on-call rotation and I can say from first hand experience that the SaaS model was only held together with a great deal of firefighting; shipping an on-prem version exposed every single weakness we had on the SaaS version in painful detail.

The project ultimately failed; millions of dollars of deals were left on the table and my entire team was laid off because the company was not capable of shipping a reliable, documented project that could be operated without constant babysitting. The product itself had a great number of incredible features and if people slowed down and focused on improved quality rather than sheer velocity then we'd be drowning in enterprise contracts.

> Which means YOU are administering it but via slacking with your customers infra engineers rather than through your own control plane.

If your product requires that you're remotely administering your software because your customers can't operate it successfully, then please invest in documentation, validation and testing, and troubleshooting. Your customers will need it as much as your on-call rotation.

You can also sell a managed on-prem setup where you set it up and maintain it in the client environment.

It’s not a huge pain in the ass as long as the software is reasonably designed to be multi-tenancy as well as multi server. For serverless architectures creating an internal service provider to route to your serverless (or theirs).

A lot of non prem can be made easier by ring azure friendly which in turn has some ways to get on prem easier.

I am not against on-prem offerings, in fact quite the opposite. However, a significant swath of our customers are small-to-medium size businesses, and many don’t have access to qualified IT to run sophisticated software systems. Our contractual stipulations are very lax in terms of requiring customers to keep their hardware, software, and environment in a conducive, working order; we recommend ____ but if you fall out of spec, you assume the risk. Fine, except our contracts still put us on the hook to make sure, with reasonable effort, that the software is running. That means we spend our sparing and critical resources trying to cut pared down or specially tailored copies of our deliverables and trying to log into countless boxes to troubleshoot every banal and tedious tech support issue as to why a package isn't running or won’t install (spoiler: it’s almost always environmental: GPO restriction, firewall rule we told them we need an exception for, antivirus problems, dependencies not installed, excruciatingly out of date system, etc).

These two things are not compatible especially as software evolves and becomes more reliant on other services. The problem is that early on the organization decided to bend over backwards to make sales, now we have to choose between letting this problem spiral out of control and drag us down with it, or renegotiate the contract with better/clearer requirements or ditching on-prem which will be a mess and probably result in losing customers. That especially sucks for many of those customers, because they live in areas with poor connectivity.

Moral of the story: if you give your customers enough rope to hang themselves with, you’re also measuring out a length for yourself.

Path A: Run a windows server VM and SQL Server somewhere in our customer's infrastructure. Our customer is 100% responsible for the health of these items and provides them as a service to us. Our customer has to notify us of any infra or app issues reactively.

Path B: We operate as custodians in our customer's Azure tenant. We deploy our software as a Function app and use Azure SQL. We are 100% responsible for the health of everything. Our customer is notified proactively of any issues.

To support both paths, we have two different executable project types (console app and azure function app) that consume a common library type. Only the barest boilerplate code actually changes between on-prem and cloud. 99% of the difference is HTTP Trigger function vs AspNetCore hosting syntax. Everything else is identical.

We work with small US banks and we haven't found one that is total stonewall against moving to cloud. Many are cautious but open. The #1 concern is the PII/data. We've been working with a few angles to cast that in a more realistic light. Most of our customers have had some degree of an incident with on-prem infrastructure that could have been made impossible if business was being conducted via a cloud provider.

I don't know why this went out of fashion but it works just fine. Your customers clearly don't want you to manage the infrastructure, based on your comment. Sell them what they want.

If the issue is that your system is so complex that a reasonably trained admin cannot manage it, you should probably fix that.

What our customers want is a technology partner who can adapt to their changing business needs over time. A one time cdrom+pdf distribution unfortunately will not get the job done in this case.

Our customers are paying for a business outcome. Our software is just a fraction of that puzzle.

Largely because "control your workflow" (a-la the toyota method) requires that a vendor is unable to change things on your behalf without your knowledge.

"On-prem" is about being permitted to invest your own resources into ensuring that your $thing will work during the hours it needs to work and you can pour as much or as little into the problem as you deem necessary.

If you use a SaaS you must somehow figure out how that factors into their pricing model and it's still largely their whim, and worse: they will still change things without notice.

It's also true that a lot of companies do not assume that the internet is always available.

Getting 100Gb/s to a server room is easy for 500 people in an office building.

But getting 100Gb/s to your SaaS platform might not even be possible without rearchitecting. (IE: Perforce in game dev, usually on-prem.)

* It’s much easier to do on-prem if you do a monolith, or at least a small number of services.

* You have to build and test your application against a range of OS and hardware combinations. You’ll almost certainly need to support RHEL, and probably Windows Server 20XX. Many companies won’t let you use Docker.

* Your software need to be tolerant of infrequent updates. Customers expect some level of stability in practice.

* You need to integrate licensing checks.

* You need to build an installer (that works out of the box…)

* You’ll need to agnostically support the common parts of SAML/OpenID for Auth only, and not rely on any bespoke features from a commercial provider like Auth0 or AzureAD.

Productisation, infrastructure expertise?

Why do anything?

I worked until recently at a traditional engineering industry focused startup company which decided to deploy a Python application on-premise to a big customer. From the use case it was very clear to me that an on-prem deployment of the software was likely to be on the cards, but there was nobody there in the non-technical leadership to say at the beginning "wait a second, maybe we shouldn't write this in Python for this use case".

It is.

> Why do anything?

Don't, unless it's 100% critical for success. You have to run super lean as a startup. You don't have time for ANYTHING except validating your business.

In particular if you deploy in Cloudflare’s style you can deploy workerd as a single binary and load it up with your functions.

It’s a little immature right now and persistent storage can be a problem though.

The company I'm with supports our releases for two years, most of our products release quarterly, and many of our customers are kind of annoyed because we release too often (in their eyes). We're looking at getting out of supporting the on-prem game, but personally I think it's unlikely because many of our clients are in government and defense.

When I went to work at a University as a research software engineer, a particularly memorable incident was a research group getting very angry that we were upgrading the cluster to CentOS 7 thus meaning the ancient version of COMSOL they were contractually obliged to use in a contract with Rolls Royce would no longer run at all. We ended up getting something working using Singularity but it was very painful.

I'm honestly shocked that Sentry, for example, doesn't have a self-hosted enterprise option. We're all terrified of PHI being caught in a stack trace and sneaking through any PHI-scrubbing feature. We had to move off of Confluent because CFK's pricing is outrageous compared to Confluent Cloud. It's clearly designed to nudge customers towards Cloud, but we can't use it due to contractual requirements with customers. Everyone is so anxious about breaches that even a BAA doesn't cut it anymore.

So there's absolutely a market for this kind of stuff.

Operationally speaking we built the products from the ground up to be available either way (at the time, it was a less contrarian take). The resulting cloud architecture is not multitenant (just high efficiency, orchestrated shared hosting). "Multi single tenant" I think they call it these days.

I think its an extremely pragmatic albeit unpopular choice for a B2B SaaS company.

Needless to say we don't have a Fortune 500 budget. When I see a nice, affordable app that says "and here's a Docker image!" next to the SaaS options, I am like ( ˘ ³ ˘ ) ♥

This comes back to nontechnical leadership, as business process issues tend to do.

Leadership has something cool, they think "what's cool for us is cool for everyone", and then they try to sell into a market that maybe doesn't do cool in exactly the same ways. Take something like SAP B1P or JIRA. Both established or very powerful cloud apps. You'd be tempted to think that anyone in manufacturing would want some of this . . but then you have to build everything so that it's able to be portable-ized, which adds a ton of development time, and an ongoing maintenance cost that also involves headcount/travel/US persons/security clearance.

How do you know if that cost could be justified? Well, that's where "nontechnical" comes in. The right way is matching up costs of dev and maintenance with what the industry will minimally pay. The wrong way - which seems dominant - is to use your Leadership Spidey Sense to determine whether or not to take the plunge. Then you get stuck in, you short the dev or the maintenance, and end up with a platform not optimized for either cloud OR on prem.

I was an engineer on a product that had an On-Premise (OP) offering. Some of our biggest moneymaker accounts were On-Premise and liked the product.

The downside was that we had to have a dedicated team just for On-Prem stuff. I was one of the engineers on that team. We had to develop custom management interfaces for the software that were only deployed in an On-Premise environment. It wasn't crazy hard or anything but it was time and effort.

Upgrades and installation were also a big track of work unto themselves. When you deploy into an On-Premise environment you are completely at the mercy of the clients infrastructure. One support call I did was clocked at 14 hours - there are so many things that can come up. Plus there's the whole hassle of actually getting the calls scheduled and making sure clients aren't too many versions behind.

Sometimes we had to spend a lot of time waiting on support calls because the customer didn't have their stuff together. Like sitting and waiting for a network team to open up the firewall so that bits could be downloaded from our distribution server. And of course the firewall had to be closed up right afterwards, so this was a common thing.

Another time we spent weeks troubleshooting a client installation. In the end it turned out that the client had our software talking to two different internal NTP servers which were slightly out of sync, which caused our SSO/SAML integration to break.

Database migrations were also challenging - in a cloud environment you often have flexibility to migrate things live while the system is running. But often with the on-premise upgrades, you had to take the whole system down while the migration was running (back to the 14 hour support call). This could mean scheduling the support calls at really weird hours.

On-premise also has challenges if you start selling to the US government, especially defense. There are super strict requirements that support personnel must be US citizens on US soil - so hopefully you have more than one person on the team meeting this requirement!

All in all, I am not a detractor of offering an On-Prem solution. It can be a real differentiator and it will open a lot of doors. There are some companies that simply won't buy SaaS, either due to internal or regulatory reasons. Once you land these clients it can be very lucrative and long-term. But, it almost always requires dedicated staff to support and manage.

That's not unique to either on-prem or the government, alas. Get involved with HIPAA, SOC2 or a few other funny acronyms and the restrictions around who may access production data or servers gets very tight indeed, even if you are a SAAS company.

Enterprise customers tend to have more edge cases and scaling requirements, along with a huge workforce with more difficult lines of communication. Anecdotally, they seem to need more follow-ups and hand-holding than smaller nimble companies.

An ideal customer is often one that subscribes, pays their bill, and rarely asks you for help. Your product already solves their problem out of the box without needing to go down any rabbit holes.

This is why consumer electronics products have those stickers that warn you you void the warranty by opening up the device. It's because nobody wants to deal with those customers who open up their device, break something, and then demand a replacement because the product is defective.

“Premise” means something very different, so I recommend you use “on-premises” or “on prem” if you’re trying to demonstrate competence to the majority of your audience who will know the difference.

But honestly, how harmful is it to your business unless your addressable market is mostly English majors? And even then, it probably prompts interaction from the English majors, who can finally show off a little with a correction.

At my current employer I chose to use their cloud offering. I thought, hey, it's just me here, and I don't have time to sysadmin yet another thing. A few weeks ago I needed to enable the REST API, and for them to create a KVstore. On-prem these are 2-minute processes (set a value in a config file and restart the process). In the cloud offering, you need to make a support ticket. It took 17 days, and that's only because I ended up doing half of it myself -- they enabled the REST API first, and I found I could use the REST API to make the KVstore.

I have some regrets.

On premise goes away and you can also setup private cloud to be filtered on firewall to give access to customer only or connect it via vpn or whatever else.

Biggest upside is that you still control deployment and are responsible for uptime but you have best knowledge in-house to deal with it.

Off-prem feels that way to me. It's "the mainframe" run by the off-prem vendor. I have a dumb terminal.

On-prem seems so much easier in today's corporate IT environments. Ship the Customer a VM. Use SAML for auth. Interact w/ clients over HTTPS.

It's worked out fine for us (of course there are tradeoffs but for us they're manageable). Definitely requires an investment by people who know what they're doing as the article alludes to.

You're putting the cart before the horse. The only places still making money off of on-prem installations are charging a premium for it. If a big government account comes up and offers to pay you extra for a special service contract, you're going to do it.

I get the appeal to developers of just throwing out a self-service experience with minimum commitments. But it's not a good, sustainable business model on its own.

If you aren't confident enough in your software to hand me that code in an install file, I guarantee I don't want to rely on your janky startup's hacked-together spaghetti.

I love the idea of on-premise for small startups (or solo startups) because it shifts the operational/oncall burden to the customer.

Also "on-premise" in 2023 can easily mean something deployable to a customer's AWS/Azure/GCP account.

not premise