I'm currently rewriting our store implementation, and it's complicated.
One example is that some currencies have 100 subunits (eg USD), some have 10 subunits (EG TWD/MOP) and some have no subunits (eg JPY). This is made slightly more difficult in that Stripe/Paypal will treat subunits differently, for example Paypal might treat JPY as having 100 subunits and Stripe will treat it with 0 subunits.
You want to be really careful here not to charge 10x/100x more/less to the end customer than you imagine.
Formatting currencies is also something you need to do carefully as dots and commas mean completely different things in various countries and can lead to unexpected charges for customers.
Don't get me started on VAT :P
All worth it in the end though, I enjoy writing this sort of thing and I do see long term benefit to our business that we do it all in-house.
I'd also recommend future proofing implementations to use longs instead of ints for storing money values. Back when Bitcoin was less valuable we allowed BTC as a currency where 1 subunit = 1 satoshi (100,000,000 satoshis in a bitcoin). It was possible to overflow an int this way.
You also can't really write payment systems with the speed you might write other code in your startup - errors and mistakes here could have real impact on peoples lives. For example, loops that create charges that don't end (possible with both Stripe AND Paypal). Loops can manifest themselves as lock collisions as well. Basically, be careful and if speed is important you probably want to outsource it at the expense of some control.
Not to mention the pain if you're selling low value (sub-penny) items, that get scaled by weight or some other dimension. The errors can often throw out invoice totals in the most annoying manner.
Lots of third party implementations will take a few % off your bottom line. That's the main one.
There's a couple of other points here as well:
- If you do enough volume, you're able to negotiate the fee with the payment processor, we've done this successfully. If you outsource, that negotiation is the third parties margin.
- I'm not sure if this is true for third parties, but if they wrap up all your payouts in your native currency (EG GBP£), you lose money on forex. We try our best to have all our EUR and USD sales land in their respective currency accounts, then convert them with low cost services such as transferwise which has saved us a lot of money in the long run.
Saving on your forex and having no third party processor fees, I think in some instances you could be saving in the region of £40,000 to £80,000 per £1mm of sales.
We also couldn't find a third party implementation that would allow us to bill for our product, varied by country, amount, currency AND billing frequency. This fine tuned control can have real benefit, for example we noticed in Ukraine we had high traffic but no sales - by introducing a low monthly UAH billing option we started to grow sales in this region.
Other third party implementations won't handle VAT properly or comprehensively which can create more of an accounting burden, or round VAT transactions unfavourably (size of that benefit is irrelevant to very small obviously depending on volume of transactions!) This gap is being slowly filled from what I've been seeing though.
Using a third party solution for SaaS will always have some level of disconnect, it just feels nicer for the customer when it's all integrated seamlessly. One example being you have full control over branding and distribution of invoices/receipts etc.
Martin Fowler's money pattern [0] solves this problem. Just make sure whenever you're dealing with currencies to use a value object for money, rather than float. Lots of languages have libraries for this pattern. [1]
I recommend his blog. He was, and I think might still be, ahead of the curve regarding software engineering patterns etc. There was an article shared on HN which hit #1 from a startups tech blog talking about their brand new architecture and someone pointed out MF wrote about an identical architecture pattern in 2004 or something.
He wrote influential software engineering textbooks on patterns, agile development, refactoring and the like. I'm not sure how relevant he is today, but in the 90s the ideas he presented were pretty groundbreaking.
> One thing I've always wondered is who the heck is Martin Fowler and why does he have such a following?
You don't know the history of your craft; study more, look into where agile came from, the extreme programming movement, and who was involved and how refactoring became a thing. Names like Ward Cunningham, Kent Beck, Martin Fowler, Ron Jeffries, Rob Martin, and Dave Thomas should be familiar to anyone who knows their craft.
I think I’ve read some of them, but it would be nice to refill my bookshelf with classics. Could you please recommend a re-starter kit of these authors’ top books?
You would think that this is a fairly rookie error and that big companies would know better, but I regularly see this on Uber: https://i.imgur.com/qDACtG0.png
Interestingly even in your link you can see that most of the money values (subtotal, total_fare, total_charged) are strings but for some reason total_owed is a float.
Presumably someone messed up but it wasn't caught in code review.
On the surface one would think big companies should "know better". I mean, a large company is the accumulation of centuries or even millennia of career experience. Unfortunately, they also contain just as much accumulated nonsense and foolishness; and enough bureaucracy and organizational cruft to hide it forever.
Plus there’s the issue that the career experience isn’t evenly distributed, so you can often have people implementing features without the background necessary to do them right.
A large company is the accumulation of a large number of people, some of who will know better and some of who will not. Hopefully the company is organized such that the people who know will be in a position to catch those kind of errors, but that's not guaranteed.
With insolvable i meant just the technical restrictions of floating point numbers. There are of course solutions. But there is just the fact that binary numbers with discrete length can only be mapped to so many numbers.
If you use numbers directly in exponential form in vanilla javascript, you already get better results.
It's not that they're not solvable, it's just that you have to go out of your way to use the proper type or library. The default floating point type still has problems.
Important caveat: BigDecimal is arbitrary-precision, not infinite-precision. Using it still requires you to think about the precision of each number you operate on.
The story I heard is that the first time this was attempted in real life, the account balance grew so fast that it attracted attention and they were caught.
One of the reasons people used COBOL is that it handled currency better. You'd say "ACCOUNT-BALANCE PIC S999999V99" to specify the account balance was signed with 8 digits, and an implied decimal point (V) before the last two. The math was done with integers so floating point inaccuracy wasn't a problem.
An unrelated issue with money handling: I once got a credit card charge for $NaN (not a number). Fortunately it was resolved without me needing to write a check for $0/0.
Photo of the receipt at http://www.righto.com/2008/05/importance-of-software-testing...
The easy way would've been to output a fixed fraction precision rather than a floating point number directly, but that doesn't mean the underlying data or checked algebraic operations are any good.
There are several things that could've prevented this, but apparently they become forgotten every few decades or so. Here's one that could help:
A fixed-point math library using a 64-bit word format like Q1:49:14. That would be a signed, 49-bit whole number with a 14-bit fraction (4 digits for nominal storage / 2 digits for settlement rounded in-place). With a ~$562 trillion limit, it would have an effective range of -562,949,953,421,311.9999 to +562,949,953,421,311.9999. This would work for almost all banking operations but could break down in hyperinflation and the distant future, where reserve and government banking would likely hit "Y2K" issues with it first. Considering 128-bit types exist (although at double the storage cost), it might be worth creating a Q:1:60+:14 format with built-in Hamming code (or similar) ECC for protection beyond that which exists at the hardware level (RAM, possibly CPU caches & IO backplane, probably storage.. but ECC and integrity isn't universal nor always possible).
Obligatory reference to the Moonpig billing system, and Mark Dominus' excellent deep-dive article on its design:
> Sometimes I see other people fuck up a project over and over, and I say “I could do that better”, and then I get a chance to try, and I discover it was a lot harder than I thought, I realize that those people who tried before are not as stupid as as I believed. That did not happen this time. Moonpig is a really good billing system. It is not that hard to get right. Those other guys really were as stupid as I thought they were.
Most lisps can handle fractional numbers as actual fractional numbers without the BS floating point rounding. 1/3 of 100 For example, is 33.333 repeating. Most languages and databases will round this off at some point. Most lisps are perfectly happy with the idea of fractions and thus circumvent this problem entirely, at least, until you need to interface with some other brain damaged system or need to give someone physical cash money.
If you're dealing with money, you should seriously consider using a lisp.
The smallest unit of US money is the mill or 1/1000 of a dollar. The smallest unit of US currency is the cent. You're supposed to work in mills and round to cents.
Or just work in floats and round as the last step. Or do both selectively depending on which rounding error works in your favor, but don't tell anyone that's what you're doing.
The real problem is that nobody at the company seems to have looked at the parts of their software that everyone sees, so what else have they not looked at?
> Or just work in floats and round as the last step
That's not as easy to get right as one might like. Let's say we have an amount in dollars, represented as an IEEE 754 double, and a tax rate, also represented as a double. Let's assume that the amount is always some integer multiple of 0.01, and the tax rate is always some integer multiple of 0.00001.
Let's say we want the tax in cents. A first try might be:
(Doing tax in cents because C/C++ doesn't seem to have a standard variant of round() that lets you say to round to the nearest 0.01. It only rounds to integers).
That will sometimes fail. The problem is it is rounding at the wrong place. It's logically rounding to the nearest multiple of 0.01, which is too crude. The rounding has to be much farther to the right.
This will do the trick:
unsigned long tax_in_cents(double amt, double rate)
{
const unsigned long M = 100 * 100000;
return (unsigned long)round((round(amt * rate * M) / (M/100)));
}
That will work for all rates from 0 to 1 that are integer multiple of 0.00001, and all amounts from 0 to 10000 that are integer multiples of 0.01, in all IEEE rounding modes. I've verified this via brute force. I'm not sure how high the amount can go before it breaks down.
I'm not at all sure that if I had come across that first try in real life I would have noticed that it is not adequate.
Yeah, nobody is going to notice or care about an error of a cent or two for something like retail, especially if it evens out in the long run. It matters for finance, not for stuff like Uber or Amazon. Just make sure you render it properly on the front end and nobody will care whether you use floats, ints, or decimals. Be consistent in the back end and careful in the front end, ideally with standard functions for translations.
Not that simple, though. That's the "division and rounding" method of the article: share a $.99 fee equally between two parties. That's either 49 cents each or 50 cents each if you're not careful. In both cases, where is the extra cent?
If you use floats and rounded that one cent would disappear with them as well.
Unless you're storing half cents in their account and not showing it.
Which is perfectly legitimate by the way, because if you earn a fraction of a cent interest accumulated over a couple months that might actually be a cent and you have to pay it.
The solution might be to store everything as 1/100 of a cent, as an integer.
Working in cents isn't always sufficient. Let's do 34.7% of $25. In cents, $25 is 2500. 34.7% of that is exactly 867.5 cents, which should round to 868 cents under the most common rounding methods in most commercial applications (round to nearest with 0.5 rounding up, or round to nearest, with 0.5 rounding to even).
You need to integerize the percentage, too. For my current financial applications, all the percentages I need are integer multiple of 0.01%, so I use that as my "percent cent". 34.7% is then represented as 3470.
That leads to a function something like this (assuming you need the 0.5 round up rule...changing it to 0.5 rounds to even is left as an exercise):
(The 2 and 4 suffixes are a naming convention I use. They are reminders that amt2 is the underlying amount x 100, and that rate4 is the underlying rate x 10000).
Unless I have to actually work with floats, I try to only do everything with int math to keep things sane. Not to mention, that it usually makes calculations faster.
Decimal floating point exists (https://en.wikipedia.org/wiki/Decimal64_floating-point_forma...), is that considered useable for financial data, or does it turn out in practice to also suffer from the same issues? Since, e.g., dividing through 3 will not be exact here either of course.
This approach doesn't work when dealing with fractional cents, like charging $0.0001 per hour for something.
I would rather recommend using an arbitrary precision decimal library like Decimal.js, and using strings to represent money in JSON. And of course using an appropriate database type for storing the data.
That works if you are 100% sure you know beforehand what the smallest possible fraction will be.
What you gain by consistently using an arbitrary precision decimal library to process all monetary values, is the freedom to process smaller fractions later if necessary. Also you'll never overflow with really big values.
In other words, when you represent monetary values with arbitrary precision Decimal objects everywhere, you can easily expand your application as much as you want on either side of the decimal dot. Changing existing integer-based code do to that is very error-prone, you quickly lose sight of what exactly an integer value or parameter means in different places.
Was going to say this too, specifically rationals with arbitrary precision should cover all general currency related issues assuming irrational values are forbidden.
Good example of a leaky abstraction. JavaScript developers almost never have to worry about or completely understand the intricacies of floating-point arithmetic, until errors like this happen. https://en.wikipedia.org/wiki/Leaky_abstraction
This is not a leaky abstraction. JavaScript doesn't pretend that the numeric type is anything other than an IEEE-754 float. It would be a leaky abstraction if JavaScript said "this type can represent all real numbers!" but that would be insanity.
Sure, it's not a leaky abstraction as long as what you want to represent is an IEEE-754 float. Let me know if you ever come across a real-world case where that's exactly what you want.
Sure, that's a good-enough abstraction for a lot of cases, which is why a lot of languages implement that standard, and it's pretty reasonable for JS to do that.
The pedantic argument you're making is basically that it was never intended as an abstraction. But when it's used as an abstraction, it's almost inherently a leaky abstraction, and it's almost always used as an abstraction.
It's not a pedantic argument. A leaky abstraction is a thing when you have some kind of abstraction where the user is supposed to not have to know anything about the implementation details, but where implementation details accidentally "leak" through the abstraction (meaning that the user does, in fact, have to know about the implementation details).
That's not the case here. JavaScript provides an implementation of 64-bit IEEE-754 floating point numbers. You do not need to know anything about the implementation (is it implemented in hardware? in software? what are the algorithmic details? who cares, they just work!) to use them, they function perfectly like IEEE-754 floating point numbers should.
You're saying "well, some programmers think floating point numbers can represent finite decimal numbers exactly", and that may be true. But it's not JavaScript's fault that some programmers are bad at their jobs and don't understand how floating point numbers work (note that this really has nothing to do with JavaScript: this is true in essentially every programming language). A misunderstanding is not the same as a leaky abstraction.
Think of it like this: no sane person would think the fact that an integral type (i.e. an "int" or a "long" in C/C++/C#/Java/whatever) can't represent the number 3.75 exactly means that "int"'s are a "leaky abstraction". Of course they can't represent 3.75, it's an integral type! Likewise, of course a floating point number can't represent 0.3 exactly: it's a floating point type! They never pretended otherwise! If you want to represent 0.3 exactly, go with a rational or decimal type, that's what they're there for.
> Let me know if you ever come across a real-world case where that's exactly what you want.
All the time! Like, 99% of the calculations you make on a computer, IEEE-754 floats are by far the best choice to represent a number! They're a stupendous numeric type! There's a reason why every CPU and GPU in the world is optimized for floating point operations (that's even how you measure their performance, with mega/tera/peta-FLOPS!), and there's a reason why every language under the sun has built-in support for them! Some languages (like Lua or JavaScript) make them essentially the ONLY built-in numeric type: it's that good!
> That's not the case here. JavaScript provides an implementation of 64-bit IEEE-754 floating point numbers. You do not need to know anything about the implementation (is it implemented in hardware? in software? what are the algorithmic details? who cares, they just work!) to use them, they function perfectly like IEEE-754 floating point numbers should.
You're missing the point: the fact that it's an IEEE-754 floating point number is always an implementation detail. I've literally never been told by a business user, "Hey, I'd like you to add an IEEE float field so we can track some IEEE floats." That doesn't happen.
> Think of it like this: no sane person would think the fact that an integral type (i.e. an "int" or a "long" in C/C++/C#/Java/whatever) can't represent the number 3.75 exactly means that "int"'s are a "leaky abstraction". Of course they can't represent 3.75, it's an integral type!
Sure, nobody will agree with your straw man argument. But de facto people want their float types to not round decimal values to binary values. And even integers are a leaky abstraction--if you expect `int` to represent an arbitrary integer, you're going to be disappointed when the values get large enough. But hey, you're the one who said it was an integral type.
> Likewise, of course a floating point number can't represent 0.3 exactly: it's a floating point type! They never pretended otherwise! If you want to represent 0.3 exactly, go with a rational or decimal type, that's what they're there for.
The difference being that the average developer or business user knows what an integer is, whereas I doubt even you can tell me whether an arbitrary case will go wrong without testing it or doing some back of the napkin math.
> All the time! Like, 99% of the calculations you make on a computer, IEEE-754 floats are by far the best choice to represent a number! They're a stupendous numeric type! There's a reason why every CPU and GPU in the world is optimized for floating point operations (that's even how you measure their performance, with mega/tera/peta-FLOPS!), and there's a reason why every language under the sun has built-in support for them! Some languages (like Lua or JavaScript) make them essentially the ONLY built-in numeric type: it's that good!
Way to not read a full 1/3 of my post. I'll just post it again, please read it this time:
Sure, that's a good-enough abstraction for a lot of cases, which is why a lot of languages implement that standard, and it's pretty reasonable for JS to do that.
I'm not sure why you're leaping to the defense of floats: just because something is a leaky abstraction doesn't mean it's not a useful leaky abstraction. I'm not saying languages or architectures shouldn't implement floats.
Floating point numbers are explicitly not trying to be an abstraction for real numbers, but rather an approximation for them that is good enough for most purposes (if you’re careful in how you use them).
That they might not be taught as such is another problem.
Everyone dealing with JavaScript should be taught about .toFixed(num) for rendering numbers as strings. In fact, anyone working on front end code in any language should learn that language's number formatting tools (e.g. sprintf). There just isn't an excuse for this type of sloppiness.
One example is that some currencies have 100 subunits (eg USD), some have 10 subunits (EG TWD/MOP) and some have no subunits (eg JPY). This is made slightly more difficult in that Stripe/Paypal will treat subunits differently, for example Paypal might treat JPY as having 100 subunits and Stripe will treat it with 0 subunits.
You want to be really careful here not to charge 10x/100x more/less to the end customer than you imagine.
Formatting currencies is also something you need to do carefully as dots and commas mean completely different things in various countries and can lead to unexpected charges for customers.
Don't get me started on VAT :P
All worth it in the end though, I enjoy writing this sort of thing and I do see long term benefit to our business that we do it all in-house.
I'd also recommend future proofing implementations to use longs instead of ints for storing money values. Back when Bitcoin was less valuable we allowed BTC as a currency where 1 subunit = 1 satoshi (100,000,000 satoshis in a bitcoin). It was possible to overflow an int this way.
You also can't really write payment systems with the speed you might write other code in your startup - errors and mistakes here could have real impact on peoples lives. For example, loops that create charges that don't end (possible with both Stripe AND Paypal). Loops can manifest themselves as lock collisions as well. Basically, be careful and if speed is important you probably want to outsource it at the expense of some control.