* The Mill build tool looks a lot better than SBT, but seems like everyone is still using SBT
* Scala minor version are binary incompatible, so maintaining Scala projects is a big pain. Upgrading Spark from Scala 2.11 to Scala 2.12 was a massive undertaking for example.
* Scala has tons of language features and lets people do crazy things in the code. Hard to win technical arguments with Scala geniuses that like using complicated language features.
* Scalatest is stil used by most projects and is annoying to use, as described here: https://github.com/lihaoyi/utest#why-utest. The overuse of DSLs in Scala is really annoying. Too many DSLs is another example of something I consider to be an antipattern, but there is no Scala community consensus on the responsible use of DSLs.
I'm optimistic about Scala. There are some folks that love the language and are continuously improving the ecosystem. Scala 3 will have to sell a better story about ditching legacy tooling and giving users a better default stack if it wants to compete with modern Go/Rust/Python.
> Hard to win technical arguments with Scala geniuses that like using complicated language features.
I was on a team building good old crud apps using monads, monoids, categories, combinators, effects cats, seamless and bunch of other nonsense that i've now purged from my brain.
You could've easily mistaken our team for a programming language research group at a university.
This is literally the number one reason i would never choose scala. There is no upper bound to amount of stupidity one can indulge in. Our code was so convoluted that even intellij had trouble understanding what the hell was going on and would spit out compilation errors when there were none.
I now work with clojure team and i feel a sense of relief and weight taken off of my head from all the useless stuff i had to learn and use.
I'm on exactly that kind of team now. They've been working on a very simple problem for several years now and have an enormous, sophisticated, but still buggy and unstable solution.
They're really smart people, and they had to be extremely capable programmers to get this far, but the embarrassing question is, how would a team of mediocre developers have tackled this problem? They would have picked a mediocre language and probably written a naive, straightforward solution, and after a few months of squashing bugs and developing production workarounds, it would have been a workable, tolerable system. They would have spent the last year working on something else.
The naive solution would have more inherent unreliability, but our vast and intricate solution, designed to scale to the moon and be more self-healing than the T2000, is never going to approach the same reliability as a naive solution, because it's too complex (and the code is too hard for mortals to read and reason about) for us ever to iron out the bugs.
They make a huge deal out of how much safer and easier to reason about our code is thanks to the FP discipline, but why do we have just as many concurrency-related production issues as a typical team using blocking operations and threadpools? Why do we have _more_ incorrectness caused by swallowed errors than I came to expect in projects that relied on exceptions?
I'm still keen on mastering this style of Scala, because I think the benefits can be had, but it annoys me that some of my teammates are happy to use those stated benefits to justify their programming adventures without seeming to care if they actually materialize.
"I'm on exactly that kind of team now. They've been working on a very simple problem for several years now and have an enormous, sophisticated, but still buggy and unstable solution.
They're really smart people, and they had to be extremely capable programmers to get this far, but the embarrassing question is, how would a team of mediocre developers have tackled this problem? They would have picked a mediocre language"
I don't mean to be rude to your colleagues, but almost by implication in what you've said, they're not good programmers. Being a good programmer is nothing to do with leveraging a fancy FP language. It's about delivering working, maintainable, testable code. Whether you write buggy spaghetti code in Go or misuse Scala, its the same root problem.
I've been a Scala dev for five years, worked on some massive projects and it's been a dream. As long as you agree on a style, don't go crazy with the language and use it with discernment, it's a joy to work with. It's a sharp tool though and it takes discernment to know how to wield it appropriately. I don't know if that's a downside, it's more a warning.
I think people can be good programmers and poor software architects at the same time. Sounds like that's what has happened in this case: brilliant abstractions that are misused and result in a buggy product and less productive team.
"I think people can be good programmers and poor software architects at the same time. Sounds like that's what has happened in this case: brilliant abstractions that are misused and result in a buggy product and less productive team."
That's fair. I guess if by good programmer we mean, fluency with the language and algorithms / abstractions /types etc then perhaps they qualify. I really meant something more like "software engineer" ie. someone who can architect a simple, sane, working solution adhering to the usual best practices of good software construction. Maybe there's a missing piece between being a good coder vs scaling this knowledge up to the application level. The latter is indeed a much rarer skill and too often, people without it influence major decisions.
I would guess that it's mainly a question of experience. When you have smart kids fresh out of school, they are going to be eager to use every tool in their toolbox. Eventually (hopefully) they will develop the wisdom to know when to deploy what.
Maybe the issue is that Scala requires a bit more wisdom that some languages. The problem is that the profession is filled with inexperienced people because of the rate at which new programmers are appearing. In that world you either want safe and, dare I say it, lowest common denominator languages, or you need more hierarchy where senior people take a stronger lead and set the rules. In other words, if you're going to use Rust, Scala, Haskell etc acknowledge the tradeoffs and potential footguns and insist that someone who knows what they're doing, leads the project in a very hands on way. Properly mentored juniors or mid level devs will have that moment of enlightenment where they see the point of these languages and don't just succumb to the blub paradox https://en.wiktionary.org/wiki/Blub_paradox.
The issue I've seen is that the help venues are overwhelmingly filled with language-"extremists" that have too much time and lead beginners and those looking for help into the rabbit hole. A bit like people that spend more time telling people to TDD-everything than coding...
Frankly, they are bad programmers. They are smart highly intelligent people who don't have either knowledge or aptitude or willingness to be good programmers.
My pet peeve in this business are supposedly good programmers who get praised despite never having actual results. And it is not like they would be rare.
Unfortunately the leetcode whiteboard interviewing strongly encourages this type. Even almost requires it. If all energy is spent on that, it is time not spent learning how to build stable products which can be maintained long term.
Because turns out that popping out algorithm exercise code after another into source control doesn't produce a good product. But that's what current interview fad measures, thus it's what it produces.
True, and with people switching jobs every 12 to 24 months (at least in the Bay Area) many of them never get to see the outcome of their choices, and don't learn from experience.
The companies in turn don't realize the outcome of their choice either and just consider the inefficiencies to be the normal status... I've never seen such a low productivity than in large companies with a lot of churn... Thankfully they have good people that can hold and patch the walls and are the real not-recognized "heroes"...
I have the same feeling with Kotlin. If you stay away from overusing DSLs and over FPing things (traduction, using Arrow or most of it) it is beautiful and easy to maintain and understand even by junior devs.
And my feeling is the same with Clojure and Scala, I'm not a specialist, but code can be written in such a way that I can even contribute to it.
I am fairly certain the jury is out on this question:
It's always better to write simple, almost dumb code that anyone can understand than to use advanced abstractions from category theory or whatever.
Why? Because every single study or anedocte I've ever heard is like yours: adding complex abstractions to code do NOT make it more reliable. But they do make it much harder to modify, understand, and fix.
FP tought us that unconstrained mutable state is very bad, and that has helped us immensely in the mundane languages of the world (Java, C#, C++, Go). But I think it's time to learn the other lesson from FP: overly complex abstractions are also very bad and should NOT be used willy nilly as they simply have a very low or negative cost-benefit.
Finding the right level of abstraction is basically the one and only skill that matters for a software engineer IMO. I've got about 15 years of experience in programming and it's still something I'm actively working on.
I disagree with the general take that "adding complex abstractions to code do NOT make it more reliable". Good use of abstraction makes code easier to write, easier to maintain and easier to extend. Good use of abstraction can make code more concise and more regular while at the same time allowing better diagnostics when you do something wrong.
As a quick example: a generic JSON serialization library that can work with any type is probably a lot nicer to use that one that requires manual reimplementation for every class in your program. It'll be more complicated to write but it's probably well worth it in the end. Similarly a logging system that can abstract over several backends and log levels depending on the environment probably beats having a bunch of if/else every time you want to log a message.
But one needs to remember the old saying: debugging code is harder than writing it, so if you write code that's as smart as you're capable of producing then by definition you're not smart enough to debug it.
You are completely right: finding "THE RIGHT" level of abstraction is extremely important. Without abstractions we would be writing code in Assembly.
This is the reason I used the term "overly complex abstractions", not just "abstractions" in general, of course. Overly complex meaning it's a failure to use "the right abstraction" where you instead go for something much more complex than the proble called for... this is what I understand most people in this thread are referring to when they refer to previous Scala projects they've worked on.
When I was young I though that good writers are the ones who write such complex sentences that nobody can understand. Later I learned that good writers are clear and understandable.
I used to write complex code because I found it elegant. Now I write simple code because I enjoy breaking down a complex problem into its simplest form, which for me means I fully understand the problem.
I agree 100%, and I always bring it back to one simple observation that has been true of almost all the projects I've worked on: the limiting resource that programmers work with is their own time and brainpower. To my mind, much of the discipline of programming is centered around making frugal use of that scarce resource.
I still think programmers should work with powerful tools, because "dumb" tools often force you to express things in convoluted ways, and you don't need a powerful language to make a huge mess. (Java proves that you only need single inheritance to produce virtually unlimited amounts of spaghettiness in real-world projects. The difference between Java and Scala is not that Scala enables teams to produce epic piles of FP crap, it's that we long ago stopped being shocked when teams produce epic piles of OO crap.)
I think the programmers doing terrible things with Scala would be doing terrible things in any language, given the chance. What they need is hands-on technical management. When they have a wild idea, they need to be walked through an appropriate engineering decision-making process. They need somebody with the authority to tell them, "Ha ha, yeah, that design with etcd and Kafka and the robot space lasers would friggin' rock, how cool would that be, but on the other hand, we could just stick a REST API in front of a database and you'd be done in two weeks, so let's compare pros and cons."
It does not matter how "simple" each line of code is.
What matters is how "simple" the whole application/solution is.
You can argue that you can write very simple code in Assembler - all instructions are very clearly defined and very simple (well, maybe not anymore in CISC...). Someone can learn them all in a day. The problem is just that now you end up with a lot of code and overall that will be hard to maintain.
Compare it to Javascript: certainly not the language where code is most maintainable or simple, but the end-result is easier to maintain then assembler.
In the end, the choice of programming language is like a the choice of a compression-tool like gzip. The more complex, the more difficult to learn and use (setting library size and various compression settings) but the output will be smaller compared to a simple tool or no compression (Assembler) while containing the same amount of information.
I write lots of F#, and I find that computation expressions (a fancy version of do-notation) makes my code more "dumb" and "simple", despite being an advanced "FP" feature. This is because it pushes the complexity from my business logic and into library code.
With computation expressions:
async {
let! foos = fetchFoos
for foo in foos do
do! launchFoo foo
do! clearFoos
}
Without computation expressions (this is probably wrong but you get the idea):
You should always reach for the simplest solution first (and Haoyi has a great set of guidelines about that: https://www.lihaoyi.com/post/StrategicScalaStylePrincipleofL... ). The point of those complex FP abstractions is to let you keep writing dumb code even when you want to do something fiddly (such as async operations with error recovery). But don't cargo-cult the complex solution if you don't have the complex problem! A plain function whose result is a plain value should be expressed that way.
> adding complex abstractions to code do NOT make it more reliable.
Sometimes false. If the (more) complex abstractions are in well-tested libraries that haven't been written solely for your code.
> But [complex abstractions] do make [your code] much harder to modify,
Not necessarily. You can separate concerns; you can isolate changes; and you hopefully reduce the amount of code you've written yourself, significantly even.
> [complex abstractions] do make [your code] much harder to understand
This is as likely false as it is true.
> [complex abstractions] do make [your code] much harder to fix
Again, this very much depends. If you're using a well-tested, widely-used external library with those abstractions, it may well be easier to fix your own code.
> the other lesson from FP: overly complex abstractions are also very bad
Lisp, in its persistent failure to achieve world domination, has been teaching us this lesson for over 60 years now!
Infinitely powerful languages are fun and, yes, powerful. But as soon as the project has more than one or two people in it, they also result in runaway complexity that will hurt the product far more than any gains in expressibility.
From the client's point of view that may be true; abstraction should be the process of pushing complexity down. But in this case I think we're talking about code that has inadvertently introduced unnecessary complexity in an attempt to be hyper-generic.
i think it depends if you need it, rust for example is quite complex, but it solves a class of issue that other language have trouble solving while maintaining speed
Reading this thread is giving me some catharsis. I worked on a large project in a big tech company where the org’s leadership was ideologically determined to build everything in Scala. We ended up with dozens of engineers and failed to deliver basic functionality. The leadership eventually left the company and now they are running a startup with a few of the Scala savants from the old team. They got lots of funding and they invited me to visit to get me to join. They showed me the awesome type system they had implemented and tried to demo it but the product was broken in five ways.
I don't think it is a Scala problem but more of a leadership/organizational problem. If you look at Jane street, they do everything in OCaml which is an interesting choice to say the least, but their stuff works, evolve and they can demonstrate...
Yeah, I don’t think Scala itself caused the problem. There was this mindset: “if we just properly model the universe in the type system, the solution will just fall out!”
Because that was the obsession, they ended up spending all their time iterating on this perfect type system and not enough on the product. I think people with that issue are more likely to want to use Scala.
I am continually perplexed at the swallowing of exceptions in certain FP communities.
Result types, Either et cetera make it extremely easy to swallow errors by flatMapping thoughtlessly losing the context of where they were - you typically /want/ the call stack when you hit into an exceptional flow.
As I am slowly acquiring the habits of thought that make it possible to read and write FP code at a reasonable speed, I'm horrified by how much idiomatic Scala FP code relies on projecting certain assumptions onto types and operations that seem, at first glance, to be neutral mathematical abstractions.
For example, I find myself hating the Either type, because I feel like there is a socially established convention that one half of the Either is the type that matters, the value that you want, the value that is the point of the computation you're doing, and the other half is a garbage value that should never be directly handled. So I really feel like I should conform to the convention and reserve Either for cases where one of the possible types doesn't matter. But how often is it true that one side of the Either doesn't matter? People want me to encode success/failure in an Either type, but if I do that, are they going to treat failure with the care it deserves?
I often handle Either (and Option) using pattern matching when I feel it's important to give both code paths equal importance and equal visibility in the code, but people change it because flatMap is supposedly more idiomatic, and they believe that eliminating pattern matching from their code is a sign of sophistication.
I feel like this stems from a strong desire among FP folks for the happy path to be the only one visible in the code, and the non-happy path to work by invisible magic. Maybe there are some brilliant programmers who achieve this by careful programming, but there are people mimicking them who seem to rely more on faith than logical analysis. They just flatMap their way through everything and trust that this results in correct behavior for the "less important" cases.
I'm sorry that this turned into a bit of a rant, but I'm entirely fed up with it, and it accounts for a lot of what I dislike about the code I work with on a daily basis.
This was great, actually.
I don't program in Scala, but it was very interesting to hear about the difference between types as abstractions vs types as they are used.
For unfamiliar topics or when presented with uncommon insight, I believe rants, monologues, even diatribes are actually some of the best things to read.
> For example, I find myself hating the Either type, because I feel like there is a socially established convention that one half of the Either is the type that matters, the value that you want, the value that is the point of the computation you're doing, and the other half is a garbage value that should never be directly handled. So I really feel like I should conform to the convention and reserve Either for cases where one of the possible types doesn't matter. But how often is it true that one side of the Either doesn't matter? People want me to encode success/failure in an Either type, but if I do that, are they going to treat failure with the care it deserves?
There's always a tradeoff between making the happy path clear and making the error handling explicit. The whole point of Either is to be a middle ground between "both cases are equal weight and you handle them by pattern matching" (custom ADTs) and "only the happy path is visible, the error path is completely invisible magic" (exceptions). Given that people in Python or Java tend to use exceptions a lot more than they use datatypes, I'd argue that a typical Scala codebase puts more emphasis on actually handling errors than a typical codebase in other languages.
Where each case really is of equal weight, consider using a custom datatype (it's only a couple of lines: sealed trait A, case class B(...) extends A, case class C(...) extends A) rather than Either.
I've been working through 'Haskell From First Principles', and it turned on a lightbulb: the 'right' half of Either is the 'important' one because its type variable is free to change.
instance Functor (Either a) where -- a is fixed here!
fmap :: (b -> c) -> Either a b -> Either a c
fmap _ (Left l) = Left l
fmap f (Right r) = Right (f r)
As a general rule, the last type parameter of a type carries special significance: the same applies to Tuples.
You _can_ trivially construct a type where the two labels are swapped; it's just labels, Left and Right aren't intrinsically important, except insofar as they reflect the positions of the type arguments in written text.
This is also the reason we have the convention in Scala as well - the inference to partially apply the type works in a certain way. But I agree with the parent post, a more descriptive name would be better.
Right; ostensibly you could create a language that lets you easily poke holes in any slot of a type, but I'm not sure you necessarily /gain/ a lot in doing so except for confusion. It would take a lot more convolution to specify types and instances for every function application.
> I often handle Either (and Option) using pattern matching when I feel it's important to give both code paths equal importance and equal visibility in the code, but people change it because flatMap is supposedly more idiomatic, and they believe that eliminating pattern matching from their code is a sign of sophistication.
Isn't this the same as letting exceptions bubble up in a non-FP language?
You don't necessarily lose the stack trace. Typically the left side of an either is an Exception (or an error ADT that wraps one). When you want to handle the left case, you can log out the full trace as you would without Either.
The Monad instance for Either means that chaining them together with flatMap has a short-circuiting effect and the first failure will stop the rest of the chain from being evaluated. I find this actually makes it easier to know where your errors are happening, and also allows you to centralise your error handling logic.
Sure - you can implicit srcloc or capture the Exception, both of which preserve it; but that's not the default behaviour and it's not what we recommend to beginners.
If you go onto the scaladoc for Either today, you see a stringly-typed Either where they discard the Exception.
Hmm, when I first learnt Scala, I haven’t had too advanced FP knowledge, so I am yet to have first-hand experience with this sort of exception-handling and I’m yet to decide how good it is.
Compared to Haskell, it is probably better in some way because you have the proper stacktrace; but it “feels” impure a bit..
In a way Java’s exceptions are already an Either type with the result type and the thrown Exception (with “auto-decomposition”, unless checked exceptions) —- is the advantages like manual management of when mapping/flatmapping happens worth it in your opinion?
Nonetheless thanks for the heads up, I might try out Scala again with the exception handling model you mentioned!
It's supposed to work like that, but it's a lot easier to screw up. Smart people screw it up all the time, and it's hard to spot in code review, whereas average programmers have no problem avoiding swallowing exceptions once they realize it's important, and if they do mess up it stands out like a sore thumb in the code.
You shouldn't swallow errors like that in FP either. You can trap the error in an effect type and throw it at the top level of your app when your effect type gets run after all of the code that processes that error value has a chance to recover. See something like Zio for an example, though you can do similar things without Zio.
If the team uses Zio, or maybe they’ve gotten lost in the religious war of Scalaz, vs Cats, vs Zio... One of the major issues I have with Scala is how fragmented the community is. Everyone has an opinion on how something should be done and everyone thinks that everyone else is wrong.
Well you have extremist who believe that you have to cull the community due to a few bad (but still contributing) apples. Additionally you have a big lightbending company that produces some questionable contributions to the community as if it's the "scala-endorsed way".
Don't get me started with the whole forcing of "lets be python like" initiatives as of late.
we use error types that extend Exception specifically for the purpose of getting a call trace. A lot of people hate doing this but it's been extremely helpful on our project
We do this too - but the other problem you get is that Exception aren't treated specially (like Any and Nothing) in the inference hierarchy so errors are liable to "collapse" to Throwable or Exception. I like wrapping them with a new type if possible to stop this behaviour.
It makes the code impure (that is with side-effects). One can also argue that if you can encode the potential errors in the type they are not exceptional.
It's not that bad though, because you get much less of these kind of errors than in a typical Java program (for example).
Yes, it is a side-effect - and does make the code impure; but it's an /exceptional/ flow like OOM, infinite looping, stack overflows which also break equational reasoning.
My argument is that as a programmer you /choose/ the base lemmas which you're comfortable with - with an exception you're saying for a large swathe of your code, it will assume a lemma. When is /does/ break you get to point the finger at it with a stack trace saying here is where the lemma was broken. As an aside there are nicer ways to do this ala contracts, but exceptions aren't a /bad/ way.
The equivalent of try.toOption is try catch everything and discarding. The equivalent of flatMapping without adding surrounding context is equivalent to try catch rethrowing. Both of these are /much/ too common in FP codebases.
This is spot on! Way too often in Scala codebases, which tend to wrap IO calls in Future or similar is that you'll have chains of side effecting futures like
And this all gets passed up the call stack to one generic thing that basically does nothing in the case of error, maybe logging it and that's it. It's tempting to write code like this because it's so easy and looks so clean and readable, but in reality it is not how you build reliable software because what one must do in response to a failure from the first call is not the same as what one must do for a failure of the second call, or third or fourth, but this flatMap().flatMap().flatMap() style seduces the programmer into thinking they've handled errors when really they've just lumped the entire workflow as one big chain that can fail at any point and in practice usually just ignore all error cases.
One strategy I find does slightly mitigate this is to use future.transformWith[S](f: Try[T] => Future[S]). This at least presents to you the opportunity to think about error handling a bit more consciously, but I still find that the chaining operator approach nudges you away from thinking about error handling because it becomes quickly difficult to read.
It's probably to do with how limited the control structures are when dealing in Futures. You don't have while/if-else/etc, you must lift all your control into flatMaps and iterations are only doable via recursive calls with explicit accumulator state (yuck!) and nesting. So whereas a properly thoughtful treatment of error cases in synchronous code may take 20 or 30 lines of legible code, this gets tranformed in the async style to 20 or 30 levels of nested callbacks and recursive calls which becomes unreadable.
Amen. I saw exactly the same problem with over-reliance on flatMap, either explicitly or in for expressions, in future-heavy Akka code that I do in cats and cats-effect code.
People love the unifying abstractions underneath these types, and they love developing instincts about how to write code based on them, but from an application programming point of view, their instincts are often counterproductive. I don't think people take a mathematical enough viewpoint. The abstractions can't tell you what is important and unimportant. They can't tell you how your code should be shaped. They can't tell you which values should be transformed further and which should be short-circuited. They can't tell you which values deserve to be given a name for readability and which values should be anonymous.
"All these values are monads, so I can combine them with a for expression" is a meaningless statement of a trivial mathematical fact, not a clue about how you should write your code.
I think error handling code can be written in a straightforward style, but it isn't as pretty as people would like. I think the trap they fall into is holding onto elegance while they reach for correctness, instead of holding fast to correctness while they reach for elegance.
That's a great observation about for-comprehensions. It's an other one of these cute niceties that in the most basic cases can come in handy but again nudges users away from consciously thinking through what they're doing (good luck handling the error path in a for-comprehension over Futures).
I dunno. You see the same pattern with chained class methods in Python and it suffers from the same exception specificity problems, but doing it is still a fair judgment call. Sometimes all you need to know is that there was an error in the chain. Like if an exception is raised by dataframe.transpose().to_dict().values() I don’t ask myself where in the chain the error occurred, I just think, “oh that’s weird why couldn’t I turn that into a list of dicts? Did I call those methods on the wrong object?”
Your point's well taken. I'd say that there is a bigger issue in the case of flatMapping side-effectful monads because they're typically dealing with things like writing/reading from databases and the error cases should be thoughtfully considered (do I have to roll something back or perform compensating actions, or clean up anything?)
Doesn't Scala have transformer libraries like ExceptT in Haskell. Aside, much of what this comment thread is about, is similar in Haskell-land as well.
Anyway, a pattern I've started to use in Haskell with ExceptT is along the lines of
So the code still remains legible (minus the plethora of Haskell operators, data wrappers and unwrappers :)), I can trap individual exceptions along the way, which I can handle. And at the end unhandled exceptions are returned by the function as Left.
where each method just returns Unit or throws exception? The version with IO/Future is superior since it at least explicitly states that you can get an error here. If you want to say that Go-style error handling is better because if forces errors handling, I can kind of buy it, but it also has some cost.
>You don't have while/if-else/etc
Maybe you are looking for ifM/whileM functions from e.g. here: typelevel.org/cats/api/cats/Monad.html.
The call stack is "magic" built into the language, so as soon as you're not using the "blessed" way of error handling, you lose it and need to rebuild the same functionality "by hand".
I agree that some kind of logical call stack is a very useful thing to have, and I'd recommend implementing something along the lines of https://github.com/lancewalton/treelog that provides it.
How do you mean? If your either turned left you should be stuck with that exact left and no amount of flatmapping should effect your stack, it simply does not kick in unless you added something like leftmap that swallows the exception
> how would a team of mediocre developers have tackled this problem?
In my experience, mediocre developers would stick to a simple language like Python and only use simple Python features... and write a completely impenetrable, 100% coupled and wholly unreadable mess that would make you wish for over-engineered Scala. Don't even get me started on what happens with "simple" distributed computing tools like Hive...
I see people complain about languages like Scala leading to over-engineering and I just don't get it: have you not worked with sub-par Python or JavaScript code? Sure, poorly designed complex solutions have problems, but so do under-abstracted codebases! Bad code is bad code, and I don't think it's meaningful to say that "over-engineering" is categorically worse than "under-engineering".
I actually worked with somebody who wrote pathologically over-complicated Haskell. Had some friction between us over that. His Python code? Somehow even worse.
No serious language I've seen at either end of the abstraction-friendly spectrum (not Haskell/Scala nor Go/Java/Python/etc) puts a meaningful floor on code quality. An inexpressive language can have a low ceiling for how good code is, but restrict or simplify the language however you like and people will still happily write utterly unworkable code.
I've found that languages—especially less common languages thanks to the "nobody gets fired for IBM" effect—get used as scapegoats for not talking about cultural issues. If people on the team are writing clearly poor, over-complicated and bug-riddled code, it means there is some sort of technical leadership shortcoming, and it would not be any better if they were using Java instead. (I mean, have you seen Enterprise™ Java™ codebases?) And, at the same time, it's clear that a Scala codebase with tasteful technical direction and leadership—conveyed through team culture, code reviews, shared expectations... etc—can be absolutely great to work in.
At this point, I'm leery of blaming languages for programming problems without a clear mechanism. "The language lets people do bad things" isn't a real mechanism because all languages do bad things. "The language attracts people who like complexity" seems specious too. (Again, let's not blame languages for hiring and cultural problems!) "The languages defaults incentivize poor code" is a better argument, although it can be a bit fuzzy. And arguments like "language A doesn't have capability B that we need" or "language C allows classes of bugs other languages don't which empirically cause problems" are the most compelling of all.
>I was on a team building good old crud apps using monads, monoids, categories
Here's the thing, if you look at a basic Spring crud app, you are also using Monads, Monoids, Categories, Traverse, etc. but you aren't expressing it in the type system. Seriously go look at modern Spring's flux stuff, it's all there minus the type classes.
I've seen teams that tried to over engineer Spring, teams that tried to over engineer Node applications, and yes there are teams that over engineer Functional style Scala.
All the teams I worked with (and managed) at Verizon leaned pretty heavily into FP style Scala without much over engineering and the experience was extremely pleasant. The only production issues in my three years there I remember were performance related, finding out how to get more throughput or lower latency out of FP Scala. I literally can't remember any 'bugs' that made it to production.
Yeah, I'd say that Spring is the Java equivalent of the same problem (over-engineering), just using reflection and runtime bytecode generation instead.
I agree that using Monads, Monoids, etc. isn't necessarily indicative of over engineering in itself. If used well they can make the code clearer/simpler.
I'm postulating that any modern CRUD app framework has Monads, Monoids, and other categorically inspired structure, even if they don't call it that or have a way to abstract over it.
If it isn't called that and doesn't have a way to abstract over it, there's a very good chance that many of the logical Monads/Applicatives/Monoids/etc. are subtly not implemented as those things, resulting in equally subtle bugs.
In the Spark world, you can use a tiny subset of the Scala features and enjoy huge productivity gains over the other language APIs (Java & Python). Those productivity gains are wiped out as more crazy language features get used.
I don't think the super complex language features should be removed. Li's libs do some crazy stuff under the hood, but provide a clean, Python-like public interface. Most devs aren't that good and complex underlying implementations leak and yield complex public interfaces.
Lots of folks would love Scala codebases that only use 10% of the available language features and none of the complex frameworks. But, like you mentioned, it's a hard language to use responsibly.
I think the trick to using Scala is to read the high level/key features of the language (case classes/pattern matching/using Option/Some/None, type aliases etc) and basically stick to them. A good yardstick for what are those features is the Scala Book[0]. This is what I read before I started to actually code in Scala.
That coupled with side-effect free functional style (as much possible without overstretching) uses the power of Scala and keeps the code concise and readable. That is how I use it and find it very pleasurable to write in Scala compared to Java.
> Lots of folks would love Scala codebases that only use 10% of the available language features and none of the complex frameworks. But, like you mentioned, it's a hard language to use responsibly.
Does this have the same problem in large languages such as C++ though? Where everyone thinks there's an optimal subset of the language, but no one agrees on what that optimal subset is?
For me, Scala is a multi-paradigm language with tons of features. The community just needs different types of style guides for different apps. vars are horrifying for the functional crowd, but cool with me for example.
On the other hand Kotlin devs can also be really conservative in what they let creep into the language. But that's what I like about the language, it doesn't try to please everybody and that create a somewhat more rigid frame in which I'm more comfortable expressing myself.
This is my biggest gripe with Scala as well. I'm sure it's a great language, but I've often seen it used to add complexity where none is necessary. It's abused to give developers a sense of accomplishment and intellectual superiority.
In fact, in one project my former employer was involved in, one main reason they picked Scala was to, on the one hand, weed out the chaff from their existing team of .net developers (in a "shape up or ship out" kind of fashion), and on the other to weed out the 95% of mediocre Java developers.
>...I've often seen it used to add complexity where none is necessary.
This reminded me of a great blurb about Niklaus Wirth's approach to languages:
>Wirth’s philosophy of programming languages is that a complex language is not required to solve a complex problem. On the contrary, he believes that l languages containing complex features whose purposes are to solve complex problems actually hinder the problem solving effort. The idea is that it is difficult enough for a programmer to solve a complex problem without having to also cope with the complexity of the language. Having a simple programming language as a tool reduces the total complexity of the problem to be solved. [0]
Scala is a very simple language at its heart. Unfortunately that empowers developers to write really complicated libraries, leaving application developers in much the same place as if they were using a language with complicated features. (Most application developers can't even tell the difference - most complaints about "Scala is a complex language" turn out to be "I was using a complex library in Scala")
> You could've easily mistaken our team for a programming language research group at a university.
I had the misfortune of working with basically this same team, on scala (of course).
I argued for maintainable simple technology but scala was too hip to pass, for many.
Still remember one meeting trying to decipher a bug and while reviewing the code in question the lead scala fan said "It is not reasonable to expect to understand what code does by looking at it. I'll need to go research this for a few days."
I hope to never see scala (or its ilk) ever again. Give me the most stable language and environment, where every question is a FAQ and every odd behavior is documented in every book and I never need to fall into the rabbit hole of language traps. Then I can just work on building the product, which is the whole point.
> I was on a team building good old crud apps using monads, monoids, categories, combinators, effects cats, seamless and bunch of other nonsense that i've now purged from my brain.
Ah I had the same experience.
That this is allowed to happen just shows that putting naive non-technical managers in charge of coders is a disaster.
I don’t know of anyone at those companies using clojure and I haven’t seen any tech blogposts by those companies about using clojure. If clojure exists at those companies I imagine it’s a small very niche team.
One very public example of a Clojure shop through and through is NuBank out of Brazil, who employs some 700 "Clojure developers" according to them. In fact they are so committed to Clojure they bought Cognitect last year.
I have no gripe with Clojure, I take issue with someone claiming that Amazon, Apple, and Netflix have systems that operate at scale because of it. Clojure is a fine niche language, but it’s not the secret sauce to big tech web scale.
I was not claiming that, just that also big companies use it. It may not be used for the core of the business but I'm convinced the services that are build with Clojure are reliable, because the language is solid.
> Hard to win technical arguments with Scala geniuses that like using complicated language features.
This is an interesting point. Perhaps part of the reason my current employer has been successful with Scala has been that we never were integrated into that part of the community.
We hire non-Scala programmers and they write Scala without any training, and it generally turns out OK and ends up converging in a pretty boring style without any fanciness
Yep, the Spark codebase is a great example of what a big Scala codebase should look like. Nothing crazy, not too many traits, mainly just "regular functions".
This is a false dichotomy. You'll run into all sorts of pain and frustration as soon as Spark touches your codebase, no matter what kind of Scala you're writing.
I've almost every big data processing system out there and Spark causes no more pain than any other. Less than most. It's also a data processing system not a library so if you're integrating it into existing code (rather than writing code for it and running code on top of it) I'd argue you're doing it wrong.
One big reason, maybe the biggest, to write Spark jobs in Scala and (not move to pyspark) is code reuse between different components. My team maintains a fairly sizeable library that is common to our Spark jobs and several web services. Spark is a library dependency if you do anything remotely complex with it and it can easily creep up everywhere if you aren't careful.
Decoupling modules isn't always obvious in a codebase that's grown organically for 6-7 years now (long before I joined) and the cohabitation with Spark is inevitably going to cause some pains. A couple examples:
- Any library that depends on Jackson is likely to cause binary compatibility issues due to the ancient versions shipped with Spark. Guava can be a problem too. Soon enough you'll need to shade a bunch of libraries in your Spark assembly.
- We have a custom sparse matrix implementation that fits our domain well, it was completely broken by the new collections in Scala 2.13. It makes cross-publishing complicated if I don't want to be stuck to Scala 2.12 because of Spark.
After a number of years with Python, I started a position working in Scala about 6 months ago. I really wanted to learn something new, and become acquainted with functional programming concepts.
I agree with most of the above. A couple of additional thoughts:
* sbt -- I still have a lot of coming up to speed to do here, but the manual is like 500 pages and it's somewhat overwhelming. There are tons of little oddities, like why can't I run `sbt --version` and instead have to do `sbt sbtVersion`?
* The functional side is fascinating -- I'm still studying the cats library. I can almost describe a Monad! It's a pretty big mountain, though, and there are times where I have doubts whether the benefits will be worth it. Would love to hear some re-assurance! ;)
* The ecosystem for microservices seems pretty closely tied to akka & lagom. These are quite complex in their own right and we've been having trouble with the latter in particular. Curious to learn about alternatives. ZIO?
* Re: DSLs. Also not a huge fan of DSLs. One refreshing thing about python is that often configuration can just be in Python itself (as in Django, for example). See also:
> sbt -- I still have a lot of coming up to speed to do here, but the manual is like 500 pages and it's somewhat overwhelming.
SBT is not worth it. Ignore it and use Mmaven.
> The functional side is fascinating -- I'm still studying the cats library. I can almost describe a Monad! It's a pretty big mountain, though, and there are times where I have doubts whether the benefits will be worth it. Would love to hear some re-assurance! ;)
You shouldn't use these things unless and until you need them. Cats is basically a library of techniques for letting you accomplish things that seem like they might need language features by instead writing plain old functions that return plain old values. If you use the fancy technique for the sake of using the fancy technique, you're putting the cart before the horse. You should use them where you'd otherwise have to use some weird language feature (exceptions, magic async, mutable variables...).
> * The ecosystem for microservices seems pretty closely tied to akka & lagom. These are quite complex in their own right and we've been having trouble with the latter in particular. Curious to learn about alternatives. ZIO?
Mostly you don't need anything too complex. I'd recommend using akka-http to start with, but don't use any akka proper - stick to the routing DSL level and use futures rather than actors. When you're more comfortable with the functional abstractions you can switch to http4s.
I learnt Scala before diving deeper into FP, but I think properly learning/understanding Monads and some other FP concepts greatly help every sort of programming I do (even though it is mostly OOP nowadays). You come to notice it everywhere and even if you can’t abstract over them (there is no one `bind` or `join` function, so it may be called different with each Monad instance), but after understanding them you will gain a better reasoning power over them.
> It's a pretty big mountain, though, and there are times where I have doubts whether the benefits will be worth it. Would love to hear some re-assurance! ;)
It’s not worth it, the entire language is a mountain of documentation and hard to understand concepts that just gets in the way of actually delivering product features for the business. It will help you to think differently about programming problems though.
Let's clarify some points for folks not so familiar with Scala.
> * Scala minor version are binary incompatible, so maintaining Scala projects is a big pain. Upgrading Spark from Scala 2.11 to Scala 2.12 was a massive undertaking for example.
Scala just chose a strange naming scheme. Other languages would have just increased their major version instead. The scala minor version is increased every few years and not every month or so.
> * Scala has tons of language features and lets people do crazy things in the code.
Actually, that's not true. Or rather: compared to what language?
Scala has surprisingly few language features, but the ones it has are very flexible and powerful. Take Kotlin for example. It has method extensions as a dedicated feature. Scala just has implicits which can be used for method extension.
> * Scalatest is stil used by most projects and is annoying to use, as described here: https://github.com/lihaoyi/utest#why-utest. The overuse of DSLs in Scala is really annoying.
I agree with the overuse of DSLs. Luckily that got much better, but older libraries like scalatest still suffer from that.
> * Li's libs (os-lib, upickle, utest) have clean public interfaces, but most Scala ecosystem libs are hard to use, see the JSON alternatives for examples
I think that just comes from using the library in a non-idiomatic way. In most applications, you will need to use the whole json anyways, and then you use (or can use) circe like that:
{
"id": "c730433b-082c-4984-9d66-855c243266f0",
"name": "Foo",
"counts": [1, 2, 3],
"values": {
"bar": true,
"baz": 100.001,
"qux": ["a", "b"]
}
}
case class Data(
id: String,
name: String,
count: List[Int],
values: List[DataValue]
)
case class DataValue(
bar: Boolean,
baz: Float,
qux: List[String]
)
import circe.generic.auto._
import io.circe.parser._
val data = decode[Data]("...").get
data.copy(name = data.name.reverse).asJson
Yeah, that is more code, but as I said, in the vast majority for projects, you need all or most of the fields anyways, so all the structure definition is a one-time thing.
The advantage is that the last line is plain Scala code. You don't even need to understand the json-library to do transformations and re-encode into json.
> Scala just has implicits which can be used for method extension.
I don't think it's fair to say it like this. Scala's implicits can mean different things, depending on where they're used. Scala 3 even divides 'implicit' to multiple keywords.
(kind of 'static' in C++ I guess, only more complicated)
Scala has one feature (implicits) but it can be used ("mean") for different things.
Essentially, you can mark definitions as implicit and you can mark parameters as implicit. Yes, Scala 3 uses different keywords to make it easier to understand which is what, but both is still just the concept of things being implicit.
Think about it: one without the other is completely useless. If you cannot define implicit parameters, then marking any value as implicit will not have any effect. The other way around too: you can mark your parameters as implicit as much as you want, if you can't define implicit values, you will always be forced to pass all parameters manually.
Even implicit classes (excentions) are just syntactic sugar for regular methods that are marked implicit.
I’m not a Scala programmer so I don’t know who is more right here, but _ai_ was saying that calling three different features by one name does not mean there’s really one feature. Which is different than saying one feature can be used in three different ways.
The C++ static example was used because in that case the same keyword was used for several literally different features to avoid adding additional reserved words.
> Scala has tons of language features and lets people do crazy things in the code.
In many ways, it's even worse than that. Scala has exactly two features (implicits and punctuation-free method calls) that allow you to build massively complicated libraries that pretend to be language features, and which work worse than equivalent features in languages that support them explicitly. A good example of this is typeclasses which are core features of haskell and rust, but are implemented by explicitly passing implicits (hah) in Scala.
Right. We don't have a full production 3.0 release yet, so I tend to think of "Scala" as Scala 2 unless it's specifically qualified as Scala 3.
Scala 3 seems to be a really good step in the right direction for me. There's a clear desire for many of the features that are currently achieved through burdensome hacks on top of implicits, and actually reifying several of them into core language features is a counterintuitive way to make things simpler rather than more complicated.
> Scala has tons of language features and lets people do crazy things in the code. Hard to win technical arguments with Scala geniuses that like using complicated language features.
I also like(d) a lot about Scala, but this is what ultimately led me to other languages. There's too many language features cranked together. This makes a) for a steep learning curve and b) exposes you to expert's code that is so dense and 'smart' that its a PITA to decipher what it does.
b) you'll encounter especially when incorporating libraries that are not fully stable, and you have to track down bugs in them (if only to know if your code is at fault, or the library's code).
I had this with Akka when it was just released. One line of code and 2hrs of debugging to determine what it did, and if it was faulty or not.
I can agree with all your points (wrote a fair bit of Scala all though I wouldn't identify as a scala programmer).
Scala is a weird case, I would normally be the perfect fanboy for it, I like functional programming, did my fair share of SML, Haskell, Clojure. Also I am not at all dogmatic and can even find joy in writing Java.
However, Scala and I never got along well. It is - I think - too magic. I'd even prefer pure Java over it tbh.
And then, the community is fairly toxic, which also made me not want to stick around.
> The overuse of DSLs in Scala is really annoying. Too many DSLs is another example of something I consider to be an antipattern, but there is no Scala community consensus on the responsible use of DSLs.
Domain specific languages are heavy users of the implicit keyword and implicit conversion; maybe that makes the code more concise, but it doesn't quite help with reading / understanding the code. For me that's the greatest problem with scala - figuring out what this code in front of me does.
The way I tend to think about it is that there are different axes of "complexity" in a language/ecosystem. Looking only at the language itself then yes Scala is quite complicated with a lot of language features (implicit, higher-minded types, macros, ...) that are uncommon in other popular languages. But another axis to consider is "how many abstractions do I need to understand in order to grok a large codebase?" And that I think is where Scala shines. There are different ecosystems but if you look at something likes cats/cats-effect then there are a core set of type classes on which everything else is built and if you understand those core abstractions then understanding any codebase which uses them is very easy. And those abstractions are extremely flexible to the point where they can be applied to an astonishing variety of problems.
Contrast that with something like Java which, while quite simple syntactically, has a huge and varied super-structure of incongruent abstractions on top of it. Every time I dive into a largish codebase of "enterprise Java" it feels like the engineers has a running bet on who could use the most GoF patterns in any given module.
I think you have something here. Scala is complez, but it lets you "push down" a lot of application-level complexity into the language where it can be better managed snd understood by tooling.
For example,
- Scala singleton objects add complexity footprint to the language, but reify a super common pattern in Java and ensures everyone does it the same way.
- Case classes add yet more resource footprint, but again reify a common pattern that in Java-land is fragmented between Beans and POJOs and other things.
- Named and optional parameters add complexity over Java's simple method calling style, but subsume a whole zoo of builders, overloading, telescoping and other patterns used to work around their absence
Each of these features certainly makes the language more complex, but arguably at the same time they make user code more simple and boring
good point! on the other hand java got some functional features since java8 (lambda/streams/option chaining) that allow to have less bloated code; i think nowadays it is (at least) possible to have a less design pattern heavy code style in java.
True. Java has improved substantially since java8 (including something kinda sorta like case classes with record types in jdk15) but I still think Scala is miles better. I don't think it is generally appreciated how essential higher-minded types and typeclasses are to making functional programming tractable and safe. Perhaps because it is something that is generally hidden away in libraries and developers working on actual application code (i.e. 99% of scala devs) never use them.
I have used python and scala for work and I find myself wishing to be able to use scala a lot when I use python (especially when using pandas or trying to multi process/thread). Not so much the other way around even though I also love python and think it’s amazing. Main reason: The type system is great and there are so many things that I can express/enforce using it that are missing for me in python even using typing libraries.
Yes, there is a large crew of FP fans around scala that can be obnoxious at times, but they have also brought a lot of good stuff around. The way I see it: You need groups that “overshoot” and then settle on the middle ground. Compromise...
I use FP when it’s not too complicated, but stay away from the larger monad systems that are hard to explain to newcomers (effects/IO/Kleisli etc) because I honestly feel like it’s too much. Let me just run my debug logger when I want to... (yes I know that’s not what it’s about)
I probably fall in the category of using scala as the (way) better Java with sprinkles of FP and non-mutable data objects everywhere. I really like the authors style (his work is awesome in general and I point to your posts a lot for our juniors, thank you for your contributions if you read this)
It has its flaws, just like any other language, but it’s also incredibly expressive and fast (enough) for the backend and data engineering that we do
Your point about the FP crew being seeming obnoxious sometimes is true. The style itself can be great, I use IO daily and the monadic style is the glue that holds our apps together. The difference with my team, or approaches that work generally, is that we don't go on and on about these abstractions. I barely mention the 'M' word and just talk about chaining things together. Once that is appreciated then IO isn't that hard to get a basic idea of either. We talk a bit about lazy evaluation but I try not to freak everyone out by mentioning purity, higher kinder types and functors every second breath. It's enough to gain a working intuition for these things without bringing theory to the foreground. That can come later if / when ppl are interested.
I guess we also choose our style and abstractions carefully and only use things when we can justify it as a team. If someone raised a PR full of novel concepts that we hadn't grokked as a team, it might well get rejected. For example we started using monad transformers a while ago only after the concept and benefits in our code was explained and bought into by the team. This doesn't just apply to Scala but is especially important when using it. Reach consensus with your team about the style and evolve it together.
I get the feeling that other language cultures, maybe Rust being one, are good at just using FP, without talking endlessly about it.
My favorite thing about the language is its flexibility, if i want to have pure imperative java style code i can do it. I dont think everyone should blame it just because there are so many haskell fans. I think ZIO is a good sign
Yeah the extreme FP stuff works for some people, but not others, and that subcommunity has traditionally been very loud. I've been pushing hard on using Scala in a more hybrid way over the years, first with my open source libraries and then with my book Hands-on Scala. I think I've had some success in pushing this.
There's definitely been a shift in the community in this direction. While the hardcore-FP folks are still around, as are the hardcore-Reactive folks, there are now many others using Scala in this more simple, simplistic way to good effect. Scala is a big tent and every
subcommunity can find things they like even if they don't always agree
In my view though, Kotlin is just a much better version of that. But of course if you have lots of scala investment already, it makes more sense to do it within scala itself.
It comes with the same easy Java interop, but is a much smaller simpler language, while in my view managing to include essentially all of the most useful ergonomic improvements (over Java) that Scala does, but with essentially none of the more complex baggage.
I definitely agree that many newer languages are strongly inspired by Scala, but I don't see a strong argument for why to choose Scala over one of a newer hybrid language. My own intuition is that these newer languages have had the benefit of being able to learn from Scala's mistakes. A lot of them have intentionally sacrificed some of the Scala's flexibility to be easier to learn and use.
I found Scala to be an extremely enlightening language: it taught me a lot about functional programming. But, the complex type system and the sacrifices needed to ensure Java compatibility entail many quirks that I find frustrating. I think it's actually a similar dynamic to what you mention: "retro-fitting these features onto an existing language never fits quite as nicely as a language that was designed with them from scratch."
BTW, I am a huge fan of your writing: the Principle of Least Power is a sacred programming text to me .
> I definitely agree that many newer languages are strongly inspired by Scala, but I don't see a strong argument for why to choose Scala over one of a newer hybrid language. My own intuition is that these newer languages have had the benefit of being able to learn from Scala's mistakes. A lot of them have intentionally sacrificed some of the Scala's flexibility to be easier to learn and use.
I don't believe any of those new alternatives has made better choices; some of them haven't had time to accumulate as many warts as Scala yet, but all of the ones I've seen have design decisions that make them inevitable. Higher-kinded types still work much better than every other way of achieving the same things that's been found - and if you want both higher-kinded types and decent tooling/library support, Scala is still pretty much the only option. (Haskell exists, but even if you consider its tooling good enough, implicit pervasive laziness has huge costs).
>the sacrifices needed to ensure Java compatibility entail many quirks that I find frustrating.
That reminds me of that dead JVM language called Ceylon. It didn't compromise anything to ensure Java compatibility. The combination of OOP+FP was executed with zero friction. Meanwhile Scala did it so poorly it created a stupid myth that OOP and FP shouldn't mix.
Of course everyone knows that nobody used Ceylon, not even Redhat who sponsored the language. The compiler was also dog slow. Typechecking complicated type unions/intersections can get really slow with bigger projects. The metamodel also created a lot of JS bloat if you were brave enough to use it in the browser. Lots of practical complaints that ultimately make you stop using it. But oh man the idea and design. It was very well thought out and enjoyable to program in.
Scala has learned from Scala's mistakes too. Scala 3 cleans up a lot of the mess that had accumulated over the years and is on the cusp of being rolled out this month.
I feel like Scala started a trend in language design, where every language adds features from other languages. Not because there's a need, but because some people are used to language X.
Example: adding OOP in PHP and Javascript. In both examples, especially the first attempts, were half-baked. Why bolt on half a language feature? Both languages would be much better served with dependency management and / or modules, which for both languages came out of the community at first.
Counter-example would be Go, that resists change - especially if it's requested with a "this feature is in language X! I NEED it!". From the FAQ: https://golang.org/doc/faq#Why_doesnt_Go_have_feature_X (followed by some features other languages have. Generics / type arguments is one that will probably be added in an upcoming version of Go, but they only considered it once they fully understood the problem and need)
JavaScript was OO form day one. There aren't even much languages that are so consequent OO like JavaScript. Everything is an object even functions.
JavaScript is actually quite similar to Self (besides the syntax), which itself is a kind of a SmallTalk like language. It's hard to be more OO than that. :-)
What newer hybrid languages besides Kotlin can cover the benefits you can get with Scala: a solid ecosystem (both for Scala and Java) for developing web applications, microservices, data processing, and good performance.
Julia, Rust, and Elixir are all great, but popularity, ecosystem are not there yet.
For Julia there are PyCall and RCall, so the ecosystem is less of an issue than with other languages. Also, its ecosystem has been growing quickly over the last 2 years or so.
I actually haven't seen a language that can do everything that Scala can do, so in a lot of senses I agree with you!
The alternative is to use multiple different languages to fill different niches. In theory this sounds suboptimal but I think it might actually be easier to learn multiple simpler languages than it is to learn Scala.
BTW according to PYPL: Swift, Kotlin, TypeScript, and Rust are now more popular than Scala.
I actually came to Scala via F#. You can even find exact dates of those transitions in my stackoverflow history.
Scala and F# are sinilar enough to basically be the same language, though I found F# had a number of warts and idiosyncracies that made me move on to Scala. Scala didn't have all of these (though of course it had warts of its own!) and is what stuck with me for the long term
What I like about F# vs Scala is the ML-style syntax and the relative simplicity. I have found Scala staggeringly complex every time I have tried to learn it, it seems like the C++ of functional programming languages. This impression may be misguided.
I to this day don't get what's wrong with Scala's marketing, as people are complaining about the "complex type system" but nobody does so for TypeScript.
Same for compile times. TS compilation is ridiculously slow yet there're no complains whatsoever out there. Scala compile times are OK on the other hands side, especially if you ever used C++ with similar complex code, or TS like said, but "the slow compiles" are a topic now for years.
Prophecy.io - low code product for data engineering on Spark - is Scala based - we love it!
We do compiler work - and packrat parser combinators are great.
Basic things are supported well. We use Play. We have GraphQL in web services - use Sangria. Slick FRM for database access is very clean. We use Akka for live web socket connections to Spark for interactive execution.
We extend Spark code from outside when needed - adding functions to classes in imported libraries.
It’s a learning curve for new engineers, but we hire the best and they like Scala after using it for a bit. SBT is confusing as hell, build times were painful but getting better. Debugging is sometimes hard (implicits)
One thing I’ve learnt us that if you’re handed a gun, you don’t have to start shooting at your feet just because you can. We used C++ at NVIDIA sensibly a decade back - using OOP but still avoiding horrible parts of C++. We use Scala sensibly now.
Can’t imagine another language that would work for us. I’ve used C, C++, F# before (building compilers and databases)- don’t know Kotlin. Only other language we use is Golang for our kubernetes operator (excluding UI)
I loved Scala until a minor version updated somewhere around 2.9-2.11 destroyed all my hope. Eight years later I still refused to use it to build anything. Yet, with that said, IMHO, it can be fucking great _second_ language.
If you haven't used a statically typed language before, i.e. you're from Ruby, JS, Python, et. al then it's one bangin' ass brain rodeo. I'd highly recommend spending a couple months trying to learn it. Once you're done, you will almost certainly spend near zero effort learning any other typesystem save the MLs (which would, in some ways, probably be a better rodeo).
A lot of time people think there's more magic going on than there is and just seeing that's how a function works with 22 arguments was pretty illuminating for me coming from Ruby.
Dang. I just checked in the Common Lisp I use and it's a bit higher. What's the reason for having to have at least 22 concrete function signatures like that?
Originally there was no way of abstracting over an arbitrary number of type parameters in the language, so a different trait was required for each arity. The choice of 22 was arbitrary, I think.
The JVM had no support for functions as values (it was designed around the idea of only having objects with methods). So if you want to very closely mirror Java at runtime (e.g. you want Java reflection libraries to work with your language) but you want to use functions as values, you have to bridge that gap with classes whose instances will represent functions.
It's for optimization, at run-time you could handle functions with more arguments, but with this it can be compiled to directly call the exact function based on the number of arguments statically used at the call-site and their type.
To do this though, you need a defined static type for the type of Functions each of a different number of arguments.
22 is just where they stopped to keep defining those types.
> Is it one of those no one is going to need more than 22 arguments?
You can work around this limit with nested tuples, or just defining Function23 yourself. Although if your function accepts 23 arguments, you should probably refactor that anyways.
I dabbled with Scala several years ago, but I've been using Kotlin for a JVM-based project and am happy with it. My main reason for choosing Kotlin is smoother interop with the JVM world. For example:
- Scala adds an Option type, whereas Kotlin adds nullability checking for existing object types.
- Scala has its own convention for getters and setters, whereas Kotlin automatically turns JVM getters and setters into properties.
- Scala defines its own set of collection classes, whereas Kotlin has zero-overhead wrappers over the JVM collection types.
I haven't tried Kotlin/JS lately, but I expect that Kotlin's strong focus on interop with existing stuff would serve it well there too, especially when it comes to minimizing bundle size.
Scala may be more powerful in some ways, but Kotlin gives me the conciseness I want, with great interop with the huge Java ecosystem.
It doesn't really matter how great they are, it kills JVM interop right there ... and its even worse if it doesn't and you get implicit conversions killing your performance.
This is where there is a bit of bait and switch with the JVM story (applies to a lot of JVM languages) - "language has great feature X" and "use any library from the JVM ecosystem" - often turn out to be mutually exclusive in a practical sense. Groovy is the only JVM language I've found that seems to deliver a real JVM interop story, but that's explicitly because it is designed from the ground up for that.
> It doesn't really matter how great they are, it kills JVM interop right there ...
That's extreme. It really doesn't "kill" interop at all. If you want to convert, convert. There are many cases where you are not calling Java in hot paths and where converting is perfectly acceptable. If you don't want to, then don't and use Java collections directly. You have the best of both worlds depending on what kind of code base and libraries you are dealing with
Also, implicit conversions between Scala and Java collections have been discouraged/deprecated for years now. You should use the explicit conversion when needed, with `.asScala` or `.asJava`. This makes it obvious whether you might pay a price in the conversion.
Finally, your position overlooks the fact that there are solid reasons to use and prefer Scala collections instead of Java collections: they are quite feature-rich compared to Java collections, are immutable by default, while providing mutable collections if you want those as well.
> that's explicitly because it is designed from the ground up for that
Scala was designed for Java interop from the get go. I am not sure how you are suggesting making things better, besides saying that Scala should never have had its own collections. That would take away a huge part of what makes Scala great, in my book.
It only interoperates with Java in a trivial sense. Using Hibernate for example is a real pain in the ass in Scala, because of its insistence on using its own collections.
> It only interoperates with Java in a trivial sense.
It's not in a "trivial sense". You can call any Java code from Scala, call Scala code from Java (except when dealing with Scala constructs that don't have direct Java equivalents). Even Java and Scala functions are the same since Java 8/Scala 2.12. I am not sure how much more interoperability you could even wish again, besides saying that Scala should be identical to Scala. And I write this having written Scala for 10 years and used dozens of Java libraries in Scala projects.
> because of its insistence on using its own collections
This reads like Scala is somehow doing this stubbornly against all reasonable arguments. That's not the case. Scala collections have benefits. Maybe this means using Hibernate is not straightforward from Scala. Ok. I don't know. But this doesn't invalidate Scala having its own collections.
There are tradeoffs in everything. We are talking about two languages and how they interoperate. As far as language integrations go, the Java-Scala interop is excellent, in my view.
> Groovy is the only JVM language I've found that seems to deliver a real JVM interop story
Have you tried Kotlin yet? If so, what interferes with its JVM interop story in your opinion? I'm happily using Kotlin with several Java libraries, and IIRC, no Kotlin-specific libraries except for the stdlib.
There are some quirks when you need to call Kotlin code from Java, so it is not as transparent as the other way around. But still, the JVM interop from Kotlin is mostly transparent (if you remember that platform types can be null).
> - Scala adds an Option type, whereas Kotlin adds nullability checking for existing object types.
Kotlin's approach amounts to doubling down on null IMO - it actually makes it harder to interoperate with newer JVM code that favours options over nulls (e.g. streams) from Kotlin than from Scala. And since "platform types" are silently treated as non-null, you still get null exceptions at runtime.
> - Scala defines its own set of collection classes, whereas Kotlin has zero-overhead wrappers over the JVM collection types.
Which means Kotlin has no actually immutable collections; a supposedly immutable collection can be silently mutated by another thread in parallel, which can be pretty surprising.
There are definitely things that Kotlin does well, but IME "smoother interop" is mostly a myth.
My experience is that most Kotlin features boil down to myths created by good marketing. When you look at them in detail they're full of gotchas and unexpected behavior (especially when interacting) due to everything being some ad-hoc special case only created to simulate some Scala feature.
Tones of ad-hoc features isn't good language design imho, even the features are desirable in isolation. That's the same issue as the biggest problem with PHP or C++.
On the other hand side Scala may seem "complicated" on the surface but it's actually just a bunch of quite simple basic principles, and some syntax sugar on top.
Scala is like math: It's build form simple first principles. Just it offers the means to combine those and create this way almost arbitrary complex abstractions.
> Scala defines its own set of collection classes [...]
One (arguably) negative consequence of this is that Scala's collections don't play nice with Hibernate and similar ORMs. Due to the way they work, both Scala and Hibernate wanted to "take over" your collections, and obviously they cannot do so at the same time. Of course, it's arguable whether this is a flaw with Scala or with Hibernate, but for the programmers who just wants to get things done that's irrelevant.
It used to be a big deal a few years ago, to the point teams in my company decided to either drop Hibernate or drop Scala.
PS: if I remember correctly, early Scala adopters had reached the consensus that ORMs didn't play nice with Scala's FP style anyway, and so using Hibernate was a mistake. Possibly this attitude changed later.
With Doobie you manually write your queries, and then map the results into the objects in your domain model. Nothing is generated for you. OTOH, nothing is hidden and you are free to write queries as optimized and specialized as you need. The real selling point of Doobie is a typesafe API for manipulating and combining queries, and fragments of queries, into larger wholes. This works very well when your application interfaces with a database it doesn't own.
With Slick you get access to a DSL that lets you layout how your tables look. From there Slick offers an api that let's you treat SQL tables as-if they are basically mutable collections, with Slick handling all the SQL generation itself. You also get DDL, so that you can automate db creation and upgrades. This work very well when your application owns and controls the database it is connecting to.
Both of these have diverged from that traditional ORM model. Slick bills itself as FRM, or Functional Relational Mapping. And Doobie is embedded queries on steroids.
I have had poor experiences with "fancy" SQL libraries in multiple languages. This includes Slick and Quill in Scala.
I don't remember what the problems were with Slick that leave such a bad feeling when I hear its name -- that was 5 years ago -- but I had problems with Quill just last year. I was trying to use it to generate an efficient "in" query against a two column composite primary key, and nothing seemed to work. Since it uses macro magic, one of my attempts triggered an internal compiler error instead of normal compiler feedback.
It seems to be less popular/active than other libraries, but it is dead simple to use, even for developers new to Scala. I write exactly the SQL I want just like I would in psql. There is little-to-no magic. I think that the only slightly magical feature I use is variable interpolation into SQL ("SQLInterpolation") that prevents injection attacks. It actually has capabilities to automatically map tables/columns into different structures and generate code for you, but my team doesn't use any of that. We just write SQL.
I didn't have a good experience with scalikejdbc, which was what we ended up using. Possibly we didn't use it well, and I can't remember the details now, but our DB code ended up looking like a mess.
they never said that, but it's basically abadonware. look at the commit history: https://github.com/slick/slick/commits/master you can also look at the contributors page: https://github.com/slick/slick/graphs/contributors the guys who primarly contributed to it basically left lightbend, and in 2019 he shifted his priorities.
it only had 28 commits since 2020 (including merge commits and changes to the build process) it's basically dead until somebody steps up.
well as said 28 commits, NOT ALL were code. MOST of them were basically "fix build"/update dep. so basically the project did not change for over a year.
AND 2019 wasn't really better. 3.3.2 (3.3.3 was mid 2020) was in JUNE 2019 and 3.3.0 was at the beginning of 2019. there won't be a 3.4 anytime soon, if nobody steps up and starts maintaining it, there won't be a 3.4 at all.
slick is heavy on macros it will not survive scala 3
I haven't used it heavily but one of our newer microservices uses Quill for read-only access to Postgres. I have found it to be great to work with, to combination of "just use case classes" and a DSL that feels like writing SQL is pretty nice.
I seem to remember Stefan Ziegler asking around the channels about questions relating to how to do certain features to uplift Slick. Don't know if anything came out of it.
> PS: if I remember correctly, early Scala adopters had reached the consensus that ORMs didn't play nice with Scala's FP style anyway, and so using Hibernate was a mistake. Possibly this attitude changed later.
Having used Hibernate in a Java project, I would argue using Hibernate is always a mistake. Way too much complexity/brokenness for basically no benefit.
> - Scala adds an Option type, whereas Kotlin adds nullability checking for existing object types.
I like null coalescing, but Scala uses a more generic way to handle nulls like Options. After working with both, I'm not sure which way I prefer. I do like not having to create new syntax for language features and could be handled with a monad. Async/await is another that can be handled with an IO, Future, ZIO, etc.
This is a good list, and this is why Scala for Android was a no starter. The huge standard library and awkward conversions with Java made it an expensive cost on early phones (and probably still today).
Kotlin also side steps the big standard library by using inline functions a lot (in addition to the mentioned advantage if extending existing Java classes).
To really appreciate Kotlin’s design, one should know some Scala.
A good chunk of the document lists things in common, not differences. The actual differences have nothing to do with dynamic typing and are deliberate design decisions:
- don't pretend the JVM and JS have identical runtimes when it comes to concurrency or numerics. Clojure is a language, not a platform abstraction.
- Emit efficient, optimally minifiable javascript code, at the cost of offering something less traditionally lisp-y when it comes to eval, macros, and other forms of code loading. Those are still possible, but some discipline is imposed.
> - don't pretend the JVM and JS have identical runtimes when it comes to concurrency or numerics. Clojure is a language, not a platform abstraction.
> - Emit efficient, optimally minifiable javascript code, at the cost of offering something less traditionally lisp-y when it comes to eval, macros, and other forms of code loading. Those are still possible, but some discipline is imposed.
This is exactly the kind of thing I'm talking about: Scala.js doesn't need to make these tradeoffs at all! Concurrency works identically (just no parallelism), numerics work identically, macros work identically. The list of pure-Scala things that behave
differently in Scala.js is shockingly short (https://www.scala-js.org/doc/semantics.html)
Scala.js sacrifices none of these and still emits efficient (sometimes faster than idiomatic JS!), optimally minifiable javascript code. The fact that Clojure needs to make these tradeoffs and Scala doesn't is largely due to Scala's static nature allowing more aggressive analysis and optimisation at compile time. For example, the reason Scala.js can provide exact semantics for numerics while maintaining high performance is precisely because it knows what exactly types things are, and can use that information to optimize the generated JS in a semantics-preserving way (http://lampwww.epfl.ch/~doeraene/presentations/jslongs-vmm20...)
ScalaJS is one of the projects which I don't understand why more people don't rave about. It's /extremely/ nice that I can have a single project which share code between them for validation; no need for swagger or OpenAPI.
I think you're overshooting how many differences there are between Clojure and ClojureScript. They're mostly identical.
The big difference is the interop, and from your link with Scala.js it seems that's also true of it.
I'd say in practice, where ClojureScript and Scala.js maybe can differ a bit is that idiomatic Clojure itself relies a lot more on interop than Scala. So probably more Scala code doesn't depend on Java interop and thus can be more easily ported to Scala.js, whereas more Clojure code relies on interop making it more difficult to port to ClojureScript.
The other big difference is that Clojure supports runtime macros and eval, and those would require the compiler be bundled with your frontend app, which would increase the bundle size a lot. So ClojureScript needs to drop some of the dynamism. It's very rare to have Clojure code that actually depends on this though.
Overall in practice, what you'd want to share between backend and frontend is sharable as-is between Clojure and ClojureScript, which is all that's really needed. You'll build a React JS based frontend in ClojureScript, and share data-model, validation, transforms over it between backend and frontend. And you'll want your render functions to be shared between them as well so you can do server side hydration for React.
P.S.: I also want to point out there is a camaraderie between ClojureScript and Scala.js, and the devs exchange ideas, taking inspiration from one another as well. Since they face a lot of the same challenges.
The JVM offers real threads, and JS not. The JVM offers nanosecond-precision time measurement, and JS not. JS has a single type for representing numbers. And so on.
How can those possibly be abstracted away and unified? If doing so, what would have the solution have to do with type inference at all?
i.e., this is simply a difference in how one approaches platform interop (raw vs abstracted/unified). Clojure occasionally offers cross-platform abstractions (e.g. core.async) but it's not its main philosophy.
How does this make a difference to the Clojure interface to threads?
After all, if you're running your JVM on hardware that doesn't support parallelism, you're not getting parallelism anyway.
> JS has a single type for representing numbers.
The comment you're replying to already covered that:
> For example, the reason Scala.js can provide exact semantics for numerics while maintaining high performance is precisely because it knows what exactly types things are, and can use that information to optimize the generated JS in a semantics-preserving way
> Clojure occasionally offers cross-platform abstractions (e.g. core.async) but it's not its main philosophy.
I can understand providing different libraries for different platforms, but language semantics should remain (as much as is possible), e.g. numerics.
> How does this make a difference to the Clojure interface to threads?
Clojure futures, promises, and some of its other concurrency primitives have synchronous semantics. So they don't take callbacks, instead the calling thread blocks on the result when it's at the point where it needs to wait for it.
So that interface doesn't work with JavaScript's concurrency model. On the other hand, Scala futures have a asynchronous interface, and you specify a callback for success and error, and cannot block waiting for them, which happen to be 1:1 with the JavaScript promise interface.
That's why on ClojureScript you don't have access to those concurrency construct, since they just don't work within the JavaScript model.
The compiler maybe could have gone through amazing lengths to rewrite all use to an async one, but it just so happens there is already in Clojure something that does so, but it rewrites a CSP interface to an async concurrent model (not futures and promises). It's called core.async and this one exist in both Clojure and ClojureScript.
> I can understand providing different libraries for different platforms, but language semantics should remain (as much as is possible), e.g. numerics
Even Scala.js has some differences in numeric types they list in their page. There's always the possibility of mismatch of a language over some runtime. In ClojureScript I think it was planned to eventually bring the same numeric semantics over, but they started without, and it seems no one minded and then never bothered.
Scala's Futures work unchanged, it just has a single-threaded threadpool instead of a multithreaded one. Anything built on Futures (e.g. my Castor actor library) is unchanged as well.
Scala Futures are more or less a 1:1 mapping to Javascript Promises, and work identically.
There isn't long running IO or sleeps, same JS, but this isn't a big problem in practice. As long as you're not juggling threads directly - and most Scala code doesn't - your code should run unchanged concurrency-wise in Scala.js just with no parallelism. Though of course you can't use OS interfaces like java.nio.file.Files in the browser!
I think in this case Clojure provides pretty strong competition with core.async - you do have to use the async specific calls for sleeping and channel IO but it still works using the same API on JVM Clojure and ClojureScript sides.
Scala is just an amazing language. I'm very happy that the worst of the 'Scala as a worse Haskell' days is over and that the Scala community is finding it's own programming style.
I remember years ago when I had a heated discussion about local mutable state. An that time it felt like heresy to even suggest it. I just recently had another discussion with the same programmer and he was like: "Yeah, that works".
Unfortunately, "Scala as a worse Haskell" is alive and well. The /r/Scala subreddit, for example, is dominated by folks who tell every Scala-curious beginner who drops by that the accepted right way to write Scala is the pure FP ecosystem and everything else died a long time ago.
There are beautiful, simple, elegant ways to write Scala, but I'm afraid that the community isn't converging on a single concrete style, and a consensus around an abstract set of ideas about style isn't useful. Everybody agrees on words like "simple" and "elegant" and "readable," but I've seen some real abominations in Scala that people are tragically proud of. I'm afraid that Scala has attracted too many of the wrong kind of programmer, the kind that identifies their professional value with their ability to implement complex solutions that are intractable to their peers. The problem is not that you can't write simple code in Scala, but that people choose not to.
I don't want to blame the pure FP style of Scala itself, because I've seen the same problems in the "better Java" style as well, and because I'm still thinking that I might be able to write good code in this style and might even come to prefer it. But oh man, the Scala programmers I've personally worked with who embrace pure FP have really screwed up priorities.
Yep this is a huge cultural problem with the Scala community. On paper and in marketing material, Scala is flexible enough to empower you to slip into mutable/OOP style or more FP style. But in practice the pure FP enthusiasts are the ones who totally dominate the ecosystem (aside from Haoyi Li). You simply cannot just ignore the Haskell Larpers/Wannabe's and get on with it, because they infect all the community watering holes like Reddit, Gitter and Discourse. Every time one talks about using non-pure-FP features of the language, they have to go through the same tired patronizing lectures about how "yes I know this is not 'safe'" and "yes I know what I'm doing" and "I'm very sorry that I am not as smart as you smart FP people" before you can actually talk about what you want to talk about.
AFAIK a lot of the wannabe-Haskell-programmers ended up transitioning fully to actual-Haskell-programmers too, which is probably a win-win that's good for everyone involved. Life's to short to use a language you hate, different strokes for different folks and all.
I still find that Scala devs think they’re writing Haskell. From what I’ve seen in the community, no one cares about shipping code, they care about whether or not it’s “pure”.
To give them their credit. The argument from the purists is that purity is itself about shipping code. As a long-time scala developer who has started using the pure-FP ecosystem (cats,cats-effect mainly) I am starting to see their point. It takes some time to grok but once you do then it is amazingly productive to work with.
Maybe. I certainly don't have data on it but am also not sure what sort of data would actually prove that point. Teams that choose Scala may just be solving different sorts of problems. Or maybe they take longer and deliver a higher quality product. All I know if that I am personally way more productive in Scala than any other language I've used (and I have used most of them at one point or another).
> Or maybe they take longer and deliver a higher quality product.
Without going into how do you define a higher quality product, does quality really matter that much..? Most code is going to be re-written every few years anyway. Software engineers aren’t building architectural wonders that will last hundreds or maybe thousands of years. So is the complexity of a language like Scala really worth it, just to deliver a product that will be obsolete in a few years anyway?
Well, I've been working in financial and data services for many, many years now and I can tell you as much: many of the code bases I've been dealing with (and am dealing with) written in less strictly typed languages (much Python, lots and lots a lot Java) are riddled with hidden runtime errors that you'll just never see in Scala.
If there's a bug, it's usually some modelling problem or some conceptual issues, not the implementation of the code. I've much more confidence in the code I've written in Scala than the code I've written in Python or Java and I never ever had to get up and pick up the phone in the middle of the night because some production issues with code written Scala.
Java has really great tooling to make it really safe if one wants to — the problem is usually the managements’ cheapness with programmers fresh out of school/bootcamp. There are areas with high security requirements where Java is used with great success.
On the margin, yes. If a language helps to deliver a better product (fewer bugs, more dependable, etc) then your prior should be that that is a good thing that will, all else equal, make for a better product. But I don't accept the premise that it is easier and faster to deliver a working product in a different language. Maybe it is in some cases but me and my time are most productive and produce the highest quality product using Scala. And I think that has a lot to do with the language and the ecosystem itself. But ymmv and if it's not a good fit for your team then you shouldn't use it.
I would make the counter argument that I would rather have a language that _may_ be more prone to bugs and be able to easily hire engineers. Also, there’s a lot of hidden costs in over engineering something. Trying to get something to be bug free is like trying to add another 9 to five 9s of reliability, the cost is rarely justified.
Over-engineering of course is bad by definition, but my point is that is not the correct characterization. Scala is both safe AND expressive which is good for both productivity and reliability. If, of course, you take the time to learn and understand the core abstractions. To your point, yes that does take time and there is generally not a ready pool of talent you can hire with that knowledge already. So I wouldn't decide to write something in Scala if the team doesn't have any knowledge already and you're on a deadline, but you could say that about any language really. Any given team is going to deliver the fastest with the highest quality using the language and tooling they already are experienced with.
But as an individual engineer I would say that it is worth learning. Even if you don't use Scala for any major projects due to various organizational reasons, it is a great, principled introduction to a lot of the patterns that are becoming ubiquitous is most modern programming languages. One of the small pleasure in life for the moment for me is when the NodeJS devs I know who always told me static typing is unnecessary and Scala was too "academic" are suddenly gaga over Typescipt and functional programming.
Basically that, yes. The idea is, as long as the mutable state starts and ends within the confines of an individual function, you can get away with treating that function as if it were "truly" FP.
The main argument against this is that it's a concession that makes the codebase less coherent. The more hardline argument being that the encapsulated mutable state is still more prone to bugs and should be almost always be avoided, except at great cost.
There are certain algorithms that either can’t or only insanely error prone way can be expressed “purely”. There is nothing wrong with locally mutable state. It is basically the same as a State monad that is evaluated/executed — which can arguably be worse
It’s rare that you need to use vars in Scala, but still if it’s scoped to the function then you know you can mentally discount it once the function terminates.
I think that the Scala experience will get even better with Scala 3, tons of well thought refactorings for metaprogramming and typeclasses, among others. Also, looking forward to (optional) indentation based syntax.
It's funny, because my experience is almost the opposite. My Scala skills are a bit rusty at this point and I promised myself to get back in the game once Scala 3 was released. Most of it I like. The new syntax, however, is really putting me off. My #1 problem with Scala has always been its huge upfront complexity and the quality of libraries, which are often over-engineered and not nearly enough UX-centric (cough implicits everywhere). Scala 3 was supposed to be a breath of fresh air, but I don't want to invest more time in a language that instead of getting simpler further complicates something as basic as the syntax. This change tells me that I strongly disagree with the core leadership when it comes to the future of Scala and what kind of language it should be.
Opposite for me. After years of Scala (which I like) I recently tried to port a few things to Scala 3 and it really feels like some of the wtf-implicit part is gone. We'll still need to make a complete idea about this version 3 though.
Yeah cold compiles are slow, but most modern tools keep a hot compiler to re-use and once warm a hot compile of hello world is more like ~200ms. There's a bit of complexity managing these daemons, but that's something that the build tool normaly handles for you transparently
Yes it gets better, and the 'fsc' (Fast Scala Compiler) does improve on the situation somewhat; but I've always found it annoyingly slow. It should be mentioned that I haven't looked very hard at that language in many years for that very reason; so things could have improved somewhat. And from what I understand, the next version of Scala (Dotty) is going address this issue (build speed).
Standard use case is to use sbt (well, or mill, since we're in lihaoyi's thread :D). Sbt starts up slow, but with bloop, or sbtn (native client of sbt), the compilation is really fast (much faster than C++ for example):
$ time sbtc compile
[info] entering *experimental* thin client - BEEP WHIRR
[info] terminate the server with `shutdown`
> compile
[info] compiling 1 Scala source to /home/.../target/scala-2.13/classes ...
[info] compile completed
[success] Total time: 0 s, completed Feb 11, 2021 4:21:51 PM
sbtc compile 0,08s user 0,02s system 21% cpu 0,492 total
Yeah, set 1.4 was a huge improvement. I wrote a lot of scala several years ago and just got back into it this year and the dev experience is much nicer now with the SBT sever model
There are very few optimizations that a JIT can improve over a static compilation of a strongly-typed language. Strong typing brings immense capability for static analysis and subsequent optimizations. The most successful JIT optimizations out there typically are ones that make dynamic languages about as fast as statically typed languages by making assumptions about the runtime types of objects, while trapping/reoptimizing when those assumptions don't hold true. It also does pretty well finding hot loops and selectively optimizing those, but PGO gives you that too without any performance hit in your final executable.
A strongly typed language like Scala has the potential for massive optimizations under the whole program optimization umbrella...and no JIT would ever dare to try those types of optimizations, because they are extremely memory and compute intensive. They'd have to pause execution for minutes at a time just to figure out what to do.
I think it is unfortunate that Scala was designed for the JVM first. It brought a lot of syntactic baggage for interop purposes, and it saddled it with weird things like reflection and dynamic class loading, making it hard to statically compile. The ideas behind scala's type system are extremely powerful and could have fundamentally changed programming for the better, but all we got was a JVM me-too that was quickly usurped by Kotlin. I don't think it is dead or a lost cause...but the legacy that the JVM has on Scala is one that slowed it down, not sped it up (like what was claimed would happen with JVM languages).
It’s not as black and white. What about GC, compacting objects and the other myriad of optimizations the JVM does?
I do want to mention that I am not overly knowledgeable on these topics, only really interested in compiler technology, but a program compiled with PGO will have to add code for each specialized function, won’t it? So perhaps not every sort of application would win by PGO AOT compilation; a structure over the running code, and manipulations over it are a great and powerful idea.
But I do agree that Scala in the early years did have some ugly artifacts from compiling to the JVM and trying to remain a “good citizen“ there, not sure about current trends.
I agree that a AOT scala would be great. But don't JITs have the ability to react to real world usage and do hotspot optimization and stuff. I assume they are very complicated to build, and if you do thing ahead of time you should, but aren't the things that can only be done in a JIT important?
I am working on an AI book using Swift. Yesterday in a meeting, my manager was asking why Swift. I compared it to Scala, but built on LLVM, and a rich CoreML and other library infrastructure. REPL + closures, very fast build times.
I haven’t had much opportunity to use Scala, but I did take Martin’s Functional Programming with Scala course years ago, and that was great.
I wish swift were like Scala. It attempts to look and feel like it, but there’s one huge aspect of Scala that is missing in swift that basically makes swift a world apart, which is expressions vs statements.
In Scala everything is an expression, meaning everything evaluates to some result even if that result is void.
In swift everything is a statement. There’s a big difference because statements don’t return things, only functions do.
It sounds minor, especially since swift has pattern matching and first class supper for functions etc, but the truth is these are all improvements over a more oop (in the Java sense) way of programming. They don’t however make the languages very similar because at a more basic so level they’re very different.
I would even go as far as to say that rust is more like Scala than swift is, although it’s been a while since I coded in rust.
Scala is a great language. I haven't done that coursera course, but many people swear by it.
To me, Swift and Kotlin look like they have taken inspiration from Scala. I think Kotlin explicitly did. That idea transfer from one area to another is how we get better langauges.
Complete aside, but how is Swift these days? I used it literally a week after release and promptly stopped because the compiler kept segfaulting - I imagine it's more stable now?
If I've said it once I've said it a thousand times: the day Scala kills sbt officially will be the day thousands of devs consider a return to it. Until then, I couldn't dare.
Maybe so... but so do Python or Javascript. It seems to me some other languages have only one stack (rust: cargo, rustfmt...; the .net world as well). Does it make them better? (honest question). Then again, what's Java official stack?
curious. what exactly is wrong with sbt? I used it once of twice when I was learning scala and I liked it a LOT more than gradle/maven. To the point where I actually opted to use it with my Kotlin projects later.
I no longer program in Scala, but I used to for my previous job and I regularly came across build files I couldn't understand -- and when I asked the person who wrote them, they couldn't explain their build properly either.
And this happened in enough cases that I wouldn't consider it an exception or a problem with a particular developer. SBT used to have two entirely separate syntaxes, and it was a mess translating from one to the other. I don't know if this is still the case (I know one of the two syntaxes got deprecated, can't remember which one now).
You need a phd in sbt to do things like "copy the jars", or "have a second set of dependencies".
The proponents of the approach that sbt takes have failed to realise that build tools have zero value whatsoever, every second spent on them is a second wasted.
What you want is a good mix simplicity (yes copy paste is fine) and some basic examples to follow, and the ability to break out into script if you are doing something unusual.
I don't know. I have no real love for SBT but I have yet to find a build tool that I actually like and that covers all the use cases and has the ecosystem that I want. Maybe it is just a really hard problem and extremely resistant to creating a simple interface. Unless of course you consciously decide to punt on the hard cases (which if fine sometimes).
I'd pick sbt over mvn no doubt. Haven't played with either fury or mill, so can't comment on those (even gradle... I never really used it). But coming from years of Java/Maven I think sbt is way better. Things start complicating a bit if you have lots of plugins or macros probably.
For new Scala devs, how much of a challenge is it to avoid sbt?
Especially for those coming from languages like Python and Go, where the former has no build system (ignoring setup.py), or an almost universal one (go build, sometimes make).
All of the projects I have touched are documented using sbt, and transitioning the build process for existing code projects while a new language learner is a double cognitive load.
It's easy; there's mill, maven and fury. Mill is quite good - but I've had to dive into the source code a few times to do things.
If you're dealing with dependencies only, then it's a straightforward change. OTOH I've never had an issue with SBT since it's cleanup a couple of years back so YMMV.
Scala compiles well enough under Gradle--got it in a roughly dozen-module multi-language project right now, and it's no trouble. I suppose Gradle might (or might not?) come up short if you want automatic deprecation/replacement updates applied to source.
Scala is nice for doing things with Spark compared to say, Java, but I really think that Kotlin can do just as well. The only advantage Scala has with Spark is the syntactic sugar. Once there is an API for Kotlin (beyond the preview version), I will be dropping Scala like a bad habit. Kotlin is also concise, has better IntelliJ support, less "implicit magic", and better Java interop.
Kotlin has lots of syntax sugar but the semantics are crazy. It's essentially an imperative language where the only way to reason about effects (suspend functions) is by reasoning about control flow. That already makes things like error handling incomprehensible, and it'll be a really bad fit for Spark's way of doing things (where values have to be serializable and shipped around the cluster) - imagine trying to use a suspend function from a Spark anything, it's completely impossible to implement in a sane way.
I'm curious what your use case is for using a suspend function in spark. I have written several spark jobs with Kotlin and have never had a reason to create one. The only difference from the Scala/Java jobs is that I had to be much more explicit with certain things (encoders in higher order funcs, etc.).
All of the reasons you'd want to use a suspend function at all - e.g. mutually recursive functions, async I/O (e.g. database queries), error handling - suspend functions seem to be kotlin's answer to anything even slightly non-vanilla. Writing in Scala I can, with care, use async and iteratees in spark and be able to reason about what's going to get executed where and when, because these things are represented as values.
That's not actually true, IntelliJ support for Java has always been much better, and still is today! Just try auto-completion in Java VS Kotlin in your IDE... it's much faster in Java files, something I've been complaining about as I've been using more Kotlin than Java, recently. Check also how many refactorings are available in your Java files compared to the Kotlin ones.
Jetbrains knows its main source of income is still, by far, Java, so they don't drop that ball.
If only Kotlin would introduce a non-ridiculous way to declare map and list literals it would be a great data science language. As it is, it falls over at the first hurdle where you want to specify some data inline. One reason I've really stuck with Groovy for JVM-based data sciency stuff.
Is Scala used outside of Spark and Akka? Spark is my company chosen analytic engine, Scala was/is the main language for majority of the production workload. In the last two years or so, more devs are writing PySpark and SparkSQL jobs than Scala.
There's a blooming ecosystem of pure functional programming libraries. Or even two ecosystems: Typelevel and ZIO.
My company uses Scala a lot with with these ecosystems (my team is on Typelevel, another one is on ZIO) and in fact Spark and Akka is what we're trying to avoid as much as possible. Once I got used to the composable nature of FP and static types - both Spark and Akka just feel clumsy and unintuitive, like maintainers of Spark chose Scala because it was a "Java with fancy syntax" back then, ignored most of benefits of static types, HKT, immutability, type classes etc. If there's future behind Scala - it definitely doesn't lie in Apache land. Akka is slightly different and probably has higher potential, but just an overkill for most of use cases it's advertised for.
def mirrors my experience too. Vast majority of spark jobs were easily ported to sql/dbt and the remaining ones are in pyspark. I used to use a lot of scala spark in backend data processing in 2016 but now its almost down to zero.
scala is real big impediment to making data processing accessible to general public in your company. order of preference now at my company is,
I've rolled out Scala based Spark interfaces to non-programmers in Databricks notebooks, so it's definitely possible, but only if you stick with the basic language features.
I think Scala Spark (using 10% of the language features) is a better technical decision (because it provides huge benefits like fat JARs, shading, better text editor support, etc), but the worse overall choice for most organizations because people are generally terrified of Scala.
They'd rather do nothing than write Scala code. I can empathize with their position.
> scala is real big impediment to making data processing accessible to general public in your company
Ding Ding Ding! Presto/Athena now is becoming huge in BI ecosystem. We don't really use Spark for ad-hoc BI anymore, we use it for data science and large repetitive workload.
I want to like Scala and Akka, but they just seem to get used very little in the “real world”. I recall some HN comments complaining about how a lot of the Scala they were writing for Spark was more or less a series of procedural API calls rather than using the language to its potential.
The author argues that dynamic languages like (JavaScript, Python) have Poor IDE and Tooling Support and are Unmanageable in large programs.
I'll counter that if you add a type checker to one of these languages, such as TypeScript (for JavaScript) or mypy (for Python), then all of those problems go away.
Additionally, RE "Poor performance" the author is presumably talking about CPU-bound performance. If you happen to be writing network-based or I/O bound programs, which you always are in JS/TypeScript and sometimes are in Python (for web applications), then the effective performance is more affected by the network/database/disk than the programming language.
I think the author as a point though. Typescript and type annotations for python will never be able to be as good. Not only because they are not "first class", but also because they always have to interopt with potentially non-typed code.
Then, creating a good typesystem is really really hard. Even professional language designers struggle with that. Take Martin Odersky, the creater of Scala. He didn't just invent Scala, no, he learnt under his professor, then created pizza-lang and did other experiments and worked with other languages (e.g. he was the one who added generics to Java) before he used all his expertise to create Scala. And even then mistakes were made.
You can see it from the language. I work with both Scala and python (with type annotations) and you can see that one is created by a specialized professional, where the other one is created by smart people but not with the same sophistication.
> Additionally, RE "Poor performance" the author is presumably talking about CPU-bound performance.
He probably does, but these languages are most often also better for IO-bound programs. The reason is that writing asynchronous and concurrently executed code is very difficult and pretty much adds another layer of problems on top of everything someone does. Languages with static types (not only Scala) can leverage a big advantage here, because the compiler can help you to indicate what is synchronous and what isn't.
Scala is awesome. It is meant to be a scalable language where you go from level 1 to say 5 as you learn. Problems arise when you put a level 5 guy into a level 1 team and letting him loose. No common style guide, no team training, just write those type signatures approaching tweets max length and let the rest wonder wth is going on
I'd argue that the real problem is finding people who think they're level 5 but aren't. Writing dense impenetrable code certainly doesn't sound like a high-level engineer to me!
Scala certainly has no shortage of these people, but at my work we throw senior engineers with zero Scala experience or training at the language and it generally turns out OK. Turns out that a 10x faster Python or a 10x more concise Java are selling points enough, even without any fancy code!
I'm a big fan of Scala, though I think the sweet spot for Scala is for a small team with pre-existing Scala's expertise. For a one-person project, I definitely use Scala. I used to use Ruby a lot.
You can move fast with safety with the caveat that the code quality is harder to manage/control.
Contrasting this with golang where they really take the guard rails to the other extreme (e.g. can't compile if a variable is unused, not even compiling just to test it out locally).
Another weak-point of Scala is that the learning curve is steep, especially when it involves SBT, the defacto build system for Scala.
We're scaling out Scala to a larger team here at work, and part of the effort is certainly about putting in guardrails.
In fact, a lot of it is putting in thr guardrails people like from Go: enforced autoformatting, banning unused variables, fatal warnings, etc. are all just a flip of a switch these days (though we only enforce these things on-merge, people are free to test locally regardless, which I understand is a big pain point in Go!)
A lot of my buddies who use Scala seem to like it. I'm sure it has some to do with the fact that they can use Intellij, which I consider to be one of the best IDE's available.
Are there any studies comparing real world impact of structural vs nominal typing?
Defects? Code size? Team size? Architectural choices, big and small, like use of Visitor design patterns?
I've looked, but no joy.
--
FWIW, metaprogramming should be reserved for personal projects and small high trust teams. Definitely not for bog standard data processing and CRUD apps.
I'm kinda curious how structural typing story plays out in this dichotomy.
Scala _can_ do structural typing, but you almost never see it in the wild so I don't think you'll be able to find any data on it. The main reason why it isn't used in Scala is because it's implemented using run-time reflection which can be slow.
From the time I've spent working with Typescript (Probably the most mainstream with with structural types), I don't think structural-by-default is the right approach if we are to design a language for safety & productivity. The error messages are a lot worse than nominal typing, and useful patterns like new type are either not safe or has a runtime cost. [1]
I suspect nominal first with good structural typing support from the compiler is where the sweet spot is, but to integrate it nicely in any language used today is probably an open research question.
I dispute the claim that compiled languages have "heavyweight build setups" or are "inconvenient for small programs". I am constantly writing small Java programs to do minor tasks. I compile and run those programs with Python scripts, not heavyweight IDEs or build tools.
In my mind, I, perhaps wrongly, equate compiled with - "runs machine code". Java is considered compiled but runs on the JVM. Python also has compilation as one of the steps and runs in its virtual machine yet it is considered interpreted. Sooo... Where do you draw the line?
Regarding your main point, I agree, compiled languages do not necessarily have complicated build setups. But maybe it is tainted by my experience in this field.
It is rare to find a language as simple and yet powerful. Much awaited Scala 3 brings more features. It is not better Java. It is an independent language which can interop with Java . It has richer set of specifications compared to any other JVM language. People think most of Scala community is about functional programming . It is not . It is a hybrid language. It is best suited for every facet of data engineering where we can make use of state of the art data abstractions and concurrent programming. There is ongoing work on improving tooling and ecosystem. I would ask of you people not to get scared of language based on others opinions. Don't go for advanced concepts when all you need as basics.
> While most programming languages allow you to take an expression (1213, "moo") and infer the type (Int, String), Scala allows you to go the other direction: take the type (Int, String) and infer a value (1213, "moo")! This is tremendously useful in a whole range of different scenarios
That's an interesting way to put it (and pretty neat), but I guess I'm not totally convinced it is that useful in practice. I saw the linked example about the parser using implicit objects but I think that could be rewritten in a number of different ways without relying on that feature.
I want to switch away from Python, at least for personal projects. I want to broaden my range, experiment with weird paradigms (Raku) or get code that’s a little faster (eg Rust).
The problem is that I use Python as a “glue language” and heavily rely on excellent third party libraries. I don’t want to learn how to write a minimum spanning theorem algorithm; I’m used to have it next to several other convenient graph utilities in networkx. There’s so much stuff for Python — where else will you find something like giotto-tda?
The JVM ecosystem is similarly rich (and basically the only one comparable; there are many js packages but they are more opinionated towards web development), with plenty of languages to choose from (and with Graal’s python implementation you can even write python code that interoperates with other JVM code)
As for concrete languages, I like Java, because it has just enough abstraction to make it useful and is not overly complex. Scala is a real gem in that it was one of the first OOP+FP hybrid, a marriage that seems really fruitful since then basically every OOP lang introduced more and more FP concepts and vice versa. There is also Clojure if you are into LISPs. If overly functional programming is not your thing, Kotlin is a modern language with some additional features to Java though I am not particularly fond of it. And there are even more niche languages running on top of it.
But the real gem is the JVM itself, and it is really actively developed and many new, exciting features are in the works.
Raku also has the Inline::Python module, that allows you to use just about any Python (3) module "natively" in your Raku program. The interface is not as polished yet as it is with the Inline::Perl5 module, but there is a GSoC proposal to improve on that! https://github.com/perl-foundation-outreach/gsoc-2021-ideas/...
Problem with Scala that most projects use sbt but it's not even packaged in many Linux distributions. With the Haskell doing strong and Rust, both compile into the native out of the box there are no reasons to use Scala unless you need particular Java libraries.
Old articles of "Why Scala" talked about how great lambdas are, pattern matching, lightweight classes without getters/setters. Now Java does those things the reason to use Scala are few.
That's because these concepts where new and difficult for many people at the time. Now that they already know them from other languages (which is great!) we can directly start with the next batch of helpful techniques in Scala.
Java is still light years behind in terms of productivity through language features. (however, therefore it has higher backwards compatibility, which also helps productivity)
* It claims to be a compiled language "like C++ and Java" - not making a distinction between targeting multiple hardware platforms and targeting a cozy VM.
* It ties itself strongly to the Java ecosystem.
So, I guess I might consider it in the 'nicer Java' category - and it does seem nicer than Java, subjectively - but not outside of that ecosystem.
That is interesting... but it's also the opposite message from the article. It even emphasizes again how Scala "leans heavily" on the JVM:
> The last thing that Scala does well is to lean heavily on the host language for both its language semantics as well as its implementation. Scala is typically run on the JVM, which together with the Java ecosystem provides a host of useful things that Scala doesn't need to worry about: etc. etc.
So, what do you do with Scala when you don't use the JVM at all, nor any Java libraries?
In Scala.js, you lean heavily on the Javascript runtime, and on the large ecosystem of Javascript libraries. That integration into the existing ecosystem is in fact one of the big selling points of Scala.js, just like how it is a big selling point of Scala-JVM
Scala has multiple host languages, and it leans heavily on each of then whenever used on that platform
I should add to lihaoyi's reply that in addition to the target environment's libraries, many Scala libraries cross-compile for Scala JVM, Scala.js, and (to a lesser extent due to its relative novelty) Scala Native.
For example I am now cross-compiling JVM/JS a fairly large project which uses, directly or indirectly, Circe, Cats, ScalaTest, Enumeratum, Shapeless, and Parboiled2, and all of that works flawlessly.
When you go to an auto repair shop, the mechanic uses four different screwdrivers or just one super ultra incredible 8-in-1 screwdriver for everything?
Why not? is the mechanic stupid? the 8-in-1 screwdriver is marvelous, has the DOUBLE of features!
At the end of the day, he has only the tools he need for his job, and thats it.
* Li's libs (os-lib, upickle, utest) have clean public interfaces, but most Scala ecosystem libs are hard to use, see the JSON alternatives for examples: https://www.lihaoyi.com/post/uJsonfastflexibleandintuitiveJS...
* The Mill build tool looks a lot better than SBT, but seems like everyone is still using SBT
* Scala minor version are binary incompatible, so maintaining Scala projects is a big pain. Upgrading Spark from Scala 2.11 to Scala 2.12 was a massive undertaking for example.
* Scala has tons of language features and lets people do crazy things in the code. Hard to win technical arguments with Scala geniuses that like using complicated language features.
* Scalatest is stil used by most projects and is annoying to use, as described here: https://github.com/lihaoyi/utest#why-utest. The overuse of DSLs in Scala is really annoying. Too many DSLs is another example of something I consider to be an antipattern, but there is no Scala community consensus on the responsible use of DSLs.
I'm optimistic about Scala. There are some folks that love the language and are continuously improving the ecosystem. Scala 3 will have to sell a better story about ditching legacy tooling and giving users a better default stack if it wants to compete with modern Go/Rust/Python.