I completely understand why, but on the other hand democracy relies on citizens being informed about what's happening. The risk is that one day, you wake up and there is no democracy any more.
Reading the news and being informed are two separate things. Being an informed citizen, the kind that democracies need to survive, also requires 1) being informed of history and 2) understanding issues in depth.
People who consume a lot of news tend to have very shallow understanding of a broad range of current events. Worse they tend to be passive receivers of news instead of active seekers of information with intent to understand the world.
As a result, they are very susceptible to manipulation through selection of what makes the news they tend to consume. They become passive pawns in political power struggles.
I'd like to get a little pedantic here and suggest it's not reading the news that's so problematic, it's 1) watching it, and 2) scrolling it. Not that print can't be effective propaganda, but it's less optimized to the task than 1 and 2. The passive pawns can't get enough of either.
On one hand I can see where you can draw this argument from. But on the other hand I don't think daily consumption of the huge quantity of news that exists is necessary for having a decent political opinion, especially given that most news is inflammatory junk (at least in my country). I just don't need a 5 page breakdown of every single event that some corpo decided to shove down our throats.
Also - and maybe I'm naive for this - I don't really need news to inform my political opinion because the current state of affairs is so far from my ideal world. Like no matter what could reasonably occur in the news, I still know who I'm voting for on polling day.
> I completely understand why, but on the other hand democracy relies on citizens being informed about what's happening.
The point being made by the author is that "following the news" nowadays has nothing to do with being informed. Instead, it became about being constantly bombarded by a barrage of noise and nonsense to constantly grab your attention.
So instead, by finding a monthly publication giving him an overview of the local, European and world news, the author is looking for a filter removing all the unnecessary noise. And the month granularity should be more than enough to allow him to be informed about important changes.
I completely agree. The author is incredibly naive on the "I asked myself how much of this actually affects my daily life". If there's one thing that absolutely affects your life, it's politics. Maybe not today, and maybe not immediately in a meaningful way, but it will affect you.
I completely disagree. The past 50-70yr of "people ought to care and be involved" type sentiment has resulted in mostly only the people who have nothing better to do and no serious problems having an outsize effect and in some subject areas completely dominating the political discourse to the detriment of literally everyone else and western society generally.
You mangled Jefferson a bit. He wrote about education, not news. He didn't imagine the the non-stop firehose of slop and advertising and propaganda we endure and call news. What passes for news today describes the opposite of critical thinking and education.
No evidence supports your sentiment. Find an example of democracy that arose from citizens "being informed about what's happening." The Athenians limited democratic participation to a small educated elite. The American Founders had the same instinct, excluding more people than they included.
Demoracy dies in front of our eyes right now, in the USA, the most media-saturated culture in history. You might blame that on an ignorant and uncritical population. You might call them uninformed, or misinformed. As Jefferson understood the problem doesn't come from people not reading the news, but rather people not educated enough to understand, think critically, or even care.
I feel like this, I honestly wish newspapers weren't bunk and there was a good "week in review" way to get the news. I find myself Doom scrolling to much.
The fact that this is downvoted really says it all. "I don't read the news" is pretty much dependent on one's profession being insulated from changing events. Which is not surprising why it's a popular opinion amongst technocrats that would rather not have democracy in the first place.
For the rest of the news, I am considering subscribing to a magazine that covers important events in Germany, the EU, or the world every few months. This kind of format filters out short-term noise and fear-driven stories.
Elections happen even less frequently than this. If your democracy disintegrates with less than a few months of warning, you were probably invaded and noticed even without the news; At this point, that would probably lead to a civil emergency notification on your phone, and by design that happens even without any apps installed.
As we said in the UK in my childhood, "Today’s news is tomorrow’s chip* paper".
In October 2024 I would not have guessed that we'd ever see masked agents killing people on the streets of major US cities, or the US administration immediately accusing the victims of being armed terrorists. Things can change rapidly. By allowing things to get this bad, we have unfortunately forfeited our right to pretend things can't change rapidly. Let's plug in; fix this situation; and then folks can go back to ignoring the news.
Last year, legal immigrants were fine. Today, their kids are kidnapped and used as bait to take them to Alcatraz. And that's not even the identity I'm mostly referring to.
Very cool stance OOP, thank you for identifying yourself as the type of centrist heaven will reject at the gate and angels will never get tired of the reaction to the shrug.
When learning basic math, you shouldn't use a calculator, because otherwise you aren't really understanding how it works. Later, when learning advanced math, you can use calculators, because you're focusing on a different abstraction level. I see the two situations as very similar.
What abstraction levels do you expect will remain only in the Human domain?
The progression from basic arithmetic, to complex ratios and basic algebra, graphing, geometry, trig, calculus, linear algebra, differential equations… all along the way, there are calculators that can help students (wolfram alpha basically). When they get to theory, proofs, etc… historically, thats where the calculator ended, but now there’s LLMs… it feels like the levels of abstractions without a “calculator” are running out.
The compiler was the “calculator” abstraction of programming, and it seems like the high-level languages now have LLMs to convert NLP to code as a sort of compiler. Especially with the explicitly stated goal of LLM companies to create the “software singularity”, I’d be interested to hear the rationale for abstractions in CS which will remain off limits to LLMs.
The "liberal" in "liberal democracy" has nothing to do with the current common meaning of "liberal" - ie, left-wing - in the USA, as it comes from classical liberalism. In short, liberal democracy means a democracy based on rule of law, separation of powers, election of representatives, freedom of speech and freedom of the press.
In theory I agree with you, in practice I do not. This is one of those terms where the precise meaning is being, or has already been, lost. For instance in the overwhelming majority of rhetoric around the lines of this article, populism is framed as being in an adversarial relationship with liberal democracy. Yet that is, from a precise interpretation of 'liberal democracy', quite nonsensical.
The entire system of democracy, of any flavor, is fundamentally populist. But populism trends towards values that are not what one would consider left-wing by US standards. And so far as I can tell, that is the only real basis for the claim of its supposed adversarial relationship with liberal democracy. It is framing "liberal" as being left-wing and US-centric left wing, and not simply of liberty.
While "liberal democracy" has a very clear meaning, "populism" has very different definitions. I'm pretty sure that the definition you have in mind is pretty different from the ones I know, if you say that democracy "is fundamentally populist".
How would you define it? Or would you disagree that democracy, in its nominal form, is a political system where political action is driven by the will of the masses?
In a liberal democracy the "will of the masses" is applied indirectly, through the election of representatives, making laws, and then applying those laws and governing in accordance to those laws. To get elected, politicians and aspiring politicians tell electors all sorts of things. Some of them tell electors that their problems have simple solutions, which go against what the intellectual elites (scientists, doctors, engineers, lawyers etc.) recommend or say is doable. Those are what are usually defined populists. Some of them actually believe that "experts" lie for some agenda. Most of them know perfectly well that those simple solutions won't work, but say what they think electors want to hear. Not all politicians/parties act like that, even if it's common to have some populists in all most parties - because populism works.
It's unclear to me how the definition you're using isn't biased to the point of meaninglessness. Let me use an example. Politicians know full well we stand no chance of meaningfully impacting climate change for a practical reason - most emissions are coming from relatively adversarial countries.
If we try to encourage them to reduce emissions via some form of ongoing compensation then we asking them to impair their development in exchange for accepting putting themselves into an exploitable dependency relationship with us. They will simply never accept this, so at best it will be superficial gestures that have no real chance of having a meaningful impact.
So is the rhetoric around climate change, and politicians/parties running on claims of being able to impact it, populist, in your usage? I'd imagine not. But is there a 'clean' way to explain how this is excluded while maintaining any degree of meaningfulness of the term as you are using it?
---
As for 'my' usage, I view populism as appealing to the genuine views of the masses, mostly in contrast to efforts to suppress or reshape them. It can certainly include demagoguery (which is largely what you're describing, but with a peculiar sort of bias built in), but it can also simply include leaders whose worldviews, or at least the worldview they espouse, happening to align largely with that of 'the masses.'
There is no such thing as "genuine views of the masses" which aren't shaped by anybody. "The masses" are made of people, all of which shape each other both at a micro and at a macro level, but the latter is disproportionately affected by mass communication - and, today, by digital social networks and their algorithms. Everybody has people who they look to for reference (ie, leaders) and that they trust on things which they don't know personally very well. Some leaders truly believe that you can eat your cake and have it too, and they tell their followers "trust me, and you'll have your cake and eat it too" - but usually these don't last long, because they're simply too stupid and/or ignorant. And this isn't an exclusive of the right or of the left: you have people who don't believe that carbon dioxide has a greenhouse effect, but you also have people who believe we can simply install solar panels and magically solve the mismatch between energy needs (temporal and geographical) and Sun irradiation.
Some other leaders know perfectly well that you can't eat your cake and have it too, and that in reality you need to make compromises between various things you would like to have - but they tell their followers the opposite, because they only care about reaching and then maintaining power. These are the demagogues, but they ally with the people from above and, together, form populist movements.
Then there are leaders who try to find the best compromise between the various things that "the masses" want/care about, and what reality allows to the best of our knowledge. Those are the non-populist, and they exist.
Well that's a rather political answer in that you're saying a lot, but avoiding the question, unless I'm expected to read into what you're saying, which can be interpreted in either way. I won't push it beyond to emphasize that I'm not picking particularly fringe arguments, as per your examples, I'm picking completely mainstream and normal rhetoric that you would hear from effectively 100% of politicians of a certain leaning.
As for 'genuine views' - contemporary politics is full of endless issues that if each person, absent any awareness of where we ended up, were to rank the importance - would end up nowhere remotely near the top. An obvious example is transsexual stuff. It's also comparably full of gaslighting on issues that may benefit the country, but hurt the people. For instance low skill immigration reduces wages of low-skill workers, while simultaneously 'growing the economy.' This is something which has been studied and confirmed endlessly, yet politicians and the media will do things like misrepresent studies or cite localized studies from 46 years ago to try to implicitly, sometimes explicitly, argue that it increases wages. It's complete gaslighting.
I view populism as stepping away from these sort of deceptions. Many if not most great presidents of the US in the past would certainly be derogatively framed as a populist now a days. JFK telling people we can go to the Moon if we truly focus on it, that America's resources can be spent better than trying to meddle in every single country around the world, and that a great country can only stay great if both the country works for the people but the people also work for the country? That certainly seems to fit the typical usage of the term now a days.
I considered that question just an example. Anyway, considering that China alone installed about 60% of the new renewable capacity in the whole world in 2024–2025, and that about 90% of new capacity is based on renewables both there and in India, I really don't see how your argument on that holds.
For low skill immigration, I fundamentally agree with you, and that's something I personally criticize left wing politicians a lot. I see the stance that Western countries can (and should) accept any amount of immigration as a left-wing form of populism.
Regarding JFK and the Moon, that's the opposite of populism - that's leading and shaping people's ideas and perceptions. How many Americans were thinking about going to the Moon before JFK made that an important issue?
It was indeed just an example, but a pointed one to try to figure out where your definition begins and where it ends. As for China's claims, beware of capacity factor. [1] Installed capacity is based on peak figures, whereas practical output tends to be a fraction of that for clean energy sources, but significantly higher for energy like coal. It makes it easy to make headlines that sound good, but don't mean what we'd think they mean, and China's not the only one doing this. In any case, CO2 levels are going to continue skyrocketing for the foreseeable future.
As for JFK - going to the Moon is something that people would somewhat naturally support. If they oppose it, it's going to be on political grounds, perhaps they think the money could be better spent in the current moment, and not because they literally just don't ever want to send people to the Moon. By contrast something like e.g. political correctness is the exact opposite. People are going to naturally oppose it, unless there is a political motivation behind supporting it. I also chose that exact example because of the comment you made about populists promising the Moon - it turns out that sometimes they deliver.
Sure, we're going to emit a lot of carbon dioxide for a long time, but we're talking about planet-level changes here, and changing the trend/trajectory has HUGE impact. At least if you care about future generations (I've got a daughter and several nephews, I don't know about you). Anyway, I asked chatgpt to estimate actual averaged capacity, and for China we have that nuclear+renewables (low-carbon) added 73% of new capacity in the last two years. I bet that the figure will become even better in the next years, as batteries and other storage methods become less expensive.
Regarding the Moon, are you sure that you're not equating "populist" with "what I personally like"?
To make a counterexample, what do you think about free universal medical care? Do you think that "the masses" would "naturally" want that, or not?
PS By the way, I'm far from what you would probably define "woke". I actually think that the excesses of wokism were a decisive contributing factor to Trump's win.
The capacity factor means you're also adding less energy than it might seem. Here is a nice graph of CO2 levels [1]. There's a breakdown by country a bit lower. The trend for CO2 levels remains quite sharp. And China is not the only factor. Asia, especially India, Africa, and many other places are due for ever greater levels of development and industrialization. For instance India has 1.4 billion people yet just 60% of the emissions of Europe. The entirety of Africa has less than 33% the emissions of Europe! These figures are not sustainable.
However, I am not that concerned about it, also as a family man. There's a finite amount of fossil fuels in the world, they will run out eventually, and become economically unfeasible long before that. So even if we do absolutely nothing, the world will likely be economically forced to start transitioning away, likely on a timeframe that is within our lives. Arguably it's already happening with places in the Mideast aggressively seeking to diversify their economies. In any case CO2 levels when dinos roamed the Earth and the oceans were full of life, were upwards of 1200ppm owing to natural processes. We're not going to hit anywhere near that even if we burn everything - in other words there's no scenario where we become Venus, or anything even remotely like it. Some places will become more hospitable, some will become less, optimal places for growing crops (and/or different types of crops) will shift, and overall there will be a lot more greenery. It's a pretty dumb experiment, but it'll be fine.
---
On free healthcare - if we are speaking hypothetically of genuinely free health care at comparable quality then obviously everybody's going to want it. The problem is that those 'political objections' are pretty tough in this case. Obviously it won't be free - it'd be paid through taxes, and the government has already shown itself in a relationship with the healthcare industry where they are, at the minimum, uninterested in reigning in healthcare costs, and government operated systems invariably balloon costs.
Outside of free likely becoming quite expensive, there's also the issue of quality and availability. Countries that have had experience running 'free' healthcare systems for decades are increasingly running into problems in modern times with declining economic growth, declining fertility, increasing health issues (obesity, psychological, etc), and so on. Even Scandiland is seeing increasing trends towards privatization in healthcare, and that's with a vastly more appropriate population for such - much less corruption, healthier, preexisting high taxes, fewer social divisions, fewer people seeking to abuse the systems in place, etc. It is still working for them, but I'm not sure if it's indefinitely sustainable at current fertility/economic trends.
---
And yeah, I definitely knew you weren't "woke" since they in general seem completely incapable of having a good old debate/discussion! I think the fear of 'wrongthink' makes people accept things that they wouldn't otherwise rationally accept which makes them unable to competently defend their views when speaking somebody of a different worldview.
Just two quick answers before we agree to disagree :)
* Life on Earth will survive any human-made change, even a full nuclear war. It's our society which won't survive if changes are too quick.
* Publicly ran healthcare systems in Europe and elsewhere are MUCH more cost efficient than the US private system, it's very easy to compare cost/performance, so the "government operated systems invariably balloon costs" is just false in this case. This isn't to say that private enterprises aren't more efficient in most cases, and the issue with private healthcare isn't that they're not efficient in terms of resources used - it’s just that maximizing profits and shareholder value when people’s lives are on the line means that you, as a health care customer, will be gouged for every penny they can get.
Brevity is a skill. Like Mark Twain wrote - I apologize for such a lengthy letter, I hadn't the time to write a short one!
In general I agree with you on both fronts - our disagreement is mostly going to be in the details and forecasts. For instance the impacts of climate change are already happening. Sea levels in parts of Florida have already risen more than 8 inches since the 50s. Yet beach front property is still selling for a premium. The point is that I expect it's going to be gradual enough that society will have time to adapt, even if the change over an extended period of time may be quite significant.
And I also completely agree that the healthcare systems pretty much anywhere in the world, government or privately operated, are dramatically more efficient than the US private system. But I don't think you can expect that to change if the government starts operating it. Medicaid's savings requires studies to measure since it's nominally more expensive/person than private healthcare. What savings there are, after a bunch of adjustments and assumptions, seem mostly explained by paying healthcare providers less per service, which is why a sizable chunk of places don't accept it. It doesn't really scream 'yeah, let's make this global and mandated' to me.
That said, I had a Norwegian friend visiting me over here in the other side of the world. He ended up getting an ear infection and went to the most premium local hospital to get it sorted out. Final charge to him = $0, even internationally. Enough to make anybody absolutely jealous, but I'm going to have a hard time believing America might be able to land on this Moon. Cheap and efficient just isn't the American way.
By the way, the emergence of LLM coding tools could make it even easier than before to reduce that dependence, as the cost of reproducing many of the mature technologies is going to cost less than it would have before. Ironically, doing that may require using US tools (like Claude Code), at least for now, but it could be a very interesting evolution/opportunity for Europe.
> the emergence of LLM coding tools could make it even easier than before
I find this highly optimistic. It will take years, maybe decades for EU to replace US clouds and tech. And if they're going to do it with LLMs, then it will take billions of euros in devs and tokens (again, all going to US tech companies).
Meanwhile, USA continues to strategically re-home TSMC to Arizona whilst simultaneously make huge investments to invigorate Intel and Micron.
Over the last decade USA and China have doubled-down on massive investments to out-compete each other while the EU seems like it's struggling to understand where to even begin.
> USA continues to strategically re-home TSMC to Arizona whilst simultaneously make huge investments to invigorate Intel and Micron.
Oh don't worry, Trump's already kneecapped both of those for a decade to come from 2025's actions alone. Y'all got time to catch up.
China, much scarier. But we all kinda let that happen over 30 years. Too late to complain now. I'd say we work together but uhh... I think we both understand (or rather, fail to understand) modern US policy these days.
What can I say, I expected more than what they actually offer. A Redshift job can fail because S3 tells it to slow down. How can I make this HA performance product slower given its whole moat is an S3 based input output interface.
As a compute engine its SQL capabilities are worse than the slowest pretend timeseries db like Elasticsearch.
Are you trying to treat an OLAP database with columnar storage like an OLTP database? If you are, you would probably have the same issue with Snowflake.
As far as S3, are you trying to ingest a lot of small files or one large file? Again Redshift is optimized for bulk imports.
Redshift does not fit into aws ecosystem. If you use kinesis, you get up to 500 paritions with a bunch of tiny files, now I have to build a pipeline after kinesis that puts all of it into 1 s3 file, only to then import it into redshift which might again put it on s3 backed storage for
Its own file shenanigans.
Clickhouse, even chdb inmemory magic has better S3 consumer than Redshift. It sucks up those Kinesis files like nothing.
Its a mess.
Not to mention none of its
Column optimizations work and the data footprint of gapless timestamp columns is not basically 0 as it is in any serious OLAP but it is massive, so the way to improve performance is to
Just align everything on the same timeline so its computation engine does not beed to figure out how to join stuff that is
Actually time
Aligned
I really can’t figure out how anyone can do seriously big computations with Redshift. Maybe people like waiting hours for their SQL to execute and think software is just that slow.
Really good to hear that. We've had AWS reps trying to push RedShift on multiple occasions after we've done our research and selected Clickhouse for our analytical workloads. Every time we have a meeting with them for some other reason - the topic of RedShift returns, they always want to discuss it again.
You realize “the pipeline” you have to build is literally just Athena SQL statement “Create table select * from…”. Yes you can run this directly from S3 and it will create one big file
I have a sneaking suspicion that you are trying to use Redshift as a traditional OLTP database. Are you also normalizing your table like an OLTP database instead of like an OLAP
And if you are using any OLAP database for OLTP, you’re doing it wrong. It’s also a simple “process” to move data back and forth between Aurora MySQL or Postgres by federating your OlTP database with Athena (handwavy because I haven’t done it) or the way I have done it is use one Select statement to export to S3 and another to export into your OLTP database.
And before you say you shouldn’t have to do this, you have always needed some process to take data from your normalized data to un normalized form for reporting and analytics.
Source: doing boring enterprise stuff including databases since 1996 and been working for 8 years with AWS services outside AWS (startups and consulting companies) and inside AWS (Professional Services no longer there)
Why are you doing this manually? There is a built in way of doing Kinesis Data Streams to Redshift
These things cost money, Redshift handling live ingestion from Kinesis is tricky.
There is no need for Athena, Redshift ingestion is a simple query that reads from S3. I dont want to copy 10TB of data just to have it in 1 file. And yes, default storage is a bit better than S3 but for an OLAP database there seems to be no proper column compression and data footprint is too big resulting in slow reads if one is not careful.
I mentioned clickhouse, data is obviously not OLTP schemed.
I don’t have normalized data. As I mentioned, Clickhouse consumer goes through 10TB of blobs and ends up having 15GB of postprocessed data in like 5-10 minutes, slowest part is downloading from S3.
I am not willing to pay 10k+ a month for something that absolutely sucks compared to a proper OLAP db.
Redshift is just made for some very specific, bloated, throw as much software pipelines as you can, pay as much money as you can, workflows that I just don’t find valuable. Its compute engine and data repr is just laughably slow, yeah, it can be as fast as you want by throwing parallel units but it’s a complete waste of money.
Thanks for having this discussion with me. I believe I don't want a time series database. I want to be able to invent new queries and throw them at a schema, or create materialized views to have better queries etc. I just don't find Snowflake or Redshift anywhere close to what they're selling.
I think these systems are optimized for something else, probably organizational scale, predictable low value workloads, large teams that just throw their shit at it and it works on a daily basis, and of course, it costs a lot.
My experience after renting a $1k EC2 instance and slurping all of S3 onto it in a few hours, and Redshift being unable to do the same, made me not consider these systems reliable for anything other than ritualistic performative low value work.
I’ve told you my background. I’m telling you that you are using the wrong tool for the job. It’s not an issue with the database. Even if you did need an OLAP database like Reddhift, you are still treating it like an OLTP database as far as your ETL job. You really need to do some additional research
I do not need JOINs. I do not need single row lookups or updates. I need a compute engine and efficient storage.
I need fast consumers, I need good materialized views.
I am not treating anything like OLTP databases, my opinion on OLTP is even harsher. They can’t even handle the data from S3 without insane amounts of work.
I do not even think in terms of OLTP OLAP or whatever. I am thinking in terms of what queries over what data I want to do and how to do it with the feature set available.
If necessary, I will align all postgresql tables on a timeline of discrete timestamps instead of storing things as intervals, to allow faster sequential processing.
I am saying that these systems as a whole are incapable of many things Ive tried them to do. I have managed to use other systems and did many more valuable things because they are actually capable.
It is laughable that the task of loading data from S3 into whatever schema you want is better done by tech outside of the aws universe.
I can paste this whole conversation into an LLM unprompted and I don’t really see anything I am missing.
The only part I am surely missing are nontechnical considerations, which I do not care about at all outside of business context.
I know things are nuanced and there’s companies with PBs of data doing something with Redshift, but people do random stuff with Oracle as well.
And you honestly still haven’t addressed the main point - you are literally using the wrong tool for the job and didn’t do your research for the right tool. Even a cursory overview of Redshift (or Snowflake) tells you that it should be used for bulk inserts, aggregation queries, etc.
Did you research how you should structure your tables fir optimum performance for OLAP databases? Did you research the pros and cons of using a column based storage engine like Redshift to a standard row based storage engine in an traditional RDMS? Not to mention depending on your use case you might need ElssticSearch.
This if completely a you problem for not doing your research and using the worse possible tool for your use case. Seriously, reach out to an SA at AWS and they can give you some free advice, you are literally doing everything wrong.
Clickhouse is column based storage, I can also apply delta compression, where gapless timestamp columns basically have 0 storage cost. I can apply Gorilla as well and get nice compression from irregular columns. I am aware of Redshift's AZ64 cols and they are a let down.
I can change sort order, same as in Redshift with its sort keys, to improve compression and compute. Redshift does not really exploit this sort-key config as much as it could.
My own assessment is that I'm extremely skilled at making any kind of DB system yield to my will and get it to its limits.
I have never used Redshift, Clickhouse or Snowflake with 1 by 1 inserts. I have mentioned S3 consumers (a library or a service, optimized to work well with autoscaling done by S3, respecting SlowDown -- something Redshift itself is incapable of respecting -- and achieving enormous download rates -- some of the consumers I've used completely saturate the 200Gbps limits of some EC2 machines at AWS). These consumers cannot be used in a 1-by-1 setting, the whole point is to have an insanely fast pipelining system with batched processing, interleaving network downloads with CPU compute, so that in the end, any kind of data repackaging and compression is negligible compared to download, so you can just predict how long the system will take to ingest by knowing what your peak download speed is, because the actual compute is fully optimized and pipelined.
Now, it might just be Redshift has bugs and I should report them, but I did not have the experience of AWS reacting quickly to any of the reports I've made.
I disagree, it's not a me problem. I am a bit surprised after all I've written that you're still implying I want OLTP, am using the wrong tool for the job. There are just some tools I would never pick, because they just don't work as advertised, Redshift is one of them. There are much better in-memory compute engines that work directly with S3, and you can create any kind of trash low-value pipelines with them, if you reach mem limits of your compute system, there are much better compute engine + storage combos than Redshift. My belief is that Redshift is purely a nontechnical choice.
Now, to steelman you, if you're saying:
* data warehouse as managed service,
* cost efficiency via guardrails,
* scale by policy, not by expertise,
* optimize for nontechnical teams,
* hide the machinery,
* use AWS-native bloated, slow or expensive glue (Glue, Athena, Kinesis, DMS),
* predictable monthly bill,
* preventing S3 abuse,
* preventing runaway parallelism,
* avoiding noisy-neighbor incidents (either by protecting me or protecting AWS infra),
* intentionally constrained to satisfy all of the above,
then yes, I agree, I am definitely using the wrong tool but as I said, if the value proposition is nontechnical, I do not really care about that.
> My own assessment is that I'm extremely skilled at making any kind of DB system yield to my will and get it to its limits.
Yes an according to my assessment I’m also very good in bed and extremely handsome.
But there is an existence proof seeing that you are running into issues yet millions of people use AWS services and know how to use the right tool for the job
I’m not defending Redshift for your use case, I’m saying you didn’t do your research and you did absolutely everything wrong. From my cursory research of Clickhouse, I probably would have chosen that too for use case
I did not do anything wrong. I had no choice with Redshift and had instructions from above. I made it work really well for what it can do and was surprised how much it sucks even when it has its own data inside of it and has to do compute. As a completely closed system, it's not impressive at all. It has absolutely shameful group-by SQL, completely inefficient sort-key and compression semantics, and absolutely can't attach itself to Kinesis directly without costing you insane amounts of money, because as you already know, Redshift is not a live service (you won't use it by connecting directly to it and expect good performance), it's primarily a parallel compute engine.
Your assessment of me is flawed. You haven't really shown any kind of low-level expertise on how actually these systems work, you've just name dropped OLTP OLAP as if that means anything at all. What is Timescale (now TigerData), OLTPOLAPBLAPBLAP? If someone tells you to use Timescale, you have to figure out how to use it and make the system yield to your will. If system sucks, it yields harder, if system is well designed, it's absolutely beautiful. For example, I would never use Timescale as well, yet you can go on their page and see unicorns using it. I have no idea why, but let them have their fun. There's successful companies using Elasticsearch for IoT telemetry, so who am I to argue I wouldn't do that as well.
There's nothing wrong with using PostgreSQL for timeseries data, you just need to know how to use it. At some point, scaling wise, it will fail, but you're deciding on tradeoffs.
So yes, my assessments have a good track record, not only of myself, but of others as well. I am extremely open to any kind of precise criticism and have been wrong bazillion times and I take part in these kinds of passionate discussions on the internet because I am aware I can absolutely be convinced of the other side. Otherwise, I would have quit a long time ago.
It's solar generation which doesn't work in winter in Ukraine, not the heat pumps. And as Russia is targeting Ukraine's energy infrastructure as their main tactic, with the goal of getting the population to accept Putin's domination to avoid freezing to death, the heat pumps would only be useful if local solar worked.
The issue is that works perfectly well when solar is a small % of the grid, but when that number grows, then you need grid scale solutions and coordination for things to continue working well. And that requires both technical skill and political will.
This isn’t remotely true. Solar / wind / nuclear / coal / gas / any electrical source including from neighboring grids can be inbound or outbound from your grid using, the grid. There are capacitors and transformers, relays and transmission lines. Any energy source can provide power. Solar used to give money back to its owners by selling power back to the grid but they killed that initiative quickly and will just use your energy you provide.
The issues you describe are from coal, oil, and gas lobbyists saying solar isn’t viable because of nighttime. When the grid is made up of batteries…
If every house had solar and some LiFePo batteries on site, high demand can be pulled from the grid while during low demand and high production, it can be given to the grid. The energy companies can store it, hydropower or batteries, for later. We have the ability. The political will is simply the lobbyists giving people money so they won’t. But we can just do it anyway. Start with your own home.
Not all prime movers are the same with regard to grid dynamics and their impact.
Solar, wind, etc., almost universally rely on some form of inverter. This implies the need for solid state synthetic inertia to provide frequency response service to the grid.
Nuclear, coal, gas, hydropower, geothermal, etc., rely on synchronous machines to talk to the grid. The frequency response capability is built in and physically ideal.
Both can work, but one is more complicated. There are also factors like fault current handling that HN might think is trivial or to be glossed over, but without the ability to eat 10x+ rated load for a brief duration, faults on the grid cannot be addressed and the entire system would collapse into pointlessness. A tree crashing into a power line should result in the power line and tree being fully vaporized if nothing upstream were present to stop the flow of current. A gigantic mass of spinning metal in a turbine hall can eat this up like it's nothing. Semiconductors on a PCB in someone's shed are a different story.
Large solar sites are required to be able to provide reactive power as well as maintain a power factor of 0.95 to avoid all of the issues you mentioned.
> There are also factors like fault current handling that HN might think is trivial or to be glossed over, but without the ability to eat 10x+ rated load for a brief duration, faults on the grid cannot be addressed and the entire system would collapse into pointlessness.
I don’t understand what you are talking about here. I don’t work in the utility world, I sell and run commercial electrical work, but handling available fault current in my world is as simple as calculating it and providing overcurrent protection with a high enough AIC rating or current limiting fuses. I don’t see why the utility side would be any different.
The utility side has found that vaporising short circuits is a useful feature, as that includes e.g. twigs hitting a power line.
There are breakers, of course, but they react slowly enough that there will absolutely be a massive overdraw first. Then the breaker will open. Then, some small number of seconds later, it will automatically close.
It will attempt this two to four times before locking out, in case it just needs multiple bursts. It’s called “burning clear”, and it looks just as scary as you’d think… but it does work.
The lack of rotating mass in a solar site means the rest of the spinning mass of the generators needs to compensate to maintain frequency and voltage, right? So when clouds roll in and the solar field output drops quickly, it’s a challenge for the rest of the system to compensate since any other generator that spins will slow down much more slowly, giving the grid more time to react.
Also, I was not aware that inverters can only handle fault current that is 1.1x the nameplate capacity, that’s a big limitation. I can buy a 20A breaker with 200kaic, which is 10,000x higher than the breaker ampacity, which is extremely helpful for handling fault current.
These do not address the concern of fault current handling. This is a much more localized and severe condition than frequency deviation. Think about dropping a literal crowbar across the output of a solar inverter. This is a situation the grid has to deal with constantly.
I'd argue that nothing that uses semiconductors would be suitable for the task. They get you to maybe 2x rated current capacity for a meaningful duration. A spinning turbine can easily handle 10x or more for a much longer duration.
We could put so many redundant transistors in parallel that we have equivalent fault handling, but then we are into some strong economic issues. There's also no room for error with semiconductors. Once you start to disintegrate, it's all over ~instantly. There is no way to control this. A synchronous machine can trade downstream maintenance schedule for more current right now. The failure is much more gradual over time. A human operator can respond quickly enough if the machine is big enough.
Grid forming inverters provide 1/3 to 1/4 the fault current of a similarly sized generator.
The other trivial solution are synchronous condensers. Or just let the generators and maybe even turbines of future emergency reserve thermal plants spin with the grid without consuming any fuel.
Just ensure the proper margins exist in the grid and call in ancillary services as needed.
Yeah, DC vs AC power. 12v vs 120v or 240v. This isn’t a limitation. All energy sources must be converted to useable energy to the grid somehow. So every power source requires an inverter or a down stepper or a really advanced rectifier or all of the above.
The people you're replying to aren't talking about converting from AC to DC or stepping voltage up or down. Rather, they're talking about grid stability. You can have mechanisms to convert from AC to DC and to step voltage up or down, but still have a unstable grid. We had a notable example of that last year: https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou....
One way to think about this problem is that our electrical grids are giant machines—in many ways, the largest machines that humanity has every constructed. The enormous machine of the grid is comprised of many smaller connected machines, and many of those have spinning loads with enormous mechanical inertia. Some of those spinning machines are generators (prime movers), and some are loads (like large electric motors at industrial facilities). All of those real, physical machines—in addition to other non-inertia generators and loads—are coupled together through the grid.
In the giant machine of the grid, electricity supply and demand have to be almost perfectly in sync, microsecond to microsecond. If they're not, the frequency of the grid changes. Abrupt changes in frequency translate into not only electrical/electronic problems for devices that assume 60 Hz (or 50, depending on where you are), but into physical problems for the machines connected to the grid. If the grid frequency suddenly drops (due to a sudden drop in generation capacity or sudden drop in load), the spinning masses connected to the grid will suddenly be under enormous mechanical stress that can destroy them.
It's obviously not possible to instantaneously increase or decrease explicit generation in response to spikes or drops in load (or alternatively, instantaneously increase or decrease load in response to spikes or drops in generation). But we don't need to: all of the spinning mass connected to the grid acts as a metaphorical (and literal) flywheel that serves as a buffer to smooth out spikes.
As the generation mix on the grid moves away from things with physical inertia (huge spinning turbines) and toward non-inertial sources (like solar), we need to use other mechanisms to ensure that the grid can smoothly absorb spikes. One way to do that is via spinning reserves (e.g. https://www.sysotechnologies.com/spinning-reserves/). Another way to do it is via sophisticated power electronics that mimic inertia (such as grid-forming inverters, which contrast with the much more common grid-following inverters).
Great explanation about the grid being a giant machine that couple smaller machines with each other. About your last point, the buffer, I think batteries (chemical and also physical) seems to be the main key going forward.
I actually have a patent in this space for demand response. I know. I was being a bit cheeky. Stability is still a concern as unstable loads and generation needs to be mitigated as well as properly phased.
Also, power companies did not necessarily kill energy export incentives. Here in Massachusetts my meter “runs backward” when I export to the grid. This does not earn me money but it does earn me kWh credits, which means that if I am net negative for energy import in the summer and net positive for import in the winter, I can be net zero (or close to it) for the year.
In MA and a few other states, polluters are also required to buy “renewable energy credits.” Since I have a solar array I can sell my RECs whether I export energy or not. It’s my first year with a solar array, so I’m not sure how much to expect, but neighbors tell me that they earn between $500-$1000 a year.
In a future with solar and batteries, daytime and nighttime electricity pricing cannot be equal - else nobody would bother to have a battery (grid scale or at home).
Rules and regulations could solve that problem (meter not allowed to go backwards, solar companies are forced to pay some kind of battery credit, etc), but the free market will always outcompete.
Therefore, I forsee the future lies in 'smart' electricity meters which can charge different rates at different times of day - perhaps with minute by minute live pricing.
Here in Ireland, night-time power prices are much lower than daytime.
I’m happy enough that a battery will serve me equally well in both modes, but there’s definitely going to be a period where all it does is support self-consumption.
This only happens if a small percentage of people have live pricing. If most people have live pricing, most people have an incentive to act on price changes - for example by turning the heating off in unused rooms to save money.
In turn, that means that at times of crisis, prices will be high, but not 1000x high.
Gasoline is another resource with live pricing, and suggesting "I want a subscription where I pay $3 per gallon fixed for a year, no matter how much I use and no matter what happens to the price of oil" wouldn't be something a fuel station would entertain, because they know that when the price was under $3 you'd buy elsewhere, and when the price was over $3 you'd buy millions of gallons and resell at a profit.
> If most people have live pricing, most people have an incentive to act on price changes
It's not latency free to act on price changes. If they spike while people are asleep, what do you expect would happen?
And would people get a notification everytime the price changed at all. The logistics are hard.
Some solar inverter systems already have a data connection to get live pricing information from the grid operator. It’s not that big of a problem to implement, although it definitely isn’t pervasive yet.
Minute by minute pricing is not crazy to expect and integration with HVAC, battery systems, and inverters isn’t crazy to expect to occur.
There's a neat way to do this that is super simple... The electricity company publishes an equation that determines the price based on the AC frequency. Ie. price_per_kwh = tan(min(max((-60 + system_frequency) * 1000, -pi/2), pi/2)).
Now every device in your home knows the price. For this to work, everyone must get the same price across the whole grid, and there must be sufficient grid capacity for energy to flow freely which isn't always the case. It will also cause issues with some very old (ie. 60+ year old) clocks with mechanical timers.
All of these issues can be fixed by updating the formula:
The published_offset would be unique to each district and adjusted from time to time to keep old clocks working properly, and sometimes to deal with limited transfer properties of the grid...
But the neat thing is that even if you don't take into account the published_offset, you still make nearly optimal economic decisions.
In reality most people will buy "smart" appliances which turn on and off based on price - eg. a water heater which picks the cheapest hour to reheat the tank for the day, or a fridge/freezer which cools everything more in cheap hours, an EV charger which starts selling rather than buying power at the highest priced hours, etc. It's all fairly simple software as soon as energy companies do live pricing, so pretty much every wifi gadget will do it.
People will choose it based on claims in the shop like "Smart timing cuts energy bills by 25% on average!".
It only takes a smallish percentage of demand to be reactive like that and really big price swings won't really happen.
Somewhere they'll still be grandad manually putting the dishwasher on at a cheap hour or turning the hot tub off whenever he sees the price is high, but I expect most to be automatic.
The whole gimmick with that supplier was that they exposed their customers more or less directly to grid pricing. You don't need to do that to charge different prices during different parts of the day.
My (very-)local utility will give you an overall rate discount if you send them a screen shot of your car charging app showing it's only charging in a certain range of hours. Surprisingly, this works fine, though supposedly they've got eventual integration plans, "nah, we'll trust you for now" is a viable MVP...
> I forsee the future lies in 'smart' electricity meters which can charge different rates at different times of day - perhaps with minute by minute live pricing.
That's what I was responding to, not the day/night predetermined pricing.
They could still have a price limit, paid for by charging a bit more when prices are lower, it doesn't have to be priced directly to the grid to have impact on usage.
A max price guarantee would also give the supplier an incentive to have their planning in order.
It's hard for people to really understand this because utilities and grid operators are using this is a headline justification for electric capital projects. In New York, they've deferred capital projects for decades and we're absorbing a massive distribution charge increase. I think my electric delivery portion of the bill is up 40%.
Well there are real challenges here. Generators which rely on massive spinning things naturally provide the grid with inertia; they resist changes to grid frequency. Power sources which rely on inverters or otherwise dynamically adapt to grid frequency don't naturally provide the same inertia.
This is a solvable problem, but it requires a solution nonetheless.
That too can be replicated. There are a few centrifuges out there. Not batteries, but spinning masses meant to keep the frequency stable. Some are looking at using air conditioning motors, of which we have millions, as such a spinning mass.
The frequency (50hz or 60hz) comes from those rotational forces from the generators and until we can eliminate them, we have to play nice with them.
Luckily, we have GFMI’s. Grid-forming inverters that can emulate 60hz push pull but you’re right that it’s more than just voltage since we are dealing with high voltage alternating current.
Solar is highly distributed. At the most basic level with a solar & battery system the production and consumption and CONTROL are all yours. You own it and it's literally on your property.
Refinements on ways to sell it to neighbours / recharge various EV's / use it for new purposes are all up to you.
There are lots of analogies to self hosting or concepts around owning and controlling your own data, when it's owned by you, you retain soverignty and full rights on what happens.
I'd expect most tech people will value the distributed nature of solar over equivilents, that by design require centralisation and commerical/state ownership and control.
Get your solar, back increasingly distributed approaches, let those pushing centralised agendas be the ones to pay for their grid. Eventually they are forced to change.
As we're finding in Australia, our high solar uptake by citizens.. is pressuring governments to respond, lest their centralised options become redundant. What we found is that as more people moved to solar, the power companies lumped the costs for grid maintenance onto those who hadnt moved yet, actually contributing to even further accelerated solar adoption and pressure to rework the system. Big corporates can lobby for themselves you dont owe them your custom.
This is not the problem. The problem is that everyone moves to solar for most of the year not using or paying for the infrastructure, then in cold winter nights everyone expects the grid to be able to supply as normal.
Cost. Useful life.
I thought about an off grid system. Batteries are expensive. Also, unless you live in a dry place in the equator, You'll need to account for things like winter, long rainy spells, so either you add more batteries to account for multiple days (weeks? months?) of low generation, or you'll need a diesel/gas generator, or have a hybrid system instead, which basically means you're using the utilities gas generator instead.
Then, subsides are drying up. Systems have a useful life, your panels can be damaged by storms, for maximizing battery life you need to ensure you don't discharge it below 20%, and neither charge it over 100%.
So, in the end, the grid needs to be there anyway, but as most grid costs are fixed, whenever you use it now, it is going to be more expensive.
Generating your own power does not necessarily mean cutting ties with the grid. I think for most people in most places being off-grid would be a real challenge. I’m not sure how Australia does it but in my neck of the woods (northeast US) staying grid-tied is the norm.
I have a relatively big battery (12kWh) which is enough to see me through the evening during the summer months. We do not get quite enough sunshine where I live to be off-grid during the winter, but I can use the battery to hedge against grid outages which are common here in the winter due to storms (eg heavy ice taking down power lines).
The battery in the winter could be used to charge during low cost time periods, assuming your have time of use energy prices. I see people in the UK doing that all the time because the peak prices are very high. I think California is the same.
Batteries have come down a lot in cost, at least the raw ones:
We do the same in Pennsylvania - I have about 10 kwh of battery. I can't put solar on my roof, so I only have a very small 800w array on top of my garden. I run it as an off grid system that can recharge from shore power, so I have to use all of the energy it produces or it goes to waste. But it saves some money and is enough battery to let me time shift to take advantage of time of use power rates, and it gives me very good run time for refrigerators and internet during outages.
There seem to be a few sweet spots in solar - a tiny array that you use all of without having to grid tie it is really cost effective. (The cost of grid tied solar adds 5-10k to the system cost). Otherwise go big. :)
There was an article that described that in UK one needs 1 megawhat-hour battery over the winter to be grid independent. Judging by current trends in few years that will be below 40K USD. While this is indeed very expensive in most of US due to much more sun available the required battery would cost below 20k. One can also have a backup generator that can run constantly at maximum efficiency to replenish the battery. Then the whole system can already be below 20K. While expensive, it provides true independence and I suspect grid cost and centralized power is more expensive for society.
This is for a single home off-grid, meaning solar over-production is already implied. You need enough solar available to charge that 1MW battery in time for it to be useful during those seasonal differences which is going to be multiples of your peak summer generation.
No need to go off grid. You getting solar and battery already positions you to be able to ‘exit the grid’. The experience in Australia has been that the major retailers keep charging infrastructure costs to those who still rely on them. The mass of solar adoption grid and off-grid shifts the playing field.
That appears to be true in places in the US that have time-of-use rates. Sadly where I live, there are no time-of-use rates for residential customers, otherwise I would absolutely do this.
> I'd expect most tech people will value the distributed nature of solar over equivilents, that by design require centralisation and commerical/state ownership and control.
I do, but I do not find value in rich folks who can afford solar wanting their cake and eating it too.
If you get a solar setup, get batteries. Then disconnect from the grid entirely. You should not be able to use the grid as a free backup energy source for the last 5% of the time you'll need it. Those last digits of reliability are the expensive hard problem to solve. That, or be charged appropriately for adding your potential usage to the capacity market. I understand that this is not legal in many places, and that folks disconnecting from the grid also cause the grid to collapse at some point as well. But at least there would be less of an individual perverse incentive involved.
Home solar folks seem to love their free battery though. Or even worse - getting paid to dump power to the grid when it's value is the smallest. Net metering is not the way to go - home solar should be being paid something around instantaneous wholesale pricing at best, plus fees to manage the more complex management of the grid they cause via being thousands of kilowatt-scale install vs. a single 50MW solar farm.
So far in the US at least, many solar programs have simply been a handout to relatively rich folks subsidized by poorer grid consumers. It's really put a sour taste on something that should be for the greater good. I don't mind that those subsidies were used to jump-start the industry, but that time has long since passed.
tldr; if your total system cost to be fully off-grid and never have to worry about a power outage is not substantially more expensive than being grid-connected, you are likely being highly subsidized by other electricity consumers.
While many rich are benefiting, they are still driving demand, that is funding continuous improvements that funding further efficiencies innovation and driving down the cost per kWh. In a very really way this makes solar cheaper and cheaper, the benefits of this one, unlike the debunked economic namesake do infact trickle down.
I think this was the case when solar panels were much more expensive. But home solar in the US has long ceased to be a useful driver of funding efficiencies and innovation. The cost of panels is now tiny and you are mostly paying for extremely overpriced installation and permitting. The 30% federal subsidy alone is enough to pay for an equivalent amount of utility scale solar outright. Australia has similar labor costs to the US but home solar is 1/3rd the cost to install.
The Australian grid shows that when solar is the dominant part of the grid, it can still work pretty well. But you need to plan for when the sun is not shining and adapt to the notion that base load translates as "expensive power that you can't turn off when you need to" rather than "essential power that is always there when needed". The notion of having more than that when a lot of renewables are going to come online by the tens of GW is not necessarily wise from a financial point of view.
That's why coal plants are disappearing rapidly. And gas plants are increasingly operating in peaker plant mode (i.e. not providing base load). Also battery (domestic and grid) is being deployed rapidly and actively incentivized. And there are a lot of investments in things like grid forming inverters so that small communities aren't dependent on a long cable to some coal plant far away.
The economics of all this are adding up. Solar is the cheapest source of energy. Batteries are getting cheap as well. And the rest is just stuff you need to maintain a reliable energy system. None of this is cheap but it's cheaper than the alternative which would be burning coal and gas. And of course home owners figuring out that solar + batteries earn themselves back in a few short years is kind of forcing the issue.
Australian grid prices are coming down a lot because they are spending less and less on gas and coal. The evening peak is now flattened because of batteries. They actually have negative rates for power during the day. You can charge your car or battery for free for a few hours when there's so much solar on the grid that they prefer to not charge you than to shut down the base load of coal/gas at great cost. Gas plants are still there for bridging any gaps in supply.
Australia is lucky, we get hot summers and mild winters, which means our electricity demand is highest precisely when we get the most solar.
That's why something like 30% of Australian houses have solar.
That said, grid prices spiked recently. Both a combination of subsidies expiring, and fewer people buying grid power (because of solar) causing fixed costs to be shouldered by fewer people.
It should be pointed out that while electricity prices went up on paper, a lot of people aren't paying those higher prices because they are on solar!
Temperature has nothing to do with the performance of solar. Solar panels perform better when they are cooled.
Also worth pointing out that much of the US is below 49 degrees latitude. Which is south of most of Europe. Washington DC and San Francisco are at a similar latitude (38) as Melbourne (-37). Most of the US is perfectly situated for getting pretty decent solar power around the year. Yes it gets cloudy sometimes. It's usually not continent wide. You can compensate with cables and batteries. The US is far behind because of policy and their local energy monopolists blocking progress. Not because of anything to do with the weather or geography.
Prices have a lot to do with scarcity. Which with monopolists has more to do with the lack of a free market than with a scarcity of resources. Installing solar is about 3-5x more as expensive in the US as in Australia. The permitting process in the US is more expensive than the total cost of buying and installing in Australia. That's a policy problem in the US. All the hand wringing around that topic isn't helping a lot. A bit of pragmatism could improve things a lot and probably very quickly. Australia is showing how to do that. And yes, they have rain there too and you can go skiing pretty close to Melbourne. That isn't stopping them.
I wasn't talking about the performance of solar, only the demand for electricity.
Someone pointed out that the big problem with solar isn't how do we store daytime solar for nighttime use - this is easily solved with batteries. The real unsolved problem is how do we store summer solar for winter use.
Australia doesn't have this problem, not to the extent of other colder places, because we don't need a lot of heating in the winter.
When you say 'Australian grid prices are coming down a lot' I don't think you're talking consumer prices.
I don't have the exact 'before' numbers on me, but our peak electricity costs went up from around 42c/kWh to 56c/kWh around 18 months ago.
At the same time that feed-in was halved from 4c/kWh to 2c. Having said that, I'm pretty sure 'Shoulder' and 'Off-Peak' went down slightly.
(I'll update this when I can access my spreadsheet with the actual numbers and dates)
I should also say that I'm fairly insulated from this price rise having recently gotten a battery installed, plus moving to a special EV plan, so I charge the car and the house battery at the very cheap off peak rate (special for EV owners) and run the house entirely off battery, topped up with solar.
It's a privileged setup, but one that I planned and worked towards for a fair while, having seen ever increasing electricity prices always on the horizon (even before AI started eating all the resources).
But it's not happening in areas that keep coal on their grid - Wyoming, Texas, Utah, China, etc.
It's primarily the places that try do both solar an fossil fuel retirement that are experiencing high energy prices - California, UK, Europe, Australia, etc.
To be clear: Australia has always had fairly high electricity prices, and Australia is also not specifically doing 'fossil fuel retirement', although there are coal plants closing they're closing because they're reaching the expected end of their natural life.
High energy prices happen when you don't do the basics to be ready for a change before making it. Or when you skip basic maintenance until everything falls apart. I'm sure there are many other complex factors I don't know about.
Texas also has the most coal power of any state. As with China, success with renewables appears to depend on a policy of compatibility with fossil fuels rather than opposition.
(Home) batteries are quickly becoming cheap and per-hour electricity rates can be implemented at a reasonable time. With that, the grid owner can influence the grid stability without having to build capacity or generation itself.
My goal is to do wholly owned solar and batteries at home, only using the grid as backup, if I move out of the city. But I think the big problem with this new demand is that it’s for data centers. I can’t see that working for them.
We see that quite often here in the summer as the energy price sometimes drops to minus 60ct/kWh (more often it hovers around -5 to -10). It is pretty much "please use everything now" to avoid grid issues. It often happens on very clear days with lots of wind.
This ignores capital and opportunity cost. Building a GPU data center or chemical plant costs a lot. If you only use it 20% of the time, you're effectively paying 5x more for that capital equipment.
The problem is the capital cost of any of that type of equipment sitting around idle or under-capacity, ready to go when the electricity price goes down. It's likely more profitable to run them most of the time, even with positive electric rates, and then only stop using them when rates are exceptionally high ("load shedding").
This is why you see most opportunistic electricity consumption systems doing resistive heating - this equipment is inexpensive.
That ‘negative value’ electricity could also be used to do something else. And actually requires a lot of capital to produce. It isn’t actually free, it’s a side effect of another process that has restraints/restrictions.
Yes…. And capital costs to capture that ‘moment’ productively are likely not in favor, if this situation exists long term.
For example, Free power for an hour is useless if someone is running an aluminum refinery, because you can’t just start and stop it; and it costs so much capital to make that only operating 1 hour out of 24 is not economic.
And that is for a situation where electrical power costs are one of the most dominant costs!
The cost of CO2 capture, and conversion into usable fuels, is in the cost of the setup of the infrastructure etc (as well as cost to run the pumps once setup, which in this case is where the free electricity goes).
The return on such an investment is likely negative, because the synthesized fuel does not sell for much (compared to the same fuel that is extracted off the ground and refined - look at natural gas as prime example). Therefore, even if electricity is negative (ala, free), you cannot make money from doing it.
Either the cost of the carbon emissions is captured as part of the cost of fossil fuel extraction (and returned to this carbon capture/conversion system) to make it break even, or something else has to happen (like massive efficiency increase in doing such conversions) in order to make it economical.
When the price of a thing is negative, the entity facing the negative price is being paid to consume it.
We don't have enough automatic integration yet to make it happen, but: Residentially, that'd be a great time to charge millions of EVs and raise the temperature of water heaters. It'd be perfect for getting a head start on heating the glass kiln for Monday morning, or to supplement the used railroad ties and other fuels that might be feeding a lime kiln.
It's pretty easy to think of loads that feature scale and/or quantity, and the ability to switch on and off rather quickly. Even if the negative price event only lasts for an hour. (Even if it only lasts 5 minutes.)
The CapEx (and planning/timing) required to actually use it would almost certainly dwarf any actual gains - notably, because we’d already be selling the electricity for a profit if we could use it productively, the negative price is precisely because the equipment just isn’t there yet.
Also, once said capex was spent so we could actually use that electricity - it’s marginal cost/value would no longer be negative.
Weird huh?
Notably, if these kinds of situations do keep occurring (aren’t just random), someone almost always ends up spending the capital to capture it, because this is obvious.
You just don’t see all the finance geeks pulling out their calculators and talking about their plans because they know secrecy is an important strategic and tactical advantage when arranging investment and building out capital equipment.
It does tend to level itself out, yes. With sufficient adoption of cost-oriented controls, negative price conditions cease to exist and money flows in the normal direction.
And no, I don't think that's weird at all -- that seems like just a natural path towards the desirable goal of balancing generation and load, and turning a negative into a positive.
In terms of implementation: There's already lot of low-hanging fruit. It only takes software to get connected things like EVs and hybrid, grid-tied battery+solar systems to be centrally commanded to take advantage of negative price opportunities.
The hardware already exists, and more of it is being built every day. And software, once written, can be copied infinitely for free.
We already have sellers who would like to sell surplus energy, but find themselves in situations where they cannot. We also have avid buyers who would like to buy energy cheaper, but who cannot take advantage of the surplus condition when it exists.
That's not a inescapable curse. It is instead an opportunity for a new market optimization.
If I wake up on some hypothetical future day and find my hypothetical EV charged to 90% instead of the 80% I might normally seek to limit it to, and this 10% increase happened for free and without any action on my part, then: I win a little bit, and the generating station with the surplus also wins a little bit, and the distribution/transmission systems still get paid for their part.
I'm happy with my tiny win. The generating station is happy with many thousands of their own tiny wins. It's good stuff.
If this happens often enough (or for long-enough periods) for me in my region, then I might seek a normal limit of 70% or even less and be able to opportunistically absorb even more of the surplus when it happens.
The advantage that participation offers me does decrease over time as things balance (if they can ever become balanced), and that's OK too: The generating station still wins.
(We already have systems that do exactly the opposite of this in the consumer space, and we've had them for a very long time. The oldest I'm aware of are radio-controlled relays for water heaters, and the newest I'm aware of involve smart thermostats. These are utility-controlled systems that are intended to shed load instead of generate load. But if it works in one direction, then it can also work in the other direction.)
The problem here is that the production of hydrocarbons, ammonia, etc. from electricity can only make back its high upfront investment when it runs basically 24/7. This is a challenge for renewables.
In China which recently opened a large off-grid green ammonia plant in Chifeng, they use multiple tiers of energy storage to ensure constant electric power availability.
Well as we all know the political will in this country seems to generally be "let's all commit suicide together", but perhaps mass installations of solar will provide material reason to improve conditions somewhat.
The bigger issue, at least in the US, is that there is a huge lack of supply in the equipment to connect to the grid at the moment. Backlogs are still 1-3 years after order, not terrible but still an issue deploying.
That is definitely not the bigger issue. If we had faster grid tie completions the problem would be even worse. If you don't believe me look at the very nearly daily negative power pricing inany areas of California.
We simply don't have the transmission and storage for significantly more grid tied solar. It's pointless to build more for purposes of grid supply, we need to build transmission and storage first.
i wonder if ppl's electricity consumption habits will change in response to this, idk like turning the heat way up during the day or using high power appliances more during the day
We have a solar electric plan - the price per kWh is much higher during the duck curve in return for cheap rates during sunshine hours. The rates are something like 1x during night, 0.5x during sunshine, 4x during the morning and afternoon peaks.
We have our heat pump water heater running during the cheap hours, and also change our use of air conditioning/heating to accommodate.
It would probably not work in our favor if we didn't work from home and were out of the home all day.
That is something you can reasonably do, but it's only useful in winter.
> or using high power appliances more during the day
Well, given that people have to work during the day, I doubt that that will work out on a large enough scale. And even if you'd pre-program a laundry machine to run at noon, the laundry would sit and get smelly during summer until you'd get home.
The only change in patterns we will see is more base load during the night from EVs trickle-charging as more and more enter the market.
I've got solar. We switched things like pool pump, hot water and so on (things already on timers) from night to day.
Dishwasher can also gave a programmed start, so that can also shift from after-dinner to after-breakfast.
I also work some days from home, so other activities can be moved from night to day. We use a bore-hole for irrigation, laundry in the morning etc. Even cooking can often be done earlier in the day.
Aircon is the least problematic- when we need it, the sun is shining.
So yes, habits can shift. Obviously though each situation is different.
At least in the US there is a push to make electric appliances smarter already. So for example, the electric hot water heater responding to the strain on the grid. The same could happen for AC, heat, EVs and other higher load appliances. At scale that can help out the grid immensely either in times of peak load or dip in demand.
I do not see a point of smart appliances besides electrical car. 10 KWt-hour battery will cover all the needs to smooth the demand from all home appliances and costs below 1K usd. It will allow also to significantly reduce maximum power that has to be supplied to a house while allow to increase peak consumption while heavy cooking/AC/heating.
At least in the US most of this is still on the research phase but if you can get a standard adopted for all new equipment you can easily adjust these high draw appliances to act as a virtual power plant. It would be a trivial implementation compared to getting batteries in homes.
So your implication that other sources of energy currently do not need scaling coordination somehow? I fail to see how that is true, maybe you can provide some insights?
Wind and solar are not in ur control. I can turn on a generator and get power. Some plants might need weeks to start up - but this is in my control. I have no idea how windy it will be in five days.
My point is that scaling coordination issues exist for everything, including all sources of energy production.
Singling out solar and continuing to not prioritize it will inevitably lead to ongoing grid issues. Whereas this has been mostly solved for other sources, due to lobbying and legacy. Thus my confusion about the OPs half-baked point.
Taking Greenland by force against a NATO (supposed?) ally would be the end of "the West" as a largely aligned block since WWII. The effects would be felt by everybody, including technologists.
reply