For most of the industry’s history, electric power in the US had largely been provided by vertically integrated utility companies that handled every part of the electricity supply: generating it, transmitting it, distributing it to customers, and managing the overall system. Utilities were granted monopoly status in their area of operations, and in return had their rates regulated by state public utility commissions. Most utility companies were private enterprises known as investor-owned utilities (IOUs), though there were several other models, such as municipally-owned utilities, rural co-ops, or the federal Tennessee Valley Authority.
What a complicated mess, we should go back to some sort of quasi public monopolies. Nobody knows how to complicate things like the US--tax code, medical insurance, utilities, no metric system, etc., etc.
This was really informative and is going to be useful to me personally, since I'll be starting an internship at an ISO in a couple days. Maybe you were planning on doing this already, but I am interested in a comparative analysis of the grid -- i.e. how are other countries running their grids? Anyway, thanks so much for this informative series!
It isn't absolutely necessary to look abroad; different US states have very different policies.
Texas, for example, is always lambasted for being largely separate from the US grid - but one of the consequences of that and Texas' "free market centric" setup is that Texas was very rapid in adopting natural gas grid electricity generation, and later on has become the single largest alternative energy producer (solar PV + wind).
California, in contrast, is the opposite of "free market centric" - no doubt at least in part due to Y2K Enron as is alluded to above.
But if you want to see a very different policy - look to China. They're clearly doing "all of the above": the most coal, the most solar PV, the most wind - and soon to be the most nuclear. And nobody is going to accuse the Chinese grid of being "free market centric".
Agree with what c1ue said: US states have significant variance in market design + China is an interesting market.
I'll add a bit about the EU perspective:
a) you can think of EU-level as US-federal and MS-level (EU Member State) as US-state level
b) Since the late 90s/early 2000s there has been a strong policy push to unify the separate energy markets of the various Member States and move it closer to the ideal of the "European Copper Plate" (i.e. the pan-european energy market)
c) In the EU, we historically talk about two energy markets (electricity and gas) → in the future we'll introduce a third distinct market for hydrogen
TL;DR (equivalents)
- ISOs/RTOs == TSO (Transmission System Operators = long-distance/inter-city transmission) + DSO (Distribution System Operators = short-distance/district-wide transmission)
-- ENTSO (European Network of Transmission System Operators) → they are responsible for setting and aligning the standard operating procedures via Network Codes (NCs).
There's one for electricity (ENTSO-E), one for natural gas (ENTSO-G), and possibly a future one for hydrogen (ENTSO-H)
- FERC == ACER (Agency for the Cooperation of Energy Regulators) → somewhat similar since it is an umbrella organization helping the different NRAs (National Regulatory Authorities) to come up with coherent strategies
A must read: “Shorting the Grid, the hidden fragility of our electric grid” by Meredith Angwin. It’s hard to think of the grid as deregulated with the thousands of pages of complex rules and layers of agencies and commissions. What a mess. Rather than a dream, I would call deregulation a nightmare. On top of that, subsidies for wind and solar distort the “market”, destabilize the grid and increase costs.
The biggest issue in the current market is traditional sources were stored fossil fuel energy and dispatchable in nature (i.e. they could be scheduled to run with differing lead times). Now we have added intermittent renewable sources that are not stored energy and thus not really fungible with existing sources yet they receive the same market price despite the lack of fungibility. This is hurting reliability and distorting market price signals. Either intermittent sources should receive a different price than stored energy sources or intermittent sources need to be paired with some low carbon storage solution to make them fungible with traditional stored sources of energy. Decarbonization is a worthy goal, we just have a bit of a free rider problem that needs to get worked out so that markets can do their part.
It is worse than you describe. AFAIK, a significant number of states have made it so that intermittents are enshrined as "first to sell" ahead of dispatchables.
1) "first to sell"/"preferential dispatch" for Renewables
The logic here would be to [a] tap into emission-free energy whenever it is possible (i.e. sun shines & wind blows) and [b] go through the marginal price cascade (see point #2 below) of energy sources until you meet demand. [c] Since the "dispatchables" are easily stored, you have an existing stock/storage IF/WHEN there's a shortage of intermittent renewables (RES-E).
2) free-riding/marginal pricing/level playing field
In the EU we have the "marginal pricing"/"merit order" system where for each hour (sometimes even 30 or 15min increments), there's a market clearing mechanism.
You take the demand/load and then start filling it with supply/capacity based on marginal cost. Since RES-E is cheaper than fossil fuels WHEN it is able to generate electricity, it wins the bid and is being dispatched.
From my perspective, the non-dispatchability of RES-E is "priced in" since they cannot participate in the market auctions when it's dark or the wind blows (i.e. the capacity factor of the PV or wind plant is generally low). RES-E facilities hence generate good profits when they can generate electricity but sit idle when they can't.
Since the energy market works as intended this presents an arbitrage opportunity. I'm sure in the coming years, we'll see more companies popping up in the energy storage space to utilize this arbitrage opportunity.
The problem here is that you are encouraging overbuild and mandating price inefficiency.
For example, nuclear power plants are forced to sell their electricity at negative prices during peak solar PV and/or wind production because they cannot ramp up or down rapidly.
Nor is this a theoretical problem: because of capacity factors, you need 3x solar and 2x wind to "equal" a 1x nuclear or fossil fuel unit of generation capacity - but this is over the span of 1 year. What this means in reality is too much electricity in mid-day low demand periods (solar) or midnight (wind) with marginal contributions during the actual peak use periods (duck curve before work am and after work pm to late evening).
So "first to sell" is not only failing to address the peak demand periods - which are the high system stress periods in general, it is introducing net system financial losses to dispatchables during the least demand periods by penalizing them for being dispatchable.
Nor is your statement "easily stored", a reality. Texas has a lot of absolute storage capacity but either 75% or 90% (I don't recall exactly which) is 1 hour or less - meaning it has zero utility even for short periods and is primarily for load shifting by 1 hour or less as opposed to true storage. Even then, the absolute storage capacity is a tiny, tiny fraction of actual steady state demand.
For example: California has 5.6 GW of storage - lots right? Except that California consumes 31 GWh on "average", meaning the 5.6 GW (note the missing time unit) is good for well under 1 hour of overall CA demand.
2) Free riding
The problem with the theoretical approach you outlined is that there is no premium for being dispatchable, which there actually should be. The difference is like Uber vs. Taxis - Uber is a superior service because you can be sure of getting a ride when and where you want it; it is the distortions of venture financing that made Uber cheaper than taxis for significant periods (at enormous losses) when Uber rides should always have been more expensive because drivers are always driving extra distance to pick you up.
The same holds for intermittent generation: there are real costs associated with running a dispatchable plant - not just the negative pricing situation noted above but the overall "spark spread" concerning natural gas price vs. price of electricity sold. This gap between theory of market pricing and reality of operational costs for dispatchables is almost certainly why electricity prices increase as renewable penetration increases - Germany and Denmark are the highest penetration renewables in Europe and also have the highest electricity prices.
Texas avoided this only because they are the epicenter of natural gas; there has been an enormous natural gas glut for the decade of fracking but the fracking overinvestment era is long dead and the glut is being eroded by progressively spiking LNG exports (to Europe, incidentally) as well as increasing domestic use and pipeline sales to Mexico. This manifested this past year as electricity prices spiked in Texas at historic levels despite the enormous renewable generation capacity there.
As for future storage: I have very low confidence in any type of mass storage except perhaps pumped hydro. Lithium or other metal type storage is prohibitively expensive, resource limited and dangerous from an energy density perspective; existing leading edge batteries are 2/3rds the energy density of gunpowder. Increasing energy density to overcome resource limitations means literally explosive risk; failure to increase energy density hits the limits of minable lithium etc which then turbocharges the cost problem.
Heat, compression, potential gravity and pretty much any other form of mechanical storage is extremely lossy and generally have even worse resource limitations.
As you can see, I do not share your view that a market based pricing mechanism will resolve these contradictions - actually I see the present market mechanisms as distorting true economic price discovery to the point of breakage particularly since renewables also enjoy massive ITC and PTC (that's installation and production) subsidies which in turn encourage overbuild precisely as intended.
If you don't believe what I'm saying - look at the daily duck curve for California, for example, since 2015. The impact of <6% solar PV generation has been to reduce daily average pricing between 10 am to 2 pm to negative numbers; the same dynamic exists for the Southwest Power Pool (SPP) due to West/North Texas wind - electricity prices are negative 30% of the year.
These negative prices are NOT good for anyone; negative prices exist because large amounts of excess electricity have a very real cost to address. Germany spent something like 900 million euros to handle excess electricity in 2021, the UK has averaged around 200 million GBP per year in handling excess wind electricity along since 2019
> The first crack in the monopoly status of utilities came in 1966, when the Minnesota town of Elbow Lake voted to establish its own municipal utility company and stop relying on Otter Tail Power Company, a small utility company Instead of getting its power from Otter Tail,
I think there’s a typo in the last paragraph. The northwest states are the ones that haven’t joined an ISO/RTO. ISO-NE is going strong!
Thanks, fixed.
What a complicated mess, we should go back to some sort of quasi public monopolies. Nobody knows how to complicate things like the US--tax code, medical insurance, utilities, no metric system, etc., etc.
This was really informative and is going to be useful to me personally, since I'll be starting an internship at an ISO in a couple days. Maybe you were planning on doing this already, but I am interested in a comparative analysis of the grid -- i.e. how are other countries running their grids? Anyway, thanks so much for this informative series!
It isn't absolutely necessary to look abroad; different US states have very different policies.
Texas, for example, is always lambasted for being largely separate from the US grid - but one of the consequences of that and Texas' "free market centric" setup is that Texas was very rapid in adopting natural gas grid electricity generation, and later on has become the single largest alternative energy producer (solar PV + wind).
California, in contrast, is the opposite of "free market centric" - no doubt at least in part due to Y2K Enron as is alluded to above.
But if you want to see a very different policy - look to China. They're clearly doing "all of the above": the most coal, the most solar PV, the most wind - and soon to be the most nuclear. And nobody is going to accuse the Chinese grid of being "free market centric".
Arun, all the best with your internship!
Agree with what c1ue said: US states have significant variance in market design + China is an interesting market.
I'll add a bit about the EU perspective:
a) you can think of EU-level as US-federal and MS-level (EU Member State) as US-state level
b) Since the late 90s/early 2000s there has been a strong policy push to unify the separate energy markets of the various Member States and move it closer to the ideal of the "European Copper Plate" (i.e. the pan-european energy market)
c) In the EU, we historically talk about two energy markets (electricity and gas) → in the future we'll introduce a third distinct market for hydrogen
TL;DR (equivalents)
- ISOs/RTOs == TSO (Transmission System Operators = long-distance/inter-city transmission) + DSO (Distribution System Operators = short-distance/district-wide transmission)
-- ENTSO (European Network of Transmission System Operators) → they are responsible for setting and aligning the standard operating procedures via Network Codes (NCs).
There's one for electricity (ENTSO-E), one for natural gas (ENTSO-G), and possibly a future one for hydrogen (ENTSO-H)
- FERC == ACER (Agency for the Cooperation of Energy Regulators) → somewhat similar since it is an umbrella organization helping the different NRAs (National Regulatory Authorities) to come up with coherent strategies
I've written a bit about energy regulation in the EU: https://delphizero.substack.com/p/a-curious-kids-guide-to-energy-regulation // maybe this helps.
A must read: “Shorting the Grid, the hidden fragility of our electric grid” by Meredith Angwin. It’s hard to think of the grid as deregulated with the thousands of pages of complex rules and layers of agencies and commissions. What a mess. Rather than a dream, I would call deregulation a nightmare. On top of that, subsidies for wind and solar distort the “market”, destabilize the grid and increase costs.
This is fascinating. Thanks for sharing!
I was hoping for something optimistic at the end, like, here's what should happen...but it never came.
The biggest issue in the current market is traditional sources were stored fossil fuel energy and dispatchable in nature (i.e. they could be scheduled to run with differing lead times). Now we have added intermittent renewable sources that are not stored energy and thus not really fungible with existing sources yet they receive the same market price despite the lack of fungibility. This is hurting reliability and distorting market price signals. Either intermittent sources should receive a different price than stored energy sources or intermittent sources need to be paired with some low carbon storage solution to make them fungible with traditional stored sources of energy. Decarbonization is a worthy goal, we just have a bit of a free rider problem that needs to get worked out so that markets can do their part.
It is worse than you describe. AFAIK, a significant number of states have made it so that intermittents are enshrined as "first to sell" ahead of dispatchables.
Adding the European regulatory perspective here:
1) "first to sell"/"preferential dispatch" for Renewables
The logic here would be to [a] tap into emission-free energy whenever it is possible (i.e. sun shines & wind blows) and [b] go through the marginal price cascade (see point #2 below) of energy sources until you meet demand. [c] Since the "dispatchables" are easily stored, you have an existing stock/storage IF/WHEN there's a shortage of intermittent renewables (RES-E).
2) free-riding/marginal pricing/level playing field
In the EU we have the "marginal pricing"/"merit order" system where for each hour (sometimes even 30 or 15min increments), there's a market clearing mechanism.
You take the demand/load and then start filling it with supply/capacity based on marginal cost. Since RES-E is cheaper than fossil fuels WHEN it is able to generate electricity, it wins the bid and is being dispatched.
From my perspective, the non-dispatchability of RES-E is "priced in" since they cannot participate in the market auctions when it's dark or the wind blows (i.e. the capacity factor of the PV or wind plant is generally low). RES-E facilities hence generate good profits when they can generate electricity but sit idle when they can't.
Since the energy market works as intended this presents an arbitrage opportunity. I'm sure in the coming years, we'll see more companies popping up in the energy storage space to utilize this arbitrage opportunity.
Let's look at what you wrote:
1) First to sell
The problem here is that you are encouraging overbuild and mandating price inefficiency.
For example, nuclear power plants are forced to sell their electricity at negative prices during peak solar PV and/or wind production because they cannot ramp up or down rapidly.
Nor is this a theoretical problem: because of capacity factors, you need 3x solar and 2x wind to "equal" a 1x nuclear or fossil fuel unit of generation capacity - but this is over the span of 1 year. What this means in reality is too much electricity in mid-day low demand periods (solar) or midnight (wind) with marginal contributions during the actual peak use periods (duck curve before work am and after work pm to late evening).
So "first to sell" is not only failing to address the peak demand periods - which are the high system stress periods in general, it is introducing net system financial losses to dispatchables during the least demand periods by penalizing them for being dispatchable.
Nor is your statement "easily stored", a reality. Texas has a lot of absolute storage capacity but either 75% or 90% (I don't recall exactly which) is 1 hour or less - meaning it has zero utility even for short periods and is primarily for load shifting by 1 hour or less as opposed to true storage. Even then, the absolute storage capacity is a tiny, tiny fraction of actual steady state demand.
For example: California has 5.6 GW of storage - lots right? Except that California consumes 31 GWh on "average", meaning the 5.6 GW (note the missing time unit) is good for well under 1 hour of overall CA demand.
2) Free riding
The problem with the theoretical approach you outlined is that there is no premium for being dispatchable, which there actually should be. The difference is like Uber vs. Taxis - Uber is a superior service because you can be sure of getting a ride when and where you want it; it is the distortions of venture financing that made Uber cheaper than taxis for significant periods (at enormous losses) when Uber rides should always have been more expensive because drivers are always driving extra distance to pick you up.
The same holds for intermittent generation: there are real costs associated with running a dispatchable plant - not just the negative pricing situation noted above but the overall "spark spread" concerning natural gas price vs. price of electricity sold. This gap between theory of market pricing and reality of operational costs for dispatchables is almost certainly why electricity prices increase as renewable penetration increases - Germany and Denmark are the highest penetration renewables in Europe and also have the highest electricity prices.
Texas avoided this only because they are the epicenter of natural gas; there has been an enormous natural gas glut for the decade of fracking but the fracking overinvestment era is long dead and the glut is being eroded by progressively spiking LNG exports (to Europe, incidentally) as well as increasing domestic use and pipeline sales to Mexico. This manifested this past year as electricity prices spiked in Texas at historic levels despite the enormous renewable generation capacity there.
As for future storage: I have very low confidence in any type of mass storage except perhaps pumped hydro. Lithium or other metal type storage is prohibitively expensive, resource limited and dangerous from an energy density perspective; existing leading edge batteries are 2/3rds the energy density of gunpowder. Increasing energy density to overcome resource limitations means literally explosive risk; failure to increase energy density hits the limits of minable lithium etc which then turbocharges the cost problem.
Heat, compression, potential gravity and pretty much any other form of mechanical storage is extremely lossy and generally have even worse resource limitations.
As you can see, I do not share your view that a market based pricing mechanism will resolve these contradictions - actually I see the present market mechanisms as distorting true economic price discovery to the point of breakage particularly since renewables also enjoy massive ITC and PTC (that's installation and production) subsidies which in turn encourage overbuild precisely as intended.
If you don't believe what I'm saying - look at the daily duck curve for California, for example, since 2015. The impact of <6% solar PV generation has been to reduce daily average pricing between 10 am to 2 pm to negative numbers; the same dynamic exists for the Southwest Power Pool (SPP) due to West/North Texas wind - electricity prices are negative 30% of the year.
These negative prices are NOT good for anyone; negative prices exist because large amounts of excess electricity have a very real cost to address. Germany spent something like 900 million euros to handle excess electricity in 2021, the UK has averaged around 200 million GBP per year in handling excess wind electricity along since 2019
Links:
https://www.ref.org.uk/constraints/index.php?tab=yr
https://www.statista.com/statistics/1332954/renewable-energy-power-curtailment-germany/
Texas averages around $0.016 per kWh in curtailment costs - use the same for the above German curtailment figures.
> The first crack in the monopoly status of utilities came in 1966, when the Minnesota town of Elbow Lake voted to establish its own municipal utility company and stop relying on Otter Tail Power Company, a small utility company Instead of getting its power from Otter Tail,
Typo here