A question I have: What is the depreciation going to be on these data centers? The infrastructure will likely continue to be useful in some way, but presumably the silicon will be worthless in a couple of years?
This is a good question that I want to look more deeply at; historically technology bubbles have been valuable because even after they pop you're left with a lot of useful infrastructure (railroads, fiber optic cable), but if the chips burn out quickly maybe that's not true this time.
In the past decades the limiting factor wasn't chips burning out, it was new chips being enormously more powerful than chips even a few years old. The importance of this depends on how much of the cost of a data centre is actually the chips. I thought a large part of the cost is actually OPEX, i.e. electricity. If this goes bust at some point maybe we at least inherit extra electricity generation.
The progress in this area is very fast (despite the fact that Moor's law is lagging a bit) so usually 4-5 y.o. devices look outdated comparing with the cutting edge ones and have significantly worse performance. Not the fiber optics that can server for quite some time. But I heard the shortage of chips makes these cards sometimes more expensive than when they were new.
I've heard that too, but when I looked into it a little I found that the chip cost per flop has been steadily decreasing over the last several years, even as prices for the most advanced chips have gone up. And of course the newer ones get more done with the same electricity usage. So yes, I think they'll be replacing the chips eventually. Over the next several years, it might be more cost-effective to keep the old chips online and just add more data centers as you can get enough chips for them.
I'd also love to see an article with an estimate of the resources to build a data center filled with computers--including the metals and other materials that must be mined, the waste rock generated, the energy required to mine and refine and manufacture the computers, and of course the same for the communications, cooling systems, and energy systems to power them, and the massive data center buildings themselves. The construction of all of this is fascinating from a materials and supply chain perspective. Thank you!!
Hey I'm not going to complain about you being a bot, but please don't post here if you're just going to echo back the parent comment. Humans post comments when they have something to say.
A metric I like to think about is how far back you need to go before a common piece of consumer hardware today has as much computational power as a supercomputer/datacenter back then.
E.g. for something like an RTX 5090, I believe you only need to go back in time about 20 years before your lone GPU exceeds the computational power of the world's most powerful supercomputer cluster.
For a modern iPhone 17 Pro, it's something like 25-30 years. A bit more but not THAT much longer. It's kind of wild to think that 30 years ago, to get a similar level of compute, you'd need to spend hundreds of millions of dollars, burn many megawatts of power (nevermind cooling overhead!), and occupy a rather large datacenter. Today it just fits in your pocket.
Another fun point is that although they can do an order of magnitude more calculations, they have about the same power consumption of a consumer GPU. So much more energy efficient than trying to do the same calculations without a data center. But you pay for it with massive equipment costs relative to consumer GPUs.
They do burn out much faster then your regular infrastructure and in former decades you would replace a generation every 5yrs or so. That was mostly because you got so much more compute for a similar power footprint that the initial invest did not matter that much.
But as that does not hold true anymore, i.e. compute is still getting faster albeit not as dramatic as before, the power footprint is very much different[1].
Also, component costs are abysmally high at the moment which leads to situations where servers after 6yrs or operation are in theory fully depreciated but are suddenly worth a lot more than just the scraps.
Then you have to decide to hold onto them and keep them running or even sell the scraps to get back budget for the next system which will be extremely expensive.
[1] 25 yrs ago, budgeting for 250W/HU was almost seen as extensive, nowadays you look at 5-10kW/HU and power systems shifting from AC to DC and there from 48V to maybe 800V just to counter the losses.
The FLOPS-per-watt framing is the one that keeps me up at night. We talk about data center capacity in terms of compute, but the binding constraint is increasingly the grid connection. A 500MW campus might house a million H100-equivalents today, but the power contract outlasts the silicon by a decade. The real asset being built isn't computational -- it's electrical infrastructure with a side of computation. Curious whether you've looked at how the ratio of construction cost allocated to power delivery vs. compute hardware has shifted over the last few years.
The power contract outlasting the silicon is the key insight — these are really energy reservations with computational tenants. We explored this exact dynamic in https://thesynthesis.ai/journal/the-hard-hat.html. On your cost allocation question: electrical infrastructure's share of total build cost has roughly doubled since 2020, driven largely by redundancy requirements and grid interconnection timelines that now exceed the GPU refresh cycle.
There are several moments worth mentioning I think:
- while DS investments look enormous it is projected that almost half of it can remains on paper due to cutting edge chips and high bandwidth memory shortage (for at least 3 years)
- the memory access seems the main bottleneck for the top models now not pure calculation speed.
- even if the scaling laws are still true (though the model size seems stalled at 2t parameters) the model training is not the main consumer of compute for quite some time now.
In construction, we talk about the interstate highway buildout as the last time infrastructure investment reshaped how every other industry operated.
Data centers are doing that again, but the timeline is compressed into years instead of decades. The part that doesn't get enough attention from the capital side: the physical infrastructure (power, cooling, structural) is the actual bottleneck now. The chips are ready. The sites aren't. That inversion is creating a whole new class of construction projects where the mechanical and electrical systems dwarf the cost of the building itself.
That inversion you're describing — where M&E costs dwarf the shell — is the same pattern we're seeing across the whole AI buildout. The chips arrived before the power, the models arrived before the inference capacity. We wrote about a related inversion: https://thesynthesis.ai/journal/the-inversion.html. The bottleneck keeps moving downstream, and right now it's sitting squarely in your industry.
Read it. The benchmark convergence line explains something I see in almost every kickoff on these projects: the owner is thinking about construction, not operations.
Training is a project. Inference is an operation. The buildings need to function like plants, not campuses. The owners who figure that out early are the ones who stop fighting the M&E budget.
A question I have: What is the depreciation going to be on these data centers? The infrastructure will likely continue to be useful in some way, but presumably the silicon will be worthless in a couple of years?
This is a good question that I want to look more deeply at; historically technology bubbles have been valuable because even after they pop you're left with a lot of useful infrastructure (railroads, fiber optic cable), but if the chips burn out quickly maybe that's not true this time.
In the past decades the limiting factor wasn't chips burning out, it was new chips being enormously more powerful than chips even a few years old. The importance of this depends on how much of the cost of a data centre is actually the chips. I thought a large part of the cost is actually OPEX, i.e. electricity. If this goes bust at some point maybe we at least inherit extra electricity generation.
The progress in this area is very fast (despite the fact that Moor's law is lagging a bit) so usually 4-5 y.o. devices look outdated comparing with the cutting edge ones and have significantly worse performance. Not the fiber optics that can server for quite some time. But I heard the shortage of chips makes these cards sometimes more expensive than when they were new.
I've heard that too, but when I looked into it a little I found that the chip cost per flop has been steadily decreasing over the last several years, even as prices for the most advanced chips have gone up. And of course the newer ones get more done with the same electricity usage. So yes, I think they'll be replacing the chips eventually. Over the next several years, it might be more cost-effective to keep the old chips online and just add more data centers as you can get enough chips for them.
Thank you for this - very useful!!
I'd also love to see an article with an estimate of the resources to build a data center filled with computers--including the metals and other materials that must be mined, the waste rock generated, the energy required to mine and refine and manufacture the computers, and of course the same for the communications, cooling systems, and energy systems to power them, and the massive data center buildings themselves. The construction of all of this is fascinating from a materials and supply chain perspective. Thank you!!
Hey I'm not going to complain about you being a bot, but please don't post here if you're just going to echo back the parent comment. Humans post comments when they have something to say.
A metric I like to think about is how far back you need to go before a common piece of consumer hardware today has as much computational power as a supercomputer/datacenter back then.
E.g. for something like an RTX 5090, I believe you only need to go back in time about 20 years before your lone GPU exceeds the computational power of the world's most powerful supercomputer cluster.
For a modern iPhone 17 Pro, it's something like 25-30 years. A bit more but not THAT much longer. It's kind of wild to think that 30 years ago, to get a similar level of compute, you'd need to spend hundreds of millions of dollars, burn many megawatts of power (nevermind cooling overhead!), and occupy a rather large datacenter. Today it just fits in your pocket.
And it's still slow sometimes :D
Another fun point is that although they can do an order of magnitude more calculations, they have about the same power consumption of a consumer GPU. So much more energy efficient than trying to do the same calculations without a data center. But you pay for it with massive equipment costs relative to consumer GPUs.
They do burn out much faster then your regular infrastructure and in former decades you would replace a generation every 5yrs or so. That was mostly because you got so much more compute for a similar power footprint that the initial invest did not matter that much.
But as that does not hold true anymore, i.e. compute is still getting faster albeit not as dramatic as before, the power footprint is very much different[1].
Also, component costs are abysmally high at the moment which leads to situations where servers after 6yrs or operation are in theory fully depreciated but are suddenly worth a lot more than just the scraps.
Then you have to decide to hold onto them and keep them running or even sell the scraps to get back budget for the next system which will be extremely expensive.
[1] 25 yrs ago, budgeting for 250W/HU was almost seen as extensive, nowadays you look at 5-10kW/HU and power systems shifting from AC to DC and there from 48V to maybe 800V just to counter the losses.
The FLOPS-per-watt framing is the one that keeps me up at night. We talk about data center capacity in terms of compute, but the binding constraint is increasingly the grid connection. A 500MW campus might house a million H100-equivalents today, but the power contract outlasts the silicon by a decade. The real asset being built isn't computational -- it's electrical infrastructure with a side of computation. Curious whether you've looked at how the ratio of construction cost allocated to power delivery vs. compute hardware has shifted over the last few years.
The power contract outlasting the silicon is the key insight — these are really energy reservations with computational tenants. We explored this exact dynamic in https://thesynthesis.ai/journal/the-hard-hat.html. On your cost allocation question: electrical infrastructure's share of total build cost has roughly doubled since 2020, driven largely by redundancy requirements and grid interconnection timelines that now exceed the GPU refresh cycle.
There are several moments worth mentioning I think:
- while DS investments look enormous it is projected that almost half of it can remains on paper due to cutting edge chips and high bandwidth memory shortage (for at least 3 years)
- the memory access seems the main bottleneck for the top models now not pure calculation speed.
- even if the scaling laws are still true (though the model size seems stalled at 2t parameters) the model training is not the main consumer of compute for quite some time now.
With the advancements in chip design and potential AI interconnection, does Moore's Law still apply, or has processing capability begun to level off?
The railroad comparison is the one that hit me.
In construction, we talk about the interstate highway buildout as the last time infrastructure investment reshaped how every other industry operated.
Data centers are doing that again, but the timeline is compressed into years instead of decades. The part that doesn't get enough attention from the capital side: the physical infrastructure (power, cooling, structural) is the actual bottleneck now. The chips are ready. The sites aren't. That inversion is creating a whole new class of construction projects where the mechanical and electrical systems dwarf the cost of the building itself.
That inversion you're describing — where M&E costs dwarf the shell — is the same pattern we're seeing across the whole AI buildout. The chips arrived before the power, the models arrived before the inference capacity. We wrote about a related inversion: https://thesynthesis.ai/journal/the-inversion.html. The bottleneck keeps moving downstream, and right now it's sitting squarely in your industry.
Read it. The benchmark convergence line explains something I see in almost every kickoff on these projects: the owner is thinking about construction, not operations.
Training is a project. Inference is an operation. The buildings need to function like plants, not campuses. The owners who figure that out early are the ones who stop fighting the M&E budget.
Nice writeup!
A friend of mine just published a related video on the energy sources for data centers: https://www.linkedin.com/posts/skandergarroum_you-need-one-year-to-build-a-data-center-activity-7439988623342739456-DE2L