How Predictable Are Laws?
An enormous amount of ink has been spilled (some of it by me) on the trials and tribulations of complying with the National Environmental Policy Act, better known as NEPA. NEPA is what requires projects to perform years-long, thousand-plus page environmental impact studies before construction can begin, and suing a project for an insufficiently detailed environmental study is one of the chief ways environmental groups are able to slow down or stop projects they don’t like. And NEPA’s influence goes beyond federally funded projects: NEPA also influenced the creation of many similar laws, both at the state level (such as California’s CEQA) and in countries around the world.
None of these effects of NEPA, however, were envisioned when the law was written. NEPA was seen primarily as an (aspirational) statement of US environmental policy, which was to “encourage productive and enjoyable harmony between man and his environment, to promote efforts which will prevent or eliminate damage to the environment and biosphere and stimulate the health and welfare of man; [and] to enrich the understanding of the ecological systems and natural resources important to the Nation.” The provision that requires environmental impact statements was added last minute as a way to try to give some teeth to these high-minded but somewhat abstract ideals, and received virtually no attention at the time. As Alec Stapp and I noted:
[The provision] was not covered in any major media publication. In Congress, it received “neither debate, nor opposition, nor affirmative endorsement.” Caldwell would later state that “most [members] had never really understood the bill and only agreed to it because it was from Jackson; it was about the environment which was a very ‘hot’ issue at the time; and it was almost Christmas and they wanted to get home.”
Not until several months after NEPA was passed did environmental groups realize what a potent weapon they’d been handed.
It’s not hard to find other examples of laws whose effect was far different than what the authors anticipated. The 401(k) retirement account, now used by tens of millions of Americans as the primary vehicle for retirement savings, was originally considered an insignificant provision of the 1978 Revenue Act. Per a Bloomberg piece about the Act:
…The initial provision was estimated to have a “negligible effect upon budget receipts.” Now, defined contribution plans are the fifth-biggest tax break for individuals, with an estimated revenue loss to the government of $61.4 billion in fiscal 2014.
“There was absolutely no discussion in ’78 that if you do this, the world is going to change,” said Daniel Halperin, then a senior Treasury official and now a Harvard Law School professor.
The tale of Richard Stanger [a primary author of the Act], who said he hadn’t been interviewed previously about his role, is also a story about accidental actors at historic moments. As Stanger himself says, if anyone had known how important 401(k) would become, the Joint Committee on Taxation never would have let him, a 28-year-old junior lawyer, write it.
In the other direction, laws aimed at stimulating the construction of housing in California have proven much less effective than predicted:
One California law was supposed to flip defunct strip malls across California into apartment-lined corridors. Another was designed to turn under-used church parking lots into fonts of new affordable housing. A third would, according to supporters and opponents alike, “end single-family zoning as we know it.”
Fast-forward to 2025 and this spate of recent California laws, and others like it intended to supercharge the construction of desperately needed housing, have had “limited to no impact on the state’s housing supply.”
That damning conclusion comes from a surprising source: A new report by YIMBY Law, a pro-development nonprofit that would very much like to see these laws work.
I wanted to better understand how common this was: how often do laws do more or less what they’re designed to do? How often do their effects diverge widely, either by having unanticipated effects or by failing to do what the authors predicted? So I used AI to analyze the effects of several hundred federal laws passed over the last 50 years.
Overall, I found that federal laws mostly do what they’re expected to do. But a substantial fraction of them — around 11% — diverge significantly, having either much smaller or much larger effects than originally predicted.
Method
To do this analysis, I first chose five random federal laws passed each year from 1976 to 2023, filtering out any laws that were less than 10 pages in length, which were mostly insignificant things like post office renamings. This yielded 240 laws total, but for one law the AI was unable to find any information, so the actual analysis was done on 239 laws. For each law, I had an AI model — Claude Opus 4.8 Max Thinking — estimate the expected effect of the law, its actual effect, and assign a score to the divergence. Divergence scores ranged from -10 to 10: positive numbers indicate the law had a larger effect than anticipated (such as the 401(k) provision in the 1978 Revenue Act or NEPA), while negative numbers indicate the law had a smaller effect than anticipated.
Scoring Rubric:
0 — actual impact matched expectations essentially exactly.
±1–2 — minor: broadly as expected; small deviations on secondary dimensions that didn’t change the essential outcome.
±3–4 — moderate: clearly noticeable gaps on one or more dimensions, but the core purpose was still substantially realized (or missed only in a limited way).
±5–6 — substantial: the primary expected outcome was materially exceeded (+) or unmet (−), or a significant unanticipated effect emerged.
±7–8 — major: the central goal greatly overshot (+) or largely failed / went unimplemented (−), or effects were largely of a different kind than intended (+).
±9–10 — extreme: actual impact bore little resemblance to expectations — dominated by unintended, larger-than-anticipated, or opposite effects (+), or near-total failure / non-implementation (−).
There were a few complications in deciding how the effect of laws should be evaluated. One is deciding when a law should get credit for having an effect. Often the largest effect of a law only happens when subsequent laws extend, modify, or build on the original law. For instance, an unanticipated effect of the 1978 Psychotropic Substances Act was the introduction of civil asset forfeiture for drug proceeds. This mechanism became a cornerstone of US drug enforcement, but much of this later expansion was due to the 1984 Comprehensive Crime Control Act. For these cases, I instructed the AI to give some credit to the original law if it was built on by other laws, but to temper it based on how much subsequent laws actually did the heavy lifting. (For the Psychotropic Substances Act, the AI assigned it a score of +4, a moderate unanticipated effect, since the later bill did most of the work.)
Another complication was trying to determine what the expected effect of a law was at the time it was passed. For this, I instructed the AI to only use contemporaneous sources, such as the bill text itself, the presidential signing statement, a CBO score, press coverage at the time, etc. But this is inherently a fraught exercise: it’s often not obvious, for instance, to what extent the goal of some law is aspirational that the authors don’t expect to necessarily happen. For instance, the Indoor Radon Abatement Act of 1988 states in the bill text a national long-term goal that “indoor air in a building be as free of radon as the ambient outdoor air.” This is almost certainly an aspirational goal that the authors did not expect the bill to actually achieve, but making these judgments requires a process of guessing and using context clues that is likely to be error-prone. (The AI scored this law as a -2, a minor shortfall compared to what was expected; the various anti-radon programs the law created stuck around, but indoor radon exposure did not improve, in part because the programs were almost all voluntary.)
This exercise is similar to a previous post where I used AI to try and estimate how early various inventions could have been invented, but this turned out to be far more difficult and annoying, mostly because of the research required. For the previous exercise on inventions, I simply relied on the AI’s knowledge of various inventions to make the judgments. But for this present effort about laws I needed the AI to thoroughly research each law: I couldn’t expect the AI to know, word-for-word, every esoteric law passed in the last 50 years, or the complete history of the downstream effects of that legislation. I ended up needing to do a fair amount of trial and error to get Claude to do a sufficiently thorough job evaluating the expected and actual effects. I kept having to modify the prompt to force increasing levels of thoroughness in the search, and even in the final version I settled on I was occasionally finding effects that the AI evaluation had missed. Because of this, I expect there to be errors in various evaluations, and I would regard these results as preliminary.
You can see the full prompt I used, and the resulting evaluations, here.
Results
The graph below shows the divergence scores of the 239 evaluated laws.
The results form something like a bell curve with a slight positive skew. Ninety-five of the 239 laws (40%) had a score from -1 to 1: either they behaved exactly as expected or had very slightly larger or smaller effects than predicted. Forty-nine had a score from -2 to -4 (20%), and 68 (28%) had a score from 2 to 4, a moderate divergence. Only around 11% of laws had a “substantial divergence,” a score of plus or minus 5 or more. Most federal laws, it seems, do more or less what they’re expected to do.
(The graph above shows scores bundled together, but if you look at frequencies of individual score values, you don’t get a smooth bell shape. Instead you get a dip, with many fewer scores at -1 and +1. This is likely an artifact of the scoring rubric, which probably pushed scores either into 0 or out to plus or minus 2, but it’s another reason why we should treat these results with a grain of salt.)
Some examples of laws that Claude scored as having a much higher effect than predicted:
“An act to amend title XIII of the Federal Aviation Act of 1958 to expand the types of risks which the Secretary of Transportation may insure or reinsure, and for other purposes”, score +5: This mundane-sounding law is described by its title as mainly about airline insurance, but one of its provisions deregulated airline cargo service, the first step in deregulation of the airline industry more broadly. This deregulation was expected to increase competition and efficiency in the air cargo market, but its effect went beyond that. With air cargo flights deregulated, companies like FedEx, which were previously confined to using very small aircraft, could now use large jets on any route they wanted, setting the stage for the entirely new “air express cargo” industry.
The Sarbanes-Oxley Act of 2002, score +5: This law, passed in the wake of business collapses like Enron and WorldCom, was aimed at restoring investor confidence by increasing financial auditing oversight, and creating stiffer penalties for compliance failures. The additional oversight was achieved, but at a cost: compliance costs were 30 to 50 times higher than expected. Another positive divergence came from the fact that one of its clauses, which penalizes “obstruction of an official proceeding,” was later used in an unexpected way: criminally charging hundreds of January 6 defendants (though this was later struck down by the Supreme Court).
The Trade Facilitation and Trade Enforcement Act (TFTEA) of 2015, score +7: This law was billed as a customs modernization and trade enforcement act: giving more resources to enforce trade agreements, streamlining various trade and customs regulations, etc. At the signing ceremony, President Obama described it as “making sure other countries are playing by the rules.” But one of the provisions of the law changed the “de minimis exemption” — the value below which imported goods were not subject to tariffs — from $200 to $800. This change is credited as a driver of the explosion of Chinese imports from companies like Shein/Temu over the next several years, until it was reversed by the second Trump administration.
And here are some examples of laws that had a much smaller effect than predicted:
The Alaska Natural Gas Transportation Act of 1976, score -7: This act was expected to create a huge 4,800-mile pipeline, the “largest privately financed energy project ever undertaken,” that would transport natural gas from Alaska to the lower 48 states. The pipeline, however, was never completed, due to a combination of rising costs and the later Natural Gas Policy Act and Fuel Use Act creating a gas supply glut, obviating the economic justification for the pipeline.
The Alabama-Coosa-Tallapoosa River Basin Compact of 1997, score -5: This law was intended to create a commission that would develop a plan to share the water of the Alabama-Coosa-Tallapoosa River Basin between Alabama and Georgia. But while the commission was formed, Alabama and Georgia never agreed on an allocation formula, resulting in continuous litigation between the two states over water distribution.
The Enhanced Partnership with Pakistan Act of 2009, score -5: This act was expected to foster a closer relationship and increase goodwill with Pakistan, by providing billions of dollars’ worth of funding for schools, roads, and other infrastructure projects. This didn’t occur: opinion of the US in Pakistan continued to fall following things like President Obama visiting India but skipping Pakistan in 2010 and the US’s raid on Bin Laden’s Pakistan compound in 2011. By 2012, 74% of Pakistanis viewed the US as an enemy.
Other than the fraction of laws with significant divergence, there are a few other notable patterns in the data. If we look at divergence over time, we don’t see much change: recent laws seem roughly as predictable as older laws.
What about differences between small/minor laws and large/major laws? If we graph a law’s divergence score against the number of pages in the law, we see a small positive correlation: large laws with many pages are somewhat more likely to have a larger-than-expected effect than laws with fewer pages.
One possible explanation here is a sort of bundling effect: major laws, like the recent 21st Century ROAD to Housing Act, are often amalgamations of many smaller laws. Because the distribution of divergences is somewhat positively skewed, when you bundle many laws together, the chance that at least one of them has a large positive effect might rise.
Another notable pattern is that different types of laws have somewhat different probabilities of a major divergence. Appropriations bills, for instance, which are often (but not always!) routine allocations of money, have a lower variance than bills that create substantially new programs. Both types of bills have the same average divergence score, but the probability of a large or small divergence is much greater with the latter than the former.
Why do laws diverge from what’s expected?
At a high level, the reasons that laws have greater or smaller effects than expected can be divided into two categories: operators of the legal machinery behaving differently than expected, and the broader world, including those who the law was designed to affect, behaving differently than expected.
On the “legal operator side,” this includes everyone who has a role in authoring, enforcing, or interpreting laws. Courts, for instance, will often interpret laws in ways that the original authors didn’t anticipate. These interpretations can greatly extend a law’s scope and influence, such as the courts’ broad interpretation of what’s required to comply with NEPA’s “detailed statement” provision. Or they can reduce a law’s scope and influence, such as by determining that provisions of a law are unconstitutional. This happened with the 1989 Ethics Reform Act, which barred all federal government employees from being compensated for giving speeches, attending events, or writing articles. In 1995 the Supreme Court ruled that this requirement violated the First Amendment, and it only survives in application to senior government officials and Members of Congress.
Likewise, prosecutors or other government agencies might behave in ways other than what was expected. They might use a law for unexpected purposes: the DNA Fingerprinting Act of 2005, which authorizes collecting DNA from federal detainees, was part of a Department of Justice Authorization bill that was primarily focused on addressing violence against women. But the act was later used by ICE to collect DNA from immigration detainees. Alternatively, they might decline to use new legal machinery introduced. An example of the latter is the 2012 STOCK Act, which on paper made it illegal for members of Congress and their staff to trade stocks based on their congressional knowledge. As of 2025, there have been zero prosecutions under this law despite suggestive evidence that congressional insider trading does occur.
This category also includes Congress itself. Future Congresses may increase the effect of some law, such as by making a temporary program permanent or otherwise expanding its scope. For instance, the 1979 Recreational Boating Safety and Facilities Improvement Act contained a provision that created a trust fund, capped at $30 million fund, to clear a Forest Service tree replanting backlog. In 2021, the REPLANT Act took this fund and massively enlarged it, using it as a vehicle for a program to plant 1.2 billion trees in national forests. On the other hand, future Congresses might reduce the effect of some law: the 1976 Parole Reorganization Act, for instance, was intended to streamline and strengthen the federal parole system, but the 1984 Sentencing Reform Act abolished federal parole, making the previous law almost entirely moot.
You see the same sorts of divergences in the world at large. The 2006 Credit Rating Agency Reform Act tried to foster increased competition in the credit ratings agency market, but even though several new ratings agencies appeared, the market remained dominated by S&P, Moody’s, and Fitch, which collectively control 95% of the market. The Air Cargo Deregulation Act failed to predict how carriers like FedEx would respond to the freedom to fly on any route with any aircraft. TFTEA failed to predict how low-price Chinese fashion companies like Shein could take advantage of the “de minimis” change to ship directly to US consumers.
And of course, unforeseen behavior of legal operators and the broader world may interact. The Alaska Natural Gas Transportation Act failed to result in a new natural gas pipeline in part due to the market’s response to new natural gas regulations passed by Congress.
Conclusion
I think of laws as sort of akin to technology. With the invention of a new technology, you create some new capability, often for the purposes of achieving some particular goal. But once that capability is out there in the world, people will find all sorts of ways to take advantage of it. Marconi envisioned radio as literally “wireless telegraphy,” a way to send and receive messages from ships at sea, but he didn’t envision the rise of broadcast radio. Vacuum tubes were first used to amplify long-distance telephone signals, and only later became components for televisions and the earliest digital computers. Teflon was first used to make pump seals in uranium separation plants for the Manhattan Project, and only later found use in non-stick cookware.
Laws often work the same way. A law will create or modify some capability — an organization, a program, a rule that must be followed — aimed at accomplishing some particular thing. But once that capability is out there in the world, people might take advantage of it in different ways, finding uses for it that the creators of that capability never expected. A modest environmental reporting requirement becomes the foundation of modern environmental litigation; a minor change in employer retirement contributions becomes a retirement account used by tens of millions of Americans; a financial reporting law gets used to charge rioters.
Conversely, just because you introduce some new capability doesn’t mean it’ll actually be useful, or that anyone wants it. Some technologies, like 3D TV, or smell-o-vision, don’t pan out, and the patent archives are full of ideas for inventions that no one had any use for. Similarly, just because you create a new legal capability doesn’t mean it will end up useful in the way you envisioned. Changing the rules for designating a “nationally recognized” credit rating agency, as the 2006 Credit Rating Agency Reform Act did, did nothing to disrupt the market share of the existing agency oligopoly.
It is, of course, notoriously hard to predict the long-term effects of new technologies. With laws, it seems like predictions are substantially easier. But divergences still exist.