Reflections on a question we’re not asking about AI-generated code
It’s a massive simplification, but code generally has two primary jobs: 1) perform some function, and 2) be maintainable. The code has to do something tangible in a correct way without undesirable side effects, and it has to be possible to make changes to it later when bugs are found, when new features are desired, or when circumstances change. When we talk about code quality, about what makes “good” code, it falls into one of those two categories.
Concerns like efficiency, security, usability all fall into the first category: does the code do what it’s supposed to do, and do it well? As a sweeping generalization, most “code quality” tools focus on these areas, because they’re easy to measure and have tangible effects. Does the code pass the functional tests? Does it have hidden logic errors? Does it run efficiently? Is it running out of resources? Does it have security issues?
This article on code quality from AWS is a pretty good high level summary of these areas. They recommend a mix of quantitative measurement and qualitative expert review for measuring and improving overall code quality.
The question we’re not really asking is this: what properties of code make it easier or more reliable for coding agents to “understand”? (Aside: yes, I know there is no “understanding” per se, but I’m using this as a shorthand for functionally parsing what the code does and being able to make workable changes. LLMs can do this.). It seems evident to me that those properties are not necessarily the same as what makes code easier for humans to understand, and if that’s the case, we’re likely to be optimizing human review for code patterns that maybe aren’t necessary. When you review code that an LLM has written, is it “good” code if it looks like what a human would write? Is the standard for “good” code that it’s comprehensible to a human? There is certainly an argument to be made that the answer to that might be no, if some future LLM (the same model or an improved one) can later look at the code, understand what it does, and still make effective changes to it. In much the same way that you no longer need to care about efficient register use in your CPU because the compiler does that for you now, it might be the case that human code readability is no longer that important.
I’ve been writing software for a few decades with a focus on reliability and efficiency trade-offs. I’ve been writing code with an LLM for a small percentage of that, and I’ve noticed some things.
The temptation to “just let the machine do what it wants if it functionally works” is very strong. I’ve written entire iOS apps where I’ve never looked at the code at all other than a very cursory glance to make sure they’re not pulling in capabilities they shouldn’t have (and why would they, the machine more or less did what I told it to do). They work. In those particular cases, they don’t really touch any external resources so the blast radius for any security issues is very small, and they’re protected from doing dangerous things by the underlying APIs. I’ve gone through rounds of architectural redesigns, massively changing what the apps do and how they work to deal with real-world constraints of running them on devices, and the LLM was able to handle that, with guidance. The LLM, within some bounds of “reason”, can indeed “understand”, refactor, and make modifications to the code.
Sometimes it went horribly wrong, but I’m experienced enough with the overall process to be able to recognize when that was happening and redirect it. Sometimes it did get into doom loops where it tried to fix an error with something it had already previously tried without “realizing” it, and just flapped back and forth between two broken implementations. A big part of the human development experience is running the code to see if it works, and LLMs have limited capacity or drive to do that out of the box without building in tooling and infrastructure to make it happen. An even bigger part of the human development experience is being terrified to make the jump to running in production to see what breaks in real-world conditions, and LLMs have no capacity to do that. In at least one of the doom loop cases, trying a more advanced model solved the problem easily. Is that a win? I don’t know, but it kind of puts to rest at least some criticisms that these tools aren’t substantially improving as time goes on, and also criticisms that they aren’t useful. But on the other hand, it sometimes said problems were fixed when they demonstrably were not. It’s a mixed bag of horrifying failure and horrifying success. But at the end of the day, I got working apps that solved a problem for me, so I call that ultimately a success.
One thing that scares me is that the LLM code isn’t even close to deterministic. I’ve done something, then lost that change for whatever reason (conversation compact, something crashed), and when asked to do exactly the same thing a second time, it came up with a very different result. Sometimes it failed to fix a bug and then when asked again, it did. There are ways to partially mitigate this with guidelines and specs, but it’s still in the nature of how these systems work. I’m on the fence about whether this is a problem or not. Does the code need to consistent if ultimately the LLM can make sense of it later? More questions.
Notably, this is a relatively small and self-contained example. There are plenty of cases where LLMs fail dramatically because they can’t fit the entire codebase into their context and code with too many interdependencies is too hard to “understand” and therefore safely modify. Of course this is an issue with human comprehension as well, but I don’t think we have a good understanding of where the limits are for this or what makes code more “readable” to machines. The solutions to this are the standard ones — simplify, modularize, document, and refactor. It seems like we’re still making an industry-wide assumption that code that’s good for humans is code that’s good for LLMs, and I’m not entirely convinced that that’s true. Will the next round of models continue to blow these assumptions out of the water? If they do, are we wasting our time when we should just be biding our time? There are a multitude of other questions for LLM-based development about efficiency, resource usage, cost, the erosion of skills, the seed corn problem, etc… Ignoring all of those for the time being and assuming this is a technology we’re still going to use to write code, I think we need to be asking what makes code good, and questioning whether the assumptions we had about that are still true.
If you enjoyed reading this, feel free to come yell at me about it on threads, bluesky, or mastodon.
