With advent of LLMs one may understand and validate which ideas of the classic theoretical Computer Science are, indeed, working, and what is mostly irrelevant.
The aim is to understand how this cognitive illusion works and why.
A text of modern Program (the source code) is highly structured and regular. This, of course, goes all the way back to Dijkstra and the discipline of Structured Programming, which among other things, postulates that one should use a small set of semantically well-defined, standard structural blocks or elements (procedures, conditionals, loops) and only these elements, instead of arbitrary mess of gotos and jumps. All this, in turn, is kind of obvious to those who have studied mathematics and logic, but the main point is that one can reason about the code in terms of these structural blocks, and this approach has fundamental cognitive advantages.
But it turns out that this very structural regularity of the modern code is exactly what allows LLMs to spew out seemingly coherent slop, that sometimes even compiles and runs. The building blocks usually have distinct shape – clear structural boundaries (with curly braces or other distinct begin and end marks). This allows the probabilistic slop generator to almost always come up with the correct whole blocks, without “knowing” any semantics whatsoever (again, all the information processing happens at the level of abstract, meaningless tokens).
Another big ideas that work are disciplined TDD and CI, which can be reduced to a small, incremental steps with a fast and tight feed-back loops on partial (incomplete) results, which, in turn, if you think for a second, underlay all animal’s [foraging] behaviour.
This is how the cognitive illusion (the Maya) works.
Now consider the less structured code – the arbitrary config files, which have an irregular structure. All LLMs fail miserably. But why?
The answer is that the hidden relations among the individual elements of a config file are not encoded in the structure of the text, so an underlying NN does not capture the relation at all. It is that simple.
Ask your favourite LLM to configure bluez with pulseaudio (both are fucking abomination and disgrace to a human intelligence) to use a high-quality “codec” for your bluetooth speakers. They will omit some necessary lines and hallucinate others. Why? Because how do they “know” that such and such option requires another one(s) to be enabled (or disabled) before that? These semantic dependencies are completely hidden and not represented at the level of syntax.
In the training set (some online forums, like Stack Overflow) people tend to give an incomplete answers, just some like that was missing or wrong, and thus LLMs missed out on the relations between other configuration options, which are arbitrary.
So, no “intelligence” whatsoever.
This can be even stated as a principle – all LLMs will fail on not sufficiently structured texts (when the semantic relations are not represented at the level of syntax).
I do not even want to think about the abstract humanities slop – it has to always be incomplete and delusional, just as it is in the real life, but now you know exactly why and how.
Here I am [metaphorically] peeing on @slutsker and @karpathy, and those who paid them hundreds of thousands.