Recently there were lots of nonsense posted on Xitter and elsewhere about “complete AA-grade games " or “ready to production CAD designs” from a single simplistic prompts, like “make me an AA grade first person shooter with photo-realistic graphics”, almost literally.
The question, of course, is not how it dances so well, but how it dances at all.
The answer seems to be “the training process and the training data”, as usual.
We all know the principles – with each training example the back-propagation algorithm based on symbolic derivatives, slightly alters all the weights, in principle. The tricks like “batching” and whatever they do nowadays does not change this fact.
This, in turn, suggests to train not on a randomly shuffled data, but on coherent and consistent data, like a whole large codebase, because good code is much more structured, consistent and coherent.
I have also suggested to train first on a primary school textbooks, then high-school ones, then university textbooks and then articles and at last all the online bullshit, so the “trails through the snow (or paths through the mountains)” develop early and stay beaten up as early as possible.
This, of course, mimics the process of a human language acquisition (by a child from the exposure to the examples within her shared environment).
There is another subtlety. Since a written prose has much less structure than a program code, especially in crappy verbose imperative languages, the inference process easily and often “jumps” between “possible paths” and the result, given what is already in the buffer, is coherent (and free of grammar and spelling errors).
This works because the grammar and spelling rules are implicitly captured in the weights – incorrect spelling and grammar just end up with way lower weights.
But the semantics is a completely different story, and the “jumps” produce so-called hallucinations. It is that simple.
With the code, however, you cannot “jump” (arbitrarily select less probable path) – everything will blow up. But the training data are so coherent and structured, that all other paths are way less probable, given what is already in the inference buffer.
This means that datacenter -sized “frontier” LLMs just represent everything “verbatim”.
It is wonderful topic in itself, how an LLM representation (and the brain’s synaptic gaps) stay “consistent” when being continuously slightly perturbed (with each flow of blood and with each backprob pass). The answer is that these changes are real but tiny, almost negligible (until they aren’t).
Look, ma, I still remember my theoretical courses!
So, what Anthropic and others are doing is that they simply feed everything in, both whole code bases and poor user’s slop sessions (this was assumed from right the start) and the model is vast, it encodes whole projects “verbatim” – via standing out [emergent] paths through the fresh snow.
This is how and why these one-shot prompts work. Again, we have a sheer brute force, not an intelligence. Only a very convicing appearance (a cognitive illusion) of it.
Paraphrasing Dijkstra – how do you know that the “adder” works for all possible combination of numbers? There is only one way to know – study how it is actually implemented.