“The Darwin Gödel Machine: AI that improves itself by rewriting its own code”
There is what is actually going on. A model trained on a large amount code is, in principle, no different from any other LLMs – it is just a statistical model that predicts the next token based on the previous ones. It does not understand the code it spews out, it does not “know” what it is doing. These are just mathematical procedures (not even functions) – given an input encoded in a particular way, it produces an output, not even the same for the same input.
When one trains a model on any code, including the code it is based on, there is, again, in principle, nothing “magical” or extraordinary happen – it does not become “aware” of anything or attained an “understanding” of any kind. It is still a probabilistic structure without any “awareness” whatsoever.
So, what appears to a naive, uneducated observer, to be a “self-improving”, is just a make-believe story, a fantasy. There is no notion of “self” in the first place. There is no “machinery” for any kind of awareness, leave alone self-awareness. Feeding “its own code” to the model achieves nothing, except that the network generates a different output, which is “classified” as “better” by a sort of wistful thinking. “better” in the context of software is a tricky concept, and it can be well-defined only in the simplest cases, like operation on linear or tree-like structures with special properties – here we can see that some code is, indeed, approaches the theoretical limit or an “optimum”.
By the way, all the limits have been already identified in the golden age of computer science, which was culminated with the “Haskell 98” report.
Anyway, one should read carefully and attain one’s own understanding (a realization) that nothing is going on there. This is, indeed, as if a bunch of monkeys are typing on typewriters, and the result is fed back to the new instances of monkeys. While this can be meme’d as a sort of “evolution” or “self-improvement”, it is just a trial-and-error process, without any “undemanding” or “awareness” whatsoever, just like biological Evolution itself is.
It is a pity that most people in the field do not even understand such simple things.