So, it feels like it’s time to write again, to see where we actually are at.
Recently there were two big events related to the LLMs. First one is the “discovery” of a counter-example for the famous conjuncture (and the large blog post by Terence Tao, that he could have discovered it too, LOL).
Another one was the “hack” of bitcoin “hardware” wallets, not the hack itself (it is rather “simple”) but the fact that, allegedly, an LLM with a given prompt like “find a security weakness in this code” (not even a “vulnerability”) have managed to find the exact RNG bug (reduction of the entropy size form the “safe” 128 bits from a hardware RNG to a “weak” 40 bits of flawed software RNG, so the attackers are just brute-forcing the 40bit “state space” and find the keys).
There are the usual things to clarify. No LLM is “thinking” or “reasoning” about anything. They are incapable of doing so in principle – there are no “reasoning engines” in them, just matrix multiplications, together with some clever “tricks” of passing the numbers between layers and masking out some of them, which they call “Attention”.
Just as the very same multi-modal model can create a consistent image for (You) based on pixel patterns it has been trained with, absolutely oblivious to any principle of human anatomy, physiology, symmetries of the bodies and faces, or the facts that hairs shall hang down when the head is upside-down, and so on, yet, it generates everything “correctly” just out of a huge amount of training data, which “out-weight” or “down-weights” the incorrect patterns (and a sheer brute-force, which appears as intelligence).
So, no, the model haven’t analyzed any mathematical abstractions in a systematic way to find a counter-example, it just operated at the level of tokens and came out with a string of symbols which human mathematicians have recognized as a correct counter-example, exactly as if they would recognize any pixel patterns in a picture.
There is, however, a catch. Some would argue that the result was “novel”, never seen before and never published before, so the LLM “came up on its own”. This is, of course, anthropomorphic bullshit all over again. It managed to “come up” with a novel result exactly because it is a probabilistic inference engine, so it produces the most likely next token, but from a range of possible tokens with slightly different probabilities (mere Numbers which we interpret as a probability) associated with each token.
This tells us a few things: first, that there is a “simulation” (an appearance) of thinking by just combining the tokens from “a less common sense”, but not a complete bullshit at the lower layers of probabilities. This is “good” for the “research” and for the “code”, provided, as in the case of the code, that there is a strict and rigorous compiler of strongly-typed language, which can catch the subtle errors and turn them into proper syntax or even semantic fixes, keeping the “novel” (less common) logic intact.
For math that would be a proof-checker – a human or an automated one.
This, obviously, answers the question (You) are repeatedly asking nowadays: “which languages are the best for LLM vape-coding” – the strictest ones, with the most advanced static tying (Haskell, Ocaml, Rust). The more constraints (typing discipline) – the better. It is that simple. Math guys know the importance of constraints and additional rigor.
What “the humanties majors” can do about subtle errors in their sectarian-consensus based slop due to lack of compiler and proof-checkers ? Probably they shall get the [long deserved] rope.
And what about the “hack” and the “discovery of a subtle cryptographic weakness” (which was hard to spot since it has been obscured by idiotic unrtyped C preprocessor macros)? Well, there are, obviously, some symbolic constants in the code, such as “RNG”, which has been captured as full (new) “tokens”. These “tokens” appear both in texts and in the code, and are “probabilisticly closer” to the discussion of potential weakness of random number generators.
Again, it did no “analysis” (in principle), it managed, however, to spew out a sequence of code from its training data (or been loaded into the context verbatim), which apparently has the commonly discussed weakness, just at the level of “token shuffling”, without any “understanding” whatsoever.
Given that the current AI race is just about “increasing the brute force to the limits” (optimizing the “compute” and the number of parameters) we can expect that the larger models will generate “better-and-better” common sense sequences (literally) but coming up with something truly “novel” and “never seen before” will be limited and a matter of a pure chance (an actual “path” the in-principle non-deterministic and non-reproducible inference algorithm will occasionally take).
One more time: what midwits see as “intelligence” is only appearance of it due to an enormous brute-force of “compute” behind it. Yes, in mathematics a brute-force approach (explicit enumeration and checking for a property to hold) is a feasible, but notice that there has to be a “checker” for subtle bullshit – a divergence from the current “common sense” verbiage.
This is where the most potential “research” breakthroughs are – in the less “traveled paths”, as good poets put it. The problem is validation. And no, another LLM cannot validate a novel result, since it requires a different kind of discipline (either proper experiments or prof-checking).
But aside from that, talking about interesting subjects with a modern LLMs, even at the shit-tier free plans, is amazing experience, and forcing it to write a minimal, “optimal” code (which it cannot, again, in principle, do by itself) is a rare joy (after hours of frustration and tension).
So, the bottom line – still a “glorified autocomplete”.