Modern coding LLMs are still a shitshow. I would not even comment on the humanties – the “sectarian consensus” abstract (ill-defined) verbiage (subtle bullshit) it could produce – not even an expert could “validate” the “correctness” of the slop (such notion is not defined in their domains).
It would be interesting to heavily prompt it about rigorous mathematics, properly captured, generalized and named from the observed aspects of What Is (which is the only proper mathematics, including the derivations of pure abstract algebraic structures, like Monoid, a Group, Latice or even a Category) … Okay, some day.
At the level of code, however, it is still a bullshit generator in principle.
The main question is – how exactly, while operating at the level of “arbitrary” tokens (not even morphemes) in principle, the particular “architecture” (which is the only thing that is different between models) “semantically accurate bridges the gap between the code and the accompanied verbiage” (hint: it does not).
Notice that the only “actual bridge” is naming – the way human language itself “works”. We tend to give to the particular “things” associated “sound patterns” to refer. Consistent naming and the knowing of the definition for a word before hearing it, are the necessary preconditions for a verbal communication to occur.
So, in an imaginary, idealized world, when something appears in the verbiage as a Frame Of Environment, it is then appears in code as a frameOfEnvironment. LOL no. Our brains have to learn the mapping between the domain concepts and the code constructs (Algebraic Data Types), and this is murderously hard (hi, dr. Bloom).
The slop generator does nothing like that. Its representation does not capture such “distant” semantic relations (no matter what the fucking theorists and talking heads would say to you). The whole process produces an illusion that it does, but no, in principle. The “relations” of being “close” (in a cluster, as they say), are not “semantic”, but “statistical” – how many times seen together (interspersed with other tokens).
The whole premise (the dogma) that this statistical “high-dimensional” model actually captures the proper semantic relations is a sectarian bullshit. Read this again, if you will.
The resent clever (I admit) “hacks” include the forced practice to heavily annotate the code with the markdown verbiage and to feed that “structured” slop into a learning feed-back loop. The slop generators nowadays are being trained almost entirely on its own slop, which precious human feedback (of accepting or rejecting the solution) in a hope that this will gradually “adjust the knobs” in the right direction (lower observable error rates) and even “converge” onto something (no, your will get stuck in the countless local optima “holes”).
When you see that the verbiage is coherent, clear, concise and affirmative about subtle semantic notions and constraints, keep in mind that the text has been synthesized independently from the code, by the process that have “no idea” that there is any code – everything is just a “next token. given the context”.
This is exactly how the illusion works – it crates an appearance that everything is consistent, while it has no means (no machinery) to enforce any semantic consistency whatsoever. Again, the text only appear to be closely related to the code and vise versa.
Only an actual expert would see the inconsistencies and sloppiness through an unwarranted dopamine spikes of tight feedback loop on vastly complex activities. The assertions stated in the accompanied verbiage simply do not hold. Yes, the overall appearance is convincing, but the semantics has been tossed out of the window.
Yes, it may use proper ADTs but a few lines down the file it will use crappy imperative tricks, like relying on the implementation and representation details, just because the code it was trained on was such kind of an imperative crap. Try to explicitly force everything to be private and everything breaks (and then being “fixed” by make everything public again, LOL).
And the unnecessary, redundant shit like .into_vec() and other “optimizations”.. You have no idea how bad the slop is apart from the fact that it somehow (eventually) runs.
The way it “fixes the errors” is a sheer brute-force (without any kind of necessary understanding whatsoever), by just feeding the code, the compiler output (errors) back as new training examples, they managed to get a measurable marginal improvement (again, without any semantics).
These are “mechanics behind the illusion”. The appearance is so convincing that the monkeys^W humans with typewriters “label” the structured (as a markdown) slop as “good” or “bad”, driven by the spikes and crashes of their neuromodulators, and this is exactly what makes the current shitshow run.
One more time: the claims in the verbiage about the code are subtly wrong and cannot be made right in principle. The process which generates the verbiage is, in principle, “unaware” of the code (and vise versa) but the convergence is happening as we speak, but only at the level of statistical “closeness” of individual tokens (not even the level of proper syntax).
The wast storage could, however, partition the network representation so fine, that distinct language-specific “clusters” will emerge, further partitioned into common problem-specific clusters within. This is their (the idiots who invested trillions of borrowed money without even looking back) only hope.
So, one last time: most of the claims about the semantics of code in the accompanying verbiage (markdown) are subtle bullshit. The actual semantics of the code come from the training data and LLMs cannot, in principle, perform transformations at the level of the code, which actually preserve the semantics (which is what optimizing compilers do). LLMs produce only a mere appearance of doing so, and this is an illusion which is being sold to you (so that you feel special, which is a necessary condition for selling illusions of any kind).