Recently I’ve re-watched some of “iconic” (LMAO) @karpathy videos, and saw (again) a lot of very fast hand-waving (literally!) when it he tried to explain the whys behind so called multi-head attention, self-attention and all the catchy memes created by disconnected from reality cocky NLP guys.

I have seen this before. There is a typical, canonical, if you will, social construct called Mandala, being usially expressed as a paining caleld Tanka.

The point is that very long ago the social consensus has been to use a Mandala (which is, originally, just a circle drawn on the ground) as the common way to express various mystical abstract ideas and notions, using sophisticated symmetries (symmetric, repetitive visual patterns) as a visual cue for deep, profound sophistication and complexity.

Then, just like any other art form, it became “solidified” and passed among generations as “the Right Way”, which is exactly how any tradition works. No one asks “why” the Right thing is the right one, it is just how it came to be.

Why do we need all these “keys”, “queries” and somethings, which are not keys atl all, not queries or even filters (every query has implicit such that) and are just matrix operations and compound manipulations.

All this is just an accidental complexity, which happen to propagate from the very specific areas of research, and the proper generalization shall, in principle, drop this accidental complexity and reveal the “essense” behind these layers of bullshit operations, which happen to propagate some numbers though.

The brain does not do any of this shit. It seemingly just maintains dynamic “gradients” or “states” in the so-called synaptic gaps and physically (and chemically) alter the “connections” (axons, dendrites) based on repeated use. The repeated use, in turn, comes from the “reward” (and “punishment”) systems at the way higher level, via hormonal signaling.

Anyway, the point is that a back-propagated DAG is enough for everything, and the operations and the layers has to “emerge”, rather that being taken for granted out of tradition.

The fact that the brain has evolved specialized areas suggests that there has to be some elaborate, multi-layer “architecture”, but the brain has its own “technical debt”, and a LOT of it.

So, which is a minimal set of operations on DAG, represented in some matrix form, which is enough to have a proper representation of “reality” as a weighted graph?

Linear algebra suggests some, numerical methods suggests the others (normalization) is the first that comes to the mind. Non-linearity (sigmoids) are also “fine”, and the weighted sums, of course, being at the core of everything.

I strongly suggest that all the “attention” crap has been dropped and a new, simplified, essential architecture has to be derived from the first principles, and trials and errors, instead of coming up from the sacred tradition.

The “old” ideas of having everything fully connected and the “pruning” what turned out to be redundant, which is what biology does, is, perhaps, the key. The brain has a LOT of redundancy.

The “communication” meme would be implicitly solved by changing the “weights” of the edges of a fully connected DAG in a way that a useful “structure” emerges from pruning.

Again, a weighted back-propagated DAG is, in principle, enough for everything, but is inefficient and costly to store and to train. The “centers” has to emerge, but definitely not from NLP -related abstractions.

As for natural languages, the idea to have a single inner representation for multiple languages is exactly right, since the underlying Reality–What Is – is what all human languages strive to captures, describe and communicate. And now we know what the encoding of any representation is (a “weighted” DAG of synaptic gaps).

But who am I compared to hand-waving @karpathy on the stage.