Sometimes, when I do not know what to do with myself I re-watch some lectures for this or that online course on Youtube. Better than doom-scrolling crap, I guess.
Something similar to that Bladerunner 2049 part, where they watched old Elvis as a hologram.
Today it was the 'Lecture 2 ｜ Machine Learning (Stanford) [5u4G23_OohI].mp4' (of the old CS229, on a blackboard).
If you watch this think of Tyson in a gym before he became the world champion. The comparison is in how much talent one has to have and then what one can achieve.
In my miserable soviet times at prole ghetto schools we never been taught maths properly, from the ground up, from the very aspects of the underlying reality, which the mathematical abstractions capture, all the way up to this kind of derivation performance. Here I explicitly laugh at @karpachy who only pastes from the web and LLMs.
Without profound understanding on mathematics any kind of bullshit with pieces of math in it sounds “serious” and, most importantly, feels like “magic” only trained magicians could perform.
On the other hand, if you really try to rebuild the whole chain, right from the first principles, everything clicks, but there is no use for this refined knowledge.
Some sufi teachers argued that accumulating knowledge is as futile as accumulating anything else, especially golden coins. We have entered the era in which this became true.
So, there are two fundamental operations on Natural Numbers – putting together, and adding to itself many times (“times” – multiplication as repeated addition).
Both addition and multiplication are commutative, which means it does not matter to which pile you add (one-by-one) the other pile, or, similarly, it does not matter which quantity became the “number of times to add”.
Commutativity tells you that the order does not matter, and this, in turn, is related to partial derivatives – how much some particular quantity involved in a commutative operation (or in a bunch nested ones) “contribute” to the result.
There is also the mathematical notion of multiplication as scaling (instead of just adding many times). Again, it does not matter which number scales which, the result will be the same.
Then it is possible to express negation and division (on $\Real $) as “adding a negative number” (implicitly multiplied by \(-1\) to “flip” a positive number on a number line) and as “scaling by a reciprocal”, which is just \(1/n\), and, indeed, division is “splitting into n equal parts”, which, by the way, is not always possible for natural numbers.
Now “just” \(+\) and \(\times\) is enough for “everything”. Everything interesting, at least.
The first “interesting” construction is a Weighted Sum. Almost everything can be thought off (or abstracted and represented as) a Weighted Sum.
A Polynomial is particular/special kind of a weighted sum, but they by themselves are much less “interesting” that more general weighted sum itself.
For example, a generalized weighed sum is the “formula for Causality itself” (while a Directed Acyclic Graph is the “shape of the Causality”).
Just list all the factors involved, sort them out by how much they contribute, make sure there none relevant missing and none imaginary added, and wallah, just like that.
Of course, the “threshold” notion for a chemical process is just that sum of relevant factors, exceeding some “energy threshold”.
So far we have talked only about “singe” Real numbers added together or Real numbers scaled by other real number constants (cx).
What if we have more than one numerical “coordinate”?
Here comes the first “aha-moment” – the “whole” Linear Algebra is just packaging of several numbers together and performing \(+\) and \(\times\) operations on them. Nothing fundamentally new has been added anywhere.
This, in turn, “explains” how some parts of Neural Networks can be expressed as matrices and why this is “enough for everything”.
There is another “aha” – no “self-attention”, no “muti-head attention”, no “communication between tokens” are necessary or even required – just matrix multiplications (as function compositions) and some “nonlinearity”.
But this is a very long and laborous climb to the top. But trust me, all the misleading (bullshit-bordering) “applied NLP” terminology (“queries” which are not queries, “keys” which are not keys, “attention” which is not attention, and so on) are irrelevant in the end, and are just different kinds of matrices being multiplied (composed).
So, the \(+\), the \(\times\) and some differentiable “compound” operations, such as “ReLU” or a “Sigmoid”.
The key “higher level” building blocks are Partial Derivatives, the “Chain Rule”, the Jacobians and some Category-theoretic composition
- No special “attention” primitive is mathematically necessary.
- No special “multi-head” primitive is mathematically necessary.
- No special “communication” primitive is mathematically necessary.
They are architectural patterns expressible within ordinary differentiable composition.
The only universal machinery is the algebra of typed values, composition, differentiation, and weighted sums.
And once differentiated:
\[ composition \;\equiv\; matrix multiplication. \]
That is the point where the whole BS Transformer vocabulary collapses back into the much older and more fundamental language of linear algebra + partial derivatives + graph composition.
Ok, lets climb.
First, all these “architectural decisions” are just artifacts of particular tradition or a “sect” (like applied NLP).
In principle, a single “large” DAG of matrix multiplications, which obey the laws of categorical composition, is enough. No special “layers” are necessary and no “communication” between these.
What is called “communication” is just one matrix multiplied by another one, in which some elements are zeros, and thus the non-zero elements are the “messages”.
Now another “aha” – since back-propagation changes all the weights, it does not matter how exactly you interpret there matrix multiplications – the weights “here” are no different from weights “there”, and if one is zeroing some or others, that just changes the “connectivity” within a DAG. Any attached “meaning” is just a social/sectarian construct and fundamentally is not “better” than any other arrangement.
Suppose that instead of the “sacred” Transformer architecture one constructs an arbitrary, “uniform”, “shallow” DAG of nested matrix multiplications initialized at random. On the same training data this one, in principle, will converge to the same results, because it is not “architectures” (arrangements) but the primitive operations that actually implement the “learning”.
Let’s now turn this intuitive knowledge into a formal mathematical system, which is a long and laborous climb.