A product-type is a generalized notion of having a bunch of things at the same physical locality (or proximity) – this and [also] that. This is what molecules are.
A sum-type is a generalized abstract notion of having this-or-that (either-this-or-that, and NOT both). This is, among other things, a “fork” on a path – never both can be taken (except when backtracking).
These notions are enough for everything and they are our basic building blocks.
These things can be nested arbitrarily.There is, however, a so-called canonical form, which is a “sum of products”. This abstraction captures the notion of that selection comes first.
Any product-type – a tuple, a record or a “struct” can be thought of as an amino acid, which has a unique “side chain” that “hangs form” a central carbon atom.
Abstractly a product-type is a [unordered] set of slots (I love the ancient terminology). We usually “select” a single slot by its name.
Conceptually, we can imagine these slots as handing from a binding or – at an implementation side – from a particular location in memory, from an “address”).
A for a representation of product-type we either access these slots by an offset, and thus by a number, or by a symbolic name associated with each slot.
Every slot thus is a nested binding (of a symbol to a value) and every value has a type.
There are various ways to encode these data structures in some “human readable” format, such as JSON, which is also used as an “in-wire” representation of the data.
The order of named slots is irrelevant and immaterial in principle, but there is always some implicit order when we represent these data structures, in memory or in any other “written form”.
So, when we use JSON there will always be some arbitrary order in which the data has been written or “serialized”. This does not imply that there is any “meaning” in this order.
The most important and the most difficult thing is to zoom in-and-out between a pure abstraction (which usually defined using some familiar and well-understood mathematical concepts and notions) and its implementations, which are defined in terms of some programming language according to its semantics.
The most important principle is that mathematics itself is a collection of captured and properly generalized abstractions, and a systematic study of their properties and relations between them. Mathematical abstractions have sets of values and sets of operations defined on them, which is what “concrete” Abstract Data Types are.
Ideally, every time we analyze some problem domain, we have to build a simple mathematical theory of the domain, which consist of properly captured abstractions, definitions of their properties and well-defined relations among them. Then “the code will write itself”. At least pure functional code will.
So, a bunch of people around the world are watching the same charts on computer screens.
Not going deep into the signals-and-systems theory, the “signal” has been “digitized” into a sequence of “numbers” captured at regular (the same) intervals or “candles”, which is a collection of numbers, or a product-type. It does not really matter [right now] how we will represent it.
A typical candle is usually two quantities – “low” and ‘high" – and a “timestamp”, which is metadata about the candle itself.
More advanced candles have four numbers – “low”, “high”, “open” and “close”, and a “timestamp”.
The fundamental principle, which makes everything meaningful, is that we are observing and “measuring” something (overall sentiment of a crowd) and what we create are “instruments” (exactly like a speedometer and tachometer or a pressure meter).
So called “indicators” are just “digital” instruments, which calculate and display some measurements and some “derived” quantities. And that is it.
We won’t be analyzing these candles with a bunch of some explicit if-then-else rules. We will feed them into a simple neural net, similar to what @karpathy did in the “make-more” series.
Just like the next “token” is a “label” for a bunch of tokens before it, each new candle is a “truth label” for a bunch of candles before it. Read the previous sentence many times.
This is the only non-bullshit application of Deep Learning (with all the resulting limitations). The whole net will “learn” some conditional probabilities, based on some input (for a particular period of time and an interval between candles).
When “sampled from”, it will produce the most probable “next candle”, which means almost nothing, just an “educated guess” or, indeed, a “forecast”.
It will be as bad as any weather forecast in principle, and this is, in principle, the best thing one can do.
We will augment (or enhance) the input candles with additional “synthetic features” (as been taught by Andrew Ng) in home that the network will capture some “hidden” (form our limited minds) relations among them.
Volume candles, some basic commonly used “indicators” (readings) and other metrics such as “funding rate” or an “open interest” has to be added.
The principle is that “Everything that humans (and bots) see and use in [emotional] decision making” has to be captured and feed forward. Intuitively, this is just the color of last few candles and a volume, along with current memes and “popular narratives”.
The theory said that no “feature” shall be presented as a ratio of other features, but this assumption can only be experimentally validated.
There are some problems, however. They are of the two kinds – the fundamental problems and ordinary problems.
The fundamental problems are: /any simulations, proper modeling, leave alone predictions, can be done only for a"closed systems", such as systems with a constant mount of energy (EE majors would understand). For any other “system” the simulations and models are necessarily flawed in these ways:
- not all relevant aspects or factors are properly captured.
- one relevant missed (or one irrelevant imaginary added) will yield nonsense.
- aspects or factors with used to be significant become irrelevant “as we model”.
- completely new factors emerge or “come from outside” (the system isn’t a closed one).
- the whole “domain” is continuously changing and evolving “as we speak”.
These observations just confirm the old fact (maxim) that “all models are wrong, but some are [to some extent] useful”.
The ordinary problem’s are as follows: the most profitable strategy to make money out of the market is not just “front-running the retail” (which is the second best, and as old as the first brokerage) but to “take actions and trade against retail using exchanges data about the open positions and basic sentiment analysis” which is exactly what SBF with friends (and literally every other institution in existence ) are doing.
SBF wasn’t big enough to move the BTC pairs, so he manipulated some popular shitcoins. Nowadays the institutional players are big-enough to move even BTC. Even Saylor moves the market.
An exchange and a brokerage profits not just from fees, but mostly when their clients are changing positions and doing stupid emotional shit, like panicing or FOMOing in. Thus it pays off to initiate certain market events, like fake reversals or fake breakouts (or simply to manipulate an unregulated marker) to “cause the maximum pain”, or to force the majority of the clients to change (close and reopen, lets say) their positions.
For futures trading liquidations are simply direct profits, but futures are just a small percentage of all trading volume. Longing shitcoins for leveraged trading (liquidations is the mechanism to guarantee that the leaased stuff will always be forcibly collected back) is a [fully automated automated] elite business too.
In general, everything that is written in popular books, even in the best ones, is just an idealized gross over-simplification at all levels. Even Trump’s post on some social media (and tweets of Musk before that) could move the markets in the direction of “the maximum pain” or “irrational bullish exuberance”. No model will be able to “predict” such events, because they are coming ‘from outside of the system’ and their magnitude varies “from zero to one”.
One more time, only “closed” (constant energy) physical systems could be properly simulated. Others just cannot in principle. You cannot sidestep this universal fact.
Now lets talk about abstract esoteric stuff.
Candles form a Semigroup, which we can artificially promote into a Monoid by introducing an “empty candle”. The operation is not just component-wise addition, but a “concatenation” by calculating the maximum and minimum over a total interval for each pair of “readings” (metrics), provided the candles are adjacent. Basically it is equivalent to a string-concatenation Monoid
This means that all the calculations are properly composable.
A candle itself is a trivial Functor (as a specialized product-type) and a list-of-candles (which represent a whole time-series) is a Functor (by the virtue of being a List).
These observations set clear abstraction boundaries and simplify the right understanding of the domain by reduction to a familiar and well-understood abstract mathematical notions. The model becomes almost minimal.
The whole literature about “timeseries forecasting” is mostly irrelevant. It is full of naive assumptions about possibility to model constantly evolving systems.
From the “digital signal processing” academic books all we need is the notion of “adding”, which for us is just capturing the extreme values with all other information being lost forever (an irreversible transformation).
As our time interval (between “snapshots”) is arbitrary but always the same, our notion of a “derivative” is just the difference in magnitude (the candle’s height).
The a concept of a Kalman Filter seems to be applicable, but being simply a moving average it is implicitly captured by a neural network.
Breaking down every common “indicator” (which is just a mathematical formula) and tracing everything back to What Is (what it is trying to capture and generalize) is the most difficult remaining task.
And this is basically it. Everything else will be more-or-less implementation details and a tedium.