I got to know HTM a year ago when my advisor told me there were internship opportunities in Jeff Hawkins’ company Numenta. Though I didn’t apply for the intern by the end, I did learn and got some basic ideas of their main product – HTM. I really appreciate the company’s persistence towards the discovery of biology-inspired artificial intelligence, especially on today when deep learning has beat more and more opponents and somehow proved the elegance of brute force.
Sparse Distributed Representation (SDR) is one of the fundamentals of HTM system. SDR can be representad as n-dimensional binary vector, in which only a small percentage of components are 1. ## 1. Biological Evidence 
Sparse coding may be a general strategy of neural systems to augment memory capacity. Sparse representation of sensory information has been observed in vision, audition, touch and olfactory systems.
To any given stimulus, only very few amount of neurons have response. And any neuron in the population can only respond to a very small subset of all kinds of stimulus.
Advantages of SDR: 1. High Efficiency (energy consumption is lower than fully active neural networks) 2. Robustness (high tolerance to noises) 3. High Capacity (the overlap of representation is highly reduced)
2. Mathematical Properties of SDRs
Some basic terminologies
w: number of 1s in an SDR (vector cardinality). \(w=||x||_1\).
Overlap: number of 1s in bitwise AND (Jeff Hawkins used ON in his paper)
Matching: a boolean value that is True when Overlap exceeds some threshold \(\theta\).
Uniqueness of SDR
Given vector dimension n and “norm” w, the number of unique SDRs is simply the w-Combination of set n: \(\frac{n!}{w!(n-w)!}\).
Therefore, the probability that two random SDRs are identical is very very low. (e.g. let n=1024, and w=4, the probability is 1 out of 45 billion.)
Overlap Sets
An overlap set is defined as the set of SDRs that excatly b bits overlap with SDR x, given n and w.
The size of the overlap set is the product of the number of subsets of x with b bits on, and the number of other patterns containing n-w bits, of which w-b bits are on.
\[ overlap(x,y)=x\cdot y \]
Inexact matching and subsampling
Inexact matching and subsampling are similar in finding partial matching between SDRs. There’s tradeoff between tolerance of errors (robustness) and the amount of false positives.
A false positive SDR y satisfies \(overlap(x,y)\geq\theta\).
Inexact matching decreases the right side: The lower the threshold is, the easier for a random SDR y to satisfy the inequation.
Subsampling increases the left side: the smaller the overlap set is, the easier for a random SDR y to satisfy the inequation.