Thanks for the Memory
While reading Jeff Hawkins’ book On Intelligence, I was struck by the resonant coherence of his memory-prediction framework for how the cortex works. It was like my first exposure to complexity theory at the Santa Fe Institute – providing a new perceptual prism for the world. So, I had to visit him at the Redwood Neuroscience Institute.
As a former chip designer, I kept thinking of comparisons between the different “memories” – those in our head and those in our computers. It seems that the developmental trajectory of electronics is recapitulating the evolutionary history of the brain. Specifically, both are saturating with a memory-centric architecture. Is this a fundamental attractor in computation and cognition? Might a conceptual focus on speedy computation be blinding us to a memory-centric approach to artificial intelligence?
• First, the brain:
“The brain does not ‘compute’ the answers to problems; it retrieves the answers from memory… The entire cortex is a memory system. It isn’t a computer at all.”
Rather than a behavioral or computation-centric model, Hawkins presents a memory-prediction framework for intelligence. The 30 billion neurons in the neocortex provide a vast amount of memory that learns a model of the world. These memory-based models continuously make low-level predictions in parallel across all of our senses. We only notice them when a prediction is incorrect. Higher in the hierarchy, we make predictions at higher levels of abstraction (the crux of intelligence, creativity and all that we consider being human), but the structures are fundamentally the same.
More specifically, Hawkins argues that the cortex stores a temporal sequence of patterns in a repeating hierarchy of invariant forms and recalls them auto-associatively. The framework elegantly explains the importance of the broad synaptic connectivity and nested feedback loops seen in the cortex.
The cortex is relatively new development by evolutionary time scales. After a long period of simple reflexes and reptilian instincts, only mammals evolved a neocortex, and in humans it usurped some functionality (e.g., motor control) from older regions of the brain. Thinking of the reptilian brain as a “logic”-centric era in our development that then migrated to a memory-centric model serves as a good segue to electronics.
• And now, electronics:
The mention of Moore’s Law conjures up images of speedy microprocessors. Logic chips used to be mostly made of logic gates, but today’s microprocessors, network processors, FPGAs, DSPs and other “systems on a chip” are mostly memory. And they are still built in fabs that were optimized for logic, not memory.
The IC market can be broadly segmented into memory and logic chips. The ITRS estimates that in the next six years, 90% of all logic chip area will actually be memory. Coupled with the standalone memory business, we are entering an era for complex chips where almost all transistors manufactured are memory, not logic.
At the presciently named HotChips conference, AMD, Intel, Sony and Sun showed their latest PC, server, and PlayStation processors. They are mostly memory. In moving from the Itanium to the Montecito processor, Intel saturated the design with memory, moving from three megabytes to 26.5MB of cache memory. From a quick calculation (assuming 6 transistors per SRAM bit and error correction code overhead), the Montecito processor has ~1.5 billion transistors of memory, and 0.2 billion of logic. And Intel thought it had exited the memory business in the 80’s. |-)
Why the trend? The primary design enhancement from the prior generation is “relieving the memory bottleneck.” Intel explains the problem with their current processor: "For enterprise work loads, Itanium executes 15% of the time and stalls 85% of the time waiting for main memory.” When the processor lacks the needed data in the on-chip cache, it has to take a long time penalty to access the off-chip DRAM. Power and cost are also improved to the extent that more can be integrated on chip.
Given the importance of memory advances and the relative ease of applying molecular electronics to memory, we may see a bifurcation in Moore’s Law, where technical advances in memory precede logic by several years. This is because molecular self-assembly approaches apply easily to regular 2D structures, like a memory array, and not to the heterogeneous interconnect of logic gates. Self-assembly of simple components does not lend itself to complex designs. (There are many more analogies to the brain that can be made here, but I will save comments about interconnect, learning and plasticity for a future post).
Weaving these brain and semi industry threads together, the potential for intelligence in artificial systems is ripe for a Renaissance. Hawkins ends his book with a call to action: “now is the time to start building cortex-like memory systems... The human brain is not even close to the limit” of possibility.
Hawkins estimates that the memory size of the human brain is 8 terabytes, which is no longer beyond the reach of commercial technology. The issue though, is not the amount of memory, but the need for massive and dynamic interconnect. I would be interested to hear from anyone with solutions to the interconnect scaling problem. Biomimicry of the synapse, from sprouting to pruning, may be the missing link for the Renaissance.
P.S. On a lighter note, here is a photo of a cortex under construction. ;-)
As a former chip designer, I kept thinking of comparisons between the different “memories” – those in our head and those in our computers. It seems that the developmental trajectory of electronics is recapitulating the evolutionary history of the brain. Specifically, both are saturating with a memory-centric architecture. Is this a fundamental attractor in computation and cognition? Might a conceptual focus on speedy computation be blinding us to a memory-centric approach to artificial intelligence?
• First, the brain:
“The brain does not ‘compute’ the answers to problems; it retrieves the answers from memory… The entire cortex is a memory system. It isn’t a computer at all.”
Rather than a behavioral or computation-centric model, Hawkins presents a memory-prediction framework for intelligence. The 30 billion neurons in the neocortex provide a vast amount of memory that learns a model of the world. These memory-based models continuously make low-level predictions in parallel across all of our senses. We only notice them when a prediction is incorrect. Higher in the hierarchy, we make predictions at higher levels of abstraction (the crux of intelligence, creativity and all that we consider being human), but the structures are fundamentally the same.
More specifically, Hawkins argues that the cortex stores a temporal sequence of patterns in a repeating hierarchy of invariant forms and recalls them auto-associatively. The framework elegantly explains the importance of the broad synaptic connectivity and nested feedback loops seen in the cortex.
The cortex is relatively new development by evolutionary time scales. After a long period of simple reflexes and reptilian instincts, only mammals evolved a neocortex, and in humans it usurped some functionality (e.g., motor control) from older regions of the brain. Thinking of the reptilian brain as a “logic”-centric era in our development that then migrated to a memory-centric model serves as a good segue to electronics.
• And now, electronics:
The mention of Moore’s Law conjures up images of speedy microprocessors. Logic chips used to be mostly made of logic gates, but today’s microprocessors, network processors, FPGAs, DSPs and other “systems on a chip” are mostly memory. And they are still built in fabs that were optimized for logic, not memory.
The IC market can be broadly segmented into memory and logic chips. The ITRS estimates that in the next six years, 90% of all logic chip area will actually be memory. Coupled with the standalone memory business, we are entering an era for complex chips where almost all transistors manufactured are memory, not logic.
At the presciently named HotChips conference, AMD, Intel, Sony and Sun showed their latest PC, server, and PlayStation processors. They are mostly memory. In moving from the Itanium to the Montecito processor, Intel saturated the design with memory, moving from three megabytes to 26.5MB of cache memory. From a quick calculation (assuming 6 transistors per SRAM bit and error correction code overhead), the Montecito processor has ~1.5 billion transistors of memory, and 0.2 billion of logic. And Intel thought it had exited the memory business in the 80’s. |-)
Why the trend? The primary design enhancement from the prior generation is “relieving the memory bottleneck.” Intel explains the problem with their current processor: "For enterprise work loads, Itanium executes 15% of the time and stalls 85% of the time waiting for main memory.” When the processor lacks the needed data in the on-chip cache, it has to take a long time penalty to access the off-chip DRAM. Power and cost are also improved to the extent that more can be integrated on chip.
Given the importance of memory advances and the relative ease of applying molecular electronics to memory, we may see a bifurcation in Moore’s Law, where technical advances in memory precede logic by several years. This is because molecular self-assembly approaches apply easily to regular 2D structures, like a memory array, and not to the heterogeneous interconnect of logic gates. Self-assembly of simple components does not lend itself to complex designs. (There are many more analogies to the brain that can be made here, but I will save comments about interconnect, learning and plasticity for a future post).
Weaving these brain and semi industry threads together, the potential for intelligence in artificial systems is ripe for a Renaissance. Hawkins ends his book with a call to action: “now is the time to start building cortex-like memory systems... The human brain is not even close to the limit” of possibility.
Hawkins estimates that the memory size of the human brain is 8 terabytes, which is no longer beyond the reach of commercial technology. The issue though, is not the amount of memory, but the need for massive and dynamic interconnect. I would be interested to hear from anyone with solutions to the interconnect scaling problem. Biomimicry of the synapse, from sprouting to pruning, may be the missing link for the Renaissance.
P.S. On a lighter note, here is a photo of a cortex under construction. ;-)
posted by Steve Jurvetson at
6:03 PM
|
25 comments
