Thanks for the Memory
While reading Jeff Hawkins’ book On Intelligence, I was struck by the resonant coherence of his memory-prediction framework for how the cortex works. It was like my first exposure to complexity theory at the Santa Fe Institute – providing a new perceptual prism for the world. So, I had to visit him at the Redwood Neuroscience Institute.
As a former chip designer, I kept thinking of comparisons between the different “memories” – those in our head and those in our computers. It seems that the developmental trajectory of electronics is recapitulating the evolutionary history of the brain. Specifically, both are saturating with a memory-centric architecture. Is this a fundamental attractor in computation and cognition? Might a conceptual focus on speedy computation be blinding us to a memory-centric approach to artificial intelligence?
• First, the brain:
“The brain does not ‘compute’ the answers to problems; it retrieves the answers from memory… The entire cortex is a memory system. It isn’t a computer at all.”
Rather than a behavioral or computation-centric model, Hawkins presents a memory-prediction framework for intelligence. The 30 billion neurons in the neocortex provide a vast amount of memory that learns a model of the world. These memory-based models continuously make low-level predictions in parallel across all of our senses. We only notice them when a prediction is incorrect. Higher in the hierarchy, we make predictions at higher levels of abstraction (the crux of intelligence, creativity and all that we consider being human), but the structures are fundamentally the same.
More specifically, Hawkins argues that the cortex stores a temporal sequence of patterns in a repeating hierarchy of invariant forms and recalls them auto-associatively. The framework elegantly explains the importance of the broad synaptic connectivity and nested feedback loops seen in the cortex.
The cortex is relatively new development by evolutionary time scales. After a long period of simple reflexes and reptilian instincts, only mammals evolved a neocortex, and in humans it usurped some functionality (e.g., motor control) from older regions of the brain. Thinking of the reptilian brain as a “logic”-centric era in our development that then migrated to a memory-centric model serves as a good segue to electronics.
• And now, electronics:
The mention of Moore’s Law conjures up images of speedy microprocessors. Logic chips used to be mostly made of logic gates, but today’s microprocessors, network processors, FPGAs, DSPs and other “systems on a chip” are mostly memory. And they are still built in fabs that were optimized for logic, not memory.
The IC market can be broadly segmented into memory and logic chips. The ITRS estimates that in the next six years, 90% of all logic chip area will actually be memory. Coupled with the standalone memory business, we are entering an era for complex chips where almost all transistors manufactured are memory, not logic.
At the presciently named HotChips conference, AMD, Intel, Sony and Sun showed their latest PC, server, and PlayStation processors. They are mostly memory. In moving from the Itanium to the Montecito processor, Intel saturated the design with memory, moving from three megabytes to 26.5MB of cache memory. From a quick calculation (assuming 6 transistors per SRAM bit and error correction code overhead), the Montecito processor has ~1.5 billion transistors of memory, and 0.2 billion of logic. And Intel thought it had exited the memory business in the 80’s. |-)
Why the trend? The primary design enhancement from the prior generation is “relieving the memory bottleneck.” Intel explains the problem with their current processor: "For enterprise work loads, Itanium executes 15% of the time and stalls 85% of the time waiting for main memory.” When the processor lacks the needed data in the on-chip cache, it has to take a long time penalty to access the off-chip DRAM. Power and cost are also improved to the extent that more can be integrated on chip.
Given the importance of memory advances and the relative ease of applying molecular electronics to memory, we may see a bifurcation in Moore’s Law, where technical advances in memory precede logic by several years. This is because molecular self-assembly approaches apply easily to regular 2D structures, like a memory array, and not to the heterogeneous interconnect of logic gates. Self-assembly of simple components does not lend itself to complex designs. (There are many more analogies to the brain that can be made here, but I will save comments about interconnect, learning and plasticity for a future post).
Weaving these brain and semi industry threads together, the potential for intelligence in artificial systems is ripe for a Renaissance. Hawkins ends his book with a call to action: “now is the time to start building cortex-like memory systems... The human brain is not even close to the limit” of possibility.
Hawkins estimates that the memory size of the human brain is 8 terabytes, which is no longer beyond the reach of commercial technology. The issue though, is not the amount of memory, but the need for massive and dynamic interconnect. I would be interested to hear from anyone with solutions to the interconnect scaling problem. Biomimicry of the synapse, from sprouting to pruning, may be the missing link for the Renaissance.
P.S. On a lighter note, here is a photo of a cortex under construction. ;-)
As a former chip designer, I kept thinking of comparisons between the different “memories” – those in our head and those in our computers. It seems that the developmental trajectory of electronics is recapitulating the evolutionary history of the brain. Specifically, both are saturating with a memory-centric architecture. Is this a fundamental attractor in computation and cognition? Might a conceptual focus on speedy computation be blinding us to a memory-centric approach to artificial intelligence?
• First, the brain:
“The brain does not ‘compute’ the answers to problems; it retrieves the answers from memory… The entire cortex is a memory system. It isn’t a computer at all.”
Rather than a behavioral or computation-centric model, Hawkins presents a memory-prediction framework for intelligence. The 30 billion neurons in the neocortex provide a vast amount of memory that learns a model of the world. These memory-based models continuously make low-level predictions in parallel across all of our senses. We only notice them when a prediction is incorrect. Higher in the hierarchy, we make predictions at higher levels of abstraction (the crux of intelligence, creativity and all that we consider being human), but the structures are fundamentally the same.
More specifically, Hawkins argues that the cortex stores a temporal sequence of patterns in a repeating hierarchy of invariant forms and recalls them auto-associatively. The framework elegantly explains the importance of the broad synaptic connectivity and nested feedback loops seen in the cortex.
The cortex is relatively new development by evolutionary time scales. After a long period of simple reflexes and reptilian instincts, only mammals evolved a neocortex, and in humans it usurped some functionality (e.g., motor control) from older regions of the brain. Thinking of the reptilian brain as a “logic”-centric era in our development that then migrated to a memory-centric model serves as a good segue to electronics.
• And now, electronics:
The mention of Moore’s Law conjures up images of speedy microprocessors. Logic chips used to be mostly made of logic gates, but today’s microprocessors, network processors, FPGAs, DSPs and other “systems on a chip” are mostly memory. And they are still built in fabs that were optimized for logic, not memory.
The IC market can be broadly segmented into memory and logic chips. The ITRS estimates that in the next six years, 90% of all logic chip area will actually be memory. Coupled with the standalone memory business, we are entering an era for complex chips where almost all transistors manufactured are memory, not logic.
At the presciently named HotChips conference, AMD, Intel, Sony and Sun showed their latest PC, server, and PlayStation processors. They are mostly memory. In moving from the Itanium to the Montecito processor, Intel saturated the design with memory, moving from three megabytes to 26.5MB of cache memory. From a quick calculation (assuming 6 transistors per SRAM bit and error correction code overhead), the Montecito processor has ~1.5 billion transistors of memory, and 0.2 billion of logic. And Intel thought it had exited the memory business in the 80’s. |-)
Why the trend? The primary design enhancement from the prior generation is “relieving the memory bottleneck.” Intel explains the problem with their current processor: "For enterprise work loads, Itanium executes 15% of the time and stalls 85% of the time waiting for main memory.” When the processor lacks the needed data in the on-chip cache, it has to take a long time penalty to access the off-chip DRAM. Power and cost are also improved to the extent that more can be integrated on chip.
Given the importance of memory advances and the relative ease of applying molecular electronics to memory, we may see a bifurcation in Moore’s Law, where technical advances in memory precede logic by several years. This is because molecular self-assembly approaches apply easily to regular 2D structures, like a memory array, and not to the heterogeneous interconnect of logic gates. Self-assembly of simple components does not lend itself to complex designs. (There are many more analogies to the brain that can be made here, but I will save comments about interconnect, learning and plasticity for a future post).
Weaving these brain and semi industry threads together, the potential for intelligence in artificial systems is ripe for a Renaissance. Hawkins ends his book with a call to action: “now is the time to start building cortex-like memory systems... The human brain is not even close to the limit” of possibility.
Hawkins estimates that the memory size of the human brain is 8 terabytes, which is no longer beyond the reach of commercial technology. The issue though, is not the amount of memory, but the need for massive and dynamic interconnect. I would be interested to hear from anyone with solutions to the interconnect scaling problem. Biomimicry of the synapse, from sprouting to pruning, may be the missing link for the Renaissance.
P.S. On a lighter note, here is a photo of a cortex under construction. ;-)
25 Comments:
I'd add that its 4000+ year old Vedic knowledge (and by direct extension Buddhist) that the brain is only a memory unit. Awareness is greatly increased when the brain ceases to interfere and ceases to obscure perception with its predictions and recollections. In meditation the sensations in the brain become still and your perception becomes much clearer. But logic is also known as a hindrance, only a tool used to sort perceptions. The Mind (in Buddhism thought to reside in the heart space, though not physically), in its form of pure awareness is aware of reality, but is not really situated inside reality. Which may sound far fetched, but so is wave-particle duality and non-locality and virtual states. Perception is also ultimately a hindrance, as is Consciousness (which is defined as a series of impressions that are under the false assumption that there are a stream of them, and that time "exists"). Awareness is beyond a concept of individual self and causual flow.
So if you model a computer after this, it probably will accomplish less than the average monk. I'm sorry Dave, I can't do that. You see the door doesn't inherently "exist" in the way you assume it does...
Fascinating blog, thanks for writing!
-felix
http://crucial-systems.com
By Anonymous, at 10:45 AM
very interesting.... gives new meaning to Ray Kurzweil's book Age of Spiritual Machines...
By Steve Jurvetson, at 8:49 PM
You raise many interesting issues--- and interconnectivity is certainly one of the most daunting.
I like to illustrate my impression of the size of the brain-technology divide by comparing a human brain to the Los Alamos ASCI Q supercomputer --- currently number 6 in the Nov 2004’s ranking of the worlds fastest computers. ASCI Q has 1.5e11 transistors and ~12,000 CPU with 1.25 GHz processors, 33TB RAM and 664 TB of total storage, it consumes 3MW of power and has a volume of about 1.2e7 cc; performance is in the 30 tera-operations (3e13) range.
The brain, on the other hand, has 3e10 neurons (3e14 synapses) with a neural firing rate ~1 KHz which equates to something like 300 peta-synaptic “operations” (3e17) per second (and does it at 20W in 1400cc of space).
I, too, try to avoid the brain-computer metaphor, but it is good for establishing context---so given the complexity of a neuron, I think the best part-to-part comparison is between the number of neurons (3e10) and the numbers of CPUs (1.2e4). As you allude to, once bio-plasticity is introduced, it makes the comparison a real apples-to-oranges affair.
So what’s a cogno-neuro-technologist to do? I think some of the often-overlooked differences between neuronal synapses and digital computing (e.g., that synapses are nonlinear, plastic, analog systems that encode information in their spatial-temporal distributions using non-binary logic) could certainly be useful for future technological design inspirations.
Yet, I think low-level neuro-inspiration will only get people so far in the quest for intelligent machines. In fact, I’m not even convinced a strictly bottom-up approach (i.e., understanding the neural implementation of the brain) is a necessary prerequisite for creating intelligent machines; understanding what is encoded is at least equally as important as how the information is encoded (for some good reading see David Marr).
If you haven’t come across the site before, there are some interesting government programs related to bio-computing that might pique your interest: http://www.darpa.mil/ipto/Programs/programs.htm
Please feel free to drop me a line.
Chris Furmanski
chris@furmanski.net
http://furmanski.net/
By cfur, at 3:24 PM
Steve,
Enjoy your Blog... I was wondering, what are your thoughts on the existence of God? Do you believe in God, or chance, or both?
-- WM
By Anonymous, at 5:07 PM
Hi Steve,
I played the traditional "predictions for the next year" game with friends and a discussion yesterday gave me the opportunity to publish a part of my contribution.
On first sight there is no connection between the two subjects, but there is one in my mind, and I would like to share it with you.
A short version of my post :
--
Semantic Internet (and not only Web), allowing access to Shared Documents all over the Net will happen in 2005, with two major consequences : a completely new way to distribute one's work, and the building over the next years or the Interpedia, composed of articles written by experts and shared.
My guess (non posted) is that Gxxxxx is heading that direction, using distributed generation of indexes (via desktop Gxxxxx Search) and planning to compile them in a central index to facilitate access to shared resources.
Technology is available, people are used to share documents, ways of claiming paternity exist, aggregating feeds and alerts is already common.
[full text in French here FR]
--
This could be the first build of a Net-wide common memory stock, each document described by keywords added automatically, by content analysis, or manually.
Accessible by "connected ones" which will have the possibility to create "personally" filtered packs, fitting their needs, and eventually share them back.
While reading your post the parallel between what I call the Semantic Internet and the "memory-prediction framework for intelligence" pop-up.
Despite the fact that this would be based on "classic" memory blocks, it looks much like Hawkins' model, based on invariant forms [documents], broad connectivity [the Net] and nested feedback loops [aggregation].
Seems that evolution is once more repeated ;-)
If my guesses [Google/Apple and maybe Yahoo! now working on that] and prediction [the building of Semantic Internet] are false, someone should take care to build that.
What do you think about it?
By Antoine Vekris, at 10:18 PM
Bonjour! The Google founders would agree with you. Every time I talk about Google's future with Larry Page, he argues that it will become an artificial intelligence.
I have a hard time factoring in the asynchronous aspect of data structure updates such as this. It seems relatively static and decoupled to me, but I have not been able to grasp it clearly. Maybe it will operate at a peculiar clock rate, where aggregation occurs over several seconds…. And so perhaps we would have a hard time recognizing it.
By Steve Jurvetson, at 10:30 PM
Steve,
it is quit ease to do if you split the job. I posted at Eclectic Thinkers Blog a more extended description and probably more explicit.
Czarandom, I'm looking forward to read your posts.
By Antoine Vekris, at 8:52 AM
When I write a book, it's like I write a chip, or maybe an architecture. When it's done it's done. When we go to the brain organization, it seems that all is done but at the same time all is on the way.
I'm always in trouble. The brain organization doesn't "store" something in some place, but is able to "restore" everything when I call the thing.
How the brain does it? One of the suggestion (not solutions, please) is in the dyanamic structure of our brain. Our brain - supposedly - doesn't stores anything, but dynamycally adds paths over paths like a grid, that's modified by experience day by day.
The grid of neurons someway "marked" as carriers, keep record of the importance of some simple message that may activate a neuronal connection that then reproduces what we call "memory".
Grids of relevant importance get more attenction than grids of few. Neurons react not to a single event, but only when a pull of events show to be similar to what the grid informations store as event.
Not a single fact or thought, but a chain of facts and thoughts that in the whole show to be meaningful.
Not one key, just to say, but almost seven (just to lift off a number) keys together, move the whole.
But, Steve, this is a very tangled field, indeed. Walk in a jungle.
By Franco Cumpeta, at 1:22 PM
Let me drop two flying thoughts. My apologises first if this is too basic, or simply non sense. I am intuitively trying to find an answer, I am not into this subject, not even near. Thank you.
1) When Medicine began to realize that their patients had something called psycology (psyque) and that this last was of a tremendous importance to their physical condition (soma) ...It was a HUGE step for this discipline.
The relationship I make here is: where in all this brain study, it is acknowledged the body that holds that brain? I do believe that our intelligence -though centered in the brain- it is not fostered only by its activity. The easiest example is to see how a person with nutritional problems has severe problems in thinking and sleeping.
It will be like saying: Our body it a whole intelligent organism, we are not just brains with legs.
This thought also leads me to ask if in this brain function study and its comparison with future possible AIs, the susbtrate (the "hardware") is acknowledged and how. I guess that as long as we don´t fully understand the exchanges of susbtances, which are they, at which temperatures and frecuencies it happens, how electrically charged the process is (at the brain and at the whole body) -btw, all this a study that lead us to the nanotech realms- ...As long as we don´t know all this it will be hard to find a way to a possible and useful AI as we dream it today. If the Protein´s structure already does its job with such grace, we should try to emulate that, take it as a model, shouldn´t we?
2) My second thought not related to the first one, has to do with the structure of the brain. I will talk from my own awareness of my thinking process. We can´t stop thinking. We even dream in our sleep. There is no "off" in the brain. The day it turns off you are death. This leads me to think that the brain function relies, it is founded in its constant dynamics, in its constant activity, which may vary in speed or intensity, but it is always present. This may make possible the lack of "storaging" (ROM) place for the brain, because everything is happenning -is present- at different layers of conciousness, but simultanously (RAM).
I tend to believe that people who are more intelligent have a better connection between these layers in surface and in depth, so they are able to connect more ideas and perceptions and memories at the same moment or in less time than ordinary people .
The other thing I thought regarding this dymanics, is about the circuits -electrical impulses?- that our brain makes which we experience as thoughts, feelings, impulses, fears, direct action, etc. I think that in this circuit formation of our mental programmming resides the pros and cons of our intelligence. There you had our capacity to learn, this is: to make habits, the good and the bad ones. This explains why someone can be excellent or brilliant at something and a disaster in another activity, or simply it may explain our inherent contradictions which we can live everyday with. i.e.: a biologist believing in the Bible. Because in the end, in its very nature, there is no moral, neither logic or ethics in our brains. There is only more or less pronounced circuits that allow more or less flowing through themselves. This should also explain why there is pleasure -and seek of it- in repetition and exhausting while learning something new, and they older you are the worst. The necessary repetition in music and visual arts that we feel it pleasant may also have to do with this. Our brain is confortable running its common rutine => that makes it so hard to quit with circular habits. Or makes us suffer the abstinence syndrome.
Ok, that´s for now... I put you to sleep? Hope I didn´t! Again apologises if these thoughts are too basic. Hope they are somehow useful. Adieu. |-)
By Gisela Giardino, at 3:12 AM
OldCola et. al. Thanks for the rich discussion and eclecticism. This is what makes blogging worthwhile.
Eliezer: Good to hear from you. Please tell me more about the massive serialism topology to overcome the connectivity bottleneck… If it's a bus/mux scheme, roughly how much logic overhead is there per bit of memory? If I imagine an IPv6 node per bit, then it seems high. =)
The logic/memory tradeoff also relates to memory design (e.g., the value of multi-port memories, multi-level memories, interconnect topologies).
Logic gates can be implemented in memory (as a look up table, as Xilinx does it) as can switch fabrics, but I am very curious about the fine-grained details of synaptic fan-out implementation or simulation (especially with self-assembling molecular electronics that are easier with simple 2D wiring grids).
Again, the motivation for this line of questions is the presumption that in the near future, the abstract capabilities embodied in Moore’s Law will continue for memory and stagnate for logic. (Computational power relies on a balanced mix of memory and logic, so the Kurzweil curves may continue, but with a reallocation of transistor functionality under the covers)
Gisela: Much of the back half of your comment resonates with Hawkins’ framework. I think you would really like the book.
By Steve Jurvetson, at 12:36 PM
Just thought that I'd add a couple of comments to the interesting points that have already been made. Yes, there is a processor/memory tradeoff to discuss when looking at neuro-inspired computation, but a more fundamental tradeoff was originally presented by Minsky and Papert back when they wrote Perceptrons - and to the best of my knowledge the limitation still stands. They called it the 'Time Versus Memory For Best Match Problem'.
If we go right back to the basic 'pattern matching' neural networks, they can be characterised as a device which, when presented with a novel pattern, will find the best match to that pattern from within a set of previously trained patterns. What Minsky and Papert did was look at the exact same problem from the perspective of conventional computing, which given that the current approach to building neural-network based systems relies on emulating parallel analog computers on serial digital computers is even more significant than it was at the time.
The two corner cases for implementing any matching algorithm are using a direct lookup memory and an exhaustive search. The direct lookup memory essentially uses the search inputs as address lines into a huge RAM (maximum memory) while the exhaustive search goes through every trained pattern in sequence comparing it against the novel input and then picks the one that matches (maximum time). Minsky and Papert observed that when looking for an exact match, there are a number of algorithms that can dramatically reduce the time memory product (eg hash tables, etc). However, they postulated that no such algorithms were possible for solving the best match problem - an observation which to the best of my knowledge still stands for conventional computers (let's reserve judgement on quantum computation for now!).
If we assume that the time versus memory for best match problem still stands, the only practicable solution is exhaustive search, since the scaling properties of direct memory lookup are horrendous. I actually spent 36 pages trying to convince my sceptical PhD supervisor of this 10 years ago but to no avail! See the following link if you don't believe me...
http://www.zynaptic.com/pdfs/Paper_03.pdf
So, my take is that if you want to build neuro-inspired computer systems using current technology, the best way forward is to bolt a 'calculate hamming distance' instruction on the side of a conventional processor and then plonk a whole load of them on a chip with vast quantities of memory bandwidth. The only things you need then in order to achieve the kind of functionality you mentioned in the original article is the right macro level structure and a good unsupervised, distributed learning algorithm... but I've decided to keep that bit of the secret sauce to myself. At least until I can work out how to make money out of this stuff!
Chris (chris@zynaptic.com)
By Anonymous, at 10:39 AM
Call me a pseudo-intellectual, or even uninformed, but the only way "memory-based models" can "continuously make low-level predictions in parallel across all of our senses", is if there's some sort of control and post-processing mechanism that directs the "simulation" at the "memory" level. Isn't that by definition a higher function -- i.e. an operation that would have to be overseen by non-memory neurons? I guess the answer depends on how loosely you define "memory"...
I'm no neuro-surgeon, or for that matter a circuit designer, but my guess would be that in nature there's no clear division b/w memory and CPU [for the lack of a better term] cells. Depending on the particular network/processing activity, a bunch of neurons can perform memory, or higher control functions (depending on the pathway invoked?).
Cheers,
-Georgi
ge0rgi@msn.com
By Anonymous, at 9:33 PM
This topic fascinates me as well. Have you read Society of Mind? Dated, but it crinkled my toes in a way that only maybe the 1970's flowing-hair Farrah Fawcett poster has. Except, I never took "Society of Mind" alone with me into the bathroom.
By Jim Rosenberg, at 1:42 PM
This comment has been removed by a blog administrator.
By Ben Of Denver, at 2:10 PM
(oops, sorry for the comment substitution, too many little things wrong in the previous incarnation to ignore...)
Phew, finally gotten through the post in my 'spare' time. Lots of really good thoughts.
Logic VS Memory VS Bandwidth
AI vs Brain Power
Searching
Patterns
Organization
The Nature of Intelligence
Great stuff. I think I need to find time to read some of the reccomended readings! Really interesting topic. I was disappointed in the actual state of AI when I took it in College. It seemed too wrote, too much human intelligence getting codified into the programs we created. Since then I've thought about AI in my spare time, even though my job doesn't require it.
Two replies on the nature of the beast especially caught my eye. Gisela, your post is very interesting. Our brain indeed would not be very capable without the rest of our bodies. The senses (allowing that touch includes all feed back of physical hapenings that aren't auditory, visual, smell, or taste) and give us the ability to form impressions of a given situation. Our 'mechanical' body itself allows us to act to change the situation. How would you say that sense and action contribute to actual intelligence besides providing the purpose for intelligence?
The second post topic was that of Pattern Matching. Improving pattern matching would yeild huge returns. I think there are actually strategies that we could teach computers to minimize the search space. The autonomic navigation grand challenge that Darpa hosted last spring has provided me a lot of food for thought. None of the vehicles went more than 7 of the course total 142 miles, but it's a great task to think about a computer tackling. It seems to me that most of the teams went after the brute force method, increasing the amount of data, and trying make up for perception failures by adding data and complexity. Add sensors, more sensors, more sensors than any human would ever need to navigate the course. Thinking about how we are able to succeed in the same task, I come to the conclusion that our brains are highly capable of shortcuts. We are expert pattern matchers. We are expert simplifiers. We are experts at measuring change over small periods of time. We know what to ignore by doing a quick categorization of patterns we see in our field of view.
If we are roaming our office looking for our misplaced keys, we quickly limit our search scope. We know they are not on the white board, the door, or the walls. We may look to see if they're hanging on something, but we usually remember these places, or come back to do a second search after the obvious search has failed. In terms of memory, we usually only remember the details of the situation we consider 'relevant' to the task we're engaging in. Things like what features are in the room, or what the name of the street was that we turned on. In emotional moments we can be 'hyper aware' and are able to remember many more details, but usually that is when we slow down and "smell the roses." We would totally fail to function in life if we studied everything with that level of detail.
It seems to me that we could make computers seem a lot smarter by teaching them to be aware of some of these shortcuts for pattern matching. Our brains have many advantages over computers, why do we demand that computers be more precise than we are to start out with? In terms of the Navigation example, the computer only needs to worry about obstacles. There are things that are attached to the ground or contours of the ground that could interfere with a planned path. Using a variation on fast forrier transform of a series of images and converting them into a collection of 'objects' or dangerous contours seems like the way to process input for the video stream. Now, that's no small feat, but there are tricks we should be able to use to simplify things. For instance, our eyes see maximum detail at the center of our field of view, maybe we could do something like that for computer vision so it wouldn't get bogged down in unneccessary detail.
Breaking down 'life' into streams of objects would seem to be useful for a computer in general. For general intelligence, there are a lot of individual facts, but they fall into a limited categorization. Shapes, known objects, colors, functions, etc. A box is this shape, it sits on things, it contains things, it hides things, it transports things. Walls are usually solid and block movement, doors are mounted in walls, etc. Any of you have recommendations for books that approach vision or machine intelligence in this way?
The ultimately interesting AI would be one that could learn from its creators from the ground up in a stream of consciousness manner. Instead of us teaching a computer facts, we would teach it how to learn facts and recall them. Persuit of this would really be cool. No hill climbing, expert systems, or neural nets that have to have their noses smooshed in the facts before they perform at all... If computers could break up information into useful chunks and organize them in a hierarchical manner by themselves it'd be very powerful. A computer could be better than humans at managing shifting perceptions too because they could note which facts were inferred and keep its eyes open for facts that validate a particular categorization. A computer could even index the source of inference so the computer could go back and re-analyze conclusions at its liesure. Eventually computers should be able to be at least as accurate at inference as intelligent humans. With the right framework, computers should be able to notice patterns, remember things, and make correct (possibly new) conclusions by using trustworthy correlation methods between the pieces of knowledge it has, and to me, that is intelligence.
Oook, I appologize for the long comment, I hope it's interesting and relevant enough to warrant the length.
By Ben Of Denver, at 7:52 PM
Oh, and yes. For this pattern matching, we need to come up with methods to access a lot of memory quickly, but this seems like it'd be conducive to processing on large clusters. Store parts of the data on 100 or 1000 even 10,000 independent computers and set them all to the task of finding the right pattern match on their personal slice of data. These would have to be moderately fast computers, but 10,000 computers sucking hard on pc 3300 memory would be 10^14 bytes cumulative throughput per second, right? Major head and power issues, of course, but enh, it's a supercomputer. Figuring out how to keep the processors busy as the results are tabulated would be interesting, but the Parallel processing people love that sort of challenge.
By Ben Of Denver, at 1:04 PM
major HEAT not head issues. blah.
By Ben Of Denver, at 1:06 PM
A short note: First, congratulations! Please, see the amount of comments -whose personal blog gets this much interaction?- the excellence of the posters (don´t mind me, seriously) and the length, diversity and thoughtfulness in each contribution. Second, I´d like to welcome you, Ben, I will be back with the answer on your question ASAP.
Lastly, I had to make this entry, given that it will be comment #23. It is a must. You understand... Thank you. =D
By Gisela Giardino, at 5:01 AM
So many fascinating contributions! I had to collect a lot of thoughts for this one…
Chris: In 1969, Minsky and Papert generalized the limitations of single layer “Perceptrons” to multilayered systems: "...our intuitive judgment that the extension (to multilayer systems) is sterile". After chilling the field for a whle, progress was then made in multilayer systems.
But most importantly, a multi-layer neural network is a very poor model of the cortex. It is not biomimicry in any real sense. It lacks run-time feedback loops, which are essential for auto-associative recall and for the very formation of memories from temporal input streams. (Back propagation of errors during training is different from nested loops in the topology of the network).
Do you recall the assumptions behind the time vs. memory tradeoff? Von Neumann architecture? Traditional single port memories? At first glace, it seems that the architecture of the cortex lies along a spectrum between the extremes of direct lookup and exhaustive search. Perhaps the degree of synaptic fanout is a hint as to where the balance was struck.
Georgi: I think you can make a compelling computational equivalence argument that the network topology and morphology are loci of computation. So I suspect you are right that "there’s no clear division between memory and CPU.” The CPU is not central; it is the matrix for the memories.
PureData: Yes! These complex systems will need to be evolved and grown, not designed and installed like Microsoft Office. This applies across evolutionary and developmental vectors.
The Google founders would agree with you. Every time I talk about Google's future with Larry Page, he argues that it will become an artificial intelligence.
I have a hard time factoring in the asynchronous aspect of data structure updates such as this. It seems relatively static and decoupled to me, but I have not been able to grasp it clearly. Maybe it will operate at a peculiar clock rate, where aggregation occurs over several seconds…. And so perhaps we would have a hard time recognizing it.
You might like the discussion here on the emergence of a hive mind.
Mr. Sun: I read that book many years ago, around the same time as Gödel Escher Bach. and Rumelhart’s PDP.Bennett: To your request for a book that discusses vision and the brain’s processing of “streams of objects”, I’d suggest the Hawkins’ book itself. =)
Hawkins notes that the brain is the only part of the body that has no senses itself. Everything we know and remember comes from the spatial and temporal stream of patterns from input axons to the brain. All of the senses, vision, touch, hearing, etc. are converted into similarly structured streams of pulses before they get to the neocortex.
So all of the inputs to the cortex are fundamentally similar, and all regions of the cortex perform fundamentally similar, but hierarchically structured functions. This is why people can be trained to see with a camera wired to electrodes on their tongue, and ferrets can develop just fine with their auditory and visual centers of the brain surgically swapped (so the vision center processes sound and vice versa). “In fact, your brain can’t directly know where your body ends and the world begins” (p.60).
And there is massive internal feedback to the sense-making process. “Your brain constantly makes predictions about the very fabric of the world we live in, and it does so in a parallel fashion… What we perceive – that is, how the world appears to us – does not come solely from our senses. What we perceive is a combination of what we sense and our brain’s memory-derived predictions.” (p.87) When the pattern stream is consistent with our memory-prediction engines, we don’t even notice it.
So all representations of reality, memory, and “self” are interwoven and internal abstractions, derived from a stream of patterns. I would imagine that some people incorporate more external feedback, and adjust internal representations more frequently, than others. Children do this naturally….
Ben & Alien23, the IO can contextually define the intelligence. Several robotics researchers fundamentally believe that we have to build a robot with a humanoid physical interface to the world if we ever want to create a human-like AI.
For example, a dog can detect identical twin humans by smell alone. Much of a dog's brain is devoted to nasal memory (related to their sensory input - a nose close to the ground), so even their “memories" would be difficult for us to relate to, not to mention any higher-order constructs.
Philosopher poet Hunter S. Thomspon had this all figured out. Watching a boxing match on TV, he blurted from an ether binge: “Kill the body and the head will die.”
But Morpheus reminds us of the converse: "the body can't live without the mind." ;-)
Update On Feb 8, Intel revealed the details of the Montecito processor at the ISSCC conference. They have more error-correction-code memory bits than my conservative assumption derived from an earlier SPARC chip (The ECC overhead is now over 2 bits per byte).
According to Intel, of the 1.72 billion transistors on the chip, 1.66 billion are memory and 0.06 billion are logic.
By Steve Jurvetson, at 2:56 PM
As far as Google is concerned, I think that in its current form no amount of data, number of network nodes and/or processing speed would ever enable it to become an AI system. This is simply because the traditional server/massive database technology is only as powerful as the underlying principles that govern the relational database that powers Google. Furthermore, the client-server architecture in itself is a limitation, because it ensures no interaction between the "spokes of the wheel".
We've probably already managed to build a system with greater memory and raw processing power capability than let's say a cat. Still, without an equally complex "pipeline control mechanism", a cat remains vastly more intelligent. We can see the neural pathway that's invoked when a person's thinking about the color blue, but do we really understand how it all works together? Do we really understand the principles that govern the development and complexity of interconnectivity?
Steve, fascinating read on "Quantum Computational Equivalence". It's interesting that the brain is not a quantum computer, and yet, in its relatively small size, it manages to achieve what only a quantum calculator can do and what "is literally impossible for any traditional computer, no matter how powerful"... Makes me realize how much of molecular biology, physiology and quantum/string theory I personally know little to nothing about.
-Georgi
By Anonymous, at 7:49 PM
you might look at what lightfleet corp is trying to do
By Anonymous, at 7:00 AM
Hear this one:
“If the human mind was simple enough to understand, we'd be too simple to understand it.”
- Emerson Pugh
How neat. How well put. Loved it.
Must be shared. =D
By Gisela Giardino, at 6:45 PM
Really interesting discussion. A couple more-or-less random thoughts:
1) A lot of the posts have the implicit assumption that memories are stored somewhere (maybe lots of somewheres, if memory is distributed) and retrieved as they are stored. But one of the most fascinating things about memory is that it's reconstructive -- there's a lot of hypothesis testing and detail-filling that goes on by logic, not by retrieving what happened. For example, Elizabeth Loftus' classic studies of eyewitness testimony: Whether or not people report seeing glass after witnessing a car accident has a lot to do with the question that is posed. If you ask, "how fast were the two cars going when they smashed into one another?" you get higher speed estimates than if you ask, "how fast were the two cars going when they collided".
2) Speed of processing (CPU speed?) has been known for a long time to matter quite a bit. For instance, one of the hallmarks of cognitive aging is that people slow down. It's thought (by some) that decreases in processing speed can then lead to problems with high-level processes, like reasoning skill. The way I think of it is that if you have one process trying to retrieve information because it's needed to integrate with some other information held in memory, and if the retrieval process takes longer, the information being held actively in memory has more time to decay. Once the already-held inforamtion is degraded, your ability to apply it to the information you've just retrieved will probably be less efficient.
3) I've been struggling with the question of what constutites a memory for a long time, and the previous posts about the mind-body connection have reminded me of that. For instance, if you exercise your biceps, later you can more efficiently lift (retrieve?) the weight. Does that mean that your muscles have developed a memory? And if it's not a memory, what definition of memory can we come up with that will exclude weight-lifting while allowing for the processes that our common sense tells us are really memory.
-Christy
By Anonymous, at 6:41 AM
hope to see you use flicker on this blog also
By Anonymous, at 3:40 PM
The comments are interesting. The study of eyewitness memory is my graduate study field. The 'anonymous' comment #3 as to whther a muscle stores a memory is one that I wrote about in a graduate paper in philosophy of mind - that was not well received.
I took the position that the very crux or definition of existence is memory in the broad sense implied by 'anonymous'. First we have to distinguish between at least perception, encoding, storing and retrieving as aspects of the broad term, "memory". It is recall, and only recall, that we can experience and test. The other features must be assessed by reasoning from measures and observations of recall. Once we make this approach, we can see that recall means the persistence in the presence of an occurence from the past. Is that not all that identifies existence? Is that not exactly what 'anonymous' is considering with respect to imprinting muscles with a pattern of responses?
By Anonymous, at 9:02 AM
Post a Comment
<< Home