Pattern Recognition vs True Intelligence - Francois Chollet
02:42:54 Share

Intelligence y is very specifically your ability to handle novelty, to deal with situations they've not seen before a and come up a on the fly with models that makes sense in the context of that situation. And this is actually something that you see the little of in elements. If you ask them to solve problems that significantly different from anything they ve been, you will fade the abstraction reasoning.

Propos g for short, you can think of as a kind of I Q test that can be taken by humans. It's actually easy for humans or V, I agents. And replace that you see every thought you get is novel, is difference from any other task in the dataset, also different from anything you may find online. R, G, S, designed to be resistant, minorities, ation, and all the other benchMarks can be hacked. I, member at all.

When I spoken to A I researchers, i've gone through arc and chAllenges together with them, and they are trying to look at their intrinsic there, saying, i'm looking at this problem and I I know it's got something to do with color. I know it's got something to do with counting and then they run the programme in their mind and they say one, two, three no.

that doesn't work. That doesn't work. I think in life is when IT comes to getting some idea of how uh your mind handlers system to thinking, I think it's not an effective for system one because system one is innocently, not something you have direct access to.

IT happens like unconsciously, instantly in in parts of your brain that you're not directly observing. Silence is is delivery. It's very slow, very low band. It's very introspective. But what's not mentioned here is.

France was, it's none to have you on the show. And honestly, this this means so much to me. You're my heroes.

So thank you so much. It's my pleasure to be here. And I would say you shouldn't have yours like it's I shouldn't. No, no IT makes for a disappointing expense.

Um not for me OK. I know what I can.

I can live up to to the .

expectations. Ih front warm, you've been critical of the idea of scale is all you need in A I can you tell me about that?

sure. So yes, so this I D that schedule need uh is a something that comes from uh the observation of scanning skating laws h when train depo network switches. So scaling laws are this relationship between the performance you see uh in in deep planning models， typically elements and uh how much data and compute h went into training them um and it's this solar flag, a log I make scaling uh of performance as as a functioning of train compute.

Typically it's formated and uh many people are extrapolated from that that well there's there's no limit to uh how much performance we can get out of these models. All we need uh is to scare up the compute by a few of those magnitude, right? Uh and eventually we get uh much beyond a human level performance, purely investigating compute with no change uh in architecture, with no change in train part.

And while the the major flaw here is the way you measure performance, uh in this case performance measured via exam style benchMarks, we shall effectively memorization games。 So effectively measuring hunger D L M is at memorizing the answers to the questions that you are going to test you on, not in set the exact answers but maybe uh the sort of flag uh program tempers that you need to apply to uh arrive dances. And if you're measuring a something that fundamental driven by memory and IT makes sense that that as you increase your model memory in the system, like the number of damages, demand of train data, uh and computers are just a proxy for that, you see a higher performance because you know of course, if you can memorize more, uh uh you're going to do Better at your memory game.

Um uh my talk is that these performance increase your job serving IT is actually altoona alone to intelligence. You are not really measuring intelligence because you are benchmark can be hacked purely by preparing for IT, by memorizing things in advance. If you want to benchmark intelligence， you needs a different kind of game, a game that you cannot prepare for and something like oak for instance. And I think if you look at uh, performance and oc over your time or the function of computer, you don't see this relationship in fact the highest performing models on oc uh today did not require tons of computer uh and some program sea approaches actually did not require uh any um training time compute because they we are not trained at all. They do require some h inference and computer, but it's not a very large month.

So you said that language models are interpretive database. And I i've spoken with sab arrow at the other day and he he caused them approximate retrieval systems. And many people say to me, tim, this is ridiculous that of course they're not database that they do. But I think as an intuition pump around memorization that that is what they do. And you wrote a substate blog about this yes.

memorialization is what they do. I think um the ball to have people get stuck is that when when they hear memorization, they think the olympic are just memorize uh answers to questions, digest memory in contents, right? And of course they do in ze a lot of content, a lot of knowledge and factors and song.

But that's not primarily what they do with that. Primarily memorizing is functions programs. And these programs do journalists to some extent that that can be with free journalizing. And um when you quit them, you are basically uh quarter uh a point in program space. So you can think of the them as a many form where uh each points uh and caused a program um and h of course you can you know uh interpret across as many forms to compose programs or combine programs interpolation like this, which means that you have uh an infinite number of possible programs to choose from.

And what happens with that arms is you are a training them, training his family, me, uh a very rich, very flexible models to predict the next token right um and if you add infinite uh uh memory capacity, which you could do of course just learn a kind of food capable right? But in practice uh the l one has uh some billions of parameters. So IT can not just learn to look table h for every sequence uh in a train data IT has to compress. And so it's actually learning is uh predictive functions that take a uh and and they take the form vector functions, of course, because is a curve. So the only thing you can encode with the curve is but a Victor functions um and so you're learning these vector or functions that take as input elements uh uh uh of the the the entry sequence and and output elements uh uh of what comes after that uh like for instance let's said yell am comes across and the works of shakespeare for the first time uh but the LLM has already learned model of the english language well now the the the the the text that is looking at is slightly different but is still in english language um so IT is possible to model IT by using a lot of uh uh functions that came from a learning to model english in general um and IT becomes much easier to model shakespeare by just learning a sort of style transfer function that we go from the model n you have to this a shakespeare sounding text that's kind of flag how he will end up um with things like the ability to do text 说 了 style transfer with with element right it's because IT turns out that uh IT is more impressive uh to learn style independently from content um uh based based on on the same kind of model uh l am is gona learn millions uh uh uh independent uh productive functions like this and IT can of course combine them the interpretation because the old vector or functions still not like uh discrete programs like you might imagine a python program points， ince still not lag that they actually Victor functions .

because when when you say programme, I think a lot of people think of a program as being something with conditional logic and with an ella.

that's not what they are.

It's it's almost like an an input sensitive way. You see this kind of reversal through the model and it's like a mapping.

It's an input to output mapping and that mapping is continuous and IT. IT is implement the curve.

But but we can describe that as a program.

Yes, of course they are functions. yes.

And and you said they were compositional.

yes, because, uh, these functions are vector functions. You can uh some them, for instance, uh, you can h interpret between them to produce new functions.

I I love this collider scope hypothesis. So can can you you dramatically introduced the collide to scope hypothesis.

sure. So everyone knows where to color. Scope is right? It's like this account all tube with a few bits of coloured glass in IT um and this uh this is just like a few bits of uh uh original information get uh mira and repeated and transformed and they create a this tremendous switches of complex patent.

You know it's it's beautiful. And the collie to cope hypothesis is this idea that uh the world in general and any domain in bartie lar follows the same structure that IT appears on the surface to be extremely rich and complex and, uh infinitely novel with every passing moment. But in reality, IT is made from the repetition and composition of just a few items of meaning.

And a big part of intelligence is the process of mining your expense of the world to identify bits that are repeated um and to extract them extractions unique at tomes of meaning and uh when we extract them we call them attractors and then uh as we build uh solar flag inner banks uh of such abstractions, then we can h reuse them two maxes of novel situations, of situations that appear to be extremely unique, novel on the surface, but actually they can be interpreted by composing togethers, is a uh uh reusable abstractions. That's the fundamental idea beyond intelligence. Intelligence is a cognitive mechanism that you use to adapt to novelty, to make sense of situations you've never seen.

And IT works by creating models on the fly of the new situation, by combining together existing building blocks, abstract building blocks, which remind from your past expense. And there, there are two key tricks here. One trick is the synthesis trick.

We will buy you take this building blocks and quickly assemble them, uh to form a program, a model that matches the current task on the current situation at your facing the syntheses and there's abstraction generation, which is the reverse process in which you're looking at, uh, the information you've you've got available about the world like your your expense, your perception also the model section of credit to respond to IT and you're gonna turn that distill IT into reusable abstraction tions which then store uh in your memory is so that you can use IT at the next time around. So synthesis and struction generation. And together they form intel's in my model, at least in my architecture of A G I.

So you've been prominent in the in the A I space for many, many years now. What experiences or insights LED you to develop such a clear perspective of intelligence? So earlier in york rate.

right? So if you read some of my old's blog posts or the the first edition of my deep planning book, you see that I started talking about how deep planning could do system one very well, but could not do system to. I started talking about the need for, uh, program sentences are roughly in meds, twenty, twenty sixteen.

I mean, I started right writing about a lot uh, in two thousand and seventeen. But in practice, I started forming these ideas uh, in two thousand and sixteen. And there are several things uh that lets me to IT.

I think one of the big uh catalist events was working on an automatic or improving using the planning a with Christians getting um and the the K I D was you know their improving. Is there a keen to program thesis? Um you you you're basically doing uh tree search uh with the Operators to and from the D S.

And the key idea was uh to use a deeper learning model to guide such process. And so I try to do IT uh for uh pretty long time, trying and trying lots of different ideas and uh everything I I was trying basically failed. I think I was doing much Better than random.

But if you analyzed high to was performing and height to uh producing that that ability perform Better than run them IT was just doing uh shadow bet on recognition. right? IT is not doing any kind of system tourism.

And IT seemed like a like a huge obstacle that I was just not inward to overcome by tweak the architecture or train that out anything. Um there was this pattern cognition shortcut available and this shortcut will be taken every single time you could not really unrealized a ball uh, despite programs. Uh, he had deep learning and that can as as a big insight to me because now before that points, uh, I was not like everybody else in the field.

I was under the assumption that deep planning models were a very general competing substrate that you could train. Deep planning models perform at any kind of competition that you know they were they were train complete, uh they were train complete. And um around the same time, you know twenty, twenty fifteen, twenty sixteen, there were a lots of a similar ideas fooling around like the concept of newer train machine sense. People thought and I thought this was a very promising direction that deep learning could ultimately uh replace uh a hundred and software you know um so I I subscribed to to these ideas when early on. But then uh in these experiments trying to get uh new networks to do math, I realized that actually they were fundamentally limited and they wear uh a pattern cognition engine and that if you wanted to do uh system to thinking you needed something, that you need program sentences. So that's when I am this a this realization I started talking about IT um but enjoy you know i've been thinking about intelligence and and how to create for uh quite a long time like my first sort of flag uh H H G I architecture, uh something I I I developed in back in twenty ten uh so that when I ten so uh the reason I developed this because I was already thinking about IT for a few years before I ve been I ve been uh in the field quick meditation on the .

shortcut role because I think this gets to the core of IT deep learning, learning basically like we're projecting into uci. An space on the only semantic metric is the ugly and distance. And you know, so so these models learn a spectrum of superior correlations and perhaps more experience than not experience strong.

So in real um the reason that they doing this is because of coalitions are always available to explain something no matter what you are looking at. There's always with some element of noise which you can wrongly interpret as being meaningful. And it's also because the planning models, they are they are curves are meaning that they are a continuous defense, broad surfaces in a higher demonstrate space.

Uh, and and we are we are fitting the damages of girls. And a curve is you can represent many things with a curve, but it's survey, bad substrates, ts to represent any sort of discrete competition. You can do IT you can and bed this could processing on the curve but is just not a very good idea, right? Um it's not easy to fit journalizing district programs in this format.

And this is why you went up with things like the fact that it's tremendously difficult to get deep network to learn how to sort the list or to learn how to add uh to sequences of her digits sensors, even in a state of the alt s. You have a hard time doing its negative, being trained on millions of examples, adding digits, but still they're only achieving something like seventy percent accent and new digits. So you've memorized a program to do IT.

But because this program uh is a Victor function, is emitted on a curve, IT is not of a good program. This is not very accurate. And you see this uh time and time again with any sort of um I kw to make type processing.

And just for those of you at home, a peace wise lenna function is still a curve as the people might get confused by that is they think of a curve. Is being this this smooth thing? But they have look at the wikipedia finial curve of, you're absolutely right. It's still, you mentioned the neural turing machine, which actually isn't a turing machine, of course, but IT IT behaves a little bit like one. What what do you see is the gap there, you with your networks not being turing machine.

So today I think, uh fitting, uh bonebreaker is grand dissent is a good fit for, uh, what I call value centric abstraction, which is D I D that you're gone to compare things. We are a continuous distance function which leads to the IT that you're gonna bed things and buy things. I mean, like instances of something like could be images, could be uh discrete concepts, could be words, right?

Uh that's that's going to lead to decide that you're going to edit them uh in uh uh on the manifold. So a space where two things that does similar end up close together and different and dimensions vision on your manifold automatically meaningful. Um you can do this with scarf. We we scarce because um the the sort of flag got to use IT naturally leads you don't open things, we are continuous distance.

But that's a very bad fit uh for any kind of uh type two abstraction like what I could program century obstruction where you actually interested in graphs and you're not interested in comparing graphs, a distance function, your interest in uh comparing when uh two grass are exactly identical to each other, or more precisely when a graph appears to be a sub component of logic graph. Um so for instance, as a software engineer, if i'm refectory some code, if I want to compress my code by expressing uh multiple functions as a swan function, I am not interested in how close the functions feel on the percept tua level. I am interested in whether they are exact uh program or uh uh uh in in uh may maybe in different forms, may be I need to injects instruction in that um and this is A A comparison that you have to do in a very uh explicit stand by step way. You can not just look at two piece of code and instantly say we are adding to think about IT all year, they looks similar and .

how would you described that capability is it's like a kind of epidemic risk rather than an alliance's risk verification might be Better way .

describing IT yeah vera subversion ation as good way of describing IT. And um you know uh I just said it's definite not like this sort of like tall uh continuous distance style comparison and that's true. But I think I can also be guided by perception.

It's like doing this a step by step. Exact comparison is very costly IT IT requires in all of your attention, uh, h expanded over some length of time. And so you are not gonna want to do IT a color in a in a brute force like way over many different possible candidate functions.

You wants to use your intuition to identify just A A small number of options and and these options you can try to get for exactly. So I do think we have a the ability to do approximate, uh. Distance comparisons between discrete objects.

But the the keeping in mind is that, uh this a this a fast h comparisons are not exact, right? The approximate, so they might be wrong. And I think you get a the the same type of output from.

And if you're trying to use IT for programming, they will often give you things that feel right, uh, but aren't execute right in sure. I think that's the thing to keep me in mind by using deep earning or when when you using a olympus instead, they are very good, giving you things at all directionally accurate, but not actually accurate. So if you want to use them well, you need this, uh, post factor verification step.

So when watching your children grow up, how is IT influences your thinking on intelligence .

and learning? One thing you notice when you children grow up is the fact that constructivist m uh is uh italy right that uh you they learn think uh in the active manner they try out uh and from these expenses is very deliberate expenses. The extract new skills which then they invest uh in the in the new goals.

And in china, you know you see pretty clearly that learning, learning in general, but especially and children is structured in whether we describe as a series of feedback loops where the child will uh notice something interesting, come up with an idea, said that as a gold, like, imagine you down the floor like crawling, then you notice something that looks are intriguing to you, like, hey, i'm gonna get this right. So that's your goal. And now you're entering this sort of feedback loop where you're trying to reach that goal, your doing something to us IT.

Then you get some feedback and you're evaluating right. You have this sort of flag, a plan action feedback back to plan uh loop. And if you reach the goal, then in the process you will have learned something and you will be able to ren invest that new skill uh in the in your next tender um and the way the way they said goals is always grounded in the things they already know about.

And you start not knowing much like when when you're born, you are animated by just a few reflexes. Um but when when you start forming these goals, they always come from uh from this layer that you've already master and your your building your own mind can fly layer by layer like at first to instance. Um one of your most important sensor enses is your math because you have the second reflex which is extremely important. It's something that you've born with.

It's not something that's a quiet extremely important because it's I feed right um and you would serve the things like to panama grasp reflects for grabbing things but you can actually use IT yet because uh you you are not in full control of your limbs so you can not really like grass grass things um but um when you start being more in control of your limbs you will want to grasp things in the reason the first thing that you that you try to do if grasp is you bring IT your math to suck IT because you set this goal um because IT sounded interesting uh, with respect to the things you already know how to do with, the things are already fine to be interesting, right? And once you know how to grab things, you're gonna add that to your to to your world to sort of like like inner world, and you're gone to build the next layer on top of those things. So next thing, uh, you you're learning to crawl, for instance, why do you crowd? Why, why, why you're trying to move along because you saw object that seemed interesting that you want to grab.

So you are learning to crawl to grab something. You are learning to grab to put IT in your mouth and you are not learning to put things in your mouth because it's already that um to your your sort of flat constructing yourself in this sort of flat clear your wise uh fashion. So basically everything everything you know everything you think about is built upon uh well level primitives, which are built on lower al level primitives and song and ultimately comes back to at this extremely basic to finances that h newborn dren have.

I do believe we construct, especially know Young, Young children, they construct their thoughts based on, there are some cemento expenses in the world. You you have to you, you cannot thank in vacuum, you have to construct to t out of something uh and that something is extracted from your experience, right? And the Young are you are of course the more uh grounded uh, your thoughts are they they they they related more directly to the things you're expands ing and doing in the world as you get to know older, your thoughts will gets increasingly abstract, increasingly disconnected from physically. But they are ultimately, you know built upon the physical layer is just that a the the tower players, uh and so tall that you you cannot see the ground anymore, but it's still connected.

So children see the collider scope, and the collider scope is created from abstractions in the universe. And then children over time derive abstractions from the collide, a scope and reason over them.

Yeah they they notice a bit in their experience uh on their own actions that appear to be reusable ah that appear to be useful to make sense of uh novel situations. And as you go, your billing up is uh vast libraries of visible bits and having access to them makes you really effective in making sense of institutions tions.

And then you said constructivist which is quite interesting. So do do you think children constructs different abstractions? Or do you think there's a kind of attractive towards representing the abstractions which the universe came up with?

You do different people come up with different models? Uh, to some extent probably yes. Uh, but because these models, they are ultimately extracted from the same kind of expenses and they extracted we are the same kind of process, they will end up being being that mini, would think.

I mean, you you do you do definitely see that different children follow slightly different elemental trajectories timah. They are all somewhat baLance. They are they are all roughly following the same dangers, maybe with different timing, you know.

So another interesting thing you've said is, you know, language models have near zero intelligence. And I just wondered if it's near zero. Which part of .

IT is not zero? sure. yeah. And you know people people think that it's a very proactive statement because they are using A A lamps all the time to find them very useful. H they seem to make sense to severe human like and so i'm like a near zero intelligence that that that sounds kind of shocking but the keys to understand that um intelligence either separate concept from skill, from behavior uh that you can always be skilled at something with a mossy being uh intelligent uh and intelligence is very specifically your ability to handle novelty, to deal with situations you've not seen before h and come up a on the fly with models that makes sense in the context of that situation.

And this is actually something that you see that little of in elements, uh if you ask them, uh, to solve problems that are significantly different from anything they have seen that train that they will fail. Um so that said, if you define intelligence in this way and you come up with a way to bench markets, uh uh like oc g iphone instance and you try at m like all the state of the alt ms on IT, uh they don't have zero performance, right um and so this is where the non zero part of my statement comes from so that said, you it's not entirely clear with that non their performance that ability to adapt to novel problems uh is actual intellect or whether it's a flaw of the benchmark. Maybe the benchmark was not actually producing entirely novel problems. Maybe there was very significant over between this or that question and something the that it's very difficult to control for that because the M S just memorize so much IT has seen you know primates ing entire internet plus uh tons of uh uh uh data annotations that will created specifically uh uh for for latam um and we don't know fundamentally what's in the train that so it's kind difficult to tell but IT does seem to me that lands uh are actually capable of some degree of recombination of what they know to depth, to something that is genuinely not quite seen before. Uh is just that the a degree of district binary, the jurist ation power is very weak, so very low.

Yeah this gets to the core of IT because a lot of people argue that this combination toral creativity or this kind of extrapolation does constitute novel model building. And I interpreted what you said as if we zoom out and think of the training process as well, that that obviously .

is model building. Yes, great, decent. Like fitting a curve to A A data set. D indecent is model building.

Uh, the measure of flaw there is IT is very inefficient, uh, model building, uh, IT requires to get a good model. You need a dance sampling of prevent everything. Uh h the model is going to have to do with at test time.

So the model is effectively only displayed weak jizan IT can adapt to think that does not seem before but only if they remain very close to think he has actually for and um where intelligent es into play is the ability to adapt to things that are way out of the of the distribution because the real world is not the distribution right. Everyday is new, everyday is different. But you have to do with that. Anyway.

critics will say, in, and I can, I can empathy. I mean, I I use claude sun in all of the time for my coding. I am paying for about doesn't two thousand requests a months on on course.

I'm using IT a lot. And IT appears clear voice in many cases. And they would argue, i'm sure, that well, because IT strained on so much stuff, the complex hole is, you know, enough to capture any novelty we we might need. Therefore.

what's the problem? sure. That something I hear a lot decided that, yeah, maybe novelties of variable I just need to train on everything.

Decided that, yes, they can exist, a dense sampling of everything you might ever want to do, everything you might ever want to know. So I mean, I D disagree with that because imagine you are, you are training at a lamps. Uh, ten years ago, you are trying to use them.

Now they're not know about programing languages that you're using the I uh, about all the all the library. And on that there something really gonna seem uh, much less uh, intelligent just because this this gap on knowledge, the world is changing all the time. And you could say, well, but what if you just train the model, uh, on a fresh descript h data uh, every single day I mean, sure uh you you can do that and will address small problems.

But still it's likely that at some point you will come up with, uh, problems at the action, novel problems that don't have a solution. Uh on the internet, uh and that's where you need intelligence, right? And i'm actually quite confident that a at some point in the future may be in the near future, uh, we will be able to create a system that can actually uh address a this issue of take quit nose and recombines uh in in truly original ways to address completely new problems.

Once we have a system like this, uh, we can start uh developing new science. For instance, like um one of the things you can then do with other lamps today, uh to develop uh uh new science, right, because the best they can do is uh speed speed back to use some interpretation of something they're read uh online, right. Um they're not they are not gonna uh set you on the way to some grand discovery OK .

the that was advocate on that. I agree that the creativity in the reasoning comes from the prompt. And because we enter promotional es the models we we may credit the the role of the human, but still inside that addressable space in the LLM whether human supervisors, i'm sure we can creatively explore the complex hole of what is known, perhaps not create new things.

sure. Uh, you can do that and that's the process, you know as as you say, to be driven by a you to human because you are gonna the judge for its interesting this is it's nonsense and we add this sort of exchange verification. Uh it's it's difficult to make good user problems.

You know that that I think that should be the thing. Always keep in mind when using a lamps is that they are very good at making uh, useful suggestions. But you should never blindly trust suggestions they make uh especially if it's uh something like cold right you should always uh use this as a starting point but verify I can make sure it's actually correct. Elms are vague that's putting you in the right direction, but they're not very good at at watching exactly correct answer.

And that that's why if we look at all of the successful implementations of the lens or applications, are you always have a human supervisor .

in the look yes or IT could also be a an next fi like sometimes the division ation process, something that you can delegate to a symbol system. So so there .

was a great segway for intelligence. Now fans of the show will know yana, can I made about eight hours of content on your measure of intelligence paper back in the day we we poured for. And it's it's fascinating. But could you just briefly introduce IT now just to give a fresh er sure.

So my definition of intelligence is uh skin acquisition efficiency. So it's decided that intelligence is separate from skill. So if you have bench mark just measures uh the skill of the eye at something IT is not a benchmark of intelligence. IT is always possible to score high without actually displaying any intelligence sweats ever a if you want to actually measure intelligence, you have to look at how efficiently the system uh acquires new skills given uh, a limited amount of data. So you have to control in particular for the dinner at the system is access to um and uh which usually takes two forms. You know IT can take the form of prior like the information at the system only has access to before is looking at uh your benchmark and the experience which is demonian information that the system will extract from the dusk, the mensmore that you giving to IT um and so if you control for prius, you control for expense measure skill, then you have some measure of skill efficiency information efficiently of uh uh acquisition of high performance on the noble tsk um in that uh uh something that i've tried to turn into a concrete benchmark and that was the archives I data set.

Just a Green point on is one of the potential issues with the measure intelligence that is non computable because we can't represent the domain of all possible tasks.

Sure um so my in in in the paper I D formalization uh of my measure of intelligence um and IT is non computable um its purpose is not to be used as A A practical tool like you're not actually wants to run this equation uh on on a system and get a number out of IT uh IT is a formalism uh that's useful to think about the promise intelligence ence precisely right it's it's a cogito device. It's not a practical device course .

that there was this wonderful figure which we are shop on the screen now, which you describe the intelligence system as being a thing which produces skill programs while adapting to novelty. But one one thing I was wondering though is you you're talking about IT is a kind of meta learning prior. And do humans come with the meta learning prior baked in? Or is that something we also learn? And should IT be the same for a system?

Yeah so that's that's very important question. Um so intellect is uh it's not skill. It's a kind of meta skill.

IT is the skill for which require new skills? And is this meta scale also something that is required to expense? Or is this something that you born with that that that comes a hot cooked in your brain? So by evolution presently, um I think the answer is that IT is both I think you are born intelligence so you are born with a sacques tion mechanism.

But this skin acquisition mechanism does not uh Operate the vacuum actually needs uh so IT IT IT it's it's composed of two bits right that's the sentences is engine which I take text look at H A new situation and new talks can will try to combine h existing parts, existing abstractions h into a model for the desk for the domain um and does the uh uh the abstraction energy bit which looks at the models to have produced so far, looks information available and we try uh to produce reusable abstractions to be to be added back uh to the library that can be uh used by the certification the next time around and um the this library of course is acquire through expense. And the Better your library of abstraction becomes, uh the more effective you are uh at ten thesis, the more effective you are at acquiring uh new skies efficiently. Uh so I believe that this uh sort of flag macro o level architecture of intelligence is something at john born with.

But as you use IT, scratch your lifetime you are getting Better at IT. You are polishing IT so you are not acquiring intelligence uh, as a skilled from scratch but you are polishing IT uh another mechanism switch I think you you you are politically um the synthetic mechanism is probably incorporating learned components so that syntheses is itself synthesis from existing abstractions is itself a skill. And you are getting Better at IT as you use IT. So I think, for instance, h fifteen year old is gonna get Better is gonna Better as the acquisition than atten owned? This is really interesting .

because in a way, you're combining rationalism, nativist within practicing because I think you are saying that they raised the creation of the oo skills programs that are not just compositions of the fundamental ones. But the broader question as well is we do this library learning so children develop that they they fitness, they refine, they build these abstractions. And surely there must be some trade off with complexities ation, because you don't want the library to be too big.

No right then you can do search IT anymore.

Is some kind of um pruning or does IT is is that the reason why our cognitive development seems to kind of platos other certain point?

Um that's quite possible um you know that that's actually via deep question. So very practical I think uh to building A G I so your G I is going have this library for use of all primitives. Do you want to expand the size? The library? Definitely of all, you wants to carpet, uh, at some number like you want at most one million programs in need of something like that. So clearly, our ability to efficiently acquire new skills or intelligence uh, does not uh improve of our lifetime in a in unbounded fashion.

Uh IT seems to peak uh relatively early on I think is actually uh a trade of here, which is that your uh rob brain power like fun since the the lot of information that you can integrate in your mind at any given point, a tree down as you age inevitably um but the quality of the abstractions that you uh uh work with and also your intuition for how to combine them. So the the learn confidence of the sense engine did you get polished over your time? Did you get Better over time? So we have this uh kind of factor that makes you smaller and this factor that makes you dare um you know and Pearly.

I think intentions probably peaks uh, in your early twenties. That's when that's when you the most the most efficient in acquiring new skills um but then again, you know independent uh I think uh uh higher level uh cognition uh pegs probably new early but there are are things that you should be learning a earlier than that right anything so you know I am mention like cognition builds layer by layer. Each layer is up the previous one, uh, the lower layers in the stack, the Crystalized, the setting storm activity, uh uh early uh, before fifteen typically. So if he wants to acquire any kind of scale that deals with a low levels instrument or primitives like you want to get pretty good at playing an instrument, you want to get really good at singing, you want to acquire and native accent in some language you should do IT before you're .

fifteen typically. Yes, me on on the the abstractions, you could argue that that it's it's kind of limited by a computational bound or you could argue that IT is just converging towards universal abstractions. But I want you to come anywhere you just said, personally, I think knowledge is very important.

So I i've spent years doing this thing with key to dogar, who's one of the smartest people like I know in the world. IT is P H D M I T. And he taught me how to be smart, just the way he thinks about things he has reprogrammed in my brain. And I would much rather be like this.

then go back to my early, twice, Better abstractions.

much Better abstractions. And but then again, I can give counter example. I spoken with them.

I I don't want to mention any names. But sometimes professors who learn too much of their knowledge are not their fluid intelligence. They can seem quite entrenched. And so too much now ledge and not enough fluid intelligence can be a bad thing as well. There seems to be some kind of optimal baLance.

yeah. So IT depends whether um you're relying on IT depends on whether you believe you already have the answers to the questions or whether you believe you you have templates that you can use to get the answers are gaining Better templates for a problem solving or for for generic earning。 Um that that makes you more intelligent.

That's one of the uh points of education. Like if you on math, you on physics programming, now you have all this uh these meet level time plates for problem solving that make you more effective problem solving that even make you more effective at learning. I think at twenty I was much more effective uh both in the in the in the the methods I was using um in in my approach at language learning. Then we'd have been at uh twelve though I twelve had uh we know more more brand plasticity and more more memory IT was easier to retain things uh but I did not have the right uh tool sets promotion and that two set is very much required. Um if you think you already have all the answers, then you're not gonna uh looking to create anything new or looking for a new information and maybe that the the beautiful that the uh some intellectuals fall into, uh they they think they've got everything figured out so they don't need to to search any further. Uh but instead, if you'll just uh carefully collecting accurate um ways to solve problems or like interesting ideas and you're not not quite I are gonna them yet but the sound uh useful to some intriguing and then your face with something new york to look into your library， look for for the the best saw the flag uh thing to connect to uh that's how you get insights like if you uh if you keep all these things in mind and then you come across something new instead of ignoring IT because you already know everything or you think you know everything, you're gonna try to connect IT with this sort of flag uh uh things in your monitor waiting for the click you know uh and then the how you gets uh uh big eura moments you know yes.

The ten plates become activated by, I can give an example xy, with the measure of intelligence paper. I spent week studying that paper, I read IT so carefully and so deeply, and I remember there were a lot of ideas and IT that I struggled with. And now I could read IT, I could just flick through IT, and I just got IT, and actually is the same with many other papers because you learn these abstractions on emblem ity. We're always focused on the abstractions, but maybe there's a cost to that because i'm just a cognitive pathway. My brain is just lighting up and then and I understand that, but maybe there's something else i'm missing.

Sure I think you know by a sort of like abstracting ing away uh the details you I want to focus on the bigger picture time um and and then you kind of find something new at a higher level yeah you don't get stuck in the details.

So at at the end of the measure intelligence paper as IT is from ninety nine, right you you introduced the arc chAllenge, the abstraction and and reasoning corpus. Can I call you .

bring up IT? Sure so yeah from from twenty nineteen uh the abstraction reasoning coop s uh it's uh a did set a benchmark that tries to capture uh the measure of intelligence that line in the paper. So um it's basically an I Q test for machines, but it's also intended to be easy for humans.

It's a set of tasks, uh the the reasoning tasks so each task you get uh a couple electrically two to four demonstration examples which are the commission of uh an input image in an output image and the input image is uh basically reed of colors. There are pretty small grades to be like from five five by five to thirty by thirty, thirty by thirty, largest, uh, into, you think some patterns in this input grade. And then, uh, you're told that IT maps to a certain output grade with some other pattern.

And so your job is to figure out what is the transformation, what is the program that goes from input to output. And you get, uh, a few pass input output path like this to learn this program on the fly. And then you are given a brand new input.

great. And you must show that you understood the program by producing yourself the corresponding output. great. And um it's pretty easy for humans uh for instance, the the so the the delicacy is split into different subset there. There's a public training subset which is generally easier.

It's intended to demonstrate uh to sort of course not as prize the the the tasks are built on. So cal knowledge is another important concept here. Um I mentioned the the the the grades feature patterns.

Well this baLance must be referring to something you know um and um in in order to build anything in in building blocks. So these building blocks are cool knowledge, which are uh solar flags, is knowledge buyers that all humans are, are expected to have messed by edge roughly four. So they are gonna be things like object ness, like quite is object, basic geometry, semey rotations and so on.

Basic apology, like things being connected. Um uh uh agent is well like go directed ness. So uh just uh are these very simple core knowledge systems and everything uh in the R G I tasks is built upon is atom of knowledge right um and uh so the the training subset is just intended to demonstrates what core knowledge looks like in case you want to apply shading approach and instead of heart good in core knowledge, you want to learning from from the data. Then there's a uh public validation subset which is intended to be as difficult uh as the private uh a private test set ah so it's intended for you to test your solutions and see what's core you get uh and then as the the private test, which is what to actually evaluates, uh uh the the competition on a on co and uh it's very easy for humans because we hand the private test set verified by two people. Uh and uh each one scored ninety seven to ninety eight percent so the only two hundred dollars in the british is week midst day actually solved, uh, with no prior exposure, nine seven to ninety eight tasks out of one hundred and together they get to one hundred right so the tasks that a uh h did not solve uh actually the at the no novel up um so that shows that if you're a small human, you should be able to do pretty much every every every task in the asset uh and IT IT turns out uh this delay set is tremendously difficult for A I systems um and so I release this in the twin eighteen today instead of the art uh was actually achieved uh uh uh earlier this morning is forty six percent right?

Yes, nice one jacket team. Yes, made mohamed a jack and Michael. Congratulations.

got yeah congrats. Um so uh so oh by the way, uh there's an actually uh an approach that's not public but that has a proof existence uh which should do uh forty thousand nine percent at least forty nine percent is uh what you get if you merely ensemble every entry that was made in the twenty twenty indication of the competition.

Wow, why has nobody done that then when .

it's not exactly an apples to apples, right? Because we are talking about hundreds of submissions. Uh each submissions was using uh some slightly different tweak on brute force program search. Uh, but you have hundreds of them in each one was consuming some number of hours of compute. So even if you had or do notebooks for all these uh h submissions, uh and you put them into one a maga notebook, IT will actually take too long to run IT in the competition right? So in a way you are uh by assembling the missions, you are in a way um scanning up brute force program search uh to more compute uh and and and you're getting uh Better results, you know in the limits. Uh if you had the infinite compute, you should be able to solve a pure of force program such right um IT is definitely possible to produce a domain specific languages that describe uh ultra solutions in the activity, concise manner in a manner so concise ed that you will never need more than like forty uh different translations to express uh a solution program um and uh in um well uh just uh uh finding uh every possible uh program that's a four Operation deep uh the idea of hundred, if you had a infante computer, you could definitely do that right.

Well, does an interesting discussion point on that? I think I raise this with ryan and jack, which is that even if you did have an infinite amount of computation, it's there's there's still a selection problem because you you could select based .

on complexity for something is coming easy because you can simply so uh for for lets you have compute. So for each program, you get quite technically, you get an infinite number of matches, right? Uh, but let's say really quickly, you get like ten uh you can simply pick uh, the simplest one like to short this one.

But is the simplest one a good heroism?

Uh, empirically, that seems to be outcomes. Razer IT seems to work in practice because the .

other potential weaknesses and you mentioned the Elizabeth silken folks at home, you should read she's from harvard. He is a professor of psychology. And know you came up with those, but those corn knowledge prize. But I think you are coming at this very much from the psychology school of thought, which is that we should understand the psychology of the human mind and build A I around that. Is that fair?

yeah. So i'm A A little bit cautious about the I D. That A I should try to emulate human cognition. I think we don't really understand enough about human mind, uh, for that understanding to be a useful guide when IT comes to creating A I so I my own ideas, but how uh how to hang integration ons might work and and how to create some software version of IT, but it's only partially derived from introspections. And and looking at people .

in the reason I said that might be a potential weaknesses, let's say we select the lowest complexity program, we have an infinite man of computation. We we do the program synthesis. And then we assume that because all of the generalization space would be in the kind of firm compositional closure of the prize that we start with, then IT will work. Yes, but but that is an assumption.

sure, but it's a reasonable assumption. You could also train a system to judge uh, with the given program is likely to journalists or not IT would use length, uh uh, uh on on the D, S, C, S, one of its features, but not the only feature.

One of the other really important things about the art chAllenge is tasked diversity. And the reason we need task diversity, I think I understand correctly, there are about nine hundred tasks in in the original arc chAllenge. Now you spoke about developer aware journalizing. What is IT is .

so important, right? So develop well janizary uh, is decided that when if journalizing is the ability to adapt. Two things that are different from the things you've expands before um then that kind of matters. What a frame of reference are taking, are you taking the frame reference of, uh, the agent does not matter if this agent is able to adapt to think that IT is not in person experience before or do you take the prefer ference of the developer of the agent 啊？ Are you trying to get to death? Two things that the develop of the system could not have anticipated um and I think the the correct from a is the front of the developer because otherwise a when when you up with is the developer is gonna a build into the system either of the A A hot codding of the retraining um the right kind of uh models and data so that the engine is gonna keep of performing very well but without actually uh demonstrating any kind of jizan just by leveraging the prior knowledge uh that uh that that is built uh .

into IT the current arc benchmark. I just wanted that if you could comment on on its weaknesses, but just to decide a couple of examples. Mellin mellin nitches l put a piece out saying that IT should be A A moving benchmark. And deli George put an interesting piece out saying that IT might be perceptually entangled in in a way that that we might not want. So what are your reflections on the potential weaknesses of IT?

Sure are. I mean, all I is the first attempt at capturing my measure of intelligence. H it's pretty crude attempt because of course, you know i'm technically limited in can produce and uh IT has, of course, uh pretty prey strong limitations. So I think the first limitation is that um IT might be falling short of its goals in terms of how much to verity there isn't to IT uh uh how much uh novelty. So some tasks in uh uh version one of A G I because but we can be version too as well.

So some tasks actually close to each other that some more um and they might also be very close to things that exist on line, some of them and which might be actually one of the reasons why you see uh x uh uh uh able to solve some percentage of all maybe actually doing this because they've seen similar things uh in the train data so I think there's the main flower um and um so yeah so many enemy should mentioned you know this a benchmark this should be a moving benchmark. Should completely agree, uh I think you ultimately, uh to measure intelligence you gonna want not, uh uh static dataset. You you want A A task generation process um and you're gonna ask you for a new tasks is gonna gonna capable of giving you something as very unique, very different, uh handcrafted just for you.

It's going to give IT to you and then um IT might try, for instance, to measure uh how data efficient you are in solving the task. So it's it's first gona give you maybe one of two examples gonna chAllenge you to figure that out um and if you cannot, then maybe you can give you a couple more and then a couple more and that way. So the reason why sometimes this would be interesting is that you can start benchmarking a approaches that are very law intelligence like for instance, a good fitting vega on the scent, uh, technically go fitting vega and is a kind of program in, so you should be able to write on arc.

Uh, the main reason why you cannot is because for each task can only have a couple examples and and the space is not interpreted, so IT doesn't really work, fit doesn't really work. But if for each desk you had one thousand examples, for instance, IT could be considerable that you could treat a curve and that will journey to to novel inputs. Um well, if you have this dynamic task generation and example generation system, then you can start the smoking uh uh techniques like this and IT would be interesting because then you can start uh grading uh, on the same scale fitting a transformer.

I got a great and the send versus program such bit force program program search planning guide program a song and then you can start seeing very conflicting what IT means uh to be more intelligent, but IT means to be more uh data efficient in your ability to produce generalization and the other thing that you can start creating uh, when you have uh uh this sort of dynamic h benchmarking eran processes, I can start greeting how much generalization power a different systems have 啊。 So you can you can measure how date a efficient 啊， your your sync, your model sentences processes, but also so how much and generalization power the output model has because you can uh, chAllenge test taker with different inputs that will be more less difficult. So you you start at the lowest levels by demonstrating a task with very few examples and let the stance very, uh, very simple, uh, test inputs, uh, as you go for that, you can add more examples, you can refine the constraint of the problem, but you also can send the a much more disco, uh, examples of the problem, uh, to kind of test how far I can generalize how complex uh, the models can produce, can bean.

I love this idea of of a generative arc and and I ultimately .

arc would be uh a generated adventure case and and I guess .

that is similar to the way things work in the world. So there's a generate function of the universe. IT produces the collider scope and we go backwards from the collider scope to the generate function. But knowing this is the thing like we in this intelligence process, we need to know what the prize are. And the prize must be either fundamental or reducable from the fundamental prize that where they were the first place.

That's right. And you know I think that the big pittman uh to avoid here is and that's actually the reason why um I did not uh release a one as uh generated even more. This was by the way, uh the first direction uh I investigated when I was uh h trying to come up with the thing that eventually became OK.

Um I was thinking that I would I would create program is benchmark. Where are the the the test examples would be created by some kind of master program um and I decided many different directions um things like this. I see the automatic like for in since you you are given the output of automatic need to reverse engineering are the rules that produce that sort of thing um and ultimately so I did not go with that for some reason. So one reason is that I wanted the tasks to be easy, intuitive for humans and that's actually divulge to chief in this way.

And I also wanted to avoid formalizing uh too much uh of the core knowledge because uh any uh formal formulation of cornwall dge might be losing something, might be missing something important that you cannot really puts into uh into words but that that this is there and uh also because and that's very important if you just write down the one master program and let you generate that said then the complexity of the tasks in your day city is fundamental, limited by the complexity of domestic program and so as someone trying to solve the benchmark, the only thing I have to do is reverse engineer the master program and then I can use IT find sense to generate uh infinite many tasks that I can fit h could fit a curve to uh or I just uh hot good system that already understand don't already understands how this uh a master genetic function behaves and can not spit right so we can hug the benchmark um and that's why ultimately ended up with this model where every task uh in okwu is actually handcrafted by me in this case and I I think you know that touching on on something that a um is is a settle but very important, which is that i'm be beautiful in the A D that the solution. To the problem of intelligence must be coevolved with uh the chAllenge. The benchmark, like the benchmark should be uh a tool that points um researchers in the right direction that that is asking the right questions.

But to ask these questions, a that is in itself that is a complex problem. So I think if you if you will keep the world of coming up with a master program that generates a test of intelligence that is, uh, reach enough complex, enough novel, enough interesting, enough to be true test of intelligence. Uh, coming up with that program is as hard as coming up with A G I IT is, in fact the same kind of thing. You basically needed G R to create, create a, uh, the chAllenge that, uh, A G R is a solution to, right?

How explainable should these programs be? I mean, as an example, you could explain to me the reason why you got a coffee this morning or something like and I would understand the agi presumably would be able to build models for things that we don't understand, like economics of financial markets or something like that. I would be incredible, uh, mess. So how could that work?

Well, yes. So, uh, E, G, I would be capable of watching a new problem and your task and your domain, uh, very quickly and vay efficiently from very till da coming up with a model of that thing. And uh, that model should be predictive.

So I should be able to anticipate the evolution of a of the system is looking at in the future. And I think IT should also be, uh, cosine. So you should you should be able to use IT uh to plan to our scores.

Like you can imagine, I have this model of the economy first. I want to get IT a towards this state. Here are the uh, interventions I can make that we actually cauSally lead a to to to desire state.

So IT should be a uh a proactive model, a causal model that can use to sort of like simulate the view of of the system. Um and I think that action makes its inherently interpretation. Uh, you will need to explain how the model works.

You can you can just show IT in action. So one example is, let's say we are looking at arc, we are not looking at the economy. We're looking at a dusk h in A G I. Uh, currently, most of the uh program since approaches, they are looking for a input to output transformation programs. And if you're not reading to content, so the program then one way you can interpret them is just running them h on on the test. Important thing uh which you get I think um the kind of model and an actual uh, in G I would produce in this case, they would not just be input to output transactions that would explain the contents of the dusk so there would be programmed that you could use, for instance, um to produce new instances of the task right or even to go from output to input, uh when when applicable is not just going from input to output. Uh, in such a kind of program is extremely interpretation because you can just uh uh ask for new examples and then look at them.

right OK so IT so I can imagine there might be some kind of mediated interface we IT does in capsule tion. You know, we understand the interface, but maybe we should think about the other ways.

So when I spoken to A I research, as i've gone through ac and chAllenges together with them and they are trying to look at their intrusive, so they are saying, i'm looking at this problem and I I know it's got something to do with color, I know it's got something to do with counting and and then they run the programme in their mind and they say, one, two, three, no, that doesn't work. That doesn't work. And and then they try and formalize that into some kind of an approach. Do you think that the way we introspect is a useful way to build a solution for the our child?

I think so I think introspection is very effective when IT comes to, uh, getting some idea of how uh your mind handles system to uh thinking. I think it's not the effective for system one because system one is innocently, not something you have direct access to IT happens like unconsciously, instantly, uh in in part of your brain that you're not directly observing ah ah ah consciousness. But system to is not like that system to the deliberate.

It's very slow, very low benefits. There's on leave. You know a few things happening to any any even time. It's it's very introspective. So I think you know what you're describing, deciding that you're looking at a new task, you're trying to describe IT uh, the asset of properties in your mind uh and then your coming up with uh a small number of different typos thesis about uh, what could be some programs that match as is uh, this descriptive constraints and then you're trying to execute them in your mind to check that your intuition contact I mean that's can I called uh system to think and right um I think that's basically how uh program sentences works in the brain but it's not uh mentioned here is all the system one fouts that that are in supports of this system to thinking.

I'm really a big bet ever in the fact that no clean ative process in the human mind is pure system one or pure system to everything is a mix of both. So even when you're doing things that seem to be extremely reasoning heavy, like sorting arc or doing math or playing chess or something um does actually a ton of pattern cognition and intuition going on, you're just not noticing IT right um and he takes the form for incense. Um the fact that um you're only looking at maybe two to four different possible I partisans for your arc ask in reality the space of potential programs uh is is immense.

There's like hundreds of thousands of possible programs you can be looking at but no you're only looking at like two or three uh and what's doing this reduction is uh your intuition right? Uh h pattern cognition eighty system one and I think that the reverse is also true even when are you're looking at currencies processes that seem to be extremely system one uh, like perception for instance. Um there's quite bit of system to elements when, uh, I think perception for stance is very, very compositional. It's not pure input to output a matching the way deplaning model would do IT. That's actually quite a bit of generalizations yet composition that happens that is actually system too.

I really great that there some strange and tanglement between the two systems. We know there is one task where color certainly had something to do of IT select. You can almost visualize IT as a secor query, you know, grouped by the colors.

Select count order in descending order. Skip one, take three, that that kind of thing. And it's similar to reduction in the sense that there's this perceptual influence happening to this set of hypotheses. And and then at some point on doing some post tog verification really does seem like system too, but that but the whole thing seems to work together in a symphony.

yes. And they they they are so in terminal that may be saying um that we're looking at system one plus stem to system one verse system. Maybe that's the wrong framing.

Maybe what we are looking we looking for that see a different kind of of data structure, ups or substrates that ended ze cognition that is inherently both system one and system two. Um but yeah, what you're doing in your mind, as you described, is basically program sentences. But that program sentences very, very heavily guided by persecution primitives and just by intuition about what you feel like, wait, what you feel might be the correct solution.

So when we implement programing synthesis in a computer that mean we could just do a knife ve greedy brute force search and then we have this combinatorial explosion. Tell me about .

that right? Um the primary obstacle let's run into if you're doing problem in theses. So problem is that the very high level it's you have uh uh a language.

So typically it's domain specific because that's a shortcut. So it's not like a language like byland, it's a language to be more specialized than that. Um and you have a bunch of functions in this language and you use them to create programs. Program is basically just a composition of these functions uh into something look like in the in case of artists specially gonna a program that takes as inputs um uh an input grade and produces a question uh output grade and um the way you do problem great and you try a bunch of compositions of these sanctions and for each each one uh each program you're going the running in practice so run its uh on on the target input. Look at the crossword output and check with the net output is I put you expected uh and you you do that across all the examples that available uh across all all the programs that you can come up to and then you look at which are the programs that uh actually match actually produced to correct outputs across all the examples, right uh and maybe have one one such program that so match and you have ten and then you you must make a selection. You try to guess which one is more likely to journeys and big is going to be a shorter one, but the huge a bottle neck that you face is that um the size of program space like the number of programs, you have to look at gross command with the number of billing blocks in the but also with the size of the program.

所以 you are looking for a program that involve， for instance, forty different function course, you're looking at a very launch space so you could not possibly a iterate of every individual element of that space。 Uh so that's the community or explosion bottle onic and uh humans clearly do not suffer from this problem like you you describe this a uh introspective process when you looking at an arctic and you're only executing a very small number of programs step by step and you're only really executing them to they fine that they actually correct uh you apparently rely on h an extremely powerful kind of intuition um that is not entirely reliable, which is why you still have to perform this one step. He does not give you the exact right answer.

Um I believe with the lambs are doing in action did some kind of community process. It's it's Better matching, right? It's tuition as you still have to valve, but it's directionally correct is doing a really, really good job um at shifting through bright much this are almost infinite space of programs and reducing IT to just a few few minutes and uh I think that's actually uh do really hot about in cognition as a this reduction process.

So there is some interesting approaches to work. So I spoke to jack o and run Green black. And then there's there's the dream code type approach. Maybe we should start with dream coder because, you know, i'm tenn band's group at M I T. You Kevin Alice was the author of the dream coder paper and and he's actually working with a to virus building a lab called basis.

I spoke with them the other day and they are very much focused on the arc chAllenge and they're are implementing a lot of its work on the art chAllenge, which is which is really cool. But I guess like the the elephant in the room is that dream coder and please introduced what that is. It's a really elegant, beautiful approach to art. But unfortunately, IT doesn't work very well yet.

right? So it's been a voice inside of the paper. But my recollection of dream color is that it's a program and this technique that tries to create a bank of reusable primitives, uh that this is actually uh developing kind like as uh in is used to to sow new desks。 And I think that's a fundamental right idea. And it's probably on the system in which i've seen a sydney idea in action, this idea of abstraction generation that you're going na use your expense your your problem solving expense to try to abstract away functions, the inputs h in your for for reuse later h or so remember uh wake, sleep, uh cycle the thing that was to um train uh so the the synthesis company that they had uh leveraged to planning and they were training the deep penning model yet the the wake sleep sitting. Can you can you correct me?

Yes so they had A A neal network generate model for programs and then they had a sleep phase where they would retrain the generated model and something called an abstraction and sleep, where they would kind of combined together programs that work very well and .

discard ones that are used well. You know that of yeah yeah generation OK. I see intelligence is having two critical components, uh, senses where you are taking your existing building rocks and assembling them, composing them together to create uh, a program that matches the situation attempt right? And then there's abstraction generation where you're looking back on the models you generated uh and or or just your your your the day you got about the world and you're trying to mind IT uh to extract reasonable billion blocks that you're sending to your memory, uh, where you can reuse them the next time around. And yeah and dream coder was actually trying to implement uh these two confidence, which I think is really uh uh the the right direction is very promising.

So what about jack or what do you think of his solution? And and that's the minds they I group .

on on the leader, right? So what they're doing is basically they're doing elem. So IT say it's good, good. A model is based on on five on the five you are they are free training on a launch code and math uh did is that because a banty types which you know on its own and interesting finding um and uh then they are further untuned on millions of generated OK like tasks to the producing pramathanath uh lots of tasks that look like ask and they are functioning a the model on IT when when I say functioning so they're basically for each desk uh they are cocooned ing uh the description reducing IT to a sequence of tokens so that's that's actually preying uh feeding that into the area and they expecting to produce uh the are put great in toga ized form and then then they could that my cat um and um so just to sit up by I described on its own as IT turns out does not perform very well IT doesn't like a few percent but they added uh really powerful twist which is that uh they're doing test time functioning ing so um taking there they are retrained uh at alarm and uh at inference time on each new task they're producing a fitness e version of the yelm so then doing that by uh producing violence uh of the desk a by applying a bunch of fundamental the hot ware formations basically um and they're turning that into a so if like mini train that said the finching on the trendy set and then there uh applying that find train model uh on the test important and produce a test output um and if you think about IT so just this uh to stand fashioning trick is actually getting the model from a performance like small percentage sold um to as you know over over forty percent uh which is very impressive so if you zoom out by a lot, I think what they are doing is not that different.

Uh from programme search is basically uh uh at a different points on the spectrum so you can think of programs such as a as a spectrum with two axis. One access is like the rich ness and complexity of your D S L, your bank of reusable building rocks. And the other access is the rich ness and complexity of the ways that you recombine as this building blocks and um this quite program search type going to Operate over there a small D S L I D S L which may be hundred two five hundred uh primitive functions in its uh but it's gonna recombine them in vay complex ways uh to get programs that may have depth twenty for instance um and what jacot is doing basically turning is and into and a little base of reusable Victor functions and has millions of IT so it's very very broad very launch the decide in the way and then this time functioning is using a great in the sense to recombine these primitives into program um in by the way to find that you have this huge uh performance jump from not using teston fighting to using teston fighting really highlights empirically the fact that recombination program search is a critical component of intelligence.

If you're just doing a static inference, you're not doing any any sort of recombination oh if you're doing IT， IT must be um uh some form in context learning. So basically uh using this uh a memorized recommended program um if if if if you're only doing that conference, you basically do not display much intelligent all uh if you are doing a recombination vats and fortune, then you are starting to implement the synthesis s components h of intelligence that I described. And the problem is that quite decent is very weak. They did an inefficient way of doing senate IT is in fact a wrong fight time.

Uh and so what you get is that uh the result programs, uh A A very shadow, a depths for combination, right? So on the on the programs in this spectrum, a the minds eyes solution is uh this point where uh they're really maxing out on the richness of the D S N access but the vivi law on the depth of a combination access was uh discrete program search as is usually implemented is uh on on the complete other side of spectrum where have very very small, very consider but very sophisticated recombination right um in intuitively my guess is that with maximum intelligence uh uh special is that it's not that even end of the spectrum it's somewhere in between. You have access to a very large marine rich uh bank of abstractions of of ideas and patterns of thought uh but you also capable of free combining them on the fly.

Uh to ever mean full degree you're not doing uh test on fighting ing in your brain when when you come up with know what ideas you're not doing great understand at all. You are doing some form uh of discrete program such uh but you're doing IT on top of this very, very rich uh bank of primitives and that enables you to solve any other problem. And pretty much in seconds I .

member reading your deep planning with patent back years ago, you were talking about the paris of fine training. You have to have the learning rate quite low because you might damage those representations in the base model. And when I spoke with jack, he said that i'm not sure how much, as I say publicly, but he encoded the defined tuning in a kind of language which would reinforce the existing manifold of of the model. So you know, who is kind of like saying, I want to use that as a foundation model by transforming the descriptions in a way that that reinforces IT and and also the the active inferences thing is not active inference from a Christinia point of view, but the test time inference that is moving away from what earlier, which is that it's not a retrieval system. I'm actually now generating new compositions as as part of the inference Price.

That's correct. It's not just a retrial, al system when when you're just doing static in france with that them you're just prompting to get eating back here some some result um that's pure retrial al uh and there's very total recombination happening.

Any recombination if IT happens must go through uh one of these uh, we learned a recommendation programs like you know some people say that um in cultures learning uh is leveraging some kind of uh hot could I get the little garish that's latent 清洁 的？ So maybe that's happening. But whatever is happening clearly, empirically, we can see that doesn't work that well。 He doesn't adapt to novelty to the ming full extent.

Ts, but um if you add this time frightening, then you are actually stalling to the real recombination, right? You're not just a reapplying uh, the the programs stored in, you are trying to to modify them, to recombine them into something that's a custom to dusk at hand. That's the process of intelligence, right? Uh, I think you know directionally, this is the right idea. Uh, the only sure have with this is that, uh, great and decent is just a way to the recommendations mean is IT and program since they are going and of course right uh is is just the the wrong approach.

So in which case had this discussion of jack when I interviewed him. But while I accepted that it's A A general method, of course it's still um demain specific in the sense that you have to come up with a prompting technique in order to find you in the language model and so on. But but IT could in principle be applied to um you know fairly broad domains of of problems but you would agree though that IT goes against the spirit of your measure of intelligent.

So there there are elements of the approach that are not quite in line with the spirits of the of the competition. I think about the idea that um is gonna be train I M on millions of generated arctic asks.

So this kind of makes me think of an attempt to anticipate what might be uh in the in the test uh the asset in private test set um try trying to joint as many tasks as possible and hope for collisions between uh which you generated in what's actually gonna be uh in the test set so that of course is trying to hack uh the benchmark for a memorization um IT is not what we intended uh but you know ultimately IT is up to us and the creators of adventure, mark, to mention that IT can not actually be hacked and that is. Resistance to a moist. If we did a bad db with that because it's actually possible to antilles the private to set, then that's on us.

So in practice, by the way, I think we did a decent job because, uh, that so if if you're not doing to some functioning right, you're only getting a very low access on the desert. So kind of shows that yes, the desert is actually decently novel, right? I think this also shown by the fact that um the best uh L L lamps right now, if shorters doing a direct prompting, they're doing, uh so the best one is uh close three point five is doing twenty one percent right so I can implies that uh about eighty percent of the list.

This is decently novel right? Even if you if you use as your frame of reference to uh entity of the internet pretty much uh so let's actually get sign uh but I think you know in the in a jackers push out. So uh the overall approach is in the in the spirits, uh uh where they are in mind because what is doing is a form of program sentences. IT is just that is gathering um if we are learning is is gathering this right and then is very a shadow combination and do you need to sky in the same, which I think is is not which should be doing, but IT ends up working right.

So why not I I agreed that so to actually in spirit is the right approach, but its bottler, ck by to caster gradient to end on on on a large language model. But this is just an interesting segway. So again, in your deep learning with python book, I think around chapter for very pedagogical for X, I want to end machine and you speak about the leakage problems. So you know the reason why we have a training set and we have a validation set and a test set is we don't want information to leak between the the sets and IT can happen inadvertantly. So for example, every time someone gets a new score on on the the art chAllenge, it's tested on the private set and that information and people then modify their approach, and it's as if theyve seen something in the private set when they haven't seen IT directly.

That's coral. And they've seen net uh this approach that they they've tested performs Better. So now we've learned something about the the concerns of the of the private test set. And yeah like many folks, even you know folks while uh machine experts, they have this a misconception that you can only over fit if you are directly training on something if if you're using this train data, that's not the case.

So for instance, uh some years ago, people we are doing a new architectural search to find a new convened arctic tus um that to perform well on the and they all used imation ate uh as as they are reference and what they were are doing is and they are mining this enormous space of possible architectures and selecting the once uh that ended up performing well when trained uh on on on imagination uh and what you ended up with was an architecture that was at the architecture level uh over fit to the the imagination uh evaluation set right um india， if you have any sort of process that extracts uh information even even just a few bits of information from your relation, a set and is rejecting information back into your model, even if it's not an automated process, even if it's if it's just sue looking at the results and then twain to approach by hand. H you are staying gradually to a effort to to to watch your your testing on and uh, you ultimately this would happen with the the the a private decide of OK. G, I, this is that because the the only bit of information you get each time you submit something is your total school.

You're really not extracting any bits information right um but eventually because uh each body band can make three submissions a day um and there are many passions uh eventually which starts uh over feeding um which is part of the reason why versions ah do something that is prety important should have been done earlier probably which is that um we are gna well I have two private tests right there's gonna do one uh that to evaluate on when you submit and and for which you see the school that's gonna the late then we also going have an extra private one, which we are only going to evaluate your your solution on at the end of the competition so that you're going to proceed competition by only getting the setbacks no let here's how value perform on the first uh private deserts, right? But at the end we're gona swab that out with the new one. And then you're gna hope that you want to IT .

hope being the Operative word.

yes. yeah.

Now might be a good time to talk about our friend ran Green black from red reset side. I interviewed him, a very smart guy. I enjoyed talking with him, and he did a kind of, you know, let's generate large and lage of candidate programs, ella lemon, then validate them. And I kind, he didn't want to call that a nearest symbolic framework, which I thought I was curious. But what would you .

think about is a project yeah I think that uh direction in that the right approach you know we we can describe how when you are solving an arc tsk， you are uh generating a small number of hypothesis that and their programs and then you are actually executing them in your mind to verify without without the correct on note, right? Uh, it's the it's the same kind of process where you're using a big, intuitive aching to produce candidate programs. And this candidate programs are hoping that they uh, more less write, but you're not sure, right? So you still have to verify them uh V A uh V A system to type uh process ah which you know in in in this case it's gonna be a could h in your case your action literally going to be uh executing the program in in your head um I think that's that's basically uh again the same type of uh uh program search approach that we are seeing uh uh among the folks are in rude on search.

Uh the minds I approach is just a different point uh on the program and to the spectrum but this is the same kind of thing right uh and enjoy you know I think the the research direction that is the most promising to me is combining uh deep planning with the script program such maybe not quite a what train Green but is doing, but the idea that are gonna use uh H A the planning model to guide program search that IT IT has to look at fewer candidate programs or so programs. That is absolutely the right idea, right? So i'm not surprised is getting good results and I do expect you are gonna keep seeing uh even Better results from science of the supporters。

So one thing I would change um is instead of generating into and python programs and then um just having a binary check is is correct or not. Um I think that might be more interesting um IT might be A A Better use of the alem to generate uh much fib graphs built on above, uh an all specific D S S L. And then instead of just checking whether the program is correct or not, you might want to do uh local decorate search around your candidate programs, basically use your candidate program as uh seed points like starting points uh for discrete search to uh reduce the amount of work that the discrete problems such processes to do um and you know you sure use I keep repeating this, but you should use a lamps as a way to get you uh, in the right direction.

But you should never trust IT to land, uh, in the exact write spot. You should assume that where you land is probably close to the solution, which is not exactly solutions. You are still going to have some some amount manner work to do to to to go from uh the points like for sense, the candidate programs and the the and produced to the actual solution. And that work has to be done by a system to tight process.

Yeah, I discuss this with him, and he still is of the mind that they are doing emergent reasoning and given enough scale that the divergence between elliot ric risk and empathetic c risk tend towards zero, which of course we don't agree with, but I agree with that wouldn't be interesting if its quite stateless.

The system at the moment wouldn't be interesting if there was some kind of program library and maybe retrieve a login ted generation into the library. He does have some interesting properties to the solution, which maybe might want to comment on. He's using vision.

He's doing some interesting prompting. He's using self, a reflection. He's got like a candidate valuation methodology. What what do you about the overall thing?

sure. Um I think it's promising. And um yeah you know I I I think we're going to keep seeing violence of this to perform well. And this is this is the reason why you introduce uh, the public truck in the charge you know we keep turing from focusing.

I'm 说 A P four can can can do this。 We like, well maybe uh let's try IT um and of course you can not uh enter the private competition with you for all because IT would IT would involve sending private a to the opening service on my private. So that's not possible.

So what we did is that to introduced an alternative test set right, which we call some my private. So it's it's private in the sense that we are not stablished IT but is also not quite private because IT is being sent h to open our service and on tropic service and so on. Um and um we did this because we want people like crying Green lands to show up and come up with some uh sophisticated a china thought by line and improve us wrong if possible.

And just before we leave this pair, are you aware of any other interesting approaches which perhaps on in the public demand? But you .

know that so I am aware of various people making claims about a about their solutions to work uh but i'm not a way of uh， specific details. They tend to be the secretary people uh and ultimately I only trust what I see. We have two trucks. We have the private track on cargo was a lot of money on the line. We have the public truck where you can use any set of the alter lemon you want um if you if you have something, you you should you summer one to one of two tracks, if it's self contain and just go for the money uh if if IT uses an L P I and use the verge track. But if it's not on the leader, more than probably are .

the organizers worried that if someone did reach human level performance, that IT would be worth more than a million dollars if they sold IT somewhere else.

Um sure maybe I I doubt that's what's gna happen though, but maybe .

interesting. And and also just on the economics of IT, this is quite an an open source approach. But what do you think the incentives are? Because if if I already had a really good solution, if I was jack cole, I mean, I would if it's worth me spending six months on IT, because as a good chance, I might win if I have nothing, then maybe i'll just have a quick look conceived as anything. But I went, invest much time VS, start up a lab and put the money into that and just hire good people to work on IT.

So of course, that is a big money prize. But you know, we don't expect that people are gonna show up and sort of ark because they want the money specially, uh je mouths money is not high enough. This is gonna en instead. Uh, the money that are putting on the line is just uh, a signal to uh indicate that this change matters. And we are sales about IT and we think it's important.

But ultimately, the real value that there is uh uh in submit a solution and winning is I would say a reputational value as like you become uh the first person to crack tice open chAllenge in open in size in ce 1 uh and personable ably your solution is a big step phone towards A G I。 Um a lot of people are talking about all right now. Uh, if you were to solve IT, uh, you would definitely make headlines right? Uh, IT would be a big deal.

So instance you mention studying a lab well, uh, IT would be a great of opportunity to start a lab around your solution and then rise them for fine, right? And you could do that. Uh just just on the momentum generated by your your winning entry.

Could you comment on the, you know had sab combat I on recently and he's got this elephant modularity tecture, which is really interesting. You know, basically you have this news symbolic, you know, L M generating ideas, critics. What do you think about that general idea?

Yeah, I think that's uh, generally the right approach. Like you should not blindly trust the acute of the alam. Instead, you should use IT as uh, an intuitive suggestion engine. IT will give you good candidates, but you should never just blindly believe that these these candidates, exactly the correction you are looking for, you should have.

And this is why I L M module, some excEllent fire is so powerful, is because you are cutting through the the community explosion problem that that that would come with, uh, trying to iteratively trying a response solution. Uh, but you're also not a limited by the fact net in lamps or table at system too, right? Uh, because you you still have this last smile verification and that's gonna done by true system to uh uh solution.

The architecture is really interesting because IT was by directional as well. So the outputs you know like that the verifiers might give you yes, no maybe or some additional information and then the elements could be fine tuned and some but but my read on IT though is that IT brittish zed IT a little bit because the verifiers, of course, are very demain specific and that seems to be slightly different to some of the solutions .

to the art chAllenge yeah ah I will tend to be the main basic and also uh it's it's not always the case that Operating in the domain where there can be an next of the fire, right uh, sometimes they can be, I think, impossible history with program sentences from input output bells. So in about this is through for OK in fact um because you know you know what output you have to expect even certain input and which your producing can be uh you posing programs so they can actually be executed. They can be very fied uh for many other programs, you have no such grantees, right?

So moving on a tiny bit. agency? Yes, now I think of agency as being defined as a virtual partition of a system that has self causing sation and intentionality, allowing for the control of the future. And I assume that is a necessary condition for intelligence. I know you don't because we bug about this the other day, but what do you think is the relationship between agency .

and intelligence, right? So you know many people can treats a agency amendment intelligence almost intentionally concepts um I like to separate them out uh in in my own model of the mind um and the west is interesting is a tool that is used buying agent to accomplish calls um but IT is IT is related to but IT is separate from uh your sentimental space for instance um or are your ability to set goals and I think you can even separate out from your world model so I know if your uh an R T S player maybe yes .

as in command and conquer .

warcraft write workflow ft worker ft exactly uh so all these games. Uh uh uh artist games and in artist game, well, uh you know you you know units moving around and you can give them commands um and you have a mini map as well. So imagine that you're selecting a unit and you're right clicking somewhere on the mini map to tell the the you need to go there.

Well um you can think of the mini map is being a world model like it's a simplify representation of the actual world uh of the game that captures uh key a elements of structure um like where things are big and where you are and when you're right clicking the mini map you are specifying a gold. And well in this in this metaphor, h intelligence is gonna. The past finding algorithm is taking in this word model, taking in this goal which are externally provided, and figuring out what is the correct sequence of actions, uh, for the ancient to reach the goal right it's it's about uh intellect es about navigating uh intelligence about navigating um future situation space.

It's it's about that finding in future situation space um and uh in in in this metaphor you can see that intelligence is a tool IT is not the agent. The agent is made of many things including uh a good setting mechanism you know in this metaphor is played by you, you are sitting the gold is made of the world model which in the agents to represent what the goal means uh and may be simulate uh planning, uh it's also gonna be uh uh including a sound metal space that an action space uh and uh they can that can receive sorry feedback as well. Um but jian is documentation for all these things in their own separate from intelligence intelligence ics, just a way to take in information and turn IT into an actionable model, uh something that can use for planning right? It's it's a way to convert uh uh information about the world into A A model that can navigate uh possible evolution of the world.

I agree with everything you've just said. I think the tension is after speaking with people like call freestone, you know, when we think about the physics of intelligence and know this epic particle system we live in with function dynamics and behaviour and and so on, the agency and the intelligence is not explicit. The world model isn't explicit.

So there seems to be something else going on, which is why, in many cases, I think of agency and intelligence as being virtual properties rather than explicit physical properties. That's not to say that we couldn't build an A, I where everything is explicit because that would be useful. We could build in computers. But there's always the tension of whether we think of the world is this complex simulation of low level particle is a nested agents. I have cells which agents and my heart is an agent, animae agent, or whether it's explicit.

right? When I think in the first stage I trigonia a be, uh, these different companies are going to be, uh, explicitly separated ads in software because they simply easiest, uh, where to get there. At least it's might take on IT and the arctic is gona be expressed. yes.

So you actually spoke about functional dynamics the other day, which was music to my years, obvious ly being a fan of of the freestone ian world view. What what's your take on that?

So to be to be honest with you, this is actually uh, something I been think about, but I do not have that crisp ideas about IT yet. But IT is my general intuition as to how the human mind performs program sentences. So think um there are there are two scales, two levels at which demand um changes itself. There's the the longtime scare which which has to do with uh abstraction mining like construction generation and memory formation is a and IT has to do with your plasticity as well. You are basically changing connections in your brain uh, to store reusable programs.

有有 formalism of intelligence focuses a lot on internal representation。 So this idea of in our minds, we have we have a world model and so on. And when I read some of your blog post from from years ago, you're talking a lot about the externalize tradition, which is that a lot of cognition happens out outside of of the brain. How do you reconcile those two world view.

right? Um i'm a big problem that most of our cognition is external ed, uh as you say, like uh when when we are talking to each other for things are using uh words that we did not events, we are using uh many images, ideas that we just read about somewhere and swan um and if we have to develop these things on, we need extremely long lives.

H to stop being intellection protective so um I don't think that really uh any any contradiction between the two views like the that show like humans, uh, as individuals are intelligence, uh, you poseuse intelligence, I poseuse intelligence, uh we can use IT sort of flag uh in isolation on our own uh uh and and uh we can h extract from our environment, from our lived experience. We can extract um reasonable bits, uh wish we can make we use to make sense of Normal stations that the process of intelligence we posses IT uh as individuals uh but also um we we able to communicate right we are not just individuals ety. So uh these ideas is a reasonable abstraction tions.

We can uh extract them uh from our brains. We can uh put them uh out there in the world, share them with others like we can write books for instance. Uh, we can type up computer programs that can be uh not even just executed by other brains, but even by computers right um and um this process is just the decoration of culture and then uh one's culture is out there.

You can download into your brain that's education as you're doing IT uh your solar flag artificially uh filling up your bank of reusable abstraction tions uh and see you short cut you know uh it's it's almost like downadup a skill is like in the matrix. Uh, it's it's little bit of that like h learning about physics, learning about math. Uh you are downloading these a very rich uh regional mental templates like really mental billing blocks and then you you can in your own brain, you can recombine them, uh, you can reply them a new problems IT makes you a more intelligent like history, more interesting ent makes you more efficient at the acquisition, more efficient at problem solving and so on.

Yeah, beautiful to cut a couple of great book. I D on the language game. Max ents book on intelligently talking about this, the plasticity of memetic information sharing, you know, allowing us to stand on the shoulders .

of of giants. A there's a an interesting an angle to the question you ask. I know I know if you where where are that but what what i've described there is the decided that humans are the source of obstruction uh human individual and brains use their lived expense to extracts abstractions and then they are uh extent icing them via language。 Typically not not exclusively but most of the time uh and then other brains can download abstraction tions and can make them their own, which is a huge shortcuts because uh you you you don't have to expand everything on your own to start leveraging obstructions。

Um but in this model, abstraction generation and abstraction recombination to form new models is always happening inside brains, right? The only part that's externalize is the memory that your uh uh moving the abstraction s the really big box out of this brand are putting them in in books and on uh and then and then double adding them back but to be useful they need to be internalized in your brain uh uh question then is could uh abstraction generation or recombination actually happened as simple as well, uh not as any uh in the context of creating A G I because you know that's exactly what what N G I would be to be uh this uh recombination and abstraction process, this syntheses and abstraction process, uh, encoded in software form. But do we have today external processes that that the implement is well, I think we sort of do.

I think science in particular uh, is doing a form of sentences, uh that is that is driven by humans. But uh, IT is not happening inside human brains like we have the ability to uh do recommended search over h spaces that actually cannot fit inside human brains. I think you said uh, in a lot of things that invented like uh when when you create uh a Better computer, for instance, uh you are doing some kind of recommendations, search of space of possible devices.

But you are naturally able to hold A A full model of the device inside you own brain. Instead the model is distributed uh across some some number of excEllent zed artifacts um and I do believe that humans citizen is implementing this highly distributed um since is the spot of the of the process of intelligence is implemented IT externally across many different brains, manipulating export symbols and artifacts. And this is what's and inning a lot of our civilization because the systems we've been creating we've been inventing or uh so complex that no one can really h understand them in food.

So you cannot run a this this invention process inside brains anymore. Instead you are using brains to a drive a much bigger, excelling zed process. So I think recognitions externalized not just in the sense that uh we have the we have the power to uh uh write down and then uh read uh ideas, abstractions and then reuse them inside of rains. We are actually your running uh intelligence outside our rains as well.

I completely great. And you've written about this about how intelligence collective situated and externalized. But there's always the question of many of the unit like science, for example, is is a kind of collective intelligence which supervenes on us and and language as well.

But do things like mmis happen outside of um biology mean certainly that happens in the world in the selfish gene that happens with genetics. but. You could argue that a kind of meet is actually happened just in any open physical system with certain patterns of functional dynamics. And so so, you know, the real question, I think, with this exterior, ed cognizing, is, where do the abstractions come from? Perhaps our brains are just very efficient at building the map from the territory and it's it's just a slightly Better way of doing what already happens naturally externally.

Yeah um I think to a large extent, uh the way with externalized cognition is uh not as efficient as the way with implemented cognition in in all brains um this exact product you know so intelligencies is a kind of such process right over specific recommendations of a thing. Um I think right now this search process is too large extent the ized when you're looking at technical, when you look at science um but it's not exercised in the smart way. I think we are roughly implementing uh beautiful search, see other spicion deplaning research um the way the deeply anning committee as a whole is finding new things is by trying everything else and eventually hitting the thing that works, you know um and I believe individual humans actually much if they if they had enough brain die ware to actually modern these things uh in their own brains， uh there would be much more effective at finding .

finding right solution. interesting. I mean, ryan Green blatt view was emblematic, emblematic of some of the x risk folks in that he was arguing that he can be in a memetics sealed chAmber, will be a brain in a vat, and it's a pure intelligence.

He would still be able to reason and solve tasks and and so on. And the the count of view is that physical and emotional ment is really important. I mean, when I asked to marry shanahan this, I said, what's the reason why we need to have physically and bodied robots? And he said, well, these robots are interacting with the real world. They're understanding the intricate cause or relationships between things, and that helps them build models more efficiently, but perhaps in service of just learning about the attractions which would exist .

in the physical work. Yes, to exercise intelligence, uh, IT needs to be Operating on something like you think out of something, uh about something like you. You need to have some concrete environment and goals in that environment that you want to completion and actions can take. So it's about something that cannot be about nothing, but it's also a made of something you are uh, making your plans for your goals based out of existing components, existing uh sub tins uh, if you have nothing at all, are you not only you have nothing to be intelligent to bots, but your uh intensions is nothing to recombine right um and that's why embodiment is important. I mean, in in humans I mentioned this, I D, that cognition is built layer by layer, each new layer, which should be more struct on the the one before IT.

IT is built a in terms of the components that can before, and if you, dig， uh, did enough, if you enforce your mind layer by layer at the the bottom, uh you will find things like uh the circle reflects for instance uh it's like starts everything starts with uh your mouth uh and then um you you start adding things like graduating objects to put them in your month and then things like crawling on the floor so that you can reach objects so you can you can grab them and put them in your model and at some point where you start putting up in your month. Uh but the new things you're learning are still expressed in internals of this sort of flat concept and skill, higher chy right and uh when when you end up doing abstract math well, you are using building blocks that eventually resolve to this extremely primitive uh uh a motor uh uh sub routines right so yeah and bedroom is important but at the same time, uh, I think the kind of body and and some matter I fall in space that you have is very much plug and play. If you have a true G, I, you could， you could basically, if if you have an G, I, you could play any environment, uh, any sense of space, uh, any D S sel as well uh into IT.

Um and we started being intelligence about IT, you know uh so in that sense, like ebola ment is important um but what kind of evidence might not might not say IT be important um and you know uh uh a another thing that really important is good sitting, by the way, which is distinct from body and is also distinct from intelligence. If you're just a brain in jawa, uh uh with with nothing to think about, well, you not going be very intelligently. So you're not really going to be doing anything because you have nothing to do.

You have no goal uh uh h to drive your thoughts um and I think especially if you if you're looking at a children, the way you learn anything is by sitting goals and accomplishing them. You can not really build uh good mental models uh uh good good word models uh passively pity by you know uh observing was going on around you uh with no goals of your ARM that's not hard to ork good setting is a critical components of any any intelligent agent. I completely.

I think the only unresolved tension in my mind is that there are many manifestations of intelligence. And IT is possible for us to build an abstract, explicit version which would run on computers. Essenic doesn't necessarily need to mimic the type of intelligence we have in the real world yeah I think so.

And I think if we d probably have uh uh at least in its for three terrans IT will probably have significant arctic crs similarity with the way interesting in people but um ultimately you know IT IT might IT might drift a way towards towards the entirely new ties of intelligence.

Now you've said that languages is the Operating system of the mind. What do you mean by that?

right? So what's Operating system right? It's not the same thing as a computer. Um IT is something that makes your computer more useful uh and more useful.

IT empowers uh competing for some museum um well IT IT empowers some user uh to to to best leverage uh the capabilities computer. I think language plays a similar role for the mind. I think language is distinct from the mind like it's it's a separate thing from from intelligence for instance, even from the world model.

But IT is a tool um that you as an agent uh is leveraging to make your mind to make your thinking more useful right? So I believe language and thinking are separate things. Languages is is a tool for thinking. And what do you use IT for? Well, I think one, where is that you can use language to uh make your thoughts, uh introspect able your your thoughts are there. They like programs in your brain and which you can uh execute to get their outputs um but you can actually look at them uh by writing them down in in in words I don't mean like right them down expressing uh as words uh suddenly you can start uh reflecting on them you you can start looking at them.

You can start comparing them and a critically you can start indexing them as well as I believe a then one of the rules of language is to enable you to uh do indexing and retry val over your own ideas and memories if you did not have language uh then to retrieve memories you would have to rely on uh x nal steamy right like you know uh a post is eating a MAdeline and it's reminding me of uh um uh uh a specific time and place and um if um positive don't have language then every time 111111 and retirement needs to uh think about the best of time place。 He would not have to eat the madeleine. This would be only access pots to that memory right this exact sim uh if he has a language and you can use language to try to a query, uh, his own world model and retrieve the memorials that he wants, so it's uh it's uh a way to express what you want to retrieve uh, inside your own mind.

Uh, it's also a way to compose together more complex thoughts if you cannot, a reflect on thoughts, if you cannot, can like me to realize them and and and look at them and and modify them in your mind. And I think you also quite limited in the in the complexity of the thoughts. You can you can format h this.

This is a very very if you have a computer, you can actually use IT to write programs. You do not in the betting system, right? You can just write in assembly code um why not but you are severely limited uh in in in terms of the the complexity of the soft can.

If you have uh in Operating system and you in in in and you have uh uh you know high level programing languages and so on, then uh there are tools that you can use as a programmer uh to to develop much more complex software and your intelligence is a programme. Your program mably has not changed, is just your tools that have gotten Better and certain you are much more capable as you are before, right? So think intelligence is using language as a similar kind of tool.

Yeah we have this information architecture of mediated abstractions at is almost like concentric circles of of complexity. And in the language going, they spoke about, you know, scissor, a physical tool, l and and language are the memetic equivalent of scissors. And of course, we can compose these tools together and and use them in different circumstances. But moving to a consciousness mean you suggested that consciousness emerge is gradually in in children. How does this, you know inform your your views of of machine .

consciousness, right? So I mean, to start with, I am not that interested in the idea of mission conscious ness. I'm specific interested in intelligence and rated uh uh spec of cognition an I think consciousness is a separate problem.

Clearly you know that IT has some relationship with intelligence. Uh you you you see for a sense in the fact net. Well, any time you you use uh system to thinking, you are aware of what you drink, consciousness is involved. So clearly there is air relationship between conscious consciousness and system to uh the natural of the relationship is not entirely clear to me and our who do not pretend that I understand conscious and that well and honestly, I don't believe that anyone does so I am all as very suspicious when I when I help people whether very very uh and and precise and category ideas about about consciousness so that you know I I do believe that it's plausible that machine cautionary is possible and in principle also believe that um we don't have anything and the resembLance machine cautious today h will probably play far from IT um for for a system to be conscious you know IT would need at the very is to need to be.

Uh much more sophisticate than a so of like input to attach mapping that you see in the deep planning models in elements um at the value you would expect the system to have some kind of permanent states um that gets uh influenced by uh uh extra Simony but that is not just fully sets uh by extensively IT has some kind of uh consistency and continuity through time uh IT can influence its own future states IT is not unior reactive right I think consciousness is is in opposition to a pune AR active types system, selective turning models or insects maybe um and I don't think we have any any system that looks like this. So they also think consciously ness require ability to uh introspect quite a bit like this sort of like uh self consistent state of the system that is meeting across time IT should have some way to represent the influence itself, should believe drifting in a way um and we don't have anything like that today but in principle, you know maybe maybe it's possible to uh build IT. And so you mention uh single mention twitter decided that babies are not warm conscious， which part is extremely so maybe I can I can say a big moment that um so feel so you know we have no real way of uh assisted with a business certainty whether anyone is conscious at any stage of development right it's basically guess um IT seems to me that uh baby uh in the wood b are very unlikely to be conscious because uh they're basically uh there they're basically free asleep all the time like there uh a sleep you know in in one of h two possible uh sleep states like nine five s of the time that a deep sleep well there just you know you know and there's active sleep ah we are they are moving around and you know the mother can can feel them uh move around.

And when they moving on they are not actually awake. They actually a sleep, is just active sleep. And the remaining five percent is not a wakefulness, is just transitions, are between deep sleep and active sleep. And the reason they are just sleeping all the time is that, a they are being sedated, right? Uh, the woman is very a low oxygen pressure environment a and hesitating them and also the presenta and the baby's itself are producing uh uh anesthetic products.

Basically the pleasant uh is actually producing an and statics and so that's keeping the the babies like in this uh dream less sleep uh pretty much which doesn't mean by the way that the brain is not learning. The brain is not like just just collected and doing nothing. Uh, they are actually learning, but they are they're learning in this very plastic ve way.

You know just computing statistics about was going on in the environment, which which is mean what brains do with your wake on your sleep um well yeah I I believe that babies in the room are not conscious and when they're born this start at at consciousness level zero pretty much um and as they start being awaken and expands ing the world, uh then consciousness starts to light up. But IT is not this sort of like instant switch, where where they go from a being unconscious to being fully conscious IT happens gradually。 So you start at zero.

And by the way, you can have to start your even uh after you wake up because uh when you're warn you have nothing to be conscious of, you know like um pretty much everything, not just actions between perception is something that you have to learn through expense when you're born. Uh you cannot even really see, because you have not learn to see, you know uh have not trained, uh, your visual code. Tex, right? So you can see maybe like blobs of flights.

Uh, you cannot you do not have A A model of yourself if you want something afford ces, uh, you have maybe a crude a portal model that you developed by moving around in the woo and your brain. A math, uh, was going on and and correlations kind of flex in in your s of space. It's not really a model.

It's not sophisticated model of anything. So you have nothing to be conscious ous of. You have no word model, no model of yourself, uh, no real h incoming perceptual stream because you have not learned to take control of a few of form is just yet. So you start zero. And then as you build up these models, uh, your world model, your model of your elephant is one, uh, you start gradually bit by bit, uh, being more conscious and uh at some points you you're rica level or where you can be said to be fully conscious the way may be like, uh, a dog might be fully conscious. Anything that happens pretty fast IT happens probably significantly earlier than the first clear external signs of consciousness ness.

Uh I think around one month old each now the babies are problem, uh conscious, uh uh the same level as you know most most mam I suppose um but still not adult level of consciousness right um and um I think a dollar level consciousness is something that children only starts expanding around age two to three doesn't mean that they they were not conscious all time like again the conscious pray much starting on day one is just too to voice small amount right um and uh so consciously something that you have to build up over a time at least that this my theory and um there are some sort of flag indications that this is not entirely made up but again um one example is uh if you try to observe uh attentional blink, uh uh try to measure IT in in here and you will see that basically up until age uh three, they have significantly slower red and attention blink and the adults uh and you're gonna pause 嗯 uh the events around them into uh uh fewer fewer events so they can have a more course grain resolution uh of time and in the world um and I think that actually。 Uh that that that time to each level a of conscious ness and also have this is very uh probably convertible idea well so you reach uh a doctor conscious this run like h two to three years roughly um but then you don't stop there. You actually keep getting more and more uh conscious several time and um your conscious ous civil will probably peaks around the age like nine to ten.

And then then in goods in reverse, you get uh less and less conscious with every every passing yeah but not too very a significant extent so that uh the the difference in the graph conscious ness between, 嗯 i don't know， and ninety old and a ten year old and a three olds actually fair and minor. But IT is still there. And I think this uh plays into some things like for something or objective perception of time.

I think the more conscious you are, the the the higher 有有 level of consciousness， the slower uh your perception of time because your perception of time is highly dependent on a how many um things you can notice in uh any any time span so one way you you could conceptualize your degree of consciousness is you can imagine consciousness is kind of flag nexus uh in your world model it's a focus point from which um uh from which span like adventure connections to other things uh connections that and call this this uh focus point and give IT meaning and these connections they can be they can be uh they can be sure them or more of them and they can be more less deep right and the deep connections the more you have the more conscious you all uh and and they also uh temple component where uh if if you're highly conscious and even in one sign you might be noticing many things, drawing many connections between these things uh in in in things you know uh that's that's a high level of conscience on the other hand are if you are not if you're noticing very few things, if you have a very qualls grain perception of realty, uh that is a old thing and and you only noticing few things, uh uh uh any any time spend them uh you are you have a faster perception of time like think things just in a best in a blink um and that's that's the other questions like uh if you drink a little boots uh you have reduce consciousness right and things will will actually seem to move faster and you will notice you are things and and the depths of connections that you establish betwen things is less um I think something like you know if you if you um uh a one year or toler uh you have uh a much uh uh tension blink your perception of time is likely they very fast. No, we have decided that children purse ve time uh, slower. I think that's true, but I really depends on your edge.

I think if you're one time is super fast because again, you you you are these lower eleven cushions, if your stress basically a dark level, but if your ten is actually pretty slow, right? If you seven, it's it's slow as well. It's actually get slow and slow and slow until until IT picks on at is like nine, they start being first. So again, because you are lesson less conscious. The time I remember .

being very bored when I was a child, I not felt bored in as long as I can remember, and I intervened. Professor max songs recent. It's got a great book called the hidden spring.

And his basic idea is that consciousness is prediction errors. So the more you like your conscious, when you first learn how to drive, so the more things become automated, the less conscious we are. And then maybe tangles faster in many ways as as we grow up. But the this idea of being more or less conscious is is really interesting, as he says, like a little switch, but on on the machine sentence thing, I remember you came on the show to talk about the chinese remarkable and you said understanding is a virtual property of functional dynamics in the system and presumably you would also argue that consciousness is a virtual property of functional dynamics in the system.

I think so. I think IT IT is not strongly tied to substrate. So in principle, you should be able to implement and consciousness, uh, using the right functional al DNA ics in silicon authentically. I don't think we have IT all are those who are close to in IT. But in principle, I don't see problem with that, yes.

And will leave the hard problem of conciousness to one side by the mark. Songs was quite dismissive about the hard problem of consciousness, you know, which is that there is something that .

is is like to be conscious. Well, I think there there like some people dismiss, yeah, some people dismiss the problem of conscious the same. yeah.

No, like something like a consciousness is what IT feels to be. Information, process, system, things like that IT really means nothing. It's just pushing a the problem back to where you can Better concentrate with words, but it's not be using in the problem.

There is clearly such a thing as square and you are producing them right now. So you cannot deny that they exist. Uh and we have no way to explain or even describe what they are like. You can describe many things about consciousness, but the count, the subjective experience is not reducible to to the explanations. There is something and we don't know what that is.

And you think we we have IT and animals have IT .

but anymore habit. I mean not animals uh and I again like I believe in this idea of degrees of conscious ness um and and uh h animals probably have IT to less extent. Video IT might IT might not be used difference by way, but it's probably less year.

Do you think the earth could be conscious to sound degree?

No, I don't think so. I think um none animal systems typically lack to basic parachutes that I would want to see in a system to even start entertaining the notion that IT might be concious like fun sensibility to maintain um this um self influenced, self consistent uh inner state across time uh that influences by perception but that that is also capable of a driving itself, prevention, producing its own future state ah that's a capable of our presenting itself introspective and so on. Uh, I don't think you see that in non biogen systems today.

Do you think the collective of all americans could be seen as a conscious being? No, why not? Again.

because IT lacks as his basic prerequisites.

So IT needs to be a physical form of connective ess to the surroundings. IT couldn't they couldn't be a virtual version distribute to over many agents.

You know you you you could definitely mention a distributed version issues that i'm not saying the collective of full americans, for instance, h implementing this uh self influenced, uh, self consistent uh state capable of h representing itself and the world, uh and so and even then, you know even if you have these things uh in self to assistance, for instance, it's not automatically conscious IT is that it's start being plausible that IT might be conscious if you also see uh uh science like, uh take your song, that might be so what what might visit to sign? Well, uh, it's difficult.

And I I don't think that you you're ever gonna see uh is um a proof of consciousness at a proof of conscious ness networks. At most of the time, I think it's always guess, but typically you know I think it's highly likely that system is conscious if IT has all these requites and IT is capable wolf expressing uh statements about its own inner state um that cannot be uh purely a product of. Repeating something the system has heard, you know like if you ask an aly lama about a hyde field and so I will answer something with it's really just, uh, rehashing something he does read so what I would want to see is, uh, the system is making statements about hide files and there seems to be a strong correlation between the behavior of the system and what he is telling me and what is is telling me is unlike anything the the system has seen elsewhere before.

Like I don't know, i'm uh, holding my two more than trying to uh consult them because um the crying and I like hash should not cry a stop crying in a dialog but I wants to cry that I feel like, well, there's a pretty strong coalition between what what what the child is doing and what they're saying about themselves so you can believe them uh and they've never heard anyone think I wants to cry. They are really expressing something that they could not have picked up from anywhere else, you know. So in these situations, just highly possible IT is not proof benefit IT is highly possible ble that the the in fact do have uh some awareness of their their own mental states and they expressing something about them and they are actually conscious. They're expensive where you .

know so from where you've been very critical of singular ratan ism and demonism, what do you think is the driving force of these extreme views?

Well, you know, I think good stories like stories about the end of the world decided that we are living uh in the end times and maybe that we have a role to play IT. Um there are there are good stories which is why you find them a lot uh in fiction like in science fiction for intense to find them a lot in religion as well uh and they're not new been you ve been around for thousands of years.

Uh so I think that's the primary driving force is just not they are they are good as he means they are good stories. People wants to believe them uh and and they also raised to retain and propagate that's that's really the the main thing. Uh you know everyone is just creating a meaning uh uh to organize their own lives around which is why court are still a problem uh in in all day on age uh and there is just an instance of that。 I think do .

you think there's a bit of a masi complex as well as .

yeah absolutely think you see that out in the in the sunshine school beria um then then all people will have kind of lashed onto this idea of building A G I um and while using IT to sort of flag picture themselves as ma as you say, basically I see a creating A G I as a scientific problem.

Not not a region quest um you know and this this is often um going to merging together with uh g ID feat our life by anyway uh which is very natural because uh uh the the storing most regions is always about uh um this this combination of um anyway um well yeah it's it's it's a kind of merging as well, a eternal life, right that if you create G I H IT will IT will make you forever pretty much so this is a very easy idea, right um and IT has become this region quest uh to get their first uh in uh whoever guess their first will become uh as god right. So i'm not really subscribing to any of that. I think building A G I science fc problem and uh once you build A G I is basically just gonna be a very useful and valuable tool IT is gonna.

You know, as as as I mention a bus finding algorithm in future situation space, it's gona be a piece of software that takes an information about the problem. And this people, people are very efficiently synthetize zing a model of that problem, uh, which you can use to make decisions about the problem. Uh, so it's so that we're tool.

But IT IT does not turn you into god. And certainly you can use its incentives research. Maybe you can use IT in longevity research, but does not automatically uh make you immortal because this is not an important I think if you start anything very powerful ways to turn information into actionable models, your bottles neck quickly starts becoming the information that you have.

So for instance, if you have uh N E G R that can uh do physics IT can quickly synthesize new physics theories。 Um the the thing is uh human scientists today, they're already a good at that. That in fact too good.

They're so good that the ability to synthesize possible new theories for exceed ability to collect uh experiments or data to validate them to switch you see with string string theory for instance. Um and that's that's a pretty star illustration of the fact net. If you are too smart then uh you start you start running can like free uh uh uh of information um and the start not being very useful anymore, right?

Uh apply intelligence is grounded uh in the in the experimental data. And if you are very intelligent, then experiment that becomes about neck. So it's not like you gna h run away a initial explosion. Is there anything that would make .

you change to mind? And when I again I D disabled sound with Green black and I try avoid having x rest discussions when when i'm actually debating and a lot of IT hinges on agency so I said because I don't think systems of a gentile or will be I don't see the problem because a lot of the the myths around this, you know the bostrom and ideas around instrumental convergence and autogas ality, it's all goals. It's all cy based.

So no agency, no problem. Presumably you agree. But you know, maybe if there was agency, would you think there was a problem?

Yeah no. I think intelligence is separate from agency is separate from gold setting. If you just have intelligence in isolation, then again, you have a way to turn information into actionable models.

Uh, but IT is not safe directed IT is not, uh, unable to set its own goals or anything like that. Go setting us to be an add on, uh, an example component bugging to IT. Now you could image that, well, what if you combine this? A G I was not in the sitting system with a value system.

Uh, you turn all of that into agent and then you you give IT access to, uh uh, the nuclear codes for something something that, uh, is that dangerous? Well, yes, but you engineered that danger in a very a deliberate fashion, right? Uh, I think once once we have A G I, uh, we love, uh, plenty of time to count. Anticipate this this kind of potential risk. So I do believe you know uh A G R will be part of the 看 了 深 uh so with this is exactly what makes IT developable and use some um anything bad form is also potential risky， but we are very much gonna be the ones in control because H G R on its own not set goals until you actually create uh uh uh not a is gold setting mechanism.

But why would you do that? You know so the the difficult parts, the tender spot is not the intelligence s bit, it's more like the the the gold sitting um in action space bits uh and if you want to create something very dangerous that creates that uh sets its own goals and text action in the your world, you do not actually need uh very high intentions to do so. You can already a do so with vacuum nixes right?

Such the thing is existential risk of in is a legitimate form of inquiry and especially nuclear arrested. For example, when I many of these folks just solely focused on A I existent al risk, they're looking at other risks as well. But how do you view the incentives of, and you could be really cynical and just say, old effective ultra ism and open philanthropy that thwing lots of money at this. And what they actually want is power and and control. How do you how do you kind of think about this?

Well, there's there's definite a bit of that also think a lot of the true believers, they are just buying to IT because they wants to believe and it's it's it's again, it's very pillar to religious ideas, uh, in many ways. So I don't I don't think it's it's very rational, you know.

Um so that said, you know once we have A G R because today we don't and figure about close to IT, but once we have IT, then we can stop think about ah the risks, the time I don't think you you're gna see you know um the day the day you just start trying the program IT becomes a self way antics, control of her, uh your love and so I don't think you're going to seen anything like that. Uh, again, intelligence A G, I is just a piece of software are that can turn data into models. It's up to you to use IT in certain way, right?

I mean, like an abstract way to think about this is framing IT as safety ism and governance in general. So if we take away the hyperbolic s, can we talk about, you know, miss information and things like that? What do you think about that?

嗯？

I mean, maybe I should be more specific. I mean, uh, you know deep fakes and and misinformation and infringement of copyright. So do you think that we should strongly regulate this or would IT .

harm innovation if we did? I think they are definitely homes that can uh by current technology uh by current and item uh users uh of A I and yes, I think some form fragment might be useful to protect the public. And so these homes I also think that um the the regulation proposals that have since so far are not very satisfactory.

They are more leaning towards hamming uh innovation uh then placing the public。 I think ultimately they will they they are more they're more likely to uh end up concentrating power uh in D A I space uh then then just product in the public. Um so I think regulating here is difficult and just relying on existing non A I regulation to product people, a might be the Better course of action. Uh given that introducing a new year special regulation um is you know it's it's difficult problem and I don't think based on what I ve been so far, I don't know about IT france.

which will let it's been an honor and a pleasure.

Thank you so much. It's my pleasure. Thanks so much of me.

amazing.