Behind Closed Doors: OpenAI's GPT-5 Deliberation
10:56 Share

Happy monday. I hope you are all doing fantastic. I'm finally back in arizona after being out of town for around a month, so feels good to be back in the studio. And today we have A G, C topic, and that is why the why open eye is specifically is its training for ChatGPT four and beyond is currently not working on GPT five.

The reason for that and also the reason why that might be a little bit of a deception and why they say they are no longer going to be making bigger A I model. So let's kick the podcast off today. Um a few, a few a bunch of the stuff that and we talking about today is coming from some interviews.

Sam altman in the CEO of opening I did recently at M I T. Um and he said that in this in this recent interview, he said that he thinks we're at the end of an era where it's going to be like these giant, giant models and that they are going to find ways to make ways to make them Better in other ways. This is really interesting where to dive into what that means um but essentially what he's saying is that this might be the end of um when we just keep seen these models bigger, bigger, bigger.

And this is interesting because they actually have not released how many parameters are in the GPT four model ah when they came up with the first GPT two, um IT was around one point seven billion parameters and when they came up with GPT three, which is you know the regular ChatGPT version that they are launch for mass action that everyone first use back in december, january, if that one had a wapping one hundred and seventy five billion parameter. So this thing went from like h this thing got massively bigger. And then when GPT forking out, they didn't actually announce how bigger got.

And my opinion on this is that, well, a couple things. Number one, they're probably trying to protect a little bit of their intellectual property, so not announcing how big the how big the model is. IT could be like they just don't want people to know how big IT is because it's so generous ous. And the reason for that be, you know competitors, my question, how did you get that much data? Um you know there's been recent um recent things, for example, in elon but twitter he shut off the twitter A P I that was directly feeding into opening eyes.

They were scraping like all the twitter data in the universe um and so I think that they may just have a really inane mount of data and they don't want people to know how big IT is or um perhaps how small IT is and how they are able to effect the work on that and and then the other interesting thought is perhaps IT didn't actually get that much bigger than uh GPT three. They just figured out ways to find tune. But for all of these different cases, they just don't want people to know what they're doing.

They don't want their competitors um to know what they doing because there has been a handful of a really big competitors that have come out with a lot of money. There's anthropic uh, elan musk is rumor to be working on something with uh twitter to decorate some A I models. There's obviously google and a lot of different people are focused, cora, on making these big AI models.

And so I think they just don't want people to know how many parameters they have, how are able to make a so effective um and there's also the the possibility that maybe they just have such an insane amount of data that people would be. I'm mad, you know that how did you gather all that data and yet yet a so really interesting but apparently these things are not going to get any bigger. A K A OpenAI probably hit the max for how much? Just share data.

They can like suck into the vacuum. I'm beyond just getting new data. So as we know, everything got cut off in twenty, twenty one. And beyond that, it's problem, not a big, big deal. Update things live because now they just plug the internet into IT and for anything nor than that, you'll just use internet to scrape and do searches and polin relevant stuff like google and binger currently doing. So it's gonna really interesting.

Um I think all men you know his statement pretty much suggest GPT might be the last major advance to emerge from um just speeding these models more and more data and other and the what the actual advancements we're going to start seeing now because apparently, uh, you're just getting diminish returns as you scale up to more data or maybe they really have so much that there isn't actually that much more to do. So I think the the new ways we going to started to see this progress um is just from how we are using these transfer MERS. So nick frost, who's a cofounder at cocher, he previously worked at A I or work on A I google. He says that when he like listening to all man talking about this um he thinks that uh this is true and that he believes that the progress on these transformer models are that run that you the the backbone of a GPT and that a type of thing um that is really that what's really gna make these things scale he says there's a lot of ways of transformers way, way Better and more useful and lots of them don't involve adding parameters to the model um he also says that he just thinks that a the architecture and further tune based on human feedback are really promising directions that researchers are probably i'm looking at right now to make these things more effective.

And if you think about IT, OpenAI and ChatGPT R T have pretty much to the biggest use case, user case on this right and tesla IT was kind of in the same thing with their self driving because they had all these cars out on the road um with self driving and are able to capture all of this data and see pretty much every time you know for example, in the case of test la, if you got autopilot and the user has to grab the stern wheel take over, tesla would be able to look at you know what was happening leading up to that uh like manual take over and that's essentially um using human feedback to help tune the model and say, oh, IT was doing something wrong, therefore the human had to take over. And so you know, i'm been saying this for a long time with cars like tesla, it's game. Be hard for other people to catch up with the same level of self driving because they just have such a massive user base feeding at data and tragedies in the exact same position they have launched.

You ChatGPT, they had over one hundred million monthly active users probably way up from there. Um and all of those users, you know there's thumbs up, thumbs on every single message that came out to ChatGPT. And so people were telling IT bad answer, good answer and so that was helping tune IT.

And in addition, you don't ask the question and they refused and asked again. IT also knows, hey, the first answer wasn't good. So they have all of these things kind of built in that they can use to find tune IT just from all of the users that they currently have.

And so IT might not actually be necessary for them to just keep scaling up with bigger, bigger data sets um in addition to other fine tuning things. But like the users, a literal human saying this was a good or bad answers is one of the best ways that they can train these things. And they just have the most they have the biggest user base.

And it's can be pretty hard for a lot of these other companies to catch up um based off of that. So you know it's pretty interesting obviously, when GPT four was about to come out, there is all these means and tech people speculating and post in these graphs of like G P, T, four or three was trained on, you know, one point seven billion meters. GPT four is going to be like, way, way is a trillium parameters or something.

We we don't really know what GPT force that. And IT doesn't seem like IT IT really matters at this point. So one other thing that is pretty interesting given all of this is that you know sam Allen also said that the company isn't going to be training GPT five. They said we are not working on this for some time.

It's interesting because this all comes on the back of, you know, elon, masking a lot of rather tech people signing an open letter to, you know, the government, but really, and like the public at large, really obviously aimed at opening the eye who's got the lead on the same. No AI models beyond a GPT capability should be trained yet. Eta um open a ee obviously didn't say, okay sure we're not going to do this because you know it's viewed elan musk for examples.

Now starting would appear starting an AI company. So IT would appear that you know a lot of these companies calling for no more advancements are maybe trying to catch up anyways. So all that would do is benefit them.

But what's really interesting is um you know sam altman n obviously doesn't want the bad P R and you know doesn't want to be viewed as the guy that like ran um blind folded straight into some giant AI disaster so he he always said he wants to appear to be a you know taking things in a measured and deliberate manner so I mean kudos to him on this but he said we're not working on GPT five at the moment. We're just working on training or like improving GPT four and boom, I think that's that's the big that's the big that's the elephant in the room kind of I guess, the big surprise, big secret. What everyone to call IT like improving GPT four is like essentially working on the next version of of A I i'm not saying that is a bad thing.

I'm just saying it's funny that people are like, oh, good, he's not working on GPT five. What is like what's the difference of GPT four point five? GPT four point six? Like you could just keep calling the GPT four point something and essentially IT could be what other people would have called G P five six seminary. It's kind of funny.

I mean, you see the same thing with the iphone, right? It's like this misconception that if the number got Better, the model got Better and IT must be like so much Better like you know when the iphone ten comes out than the iphone eleven comes out or whatever, it's like a lot of the times you know these iphones aren't actually that different. And um the reverse seems to be true where you just don't have to label IT a bigger number with the like G P T four GPT five and all of a sudden um you know people are like, oh cool you know going to GPT five one know.

Is there is no different or you are to connect to the internet to GPT four and created plugging and created all this crazy stuff that might be GPT ten, but we're just call in the lower thing. So I think that's A I mean, good job for P R control, a know damage control, for open a eye. And personally, I think you know improving IT obviously has to be done for a lot of different things, making sure that is safe and capable.

But any is just a misconception that like people are celebrating if they cared about A I advancement you know that jep t five is in in the works. So I think it's not really it's not really seen much by that statement. But that is interesting um and IT is interesting to note that given the previous conversation, IT would appear that GPT five, even when they do start working on that, probably isn't going to be much bigger as far as parameters and data goes um then GPT four. So really interesting to see what's going on with all of this uh advancements in A I in tech. It's going to be interesting space to watch and to see what exactly these improvements in GPT four are in the future.