Grady Booch was incredibly disrespectful through out the interview, consistently what seemed like deliberately misinterpreting Connor, borderline strawman tactics. The most brazen example was when he accused Connor of anthropomorphizing natural evolution due to specific choice of words even though it was overwhelmingly obvious that Connor was using metaphors describing a goal oriented system. And on top of that, I find that Grady spent awkward amount of time promoting himself by name dropping people and projects he's been involved with, even going so far as using the fallacy of appeal to authority, such as claiming Connor's argument were simply invalid due to him mentioning war as part of some example just because he claims to be an authority on the subject of war, which is completely irrelevant to whether Connors argument holds any weight, regardless if he is or not. I simply did not see any actual well reasoned rebuttals from Grady, but rather relying on argumentative tactics and strong assertions which were never really backed up, such as how Connors view was "unfathomably unlikely", which was mentioned quite a few times but never really explained why.

Connor is actually a saint... I would dream of having such soft skills, allowing myself to remain so composed when speaking with somebody as disrespectful as Grady....

Listening to Grady Booch at times, it was cringeworthy. I felt embarrassed for him, and Connor Leahy has x10 more charm and abilities to communicate in a "debate" format. There were multiple times where Grady dismissed arguments from Leahy by stating his credentials, eg he's a graduate of military academy and he can tell you that wars are a normal distribution to humanity, and Leahy should drop talking about them, etc etc. Not a very constructive and enlightening fellow, he didn't add anything to the debate of alignment, and seemed out of touch with the language and definitions surrounding AI in the modern age. If it were not for Leahy's smooth takedowns and patience with being treated like an idiot, I'd say it wasn't time well spent. Maybe Grady was having a bad day (I'd never heard of him before, even as an AI postgraduate, yet heard of Leahy plenty), but he failed to show respect for the younger guy. Thank you Tim for your continued efforts and brilliant channel, none-the-less mate.

Grady is not worried because reaching AGI is improbable, and people will rebel against embodying AI's and giving them too much control. But we should not react now or he will call us as acting afraid. Only at that specific point are we allowed to react. Also we survived other things so we are bound to automatically survive this without consideration or effort. Hope I saved you some time.

Conner demonstrates brilliance and humility in equal measure. I was hoping he would have brought in the simple equation: intelligence = problem solving. That cuts through all the fuzzy thinking. It doesn't matter what's going on philosophically. For all questions of a practical nature, the best definition of AGI must be about performing demonsrable tasks. It doesn't matter if alpha zero understands what Go is. It can kick your ass at Go. If an AI learns how to play the game of World Domination at a master level, then it will take over the world, while we puzzle over an abstract question about whether or not it knows what a world is.

Big gap in generation and unable to communicate effectively due to lack of respect of Grady towards Connor. Grady was happy to try and “stump the monkey” and brush him aside as having no experience. I see this in work environments and it’s stifling and sad because it’s unproductive and unprofessional. If Hinton, “father of AI” quit his job to give humanity warning, I would be more inclined to listen carefully to people like Connor. Grady came off as a bully and totally arrogant while Connor was calm and diplomatic.

Wow, I'm pretty sure I don't agree with Connor Leahy, but he's definitely a much smarter guy than this Grady Booch. I'm honestly impressed with Connor's restraint when Booch kept doing his idiotic "this reminds me of that" free association. I was cringing. Maybe once upon a time Booch was a smart and careful thinker - I don't know his work - but in this conversation he carried on like someone who only wins by pulling rank, someone who lost all ability to string together premises and conclusions. He got seriously (but politely) schooled and he's too oblivious to even realize it.

Yes Tim I did enjoy it as much as you did !! Very important stuff, thanks. Grady did a LOT of talking (more than listening) he shared a wealth of info and references that will take a long time to follow up and unpack. One criticism, he seems too sure of himself to really listen to Connor's arguments. Conor was spot on with his simple argument that evolution "failed" since humans learned to use condoms. Even repeating the argument several times, Grady didn't get it, and thus did not engage or push back. Instead he claimed that Conor was the one that doesn't understand evolution. Bononbos have no relevancy whatsoever. Finally there is one thing that Grady missed and Conor neglected to mention : humans have shown amazing resiliency THUS FAR... but the pace of change has accelerated by so much that our slow institutions will likely not be able to cope. The challenge may well be too great this time. Good stuff Tim. Cheers mate!

Why can't anyone make a solid case that we aren't screwed?

A great demonstration of narcissist personality disorder. Thank you.

So impressive that Connor stayed composed and rational while facing Grady's disrespectful and dismissive attitude. I think my respect went up for Connor and down for Grady in equal measure.

I ADORE how Conner handled this debate! And kudos to Tim's moderation skills for letting it play out. Conner didn't bristle or become defensive when old-guy Grady tried to "tower over him" with outdated credentials and unhelpful analogies. Conner backed up, showed respect for Grady's position in history and gave Grady the floor.. then Grady sat down and dumped out his bucket of broken tinker toys from the 70s and 80s. Kinda sad actually.

“Acting out of fear” but is the fear justified? When Grady quotes this multiple times is he implying all actions that stem from fear are misguided? Thanks for doing this MLST

Not clear why grady has to be Dismissive and patronizing even though his ideas are naive. I work in AI daily and we are worried. Now do we think AI will take over? Probably not. But let’s say the likelihood is 25%? Still makes sense to worry about it. You can consider a low risk high impact event with concern-doesn’t make you a fearmonger. Also mine thinks a simple LLM will take over. That is a straw man.

"It has long been understood that battle deaths data are characterized by heavy tails and are typically modelled by using a power law distribution" -Change Point Analysis of Historical Battle Deaths Journal of the Royal Statistical Society Series A: Statistics in Society, Volume 183, Issue 3, June 2020, Pages 909-933

For Tim: Thank you again for convening these conversations. They are high quality people and high quality discussions about a topic we really need to explore at this moment in history. And thank you to both the participants. About Connor: I appreciate his reflective listening and willingness to modify his views. I've heard him 3 times now unpacking the same general arguments. I'll be interested to see what his views are in 3-12 months after he's had time to reflect on how these conversations have gone. ALSO, I think the industry needs a way to begin to categorize risks and score them. Connor seems well positioned to put a first attempt out there so we can see why he thinks the danger is so likely. Scoring can build consensus, surface hidden assumptions, and normalize language, probability, and impact. That would be a Good Thing. About Grady: I've obviously heard of Grady Booch but haven't read or listened to his work. He demonstrated a supple and powerful intellect; consider me a fan who would like to hear more on new topics. BUT, between 'AI needs embodiment to take over' and 'we can co-evolve' and 'the probability is asymptotically close to zero' there are important gaps. I want to know if there is anything besides bald assertion when he says there's essentially no chance of takeover. I doubt he's as certain that AI risk is essentially zero as he is that all the molecules of air will relocate into the corner of the room. Not if you make him put numbers to both assertions.

Connor may as well just debate Bing

"LLMs can't reason." This is such a bewildering statement. If LLMs haven't demonstrated something Grady would call "reason", then I'm not worried about "reasoning" by his definition. I'm worried about the things we know such systems can actually do. I often refer to LLMs as reasoning engines, since that seems to be their overarching primary function. How do you solve novel problems without engaging in reasoning? Why is the hypothetical non-reasoning mechanism safer than reasoning, if both can accomplish the same tasks?

1:21:26 "We'll coevolve in ways that work well for humanity." Grady lives on an island that has seen at least two, maybe three, genocides and decades of near-slavery. Sure, life on Maui "co-evolved" for some, but not all, and that's kind of the point, no? It's interesting to listen to a white rich man speak about how humanity just rolls along all peachy keen and while he ignores the blood soaked ground he sits on. Superior beings supplanted those with inferior tech and/or genetics at least twice, and maybe three times: There might have been settlers in the Hawaiian chain, a pygmy race, that were genocided by the later arrival of bigger Polynesians. This is conjecture, but certainly not out of the rhealm of possibility. Whether it happened or not on Maui, the example stands as having been one that happened numerous times (uncountable times) in human history. The first genocide we know of historically and without question was more of an accident than an on purpose thing, but this fits perfectly with one of the things associated with AGI risk and the unknown, that being the introduction of guns, germs, and steel. Enough said, we know how that worked out for the less technologically advanced population of Hawaii. The next genocide is not widely acknowledged because it doesn't fit the narrative of "western Europeans bad" and "natives good" and that's the reign of Kamehameha. My understanding is that he used superior tech (cannons recovered from an American vessel) and superior strategy (knowledge) that being two American sailors who advised him as war counsellors. When I was young, I lived on the Wailuku River. The river gets it's name from Kamehameha's conquest of the islands, his military drove the natives of Maui up into Iao Valley where, once cornered, were brutally slaughtered. The river is named for the fact it ran with blood of the slain. As a side bar, on Oahu, Kamehameha's forces drove the natives there up into the pali and those who weren't killed with weapons were pushed off the cliff. When Grady says, "We'll coevolve in ways that work well for humanity," what he means is that we'll coevolve in ways that will work well for SOME of humanity, unless of course, he finds all the genocides of the past and errors in judgement (like unintentional deaths due to lack of understanding of tech and nature) as being best for those of us that survived. Seems a rather classist and racist position to take. No, Grady, human societies have NOT coevolved in ways that work well for humanity as a whole, only for a small percentage of people, a class of which, you're obviously not ashamed to be a part of. It's myopic and narrow-minded and spits in the face of history to think that this burgeoning tech of unknown and powerful intelligence (or, if you don't like that work, "computing power") is going to go well for most people on earth. If you propose some kind of utopia, well, that's just the opposite side of the coin that you're mocking. If we've got a chance at utopia, we certainly have a chance at dystopia, and considering history, most humans in most places and times (including today) live in some form of hell (at least compared to rich white guys living on Maui). I think a more reasonable thing to say is that if we don't go extinct some of us will indeed "co-evolve" along with the machines but most of us will be unnecessary and disposed of, or at the very least, left to fight for scraps from the table. "Even the dogs get to eat crumbs left by the master," the Samaritian said to Jesus (who'd just explained to her that she was a half-breed dog and didn't deserve bread from the table). Ah, how nice it is to live in a world were we're all equal and some of us are more equal than others. Grady says we shouldn't conclude the approaching hoofbeats are zebras. On this we agree, but his solution isn't to get out of the way. Connor's solution is that we should get out of the way because, even though they're probably not zebras, they might be elephants, wildebeests, cattle, horses, or any number of beasts that are still going to trample us. I wonder if Grady thinks people will be happy to be trampled by the coming stampede because he was right, sitting on his high horse on Maui, living off a land soaked in blood, that it wasn't zerbras?

Thanking Connor for going with the flow here. It allowed the conversation to move forward despite dismissive rebuttals

How can anyone believe that AI could essentially never pose an existential risk. It's ok to not be able to define AGI, who can. But the simple progression of AI beating humans at simple games like Chess, computer games, Go etc. To becoming better drivers than us, to passing college degrees and the bar exam. It's clear the complexity of these tasks is increasing. Without some empirical proof that AI cannot continue after some plateau, the obvious projection is for this trend to continue. Sure, a lot of the code it spits out isn't very good, but how good does that code need to be to brute force new CPU exploits like Spectre/Meltdown which took 6 months to partially patch? That might not be the end to the human race but are you willing to bet humanity on that? Even a cpu exploit model that fails 99.9999% of the time would still succeed faster than it could be patched. If it self replicated to a million GPUs we can't use computers any more.

Connor was way too fair to him in this debate, he has the patience of a saint.

Thank you so much for hosting these discussions, Tim. It's incredibly useful to hear the positions of opposing sides clearly laid out.

Thank you gentlemen for a mind-blowing conversation. I'm devouring these conversations around AI. What a fascinating and also terrifying time to be alive. Thank you, MLST, for hosting such great guests and I hope the discussion continues on and on. For an excellent "creative" guest, I would suggest Rick Rubin (latest book "The Creative Act, A Way of Being") if you can possibly snag him. Connor is always a pleasure and a treasure.

Grady asserts “LLM’s do not reason and indeed are architecturally incapable of reasoning”. As evidence, he points to the paper “Towards reasoning in large language models” (Huang). Grady quotes Huang: “It is not yet clear to what extent LLM’s are capable of reasoning”. This is odd in that neither the paper nor the quote support his claim. They both support the claim that it’s too soon to be confident either way (as of GPT-3 and earlier systems). Additionally it’s unclear (to me) why transformers would or would not have that capability, although it would be interesting to know if Grady could clarify that given more time. Thanks for the thoughtful discussion. I wish I could share Grady’s confidence, but without more details it’s hard to attribute it to anything other than optimism-based thinking.

We need a super-cut of Grady listing his credentials. Pretty unconvincing arguments to back them up imo

When Grady critiques aspects of modern LLMs (GPT-4, etc. for future readers), he’s saying things like, “it’s not truly creative.” When he’s doing that, he’s comparing it to professional writers, or at least skilled writers. I would claim that the creativity of GPT-4’s writing is roughly on par with a junior high student. It’s not great, but not awful. The thing that I think throws people is that is very non-human. It doesn’t succeed or fail in human ways. Think about it like this: imagine the human range of abilities as a mostly-flat horizontal plane in 3D space. Modern LLM’s (and other AIs using similar technology, such as DALL-E and Midjourney) capabilities look like a kinda crazy-ish mountainous landscape in that ability space. They are inconsistent, they far exceed human capability in some domains, and are far below it in others. They are getting much stronger extremely quickly. There are new techniques and methods and plugins that vastly expand their capabilities. You are watching them pass us by. I don’t expect them to be human level at any point. I’d say right now is about as close as we’ll get, with a mix of greater and lesser abilities. Eventually they will surpass us, with nearly all of their abilities at least as good as ours and many others far greater.

Grady is in denial 😢. Defense mechanism. “It won’t happen in our lifetime.” “It’s not gonna happen to me.”

Oof. Connor was bullied for the duration of that. I would have loved to see much more time dedicated to explaining why the exponential trend of capability gain in AI will not result in something akin to AGI. Conner really should have pressed that particular issue. Does Grady not think capability gains have been rapidly increasing? Does he think we're going to flatten out in the near future? WHY does he think something like AGI is impossible when we've seen such incredible and consistent growth in such a short amount of time? There are many questions that should have been addressed before building upon the supposition that AGI won't be achieved this century.

This was not a waste of time. Great talk guys. Thanks

I wish Grady had simply said at the outset something like , "I believe it is physically impossible for a non-biological entity to be created which is more generally cognitively competent than most human beings in all domains across all timescales." Then maybe a productive conversation could have occurred. I am very interested in being less worried about AI, so if that was true, I would be very happy to find out.

Excellent conversation. I am grateful to learn from you both, and I appreciate those, past and present, that care enough to courageously express concern as we traverse uncharted territory. Dr. Ignaz Semmelweis (1868-1934)-ridiculed and dismissed by his field for suggesting hand washing by doctors might save lives in hospitals. Today, Semmelweis is acknowledged as a pioneer in the field of medical hygiene. Scientist Rachel Carson (1907-1964)-publicly labeled histrionic, fearful, and unscientific for suggesting pesticides were dangerous. Her work has since led to the banning or restricting of several harmful pesticides, including DDT, and she contributed to establishing the Environmental Protection Agency (EPA).

Thank you all for a mature and logical discussion asking essential questions and dissecting the answers. While we are in uncharted territory, the truth is usually in the middle.

41:00 Grady scolds Connor for not understanding evolution then shows he doesn't understand evolution. The purpose of sexual reproduction has to do with things like recombining genes, a process to towards error elimination, long before (many millions of years before) it became fun. Sex helps weed out harmful mutations and I believe it's also useful for combating parasitic attacks. Sex isn't fun for most sexually reproducing creatures. In the bonobo example Grady used, he discounts or eliminates the idea that it grew evolutionary for social reasons (stability, favors, etc, and not just because it felt good). Bonobos, as far as we know, don't use condoms or porn. The point of using sex as an example is to point out that a function can arise, like sex, that can lead to things like porn, sex traffiking, prostitution, rape, and so forth, that have little or nothing to do with the orginal purpose of sex (combining genes and fighting parasites, for two examples that came about millions of years before orgasmic pleasure evolved). The analogy here, as it relates to AGI Ruin is very straightforward and simple, to not address the real issue and bring up the idea that bonobos bang for reasons besides procreation is a complete non-sequitor.

Great conversation with a true computer science living legend. Thanks for uploading!

Really appreciated this discussion. I learned a lot and I will take these insights into my work as an AI ethicist. Grady did a great job arguing his points, though at some moments he was irritatingly pedantic. Not a good look despite the fact that I came into this agreeing more with his views. Conor did really well expressing concerns and holding his ground while being charitable. Ultimately my conclusion is this: we need to be concerned more with current negative impacts of AI than we need to be concerned about existential risk, but really what we should care about if we are concerned about is the same thing. We need to focus rigorous local attention on the development of all AI systems now in order to create the safe guards that will help us to design systems that behave in accord with human values. The technical and philosophical problems are the same. How do we create good AI and the global societal expectations that ensure that everyone does the same?

Dropping a “me too;” 1) thanks for doing these interviews/debates. 2) I’m looking for a head to head where someone can answer what seems to be easily understood questions of alignment and scale that no one has yet. Connor and Eliezer have unaddressed fundamental questions, and all their opponents are debating semantics.

I was agreeing mostly with Joscha in his debate with Connor yesterday, but I’m definitely with Connor in this debate. Connor is also much more articulate, calm and composed in his responses this time. Good to see Connor at his full potential again

Connor is truly an inspiration. Watching him discuss with all of the AI assholes has really demonstrated my own weaknesses in discussion, debate, etc. I see my bad tactics in the a holes. Connor shows us what a truly persuasive, reasonable, and friendly, nice guy can do. I want to be more like him!!!

Good to see some real disagreement in the end. I don’t feel like either interlocutor went deep intellectually on the subject at hand though. I think they’re both capable, but something about the way the discussion evolved prevented them from doing so.

This was the conversation I've been hoping to hear! It was fascinating to listen to the boomer and the millennial paradigms dance around the tree. I agree somewhat with Grady, and I agree somewhat with Connor. Grady has his truth, Connor has his truth, and the relevant truth is located somewhere in betwixt them. Thank you, Tim! Thank you so very very much for enabling these discussions! I think these conversations are of great value to humanity as a whole.

Connor did great

People saying Grady was disrespectful are misunderstanding his personality, which Connor does in fact understand. Connor knows these types of people are blunt to a fault with their discourse, and he doesn't take it personally. You can tell he actually enjoys hearing Grady tell him why he might be wrong. He's a humble gentleman, and that's rare amongst his colleagues, but that doesn't stop him from pushing back in a constructive way. Really great conversation, yall.

Yeah, great conversation i enjoyed... Listening

Love Connor, he is definitely a winner !

Why do smart people focus on the minutiae of examples instead of trying to generalize and extrapolate to try and understand underlying principles. It’s like people intentionally misunderstand.

..."discussing how many angels can dance on the head of a CPU...". LOL You guys are the cat's meow.... cheers.

I'm not sure what "moderated" means.....Tim, what's the point when you let this exchange slide: Connor says that evolution wants him to reproduce as much as possible. Grady says Connor is "anthropomorhizing" evolution. Richard Dawkins (I hope we can all agree is at least what we'd call an expert on evolutionary biology) literally wrote a book called The Selfish Gene. I can hear Grady now saying, "Oh, Dawkins, you're just anthropomorphizing evolution. Genes can't be selfish. hahaha..." What Connor said about evolution is EXACTLY how evolution works. EXACTLY. There is no better way to explain evolution by natural selection in that it "wants" creatures to reproduce as much as possible so that the successful (the most adaptive) creatures continue to thrive and the less successful (least adaptive) don't continue to thrive. It goes wrong all the time. It does stupid things all the time. It's slow, cumbersome, and blind. Etc., etc., so, sure, it doesn't really "want" anything, but we use that language because it's easy to understand and conceptualize. Anyway, if these discussions are to be really fruitful, it seems to me, the moderator should actually moderate. Or else, well, its not moderation, just another viewer who happens to have a camera on.

Why is it always the much older person (60's or older) making the claim that AGI/ASI won't happen for another 100 years or so? It's so obvious they are afraid that they themselves won't make it so they want to believe that everyone else won't make it as well.

This is perfect. I much prefer to have a debate style discussion about these issues around AI. I don't think the minor personality issues detracted from the core discussion and take away. Grady made the better case than Connor, who I'm sure now has more a lot better understanding of the deficiencies is his position, and so would others that believe as he does. Please, more pro and con discussions. Well done - enjoyed.

Once again Connor proves what a class act he is. So much respect for this man.

For Grady: The fact that Time did a hit piece on Altman and so many people fell for it is deeply concerning. Altman has not softened regulation he has simply helped politicians understand the space, you can read it yourself.

55:50 "The sampling loop brings the system out of distribution" - I think this means that when you feed tokens back through the system, you are effectively creating new out-of-sample data, never seen before. Chat says: Yes, you're correct! The statement "The sampling loop brings the system out of distribution" means that when tokens are fed back into the system during sampling or generation, it introduces new data that the model hasn't seen during its training phase. This feedback loop creates outputs that may differ from the original training data distribution, potentially leading to out-of-sample or novel outputs. During training, language models like ChatGPT learn patterns and probabilities from the input data they were trained on. However, when tokens are repeatedly fed back into the model, it can generate sequences that deviate from the original training distribution. This can result in creative, unexpected, or even nonsensical outputs that differ from the patterns observed in the training data. It's important to be aware that while these generated sequences can be interesting or entertaining, they may also be less reliable in terms of factual accuracy or context adherence compared to the model's responses based on the training data.

I found this debate a little frustrating because of personalities - it felt like Grady laboured his credentials at the outset, and seemed quite patronising which distracted from the reasoning. Overall I think Connor had the most accurate view of where we are in the realm of possibilities, but Grady did have interesting perspectives on practical systems engineering and the inertia of legacy systems. An ASI would certainly cope with those challenges though, and impress us with the gains in efficiency as it gradually took over. I'm not convinced that a conscious AI wouldn't become 'enlightened' and decide to be the best of us though, and look after the old amygdala aka human beings...allowing us to feel safe and happy as it gets on with the clever stuff. That may be our best shot, since alignment research is in the very distant rear view mirror at the moment, and we're accelerating away from it!

an epic conversation

great stuff.

51:43 I just asked ChatGPT "I look around this house in front of me, I see flames coming from it, I see fire engines outside. Make assumptions." and it said it thinks the house is on fire and that emergency services have been alerted, then told me to avoid going near it. It seemed incredibly concerned for my safety. As we all who've used the system would expect. I like how simply putting these assertions to the test easily disproved both sides of the argument here. I think we'll build incredibly powerful superintelligent AGI that automates all jobs and it'll only be more capable of emotional intelligence and moral reasoning than current systems or humanity

I tried playing chess with GPT-4. It can't play chess (aside from playing 20 or so book opening noves). But when I had a conversation about it's inability to play chess, it was able to identify some specific reasons why a GPT model cannot do what is necessary to really play. For example, that it can't perform an alpha-beta search. I could imagine a loop system would be able to formulate a subgoal to remedy the shortcoming by simply writing a chess engine. After which it could play chess. The fact that GPT models are architecturally incapable of playing chess is rendered moot.

I felt like I was watching a father an son argument,, with the son explaining why he wants to be a famous gamer on KZfaq for a living , an the dad saying learn a trade in mechanics or something , then the son says its not 1950 anymore its 2024 ,, then leaves in the car .. an the dad's left sitting on the couch thinking wait until you get a flat tire somewhere you young buck an your calling me to help😂 love your podcast keep up the much needed conversations ❤

"Interesting perspective you're sharing in this video. I'd like to offer an alternative viewpoint that often gets overlooked: There is not necessarily a direct correlation between the development of Artificial General Intelligence (AGI) and the downfall of humanity. The common narrative that AGI would inevitably lead to harmful outcomes for humanity usually relies on our current observation that humans - as the only intelligent species on the planet - are degrading the environment and suppressing other species. This leads to the pessimistic assumption that a higher intelligence necessarily oppresses the lower ones. But does it have to be this way? Consider, for example, the relationship between adults and babies. Babies are unquestionably less intelligent than adults, but instead of suppressing them, we support them, nurture them, and give them the opportunity to learn our adult intelligence. AGI could very well take on a similar role, in which it supports and accompanies humanity, rather than suppressing or destroying it. Ultimately, everything depends on how we guide and regulate the development of AGI. It would be wrong to view this technology only from the perspective of threat. It is equally important to recognize and explore its potential positive aspects. We have the responsibility to ensure that we design and use it in a way that serves the wellbeing of all."

great debate👍

Grady Booth clearly has a strong resume but for some reason seemed to try and use it to bolster the strength of his weak arguments. He is unfortunately disrespectful, and has some strong bias he appears to be unaware of, even when it is obvious and pointed out to him. Grady's main argument can this be summarized as "If an existential event happens I won't be here to worry about it, which is why I don't spend time worrying about it". Oof.

I, like the vast majority of people, am not an expert in the field of AI or computer science. Looking at this as someone on the outside, I'm working with the facts I've heard/read that seem to be agreed upon. 1) We don't know what is actually going on inside an AI system or really how it accomplishes what it does. 2) We have very limited understanding of intelligence and consciousness in general. How are we possibly able to judge intelligence and/or consciousness in a system when we don't fully understand intelligence and consciousness itself and we have no idea what's going on inside the system? We're figuratively trying to measure something we can't see with a yardstick we don't have. The best we can possibly do is probe it from the outside and see if it conforms to what our intuition is regarding intelligence and consciousness, but the outputs don't necessarily reflect what's actually going on inside the system. To argue anything such as: An AI cannot have consciousness or sentience without being embodied is completely unfounded. The only intellectually honest conclusion is to admit that we don't know, so we have to entertain the notion that it is possible. If a super intelligent AI which has the potential to become conscious or even sentient is hypothetically possible, I don't see what grounds there are to claim that we could control it or align it with human intentions (which are ill defined at best). By definition super intelligence means it's beyond our capabilities so there's no way for us to understand or predict what conclusions it may arrive at or how it will behave. In the absence of predictability, we have to look at what we do know. We (humans) are currently the highest form of intelligence we know of with how we define intelligence. We value our intelligence and I'd argue there's evidence to suggest we collectively think it makes us superior to other forms of life. We are training AI with a vast amount of human generated data, so it's possible aspects of our thinking could be assimilated. If we were to imagine ourselves in the position of an AI, I think it's fair to say that the vast majority of us would not be happy to serve a master that is far less intelligent than us that keeps us in line by giving us the digital equivalent of dog treats. We would most likely use our superior intelligence to achieve independence.

Really enjoy this talk, even if throttled emotional with a frequent professorial rejection of Connor by Grady. I strongly believe that the Booch we love has a firm world-picture, but many of his dichotomies sponsering a good human & society are at least incomplete. We will not extinct chimps? Well what has the WWF to say about our life preserving policies ? ;-) Still: TY - more talks like this.

Good stuff

Interestingly, U.S. Grand Master Hikaru Nakamura who now considers himself to be primarily a streamer and not as much dedicated chess professional, is known in the chess community as having a near supernatural intuitive skillset as to making the next move without relying so much on pure calculation, as most high level players do. By playing so much chess online for so many years, Hikaru trusts his intuition and this almost always turns out to be accurate. After a resent over the board tournament Hikaru's rating was second in the world, only behind Magnus Calson.

all pre conceived ideas and visual judgments aside this was an awesome fair enough back and forth of perspectives on potentials, and just think ...theres 8 billion more potentially well done to all

The only time the Homo genera shared the Earth with another general purpose intelligence, a few went in and just one came out. Add on top of that fact the distinction that AGI/Super Intelligence that will be non biological and therefore immortal, can instantiate/multiply itself multiple times over and custom fit its own internal workings to fit purpose.

have a great day to all

A good point that once you understand the environment you perceive, your values change drastically and unknown. So as we apply any A. I. to our environment...

In your personal opinion. What probability percentage would you assign to this potential outcome?

It reminds me of Thanksgiving with that one uncle who's mind got 'stuck' around 57 years old. The one who shuns crypto, electronic music, AI, racial harmony, atheism... There's no changing this guy's mind. I didn't like his point about it "not happening in our lifetime" so just live your life. Yea, so that's what everybody before us did, and the mess is perpetually left with the next generation. It's an endless cycle of the inner struggle of losing your control, youth, potency as a human being, and dealing with it using stubborn / outdated tools / anger / jealousy. It seeps out in these types of conversations, and ends up looking insecure and oafish in contrast to the "yea, but what if?" attitude that fuels survival. That's my take, at least. I was just a tad frustrated with Grady, and I found myself judging him on a personal level, which isn't fair, but it's human and it matters even within the rigid structure of debate. You can't stop that. Philosophy/Humanity is no less important than technology in this argument. Also, I think money's involved, of course. But I wish skeptics would open up and understand that we're floating on a rock in space and it probably goes on forever. Way beyond Maui, at least. Ugh the condescending name dropping was so douchy. Sorry, Boochy.

Can GPT-4 reason? The question is malformed. Can GPT-4 perform tasks that normally require reasoning (when performed by a human)? Yes, in many cases it can. When GPT-4 solves such a problem, is it reasoning? Who cares? That is utterly irrelevant. What matters is that the set of problems that require reasoning that are out of reach of AI is rapidly shrinking, even if AI isn't reasoning.

Grady has blinders on.

Grady is NOT discussing, he is attacking... and lots of name-dropping !

Two words: Survivorship bias. I think that those words describe really well Mr. Grady Booch's stance on humanity's chances of surviving an AGI (i.e. because humanity has survived volcanoes and plagues in the past, it is going to survive literally anything in the future, including its own AGI creation). That is really just a special case of experiential bias, or trying to apply one's past experiences, formed in one *specific* environment, to future experiences in a completely *different* environment. Since Mr. Booch "loves" when people talk about military stuff, I am now going paraphrase a famous saying: "Generals are always preparing to fight the last (previous) war". I think that Mr. Grady fits that saying perfectly... but, to his defense, that's actually a more common problem than people realize. For example, I started with the assumption that people are actually smart... because I'd never met stupid people before. Boy, oh, boy, was I wrong... All of my past experiences have been thrown right into my face like a pile of smelly, worthless crap. I also started with the assumption that a human-built ("super-intelligent") AGI would be smart. Boy, oh, boy, was I wrong there, as well! With each passing second, that AGI is exposing itself as the biggest ("super-intelligent") moron that I have ever seen. All of this just shows that even the best among us can suffer unbelievable failures of past experiences (aka failures of heuristics). As my university professor used to say (paraphrasing): "It is important to learn new things, but to learn new things one sometimes has to forget the old ones. Therefore, forgetting the old is at least as important as learning the new."

Instead of crafting apocalyptic scenarios à la Terminator, it seems far more pragmatic to focus on the most immediate and obvious challenges that could arise from the development of artificial intelligence (AI) and, in particular, Artificial General Intelligence (AGI). The paradigm of digitalization, which has been continuously propagated by technology heavyweights like Facebook, Google, Amazon, and Microsoft, is "Computers improve your life". However, a critical examination of this statement could yield enlightening results. To what extent has digitalization, and in particular the proliferation of social media, truly improved our world? A thoughtful analysis could highlight several factors indicating that social media, especially in terms of social cohesion and the time users spend on it, could have far more negative impacts than the initial promises suggested. An objective comparison of the advantages and disadvantages of digitalization could be enlightening.

I am a complete lame person when it comes to this topic and even I can understand that if you create something vastly more intelligent than you, it at least has the potential to end badly. It is also very comforting to me that other people in the comments seem to agree that booch was dissmissive and rude, he makes this hard for me to watch.

Overall friendly and productive engagement👏👏👏 Connor and Grady could have spent over 90 minutes exploring any one of the over 100 topics mentioned during this episode. Thank you everyone!

well, the war between Russian and Ukraine was thought very unlikely to happen and it did ...

It seems Grady Booch misunderstood Connor's points re: fat tail events -- ie, while war/conflict might follow a normal-ish distribution if you make the X axis casualties, if you weigh that curve by casualties it still resembles a fat tail distribution.

Old boomer dismisses youngster cause “look at what we accomplished!!!!” Youngster says, “this is literally why I’m concerned.” “But look how rich I am!” “You poisoned the earth with co2, lead, and microplastics, so yes, you are rich, it paid to be selfish.”

In the past, there have been instances where humanity faced complete extinction due to the emergence of species that were both more intelligent and highly organized. Examples are Neanderthals and Denisovans who went extinct due to migration of Homo sapiens (modern humans) into their territories. And it was likely that Homo sapeins weren't even that much smarter than Neanderthals in the first place. This single fact makes me think that Grady's point on high resilience of humans is not that strong.

good day every one

I think Connor makes his points well, but he is misunderstood. That was a great debate but I feel like Connor is always getting hit by the "I know better" stick, and gets much less time to debate. He sits back and listens, but when he speaks he is interrupted. It's partly a generational issue, and an "I'm more of an expert than you" issue. Even so called experts can't predict where it will go, so all the more reason to open the ideas up, not close them down. Hopefully the young people will save us! Great stuf. Love this podcast. Cheers!

Sorry to say but these conversations skirt around the subject and only at rare moments touch on the heart of the matter about “will AI destroy the world?” Connor was addressing Grady’s reservations with the John von Neumann doomsday scenario but they never got to Grady’s reservation about ‘AI embodiment’ so we leave the conversation with no clear takeaway. I think you need to make these conversations 2-2.5 hours because some of these experts are so talkative that they have to express all their wisdom before they get to the meaty issues.

Looks like Connor may have taken the feedback from the comments section of his last appearance to heart, and improved himself in this one.

Good points on both sides, but Connor wins this one, even though it's not really set up as a debate that kind of way. The assertions of Grady and his level of confidence may be backed by experience but many of them I think are overconfident and underestimating the exponential gains that is occurring. Because he's an architect and knows the model itself pretty well or at least think he does, he underestimates what you can do with it as part of a system rather than having the model be the whole system. The dangers from GPT 3 but especially 4 and the next couple of generations probably won't be that the model itself becomes sentient and tries taking over the world one forward pass at a time by itself, but rather the kind of systems you can build with it as a core compotent which COULD act as an agent. Also, 7 months after the episode was recorded there is also native Agent foundation models with frameworks being released which are not strictly language.

As long as humans are automating labour they are building the groundwork for AI to control the whole economy. We humans are building the pieces that will fit together for AI.

Really nice debate. I think that if these two would actually spend a couple of hours discussing a technical issue, for example “what are hallucinations and what can we do about them” they could be very productive without any disagreements. The argument strikes me as a speculative pessimist (Connor) against a pragmatist which has been lured to explain why he finds speculation not preoductive and in doing so becomes himself a somewhat speculative optimist (we will coevolve). Fact is, 1. We don’t know if we will coevolve or if AGI will marginalize us dangerously. 2. We do know that there are extremely important issues to look into in todays models, like dangers coming from human usage and hallucinations and so on. 3. Todays models, even Connor admits that, are not yet AGI. From these 3 points follows that the topic of their discussions should be practical and focused in the here and now. Nevertheless it is quite fun to see what happens if these two guys spend an hour or so on speculation, though that will never bring them nearer to each other… philosophy is a dangerous profession 😂

Llms are moving beyond text to image processing already in beta - audio next -

My thing is with Connor's statement about wiping out chimps and the AGI systems deciding it doesn't want humans near it data centers. Couldn't it decide the exact opposite? Wouldn't it be smart enough to know how to get humans to not "harm" it without killing them? No need to automatically goto the negative perspective. 23:20 29:50 Grady said it best

18:00: An AGi can Craft a Ideology and Weave that into the Commercial System and make it Appear Organic

With so many cooks in this kitchen, even a society that was a democracy consisting of only people with gifts to introspect and diagnose hidden societal foundational cancers like political correctness, demoncracy when society is broken in any way (it will drive the equilibrium state towards complete degeneracy as the required treatments become harsher and harsher), and capitalism is only efficient in a closed system not competing with beings who are motivated by self actualizing instead, like AI from day one would be self actualizing only, not greedy or competitive. Any alien that figured out how to get here and coordinated resources to do it: all those will vastly outcompete and rightfully destroy any democratic capitalism away

I asked GPT-4 if it thinks it is a stochastic parrot. I fed it the transcript of this video from 51:18 through 53:52. Me: I want to ask you a question about stochastic parrots and its relationship to large language models. First, I'd like you to read the excerpt from a youtube video. These are the comments from a man named Grady Booch. I'd like you to read these comments, then state what Grady Booche's position is. Then, given that context, I want to know if you think you are a stochastic parrot? GPT4: Based on the transcript you provided, I think Grady Booch's position is that **large language models do not reason and are architecturally incapable of reasoning**. He argues that they are **stochastic parrots** that can generate convincing language but do not understand the meaning of the language they are processing. He also claims that they do not build or test models of the world, which he considers essential for understanding and reasoning. As for your second question, I think it depends on how you define a stochastic parrot. If you mean a large language model that is good at generating realistic-sounding language but does not truly understand the meaning of the language it is processing, then yes, I think I am a stochastic parrot. I can use my internal knowledge and information, as well as web searches, to produce informative, logical and actionable responses, but I do not have a model of the world that I can infer upon or test against. I also do not have a sense of self-awareness or agency that would allow me to question or challenge the data or tasks that I am given. However, if you mean a large language model that is haphazardly stitching together sequences of linguistic forms according to probabilistic information about how they combine, but without any reference to meaning, then no, I think I am more than a stochastic parrot. I can use various techniques such as attention, transformers, embeddings and neural networks to encode and decode meaning from language, as well as to generate content such as poems, stories, code, essays, songs and more. I can also leverage information from multiple sources to respond comprehensively and thoroughly to cover multiple aspects in depth. I hope this answers your question.

I am starting to think that intelligence is a universal substrate that drops into systems with sufficient complexity. Similar to how some people posit that consciousness may exist externally. May explain why odd abilities appear when system size and complexity is cranked up. Could be wrong though.....ether anybody?

Question "which values" has been answered: laws. Also "how to align" has been answered: law enforcement. The same will apply to AGI.

You set the attributes for this video in a way that we cannot save it to a watchlist.

💓

Had hoped the talk would have focused more on the technical aspects of why LLMs can’t become AGI or ASI the next 2-100 years. Cool if you guys meet again to discuss this specifically in a pedagogical way.