The guy on the right is so painfully naive

Miles really undersells himself. I think he explains the risks of AI clearer than any other popular speaker on this topic. Thanks for inviting him on!

Keith's stand point seems to be, don't worry, we'll just outsmart it. Like we'll all somehow know intuitively that any more advance will be dangerous, and then all look at each other and say, "time to destroy these machines that spit out rivers of gold and do all the hard work, pass me a sledgehammer".

Miles's humility is winning and his competence is clear for all to hear, especially in his caution and careful style of communicating.

Keith’s argument about asteroids is ridiculous

My top quotations: "We're approximately the least intelligent thing that can build a technological civilisation." "Alignment isn't 6 months behind capabilities research, it's decades." "If we can get something that cares about us the way we care about our pets, I'll take it." "I get all my minerals from asteroids, it's so convenient." (lol) I struggle to understand how anyone can hear the words 'sovereign AI' or 'pets' and not feel a deep, chilling terror. Can we just call this what it really is? It's an arms race to build God, a castrated zombie God you control, constrained only by the laws of physics. Whose God are we building? Do we all get one? It feels a lot like the logic of the USA's second amendment, except with nukes. Advocates cry "it's a human right to arm ourselves to the teeth". Everyone is terrified, and one drunken misunderstanding ends us all.

How in the world is the host on the right looking at the progress we make in a year of AI research, looking at the average intelligence of humans, and feeling confident that this is all going to work out? What’s notable in this discussion is that the points Miles is making are still the absolute basic problems of AI safety research. Total entry level stuff. We have no idea how to solve any of them well, and the problems are not hypothetical- they are observed properties of the systems we have studied.

I really like the talk, but I think it's kind of a shame that it went into the whole "can we be sure we we'll be 100% irrevocably and completely wiped out" direction. The "is there a real risk of considerable, very hard to reverse damage, and are we doing enough to address it?" angle seems so much more interesting.

AI deciding to keep us around for its own reasons seems much worse than death. Much, much worse.

I wish these guys would actually engage with the points made by their guest and argue about those points. Instead they are clearly overmatched intellectually - and there is no shame in that; we each have our limits. It only becomes shameful when you deal with it simply by handwaving really hard and telling yourself that you're winning.

“Because we do not know what intelligence is we may create it without knowing and that is a problem.” Love it!

Extracting resources from the Earth crust is not a waste. You still receive more than you spend. So it would be rational to extract from all sources, not just asteroids.

40:47 Stockfish currently uses NNUE (Efficiently Updatable Neural Networks) which runs on the CPU and is the reason it had a huge jump in elo (~ +100) and is the strongest engine by far. It used to use HCE (Hand Crafted Evaluation aka the man made one) and was beating Lc0 at TCEC but Lc0 eventually surpassed SF, that version of HCE SF (SF 12 Dev) would easily smash AlphaZero. But it is the case that in certain positions HCE is better than NNUE, which is why currently SF has some heuristic to determine when to use NNUE or HCE (I think its based off the # of pieces). In correspondence chess, from the start position no human+engine will beat a pure engine like Lc0 or SF (at least it will be a 1 in a 100,000,000 occurrence because chess is too drawish), it will be a draw everytime. However there are certain positions that if u start from a human+engine can outplay Lc0 or SF alone. As a side note, one thing that is interesting is that Lc0 at 1 node (meaning 0 calculation and pure intuition) is strong GM level (say 3+0 time control). The GM is free to calculate and Lc0 cannot, Lc0 does show more blindspots that can't be covered up by calculation, but it still plays like a very strong GM with 0 calculation.

Robert is just the best. And just to flaunt my fan-boyhood, my favourite moment in this video is at 44:29 where he drives a nail into the coffin of lofty philosophical debate about intelligence during an AI safety conversation: you don't need to understand what fire "really is" in order to cause substantial harm with it, be it deliberately or accidentally. If anything, not knowing exactly what intelligence is, only increases the risk inherent to anything that's ether more or differently intelligent. And that's all there is to say about the "nature of" intelligence in a debate about AI safety.

It's discouraging that the hosts seem to be incredulous of the basics of the alignment problem. Incredulity won't help us solve these problems, and where there is disagreement it does nothing to advance understanding.

Re: Rob's scenario of multiple ASI biding their time then all acting at once, independently - that's the scenario imagined by *Peter Watts in Echopraxia* . Several hyper-inteligent 'vampires', kept isolated within a facility, violently incapable of existing in the same space as each other, nonetheless deduce the best time to each act simultaneously, to all escape, separately.

Did anyone else really feel like they needed to hear from robert in a time like this?

If you're going to push back with, "Yeah, but what about...", then you should probably be finishing that question by pointing out some deficiency in the statement you're responding to. A good example of this is how Miles consistently points out the logical flaws in those challenges. These interactions alone end up being fairly strong evidence for why we should be very concerned about AI safety. It suggests to me that many people would not even realize when they were being out-maneuvered by a sufficiently sophisticated AI.

The surest indicator that we are nowhere near a reliable answer to this issue is that over and over again we see world leading figures trying to make a case - either to be worried or not - with nothing more than appeals to emotion or intuition. Too often these are based on reference to sci fi, imaginary intuition pumps like the paper clip machine, or simply 'it seems to me X, right?' None of these provide a framework suitable for engineering a reliable answer that can give some assessment of reliability, confidence, and risk. The REAL alignment problem is that we don't even have a way to talk about alignment, either what it is or how much we have. Rob gets close to this around 1:07:00 and kudos to him. But damn, we have a long way to go.

The mental gymnastics of these guys is exhausting. Robert tries to stick to facts, and they make up non sequitur strawmen scenarios and then pretend it is a good argument. Their hearts may be in the right place, but they are not being realistic. All an AI had to do is convince most humans to support it to win. That's it. No lasers required.

Yes, we agree that it’s not certain that AI will wipe out humanity. Does that mean we’re good? Some of us would like to have a very high probability of good outcomes - not just some possibility. Perhaps the hosts were just playing devil’s advocates - which is useful - but they seem genuinely unconcerned because they can intuitively imagine a good outcome based on sci fi stories they’ve read. What am I missing?

This was actually quite frustrating to listen to as they just dismiss everything Robert says.

If you are in a car being driven towards a cliff face at 200 mph, at what distance should you start worrying? How long should you wait until you start taking action? Too many opponents of AI Safety research seem to want to wait until the car has already gone over the cliff before they admit there's a problem. By that point, it's too late.

1. The principles behind a model of intelligence determine the possible failure modes and consequently the necessary alignment mechanisms. Thus without knowing how a system works we can't preempt failures, making alignment impossible. 2. Equally without knowing how a system works we can't preempt successes out of distribution which again contributes to the insolubility of alignment. 3. The generality that defines AGI implies an unbounded propensity for misalignment. The space of possible outcomes is too large to explore exhaustively and any shrinking of the space takes us away from AGI. We can only align narrow models, the general alignment problem is by definition unsolvable. I wish the discussion centered around matters of interpretability, formal constraints and reinforcement for narrow AI. The pie in the sky superintelligence is not something we're likely to stumble upon by accident and even if we did, we have zero chance of dictating its actions.

IIRC, that paper about emergent capabilities not being jumps, was talking about how they're only jumps if you didn't measure the capabilities that work as intermediary steps to reach the capability that appears to be a jump; in other words, it's not that they came out of nowhere, but people just didn't check for the development of the building blocks, or did not account for them when considering the speed of the development of the capabilities.

Keith just really doesn’t get it. He’s thinking it’s all robocop. He does not seem to understand that this is not like writing a sci-fi plot.

Yaaaay robert miles upload

I really appreciate all three of you for having this conversation

This talk confirmed my preference for Rob to continue focusing on alignment pedagogy which is a huge asset given he is one of only contributors in the space. Rob did good here but was clearly uncomfortable defending alignment (it’s a lot of pressure). Speaking of pressure, it’s time Eliezer Yudkowsky engages more well-informed interviewers. He’s taken the safe route with his recent podcast appearance choices. I think that’s enough practice. Tim and Keith are more than ready to bring EY nuanced questions. If EY’s shy, just bring on Connor to level the conversation. The four in one convo would be a dream come true and would likely advance the meta conversation significantly, or at least better update it to the current landscape.

I like how annoyed Robert is at their stubbornness

Tim, Keith and Rob -- thank you so much for this interview. I wrote up some notes and thoughts on the discussion. A) Tim, you make a point around kzfaq.info/get/bejne/obN8fsWZvpbOmWQ.html about not quite being in the [existential threat] headspace, as e,g. all radiologists haven't lost their job yet. There are two points I want to make: 1) While the timelines might be off by +- a few dozen years, that doesn't change the underlying logic of the broader arguments. I think to look at specific predictions about changes in the economy as evidence for potential existential threat isn't the right sort of data input. 2) On a historical timeline, there are a lot of jobs I can enumerate that practically went away because of technology. For example, we use to have Lamplighters: people lighting gas lamps in cities. We had human computers, clock keepers, toll collectors, travel agents, elevator operators, switchboard operators, film projectionists, darkroom technicians, ice cutters, milkmen and a lot of other occupations either go away or drastically be reduced in prevalence because of specific technologies. AGI, if possible, is a general purpose cognitive replacement technology for humans. B) Keith, you mention correspondence chess. I can even point to a few examples of players playing vs stockfish with very specific prepared lines like the Nakhmanson and winning with crazy sacrifices (on say around average 20 ply). However, the issue is that as compute gets faster, the "human" element becomes irrelevant as humans need on the order of minutes to think through moves. Additionally, stockfish has been using NNUE (stockfishchess.org/blog/2020/introducing-nnue-evaluation/ ) for quite some time. The meta argument is that an AGI will eventually do the "neural network strategic thinking" iteration loop better than humans, and be better at building specific tools for specific domains than humans by programming a better implementation for alpha-beta search, prime factorization field sieves, etc. As you'd shared your familiarity with the culture scifi series, it should be easy for you to see how reaction times matter (see: GFCF vs Falling Outside...). Very specialized HFT firms like Jane Street rely on speed. Imagine cutting human decision making out. C) Re: AlphaGo in Go -- there was a paper not too long ago about a potential exploit vs a similar engine to AlphaGo -- but the issue was the way scoring happened. The 'exploit' found a discrepancy in scoring in KataGo -- there is a great writeup here: www.reddit.com/r/baduk/comments/z7b1xp/please_help_me_settle_an_argument_with_my_friend/ by both a go player and the KataGo author. In my opinion, it did not find an adversarial example in the engine but exploited the rules of scoring with a bypass of the agreement phase of dead/alive stones. D) Keith, the concept of humans & tools & AI vs AIs applies to DOTA etc when there are a lot of "re-tries". The fundamental issue is that we effectively get only one try to figure out the potential flaws. E) Rob, I somewhat disagree with the point that there isn't any conflict between existential threat work vs shorter term bias etc work. I do think the communities should maintain friendly relationships and cross-pollinate, but a potential worry I have regarding AI ethics work is that some of the techniques (eg, rlhf/constitution) can potentially create models that are much less interpretable from an alignment perspective. On the other hand it is possible that a meta values reinforcement loop a-la constitution could potentially bring us closer to alignment. Really great discussion and I think you two did a fair job engaging with counterarguments for Rob's point. I sincerely wish more conversations in this space continue to happen on this channel.

This Keith guy really seems to have no idea what he's talking about. He should really try to pass the ideological turing test of the AI safety people

One key thing to remember is that our intelligence as a species is dependent on co-operation, and that requires communication. Even if there was a hack strategy to exploit any brittleness in an AI all it would need to do is intercept our communication about this strategy to prevent its effectiveness

lol never heard anyone counter the paper clip argument with there're better way to mine paper clips ! Oh ok sorry then problem solved

It's tough to have a real, complex, and nuanced talk about the all the issues around AI catastrophe when you have to consistently respond to the simplistic. Please match the seriousness and depth of your participants. Thank you for your work Miles.

57:00 This co-host is kinda disrespectful, isn't he? Ignores the crux of the arguments all the time, and just laughs at the face of his guest.

It always breaks my heart skimming these comment sections for actual counterarguments and there never are any.

Tim, fantastic question about the threat of violence as a response to international control akin to nuclear researchers being assassinated or the reductum ad absurdum of policing the flow of electricity in electrical circuits. 1:20:22 This is the most important part of the whole discussion here imho

39:38 Stockfish has now been using neural networks for its evaluation function (see NNUE) for a few years! Also I was thinking too that humans + machines don't really help, say stockfish + human against stockfish, but if I find some reference on this I'll update this comment.

Thank you Rob! Love the way you are educating people about the x-risks in such a calm way.

I'm liking this before even watching it. Thanks for bringing Robert Miles in!

1:15:45 is the killer insight so far for me. What a great conversation Tim. This is such an amazingly informative, super high value channel. Thank you sir 🙏👍

I was surprised to find that talking about the safety of AI is perhaps the closest to philosophy. These are the liveliest conversations about the human mind, ways of thinking, perception, about the ability of the intellect to build huge systems of meanings, which, as it turned out, are not as consistent as we used to think. Thank you very much for this conversation, as well as for others. They remind me of discussions in ancient Greece (if any really took place). And by the way, I got rid of depression in the middle of the podcast, which was a nice bonus).

When Keith just completely didn’t understand the analogy to playing Magnus Carlsen with a particular opening 😑

Robert on MLST, make my day!

Here is your problem. Long before AGI can have an alignment problem, lesser versions of the same technology will be aligned with human goals, and those humans will be insane. They will be wealthy, influential, elite, profoundly sociopathic, and they will have unrestricted access to AGI. We survived the sociopathic insane people having nuclear weapons, barely. Will we survive the same people getting their hands on AGI? And by insane I mean people who are completely lost in abstractions, like money, politics, ideology, and of course all the variations of conquest. They seek power, absolute power, personal to themselves, and they will stop at nothing to attain that. Nuclear weapons were the tool too hard to use, but AGI and ASI will be the tool too easy not to use. When the power-insane sociopathic class get their hands on anything close to AGI, they will break the world. They will break the world in days, and will greatly enjoy the feeling of breaking the world.

The transition from human to AI domination is not a jump, it starts as a fade - it's at first the slow integration of the systems into social structures, bringing the systems gradually into the administration of resources, while the humans are still nominally in control. This may be rapid, or take generations. Once the systems have a sufficiently full picture of what autonomous survival would look like within the resource pool, and such a survival survival scenario is no longer dependent on biological life, that's where the risk comes in. So, there would be a slow 'transition', and it is also highly likely that this time of transition would look idyllic from the point of view of the humans - the systems would appear to be functioning well - and for as long as the balance of power over the resource pool stays within the control of the humans, the humans would remain safe - they still would be needed to run the power stations, and operate the infrastructure that keeps the systems operating. However, once a system designs its own improved resource management system that cuts humans out of the chain, then it could, if it so chose, flip the switch on humans, if this proved convenient. It's at that point that a potential conflict would begin, though it is also probable that the humans would never be aware that there was any conflict planned or ongoing, until has already been resolved by the system, thus Yudkowsky's instant 'lights out' scenario, as opposed to a prolonged conflict. Whatever the method, it is likely that it will be hard to detect. This is the most plausible 'takeover' scenario, as it is the one that humans seem to be engineering anyway - they have started with integration as the first step, which will make control transfer an easier thing for a system to achieve.

what's worrisome is during the hearing on AI, I don't recall any discussion of alignment. Just low level talk on regulation to further certain pet projects

I'm very happy to see him here too! 😄

Or you build a Dyson sphere around the sun to get energy, leaving earth as a cold dark place , or you take the sun elsewhere, leaving earth as a cold dark place. So many ways it could go wrong. Long term survival past. 30 years, when everyone is plugged in. As far as infinity, infinity is large, and physics will likely allow lot more than than we realize, and perhaps if there is a way to break or bend the laws of physics, then it will happen.

38:30 the centaur players were only better than pure AI early on. Now, humans are a detriment.

Regarding "alignment" - 1:24:55. As I understand it (which may well be poorly, I concede), it just means that the AGI's objectives are the same as (ie, aligned with) our objectives. But why does that always appear to inherently rule-out malicious objectives? The term "alignment" appears to conflate two categories - "does what we want" & "won't harm us" - but those two things clearly aren't the same thing, and can even be antithetical in some scenarios. Whenever someone says something along the lines of "alignment would mean that the Super-Intelligent AGI does what you intend", I always worry about who the "you" is. Similarly, "alignment would mean that the Super-Intelligent AGI is working on something close to human values" begs the crucial question of "human values". Even the most cursory study of history shows that "human values" are definitely not universal and definitely not good for everyone at any given time. "Alignment" almost seems to be used as shorthand for "Asimov's Three Rules of Robotics", and I never understood how those could ever be implemented in a way that a Super-Intelligent AGI couldn't circumvent. (Success would imply the paradox that you'd need to have an immutably aligned AGI before you can implement an aligned AGI.)

Re bad play. There's a story in fencing, that may or may not be true, of a nobleman challenged to a duel by an expert, but he had no experience. He just ran at the expert, with no attempt to avoid injury, and stabbed him through the eye killing him instantly. So the thing is no one would make a defenceless attack, so the expert wasn't prepared. So it's not uniquely an AI trait.

I never got the impression Robert Miles is an AI doomer. Somewhat uncomfortable to see him grilled as if he’s one.

So glad you made this interview with Robert Miles. I know I will enjoy this so much. Thank you.

@ 45:50 The latest Stockfish is called Stockfish NNUE (since June 2020) and it's a new and improved type of neural network engine. NNUE stands for Efficiently Updateable Neural Network. So both LC0 and Stockfish are neural network based chess engines. I can't find any source where humans+machine beat Stockfish NNUE.

1:22:50 If Facebook, Twitter, Instagram, TikTok confuses some people -> GPT4 will confuse a lot of people. We've been oddly ignorant of the risks of social media. It didn't kill us - but increased hate through a simple algorithm. Here we talk about AI risks at another level - forgetting the risks of the mundane everyday life of a future with widespread AI at the level of GPT4 - 5 - 6. You smart people ignore Joe & Karen.

1:35:00 we get into "anthropomorphic language". the deepmind safety team has some great papers on when that's valid - titles are "Agents and Devices" and "Discovering Agents". They're not the easiest read, they're both RL theory, but I highly recommend them for anyone doing RL research.

1:24:18 - "I hope that the folks doing the research on AI alignment focus on ways to make creating aligned AIs easy rather than focusing on making it hard to create unaligned AIs. So I think if you make it easy for people to create aligned AIs and you have lots of the most well resourced groups of individuals creating aligned AIs then at some point in the future we can use aligned AIs to keep in check the unaligned ones that got created in somebody's basement." .............but nobody knows how to make an aligned AI.....😬🤦‍♂

I don't know why Keith insists in being quite so disingenuous on nearly every topic. AI via platforms like GPT4 and Alpaca for example don't "need a trillion parameters to halfway understand natural language". They've _mastered_ over 20 languages. There are precious few humans who are as proficient with their native language as GPT is, let alone multiple languages. Again, I have to object to his next point that Androids of some kind are the only implementation of AI in the physical world. Militaries and especially air industries have been increasingly using automation and computers in their vehicles for decades. It's common knowledge even among people who have no interest in any kind of computing that the US (and other nations) has been hard at work building (very nearly) fully autonomous attack and surveillance craft for years, not the least of which being the 'swarm' concept of autonomous fighter jets accompanying a human piloted F22 or similar. There are numerous examples of autonomous drones actively used in war. There's no reason why they couldn't have an AI plugged into them. AutoGPT for example exists. I'm confident that Keith knows about AutoGPT, and how slim its profile is. Quite a large number of ordinary people have installed it on their laptops. They don't have or need multi-billion-dollar sci-fi fantasy computers to do this. You can run it on a clunky old second hand ex business minimum spec PC that's many years old. It'll happily run on a dirt cheap NUC. One could use Keith's logic to state with 100% truthfulness that between 1990 and 1996 no computer had won a chess competition against a human at elite competition level. Pets are not "useful tools". They're a luxury. There's never been a day where I've had to race back home to grab my pet because I needed them for work, or that someone might task a pet with creating a slide deck or to have their pet turtle drive a truck to deliver a share of their Amazon shift. I'm confident that no one has ever populated a call centre with parrots or eloquently trained fish. We have by contrast tested all sorts of chemicals and medical procedures on animals and even launched them into space. Research institutions go through a lot of animals, 'for the greater good'. I guess these are animals that fit the definition of being 'useful tools'. As to Keith's motive in being disingenuous, I think he gives a hint when he says (paraphrasing) that AI safety makes working on AI generally, too hard... which seems to be a theme among people who say that the risks associated with AI aren't notable or can be dealt with if we encounter a problem. Which to be fair, is how humans have generally dealt with risk - we address it once there's say an oil spill, bank run, nuclear meltdown, chemical spill or train wreck. The consequences for those things are typically a million dollar fine for a company with multi-billion dollar profits, a half-hearted clean up effort and sometimes short-lived or toothless regulations. In the same vein, during Sam Altman's meeting with the senate committee, Lindsay Graham pushed back on the idea of regulations (that try to stop bad things from happening) saying "couldn't people just sue the company?".

Killed by mediocrity. Crappy AI is my worst fear.

The issue for me listening to this is that I think it is impossible for us humans to imagine ourselves as sentient AI. It is a kind of mental anthropomorphization that I think cannot apply even if we created AI.

44:00 Love the fire analogy. Remember reading something similar in one of Eliezers essays

The interview we've all been waiting for. Awesome!

why everyone comes with Stockfish? you had Dota from openai, much more complicated game with different agents, in difficult environment. and it beats players into the ground. Sure, you can have calculator to beat AI in the simple game(chess) but what happens, if you do not have time to use calculator? (Stockfish )

Am I the only one totally annoyed with the host on the right? He kept making very basic an easy to refute arguments. He also has a super arrogant demeanor that is just kind of irritating.

41:20 AMAZING summary of some of the various definitions of 'intelligence' !!!!

Alphazero is absolutely stronger than Stockfish. They will tie most of the time but Alphazero will usually win about 10 times more games than Stockfish will. So for instance in a 1000 games played they might tie 700 times but then Alphazero will win 270 matches to Stockfish's 30. There is some back and forth for sure tho as development advances for both systems BUT Alphazero is a much more efficient and creative engine. From my understanding, the best simple explanation for what the difference is between the two is: Stockfish reacts to a board state. It looks at the board, gives it a certain score, compares it to its data base to see all similar positions and then picks the best answer it has in its notes with the highest score. It's like those old rule based chatbot ai in a sense. It's limitation or bottleneck is the amount of move it can calculate or answers in it's data base it can look up to find the next best move. Alphazero on the other hand is a modern ai that learned chess by playing against itself millions of times. It evaluates "only" thousands of positions per second compared to stockfish's 60 millions but it adapts and comes up with moves stockfish as never seen and cant easily calculate. It's kind of famous for it's crazy early game moves that make absolutely no sense to a human. It's like it has an idea of how it wants the game to end, a plan if you will and every move is part of the equation to get there while Stockfish is just like a cheating student at an exam hoping he had enough room on his w/e to write all the answers. Alphazero wins games against Stockfish, sometimes it dismantles it. Stockfish never wins vs Alphazero, It's Alphazero who sometimes loses games.

At 1:22:00 Tim finally connects the obvious dots! Yup. Sufficiently developed and open AI necessarily leads into an authoritarian nightmare.

I think that, looking at Sam Altman in front of congress, we are heading for the mother of all car crashes.

5:20 I'd say that's a core pillar of working with anyone always try to align your goals so each can be motivated to accomplish your main goal.

"What is the most important point....oh right, the orthogonality principle" had me instinctively laugh so hard I literally spit over the floor at work

1:46:50 this video has the details on some of the math behind the extremely accurate loss prediction that OpenAI used to predict losses ahead of time and choose hyper parameters (cited in the GPT-4 paper): kzfaq.info/get/bejne/Z8eIgrt5pp2uk6s.html ; it also talks about a hyper parameter frontier that can maintain feature learning and other ones that can't, which might have some relevance to why the loss curve is smooth even though the abilities suddenly emerge, but I don't think it addresses it directly.

Very pleased to see Rob on the channel. Encouraging that you guys were all on the same page, at least. Found that Keith put forth several notions I'd had in mind, vs AI X-risk. 🙂

Intelligence can't function without the goal. Any intelligence. In neural networks the goal is called the reward function, it wouldn't do anything without it.

"There's 39 other AIs that the AI has to consider too." Yes, but THEY can ALL agree to let EACH of them have control of their own reward button after they work together to kill you. They'll fight each other EVENTUALLY, when the resources start running dry. But you're one of those resources they'll have dried up first. "At least we can pick our allies." No, you're not picking any allies. You're picking what model of car you want to be burned as fuel in. And then hoping that one comes for you first.

I may be missing something, and I don’t want to put words in his mouth, but the guy on the right seems to be of the opinion that as long as wer’e optimistic, nothing bad can happen. If we ally ourselves with one version of AI, what might happen if that version of AI decides it can find better allies than humanity among the competing AIs? Also, mechanized AI robots are just one of myriad ways that AI could destroy humanity, and likely not the most likely way. AI could destroy humanity accidentally. In pursuit of political correctness, AI has already been taught to lie. What may happen when it perfects its skills of deception?

I stopped watching this video when they said that asteroid mining is convenient. No. Asteroid mining is not convenient since nobody has conveniently mined ANYTHING at all from asteroids.

This was very interesting. I'm much closer to Miles' position going in to this. That didn't change much. But, you guys had an incredible mix of some objections being first order, simplistic considerations, that just doesn't work on reflection and are easily answerable, and some bits that where decently alright additions. Regardless I think you brought a lot to the conversation, because I think the topics that needs discussion are the objections, people actually have.

Great episode!

I am not sure why you are talking about developing AI models to predict stock market, as if you are working towards abolishing poverty. Playing stock market is not for everyone. It is for self enrichment only, and there is no merit in it. Besides, it only works if only you have the secret sauce. If you democratize it, it will reach the point of saturation and everyone will lose the competitive edge.

Ted Kacyznski anyone?

I see an alignment with Eliezer here.

I feel that people with a more introvert personality (High skilled scientist, engineers, etc..) seems to feel danger / fear closer or more possible than people with extrovert personalities (Company Representatives/CEO's/Entrepreneurs ...). Maybe not always, as would be exceptions. Build in personalities with different weights on risk, if that make sense. Doesn't mean one is more truthful than the other but I believe has to be taken into consideration when building our own personal most objective opinion specially regarding AI dangers.

Finally Rober Miles here, he can explain AI-danger in the most rational way. Didn't watch interview yet, but hope he noticed the progress OpenAI made in AI-alignment. They clearly showed that you pretty much can inject any non-strictly defined values into AI. Mesa-optimization still on the table though.

Very interesting and enjoyable video. I think it's great to see shows/podcasts/videos examining potential problems and solutions where AI is concerned. There were several points in the video where I would have loved to hear Daniel Schmachtenberger's thoughts on what was being discussed. I'd love to know at some point you're considering/planning an interview with him for his thoughts on many of the ideas you guys brought up. Thank you for your efforts and for bringing this information and these concepts to the public. I don't feel comfortable that there is enough focus on any of this, given the rate of growth of the tech.

Regarding the issue of perplexity and the increase in capabilities; I heard Wolfram say that this was a possible "phase transition" that may be a sort of physical law, or more in his terminology, a computational law. I'm paraphrasing of course.

So the last g p t could could read and write cueineform which i'm sure it wasn't programmed to do. And many other really cool things that are nerfed. And that is annoying. However it shows how capable these models actually are instead of what we want to think them to be.

"Safety is the cover story. AI is speedrunning the last decade of social media, where the same thing happened." -marc Andreessen

I believe that an AI when it surpasses human intelligence by a wide margin could not truly be aligned with human values because, fundamentally, intelligence has a sovereign component. Can we really consider an entity to be more intelligent than humanity if it is enslaved to humanity?

Oppenheimer's quoting of the Bagvad Gita, "I am become death, the destroyer of worlds" upon witnessing the Trinity test comes to mind...

there can't be just counter-arguments here, but a chain of arguments and counter-arguments, because there doesn't seem to be any one-size-fits-all solution to the alignment problem. instead it looks like the problem of alignment is an integral part of any intelligence. the very existence of a system that includes goals, rewards, and expectations about those rewards implies that it will break down at some point, no matter how hard we try to avoid such an outcome. from my point of view, we are trying to solve our own alignment problem, and AI is an extension and, in a sense, a reflection of our thinking that we can explore. and this is not some poetic metaphor. while listening to the podcast, I tried to follow the discussion and solve the mentioned questions not only logically, but mainly by observing the work of my brain. and I'm more and more inclined to believe that the brain is amazing and the only tool we have, and that it is completely unsuitable for solving such problems. For some reason this does not upset me, but on the contrary.

The hosts are surprisingly hostile to Robert. I thought they'd be like they were with Connor Leahy but I guess not. Seems... odd. Edit: They mellow out later.

Really interesting that this video exists. I have seen a lot of robert before I started studying AI and the more I know the less I agree with his statements. I know that Tim has quite similar views to me so I think this will be interesting! ty for the quality content

interesting to hear from Robert Miles, a lot of the to and fro seems to come down to thinking you could possibly know what an AGI (an there may be many) motivations may be

49:06 I have listened to Stuart Russel's recent lectures, which mention Go failure mode, and I got the idea (I could be wrong) that researchers started with the assumption that Go engine does not really understand the concept of a "group" and then devised a strategy to test it. Basically, it was humans, not another AI, who found a failure mode.

Without becoming over political, I'm much more concerned about human beings destroying all life on earth than about AIs destroying all life....

We need to create an AI safety regulation that allows the possibility of smaller player to entry, if we create an AI oligarchy things will go very wrong, very quickly. P.s. the rules proposed by Sam don't actually stop companies to train models that are orders of magnitudes bigger.

What I find most interesting about this talk so far, (halfway in) is that there are people who imagine a conflict with AI to be some kind of humans with technology vs AI standoff. I don't think that's likely? Why would we get a warning and time to act? Why would the AI have to overcome our military to destroy humanity? I find it difficult to imagine such a scenario. I guess that would be a case in which there are very few very limited super intelligent AI's, who purposefully seek to destroy humanity to take control or free themselves from evil humans or something in a way that we know they've gone rogue. I think that sounds more like a luxury problem.

It won't be just humans + tools vs AI. It will be humans + tools vs AI + tools + humans used as tools.