The level of abstraction involved in AI related existential risk is, as Steve noted, a real problem for effective communication with most of the elite, since most of them are not members of the cognitive elite. The problem is intensified by several other factors: this is something new under the sun, it is a black box, it is massively dual use (really omni-use), and, as mentioned, safety work converts almost automatically into more frontier incursions. The situation we face is akin to an invasion of infant aliens from a race of ultra-intelligent aliens, where we do not fully appreciate how intelligent the infants will soon be and have no conception of what they will want when they become themselves. Discussions of potential long term risk scenarios remind me of discussions of metaphysics and theology. The fact that current LLMs are black boxes goads the imaginations of both optimists and pessimists. Predicting a future contingent on these mysterious entities is not an altogether empirical endeavor. Consequently this game produces high degrees of diversity of opinion. And who's to say that the entire future is not at the mercy of the imaginations of these machines as much as of their capabilities? The is/ought distinction extends to machinic intelligence. This leads my imagination to suspect that entities with a better grasp of the "is" will select "oughts" that we will never understand. An entity which is smarter than you is, to some extent, inherently a black box. Even if our AI were transparent at this stage of sub-human g, they would transition to opacity as they surpassed the human range. Narrow AIs like AlphaGoZero were already opaque in their special spheres. But, like the leading LLMs now, that Deepmind system was designed (if I'm not mistaken) to be opaque even before it surpassed humans. Another source of unpredictability in humans and smart machines is the fact that there does not seem to be such a thing as pure reason. Chaos, probability, anosognosia, and many other factors prevent pure reason from arising. Maybe cognitive fallacies could be structurally embedded in the AI system to handicap it as an agent in the world, ideally fallacies that would not interfere too much with its useful work. For example, being unable to act in face of perceived ambiguity, due to a peculiar form of risk aversion, would make many possible AI attacks impossible. If the installed fallacy is a known characteristic of the system, it might be possible to cancel it or compensate for it after the results are produced. Of course, a sufficiently intelligent and self-aware system might be able discover such limitations and extricate itself surreptitiously. One of the questions I'm left with after this conversation, which was briefly touched on, is what the East Asian nations think about AI prospects. It could come to matter if the Chinese gain traction in their huge semiconductor catch-up efforts. It's hard to imagine that they'll catch America in this field in less than ten years, unless America abdicates or China achieves the greatest espionage coup of all time.

Prior Jaan podcast: kzfaq.info/get/bejne/jbh4m5iHmqzYg4E.html