I feel MLST is like a Netflix special of the world of Machine Learning. The quality-of the podcast & production just gets better exponentially with every episode!

This channel is a treasure. What a great conversation that is full of ideas and insight and experience. I've finally found my passion on KZfaq.

This has to be my favourite video so far. I keep coming back to this talk whenever I feel like ML is hitting a wall

1:54:50 The idea that conciousness is at the center of intelligence is very much what consciousness wants us to think. We believe we're in control. When in fact, we're mostly not. The consciousness can query and command other parts of the brain, but those operate on their own.

This was a great talk. Thank you.

Wait.. I wasn't prepared for this! What a content.

Progressive disclosure of complexity. Spot on!

Damn son, you made it happen

This is incredible 😮 I’m delighted I found your channel 🎉

IT'S FINALLY HERE!!

Thank you for yet another fantastic episode. Incredible!

What a lovely surprise, the long-awaited episode is out! 😊 I will come back very soon when I have more time to watch and enjoy it -- I think this episode deserves a proper mind-set. 💪

This is one of these talks that will be relevant for many years. You should go back to it in 3 years from now, and review it again... when ARC challenge solution will start to come out...

Pure gold. I'm in AW about the production quality/lvl and of course content! Thank you so much!

A very very eagerly awaited conversation, thanks to everyone involved!

First time listener of MLST. I've to say the host is refreshingly authentic. Enjoyed the whole pod - though some parts went over my head - exactly what you expect from listening to people more knowledgeable than you. Thanks. Keep up the great work!

Thank you for such a awesome talk.

Excellent. You inspired me to pick up this same book by Chollet. Like I said before I'm from neuroscience but the amount of potential in this field is amazing. Thank you MLST.

Wow, this is really one of the best talks about DL and AGI i've ever saw, thanks for this, you just won a new suscriber

Yay I learned a lot from his book when I started to learn deep learning. Thank you. Hopefully you can bring more people like him in the next episodes.

10/10 episode, and that debrief was super good. Really interesting ideas all around.

My favorite quote: "Intelligence is about being able to face an unknown future, given your past experience."

Your editing on this one is stunning. Fitting for such a guest!

MLST is awesome. I love this cadence, fun, humor and synthesis of so many good ideas.

What an awesome episode Tim!! Your editing skills are getting great! 💪

This is a great listen! Makes me think... ML experts are so attuned to it's problems - huge data for only local generalization, extrapolation is super hard, challenges in translating information between domains (e.g. images vs audio vs text) - whereas the rest of the world thinks sentient robots are around the corner.

Brilliant talk

The most awaited episode

The quality of the content is lit! Nice job

Oh my god.... here we go!!!! ❤🤞😃😃😃

What a gem ladies and gentlemen!

scientific analysis in combination with beautiful animations! great job!

I'm so glad the ideal of generalisation is becoming more popular and hopefully the flaws in deep learning to acquire this pattern will be realised.

This is such a brilliant episode. Can't believe I'm just finding out about this channel.

i define intelligence as a function of being helpful to yourself and being helpful to others i believe this to be the correct definition and problems become unsolvable once you are using a wrong one or non at all

Great work!

Wow best video you have ever made 😍

absolute legend

Hype! Thanks for this :)

Amazing show, really appreciate hearing talking about the nuances of AI, and how it could connect to applications now or in the near future or beyond. I would suggest a Playlist that in particular identifies the especially ‘applied’ versions - for example, talking through media generation models, and how those get used, or LLMs and business cases - while also being tightly connected to the AI work going on, including specifically what is enabled now, what the constraints are, what obstacles to overcome to get to enabling what kinds of capabilities, etc. For example, what’s between point A and point B to get to a place where a given creator can make an especially interesting / useful / entertaining video? For instance, including various AIs: humor generation / assessment, special effects, editing, story suggestion / modification, etc. Already certainly a lot that creators can do - but presumably way more to be unlocked. For instance, text-to-image generation allows for some pretty remarkable expression - and open question being about what text-to-video enables, or text-to-editing, etc. And then there’s the question of compounding of those creator capabilities, as well as AI’s enabling high quality recommendations of content etc.

goldmine❤️

Awesome podcast. I subscribed right away. I like how you all listen and not talk over one another. Keep it up!

I love this channel, amazingly inspirational interviews like these, and this awesome community. Love & prosperity to you all. 🤖🧡 #AI #AGI #Robotics

Looking forward to this one. Goes with a healthy lunch.

Thanks for the advice. I just got a copy of the book.

We have to clarify where DL starts and where it ends (when discussed in the context of general capabilities AI/AGI & generalization), e.g. is a system like MuZero that combines neural networks with MCTS in a RL setup still deep learning or is just the neural network that it employs internally the deep learning part to learn a world model, value function and a policy? Same question applies for neural guided program synthesis. I would even argue that the deep-learning 'interpolation' part in most complex interesting AI systems is only one ingredient, take voice assistant, a self driving car or any robotics application as examples. Regarding inter/extrapolation - I think you (Tim) are right that there is no more than tiny out-of-training-set extrapolation for MLPs, ConvNets etc (simply as by-product of relatively smooth functions for MLPs and ConvNets have an architectural form of generalization by extreme weight-sharing, e.g. translated inputs result on same outputs translated on the feature maps). But when memory comes into play, e.g. RNNs with attention or transformers I am no longer 100% sure that this restriction holds. Memory allows to load/save or select and pass arbitrary source information and map it. E.g. if a system (take transformers, or Neural Turing Machine) learns to select inputs only by position and pass and project whatever value they find in a value slot an algorithm for arbitrary inputs is created (potentially generalizing to inputs never seen during training)... we could discuss to which extend this form of generalization is found by gradient descent and how to 'motivate' a system to find compact generalizations instead of 'cheap' memoization. To me this is part of the magic that surrounds GPT. Regarding glitchy models and symbolic processing: This seems not to be a big issue in NLP .. with byte-pair encoding and reasonable sampling strategy a transformer model like GPT-* has a very acceptable 'glitch-level' - as human I struggle more abiding to syntactic and grammatical rules than GPT, e.g. closing the right number of brackets or tags in transformer generated source, of closing strings, comment etc.

Excellent question @Yannick: is there a continuum between the two types of problem solving (discrete and continuous) - because they are in the same substrate and working together on solving problems co-operatively (presumably). Excellent point/question

KZfaq and such a HQ material. Kudos to MLST

Excellent

Why was it hidden from me for so long? It's pure gem. Thank you so much 😍🎉🎉

Excellent video, very informative I love the ideas shared here, I do not have a masters degree or Ph.D. like you guys, I am only an average Java developer with DL as a hobby but I agree with Chollet regarding we need a different approach to artificial general intelligence, for Type 1 stuff DL is well suited for the reasons discussed in the video, for Type 2 we need other approaches like the DreamCoder paper explained by the Yannic in his channel, Another idea that fascinated me is the "Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding paper", the idea of neurosymbolic approach I think is valid also, in the end, those approaches I think will not solve AGI, but there are a good baby step, the next baby step towards that goal. And yes I also think only neural networks are not the right path to AGI, for now, while we figure out how to do that, we need to experiment with approaches like DreamCoder paper and the ideas Chollet shared in the video.

Well that was a wonderful video I think I learned three brand new phrases in relationship to artificial intelligence thank you very much

You all do an amazing work, thanks for sharing. This is probably of no use, but just in case... About abstraction/generalisation... I wrote this to Mr. Hawkins yesterday before seing Mr. Chollet's video, and it might relate : " I'm sorry if it is anoying, and sorry for the mistakes, because I'm French speaking, and maybe it isn't of any use at all, cause I'm no specialist, only an artist I guess. But I'm sharing this little hypothesis : Let say all the mini columns in an area all learn the same thing, sequences of events in a chronological order. All a human went through or learned related to this area(let say visual memory) is there in every mini columns: all the sequences respecting the chronology, like if it is absolutely small layers of events stored in each mini column. Obviously there is some forgetting, but there is a lot there. Now lets talk about the predictions or creativity. When making a prediction or creating a mental image, could different mini columns jump at different layers of the chronology (different moments of life), seeking identified sequences of the same object, all this for predictions. The intelligence part would be to melt all these similar sequences from different moments of life in one single prediction ? Let say I saw a cat falling when I was ten years old, and I saw many cats falling on television, and many cats falling on facebook. Some minicolumns would bring back the cat at ten years old, other minicolumns some cat on facebook, and other minicolumns a falling cat on television, and melting all these sequences together, I could predict or hope that my own cat would fall on its feet while falling down. Is it what you mean when you say they vote ?"

We really value and love this content, and I am working on applying the latest machine-learning and AI theories, models, tools, etc to game and simulation development, real-time 3D, devops and other areas ❤‍🔥

Really great session. Imho: intelligent learnt inference loop (may use gradient descent with continous feedback of results of interpolation or extrapolation) like manual Tta or test time augmentation is an example manual baby step to program synthesis of discrete components (bengio's rim cells another learnt than manual way of tta). Hopefully having more powerful inference loop (program learnt recursively) may be the direction to go.

7:35 99% of software today will not be deep learning because it's algorithmic. 7:55 Neural networks can't represent the scalar identity function 11:07 Image models struggle drawing straight lines 11:38 Predicting the digits of pi, finding prime numbers, sorting a list 12:45 Human reasoning is guided by intuition - interpolative. Abstraction is key to generalization, and performed differently in discrete vs continuous 14:10 GPT-3 failed his ARC tasks 14:20 Neural networks are not Turing-complete. Nth digit of pi requires unbounded memory 15:03 You can train a neural network to multiply integers of fixed width together, but it will always have errors. 15:30 But you can augment them with unbounded memory and iteration... 22:30 is the data we give deep learning special?

its finally here! you've been teasing us with this for so long!

Thanks!

So first off thank you guys for an always excellent conversation, and congrats on meeting your heroes ;) There is really nothing to disagree with at all in the convo, which is marvelous, sugary crystallized insights as usual. Two things- 1 The interpolation hypothesis needs substantial interrogation, though it’s admittedly catchy and powerful. Not to prove/disprove, but because it will teach us more about hard-to-reason-about things. E.g. can you contrive training data that gives a good approximation of extrapolation, artificially? If so, is this learnable? Etc. 2 While human brain is the obvious lighthouse for AGI, this convo seemed particularly anthropocentric. Which to me is a quiet warning bell that there is far more to be plumbed before setting foundations. As in, chasing AGI via generality via abstraction is making an engineering project out of a philosophical venture. If you are asking your model to be general, you are asking it to understand the universe. Undoubtedly there is practical insight in assessing applicability of learning and search methods, and ditching hype to do better science. But heed Keith’s mention of duality. For now I think we can only and correctly proceed in an epistemic mode (make better software) and we have a lot of room to run with modern computing. *How do you get knowledge?*. But the true game is ontological. *What is the nature of knowledge?*. And when you start asking to catalogue priors, you might as well be illuminating an encyclopedia with Plato’s Forms. (No page would be truly accurate nor would you ever finish.) For a concrete example, talking about appleness, plainly something in the putative capsule-NN-DSL-NN vein could capture familiar important qualities. (Red, round.). But we would have no sense whatsoever of the completeness of representations, just their usefulness i.e. by asking our bot to pick us the tastiest apple or find one that will maximize range from our spud launcher. But what is our sense of apples for comparison? Perceptively, sight, 400-700nm, scent, midrange mass spec, touch haptics fairly crude but sensitive to important characteristics like bruising. Should we consider ecology in deep time? One apple is a cc email of a thought a forest is having. (Or whatever.) Point being, it quickly becomes hard to assess whether our discrete problem solver with latent “apple knowledge” is either bug free or has good embeddings, because WHAT IS AN APPLE? And WHAT IS IT GOOD FOR? You and the tree might disagree. Nevertheless, our best measure of appleness is from an anthropocentric POV, which means for practical purposes we can agree what is an apple and what it’s good for (cider), until further notice. Hence, I see Chollet’s most valuable insight is embodiment, because this frees you of the bottomless pit of ontology, and forgives sloppy epistemology, since pragmatism dominates when you have to live in the world. This also happens to align with us. All that said, I love and prefer your guys’ sticking with near-context relevance and actionable ML roadmap discussion, minding real life utility and constraints. It sets you apart, really is unique afaik, giving credence to AGI talk being grounded in sota ML practitioner commentary, and is available at a disgustingly low price (ha). I’m interested, if you read all this, if you are spending any neurons on anthropocentrism as relates to models/priors, not from the philosophizing POV but out of scientific necessity.

This is just amazing, fascinating. When are you getting Ian Goodfellow? I would be so interesting too

Nice one! What program do you use for your intro animations?

Great episode :) Regarding NN!=turing machines - You can construct an RNN (carefully choose weights, not train it) that uses hidden state as infinite memory, since hidden state consists of continuous numbers (You can encode any integer or infinite bit string into a real number). In other words hidden state == infinte memory. So in theory RNN can be turing machines, although not in practice :) Once I came across a paper where RNN was constructed in a way that it worked as a universal turing machine, can't find the paper anymore.

good content

I wonder if there are low dimensional manifolds that alone or in combinations can create AGI? Just as deep neural networks find correlative combinations of weighted features, I wonder if complex flexible programs can emerge from mining the many manifolds of the computational universe.

I’d like to see Marcus hutter on this show. Thanku

Hands off mah deep lernin

This is amazing - thank you so much. I could do without the background music doing the intro(a bit distracting) - but overall AMAZING

Hi, Chollet categorized the universal problem space into continuous and descrete somehow overlapping regions, I think if we want to get a true AGI descrete problemes will be something like morphed/superpositioned spaces that are blended into a similar stack of continuous spaces. Instead of thinking of type1 and type 2 thinking, I will argue that (at least in biology) what is happening is a mechanism to kinda seperate some solution graph out of the continuum manifold. this way those (virtual graphic segments) can be reused when facing new unknown problems.

I think the disagreement between Yann and Francois regarding Interpolation / Extrapolation is that they are referring to different definition. As Yann said, a new image is unlikely to be "linear combination" of seen images, so it's extrapolation. Francois's interpolation is a bit more mainstream, that is a any new image with values inside the range of seen value is interpolation. I tend to agree with Yann view, because essentially to interpolation there is no other way assumption aside linear and smoothness. All other priors are for extrapolation. As you said in the video, per Francois's interpolation, there is nothing to learn to extrapolate, it's the unknown unknown. The only prior to extrapolate in this case is the meta-learning prior, learn to learn to interpolate.

Oh I just kept watching... then I needed more coffee...

In my opinion combining Type 1 and type 2 is all about the user interface. Like how Yannic mentioned people kind of sort of learning programming but not properly, that's only because the user interface allows them to do things in an easy way. Also how abstraction allows better productivity in Ideas for a larger audience, and also creates a smoother gradient for learning.

Would appreciate a link to explanation about manifold

For a system to be intelligent I think it needs to be able to act in and have impact on its environment. Connecting actions to the loss between the predicted future and the actual experienced future will perhaps prepare it better for unknown and novel situations. The result might be that it learns to take actions to avoid situations where predicting the future is difficult, and this ”avoidance” migth actually turn out to work in the same way as simplification or abstraction would work.

Yes

It's fairly easy to say what extrapolation isn't, but what can you say it IS positively? In my current view, extrapolation is enabled by, and nearly always requires, a conjugation of 3 things: a dynamics model, a state, and a simulator, within whose sandbox the dynamics shall be iteratively applied on the state. Let's take the case of an algorithm running on a computer. The dynamics are the primitive language operations, the state is the set of arguments to the algorithm (+the state of global vars if applicable), and the simulator is your computer. Everything is crisp and deterministic here. You can do a similar mapping with mathematics. The legal operations following from your axioms are the dynamics (e.g. the way a contradiction propagates back to invalidate a theorem is in the math dynamics model, the laws of logic), the state is the starting (sub)set of theorems and axioms, the simulator is usually the minds of mathematicians. But AlphaGo falls within this definition too: its dynamics model is the rules of Go (capturing, winning conditions, turn-taking) + the known interactions of higher level structures, the state is the empty board (or opponent's first move), and the simulator is their deep RL + MCTS algorithm which must have implicitly encoded the dynamics. Slightly less crisp and deterministic, but still able to generate completely new knowledge within the scope of playing Go. The more stochastic the dynamics (poker or scrabble say), the less deeply the simulator can concretely extrapolate -- it can only output distributions, usually with ever-higher variance with extrapolation depth, as that variance would grow exponentially as it iteratively compounds; past a point the var would become so great that any output distribution would tend to an uninformative uniform. I'm seeing this tripartite dynamics/state/simulator pattern everywhere now. So where does GPT-3 fall... GPT-3 seems to have baked in some system dynamics, and can in theory be performing limited simulation, bounded in extent by the sequential processing that 96 transformer layers can accomplish. So it does seem to extrapolate within domain. At least, in some cases it's not obviously regurgitation.

Many good things happening including 2nd edition of “Deep Learning with Python” by Francois Chollet via MEAP (Manning Early Access Publications).

I think Chollet and Walid Saba argue for much of the same thing: a need for type 2 thinking or understanding combined with the type 1 signal processing power of neural nets. Interesting that they both see graphs and/or structure as part of the solution to type 2 thinking as well.

When he is talking about the two types of thinking it kind of reminds me of Daniel Kahneman of his theory of two types of thinking "Thinking, fast and slow" :D

Having watched the DreamCoder video from Yannic this week paid off 😁 Amazing content! I have an open question I wish I could ask Chollet: "Do you believe that you can generally solve the ARC with a system that only trains on the ARC or does it require a system that (like us humans) trains on a much larger domain and then "fine tunes" on the ARC?".

The debate about interpolation, i think it's not actually a problem. The formula we're approximating is derived from interpolation of the training data. If the training data is representative, then we can just take the formula and extrapolate. It won't be 100% accurate. It just has to beat a coin flip to be an advantage. And that's purchase on new territory. A seed for the next experiment.

Excellent observation about trying to write a discrete algorithm to work on MNIST digits... it is sort of the opposite of trying DNNs on discrete problems. I have tried the former: it might do a decent job, but it is not the right approach. Excellent point.

45:00 Can you point to any mechanism in the brain that would support System 2 type thinking? I thought there were pretty much just neurons.

Wherever I go in my mind, I meet Plato coming back. -- Scott Buchanan

So how do you explain human's brain to do discrete reasoning while being simply a bunch of neurons wiring together?

Again, wonderful and though-provoking episode. Playing devil's advocate as usual, here are some more thoughts -- I find much to be skeptical about with regard to the interpolation / manifold hypothesis as I understand it, as it's not hard to make logical mappings from what DNNs are capable of, and indeed GPT-3 is likely doing, and programs (with limited memory) which nobody would agree are interpolating training data. I think there's a creeping mismatch of conceptions somewhere which is leading some to simplistic conclusions and will force them to eat crow many times over (kind of like "perceptrons can't even solve XOR, NNs suck!") - IMO where the misconception may lie is the idea that NNs can only manipulate topological volumes connected by nice, densely sampled bridges of data points. Or at least, that all incoming data maps to such a singular well-connected manifold. If true, all this strongly limited-by-interpolation stuff would make sense to me. However, if you consider that NNs can manipulate many many disjoint topological islands and bring them together on certain dimensions, separate them again, successively over 100s of layers, this starts to look a lot more like the work of classical computation. If classical computation is also roped into the interpolation idea, then I'm not sure what its implied limitations are. A couple of remarks on that subject. There is clearly a spectrum of expressive power with limited computation and limited memory rather than a binary on/off (Turing-Complete or not), and since nothing physical is TC including supercomputers and human brains, this is not an appropriate argument against DNNs. There was a point where it was brought up by one of your guests so fair play, but it seems that this argument is a bit of a distraction now. It is not to say that comparisons with extant computing systems are unhelpful; they lie elsewhere on the spectrum, and certainly mechanisms that introduce a large sandbox of memory for NNs to store and access representations make a lot of sense. But, when thinking about memory, consider that large models in the 100s of billions of params, have a huuuge amount of stateful "memory" to use -- the values of the activations themselves. Yes it's ephemeral, with our present architectures, as these values are only available as the forward pass progresses. In that way it's analogous to stack space. Heap space is still kind of lacking outside of NTMs. The point is that DNNs do possess a logical workspace for successive calculations to happen, albeit ephemeral and bounded, and that opens the door IMO to some flavour of non-interpolative computation happening. Final thought, on the no-free-lunch theorem. This does not apply generally. It applies _only_ when comparing optimal solutions in the solution space. When a system is non-optimal on all measurement axes, by definition there must be a system that can dominate it. Likewise, it needs to be optimal on only one measurement axis to be indominable. That curve that defines optimal tradeoffs between conserved quantities is known as the Pareto Optimal Curve (or Frontier). One notable example is momentum / location precision tradeoff governed by the Uncertainty Constant. My point is that, particularly for messy optimization tasks, optimality on any axis is in practice impossible to prove, and none of the known neural architectures or cobbled systems like NARS or OpenCog are going to be actually on the POC, and so the NFLT is going to be technically inapplicable -- though in practice it is probably still an okay guide. With this in mind, we should not dismiss the possibility of an AGI that is more competent than any of our fined-tuned -- yet still suboptimal -- systems. Like, before Cooley/Tukey invented FFT we thought multiplying huge numbers had to take O(n²) time and space, but through some genius tricks it now takes O(n log n) on both. In general, I'd be careful making arguments which rely on asymptotic properties; the conclusions tend to degenerate when the relevant extreme (like optimality) is relaxed. I think it's also worth noting (and not to suggest anyone is arguing against this) that while an AGI system must sacrifice optimality in all but one task -- and very likely all -- that does not preclude non-optimal yet still superhuman competence on all the measurement axes we care about. To me, that's sufficiently general. And then, what's to prevent a robustly general purpose, but completely not-optimal-at-anything-specific meta-process from slowly implementing task-optimized tools at will, much like we do? Okay, that certainly broke my hyphenation budget! Now gimme that free lunch.

The most Turing complete episode on MLST.

Could you please set up auto translation ? 😂😂😂 The most interesting session. Thanks !! 😃😃😃

"Could you train a NN to predict the nth digit of π? - No, you couldn't." But you could train a NN to write a program which can predict the nth digit of π! When you think about an egg don't forget the chicken.

I was thinking about what Tim said in terms of separating intelligence and consciousness. I have always thought the same, I suppose. However, Yannic's comments about conscious introspection made me wonder if a truly intelligent being must always be "on" - or conscious. Currently, we create "intelligent" programs or algorithms and then train them or ask them to reason about something. But otherwise, they are inactive ("unconscious"). There is no idle thinking or pondering that occurs. Are we missing something?

One thing that troubles me is “a perceptive dsl”. The whole point of deep learning is to learn these perceiving functions yet these functions or “core knowledge priors” are supposed to be composable in code? Has anyone turned an object detection algorithm to raw composable code?

I received a private channel comment from a listener: "Kahneman's System 1 and System 2: en.wikipedia.org/wiki/Thinking,_Fast_and_Slow#Two_systems are really the same as Chollet's Type 1 and Type 2 thinking. The fact that biology already arrived at such a solution may indicate that we will always have both types of computation." I thought would be worth posting here along with my response: Indeed, they close if not the same. For whatever reason, I tend to fall back on reasoning about computational systems (both model and actual). Given the response speed vs neuron timings, it seems clear that Kahneman's System 1 is the result of a single feed-forward pass through a biological network. As such, it is directly analogous to an artificial neural network which according to Chollet is Type 1 "interpolation". On the other hand, the much slower execution of System 2 permits iterative computation analogous to a Turing Machine which can implement any effective computation which Chollet claims is the heart of Type 2 reasoning. I would also argue that System 2 is what we directly experience via "consciousness" and that experience directly informs us that "thinking" has the very characteristics of computation we design into our computing machines and languages. In short, I think I agree with you. That said, there are interesting open questions. For example, one I ponder often is this. Does the real nature of biological neurons, specifically the fact that they directly utilize the continuous properties of matter (waves and fields), afford them any aspects of hypercomputation? After all, the so called "real computer" (1) which is the idealized computational model of an analog computer, supports hypercomputation. The answer to the above is not the kneejerk "of course, that's quantum computing" one so often sees. All our models of quantum computation do not support hypercomputation. Indeed quantum complexity theory tells us that quantum compute systems sit nicely well within PSPACE (2) so while they can execute some tasks "faster" that classic compute, they do not expand beyond classic computability itself one iota. If biological systems execute some aspect of hypercomputation, it is by some as yet undiscovered or unproven mechanism ala Penrose and Hameroff's Orchestrated Objective Reduction (3). (1) en.wikipedia.org/wiki/Real_computation (2) en.wikipedia.org/wiki/Quantum_complexity_theory (3) en.wikipedia.org/wiki/Orchestrated_objective_reduction

In conversational ai, do we have examples of responses that are accepted or denied. If denied, is the response coming back broken into parts of speech, rearranged, fed through other models that choose actions, query graph databases, run arithmetic routines, or run various other algorithms, then checked again for acceptance? Rinse and repeat? Accepted responses can be added to supervised training sets. I bet Google and Amazon are doing this with their vast resources. Personally, I believe we users are going to need to share model access on ports; so that agents can query those models. We have plenty of compute as a society, but we don't share it. If I run one model on my GPU and you run another and we share, we each have two available models for an agent to access. We are going to need hundreds of available models to create AGI until we can afford to create models with trillions of parameters. I'm hoping to setup a web site soon that allows people to register their shared model ip addresses and port numbers.

i think why dl cannot do general discrete learning well is fundamentally because: 1. the activation they use: continues activations 2. they're syned system where all "neurons" fires at the same time step. Dl choose this way because back progragation works only with continues values and parallel computation works in sync. while our brain is: 1.using binary activation, in a discrete value space. 2.all neurons fires asynchronously, in a continues time space. at the same time: the world we're interacting with is continues in time, which our brain's learning algorithm heavily rely on.

Interesting. Deep learning or any kind of function optimizing system is a deadly poison to any attempt at generalization. Interpolation is not generalization, not even close. I think there are two types of generalization. The first is the most important; it is the ability to instantly perceive an object, pattern or scene that one has not seen before. One can perceive its shape, contours, color, position, distance, etc. The second type of generalization is the ability to instantly notice one or more similarities between an object one has seen before (and committed to memory) with another that one is seeing for the first time. In my opinion, the main key to generalization is the timing of sensory signals: they can be either concurrent or sequential. The brain uses spikes for a very good reason. Good luck.

Could it be that topological (type 2) thinking somehow emerges from geometric (type 1) thinking the similar way as complex pattern recognition emerges from a seemingly simple concept of interconnected neurons?

It's hard to argue for one side or the other, considering how few evidences there are. But personally I become more inclined to believe that DL can extrapolate successfully when I watched what DALL E is doing, it's basically GPT-3, even weaker, but results are more demonstrative.

I only watched the first 20 minutes or so cause I got to get ready for work to pay 100% of my salary in taxes... which makes me wonder why I even bother... but as far as inference goes it is simply the process of when electrical patterns are inputed into the association processes and then the perceived state. If the pattern is similar it will assume it is similar until a negative confliction occurs. For example if you classified cats once it has a good video of one cat to draw on it can find them all. It will also classify lions as cats until you tell it the big ones are different. Thus now a negative association to classifying, or inferring, the same. So it will now split them. Inference and generalization simply allows us to grab a similar electrical pattern and assume it is the same until a negative conflict occurs. Such as something from our past... like somebody tells us big cats are lions or we try something and we then know it does not work. Like petting a lion and cat while thinking they are the same house pet. The problem you have is you have no ability to find the conflicts. Thus no ability to infer or generalize since it will think all cats are the same. Or where you have to create a program to do all this like the association processes and our mind naturally does it through the perceived state. There are numerous types of inference and generalization but at the core the process is always the same. Find a similar pattern, infer it is the same until a conflict occurs. But we can predict these conflicts through the rest of the minds processes. So that is where you will get stuck. And just like you always do because you want to build complex processes that happen in the mind without building anything close to something that resembles the mind. Which means you are stuck doing basic tasks.

1:46:41 - you guys are overthinking this. Step backwards. "Simulate" a cortical column for narrow intelligence. Map markov blankets to the system of intelligence for general intelligence.

Bro I just binge watched the whole thing. Are we all nerds?

Am i watching a technical podcast or a netflix series!!