Yet another interesting video, which condenses a lot of information into a manageable chunk. Keep it up!

As someone who actually deploys edge AI I heavily disagree with how "easy" you make FPGA's and ASICS seem. For the vast majority of "smaller" projects the Nvidia Jetson series is a far better choice since they support newer algorithms and functions. Especially with the speed the field of AI is progressing at. Furthermore fp16 GPU tensor cores are basically optimized for ML inference and provide good performance if you want to spend a little extra time on converting the model, though often even that has compiling issues with newer ML models.

Dude your content is as outstanding as it is random. From soviet oil export over chip manufacturing to edge AI models xX

I write inference pipelines on jetsons for my work. The latest generation have some very attractive performance characteristics. They do pull 60W, which definitely isn’t nothing, but for our use case it’s manageable. Something that wasn't stressed is that those devices are heavily optimized around fixed point 8 bit throughput. A high-spec Orin can put out 275 TOPS, which is more than triple a 4090s 83 TFLOPS. Even if the models are much weaker, the increase in throughput opens up a lot of flexibility with system design.

Post-training pruning is very much how human neural networks learn - massive connections anywhere and everywhere over the first few years - during the initial training phase, then massive pruning to get rid of the unnecessary connections.

My fave video of yours recently! Thanks for making it!

for a second, I thought the cow photo at the beginning said "dont eat ass"

That solder bridging in the stock video at 2:56 and 3:00. Yikes! Great video BTW!

Wake up babe! Asianometry posted!

Amazing video (as always), I am a ML engineer, it's not (always) true that the less weight the model has the less memory/faster it is. I know it's a little counter intuitive. It all comes down to how many GFLOPs the model need on some hardware

Whipping and nae naeing simultaneously should be an Olympic sport

I thought you would have included the mobile processors from Apple, Samsung, Qualcomm, etcetera. They all include a Neural Processor these days but I never hear much discussed in relation to them, how powerful they are, what they’re actually able to do and so on. Given the many millions of people own phones that are powered by these processors, surely the potential to bring Edge AI to everyone is now here if they are used effectively.

I would like to write some personal sentences since this is something I am recently working on. There are already existing ASICs around, and it has a huge benefit compared to FPGAs, that is the "area". It's nice that we can reprogram our FPGAs anytime, but considering the edge-AI commercial devices (E.g., VR headsets), we have a very tight form factor for the chip. Then, it is feasible to say ASICs will dominate the edge-AI products. Even more, we will find the architecture of edge-AI, next to the processing units with more caches (everything on a single package = inside of our mobile devices in the future(?) ). However, we need to consider some issues with AI accelerators.. They have to work quite busy, which makes their thermal profile a bit annoying, and we need too many buffers next to cache memories (too much memory on-chip = too much area... ). We have nice cooling solutions already, but we definitely need more, or we need new methods to reduce the sparse computation of neural networks. Maybe you heard about "Spiking Neural Networks (SNN)". They provide a nice "event-based" structure to the network, which allows you to create "idle" states for your computation... That is already a nice idea to have a nice edge-AI chip with low power option !! Next, what if we make this chip in 3D? Considering the memory domination in the AI chips, what about stacking the memory die, vertically on top of the logic die? We try to answer this question in imec.

Natural language processing is what will end me. There is literally nothing else I can do as a paraplegic with dyscalculia on the ass-end of Europe besides translation, and Deepl is already near perfect between every language pair I can speak. I've got every recommendation under the sun from becoming a musician (been playing the guitar for 6 years daily, can't memorize the fretboard because of dyscalculia), to programming (can't memorize the syntax because of dyscalculia), to 3D modeling... (you get the gist, nothing even remotely related to manipulating quantities or memorizing arithmetic relations is viable), to becoming a novelist (sure, because we all know how great those people earn). Aanyway, that was my comment for the algorithm.

2:35 The AI-generated PCB image is a nice touch

ty for your hard work =)

Some of the Nvidia Jetson boards like the AGX Xavier and Orin have separate Nvidia Deep Learning Accelerators built in as well as the GPUS. There is the Hailo-8 M.2 accelerator too.

You should take a look at Perceive's ERGO chip it seems to be a gamechanger in this field

Almost 20 years ago I led a project to port the neural models we ran on a Beowulf cluster to a more mobile platform. Our goal wasn't to create a processor to run solved networks like the current crop of AI processors - we built it with full plasticity so that the learning and processing could be performed on the same piece of hardware. I am disappointed that what is available today is a shadow of what we did in 2004. None of the current processors are constructed to model the neural structures required for real intelligence. They just keep modeling the same vacuous networks I used as an undergrad in the cog sci program at UCSD in the late 80's. Most of the people using the technology don't understand intelligence and sadly don't care what is required. One example: Lately I've seen numerous job postings for AI engineers who can do prediction - what they don't understand is that it isn't prediction, the missing component of these networks is expectation - facilitated by prospective memory.

The main issue I have with AI is that we let silicon/software learn under some evolutionary pressure, and we pick the one that does the job best. But we don't actually understand what made that particular resulting structure do the job best. We harness the gains from complexity without learning about the part of the structure that makes this so efficient and why. It's like someone finding a wand and wielding it without knowing how it does what it does. Part of this is that more complex problems require an AI network that is so complex, that we have no hope of understanding what makes the best working model tick. I don't think we're in any danger of being taken over by AI, but that the information we could learn from AI for making our own designs are rarely or never learned. As edge AI is concerned, I suspect we'll get finished AI solutions preprogrammed. This leaves a small requirement of processing for the edge device, which should keep power requirements low. Most devices are VERY specialized in their purpose. Much easier to run an expensive AI centre and churn out cheap pre-programmed devices.

Love the potato PCB ai generated image at 2:44

Thank you and much Love from the Philippines.

Fantastic video

Brainchip AkidaTM is the world’s first commercial neuromorphic processor. It mimics the brain to analyze only essential sensor inputs at the point of acquisition-rather than through transmission via the cloud.

Fwiw, CUDA doesn't really use C or C++. It has a very restricted API from which you can call upon its features, and yes it is accessed from "C", but its semantics are very different. Branches are expensive, and recursion is not allowed. And malloc doesn't exist.

Video summed up: There is no shortcuts, this is going to require technology we just dont have yet to really get what we want from AI/Deep Learning

Qualcomm AI100 DM.2e form-factor edge AI accelerators scale 70-200 TOPS at 15-25W, also various Qualcomm Snapdragon mobile SoCs fit the role of less powerful 30 TOPS and under edge AI accelerators. Qualcomm is pushing pretty hard into that direction with SoCs for edge AI, vision AI, drones/robotics, ADAS, and level 4-5 autonomous driving including software stacks. They even have AI research division dedicated to solving problems with edge AI model optimizations and other things.

You're missing the latest development in edge AI: simplified models running on a physical quirk flash memory. Mythic AI and Syntiant are two companies taking advantage of this to do simple, this tech is in the earliest days and has a lot of future potential.

I was wondering whether it is possible to un-learn or de-learn (undo a learning) of a learned neural network? So it will forget things or patterns

you didnt mention in-memory computing or analog computing as possible solutions. Take a look at Mythic AI, looks like the will get around the perf and cost by using an analog and in memory computing methodology to get around the von nueman bottleneck and use a flash memory manu process which is relatively cheap and dense.

not sponsored but arduino nano 33 BLE with ARM M4F chip is so good, I have pretty good accuracies with some custom applications (shadow gesture sensing). There are also concepts on reservoir computing, recurrent networks, echo state networks, would love to hear your take

This is gonna be interesting.

Any thoughts on hailo?

thanks for highlighting the NPU on a stick... might have to check that out

The thing with fpga is that you have to manually program entire network in vhdl or something similar. Which is cumbersome and not very easy. Belive me I tried.

'FPGAs have a lot of potential'; if I got a dollar for every decade thats been said... id be able to buy a fancy cup of coffee, at least. Not sure if FPGAs have some kind of massive marketing campaign, or if the general idea just appeals to nerds; its like FPGAs are to EEs as what fusion is to physicists. To present FPGAs as having even a fraction of a precent of the relevance to edge AI as jetson nano or GPU type architectures does, makes presenting ITER as a practical solution to power generation look intellectually honest and well informed by comparison.

I have "The Edge" open in a back tab. Yt free movies has it up now. Or did, they only stay free for a few weeks sometimes and when I get to it its gone and private. I had to sit through "Rob Roy" dozens of times myself in earlier days.

I’ve been studying neural architecture search for my PhD but ive not heard much about joint architecture and hardware search. Currently in an internship in industry and that sounds like quite an ambitious goal based on whats going on meta.

Am I mistaken is not Tesla with their dojo configuration among the leaders?

Video on US chip restrictions?

> Silicon Compiler : kzfaq.info/get/bejne/fbNpg65yy7rQnYU.htmlsi=b7se3OuI42jfNYkM > Parallella : kzfaq.info/get/bejne/rLxpmcaiurvJYY0.htmlsi=2dtIp--sL6L4iiKP&t=830

Is there an rss feed for the news letter

more ai vids love this

Pretty sure the future is completely distributed compute, be that edge or cloud, everything will be automatically optimized to run as low cost/latency via markets. You'll be able to buy and sell the tiniest useful sliver of computation and your devices will automatically buy/sell from/to the lowest/highest bidder. So in the end there will be real time market forces shaping which computing devices will be where. If you have some large computation that isn't latency sensitive, it'll automatically be outsourced to the cloud, taking into account the size of the problem, the time, the latency, the size of the output. Even how to most cheaply get the end result back to you will be market driven, e.g. what's cheaper, storing it in the cloud, only accessing singular elements or should it be compressed and sent to you or is compression too expensive vs sending it uncompressed etc.. In areas of high computation costs/latency that will locally encourage the aggregation of computational power. You might get recommendations how to save money by buying/renting edge devices or to make money by increasing your local area's computational power, similar to miners. All based on the buying and selling of tiny packages of computations. And I suspect edge will definitely have it's place in that future. Maybe in the far future there's even drones that fly computational power from place to place wherever the highest prices are at the moment, like pidgeons looking for food. Some scientist starts a calculation and a giant swarm of drones flocks to his house, crawling over each other like a bee hive. Who knows lol.

Valeu!

So AI computing is nothing more than a super-fast computer either sitting at the data center or edge. It's like a super IBM PC vs IBM mainframe in the old days 🙂 People just love to put lipstick on a pig and call it something different, but it's actually the same sh*t 🙂

O man i got this 18 minutes fresh? nice

5:35 It is really silly to think you can try pruning something that nobody has an idea of how or even why it works, to make it work more efficiently. I imagine it would be similar to cutting cables in an electrical panel and checking if the machine powered by it is working faster or slower, without knowing which room or component had had the power cut out.

I don't feel like any of those are AI-specific issues but apply to all kinds of computation.

wait. Isn’t it an AI generated image at 2:40 ?

I work with NLU and it is almost all snake oil

My father LOVES the same movie. Must be a dad thing.

The edge NPU with arguably the best chances of widespread adoption are ARM's Ethos-U and Ethos-N NPUs which are made to complement their Cortex-M and Cortex-A processors respectively. To my knowledge they do not exist in (commercial) silicon yet but will like have a head start over the offerings of small startups due to their seamless integration with an already established ecosystem.

4:50 *realizes that I have been using premier pro the hard way*

I’m looking at using multiple edge ai devices ($3) in parallel. Each running a real time algo and then combining outputs. NNs will get progressively more efficient and will easily run on modern micros. Micros are the future over fpgas but an fpga can form part of the parallel to serialising etc.

Maybe ASICs for standard image processing layers like edge and depth detection, where a lot of the hard work can be done efficiently and in a standard way, then FPGAs or GPUs for more specific neural nets that take the final layer from the standard ASICs and do something interesting with them.

And then there came software programmable ASICs to run inferencing for generic AI models

thanks

The way you close your video sounded exactly like my highschool coaching tutor. He would give really in depth and well explained lessons, but the hour ended, closed shop and got out in his car before the class' bags were even packed 🤣

Another edge AI techonolgy is spiking neural networks, which can reduce consumption a lot. Brainchip is a company offering this technology

Using edge processors for neural networks at work, it's amazing to see how quickly the field is growing and how quickly their capabilities grow.

Honestly i cant see how this problem could be "solved". on an Edge device like for traffic recognition you need fast AND precise Inference. Like the Video said, most ways of cutting corners just dont apply. I fear that this will result in overspecialized hardware which will flood the market after some cheap product with good software support gets released...

Interesting topic indeed, My own final year college project is based on post training quantization of models Currently working on performing 2 bit quantization

Thanks for your enjoyable content

That was a good movie

Its all about uJ/inference and low Iq; checkout Syntiant and Greenwaves for example...

Will be interesting to see how Musk's, not sure if it is apples, DOJO works out.

Learned a lot more thanks to your video and (I assume) the ai generated voice. Anyway ai could maybe one day optimize its power consumption and manage a power grid to optimize the distribution.

0:49 It's the dude from Ireland. Duh ;)

Love your use of AI generated images throughout the video.

As an electrical engineer, a video like this is music to my ears.

interesting 🤔

Okay I love you goodbye!

Power not processing is the problem

a topic i'm really interested in is the use of analog computers for NNs. most NNs are not entirely precise by nature so I think it would make sense to run it with analog chips

0:06 Eat mor chikin

I’m amazed you didn’t mention Qualcomm, Apple or Google. All of them have very good edge AI solutions compared to everything you mentioned in this video. Whenever you open up your phone’s camera app, data is being fed to multiple neural networks for autofocus, lighting, super resolution, noise reduction… etc. Qualcomm even has their AI hardware exposed to app developers. Not sure about apple and Google, though.

Optical cpu🤔

We should demand open source firmware, or edge AI could be insecure at brain stem level. 😮

Constrained (edge) models will perform well once we have pretrained ANN based online reinforcement learning algorithms that solve discretization (episode length and SAR map resolution) and search strategy (not epsilon decay, a better bellman equation). I also don't think parameter quantization is reliably stable enough for chaotic systems (deep nets) and stochastic dropout mine as well be Monte Carlo architecture search. Quantization is a useful techniques but I don't think its the future of edge learning and is a nascent attempts at porting big data to the edge instead of finding an edge first solution or integration to deep net models. Inference at the edge is underutilizing the edge. We need fine tuning at the edge and continual learning. Tesla's closed loop data feedback and update pipeline is an interesting approach that utilizes the constantly shifting domain seen by the model at the edge.

I don’t think I’ve ever heard of the terminology ‘edge’ devices, I believe they’re called IOT (internet of things) devices in Cisco and maybe elsewhere

You need to start charging for this. By no means stop posting.

Today I learned Github Copilot exists.

The real problem is that computer neural networks don't work like biological neural networks.

Why not offload computing to an edge cloud? This combined with 5G would solve the latency constraint.

I hope we eventually destroy disease with better computers especially cancer

Second

Why do I always get you in my recommendations, also when I blocked you in every way possible?

What about the Akida 1000 by Brainchip Corp ? You seem to have missed a winning entry. And deleted my post ?

tinygrad

Good lord, what tosser bought a license plate with "TSLA S1"!?!

Deep learning ≠ A.I. Actual A.I. should be able to learn just by watching once or twice, then a little practice just like a human

Edge, edge, the edge, the Edge -- what is the Edge, edge?

Amazing video and thank

Sounds like you’ve never heard of Palantir

The reason there aren’t that many good edge chips is because the market is small.

I am really not all that impatient for ultra powerful AI to appear on The Edge, as so much of this tech gets misused. Generally, it's just not making the world a better place.

If I hear about the Asianometry Newsletter one more time today I'm actively spending resources against it.

Great video @asiaonometry! Iḿ suspect but I tend to favor NVidia basically because is easier to deliver the models and inference code, basically what changes is the Docker container and a few libraries. Besides that flexibility comes at a cost of power consumption. For instance if I want them running 24x7 on solar independent power I need a large solar panel AND lot's of batteries

thanks