Running LLMs on the NPU of the Rockchip RK3588
Рет қаралды 2,454
Күн бұрынIn this video I show you running a Large Language Model (LLM) on the NPU of the Rockchip RK3588.
With the Ubuntu 24.04 version by Joshua Riek for Rockchip RK3588 SBCs, it comes with the NPU driver version 0.9.6.
You can find it here: github.com/Joshua-Riek/ubuntu...
Someone on Reddit posted this information: / psa_ubuntu_rockchip_fr...
You can find the program to use the NPU here: github.com/Pelochus/ezrknn-llm
Converted model files: huggingface.co/Pelochus/ezrkl...
Allow more open files: ulimit -n 16384
Check NPU load: sudo cat /sys/kernel/debug/rknpu/load
00:00 Intro
00:13 Reddit RockchipNPU Community
01:13 ezrknpu ezrknn-llm
02:26 Installation
04:33 ulimit
05:23 Llama 2 7B on the NPU
09:25 Llama 2 7B on the CPU
If this video was helpful, please like, comment and subscribe!
Bluesky: bsky.app/profile/livinglinux....
#Radxa #Rock5 #rockchip #RK3588 #ubuntu #ai
@ribeiro4642 23 күн бұрын
Obrigado pelo vídeo!
@devlogschannel 28 күн бұрын
hi, thanks for sharing great video, but is there anyway to fully use 3 cores of npu
@stegofroggy Ай бұрын
Was the power consumption 8 watts when running two of the three NPU cores? I wonder how the power consumption changes as a function of the number of NPU cores in use.
@LivingLinux Ай бұрын
Idle is around 4W. Yes, it was running on 2 cores (but not at 100%) at around 8W. So my estimate is 12W total when running the 3 NPU cores at full load. Do note that I was running from a micro SD. Running from a m.2 drive might even use more power.
@StasNsky Ай бұрын
What was the exact speed in t/s for the llama 2 on the NPU?
@Crftbt Ай бұрын
Is the NPU failing to complete with the 7billion model due to running out of memory? Is there a log file somewhere?
@LivingLinux Ай бұрын
I don't think it's running out of memory. According to htop, memory usage is around 72% and stable. Perhaps there is a reason that the NPU driver still doesn't have a version number at 1 or higher.
@Freshbott2 Ай бұрын
Hi, sorry it's not really related to your video but did you compile uboot for this device? I'm at my wit's end trying to follow the Rockchip Wiki for uboot.
@LivingLinux Ай бұрын
No, I have never compiled uboot. Do you have a Radxa or Orange Pi board (or other)? It's probably better to ask in their forums. forum.radxa.com/ www.orangepi.org/orangepibbsen/
@Freshbott2 Ай бұрын
@@LivingLinux I've got the FriendlyElec CM3588 and a lot of regret, as I don't want to be dependent on someone's Google Drive for OS support (now) or into the future. But thankyou though I'll see if someone's got more detail for an Orange Pi.
@user-gq7kq7ju4t Ай бұрын
Hi. I think this content is Ubuntu on rk3588 and use rk3588's npu. If i use rk3568, can i use this source?
@LivingLinux Ай бұрын
It needs NPU driver 0.9.6. You can check it with this command: dmesg | grep -i rknpu
@user-gq7kq7ju4t Ай бұрын
Thanks for response. But rk3568's rknpu driver version is 0.9.0. I tried uploading kernel , but it doesn't easy. Could you tell me what is your devlopment board?
@LivingLinux Ай бұрын
@@user-gq7kq7ju4t I have the Radxa Rock 5B and 5A. I also have some Mekotronics devices, but I mainly use Android on them.
@ps3301 Ай бұрын
It is so slow. It might as well be useless.
@LivingLinux Ай бұрын
It's not fast, but it is energy-efficient.