Indeed.

I was thinking the same about offloading to RAM, as it has become so much cheap. However on my quick search could not find that option yet with that lib. Will need to investigate more. If you find please let me know as well.

... isnt this question ironic? doesnt LLM naturually load into RAM / VRAM, and the whole reason of this project is to switch it to an Actual Storage Drive so You can use the 70B in the Drive instead of having issues with over loading VRAM / RAM

@@i6od Indeed, this project brings a completely new way to deal with LLMs beyond RAM/VRAM.

Really simple to do just create a ram Drive a simple application you can download and then put your model into the Rand Drive and load it from there and you're all set

I would then just try to use a RAM-disk..

theoretically possible . The layered inference approach will just do the sequential loading and unloading of model layers. Ofcourse, the latency will accumulate and result in super super slow inference.

Yes, I think its possible i.e. you can managing the num of layers to allocate to GPU reducing the frequency of SSD reads. Here's the long ans. - In the current implementation of their code (check the github repo), the `AirLLMBaseModel` class in `airllm_base.py` loads and processes one layer at a time during the forward pass. However, you can modify the `forward` method to load and cache a certain number of layers based on the available GPU memory. For example, you can introduce a configuration parameter to specify the number of layers to cache in GPU memory. Then, in the `forward` method, you can load and store the layers in a cache until the specified number of layers is reached. When processing the next layer, you can check if it is already in the cache before loading it from SSD. Here's a simplified example of how you could modify the `forward` method to cache multiple layers: ```python def forward(self, ...): ... cached_layers = [] max_cached_layers = 4 # Specify the maximum number of layers to cache for i, (layer_name, layer) in enumerate(zip(self.layer_names, self.layers)): if layer_name in cached_layers: # Layer is already cached, use it directly layer = self.cached_layers[layer_name] else: # Load the layer from SSD and add it to the cache state_dict = self.load_layer_to_cpu(layer_name) self.move_layer_to_device(state_dict) cached_layers.append(layer_name) self.cached_layers[layer_name] = layer # Remove the oldest cached layer if the cache size exceeds the maximum if len(cached_layers) > max_cached_layers: oldest_layer = cached_layers.pop(0) del self.cached_layers[oldest_layer] # Process the layer ... ``` In this example, the `max_cached_layers` variable determines the maximum number of layers to cache in GPU memory. The `cached_layers` list keeps track of the currently cached layers. When processing a layer, it first checks if it is already cached. If not, it loads the layer from SSD, adds it to the cache, and removes the oldest cached layer if the cache size exceeds the maximum. - By caching multiple layers in GPU memory, you can reduce the number of SSD reads required during inference. Additionally, you may need to handle the case where a single layer itself exceeds the available GPU memory. In such scenarios, you might need to explore other techniques like tensor parallelism or model sharding to distribute the layer across multiple GPUs or devices.

No haven't tried on that yet, but will try.

Dont think ollama supports this.

Yes I think you are talking about "LLM in a Flash: Efficient Large Language Model Inference with Limited Memory" twitter.com/rohanpaul_ai/status/1737425137451073573

google "RAM disk" still a thing in Win 11...

Yes offloading to RAM will always be better given how cheap it is. But this library wanted a new way to deal with LLM bypassing RAM/VRAM as much as possible.

You just need to be able to accomodate the entire model into your SSD.

Depends on SSD read speed. It may vary but in Mac hardware was getting 1 tok/2sec.

@@RohanPaul-AI is this an regular SSD or an NVME?

@@Gatrehs its NVME

Yes, but actual speed may not improve much as you still have to do Disk IO. So you will always be bottlenecked by your SSD read speed.

Exactly. The smaller the model the higher the proportional IO overhead compared to compute… at 8B paging memory like this in and out makes it far slower than it is right now. That is because the compute time needed per additional parameter in an XB model grows exponentially. So large models are so slow in compute, that IO overheads like these can become neglectable. But there are interesting developments like vLLMs that use something like virtual memory management to pack a very large model still in small GPU memory. Skipping the need for IO speed (since there is no IO to disk), since everything is still in memory on the graphics card.

@@dinoscheidt very well explained. Thanks.

To my understanding, Groq uses a similar trick as their LPU has only 250MB(yes MB) memory

@@dinoscheidt can't wait for vLLM Llama 3 400B. For a few years I've been hoping for something like that, then really top level AI can be run locally by anyone with a reasonable computer and ok graphics card... Will be amazing for productivity, gaming, etc.

Yes, you can use the locally downloaded model's local path like below model = AutoModel.from_pretrained("/home/ubuntu/.cache/huggingface/hub/models--garage-bAInd--Platypus2-70B-instruct/snapshots/b585e74bcaae02e52665d9ac6d23f4d0dbc81a0f")

​@@RohanPaul-AIThank you! ❤