You had to apply?

At first need to apply for groq API, but now don't need it.@@cjenkinsiv

is there a difference between the two model though ? When you test them via groq ?

It's super fast but they don't follow long-form instructions as well as ChatGPT.

lol nailed

Here's a long wall of text about the thing you're interested in. Now I think this is very interesting myself, what I understand is that it allows llms to be used on lower end hardware or something? I used AI to do this. Understanding AirLLM and Its Significance AirLLM is a groundbreaking technique that facilitates running a 70-billion-parameter large language model (LLM) on a single 4GB GPU, overcoming traditional hardware limitations. Traditionally, such large models would require more powerful and expensive hardware, restricting accessibility. **What is AirLLM?** AirLLM stands for a method allowing large language models to operate on smaller, less powerful GPUs. It employs two main techniques: 1. **Layer-wise Inference:** Breaks down the model into individual layers, loading only the necessary layers into memory during inference, reducing the overall memory footprint. 2. **Flash Attention:** An optimization within layer-wise inference, focusing on loading and executing only one layer at a time, further minimizing memory requirements. **Why is AirLLM Significant?** The significance lies in democratizing the use of large language models, making it accessible for personal projects, educational purposes, and small businesses. Practical implications include the ability to run advanced AI models, like chatbots, on smaller hardware, fostering innovation in various fields. **Running 70B LLM on a 4GB GPU with AirLLM Technique** **What it is:** AirLLM is a groundbreaking technique enabling the operation of a 70-billion-parameter large language model (LLM) on a single 4GB GPU, overcoming traditional hardware limitations. **How it works:** The technique employs layer-wise inference, dividing the model into layers and loading only the necessary parts into memory, drastically reducing memory requirements. **Key Features:** - **Layer-wise Inference:** Splits the model into layers, significantly cutting down on GPU memory usage. - **Flash Attention:** Optimizes single-layer execution, further reducing memory needs per layer. **Benefits:** Allows for efficient and effective inference on a 4GB GPU without compromising model performance, bypassing the need for high-end GPUs or excessive RAM. **Applications:** Ideal for scenarios where hardware resources are limited, such as personal projects or low-budget research. This technique represents a significant leap in making advanced AI models more accessible, enabling users with modest hardware to leverage the power of large LLMs for various applications.

The code is available in the playground.

nvidia killer is a company that will beat them in: 1. watt/token 2. $/token 3. first token latency 4. total generation time so far i see 4 was beaten, but it's the least important aspect.

257 LPU vs 1 H100? do the math and watch your wallet

How? I haven't tried out their API or service

What do you mean?