Into Generative AI with PyTorch Lightning 2.0 - Luca Antiga & Carlos Mocholí, Adrian Walchli Lightning AI
Four years after its initial release, PyTorch Lightning has become one of the most used deep learning frameworks. It has been adopted by tens of thousands of companies and academic groups, contributing to drive the adoption of PyTorch to where it is today. The advent of generative AI has led to new challenges in training large models, and PyTorch Lightning has empowered the industry by making many of the latest innovations accessible and robust. PyTorch Lightning powers state-of-the-art generative AI models like StableDiffusion and SDXL. It has been adopted in exciting new directions for LLMs like Hyena Hierarchy and State Space models, as well as the new RWKW recurrent architecture. On top of all that, the PyTorch Lightning-based NVIDIA NeMo, which includes NeMo Megatron, is enabling companies to train LLMs up to hundreds billions parameters. In this talk we will explore generative AI applications powered by PyTorch Lightning, and cover the latest PyTorch Lightning 2.0 features that make working with large models easy. We will also discuss how Lightning Fabric powers lit-gpt, which has been adopted as the starter kit for the recent LLM Efficiency Challenge at NeurIPS 2023.