Workshop: Optimizing Kubernetes Cluster Scaling for Advanced Generative Models - Shivay Lamba & Shivanshu Raj Shrivastava, SigNoz
As more organisations are seeking automation, the demand for scalable infrastructure capable of supporting complex generative models is at an all-time high. Kubernetes has emerged as a leading solution for orchestrating and managing containerized applications, including machine learning workloads. Some recent enhancements in Kubernetes have provided a way to effectively use GPU resources, and build drives for GPU to dynamically share the resources within a Kubernetes node. This talk will delve into the intricacies of scaling Kubernetes clusters to accommodate the computational demands of cutting-edge generative models.
We will explore how we have evaluated and adopted various tools and frameworks such as Flyte, MLflow, Kubeflow, Kserve, vLLM, and Argo to seamlessly integrate into our hybrid Kubernetes clusters to streamline the development, deployment, and management of generative models. Attendees will gain insights into the unique features and capabilities of each tool and understand how they contribute to the scalability, efficiency and maintainability of our hybrid Kubernetes clusters.