Watch additional Chill Data Summit 2024 recordings at www.upsolver.com/resources/ev...
Description
Jack from Amazon EMR and Athena shares his expertise on integrating the REST Catalog Specification with Spark and Trino. As a member of the Iceberg community and PMC member, Jack delves into the details of how his team manages table formats and storage services, focusing on the integration of REST Catalog Specification with Spark and Trino.
Key Topics Covered:
- Introduction to REST Catalog Specification: Jack explains the differences between Glue Data Catalog and REST Catalog, highlighting the unique needs of different customer types.
- Customer Use Cases: Learn about various customer scenarios where REST Catalog Specification is preferred, including third-party vendors and in-house data catalog solutions.
- Internal Experimentation at Amazon: Discover how Amazon experimented with building an internal data catalog service, leading to performance improvements and optimization research.
- Performance Improvements: Jack showcases intelligent scan planning techniques, reducing scan times from minutes to seconds, and the implementation of a scan API to enhance performance.
API Enhancements: Explore the introduction of new APIs for better scan and commit operations, promoting faster and more efficient data handling.
- Future Prospects: Jack discusses potential enhancements and optimization directions for Iceberg, driven by practical customer feedback and production use cases.
Tags: #AmazonEMR #Athena #Spark #Trino #Iceberg #DataIntegration #RESTCatalog #PerformanceImprovements #APIs #BigData