At Splunk we have built a Flink streaming infrastructure and scaled it to petabytes of data per day and millions of events per second. Along the way, we've learned a lot about writing performant operations in the DataStream API and putting together high-throughput pipelines. This talk will cover our real-life experiences in scaling for this throughput and the best practices we've learned along the way. We'll talk about aggregation, built-in functions, user-defined functions, the async I/O API, as well as discoveries around GC, Java object management, state backends, and serialization.
0:00 Introduction
1.07 Environmental & Performance Testing
3:57 The basic challenges
10:17 Sources
21:22 Sinks
25:10 Putting it all together