Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark

Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark | Databricks

Рет қаралды 61,277

Data Council

Күн бұрын

Пікірлер: 18

@tejusization 4 жыл бұрын

28:45 to 29:40 is the best!!! :D just don't miss that. sets the context

@HridyanshiB. 5 жыл бұрын

Good explanation about streaming..Thanks

@venkat.k4392 3 жыл бұрын

Appreciated. Thanks you for a great knowledge share.

@ashwinkumar5223 Жыл бұрын

Superb Explanation

@danielmackie82 5 жыл бұрын

What would be an open source equivalent of DB delta?

@mohitmehta3788 4 жыл бұрын

Very simplified approach of explaining streaming.

@yourstruly5DA 5 жыл бұрын

Good presentation. Would like to understand more how it could integrate and scale with Apache Kafka.

@satria5403 3 жыл бұрын

Hi, please let me know if u have good resources for this. thank you

@zhengfang303 4 жыл бұрын

When the data has entered the dataframe, if the data has been updated or deleted, how can I update or delete it in the dataframe?

@parthadeb3723 3 жыл бұрын

A dataframe is immutable. You cannot update a dataframe. You have to create an new dataframe.

@GrayMatterSoftware 3 жыл бұрын

Want to know about the best Practices for Real-Time Analytics Architecture on Big Data? Read here: www.graymatter.co.in/real-time-analytics-bigdata-architecture/ Know more: www.graymatter.co.in/real-time-analytics/ Watch here: kzfaq.info/get/bejne/or6UjM5kzcfPZWg.html

@thesleepyhead7273 5 жыл бұрын

How about integrating this with Tensorflow Serving for end to end Analytics paradigm

@michaelbrenndoerfer9908 5 жыл бұрын

Google Hydrogen

@sujithkumar804 4 жыл бұрын

@@michaelbrenndoerfer9908 lol

@JanekBogucki 4 жыл бұрын

23:30 A single rogue timestamp which is one hour ahead of the second max timestamp would drop all earlier buckets except one bucket corresponding to this single anomalous value. This is fragile.

@onewithsixonewithsix601 4 жыл бұрын

Unless there is crazy issue in code manipulating timestamp. It is not a probable scenario to get timestamp ahead of actual unix time.

@albertoandreotti7940 4 жыл бұрын

This is a big disappointment. You cannot stream pipelines built with dataframes. Unified processing framework?? come on! You have to build new versions of all your algorithms so now they can work with a DStream? What a waste of time.