In this video, we'll dive into how to use Data Wrangler in Microsoft Fabric to cleanse and group by/aggregate data as part our ETL pipeline. We'll walk through some common steps used in data transformation, and include a few examples on deriving new columns from existing ones, such as extracting the hour from a timestamp.
Links:
Source Code aka.ms/fabricrealtimelab
Data Wrangler learn.microsoft.com/en-us/fab...
Chapters:
00:00:00 Intro
00:00:25 Eventstream to lakehouse
00:01:26 Notebook overview
00:03:17 Creating a "anomaly" test dataframe
00:06:00 Loading Data Wrangler
00:08:37 Drop missing values
00:09:21 Filter invalid data
00:11:58 Alter the generated code
00:14:00 Derive date from timestamp
00:17:00 Derive hour from timestamp
00:19:00 Derive minute from timstamp / handling incorrect flash fill examples
00:20:12 Convert to integers
00:21:38 Group by and aggregate
00:23:58 Modifying the generated code
00:28:09 Additional aggregations
00:31:05 Query the tables