Abstract
The Azure Databricks ecosystem has been around for about five years. During that time, the vendor has kept on improving both the interface and the design patterns. There are three new things that you should be using in your data engineering projects.
First, scheduling jobs has always been a part of the product. However, with workflows, one can control the pattern in which notebooks are executed.
Second, delta live tables allow the developer to specify the source, the transformation, and the destination of data using Python. This complete workflow can be scheduled as either a batch or real-time streaming job.
Third, cloning technology has come to delta tables. One can use shallow clones to keep different environments in synch with the master. On the other hand, deep clones allow the developer to stamp out a table at a given point in time.
To recap, come to this presentation to learn about these new design patterns and get a demonstration on how to apply them in your data engineering projects.
Speaker Bio
John Miner is a Senior Data Architect at Insight Digital Innovation helping corporations solve their business needs with various data platform solutions.
He has over thirty years of data processing experience, and his architecture expertise encompasses all phases of the software project life cycle, including design, development, implementation, and maintenance of systems.
His credentials include undergraduate and graduate degrees in Computer Science from the University of Rhode Island. Also, he has earned certificates from Microsoft for Database Administration (MCDBA), System Administration (MCSA), Data Management and Analytics (MCSE), and Data Science (MPP).
John has been recognized with the Microsoft MVP award eight times for his outstanding contributions to the Data Platform community.
When he is not busy talking to local user groups or writing blog entries on new technology, he spends time with his wife and daughter enjoying outdoor activities. Some of John’s hobbies include woodworking projects, crafting a good beer, and playing a game of chess.