How to Log Pipeline Audit Data for Success and Failure in Azure Data Factory

How to Log Pipeline Audit Data for Success and Failure in Azure Data Factory - ADF Tutorial 2021

Рет қаралды 17,098

2 жыл бұрын

How to Log Pipeline Audit Data for Success and Failure in Azure Data Factory - ADF Tutorial 2021, in this video we are going to learn How to Log Pipeline Audit Data for Success and Failure in Azure Data Factory - ADF Tutorial 2021, Azure Data Factory Step by Step - ADF Tutorial 2021 - ADF Tutorial 2021 Step by Step ADF Tutorial - Azure Data Factory Tutorial 2021
Azure Data Factory Tutorial for beginners
Azure Data Factory Tutorial 2021
Step by step Azure Data Factory Tutorial
Real-time Azure Data Factory Tutorial
Scenario-based training on Azure Data Factory
Best ADF Tutorial on youtube
Failure
SELECT '@{pipeline().DataFactory}' as DataFactory_Name,
'@{pipeline().Pipeline}' as Pipeline_Name,
'@{pipeline().RunId}' as RunId,
'TotalSaleTable' as Source,
'TotalSale CSV File' as Destination,
'@{activity('CopyTotalSale').output.executionDetails[0].status}' as Execution_Status,
'@{activity('CopyTotalSale').output.errors[0].Code}' as ErrorCode,
'@{replace(activity('CopyTotalSale').output.errors[0].Message,'''','')}' as ErrorDescriprion,
'@{utcnow()}' as ErrorLoggedTime
Success
SELECT '@{pipeline().DataFactory}' as DataFactory_Name,
'@{pipeline().Pipeline}' as Pipeline_Name,
'@{pipeline().RunId}' as RunId,
'TotalSaleTable' as Source,
'TotalSale CSV File' as Destination,
'@{activity('CopyTotalSale').output.rowsRead}' as RowsRead,
'@{activity('CopyTotalSale').output.rowsCopied}' as RowsCopied,
'@{activity('CopyTotalSale').output.copyDuration}' as copyDuration,
'@{utcnow()}' as ErrorLoggedTime
#AzureDataFactory #ADFTutorial2021

Пікірлер: 12

@PaulLundgren1970 3 ай бұрын

This tutorial was very helpful and easy to follow. Thank you very much.

@franciscoanalytics Жыл бұрын

Really interesting video. Is it possible to do the same in a pipeline with more than one activity? How would you do in that case?

@prashanthbalakrishnan4541 2 жыл бұрын

Can we use Synapse Linked services instead of Sql DB for running the query ?

@etcindy 2 жыл бұрын

Is there a way to set up the expression so that it can pass error, status and Erroredlogtime from any copy data task in the pipeline, or does each copy task have to have its own

@vivekvp Жыл бұрын

Hello - very helpful - thank you! In my case I want to log the status of a copy from from on on-premise file share to blob storage, not a database. I won't be able to write select statements as I have no DB. Suggestions or thoughts?

@yedukondalu9100 Жыл бұрын

you can use stored procedure bro

@dpusplatha4489 Жыл бұрын

Can we send custom message codes to the response json in case of success?

@deepjyotimitra1340 2 жыл бұрын

How to know who triggered the pipeline on run time?

@kunalbhatia5404 2 жыл бұрын

I tried all the same steps which you showed in the video, but not succeeded. Error Message is not logging in my blob .. How can I get the answer or suggestion..?

@kunalbhatia5404 2 жыл бұрын

Getting Error : The expression 'activity('Data flow1').output.Error' cannot be evaluated because property 'Error' doesn't exist, available properties are 'runStatus, effectiveIntegrationRuntime, billingReference, reportLineageToPurview'. Any Solution ??

@lalitb20 Жыл бұрын

Same error, did u get any solution?