Scribd
Upload
12 views14 pages

09 Chapter9+ +Controlling+Pipeline+Flow

Uploaded by

thulasi narravula
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views14 pages

09 Chapter9+ +Controlling+Pipeline+Flow

Uploaded by

thulasi narravula
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

1 Asserts

Overview of Assert Transformation

2 Implementing Asserts

Controlling
Implementing an Assert Transformation

Pipeline Flow 3 Error Handling


Overview of Pipeline Error Handling

4 Implementing Error Handling


Implementing Pipeline Error Handling

MODULE 9
SECTION 1
Asserts
Asserts
Overview of Assert Transformation

• Build custom rules in mapping data flows for data


quality
• Rules ensure that data quality constraints are met

Data Quality

01

Data Validation 02 03 Control Data Flow


• Rules to ensure that data values meet an expected • Control data flow by raising error messages when
value domain constraints are not met
• Validation rules enforced for each row
Asserts
Overview of Assert Types in an Assert Transformation

• Result of the expression must evaluate to a boolean


true result
• Used for validating values as per a constraint

Expect True

01

Expect Unique 02 03 Expect Exists


• Set a column or an expression as a uniqueness rule • Used in checking if a particular row exists within both
• Used to tag duplicate rows input data sets
• This assert type is only available when two input data
sets are provided
SECTION 2
Implementing Asserts
Implementing Asserts
Implementing an Assert Transformation

Screencast
Describe the use case – that our data should only contain sales regions as “UK” or “EU” and any other
values are considered as error rows

Show how to add an assert transformation and use the assert type “expect true”

Show how to use the derived column transformation and the conditional split transformation to identify
error rows

Show how to direct error rows

As a recap show the final dataset without the error rows and the separate date file that
contains error rows
SECTION 3
Error Handling
Error Handling
Overview of Pipeline Error Handling

Data Factory allows conditional logic and provides pipeline control flow based upon outcomes of
an activity

(Default Pass) Execute this path if the current activity


succeeded Execute this path if the current activity failed

Upon Success 01 02 Upon Failure

Upon Completion 03 04 Upon Skip


Execute this path if the current activity completed, Execute this path if the activity itself didn’t run
regardless if it succeeded or not
Error Handling
Overview of Pipeline Error Handling

Common Error Handling mechanisms

Try Catch Block Do If Else Block Do If Skip Else Block


Defines only Upon Failure Path Defines Upon Success and Upon Failure Path Defines upon Success and Upon Failure Path (dummy Upon
Skip Path)
• Activity succeeds => Pipeline succeeds • Activity succeeds => Pipeline succeeds
• Activity succeeds => Pipeline succeeds
• Activity Fails => Pipeline succeeds • Activity Fails => Pipeline Fails
• Activity Fails => Pipeline succeeds
SECTION 4
Implementing Error Handling
Error Handling
Implementing Error Handling

What we will learn


Add Error Handling to the Pipeline

Capture Errors to an Error Log File

Create an Error condition and test the Pipeline


Implementing Error Handling
Implementing Pipeline Error Handling

Screencast
Describe the use case – that could cause an error

Run the data pipeline to show the pipeline failure

Show how to implement error handling for the use case

Run the data pipeline to show how the error is handled


Module Summary
In this module we learnt

Overview Control Flow Hands-On


We got an overview of the assert We learnt about pipeline error handling and We learnt how to implement asserts to
transformation and where it is used the different error handling mechanisms manage data quality

We learnt how to implement error handling to


control pipeline flow
References
Assert Transformation
https://learn.microsoft.com/en-us/azure/data-factory/data-flow-assert

Pipeline Error Handling


https://learn.microsoft.com/en-us/azure/data-factory/tutorial-pipeline-failure-error-handling

Monitor Pipelines with Email Notifications


https://learn.microsoft.com/en-us/azure/data-factory/how-to-send-email

You might also like