Scribd
Upload
121 views

Elt Process With Snowflake Stored Procedure and Task

Uploaded by

Mahesh Reddy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
121 views

Elt Process With Snowflake Stored Procedure and Task

Uploaded by

Mahesh Reddy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 6

ELT PROCESS

with Snowflake Stored Procedure and Task


ELT
ELT is the process of extracting raw data from one or multiple sources and loading into the target
table/data warehouse. On a high-level ELT has the following steps:
• Step I: Raw data will be extracted from various source systems.
• Step II: Raw data will be loaded into target tables/data warehouse without any transformation.
• Step III: Transformation will be applied to the target system to perform the data transformation.

ELT Highlights
• Cloud-based data warehouses offer unlimited storage as well as compute processing power.
This make ELT more viable on cloud-based servers.
• In the ELT world, it transforms only the data required for business decision.
• ELT allows you to load all forms of data immediately once it is available.

ELT Process

This article attempts to discuss how one can perform data integration from snowflake staging area to final table,
using snowflake stored procedure and schedule it using “Task”.
Stored Procedure
A stored procedure is useful to perform one or more SQL, data transform and data validation. A stored procedure
may contain one or many statements and even call additional stored procedures, passing parameters through as
required.
Currently following UDF’s are supported in snowflake:
• SQL
• Javascript
SQL UDF:
• A SQL UDF evaluates a random SQL expression and returns the result in the form of tabular or scalar format.
Javascript:
• A JavaScript UDF lets one use the JavaScript programming language to manipulate data and return either scalar
or tabular results
ELT Process

At the end of this article, one would have preliminary information on,
• How to create a stored procedure in snowflake
• How to call one stored procedure from another procedure
• Variable concatenation / binding
Snowflake stored procedure will read the metadata table, execute respective SQL and return the status:

ELT_SCHEMA_DETAILS

Column Name Data Type Constraint Default Value

ELT_ID NUMBER(10) NOT NULL NULL

ETL_SQL VARCHAR NOT NULL NULL

Note: Loading data from the stage table to the target table can be done using snowflake Merge statement. In order
to process the data from the stage table to the target table using a merge statement, we need the primary key.

Snowflake merge syntax

MERGE
INTO <target_table>
using <source>
ON <join_expr> { matchedClause | notMatchedClause } [ ... ]

Snowflake Stored Procedure


Calling Stored Procedure

How the procedure works

• Main stored procedure accepts the following input parameter:


o ELT_ID
• Procedure reads respective SQL associate with ELT_ID from snowflake metadata table.
• Execute the merge / SQL statement to catch an error, if any.

Scheduling stored procedure using Task

Snowflake task allows to schedule a SQL script or stored procedure on snowflake instance. Snowflake support
CRON based job schedule. CRON is the Linux version of windows task schedule, and it has a simpler mechanism
regards to run a job.

Min Hour Day Month Week Day Description

* * * * * Run every min, hour and day

10 16 * * Mon Every Mon day 4:10 PM

10 * * * * Every 10 Min
Create Task

Following task will run the stored procedure every minute.

Assign privileges to named role

Validate the status of task

Alter the status of task

Task execution history


www.youtube.com/user/techmahindra09
www.facebook.com/techmahindra
www.twitter.com/tech_mahindra
www.linkedin.com/company/tech-mahindra
www.techmahindra.com

Copyright © Tech Mahindra 2021. All Rights Reserved.


Disclaimer. Brand names, logos and trademarks used herein remain the property of their respective owners.

You might also like