100% found this document useful (5 votes)
72 views

SQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks with T-SQL, SSIS, and Azure Data Factory Andy Leonard pdf download

The document discusses the creation of metadata-driven frameworks for SQL Server data automation using T-SQL, SSIS, and Azure Data Factory. It includes detailed chapters on stored procedures, automation, SSIS frameworks, and Azure integration, providing practical examples and guidelines for implementation. The authors, Andy Leonard and Kent Bradshaw, aim to equip readers with the knowledge to build efficient data automation solutions.

Uploaded by

risketsevon6u
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (5 votes)
72 views

SQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks with T-SQL, SSIS, and Azure Data Factory Andy Leonard pdf download

The document discusses the creation of metadata-driven frameworks for SQL Server data automation using T-SQL, SSIS, and Azure Data Factory. It includes detailed chapters on stored procedures, automation, SSIS frameworks, and Azure integration, providing practical examples and guidelines for implementation. The authors, Andy Leonard and Kent Bradshaw, aim to equip readers with the knowledge to build efficient data automation solutions.

Uploaded by

risketsevon6u
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 55

SQL Server Data Automation Through Frameworks:

Building Metadata-Driven Frameworks with T-SQL,


SSIS, and Azure Data Factory Andy Leonard
download
https://textbookfull.com/product/sql-server-data-automation-
through-frameworks-building-metadata-driven-frameworks-with-t-
sql-ssis-and-azure-data-factory-andy-leonard/

Download more ebook from https://textbookfull.com


We believe these products will be a great fit for you. Click
the link to download now, or visit textbookfull.com
to discover even more!

SQL Server Execution Plans For SQL Server 2008 through


to 2017 and Azure SQL Database 3rd Edition Grant
Fritchey

https://textbookfull.com/product/sql-server-execution-plans-for-
sql-server-2008-through-to-2017-and-azure-sql-database-3rd-
edition-grant-fritchey/

Building Custom Tasks for SQL Server Integration


Services: The Power of .NET for ETL for SQL Server 2019
and Beyond 2nd Edition Andy Leonard

https://textbookfull.com/product/building-custom-tasks-for-sql-
server-integration-services-the-power-of-net-for-etl-for-sql-
server-2019-and-beyond-2nd-edition-andy-leonard/

Building Custom Tasks for SQL Server Integration


Services The Power of NET for ETL for SQL Server 2019
and Beyond Second Edition Andy Leonard

https://textbookfull.com/product/building-custom-tasks-for-sql-
server-integration-services-the-power-of-net-for-etl-for-sql-
server-2019-and-beyond-second-edition-andy-leonard/

Biota Grow 2C gather 2C cook Loucas

https://textbookfull.com/product/biota-grow-2c-gather-2c-cook-
loucas/
Quick Start Guide to Azure Data Factory Azure Data Lake
Server and Azure Data Warehouse 1st Edition Mark
Beckner

https://textbookfull.com/product/quick-start-guide-to-azure-data-
factory-azure-data-lake-server-and-azure-data-warehouse-1st-
edition-mark-beckner/

Data Integration Life Cycle Management with SSIS: A


Short Introduction by Example 1st Edition Andy Leonard

https://textbookfull.com/product/data-integration-life-cycle-
management-with-ssis-a-short-introduction-by-example-1st-edition-
andy-leonard/

SQL Server Big Data Clusters : Data Virtualization,


Data Lake, and AI Platform Benjamin Weissman

https://textbookfull.com/product/sql-server-big-data-clusters-
data-virtualization-data-lake-and-ai-platform-benjamin-weissman/

Azure SQL Revealed: A Guide to the Cloud for SQL Server


Professionals Bob Ward

https://textbookfull.com/product/azure-sql-revealed-a-guide-to-
the-cloud-for-sql-server-professionals-bob-ward/

Azure SQL Revealed: A Guide to the Cloud for SQL Server


Professionals 1st Edition Bob Ward

https://textbookfull.com/product/azure-sql-revealed-a-guide-to-
the-cloud-for-sql-server-professionals-1st-edition-bob-ward/
SQL Server Data
Automation
Through
Frameworks
Building Metadata-Driven Frameworks with
T-SQL, SSIS, and Azure Data Factory

Andy Leonard
Kent Bradshaw
SQL Server Data
Automation Through
Frameworks
Building Metadata-Driven
Frameworks with T-SQL, SSIS,
and Azure Data Factory

Andy Leonard
Kent Bradshaw
SQL Server Data Automation Through Frameworks: Building Metadata-Driven
Frameworks with T-SQL, SSIS, and Azure Data Factory
Andy Leonard Kent Bradshaw
Farmville, VA, USA Providence Forge, VA, USA

ISBN-13 (pbk): 978-1-4842-6212-2 ISBN-13 (electronic): 978-1-4842-6213-9


https://doi.org/10.1007/978-1-4842-6213-9

Copyright © 2020 by Andy Leonard, Kent Bradshaw


This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with
every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an
editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the
trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not
identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to
proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication,
neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the
material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Jonathan Gennick
Development Editor: Laura Berendson
Coordinating Editor: Jill Balzano
Cover image designed by Freepik (www.freepik.com)
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street,
6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-
sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member
(owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a
Delaware corporation.
For information on translations, please e-mail [email protected]; for reprint,
paperback, or audio rights, please e-mail [email protected].
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and
licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales
web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to
readers on GitHub via the book’s product page, located at www.apress.com/9781484262122. For more
detailed information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
For Christy
—Andy

For Ann
—Kent
Table of Contents
About the Authors���������������������������������������������������������������������������������������������������� xi

About the Technical Reviewer������������������������������������������������������������������������������� xiii


Acknowledgments���������������������������������������������������������������������������������������������������xv

Introduction�����������������������������������������������������������������������������������������������������������xvii

Part I: Stored Procedure-Based Database Frameworks������������������������������������������� 1

Chapter 1: Stored Procedures 101��������������������������������������������������������������������������� 3


The Need for a Framework����������������������������������������������������������������������������������������������������������� 3
Demonstration of a Framework���������������������������������������������������������������������������������������������������� 4
An Example Schema���������������������������������������������������������������������������������������������������������������� 5
The Daily Process�������������������������������������������������������������������������������������������������������������������� 6
Executing the Daily Process�������������������������������������������������������������������������������������������������� 12
Including a Monthly Process������������������������������������������������������������������������������������������������� 12
Summary������������������������������������������������������������������������������������������������������������������������������������ 18

Chapter 2: Automation with Stored Procedures����������������������������������������������������� 19


A Daily Process Controller����������������������������������������������������������������������������������������������������������� 19
Monthly Process Controller��������������������������������������������������������������������������������������������������������� 23
Summary������������������������������������������������������������������������������������������������������������������������������������ 26

Chapter 3: Stored Procedure Orchestrators����������������������������������������������������������� 27


The Orchestrator������������������������������������������������������������������������������������������������������������������������� 27
A Quick Process Overview���������������������������������������������������������������������������������������������������������� 31
A Look at Troubleshooting Issues������������������������������������������������������������������������������������������������ 31
Summary������������������������������������������������������������������������������������������������������������������������������������ 32

v
Table of Contents

Chapter 4: A Stored Procedure–Based Metadata-Driven Framework�������������������� 33


Building the Metadata����������������������������������������������������������������������������������������������������������������� 33
Metadata-Ready Controllers������������������������������������������������������������������������������������������������������� 40
Metadata-Ready Orchestrator����������������������������������������������������������������������������������������������������� 47
Troubleshooting with Metadata�������������������������������������������������������������������������������������������������� 50
Summary������������������������������������������������������������������������������������������������������������������������������������ 56

Part II: SSIS Frameworks���������������������������������������������������������������������������������������� 57

Chapter 5: A Simple, Custom, File-­Based SSIS Framework������������������������������������ 59


An SSIS Framework, Defined and Designed������������������������������������������������������������������������������� 60
Functionality�������������������������������������������������������������������������������������������������������������������������� 60
Empathy�������������������������������������������������������������������������������������������������������������������������������� 60
Simplicity������������������������������������������������������������������������������������������������������������������������������� 60
Building a File-Based SSIS Framework�������������������������������������������������������������������������������������� 61
Obtain the Code��������������������������������������������������������������������������������������������������������������������� 61
Metadata-Driven Execution Management����������������������������������������������������������������������������� 62
The SSISConfig Database������������������������������������������������������������������������������������������������������ 62
Config Schema���������������������������������������������������������������������������������������������������������������������� 63
The Config.Applications Table������������������������������������������������������������������������������������������������ 65
The Config.Packages Table���������������������������������������������������������������������������������������������������� 67
A Sample SSIS Solution�������������������������������������������������������������������������������������������������������������� 75
Configure Visual Studio 2019 for SSIS Development������������������������������������������������������������� 76
Create the Sample SSIS Solution������������������������������������������������������������������������������������������ 78
SSIS Framework Metadata Management����������������������������������������������������������������������������������� 85
Add an SSIS Application�������������������������������������������������������������������������������������������������������� 85
Add SSIS Packages��������������������������������������������������������������������������������������������������������������� 88
Assign SSIS Application Packages���������������������������������������������������������������������������������������� 91
Conclusion���������������������������������������������������������������������������������������������������������������������������������� 96

vi
Table of Contents

Chapter 6: Framework Execution Engine��������������������������������������������������������������� 97


Create a Parent SSIS Package���������������������������������������������������������������������������������������������������� 97
Log Execution Values������������������������������������������������������������������������������������������������������������������ 98
Retrieve SSIS Application Packages from SSISConfig�������������������������������������������������������������� 104
Iterate Application Packages����������������������������������������������������������������������������������������������������� 120
Log Application Package Values����������������������������������������������������������������������������������������������� 130
Execute Application Packages�������������������������������������������������������������������������������������������������� 135
Log Execution Results��������������������������������������������������������������������������������������������������������������� 150
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 158

Chapter 7: Framework Logging���������������������������������������������������������������������������� 159


Create a Log Schema���������������������������������������������������������������������������������������������������������������� 159
Add Application Instance Logging to Parent.dtsx���������������������������������������������������������������������� 164
Add Application Package Instance Logging to Parent.dtsx������������������������������������������������������� 176
Viewing Execution Report Data������������������������������������������������������������������������������������������������� 191
Configure Fault Tolerance in SSIS Framework Application Metadata��������������������������������������� 197
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 206

Chapter 8: Azure-SSIS Integration Runtime��������������������������������������������������������� 207


Getting Started with Azure�������������������������������������������������������������������������������������������������������� 207
Provisioning an Azure Data Factory������������������������������������������������������������������������������������������ 208
Provisioning Azure Storage������������������������������������������������������������������������������������������������������� 217
Provisioning an Azure-SSIS Integration Runtime for SSIS Package Files��������������������������������� 223
Stopping the Azure-SSIS Integration Runtime�������������������������������������������������������������������������� 236
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 240

Chapter 9: Deploy a Simple, Custom, File-Based Azure-SSIS Framework������������ 241


Provisioning the SSISConfig Database�������������������������������������������������������������������������������������� 241
Deploy the Simple, Custom, File-Based Azure-SSIS Framework���������������������������������������������� 249
Provision an Azure File Share��������������������������������������������������������������������������������������������������� 267
Upload SSIS Packages��������������������������������������������������������������������������������������������������������� 270
Update PackageLocation Values������������������������������������������������������������������������������������������ 272

vii
Table of Contents

Build the SSIS Framework ADF Execution Engine�������������������������������������������������������������������� 273


Retrieve a List of SSIS Packages����������������������������������������������������������������������������������������� 273
Execute the Retrieved SSIS Packages��������������������������������������������������������������������������������� 287
View Test Execution Logs���������������������������������������������������������������������������������������������������� 296
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 299

Chapter 10: Framework Logging in ADF��������������������������������������������������������������� 301


Add the ApplicationName Parameter���������������������������������������������������������������������������������������� 301
A Quick Review of SSIS Framework Applications and Packages���������������������������������������� 302
Add Application Instance Logging��������������������������������������������������������������������������������������������� 303
Modifying the Log.ApplicationInstance Table���������������������������������������������������������������������� 303
Adding the Log.InsertApplicationInstance Stored Procedure���������������������������������������������� 304
Logging Application Instance���������������������������������������������������������������������������������������������� 306
Adding the Log.UpdateApplicationInstanceStatus Stored Procedure���������������������������������� 311
Updating Application Instance��������������������������������������������������������������������������������������������� 312
Let’s Test It!������������������������������������������������������������������������������������������������������������������������� 316
Updating Application Instance on Success�������������������������������������������������������������������������� 317
Adding the ApplicationPackageId Field������������������������������������������������������������������������������� 320
Adding the ApplicationInstanceId Pipeline Variable������������������������������������������������������������� 321
Add the Child Pipeline��������������������������������������������������������������������������������������������������������� 325
Let’s Test It!������������������������������������������������������������������������������������������������������������������������� 333
Add Application Package Instance Logging������������������������������������������������������������������������������ 334
Modifying the Log.ApplicationPackageInstance Table��������������������������������������������������������� 335
Adding the Log.InsertApplicationPackageInstance Stored Procedure��������������������������������� 336
Logging Application Package Instance�������������������������������������������������������������������������������� 338
Adding the Log.UpdateApplicationPackageInstanceStatus Stored Procedure�������������������� 344
Updating Application Package Instance������������������������������������������������������������������������������ 345
Let’s Test It!������������������������������������������������������������������������������������������������������������������������� 348
Updating Application Package Instance on Success����������������������������������������������������������� 350
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 353

viii
Table of Contents

Chapter 11: Fault Tolerance in the ADF Framework��������������������������������������������� 355


A Brief Introduction to Fault Tolerance�������������������������������������������������������������������������������������� 355
Add ADF Managed Identity to Contributor Role������������������������������������������������������������������������� 356
Add Application Package Fault Tolerance��������������������������������������������������������������������������������� 359
About Fail Application on Package Failure�������������������������������������������������������������������������� 359
Implementing Application Package Fault Tolerance������������������������������������������������������������ 359
Checking the Logic�������������������������������������������������������������������������������������������������������������� 362
Implementing the Fault Tolerance Logic������������������������������������������������������������������������������ 363
Let’s Test It!������������������������������������������������������������������������������������������������������������������������� 378
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 383

Index��������������������������������������������������������������������������������������������������������������������� 385

ix
About the Authors
Andy Leonard is Chief Data Engineer at Enterprise Data
& Analytics, creator and Data Philosopher at DILM (Data
Integration Lifecycle Management) Suite, an Azure Data
Factory and SQL Server Integration Services trainer and
consultant, and a BimlHero. He is a SQL Server database and
data warehouse developer, community mentor, engineer,
and farmer. Andy is coauthor of SQL Server Integration
Services Design Patterns, Data Integration Life Cycle
Management with SSIS, and The Biml Book.

Kent Bradshaw is the founder or Tudor Data Solutions,


LLC. With over 40 years of IT experience, he is a SQL
Server database/ETL developer and database architect
with a background in Medicaid claims, public schools,
government, retail, and insurance systems. In 2011,
Kent founded Tudor Data Solutions, LLC, to pursue new
development opportunities which led to his association with
Andy Leonard and Enterprise Data & Analytics. In 2017, Kent
received the MPP certification for Data Science.

xi
About the Technical Reviewer
André van Meulebrouck has a keen interest in functional
programming, especially Haskell and F#.
He also likes data technologies from markup languages
to databases and F# type providers.
He lives in Southern California with his wife “Tweety”,
and is active in athletics: hiking, mountain biking, and
gravity/balance sports like freestyle skating (in-line and ice),
skateboarding, surfing, and sandboarding.
To keep his mind sharp, he does compositional origami, plays classical guitar, and
enjoys music notation software.

xiii
Acknowledgments
This book would not have been possible without the help of Kent Bradshaw, my
coauthor, friend, and brother. This book represents Kent’s first foray into authoring, and
he delivered outstanding work.
Thanks André van Meulebrouck for proofreading, working through the samples, and
providing feedback on the manuscript.
I owe my coworkers at Enterprise Data & Analytics (EntDNA.com) a debt of gratitude
for their encouragement and for covering for me when I stayed up too late trying to
figure out how to make the code work. We have an awesome team, especially my
brothers from other mothers – Nick Harris, Shannon Lowder, and Reeves Smith.
Donald Farmer inspires me every time we interact. As Principal Program Manager at
Microsoft, Donald worked extensively with SSIS and helped shape the product. Donald
continues to shape data software by providing vendors unique strategic guidance at
TreeHive Strategy (treehivestrategy.com).
I am certain there are many excellent editors in this business. Jonathan Gennick is
the best with whom I have had the privilege to work. Without Jill Balzano’s input and
project management, I would have been lost!
Finally, I thank my family for their understanding. My children Stevie Ray, Emma, and
Riley who live at home at the time of this writing, and Manda and Penny who have children
of their own. And Christy, to whom I dedicate this book, my wife, my love. Thank you.
—Andy

Writing this book never really entered my mind until my great friend and EDNA
colleague, Andy Leonard, convinced me to do it. I truly appreciate his confidence in me,
and it was a very interesting experience. I’m really glad that I could work together with
him on this project.
Many thanks to Jonathan Gennick, Jill Balzano, and André van Meulebrouck for all of
their help and suggestions. They made things so much easier.
Most important of all, I want to thank my wife, Ann. Much of where I am today is due
to her love, support, and encouragement. I couldn’t begin to do it without her. Thank you.

—Kent

xv
Introduction
Frameworks have existed for as long as people have been building things. When
people began developing solutions using computers, frameworks soon followed. Like
woodworking jigs, frameworks exist to simplify work and speed up the process of
developing a solution, whether the problem is to complete a woodworking project or
deliver an enterprise data integration solution.

Who Is This Book For?


Anyone who is interested in data-related automation or executing multiple chunks
of code with a single command will find value in these pages. Three audiences will
particularly benefit from this book:

1. Database developers

2. Data engineers and data integration developers

3. Business Intelligence Markup Language (Biml) developers

Database developers will learn about driving database object execution from stored
procedures and will see one example of a database framework.
Data engineers and data integration developers will learn how to use SQL Server
Integration Services (SSIS) to implement a metadata-driven data integration framework.
Biml developers will see one example of using BimlExpress and metadata to
rapidly produce several SSIS packages in selected design patterns and an example of
interrogating flat files to generate flat file formats and flat file connection managers via
Biml.

How This Book Is Structured


In the following sections, we will explore the chapter layout of this book. There are two
parts that guide you through the process of building a database framework using stored
procedures and an SSIS framework. Each of those parts is composed of several chapters.

xvii
Introduction

Part I: Stored Procedure–Based Database Frameworks


In the first part of this book, we will focus on building a database framework using
stored procedures. These chapters are targeted primarily at those who develop database
solutions. We do not endeavor to create the most comprehensive primer for every aspect
of database development; we cover the basics well.

C
 hapter 1: Stored Procedures 101
Before we dive into database development, let’s cover some basics of stored procedures.

C
 hapter 2: Automation with Stored Procedures
Once you have a solid grasp on stored procedures, we will use what we’ve learned about
stored procedures to automate execution using the controller pattern, which executes
one or more stored procedures.

C
 hapter 3: Stored Procedure Orchestrators
In this chapter, we examine high-level execution management using the orchestrator
pattern, which executes one or more controller stored procedures.

C
 hapter 4: A Stored Procedure–Based Metadata-Driven
Framework
Metadata-driven frameworks are a recurring theme in this book. In this chapter, we
store information about stored procedures, controllers, and orchestrators to execute
framework applications.

Part II: SSIS Frameworks


If your enterprise is executing a few SSIS packages, there is no need for an SSIS
framework. However, if your enterprise is executing several hundred (or several
thousand) SSIS packages, you definitely need an SSIS framework.

xviii
Introduction

C
 hapter 5: A Simple, Custom, File-Based SSIS Framework
In this chapter, we define a data integration/engineering framework metadata database
that manages configuration and execution. We introduce a version of an SSIS framework
that manages these subject areas for SSIS executed on-premises.

C
 hapter 6: Framework Execution Engine
In this chapter, we add instrumentation in the form of Information and Error events.
Built-in SSIS logging will surface these messages, which are useful when troubleshooting
SSIS framework application issues.

C
 hapter 7: Framework Logging
In this chapter, we persist execution metadata to “Instance” tables. The
ApplicationInstance table stores one record for each instance of an SSIS framework
application execution. The ApplicationPackageInstance table stores one record for each
instance of an SSIS framework application package execution. Important execution
instance attributes – such as start and end times and execution status – are maintained
in the “Instance” tables.

C
 hapter 8: Azure-SSIS Integration Runtime
In this chapter, we introduce Azure Data Factory (ADF) and the Azure-SSIS integration
runtime by walking through the provisioning processes.

C
 hapter 9: Deploy a Simple, Custom, File-Based Azure-SSIS
Framework
In Chapters 5–7, we designed and constructed an SSIS framework aimed at on-premises
SSIS execution, configuration, and logging. In this chapter, we begin migrating the SSIS
framework designed in Chapters 5–7 by provisioning a new Azure SQL database and
then deploying the metadata database to our new Azure SQL database. We provision an
Azure File Share and migrate test SSIS packages to the Azure File Share. Finally, we begin
building the Azure Data Factory parent pipeline that serves as the execution engine for
the ADF version of the SSIS framework.

xix
Introduction

C
 hapter 10: Framework Logging in ADF
In this chapter, we add logging functionality to the parent ADF pipeline, much like the
functionality we added to the on-premises version of the SSIS framework in Chapter 7.

C
 hapter 11: Fault Tolerance in the ADF Framework
In this chapter, we complete ADF execution engine functionality by implementing fault
tolerance to programmatically stop (or not stop) pipeline execution based on SSISConfig
metadata configurations.

C
 onclusion
This book is for people who want to dive deeper into SQL Server and SSIS automation.
We will be discussing and demonstrating database and Integration Services frameworks
and covering such topics as SQL Server, SSIS, Azure, and Azure Data Factory.
As you can see, we have a long journey filled with learning and new insight ahead of
us. Let’s get started!

xx
PART I

Stored Procedure-Based
Database Frameworks
CHAPTER 1

Stored Procedures 101


One of the most common issues continually facing IT organizations is finding the proper
balance between the effort to develop and deploy processes into production against the
efficiency and effectiveness of production control operators. The effort to develop and
deploy and the effectiveness of production control seem to be diametrically opposed.
Making it easier to develop and deploy processes usually means more work and manual
intervention for production control. The real question that needs to be considered is
where is it better to “feel the pain”? Pushing the effort toward the development side of
the equation can slow down the throughput but minimizes the liability of issues at the
production process level. This and the next few chapters are going to concentrate on
processes that are executed using stored procedures. In this chapter, you’ll get a basic
introduction to stored procedures that is the foundation for the chapters that follow.
You’ll see how to create a child stored procedure, and we’ll provide you a template that
you can use to create similar procedures in your own work.

The Need for a Framework


When just in development mode, it is awfully easy (and impressive) to construct
large, monolithic procedures that do everything from beginning to end. They are the
proverbial “black box” where something goes in, many gyrations take place, and then the
desired result occurs. Those are great until a problem occurs, or just a change in business
requirements means that modifications need to be made. When the procedure does
so much, what does it take to test it once the modifications are done? Even though the
change only impacts 10 percent of the procedure, the entire procedure has to be tested.
What does it take to accomplish that? What if there are several intricate modifications
that need to be made? How difficult does it become for more than one developer to work
on the changes and coordinate their efforts?

3
© Andy Leonard, Kent Bradshaw 2020
A. Leonard and K. Bradshaw, SQL Server Data Automation Through Frameworks,
https://doi.org/10.1007/978-1-4842-6213-9_1
Chapter 1 Stored Procedures 101

Now, consider that monolith broken up into multiple procedures, each of which
performs a unit of work. When a change is made and needs to be tested, that effort can
be isolated to just what is necessary to perform that unit of work, and the validation is
concentrated on the result of the procedure. And, with it now being multiple procedures,
modifications can usually be done simultaneously by multiple developers, and their
efforts can be mutually exclusive. Over time, that approach can prove to be much more
cost effective and efficient.
That is where a framework helps to organize and manage processes to provide
the most flexibility in development and can minimize the maintenance effort (which,
sometimes, is not considered until it becomes an obvious issue). A framework provides
a consistent methodology for assembling and executing processes. It also promotes
writing code in small units of work that can potentially be mixed, matched, and reused.
It adds complexity to the development and deployment processes but can reduce the
effort for production scheduling. The framework can also provide greater flexibility for
managing the execution of the process.

D
 emonstration of a Framework
To begin the analysis of the framework concept, we need a process. Our example to
follow shows a framework built to run a daily process against an example schema. The
details of that process don’t matter to the example. Just consider that any production
system might have something that needs to be done each day, and what follows is a
framework by which to make those daily processes happen.
Also, part of the example is a monthly process. Just as a system might need certain
tasks to be done each day, it’s also common to have certain things that need to be done
once monthly. In designing such a system, one must take into account the order in
which daily and monthly processes execute when their schedules intersect on – in our
example – the first of each month.
For the purpose of this book, a simple process has been developed (NOTE: all of the
code described can be downloaded at entdna.com. You can also find a link to the code
from the book’s catalog page on Apress.com). Downloading the example code enables
you to follow along with the upcoming examples on your own machine.

4
Chapter 1 Stored Procedures 101

An Example Schema


Listing 1-1 shows code to create a schema called FWDemo that will contain everything
needed for the demonstration. Also, there is code to create a table called FWDemo.
ProcessLog. Including a pattern for writing to this table throughout all of the procedures
certainly adds some complexity and overhead to the procedures, but what it provides in
monitoring and troubleshooting more than makes up for the upfront effort.

Listing 1-1. Schema and log table creation

print 'FWDemo Schema'


If Not Exists(Select name
              From sys.schemas
              Where name='FWDemo')
begin
  print ' - Creating FWDemo schema'
  declare @sql varchar(255) = 'Create Schema FWDemo'
  exec(@sql)
  print ' - FWDemo schema created'
end
Else
print ' - FWDemo schema already exists.'
print ''
GO

IF  EXISTS (SELECT * FROM sys.objects


            WHERE object_id = OBJECT_ID(N'FWDemo.ProcessLog')
              AND type in (N'U'))
    DROP TABLE FWDemo.ProcessLog
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO
5
Chapter 1 Stored Procedures 101

CREATE TABLE [FWDemo].[ProcessLog](


      [ProcessLogID]      [int] IDENTITY(1,1) NOT NULL,
      [ProcessLogMessage] [nvarchar](255)     NOT NULL,
      [CreateDate]        [smalldatetime]     NOT NULL
)
GO

SET ANSI_PADDING OFF


GO

Do you have a SQL Server instance that you can use for learning purposes? Connect
to that instance as an administrator, for example, as the sa user. Then, in SQL Server
Management Studio (SSMS), open a “New Query” window, copy the code from Listing 1-1,
and execute it to create the example schema used in this and subsequent chapters.

The Daily Process


Listing 1-2 shows the code to create two stored procedures that will make up our
demonstration Daily Process. We are providing two procedures in our example because
it’s common to have more than one, and having two allows us to show how to make the
execution of subsequent procedures depend upon the success of earlier ones – because
the need to execute a series of procedures and halt or take other actions when an error
occurs is the real-life scenario that most of us face.
These procedures (as well as all others that we will be using) can be compiled and
executed for your own testing. You will notice that there is some code commented out
(lines that are preceded with ‘--’) in each procedure that can be invoked (remove the
‘--’, then recompile) to create an error condition. This ability to create an error condition
allows testing for successful and unsuccessful completions that will become more
important as we progress through demonstration iterations in later chapters.
For the sake of this exercise, we will declare a business rule for the Daily Process
stating that FWDemo.DailyProcess1 must complete successfully before FWDemo.
DailyProcess2 can be executed. FWDemo.DailyProcess2 must then complete
successfully before the Daily Process can be deemed successfully executed.

6
Chapter 1 Stored Procedures 101

Listing 1-2. Daily Process stored procedures

If Exists(Select s.name + '.' + p.name


          From sys.procedures p
          Join sys.schemas s
            On s.schema_id = p.schema_id
          Where s.name = 'FWDemo'
            And p.name = 'DailyProcess1')
begin
  print ' - Dropping FWDemo.DailyProcess1 stored procedure'
  Drop Procedure FWDemo.DailyProcess1
  print ' - FWDemo.DailyProcess1 stored procedure dropped'
end
GO

CREATE PROCEDURE FWDemo.DailyProcess1


AS
---------------------------------------------------------------------------
---------------------------------------------------------------------------
--
-- Purpose: This procedure is part of the Stored Procedure Framework Demo.
--
-- NOTE: An Error situation can be created for testing/demo purposes by
--    un-commenting the Error code in the body of the procedure.  To return
--    to a procedure with a successful execution, re-comment the code or    
--    recompile the original.
--
---------------------------------------------------------------------------
---------------------------------------------------------------------------

SET NOCOUNT ON

/*********************************************/
/*  Log the START of the procedure to the process log  */
/*********************************************/

7
Chapter 1 Stored Procedures 101

INSERT INTO FWDemo.ProcessLog (


      ProcessLogMessage,
      CreateDate
)
Values ('Procedure FWDemo.DailyProcess1 - STARTING',
      GETDATE()
)

DECLARE @RetStat int

SET @RetStat = 0

/******************************************/
/*  Force an ERROR CONDITION for this procedure  */
/******************************************/

--INSERT INTO FWDemo.ProcessLog (


--    ProcessLogMessage,
--    CreateDate
--)
--VALUES ('Procedure FWDemo.DailyProcess1 - Problem Encountered',
--    GETDATE()
--)
--SET @RetStat = 1

/****************************************************/
/*  Log the COMPLETION of the procedure to the process log     */
/****************************************************/

IF @RetStat = 0
   BEGIN
      INSERT INTO FWDemo.ProcessLog (
            ProcessLogMessage,
            CreateDate
      )
      VALUES ('Procedure FWDemo.DailyProcess1 - COMPLETED',
            GETDATE()
      )

8
Chapter 1 Stored Procedures 101

   END
ELSE
   BEGIN
      INSERT INTO FWDemo.ProcessLog (
            ProcessLogMessage,
            CreateDate
      )
      VALUES ('Procedure FWDemo.DailyProcess1 - ERROR',
      GETDATE()
      )
   END

RETURN @RetStat
GO

If Exists(Select s.name + '.' + p.name


          From sys.procedures p
          Join sys.schemas s
            On s.schema_id = p.schema_id
          Where s.name = 'FWDemo'
            And p.name = 'DailyProcess2')
begin
  print ' - Dropping FWDemo.DailyProcess2 stored procedure'
  Drop Procedure FWDemo.DailyProcess2
  print ' - FWDemo.DailyProcess2 stored procedure dropped'
end
GO

CREATE PROCEDURE FWDemo.DailyProcess2


AS
---------------------------------------------------------------------------
---------------------------------------------------------------------------
--
-- Purpose: This procedure is part of the Stored Procedure Framework Demo.
--

9
Chapter 1 Stored Procedures 101

-- NOTE: An Error situation can be created for testing/demo purposes by


--    un-commenting the Error code in the body of the procedure.  To return
--    to a procedure with a successful execution, re-comment the code or
--    recompile the original.
--
---------------------------------------------------------------------------
---------------------------------------------------------------------------

SET NOCOUNT ON

/*********************************************/
/*  Log the START of the procedure to the process log  */
/*********************************************/

INSERT INTO FWDemo.ProcessLog (


      ProcessLogMessage,
      CreateDate
)
Values ('Procedure FWDemo.DailyProcess2 - STARTING',
      GETDATE()
)

DECLARE @RetStat int

SET @RetStat = 0

/******************************************/
/*  Force an ERROR CONDITION for this procedure  */
/******************************************/

--INSERT INTO FWDemo.ProcessLog (


--    ProcessLogMessage,
--    CreateDate
--)
--VALUES ('Procedure FWDemo.DailyProcess2 - Problem Encountered',
--    GETDATE()
--)

10
Chapter 1 Stored Procedures 101

--SET @RetStat = 1

/****************************************************/
/*  Log the COMPLETION of the procedure to the process log     */
/****************************************************/

IF @RetStat = 0
   BEGIN
      INSERT INTO FWDemo.ProcessLog (
            ProcessLogMessage,
            CreateDate
      )
      VALUES ('Procedure FWDemo.DailyProcess2 - COMPLETED',
            GETDATE()
      )
   END
ELSE
   BEGIN
      INSERT INTO FWDemo.ProcessLog (
            ProcessLogMessage,
            CreateDate
      )
      VALUES ('Procedure FWDemo.DailyProcess2 - ERROR',
            GETDATE()
      )
   END

RETURN @RetStat
GO

In a “New Query” window in SSMS, execute the code from Listing 1-2 while
connected to the FWDemo schema. The code creates two stored procedures that
together make up a daily process. With those procedures in place, you can turn your
attention to the next problem, which is to schedule those procedures to actually run each
day.

11
Chapter 1 Stored Procedures 101

Executing the Daily Process


Now that an environment has been built and a process created, let’s turn to the
execution. Operations staff will have to set up or schedule the procedures to run and
either monitor for any error conditions that are raised or set up precedence rules if such
a function exists in any scheduling tool used. In a basic sense, we now have a Daily
Process that is ready for production. Listing 1-3 shows the statements that can be used
to execute the Daily Process procedures and also a SELECT statement that can be run
to view the output written to FWDemo.ProcessLog. You will notice that we are ordering
the output in a descending order. This will show the most recent messages at the top and
eliminate the need to scroll down to get to the messages for the current execution and
much easier once the log starts to become heavily populated.

Listing 1-3. Daily Process execute statements and process log SELECT statement

EXECUTE FWDemo.DailyProcess1

EXECUTE FWDemo.DailyProcess2

SELECT ProcessLogID
      ,ProcessLogMessage
      ,CreateDate
FROM FWDemo.ProcessLog
ORDER BY ProcessLogID desc

Including a Monthly Process


Now it’s time to add another layer to our production process. In Listing 1-4, there is
code to create two more stored procedures that will make up a Monthly Process. The
procedures operate the same as our daily process procedures, and there are some
business rules associated with them. First, the monthly process will run on the first day
of the month. Second, it will run after the successful execution of the Daily Process,
and third, FWDemo.MonthlyProcess1 must complete successfully before FWDemo.
MonthlyProcess2 can be executed.

12
Chapter 1 Stored Procedures 101

Listing 1-4. Monthly Process stored procedures

If Exists(Select s.name + '.' + p.name


          From sys.procedures p
          Join sys.schemas s
            On s.schema_id = p.schema_id
          Where s.name = 'FWDemo'
            And p.name = 'MonthlyProcess1')
begin
  print ' - Dropping FWDemo.MonthlyProcess1 stored procedure'
  Drop Procedure FWDemo.MonthlyProcess1
  print ' - FWDemo.MonthlyProcess1 stored procedure dropped'
end
GO

CREATE PROCEDURE FWDemo.MonthlyProcess1


AS
---------------------------------------------------------------------------
---------------------------------------------------------------------------
--
-- Purpose: This procedure is part of the Stored Procedure Framework Demo.
--
-- NOTE: An Error situation can be created for testing/demo purposes by
--    un-commenting the Error code in the body of the procedure.  To return
--    to a procedure with a successful execution, re-comment the code or
--    recompile the original.
--
---------------------------------------------------------------------------
---------------------------------------------------------------------------

SET NOCOUNT ON

/*********************************************/
/*  Log the START of the procedure to the process log  */
/******************************** ************/

13
Chapter 1 Stored Procedures 101

INSERT INTO FWDemo.ProcessLog (


      ProcessLogMessage,
      CreateDate
)
Values ('Procedure FWDemo.MonthlyProcess1 - STARTING',
      GETDATE()
)

DECLARE @RetStat int


SET @RetStat = 0

/******************************************/
/*  Force an ERROR CONDITION for this procedure  */
/******************************************/

--INSERT INTO FWDemo.ProcessLog (


--    ProcessLogMessage,
--    CreateDate
--)
--VALUES ('Procedure FWDemo.MonthlyProcess1 - Problem Encountered',
--    GETDATE()
--)
--SET @RetStat = 1

/****************************************************/
/*  Log the COMPLETION of the procedure to the process log     */
/****************************************************/

IF @RetStat = 0
   BEGIN
      INSERT INTO FWDemo.ProcessLog (
            ProcessLogMessage,
            CreateDate
      )
      VALUES ('Procedure FWDemo.MonthlyProcess1 - COMPLETED',
            GETDATE()
      )
   END

14
Another Random Document on
Scribd Without Any Related Topics
he may lefully
recouer apon him
be were his thing,
gif he may reclame
him in jugement.
Gif a Knycht deis in
bataill in his Princis
querele, quhethir
his saule be sauf, or
nocht. 52
Quhethir Rychtwise
men or sinnaris ar
starkar in bataill. 53
Quhy is there sa
mekle Were in this
warld. 54
Gif a Prisouner be
suorne to hald
prisoun, and his
takar put him atour
his ath in stark
prisoun or
festnyng, gif it be
lefull to him to
escape, and brek
presoun. 55
Gif a Man be
presonare till ane
othir, and he put
him in a stark close
toure, in sekir
festeynyg, quhethir
he be haldin to
brek prison, and
eschape. 56
Gif a Man has sauf
condyt to com
seurely, nocht
spekand of his way-
passing, quhethir
he may be haldyn
prisonar in his
passing. 57
Gyf a Man that has
sauf conduct may
bring on his sauf
conduct gretare
man na himself is. 58
Gif a Man be tane
prisoner apon ane
otheris sauf condyt,
quhethir he that aw
the sauf condyt
suld outred him of
prisoun on his awin
cost. 59
Gif a Man suld enter
agayn in prisoun,
and he war rycht
dredand for to be
put to dede. 60
Gif a Prince may
lefully refuse ane
othir Prince to pas
his voyage throu
his contre but
scath. 61
Quhethir Kirkmen suld 62
pay tailles, tributis,
and inposiciouns to
Seclere kingis or
princis.
Gif the Kirk may mak
were agayne the
Jowis. 63
Gif a Man may ficht
for his wyf in
armes. 64
How the ta Brothir
may defend the
tothir in armys. 65
Gif a Baron be vassall
to twa Lordis that
makis weris in
syndry contreis, to
quhilk of tham sall
he mak seruice till. 66
And gif a Baron be
vassall to twa lordis
the quhilkis makis
were ilkane apon
othir, quham to sall
he mak seruice. 67
Quhethir Bondis suld
be constreynit to
the weris. 68
Quhilk Folk may nocht
be strenȝeit to mak
weris, supposs thai
be chargit. 69
Gif a Man be hurt
sarely be ane othir,
and he hurt him
agayne, gif he sal
be punyst. 70
Gif a Man bonde
makis slauchter be
the bidding of his
Lord, suld he be
punyst. 71
Quhethir a Bonde
may defend him
again his Lord, and
he war sett to sla
him. 72
Gif a Monk may
defend him fra his
Abbot, and he wald
sett to sla him. 73
Gif the sone may
lefully defend him
agayn his fader,
and he wald sla
him. 74
Gif a Man may lefully
defend him agayn
his awin juge, or
nocht. 75
Gyf a Man be banyst
a realm, and
happin to cum in
agayne be ony cas,
gif men wald set on
him, to tak him,
quhethir he aucht
to defend him. 76
Gif a Preste be assailit 77
wyth his inymyes
berand Goddis
sacrit body on him,
quhethir he aw to
lay doun Goddis
body, and defend
him, or nocht.
Gif a Man may for
mark be prisoner
that maid neuer
caus bot for
otheris. 78
How, and in quhat
maner, mark suld
be tholit or gevin
be the Prince. 79
How suld mark be
gevin aganis a
Citee that allegis to
na soverayne. 80
Gif all Lordis may
graunt markis. 81
How, or be quhat
resoun, may it be
steynd that the
King of France be
nocht subject to
the Empire. 82
Quhethir gif the King
of Ingland be
subject ony way to
the Empire. 83
Gif a Burgess haldand 84
change and house
at Parise be tane
and robbit be the
way cummand to
Parise-wart,
quhethir he is to be
gevin power of
merk to for the
gude recouering.
Quhethir a Scolare at
the study in Parise
of Ingland borne,
aw to be prisonare. 85
Quhethir a Seruand
suld joyse the
priuilege that his
maister has lang
joisit. 86
Gyf ane Inglissman
cummys to Parise
to visyte his sone at
the scule, beand
seke, quhethir he
aw to be prisoner,
or nocht. 87
Gyf ane Inglissman
cummys to Parise
to visyte his brother
seke at the scule,
quhethir he aw to
be prisonare, or
nocht. 88
Quhethir a Studyand
may lefully be
haldin in prisoun for
ony mark. 89
Quhethir a Wode man 90
may be haldyn
presoner and
ransound in the
weris.
Quhethir a Wode
man, efter that he
be cummyn again
to his wit, may be
haldyn presoner. 91
Quhethir a passand
alde Ancien man,
be law of armes,
may be haldin
prisonere. 92
Quhethir a Childe
may lefully be tane
and haldin
presonere be the
law of armes. 93
Quhethir a Blynd
man, be law of
armes, may be tane
and haldin
presonere. 94
Quhethir
Ambassadouris or
Legatis cummand
to the King may
lede his inymyes
throu his realme
with thame, or
nocht. 95
Quhethir a Bischop 96
may be tane
presonere be a
Franch man, the
Bischop beand of
Ingland.
Quhethir a Kirk man
may be tane for
mark. 97
Quhethir gif Pilleryns
may be maid
presoneris be ony
maner of weris of
armes. 98
Quhat thingis in tyme
of were has sauf
condyt be priuilege
unaskit at the
Princis. 99
Quhethir, in tyme of
were, the ass and
the ox suld bathe
joise a maner of
priuilege. 100
Quhethir gif the varlet
aw to joyce the
priuilege of the
husbandman. 101
Quhethir, in tyme of
weris, folk may
ledder castellis and
wallit townis lefully. 102
How suld be punyst
folk that brekis the
Princis sauf condyt,
or his assurancis. 103
Quhethir a grete Lord 104
suld traist in a sauf
condyt, or ony othir
lawar person.
Quhethir gif a Cristin
King, Prince, or
Emperour, may gif
a sauf condyt till
ane othir King,
Prince, or
Emperoure
Sarraȝene. 105
Gif twa Lordis has
made trewis
togidder suorne,
quhethir gif the
tane brek trewis gif
the tothir suld rycht
sa brek. 106
Quhether better be to
fecht fastand
before mete, or
efter mete quhen
men has dronkin. 107
Quhethir bataill may
be set before
Ladyes. 108
Quhethir the Quene
Jonat of Naplis
mycht lefully
assailȝie the King
Lowis de Cicile. 109
Here previs the
Autour playnly how
gage of bataill is
reprovit be all
maner of lawis. 110
Here he puttis the
case, in the quhilk
it is lefull to geve
gage of bataill. 111
And ȝit he puttis ane
othir case in the
quhilk law of armes
will thole gage of
bataill. 112
And ȝit ane othir case
efter the lawis of
Lumbardy. 113
And ȝit ane othir case
efter the law of
Lumbardy. 114
And ȝit ane othir ease
efter the law of
Lumbardy. 115
And ȝit ane othir case
efter the law of
Lumbardy. 116
And ȝit ane othir efter
the law of
Lumbardy. 117
And ȝit ane othir efter
the lawis of
Lumbardy. 118
And ȝit ane othir case
efter the lawis of
Lumbardy. 119
And ȝit ane othir case
efter the law of
Lumbardy. 120
And ȝit ane othir case 121
efter the law of
Lumbardy.
And ȝit ane othir case
efter the law of
Lumbardy. 122
And ȝit ane othir case
efter the law of
Lumbardy. 123
And ȝit ane othir case
efter the said
Lawis. 124
And ȝit ane othir case
efter the law of
Lumbardy. 125
And ȝit ane othir case
efter the law of
Lumbardy. 126
And ȝit ane othir case
efter the said
Lawis. 127
How oft tymes the
bataill in listis is
nocht done be the
principale persouns
bot be otheris. 128
The form and maner
of thair Aithis that
suld fecht in
barrieris of close
listis in felde. 129
Gif a man passit age,
may put quham
him list to
campioun to ficht in
barreris for him. 130
Gif ane of the
campiouns brekis
his suerd, quhether
ane othir suld be
gevin him agayne. 131
Gif the Lord may
nocht knaw the first
day quha has the
lyklyar, gif thai suld
cum again on the
morn, and enter in
felde as before:
Quhilk of the twa
campiouns suld first
stryke. 132
Gif the vencust man
suld pay the costis,
thouch the Kyng
remytt his actioun. 133
Gif a man has bene
vencust of ony
crime in barreris,
gif he may be
accusit in jugement
tharof. 134
Quhethir gif the
campiouns may
fecht in playne
felde, but barreris,
gif thai lykis. 135
How he suld be
punyst that has
grantit his crime,
and vencust in
barreris opinly. 136
Gif a Knycht appelis
ane othir, quhether
gif thai may leue of,
and forthink the
appele. 137
Here, he spekis of
Armes and baneris
in generale. 138
Here he spekis of
Armes and baneris
in specialitee. 139
Gif a Man may [tak]
otheris armes at his
lyking. 140
Gif ane Allemain
fyndis a Frenchman
berand the samyn
armes that he beris
in felde, quhethir
gif he may appele
him of battaill. 141
How suld be punyst
folk that beris othir
mennis armes but
leue, to do tham
ony lak. 142
Here spekis he of
colouris in armes,
quhilkis are the
maist noble; and of
thair diuisiouns. 143
And first, he spekis of
the colour that is
rede. 144
And syne he spekis of
asure that is the
blewe coloure. 145
And syne he spekis of
the quhite colouris. 146
And syne he spekis of
the colour that is
blak. 147
And syne he spekis of
the condicioun and
nature of the
ordinance of the
closing of the
Barreris. 148
And ȝit spekis he of
the condicioun of
the close felde,
ordanyt for fechting
in barreris, as said
is. 149
And ȝit ane othir thrid
reule of the
condicioun and
nature of close
barreris. 150
And ȝit spekis he of
the ferde
condicioun and
nature of the close
felde, that is callit
barreris. 151
And ȝit the fyft 152
doctryne gevis he
of the form and
maner and
condicioun of the
close felde.
And ȝit the sext
doctrine spekis he
of the form, maner,
and condicioun of
the close barreris. 153
Here he speris quhat
condicioun suld be
in a gude
Emperoure be the
nature of his hie
office. 154
Quhat thingis
appertenis to be in
a gude Prince,
King, or othir. 155

CAPITULUM CXXXVIII, &c.

HERE SPEKIS THE DOCTOURE OF ARMES THAT AR IN BLASONS, AND OF


BANERIS AND PENOUNS.[18]

Fol. 77.
b.

ow efter that he has determynit of Bataillis bathe in


generale and in speciale, than will he declare the Armes
that all Princis and Nobles and othir Gentillis aw to
were, and of thair colouris, and discripciouns: And fyrst,
Quhethir a man that is nocht of thair lygnie may bere
leuefully thair Armes at his plesaunce? the quhilk mater is nocht
lycht to declare, bot of grete difficultee for mony caussis. And first,
men suld vnderstand that sum Armes was gevin of power of
autoritee of Emperouris, Kingis, and Princis, to Lordis, and otheris
Barouns, or to thair predecessouris, the quhilkis ar of alde tyme, and
of alde ancestrye, that nane suld bere, bot thai war cummyn of that
lignie, that is to say, in the realme of the Emperoure, King, or Prince
that gafe the said armes; ffor and the King of France had gevyn a
lyon of gold to bere till a Lord of his contree, quhat wrang dois that
Lord till ane othir Lord of Spaigne or of Almane, that the Emperoure
or the King of Spayne had gevin it till. Bot thare is othir maner of
Armes, the quhilkis ilke man that beris thame, tuke at thair awin
plesance to mak difference and knaulage amang Lordis, Gentilis, and
noble men of armes, to knaw ilk ane be othir. And all namys and
surnamys of men was foundyn ffor the samyn cause. Or ellis all suld
haue bene in confusioun that nane suld haue had knaulage of ane
othir. And this name may ilke man tak, and geue his barne at his
awin plesance, or the godfader, or godmoder, or frendes may geue
namys to thair frendis; and rycht sa in the samyn wyse is it of
armes, that in the begynnyng quhen the weris began, till haue sum
defference amang nobles, sik armes was, sum assignit be Princis and
Lordis; sum was tane at the plesance of partye; sum be thair frendis
consent and consale, sa that men of honour and of estate suld be
knawin be thair armes, the quhilkis ar callit thair takenys in armoury.
And thus in were tyme, new men of armes that has nane armes of
propertee, may in this wise tak armes at thair lyking syk as thame
lest. Bot nocht to tak nane otheris Armes.
HERE SPERIS THE DOCTOUR GIF A MAN MAY TAK ANE OTHERIS ARMES AT HIS
LYKING.

s thus a Man has tane to bere in his Armes a low of


gules in a champ of siluer, ane othir of that ilke toune
has tane the samyn efter that he has it wynteris and
ȝeris. Than speris the questioun, Quhethir the first may
gaynstand this armes and plenȝe to the Prince, and ger
this be reformyt and forborne. And first he sais, Nay; ffor quhy ony
man may tak lefully, as before said is, ane othir mannis name, and
call his barne in the samyn toun, ffor it befallis oft tymes that syndry
men ar callit be a name in the samyn toun; and may do it but lak.
And quhy then may thai nocht alswele tak twa ane armes, or thre, or
alsmony as lykis. Bot the countre party sais agayne, That it is a
commoun vse and custume in mannis lawe, and approuit be othir
lawis, that quha sa euer may first tak wilde foule, or fysch, or wylde
beste in the wilderness, it is his be the lawe. And than, sen this
noble man has first tane sik a beste, or sik a foule, or sik a fisch, to
bere in his schelde and on his cote of armes, and on his banere,
pannoun of armes, or in blasone apon his heraulde or perseuandis
brest, or othir wayis to paynt in hall or chaumer at his lyking; Quhy
suld ony othir tak it efterwart to bere that war in toune or in the
samyn contree, quhare it war borne? And als it war mare thair
scathe na thir prouffit, ffor it wald quhilom mere men that had grete
dedis ado, quhen thai wend to cum to thair awin maister in
werefare, thai mycht failȝe, and othir wayis in syndry wyse mycht
erre in thair dedis that mycht hynder bathe the partis that nedit
nocht, na is na poynt of gude gouernaunce, na gude policy in dedis
of armes: And as langand this questioun the Doctour makis sic a
conclusioun, That gif a gentill man or lord had tane ane armes at his
plesance, and borne it lang tyme opynly kend in dede of armes, and
in weris, or othir wayis in tyme of pes, that it war kyd and knawin till
him and his lignage; Thare aw nane othir in that contree to tak the
samyn to bere, na the Prince na the lord of that contree, suld nocht
lat tham bere it on na wise, ffor than war the principale cause of
Armes-taking all forletyn; ffor the principale cause of Armes-taking is
for to knawe the personagis of noble men in bataill, or in armes, or
in tournamentis, or to knawe a lord in felde be ane othir, with his
men, and his frendis and wele willaris suld draw till him and knawe
him be his takin, and sa suld the diuersitee of armes mak the
knaulage of the diuersitee of personnagis; and gif it be hapnyt ony
lorde or othir man to be slayn in felde, and sa manglit that his visage
mycht nocht be knawin, be his cote of armes he suld be knawin and
brocht to Cristin beriss: And alssua, that be the defference of armes
euer furth quhill the warld lestis, men suld knaw be the takynnyng of
thair armes, thair sepulturis quhare thai ar beryit; and quha was in
thai tymes maist honourable and worthy men, as oft tyme men seis
apon thair sepulturis be thair frendis maid efter thaire decesse, and
sum be thame self or thai be dede; be the quhilkis, quhen all the
lygnie is failit, and the surname, ȝit will the valliance of thame be
knawin be thair armes, the quhilkis in armes are callit Takenis: ffor
and syndry lordis or gentilis tuke all ane armes, or takenis, it was bot
a confussioun. And a mare grete resoun ffor be all gude custumes of
noblesse, lordis, and gentilis, makis thair selis efter thair armes, and
gif ane suld bere ane otheris armes in his sele, men suld nocht wit
quhais it war. Item, all Kingis suld kepe that na man do till othir
dishonour, schame, na villany, na injure, na new novelliteis. And it is
to presume, gif ony man wald newly tak ane alde armes of ane
otheris that it war for dispyte or injure to despise him to prouoke
noise and debatis for alde fede or enuy, the quhilk the Prince suld
stanch. And as to the argument, it is na thing lyke till a man to be
callit lyke till ane othir or syndry in a town; For quhy? For sik cause
men has gert geve ilke man his surname that makis the difference.
Bot and mony men bare ane armes, how suld ony man, Haralde, or
othir, knaw men, na geve the honoure of gude dede till him that had
honourabily deseruit it, or to geve lak and dishonour till cowardis or
flearis fra bataillis: And tharefore, in all sik debatis, the Prince suld
ger sett remede. And gif ony complaynt war, se, be Harraldis and
men of knawlage, quha had rycht, quha wrang, and do justice.

HERE SPERIS THE DOCTOURE GIF A FRANCH KNYCHT SAW A DUCHEMAN OF


ALMANE BERAND HIS ARMES BUT DIFFERENCE, AND HE APPELLIT HIM OF WERE
TO FECHT WITH HIM, OR FORBERE THE ARMES; QUHETHIR AW HE TO BE
ADMYTTIT, BE PRINCE, OR NOCHT TO FEICHT IN BARRIERIS WITH HIM.

s gif a Knycht of Almane wald cum to see noblesse in


France, as tournamentis or othir wasselage, and that he
fand a Knycht in France at the tournayment, that bare
the samyn armes that he beris: And thus he maid
questioun bustously, sayand, He traistit he wrangit him
and his lignie to bere thai armes, sperand, be quhat title of rycht he
bare thame? The Knycht of France ansuerd, sayand, That he traistit
that he mycht bere lefully the takyn of armes that his fader, and
forefader, and all his ancestris had of sa lang tyme borne, that thare
was na memorye in the contrarye. The Duche Knycht replyis agayn,
sayand, Gude Sir, suppose your fader and ancestris haue borne
thame sik a tyme, my kyn and ancestris ar of eldare begynnyng na
ȝouris, and als ar mare noble of lignie; quharefore, sen ȝe and ȝouris
has tane thame efter us, and ȝe ar nocht of sa grete noblesse of
alde ancestry, me think ȝe suld deferr till us, and nocht we to ȝow:
Quharefore, I say ȝe bere thame euill and wrangwisly, and that I sall
preue with my persone. And with that the Franchman sais, That he
dois him na wrang that beris the armes that his ancestres has of sa
lang tyme borne, and that he denyis his wrang, and that sall he
defende. Than is this the question, Quhethir the King aw to geve
leve to thir twa Knychtis to feicht, or nocht? And as to the first
visage, it semys thai suld be tholit, be the resouns that the Duche
Knycht allegis. Bot the Doctouris accordis nocht to that opynion, ffor
as we have before said, thai ar nocht of a realme, na of a Prince
haldand, quharfor the Naciouns makis the defference sufficiand, sa
that it war nocht done for despyte, na othir barate; ffor gif a
trauailand Knycht of France had tane sik armes, and he war a wikkit
man, of lyf a tyran, and unhonourable, that mycht defame the armes
in ferre contreis, the Duche Knycht mycht haue sum coloure and
resoun tharefor, gif the Franch Knycht past in Burgone, or Barry, or
Lorane, and brynt and slewe, and reft and forsit women, and had
renoun to be a wikkit man of lyf, and men wist nocht his surname,
na of quhat contree he war, and the tothir Knychtis armes war kend
our all thai countreis, and sum men mycht traist that it war he; and
in this cas, the Duche Knycht had resoun to ask him to be depriuit of
his armes at the King of France, and the King to grant it him, gif the
said Duche Knycht and his lignie war approuit men of honour, and
thareapon grant him leue of bataill in listis, as said is, gif him lykis
for the cause, efter the custumes may be tholit.
HERE SPERIS THE DOCTOURE HOW THAI SULD BE PUNYST THAT BERIS
OTHERIS ARMES WRANGWISELY, IN ENTENCIOUN TO DO MYS VNDER SCOUG OF
THAME COUERTLY.
s gif a Souldiour of symple state tuke the armes of a
Knycht Noble of France that war of gude renoune, bathe
in honour of armes, and othir wayes of alde ancienetee,
and that Knycht of Ducheland had tane thai armes
newly, in entent to be mare presit and honourit, and to
be hyar auansit, and tak mare wagis, in faith I traist that the King, at
the persuyt and request of partye, the King aw to punyse him be law
of Armes. As in the lyke maner, gif a maister armoureur of Parise,
that had renoun to be the best of that craft, that war in France, and
in his werkis had a takyn that his werkis were knawin by, and ane
othir of Troyes in Champaigne tuke that ilke takyn, sa that for the
renoun of the Parisien, his werkis suld be the better sauld; and rycht
sa of coultellin, or ony othir craft, or of notairis, gif ane dois falset
vnder the sailign of ane othir, I say, all sik men suld be well and
cruelly punyst be justice; and gif the contrary war tholit, it war grete
damage to the realme.
HERE SPERIS THE DOCTOUR QUHAT ARMES AR MAIST NOBLE BE THE
COLOURIS, AND QUHAT COLOURIS AR MAIST NOBLE IN ARMES.

ot be cause the Princis and Lordis beris Armes of mare


noblesse na otheris; and that the Doctouris has spokyn
in othir tymes, and othir placis, of Princis armes, and of
thair baneris, quharefore I will nocht here mak
questioun, na dout the quhilk armes are the maist
noblez and the maist rychez; ffor quhy, that alwayis comparisoun is
odious.[19] Bot it plesis me to speke sum thing of Colouris of Armes,
and of thair descripciouns. And as the Doctour sais, that sum of
thame is mare noble na otheris, for the representatioun that thai
mak be thair propre nature, and be this cause, we say, that colour of
Gold is the maist noble colour that is in this warld here; and the
resoun quhy is, ffor be the nature of gold, it is clere and schynand,
rich, vertuouse, and confortand; ffor oure Maisteris, Doctouris, and
Medicinaris, and Philosophouris, gevis the gold in syndry wise in
medicyne to folk that ar debilitez in thair nature, that thai can get
nane othir remede for souerane remede; and is lyknyt be his
condicioun and nature to the Sonne, the quhilk is the maist noble
planet that euer God maid, and beris lycht till all the warld, and
encrescement and confourt till all naturale creaturis. And the lawis
sais, that of all things that God maid, the claritee and licht is the
maist noble; and, tharefore, the Haly Wrytt sais, that the sanctis in
hevyn schynis as the sonne; and alssua oure Souerane Lord, quhen
he transfigurit him before his Apostlis, his visage apperit to thame as
the Sonne in someris day brycht: And because the Gold is comperit
to the Sonne, as the propre effect of the Sonne, the quhilk is king
and lord of all planetis, and alssua is figurit be Haly Wrytt be the
visage of our Lord; and be that cause the ancien Princis, in ald lawis
of armes, ordanyt that na noble man suld bere gold in his armes, bot
Princis, Kingis, and Emperouris, for the nobless of him: And thus
conclude we, that the maist noble coloure is Gold. And suppose sum
ignorant men wald say, gold is metalle, and na coloure, that makis
nocht; ffor largely to tak colouris, be all oure Maisteris and
Philosophouris, all metallis, all low and lychtnes, that lemys and
gevis sycht to the eyne, is of the nature of colouris.

The Secound coloure that is in Armoury, is callit be thir maisteris


Purpre; the quhilk he callis here Rede colour; the quhilk representis
the lowe of Fyre, the quhilk is the maist clere, and lycht efter the
Sonne, and the maist noble of all the elementis; the quhilk colour
suld nane in armes were, bot anerly Kingis or Princis, be the alde
custumes of Princis and Faderis of Armes, of alde tymes.

The Thrid colour is Asure; the quhilk, be his figure and coloure,
representis the Ayer, the quhilk is next the fyre, the maist noble
element; ffor it is in itself lignie and sutile, and penetratys,
ressauand the lycht throu it, and hable till rassaue all influences of
the planetis and of the hevynly constellaciouns of nature, throw the
quhilkis all this Erde is gouernyt, and all Nature: and sum callis the
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and


personal growth!

textbookfull.com

You might also like