Introduction to Data Migration
0 likes989 views
Data migration involves transferring data between different storage types or systems and is typically automated to save human resources. The process consists of three stages: pre-migration (planning and analyzing data), migration (executing the transfer), and post-migration (quality control and updates). Key considerations include normalizing data to adhere to standards and ensuring integrity throughout the migration process.
![E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard
MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide
Pre-Migration: Mapping
The goal of normalization is to transform or
clean up your data values so they conform to
accepted standards, are more consistent, and
can be understood by any user of your images
Can be done during
Pre Migration
Migration
Post Migration
E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard
MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide
Pre-Migration: Mapping
Data to Normalize
Abbreviations
e.g. Material= ol on cv.; source = DOA; attribution=sch of
Inconsistencies
e.g. expressions of circa: Date= c1947; c. 1988; c 500 AD;
ca. 15th Century
Formatting conventions
e.g. title=[Grapevines at Mission de San Ignacio]](https://image.slidesharecdn.com/rose-100211150405-phpapp01/85/Introduction-to-Data-Migration-7-320.jpg)
Introduction to Data Migration
- 1. INTRODUCTION TO DATA MIGRATION Trish Rose-Sandler VRA Conference Kansas City, MO March 28 2007 Definition “Data migration is the transferring of data between storage types, formats, or computer systems. Data migration is usually performed programmatically to achieve an automated migration, freeing up human resources from tedious tasks. It is required when organizations or individuals change computer systems or upgrade to new systems.” Wikipedia 2/21/07
- 2. 3 Stages of Data Migration Pre Migration Migration Post Migration 3 Stages of Data Migration Pre Migration – Analyzing, Mapping, Normalizing/Transforming, Testing, Backup Migration Post Migration – Quality Control, Cleanup, Update Cataloging Guidelines
- 3. Pre-Migration: Data Analysis Evaluate data for: Consistency Unnecessary redundancy across records Identify errors Re-evaluate relationships & structures Pre-Migration: Data Analysis Screen shot of Oxygen XML editor
- 4. Pre-Migration: Mapping Source Target CreatorName vra:agentName dc:creator Date vra:date dc:date Site vra:locationName dc:coverage General Notes vra:description dc: description Pre-Migration: Mapping Example of Source to Target map
- 5. Pre-Migration: Mapping Source Set Examples of parsing Source data stores all agent names in creator field w/o a qualifying role Creator Target data stores all agent names separate from but linked to roles Agent.name Agent.role Pre-Migration: Mapping Target Set Mix of community standards and local elements Descriptive – e.g. Core 4.0 Technical – e.g. Core 4.0, MIX Rights – e.g. Core 4.0, METS Rights Local – e.g. dateImagePurchased
- 6. E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Pre-Migration: Mapping Familiarize yourself with standards’ documentation Examples VRA Core 4.0 standard Core 4.0 documentation Cataloging Cultural Objects (CCO) content standard MODS standard MODS documentation guidelines <http://www.loc.gov/standards/mods/> DLF/Aquifer Implementation Guidelines for Shareable MODS Records <http://www.diglib.org/aquifer/dlfmodsimplementationguidelines_ finalnov2006 pdf> E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Pre-Migration: Mapping Crosswalks may be useful e.g. Getty crosswalk
- 7. E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Pre-Migration: Mapping The goal of normalization is to transform or clean up your data values so they conform to accepted standards, are more consistent, and can be understood by any user of your images Can be done during Pre Migration Migration Post Migration E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Pre-Migration: Mapping Data to Normalize Abbreviations e.g. Material= ol on cv.; source = DOA; attribution=sch of Inconsistencies e.g. expressions of circa: Date= c1947; c. 1988; c 500 AD; ca. 15th Century Formatting conventions e.g. title=[Grapevines at Mission de San Ignacio]
- 8. E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Pre-Migration: Mapping Other types of normalization Source system uses codes for values e.g. Name=Johnson, Ben type=1 (1=personal) Create indexed versions of dates e.g. 2/9/01, mid 15th century, October 1945 YYYY-MM-DD, YYYY-MM, YYYY (ISO 8601 standard) E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Pre-Migration: Mapping Assess metadata granularity Examples of narrow and broad worktypes Narrow: City planning, urbanism, landscape design, garden design, environmental design Broad: Architecture Narrow: costume design, fashion design, clothing, jewelry, ornament, body decoration Broad: Costume and Jewelry
- 9. E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Pre-Migration: Testing Hand pick records to evaluate mapping decisions and to test for normalization and diacritics problems Diacritics- best to encode in UTF-8 Unicode or use Unicode decimal or hexadecimal character references Display Karls tejn (Str edoc eský kraj, Czech Republic)--Castle Exported using Unicode decimal character references Karlštejn (Středočeský kraj, Czech Republic)--Castle E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Pre-Migration: Final Word BACK-UP, BACK-UP, BACK-UP
- 10. E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Migration: A few words Enlist the help of a programmer or database administrator Need for assistance will depend on your source and target systems (what tools they provide for migrating data), how much data normalization you’ll need, restructuring of the data, etc. A db administrator can help with target system setup (forms, reports, security, backup, etc) E.g. VRA Core has its own documentation and is meant to be used with the Cataloging Cultural Objects (CCO) content standard MODS schema has both the MODS documentation guidelines <> and a best practice guideline written by DLF members called DLF/Aquifer Implementation Guide Post Migration Quality Control (QC) Data Cleanup Update Cataloging Guidelines