Data mapping
| Data transformation |
|---|
| Concepts |
| Transformation languages |
| Techniques and transforms |
| Applications |
| Related |
In computing and data management, data mapping is the process of defining correspondences and transformation rules between data represented in a source model and data represented in a target model. A mapping can specify direct correspondences between data elements as well as transformations such as conversion, concatenation, lookup, or restructuring.[1][2]
Data mapping is used in data integration, data migration, extract, transform, load (ETL), electronic data interchange (EDI), application integration, and other processes in which data must be exchanged or reconciled between different representations.[3][2]
For example, a source system might contain fields named first_name and last_name, while a target system expects givenName, familyName, and fullName. A mapping could associate the first two fields directly with their corresponding target fields and define fullName as a concatenation of the two source values.
Concepts
A data mapping relates a source representation to a target representation. The source and target can be database schemas, files, messages, application data models, metadata schemas, or other structured representations.
Mappings range from simple one-to-one correspondences to transformations involving multiple fields. Common forms include direct mappings, conversions between units or formats, lookup rules, composition of several source values into a target value, decomposition of a source value into several targets, and structural transformations.
Mapping and schema matching
Data mapping overlaps with schema matching, and terminology varies in the literature. Schema matching generally concerns identifying semantic correspondences between elements of different schemas, while schema mapping commonly refers to an explicit specification of how data represented according to a source schema is related to or transformed into a target schema.[3][1]
For example, determining that surname in one schema corresponds to familyName in another is a matching problem. Defining a rule that copies surname into familyName, or that constructs fullName from several source fields, is a mapping. Matches can therefore serve as input to the construction of mappings.[1]
History
Data mapping developed within several overlapping areas of information systems, including electronic data interchange, database schema integration, data warehousing, and data integration.
Electronic data interchange
Electronic data interchange became an important application of mapping between organizational data representations and standardized business messages. Early EDI work sought to replace proprietary organization-to-organization formats with common definitions and message structures. In the United States, transportation-industry work produced integrated EDI standards in the 1970s, and ANSI accredited the X12 standards in 1977.[4] X12 has since developed transaction sets for business processes such as ordering, shipping, invoicing, and payments.[5]
International EDI developed through efforts that led to UN/EDIFACT, a set of standards, directories, and guidelines for exchanging structured data between independent computerized information systems.[6] Systems using EDI commonly map between an organization's internal representation and the standardized message structure used for interchange.
Schema integration and data warehousing
As independently designed database schemas became an integration problem, research investigated methods for identifying corresponding schema elements and reconciling heterogeneous representations. Schema matching became a basic problem in areas including data integration, electronic business, data warehousing, and semantic query processing.[3]
Data warehousing became another major application of source-to-target mapping: data from heterogeneous operational systems must be related to a warehouse schema and transformed during extraction and loading.[3][2]
XML and schema matching
The growth of XML created another important environment for explicit data transformations. The World Wide Web Consortium (W3C) published XSLT 1.0 as a Recommendation in November 1999, defining a language for transforming XML documents.[7]
In 2001, Erhard Rahm and Philip A. Bernstein published an influential survey of approaches to automatic schema matching. They described matching as a largely manual task in contemporary implementations while surveying research intended to partially automate it using schema-level, instance-level, structural, linguistic, and constraint-based evidence.[3]
The same year, Jayant Madhavan, Bernstein, and Rahm presented Cupid, a generic schema matcher that combined linguistic and structural information, including names, data types, constraints, and schema structure. Its stated applications included XML message mapping, data-warehouse loading, and schema integration.[8]
Semantic Web and RDF mapping
Mapping also became important in the Semantic Web, where heterogeneous data can be related through shared vocabularies and ontologies. In September 2012, the W3C published R2RML as a Recommendation for expressing customized mappings from relational databases to Resource Description Framework (RDF) datasets.[9] R2RML specifies how relational tables, rows, and columns can be represented as RDF resources and properties.
Assisted mapping
Research into automating mapping and matching has used several kinds of evidence, including element names, data types, schema structure, constraints, dictionaries and ontologies, and values observed in datasets.[3][1]
More recent mapping-generation research has combined relational metadata with profile data to search and rank candidate mappings over collections of independently produced datasets. Such systems are intended to assist data engineers in the labour-intensive process of producing integrations rather than assuming that source and target semantics can always be inferred automatically.[10] Contemporary semantic-integration research likewise reports that many mapping and semantic-annotation tasks continue to require substantial human intervention.[1]
Mapping techniques
Data mappings can be created and represented in several ways. These techniques are not mutually exclusive, and integration systems can combine them.
Hand-coded and declarative mapping
Mappings can be implemented directly in procedural programs, database queries, scripts, or declarative transformation languages. Examples include SQL expressions for relational transformations and XSLT for XML transformations.
Hand-coded mappings provide direct control over transformation logic but can become difficult to maintain when large numbers of source and target elements or systems are involved.
Graphical mapping
Some integration tools provide graphical source-to-target interfaces in which users connect schema elements visually and attach transformation operations such as conversion, concatenation, filtering, or lookup. The resulting mapping can be stored as metadata and interpreted or translated into executable transformation logic.
Schema-based and semantic approaches
Schema-based approaches use information such as element names, data types, constraints, and structural relationships to propose correspondences between source and target schemas.[3][8]
Semantic approaches can additionally use dictionaries, controlled vocabularies, semantic annotations, or ontologies as domain knowledge. This can identify relationships that are not apparent from element names alone and can support mappings between local schemas and shared conceptual models.[1]
Instance-based and data-driven approaches
Instance-based approaches examine values stored in data sources rather than relying only on schema definitions. Value distributions, formats, patterns, and statistical relationships can provide evidence that elements correspond or that transformations may be required.[3]
Schema-level and instance-level evidence can be combined. Modern mapping-generation research, for example, can use inferred profile data together with relational metadata to evaluate candidate mappings.[10]
Automated and assisted mapping
Mapping systems can automate parts of the process by generating or ranking candidate correspondences and transformation rules. Techniques include string and linguistic similarity, structural comparison, constraint analysis, data profiling, semantic knowledge, statistical inference, and learned models.[3][1]
Automation is particularly useful when schemas contain many elements or when mappings must be constructed across large collections of heterogeneous datasets. However, structural or statistical similarity does not by itself establish semantic equivalence, and current research continues to treat human knowledge and validation as important parts of many integration workflows.[10][1]
Applications
Data mapping is used wherever information must cross a boundary between representations or systems. Common applications include:
- Data migration – translating data from a legacy system into the schema and formats of a replacement system.
- ETL and data warehousing – defining how operational source data is transformed into warehouse tables and analytical models.[2]
- Application and message integration – translating records or messages exchanged between applications with different data models.
- Electronic data interchange – mapping internal business data to and from standardized messages such as X12 or UN/EDIFACT.[5][6]
- Data and schema integration – reconciling heterogeneous sources into a common or mediated representation.[3]
- Semantic-web publishing – mapping relational data into RDF vocabularies and graphs.[9]
- Data lineage and governance – mapping metadata can record source-to-target relationships and transformations, contributing to broader records of data provenance and movement.
Standards and mapping languages
No single standard represents all forms of data mapping. Different domains have developed interchange formats and mapping languages for particular kinds of data.
| Standard or language | Domain | Role in mapping |
|---|---|---|
| ASC X12 | Electronic data interchange | Defines standardized structures for electronic business messages to which internal business data can be mapped.[5] |
| UN/EDIFACT | Electronic data interchange | Defines internationally agreed standards, directories, and guidelines for structured electronic business messages.[6] |
| XSLT | XML | Defines transformations between XML documents.[7] |
| R2RML | RDF / Semantic Web | Defines customized mappings from relational databases to RDF datasets.[9] |
See also
References
- 1 2 3 4 5 6 7 8 Masmoudi, Maroua; Ben Abdallah Ben Lamine, Sana; Karray, Mohamed Hedi; Archimède, Bernard; Baazaoui Zghal, Hajer (2024). "Semantic Data Integration and Querying: A Survey and Challenges". ACM Computing Surveys. 56 (8): 1–35. doi:10.1145/3653317.
- 1 2 3 4 Shahbaz, Qamar (2015). Data Mapping for Data Warehouse Design. Elsevier. ISBN 978-0-12-805335-5.
- 1 2 3 4 5 6 7 8 9 10 Rahm, Erhard; Bernstein, Philip A. (2001). "A Survey of Approaches to Automatic Schema Matching". The VLDB Journal. 10 (4): 334–350. doi:10.1007/s007780100057.
- ↑ "Awards". X12. Edward A. Guilbert Lifetime Achievement Award; Earl "Buddy" Bass Achievement Award. Retrieved 6 September 2026.
- 1 2 3 "About X12". X12. Retrieved 6 September 2026.
- 1 2 3 "Executive Guide on UN/EDIFACT". United Nations Economic Commission for Europe. Retrieved 6 September 2026.
- 1 2 "XSL Transformations (XSLT), Version 1.0". World Wide Web Consortium. 16 November 1999. Retrieved 6 September 2026.
- 1 2 Madhavan, Jayant; Bernstein, Philip A.; Rahm, Erhard (2001). "Generic Schema Matching with Cupid" (PDF). Proceedings of the 27th International Conference on Very Large Data Bases. pp. 49–58.
- 1 2 3 "R2RML: RDB to RDF Mapping Language". World Wide Web Consortium. 27 September 2012. Retrieved 6 September 2026.
- 1 2 3 Mazilu, Lacramioara; Paton, Norman W.; Fernandes, Alvaro A. A.; Koehler, Martin (2022). "Schema Mapping Generation in the Wild". Information Systems. 104: 101904. doi:10.1016/j.is.2021.101904.
{{cite journal}}: CS1 maint: article number as page number (link)
Notes
This article is a direct transclusion of the Wikipedia article and therefore may not meet the same editing standards as LIMSwiki.
