How To Get The Most Out Of Data Lineage?

Global IDs Presents How To Get The Most Out Of Data Lineage? A Business User's Guide To Know About Data Lineage!

INTRODUCTION Business users must understand key concepts about data lineage so they effectively participate if they become involved in the acquisition of data lineage technologies. If business users do not understand these concepts they will not be able to communicate their requirements effectively and may get saddled with a technology that does not give them what they need. 1

What Is Data Lineage? Typically, a business user comes from the perspective of an endpoint — a production report — and wants to look back through the data pipeline to understand how the data got into that report. That is clear enough, but the term “data lineage” is used in many different ways by consultants and vendors in the data industry which often drives confusion. The three main versions for data lineage are: 2

Data Relationships The visualization of any relationship in the data is sometimes branded as “data lineage.” For instance, consider the relationship where one Customer can have many Accounts. This can be represented in a diagram generated on a screen by a box that appears for a Customer, some boxes for Accounts, and lines joining the Customer box to the Account boxes. This is described as a “data lineage diagram.” However, it is not. There is no flow of data in these types of diagrams. They are structural, logical diagrams. They certainly have a lot of value, but they are not data lineage, and they will not help you find out how strange numbers got into a report. 3

Data Roadmaps We’ve probably all used Google Maps, Waze or some equivalent of GPS navigation, and we know how incredibly useful they are. If one of us invites a guest to our house, we just need to give the guest our address, and they can use one of these tools to figure out how to get there. There is a type of data lineage that is like this, too. It is essentially a map that shows the route or routes that data can take as it flows through data pipelines and gets to the report where we found the problem we are trying to deal with. 4

Data Provenance In the world of fine art, notorious for forgery, theft and ownership disputes, “provenance” is a key concept. It is the chain of custody of legal ownership, back to the original artist, which must be documented, warranted and provable. In data lineage, this is being able to prove how a particular piece of data — a data value — got to an endpoint, like a report where a Business User suspects there is a problem. It is very similar to provenance in fine art; it is the chain of custody of an individual data value. 5

Understand Data Lineage When Selecting Tools Business users are increasingly voicing their need for data lineage solutions. Getting new enterprise-class software is nearly always the responsibility of the IT department, so business users will have to partner with IT for any data lineage tool acquisition. This raises the risk of business users being confronted by a dizzying array of technical features and undoubtedly beautiful visualizations that may or may not be useful. It is vital that business users understand the concepts we have described to get to a solution that will truly satisfy their needs. 6

www.globalids.com Founded in 2001 to address the challenges of next- 182 Nassau Street, Suite 202 Princeton, NJ 08542 generation data ecosystems, Global IDs is led by Arka +1 (888) 514-0192 Mukherjee and a management team of proven executives with key industry and CIO level experience. We are passionate about data design and information management and take great pride in building software that solves complex problems for the world's most demanding institutions. A data management industry expert with extensive experience in master data management and data warehousing.