Go back

Based on the provided sources, here is a comprehensive summary of Chapter 5: "Data as a Product."

Chapter 5: Data as a Product

This chapter focuses on the second principle of the data mesh: shifting the perspective from data as a by-product to data as a product. While technical aspects like schemas are important, data often lacks the coherence required for easy management and consumption. Organizations frequently have valuable datasets trapped in private drives or unprepared formats. To solve this, the data mesh mandates treating data as a well-defined unit tailored to user needs, ensuring it is findable, accessible, interoperable, and reusable,.

5.1 Applying Product Thinking

To transform data into a product, organizations must apply product thinking. This problem-solving technique prioritizes defining the problem a user wants to solve before designing the solution. It is guided by two main principles:

  1. Love the problem, not the solution: One must deeply understand the users and their specific problems before designing aspects of the product.
  2. Think in products, not features: Instead of merely exposing existing data or adding features, the focus must be on satisfying user needs. The final product results from the interaction between users, the team, and the technology.

Before exposing a dataset, teams should ask critical questions: What problem is being solved? Who is the user? What is the vision and strategy? What features should be included?.

Product Thinking Analysis: Messflix Case Study The chapter applies product thinking to several candidates from the "Produce Content" domain identified in the previous chapter:

  • Cost Statement Data Product:

    • Problem: Production costs are currently managed via manual Excel files and manually imported into data warehouses, limiting complex analysis,.
    • Vision: Create a solution for automated, complex cost comparisons and budget forecasting.
    • Implementation: Files must be standardized (e.g., consistent date formats and currency), validated, and exposed via shared drives for data engineers and financial analysts.

  • Scripts Data Product:

    • Problem: The selection process for scripts is manual and error-prone.
    • Vision: Enable a script recommender system that automatically pulls data to rank scripts for production.
    • Implementation: Since script functionality exists as a microservice, the team must expose this data via a REST API containing script metadata (type, character count, etc.).

  • Movie Popularity Data Product:

    • Problem: Data is currently fetched ad-hoc from an external API (World Movie DB) and allows only manual use via a custom UI.
    • Vision: Automate data retrieval to support the script recommender.
    • Implementation: Create a solution to download data weekly and store it as analytics-optimized files containing production rankings.

The Data Product Canvas To structure this analysis technically, the authors introduce the Data Product Canvas, a visual tool for collaborative design. It details essential components such as:

  • Domain/Business Capability: The area the product belongs to (e.g., Produce Content).
  • Classification: Is it source-aligned (direct from source), consumer-aligned (transformed for specific needs), or shared core? Is it virtual (computed at runtime) or materialized (persisted)?
  • Ports: How data is exposed (e.g., REST API, CSV files).
  • Flow: Inbound and outbound datasets.
  • Security: Public or restricted access,.

For example, the Cast Data Product is classified as source-aligned and stable, exposing actors and movie roles via a REST API. The Movie Trends Data Product is materialized and source-aligned, derived from the Movie Market Monitor system, and exposed via API and CSVs for stable production use.

5.2 What is a Data Product?

The chapter provides a precise definition to distinguish a data product from a by-product or a generic analytical tool.

Definition A data product is an autonomous, read-optimized, standardized data unit containing at least one domain dataset, created to satisfy user needs.

Key Characteristics:

  • Autonomous: It is a self-contained unit (system or dataset) that can be developed independently without affecting other systems. For instance, the Movie Popularity product is a separate component (a local data lake) that independently fetches, processes, and stores data.
  • Read-Optimized: Storage formats are selected for analytical efficiency. For example, movie rankings may be stored as files optimized for Python-based data engineering tools.
  • Standardized: It adheres to organizational rules regarding metadata descriptions, protocols, ports, and identifiers.
  • Node in a Mesh: It integrates into the larger ecosystem by registering with a data catalog and enabling self-service.
  • Contains Domain Datasets: It must hold data with intrinsic business value (e.g., movie rankings or script metadata).
  • Satisfies User Needs: It is designed with product thinking to serve specific users, such as accountants controlling production costs.

Product vs. Project Creating a data product is not a one-time project. It requires a long-term perspective involving continuous improvement, lifecycle management, and adaptation based on user feedback.

What Can Be a Data Product? Any data representation offering user value can be a data product. Examples include:

  • Raw unstructured files (images, videos) with metadata.
  • Simple files (CSV, Excel).
  • Read-optimized databases (SQL, NoSQL).
  • REST APIs (potentially supporting HATEOAS).
  • Data streams (change history or snapshots).
  • Data marts and denormalized tables.
  • Graph databases-.

5.3 Data Product Ownership

Treating data as a product requires specific roles to ensure its long-term value and evolution.

Data Product Owner (DPO) The DPO is responsible for the business vision, lifecycle, and evolution of the data product. This role is distinct from a project manager; the DPO thinks long-term about maximizing data utility, gathering user feedback, and managing the product roadmap,.

DPO Responsibilities:

  • Vision and Needs: Defining the purpose and understanding user expectations.
  • Strategic Planning: Creating roadmaps, defining KPIs and SLAs.
  • Requirements Assurance: Ensuring metadata quality and compliance with governance standards.
  • Backlog Management: Prioritizing and clarifying requirements.
  • Stakeholder Management: Managing conflicting requirements between users and management.
  • Governance Participation: Influencing organizational data rules-.

Data Product Development Team This is a cross-functional, Agile/DevOps-style team responsible for the end-to-end implementation and maintenance of the product. Unlike centralized data teams, this team works within the domain,. It includes diverse competencies such as data engineering, operations, software development, data science, testing, and security.

Relationship with Product Owners The DPO role relates to the traditional software Product Owner (PO) in three ways:

  1. DPO is the PO: Best when the data product is a natural, low-complexity extension of a source system (e.g., a subscription module exposing purchase data).
  2. DPO is the PO, but teams differ: Used when analytical data requirements are extensive enough to require a separate backlog and team (e.g., a module providing predictive analytics).
  3. DPO is not the PO: Necessary when the data product is a complex, independent solution (e.g., a data mart with ML models).

5.4 Conceptual Architecture of a Data Product

Designing a data product involves defining its external interfaces (for consumers) and internal components (for implementers).

External Architecture View To function as an autonomous node, a data product exposes various interfaces:

  • Communication Interfaces (Ports): Define the format/protocol for reading data (e.g., SQL, API, Files).
  • Information Interfaces: Provide metadata, lineage, and quality metrics to users.
  • Configuration Interfaces: Allow the platform to configure security and ecosystem integration-.

Data Product Ports A single data product can expose the same data through multiple output ports to suit different user personas:

  • Database-like Storage: SQL/NoSQL interfaces for technical users processing large datasets.
  • Files: CSV or Excel files for data scientists or business users, suitable for large amounts of data.
  • REST API: Good for system integration and small data subsets. Adding GraphQL allows complex queries; HATEOAS supports automated browsing.
  • Streams: Messaging systems (like Kafka) for real-time, distributed processing or event sourcing.
  • Visualizations: Charts and dashboards for non-technical users.

Internal Architecture View The internal implementation depends on technology but generally includes:

  • Datasets: The actual data (tables, files, streams).
  • Metadata: Machine-readable descriptions (e.g., JSON files) defining the schema, owner, and quality metrics.
  • Code: Scripts for data cleansing, ETL/ELT transformations, infrastructure as code (IaC), CI/CD pipelines, and scheduling.

For example, the Cost Statement data product internally uses Python scripts scheduled by a workflow engine to read spreadsheets, clean them, and write them to a database (for the API port) and shared drive (for the file port), while generating logs and quality reports.

5.5 Data Product Fundamental Characteristics

A data product must be findable, understandable, addressable, secure, usable, and trustworthy.

Self-Described Data Product Metadata is the "fuel" of the data product. Unlike traditional approaches where metadata is scattered in external catalogs, a data product must be self-described, containing all necessary information to be used autonomously,.

Metadata as Code The chapter advocates for metadata as code, stored alongside the data (e.g., in JSON format) to enable machine interpretation and versioning. Key metadata categories include:

  • Data Product Metadata: Business-oriented info (ID, title, description, owner contact, business unit, terms of use).
  • Domain Dataset Metadata: Technical info (ports, schema, download URLs, version).
  • Other Metadata: Lineage, data quality metrics, security rules, and semantic definitions.

Adopting standards is crucial. The authors recommend W3C standards like DCAT (Data Catalog Vocabulary), PROV (Provenance), and ODRL (Rights Expression) to ensure consistency.

5.6 Additional Data Product Characteristics: FAIR and Immutability

To maximize utility, data products should adhere to FAIR principles (Findable, Accessible, Interoperable, Reusable), originally designed for scientific research but essential for the data mesh.

Findability Data and metadata should be easy for both humans and machines to find.

  • Assign globally unique, persistent identifiers to data and metadata.
  • Describe data with rich metadata.
  • Register metadata in a searchable resource (e.g., a data catalog).

Accessibility There must be clear rules and standard protocols for accessing data.

  • Retrieve metadata/data by identifier using open, standardized protocols (e.g., HTTP/HTTPS).
  • Ensure the protocol supports authentication and authorization.
  • Key Requirement: Metadata should remain accessible even if the data itself is no longer available (e.g., for historical auditing).

Interoperability Data needs to be easily integrated with other data products.

  • Use formal, shared languages for knowledge representation (e.g., JSON-LD, RDF).
  • Use standard vocabularies and ontologies.
  • Include qualified references to other data (e.g., linking a "gender" field to a standard enum definition via URL).

Reusability Data should be usable in different contexts.

  • Release data with clear usage licenses (e.g., "Internal Use Only").
  • Associate data with detailed provenance (lineage).
  • Meet domain-relevant community standards.

Immutability While not always mandatory, immutability is a highly desirable characteristic. Data products should ideally allow access to data from any point in the past (time travel). This is critical for reproducibility (e.g., verifying conclusions drawn from specific data versions) and auditing. If data is not immutable, the product must explicitly declare how past data is accessed or if it is lost-.

5.7 Data Contracts and Sharing Agreements

Product thinking implies a mutual relationship between producers and consumers. Often, internal company relationships become one-sided. To fix this, the data mesh utilizes Data Contracts and Sharing Agreements.

Data Contracts A data contract acts as a delivery or service guarantee from the producer. It specifies:

  • The nature of the data (production vs. test).
  • Quality guarantees (e.g., "checked for broken records").
  • Service Level Agreements (SLAs) regarding uptime and delivery speed.
  • Versioning strategy (e.g., semantic versioning).
  • The schema and metadata.

Data Sharing Agreements This extends the contract by specifically targeting the consumer's intent. It involves collaboration to define:

  • The goal of sharing the data.
  • Intended usage (e.g., joining with other datasets).
  • Security levels (e.g., is public access acceptable?).
  • History requirements (e.g., need for a 3-year backfill).

Implementation Strategy Implementing these agreements should not block development. The chapter suggests a three-step evolution:

  1. Cultural Shift: Start with simple web forms or documents to force the conversation between producer and consumer. Focus on the discussion, not the artifact.
  2. Integration: Once the mesh grows, integrate these contracts into the GUI of the self-serve platform/data catalog.
  3. Automation: Finally, automate checks against the contracts (e.g., automated SLA monitoring or access policy enforcement).

In summary, treating data as a product involves a shift from ad-hoc data generation to a deliberate, user-centric design process supported by clear ownership, robust architecture, rich metadata, and strong governance agreements.