Go back

This is a comprehensive summary of Chapter 8 from Data Mesh in Action. This chapter transitions from the theoretical and MVP concepts discussed in previous chapters to concrete, real-world technical architectures. It presents four distinct "blueprints" for building a self-serve data platform using different technology stacks: Google Cloud Platform (GCP), Amazon Web Services (AWS), Databricks, and Apache Kafka.

Each section analyzes the architecture, identifies platform and data product components, examines user workflows, and evaluates the pros and cons of the specific approach.

Introduction: The Role of Physical vs. Logical Centralization

The chapter emphasizes that while the Data Mesh paradigm requires logical decentralization (domain-oriented ownership), the underlying infrastructure often utilizes physical centralization (shared cloud services) to reduce friction and maintenance costs. The goal of the architectures presented is to balance team autonomy with platform maintainability,.

1. Data Mesh on Google Cloud Platform (GCP)

This architecture is designed to balance ease of setup with a high degree of autonomy for data-producing teams. It relies heavily on GCP-native serverless offerings.

1.1 Architecture and Team Topology

The architecture centers around four specific participants:

  • Black Team (Development): Creates data products using direct pipelines.
  • White Team (Data Science): Creates and consumes data products.
  • Gray Team (Platform): Owns the self-serve platform, providing configurable Terraform templates.
  • Consumers: Business analysts (using SQL) and automated systems (Recommendation Engine).

1.2 Platform Components

The platform is divided into the Platform Interface and the Platform Kernel:

  • Interface: A Git repository containing documentation (README) and Terraform templates. This is the entry point for data producers.
  • Kernel: The configurable Terraform templates themselves. This represents a low-abstraction platform where the "service" provided is infrastructure-as-code (IaC) that teams apply themselves.

1.3 Data Product Components

Data products in this architecture share a common toolset based on GCP services,:

  • Storage/Ports: BigQuery datasets serve as the primary container for data products. Each data product corresponds to one BigQuery dataset (which may contain multiple tables).
  • Transformation: Google Cloud Dataflow (serverless batch/stream processing) is used for ingestion and transformation pipelines.
  • Ingestion: Google Cloud Pub/Sub is used for event messaging, and Google Cloud Storage (GCS) is used for file-based inputs.
  • Access Control: Service Accounts are critical here. Each data product includes a specific service account to manage permissions at the dataset level, acting as the technological boundary for domain ownership.

1.4 Workflows

  • Producers: Access the platform interface (Git repo), pull the latest Terraform template, configure it (e.g., input storage options), and apply it to create their stack. They then configure Dataflow pipelines to populate their BigQuery datasets.
  • Consumers: Use the BigQuery SQL interface to browse datasets. Access requests are handled by the owning team modifying service account permissions.

1.5 Variations and Governance

The text suggests enhancing this architecture by,:

  • Data Catalog: Integrating DataHub directly with BigQuery to serve as a central discovery tool.
  • Automated Quality: Using GCP logging routers to send logs to a sink, where a custom component computes usage statistics or quality metrics (e.g., counting NaN values) and pushes them to DataHub.
  • Metadata Management: Wrapping the Terraform template in a GUI to enforce richer metadata collection during product creation.

1.6 Summary Profile (GCP)

  • Pros: High autonomy for teams; serverless components reduce maintenance; easy integration of native services.
  • Cons: Vendor lock-in; requires extra effort to integrate non-GCP data sources.
  • Best For: Small- to medium-sized companies seeking a middle ground between autonomy and maintenance costs.

2. Data Mesh on Amazon Web Services (AWS)

This architecture is inspired by implementations at companies like Zalando and BMW. It differs from the GCP example by focusing on object storage (S3) rather than tabular storage (BigQuery) and places a heavier emphasis on enabling data consumers.

2.1 Architecture and Team Topology

The setup is similar to the GCP model but includes advanced Business Analysts who act as both consumers and producers (creating ETL jobs). The architecture utilizes distinct AWS accounts to enforce domain boundaries.

2.2 Platform Components

The Gray (Platform) team provides four key deliverables:

  1. Producer Template: Terraform configuration for deploying data products.
  2. Consumer Template: Configuration for analysis tools (Athena, SageMaker).
  3. Access Management: A central AWS Lake Formation instance combined with an IAM role repository to manage cross-account access.
  4. PII Service: A central API service that hashes personalized data, simplifying GDPR compliance for producers.

2.3 Data Product Components

The technical stack relies on AWS Glue and S3,:

  • Storage (Output Port): S3 buckets serve as the physical storage for data products.
  • Transformation: AWS Glue (Python/Scala) handles ETL jobs.
  • Discovery: A federated Glue Data Catalog is used. A Glue crawler updates metadata, which is synced to the central catalog via Lake Formation.
  • Streaming: Amazon Kinesis Data Streams are used for real-time data ingestion, pushing data to Firehose and then S3 for processing.

2.4 Workflows

  • Producers: Pull templates and apply them to their AWS accounts. They can opt-in to use the central PII service within their Glue code to automatically handle sensitive data.
  • Consumers: Use Amazon Athena (SQL query engine) via a default "open-for-all" IAM role to browse metadata. For deeper analysis, they use the Consumer Template to spin up Amazon SageMaker notebooks.

2.5 Variations

  • Legacy Integration: Using AWS Fargate containers to pull data from legacy non-AWS systems and push it into Kinesis streams.
  • Query Engines: Replacing Athena with Trino for more advanced SQL capabilities.
  • Deployment: Using AWS Cloud Development Kit (CDK) or CloudFormation instead of Terraform for tighter cloud integration.

2.6 Summary Profile (AWS)

  • Pros: Maximum flexibility and autonomy (teams own their infrastructure); wide support for diverse workloads; separates compute from storage.
  • Cons: Moderate vendor lock-in; requires a dedicated platform team to manage the templates and Lake Formation.
  • Best For: Medium- to large-sized companies prioritizing maximum flexibility for development teams.

3. Data Mesh on Databricks

This architecture represents a more centralized technological approach. While logical domains remain separate, the infrastructure is unified on the Databricks platform. This reduces maintenance significantly but limits architectural freedom.

3.1 Architecture and Team Topology

  • Platform Team: Can be very lean (potentially a single engineer) because Databricks manages the underlying complexity.
  • Context: Data domains are separated by folders or workspaces within Databricks rather than separate cloud accounts.

3.2 Platform Components

  • Interface: Instead of complex templates, the platform interface consists of Guidelines—documentation on how to use Databricks namespaces, access rights, and storage locations.
  • Kernel: The Databricks environment itself, including Delta Lake (storage layer) and Apache Spark (compute engine).

3.3 Data Product Components

Data products are built entirely within the Databricks ecosystem:

  • Ingestion: External tools (like AWS Fargate or Kinesis) are often needed to get data into the Delta Lake initially.
  • Processing: Databricks Notebooks (Python/Scala/SQL) are used for batch and streaming transformation jobs.
  • Storage: Delta Lake tables act as the data output ports.
  • Composition: Creating new data products from existing ones is simple—users just reference existing Delta tables in a new notebook.

3.4 Workflow Considerations

  • Lock-in: This architecture is highly intertwined with Databricks. Teams cannot easily "opt-out" or use different engines without breaking the mesh pattern.
  • Target Audience: It favors data engineers and data scientists over software developers, as the workflow revolves around notebooks and Spark rather than microservices.

3.5 Summary Profile (Databricks)

  • Pros: Extremely low maintenance; excellent support for analytics and ML; "Lakehouse" architecture simplifies storage.
  • Cons: High vendor lock-in; limited support for standard software engineering workflows; requires external ingestion tools.
  • Best For: Small- to medium-sized companies, or those focused heavily on analytical/ML data products rather than operational software.

4. Data Mesh on Apache Kafka

This architecture focuses on data-intensive applications, particularly those requiring real-time processing. It shifts the "center of gravity" to the development teams and event streams.

4.1 Architecture and Team Topology

The Kafka cluster acts as the physical center of the infrastructure. Logical separation is achieved through Topics and Streams. This setup is ideal for companies where data needs to be integrated back into operational applications quickly.

4.2 Platform Components

  • Interface: Documentation and standard configuration for connecting to the Kafka cluster and Schema Registry.
  • Kernel: Apache Kafka cluster, Kafka Connect, and ksqlDB.

4.3 Data Product Components

Data products here are defined by streams and connectors,:

  • Input Ports: Standardized using Kafka Connect Sources (e.g., pulling from a PostgreSQL JDBC source) or the Producer API.
  • Transformation: ksqlDB is the default tool for transforming streams using SQL syntax (e.g., filtering, joining streams).
  • Output Ports:
    • Kafka Topics: Event streams accessible via the Consumer API.
    • Kafka Sinks: Connectors that dump data into external storage like Snowflake tables or S3 buckets for batch analysis.
  • Schema: Relies on the Confluent Schema Registry to ensure contract validity between producers and consumers.

4.4 Workflows

  • Producers: Development teams use Kafka Connect to "source" data from their microservices into topics. They use ksqlDB to transform this data into public "Data Product" topics.
  • Consumers: Other apps consume these topics directly.
  • Analysts: This architecture presents a challenge for traditional analysts. To support them, the platform must usually include a query engine like Trino (formerly Presto) or sink data into a warehouse (like Snowflake) to allow SQL querying across streams.

4.5 Summary Profile (Kafka)

  • Pros: Native support for real-time applications; strong decoupling via event streams; high performance.
  • Cons: Steep learning curve (streaming paradigms); harder for data analysts to query directly; strong technological centralization on Kafka.
  • Best For: Companies building data-intensive, real-time applications where development teams are the primary data producers and consumers.

Comparative Summary

The chapter concludes that there is no single "correct" technical implementation of a data mesh. The choice depends on the organization's specific needs regarding autonomy vs. maintenance:

  1. GCP: Balanced approach. Good autonomy, serverless ease-of-use.
  2. AWS: High autonomy. Teams own their infrastructure (accounts). Best for flexibility.
  3. Databricks: High centralization. Teams share a workspace. Best for low maintenance and heavy analytics.
  4. Kafka: Application-centric. Focus on real-time streams. Best for software engineering-led data organizations.

Regardless of the stack, the architecture must support the core Data Mesh principles: Domain Ownership (via accounts/datasets), Data as a Product (via defined ports/catalog), Federated Governance (via central policy engines like PII services), and Self-Serve Platforms (via templates and automation).