Get started with DataHub

DataHub is an extensible data catalog that enables data discovery, data observability, and federated data governance. This tutorial describes how to get started with Charmed DataHub, the Kubernetes operator that deploys and manages it. It takes you from setting up a local Juju environment, through deploying DataHub and its supporting charms, to ingesting metadata from a PostgreSQL database and exploring it in the DataHub UI.

DataHub relies on three backing services: PostgreSQL for metadata storage, Kafka for event streaming, and OpenSearch for search and graph indexing. OpenSearch runs on machines rather than Kubernetes, so this tutorial builds a small two-cloud environment on a single host: an LXD cloud for the data platform and a MicroK8s cloud for DataHub itself. This mirrors how production deployments are structured, just at a smaller scale.

By the end of this tutorial, you will have a working DataHub instance and a searchable data catalog containing the schema of a real database table.

Prerequisites

This tutorial installs both LXD and MicroK8s on the same host and writes host-wide kernel parameters. Run it in a disposable VM rather than your personal machine. Multipass is a quick way to get one:

sudo snap install multipass
multipass launch --name datahub-tutorial --cpus 8 --memory 24G --disk 80G noble
multipass shell datahub-tutorial

Before you begin, make sure the machine (or the Multipass VM above) has:

  • Ubuntu 22.04 LTS or later

  • Snap installed

  • 8 CPU threads

  • At least 24 GB of RAM

  • At least 80 GB of available disk space

The DataHub stack is memory-hungry: it runs three JVM-based workloads alongside Kafka and OpenSearch. Machines below these requirements may see workloads restart or fail to settle.

Tutorial steps

  1. Set up your environment: install LXD and MicroK8s, and bootstrap the two Juju controllers this tutorial uses.

  2. Deploy supporting charms: deploy PostgreSQL, Kafka, and OpenSearch, and offer them for cross-model access.

  3. Deploy DataHub: deploy DataHub, relate it to its backing services, and log in to the UI.

  4. Ingest metadata from a database: create a demo table and pull its schema into the DataHub catalog.

  5. Clean up