Predli Studio — Document Ingestion on Dagster

Context

Predli Studio is Predli’s enterprise RAG product — the platform clients log into to ask questions over their own documents and get cited, grounded answers. It serves multiple enterprise tenants from one codebase.

I owned the ingestion layer: the path from a tenant’s raw documents to the searchable, citation-ready knowledge the Studio app reads from. The single most consequential decision in that build was the orchestrator: Dagster, with its asset-based model, rather than a traditional task-based scheduler like Airflow or Prefect.

That choice is what this case study is mostly about, because it’s the part most worth carrying into other systems.

The problem with "just write a pipeline"

The first version of any ingestion system is a script. Read PDFs from a folder, chunk them, embed them, push to the vector store. It works on a laptop with one tenant and 50 documents. Almost nothing about it survives contact with an enterprise rollout:

• Documents change. A document gets updated, deprecated, withdrawn for compliance. The chunks and embeddings derived from it have to follow.
• Connectors retry. A scheduled sync re-emits the same file. Without a story for "I’ve seen this exact bytes before", you get duplicate chunks in the index every time the network blips.
• The chunker isn’t fixed. Six months in you’ll change your chunking strategy or your embedding model. Re-deriving the affected stage and everything downstream — without re-fetching every PDF — needs to be a normal operation.
• You’ll need to explain failures. When a tenant asks "why isn’t this document showing up in answers?", the answer cannot be "let me grep the logs."

A task-based scheduler treats each of these as exception handling. An asset-based orchestrator treats them as the default operating mode. That asymmetry is why the orchestrator choice matters more than it looks.

Why Dagster, specifically

Dagster’s core abstraction is the software-defined asset: a chunk of stored data — a parsed document, a chunk row, an embedding vector — declared as code, with lineage, metadata, and a function that produces it from upstream assets.

Three things that follow from that, more or less for free:

1. Asset lineage. Every embedding traces back to the chunk it came from, the parsed document that produced the chunk, and the source file the parse came from. When something looks wrong in retrieval, the lineage answers "why" without forensics.
2. Selective reprocessing. When an upstream asset changes, you re-materialize the affected asset and let the dependency graph fan it forward. The unchanged upstream stays cached. Reprocessing is a one-line operation, not a rebuild.
3. A real operations surface. Dagster’s UI shows the asset graph, run history, and materialization metadata. Support questions about "what was ingested for this tenant, when, and what happened" become a UI navigation, not an archaeological dig.

I evaluated Airflow and Prefect; both would have worked. But neither models the data that flows through the pipeline. They model the tasks. For an ingestion system whose entire job is producing correct derived data, that’s the wrong center of gravity. You end up rebuilding asset-style abstractions on top of a task scheduler — and doing a worse job of it than the system that ships them by default.

Runtime topology

Before the asset graph, the deployment topology — how Dagster itself is wired to run in production:

Three processes, with separate concerns:

• Web Server — fronts the UI and the GraphQL API. This is what humans interact with (operations dashboard, asset graph view, run history) and what external systems integrate against. It does not, itself, execute pipeline code.
• Daemon — long-running background process that owns time-based behavior: schedules (cron-style runs) and sensors (event-driven runs that fire when something changes upstream — a new file, a new connector record, a new tenant). When a schedule or sensor fires, the daemon asks the user-code container to launch a job.
• User Code Container — the only process that actually runs pipeline code. Asset definitions, job definitions, the parsing/chunking/embedding logic — all live here. Code changes redeploy this container alone, without touching the web server or daemon.

External triggers (a tenant uploading a file in Studio, a scheduled connector pull, a sensor detecting a new document) come in through the web server. The daemon launches scheduled and sensor-driven runs against user code. Manual runs and ad-hoc asset materializations launch directly from the web server. Either way, runtime logs, materialization metadata, and events flow back to the web server so the UI can show what happened.

The reason this separation matters: it lets each process be deployed, scaled, and restarted independently. A web-server restart for a UI fix doesn’t interrupt running ingestion. A user-code redeploy (the most frequent change) doesn’t require restarting the scheduler. The daemon and the web server are stable infrastructure; the user-code container is where the actual work churns.

The asset graph

The pipeline is a graph of assets, each declared as code with a clear function, an explicit dependency on the assets upstream of it, and metadata on what was produced.

Parsing branches by document type, carrying over the file-type-aware approach that drives the Studio RAG product itself: PDFs preserve headings and table boundaries, decks parse slide-by-slide with notes as metadata, spreadsheets keep header context so numerical regions survive, web pages get readability extraction. The contract downstream of parsing is uniform — a document object with typed regions that chunking can reason about.

The pattern most worth calling out — content-hash idempotency

Of the engineering decisions in this build, the one I’d carry into every ingestion system I touch from now on is the boring one: the raw-document asset is keyed off a content hash of the source bytes.

The consequence: re-uploading the same file is a no-op, not a duplicate. A connector that retries during a deploy doesn’t silently double the chunks in the index. A user who drags the same file in twice doesn’t pay for two embeddings. The pipeline becomes deterministic on its inputs — the same file, run through the same pipeline, produces the same downstream assets, every time.

This sounds obvious. It is not what most ingestion pipelines actually do. The first time a connector misbehaves — and they all eventually do — you find out which side of this you’re on. The fix after the fact is hard (you have to deduplicate a vector store with poisoned chunks); the choice up front is one line of code at the asset boundary.

Resources, not hardcoded clients

A smaller but worth-mentioning pattern: every external system — vector store, blob store, embedding API, parser library — is wired in as a Dagster resource rather than a directly-imported client. Asset code never instantiates a vector-store client; it asks the run for one. Environment swaps (local containerized stack → staging → production) are configuration, not code changes. New tenants pointed at different backends are configuration, not code changes. Mocking in tests is configuration, not code changes.

It’s the kind of decision that costs five minutes to set up and saves a week per major refactor for the life of the system.

Reflection

Two things from this build that I’d carry forward:

Pick the orchestrator that models your output, not your tasks. Airflow models tasks; Dagster models assets. Pick the one whose abstraction matches what you actually care about being correct. For a system whose product is derived data — chunks, embeddings, indexes — an asset-based orchestrator is a different category of tool, not a slightly nicer DAG runner. The "asset graph as the source of truth" framing changes how you think about every later question: backfills, debugging, observability, even tenant onboarding.

Make idempotency a property of the asset boundary, not of the application. The day a connector retries during a deploy is the day every shortcut you took on idempotency presents its bill. Hashing the source bytes and keying the asset off the hash is the cheapest correctness investment available; do it before anything else.

There’s a broader recommendation I’d make for any enterprise multi-tenant RAG system — that cross-tenant isolation belongs at the storage layer (per-tenant namespaces in the index, partitioned schemas in the relational mirror) rather than in application-layer query filters. Application-layer filters fail open; storage-layer isolation fails closed. For an enterprise product, that distinction is the contract you sign.

The product story this infrastructure feeds — what RAG looks like when chunking and retrieval actually respect document structure — is documented separately as the Enterprise RAG Platform.