10 Design Principles for Building a Modern Journey Analytics Solution
By Trevor Paulsen
Principles First, Technology Second
A core belief I have about both good engineering architecture and good product management is that you need conviction about the problems you want to solve - and almost more importantly, about the problems you won't solve. This conviction gives you useful constraints, and is especially important when working with AI coding tools like Claude Code. If you've read my last post on building journey analytics with AI, you already know what I believe a journey analytics solution should do - so in this post, I want to establish the design principles that guide how we'll build it.
To start, I think it's important to recognize that while AI coding tools are very powerful, they don't change the fundamental economics, security, or complexity of data software. Some problems - like query engines or data synchronization - have been solved by thousands of engineers over many years. Reinventing those wheels would be foolish, even with AI assistance.
Additionally, it's important to make architectural choices that set us up for success: designing for historical rewrites, optimizing for speed, structuring code so AI agents can work in parallel. Ultimately, the goal will be to let AI do what it can do well, and leverage other technology choices everywhere else.
So here are the 10 principles that will shape the technology and architecture choices we make before we start writing any code. Let's dive in!
10 Principles for Building a Journey Analytics Solution
1. Keep the Warehouse as the Source of Truth
This is about privacy and consistency. The analytics layer may need its own optimized copy of the data for query performance, identity stitching, etc. - but that copy should stay continuously synced with the warehouse via CDC (Change Data Capture - a pattern for tracking and propagating data changes between systems). If a customer deletes a user's data from their warehouse (say, for GDPR compliance), that deletion should automatically flow through to the analytics layer. If you need to correct a data quality issue, that correction should propagate.
The key is that all data governance should happen in one place: the warehouse. Anything else should be a synchronized replica, not an independent data store that needs its own governance.
2. Don't Build Data Synchronization Infrastructure
Keeping data in sync between systems is a solved problem. Tools like Airbyte, Fivetran, and dozens of others have spent years building reliable and secure CDC pipelines, handling edge cases, managing retries, and dealing with the thousand ways data synchronization can fail.
Rebuilding this from scratch would be months of work on problems (and security risks) that have nothing to do with journey analytics. Hard pass.
3. Be Data Collection Agnostic
Journey analytics shouldn't care where your data comes from. Adobe Analytics, Google Analytics, Snowplow, Segment, custom pipelines - you should be free to use whatever collection technology suits your company best. Most companies already have data flowing into their warehouse, and the last thing anyone wants is another collection tool to maintain.
The journey analytics layer should work with whatever data collection pipeline you prefer, not force you to roll out more tags on your site or app.
4. Don't Build a Query Engine
This one should be obvious, but I'll say it anyway: building a query engine that can handle billions of rows with sub-second response times is hard. Like, PhD-thesis-hard. Companies like Adobe, Google, Snowflake, Databricks, and ClickHouse have large teams dedicated to query optimization.
This isn't something you should attempt to build on your own - and I don't believe Claude and I could realistically recreate such a thing.
5. Design for Historical Rewrites
Identity stitching is at the heart of journey analytics. When you learn that device_abc and device_xyz belong to the same person, you need to go back in time and restate all their historical events under a unified identity. The same goes for privacy deletions, data corrections, and any other changes that need to propagate retroactively.
This means the system must handle continuous, large-scale rewrites of historical data without breaking a sweat.
6. Optimize for Speed at Every Layer
Journey analytics is inherently exploratory. Analysts iterate rapidly, adjusting queries as they learn. If each query takes 30 seconds, that exploration grinds to a halt.
Speed isn't just about the query engine - it's about every layer. The config database, the semantic layer lookups, the API itself - all of it needs to be fast. And as query complexity grows (which it will for journey analytics), slow components compound dramatically. A 100ms delay that's barely noticeable for a simple query becomes a major bottleneck when that component gets called dozens of times in a complex analysis. For journey analytics, speed isn't just a nice-to-have - it's life or death.
7. Build an API Layer, Not a Database UI
If all you need is a thin UI that passes SQL queries straight to a database, there are many great tools out there to choose from. For journey analytics, however, we need something that allows us to add flexible security and access controls and evolve the underlying implementation without breaking things.
Perhaps more importantly, an API layer allows us to abstract away complexity. Users (and AI assistants) shouldn't need to understand how to de-nest complex JSON structures, write window functions for sessionization and attribution, or handle the quirks of whatever database sits underneath. The API can be dead simple - "give me orders for new users for this date range" - while the implementation handles all the gnarly SQL stuff under the hood.
8. Focus on Product, Not Servers
No one wants to be paged at 2am because a server went down. Serverless architecture isn't just about cost - it's about operational simplicity. Time spent on infrastructure management is time not spent on the actual product.
Cloud-managed services (ClickHouse Cloud, Airbyte Cloud, Firebase) handle the heavy lifting - and they're available across AWS, Azure, and GCP regions, so you can co-locate with your existing warehouse. No moving data between data centers means lower costs and fewer security headaches.
9. Design for AI From the Start
Modern analytics needs to work with AI assistants, and that has real architectural implications. The underlying databases and APIs need to support conversational AI patterns - storing context, retrieving information quickly, and maintaining conversation state. If AI is going to help users set up data sources, configure metrics, and run analyses, it needs clean APIs to work with.
Every endpoint needs to be well-documented, predictable, and designed so they can be chained together. The API isn't just for human developers - it's the interface that AI agents will use to accomplish tasks on behalf of users.
10. Structure Your Codebase for AI Parallelization
I believe this is the secret to how one PM can build something this ambitious with AI coding tools: treat Claude Code agents like engineering teams.
I've spent years as a PM working with multiple engineering teams in parallel - backend, frontend, data infrastructure. Each team has its own context, priorities, and domain expertise. They coordinate through APIs and contracts. I think that mental model can translate directly to AI-assisted development.
By keeping separate codebases (e.g. Firebase Cloud Functions for the API, React on Vercel for the UI), each with its own CLAUDE.md file containing domain-specific context, I can spin up multiple Claude Code instances that work in parallel without stepping on each other. One "team" focuses on API endpoints while another builds UI components. They communicate through the same API contracts that human teams would use.
The PM skills I already have - breaking down work, defining interfaces, coordinating across teams - may turn out to be exactly what's needed to orchestrate AI agents effectively (at least that's the hope!).
The Technology Stack
With these principles in mind, here are the technologies I'm going with and why.
Airbyte Cloud for Data Synchronization
Airbyte handles all the CDC/ETL work of keeping customer warehouse data synced to the analytics layer. It's open source, which matters for trust when you're moving customer data between systems. It has pre-built connectors for BigQuery, Snowflake, and dozens of other sources. I don't have to think about retry logic, schema changes, or incremental sync strategies - Airbyte's been solving those problems for a while now.
ClickHouse Cloud for Analytical Queries
ClickHouse is purpose-built for the kind of large-scale, read-heavy analytical queries that journey analytics requires. Like Airbyte, it's open source, so you can inspect exactly how your data is being stored and queried, and it's fast. When partitioned and pre-sorted correctly, ClickHouse outperforms general-purpose warehouses like Snowflake and BigQuery by 10x in my experience. It handles billions of rows with sub-second query times, supports complex window functions, nested json structures, and aggregation types that journey analysis requires, and its ReplacingMergeTree engine is perfect for historical rewrites when identity stitching updates past events.
Firebase for the API and Configuration Layer
Firebase gives us several things we need:
- Cloud Functions for a serverless API layer - no servers to manage, automatic scaling, pay-per-use
- Realtime Database for storing the semantic layer (metrics, dimensions, segments) with sub-millisecond reads
- Firestore for AI conversation state and user notebooks - document-oriented storage that works well with conversational patterns
- Authentication for robust user auth and permissions out of the box - no need to build login flows or manage sessions ourselves
Vercel for the Frontend and AI SDK
The frontend is built with Next.js and hosted on Vercel - zero configuration, just push to Git and it deploys. Next.js is also one of the frameworks AI coding tools work best with, thanks to its popularity and extensive documentation. Keeping the UI in a separate codebase from the API means cleaner separation of concerns and enables parallel development.
Vercel also provides the AI SDK, a useful framework I want to try for building AI-powered interfaces. It handles streaming responses, tool orchestration, and multi-step reasoning out of the box - saving me from building custom infrastructure for AI conversations.
And of Course, Claude Code for Development
Claude Code is the not-so-secret weapon that makes all of this possible. Because it runs directly on my development server, I can build from anywhere - laptop, iPad, wherever. I store all my product requirements, API docs, and architecture decisions in markdown files that Claude can reference anytime. The CLAUDE.md system means each codebase gets its own context file with domain-specific knowledge baked in. You can spin up multiple instances across different codebases and they each understand their domain deeply. It's the closest thing I've found to actually having multiple engineering teams at my disposal.
What's Next
So... with these principles and technology choices in place, upcoming posts will dive deeper into how each piece works in practice:
- How Airbyte syncs data from warehouses to ClickHouse (and the gotchas I encountered along the way)
- How ClickHouse handles the complex queries that journey analytics requires
- How the semantic layer works - storing SQL snippets in Firebase and composing them at runtime
- How identity stitching actually functions under the hood
If you're interested in following along as I build this out, I'd love to have you!
Trevor Paulsen is a product leader at UKG and former PM of Adobe Customer Journey Analytics. All views expressed are his own.