RiskNow: a lakehouse under a portfolio-monitoring product

Context

Portfolio monitoring is a data product with an operations problem attached: the analysis is only as good as the pipeline behind it, and the pipeline is only as good as the deployment and support process around it.

Problem

Product, data architecture and operations were being decided separately, which is how a release becomes an incident: a schema change lands, the ELT degrades quietly, and support discovers it before the roadmap does.

Approach

I led the portfolio-monitoring product and every deployment and maintenance operation around it. That meant designing the Data Lake, Warehouse and Lakehouse architecture that fed the product, the rollout models by which a release reached customers, and the continuous improvement of the ELT and support processes as one loop rather than three backlogs.

Non-obvious decisions

• Separate lake, warehouse and lakehouse by how the data is read, not by how it arrives. The ingest shape is the least stable thing in the system and the worst organising principle available.
• Design rollout as a model, not as a date. Releasing in stages per customer segment, with a rollback path, is what lets a product change without producing an incident per release.
• Feed support and deployment failures straight into the roadmap. Their volume is the most honest product signal available and it costs nothing to collect.

Outcome

The portfolio-monitoring product ran on an architecture designed for how it was read, released through staged rollout models with a rollback path, and improved continuously from the failure modes its own ELT and support surfaced.