DataAIChronicles

Durable models on variable compute.

Practitioner writing on data engineering, AI infrastructure, and the systems that make intelligence work at scale.

The decisions, incidents and numbers behind platforms that shipped, and help for teams that want the same experience applied to theirs.

Ved Prakash, Principal Data Engineer at GitLab and one of 17 members of Snowflake's global Data Superhero Council.

Read the ChroniclesWork with me
AI, models and data: the three layers everything here is about.

Three layers, read top to bottom

The mark is the map. AI sits on top, the durable models in the middle, and the data everything depends on at the base. Every piece belongs to one of them.

  1. AI

    Agents and the governance that keeps them affordable and safe.

  2. Models

    The durable layer: semantic definitions, open tables and catalogs.

  3. Data

    Pipelines, change data capture and the foundations under everything.

Latest from the Chronicles

All writing

  1. The quietest process in Airflow: what the triggerer did for 200 parallel Dataproc jobs

    How Airflow's triggerer and deferrable operators let a single scheduler supervise 200 concurrent Dataproc jobs without burning worker slots.

    Real-time & pipelines7 min readPublished on Medium

  2. Connecting Atlan to Snowflake's ontology: tags, lineage and semantic views across the boundary

    Keeping tags, lineage and semantic views consistent when governance lives in Atlan and the definitions live in Snowflake.

    Semantic layer & ontology10 min readPublished on Medium

  3. The semantic layer: semantic views, dbt metrics and business definitions

    Where business definitions should live, and how Snowflake semantic views and dbt metrics divide the work.

    Semantic layer & ontology7 min readPublished on Medium

  4. Data & Engineering Innovators Spotlight: Ved Prakash

    An interview with Artie on real-time CDC with Project Siphon, proactive cost governance for Snowflake Cortex, and where AI will change data engineering next.

    Real-time & pipelines10 min readPublished on Artie

  5. How a Gen AI model reaches production

    From raw data to millions of conversations. Seven stages, from data collection and pre-training through alignment, evaluation, infrastructure, deployment and monitoring.

    Foundations10 min read

Measured in production, not on slides

Results from platforms I designed and run at GitLab. The same thinking goes into every engagement.

See how we can work together
100%
of database changes captured by Project Siphon, closing a 30–40% gap
$100K+
a month in potential AI overruns prevented by Cortex governance
62%
pipeline runtime reduction, 1,456 compute hours saved a year
100 TB+
migrated from on-premise Kognitio to Snowflake on AWS

On stage

All talks

Have a platform question that will not wait for the next post?

Tell me what you are building, what it runs on and where it hurts.

Email hello@dataaichronicles.com