We build for a buyer that has options. Some of those options are the clean version of the 0-UI pitch you can hear in every AI discourse thread on LinkedIn right now. Voice in, answers out, agent does the rest. Lives inside the email client so nobody has to learn anything. Innovative on paper, beautifully demoed, the kind of thing that gets written up as “the end of dashboards.”

Our pilot users prefer the boring UI. Not because the AI underneath is dramatically better. Because the field reps who actually use this stuff every day do not want to learn a new conversational habit. They want the same view, the same shape, the same numbers in the same place, with someone smarter than themselves having already done the work.

That is the data point I keep coming back to.

The thesis

The 0-UI thesis goes something like this. AI keeps getting better. Interfaces start to feel like overhead. The natural endpoint is no interface at all: agents do the work, humans get notifications and approve. Chat replaces the dashboard. The screen, as a thing, was a workaround for software that could not understand what you wanted.

The argument has a real piece in it. We did build a lot of bloated software, and the AI shift is going to delete some of it. But the conclusion (kill the screen) is the wrong answer to a right question.

Input method vs trust artifact

The mistake is treating UI as one thing. UI is two things and they do different jobs.

The first is input method. How does the user tell the software what they want? Click a button, fill a form, choose from a dropdown. Chat is a great input method for fuzzy intent. Anyone who has ever written a SQL query that should have been one sentence in English knows the chat version is a real upgrade. Approve-this, run-that, find-me-customers-who, summarise-the-account. Replacing those with chat is mostly fine and sometimes much better.

The second is trust artifact. The output that proves the work happened, in a form a human can scan and act on under pressure, and the control surface for when the AI gets something wrong. The thing the field rep looks at for thirty seconds in the parking lot before walking into a customer meeting. The page the manager opens for a Monday review. The screen the auditor takes a screenshot of because something looked off. And the field the user clicks into to override a date the agent picked badly.

Chat is terrible at this. The output is variable, ephemeral, hard to scan, hard to share, hard to defend if something later goes wrong. A conversation is a fine way to ask. It is a bad way to remember what was said, prove what you saw, or compare today’s answer to last week’s.

It is also a bad way to steer. Anyone who has tried to correct a long agent response by typing knows the pain: “no, keep the first three but change the date on the fourth and drop the second one.” The agent might get it right. It might not. Either way you are doing the cognitive work of pointing at things in prose, instead of just clicking on the wrong one and fixing it. UI artifacts are not just for looking at. They are the control panel for when the AI inevitably guesses wrong.

Chat replaces the input. It does not replace the artifact.

Why this lands harder in B2B

The dividing line is not exactly B2B versus consumer. It is low-stakes/simple versus high-stakes/complex. For a low-stakes consumer query (summarise this article, what’s a good recipe for the leeks in my fridge) the input/artifact distinction is mostly invisible. You ask, you act on the answer, you move on. The cost of a wrong answer is low. You self-correct in the next message.

The same person, doing a complex consumer task, runs into the same trust-artifact problem the B2B user has. Try planning a multi-city trip with hotels and trains by chat alone, or comparing three mortgages through a conversational agent, and watch how fast you reach for a spreadsheet. The dividing line is not “I am at work.” It is “this is complex enough that I cannot hold it in my head.”

In a real B2B workflow, that line is crossed daily. The cost of a wrong answer is somebody losing a customer, missing a renewal, or sending the auditor on a six-week scavenger hunt. The user cannot afford a conversational interaction that varies day to day. They need the same view, the same shape, the same numbers in the same place. They need to know that if they screenshot it and put it in front of their boss, the next person looking at the same screen will see the same thing.

A field rep in distribution is not a power user. They have a 90-account territory and a 45-minute drive between visits. The valuable thing the software does for them is reduce decisions, not add them. “Here are the three accounts to call today and the one talking point each” beats “ask me anything about your accounts” every time. The first is a tool. The second is a homework assignment.

The best case for zero UI

The strongest case for 0-UI is search. It is stateless, one-shot, and retrieval-based. You want an answer, not a workflow. Blue links were always a workaround for the fact that the engine could not give you the answer directly. Now Gemini can. Of course the interface collapses there. The job was retrieval and retrieval is exactly what an agent does best.

At I/O 2026 Google shipped the biggest redesign of the search box in 25 years. AI Mode now serves a billion users a month with conversational, follow-up search. Information Agents run in the background and ping you when something you care about changes. And Generative UI builds a one-shot custom interface per query: a small simulation, a comparison chart, a mini-app, generated on the fly for that one question.

Notice what Google did and did not do. They did not kill the UI. They built more of it, on demand, for each query. The thing they killed was the assumption that the interface had to be static and built ahead of time.

That is the move. Even in the cleanest case for 0-UI, the answer was not zero. It was situationally appropriate UI, generated on demand. The interface stays. It just stops being a fixed artifact that someone designed last quarter.

That is not the same job as preparing for a customer visit. A search query has no state, no history, no accountability, no follow-up four weeks later when something looked off. A decision-support workflow has all four. The UI you generate for one is not the UI you generate for the other. But in neither case is it zero.

Where the discourse goes wrong

The 0-UI argument tends to start from the right observation (a lot of software is a frustrating maze of screens that should not exist) and end in the wrong place (so we should replace screens with chat).

The actual conclusion is more boring. Replace the analysis layer. Keep the interface.

A workflow with real stakes has three layers, and “what AI replaces” is the only thing that really differs between the three answers people are giving right now.

In the old version, the human was the intelligence layer. They opened three reports, compared quarters, spotted the anomaly, decided who to call. The software showed the data.

The 0-UI version replaces two layers (interface and analysis) with an agent. Just type at it.

The version that has worked for us, and that I keep seeing work elsewhere, replaces only the analysis layer. The human stops being the analyst. The AI does the pattern recognition, the ranking, the anomaly detection. The interface stays, often barely changed, because the interface was not the problem. The cognitive load of doing the analysis was the problem.

People adopt new results, not new interactions. The rep does not want a new way to work. They want to walk into the meeting better prepared. You can deliver that without making them learn anything.

When zero UI is actually right

I do not want to swing too hard. There are cases where 0 UI is the correct call.

Pure automation, no decision: invoice arrives, gets categorised, posted, ledger updated, you get a notification if something went wrong. There is no analyst here. There is no artifact to scan. The work is fully invisible by design.

Pure exploration, low stakes: “summarise this thread for me,” “what did we say about Acme last quarter,” ad-hoc questions a human would otherwise hand-build a report for. Chat is great. The artifact is the next decision, not the answer itself.

Augmented input on top of a structured UI: voice-to-fill on a form, natural-language search inside an existing list, an AI suggestion in a sidebar. Almost always wins. This is chat as input method without the artifact regression.

The case it does not work, and the case the 0-UI evangelists keep skipping, is the one in the middle. Daily professional workflows with real stakes, repeated by the same person, where consistency and scannability matter more than flexibility. Field sales, clinical decision support, ops dashboards, financial review. The places where dashboards exist for a reason.

I might be wrong

I should be honest that my position has a half-life.

The argument rests on two assumptions about today’s reality. The first is that conversational answers are still variable enough, slow enough, and hard enough to share that they fail the trust-artifact test in a way scannable dashboards do not. The second is that the users I care about (field reps, ops people, anyone with a tight budget of attention and a high cost of being wrong) are not going to retrain themselves around a new interaction model on someone else’s timeline.

Both assumptions are eroding. Models get more consistent. Tooling gets better at producing structured, comparable output instead of variable prose. A generation of users who grew up talking to assistants is entering professional roles where their boss did not. If any of those curves move faster than I expect, the trust-artifact argument shrinks. The “boring UI” that wins pilots today wins fewer of them in three years.

So I would not bet against the shift happening. I would bet against it happening as cleanly and as soon as the loudest version of the 0-UI argument predicts. The transition is going to look like augmented UIs slowly absorbing more chat affordances, not chat replacing UIs in one move. And the products that survive the middle of that transition are the ones that keep the artifact intact while the analysis layer underneath quietly turns into an agent.

If I am wrong, I am wrong on the timeline, not the direction.

The middle ground

The interesting framing, the one worth stealing from this whole discourse, is not “kill the dashboard.” It is zero manual work, not zero UI. The interface stays. The labour underneath disappears.

Google’s I/O move is a hint of where this is heading. So is the agent-emits-UI pattern that has quietly become the way interactive agents ship in production. Instead of the agent answering with text, the agent calls a tool that returns a structured component, a chart, a table, a form, that the user sees and interacts with in the same turn. The artifact is still an artifact (visible, scannable, shareable) but it gets built per moment instead of designed once and reused for years. Generative UI.

Databricks’ Genie is the same idea on the enterprise side. The user asks a natural-language question, the system generates a chart, a SQL query, a structured answer, and (the part that does the trust-artifact work) a “thinking steps” panel that shows how the question was interpreted and which tables were touched. That panel exists because explanation is the price of being trusted twice.

That is the bet I would defend. Not chat instead of UI, but agents underneath a UI that can rebuild itself when the job changes. The interface stops being a fixed monument and becomes another thing the AI does.

The boring static UI wins today. The dynamic, generated, AI-shaped UI wins tomorrow. The chatbox in the middle is a distraction.

The field reps using the boring UI have already voted, though. They will keep voting that way for a while.

Related reading

• The SaaSpocalypse Has Begun (heise.de, in German). The mainstream version of the 0-UI argument I’m pushing back on.
• Generative User Interfaces (Vercel AI SDK). Practitioner docs for the agent-emits-UI pattern.
• DynaVis (CHI 2024 Best Paper). Academic anchor: natural-language input synthesises persistent GUI widgets.
• AI/BI Genie is now Generally Available (Databricks, 2025). The “thinking steps” panel as a trust-artifact pattern.
• The Agent Doesn’t Know Your Stack. My previous post. Same underlying thesis (AI replaces the analysis layer, not the interface) applied to coding agents instead of dashboards.

That line was earned the slow way. We started with one big CLAUDE.md, watched the agent fail in five different shapes, and ended up with one layer per failure mode.

Most Claude Code setup writing starts at configuration: CLAUDE.md patterns, skills, subagents, plugins, hooks. Martin Fowler’s Context Engineering for Coding Agents, Shrivu Shankar’s How I Use Every Claude Code Feature, and Anthropic’s Skills documentation all cover that layer carefully. None of it is wrong. It is just not the part that breaks first in a data-heavy codebase.

In production, configuration only works after the agent has access to the data model. If it does not understand the shape of your data, no amount of CLAUDE.md saves you. It will write code that compiles, passes tests, and ships nonsense. The agent is not “helpful.” It is confident, fluent, and prone to hallucinating joins that do not exist.

AI/BI Genie lives in the same neighbourhood. It does not strictly refuse to work on a messy schema, but its docs and best-practice guides are very clear that it leans heavily on documented tables and columns in Unity Catalog. A star schema with good descriptions: useful answers. A junk drawer of tables with no metadata: confidently wrong answers. The coding agent is in the same spot. It is just less polite about it. It does not surface the gap. It just guesses.

This is the same conversation the semantic-layer crowd has been having for years: dbt’s semantic layer, Cube, Malloy, the broader push to make metrics and data semantics explicit. The shared bet is that humans, BI tools, and now LLMs all do better when the meaning of the data lives in one place. A coding agent is one more consumer of that semantic layer, just with the worst manners.

For context: at Plato we run a multi-tenant ML platform on Databricks, 50+ wholesale-distributor tenants, Python monorepo on top. What follows is the scaffold we have on top of that. Five layers, each earned by a failure I have watched the agent produce.

Layer 1: The data model

This is where most of the work is. It is also the layer that is invisible until it bites you.

Our model is a dimensional warehouse. Facts (orders, line items, quotes), dimensions (customer, article, time), aggregates published into StarRocks for the app to query. The names are conventional. The semantics are not.

The order-status example is the one that hurts the most. Our order_line_transaction_facts table has an order_status column. For some tenants it has a small set of clean values (BILLED, OPEN, PARTIALLY_BILLED, NULL) and the header and line statuses always match. For others, with messier ERPs, it has dozens of header/line combinations that can diverge, plus values like CANCELLED, LOST, RECEIVED, REVISED, RELEASED_FOR_BILLING. If the agent writes a “revenue last quarter” query without filtering by status, it will quietly sum cancelled and lost orders along with the real ones. Depending on the tenant, that can materially overstate the number. The query runs. The number looks plausible. The dashboard ships.

This kind of detail does not live in column names. It cannot. There is no naming convention strong enough to encode “two tenants disagree about which states count as revenue.” It has to live in prose.

So it lives in DATA_MODEL.md, alongside a .claude/rules/data-querying.md rule that codifies how the agent should approach SQL: which tables are canonical, which join keys to use, which columns are tenant-specific, what to read before writing a query. We update both whenever a new tenant onboards with a new mapping quirk (most of them do).

The rule loads on demand whenever the agent does anything data-related. Before the agent writes a query, it has already read the relevant section of DATA_MODEL.md.

Before this layer existed, the agent invented joins and unfiltered status queries. Confidently. Fluently. After this layer existed, the agent looked things up and asked questions.

The cost is not zero. Keeping DATA_MODEL.md honest is a real maintenance job (someone owns it, the way someone owns the build). When a new aggregate ships and nobody updates the doc, the agent regresses to its old habits within a week. This is not free scaffolding. It is documentation that the agent reads, which means the documentation has to be correct, which means someone has to maintain it. The team underestimated this part. Twice.

Everything else in this post sits on top of this layer working.

Layer 2: CLAUDE.md

The constitution. The only file guaranteed to load at the start of every session. Also the most over-loaded file in every repo I have seen, including ours for the first six months.

Keep it short. Ours is roughly:

• check for a relevant skill before implementing anything (with the keyword triggers spelled out)
• multi-tenant Databricks ML monorepo, conda env (conda activate platoml), make test / make lint
• catalog naming: source ERP data in customer__{tenant}__{env}, ML outputs in internal__{tenant}__{env}
• Databricks code uses import pyspark.sql.functions as F, type annotations on signatures, ruff format
• never use print(); get_logger(__name__) for ops logs, DeltaLogger for dashboard metrics
• always read docs/DATA_MODEL.md before writing SQL against ERP tables

That is most of it. The point of CLAUDE.md is to point at the other layers (six rule files do the heavy lifting).

Layer 3: Rules

CLAUDE.md is the constitution. Rules are the case law.

Project-specific guidance that loads only when the agent is doing a specific kind of work. SQL style. Spark conventions. Logging patterns. The data-querying protocol. Lives in .claude/rules/*.md and gets pulled in by the skills that need it.

Our .claude/rules/ folder has a handful of these: data-querying.md, spark-sql.md, bundle-operations.md, logging.md, code-standards.md. Each one loads when the agent does the relevant kind of work. This is where the configuration layer earns its keep: project-specific judgment, loaded on demand, not paid for on every session.

Layer 4: Skills

A skill is a small package: description, body, optional resources, metadata telling the agent when to invoke it. This is the layer the rest of the field is writing about, and they are right to. It is the most useful new piece.

We have around eight in active use. Two carry most of the weight:

• tenant-onboarding: drives a new wholesaler onboarding end to end. One invocation, 16+ SQL queries, seven config decisions, two human confirmations. Replaces a 12-step Notion runbook that took a senior engineer a full day.
• bsr-issue-resolver: takes a triaged business-rule report, queries Databricks, drafts the annotation, pushes a PR.

The orchestrators sit on top of specialists (bundle-generator, business-rules-annotation-generator, business-rules-executor) which sit on top of inspectors and primitives (job-run-inspector, databricks-notebook-runner, insight-triage-analyzer, the MCPs). Each layer of skill knows less about the world and more about its specific job. A primitive does not know it is part of an onboarding. The onboarding does not know which notebook will run.

A script encodes one path. A skill encodes a capability: when to use it, what to do when it fails, what to fall back to. At 50 tenants something is always going sideways. A script gives up at the first unhandled case. A skill gives the agent enough structure to recover.

Layer 5: Tools

The execution surface. CLIs, MCP servers, shell, the file system.

We prefer CLIs over MCP servers (the recap post has the long version). CLIs are one invocation, one stdout, done. MCP servers are tokens, schemas, and a hosted service that occasionally has a hiccup mid-flow. Default to a CLI, expose as MCP only when the convenience tax pays for itself.

Our tool surface is intentionally small: query_databricks.py, dabgen, plus standard file/git/grep, plus one or two MCP servers where the integration is worth it. Small sharp set beats large overlapping set every time.

What broke before each layer existed

The layers are cumulative. CLAUDE.md could tell the agent to be careful with data, but that did not help until there was an actual data model to read. Skills could orchestrate onboarding, but only after the publication-schema and Spark-SQL rules were pulled out of people’s heads.

What the stack does not do is reason about your business. That part is still yours.

How it actually got built

Not up-front. Incident by incident.

The migration out of a bloated CLAUDE.md happened when sessions started feeling heavy from the first turn. The data-querying.md rule was written after enough fanciful joins. The spark-sql.md schema rules were written down by the people who had been silently fixing the same three mistakes in PR review for months. The narrow tool surface came from a Saturday spent watching MCP roundtrips burn token budget on a job a CLI could have finished in half the time.

Each layer is a scar. Each scar is now scaffolding.

If your coding agent is sometimes brilliant and sometimes confidently wrong, do not start by adding another prompt. Look at the bottom of the stack. In a real B2B codebase, the agent usually does not understand the data model. Build that floor. Then the skills will start to mean something.

Related reading

The configuration layer is already well covered:

• Martin Fowler, Context Engineering for Coding Agents
• Shrivu Shankar, How I Use Every Claude Code Feature
• Dean Blank, A Mental Model for Claude Code
• Anthropic, Skills documentation
• Anthropic, Best practices for Claude Code

What stays under-discussed in Claude Code setup guides is the layer underneath: the data model the agent is supposed to operate on.

The talk

The premise: at Plato we ship 25+ ML algorithms across 50+ wholesale-distributor tenants, and we do it without hand-rolling deployments. The plumbing under that is Databricks Asset Bundles (recently rebranded to Declarative Automation Bundles, or DABs) plus a small generator we wrote called dabgen, plus a Claude Code skill that drives the onboarding end-to-end.

Title was “From Days to Minutes: How We Taught an AI to Onboard 50+ Tenants on our AI Features.” Slides are on Speaker Deck.

It went well. The food was great. The conversations after were better.

The surprise

I had built the talk assuming most of the room was already using DABs in production, and most engineers in the room were using AI coding assistants daily. The whole framing was “advanced workflow tricks.” Stuff like: how to layer overrides cleanly, where to draw the line between Jinja templates and runtime config, what a CI/CD pipeline looks like when an AI is the one writing your tenant configs.

Turns out fewer people than I thought were actually using DABs in production. Same story with the AI coding wave (Claude Code, Codex, Cursor, the whole stack). I was talking like these were table stakes. They are not, yet.

That changed how I read my own talk afterwards. For a lot of the audience, the value was not in the specific tricks. It was in seeing that this stuff is real, and that small teams are running it, and that the resulting workflow is calmer than the one they have today. Good signal for the next version of the talk. Less “here is the advanced pattern,” more “here is what a working setup actually looks like, and why the boring parts matter.”

Two questions worth writing down

Two of the questions during Q&A are still in my head, because the answers are the kind of thing I had not bothered to articulate before someone asked.

“Why skills and not just scripts?” This came up because half of what a skill ends up doing is “run a thing, parse the output, decide what to do next.” Which is what scripts do. So why the indirection?

The honest answer is that a script encodes one path. A skill encodes a capability: the description, the inputs it expects, the failure modes it knows about, the tools it composes. When something goes sideways (and at 50 tenants, something is always going sideways), a script gives up at the first unhandled case. A skill negotiates: it tries an alternate path, asks the operator for a confirmation, drops back to a fallback tool. The five-tier scaffolding around the skill is what makes that negotiation actually go somewhere instead of into a loop.

The TL;DR I gave at the meetup: scripts are great when the world is fixed. Skills earn their keep when the world is messy and you want the agent to keep going. (More on this in a follow-up.)

“MCP server or CLI tool?” This one I have a strong opinion on. I prefer CLIs.

Two reasons. First, every MCP roundtrip is tokens. Tool definitions, schemas, the wrapper boilerplate. It adds up fast on long tasks, and on a tenant onboarding the agent is spending most of its budget on glue, not on actually thinking about your problem. A python query_databricks.py "..." call is one tool invocation, one stdout, done. Specifically the Databricks SQL MCP is the one we kept reaching for, and the one a small query_databricks.py script replaces nicely. Second, CLIs degrade better. When the hosted MCP service has a hiccup mid-flow (which has happened to us in production), the agent that also knows how to invoke the local CLI finishes the job. The one that only knows the MCP gets stuck. So our preference is: build the CLI first, expose it as an MCP later if it earns the convenience tax.

I want to write up the MCP-vs-CLI argument properly, because it cuts against the default advice you’ll see, and it has cost-and-reliability evidence behind it. That’s on the queue.

The personal part

This was my first talk since before corona. Five-ish years of public-speaking rust. I had forgotten how much I miss the feeling of a real audience asking real questions, instead of typing into a void on LinkedIn.

What’s next

I am going to expand a few of the bits I had to cut from the slides into separate posts. The current shortlist:

1. Knowledge scaffolding for AI agents. The five-tier pattern (data model → CLAUDE.md → rules → skills → tools) we use to make a coding agent productive in a real production codebase. This is the most stealable idea from the talk and the one I get the most questions about.
2. Generator of generators. What dabgen actually does, and why “the same Jinja template renders the bundle template and the bundle” turned out to be the right design.
3. MCP vs CLI: a token and reliability argument. The longer version of the answer above. Why we default to CLIs, when MCP is worth the cost, and what we measured.
4. Tool-teaching beats prompt-engineering. When a hosted MCP drops mid-flow, the agent that also knows how to call your fallback Python script will finish the job. The one that just had a great prompt will not.

If you run a Databricks or AI engineering event in Europe and want a longer version of any of this, you can find me on LinkedIn.

The slides

In this blog post, I review some interesting libraries for checking the quality of the data using Pandas and Spark data frames (and similar implementations). This is not a tutorial (I was actually trying out some of the tools while I wrote) but rather a review of sorts, so expect to find some opinions along the way.

Intro - Why Data Quality?

Data quality might be one of the areas Data scientists tend to overlook the most. Why? Well, let’s face it, It is boring and most of the time it is cumbersome to perform data validation. Furthermore, sometimes you do not know if your effort is going to pay off. Luckily, some libraries can help with this laborious task and standardize the process in a Data Science team or even across an organization.

But first things first. Why I would choose to spend my time doing data quality checks, while I can spend my time writing some amazing code that trains a bleeding-edge deep convolutional logistic regression? Here are a couple of reasons:

• It is hard to ensure data constraints in the source system. Particularly true for legacy systems.

• Companies rely on data to guide business decisions (forecasting, buying decisions), and missing or incorrect data affect those decisions.

• The trend to feed ML systems with this data (these systems are often highly sensitive to input data as the deployed model relies on the assumption on the characteristics of the inputs).

• Subtle errors introduced by changes in the data can be hard to detect.

Data Quality Dimensions

The quality of the data can refer to the extension of the data (data values) or to the intension (not a typo) of the data (schema).

Extension Dimension

Extracted from Schelter et al. (2018):

• Completeness: The degree on which an entity includes data required to describe a real-world object. Presence of null values (missing values). Depends on context.

Example: Notebooks might not have the shirt_size property.

• Consistency: The degree to which a set of semantic rules are violated.
  • Valid range of values (e.g. sizes {S, M, L})
  • There might be intra-relation constraint, e.g. if the category is “shoes” then the sizes should be in the range 30-50.
  • Inter-relation constraints may involve multiple tables and columns. product_id may only contain entries from the product table.
• Accuracy: The correctness of the data and can be measured in two ways, semantic and syntactic.
  • Syntactic: Compares the representation of a value with a corresponding definition domain.
  • Semantic: Compares a value with its real world representation.

Example: blue is a syntactically valid value for the column color (even if a product is of color red). XL would neither semantically nor syntactically accurate.

Most of the data quality libraries I am going to explore focus on the extension dimension. This is particularly important when the data ingested comes from semi-structured or non-curated sources. On the intension of the data is where the richest set of checks can be done (i.e. checking the schema would only verify if a field is of a certain type but not some additional logical like that what are the valid values for a string field).

Libraries

The following are the libraries I will quickly evaluate. The idea is to display writing quality checks works and describe a bit of the workflow. I selected these libraries as are the ones I have either been using, reading about, or seeing at conferences. If there is a library that you think should make the list, please let me know in the comment section.

• Great Expectations
• Pandera
• Deequ/PyDeequ

Sample Data

I will use a sample dataset to exemplify how different libraries will check similar properties:

import pandas as pd
df = pd.DataFrame(
       [
           (1, "Thingy A", "awesome thing.", "high", 0),
           (2, "Thingy B", "available at http://thingb.com", None, 0),
           (3, None, None, "low", 5),
           (4, "Thingy D", "checkout https://thingd.ca", "low", 10),
           (5, "Thingy E", None, "high", 12),
       ],
       columns=["id", "productName", "description", "priority", "numViews"]
)

Things that I will check on this toy data:

• there are 5 rows in total.
• values of the id attribute are never Null/None and unique.
• values of the productName attribute are never null/None.
• the priority attribute can only contain “high” or “low” as value.
• numViews should not contain negative values.
• at least half of the values in description should contain a url.
• the median of numViews should be less than or equal to 10.
• The productName column contents matches the regex r'Thingy [A-Z]+'

Great Expectations

Calling Great Expectation (GE) as library is a bit of an understatement. This is a full-fledged framework for data validation, leveraging existing tools like Jupyter Notebook and integrating with several data stores for validating data originating from them as well storing the validation results.

The main concept of Great Expectations (GE) are well expectations, that as the name indicate, run assertions on expected values of a particular column.

The simplest way to use GE is to wrap the dataframe or data source with a GE DataSet and quickly check individual conditions. This is useful for exploring the data and refining the data quality check.

import great_expectations as ge
ge_df = ge.from_pandas(df)
ge_df.expect_table_row_count_to_equal(5)
ge_df.expect_column_values_to_not_be_null("id")
ge_df.expect_column_values_to_not_be_null("description")
ge_df.expect_column_values_to_be_in_set("priority", {"high", "low"})
ge_df.expect_column_values_to_be_between("numViews", 0)
print(ge_df.expect_column_median_to_be_between("numViews", 0, 10))

If run interactively in a Notebook, for each expectation we get a json representation of the expectation as well some metadata regarding the values and whether the expectation failed:

{
  "expectation_config": {
    "meta": {},
    "expectation_type": "expect_column_median_to_be_between",
    "kwargs": {
      "column": "numViews",
      "min_value": 0,
      "max_value": 10,
      "result_format": "BASIC"
    }
  },
  "success": true,
  "exception_info": {
    "raised_exception": false,
    "exception_traceback": null,
    "exception_message": null
  },
  "meta": {},
  "result": {
    "observed_value": 5.0,
    "element_count": 5,
    "missing_count": null,
    "missing_percent": null
  }
}

However this is not the optimal way to use GE. The documentation states that is better to properly configure the datasets and generate a standard directory structure. This is done through a Data Context and requires some scaffolding and generating some files using the command line:

[miguelc@machine]$ great_expectations --v3-api init

Using v3 (Batch Request) API

  ___              _     ___                  _        _   _
 / __|_ _ ___ __ _| |_  | __|_ ___ __  ___ __| |_ __ _| |_(_)___ _ _  ___
| (_ | '_/ -_) _` |  _| | _|\ \ / '_ \/ -_) _|  _/ _` |  _| / _ \ ' \(_-<
 \___|_| \___\__,_|\__| |___/_\_\ .__/\___\__|\__\__,_|\__|_\___/_||_/__/
                                |_|
             ~ Always know what to expect from your data ~

Let's configure a new Data Context.

First, Great Expectations will create a new directory:

    great_expectations
    |-- great_expectations.yml
    |-- expectations
    |-- checkpoints
    |-- notebooks
    |-- plugins
    |-- .gitignore
    |-- uncommitted
        |-- config_variables.yml
        |-- documentation
 (...)

Basically, the process goes as follows:

1. Generate the directory structure (using for example the command above)
2. Generate a new data source. You can select - This opens a Jupyter notebook where you configure the data source and store the configuration under great_expectations.yml
3. Create the expectation suite, using the built-in expectations using also Jupyter Notebooks. You store the expectations as json in the expectations' directory. A nice way to get started is to use the automated data profiler that examines that data source and generates the expectations.
4. Once you execute the notebook, the data docs are shown. Data docs show the result of the expectations and other metadata in a nice HTML format that can be useful to learn more about the data.

Once you have created the initial set of expectations you can edit them using the command great_expectations --v3-api suite edit articles.warning. You will have to choose whether you want to interact with a batch (sample) of data or not. This will also open a Notebook where you depending on your choice will be able to edit the existing expectations in slightly different ways.

Now that you have your expectations set up you can then use them to validate a new batch of data. For that, you need to learn a new additional concept called Checkpoints. A Checkpoint bundles Batches of data with corresponding Expectation Suites for validation. To create a checkpoint you need, you guessed right, another nice command line and another Jupyter Notebook.

[miguelc@machine]$ great_expectations --v3-api checkpoint new my_checkpoint

If you can execute the above command, it will open a Jupyter Notebook where you can then configure a bunch of stuff using YAML. The key idea here is that with this Checkpoint you link an expectation_suite with a particular data asset coming from a data source.

Optionally, you can run the checkpoint (the full expectation on the data source) and see the results on the already familiar data_docs interface.

As for deployment. one pattern would be to run the checkpoint as a task in some sort of workflow manager (such as Airflow or Luigi), you can also run the Checkpoints programmatically using python or straight from the terminal.

I recently found out that if you use dbt, you get GE installed by default and can be used to extend the unit tests of the SQL queries you write.

The Good

• Interactive validation and expectation testing. The instant feedback helps to refine and add checks for data.
• When an expectation fails, you get a sample of the data that does make the expectation fail. This is useful for debugging.
• It is not limited to pandas data frames, it comes with support for many data sources including SQL databases (via SQLAlchemy) and Spark dataframes.

The not so good

• Seems heavy and full of things. Getting started might not be as easy as there are many concepts to master.
• Although it might seem natural for many potential users, the coupling with Jupyter Notebook/Lab might make some uncomfortable.
• Expectations are stored as JSON instead of code.
• They received some funding recently and they are changing many of already existing (and already large) concepts and API, making the whole process of learning even more challenging.

Pandera

Pandera is “statistical data validation for pandas”. Using Pandera is simple, after installing the package you have to define a Schema object where each column has a set of checks. Columns might be optionally nullable. That is, checking for nulls is not a check per se but a quality/characteristic of a column.

import pandas as pd
import pandera as pa

df = pd.DataFrame(
       [
           (1, "Thingy A", "awesome thing.", "high", 0),
           (2, "Thingy B", "available at http://thingb.com", None, 0),
           (3, None, None, "low", 5),
           (4, "Thingy D", "checkout https://thingd.ca", "low", 10),
           (5, "Thingy E", None, "high", 12),
       ],
       columns=["id", "productName", "description", "priority", "numViews"]
)

schema = pa.DataFrameSchema({
    "id": pa.Column(int, nullable=False),
    "description": pa.Column(str, nullable=False),
    "priority": pa.Column(str, checks=pa.Check.isin(["high", "low"]), nullable=True),
    "numViews": pa.Column(int, checks=[
        pa.Check.greater_than_or_equal_to(0),
        pa.Check(lambda c: c.median() >= 0 and c.median() <= 10)
        ]
    ),
    "productName": pa.Column(str, nullable=False),

})

validated_df = schema(df)
print(validated_df)

If you run the validation an exception will be raised:

Traceback (most recent call last):
  File "<stdin>", line 26, in <module>
  File ".../lib/python3.9/site-packages/pandera/schemas.py", line 648, in __call__
    return self.validate(
  File ".../lib/python3.9/site-packages/pandera/schemas.py", line 594, in validate
    error_handler.collect_error("schema_component_check", err)
  File ".../lib/python3.9/site-packages/pandera/error_handlers.py", line 32, in collect_error
    raise schema_error from original_exc
  File ".../lib/python3.9/site-packages/pandera/schemas.py", line 586, in validate
    result = schema_component(
  File ".../lib/python3.9/site-packages/pandera/schemas.py", line 1826, in __call__
    return self.validate(
  File ".../lib/python3.9/site-packages/pandera/schema_components.py", line 214, in validate
    validate_column(check_obj, column_name)
  File ".../lib/python3.9/site-packages/pandera/schema_components.py", line 187, in validate_column
    super(Column, copy(self).set_name(column_name)).validate(
  File ".../lib/python3.9/site-packages/pandera/schemas.py", line 1720, in validate
    error_handler.collect_error(
  File ".../lib/python3.9/site-packages/pandera/error_handlers.py", line 32, in collect_error
    raise schema_error from original_exc
pandera.errors.SchemaError: non-nullable series 'description' contains null values: {2: None, 4: None}

The code would look similar to other data validation libraries (e.g. Marshmallow). Also, compared to GE the library offers the Schema abstraction, which you might or not like it.

With Pandera, if a check fails, it will raise a proper exception (you can disable this and turn it into a RuntimeWarning). Depending on how you might want to integrate the checks into the larger pipeline, this might be useful or plainly annoying. Furthermore, if you look closely, Pandera only displays one validation error as the cause of the validation error, although there is more than one column that does not comply with the specification.

Given that this is Python library is relatively easy to integrate into any existing pipeline. It can be a task in Luigi/Airflow for example or something that could be run as part of a larger task.

The Good

• Familiar API based on schema checking that makes the library easy to get started with.
• Support for hypothesis testing on the columns.
• Data profiling and recommendation of checks that could be relevant.

The not so good

• Very few checks included under the pa.Check class
• The message is not very informative if the check is done through a lambda function.
• Errors during the checking procedure will raise a run-time exception by default.
• It apparently only works with Pandas, it is not clear if it would work with any other implementation or Spark.
• I did not find a way to test for properties on the size of the dataframe or to do comparisons across different runs (i.e. the number of rows should not decrease between runs of the check).

Deequ/PyDeequ

Last but not least, let us talk about Deequ. Deequ a data checking library written in Scala targeted towards Spark/PySpark dataframes and thus aims to check large datasets making use of Spark optimization to run in a performant manner. PyDeequ, as the name implies, is a Python wrapper offering the same API for pySpark.

The idea behind deequ is to create “unit tests for data”, to do that, Deequ calculates Metrics through Analyzers, and assertions are verified based on that metric. A Check is a set of assertions to be checked. One interesting feature of (Py)Deequ is that it allows comparing metrics across different runs, allowing to perform assertions on changes on the data (e.g. an unexpected jump in the number of rows of a dataframe).

from pydeequ.checks import Check
from pydeequ.verification import VerificationSuite

check = Check(spark, CheckLevel.Warning, "Review Check")

checkResult = (
    VerificationSuite(spark)
    .onData(df)
    .addCheck(
        check
        .hasSize(lambda sz: sz == 5)  # we expect 5 rows
          .isComplete("id")  # should never be None/Null
          .isUnique("id")  # should not contain duplicates
          .isComplete("productName")  # should never be None/Null
          .isContained_in("priority", ("high", "low"))
          .isNonNegative("numViews")
          # at least half of the descriptions should contain a url
          .containsUrl("description", lambda d: d >= 0.5)
          # half of the items should have less than 10 views
          .hasQuantile("numViews", 0.5, lambda v: v <= 10)
        )
    .run()
)

checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, checkResult)
checkResult_df.show()

After calling run, PyDeequ will compute some metrics on the data. Afterwards it invokes your assertion functions (e.g., lambda sz: sz == 5 for the size check) on these metrics to see if the constraints hold on the data. The metrics calculated can be stored in a MetricRepository (e.g. S3 or disk) for future reference and to make comparison between metrics of different runs.

(Py)Deequ allows for differential calculations of metrics, that is, the metrics calculated for a dataset can be updated when the data increases without having to recalculate the metrics from the whole dataset.

Another unique feature of (Py)Deequ is anomaly detection, whereas GreatExpections allows for single thresholds, (Py)Deequ allows for a checks based on a running average and standard deviation of the metrics calculated.

Similar to Pandera, PyDeequ is easy to integrate to your existing code base as it is PySpark/Python code.

Deequ for Pandas DataFrames

You might be wondering if you can use (Py)Deequ for Pandas, and it is sadly not possible. However, almost a year ago I developed an experimental port Deequ to Pandas. I called it Hooqu. However, due to personal constraints, I haven’t been able to maintain it, but it is still functional (albeit by using a lot of Pandas hacks) and you can install it via pip.

The Good

• Use PySpark to parallelize otherwise expensive checks.
• Support for external metric repositories.
• Data profiling.
• Constraint suggestion.

The not so good

• This is not a pure Python project, rather a wrapper over a Scala/Spark library, and thus the code might not look pythonic.
• Only make sense to use it if you are already using a (py)Spark cluster.
• It is your responsibility to load the data from whenever it resides into a Spark dataframe. There are no “connectors” or “loaders” off-the-shelf.

Comparison table

Let’s finish with a table summarizing the features of the different libraries:

1. Hooqu offers a PyDeequ-like API for Pandas dataframes.
2. Using running averages and standard deviation of incremental computation.

Final Notes

So, after all this deluge of information, which library should I use?. Well, all these libraries have their strong points and the best choice will depend on your goal, which environment are you familiar with, and the sort of checks you want to perform.

For small Pandas-heavy projects, I would recommend using Pandera (or Hooqu if you are a brave soul). If your organization is larger, you like Jupyter notebooks, and you do not mind the learning curve, I would recommend GreatExpectations as it has currently a lot of traction. If you write your pipelines mostly in (Py)Spark and you care about performance I would go for (Py)Deequ. Both are Apache projects, are easy to integrate with your codebase, and will make better use of your Spark cluster.

TL;DR

In this article I use mypy to document and add static type checking to an existing codebase and I describe the reasons why I believe using mypy can help in the refactoring and documentation of legacy code while following the The Boy Scout Rule.

Intro

We all love Python, it is a multi-paradigm dynamic programming language very popular in Data Science and Machine Learning. Besides some small quirky things in the language, I am quite happy with how it is evolving. However, there are some areas where I thought Python could do better for improving programming productivity in specific contexts:

• While is easy to hack around scripts and get something running, managing a large complex codebase becomes an issue. You can get something working really fast, but maintaining it can become an issue if your code base becomes large enough.

• Many times while reading other people’s code (heck, even my own code), and even when documented, it is really hard to figure out what a method or function is doing without a clear knowledge of the types you are working with. In many cases having just the type information (i.e. via a simple comment) would make understanding the code a whole lot faster.

I have also spent a lot of time debugging just because the wrong type was passed to a function/method (e.g. the wrong variable was passed to a method, wrong argument order, etc.). Because of Python’s dynamic typing the interpreter and/or linter could not warn me. Plus, some of those errors only were evident at execution time, generally in edge cases.

Although we all like working on greenfield projects, in the real world you will have to work with legacy code and it will generally be ugly and full of issues. Let’s take a look at at some Python 2.7 legacy code I have to maintain:

# snnipets.py
def get_hotel_type_snippets(self, hotel_type_id, cat_set):
    snippets = self.get_snippets(hotel_type_id, "pos")
    snippets += list(it.chain.from_iterable(
        self.get_snippets(
            rel_cat,
            cat_set[rel_cat].sentiment
        )
        for rel_cat
        in cat_set[hotel_type_id].cat_def.related_cats
        if rel_cat in cat_set and cat_set[rel_cat].sentiment == "pos"
    ))
    return snippets[:self.max_snippets]

Don’t focus too much on the fact that it has no documentation and forget about the ugly comprehension inside.

In order to understand this code I have to answer the following questions:

• What type is hotel_type_id (is it an int?)
• What type is cat_set, it looks like a dictionary containing something else.

These two issues could be fixed with a proper docstring, however comments sometimes don’t contain all the information required, don’t include the type of the parameters being passed or can be easily inconsistent as the code might have been changed but the comment not updated.

If I want to understand the code I will have to look for its usage, maybe grepping through the code for something called related_cats or sentiment. If you have a large codebase, you might even find many classes implementing the same method name.

I have two choices when I need to modify existing code like this. I can either hack my way around, modifying it enough to make it do what I want, or I can look for a way to make this code better (i.e. the The Boy Scout Rule). Besides adding the needed documentation, it would be cool to have a way to specify the types that could be potentially used by a static linter.

Enter mypy

Luckily I was not the only one with this problem (or desire), and that’s one of the reasons PEP-484 came to life. The goal is to provide Python with optional type annotations that allow an offline static linter to check for type issues. However I believe making the code easier to understand (via type documentation) is an awesome side-product.

There is an implementation of this PEP called mypy that is in fact the inspiration for the first. Mypy provides a static type checker that works in Python 3 (using type annotations) and Python 2.7 (using specific crafted comments).

At TrustYou we have a lot of Python 2.7 legacy code that suffers many of the issues mentioned above, so I decided to give it a try in a new project I was working on and I have to say it helped catch some issues early in the development stage. I also tried in it in an existing code base that because of its structure was hard to read.

Let’s go back to the example code I shared before and let’s document the code using type annotations:

from typing import Any, List, Dict
from metaprecomp.tops_flops_bake.category import CategorySet

def get_hotel_type_snippets(self, hotel_type_id, cat_set):
    # type: (str, CategorySet) -> List[Dict[str, Any]]

    snippets = self.get_snippets(hotel_type_id, "pos")
    # (...) as before

As you might guess, (str, Category) are the types of the method parameters. What follows -> is the return type, in this example, a list of dictionaries from str to Any. Any is a catch all-type. It helps when you don’t know they type (in this case, i would have had to read the code further, and I was too lazy) or when the function can return literally any type.

Some notes from the code above:

• You might have noticed the from typing import Any, ..., the typing library brings the required types into Python 2.7, even when used only as comments. So yeah, you will need to add it to your requirements.txt.
• You also noticed I had to import explicitly CategorySet from the category model (even if I used it as a comment). I find that good as I am stating there’s a relationship or dependency between those modules.
• Finally, you also noticed the # noqa: F401. This is to avoid flake8 or pylint to complain about unused imports. This is not nice, but it is minor annoyance.

Installing and running mypy

So far we have used mypy syntax (actually PEP 484 - Type Hints) to do some annotation, but all this hassle should bring something to the table besides a nifty documentation. So let’s install mypy and try the command line.

Running mypy requires a Python 3 environment so if your main Python environment is 2.7 you will need to install it in a separate one. Luckly you can call the binary directly (even when your Py27 environment is activated). I you use Anaconda you can easily create a dedicated environment for mypy:

[miguelc@machine]$ conda create -n mypy python=3.6
(...)
[miguelc@machine]$ source activate mypy
(mypy)[miguelc@machine]$ pip install mypy  # to get the latest mypy
(mypy)[miguelc@machine]$ ln -s `which mypy` $HOME/bin/mypy   # I have $HOME/bin in my $PATH
(mypy)[miguelc@machine]$ source deactivate
[miguelc@machine]$ mypy --help    # this should work

With that out of the way, we can start using mypy executable for checking our source code. I run mypy the following way:

[miguelc@machine]$ mypy --py2 --ignore-missing-imports  --check-untyped-defs  [directory or files]

• --py2: indicates that the code to check is a Python 2 codebase.
• --ignore-missing-imports tells mypy to ignore error messages when imports cannot be resolved, e.g. when they don’t exist on the env mypy is running.
• --check-untyped-defs: checks functions but does not fail if the arguments are not typed.

The command line tool provides a lot of options and the documentation is very good. An interesting feature is that it allows you to generate reports that can be displayed using CI tools like Jenkins.

Checking for type errors

Let’s take a look at another method I annoated for the purpose of exemplifying the type of errors you can find using mypy after adding type annotations:

from typing import Any, List, Dict, FrozenSet  # noqa: F401

def get_snippets(
        self, category_id, sentiment,
        pos_contradictory_subcat_ids=frozenset(),
        neg_contradictory_subcat_ids=frozenset()):
        # type: (str, str, FrozenSet[str],  FrozenSet[str]) -> List[Dict[str, str]]

        # (...) not relevant code...

Indeed, another method with no documentation whatsoever. So I had to read a little bit of the code to figure out what are the input and return types. Now let’s imagine that somewhere in the code something like this happens:

# bake_reduce.py
cat = 13
# (...)
snippets_generator = SnippetsGenerator(
    snippets_by_cat_sent,
    self.metacategory_bundle[lang]
)
snippets_generator.get_snippets(cat, "pos")

If I run mypy I would get the following error:

[miguelc@machine]$ mypy --ignore-missing-imports  --check-untyped-defs  --py2  metaprecomp/tops_flops_bake/bake_reduce.py
metaprecomp/tops_flops_bake/bake_reduce.py:238: error: Argument 1 to "get_snippets" of "SnippetsGenerator" has incompatible type "int"; expected "str"

If you come from the static typed language world this should look really normal to you, but for Python developers finding an error like this (in particular in large code bases) requires to spend quite a bit of time debugging (and sometimes the use of Voodoo magic).

When to use mypy

Optional type annotations are that, optional. You can start hacking as normal using the speed that Python dynamic typing gives you and once your code is stable enough you can gradually add type annotations to help avoid bugs and to document the code. The mypy FAQ contains some scenarios in which a project will benefit from using static type annotations:

• Your project is large or complex.
• Your codebase must be maintained for a long time.
• Multiple developers are working on the same code.
• Running tests takes a lot of time or work (type checking may help you find errors early in development, reducing the number of testing iterations).
• Some project members (devs or management) don’t like dynamic typing, but others prefer dynamic typing and Python syntax. Mypy could be a solution that everybody finds easy to accept.
• You want to future-proof your project even if currently none of the above really apply.

In the particular case of my team, a lot of the code we write ends up running for quite a long time inside of MapReduce (Hadoop) jobs, so being able to detect bugs ahead of time would save precious developer time and make everyone happier.

Adding support to Emacs

By now you might be thinking that it would be cool to integrate mypy checks into your editor. Some, like PyCharm, already support this. For Emacs you can integrate mypy into Flycheck via flycheck-mypy. You can install it via M-x package-install flycheck-mypy. Configuring it is a matter of setting a couple of variables:

(set-variable 'flycheck-python-mypy-executable "/Users/miguel/anaconda2/envs/py35/mypy/mypy")
(set-variable 'flycheck-python-mypy-args '("--py2"  "--ignore-missing-imports" "--check-untyped-defs"))

Mypy recommends disabling all other linters/checkers like flake8 and others when using it, however I wanted to keep both at the same time (call me paranoid). In Emacs, you can accomplish this with the following configuration:

(flycheck-add-next-checker 'python-flake8 'python-mypy)

Final words and references

Using mypy won’t magically find errors in your code, it will be as good as the type annotations you add and the way you structure the code. Also, it is not a replacement for proper documentation. Sometimes there are methods/functions that become easier to read just by adding type annotations, but documenting key parts of the code is vital for ensuring code maintainability and extensibility.

I did not mention all the features of mypy so please check official documentation to learn more.

There are a couple of talks that can serve as a nice introduction to the topic:

• Introducing Type Annotations for Python - by Guido, Greg Price and David Fisher
• Static Types for Python PyCon 2017 - by Jukka Lehtosalo and David Fisher

The first one of them is given by Guido, who’s pushing the project a lot. Thus, I expect mypy to become more popular in the following years. Happy hacking.