Everyone's Using AI. Nobody's Measuring Whether It's Working.

Everyone has AI now.

The engineering team has code assistants. The support team has chatbots. The sales team has AI-powered CRM workflows. The marketing team has content generators. Every team lead, every VP, every board deck has a slide that says "AI-integrated" with a green checkmark next to it.

And almost nobody, genuinely, almost nobody, is measuring whether any of it is actually making a difference.

I don't mean measuring adoption. Teams are great at measuring adoption. "85% of our developers use AI code assistants daily." "We've deflected 40% of support tickets to our AI bot." "We've generated 200 blog posts this quarter using AI."

Those are activity metrics. They tell you that AI is being used. They tell you nothing about whether it's working. And in my experience, the gap between "we use AI" and "AI is creating measurable value" is enormous, and most organisations aren't even aware the gap exists.

The vanity metrics problem

Let me give you a specific example from engineering.

There's a category of tools now that measure how much code is written using AI assistants. Lines of code generated. Number of commits with AI involvement. Percentage of pull requests that include AI-assisted code. These numbers go into dashboards. The dashboards go into leadership reviews. Everyone nods.

But here's what nobody's asking: what was the actual impact of that code?

A developer writes a single prompt. The AI assistant generates a hundred lines in five minutes. What used to take an hour now takes five minutes. That sounds impressive. But what happened next? Did those hundred lines ship? Did they require significant review and rework? Did they introduce subtle bugs that took someone else two hours to find? Did the feature they enabled actually move a metric that matters to the business?

Nobody's measuring that. They're measuring the input, lines generated, time saved on the initial draft, and assuming the output is proportional. It usually isn't.

Lines of code were never a good productivity metric, even before AI. Adding "written by AI" doesn't make it one. The metric that matters is: are we shipping more features, of higher quality, that have a measurable impact for customers? If we were shipping X features per sprint before AI and now we're shipping X plus Y at the same time, and that Y difference is driving real customer value or revenue, that's impact. If we're shipping more stories that don't move the needle for customers, we've just accelerated irrelevance.

The support bot illusion

Here's another one I see constantly.

A company deploys an AI support bot. The immediate metric looks great: ticket volume to human agents drops by 30%. The team celebrates. Leadership calls it a success. Headcount planning adjusts.

But did anyone ask whether the customers who interacted with that bot actually got their problems solved? Not "did they stop contacting us", that could mean they solved their problem, or it could mean they gave up. Did customer satisfaction change, not through a survey that the bot asks at the end of the conversation, but through downstream behaviour? Did those customers renew? Did they expand? Did they churn at a higher rate three months later?

And there's a dimension most teams don't even think about: did the support bot generate revenue? Not directly, but if a customer enquired about a product capability and the bot understood the context, guided them to the right solution, and helped them convert from a free tier to a paid plan, that's top-line impact. If the bot just deflected the ticket and the customer figured it out themselves, or didn't, that's cost reduction at best and customer erosion at worst.

The support bot looks like a success on the dashboard because the dashboard measures activity. It measures tickets deflected, not problems solved. It measures cost savings, not customer outcomes. And the difference between those two things is where the real story lives.

Every use case needs its own measurement

This is the part that makes AI impact measurement genuinely hard, and it's why most teams don't do it properly.

You can't apply one framework across the board. The way you measure whether an AI code assistant is delivering value is fundamentally different from how you measure a support bot, which is different from how you measure an AI-powered sales workflow.

For engineering productivity, the measurement isn't "how many lines did AI write." It's: Are we shipping more meaningful features, faster, with the same or better quality? Are those features driving customer value? Has the cycle time from spec to production improved? Has the defect rate changed? The unit of measurement is customer-impacting output, not developer activity.

For customer support, the measurement isn't "how many tickets were deflected." It's: Did customer satisfaction hold or improve? Did resolution quality stay consistent? Did churn rate change for customers who interacted with the bot versus those who didn't? Did the bot create any conversion or expansion opportunities? The unit of measurement is customer outcome, not cost savings.

For sales and CRM workflows, the measurement is closer to revenue attribution. If AI is integrated across the sales pipeline to track payments, manage tickets, monitoring customer signals, can you trace a line from "the system identified this opportunity" to "the customer converted"? Can you show that the AI-augmented workflow is generating more pipeline, closing faster, or catching signals that humans were missing? The unit of measurement is dollars, top line increased or bottom line saved.

The common thread across all of these: the metric that matters is the business outcome, not the AI activity. Dollars earned. Dollars saved. Customers retained. Quality improved. Time reclaimed and reinvested in something that produces tangible results. If you can't draw a line from "we added AI here" to one of those outcomes, you're measuring the wrong thing.

Saving time is not the metric

This is where I see the most self-deception.

"AI saves us 10 hours a week." Great. What are those 10 hours being used for? If the answer is "more of the same work, marginally faster," you haven't created value; you've created capacity. Capacity is only valuable if it's directed at something that produces a measurable result.

If those 10 hours are being reinvested into shipping an additional feature that drives customer retention, that's value. If they're absorbed into meetings, context-switching, or tasks that don't move any business metric, the time savings are real, but the impact is zero.

John Doerr wrote a book called Measure What Matters. The title alone is the whole point. The trap with AI integration is that it's incredibly easy to measure what's visible, usage, adoption, speed- and ignore what matters, outcomes, impact, value creation. The visible metrics feel good. The meaningful metrics require work to define, work to track, and sometimes to deliver uncomfortable answers.

The uncomfortable question

Here's what I think most organisations are avoiding: some AI integrations, if measured honestly, would turn out to be net negative.

Not because AI doesn't work. It does, in the right context, for the right problem, with the right measurement. But "we added AI because everyone's adding AI" is not a strategy. It's a reaction. And reactions don't come with measurement frameworks attached.

A code assistant that speeds up initial code generation but increases review cycles, introduces subtle bugs, and creates a false sense of productivity, that could be net negative. A support bot that deflects tickets but silently degrades customer experience, that could be net negative. An AI-powered workflow that adds complexity, requires new tooling, and demands ongoing prompt maintenance, if the output doesn't justify all of that overhead, it's a net negative.

Most teams will never discover this because they're not measuring outcomes. They're measuring activity. And activity always looks positive: more usage, more output, more AI. The dashboard is green. The impact might be red. Nobody's checking.

Outcome-driven AI

I want to be clear: I'm not arguing against using AI. I build AI systems for a living. I believe in what this technology can do when it's applied thoughtfully.

What I'm arguing for is a different starting point.

Instead of "where can we add AI?" the question should be "what outcome are we trying to improve, and is AI the right lever?" Instead of measuring adoption, measure impact. Instead of celebrating usage, celebrate results.

This means doing the hard work upfront: defining what success looks like before you integrate AI, establishing a baseline before you change anything, and committing to measuring the outcome, not the activity, after deployment.

It also means being willing to turn things off. If you measure honestly and the AI integration isn't delivering the outcome you defined, that's not a failure of AI. It's valuable information. Maybe the use case wasn't right. Maybe the implementation needs work. Maybe the workflow needed a different kind of improvement entirely. But you'll only know if you measure what matters.

The organisations that will get the most value from AI over the next five years aren't the ones that integrate it the fastest. They're the ones who measure it the most honestly.

Where to start

If you're reading this and recognising your own organisation, here's how I'd approach it.

Pick one workflow. Not your whole AI portfolio, one integration. The one your team is most proud of. The one with the green checkmark on the board deck.

Define the outcome you expected. Not "we'll be more efficient." Specific: "We expected to reduce customer resolution time by 20% without degrading satisfaction scores." Or: "We expected to ship two additional features per sprint." Write it down. Be precise.

Establish the baseline. What were the numbers before AI was integrated? If you don't have this, you're already in trouble, but you can start measuring now and compare forward.

Measure the outcome, not the activity. Ignore adoption rates, usage numbers, and lines generated. Look at the business metric you defined. Did it move? In which direction? By how much?

Be honest about what you find. If the numbers are good, double down. If they're not, investigate why. Maybe the integration needs tuning. Maybe the workflow wasn't the right fit. Maybe, and this is the hard one, the AI isn't adding the value you assumed it was.

One workflow, measured honestly, will teach you more about AI's real impact on your organisation than a hundred adoption dashboards.