When Output Stops Being a Signal

It used to be easier to tell who the strongest engineers were on a team, often just based on output. They were the ones shipping the most, moving the fastest, and getting the most done.

AI is making that harder to do. It made it easier for everyone to produce output, even non-engineers, which means output got a lot noisier as a signal.

To be clear, I don’t think output is or has ever been our best signal. I’ve never loved using Jira metrics as a proxy for performance. I’ve always cared more about whether we shipped what we intended, when we intended to.

But AI makes it harder to ignore.

On one of my teams, an engineer systematically cleared a backlog of low-level tech debt in a single sprint. That was real, valuable work. But when I looked at the metrics alongside an engineer who spent the same sprint on one high-impact architectural change, the numbers looked identical. Output has never been a great signal — AI just made that harder to pretend otherwise.

You can have two engineers shipping similar amounts of work and operating at completely different levels.

One uses AI to move faster but validates decisions and simplifies the design. The work holds up. The other uses AI to generate most of the implementation. It passes tests, gets approved, and ships. A few weeks later, it’s the source of follow-up work, edge cases, and confusion. On paper, they look equally productive.

The hardest part isn’t even the obviously bad code. It’s the work that looks reasonable - clean PRs, passing tests, sensible structure - but work that’s built on shallow understanding.

I’ve seen this show up in moments of discovery where an engineer will use AI to explore a problem space and come back with a long list of possible issues and directions. It looks thorough and it sounds informed. But without deeper judgment, that list can turn into rabbit holes that don’t actually matter.

I’ve even felt this myself. I once relayed an inflated estimate to product leadership based on that kind of exploration, only to realize later that most of it wasn’t relevant. It felt so convincing in the moment. 

We’re all learning how to use AI. I use these tools too, but as a manager, I’m often in situations where I have to rely not on my own ability to use them but on my engineers’. I’m one degree removed, which puts a lot of trust in a skill we’re all still figuring out.

With AI, you can have similar output and very different depth. It’s easier to skip the deep digging that used to build real intuition, and that gap shows up later in edge cases and design decisions. You can get to something that works without fully internalizing why it works.

I’ve seen the opposite too.

Some of the most impactful engineers I work with don’t have the highest Jira velocity. They spend time building internal tools, reviewing others’ work, and keeping the team aligned to the technical direction.

Their impact shows up in everyone else’s output, not just their own.

If I only looked at tickets closed, I’d have undervalued them, which means I’d be optimizing for the wrong things.

For a long time, managers have used output as a rough proxy for capability. It was never perfect, but it mostly worked.

It works a lot less reliably now, which means the job of being a manager changes.

Now managers have to evaluate the things that are harder to see: how engineers make decisions, whether they can explain why something is the right approach, whether they improve the system around them, not just their own output.

Testing, observability, and guardrails matter more but as system support, not as a replacement for judgment.

None of this is about AI being good or bad.

It’s about what it changed. Output is easier to produce, which means it’s less useful as a signal.

The difference between engineers hasn’t gone away. It’s just harder to see at first and more expensive to ignore later.

That means more direct conversation about decisions, more investment in code review as a teaching tool, and more weight given to the engineers whose impact multiplies others rather than just their own.

When output stops being a signal, you have to get better at seeing what actually matters.