Chapter 9: Measuring What Matters (and Ignoring What Doesn’t)
There are known knowns; there are things we know we know. We also know there are known unknowns. But there are also unknown unknowns — the ones we don’t know we don’t know.
Department of Defense press briefing, February 12, 2002
Abi Noda, founder of DX, a developer experience research firm, describes the moment that most CTOs recognise. "I personally experienced this challenge in my first job as a CTO. Our executive team had started meeting monthly to review each department’s progress. Our CEO wanted more than just the anecdotal release updates I had been providing, and asked me to come up with a hard metric to report. The problem: I couldn’t think of a good metric to use."[1]
The CTO who measures nothing is flying blind. The CTO who measures everything is drowning in data. Both end up in the same place: unable to answer the question the CEO is actually asking, which is not "how fast is engineering moving?" but "is the investment in engineering producing results?"
This chapter provides the measurement system that answers that question — lightweight enough for a five-person team, rigorous enough for a Series B board, and structured to avoid the traps that turn measurement from a tool into a distraction. The reader who has built (Chapters 4–5), shipped (Chapter 6), and defended (Chapter 7) now needs to know if any of it is working. The measurement language developed here is also the raw material for Chapter 10’s board communication framework: the CTO who can measure well can translate those measurements into the business language that boards evaluate.
What DORA Actually Measures
The DORA metrics — deployment frequency, lead time for changes, change failure rate, and time to restore service — are the closest thing the software industry has to a validated measurement standard. Developed by Nicole Forsgren, Jez Humble, and Gene Kim through research spanning more than a decade and tens of thousands of professionals, the metrics have demonstrated a consistent finding: teams that deploy more frequently also have lower failure rates and faster recovery times. Speed and stability are complements, not trade-offs.[2]
The metrics are useful. They are also widely misunderstood, frequently misapplied, and occasionally dangerous. The CTO who adopts DORA without understanding its limitations will make three common mistakes.
The first mistake is treating DORA as a productivity measure. It is not. Forsgren herself has stated: "One of DORA’s limitations is its scope. We focused on the software delivery process — mostly from code commit to release. There’s a lot of important work and systems behavior that happens before and after that which are not captured."[3] Noda’s formulation is sharper: "Conflating productivity and software delivery performance is like conflating cholesterol and blood sugar — while both are important factors, they are fundamentally distinct things."[4] DORA tells you how efficiently the team delivers code to production. It does not tell you whether that code produces value for customers.
The second mistake is treating the performance clusters as fixed benchmarks. The elite/high/medium/low designations that practitioners cite — deploy on demand, lead time under one hour, change failure rate below 5%, recovery time under one hour — are not targets. They are clusters that emerge from statistical analysis of survey responses, and they change every year. Rachel Stephens at RedMonk observes: "These values are not pre-defined benchmarks. The performance levels that a team needs to achieve to be placed in a specific low-medium-high-elite group can and do change year-to-year based on survey responses."[5] In 2025, DORA abandoned the four-tier system entirely, replacing it with seven archetypes based on eight measures.[6] The CTO who tells the team "we need to reach elite by Q4" is chasing a moving target that was never designed to be chased.
The third mistake is using DORA metrics as individual performance measures or as a tool for comparing teams. Bryan Finster, a DevOps practitioner who has written extensively about DORA misuse, warns: "Having good DORA metrics does not mean you are a high-performing organization. Delivering very stable, very small batches of useless crap doesn’t make you high performing."[7] DORA’s own documentation warns against Goodhart’s Law — the principle that a measure that becomes a target ceases to be a good measure. Publishing DORA metrics without context, Finster argues, "is usually a destructive act."[7]
Where DORA earns its keep at startup scale is as a diagnostic tool rather than a scorecard. Charity Majors, CTO of Honeycomb, provides the most useful framing: "That’s the great gift of the DORA metrics. If you start focusing on one of them, you will lift your org out of poverty, and the others will get better too."[8] If the team’s deployment frequency is weekly, improving it to daily will likely improve lead time, recovery time, and failure rate simultaneously — because the underlying practices that enable frequent deployment (small batches, automated testing, continuous integration) are the same practices that improve all four metrics. The CTO does not need to optimise each metric independently. They need to identify the one that is most constrained and address that constraint.
The 2024 DORA report introduced a finding that connects directly to Chapter 12’s AI argument: as AI adoption increased across the surveyed organisations, delivery throughput decreased by an estimated 1.5% and delivery stability decreased by 7.2%.[9] The 2025 report refined this: AI shows positive impact on individual throughput but continues to have a negative relationship with delivery stability.[6] The implication for the CTO: AI tools may be making individual engineers faster while making the system less reliable. DORA metrics are the instrument that reveals this gap — if the CTO is tracking them.
|
AUTHOR: How CorralData uses (or does not use) DORA metrics — are you tracking deployment frequency, lead time, or failure rate? If not, what do you measure instead? If yes, what has the measurement revealed? The reader needs to see a working CTO’s relationship with these metrics, not just the theory. |
Beyond Delivery: Measuring What Engineers Experience
DORA measures the output of the delivery process. It does not measure the experience of the people producing that output — and the experience turns out to be a leading indicator of the output.
The SPACE framework, published by Forsgren, Margaret-Anne Storey, and colleagues in ACM Queue in 2021, provides five dimensions of developer productivity: Satisfaction and well-being, Performance, Activity, Communication and collaboration, and Efficiency and flow.[10] The framework’s core contribution is a constraint: any measurement system must capture at least three of the five dimensions, combining perceptual data (surveys) with workflow data (system metrics). A system that measures only activity — commits, pull requests, lines changed — misses whether those activities are producing satisfaction, enabling collaboration, or being performed efficiently.
Laura Tacho, CTO of DX, clarifies a common misconception: "’`SPACE metrics’ simply don’t exist. SPACE is a framework that does not come with a punch list of things to measure. Instead, SPACE provides guardrails and mental models when crafting your organization’s definition of productivity.`"[11] Forsgren herself positions the relationship between the two frameworks: "DORA is an instance of SPACE, as a measure of the performance of software."[3]
For the startup CTO, the practical question is not whether to implement SPACE — the full framework requires a survey programme and tooling infrastructure that a five-person team does not need. The practical question is which of the five dimensions to track, and how.
Lena Reinhard, a VP of Engineering who has led teams at CircleCI and Travis CI, provides the most useful guidance for scaling measurement to team size. At one to ten engineers, she writes, "Everyone works more or less on everything and the team size is small enough so that everyone still has visibility into what’s going on. Use a rudimentary project management tool." At this stage, satisfaction is the most important dimension and the easiest to measure: a monthly one-on-one conversation that asks whether the engineer is productive, frustrated, or blocked. "At this stage, anything more would probably add too much overhead and slow teams down."[12]
At the 25-to-50 person threshold, Reinhard identifies the critical transition: "Your organization is growing. Visibility across and between teams naturally decreases, while complexity increases. This means that your delivery process should become more robust."[12] This is where efficiency and flow become critical — are engineers spending their time on the product or on tooling, review cycles, and context-switching? It is also where the temptation to over-instrument is strongest. Reinhard’s warning applies directly: "Resist the temptation of making too much information mandatory and standardizing too much."[12] The goal is to see the patterns, not to surveillance the process.
Her framing of the cultural stakes is the most important insight in the measurement literature: "Metrics create incentives, that create behaviors, that create culture."[12] The CTO who measures lines of code is creating a culture that values volume. The CTO who measures cycle time is creating a culture that values flow. The CTO who measures developer satisfaction is creating a culture that values the people who do the work. The choice of metric is not a technical decision. It is a cultural one.
Noda and his collaborators distilled SPACE further in 2023 with the DevEx framework, which identifies three dimensions that most directly affect developer productivity: feedback loops (how quickly developers get information about their work), cognitive load (how much mental effort is required to complete tasks), and flow state (how often developers can sustain uninterrupted focus).[13] In 2024, Noda published the DX Core 4, which combines one system metric (pull requests per engineer) with one perceptual metric (a Developer Experience Index derived from surveys), one quality metric (change failure rate, borrowed from DORA), and one business-alignment metric (percentage of engineering time spent on new capabilities versus maintenance).[1]
The DX Core 4’s business-alignment metric — what percentage of engineering time goes to new capabilities — connects directly to Chapter 5’s debt analysis. The Stripe data showed that 33% of engineering time goes to technical debt service. The DX Core 4 makes that ratio measurable and visible. The CTO who can say "62% of our engineering capacity is going to new capabilities, up from 48% last quarter after the authentication migration" is speaking in outcomes. The CTO who says "we completed 47 story points" is speaking in noise.
|
AUTHOR: Which of these dimensions CorralData tracks — formally or informally. Does the team do developer experience surveys? Is there a measure of how much time goes to new features versus maintenance? The healthcare B2B context may introduce specific dimensions: compliance work as a percentage of engineering time, for instance. |
The Vanity Metrics Trap
Story points are the most widely used engineering metric in the startup world. They are also, by the admission of the person who invented them, a mistake.
Ron Jeffries, one of the three founders of Extreme Programming, introduced story points in the late 1990s as a planning tool — a way for teams to estimate the relative complexity of upcoming work. He has since disavowed their use as a productivity measure: "Well, if I did invent story points, I’m probably a little sorry now, but not very sorry. I do think that they are frequently misused and that we can avoid many pitfalls by not using story estimates at all."[14] Chet Hendrickson, who co-created the concept with Jeffries, is blunter: "It was the best we could think of in 1996, by 2001 we had discovered better ideas, and by now it should have been kicked to the curb long ago."[14]
The problem with story points is not that they are useless for planning. Within a single team, over time, they provide a rough sense of how much work the team can absorb in a sprint. The problem is that they inevitably get used for something they were never designed to do: cross-team comparison, individual performance evaluation, and board-level reporting. Martin Fowler identified this dynamic in 2004: "The dangerous aspect comes from once you have a standard unit for measurement across teams, someone is inevitably going to use it to compare the performance of teams. This will cause teams to distort their measurements."[15] Fowler’s assessment of velocity as a productivity tool is equally direct: "Velocity is a tool for calibrating estimations. It is not a measure of productivity. Using velocity as a productivity measure kills agility."[16]
Lines of code is the other perennial vanity metric, and it has been discredited for even longer. Edsger Dijkstra, writing in 1988, provided the definitive takedown: "`If we wish to count lines of code, we should not regard them as ‘lines produced’ but as ‘lines spent’: the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger.`"[17] The GitClear data from Chapters 5 and 11 demonstrates the consequence of measuring code volume: AI tools produce more lines while refactoring less, creating codebases that are larger, more duplicated, and harder to maintain.
The McKinsey developer productivity controversy of 2023 provides the cautionary tale for the CTO who is tempted — or pressured — to adopt simplistic measurement. McKinsey published an article arguing that developer productivity could be measured through a combination of system metrics and individual contribution analysis. The engineering community’s response was swift and damning. Kent Beck and Gergely Orosz published a joint rebuttal arguing that McKinsey’s framework measured only effort and output, not outcomes and impact, and would "do far more harm than good to organizations — and to the engineering culture at companies."[18] Dan North, creator of Behaviour-Driven Development, pointed out the irony: "Software development is one of the most ruthlessly scrutinized and measured of all business activities. The one thing you cannot call this era is under-measured. Mis-measured, for sure, but never under-measured."[19]
North’s most important insight is about where value actually resides: "Your highest-value developers are 10x by enabling other developers. Often the least useful thing they can be doing is producing code themselves."[19] Any metric that measures individual code output — story points, commits, pull requests, lines of code — systematically undervalues the engineers who make everyone else more productive. The CTO who reports velocity to the board is not just using the wrong metric. They are creating incentives that reward the wrong behaviour.
David Subar frames the stakes in career terms: "Velocity is important to Engineering and partly to Product, but is completely the wrong thing to discuss with CEOs. Communicating results using the wrong metrics can lead CEOs to believe that their CTO is poorly focused, feckless, or worse, an impediment to the organization."[20] The average CTO tenure of two and a half to three years, Subar argues, is directly connected to this communication failure.
The alternative to vanity metrics is not more sophisticated vanity metrics. It is metrics that connect engineering activity to business outcomes — which is the subject of the next section.
|
AUTHOR: Has CorralData ever used story points or velocity as a metric? If so, what happened? If not, what replaced them? A concrete example of replacing a vanity metric with one that connects to business outcomes would be the most useful passage in this section. |
Goals That Survive a Pivot
OKRs — Objectives and Key Results — are the dominant goal-setting framework in the startup world. They are also, for most startups, a waste of time.
Marty Cagan, the most influential voice in product management, stopped recommending OKRs to most companies: "In the majority of companies I meet, I have stopped recommending the practice. That’s because, in so many companies, even though conceptually the technique is simple and sound, it ends up proving a waste of time and effort, and yields little if any results."[21] His diagnosis: OKRs work at companies like Google because those companies already had empowered product teams — teams that were trusted to solve problems, not just deliver features. At companies where engineering teams are treated as delivery functions, OKRs become a bureaucratic layer on top of a feature factory.
Angelo Saraceno, co-founder of Railway, describes the firsthand failure: "We tried OKRs, they created more ceremony than clarity. We implemented them faithfully, we all read the John Doerr book. And it worked — somewhat." The problem: "At Railway, we would spend a significant amount of time discussing what is a valid OKR, only for us to realize we were two days into the planning process and we’ve yet to decide what we should build."[22]
The startup CTO faces a specific version of this problem. OKRs assume you know what you are trying to achieve. Pre-product-market-fit startups are still figuring that out. Setting quarterly objectives when the company might pivot next month is not strategic planning. It is performance art.
Dave Bailey, an executive coach who works with early-stage founders, proposes a lightweight adaptation: four-to-six-week cycles instead of quarterly, one or at most two key results per objective, and a constraint on scope. "In a startup, there are only two things that matter: your customers and your team. Therefore, every OKR across the business should serve either your customer or your team. Period."[23]
Ravi Mehta, former Chief Product Officer of Tinder and former product leader at Facebook and Tripadvisor, identifies a structural gap in the OKR framework: it answers "what?" and "how?" but not "why?" He observes that John Doerr himself, in his TED talk, says good goals should answer all three questions — but the OKR format only provides fields for two. Mehta’s alternative — Narratives, Commitments, and Tasks — adds the missing dimension. The Narrative is one to three sentences explaining what the team is doing and why it matters. The Commitments are three to five measurable goals the team commits to completely. The Tasks are the specific work items, which can change as circumstances shift. The key distinction: "If you’ve achieved all of your Commitments, but not completed any of your Tasks — that’s a win."[24]
Spotify abandoned OKRs entirely in 2014. Henrik Kniberg, who documented Spotify’s engineering culture, described the replacement: a DIBB framework (Data, Insight, Belief, Bet) with a strategic cadence of six months and a tactical cadence of six weeks.[25] The framework was less prescriptive than OKRs and better suited to an organisation that was still learning what worked. The lesson for the startup CTO: the goal-setting framework that Google uses at 100,000 employees is not the framework that a 10-person team needs. The tool should match the stage, and at the early stage, the tool should be light enough that the team spends more time building than planning.
For the CTO at the Coder stage, even Bailey’s lightweight OKRs may be too much process. The question to answer is simpler: what are we trying to learn this month, and how will we know if we learned it? At the Manager stage, Bailey’s framework or Mehta’s NCTs provide enough structure to coordinate multiple workstreams without the overhead of full OKRs. At the Director stage, OKRs may begin to earn their keep — but only if the team culture is one where engineers solve problems rather than deliver features. Cagan’s diagnostic is the test: if the engineering team is empowered to figure out how to solve the problems they have been assigned, OKRs can help align that work. If the engineering team is told what to build and asked to estimate when it will be done, OKRs are a bureaucratic veneer on a delivery function.
|
AUTHOR: CorralData’s goal-setting process — do you use OKRs, NCTs, or something informal? How do you set engineering goals when the product roadmap is evolving? The reader at your stage (scaling toward $6MM ARR) needs to see what goal-setting looks like in practice at a company that is past MVP but not yet at the scale where formal OKRs are natural. |
Leading Indicators and the Gut Check
The metrics described so far are lagging indicators. They tell the CTO what has already happened. The CTO who relies only on lagging indicators will discover problems weeks after the team has started to struggle.
Will Larson provides the most systematic framework for measurement, organising it into four modes: measure to plan (project selection and prioritisation), measure to operate (incidents, downtime, latency, costs), measure to optimise (engineering productivity and developer experience), and measure to inspire (engineering’s transformational impact on the business).[26] His most important warning is about which of these to share with the CEO: "The biggest mistake to avoid is asking your CEO to measure engineering through your Metrics to optimize. Although intuitively you can agree that being more efficient is likely to make engineering more impactful, most CEOs are not engineers and often won’t be able to take that intuitive leap."[26] The CEO does not need to see deployment frequency or PR cycle time. The CEO needs to see whether engineering is delivering on its commitments and what the risks are. Optimisation metrics are for the CTO’s internal use. Business-outcome metrics are for the board.
Nicole Forsgren articulates the principle that connects leading indicators to the vanity metrics problem: "The challenge with single metrics is that they can be very easily gamed. I prefer two or three metrics that are held in tension, making it difficult to be myopic in your focus."[27] The CTO who tracks deployment frequency alone will optimise for small, frequent deploys — even if they are deploying trivial changes. The CTO who tracks deployment frequency alongside change failure rate and customer-reported issues holds the metrics in tension: speed is only valuable if it does not degrade quality or customer experience.
Charity Majors provides the most vivid language for the leading indicator the CTO should watch most closely: deployment frequency as "the heartbeat of your company. It should be regular. It should happen consistently. It should be boring."[28] When the heartbeat becomes irregular — when deploys start spacing out, when the team begins batching changes, when someone says "let’s wait until next week to ship this" — the CTO is seeing the early symptoms of what Majors calls "a software engineering death spiral." The cycle: slow deploys make each deploy riskier, which makes the team more cautious, which makes deploys slower, which makes them riskier. The staffing cost is exponential: "If it takes X engineers to build, maintain and run your software systems with an under 15-minute delivery frequency, then it will take twice as many engineers for an hour deployment frequency. Quadruple the staff if it takes you days to ship."[29]
The leading indicators that predict trouble before the lagging metrics move include: PR review time increasing (engineers are either overloaded or disengaged), the ratio of bug-fix commits to feature commits shifting (the team is spending more time reacting than building), on-call burden growing (the system is becoming less stable), and — the one that no dashboard captures — the senior engineer who stops proposing improvements and starts just completing tickets. That last indicator is where the CTO’s gut matters more than the dashboard.
Larson names the trap that makes gut-versus-data a false choice: "I’m a believer in measuring something imperfect, but useful, versus holding out for the perfect metric while measuring nothing in the interim."[30] The CTO who trusts their gut without data cannot persuade the CEO. The CTO who trusts the dashboard without gut-checking it will miss the problems that metrics cannot see. The right practice is to use metrics to confirm or challenge what the CTO already suspects — and to investigate when the metrics and the gut disagree, because that disagreement is itself the most valuable leading indicator of all.
|
AUTHOR: A CorralData example of catching a problem early — through metrics, through gut feel, or through the disagreement between the two. The reader needs to see what "the dashboard says fine but something feels wrong" looks like in practice. |
The measurement system for a startup CTO should be lightweight enough to maintain without a dedicated tooling team, focused enough to fit on a single dashboard, and connected to the business outcomes that the CEO and board evaluate. At the Coder stage, measure three things: how often you deploy, how quickly you recover when something breaks, and whether the team is frustrated or energised. At the Manager stage, add a developer experience dimension — a quarterly survey or a regular check-in — and begin tracking the ratio of new capability work to maintenance work. At the Director stage, add cross-team delivery metrics and begin reporting engineering investment in business-outcome terms: not "we shipped 12 features" but "we reduced customer onboarding time from three weeks to four days."
The measurement vocabulary developed in this chapter is the prerequisite for Part III. The CTO who knows what to measure, who can distinguish leading from lagging indicators, who can hold metrics in tension rather than optimising a single number, and who can translate engineering measurement into business language — that CTO is ready for the conversation that Chapter 10 describes: the one that determines whether they keep their job.