Metrics and their usage are incredibly important topics for all entrepreneurial teams, and there is often an overwhelming amount to think about. Here’s my primer on topics of metrics, dashboards, and alerts.
There are a few categories of metrics, each with a different purpose.
A North Star metric typically encapsulates a lot of complexity and represents the one thing that matters most to the overall health of the mission. A North Star metric should measure impact, not input, but which impact is not always obvious without investigation.
Consider a social app. The total number of active users might seem an obvious choice, but a better North Star might be the number of social connections first-time users make in their first week because that metric is the best indicator of long-term retention and engagement. [1]
As a result, your North Star might initially be aspirational, but it should ultimately be a fundamental and discovered property of the system you’re building.
North Star metrics require iteration, but once found, they are powerful guides to higher levels of impact.
For entrepreneurial teams, there should be one and only one North Star for everyone. The North Star might change over time, iterating with each strategic phase of the team’s mission, but it should be universal within each phase. It is the drumbeat to which everyone marches.
Not every individual or team can operate directly on the North Star, but everyone needs to be operating toward it nonetheless. In these cases, Proxy North Stars can be defined, like substituting one-year retention with first-month engagement.
Proxies are actionable metrics that are believed to align with the North Star. And “believed” is a key word. Just because you choose a proxy doesn’t make it a good one; its alignment only shows up in practice. If a team nails its proxy measurement but the proxy turns out to be poor, that’s a learning for all involved, not a reflection on the team that nailed its metric.
And here’s one other word of caution. Proxies are too often abused. Take care to ensure that each proxy is authentic and not a smoke screen to include a bunch of favored ideas that don’t currently align.
Key Results track the progress of an OKR (Objective and Key Results), and they often relate to the North Star or its proxies. They provide concrete feedback for steering, refining tactics, uncovering learnings, and disrupting incorrect strategies. As such, their tracking should be put into place soon after OKR launch, and that tracking should be automated. If you can’t measure your progress, you don’t really have Key Results.
The status of Key Results should also be widely available across the team to leverage brainpower and build trust. Unfortunately, transparently sharing status can sometimes create friction when those vested in outcomes hover over dashboards, reacting to the slightest changes. Address micromanagement directly instead of reducing transparency in response. Complementary dysfunctions don’t a highly functional team make.
Key Performance Indicators (KPIs) track an amalgam of strategic metrics, including North Star candidates, former and future Key Results, and other metrics that capture critical aspects of the product, mission, operations, or entire endeavor.
KPIs are metrics the team is committed to maintain but not necessarily improve in the near term (unless also tied to OKRs). By definition, if any performance degrades in a KPI, the team disrupts other work and fixes it. For that reason, KPIs should be the minimal set that stops activity against OKRs.
It is therefore necessary to regularly prune KPIs, promoting new metrics and demoting ones no longer providing value. A good way to demote KPIs is to recategorize them as diagnostic metrics—metrics that describe your system in detail but are only viewed during a forensic deep dive, not as a regular practice. This has the benefit of keeping the metrics pipeline alive in case your demotion is made in error.
Diagnostic measurements are used to understand changes to KPIs and are not necessarily operated on directly, either as a KPI interrupt or a North Star. Low-level server metrics, for example, may feed into weekly scaling models or impact long-term performance road maps. But most often they are used to understand what’s happening if a KPI is deviating from its norm.
Aggregation is a key part of building an effective metrics system, and there are two related concepts: cadence and rollups.
Cadence refers to the frequencies at which data is sampled and then reviewed. The cadence at which metrics are sampled defines a team’s responsiveness, and the cadence at which metrics are reviewed defines its urgency.
Teams can only respond to new information as frequently as data is sampled, so if your metrics are collected monthly, you can be no more responsive than that. Similarly, if you review dashboards monthly, even if you collect data daily, you set a team pace that acts in months, not days or weeks. The cadence with which your team reviews metrics defines culture, so make sure your cadence aligns with success.
Different teams have different frequencies of change (e.g., consumer Internet versus established enterprise software), and different metrics within the same team also have different frequencies of change (e.g., server performance versus daily users versus monthly sales).
In general, metrics should be collected at the highest frequency that captures meaningful change, defined as change that leads to action. But capture frequency is not the only frequency that is meaningful or inspires action, which leads to rollups.
Rollups are mathematical aggregations of captured metrics that produce new metrics or statistics of existing metrics. Examples of new metrics include ratios and rates of change, like a first derivative (velocity) or second derivative (acceleration). Both are powerful forms of aggregation, often superseding the captured metric in operational value. Active user count is a great vanity metric, for example, but what often matter more are the velocity and acceleration of user count and the ratio of revenue to users (both of these were critical metrics on the Roblox ops team during my tenure).
Statistics are another powerful form of aggregation, particularly for performance. The 95th percentile in server response time, for example, is a better indicator of audience impact than the average because it captures the experience of all but a small percentage of outliers. And finally, time-based rollups that aggregate high-frequency metrics into longer time frames (such as daily active users rolled up into monthly and yearly actives) allow teams to analyze lower frequency changes that could otherwise be missed.
Dashboards are a codified collection of metrics organized to help people quickly digest a lot of data in order to understand the status of a system. Examples include a dashboard of all goals for the team or an individual group, a dashboard of the top five audience performance metrics that tie most to engagement and retention, and one that shows the current health of the major components of a system.
Dashboards can be visual or tabular, automated or manually updated, app or spreadsheet.
Dashboards must be continuously refined around three traits. First, they must capture the essentials of performance for a subject (if someone looks at a dashboard and assesses that the system is healthy, the system must be healthy or the dashboard corrected). Second, dashboards must lead to concrete action. However codified, a team must understand what concrete actions to take when norms deviate. Finally, dashboards must be culled to reduce signal to noise. Without these three traits, dashboards degrade in usefulness over time.
These traits also mean that dashboards should have some friction to change. An individual or group can’t add a metric they think matters without some collaboration with other viewers. Some friction is healthy, but it also restricts innovation through bureaucracy. To complement this, individuals should have access to do-it-yourself tools to experiment on a faster cadence than codified dashboards. This access can be sophisticated third-party tools, access to an analyst, or access to raw data. Just be sure to incorporate all critical system metrics into common dashboards for broad visibility and response.
Together, codified dashboards that represent the system and ad hoc tools to experiment with new insights allow teams to reap the many benefits of metrics.
Alerts are automated notifications based on metrics. Everything that applies to dashboards applies to alerts. They must accurately represent health, they must be actionable, and they must be culled.
Like dashboards, for peak effectiveness there must be a codified set of alerts and the ability for do-it-yourself alerts, the latter restricted to alerting those creating them unless promoted.
Alerts have the potential for much greater signal-to-noise problems than dashboards because of cascading failures or higher-than-needed sensitivity. There are a few ways to reduce noise. First is regular culling, just like with dashboards. Second is targeting, sending alerts only to those able to act. Third is real-time aggregation, sending a single multi-subject alert when many alerts fire at once. And finally, there is escalation, targeting those on the front lines first and then alerting to higher levels of accountability when alerts are not resolved within an agreed time frame.
For this last approach, it helps to have a response system with equal transparency to the alerting system so all involved understand if an issue is being addressed or ignored.
[1] North Stars are sometimes, but not always, associated with aha *moments—*the moment a user understands the value of your service as measured by retention, engagement, or a similar metric. For early Facebook, the aha moment was the number of friends made in ten days. Here’s one of many references: Benn Stancil, “Facebook’s ‘Aha’ Moment Was Simpler Than You Think,” Mode Analytics, January 30, 2015, https://mode.com/blog/facebook-aha-moment-simpler-than-you-think.
[2] Image: assets.science.nasa.gov/dynamicimage/assets/science/psd/solar/2023/06/North-star_star-trails_Credit_Preston-Dyches.jpg