We've talked to more than 50 innovators. The value of impact measurement is coming into focus.

A Simple (but Not Easy) Mission

Cobalt Collective was built on a simple premise: when innovators in education, health, and work measure their impact, they end up with more effective and successful solutions. Historically, this simple premise has been challenging to bring to life because impact measurement, as traditionally defined, often has not met the needs of early-stage innovators.  So my first job at Cobalt was to identify barriers to impact measurement and clarify what kind of impact measurement is feasible, cost-effective, and valuable for them. Here are key takeaways from 50+  conversations.

One Size Does Not Fit All

When most people hear “impact measurement,” they think of causal impact studies, which are typically studies that compare outcomes from a group of people who used a solution to outcomes from a similar group who did not. There is no question: causal impact studies are the gold standard for proving that a product is responsible for improved outcomes. 

University researchers (full disclosure: I am a recovering academic) often operate under the assumption that the best R&D sequence is: design, build, evaluate causal impact, commercialize. This makes sense in the university context because faculty are incentivized to produce scientific evidence using rigorous methods. But there is a problem: This R&D sequence optimizes for solutions that work in a structured study, not for usability or feasibility in everyday practice. As a result, innovations with strong evidence of causal impact never see meaningful adoption. For example, several RCTs support the efficacy of well-implemented project-based learning, but because PBL is difficult to practice, it has not been widely adopted.

Alternatively, non-university-based innovators (more disclosure: I am also an entrepreneur) often build and commercialize before evaluating impact. Doing so, they optimize for usability and feasibility in varied real-world settings. The result is often a scalable solution designed to produce positive outcomes for the people it serves. But here is a problem too: there is often no credible evidence that the innovation actually works. This is, or should be, unsatisfying to anyone who counts on a product or solution to fulfill its impact commitments.

Still, a causal impact study is often not the right answer. In my conversations with innovators, many reported that causal impact studies are expensive, time-consuming, and not helpful. Some organizations have run a causal impact study only to find that school district buyers want proof that the solution will work with their students in their context, or that the buyer weighs a colleague’s recommendation much more heavily than rigorous evidence of impact.

It’s not that causal impact studies are always the wrong answer; it’s just that they are not the right answer for every organization given their stage (pre- or post-PMF, for example), strategy, and stakeholders. (For guidance on when a causal impact study is the right move, stay tuned – I’ll be writing about that soon.)

Measuring User Behavior Is a High-Leverage Impact Move

Okay, you say: What is “impact measurement” if not a causal impact study? 

I’ve learned through my conversations that with the right questions, innovators can usually explain in detail (and with conviction!) a product’s “active ingredients” – user behaviors they believe will produce the intended outcome. The challenge is that once the product has scaled to the point where innovators can no longer talk to each customer, they often no longer know whether user behavior continues to reflect its active ingredients. When this is true, impact measurement means measuring how, specifically, users are interacting with the product. 

Take, for example, a school-based behavioral health company that provides a life skills curriculum delivered via a digital platform. Their theory of the program’s active ingredients is that it creates that shared language about appropriate behavior, reinforced in classrooms, PA announcements, assemblies, posters, the carpool line, and everywhere else. This shared language in turn causes improved student outcomes. But beyond anecdotes, they have very little information about whether, how, and how often shared language is being reinforced. At the same time, account managers regularly speak with district partners, providing a natural, low-cost forum for measuring implementation with a few standard questions. 

Another example: A problem-solving platform where students solve challenging problems and describe their thought process while doing so. An AI engine provides instant feedback on their problem-solving. The product team has a theory of the active ingredients: when students speak aloud as they solve problems, this leads to greater learning than when they type their thinking process. In addition, teachers who let students make a few attempts at the problem before intervening learn more than teachers who let students interact with the platform indefinitely. Finally, specific teacher pre-teaching and debriefing behaviors maximize the product’s impact on student learning. All of these signals are available through data streams captured by the product. But they are not organized and reported to give the team visibility into implementation.

These examples share four commonalities. First, the product developers have a strong theory of what had to be true for the product to have its intended impact. Second, they have insufficient visibility into user behavior. Third, each has clear and relatively low-cost opportunities to measure, review, and act upon data on user behavior.  Finally, being able to measure implementation quality has the  potential to inform product improvements and to direct care team attention to improve product use (and customer satisfaction and renewal).

This pattern emerged across many early-stage ventures. The good news is that for most organizations, reclaiming insight into whether a product is working as designed requires a modest investment with a potentially big payoff.

Measuring User Behavior is a Short Hop to Measuring Outcomes

Insight into implementation is a stepping stone to measuring impact.  Imagine that the life-skills curriculum provider gathered, organized, and reported implementation quality across all of its schools. Because their products include in-app measurement of student outcomes in the fall and spring, they can now analyze the correlation between implementation quality and student skill development. 

Take it a step further: at the end of every school year, they can present evidence of the association between using their product and student outcomes. Although this is correlational (with the usual caveats about correlation versus causation), it addresses the question district decision-makers care about: How well does this solution work with my students? And the renewal discussion becomes rooted in evidence, not anecdote, a key differentiator in an era of tight educational budgets.

Many innovators I spoke with expressed strong interest in being able to gather, analyze, and report this kind of district-level correlational evidence. The challenge: Not every product collects its own outcome data. Being able to obtain student test scores (and other data) from districts would help innovators measure impact, but student data privacy law can seem like a scary maze–I’ll be writing about that challenge, and how Cobalt can help, soon.

Discovering Impact Measurement Opportunities

To identify impact measurement opportunities, I ask innovators what problem they are trying to solve, how their solution to the problem works, what the intended outcome is, what has to be true for the solution to produce its intended outcome, how they would know it if the solution were working as designed, whether they currently measure those aspects of user behavior tied to the active ingredients. 

Answers to these questions inform Cobalt’s Impact Measurement Assessment (IMA). This is a light-touch, no-cost advisory service. After one or two conversations with me, if the organization’s needs are in Cobalt’s wheelhouse, we write and share an interactive IMA report that includes prioritized impact measurement opportunities explicitly linked to the organization’s strategic priorities. Some organizations take that report and build on their own; others engage Cobalt for additional support to help execute the highest value impact measurement opportunities.

If you are interested, the low-friction way to test this out for yourself is to complete a 10-minute Impact Discovery Agent chat. The app takes user responses and drafts a theory of change and impact measurement opportunities. (My anecdotal, non-causal findings: most users report that interacting with the app is fun, fast, and helps advance their thinking.)  At the end of the chat, if you want to learn more, you can set up a meeting with me to discuss how we can help.