The recipe for succeeding in any given field is hardly a mystery: good ideas, hard work, discipline, imagination, perseverance and maybe a little luck. Oh, and let’s not forget failure, which Dashun Wang and his colleagues at Northwestern University call “the essential prerequisite for success” in a new paper that, among other things, is based on an analysis of 776,721 grant applications submitted to the National Institutes of Health from 1985 to 2015.

In their effort to create a mathematical model that can reliably predict the success or failure of an undertaking, the researchers also analyzed 46 years’ worth of venture capital startup investments. They also tested the model on what Wang calls their “least conventional” but nevertheless important data set — 170,350 terrorist attacks carried out between 1970 and 2017.

The takeaway? “Every winner begins as a loser,” says Wang, associate professor of management and organizations at Northwestern’s Kellogg School of Management, who conceived and led the study.

But not every failure leads to success, he adds. And what ultimately separates the winners from the losers, the research shows, certainly is not persistence. One of the more intriguing findings in the paper, published this week in Nature is that the people who eventually succeeded and the people who eventually failed tried basically the same number of times to achieve their goals.

It turns out that trying again and again only works if you learn from your previous failures. The idea is to work smart, not hard. “You have to figure out what worked and what didn’t, and then focus on what needs to be improved instead of thrashing around and changing everything,” says Wang. “The people who failed didn’t necessarily work less [than those who succeeded]. They could actually have worked more; it’s just that they made more unnecessary changes.”

As they explored “the mechanisms governing the dynamics of failure” and built their model, Wang’s team identified what they describe as previously unknown statistical signatures that separate successful groups from nonsuccessful groups and make it possible to predict ultimate outcomes.

One such key indicator (besides keeping the stuff that works and focusing on what doesn’t) is the time between consecutive failed attempts, which should decrease steadily. In other words, the faster you fail, the better your chances of success, and the more time between attempts, the more likely you are to fail again. “If someone has applied for a grant and they are three failures in,” Wang says, “if we just look at the timing between the failures, we will be able to predict whether they will eventually succeed or not.”

The massive NIH database, which Wang calls a “graveyard full of human failures,” turned out to be a researcher’s dream come true. “For every principal investigator,” explains Wang, “we know exactly when they failed, and we know how badly they failed because we know the scores of the proposal. And we also know when they eventually succeeded, after failing over and over, and got their first grant.”

For the startup domain, success was either an IPO or a high-value merger and acquisition. And for terrorism, attacks that killed at least one person were classified as successes, whereas failures were attacks that did not claim casualties. The average number of failures for those who failed at least once before success was 2.03 for NIH, 1.5 for startups and 3.90 for terrorist groups.

Working with such large-scale data, Wang and his colleagues were able to identify a critical point common to each of the hundreds of thousands of undertakings they analyzed, a fork in the road where one path leads to a progression region and one leads to a stagnation region. As the paper explains, “two individuals near the critical point may initially appear identical in their learning strategy or other characteristics yet depending on which region they inhabit, their outcomes following failures could differ dramatically.”

This diverging pattern of performance increases with each new attempt, says Wang, although in some cases it is apparent which region a person is in as early as the second attempt.

Wang points out that the existence of the tipping point cuts against the traditional explanations for failure or success, such as luck or a person’s work habits. “What we’re showing here is that even in the absence of such differences, you can still have very different outcomes,” he says. What matters is how people fail, how they respond to failure and where those failures lead.

Looking ahead, Yian Yin, the first author of the study, says next steps include refining the model to quantify other individual and organizational characteristics besides learning from past failures.

Wang’s model, tested now in three disparate domains, shows promise as a tool in other arenas, says Albert-László Barabási, director of the Center for Complex Network Research at Northeastern University and author of The Formula: The Universal Laws of Success. “There are countless works trying to understand how people and products succeed,” he says. “However, there is very little understanding of the role of failure. Wang's work fundamentally rewrites our understanding of success, showing the key role failure plays in it, finally offering a methodological and conceptual framework to put failure where it belongs within the canon of success.”