During the festival, around a thousand people played the game, generating more than 75,000 annotated cells in tumour tissue. According to Horlings, the lack of annotated data is one of the main bottlenecks for AI in pathology, and a festival turned out to be a good place to tackle it: thousands of motivated participants from all walks of life, in a short time, in a playful mood. Players turned out to be strikingly precise and competitive, with some reaching up to 100 percent accuracy, and regularly asked sharp questions about immunotherapy and pathology.

The biggest challenge was making the game both fun and scientifically rigorous at the same time. Annotations are normally done by highly trained specialists, so the researchers had to find out whether non-experts could reach a level useful for training AI models. That required designing a game that trains players well, clever ways to combine answers from many players, and ground truth based on pathologists’ own annotations. In a recent peer-reviewed paper, Horlings and colleagues showed that a model trained on players’ data performs as well as one trained on pathologists’ annotations.

According to Horlings, this approach also opens up a scale of data collection that would be impossible among the Netherlands’ roughly 400 pathologists, while helping people outside science understand why the research matters.