A patient who has trouble reading spends a whole day getting a referral arranged through ZorgDomein. Another patient, who has their first hospital appointment today, is immediately asked in the consultation room: why didn’t you register digitally? With a few examples, moderator Marit Blom of Sigra makes clear right away where digital healthcare can cause problems. What is convenient for an organization can be a considerable hurdle for a patient.

That is why digital inclusion is today’s theme. Who benefits from new technology, and who risks being left behind? And who may be missing entirely from the data that AI applications use? This edition of Zorg2040 takes place during the scientific conference Governing Algorithms for Care, at Felix Meritis in Amsterdam. Zorg2040 has existed since 2015 and is organized by Sigra, Rabobank, Amsterdam AI, Amsterdam Economic Board and ROM InWest.

The question of who benefits from new technology already arises with the data used to develop AI applications, as Julia van Weert shows. She is Professor of Health Communication and co-founder of the ELSA lab AI for Health Equity at the University of Amsterdam. If certain groups are insufficiently represented in that data, a system may work less well for them.

Think of facial recognition that is less accurate at recognizing dark-skinned women’s faces. Or of detecting heart problems in women, whose symptoms can present differently than in men. Who is visible in the data? For which groups does an application work better or worse? According to Van Weert, those questions need to be on the table from the start.

Offering everyone the same technology therefore does not mean everyone gets the same opportunities. One person needs more support than another. The ELSA lab focuses not on health equality but on health equity, says Van Weert. “You want everyone to get the support they need to reach their full potential for health and well-being.” To that end, the lab investigates the ethical, legal and societal aspects of AI applications. Who, for example, is ultimately responsible when such an application is used? And what should people be told when AI plays a role in the care they receive?

In the debate on AI in healthcare, it is important to distinguish between different AI applications, Van Weert stresses. Asking a chatbot a medical question is something different from a doctor using a decision support system. As an example, she mentions a system that gives a GP insight into a patient’s fall risk with certain medication use. Here there is a human in the loop: the doctor assesses the outcome and takes it into account in the treatment. And for now, that is the better variant.

What ends up in a prompt?

Maarten Dirksen of Basic Orange looks at AI from the perspective of the organizations that work with it. Basic Orange develops digital applications, including for many healthcare organizations. Dirksen sees how employees are already getting started with AI while the organization itself still lacks policies and agreements. As a result, sensitive data can unknowingly end up with an external party, with possible privacy law violations as a consequence. “We use AI as if it were an internal tool, while legally it is an external party.”

His company therefore developed GDPR Filter. In a demonstration, Dirksen shows how the filter replaces personal data with labels before a question is sent to an AI model. In the answer, those labels are converted back into the original data. This is how he wants to help organizations use the possibilities of AI while protecting sensitive information. Dirksen: “The question is not whether your employees will use AI, they already do. The question is how to set this up safely.”

And that other twenty percent?

Nikki van de Veerdonk, project leader digital innovations & AI at Pharos, recognizes in the use of AI the pattern of other forms of digitalization in healthcare. Developers build an application and test it with people who are similar to themselves. Then it turns out that part of the users cannot cope with it.

Adjustments are then needed, while the budget has already run out or developers would rather spend it on extra functionality. “An app that reaches eighty percent of the target group and gets an eight from its users may seem successful. But who are the other twenty percent? Precisely the people who already have less easy access to care risk being missed again. It’s great when AI makes an understandable summary of a conversation. But if you then have to download it in the practice’s app, that is far too complicated for some people.”

Pharos, the national center of expertise working to reduce health disparities, helps make digital applications more understandable and accessible. Sometimes a small adjustment is enough: showing past and upcoming appointments separately in the patient portal, instead of in one long list. “But of course I would rather not have to make AI applications inclusive retroactively. I would prefer this to play a role from the start of development. Now there is a risk that these applications cause health disparities to increase exponentially.”

Plenty of questions

The three speakers take their seats on stage. They continue the conversation with each other, with moderator Marit Blom and the audience. Is digital exclusion mainly about access, asks one of those present, or about understanding and skills? Both, answers Van Weert. Making an app available is not enough. Some people don’t even have a smartphone, others need personal explanation. In her research, she has users say out loud what they are thinking while they try an application. It then turns out that steps that are self-evident to developers are not logical at all to users.

In the room, one hand after another shoots up. The questions are about understandable applications, the protection of patient data and the responsibility to bring everyone along. There are also many questions about the data behind AI. What does a system learn from? And don’t you need personal information for an appropriate answer? The discussion thus touches on a dilemma: how do you give a system enough context while protecting the patient at the same time? Van Weert explains that her lab works, among other things, with synthetic data: artificially compiled data that researchers can work with without using real patient data.

Who should do it?

A youth doctor brings the conversation back to daily practice. Healthcare providers are willing, but do not always have the means to use digital applications well. What can they do with today’s insights? And who ensures that the voice of their patients reaches developers and policymakers? Van de Veerdonk hesitates to give healthcare providers yet another assignment. They already have a lot on their plate, and now knowledge about AI is added to it. She would rather see the experiences of people with complex care needs taken into account during development. Van Weert agrees. “Tech companies are well represented in The Hague. But we cannot leave it to them alone. Social scientists and civil society organizations must also make themselves heard there.” Moderator Blom also refers to Sigra’s Wegwijzer Digitale inclusie (Digital Inclusion Guide).

What should have changed by 2040 if it were up to today’s speakers? The three agree that by 2040, users should no longer have to adapt to the technology, but the technology should take them into account. Dirksen is glad to see that awareness of this is already growing. Van Weert wants involving end users from the start to be a given. “I don’t want to have to fight for that anymore.” Van de Veerdonk adds, laughing: “And in fifteen years I really don’t want to have this job anymore.”