FST JOURNAL

AI systems are transforming healthcare. They can analyse chest X-rays, read heart scans and flag potential health issues faster than human doctors – in some cases, in seconds rather than minutes. Hospitals are adopting these tools to improve efficiency, reduce costs and standardise care. As the Science Technology, Engineering Mathematics and Medicine (STEMM) community and the public collectively look forward to advances in the field of medical AI and the resulting benefit on patient care, it is vital that AI models are trained responsibly so that the unique needs of all patients can be addressed to ensure societal equity and patient safety. 

Fairness and bias impact AI for healthcare

 In her invited talk, Dr Tiarna Lee (Kings College London) emphasised that fairness encompasses both equality (treating everyone the same) and equity (allocating resources based on need), and that protected characteristics (e.g. age, sex, and race) must be carefully monitored when developing and deploying AI tools. Dr Lee explained that bias can arise at any stage of the AI pipeline, from defining the clinical problem to data collection, preprocessing, model training, and downstream clinical decision-making, with early-stage biases propagating through the pipeline. 

The datasets that are used to train many AI models are often imbalanced in terms of subject demographics and so the resulting AI models have a better performance for the overrepresented group(s) and worse for the underrepresented group(s). In her research on cardiac MRI segmentation, Dr Lee identified that models trained on imbalanced datasets demonstrate significant racial performance differences. This can lead to delayed or poorer-quality treatment for heart disease with worse outcomes overall, which then reinforces or worsens existing disparities. The talk concluded by calling for intentional and transparent development practices supported by frameworks such as FUTUREAI, which is an international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. 

Groups of conference attendees discussed what incentives could make sectors share responsibility for creating and maintaining diverse, representative datasets, and what practical steps could ensure how health technologies are regulated, commissioned or deployed. 

 Key themes

1. Data representativeness is essential: Participants consistently emphasised that AI systems cannot be fair without representative datasets but, in practice, current datasets are often skewed by age, ethnicity, geography or scanner differences. 
2. Bias is multidimensional: Bias arises not only from data imbalance but also from data curation, variable selection, annotation practices and institutional workflows. 
3. Transparency and traceability are lacking: Participants highlighted a lack of visibility into how datasets are constructed, what populations they represent, and how AI models are trained and validated. 
4. Regulatory gaps hinder safe deployment: There are currently no dedicated UK regulatory requirements for assessing dataset representativeness or model bias. 
5. Public trust must be strengthened: Given historical inequalities in healthcare, underrepresented communities may mistrust data collection initiatives. Trust was widely recognised as essential for gathering representative data, necessitating clearer communication and community-led engagement. 
6. Incentives are misaligned with fairness objectives: Both researchers and companies are often incentivised to demonstrate successful results rather than to interrogate bias. Developers may avoid rigorous fairness evaluations because finding bias can slow publication or complicate regulatory processes. 
7. Collaboration across stakeholders is essential: Fairness cannot be solved by technical teams alone. It requires collaboration between AI developers, clinicians, regulators, funders, and the public. 

 Recommendations for stakeholders:

 AI developers

• Build trust to incentivise inclusive user participation, using transparent communication to address historical mistrust and encourage the involvement of underrepresented groups.

• Build a culture of reputational risk around non-compliance and opportunistic sampling, encouraging equity as an explicit requirement not an optional addition.

• Educate AI developers on the risks of using imbalanced training data, including how bias propagates through the pipeline from early design through to deployment.
• Prioritise representative and diverse datasets, actively seeking data that reflect the patient populations the AI tools will serve.

 Market end-users

• Build a digitally skilled health workforce with training on equity issues and patient engagement barriers to avoid perpetuation of systemic bias and health inequalities.

• Implement standardised and transparent data practices, ensuring clinicians and patients understand how to interpret results and their limitations.

• Use subgroup performance checks and bias-focussed, pre-market safety checks before deploying AI tools in healthcare settings.