Transparent and Reliable AI Lab (TRAIL)

Chakraborty Group / TRAIL

TRAIL seeks to push the envelope of scientific knowledge towards Frontier AI Assurance, firstly through tackling open grand challenges like marginal vs individualised coverage in conformal prediction, proximal vs distal causal machine learning, leakage problem of concept bottleneck models, and then apply these methodological innovations to Frontier AI methodologies (generative AI, agentic AI, multimodal AI, ,etc) for high impact practical applications like personalised healthcare and precision biomedicine.

TRAIL is shared between UCL and the Alan Turing Institute (the UK's National Institute for AI). It is led by Dr Tapabrata Rohan Chakraborty, a Principal Research Fellow at UCL and a Theme/Group Lead in Frontier AI Assurance at the Alan Turing Institute. Research at TRAIL is conducted through 3 overlapping thematic workstreams as described below, but the overarching current North Star of TRAIL can be summarised by the following research question:

Theme 1: Frontier AI Assurance

Recently, there has been a rapid escalation in AI capabilities (like foundational AI, generative AI, multimodal AI, agentic AI, etc) in a range of complex tasks scaled over big data, thus forming a new frontier of emergent AI. However, for Frontier AI to be deployed with confidence in high-risk applications, it must come with design-enforced checks and balances that it is “fit for purpose”. This gives rise to the concept of AI Assurance, a term derived from the industrial concept of quality assurance, that would provide a “justified trust” based on guarantees of performance (accuracy, transparency, fairness, certainty) at the desired level for a particular task. Innovative design of these assurance benchmarks for emerging Frontier AI is yet unsolved and forms the overarching thematic scope of TRAIL.

There can be different ways of approaching the problem of Frontier AI Assurance. At TRAIL, as the name of the group suggests, we divide the effort into two workstreams –AI Transparency and AI Reliability. Within the AI transparency work package, we are interested in building new methods in AI explainability and interpretability like hybridisation of deep learning with mechanistic models, concept bottleneck models, causal machine learning, etc (more details in Theme 2 below). Within the AI reliability work package, we are interested in AI trustability and fairness through conformal prediction-based uncertainty quantification (more details in Theme 3 below).

There is also a thought leadership aspect to this theme related to AI policy and good governance, given the recent UK Govt white paper on AI assurance. Rohan (TRAIL group lead) is an invited expert in Responsible AI with the Global Partnership on AI (GPAI, part of OECD) and was the PI for the GPAI India project on Generative AI in 2023-2024. Following that, Rohan is currently leading the AI workstreams of the UK-India Vision 2035 programme on behalf of the Alan Turing Institute, the UK’s national institute for AI. Rohan also recently co-founded the Social Data Science Alliance (SDSA) to help businesses ensure that their AI tools are compliant with the Digital Services Act.

Featured publications

• Dey S, Banerji CRS ... Chakraborti T. Generating crossmodal gene expression from cancer histology predicts multimodal decision utility. arXiv:2502.00568, 2025. (Under review in Nature Communications) [Code]
• Banerji C, Shah A, Dabson B, Chakraborti T, et al. Clinicians must participate in the development of multimodal AI. Lancet EClinicalMedicine, 84, 103252 (2025).

Theme 2: Transparent AI with Concept Bottleneck Models

For powerful AI systems to be widely adopted in high-stakes applications and reach their full transformative potential, they need to be trusted, given the high risk. However, these deep learning-based AI systems are known to be complex and opaque; for such systems to be translated to the clinic, the AI black box must be opened. There are off-the-shelf techniques available for explainability, like SHAP, LIME, GRADCAM, etc but these are post-hoc visualisation-based techniques; what we need are more rigorous techniques that ensure interpretability by design.

One such method we are particularly interested in currently at TRAIL is the concept bottleneck model, where domain-established concepts for a particular task are learnt along with the predictive output to ensure that the decision is interpretable. One additional advantage of such models is that they provide the user with the opportunity to intervene on the concepts if wrong and thus provide a human-in-the-loop feedback to improve the system over time. Furthermore, one can add an unknown concept which might help to discover new clinically relevant information in an exploratory manner.

Two open questions in concept bottleneck models that TRAIL is working to address are: 1) rigorous quantification and then mitigation of the leakage problem between learnt concepts which is often not addressed in the literature and defeats the purpose of true interpretability, 2) extend the concept bottleneck framework to the multimodal AI scenario to measure information sharing (compatibility, orthogonality or redundancy) between modalities to assure optimal and transparent integration.

Featured publications

• Banerji C, Chakraborti T, MacArthur B, Train clinical AI to reason like a team of doctors. Nature 639, 32-34 (2025).
• Parisini E, Chakraborti T, et al. Leakage and Interpretability in Concept-Based Models. arXiv:2504.14094 (2025). (Under review in the Journal of Machine Learning Research) [Code]

Theme 3: Reliable AI with Conformal Uncertainty Quantification

Having transparency in AI Assurance frameworks is a necessary but not sufficient condition. In addition to the decision process being explainable and the decision itself being accurate at a population level, it also needs to have a high level of confidence or certainty for each individual. If not, that uncertainty needs to be carefully quantified and bounded with some sort of guarantee (hence the assurance), so that the human user can make the decision to take or leave the AI system's suggestion. In fact, the same logic extends to agentic AI too, as the AI agent can decide whether to act on the decision based on the confidence.

TRAIL is developing new conformal prediction-based methods to quantify uncertainty by providing coverage guarantees as well as individual-level bounds. A simple way to understand would be a classification task, where conformal prediction at a certain level of significance alpha would provide (through a calibration process) the minimal set of predictions for each test sample in the inference stage to ensure that the correct prediction is included in the conformal set, thus the length of the set of predictions itself is a quantification of predictive uncertainty for that individual.

Two open questions in conformal prediction that TRAIL is working to relax are 1) conformal prediction provides marginal coverage guarantee, but for high-risk applications like healthcare, we need personalised guarantees, 2) conformal prediction assumes exchangeability of test and calibration set, but this condition is not practical in case of evolving real-life datasets with drift/shift. On the application end, TRAIL is working towards expanding the conformal framework to encompass Frontier AI applications like foundational AI, generative AI, agentic AI, multimodal AI, etc.

Featured publications

• Chakraborti T, Banerji C, et al. Personalized Uncertainty Quantification in Artificial Intelligence. Nature Machine Intelligence, 7, 522–530 (2025).
• Banerji CRS, Chakraborti T, Harbron C, MacArthur BD. Clinical AI tools must convey predictive uncertainty for each individual patient. Nat Med. 2023 Dec;29(12): 2996-2998.

Group Members

Current members of the Transparent and Reliable AI Lab (TRAIL), led by Dr Tapabrata Rohan Chakraborty, including those co-supervised through his research network.