Making AI Safe Enough to Scale: Why Responsible AI is a Systems Transformation Task
By Ritik Raj
A widely used sepsis prediction model, built into an electronic health record, was designed to alert clinicians when patients might be at risk. Sepsis is one of the most time-sensitive conditions in healthcare. Early detection saves lives. That is precisely why AI-based warning tools have moved so quickly into hospital settings. But an external validation study at Michigan Medicine found that the model missed around two thirds of sepsis cases and generated frequent false alerts. The model existed, but the validation, workflow alignment and the clinical oversight around it were not sound.
The AI failures the sector is starting to see are not, in the main, technology failures. They are failures of the systems into which AI has been introduced.
Confidence Is Not Accuracy
A recent research report by OpenAI highlighted a growing concern with generative AI: these systems can produce answers that sound confident, polished and credible even when the information itself is incomplete or wrong. The risk is not that AI makes mistakes, the risk is that it presents them in a way that feels authoritative enough to be trusted.
In the sectors where AI is now moving fastest, healthcare, life sciences, medical technology, that confidence can cause real harm. Every recommendation, summary or insight in these fields must be backed by evidence, human judgement and accountability. When AI outputs feel credible enough to skip the checks, the checks quietly stop happening.
The Real Question Is Not About AI
We tend to frame the AI conversation as a technology question. What can this model do? How accurate is it? How fast can we deploy it? Those are useful questions.
But, the important question is what the surrounding system has to look like for the AI to be safe. What governance sits around it? Who owns the decision it accelerates? What data is it drawing on, and how confident are we in that data? What workflow is it being introduced into, and what does the workflow look like when the AI is wrong? Who validates it after it has gone live?
These are systems transformation questions and the answers are being consistently underinvested in.
At Tektology we work with organisations as a systems transformation advisory. Making AI safe enough to scale is about the systems around governance, data flow, workflow integration, human oversight and accountability that surrounds the AI, without which even a well-built model will fail in ways that damage patients, clients, and institutional trust.
Where Governance Breaks
In our work we see the same governance gaps. The first is that AI is often introduced without a clear owner. Someone has procured the tool. Someone else is using it. Someone else again is accountable for the outcome. When responsibility is diffused, verification tends to fall through the gaps.
The second is that the data underneath the AI is not always strong enough to support what is being built on top of it. We have written before at Tektology about the data mirage, the way data can look coherent on the surface while being fragmented, inconsistent or unreliable underneath. AI does not fix that problem. It scales it.
The third is that AI is often introduced into workflows that were designed for a pre-AI world. The tool assumes a set of behaviours, checks and handovers that either do not exist in the workflow it has been placed into, or that quietly fall away as staff come to trust the AI's outputs. When that happens, the safety net the tool was assumed to be sitting inside is no longer there.
And the fourth, which underpins all of the others, is that few organisations have a clear discipline for what happens after AI goes live. Validation is treated as a pre-deployment activity. In practice, the risk profile of an AI tool changes constantly, as the underlying data changes, as staff behaviour adapts around it, and as the model itself is updated. Governance has to be live, not one-off.
AI Adoption Is Easy. Responsible AI Is a Discipline.
The organisations that will get this right are not the ones that adopt AI the fastest. They are the ones that treat responsible AI as a discipline rather than a principle.
That discipline has structure to it. Frameworks give organisations a way of asking the right questions before, during and after AI is deployed. But no framework closes the gap by itself. The work that closes it is the systems work around the framework. Redesigning the workflows the AI is entering. Strengthening the data underneath it. Clarifying who owns what. Building the human oversight and validation loops that catch the tool when it is wrong. Setting up the monitoring that keeps working after go-live, when the pressure to declare success is at its highest.
This is not about slowing innovation, it is providing the discipline that makes AI innovation safe enough to scale.
AI will continue to move deeper into healthcare, life sciences and medical technology. In our own work at Tektology across the UK, Australia and India, we are helping clients shift the conversation from ‘is this useful?’ to ‘is our system ready to hold this responsibly?’
The organisations that succeed will be the ones that understand where AI creates value, where it introduces risk, and what the wider system around it needs to look like before it becomes part of real decisions.
Ritik Raj is a Senior Associate at Tektology, based in India. He works across digital health strategy and healthcare transformation programmes, currently contributing to the digital strategy workstream for the NHS Smart Building programme. Before joining Tektology, he was an Associate Consultant at Infosys, working on enterprise strategy for Fortune 500 healthcare clients. His work focuses on the practical space where transformation ambitions meet the systems, data and workflows that have to carry them.