Many banks have started delegating an increasing number of operational functions to AI. From assessing credit scores and detecting fraud to delivering customer service, AI is helping banks make decisions and respond swiftly. However, models can hallucinate, drift from training conditions, and produce inconsistent outcomes as data and customer behaviours evolve, which is why the Reserve Bank of India (RBI) has proposed new rules for banks to manage AI risk, calling for continuous monitoring and, where necessary, a “kill switch”.
In order to spot drift, bias, or harmful output the moment it happens, and trace it back to where it started, teams need visibility across the entire chain a model runs through. When something goes wrong, organisations often see the outcome but struggle to identify where in the chain the failure actually occurred. How can organisations navigate this complex situation?
Never set and forget
With AI systems continuously evolving and the threat landscape growing more sophisticated, AI deployments cannot be set and forgotten. The tech world has already seen what happens when this approach is taken. Multiple incidents have surfaced where AI went against instructions despite explicit guidance not to. Against this backdrop, the RBI’s move is very timely. It holds organisations responsible for outcomes, even when models are outsourced.
This is why the RBI’s proposed framework shifts the conversation from adoption to accountability. Once AI is deployed, organisations must look at how decisions evolve. They should also track how models interact with applications, and when behaviour begins to deviate from expected outcomes. It requires oversight and the ability to intervene when AI systems begin behaving outside expected boundaries. This is where AI-powered observability becomes pivotal. Before an organisation can decide whether to pause, roll back, or switch off an AI application, it should have continuous visibility into how the application behaves in production. Observability provides that visibility. It helps organisations detect anomalies early, understand where issues originate across the AI application stack, support timely human intervention and generate the operational evidence needed for governance, compliance and audits.
Removing the guesswork
Visibility begins with seeing the AI and application stack as one system, not a collection of dashboards. AI-powered observability traces model responses, agent interactions, and the full Model Context Protocol (MCP) request lifecycle end to end. When it comes to which tool an agent called, in what order, how long each step took, or where a request actually spent its time: all of this can be captured through distributed tracing instead of being pieced together from logs after the fact. When combined with AI, distributed tracing can track prompts, model responses, and tool executions exchanged across multi-agent workflows. This enables developers to instantly pinpoint which agent hallucinated, lost context or caused a bottleneck to occur.
This level of granular visibility makes it possible to say where a problem started, or where it came from. It also removes guesswork and helps organisations identify abnormal model behaviour, unexpected output patterns, latency spikes and failures across the AI workflow before they escalate into customer or regulatory issues. Yet, none of this replaces human judgment, and the RBI’s draft is explicit on that point: institutions need the ability to intervene, and accountability has to stay with people, not with the model.
A kill switch is only as effective as the signals that trigger it. Visibility tells organisations something is wrong. Automated remediation and human oversight determine what happens next, whether that’s rolling back a deployment, pausing an AI workflow or escalating the incident for approval. BFSI organisations should arm themselves with the visibility to know when to reach for the kill switch, and the judgment to decide when to use it. The right decisions can only be carried out when you have the right signal to support them, and observability is part of that signal. It provides automation that allows BFSI to move at machine speed but never without a human in the loop.
The changes ahead
While the government’s framework is open for comment, and some specifics could change before it’s finalised, what won’t change is the underlying ask: organisations need to know what their AI systems are doing continuously, and to step in when something goes wrong. AI can take over more tasks in banking, but it can never be the one holding its own authority. That responsibility has to sit with the people running the institution, not the systems they’ve built.
Views expressed by: Ganesh Narasimhadevara, Director of Solutions Consulting, India at New Relic
