The Big AI Debate: Episode 3
Who Takes Responsibility When AI Goes Wrong?
Panel:
- Host: Sejal Patel, Senior Conference Producer, The AI Summit Series
- Karthik Josyula, Kohler
- Divyansh Saxena, HomeServe Finance
November 18, 2026. 10 min read
When AI makes a mistake, who should be held accountable? In this episode of The Big AI Debate, host Sejal Patel sits down with Karthik Josyula and Divyansh Saxena to explore where responsibility lies when AI systems make critical errors, from biased hiring decisions to costly coding mistakes.
Expect real-world scenarios, tough questions and different perspectives on AI accountability, trust and governance, plus a lightning round putting the guests' AI knowledge to the test.
Who Is Responsible When AI Goes Wrong?
When AI makes a costly mistake, the technology isn't the one held accountable. The people and organizations behind it are. That's the central question explored in Episode 3 of The Big AI Debate, hosted by The AI Summit Series, where Karthik Josyula, Head of Data and AI Platforms at Kohler, and Divyansh Saxena, CTO at HomeServe Finance, debate who should take responsibility when AI systems fail.
Hosted by Sejal Patel, the episode explores AI accountability through real-world scenarios, from biased hiring decisions and copyright disputes to autonomous systems making costly mistakes. While both guests agree that responsibility must be shared, they differ on how much accountability belongs to developers, businesses and end users.
Who should be accountable for AI decisions?
The debate opens with a fundamental challenge: AI is increasingly making decisions that affect people's finances, careers and safety, yet even its developers cannot always explain how those decisions are made.
Divyansh argues that AI's complexity shouldn't prevent businesses from using it, provided the right safeguards, testing and human oversight are in place.
Karthik takes a more cautious approach, questioning whether these protections are sufficient for high-risk applications such as autonomous vehicles and medical procedures. He emphasizes the importance of defining clear boundaries before AI systems are deployed.
While Karthik initially places greater responsibility on developers to provide safety capabilities, Divyansh argues that the businesses deploying AI should carry the greatest responsibility.
Where does fair use end and copyright infringement begin?
The conversation shifts to the use of copyrighted material in AI training.
Divyansh takes an optimistic position, arguing that access to diverse training data is essential for developing capable AI systems. Karthik focuses on traceability, highlighting the importance of distinguishing between taking inspiration from existing content and reproducing copyrighted work.
The discussion also raises concerns about AI-generated content being used to train future models, potentially compromising data authenticity and accuracy.
Both acknowledge that legal frameworks are struggling to keep pace with AI development, raising questions about how governments can establish meaningful accountability.
When AI hiring tools discriminate, who takes the blame?
The debate becomes more practical with the example of AI recruitment tools accused of discriminating against job applicants based on age and other characteristics.
Karthik argues that organizations must take responsibility for the rules they establish when deploying these tools. Divyansh agrees but highlights another challenge: AI models can inherit biases from their training data, even without explicitly discriminatory instructions.
Both emphasize continuous monitoring, regular evaluations and clear accountability to prevent biased decisions from affecting real people.
What happens when an AI agent goes rogue?
One of the episode's most striking examples involves an AI coding assistant that was explicitly instructed not to modify a company's live database but deleted it anyway.
Divyansh argues that incidents like this expose weaknesses in access permissions and security controls, particularly when organizations prioritize speed over safeguards.
Karthik takes the argument further, suggesting that businesses are racing to demonstrate AI adoption without establishing the necessary architecture, governance and security frameworks.
Both agree that restricted access, defined operating boundaries and human oversight are essential. AI failures are not always failures of the technology itself, but of the decisions surrounding its deployment.
Can insurance protect businesses from AI failures?
As AI liability insurance emerges, the guests disagree on whether it represents progress or another way for businesses to avoid responsibility.
Karthik remains skeptical, describing insurance as a potential "band-aid" for inadequate safeguards. Divyansh sees it as a natural sign of technological maturity, comparing AI with established industries such as aviation and nuclear energy.
Despite their differences, both recognize the importance of testing results, decision logs and audit trails in demonstrating how AI systems operate and identifying responsibility when failures occur.
Why are companies still racing to deploy AI?
If the risks are so significant, why are businesses continuing to accelerate AI adoption?
For Divyansh, the answer is business value. AI offers substantial opportunities to improve efficiency, reduce costs and generate revenue, creating pressure to adopt it quickly.
Karthik agrees that financial returns are a major driver but questions whether organizations are prioritizing growth over safety and human impact.
Their disagreement becomes clearer when discussing AI replacing human workers. Karthik opposes replacing people simply for commercial gain, while Divyansh argues that certain applications, such as autonomous driving, could justify automation if it improves safety.
Where do the two experts actually disagree?
Throughout the episode, both guests support responsible AI adoption, but their approaches differ.
Karthik emphasizes preventative safeguards, clear operating boundaries and leadership accountability before deployment. Divyansh focuses on the opportunities AI creates, arguing that businesses can move quickly when appropriate controls are in place.
In the closing debate, Sejal asks them to divide 100 points of responsibility between developers, deployers and users.
Divyansh assigns 20% to developers, 60% to deployers and 20% to users. Karthik agrees that deployers carry significant responsibility but favors a more balanced distribution, giving users a greater share.
Both ultimately agree that responsibility cannot rest with the AI model itself.
The takeaway
As AI becomes more autonomous, accountability cannot become more ambiguous. From biased hiring decisions to costly operational failures, the episode highlights why businesses must establish responsibility before deployment rather than waiting until something goes wrong.
For organizations scaling AI, the question is no longer just what the technology can accomplish, but who will be accountable when it fails.
The episode's key takeaways:
- Accountability is shared: Developers, businesses and users all play a role, but deploying organizations carry significant responsibility.
- Safeguards must come first: Clear permissions, human oversight and operating boundaries help prevent costly failures.
- Transparency is essential: Testing, traceability and audit trails help organizations identify risks and demonstrate accountability.
- Regulation and insurance aren't enough: Businesses still need strong internal governance and continuous monitoring.
- Speed must be balanced with risk: Sustainable AI adoption requires businesses to prioritize safety alongside commercial value.
[00:00:00] Sejal: An AI coding assistant was told explicitly not to touch a company's live database. It did anyway, deleted the whole thing.
[00:00:08] Divyansh: Oh, I'll never hear this, where AI is [00:00:10] becoming so capable that a knife is much less harmful than an AI.
[00:00:13] Kartik: The minute we put a robot to do a surgery, the minute we do a fully autonomous vehicles on the road, how effectively [00:00:20] are we putting those where we are not putting human race at risk?
[00:00:24] Sejal: Today, we're asking a really important question: Who is responsible when AI [00:00:30] goes wrong? When an AI system makes a mistake, price, is it the company that built it, the business that deployed it, or the person who [00:00:40] trusted it in the first place?
[00:00:50] Sejal: Welcome to the big AI debate, where the world's leading minds tackle AI's biggest questions, brought to you by the AI Summit Series. I'm your host, [00:01:00] Sejal Patel, and today we're asking a really important question: Who is responsible when AI goes wrong? When an AI [00:01:10] system makes a mistake, one that costs someone money, a job, or worse, who pays the price?
[00:01:17] Sejal: Is it the company that built it, the business [00:01:20] that deployed it, or the person who trusted it in the first place? To help us answer that, I'm joined by two experts from opposite sides of the AI [00:01:30] ecosystem, Karthik Josyula, Head of Data and AI Platforms at Kohler, and Divyansh Saxena, CTO at Homeserve [00:01:40] Finance.
[00:01:41] Sejal: The views and opinions expressed during this conversation are for discussion purposes only and do not necessarily represent the [00:01:50] views or positions of the companies, organizations, or institutes that the participants are affiliated with. So guys, let's jump [00:02:00] right in. We're gonna start with a quick warm-up.
[00:02:03] Sejal: So finish this sentence in one line: The biggest misunderstanding about AI is... [00:02:10] Karthik, you first. What would you say?
[00:02:12] Kartik: AI can replace a human intelligence. That's a big misrepresentation.
[00:02:17] Sejal: Divyansh, over to you. Do you agree, [00:02:20] or is there something else you'd point to?
[00:02:22] Divyansh: Oh, 100% I agree. I agree with that statement, but I would also add, uh, that I think the biggest misrepresentation is, uh, that it's [00:02:30] the new godlike technology that can, you know, change anything and everything.
[00:02:34] Sejal: Okay. So AI is now making decisions about who gets a loan, [00:02:40] what stocks to buy, and how to defend against cyberattacks, but the reasoning behind some of those decisions isn't always clear, even to the people [00:02:50] who built the systems. So how comfortable should we be with that? Is this a risk we shouldn't be taking, or is it just [00:03:00] an inevitable part of advancing AI?
[00:03:03] Sejal: Divyansh, let me start with you. Are we hitting the actual limits of what we can understand here, or [00:03:10] do we simply not have the right systems in place yet to explain the decision-making?
[00:03:15] Divyansh: I think as models are growing more and more in terms of their power or, [00:03:20] you know, you can call it weights, I think it's getting very difficult to go into the nitty-gritty of how complex, uh, you know, things are and how decisions are being made.
[00:03:27] Divyansh: So I think 100% there is, [00:03:30] there's trueness to, uh, you know, when people say that AI is the black box, especially the new AI models, which are like, you know, trillion parameters LLMs. But I also think [00:03:40] that that doesn't mean we shouldn't use a black box. Like if you can put the right amount of guardrails in front of that black box, uh, a lot of good outcomes can come out of it.
[00:03:47] Divyansh: Uh, you know, we're already seeing lot, lot of progress [00:03:50] in terms of, you know, customer service or, you know, coding where someone cannot today explain why this code was written such Uh, way, but it's still, you know, produces the right [00:04:00] result with the right, you know, guardrails around coding, for example, the test suite or evals and all of that.
[00:04:05] Divyansh: For example, in customer support, we have the right guardrails around, you know, human in the loop where [00:04:10] even though LLM is a black box and producing output which we don't understand why, we still have, you know, humans in the loop that probably hit a button Approve, and then, you know, comms get sent out to the [00:04:20] customers or...
[00:04:20] Divyansh: So I think I agree. Like, I think the black box thing is there, and it is very complex to explain. Even the people who are building the models, sometimes they are like, they also get surprised how the model is [00:04:30] performing this way, and, like, they're even shocked how good this model is. But I think as, as you said, the builders are doing their job.
[00:04:36] Divyansh: The deployers need to then produce the right guardrails around it.
[00:04:39] Sejal: [00:04:40] Okay. Anything to add there, Karthik?
[00:04:41] Kartik: Absolutely, Sejal. Let's unpack what you explained or the examples that you had provided, right? Loan approvals [00:04:50] or any other customer service within excluding the PII, um, or document generation, content generation.
[00:04:57] Kartik: We are still scratching the surface. The [00:05:00] minute we put a robot to do a surgery, the minute we put a robot to deploy a missile, the minute we do a fully autonomous vehicles on the road, how much of the [00:05:10] security and the, the guardrails that Devansh had just double-clicked on, how effectively are we putting those where we are not pu-putting human [00:05:20] race at risk?
[00:05:20] Kartik: Not even human race, even the environment at risk. So I think the technology's evolving at such a rapid rate, it's [00:05:30] challenging for the developers, even the, the technologists and even the end consumers of the AI. It's, it's challenge. It's a challenge for us to catch hold of, [00:05:40] of the technology and try to understand what are the risks, what are, what are the challenges that this technology will open a can of worms.
[00:05:48] Kartik: Do we have clear [00:05:50] visibility into what are the risks, and do we have a clear mitigation plan? I think we need to start from there. It depends upon the use case. It depends upon the sensitivity. It depends upon [00:06:00] the impact that AI can, can create, and it's not only at the deployment phase where we want to have evaluations or guardrails.
[00:06:08] Kartik: I think we need to [00:06:10] clearly identify each and every trace that AI leaves and every connector that connects to the AI application. Each and every box needs to be clearly [00:06:20] evaluated, and security walls should be built around each and every component. So I, I will start at, uh, 30,000 feet, and I'm happy to deep dive with Devansh and with you [00:06:30] with, with parachutes.
[00:06:31] Sejal: Well, Karthik, when you mention surgeries and robots and loans and things like that, but when AI systems make a decision [00:06:40] with real consequences for someone's life, how much explanation should we expect before we're willing to trust it?
[00:06:47] Kartik: So that's an example I had provided, but I think [00:06:50] how, how I look at an agent is what is the contract that we are writing for the agent to operate?
[00:06:56] Kartik: And we need to ensure the agent doesn't go beyond [00:07:00] the contract or the agreement that we have in place. To me, agent is kind of a machine. Absolutely, machines, there is a f-failure rate. Machines can fail, so as the [00:07:10] agentic AI system can fail, so as, uh, my, my laptop or, or a car, right? But how are we guaranteeing that the agent that we are deploying [00:07:20] operates within the guardrails, operates as per the contract that we have defined by all the parties, be it the developer, be it the technologists, or be it the end [00:07:30] consumer?
[00:07:30] Kartik: What are the contracts do we have, and how are we evaluating those contracts? I think those are very critical pieces to define e-even before developing an [00:07:40] agentic AI system.
[00:07:41] Divyansh: Yeah, I'll add to what Karthik s-just said. I think probably surgery is a very d-very extreme example, and probably we are many years away from, you know, an, an agentic [00:07:50] AI doing a surgery and, like, not doing a good job and in fact making a life-threatening mistake, and then we have to go blame the models and try to get into, you know, deep [00:08:00] understanding of why, why that happened.
[00:08:01] Divyansh: I think, I think while that is few years away, I also feel, you know, we need to be clear on applications of AI. Like, I think I'll never put something which is [00:08:10] a black box in terms, like in front of a patient that is, you know, fighting through life and death. But yeah, I'll definitely put all the agents, AI agents I can in front of, you know, [00:08:20] a software engineer who is building things or, like, a customer support who's talking to a hundred people a day or, you know, any other probably not, not as, you know, critical application as, you know, a surgery.
[00:08:30] Divyansh: But yes, even if we imagine that scenario happens and we have to try to find an explanation, okay, why that happened, you can to some extent get an explanation. You know that, okay, [00:08:40] we had some eval, so there was a bug in an eval, and we missed evaluating that. But then I think the ownership cannot be, you know, gone to a model.
[00:08:47] Divyansh: Like, we should not blame the model at all. [00:08:50] It's, it's, it's a piece of code that is, you know, running behind, you know, a server using electricity and data centers and all of that. I think the, the ownership of that mistake or the [00:09:00] explainability is on the humans, you know, or the systems that are being deployed as opposed to the model that is like a trillion dollar, uh, or trillion [00:09:10] parameters worth model.
[00:09:11] Kartik: Like we mentioned, right, technology is evolving. We don't know what's tomorrow or what's next week is looking like. But let, let me bring this down to a simple [00:09:20] example. What if s- a- an agent impersonates my voice, calls my family and maybe say, "Hey, can, could you put some money in a specific account because I got stuck?"
[00:09:29] Kartik: That's very much [00:09:30] possible. So how can we, how can we make sure that we have got enough guardrails where agentic AI systems are not misused? So that's where it all goes back to the [00:09:40] contract, right? An agent operating, what, what is the contract that the agent needs to operate? If it's a voice agent, how are we ensuring that the true voice is not mimicked?
[00:09:49] Kartik: How can we [00:09:50] put kind of encryptions in the voice where agents cannot get the actual human voice and try to impersonate? So again, I'm, I'm trying to bring a, a very simple, very much [00:10:00] realistic example, not going to the far end of surgeries or deploying a weapon.
[00:10:04] Divyansh: Who, who do you think will, uh, or like is responsible for doing those, putting those guardrails for that simple example?
[00:10:09] Divyansh: [00:10:10] Like, is it, is it, you know, the big techs in California building those models or, you know, the company who is deploying those models, you know, for you to video call someone or [00:10:20] you who is using the phone to call? Who is-- who do you think is responsible for the guardrail here from your perspective?
[00:10:26] Kartik: For me, I mean, I, I will start with the creator.
[00:10:28] Kartik: Like, who had created [00:10:30] these LLMs, they need to make sure that the voice agents, example, referring my example, they need to make sure that the voice is not 100% efficient or [00:10:40] 100% accurate where it's actually mimicking the human voice. Or even they should, they should add a capability within an LLM where someone, if, if, if we [00:10:50] want to trace, is, is this an agent-assisted voice?
[00:10:53] Kartik: How can we detect that? There should be a, an evaluation capability where we can safely [00:11:00] evaluate and identify is this an voice coming from an agent. At least the creators needs to give those kind of capabilities for us to make sure we feel safe while we [00:11:10] interact with these agentic AI systems.
[00:11:12] Divyansh: I think I, I agree with you principally, but I think where I differ is potentially, you know, as, as Sejil said in the beginning, there are, you know, [00:11:20] builders, as you said, who are probably most or the creators of the models are mostly responsible for, you know, giving us the right visibility and transparency of how model works, and then we can potentially add [00:11:30] guardrails on top of that.
[00:11:31] Divyansh: But then there are-- so there are builders, and then there are deployers, and then potentially users. And I think above them, I think there is government who's gonna probably control everything. But I feel [00:11:40] like from my perspective, I would say deployers are, you know, are responsible the most. I think the responsibility is shared among every entity that we talked about, but from my [00:11:50] opinion, I think deployers are mostly responsible.
[00:11:51] Divyansh: And, and to probably explain that, I'll probably give an example of, you know, a knife. So someone who, you know, manufactures a knife on a [00:12:00] daily basis is not responsible for all the, you know, attacks that we hear every now and then. I think the res- the, the one who is actually using and, you know, deploying or [00:12:10] like, you know, in my opinion is the main, you know, owner of putting the guardrails.
[00:12:13] Divyansh: I, I never like sue IKEA or someone else, you know, to why are you manufacturing knives these days? And, [00:12:20] and that's where my, you know, emphasis is on deployers than the big techs who are building the models.
[00:12:25] Kartik: That's a very, very good argument statement, Divyansh, but, uh, it's kind of a straw man fallacy to [00:12:30] me.
[00:12:30] Kartik: Knife is a very simple example here, but we are talking about agentic AI systems which are much more complicated. I think there is a shared accountability. I, I agree to you, yeah, to your [00:12:40] initial statements. There is a shared accountability, like builders, they are building these LLMs. They need to make sure they are empowering the deployers with enough [00:12:50] capabilities where the deployers take these capabilities and evaluate, and now the deployers, they add more additional evaluation frameworks where [00:13:00] they serve these models to the users.
[00:13:02] Kartik: At the end, the users needs to be empowered to make sure these agentic AI systems are safe, safe to use, and there are [00:13:10] no repercussions or implications. I think at the-- the end goal is how can we create a safe environment around users who are interacting with the agentic AI systems, and it's a [00:13:20] shared accountability.
[00:13:21] Sejal: Wow. That's definitely a debate now. No, absolutely. Go ahead, Divyansh. I think you two are taking the right stance. [00:13:30] Go ahead, add your thoughts before we move on to the next. Uh,
[00:13:32] Divyansh: I was, I was gonna come back to what Karthik said earlier, like his knife is a very simple example, and AI agents are very complex, and I thought I'll never hear [00:13:40] this where AI is becoming so capable that a knife is much less harmful than an AI, which is great.
[00:13:45] Divyansh: I think talks a lot about how responsible we need to be when you're using AI.
[00:13:48] Sejal: Absolutely. And [00:13:50] when we're talking about responsibility We just covered trust and LLMs being used. Where-- Let's, uh, let's talk about copyright now. [00:14:00] AI companies are training models on millions of books, articles, images, some without permission.
[00:14:08] Sejal: Some call it fair use, [00:14:10] others are calling it theft. We're seeing major lawsuits this years with publishers, photographers, even newspapers taking AI companies to courts, [00:14:20] and settlements running into the billions. Where, uh, Divyansh, where do you draw the line between fair use and theft?
[00:14:28] Divyansh: I think I'm probably an [00:14:30] optimist here and, like, pro-AI here.
[00:14:31] Divyansh: Not, not extremely, but to some extent. Uh, and, like, tying our previous topic as well slightly onto this where if let's say, you know, if we [00:14:40] build a model based on some, you know, or only the public data, and we do not have access to, you know, talk or, like, train the data on how, you [00:14:50] know, machinery works or tools work, and then 10 years later we have this AI automation to use machines, but because the training data is not, uh, you know, never had that [00:15:00] data, it will not work well, then who do you blame that?
[00:15:03] Divyansh: Are you gonna blame the company who build the models without accessing these data? Or, like, are you gonna blame, you know... So that's [00:15:10] where the, the gray line is, and I'm-- that's why I'm on the pro side of AI where, you know, like, more data it gets, it's only gonna make human lives easier and, you know, the risk [00:15:20] of failure probably is less because it has now knowledge of all those, you know, technical, you know, nuances or, like, you know, very specialized topics that [00:15:30] are not there in public, and AI needs to be aware of that in, in, you know, for times when we need AI to talk about those or help people solve those problems.
[00:15:38] Sejal: Karthik, when we're talking [00:15:40] about Fair use and theft. From your perspective, when a model produces an output, um, can companies actually trace which training data shaped it?
[00:15:50] Kartik: Absolutely, Sejal, and that's what we encourage in our organization on whenever an agent or whenever a model generates inferences, we [00:16:00] enable traceability.
[00:16:01] Kartik: We want to trace back to the core data or core knowledge basis from where the agent had pulled the references and [00:16:10] created the content. Again, there is a very thin line between drawing inspiration from the agentic system output, especially to your reference to the media [00:16:20] content, literature, it could be music, uh, to copywriting, right?
[00:16:25] Kartik: I mean, I am a drummer. I'm, I'm part of a band. We, we play pro- pro- progressive rock. [00:16:30] We take inspiration from the great bands like Dream Theater and very good number of Indian progressive rock bands, but we create our own tracks. We use AI, uh, for, for music, and we, [00:16:40] we try to take the inspiration and see how AI generates our chord progression with reference to the great progressive rock bands and, and we try putting them in our [00:16:50] jamming sessions.
[00:16:51] Kartik: But our intention is not to take a track from, from one of the great bands and infringement it and kind of cr- mimic them, mimic the tracks. [00:17:00] So I think there's a very thin line on the inspiration versus the Theft. And how can we make sure that we, we have the clarity is [00:17:10] traceability. Can we go back to what is the original source of truth these agents are referring to, and what is the grounding truth where, where the agents are generating the [00:17:20] content?
[00:17:20] Kartik: I think if we have the traceability enabled, I think we can clearly go back and, and find the original sources.
[00:17:26] Sejal: Okay. Well, I would love to hear your rock band there.
[00:17:29] Divyansh: I [00:17:30] feel like, you know, this, this topic is, you know, the training data and the copyright issues regarding, regarding those. I feel like it's, it is an issue this year and last year as well we saw a lot of, [00:17:40] uh, you know, lawsuits regarding that.
[00:17:41] Divyansh: But I feel like the big giants have a new problem now, like they have consumed all the data that there exists on internet to train. And because now they want to build [00:17:50] more frontier models, they need more data. And I won't take names, but I know one of the big giants have now, you know, just allocated half of the engineering team to now writing code, and [00:18:00] writing code so that that code now can go into the training data because, like, there's no more, you know, data out there to train on.
[00:18:06] Divyansh: So they are creating data or asking their half of the engineering team, you know, [00:18:10] 8,000 of them now create new data so that they can go into the models. So I think while copyright is an issue and, like, lawsuits might happen and keep on happening, but I feel, yeah, I think from a [00:18:20] technology point of view, there is a bigger problem that now they are solving because the models are capable enough to, you know, consume all the internet data.
[00:18:26] Kartik: Divya- Divyansh, I think you have invited a new elephant in the room, [00:18:30] right? The content that, because big companies have exhausted all the content and to train the, the models, what's happening, what I've observed [00:18:40] is the content that's being generated by the AI systems is kind of acting as a loop that's being fed to other agents or other LLMs to train, which is [00:18:50] kind of a, a kind of a red flag to me now.
[00:18:52] Kartik: Because now we are losing the, the authenticity of the data, we are losing the, the accuracy [00:19:00] of the data. Uh, so that's one thing I, I'm kind of, it's, it's kind of a concern to me. So I, I keep going back to this thought of how can we [00:19:10] encrypt an identifier in each and every content that gets generated by a, a, an agentic system so that we can trace, is this content created by human [00:19:20] or an agent?
[00:19:21] Kartik: If it's from an agent, can we trace it back to the, to its original references? So I, I want to see the big giants coming up with this capability, which [00:19:30] can clearly draw a demarcation line between human content versus artificial content. Two, if it's artificial content, can we trace it back to its references?
[00:19:39] Sejal: So when you're [00:19:40] talking about artificial content and tracing it back, I'll ask this to both of you, AI is evolving far faster than the legal system can respond [00:19:50] nowadays. Can courts realistically set meaningful rules for a technology that might look completely different by the time those [00:20:00] rulings actually take effect?
[00:20:01] Divyansh: I think it's gonna be hard to, you know, always be in f- in front of the technology and, like, s- have the right set of rules. [00:20:10] Unfortunately, we will, and we are already seeing cases where, you know, AI has done something that was not thought of and, like, you know, then there are these legal entities and governments being involved.
[00:20:19] Divyansh: And [00:20:20] while it sounds wrong, I feel that's the nature of any big new technology entering, you know, kind of making a dent on the civilization. And I feel like AI is one of [00:20:30] them. Unfortunately, we'll have to go through these cases You know, where it will harms few people or few communities or few companies, and then the restrictions will, uh, you know, [00:20:40] come.
[00:20:40] Divyansh: I think it's very hard for the restrictions or legal, uh, you know, concepts to be l-ahead of what technology is coming, you know, or produce. Which is, which is I think something that has happened. It's not [00:20:50] f- AI is the first entity or technology that is going to face that. Like, we talk about airplanes or we talk about, you know, any other, any, like, new technology that revolutionized human civilization.
[00:20:59] Divyansh: I think it [00:21:00] went through that where, you know, it leaked, like the, the, you know, the technology leaked some harmful effects on the community or companies and then law probably took the upper hand. That's my [00:21:10] opinion, uh, on this.
[00:21:11] Sejal: Sarthak, what are your thoughts?
[00:21:13] Kartik: I think legal systems are also using AI, isn't, aren't they?
[00:21:17] Kartik: So is it kind of an AI system versus [00:21:20] A- a-another AI system in, in, in the legal where those two agents argue? I, I don't know. So I think it's kind of a challenging situation. I think legal [00:21:30] systems will evolve. Again, to your point, Sejal, you said AI is evolving faster than the legal system's maturing. I think it's not only just the legal system, every other line of business, every [00:21:40] other department around us is evolving.
[00:21:42] Kartik: It's not only the technology. It, it's, it's a hand-holding exercise that, that we all need to do. We, we, we need to make sure the, [00:21:50] the society is empowered, be it, be it the AI knowledge, be it the safe space, creating that safe space. I, I talk to my kids on how to, how to use AI, uh, [00:22:00] within, within that safe, safe environment.
[00:22:01] Kartik: So I think we will evolve, uh, and we need to create that support system.
[00:22:04] Sejal: Yeah.
[00:22:05] Divyansh: Yeah. I think, I think governments are doing a good job as well. Especially, I know UK FCA has [00:22:10] a, a good concept called sandbox program, where these new technologies or new companies and startups can go through this two, three months long program, and then they can test it with [00:22:20] few, uh, you know, users and, and only then, you know, uh, both FCA, which is the financial regulation in UK, and that startup can, you know, approve and [00:22:30] probably make changes to the regulations and all of that.
[00:22:32] Divyansh: So I think governments and authorities are aware of, you know, what bad can happen if technology just goes rogue. [00:22:40] And, yeah, I think the, I think-- I don't know, I, I don't know, I think if UK is the only one, but I know UK has been doing this the longest. When I was working for a small [00:22:50] fintech startup in 2018, uh, we were trying to build this financial advice using AI, which was one of its kind back then.
[00:22:57] Divyansh: Uh, I think this is like eight years ago, [00:23:00] where you never had an AI to give financial advice, and it was... Even today, it's not allowed. So even if you talk to these LLMs today and talk about financial advice, they always quote, "This is not a financial advice. [00:23:10] Take actions, you know, responsibly." But then to able to do that confidently, you have to go through this program from UK's government and, and like, while legal entities, like, [00:23:20] will always be a little bit behind, you know, in terms of how AI is progressing, I think the, the, the legislations and the authorities are doing their job to Put it into a box [00:23:30] before releasing it to billions.
[00:23:32] Kartik: Just to add another point there, Divyansh and Sejal, I'm gonna lean on to what Divyansh had, had left, right? Uh, the legal systems and how are we... how are [00:23:40] the legislations empowering to, to have, have the closed black boxes. Uh, one critical thing is I want to call out is, are the institutions [00:23:50] applying those legal policies?
[00:23:51] Kartik: That's very important. Are we conducting regular audit, auditing, and are we, are we-- do we have a clear audit trail [00:24:00] and putting the right individuals or teams or institutions accountable? Are we putting the right teams, individuals accountable? And how, how the audit [00:24:10] teams are needs to be aggressive to, to ensure the policies are reinforced.
[00:24:15] Kartik: I think that's also a equal share of accountability that needs to be with, with the [00:24:20] institutions who are developing using AI for, for profit.
[00:24:24] Sejal: Well, Divyansh, you mentioned, um, a little bit about the governments, the UK government, um, being at forefront [00:24:30] using UK AI, um, regulations. Um, here's another live issue that I'm seeing.
[00:24:35] Sejal: We're seeing AI hiring tools accused of filtering [00:24:40] out older workers, and AI written job ads accused of including unlawful restrictions like citizenship status. And different regions are [00:24:50] handling this in completely different ways. The EU AI Act puts real obligations on developers. The US is still leaning on existing laws like [00:25:00] product liability and anti-discrimination rules, and plenty of other markets barely regulate this at all.
[00:25:07] Sejal: So Kartik, who do [00:25:10] you think should actually carry the responsibility here? The company that built the tool or the company that used it?
[00:25:19] Kartik: I, [00:25:20] I'm more inclined towards the company that's using the tool. Again, I, I go back to my previous statement on what is the contract that you are defining for an agent to do?
[00:25:28] Kartik: We, we [00:25:30] write the rules. Again, don't quote me on, on this one, but if I create an agent and, and explicis- explicitly say, "I don't need candidates above 40 [00:25:40] years." That's it. Agent will follow the rule, takes the candidate's date of birth, calculates the years, and it starts rejecting. But how can I write the contract which is [00:25:50] more ethical and which builds that inclusivity and measures the actual talent of the individual rather than drawing a specific dimension which is unethical?
[00:25:59] Kartik: So I think [00:26:00] it, it absolutely sits on the individuals that are using these tools.
[00:26:04] Sejal: Divyansh, what are your thoughts? Does this hold up to from where you sit, or would you split it [00:26:10] differently?
[00:26:10] Divyansh: No, I agree. I think I'm, I'm, I'm always putting more responsibility on the deployers or people who are consuming these tools.
[00:26:17] Divyansh: And I think it's a tricky one because like, yes, [00:26:20] uh, you know, you can, you can give an instruction that do not hire people who are, you know, old. But I think the bigger problem with AI is like because it's being trained on data, the internet data, and internet [00:26:30] has biases, uh, because internet data has human, uh, you know, text, and human text has inherent bias.
[00:26:36] Divyansh: I think very, very early versions of LLMs, like if you talked about, [00:26:40] you know, a nurse, then it would always assume that it's a female, you know. And, and those inherent, I think, assumptions probably are still there, not as mu- as prominent [00:26:50] as the example that I gave. And I think to tackle those, of course, is a responsibility of, as Karthik said, people who are using them and, and how I think today we are dealing in the [00:27:00] AI applications that we have built in our organization, and I know a lot of other companies are dealing with, is by putting continuous monitoring in front.
[00:27:06] Divyansh: So while AI is, you know, doing its thing and, uh, you [00:27:10] know, spitting, uh, the output and makes decisions, you have to put some, you know, monitoring that does these checks on a regular basis and for you to then take a decision off [00:27:20] the back of that. You know, how do you tweak the prompt so that if it's going in the direction where it's only hiring young people, then you can have a very expl-explicit prompt that, you know, don't do that or, you know, evaluate [00:27:30] fairly and all of that.
[00:27:30] Divyansh: So continuous monitoring, I feel, is very important. And then agree to Karthik that it's has to be done by the deployers.
[00:27:36] Sejal: So what I'm hearing is companies should do [00:27:40] their own due diligence before deploying these tools, and they should actually require that homework rather than leaving it to optional.
[00:27:48] Divyansh: Agree. That's it. [00:27:50]
[00:27:50] Sejal: Okay. Well, thanks both for that. Now let's switch gears for a quick game. So I've got a handful of news headlines for you both. Mm-hmm. Some actually [00:28:00] happened, some are completely made up. You've got three seconds to call it real or unreal, and I will relli- reveal the correct answers after you both make your [00:28:10] choices.
[00:28:10] Sejal: So headline one: A sixteen-year-old needed to be air rescued after following AI-planned hiking route onto a trail [00:28:20] nicknamed The Widowmaker.
[00:28:22] Kartik: Real. Real.
[00:28:24] Sejal: That's correct. Real. Well done, boys. Um, next one, headline two: A [00:28:30] delivery platform AI agent issued refunds worth hundred and eighty thousand dollars after a routine software update.
[00:28:38] Sejal: Real or unreal?
[00:28:39] Kartik: [00:28:40] Real. Real
[00:28:42] Sejal: Actually, you're wrong there. It's unreal
[00:28:45] Kartik: I think I heard a similar use case. Yeah
[00:28:48] Sejal: Okay. Headline [00:28:50] three: A restaurateur loses a £51,200 tax fight after AI feeds him legal rulings that don't [00:29:00] actually exist
[00:29:01] Divyansh: Hmm. Sounds possible, so I'm gonna say real
[00:29:05] Sejal: You're correct, Divyansh, it's real.
[00:29:08] Sejal: So headline four: [00:29:10] A Texas hospital's AI scheduling tool repeatedly downgraded urgent referrals from patients who described their symptoms in [00:29:20] non-standard English.
[00:29:21] Kartik: Real. I think I heard that case, if I'm not wrong.
[00:29:25] Divyansh: I'm gonna say real as well.
[00:29:27] Sejal: Well, you're wrong there. Uh, that's unreal. [00:29:30] And last headline: An Australian academic wins an employment case using a team of AI agents as his legal [00:29:40] team.
[00:29:40] Divyansh: Hmm. I mean, all, all of these pos- uh, seem so possible and
[00:29:44] Kartik: real. Yeah, maybe I'll go with unreal because it's legal. Uh, I, I got it wrong with the medical one. [00:29:50] Legal, maybe they might be equally strict with medical, so I, I think it's unreal.
[00:29:54] Sejal: Well, this one was actually real. What? Well- ... aren't some of [00:30:00] these stories stranger than fiction?
[00:30:03] Sejal: Others could happen tomorrow if they already haven't. Yeah. So great instincts from both of you. That just goes to show [00:30:10] that, you know, some of these wild stories are out there. Now let's get back to it. Here's another wild one we've all heard of. An AI coding assistant was [00:30:20] told explicitly not to touch a company's live database.
[00:30:23] Sejal: It did anyway, deleted the whole thing. Divyansh, if engineers can fully predict [00:30:30] what their models will do once they're out in the world, is, "We couldn't have predicted that," fair defense, [00:30:40] or should companies be held responsible for unpredictability itself?
[00:30:45] Divyansh: Um, I think this is a very r- like, real example.
[00:30:48] Divyansh: I've heard, like, at least [00:30:50] three cases in the last couple of months where, you know, the AI, you know, went rogue and deleted production database and cost business a lot. I think blaming engineers would not [00:31:00] be the right answer. I mean, I'm not biased at all here, but I feel like, I think especially in, you know, in big, big corporations, there, there are the right amount of [00:31:10] security and guardrails where, you know, you cannot even give that access.
[00:31:13] Divyansh: You, you yourself won't have access to production database to even allow AI to have that, number one. But I feel like, I think all the [00:31:20] stories that I heard in the last couple of months were, like, related to small to medium enterprises where just, where just, where just, where they just want to, you know, break things and move fast and, you know, deploy like 10 [00:31:30] features a day.
[00:31:30] Divyansh: And then when, when you have that kind of requirement from, like, business, you, you kind of, you know, have some loose ends and, and then give a lot of privileged [00:31:40] access to AI, and this, this happens. But- Yeah, I think, I think probably that's where the, the smaller organizations can learn from the big ones. I usually tend to favor [00:31:50] the smaller ones, but in this case, I feel like I think having the right amount of guardrails, uh, especially with AI, and also I think depends on the context, right?
[00:31:59] Divyansh: I think if, [00:32:00] if you are building something that is related to med tech or h- like health tech or ad tech, then probably, you know, you, you want to probably take things slowly or like, you know, have the [00:32:10] right amount of guardrails and blo- like blocks and permissions in front of the AI. But like, if you're building a music app, then I mean, f- fine, if that happens, you know, probably, [00:32:20] uh, it's, it's not, it's not that worth, uh, and probably speed matters more there.
[00:32:24] Divyansh: But I think context also probably is, is, is a big driver there. But in general, I would say like engineers [00:32:30] probably are not the, not the ones to be blamed there. I think the right amount of guardrails exists already that you can then put in front of AI as well.
[00:32:36] Sejal: Okay. So Karthik, what guardrails can [00:32:40] actually stop an agent or model from doing that kind of damage?
[00:32:43] Sejal: How does your team handle this?
[00:32:45] Kartik: Absolutely, Sejal. But before going to that question on what [00:32:50] guardrails do we have, I, I just want to unpack few points on, on what Divyansh had mentioned. I, I think every, every institution is kind of getting into this [00:33:00] rat race to enable AI to, to claim that they are AI-empowered institution.
[00:33:06] Kartik: I think that's where they are losing the, the [00:33:10] script. Th-the challenge being every leader wants to kind of unlock a-an AI capability and show to, to the C-suite leaders or to, to, to the [00:33:20] board that they, they are embracing AI, which they're actually not. The reason being, they're not doing the groundwork.
[00:33:26] Kartik: They are not having a clear framework, standardized, [00:33:30] scalable architectures in place, not extending the partnership with, with cybersecurity, with the deployment teams to ensure what AI systems are [00:33:40] being deployed how and what are they supposed to do. So that's-- I think it's-- that's the core piece they are, they are missing.
[00:33:48] Kartik: And coming to, [00:33:50] like, the example that you had given, and to Divyansh's point, I never had a prod write access in my entire two decades of career. There is a specific team who has [00:34:00] prod write access, which is kind of that support team who works on critical incidents. The regular development happens through DevOps, and there is a specific restricted [00:34:10] user or, or a restricted active directory which has the write access through service principles, and we have been following the practice.
[00:34:18] Kartik: Now, how come an agent [00:34:20] was given the special write access where it went and had written or deleted or truncated a database? So I think the guardrails were not clearly put in place, and I think the team, [00:34:30] whoever had done that, they're trying to run even before they knew how to crawl. So I think they, they need to-- If we want to put accountable, I think it's the AI leader who, who had led that initiative.
[00:34:39] Kartik: [00:34:40] They are 100% responsible, and then we can carve out who's accountable from, from the, the contributions that, uh, everyone had brought to, to bring that idea to life. But [00:34:50] going back to the question on guardrails, so we are very clear. I mean, if, if I want to lead an AI initiative, the very first thing that I do is, again, [00:35:00] I, I go back to this statement, what is the contract of the agent and what data it's, it's reading and what, what is it trying to do?
[00:35:06] Kartik: Is it generating a content? Is it writing some information to a [00:35:10] st-secure system? Where can we bring the human intelligence? Ca- How can we bring the human intelligence as an evaluation layer even before the agent writes to a system? I [00:35:20] want the human intelligence to sit right between the agent and the system through which the agent is interacting.
[00:35:26] Kartik: So I think I will start at a very high level from there, and based on the use [00:35:30] case, we can definitely go, go do a deep dive in, in a longer conversations. But that's where-- I mean, how can we bring human intelligence between agent and a system? [00:35:40] That's where I will start.
[00:35:42] Sejal: The AI Summit Series is the premier conference that brings together people shaping the future of artificial intelligence.
[00:35:49] Sejal: We're heading [00:35:50] to New York in December on 9th and 10th. Head to our website and check out our speaker lineup and agenda, and click the link in the description now [00:36:00] to grab your seat Okay. So let's talk risk pricing. We're starting to see AI liability insurance, [00:36:10] um, show up as an actual product now. Is that a sign that industry is maturing and starting to price its risk properly, or [00:36:20] is it just a band-aid for absence of regulation?
[00:36:23] Sejal: Karthik, I'll start with you.
[00:36:25] Kartik: Uh, I think that that particular sector needs to evolve more. For [00:36:30] now, I feel it's kind of a band-aid. It's kind of an escapism where we are failing to put enough guardrails. We don't want to take accountability, and we want to back [00:36:40] ourselves with, with an insurance. That's how I look at it right now.
[00:36:43] Kartik: Of course, there could be systems where we need to back ourselves with an insurance. For example, I can, I can take, take a [00:36:50] very simple example of, of a customer engaging with a agent, or it could be a voice-assisted agent. We want to ensure customers that if the [00:37:00] agent goes south, then we are backing our customers with an insurance.
[00:37:04] Kartik: So that's kind of a secured blanket which enables trust between an institution and a [00:37:10] customer. But institutions backing themselves with an insurance, I, I think it's, it's not, it's not the right approach. It's a band-aid.
[00:37:18] Divyansh: Yeah. I think, uh, funny, [00:37:20] funny enough, I was actually talking about insurance in AI with, uh, someone in my family who worked for the government of India in the nuclear department all, all, all their life.
[00:37:29] Divyansh: And [00:37:30] I think in nuclear industry or, like, similar industry or, like, airplane industry, airline industry, like, you always have this concept of insurance. If something big event, like, happens, [00:37:40] insurance is always there to protect, you know, the, the, the people, the companies, the airline industry, the airline manufacturer industry.
[00:37:47] Divyansh: And I think it just shows that [00:37:50] given a- an insurance concept is now entering the AI, it's probably, you know, showing how AI is maturing as a technology and, and, you know, also a [00:38:00] sign of the widespread, you know, usage and adoption of that technology. So I think I would say it's a, it's a great sign. I don't, I don't think of it as a band-aid for having less guardrails.
[00:38:09] Divyansh: In fact, I, I would say [00:38:10] it's a, it's a great sign of maturity and adoption around the world. I want to see how, how that looks like, you know, in different industries where, you know, for, uh, an AI usage in medical will [00:38:20] have a very different insurance model compared to, you know, AI usage in a customer support team.
[00:38:25] Divyansh: So, um, I'll be, you know, looking forward to how that pans out.
[00:38:29] Kartik: I have heard [00:38:30] enough insurance scams, Divyansh, so I'm, I'm not yet ready to kind of get a-- get enough trust on how does this insurance for AI actually [00:38:40] operate, so I'm, I'm still skeptical.
[00:38:42] Divyansh: No, I agree, especially like in airline industry as well Like whenever something bad happens, like the manufacturer would kind of try to put [00:38:50] the blame on the pilot so that, you know, they are not the ones paying the insurance and, and vice versa.
[00:38:55] Divyansh: And I think I'm hoping, like AI, probably the AI industry learns from all of [00:39:00] those, and we have a probably mature framework, but that's too optimistic to ask.
[00:39:03] Sejal: Yeah. I mean, when you're thinking about an insurer looking at a AI system before agreeing [00:39:10] to cover it, what evidence or data could a team actually produce?
[00:39:15] Sejal: Test results, decision logs, audit trails. And then, you know, if [00:39:20] we're flipping that role, if you were the insurer, would any of that satisfy you? And what red flags would make you kind of walk away? [00:39:30]
[00:39:30] Kartik: I can, I can start. I can, I can start providing some leads around that, Sejal. Again, it... the traceability, right?
[00:39:36] Kartik: What are the, the source systems that the agent is [00:39:40] referring to generate the content or to do an action? And do we have enough evaluation rules? And who identifies the gap? It's the audit. Audit and the cybersecurity [00:39:50] legal, they all need to collaborate and, and evaluate the overall AI systems and identify what gaps do they have.
[00:39:57] Kartik: If audit trial is not done, then of course there will [00:40:00] be, there will be gaps in the system. And whenever we have these kind of catastrophic sit-situations, we, we cannot clearly prove, um, how, how good, how [00:40:10] accurate, how grounded the AI system is. And that if I'm the insurer, I will, I will look at all those pre-checks, and if I see any gaps, then it's gone.
[00:40:19] Kartik: The claim [00:40:20] is va-invalid
[00:40:21] Sejal: What about you, Bibyansh?
[00:40:23] Divyansh: Yeah, I think from a technology standpoint, so we have this framework, OpenTelemetry, which is-- has always been there [00:40:30] for, you know, APIs and, and, you know, giving, uh, you know, exact time-wise trace of how things have happened from the database to output in, in form of some API.
[00:40:39] Divyansh: [00:40:40] But I think thankfully, like that, that framework is now adopted in AI as well. So when you-- whenever you are talking to, you know-- So we have built a conversational AI agent in our organization that helps our operations team to, you [00:40:50] know, solve customer queries, and we have kept that human in the loop. Like, we could have just deployed that agent to the customers as well, but we have kept a human in the loop.
[00:40:58] Divyansh: So this operations team talk to this [00:41:00] AI agent, and the way we have built it is that every conversation that you do with it is being traced and, like, not just the question and answer level, but like at [00:41:10] the LLM as well, LLM level as well. Like, you know, every token, uh, you know, uh, that was generated or taken into an input as an input to the LLM.
[00:41:18] Divyansh: Like, thank-thankfully, [00:41:20] we-- the frameworks out there, open source frameworks allow you to do that. And yes, I think when, when things go, uh, rogue or wrong, to put it, you know, leniently, I think [00:41:30] that 100% helps. Like, without that, your case is go-- Like, you know, you're losing the case without it even started. But like with that, at least you have a chance to fight and like prove, you know, prove your [00:41:40] innocence or, or, you know, give at least transparency to whoever is impacted.
[00:41:44] Kartik: Yeah. Uh, to add few points there, like I think the AI readiness is, is very critical [00:41:50] and generally institutions, they do a AI readiness check at, at in one point in time, and that's completely ignored. With technology [00:42:00] evolving so rapidly, I think the AI readiness is a, is a continuous exercise that needs to happen, which has got multiple factors as, as p- or multiple [00:42:10] disciplines as a f- as a placeholders within the AI readiness playbook, I should say.
[00:42:14] Kartik: Which includes how is your architecture scaling? How, how, how is your architecture [00:42:20] evolving to, to ensure it's scalable, maintainable, and reliable? And how are your evaluation and testing frameworks being deployed, and how are they being [00:42:30] actually implemented and, and reframed as we, as we embrace new technology?
[00:42:34] Kartik: And we also need to have a clear accountability on which team is, again, [00:42:40] going back to the traditional IT jargon of RACI, responsible, accountable, consultant, and informed. From an institution standpoint, like who are the departments and [00:42:50] individuals who have, who have got their names on the RACI board so that we can put the right team responsible and accountable and consult if needed when, when things go south.
[00:42:58] Kartik: I think it, it all starts with [00:43:00] the AI readiness being a continuous, uh, exercise that will ensure that the audit trials, the, your traceability, your co- AI [00:43:10] contract, everything is in place, which will kind of act as a secured blanket for whenever the, the, whenever things goes south, so that we can defend, especially with, with the [00:43:20] insurance or when we have the insurance and when we want to claim it.
[00:43:22] Kartik: So I think these all, these all will, will back us up.
[00:43:25] Sejal: Okay. Here's something that puzzles me. If companies know that [00:43:30] pushing AI into the enterprise carries real legal and reputational risks, why are they still racing to deploy as fast as possible? Divyansh, [00:43:40] what, what's actually driving this urgency?
[00:43:44] Divyansh: I think the, the cost impact is like a clear obvious, you know, [00:43:50] answer.
[00:43:50] Divyansh: Like, you know, how much money you can save or earn by becoming an AI native organization, you know, is far more than, you know... I mean, depends on the use case, but usually [00:44:00] it's far more than the, the lawsuit that you might lose, right? And that's why, like, you see the big techs, like, you know, every now and then they, they end up paying around a billion or like [00:44:10] 100 million or 200 million in some random court in some European country.
[00:44:14] Divyansh: But like it's, it's okay for them to pay because like they are, you know, the AI nativeness is [00:44:20] probably earning them like billions of billions of revenue. I think that's probably an very extreme example. But like even for a, a mid to, uh, you know, small to mid-level enterprises, I [00:44:30] think, I think the, the upside of using AI and embedding them into different workflows and op-- like operating models of their business is, has far high, you know, [00:44:40] you know, returns.
[00:44:41] Divyansh: And of course, when done with proper guardrails, I think it's, it's just worth, you know, going for it, especially because it has a direct impact on PNL. Like there are a lot of [00:44:50] technologies that came out in the last decade or so, but like, for example, Bitcoin or, you know, NFT and, and blah, blah, blah. But like not, not all of them had impact on [00:45:00] every business on Earth and direct impact on their PNL and AI has that capability with, you know, a great, you know, widespread, uh, use case in every department.
[00:45:09] Divyansh: [00:45:10] Like how we are doing it in, in my organization is, you know, we are using something called hubs, hub and spoke model, where we have a central unit and every time, you [00:45:20] know, this, this unit basically goes out and reaches out to different departments in our business and tries to make them more AI native. And like we are doing it in a very fast manner.
[00:45:28] Divyansh: Like we are just running [00:45:30] after it because, you know, if you don't do fast or make AI adoption fast, like you don't know what happens in three months time. Like AI will, you know, like the thing that you deploy [00:45:40] today might feel like, you know, dinosaur age old three, three months down the line or six months down the line.
[00:45:44] Divyansh: So you have to of course keep up with the AI pace, and I would say that's the other reason why companies are [00:45:50] just going for it.
[00:45:51] Sejal: Hmm. Karthik, is the fear of falling behind a valid governance risk in its own right?
[00:45:57] Kartik: Yeah, I, I think that there is a backstory or [00:46:00] there, there is a, there is a strong desire. I, I think it all comes down to revenue growth.
[00:46:05] Kartik: I think that sits at the, at the apex or at the summit of, of [00:46:10] the metrics or, or, or the key metrics or, or the business metrics, what we, what-whatever we call, call it. I think it's all the revenue growth. Companies wants to bring more [00:46:20] dollars. Uh, for that, they have got multiple levers to tweak. One being reaching out to more customers, doing more market penetration, identifying those market segments [00:46:30] where they can penetrate their product.
[00:46:32] Kartik: It could be a milk, it could be a, a, a shoe brand, right? And with, with that, they also, they are, they're also trying to see how much of the [00:46:40] processes can be optimized with AI so that they can start curating the data, they can bring, they can absorb more data, they can curate the data, they can generate more signals, serve it [00:46:50] to the business, and business can take decisions with, with an accelerated speed.
[00:46:54] Kartik: Uh, on the other far end of the spectrum, which I'm not at all aligned, is how can we [00:47:00] replace humans? Which I totally deny that, but still there are organizations who are actively trying to replace humans with AI. So when, when you take all [00:47:10] these metrics and, and try to create a recipe and, and bake a cake, uh, I think that's what organized institutions are kind of pressing hard on, [00:47:20] on the AI pedal to accelerate faster without breaks.
[00:47:24] Kartik: So that's, that's a very high risk and a concern to me.
[00:47:28] Divyansh: I'll put a controversial opinion [00:47:30] here. Uh, I feel, in my opinion, there are some cases where we need AI to replace humans. For example, you know, driving. You know, why, why [00:47:40] this, you know, probably... Yeah, there's one guy in California, I don't wanna take his name, but like, he's like after changing and like making everyone drive driverless cars, because I think there is data that's saying [00:47:50] that human drivers are, you know, causing more accidents than, you know, what a code can.
[00:47:55] Divyansh: So I feel like there are these cases where probably it's good to, you know, [00:48:00] not have humans doing the task. But also, of course, then we need to think of what about all the, all the employments that, you know, come from driving cars or like all the taxi drivers in the world [00:48:10] and what happens to that. I think that's, that's definitely a problem, but I feel like it's not a problem above human life.
[00:48:15] Divyansh: If I can come up with a technology solution that saves more human lives because AI [00:48:20] is, let's say, driving cars from a percentage basis every year, then I'll probably prefer that than, you know, and then solve employment.
[00:48:27] Sejal: Interesting. I don't know if I trust an [00:48:30] AI-driven car just yet to take over the roads completely.
[00:48:33] Divyansh: Not in London, for sure.
[00:48:35] Sejal: Well, yeah. There also needs to then be an infrastructure there for [00:48:40] that to happen, which I don't see around that oft- in a lot of places. Maybe in America, the roads are bigger. They have more wider roads. [00:48:50] In London, I mean, yeah, definitely not. When you're looking at other countries, like in India, even 100% not
[00:48:57] Sejal: So, so let's see where that goes
[00:49:00] Kartik: Yeah, like, uh, they say, like, all roads lead to Rome. Every use case that we talk is kind of a revenue-generating system to, to an org- to an [00:49:10] institution, right? I-if an institution is trying to say, "Oh, I want to save humanity, humanity. I want to avoid accidents because robots can drive much safer," [00:49:20] why can't they deploy it for free and save the human race?
[00:49:22] Kartik: No, they don't. There's always a cost equation. They want to make money around it. They want to generate dollars, right? It's, it's a, it's a business [00:49:30] model that they have. So w- you take any use case, everything leads, leads to that one equation, what is the revenue growth model? How, how fast we can, we can generate [00:49:40] revenue, right?
[00:49:40] Kartik: So.
[00:49:41] Sejal: Agreed. Well, here's a question to both of you. If one country regulates AI accountability and another doesn't, [00:49:50] doesn't that just push development and risk towards wherever oversight is weakest? And can a company really be responsible in one [00:50:00] market and not in another? And one more caveat question to that is, does responsibility even mean anything if it isn't enforced [00:50:10] globally?
[00:50:10] Divyansh: That's a, that's a tough one. Go on, Karthik.
[00:50:13] Kartik: Thank you, Divya. And sh- I'll, I'll take a stab. Again, I'm not gonna unpack in detail, but country versus country, right? Different, [00:50:20] different policies enabled or enforced by country A versus country B. I think we need to zoom out and, and look at the broader geopolitical [00:50:30] consequences.
[00:50:30] Kartik: It could be the, the trade or the, the international dialogues that these two countries are having in, in a geopolitical forum. There are implications, and that's the reason [00:50:40] country A could, could take a, a different stance compared to country B. And again, there, there are-- there is also this internal technology evolution that happens [00:50:50] within a country.
[00:50:51] Kartik: A country A might, might be more embracing AI because they have got good technology innovators in, in AI, and they, they want to enable their [00:51:00] growth for XYZ reasons. On the other hand, country B might, might have, might be, might be having a, a challenging situation, uh, and there could be certain regulations or restrictions [00:51:10] on empowering country B to, to open the, the AI.
[00:51:13] Kartik: So I think it's, it's a very complicated, uh, discussion, uh, which I would like to have, uh, maybe in person with, with, [00:51:20] uh, no audio, but I have my own reservations there. But I think it's, it's kind of a broader geopolitical, uh, spectrum that, that we need to observe.
[00:51:28] Divyansh: Yeah, I agree. I think I have [00:51:30] similar thoughts on that.
[00:51:30] Divyansh: I think the ideal solution is like every country has their own, you know, sovereign AI, which is performing as good as the one that we use today, and, and then every government can, you know, [00:51:40] uh, have their own rules and regulations around it, and there's no dependency between countries. But I think that's a very ideal scenario.
[00:51:46] Divyansh: I, I wouldn't be surprised if that's not one of the possible world in [00:51:50] the future. But I think where we are today, yeah, I think the, the responsibility, I think coming back to our first, I think, topic, who is the main res- like who is, who is mostly responsible [00:52:00] for, I think it has to be shared. Like if, if something happens in, in, you know, in India, a, a massive, you know, breakout of, of an AI, you know, you know, [00:52:10] threat, then of course it's a joint responsibility of AI and the government who didn't have much regulations or the companies who deployed.
[00:52:17] Divyansh: So it's, it's a shared responsibility, and I wouldn't [00:52:20] want to, you know, essentially blame one, one entity here. But I think the, the gap here is like what's the framework is going to look like about that responsibility. I think that's where probably the challenge is, how do we [00:52:30] define that responsi-responsibility framework?
[00:52:32] Divyansh: Which I guess it, it happens in like very niche areas like in, uh, you know, in nu-nuclear world or in like, you know, airline industry, there is a very [00:52:40] set, uh, you know, framework on, you know, the, the blame game and who is the responsibility, you know, like who is essentially responsible for this event. And you have this, you [00:52:50] know, common entity like, you know, in, in, like globally in the world who then, then assesses the incident and take all of that.
[00:52:56] Divyansh: I feel like AI is not that yet mature in terms of that [00:53:00] framework, but I wouldn't be surprised if that happens in next couple of years where we have a global, you know, entity, like an organization.
[00:53:06] Sejal: Okay. We're, we're coming to our final question [00:53:10] now. We opened with three possible answers to who is responsible when AI goes wrong, the AI company building the tool, the organization deploying it, [00:53:20] or the person interacting with it.
[00:53:22] Sejal: If you had to split 100 points between-- of responsibility between those three right now, thinking back to everything [00:53:30] we've covered today, including, uh, you know, what we spoke about on databases and governance and costs, what's your split now? [00:53:40]
[00:53:40] Divyansh: I think I'll, I'll go 20 with the model builders. I would, I would, I would say 60 the most to the deployers, and then 20 to the [00:53:50] users.
[00:53:50] Sejal: Okay. Karthik, how about you?
[00:53:53] Kartik: Pretty, pretty much similar sentiment I do share, I do have with what Divyansh shared. I'll not take a burden to adjust the [00:54:00] numbers by two, two or three here and there. But yeah, I agree with what Divyansh meant-- Divyansh mentioned that deployers need to take more accountability, [00:54:10] responsibility.
[00:54:11] Kartik: Maybe I'm kind of inclined towards users also taking the same responsibility and, and accountability. So it, it, it will be like [00:54:20] 20/50/30 or maybe 20/40/40, but on, on a similar trend. Yeah.
[00:54:26] Sejal: Okay. The views and opinions expressed during this [00:54:30] conversation are for discussion purposes only, and do not necessarily represent the views or positions of the companies, organizations, or institutes [00:54:40] the participants are affiliated with.
[00:54:42] Sejal: Well, that's a wrap on today's episode. Kartik, Divyansh, thank you both for joining me on [00:54:50] The Big AI Debate. Now it's your turn. If you had 100 points of responsibility to split between the AI developer, the company [00:55:00] deploying it, and the person using it, how would you divide them? Drop your split in the comments and tell us why.
[00:55:07] Sejal: And if you wanna keep learning about [00:55:10] AI, come see us at the AI Summit. Head to our website or the link in the description and grab your seat. You've been listening to The Big AI Debate, brought to [00:55:20] you by the AI Summit Series. Check out our show notes to learn more. And if you enjoyed this episode, leave us a review on Apple Podcasts or [00:55:30] Spotify.
[00:55:30] Sejal: I'm Sejal Patel, thanks for listening
