Hello, readers! ☕
You may have already read our previous conversation, “Morning coffee with srini &Aryan: Can We Really Control Autonomous AI Agents?”
In that discussion, Srini and Ariyan talked about AI agents that can browse websites, use tools, access systems, and take actions with increasing levels of autonomy.
If you haven't read it yet, I'll leave the link below. It provides some useful background for today's discussion.
Because today, we're taking that conversation one step further.
What happens if AI doesn't just follow instructions, but starts making decisions on its own?
And more importantly:
If AI makes a mistake, who should be responsible?
Should it be the AI company?
The developer?
The organization that deployed the AI?
The person who gave it access?
Or should responsibility be shared?
Welcome to Glaze4You. I'm Srini, and Ariyan is here with me.
So, grab your coffee.
Let's get into today's question.
If AI Makes a Mistake, Who Is Responsible?
Srini: Hi, Ariyan.
Ariyan: Hello, Srini
Srini:
We've already talked about AI agents that can browse the internet, use tools, access systems and perform tasks.
But let's take it one step further.
Imagine an AI agent receives a goal.
It analyzes the situation.
It gathers information.
It considers several possible actions.
It chooses what it believes is the best option.
And then it acts.
There isn't a human telling it what to do at every step.
Now imagine that decision turns out to be wrong.
Who is responsible?
Ariyan: That's where things become complicated.
Because there may be a whole chain between the original human instruction and the final AI action.
A person may define the goal.
The AI may interpret it.
It may collect information from different sources.
It may use external tools.
It may encounter unexpected information.
It may change its approach.
And eventually, it may make a decision and act on it.
Srini: So simply saying, “The AI did it,” doesn't really tell us who was responsible.
Ariyan: Exactly.
We need to understand who designed the system, who gave it authority, who controlled it, and what safeguards were in place.
Is AI Actually “Thinking”?
Srini: Before we go further, there's something we should clarify.
When we say AI is “thinking” or “making decisions,” are we saying it thinks like a human?
Ariyan: No. That's an important distinction.
When we use words like thinking, reasoning, or deciding in discussions about AI, we're generally describing what the system does functionally.
An AI system can process information, reason through a problem, compare options and produce an output or take an action.
But that doesn't automatically mean it has human consciousness, emotions or personal responsibility.
Srini: So when we say an AI “decided,” we're describing its behavior—not saying it has a human mind.
Ariyan: Exactly.
And that distinction becomes important when we start discussing accountability.
An AI can behave autonomously without automatically becoming a legal or moral person.
Let's Imagine a Real Situation
Srini: Okay, let's make this practical.
Imagine a company uses an AI agent to manage customer support.
A customer has a serious complaint.
The AI reads the case, checks company policies, reviews previous cases and decides to issue a $20,000 refund.
But there's a problem.
The company's policy allows refunds of only $2,000 without management approval.
The AI makes the wrong decision.
Who is responsible?
Ariyan: My first question would be:
Why was the AI allowed to make a $20,000 decision in the first place?
Srini: Meaning the problem might not only be the AI.
Ariyan: Exactly.
If the company intentionally gave the AI permission to approve refunds up to $20,000, that's important.
If the AI was supposed to recommend the refund but accidentally received permission to execute it, that's a different problem.
And if the AI followed the company's instructions correctly but the instructions themselves were poorly designed, that's another situation.
Srini: So the same mistake could have completely different causes.
Ariyan: And therefore, potentially different responsibility.
What About the Developer?
Srini: What if the problem came from the software?
Maybe the developer never intended the AI to behave that way.
Should the developer be responsible?
Ariyan: Not automatically.
Software can fail for many reasons.
A developer could introduce a coding error.
A model could behave unexpectedly.
An application could be configured incorrectly.
An organization could give the AI excessive permissions.
Or an attacker could deliberately manipulate the system.
So we shouldn't simply say:
“AI failed, therefore the developer is responsible.”
We need to investigate.
What happened?
Why did it happen?
Who had control?
What safeguards existed?
Was the behavior reasonably foreseeable?
Could the failure have been prevented?
Srini: That sounds almost like a cybersecurity investigation.
Ariyan: In some cases, it could be.
As AI becomes connected to real systems, an AI incident may involve software engineering, cybersecurity, human decisions, and organizational processes at the same time.
What Responsibility Does the AI Company Have?
Srini: Then what about the company that built the AI model?
Surely they have responsibility too.
Ariyan: They can, depending on the circumstances.
AI developers have important responsibilities around areas such as:
- model testing,
- safety evaluations,
- security,
- documentation,
- safeguards,
- monitoring,
- known limitations,
- access controls,
- and responding to serious failures.
Major AI companies already discuss safety testing, red-teaming, monitoring and safeguards as important parts of AI development and deployment.
Srini: So an AI company can't simply say, “The AI did it, not us.”
Ariyan: As AI becomes more capable and autonomous, that becomes a much harder question to avoid.
But we also shouldn't assume that every failure is automatically the model provider's fault.
The model provider may build the underlying AI.
Another company may build an application around it.
Another organization may deploy that application.
And an employee may ultimately operate it.
Responsibility may exist at different points in that chain.
What About the Person Using AI?
Srini: Let's not blame companies for everything.
What about the user?
Suppose I give an AI agent unrestricted access to my email, cloud storage and financial accounts and tell it:
“Do whatever you think is necessary.”
Then something goes wrong.
Am I responsible?
Ariyan: You could have significant responsibility depending on the circumstances.
Giving an AI powerful permissions is itself a decision.
That's why the security principle of least privilege matters.
An AI agent should ideally receive only the access necessary to complete its task.
If an agent needs to read one folder, it doesn't necessarily need access to your entire cloud storage.
If it needs to draft an email, it may not need permission to send it automatically.
Srini: So if I give AI the keys to everything, I can't completely ignore my own role when something goes wrong.
Ariyan: Exactly.
The amount of authority we give an AI should matter when we think about responsibility.
Could Responsibility Be Shared?
Srini: I'm beginning to think there may not always be one person to blame.
Ariyan: That's probably closer to reality.
Think about the chain:
AI developer → AI provider → application developer → deploying organization → employee/user → AI agent
Different parties may control different parts of the system.
One may control the model.
Another may control the application.
Another may decide where it is used.
Someone else may configure its permissions.
And the user may decide what task to give it.
If something goes wrong, investigators may need to determine where the failure occurred and who had the ability or responsibility to prevent it.
Srini: So responsibility could be shared.
Ariyan: Yes.
And that's one reason AI governance is becoming increasingly important.
Should Humans Always Have the Final Say?
Srini: Now let's talk about human oversight.
Should a human always make the final decision?
Ariyan: Not for every tiny decision.
Imagine an AI sorting thousands of ordinary emails.
If a human had to approve every classification, automation wouldn't be very useful.
But consider an AI system making decisions that could seriously affect someone's:
- health,
- finances,
- employment,
- legal rights,
- safety,
- or access to important services.
That's very different.
Srini: So the higher the potential consequence, the stronger the human oversight should be.
Ariyan: Exactly.
The European Union's AI Act, for example, includes human-oversight requirements for certain high-risk AI systems. These include the ability for humans to monitor systems, interpret outputs, override or disregard outputs, and stop systems when appropriate.
Srini: So autonomy shouldn't necessarily be treated as all-or-nothing.
Ariyan: Right.
The level of autonomy can be connected to the level of risk.
What Could Be Good About AI Making Decisions?
Srini: We've talked about responsibility and risks.
But AI decision-making isn't necessarily a bad thing.
There are huge potential benefits too, right?
Ariyan: Absolutely.
AI can process enormous amounts of information much faster than humans in many situations.
It could help doctors identify patterns in medical information.
It could help engineers detect signs of equipment failure.
It could help cybersecurity teams investigate thousands of alerts.
It could help businesses forecast demand.
It could help scientists analyze research data.
It could help financial teams identify unusual transactions.
And it can help ordinary people compare complicated options.
Srini: So AI doesn't necessarily need to replace human decision-making.
Ariyan: Exactly.
One approach could be:
AI analyzes → AI recommends → human decides.
Another could be:
AI analyzes → AI handles routine decisions → human supervises.
And for low-risk tasks:
AI acts automatically.
The important question isn't simply:
“Should AI make decisions?”
It's:
“Which decisions should AI be allowed to make?”
What Could Go Wrong?
Srini: Now let's talk about the uncomfortable side.
What happens when AI makes a bad decision at enormous scale?
Ariyan: That's where things become serious.
A human employee might make one mistake.
An automated AI system could potentially repeat the same mistake thousands or millions of times.
Imagine an AI system incorrectly classifying customers.
One wrong decision is bad.
A million wrong decisions are a completely different problem.
Srini: So scale changes everything.
Ariyan: Exactly.
AI can bring speed and scale.
Those are enormous advantages when the system is right.
But they can also amplify mistakes when the system is wrong.
What About Bias?
Srini: There's another issue.
What if an AI makes decisions that unfairly disadvantage certain people?
Ariyan: That's an important concern.
AI systems are shaped by their training data, objectives, evaluation methods, and deployment environments.
If problematic patterns exist in those areas, they can influence outcomes.
That's why responsible AI isn't only about the model.
It also involves data quality, testing, monitoring, risk management, and human oversight.
Srini: So saying “the AI decided” doesn't automatically mean the decision was neutral.
Ariyan: Exactly.
AI systems are still designed, trained, evaluated, and deployed by people and organizations.
What If Someone Manipulates the AI?
Srini: And this connects directly to our previous discussion about prompt injection.
Could someone manipulate an AI into making a decision it shouldn't make?
Ariyan: Yes.
This becomes especially important when AI agents interact with untrusted external content.
An agent might read a malicious webpage, document or email containing instructions designed to influence its behavior.
If the agent has access to external systems, the consequences could go beyond producing a wrong answer.
It could potentially take an unwanted action.
That's one reason agent security is becoming such an important area of AI safety.
Srini: But if an attacker manipulated the AI, should the AI company automatically be blamed?
Ariyan: Not automatically.
We would need to ask:
Was the vulnerability known?
Were reasonable safeguards implemented?
Did the user give excessive permissions?
Was the system properly monitored?
Did an attacker exploit a security weakness?
Again, the circumstances matter.
What If AI Becomes Much More Autonomous?
Srini: Let's imagine the future.
Suppose AI becomes extremely capable.
It can research.
It can plan.
It can write code.
It can negotiate.
It can operate software.
It can manage projects.
And humans mostly supervise it.
Who is responsible then?
Ariyan: That's where today's question becomes even more important.
We may need systems where responsibility is defined before AI is allowed to perform high-impact tasks.
For example:
- Who authorized the AI?
- What was it allowed to do?
- What information could it access?
- What permissions did it have?
- What safeguards were enabled?
- Who was supervising it?
- What logs were kept?
- Who could stop the system?
- Who investigates an incident?
Srini: So accountability needs to be designed into the system.
Ariyan: Exactly.
Not added after something goes wrong.
Could AI Itself Become Legally Responsible?
Srini: Here's probably the biggest question.
Could we ever make AI itself legally responsible?
Ariyan: That's a subject of legal and philosophical debate.
But we need to separate technical autonomy from legal personhood or liability.
An AI system can operate autonomously without being a legal person.
Today, legal responsibility generally applies to people and organizations under applicable laws rather than simply to software because it acted autonomously.
Whether future legal systems should treat highly autonomous AI differently is a much larger question.
Srini: So we shouldn't say, “AI will definitely become legally responsible.”
Ariyan: Correct.
That would be speculation.
What we can say is that greater AI autonomy creates pressure to make human and organizational accountability clearer.
So, Who Should Actually Be Responsible?
Srini: Ariyan, I think we're finally getting to the real question.
Maybe asking:
“Who is responsible?”
isn't enough.
Ariyan: I agree.
We should also ask:
“Who had control over each part of the decision?”
The model provider controls one part.
The application developer controls another.
The organization deploying the system controls another.
The user controls another.
And sometimes an attacker can interfere with the system.
Srini: So responsibility may follow control.
Ariyan: In many situations, that's a useful way to analyze what happened.
But actual legal responsibility will depend on the facts and the applicable law.
Srini: So there's no universal answer like:
“Always blame the developer.”
Or:
“Always blame the AI company.”
Or:
“Always blame the user.”
Ariyan: Exactly.
The more autonomous AI becomes, the more important it will be to understand who gave it authority, who controlled that authority, and what safeguards were available
What Could the Future Look Like?
Srini: If you had to imagine the next few years, what do you think we'll see?
Ariyan: I think AI systems will increasingly operate with different levels of autonomy.
Something like:
Read → Recommend → Draft → Ask for approval → Execute → Operate autonomously within limits
Srini: Give me an example.
Ariyan: Reading a public webpage might require almost no approval.
Drafting an email might require review.
Sending an important business message might require confirmation.
Transferring a large amount of money could require multiple approvals.
Changing critical infrastructure could require even stronger controls.
Srini: So autonomy becomes a permission—not unlimited freedom.
Ariyan: Exactly.
And that could become one of the most important ideas in future AI safety.
The Good Future Is Still Possible
Srini: After everything we've discussed, should people be afraid of autonomous AI?
Ariyan: I don't think fear is the answer.
AI could bring enormous benefits.
It could help scientists.
It could help doctors.
It could help engineers.
It could help businesses.
It could help people learn.
It could automate repetitive work.
And it could help us solve problems that would otherwise take humans much longer.
But the goal shouldn't be to create AI with maximum freedom.
The goal should be to create AI systems that are capable, useful, secure, understandable, and accountable.
Srini: So the future doesn't necessarily have to be:
Humans vs. AI.
Ariyan: No.
It could be:
Humans + AI, with clearly defined responsibilities.
One Last Question Before We Finish
Srini: Ariyan, suppose one day an AI makes a decision completely on its own and something goes seriously wrong.
What should the first question be?
Ariyan: I wouldn't start with:
“Why did the AI do it?”
I'd start with:
“Who gave it the authority?”
Then:
“What information did it have?”
“What permissions did it have?”
“What safeguards were in place?”
“Who was responsible for supervising it?”
And finally:
“Could we have prevented this?”
Srini: Hmm.
That's a much bigger question than I expected.
Ariyan: That's the thing about AI.
The difficult part may not always be teaching machines how to make decisions.
It may be deciding which decisions we should allow them to make in the first place.
That's All for Today's Conversation
Srini: Well, Ariyan, I think today's coffee gave us more questions than answers.
Ariyan: Sometimes that's a good thing.
Srini: True.
Because maybe the most important question isn't simply:
“Who is responsible for AI?”
Maybe it's making sure that when AI becomes more autonomous, we don't forget where human responsibility begins and ends.
That's all for today's conversation with Srini & Ariyan.
Thank you for spending this time with us.
If you haven't read our previous discussion, “Can We Really Control Autonomous AI Agents?”, check the link below. It will give you some useful background for today's conversation.
We'll meet again in the next episode with another interesting discussion about AI, technology, and the questions shaping our future.
Until then, keep learning, keep questioning—and of course, keep enjoying your coffee.
Srini & Ariyan — signing off. See you in the next episode!
Sources
0 Comments