Morning Coffee with Srini & Ariyan
AI, Technology and the Future
Imagine applying for your dream job. Your qualifications are strong, your experience is relevant, and you believe you have a fair chance.
Then an email arrives.
Your application has been rejected.
You ask the company why. The answer is simple: an AI system evaluated your application and decided you were not the right candidate.
But when you ask what went wrong, nobody can give you a clear explanation.
The system processed your information, compared your profile with other applicants, and produced a result. The company trusts that result, but you have no meaningful way to understand or challenge it.
Would you accept the decision?
Or would you insist on knowing why an algorithm was allowed to influence your future?
Good morning, everyone! ☕ Welcome to another episode of Morning Coffee with Srini & Ariyan, where we explore the questions behind the technology changing our lives.
Today, we're asking a question that will become increasingly important as AI systems take on more responsibility: Should we trust an AI decision when we cannot fully understand how it was made?
When AI Knows the Answer, but We Don't Know Why
Srini: Ariyan, let me start with something simple. If an AI system makes a decision that turns out to be correct, but nobody fully understands how it reached that conclusion, should we trust it?
Ariyan: I wouldn't answer with a simple yes or no, Srini. I'd want to know what the system was deciding, how reliable it had been, and what could happen if it made a mistake.
Srini: But humans make decisions without explaining every detail of their thinking, too. A doctor might recognise a pattern through years of experience. A mechanic might hear an engine and immediately suspect a particular problem. Why should AI have to explain everything?
Ariyan: That's a fair challenge. We shouldn't automatically reject an AI system just because its internal calculations are complicated. Sometimes it may identify patterns that people miss.
But imagine that same system deciding whether someone receives a loan, gets shortlisted for a job, or qualifies for an important service. If the decision is wrong, the person affected needs a way to question it.
Srini: So you're saying the problem isn't simply that we don't understand everything about AI. It's that we might give it authority without knowing whether its decisions deserve our trust.
Ariyan: Exactly. A useful prediction is not automatically a fair decision. And a confident answer isn't proof that the system is right.
That distinction matters more than it might seem.
The Black-Box Problem: What's Happening Inside AI?
Modern AI models learn patterns from enormous amounts of data. Their internal calculations can involve billions of parameters interacting in complicated ways. Researchers can study these systems, test their behaviour, and develop methods to explain particular outputs, but a complete explanation of every decision is not always available.
This challenge is commonly called the black-box problem.
Consider an AI tool that helps doctors examine medical images. It identifies a pattern that suggests a possible disease, perhaps one that is difficult to notice immediately.
That could be extremely useful. But should a doctor accept the result without examining the evidence?
Of course not. The finding needs to be considered alongside the patient's symptoms, medical history, other test results, and professional judgement.
The same concern applies when AI is used in hiring, financial services, education, and public services. The higher the consequences of a decision, the more important it becomes to understand the system's limitations and establish a way to challenge its output.
There is also a subtle problem: an explanation can sound convincing without proving that the original decision was correct.
A system might highlight certain factors that influenced a result, but that does not automatically establish that the decision was fair, that its data was accurate, or that it reached the right conclusion.
Understanding an answer and verifying an answer are two different tasks.
A Real-World Concern: Can AI Make Hiring Less Fair?
AI-assisted recruitment provides a useful example of why these questions matter.
Employers may use automated tools to screen applications, rank candidates, or help make employment decisions. Such tools can save time, particularly when organisations receive large numbers of applications.
But there is a potential problem.
If a system learns patterns from historical hiring data, it may reproduce unfair patterns contained in that data. A ranking that appears objective could disadvantage qualified applicants if the system relies on unsuitable signals or reflects past discrimination.
This is not merely a hypothetical policy concern. The U.S. Equal Employment Opportunity Commission has specifically addressed the risks that AI and other algorithmic tools can create in employment decisions, including the possibility of discriminatory barriers to jobs.
Its guidance highlights an important principle: using technology does not remove an employer's responsibility to comply with applicable anti-discrimination laws.
Now imagine two applicants with similar qualifications. One is shortlisted, while the other is rejected. If an AI tool influenced the decision, the rejected applicant should not automatically be told that the algorithm is objective and therefore cannot be questioned.
The company should be able to examine how the tool was used, whether the process produced unfair outcomes, and whether the decision needs to be reviewed.
Efficiency is valuable, but efficiency alone does not make a decision fair.
What If AI Is More Accurate Than a Human?
Srini: Let me challenge your argument, Ariyan. Suppose a particular AI system consistently makes more accurate predictions than human experts. If we insist on understanding every decision, couldn't we end up rejecting a tool that actually helps people?
Ariyan: Yes, and I think that's an important point. We shouldn't confuse a decision we don't fully understand with a decision that's necessarily wrong.
If an AI system has demonstrated strong performance under rigorous testing, it could be valuable even when some of its internal processes remain difficult to interpret.
Srini: Then perhaps explainability isn't always the most important thing.
Ariyan: It depends on the situation. For a low-risk task, such as sorting personal photographs, a high level of internal explanation may not be necessary. For a decision that could affect someone's health, freedom, livelihood, or financial security, we need stronger safeguards.
And accuracy itself needs careful examination. Accurate according to which measure? Tested on which people? Under what conditions? Does the system perform equally well when the situation changes?
Srini: So a good benchmark isn't the same as a guarantee that the system will work for everyone.
Ariyan: Exactly. We need to look at accuracy, reliability, fairness, safety, and accountability together.
That is a much stronger standard than simply asking whether the AI performs better than a person in one test.
Explainability Is Important, but It Isn't Everything
The U.S. National Institute of Standards and Technology (NIST) offers a useful framework for thinking about this problem.
Its AI Risk Management Framework identifies several characteristics associated with trustworthy AI, including validity and reliability, safety, security, accountability, transparency, explainability, privacy, and fairness.
The framework also recognises that these characteristics can involve trade-offs. Improving one aspect of a system does not automatically make the entire system trustworthy.
For example, a model might provide a more understandable explanation while still making an inaccurate prediction. Another model might achieve strong predictive performance but require additional safeguards to ensure that its use is fair and appropriate.
The goal is not to maximise a single quality and ignore the rest. It is to evaluate the system in the context of the task it is being used to perform.
This gives us a more practical way to approach AI decisions.
Instead of asking only, "Can the AI explain itself?", we should also ask:
- Has the system been tested under realistic conditions?
- How does it perform when information is incomplete or unusual?
- Could its results disadvantage particular groups of people?
- Can an error be investigated and corrected?
- Is there a clear way to challenge a consequential decision?
These questions turn trust from a feeling into something that can be examined.
Who Is Responsible When AI Gets It Wrong?
Srini: Let's imagine a company follows an AI recommendation, and someone suffers because the decision was wrong. Who should be held responsible—the developer, the company, or the person who approved it?
Ariyan: We would need to examine the circumstances. Did the developer fail to address a known limitation? Did the company deploy the system without appropriate testing? Was the information inaccurate? Did a human ignore a warning?
The answers could differ from one case to another.
Srini: But that creates another danger, doesn't it? Everyone could point at someone else.
The company says the AI made the recommendation. The developer says the company used it incorrectly. The employee says the system was approved by management. Meanwhile, the person affected is left waiting for an answer.
Ariyan: And that's why responsibility needs to be established before deployment, not invented after a failure.
Organisations should decide who is accountable, what records need to be kept, how errors will be investigated, and when a human must intervene.
They also need a practical process for correcting mistakes and allowing people to challenge important decisions.
Srini: So even if AI becomes better than humans at certain tasks, it shouldn't become an excuse for humans to avoid responsibility.
Ariyan: Exactly. We can delegate work to technology without delegating away our responsibility for how that technology is used.
That may be one of the most important principles of responsible AI.
Can We Build AI That Explains Itself?
Researchers and technology companies are working on ways to make AI systems easier to evaluate and understand. There is no single solution that makes every complex model completely transparent, but several approaches can help.
Explainability tools can investigate which inputs or patterns may have influenced a particular output.
Independent evaluations can test how a system behaves across realistic situations, including unusual cases and potential failure modes.
Decision records and audits can help organisations investigate what happened when an important decision goes wrong.
Human review and appeal processes can give qualified people the authority to question, override, or correct consequential decisions.
Clear communication about limitations can help users understand where a system is reliable, where uncertainty remains, and when expert judgement is needed.
These measures are useful, but none is perfect. Explanations may be incomplete, evaluations may miss unusual failures, and human reviewers can make mistakes.
The goal is not necessarily to make every calculation inside every model understandable to every person. It is to ensure that AI systems can be tested, monitored, challenged, and controlled appropriately for the tasks they perform.
Should Every AI Decision Require Human Approval?
This is another question worth considering.
If a system is helping someone organise files, recommend a film, or sort photographs, requiring a person to approve every action would be unnecessary.
But consider an AI system involved in a medical decision or a financial application. In such cases, the consequences of a mistake can be much greater.
The appropriate level of oversight should reflect the risk.
For lower-risk tasks, automated decisions may be reasonable when the system is reliable and users have appropriate controls. For high-impact decisions, stronger testing, meaningful human review, documentation, and routes to appeal may be necessary.
Human oversight must also be genuine. Asking an employee to approve hundreds of recommendations without enough time or information to examine them does not provide much protection.
A person should have the ability to understand the relevant evidence, question the system, and intervene when necessary.
The aim is not to slow down every AI application. It is to make sure that the safeguards match the potential consequences.
One Last Thought Over Coffee
Srini: Ariyan, I think I've changed how I'm looking at this question. At first, I thought the biggest problem was that AI might make decisions we couldn't understand.
Now I'm wondering whether the bigger problem is that we might stop questioning those decisions simply because the technology appears intelligent.
Ariyan: I think that's worth thinking about, Srini. We don't need to understand every internal calculation to use AI responsibly. But we do need evidence that it works, safeguards against foreseeable harm, and a way to challenge decisions when something important is at stake.
Srini: So the future shouldn't be about blindly trusting AI—or blindly rejecting it.
Ariyan: Right. It should be about knowing when to rely on it, when to question it, and when a human needs to take responsibility.
Perhaps the real test of intelligent technology is not whether it can always explain itself perfectly. It's whether the people who build and use it are willing to examine its decisions honestly.
As AI becomes more capable, that responsibility will become more important, not less.
Because the ability to make a decision and the right to make that decision are not necessarily the same thing.
Your Turn: Would You Trust an AI You Couldn't Fully Explain?
Imagine an AI system makes a decision that affects your job, health, or financial future. Its results look accurate, but nobody can clearly explain how it reached its conclusion.
Would you accept the decision because the system has a strong track record? Or would you insist on a meaningful explanation and a way to appeal?
There may not be one answer that fits every situation. But the question is worth discussing before AI becomes an even bigger part of our lives.
Share your thoughts in the comments. Should trust in AI depend more on its accuracy, its explainability, or our ability to hold someone accountable?
This has been Morning Coffee with Srini & Ariyan. Until next time, stay curious, keep asking questions, and never stop thinking for yourself.
Further Reading
- "NIST — AI Risk Management Framework" (https://www.nist.gov/itl/ai-risk-management-framework)
- "NIST — AI Risks and Trustworthiness" (https://airc.nist.gov/airmf-resources/airmf/3-sec-characteristics/)
- "U.S. EEOC — Artificial Intelligence and Algorithmic Fairness" (https://www.eeoc.gov/newsroom/eeoc-launches-initiative-artificial-intelligence-and-algorithmic-fairness)
.webp)
.webp)
.webp)
0 Comments