One in five Australian workers said the most significant artificial intelligence (AI) error they encountered at work caused a customer service issue, according to new research from Appian (Nasdaq: APPN), raising questions about the risks of relying on AI in customer-facing operations.
The study found that 60% of Australian workers had personally identified an AI mistake at work, while 13% reported that the most significant error they encountered caused operational disruption. Almost half said their most serious experience with an AI error had consequences greater than a minor inconvenience, including financial, compliance or reputational impacts.
For retailers increasingly using AI to support customer enquiries, order management and other business processes, the findings highlight the importance of ensuring mistakes can be identified and corrected before they affect the customer experience.
Despite these risks, more than one in five workers said employees in their organisation are discouraged from questioning AI recommendations or decisions, while just 14% fully trust the AI-generated outputs used at work.
Charlie Hutchinson, SVP Asia-Pacific and Japan at Appian, said employees need to feel confident questioning AI outputs, particularly when mistakes could affect customers.
“Every visible error gives workers another reason to question whether AI should be trusted with more consequential work,” said Hutchinson. “Employees are often the last line of defence before an incorrect output reaches a customer or becomes part of a business decision. If they are discouraged from speaking up, organisations remove one of their most important safeguards.”
Human oversight remains critical
The research found that 94% of Australian workers opposed allowing AI to make decisions affecting employees or customers without limits or human oversight.
Two-thirds (66%) believed these decisions should either require human review and approval or remain entirely in human hands, while a further 28% supported restricting AI to low-risk decisions.
Separate global research from Harvard Business Review Analytic Services, sponsored by Appian, found that while 92% of respondents agreed AI agents need rules-based guardrails, only 48% said their organisation had defined them.
As retailers expand their use of AI, establishing clear processes for reviewing automated recommendations and escalating decisions to employees could help reduce errors and protect customer relationships.
“Australian workers are rejecting unchecked AI. Businesses will only realise the full value of AI if employees build trust, are willing to use it within critical processes, and confident enough to act on its outputs. That requires governed processes where AI recommendations and autonomous agent outputs can be systematically reviewed, validated, or overridden and human judgement remains part of important decisions,” concluded Hutchinson.
