Madison, a server in New York City, greets every table by asking about each diner’s allergies. Lately, there have been some close calls. “Sometimes people will tell me they have a shellfish allergy, and I’ll come back, and they won’t ask me any questions,” says Madison, who asked that her last name be withheld to protect her identity. “And they’ll order a fish dish that comes with a broth that is made of shellfish.”
When she tells them a dish contains an allergen, she’s had guests push back, saying ChatGPT disagrees. “They just want to talk to ChatGPT,” Madison explains. “You’re allergic to shellfish, and ChatGPT says there’s no shellfish in this, and we are telling you, yes there is!”
Mass adoption of generative AI tools is rewriting the old adage that has long been a thorn in the side of restaurant and retail workers: “The customer is always right.” But what happens when the customer expects you to play along with something AI made up?
Beyond ignoring her warnings about allergens, Madison has noticed an increased reliance on generative AI among patrons at the Michelin-recommended Italian restaurant where she worked until recently. “The amount of times I’ve suggested, ‘Let me grab the sommelier for you,’ and they say, ‘No, I’ll just ask ChatGPT,’ or somms have been dismissed from the table because the guest has a preference for ChatGPT, that has been increasing quite a bit,” she says. She’s had customers request bottles that the restaurant doesn’t carry — or, in some cases, that don’t exist at all.
For Madison, these AI hallucinations exacerbate some people’s preexisting anxieties about wine. “Some people have always had an issue with sommeliers, thinking ‘These people are like used car salesmen trying to get us to spend way more than we want to on a bottle of wine,’” she says, “or there are people who get in their heads and think, ‘The sommelier is going to think I’m stupid.’” To “soften the blow” of AI hallucinations, Madison and her colleagues will say things like, “We have this really hyper-specific wine list, maybe it’s showing you a French wine, and we don’t carry French wine,” instead of telling the customer their AI chatbot completely made up a wine brand.
You’re allergic to shellfish, and ChatGPT says there’s no shellfish in this, and we are telling you, yes there is!
The AI creep is seemingly ubiquitous in customer service jobs now. The launch of Muse, Meta’s new AI agent, will likely exacerbate the issue — and the headaches for workers who have to field customers’ occasionally nonsensical requests. In reporting this story, I heard from movie theater workers, baristas, and Apple Store employees, all of whom have had customers ask for products or features that don’t exist. On Reddit and X, customer service workers are complaining about customers who take ChatGPT’s word as gospel.
Jason, a river ranger who works on federal lands in the Mountain West, says he’s noticed an increase in visitors arriving to the park with nonsensical AI-generated itineraries.
“My colleagues and I have explained that the AI hallucinated campsites or scheduled things out of geographical order, so the group would have to row their boats upstream some days (not possible) and float with the river current other days,” says Jason, who asked to be referred to by a pseudonym because he’s a federal employee. “The craziest part is that even with a ranger explaining this while pointing at a map, we’ve had multiple groups remain skeptical. A certain type of AI user is inclined to trust ChatGPT over rangers, maps, and printed guidebooks.”
Casey, an after-school program teacher in New York who asked to be referred to by a pseudonym, says she’s noticed many students’ parents, nannies, and other caretakers are now turning to ChatGPT or Claude for their questions on childhood development.
“We had one family that consulted ChatGPT to come up with a bedtime routine and sleep schedule for their kid, but their kid wasn’t getting quite enough sleep,” Casey says. Based on his behavior in the program, Casey estimated that the child needed two to three more hours of sleep a night, and told his parents this. “The parents just insisted that their ChatGPT was telling them that their schedule was fine.”
Some parents, she says, now see generative AI tools as experts whose knowledge surpasses that of childcare professionals. “It can be really hard to talk with somebody who has made their mind up,” Casey says, “because they trust this outside entity as being invaluable.” She’s also noticed more anodyne uses of the technology, like kids coming in with AI-generated coloring sheets.
Lauren Wilford, who has worked in food service for 15 years, has noticed AI creeping into her recent interactions with guests as well. It started with the menu: Someone would upload a photo of it to ChatGPT or Claude and ask it what to order, or for guidance on a wine pairing, instead of asking the server.
But now, it appears, some customers have dropped the intermediary altogether. Over the past year, Wilford’s restaurant has been getting calls and emails from agents trying to make reservations on users’ behalf. The rise of agentic AI makes these interactions that much more confusing.
Wilford was surprised when a hands-off client suddenly became demanding on the eve of a big event, making last-minute requests for podiums and AV equipment — neither of which were readily available at the Providence, Rhode Island, restaurant she manages — and eleventh-hour changes to the catering menu. But Wilford was even more taken aback when the clients arrived the next day and didn’t mention any of it. Though she can’t prove it, Wilford believes the requests may not have come from anyone at the company at all; she suspects they were the work of an AI agent.
“I’m sympathetic to the reasons people use AI agents,” Wilford says, especially for bureaucratic and administrative tasks. “But anything where there’s value to two humans seeing each other and working something out together, I think we’re not sensitive to what might be lost if we try to skip all of that stuff.”