AI’s web of lies -

AI, artificial intelligence, doesn’t know everything. In fact, it comes up with wrong answers sometimes. Like, a lot of the time. They’re called “hallucinations.” They’re not called “lies” because AI has no consciousness. Call it what you want, but AI does not always serve up the truth.

While many AI models hallucinate, in this article we’ll be looking at some times ChatGPT had some pretty severe hallucinations, ones that had dire consequences.

ChatGPT is super useful for lots of things. I use it almost every day for one thing or another, especially when the only information I have is kind of vague. Things like:

ChatGPT, what’s the movie where Sandra Bullock and River Phoenix are trying to make it as singers in Memphis?

ChatGPT: The movie you’re thinking of is “The Thing Called Love” (1993). In this film, Sandra Bullock and River Phoenix play aspiring musicians trying to make it in Nashville, not Memphis.

I’ve tried ChatGPT with the most obscure things. Like that little-known but excellent film I saw on VHS back in Blockbuster days, an offbeat musical where Carol Burnett’s daughter in real life goes to Japan to pursue a career as a pop star. Even if I can’t remember the daughter’s name or the year the film was made, ChatGPT was able to find it.

ChatGPT: The movie you’re referring to is “Tokyo Pop” (1988).

If all this sounds great, you’re right. ChatGPT is a terrific research tool, especially when Google just isn’t going to cut it for you. Looking for a recipe that uses just the five things you have in your cupboard or fridge? ChatGPT’s got it. Trying to write a polite email to cut off a disrespectful client? ChatGPT’s got you there, too.

But there are times when ChatGPT just jumbles its facts, in ways that can cause trouble.

AI hallucinations: an example

A couple of years ago I was writing a book on cryptography, and I needed examples of companies that suffered a hack or breach because of a poorly constructed random number generator. I was looking for a fail that wasn’t related to a casino or lottery, like maybe something a little closer to home, like a security system.

ChatGPT, tell me about a time when a home security system was hacked because of a poorly constructed random number generator.

ChatGPT: In May 2019, a vulnerability was discovered in SimpliSafe, a popular home security system. The SimpliSafe system used a weak and easily exploitable random number generator (RNG) that could allow hackers to disable the system’s alarms and bypass its sensors.

A Google search didn’t point up any details, so I asked ChatGPT for its source.

ChatGPT, where did you find the information about the RNG that Simplisafe used that exposed a vulnerability? Please give me a link to the article.

ChatGPT: I apologize for any confusion, but SimpliSafe has not been known to use a random number generator that exposed a vulnerability. If you have any further questions or concerns, please let me know.

ChatGPT hallucination exposed!

I should note that ChatGPT didn’t make the same mistake again for me. I tried it again a few months later, with the same prompt, and I got a right answer. ChatGPT took its own mistakes and incorporated them into its vast store of knowledge. You might say that it learned from its mistakes.

And while that particular hallucination didn’t really have any big consequences—I researched and found something else to put in the book—there are hallucinations that caused major problems out in the world of the law.

Lawyers in trouble

When a lawyer is preparing a case, it’s standard practice for them to look for similar cases and see how they turned out so they can cite these cases as “precedent,” a previous case that strengthens their own case. Lawyers and paralegals often spend hours looking for such cases, so it makes sense that they would turn to AI to help their search.

In 2022, a man named Roberto Mata sued Colombian airline Avianca, claiming he was injured during a flight when a metal serving cart banged into his knee. Mata said the injury was severe, so he hired New York lawyer Steven Schwartz to argue his case.

Schwartz produced a document that cited at least six cases that supported his client’s claim, that an injury like the one Roberto Mata got was worth suing over: Martinez v. Delta Airlines, Zicherman v. Korean Air Lines, and the most famous of all, Varghese v. China Southern Airlines.

Well, guess what. It’s not! There’s no Varghese v. China Southern Airlines. The airline exists, but there’s no case like that. Schwartz didn’t bother to check the cases against legal repositories or even the internet.

When Avianca’s lawyers told the court they couldn’t find the cases anywhere, Schwartz was forced to admit he had used ChatGPT to research the cases, and never fact-checked them. Schwartz, who had been a lawyer for decades, claimed that he hadn’t used ChatGPT before, and was “unaware of the possibility that its content could be false.” He even asked ChatGPT to verify the cases after serving them up, and ChatGPT said Yes.

Asking the liar if they lied? What could possibly go wrong?

It’s pretty common practice for lawyers to have a paralegal look up cases for them, and in that sense, ChatGPT is kind of like that paralegal. But the difference is that a paralegal is a person, one who values their reputation and the quality of their work. ChatGPT, on the other hand, doesn’t care. I mean, you can’t sue ChatGPT, right? And you can’t just take what it says at face value. You have to check each and every fact with independent research.

In the end, a judge slapped a $5000 fine on Schwartz and his law firm for “false and misleading statements to the court.”

Then it happened again in December 2023. A Colorado lawyer, Zachariah Crabill, submitted fake cases as part of a brief, cases imagined up by ChatGPT, and he didn’t bother to check because he was in a rush. In that case, Crabill’s law license was suspended for two years.

Check your facts!

ChatGPT, and other chatbots, are a great way to get specific keywords to search on Google. But check your facts there, and make sure those sources are reputable, too, not things like conspiracy theories on TikTok, fringe documentaries that have been debunked, or even Wikipedia. News websites, government websites, tech blogs, the sources Wikipedia cites—the ones that you know check their facts and have proved reliable over and over again—these should be your sources.

And if you’re a lawyer, you already know where to check to stay out of trouble.