Can artificial intelligence chatbots outperform human doctors? Here's what new Stanford study found

SAN FRANCISCO (KGO) -- There's been a lot of conversation and consternation about artificial intelligence and its role in our society. Can platforms like ChatGPT be trusted?

For writing a customer service complaint or cover letter, certainly. But how about diagnosing what's ailing you or considering medical ethics? Can a chatbot be trusted to help make life-or-death decisions?

A pair of recent studies out of Stanford University, led by Dr. Jonathan H. Chen, suggests the answer might be "yes."

In fact, AI defeated human doctors at both diagnosing illnesses and reasoning through tough decisions. Dr. Chen, faculty director for medical education in AI, appeared on ABC7 News at 3 p.m. on Thursday and spoke with ABC7 News anchor Kristen Sze about the implications of his stunning findings.

"We had two studies now, one in diagnosis. 'Can you guess what it is? And can you explain why,' the computer actually did better than humans, plus computer that was absolutely not what we expected," Dr. Chen said.

In the first experiment testing diagnosis powers, doctors who did not have access to ChatGPT received a score of 74%. Doctors who had access to ChatGPT scored slightly better at 76%. But the AI alone scored 90%.

In the second experiment testing decision making in complex clinical situations, Dr. Chen created three test groups: standalone chatbots, 46 doctors using chatbot support, and 46 doctors relying on traditional resources such as internet searches and medical references.

Each group was asked to analyze five patient cases, providing detailed responses about their decisions and reasoning. Even here, AI won out.

So, what's the upshot? If the AI is so good, should human doctors get out of its way? Dr. Chen says chatbots may be a valuable complement to human expertise, but warns it's not a replacement, and cautions patients against relying solely on AI for medical advice.

"I used to always say this because it sounds good, human plus computer, when combined, will deliver. Now I'm a little bit unsure. I don't say that as much. What does this mean? Yeah, what I actually think it means, obviously the combination is better, but it's a whole new tool. Most of the doctors in the study at this time, they didn't even know what a chatbot was. They really just didn't know how to use it. And that's why I've taken on this medical education role, because in addition to tech development, we clearly need to train and retrain all of us on how to use powerful new tools," he said.

He also says this technology has the potential to reshape how doctors make decisions and deliver care. The question is, how soon we'll see this collaboration in action and how it might make human doctors better.

Watch the full interview in the media player above