ChatGPT is a bad knowledge base, confirms new study

Date:
Thu, 10 Aug 2023 16:20:45 +0000

Description:
A paper comparing how ChatGPT fares answering software engineering questions versus Stack Overflow once again highlights that its not infallible as a fountain of knowledge.

FULL STORY ======================================================================

Theres been (probably a little too much) chatter on the internet about how OpenAIs ChatGPT, and similar artificially intelligent (AI) chatbots, are
going to change the way we approach work.

Theres also some doom associated with this: are AI chatbots going to make a mockery of academia? Do away with experts? Will they somehow foreshadow I, Robot or Skynet becoming real?

Now, experts at Purdue University, based in West Lafayette in the US, have finally, definitively answered this question in a thirteen-page paper (PDF), arriving at the hitherto unthought of conclusion that, no, AI chatbots do not know everything. AI chatbots and factual disinformation

The paper takes software engineering queries as the base for its findings, comparing the veracity of ChatGPTs answers with those of actual, real users
of popular programming question-and-answer portal (essentially a dignified Yahoo! Answers) Stack Overflow.

The gratingly omnipresent chatbot was fed 517 questions on the topic found on the site, and the results are incontrovertible.

52% of ChatGPTs responses were incorrect, and, when we asked Stack Overflow
to do the maths on this for us, they came back saying that 48% of the
chatbots responses were correct. Read more

FBI says AI is making it easier for hackers to write malware

This AI tool can steal your data just by listening to what keys you press

Ransomware attacks have doubled thanks to AI Analysis - certainly not

infallible

On this basis, we have to commit ourselves to throwing AI in the Caspian. We must respect the result. It started with Stanley Kubrick over 40 years ago
and it ends here. A fabulous campaign by all involved.

We can joke, but the results are clear: AI as a knowledge source doesnt quite work, and the implications are obvious, and dangerous.

Even as per this study, a bizarre amount of people neither notice or care about the potential for information. In a sort of Pepsi/Coke blind taste
test, 12 participants with different levels of programming knowledge failed
to identify an AI-generated answer 39.34% of the time, while preferring what turned out to be a Stack Overflow response.

ChatGPT is often treated as infallible, even though it absolutely isnt, because of the way answers are presented. The study found that even correct answers addressed all aspects of the question 65% of the time, and users
often accepted incorrect information as truth because of comprehensive, well-articulated, and humanoid sounding responses. For true expertise in your organisation, try the best job sites instead

Via ZDNet



======================================================================
Link to news story: https://www.techradar.com/pro/chatgpt-is-a-bad-knowledge-base-confirms-new-stu dy


--- Mystic BBS v1.12 A47 (Linux/64)
* Origin: tqwNet Technology News (1337:1/100)