A simple trick
ELIZA looked for keywords and rephrased sentences. To “I am sad” it replied “Why are you sad?”. It understood nothing, yet it seemed to be listening.
The ELIZA effect
Weizenbaum's secretary reportedly asked him to leave the room while she talked to the program. This frightened its author, and he became a critic of excessive trust in computers. The tendency to attribute understanding to machines has been called the ELIZA effect ever since.
From ELIZA to ChatGPT
Today's language models are incomparably more capable, but Weizenbaum's warning still holds: even a convincing answer is not necessarily true.