When AI starts making things up

One of the biggest problems with generative AI is that it can produce information which looks completely correct but is actually wrong.

This is especially dangerous because the answer can be written very confidently. A person reading it may not realise that the information is incorrect.

Scientific references are a good example. There have been cases where language models generated scientific references that did not actually exist. Recent research and reporting has shown that fabricated or incorrect citations are becoming a problem in scientific literature.

This creates a strange situation where a system can produce something that looks very scientific while the source behind it may not exist at all.

So we have to remember that something sounding correct does not mean that it is correct.