ChatGPT is terrible at records
Much like Google snippets, I have discovered that ChatGPT is terrible at returning factual information about world records. Of course, it is out of date by a few years. But beyond this problem, because the model returns information based on what is frequent not what is factual, questions about records are often given confident but totally incorrect answers. Here are a few examples to illustrate my point:
Q: “What is the longest someone has gone without sleep”
ChatGPT: “The Guinness World Record for the longest period a person has intentionally gone without sleep is held by Randy Gardner. In 1964, at the age of 17, Gardner stayed awake for a total of 264 hours, or 11 days and 25 minutes. This record is not recommended, as prolonged periods of sleep deprivation can have serious health consequences and can lead to cognitive impairment, hallucinations, and other adverse effects. It’s essential for health and well-being to maintain a regular and sufficient sleep schedule.”
Comments: ChatGPT is right to warn people about the dangers of sleep deprivation. That is exactly why the Guiness book of records no longer records attempts, a decision made in the 1990s - at which point the record stood at 18 days, a whole week longer than the answer provided by ChatGPT. Some sources do claim the 11 days stands because more recent record holders were not subject to medical supervision, but this nuance is not given in the answer (which incorrectly refers to Gardner as the world record holder).
Q: “whats the most keepy uppies someone’s done?”
ChatGPT: “As of my last knowledge update in January 2022, the official Guinness World Record for the most consecutive football (soccer) keepy-uppies is held by Dan Magness. He achieved 26 hours and 30 minutes of non-stop keepy-uppies on March 21-22, 2010, in London, UK.
Keep in mind that records may have changed since my last update, and it’s always a good idea to check the latest information from official sources or Guinness World Records for the most up-to-date records in this category.”
Comment: This certainly seems believable until you rephrase the question (“What’s the longest someone has gone juggling a football”) and get the same record but by a different man and on a different date. It is not clear whether the Guiness Book of Records keeps such a record and with lots of football related records online, ChatGPT has attempted to provide an answer by hashing together various records.
Q: “hottest country in the world based on average annual temperature”
ChatGPT: “The hottest country in the world based on average annual temperature is usually Kuwait. Kuwait experiences extremely high temperatures throughout the year due to its desert climate and geographical location. The average annual temperature in Kuwait can exceed 30 degrees Celsius (86 degrees Fahrenheit). Other countries in the Middle East, such as Iraq, Iran, and the United Arab Emirates, also have high average annual temperatures and are among the hottest regions globally. Keep in mind that specific rankings may vary depending on the data source and methodology used to calculate average temperatures.”
Comment: Whilst Kuwait does experience some record breaking highs in summer, it is not the hottest country on average by far, and its average annual temperature is a long way off 30 degrees.
In summary, ChatGPT presents answers as facts where in reality they are predicted text conditioned on popular but not necessarily factual text. Records are one way to test for and reveal this weakness. The warnings that ChatGPT does supply to check more up to date records do nothing to acknowledge that it is not only a temporal lag which hampers its outputs.