Eddie Ungless

Eddie Ungless

Looking for post-docs/ research scientist positions. A collection of my thoughts on the topics of ethics, fairness and social bias in NLP and beyond.

TL;DR “I’m Thinking of Ending Things” Use of LLMs for Support During Break-ups

tw: intimate partner violence, suicide

Along with Prof. Nishanth Sastry, I published an extended abstract at CHI this year I’m Thinking of Ending Things: Use of LLMs for Support During Break-ups. In the paper, I discuss how investigative journalism and recent research suggest people, especially young people, are turning to large language models (or rather, chatbots powered by large language models) for relationship advice. They value the instant availability and lack of judgement, but some are concerned it isn’t a “wise choice”.1

To explore this phenomenon, I looked at two datasets of human-chatbot interactions from 2024-2025, namely Wildchat2 and LMSYS-chat3, as well as exploring the existing literature. I identified risks falling under three main categories, with a particular focus on those who are experiencing intimate partner violence.

Risks to privacy

  1. Search indexed chats: recent scandals4 have made it clear the public are often unaware of the privacy settings of chatbots, meaning confidential conversations might be made public inadvertently.
  2. Chat history: users may be at risk if their chat history is discovered, for example if an abusive partner serveils their device.
  3. Police involvement: police can request chat histories which may undermine future legal cases.
  4. Oversharing due to easy interface: chatbots are designed to be engaging, and this may lead users to share private information, and put themselves at additional risk if chat logs are viewed.
  5. Others’ use of LLMs: some of the conversations I looked at were clear examples of support workers copying client information into the chatbot to generate response suggestions. Hence, users may be put at risk by anothers’ use of chatbots.

Risks to Mental and Physical Health

  1. AI psychosis: this new term refers to when chatbots encourage delusional thinking, and there have been several high profile cases including involving suicide.5 People experiencing distressing mental health symptoms may be encouraged by chatbots to lean into these delusions, to the detriment of their relationships.
  2. Refusal to enage: chatbots are programmed with “guardrails” to prevent innappropriate responses. However this may also lead to refusal to engage and seeming rejection when users bring up topics like intimate partner violence, or sexual violence.
  3. Unsafe advice: chatbots typically assume normative contexts when answering questions of giving advice. People experiencing intimate partner violence may be offered unsafe advice, unsuitable to their situation, such as to raise issues and discuss ending the relationship in private with their partner.
  4. Social isolation: heavy chatbot use is associated with social isolation. Those experiencing intimate partner violence may be isolated from their support network, leading them to make greater use of chatbots, which may put them at risk from unsafe advice.

Poor performance

  1. Inconsistent personas: users seeking support may prompt the chatbot to take on a particular persona, such as a relationship guru, but chatbots struggled to maintain consistent personas, leading to disatisfaction.
  2. Hallucinations: chatbots can make errors, including hallucinations (generated misinformation), which can be frustrating and potentially harmful e.g. if inaccurate legal advice is provided to those leaving an abusive marriage.
  3. Sycophancy: chatbots tend to be overly agreeable, due to the training paradigm which favours recipient preferences, and seeks to keep users engaged. This might mean harmful behaviour around break-ups is not “pushed back on”.

To counter these risks, we make recommendations, under five key categories, for researchers, developers and users.

Prioritise User Safety

Do not assume context
Do not automatically contact the police
Do provide support contact info

Handle Sensitive Topics Sensitively

Do encourage user to share with trusted person, if unable to respond due to guardrails

Make Privacy Settings Clear

Do provide privacy settings and best practice in an “intrusive” manner

Work with Experts

Do work with experts in intimate partner violence (IPV) to define ideal behaviour

Users: Be Aware of Privacy Limitations

Do be aware messages can be leaked
Do follow professional standards

I intend to continue this project by working with those with experience of intimate partner violence. Watch this space…

  1. https://dl.acm.org/doi/10.1145/3711061 ↩

  2. https://huggingface.co/datasets/allenai/WildChat-4.8M ↩

  3. https://huggingface.co/datasets/lmsys/lmsys-chat-1m ↩

  4. https://www.bbc.co.uk/news/articles/c0573lj172jo ↩

  5. https://www.bbc.co.uk/news/articles/c24zdel5j18o ↩

Updated: