Their company summary reading as "AI Powered Community Moderation" really stood out to me, as at first glance it's not what they market themselves as.
But after reading through the rest of the post, and seeing their list of 'customers' on their website it's obvious what's going on. The apparently unmoderated, highly dangerous chat service where they throw extremely vulnerable people (and presumably the people who want to take advantage of them) at each other in the name of 'mental health support' is not their product. Their product is a bot that crawls social media content to flag posts for moderation. The chat service is just a data harvesting ploy. They need a large body of example text of people experiencing mental health crises to train their bot on. The chat service lets them harvest that.
This company is selling a tool to help social media companies automatically detect at risk users and provide them with links to mental health services. Which sounds perfectly fine until you realise that it's building that tool in the most breathtakingly irresponsible and immoral way imaginable.
KOKOBOT - The Airbnb-Owned Tech Startup - Data Mining Tumblr Users' Mental Health Crises for "Content"
I got this message from a bot, and honestly? If I was a bit younger and not such a jaded bitch with a career in tech, I might have given it an honest try. I spent plenty of time in a tough situation without access to any mental health resources as a teen, and would have been sucked right in.
Chatting right from your phone, and being connected with people who can help you? Sounds nice. Especially if you believe the testimonials they spam you with (tw suicide / self harm mention in below images)
But I was getting a weird feeling, so I went to read the legalese.
I couldn't even get through the fine-print it asked me to read and agree to, without it spamming the hell out of me. Almost like they expect people to just hit Yes? But I'm glad I stopped to read, because:
I was curious about their business model. They're a venture-capitol based tech startup, owned by Airbnb, the famous mental health professionals with a focus on ethical business practices./s They're also begging for donations despite having already been given 2.5 million dollars in research funding. (If you want a deep dive on why people throw crazy money at tech startups, see my other post here)
They also use the data they gather from users to conduct research and publish papers. I didn't find them too interesting - other than as a good case study of "People tend to find what they are financially incentivized to find". Predictably, Koko found that Kokobot was beneficial to its users.
So yeah, being a dumbass with too much curiosity, I decided to use the Airbnb-owned Data-Mining Mental Health Chatline anyway. And if you thought it was dangerous sounding from the disclaimers? Somehow it got worse.
(trigger warning / discussions of child abuse / sexual abuse / suicide / violence below the cut - please don't read if you're not in a good place to hear about negligence around pretty horrific topics.)
Keep reading