"Friendly" apps are good at maximizing engagement, but their context-blindness is a cesspit of algorithmic cruelty

Designers use metrics and a/b splitting experiments to maximize engagement with their products, seeking out the phrases that trigger emotional responses in users -- like a smart scale that congratulates you on losing weight -- but these systems are context-blind, so they are unable to distinguish between people who might be traumatized by their messages (say, a woman who's just miscarried late in her pregnancy, being congratulated on her "weight loss"). (more…)