Phaedra Haywood, the Santa Fe New Mexican:

A state district judge in Santa Fe on Thursday ordered social media giant Meta Platforms Inc. to pay $567 million into an abatement fund to address public harm to New Mexico children and teens.

Judge Bryan Biedscheid’s ruling resolves the second of a two-part civil proceeding in a landmark case filed in 2023 by New Mexico Attorney General Raúl Torrez, who argued Meta’s social media platforms have led to a youth mental health crisis in the state and have exposed kids to exploitation by sexual predators.

This is in addition to a $375 million penalty issued by a jury in March.

The judge’s orders are worth reading, I think, because they show the careful reasoning that guided the state’s new requirements of Meta. People like Eric Goldman and Techdirt’s Mike Masnick who worried that Section 230 of the Communications Decency Act would be fundamentally undermined by the verdict of this case might be pleased. For example, the judge declined to mandate changes to “features designed to maximize screen time, such as autoplay videos [and] infinite scroll” because it would risk “running afoul of the First Amendment and Section 230 because of the direct effect those features have on content presentation”. If features like those — or algorithmic recommendations, about which the state’s proposed changes “are vague and aspirational” — are to be altered, it is something this judge punted to the two other branches of U.S. government.

Similarly, the judge declined the state’s proposal to restrict or eliminate end-to-end encryption in Instagram messages. In part, that is because Meta already removed the feature in March. But it is also because the evidence did not point to end-to-end encryption being of particular concern (paragraph 149):

Fundamentally, regarding sextortion, grooming and other exploitative activities, it is the algorithmically recommended connections of adults and adolescents that creates the most significant harm.

To that end, the judge says children’s accounts must not be recommended to adult users (paragraph 143), something Meta apparently does not already do.

Meta must make many other changes to the way it handles accounts belonging to children, including this curious restriction (paragraph 170):

Meta shall implement a mandatory usage time limit for accounts belonging to users under 18 years of age. Meta shall restrict the usage of all such users to not more than 90 hours of use per month cumulatively across Facebook and Instagram.

However, because of the Children’s Online Privacy Protection Act of 1998, the judge cannot order that Meta “request children to submit personal data or be passively tracked online, even for age-verification purposes”. Therefore, all of these age-based limitations will be based on estimations or information derived from other interactions.

Many of the most damning statements in this decision are quotes from internal Meta research, and the company could have made lots of positive changes itself. Instead, it exploited its own findings. A May 2020 presentation, for example, “explicitly connected notifications to time spent: a graphic included in the presentation shows the phrase ‘Fewer Notifications,’ which is then followed by an arrow pointing to ‘Fewer Sessions,’ which is followed by an arrow pointing to ‘Less Time Spent.'” while a presentation from June 2023 found “[p]erceived life interference from app use is highest for younger users”. Now that it has been forced to react by this court, it suggested a bunch of stuff it could do, which often agreed with its recommendations: limiting push notifications during school hours and at night, removing like counts, comparing itself to a polluting factory, and so on. Meta and its peers cannot and will not self-govern, even when they have the research. I have low expectations these changes will be rolled out to child users worldwide.

⌥ Permalink

In 2023, Bloomberg Businessweek asked TikTok for comment as it reported out a story about users who were recommended videos about self-harm and suicide, particularly teenagers like Chase Nasca. It is a difficult read. As a result of this press outreach, TikTok analyzed Nasca’s account and why he was given so much of this material that it likely played a role when he took his own life at just sixteen years old.

Olivia Carville, of Bloomberg Businessweek, recently obtained that internal report:

Chase’s account was stuck in a so-called filter bubble, an online echo chamber that pumped out repetitive content the algorithm predicted he’d want to watch. There he received an “onslaught” of suicide and self-harm content, according to the document. Because Chase was in the control group, it says, “TikTok’s filter bubble prevention strategies did not take effect on this user by design.”

Those two words, “by design,” carry extra weight against the backdrop of litigation over social media addiction that’s playing out in US courts. The crux of the legal argument being wielded against the world’s biggest social media platforms is that their products are dangerous by design. The companies, including TikTok, vehemently deny this and point to safety features they’ve introduced to protect users. But the document shows TikTok intentionally withheld a safety feature from millions of people, and why.

The “why” is simple: TikTok made a change to its recommendations system and wanted to see how it performed. TikTok rolled out this change to 90% of its users but kept 10% as a control group; Nasca’s account was in that group.

That split is odd to me — when Google experiments with new features, it first tries them with a “small” and “narrow” group, only expanding them later. Trying a new feature with nearly every user suggests to me that TikTok had a hunch this was a promising enhancement and it only needed to keep a small number as validation. In the report obtained by Carville, one consideration was “the ability to measure impact on [daily active users] and core metrics”, which suggests an alarming followup question: if a better-moderated feed hurt these metrics, would TikTok have considered rolling it back?

A steelmanned argument in TikTok’s favour is that such metrics are only one consideration in adjusting recommendations, and that depressive material was probably still seen by users who were using the newer system. And, yeah, those two things may be true, but the simple fact is that TikTok differed its recommendations system for different users, and one of them was fed an obsessive number of similar videos advocating suicide. That is a choice the company made about what videos would be seen by which users. TikTok has agency, and it should have responsibility.

⌥ Permalink