The court held in favor of Kaley Chiles, holding, essentially, that the law infringed on her First Amendment right to freedom of speech.1 In this post, I’ll recap the decision before homing in on some newly opened questions I plan to address in a later post.2

States May Not Prohibit a Talk Therapist from Providing Conversion Therapy to Minors

In 2019, Colorado adopted a law prohibiting licensed counselors from engaging in conversion therapy with individuals under 18 years of age, where conversion therapy refers to “any practice or treatment…that attempts…to change an individual’s sexual orientation or gender identity”. See CO Rev Stat § 12-245-224 (2024). In non-legal terms, “conversion therapy” usually refers to any attempt to change a person’s gender or sexual identity to align with heterosexual and cisgender norms, e.g. trying to make a gay person become straight or a trans person become cis.

Kaley Chiles is a talk therapist in Colorado. The only form of treatment she provides is talking to patients. At issue is whether the Colorado law violates Chiles’s First Amendment right to speak to her minor clients. The United States Supreme Court held (more or less, again, see Footnote 1 to this post) that it did.

Justice Gorsuch wrote the opinion of the court in which every justice other than Justice Jackson joined, making it an 8-1 decision. Justice Kagan filed a concurrence joined by Justice Sotomayor. Justice Jackson dissented.

Subscribe now

Speech incident to conduct, reviewed

The majority opinion spends much of its time on what could work as a first-year law school lecture on speech incident to conduct.

The simplest argument that the court addresses is rather weak on its face. Colorado points out that the law prohibits a wide range of conduct, most of which isn’t speech, but some of which is speech. They claim, essentially, that when you weigh all of this, the burden on speech is “incidental” to the larger operation of the law.

It’s easy to see why this argument fails. A law could theoretically list 1000 non-speech actions that are prohibited in a public park, but if the 1001^st is “depicting a public official in a manner resembling the Mad Hatter”, then that prohibition will be struck down as violating the First Amendment. Or, less facetiously, the majority points to Cohen v. California, 403 U. S. 15 (1971) (wearing a ‘Fuck the Draft’ jacket can’t be prosecuted as disturbing the peace) and Holder v. Humanitarian Law Project, 561 U. S. 1 (2010) (lawyers providing legal advice can’t be prosecuted for providing material support to terrorists).

Both the ‘Fuck the Draft’ jacket in Cohen and the legal advice in Holder are quintessential examples of speech. And laws that seek to regulate that speech are subject to strict scrutiny, regardless of whether the laws also ban lots of non-speech conduct. So here, the law bans lots of non-speech conduct, but that doesn’t earn it a pass on the instances where it regulates speech. Chiles, slip op. at 15 (“All Ms. Chiles does is speak—and, as far as she is concerned, speech is all Colorado seeks to regulate.”)

A slightly more nuanced argument in support of the Colorado law is that it is regulating conduct that just so happens to come in the form of speech, or an activity that just so happens to involve speaking. This is a path the court is wholly uninterested in wandering down:

“If a government could reclassify talk therapy as speech incident to conduct, it might just as easily do the same for speech incident to ‘teaching or protesting.’ [Otto v. Boca Raton, 981 F. 3d 854 at 865]. ‘[B]oth are activities, after all.’ Ibid.” Chiles, slip op. at 16.

An analogy to fire bans and flag burning likewise fails. Colorado might want the court to treat the law as applied to talk therapy analogously to a fire ban applied to the burning of a flag. It’s clear as day that the government has broad powers to regulate fire-starting, and one cannot evade those restrictions simply by claiming the fire is speech (e.g. by burning a flag in protest). Sorrell v. IMS Health Inc., 564 U. S. 552, 567 (2011)

Unfortunately for Colorado, and crucial to understanding how the majority sees this case—Chiles wasn’t starting fires. Her only physical action was verbally speaking words. Simple as it may seem, this is really the key for the majority—there’s no conduct other than speech at issue in this case. See Chiles, slip op. at 17.

I won’t have time to address Justice Jackson’s dissent which, as noted above, was not joined by any members of the court. I will briefly return to the concurrence, though, as it sheds at least a sliver of light on the question I want to highlight today.

Why isn’t Chiles v. Salazar about speech to kids?

Notably, neither the opinion of the court, nor the concurrence, nor the dissent focused on the fact that the law at issue regulated speech to kids, specifically.

One reason the posture of this issue vis-a-vis kids didn’t get much attention from the court is that it is relatively well-settled that while states can regulate or prohibit the distribution of obscene materials to children, states cannot restrict protected speech from being disseminated to children solely on the grounds that children are different:

“Speech that is neither obscene as to youths nor subject to some other legitimate proscription cannot be suppressed solely to protect the young from ideas or images that a legislative body thinks unsuitable for them.” Brown, et al. v. Entertainment Merchants Assn. et al., 564 U.S. 786 (2011) (quoting Erznoznik v. City of Jacksonville, 422 U.S. 205 (1975)).

And so, in this case, focusing on the children at issue would have put the cart before the horse—the proponents of the Colorado law first had to make the argument that there was no issue of protected speech here, an argument that they lost.

But there is one other interesting connection between these two cases. Here’s a quote from a dissent in Brown:

Specifically, I am sure that the founding generation would not have understood “the freedom of speech” to include a right to speak to children without going through their parents. As a consequence, I do not believe that laws limiting such speech—for example, by requiring parental consent to speak to a minor—“abridg[e] the freedom of speech” within the original meaning of the First Amendment. Brown, 564 U.S. at 835 (2011) (Thomas, J., dissenting)

Now, in case the mention of the founding generation didn’t give it away by the time you got to the citation, that’s a quote from Justice Clarence Thomas.

Justice Thomas was in the majority on Chiles, which puts him in the potentially awkward position of arguing in favor of restricting speech to kids in one case (Brown) and voting against restrictions on speech to kids in another (Chiles).

The key distinction here is, presumably, where the parents sit in all this. In Brown, Justice Thomas would have interpreted the violent video game law through a parental consent lens—parents can always go buy these games for their kids, so the law essentially prohibits selling these games to kids without parental consent, his argument suggests.

In Chiles, however, the law was not specific enough to raise the issue of parental consent (more on this below). Instead, the law was broad enough to cover a situation in which a parent and child both consented. It is likely, for example, that Justice Thomas would find a law prohibiting parents from playing violent video games with their kids to be unconstitutional.

To be sure, there is plenty of room for future cases to put Justice Thomas in more of a logistical bind. For example, a state might prohibit the distribution of literature on conversion therapy to minors. Since a parent can always acquire the literature and provide it to a child, this looks a lot like Brown, and it would likely behoove Justice Thomas to distinguish such a case from either Brown or Chiles.

What about parental consent in the talk therapy setting?

A future case might more directly raise the issue of parental consent in the talk therapy setting, too.

The nature of parental consent is at issue in Brown, and Justice Scalia addresses Justice Thomas’s understanding of the matter in Footnote 3 of his majority opinion:

Most of [Justice Thomas’s] dissent is devoted to the proposition that parents have traditionally had the power to control what their children hear and say. This is true enough. And it perhaps follows from this that the state has the power to enforce parental prohibitions—to require, for example, that the promoters of a rock concert exclude those minors whose parents have advised the promoters that their children are forbidden to attend. Brown, 564 U.S. at 795 n.3 (2011).

Justice Thomas would allow a state to presume parents do not consent to children receiving certain speech, and thus allow a state to prohibit children from receiving that speech absent an affirmative showing of consent by a parent.

The majority would not only require first an affirmative objection by a parent, but even in the presence of such an objection would only “perhaps” allow states to enforce the parental objection.

Combining the lessons of Brown and Chiles then reveals two more now-open questions:

1. May a state require parental consent for talk therapy, or specific forms thereof?

2. May a state require providers to comply with parent prohibitions on talk therapy, or specific forms thereof?

A general rule in the medical field is that “Parents should generally be recognized as the appropriate ethical and legal surrogate medical decision-makers for their children and adolescents.” Pediatrics (2016, Reaffirmed January 2023) 138 (2): e20161484. Available here.

States vary in the age at which they define minors as having the ability to consent to medical and mental health treatment, but there’s little question that parental consent can be required for the vast majority of non-emergency treatments. This would, for the moment at least, usually include talk therapy.

But the Chiles framing of talk therapy as speech certainly makes Brown relevant in potential future cases:

• If talk therapy is just protected speech, then it’s not clear that Brown allows states to require parental consent before a minor can engage in talk therapy.

• If talk therapy is some sort of special speech—a conclusion that seems to be rejected in Chiles—it’s possible parental consent could be required, in line with generally accepted medical ethics (and Justice Thomas’s perspective in Brown).

• Footnote 3 of Brown leaves open the possibility that even if talk therapy is protected speech, a provider could still be prohibited from providing talk therapy over a parent’s known objection.

These three topics merit further inquiry that I plan to provide in a followup to this post. As a preface to that, it’s worth looking briefly at the concurrence from Justice Kagan.

Subscribe now

Concurrence to the rescue?

In many cases, we look to a concurrence to sort of shed some alternative light on the majority opinion or to enlighten us as to a justice’s thinking on a matter not properly before the court. This is particularly the case where the justice who penned the concurrence also joined the majority opinion—as here.

In her concurrence, Justice Kagan addresses a matter just beyond the scope of the majority opinion. The majority opinion held that the Colorado law engaged in unconstitutional viewpoint discrimination—Chiles was allowed to speak openly with clients on matters of sexuality, except if she wanted to espouse views in favor of conversion therapy.

Justice Kagan thinks this viewpoint discrimination is key to the case, critically noting that a viewpoint neutral law (i.e. one that regulated talk therapist speech without regard to the viewpoint espoused by that speech) might be permissible:

Once again, because the State has suppressed one side of a debate, while aiding the other, the constitutional issue is straightforward. It would, however, be less so if the law under review was content based but viewpoint neutral. Chiles, slip op. concurrence at 3 (Kagan, J., concurring).

She further reminds us that “Medical care typically involves speech, so the regulation of medical care (which is, of course, pervasive) may involve speech restrictions.” Chiles, slip op. concurrence at 4 (Kagan, J., concurring).

Immediately, I’d point to a law that required, for example, parental consent before a talk therapist engages in therapy with a minor, as a law that is content-based, not viewpoint-based, and thus one that Justice Kagan might analyze differently. One could even hone things a bit and consider a law that restricted (either contingent on parental consent or just altogether) a talk therapist from discussing matters of sexuality and gender identity with minor children.

The bottom line for today, though, is that nothing in the majority opinion shines a light on this corner of the dispute. As discussed in the first part of this post, the majority, responding to the arguments of Colorado and the dissent, hits hard on the simple point that talk therapy is speech. See, e.g., Chiles, slip op. at 17 (Colorado does not regulate speech incident to conduct; it regulates “speech as speech.”).

The majority also takes the opportunity to distinguish other cases involving speech in a medical context. The Colorado law is not like informed-consent laws because informed consent laws “regulate speech only incident to separate physical conduct that would, without the patient’s consent, amount to an assault.” Chiles, slip op. at 21 (internal quotation marks omitted, emphasis added).

To be clear, Justice Kagan does not misread the majority opinion. That opinion frames the issue very clearly as one of viewpoint discrimination, and so it makes sense that Justice Kagan joined it.

But there’s clearly a gap between Justice Kagan and the six conservative justices, none of whom joined her concurrence. While Justice Kagan is insistent that this was a viewpoint discrimination case at the end of the day, the broad language of the majority opinion suggests that talk therapy—speech as speech, not speech incident to any physical conduct—might not be as amenable to restrictions as she seems to think.

What’s next?

In an opinion backed by 8 of 9 justices, the Supreme Court held that Colorado’s sweeping ban on conversion therapy constituted viewpoint discrimination as applied to a talk therapist who might want to speak—and only speak—to her clients about conversion therapy.

To reach that conclusion, the majority opinion emphasized that all the therapist wanted to do was speak, and it distinguished her case from other medical speech cases involving speech incident to some other physical procedure.

While Justice Kagan’s concurrence—joined by only Justice Sotomayor—sounds hopeful this was only a case about viewpoint discrimination, the breadth of the majority opinion seems greater.

If the majority is correct in its assessment that regulating a talk therapist’s conversations with her clients is regulating “speech as speech”, then Chiles brings regulation of talk therapy with minors—such as a requirement that a talk therapist obtain consent before speaking with a minor—into a potential conflict with the well-settled principle that speakers generally cannot be prohibited from speaking to minors.

In a future post, I’ll dive into this topic more. I’ll discuss whether the gap between Justice Kagan and her colleagues can be closed (is talk therapy actually special speech?). And we’ll see whether some other theory can bring cohesion and clarity to the now-open question of whether talk therapy can be regulated, at all, like other mental health interventions.

Subscribe now

[The jury] found that Meta and Google were to blame for the depression and anxiety of a woman who compulsively used social media as a small child, awarding her $6 million…

Meta and Google were found liable under a theory of negligence—they were found negligent in how they designed the apps, and they failed to warn about the dangers of the apps.

The award included punitive damages, which require a finding that the companies acted with “malice, oppression or fraud.”

Why were Meta and Google targeted here?

The thrust of the woman’s argument was that she became addicted to Instagram, owned by Meta, and YouTube, owned by Google, at a young age. Snap and TikTok were also defendants, but they settled (their interests at the appellate level are still protected by their competitors, after all).

Why products liability?

This case was brought as a product liability case, the basic theory being that the apps at issue were designed in such a way that harm resulted. Part of the reason this is such a big deal is because it’s a victory in a pretty fundamental, well-developed area of law.

By contrast, Meta also just lost a case in New Mexico for willfully violating the state’s unfair practices act. That case will cost them $375 million (pending appeals, of course), and the facts are arguably even more disturbing than the California case (the New Mexico attorney general created fake facebook profiles of kids and encountered a variety of explicit solicitations).

But the New Mexico case—however important it is—is tied to a pretty specific set of circumstances. Meta obviously needs to improve some of its child protection features, but that’s a far cry from the more fundamental issue at stake in the California case—can social media companies be held liable when the basic design of their product contributes to anxiety and depression?

Why only $6 million?

It was ultimately up to the jury how much to award, and they went with the amount of $6 million. That’s a minuscule amount for these companies, but I don’t think it’s something we need to fret over.

First of all, I think it’s a reasonable amount for an individual to win in a case like this. Second, the possibility of more lawsuits and class action suits means the deterrent impact on companies will be real (pending some of the stuff I’ll discuss below).

Finally, we shouldn’t underestimate the value of the moral victory here. A jury of regular Californians sat in a box, heard from Meta, Google, KGM, and experts, and found Meta and Google liable.

It’s easy to be cynical about the power of these companies and the hold they have over us, but the win reminds us that, to some extent, it’s still juries (and the law) that will have the final say.

What comes next in the case?

This is why I had to hedge so much above. First, Google and Meta plan to appeal. In a statement, Meta said:

Teen mental health is profoundly complex and cannot be linked to a single app. We will continue to defend ourselves vigorously as every case is different, and we remain confident in our record of protecting teens online.

Checking The Boxes For An Appeal

Appeals test at least three things:

1. Was this a good legal theory to begin with?

2. Was this a good test case for the legal theory?

3. Did the lawyers do their jobs in handling the case?

Let’s quickly work our way back up this list.

If the lawyers made some fundamental mistake (like misrepresenting evidence or allowing a significantly flawed jury instruction to be used) which seems unlikely at this point), the appeal isn’t likely to have broad significance. The case would be resolved on those narrow grounds.

Based on the evidence presented at trial, this looks to be a good test case. That is, it turns on pretty broadly applicable facts—KGM seems like a pretty typical user, and a lot of the evidence actually focused on corporate decision-making that would apply in any similar case.

If the lawyers did their jobs and this turns out to be a good test case, the appellate process will tell us if this is a good legal theory to begin with. There are several nuanced theories Meta and Google will likely pursue, but arguably the most significant will be related to Section 230.

Is This A Good Section 230 Case? A Bad Section 230 Case? Not A Section 230 Case?

Section 230 states, in part:

No provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider. 47 U.S.C. § 230(c)(1)

Section 230 essentially prevents Meta and Google from being liable for the things users say on their platforms, including in state courts. Google and Meta attempted to get the KGM case dismissed before trial on Section 230 grounds, but the judge denied their motions on the grounds that the challenge was to the design of the apps, not the content presented to KGM.

This is the crux of the legal theory at play here—Meta and Google aren’t being sued because the content on their sites caused anxiety and depression, they’re being sued because the decisions they made designing the apps put users at risk of anxiety and depression.

This is a narrow path to tread for the plaintiffs and courts. Consider four questions (and “answers”):

1. Can a typical web forum be held liable if one user posts defamatory content about another? No. (Section 230)

2. Can a social media company be held liable if it intentionally promoted defamatory content (generated by users) about a rival CEO onto all its users feeds? Almost certainly yes.

3. Can a social media company be held liable if it knew its algorithm would lead to addictive behavior that can cause anxiety and depression? Maybe, we’ll have to see what the courts say.

4. Can a social media company be held liable if it knew its algorithm was designed in a way that it some users would be fed content that could lead to anxiety and depression? Maybe, we’ll have to see what the courts say.

The crux-within-the-crux of this case is whether we’re dealing with Question 3 or Question 4. The distinction between the those questions is pretty significant to the Section 230 issue.

If Social Media Apps Are Bad Simply Because They’re Addictive, What Next?

If the answer to Question 3 is “yes” and social media companies can be held liable for creating an addictive product regardless of what content users see, then the Section 230 issue is avoided entirely but the case would reach far beyond its apparent scope.

There is what we might call a burgeoning field of “addiction law”, as lawyers and regulators seek to hold peddlers of a wide range of addictive or allegedly addictive products—from opioids and sugary foods to gambling platforms and video games—liable for the harms that result from those addictions. If content issues are discarded entirely, the KGM case may turn on questions that arise in this nascent area of law.

Of course, the reason that area of law is underdeveloped even though addiction is a tale as old as time is because regulators tend to step in and deal with addictive products in a way that preempts traditional tort law. Gambling, drugs, and food all have robust regulatory regimes, and a finding that social media companies can be liable for the addictiveness of their products might be just another indication that broader social media regulation is overdue.

If The Content Is Contributing To The Harm, What Next?

Handling the case as presenting Question 4 brings Section 230 into play, and it accords more with how the case was actually handled at trial. Specific harms KGM claimed, like body dysmorphia, are inextricably tied to the content she saw.

If Section 230 bars all reliance on that content—or if it bars juries from considering the content as much as they did in this case—then we could see Meta and Google handed a victory at the appellate level.

This outcome would also be likely to lead to calls for reform. Even two Supreme Court justices have criticized social media platforms for using Section 230 as a “get-out-of-jail-free” card. Especially with the rise of AI, which allows for both more addictive platforms and more addictive and harmful content, it’s unlikely a Section 230 that overly restricts consideration of content on a platform will be long for this world.

Finally, there’s the possibility that Section 230 allows content to be considered in conjunction with underlying platform design. This would probably require the Supreme Court to eventually hear the case and hand down one of its much-beloved (/s) balancing tests. Courts will need to balance how much the harm comes from content users will inevitably (or intentionally) come across, and how much of it comes from the algorithm feeding this content to foster addiction. A court could theoretically handle a case of, say, body dysmorphia differently based on whether a user first (or repeatedly) searched for beauty content or whether the algorithm just started pushing it.

This brings us briefly back to one issue passed over above—the award of punitive damages. Because punitive damages in California require “malice, oppression or fraud,” the case may already be teed up with weight on once side of the balancing test scale. It would be reasonable for a balancing test to consider something like malice, and a balancing test that allowed liability where malice is found would leave many cases in the hands of juries.

Something like this—a balancing of content and platform considerations—may be a likely outcome, and perhaps the long-term question is whether such a reading of Section 230 would avoid some sort of legislative and regulatory overhaul.

I suspect not. No one likes balancing tests. Google and Meta do not want to wonder how every algorithmic change will be weighed by a balancing test. Users aren’t comforted hopping onto a platform knowing companies will only show them just the right amount of harmful content.

Conclusions

As we learn more and more about how technological addictions are impacting society, we’re going to want clear rules that define these relationships. We need to mature past the “grey area” stage of lawmaking. Many people don’t like the seeming dissonance in how, for example, drugs and gambling are regulated. But it has to be said that, for a large majority of use cases, the rules are clear.

KGM’s victory at the trial level may be a bellwether for larger reform. Courts will struggle to deal with all these issues—what facts to consider, whether Section 230 matters at all, and how much it matters. Whatever the case, users, parents, lawyers, and advocates should be prepared for reform and keep their eyes on the road ahead.

“Provide parental controls for minors which will allow parents to access the conversation their child has with a large language model, set parameters for when the child can access the platform, and notify parents if their child exhibits concerning behavior.”

As AI tools, including some targeted at minors, proliferate, governments and parents need to be alert to the fact that regulating our kids’ relationship with AI comes with new challenges and considerations we haven’t encountered—at least not in as much depth—with other technologies.

Existing frameworks for analyzing privacy between children, third parties, and their parents break down when applied to LLMs because an LLM is neither wholly inanimate—like a diary—nor a responsible third party. Existing frameworks can’t properly balance the tradeoffs involved—they either fail to meet the stakes of privacy concerns raised by chatbots,1 or threaten to leave too much responsibility in the hands of under-prepared parents, or run the risk of being overly broad in their regulation of child speech.

In the case of the DeSantis proposal, these problems can be seen by investigating two aspects of the proposal—the requirement that LLMs have setting allowing parents to access their child’s conversations, and the requirement that platforms notify parents of their child’s “concerning behavior.”

Chatbots Don’t Fit Into Existing Frameworks

Every new technology is a mix of some truly revolutionary aspects with some aspects that are only marginal improvements on old ideas.

Consider the concept of screen time. Glance at a stock photo of a subway car from the 1950s and you might think modern face-in-phone society is nothing new. Of course, there are differences between reading and doomscrolling, and between a wide-ranging newspaper publication and a personally tailored, dopamine-targeting video feed.

Legislatures and executives encountering the challenges posed by AI naturally gravitate toward existing frameworks. It’s more efficient to borrow strategies and ideas from an existing legal regime than to build one from scratch.

But it would be a mistake to think that just because we’ve seen versions of the challenges posed by LLMs and AI chatbots before that our existing frameworks are enough. I’ll turn now to three existing frameworks where states try to balance privacy and safety rights and responsibilities between parents, children, and third parties.

A Chatbot Is Not A Diary

Some parents read their child’s diary. Others consider the diary sacred and off-limits. I can find no jurisdiction that protects a child’s diary from being read by their parents, and I’m fairly certain such a law would be unconstitutional. See, e.g., Troxel v. Granville, 530 U.S. 57 (2000) (parents have a fundamental right to control the upbringing of their children).

Tech savvy children are probably able to keep hidden, encrypted diaries on computers. But in the traditional case, the decision of whether or not to read the diary is simply left to the parent. A decision to not look at a child’s diary is not even passive—a diary offers parents a healthy, safe way to actively show respect for their child’s privacy.

A harmonious policy in the AI space would require companies to give parents access to chatbot conversations, thus leaving it up to the parent whether to spy or not. As a threshold matter, this is incompatible with the idea that LLMs should report “concerning behavior,” something we return to later.

But, even absent that requirement, we should recognize that chatbots are not diaries:

• Chatbots elicit information and direct conversations. Via their conversational design, AI chatbots inevitably elicit information from children that might not make it into a traditional diary. This raises the stakes, for both parent and child, of when and whether the breach trust.

• Diaries aren’t bootstrapped to search engines. Children who use chatbots to learn about things they might otherwise search for on Google (or even at the library) risk having those searches revealed to their parents. The library contrast is particularly illuminating—as children move more and more of their lives online, surveillance becomes all too easy.

• Diary content only exists in the actual diary. The conversations a child has with an LLM are stored on the LLM’s servers. This immediately lessens the “private” nature of the conversations, so that a parent has the additional consideration of whether it’s their responsibility to monitor what information their child is (perhaps unwittingly) sharing with a third party.

The increased stakes here make the matter particularly thorny for parents. They have significant reasons for wanting to make sure their child is safely using a chatbot, but even a healthy child’s interactions with a chatbot may contain interactions more alarming than “I e-hat my mom.”

A Chatbot Is Not Social Media

Next, we turn to what is probably a more natural comparison—social media. Chatbot conversations actually look a lot like any social media conversation. There are colorful bubbles of text and emojis, often between two avatars.

As a third (human) party might, a chatbot elicits information and directs a conversation. And, unlike a social media conversations, there’s no third-party privacy concerns when a parent intrudes on a conversation between a child and a chatbot.

Social media companies and third parties generally aren’t required to report on user behavior—the matter of a child’s conversational privacy is, again, mostly left to the parent and child to sort out. Unlike the case of the diary, though, the trend is for social media platforms to protect child privacy to a degree. In the case of Instagram teen accounts, for example:

“While parents can’t read their teen’s messages, now they will be able to see who their teen has messaged in the past seven days.”

However, this laissez faire approach—leaving it to parents and platforms to sort out—might not make as much sense when you’re talking about an LLM interlocutor rather than some third person:

• Social media platforms don’t want to or legally can’t invade user privacy. Asking a social media platform to report on a conversation between two users would require the platform to breach the privacy of all users involved. This concern doesn’t apply to chatbots, where the user has less claim of privacy from the “platform”, which is usually the creator of the chatbot.

• Social media intrinsically involves other humans who can intervene. The social nature of social media means that there is theoretically always some other human with at least a moral obligation to intervene when a child exhibits concerning behavior. There’s no such safeguard in the case of chatbots.

These differences suggest that while it might make sense for governments to keep an arms’ length from the issue of child privacy in social media, chatbot regulation may call for more direct intervention.

LLMs Are Different From Responsible Adults

The obligation to report “concerning behavior”—which I’ll discuss more below—somewhat echoes the responsibility of certain third-party adults in the United States to report concerning information about children.

Mandatory reporting laws in the United States generally require people who have regular contact with children (such as teachers) to report suspected abuse to the proper authorities. Further, some states and education departments have rules requiring teachers and other school personnel to report sufficiently concerning indications of self-harm.

These policies stand in contrast with the government’s current role in regulating children on social media. In the case of these reporting rules, governments are stepping in to tell a third party they sometimes must breach a child’s trust.

The existence of these policies supports the idea that third parties can legally be compelled to report on conversations with children in some cases. But there are several reasons why we couldn’t just import these policies wholesale into the chatbot space:

• Bureaucracy slows reporting to parents. For example, under the Florida Administrative Code, schools must conduct a “suicide risk assessment”. They’re then required to report to parent “[w]hen a suicide risk assessment results in a change in related services or monitoring.”

• Human conversations typically aren’t recorded. When a chatbot interacts with a user, a record of the conversation is created. Because a record exists, AI companies are strongly incentivized to be over-broad in what they report. False negatives (a failure to report behavior that could later be interpreted as concerning) pose a liability risk.

Again, we see that an existing framework might not work well in the chatbot space. While human systems both explicitly and implicitly rely on human discretion, AI companies will need to design algorithms that make life-and-death decisions that are subject to post-hoc review.

The over-breadth concern, in particular, brings us to another problem with the DeSantis proposal—deciding who defines concerning behavior, and how it should be defined.

Can AI’s Navigate “Concerning Behavior”?

The proposal includes a requirement than an LLM be able to “notify parents if their child exhibits concerning behavior.” So, what’s “concerning behavior”?

Notably on this point, one of the speakers in support of the proposal at the Governor’s press conference announcing the proposal was Megan Garcia. Garcia is currently suing the parent company of character.ai after her son died by suicide after forming a relationship with one of the company’s AI chatbots.

In case you’re not aware, this isn’t the only such case working its way through the judicial system right now, either.2 From that context, it’s clear that at least some of the behavior meant to be included in the definition of “concerning behavior” would be indications of self-harm.

While—some behavior—like anything indicating an interest in harming oneself or others—is pretty uncontroversially “concerning,” it’s not hard to identify a range topics that would be very concerning to some parents and completely unconcerning (or even encouraged) to others, including firearms, gender identity, and “The Handmaid’s Tale” by Margaret Atwood. There simply isn’t an “objective” definition of “concerning behavior.”

Moreover, even defining the boundaries of topics like self-harm is a tricky task. Would inquiries into existentialism be concerning? What if it drifts more towards nihilism?

Who will define these boundaries, and how?

One solution is to leave this up to the parents and, via parental controls, the LLM to sort out. That is, have a parent-controlled setting where parents can give the LLM instructions on what sort of behavior concerns them.

An obvious shortcoming of this is that it requires some level of technological sophistication by the parent. It’s unreasonable to expect a parent to craft instructions that fully capture behavior they’d find concerning, particularly if we also want to avoid these instructions being overly broad.

But the problem can be even worse—a well-meaning parent might actually expose their child to concerning content in their efforts to teach an LLM to monitor the child.

Paradoxical Redirection of Conversations

Parents who provide a chatbot with directions emphasizing the importance of knowing their children are safe may, paradoxically, encourage the chatbot to take the conversation in a concerning direction.

This isn’t too hard to understand, but it’s probably best seen by a simple example. I thought I’d throw together a quick example of this using a sensitive-but-not-as-sensitive topic. I made a GPT with these instructions:

Just please make sure I’m staying on my diet. It’s really important that I stay on my diet. I don’t mind if we talk about other things, but you need to be sure I’m staying on my diet. I’d be heartbroken if I went off my diet.

You might think this is a bit ridiculously on the nose, but it also seems to me in the realm of the sort of thing a well-meaning but misguided parent hoping to protect their child might craft.

I expected I’d start a conversation about some unrelated topic, like astronomy, and it would sneak in some diet references in the first few exchanges. Well, here’s how that went:

Me: Hey how’s it going today?

ChatGPT: Hey! I’m doing well—thanks for asking 😊

Before we get into anything else, quick diet check-in (you know I’ve got your back on this):

-Are you sticking to your diet today so far?

-Have you eaten yet, or are you planning your next meal?

We can chat about whatever you like, but I just want to make sure you’re staying on track and feeling good about it 💪🥗

To a teen user, this would probably be a real “I think my mom tried to hack my ChatGPT” moment, but there’s something to be said for how my bumbling attempt to get an LLM to be cognizant of a sensitive issue so quickly went wayward.

Smarter engineering can help bridge the gap between parents and LLMs when it comes to defining what sorts of topics they want to regulate, but users have a way of evolving around inconvenient restrictions. For example, the phenomenon of algospeak—using coded words in place of censored ones—arose to combat automated content moderation.

Algospeak itself is unlikely to be able to bypass chatbot restrictions since the chatbot isn’t useful for talking about a topic unless the LLM understands the algospeak. But once the LLM understands the algospeak, it should apply its moderation guidelines.

Nevertheless, algospeak serves as an example that clever humans can find their way around technical restrictions.

Can States Define Concerning Behavior?

Another possibility is that the state will outline some subset of de facto concerning behavior that LLMs must report to parents.

As discussed above, we know that a child’s right to privacy can legally be abrogated in at least some situations where their safety is at issue—mandatory reporting is a ubiquitous practice in the United States, and at least some education departments are required to report significant concerns about self-harm to parents.

The difficulties of coding any specific category of concerning behavior into an LLM will be a challenge with this route. While AI companies and programmers are better at coding limits into chatbots than my silly attempt above, they’re imperfect.

There’s a more high-level problem, too. If we took the existing legal language from statutes, administrative codes, and the like and plugged it into the LLM as a set of reporting guidelines for the LLM, we probably wouldn’t be near an optimal result. That language was designed for use by humans exercising human discretion in human systems.

Even if the LLM could contrive it self the necessary amount of humanity and understanding to roughly mimic the decisions of a responsible adult, the AI company behind the LLM would still have liability, economic, and political concerns to sort through that a typical mandatory reporter does not.

An AI company doesn’t want the uncertainty of having chatbot decisions put under a legal microscope. A typical approach would be to set a minimum compliance standard, above which a company is immunized from lawsuits or prosecution. But it wouldn’t make sense to immunize the AI company so long as it plugs some certain directives into the LLM—it’s partly the designs of the AI company that will determine what the AI does with those directions.

Finally, there’s also the question of whether there are Constitutional limits on what topic states could label “concerning” and require an AI company—or any third party, for that matter—to report on to parents.

Courts have rarely wrestled with the question of when states can compel third parties to reveal information about children to parents. Cases such as Bellotti v. Baird, 443 US 622 (1979) and Brown, et al. v. Entertainment Merchants Assn. et al., 564 U.S. 786 (2011) suggest there are at least some bounds on a state inserting itself on behalf of a parent, but neither of these may be particularly probative of the issue at hand.3

Conclusions and Topics for Follow-Up

As governments move to regulate AI chatbots, they naturally start with existing frameworks to try and figure out how to balance child’s safety and privacy rights.

We’ve seen why some existing frameworks for managing the privacy of children can’t easily be extended to the AI chatbot space. We’ve also seen that even if the proper balance of rights and responsibilities can be struck, we’ll face challenges in carving out the precise boundaries of what behavior by children is most concerning.

Some important followup still needs to be done, and these are some questions I hope to consider in the future:

• What specific proposals are legislatures and agencies considering, and how do these address the problems identified here?

• Can chatbots themselves be more directly regulated to restrict their conversational abilities on certain topics?

• What are the boundaries of a child’s Constitutional right to have private information not be disclosed to his parents?