As governments worldwide move to restrict teenagers’ access to smartphones and social media, a fierce scientific debate has erupted over whether these digital technologies actually harm young people’s mental health.

The controversy, sparked by an influential recent book blaming phones for rising youth anxiety, has exposed deep uncertainties in the research evidence – even as policymakers from Arkansas to Australia forge ahead with sweeping bans and restrictions.

A timeline of the controversy

In March, New York University social psychologist Jonathan Haidt published a popular science book called The Anxious Generation. This blames a rise in youth mental illness over the past 15 years or so on the advent of smartphones and social media.

One early review of Haidt’s book by Duke University psychological scientist Candice Odgers, published in Nature, voiced a common criticism among expert readers: while social media is sometimes associated with bad outcomes, we don’t know if it causes those bad outcomes.

In April, Haidt responded that some recent experimental studies, where researchers get people to reduce their social media use, show a benefit.

In May, Stetson University psychologist Christopher Ferguson published a “meta-analysis” of dozens of social media experiments and found, overall, reducing social media use had no impact on mental health.

Next, in August, Haidt and his colleague Zach Rausch published a blog post arguing Ferguson’s methods were flawed. They said doing the meta-analysis in a different way shows social media really does affect mental health.

Not long afterwards, one of us (Matthew B. Jané) published his own blog post, pointing out issues in Ferguson’s original meta-analysis but showing Haidt and Rausch’s re-analysis was also faulty. This post also argued properly re-analysing Ferguson’s meta-analysis still does not provide any convincing evidence social media affects mental health.

In response to Jané, Haidt and Rausch revised their own post. In September and October they came back with two further posts, pointing out more serious errors in Ferguson’s work.

Jané agreed with the errors Haidt and Rausch found and has set out to re-construct Ferguson’s database (and analyses) from scratch.

The discussion and further work is still ongoing. Yet another team has recently published an analysis (as a preprint, which has not been independently verified by other experts) disagreeing with Ferguson, using similarly unreliable methods as Haidt and Rausch’s first blog post.

The evidence is varied – but not very strong

Why so much debate? Part of the reason is experiments where researchers get people to reduce their social media use produce varied results. Some show a benefit, some show harm, and some show no effect.

But the bigger issue, in our opinion, is simply the evidence from these experimental studies is not very good.

One of the experiments included in Ferguson’s meta-analysis had some German Facebook users reduce their use of the social media platform for two weeks, and others continue using it normally. The participants then had to self-report their mental health and life satisfaction.

People who were asked to use Facebook less did report spending less time on the platform. However, there was no detectable impact on depression, smoking behaviour, or life satisfaction at any time point between the two groups. There was a difference in self-reported physical activity, but it was very small.

Another famous study recruited 143 undergraduate students and then randomly assigned them to either limit their Facebook, Snapchat and Instagram use to ten minutes per day for a month, or to make no changes. The researchers then asked participants to report their anxiety, depression, self-esteem, autonomy, loneliness, fear of missing out and social support.

At the end of the month, there was no difference between the two groups on most measures of mental health and wellbeing. Those who reduced social media use showed a small decrease in self-reported loneliness, and there was also a small improvement in depression scores among people who reported high levels of depression to begin with.

Existing social media experiments can’t answer big questions

Studies like these address narrow, specific questions. They are simply unable to answer the big question of whether long-term reduction in social media use benefits mental health.

For one thing, they look at specific platforms rather than overall social media use. For another, most experiments don’t really define “social media”. Facebook is obviously social media, but what about messaging services such as WhatsApp, or even Nintendo’s online gaming platform?

In addition, few if any of these studies involve interventions or outcomes that can be measured objectively. They consist of asking people – often undergraduate students – to reduce their social media use, and then asking them how they feel. This creates a range of obvious biases, not least because people may report feeling differently based on whether they were asked to make changes in their life or not.

In a medical study assessing a drug’s effect on mental health it is common to administer a placebo – a substitute that should not have any biological effect on the participant. Placebos are a powerful way to mitigate bias because they ensure the participant does not know if they actually received the drug or not.

For social media reduction studies, placebos are virtually impossible. You cannot trick a participant into thinking they are reducing social media when they are not.

Individual changes and a social problem

What’s more, these studies all work at the level of changes to the behaviour of an individual. But social media is fundamentally social. If one college class uses Instagram less, it may have no impact on their mental health even if Instagram is bad, because everyone around them is still using the platform as much as ever.

Finally, none of the studies looked at teenagers. At present, there is simply no reliable evidence that getting teenagers to use social media less has an impact on their mental health.

Which brings us back to the fundamental question. Does reducing social media improve teen mental health? With the current evidence, we don’t think there’s any way to know.