(From: theguardian.com)
Part of our mission at DRGO is to keep you up-to-date on research about the use and abuse of firearms. We take some satisfaction in revealing when “research” turns out to be made of recycled memes, illogical methods and forgone conclusions.
So when this year’s annual Epidemiologic Reviews (Vol. 38) was announced in March, we were thrilled by the opportunity to criticize 9 gun-related studies all in one convenient place. Then we felt overwhelmed by the challenge that would be.
Thus we were delighted to hear from our friend, Caillin Langmann, MD, PhD at McMaster University Faculty of Health Sciences, who took a close look at one of these articles, What Do We Know About the Association Between Firearm Legislation and Firearm-Related Injuries? Dr. Langmann is an assistant clinical professor in McMaster’s Division of Emergency Medicine, a clinician (the MD) and researcher (the PhD).
The short version is that you can find “associations” between many things, which may not mean anything.
The longer explanation begins with news coverage about the review. Quoting from Science Daily, “While the research did not conclusively prove that restrictions, or relaxation of laws, reduce gun deaths, the results indicate that gun violence tended to decline after countries passed new restrictions on gun purchasing and ownership.”
Observe the cautioius use of “did not conclusively prove. . .results indicate. . .tended to”. Now that’s not uncommon, since scientific work usually adds very incrementally to knowledge rather than definitively removing all doubt in some way beyond questioning. Still, there’s a lot of wiggle room.
Put more formally, from the paper’s introduction, “implementation of laws targeting multiple firearms restrictions is associated with reductions in firearm deaths. Laws restricting the purchase of (e.g., background checks) and access to (e.g., safer storage) firearms are also associated with lower rates of intimate partner homicides and firearm unintentional deaths in children, respectively. Laws restricting the purchase of (e.g., background checks) and access to (e.g., safer storage) firearms are also associated with lower rates of intimate partner homicides and firearm unintentional deaths in children, respectively.”
But, from Dr. Langmann:
- “Overall this paper is a simple summary of some of the available literature.” Because of the wide variety of approaches in the 130 studies examined, “results [could not be] pooled to produce a statistical summary”. . “Thus there is nothing new.”
- The hypothesis being tested is whether laws imposing “gun control [have any] effect on homicide or suicide or overall firearm caused morbidity or mortality.”
- The authors are willing to credit some benefit from laws even when the studies they reviewed reached contradictory conclusions. Yet by the same standard, “they would have to conclude that concealed carry of firearms is beneficial in reducing violence and homicide. This is a dramatic conclusion and should be mentioned in the abstract and it is not.”
- “Background checks don’t seem to conclusively have any benefit in overall reduction of homicide and suicide.”
- They may have some small “benefit. . . in domestic violence however the results there are hard to present as conclusive.”
- The review suggests “a small reduction in firearm homicide from laws that prevent people with domestic violence restraining orders from purchasing firearms, but laws that remove their possession of firearms or prevent those convicted of domestic violence from obtaining firearms have no effect. This is counter-intuitive and suggests there might be other confounding variables which may prevent domestic homicide rather than firearms restrictions.”
- “Placing studies with different results and conclusions under the same heading on the same graph presents the implication of association when there is not. For instance [in a single figure, one] study looks at different types of homicide while [other studies] look at domestic firearms homicide. These studies cannot be compared.”
- The authors’ conclusion “that laws targeting specific firearms such as ‘assault weapons’ or semi-automatic firearms, magazine size restrictions etc. have no benefits on homicide or suicide” does seem valid.
- “The conclusions from the Canadian studies point “toward. . . firearms laws hav[ing] no benefit on preventing suicide or homicide as the majority [of studies] show no benefit.”
- “One Canadian study cannot [be used as it was to] examine for laws such as licensing and registration of firearms since these were implemented in 2001 and 2003, [and the] study only examines up to year 2004.”
- Dr. Langmann found one of his own studies in the pool, in which he had included types of homicide and many control variables that the authors ignored. He found no “beneficial association between legislation and firearm homicide rates.” This raises questions about the accuracy of their analysis of every other study, too.
- “There have been in fact a few mass victim shootings since” the Australian National Firearms Agreement in 1996, contrary to the authors “statements that there have been an absence of mass shootings since” then.
- “Their claim that one study shows restrictive laws have had a benefit on homicide is actually incorrect. While laws reduced overall deaths, homicide deaths were not affected by specific laws.”
The paper’s abstract is freely available but, in a common offense to scientific review, the full article is only available behind a pay wall for a minimum of $40. Those of us with academic appointments can get it via our university libraries, but we can’t disseminate it further.
We can’t vouch for problems in the rest of Epidemiologic Reviews’ eight other investigations into such things as the relationship between firearms use and alcohol and substance abuse, suicide, domestic violence, storage requirements, “transmission” of behaviors across social networks, and the effectiveness of possible interventions. But it’s not unlikely that such mistakes in design and analysis are frequent there, because they are not uncommon in scientific research generally and are rife throughout social science research like this.
I have to agree with Santaella et al that “High-quality research overcoming limitations of existing studies in this field would lead to a better understanding of what interventions are more likely to work given local contexts.” Higher-quality and better than this review at least.
We can’t pick through every one of the stream of publications coming our way that suggest inherent bias for restrictive gun laws. We have lives, too. But we hope examples like this show the sort of thinking (or lack thereof) that is too often obvious in such work.
It takes expert reading like Dr. Langmann’s to pinpoint the flaws, but the principle remains: even if “association” can be shown, that is far from proving causation.
Check out our resource document, Reading ‘Gun Violence’ Research Critically. If it walks like an anti-gun duck and talks like one, it probably is anti-gun quackery.
— DRGO editor Robert B. Young, MD is a psychiatrist practicing in Pittsford, NY, an associate clinical professor at the University of Rochester School of Medicine, and a Distinguished Life Fellow of the American Psychiatric Association.