[Ed: After testifying to the House Appropriations Committee in 1996, Dr. Faria was tapped to serve at the CDC on the NCIPC‘s grant review committee during the George W. Bush administration. This two-part series (Part II here), republished with permission, describes his tenure there. Originally published by World of Neurosurgery in February, 2007]
During the years 2002 to 2004, I served in the Injury Research Grant Review Committee (more recently the “Initial Review Group”) of the Centers for Disease Control and Prevention (CDC)—more specifically, the National Center for Injury Prevention and Control (NCIPC).
I participated not only in the major meetings in Atlanta, but also in on-site reviews and inspections of Injury Research Centers reviewing thousands of pages of grant applications requesting funding for medical and public health scientific research proposals. I have deliberately let some time elapse before writing this analysis with the purpose of being able to take a step back and write from a distance, objectively.
I should also inform the reader that I must write in generalities, for I cannot disclose by CDC rules specific details of any grant proposal requesting funding, or discuss the content of the review of any specific grant application in which I participated or that came to my knowledge while working at the CDC in the capacity of grant reviewer. This secrecy seems, in retrospect, even more stringent than those that were in place at Los Alamos during the Manhattan Project! So my discussion necessarily will be lacking specific examples to illustrate the thread of my arguments. Nevertheless, I ask you to bear with me, gentle reader, for enough general material discussing points of scientific interest will, I think, make it worth your while—that is, if you have an interest in the present interrelationship between public health, social science, and the purported relationship these disciplines bear today with medicine, including neuroscience, and the scientific method.
Before proceeding, as a further introduction, I would like to quote several excerpts from a magnificent article entitled, “Statistical Malpractice” by Bruce G. Charlton, MD, of the University of Newcastle upon Tyne. It is perhaps no coincidence that Dr Charlton is associated with the same great university that gave us Dr John Snow, the illustrious physician who in 1849 successfully applied the scientific method to epidemiology. (In the process, Dr Snow proved that cholera is a waterborne disease. This discovery led to the conquest of epidemic diseases such as dysentery and typhoid fever.) Dr Charlton’s comments that follow cite the growing misuse of pure epidemiology and statistics as science. As my narrative unfolds, the relevance of these momentous passages to my narrative will become obvious.
Science versus Statistics
There is a worrying trend in academic medicine which equates statistics with science and sophistication in quantitative procedures with research excellence. The corollary of this trend is a tendency to look for answers to medical problems from people with expertise in mathematical manipulation and information technology, rather than from people with an understanding of disease and its causes. Epidemiology [is a] main culprit, because statistical malpractice typically occurs when complex analytical techniques are combined with large data sets…. Indeed, the better the science, the less the need for complex analysis, and big databases are a sign not of rigor but of poor control. Basic scientists often quip that if statistics are needed, you should go back and do a better experiment…. Science is concerned with causes but statistics is concerned with correlations…. Minimization of bias is a matter of controlling and excluding extraneous causes from observation and experiment so that the cause of interest may be isolated and characterized…. Maximization of precision, on the other hand, is the attempt to reveal the true magnitude of a variable which is being obscured by the “noise” [from the] limitations of measurement.
Science by sleight-of-hand
The root of most instances of statistical malpractice is the breaking of mathematical neutrality and the introduction of causal assumptions into the analysis without scientific grounds…
Practicing science without a license
…Medicine has been deluged with more or less uninterpretable “answers” generated by heavyweight statistics operating on big databases of dubious validity. Such numerical manipulations cannot, in principle, do the work of hypothesis testing. Statistical analysis has expanded beyond its legitimate realm of activity…. From the standpoint of medicine, this is a mistake: statistics must be subordinate to the requirements of science, because human life and disease are consequences of a historical process of evolution which is not accessible to abstract mathematical analysis .
The following discussion is intended to describe some problems I encountered in the honorific service of grant reviewer, as specified above, while making a few critical observations that will hopefully improve the quality of grant proposals at the CDC.
Moreover, research appropriateness and cost-effectiveness of these grant proposals are included in this discussion because the allocation of public health research funding is of concern to the American public (taxpayers), who are already shouldering a significant burden of their own rising health care costs and yearly are asked to increase that burden in tax dollars for social and public health research programs that may yield relatively little in terms of cost-effectiveness.
Despite definite improvements in the last several years, it is my opinion, after working in the belly of the NCIPC beast, so to speak, that much work still needs to be done. Much public health research in the area of injury control and prevention is duplicative, redundant, expensive, and not cost-effective. Many of the grant proposals for the study of spousal abuse, domestic violence, inner city crime, juvenile delinquency, etc, belong more properly in the sociologic and criminologic disciplines than in the realm of public health and medicine. When we discuss the lack of science in many of these proposals, it is not surprising that I found (and many reviewers privately acknowledged) that the scientific method, or for that matter, the application of the public health model, is found wanting when applied to the study of violence and crime. But funding continues to roll into the CDC’s NCIPC and it continues to be squandered and misapplied by public health researchers.
Perhaps the biggest problem of all has been created and promoted by Congress in the allocation of ever-increasing amounts of taxpayer dollars to public health “research” in the area of injury control that, frankly, in many dismaying instances is of questionable scientific validity and even less cost-effectiveness. Oversight, accountability, and clear demonstration of cost effectiveness have been clearly lacking. And yet, the Department of Health and Human Services shares some of the blame as the executive federal agency has considerable leeway in how these tax dollars are spent by the CDC and the public health establishment.
The unfortunate reality is that in the areas of injury (and “violence”) research, much money is being wasted in programs of questionable medical value and scientific validity. Many of the grant proposals submitted in the name of “violence prevention” and other goals of the Healthy People 2010 agenda are generally geared toward promoting social engineering and enlarging the scope and collective role of government in the lives of citizens. For example, numerous times I opposed and voted against proposals that required intrusive home visits to families by social workers (“home observations”), raising serious concerns of privacy violation. Ditto for proposals establishing databases of dubious validity where no hypotheses were tested. Behind many of the health proposals is a compulsive need to protect people from themselves, as if they were children requiring the helping hand of health academic bureaucrats for the people’s own survival. All of these collective patronizing social programs, unrolling year after year, are borne out at the expense of personal freedom, not to mention taxpayers’ pocketbooks.
Likewise, many of the proposals made in response to Healthy People 2010, year after year, sponsored by the CDC’s NCIPC, have more to do with “social justice,” expanding socialistic agendas, and perpetuating the welfare state than with making genuine scientific advances and improving the health of humanity. In some cases, these grant proposals (many of which are actually funded) use or misuse statistics, collect data of dubious validity, and even use “legal strategies” to reach social goals and formulate health care policies that the public health researchers believe may achieve “social justice.” Many of these studies aimed at nothing less than the gradual radicalization of society using “health” as a vehicle [, , , , ]. “Scientific” peer review in many instances, frankly, is not working or is nonexistent. The reader will be surprised to learn that I found probably as many lawyers and social workers as practicing physicians and nurses applying for public health “scientific” research grants!
Healthy People 2010 and injury control programs (particularly “Grants for Violence”) have become vehicles for the creation of ever-increasing social programs (and the remodeling of society in the image of public health researchers) through the back door. “Health” and “science” are used as covers because, to a significant extent, many of these proposals have relatively little to do with improving the general health of the public and even less with scientific merit.
Frequently, proposals under these Grant Announcements (perhaps with some notable exceptions for Traumatic Brain Injury and Acute Care and Rehabilitation) are submitted to dovetail previous social “research” in which statistical significance is frequently not established in prior studies. But the money keeps rolling in, and more shoddy research is funded, year after year.
SIMPLE STATISTICAL TOOLS FREQUENTLY MISSING
From the scientific point of view, a trend most troubling is the misuse or nonuse of the very simple but very helpful traditional statistical tools in the statistician armamentarium. I refer to the useful methodology of relative risks (RRs), confidence intervals (CIs), and the increasingly ignored P values. These traditional statistical parameters are essential in determining the strength of statistical associations. These tools are actually tough tests that are applied to statistical studies to ascertain their validity. Although these tests do not establish cause-and-effect relationships, they are essential in the process of establishing statistical associations, particularly in these social “science” (sociological) investigations that are carried out now routinely under the aegis of public health research. Their time-proven places in statistics are being replaced by complex, inscrutable statistical methodologies, such as regression models, multivariate stratified computer analysis, and other complex statistical manipulations that, devoid of the required clinical experience, befuddles everyone including the statisticians themselves!
FISHING EXPEDITIONS IN SEARCH OF SOCIAL PROBLEMS
Not infrequently I found it difficult to discern in any of these ever-proliferating health (social) proposals strong statistical associations leading to groundbreaking, scientific research. Fishing expeditions in hypothesis searching and solutions in search of social problems are frequent, whereas hypothesis testing is poorly formulated. One reason for this misuse of statistics universally ignored is that epidemiology should be applied to rare diseases that occur at high rates in a defined segment of the population preferably over a short defined period of time (eg, lung cancer deaths in smokers, or even better, an outbreak of salmonella or shigella poisoning at a convention hall on a specified date, etc) and not applied to low rate of disease or injury extending over a long period and over vast populations (eg, ongoing studies of juvenile crime in the inner city, health consequences of environmental toxins over a large population, etc). Moreover, investigation of these diseases or injuries should be carried out in conjunction with clinical investigation.
Koch’s Postulates of Pathogenicity seemed to have been completely forgotten by the public health establishment. Koch’s Postulates, as the reader will recall, are the steps that a medical researcher follows in proving that a specific microorganism causes a specific disease: (1) The organism must be found in diseased animals and not in healthy individuals. (2) The organism is then grown and isolated in the laboratory in pure culture. (3) The organism from such culture should cause the disease in susceptible individuals upon inoculation. (4) The pathogenic organism must be reisolated from the diseased individuals and cultured again in the laboratory .
Epidemiological studies, whether case control or ecologic, are unreliable population-based investigations that should not be routinely used for investigating the etiology and course of common diseases, not to mention preventing injuries and violence among the general population. And yet, despite these objections, epidemiological studies are funded routinely to investigate the pathophysiology, course, and outcome of common diseases and injuries that occur slowly or repeatedly over extended populations.
PREMATURE DISCLOSURE OF “SCIENTIFIC FINDINGS”
It is no wonder that the media picks up on these reports prematurely and sensationalizes conclusions that often contradict one another as soon as they are published, sometimes in the same issue of the same journal! Disconcertingly, we have learned from researchers and the media that coffee can cause cancer as well as prevent cancer, and that silicone breast implants are harmless and that they are not, etc! We continue to be bombarded with prematurely reported headline grabbing studies, day after day, to the detriment of the trust that the public has vested in science and medicine. Prevention of premature disclosure of scientific findings to the media in taxpayer-funded research is something that both the directors of the CDC and NCIPC may be able to do something about.
The public, befuddled and confused, loses faith in “science” and looks elsewhere for answers, conducting its own research on the internet, not always with salutary results. Public health proposals commonly tout their complex statistical analyses (eg, multivariate and regression analysis to eliminate confounding variables, etc), yet important statistical tools, such as P values, RRs, and CIs, are frequently not even mentioned in the body of the research!
Although RRs do not establish cause-and-effect relationships, it is an invaluable tool in statistics. Relative risk is used to determine whether there is a difference in the rate of a disease process or injury between 2 given populations. A RR of 1.0 signifies no difference. A RR of 2.0 means that the exposed population has twice (100%) the rate of disease (a positive association) as compared to the other population. Statistics is not science, and a 100% increase in this context is a very small number that could be explained solely by the quality of the data, thus denoting a weak statistical association.
Likewise, a RR value of 0.5 carries a risk of half (50%) as compared to the other population (a negative association) and likewise conveys a weak statistical association. Thus, experienced statisticians ignore the statistical significance of a RR ranging between 0.5 and 2.0 which relates no significant risks between the rate of disease or injury in the 2 populations under study. Remember, RR values between 0.5 and 2.0 do not reach statistical significance. These figures should be noted, but can be safely ignored (no cause-and-effect relationship) by the epidemiologist. To find out further whether a RR value outside those limits truly carries any statistical significance, 2 other tools are used: The P value, the killer test that determines whether the difference in the 2 study groups are due to chance, and the CI .
A P value of .05 corresponds to the 95% level of confidence that traditionally has been set by scientists as the gold standard for epidemiological research. A 95% CI represents a range of values within which we are certain the true value of the RR is found (with 95% confidence). One must be aware that many public health researchers want to lower this standard to 90% (increasing uncertainty) to increase the chance of reaching statistical significance and confirm the validity of their research findings. Other researchers do not disclose P values, neglecting the possibility that their findings are due to chance. Those researchers who report and pass the P value (P < .05) test usually disclose it in their studies; those who ignore it make their study suspect and their conclusions may not have reached the touted statistical significance they profess to have reached in their conclusions. As we shall see, the 95% CI corresponds to a P value of .05, but low P values, even as low as P = .006 or P = .009, do not establish causation; they only denote strong statistical associations not due to chance. Statistics is not science and cannot prove cause-and-effect relationships. For an amusing yet informative review of this subject for the busy clinician, neurosurgeon, and neuroscientist, I strongly recommend Steven J Milloy’s book, Junk Science Judo, published by the Cato Institute in 2001 .
The next step that should be performed to test RR for statistical significance is the 95% CI. Scientists know that the true RR value is found with a 95% level of confidence. When a RR is greater than 1.0, the range (for both the lower and upper boundaries) must be greater than 1.0 to satisfy the CI test. Conversely, when the RR is less than 1.0, to satisfy the CI test, the range of CI values must fall below 1.0 (again both lower and upper reported boundaries included). Neither statistical association must include 1.0 within the range. If 1.0 is included in either range, a statistical significance is not established.
Frequently, I found that one or more of these traditional tools (RR, P value, CI) are not mentioned or are incompletely disclosed in public health research proposals, apparently to avoid disclosing the inconvenient fact that drawn conclusions do not reach the level of statistical significance. Weak statistical associations, if honestly reported, would have disqualified them from reaching their preordained conclusions. And therein lies the rub: full reporting of values reaching no statistical significance may lead to no funding of further research grants! Thus, many grant research proposals in the violence prevention area of public health pad their numbers, increase uncertainty (90% CI), ignore P values, and tout RRs that have no statistical significance—to apply and reapply for more funding for shoddy research grants. Remember, RR values between 0.5 and 2.0 denote no difference in rate of disease or injury between the 2 tested populations and carry no statistical significance. The difference may be due solely to the quality of the data.
Another situation arises when RRs (cohort studies) or odds ratios (case control studies) are mentioned, but then the discussion proceeds as if statistical significance has been reached (it had not), and on the basis of statistically insignificant findings, presto, further social programs are proposed masquerading as ongoing public health studies. The money keeps rolling in. Broad hypotheses that should have been refined or discarded are kept and tested and retested. And, in subsequent studies, efforts at hypothesis searching are amplified (ie, fishing expeditions), eventually obtaining data of poor quality (ie, data dredging) but conducive to “positive” results.
In the process, databases of dubious validity are compiled for further social research! Large data sets are hardly ever verified for accuracy and are used by some researchers in the hope of finding statistical association where none exists (ie, data mining).
In other words, much of this work entails finding solutions in search of problems, proposals in which narrow hypotheses are lost in nebulous seas of data. Immense sets of data are collected, tested, and retested, and statistics are tortured until finally they are made to confess to reach predetermined social objectives. These social goals are then put in place for dissemination to health policy makers, who believe that these conclusions have arisen via the arduous trials of the scientific method. More funding is allocated to implement public policy, and for more public health (social) research. And the beat goes on!
— Miguel A. Faria, Jr., M.D. is a retired Clinical Professor of Neurosurgery and Adjunct Professor of Medical History at Mercer University School of Medicine. He is Associate Editor in Chief and World Affairs Editor of Surgical Neurology International. He served on the CDC’s Injury Research Grant Review Committee.
- Arnett JC. Book review: Junk science judo by J. Steven Milloy (ed). Med Sentin 2002;7(4):134-5.
- J.T. Bennett, T.J. DiLorenzo (Eds.) From pathology to politics—public health in America. Transaction Publishers, New Brunswick (NJ); 2000: 21–115
- Brock, T.D. Biology of microorganisms. in: Prentice Hall Inc, Englewood Cliffs (NJ); 1970: 9–12
- Charlton, B.G. Statistical malpractice. J R Coll Physicians Lond. 1996; : 112–114
- Faria, M.A. Public health—from science to politics. Med Sentin. 2001; 6: 46–49
- Faria, M.A. The perversion of science and medicine (Parts I-II). Med Sentin. 1997; 2: 46–53
- Faria, M.A. The perversion of science and medicine (Parts III-IV). Med Sentin. 1997; 2: 81–86
- Milloy, S.J. Junk science judo—self-defense against health scares and scams.(www.cato.org) in: Cato Institute, Washington (DC); 2001: 41–114