The below article, written by Frederick J. Lee of Boies, Schiller & Flexner LLP, and Dr. David Tabak of NERA Economic Consulting, was originally published in Law360's Expert Analysis section. An extended version is also available on the NERA website, here.
Materiality And Statistical Significance Explained
By Frederick J. Lee, Boies, Schiller & Flexner LLP, and Dr. David Tabak, NERA Economic Consulting
Law360, New York
December 16, 2010
The Supreme Court will soon consider whether information needs to be statistically significant for it to be material in an appeal of a shareholder class action from the Ninth Circuit by Matrixx Initiatives Inc. Several doctors and plaintiffs between 1999 and 2004 claimed that Zicam nasal spray caused loss of smell. While there had been some 12 to 23 adverse event reports (AERs) of users losing their sense of smell after using the popular cold remedy, the reports were just stories, and the number of reports was miniscule relative to the millions of units sold. The number was so small that Matrixx, the drug’s manufacturer, described it as not statistically significant.
On Feb. 6, 2004, Good Morning America aired a segment alleging that Zicam causes users to lose their sense of smell. Matrixx’s stock price plummeted, and a federal securities class action lawsuit quickly followed, alleging that the company should have disclosed the adverse event reports. The question currently pending before the court is whether undisclosed information can be material when it is not statistically significant. To understand this question, one must understand both statistical significance and materiality.
Statistical Significance: Definition and Discussion
A discussion of how materiality and statistical significance are connected in the law requires an understanding of both concepts. One publication for lawyers that provides a helpful discussion of statistical significance is the "Reference Guide on Statistics," a part of the "Reference Manual on Scientific Evidence, Second Edition," published by the Federal Judicial Center.
According to the Reference Guide on Statistics, statistical significance is determined by “comparing a p-value to a pre-established value, the significance level.” A p-value “is the ‘probability of getting data as extreme as, or more extreme than, the actual data, given that the null hypothesis is true.’” Id. A high p-value means the data, or data more extreme, are more expected or likely, given that the null hypothesis is true. A low p-value means the data, or data more extreme, are less likely, given that the null hypothesis is true. Once the p-value drops below the pre-established value, the analyst rejects the null hypothesis and considers the data statistically significant.
Although this definition should feel intuitive to an expert in statistics, it will often sound confusing to others. To understand the definition, consider the case of flipping a coin as a test of whether the coin is unbiased (i.e., whether it should come up heads 50 percent of the time and tails 50 percent of the time). Suppose that we flip the coin twice. If we used a fair coin, the probability of getting both flips in the same direction (HH or TT, where H and T represent heads and tails, respectively) is represented by two out of the four possible outcomes, or 50 percent. The probability of getting one flip in one direction and one in the other is 50 percent (HT or TH).
Our null hypothesis for this test is that the coin is unbiased, meaning that half the flips should be heads and half should be tails. With two flips, a result of HH or TT would have a p-value, also known as the observed level of statistical significance, of 50 percent, because that is the likelihood of getting results as extreme or more extreme than 100 percent in one direction if the coin was fair (i.e., if the null hypothesis that the coin is unbiased is true). Any result with one flip of one type and one of the other would have a p-value of 100 percent, consisting of the 50 percent for an outcome as extreme of two flips in the opposite directions plus the 50 percent for the more extreme outcome of both flips in the same direction. While a p-value of 100 percent, or 1.0, may sound strange, its interpretation is in fact quite intuitive: if the coin is fair, we will always get results at least as extreme (i.e., as far from 50/50) as having one flip in each direction.
One important issue in considering how statistical significance relates to materiality is the difference between practical and statistical significance. In a nontechnical sense, statistical significance examines some feature of the data and asks the question, “How unusual?” while practical significance looks at that feature and asks, “How large?” For example, imagine a diet pill that causes people to lose one ounce of weight over the course of a month. Suppose that by carefully controlling all the food intake and exercise of the group receiving the diet pill and the group receiving a placebo, we reduce the other sources of variation of weight change in each group and show that there was a difference between the two groups, of one ounce of weight loss per month, that is of a magnitude that is highly unlikely to be caused by chance alone (the definition of statistical significance). The result would then be statistically significant. But a weight loss of one ounce per month, less than a pound per year, would generally not be considered practically significant, meaning that it is not, in some sense, important.[1]
Materiality depends on a combination of statistical and practical significance. Just as a small, statistically significant amount of weight loss may not be material, so too might a finding of a large amount of weight loss in a single person: while the amount may be large (practically significant), it might be a weight change that could easily happen by chance alone (statistically insignificant).
Materiality
The “Total Mix” of Information
In TSC Industries Inc. v. Northway Inc., the Supreme Court held that an omitted fact is material if there is a “substantial likelihood” that a “reasonable investor” would have considered its disclosure to have significantly altered the “total mix” of information available.[2] The court later expressly adopted this standard for § 10(b) and Rule 10b-5 causes of action.[3]
Although statistical significance can be a factor to consider under this standard, it does not encompass the “total mix” of information available to market participants. Nonfinancial information may not be material even if statistically significant. For example, if most patients developed a minor cold from a product that accounts for less than one percent of a pharmaceutical company’s revenues, the medical effects might be shown to a high degree of statistical significance, but the financial implications may be trivial and, therefore, immaterial.
Probability x Magnitude
Although the court’s “total mix” standard in TSC Industries remains the touchstone for materiality, its broadness and fact-intensive nature have engendered several alternative or supplemental standards. In Basic Inc. v. Levinson, the court expressly adopted the TSC standard in the § 10(b) and Rule 10b-5 context, but it noted that the standard “admits straightforward application” only where the impact of information is “certain and clear.”[4] Where events are “contingent or speculative in nature,” materiality depends instead on “the indicated probability that the event will occur and the anticipated magnitude of the event.”[5]
Recall that statistical significance relates to how unusual a result is, and particularly the probability that results as extreme as, or more extreme than, those actually observed would be found if the null hypothesis is true. Thus, statistical significance has some relationship to the idea of probability in Basic, though these are measurements of different forms of probability. Practical significance, on the other hand, goes to magnitude. Thus, statistical significance will speak to the probability factor of Basic’s formula, but not the magnitude.
For data other than securities’ prices, all that statistical significance can tell us is whether the data are unusual, meaning unusually different from some hypothesized level. For example, if there were the same number of reports of death and sneezing, and if the background rates of the two were the same, the measurements of statistical significance for these two medical outcomes would be identical.[6] The importance of the two types of adverse reactions, however, is very different both medically and with respect to the valuation of the company’s stock. With stock price data, some courts have used statistical significance as a proxy for materiality under the assumption that if a stock price movement is not statistically significant, the misstatement or omission must have been immaterial. This assumption may be justifiable if a statistically insignificant stock price movement (originally meaning that it signified nothing) precludes a finding of loss causation.
Rule of Thumb
The desire for certainty has led some lower courts to adopt an even simpler standard of materiality: a rule of thumb. In Ganino v. Citizens Utilities Co., for instance, the Second Circuit reversed the district court’s holding that an event is material if it “might affect a company’s earnings, positively or negatively, by 3 percent to 10 percent.”[7]
A rule of thumb is arguably loosely tied to statistical significance in that both examine deviations from an expected value. The difference is that rules of thumb are often given to be the same for all firms and for all types of financial (and potentially nonfinancial) measures. In contrast, statistically significance is tailored to the particular data being measured, so that it may take a three percent change in quarterly sales for one company’s misstatement or omission to be statistically significant at the standard five percent level, but quarterly changes of eight percent for another company.The rule of thumb, while attractive in its simplicity, has yet to gain much legal traction.
Market Reaction
Finally, we come to a metric with perhaps the most promise but also the greatest pitfalls: the market. If materiality is what a rational investor considers pertinent, then there is no better measure than the market price, which in an efficient market should impound all material information of a certain type.[8] Here, there is a crucial difference between the statistical significance of a market reaction and the statistical significance of the underlying information. The underlying information may be statistically significant, but its magnitude or practical significance may be small, resulting in a statistically insignificant market reaction.
Conclusion
Statistical significance can help litigators and courts examine materiality. The first step to using the tool is to understand what statistical significance does and does not show. Statistical significance sheds light on how unusual certain results or data are, and specifically how unlikely it would be to see such results if there were no true causal effect or relationship present. Statistical significance, however, does not go to the magnitude of an effect, a concept referred to by terms such as practical, economic or clinical significance. Consequently, depending on the time and the data being examined, statistical significance has different implications for whether information would be material for investors.
--By Frederick J. Lee, Boies, Schiller & Flexner LLP, and Dr. David Tabak, NERA Economic Consulting
Frederick Lee ([email protected]) is an associate in Boies, Schiller's New York office. David Tabak ([email protected]) is a senior vice president of NERA Economic Consulting in New York.The opinions expressed are those of the authors and do not necessarily reflect the views of their firms, their clients, or Portfolio Media, publisher of Law360.
[1] For additional information, see the Reference Guide on Statistics, p. 124 and the Reference Guide on Multiple Regression, p. 190.
[2] 426 U.S. 438, 449 (1976).
[3] Basic Inc. v. Levinson, 485 U.S. 224, 232 (1988) (“We now expressly adopt the TSC Industries standard of materiality for the § 10(b) and Rule 10b-5 context.”).
[4] Basic, 485 U.S. at 232.
[5] Id. at 238 (quoting SEC v. Texas Gulf Sulphur Co., 41 F.2d 833, 849 (2d Cir. 1968)).
[6] This assumes that such measurements are even possible given the nonsystematic nature in which AERs are collected.
[7] 228 F.3d 154, 161 (2d Cir. 2000).
[8] Under the semi-strong form of the efficient market hypothesis, an efficient market impounds all relevant, publicly available information, but not private information.