Not All Tests Are Created Equal

Though the state has outlined which tests must be performed on which products, as well as the established limits as to the permissible quantity of contaminants in a sample, certain tests are of more interest than others in terms of monitoring the safety of marijuana products. Foreign Matter, for example, is off less interest to policy makers than Microbial and Residual Solvent testing. Microbes or residual solvent found in marijuana can have serious health implications for consumers. A more scrupulous evaluation of these tests is necessary for evaluating the safety of these products.

Measuring the Amount of Contaminants in Marijuana Samples from Producers (Growers)

Labs upload lab results into the Washington State Database (with few exceptions). The lab results include the sample number, the tests run on that sample, the value of the content found for each test, whether or not the sample passed or failed the test, as well as the time and date of the test.

The value recorded is different for each test. For example, for residual solvent tests, results are measured in parts per million (ppm). The limit set by the state for residual solvent testing is 500 ppm. Any sample containing more than 500 ppm fails the test. Many microbial and mold tests are measured in Colony Forming Units per gram (CFU/g), though some are measured in grams. Moisture content measures the percent of the sample that was moisture.

The graphs below illustrate the density distribution of the values of contaminant found for each test, divided into passing and failing groups. The vertical red lines illustrate the maximum amount of contaminant found for each test type. For the yeast and mold test, the red line is for flower products and the black line is for edibles.

Foreign Matter, Moisture Content and Residual Solvent tests are straight forward tests that yield one result—a value and an indicator of pass or fail.

Microbial tests are subdivided into 5 subtests. Lab results for microbial tests include a value and an indicator of pass or fail for each of the 5 subtests performed on the sample.

It appears that there are no passing tests that have a higher value than is permitted by the state. Though in some cases, for instance Foreign Matter and Bile Tolerant Gram Negative Bacteria tests, there is a high number of samples with values just below the limit.

The Number of Samples that Each Lab Fails is Relatively Low—Some Labs Even More So than Others

One of the ways to gauge the reliability or trustworthiness of a lab is looking at its fail rate (the number of samples that fail out of all samples tested. Interestingly, it appears that the labs with the highest market share, that is to say labs that process high numbers of tests, tend to offer lower fail rates. The graphs below illustrate the overall fail rates for each lab. Each letter along the horizontal axis represents a lab. The labs are arranged in order of increasing market share, meaning the labs farthest to the right process the most tests. Foreign Matter, Micro Screening and Moisture Content are shown together since they have similar fail rates. Residual Solvent is shown separately with a larger range on the y-axis because there are higher fail rates for this test—a good sign since residual solvents pose the highest risk to human health.

Fail Rates drop and stay low over time

The legal marijuana market has grown rapidly since its opening in July 2014. At the start of the market, there were fewer labs and more have opened over time. The labs have had to adjust to the demands of the state and the producers as well as compete with one another for business. One of the critical factors on which labs compete is fail rate. Producers have complete liberty to choose where they send their samples. If a lab is not giving them the results they are hoping for, the producers have total freedom to choose another lab. The animation below illustrates how the fail rates have changed for each lab over time. Each ball represents a lab. Time is represented through the animation (monitoring the labs month by month). The horizontal axis measures the total number of tests processed, representing increasing market share. The vertical axis measures the monthly fail rate. Balls that move faster towards the right are gaining market share quickly. Labs that move slowly or do not move, do not process many tests.

Initial Period of Fluxation before Labs Settle into a Pattern

It appears that often when a new lab initially opens, there may be high fail rates that then come down over time. While this may in part be due to the newness of the legal market overall, when we look at the fail rate given the number of months a lab has been open as opposed to their fail rate over time that the market has been open, this spike appears to be more closely related to the lab opening than to the market opening.