Statistical sampling is routinely used in audits and investigations when seeking to reach conclusions about large volumes of data. A probe sample can be a key tool in increasing the efficiency and reducing the cost of statistical sampling analysis. A probe sample can also be helpful in identifying risk and quickly evaluating whether a full statistical sampling is required. More importantly, when planned and implemented properly, a probe sample can also be incorporated as part of future sampling analysis to eliminate duplicated efforts and to further minimize cost and effort. This post discusses the proper design and implementation of probe samples, with particular emphasis on developing valid and defensible analysis of healthcare claims.
“Full” Statistical Sample
To begin, it is worthwhile to understand the purpose and role of statistical sampling in the first place. Statistical sampling analysis is most commonly used when one seeks to infer useful information about a relatively large population without examining every unit in the population, and instead by examining only a subset of that population (i.e. a sample). As part of sampling analysis, estimation or extrapolation is a procedure by which measured characteristics of a sample yield estimates, inferentially, about unknown characteristics of the population from which the sample was drawn. The term “probability” or “statistical” sample arises from the fact the sample is selected in a manner that is predictable in terms of the laws of probability, which eliminates both conscious and unconscious selection bias on the part of individuals performing sample selection. Such a sample must be obtained in a certain way (i.e. randomly), to be objective and defensible. Review an in-depth discussion of statistical sampling here.
A “full” statistically valid sampling analysis may involve the selection of several hundred sample units (e.g. patients, claims, etc.) in order to achieve results within a specified degree of uncertainty, which is typically described in terms of confidence and precision. Such analysis may be time consuming and expensive, particularly when the objectives of a particular audit are somewhat uncertain. For example, a hypothetical analyst auditing a healthcare system with 12 facilities and 45 providers may need to sample over 300 patient records to conduct a “full” sampling analysis. This can be cost prohibitive, particularly for routine audits where the existence of overpayments may not even be anticipated.
Using a Probe Sample or Discovery Sample
Enter the Probe Sample, sometimes called a Exploratory or Discovery Sample. Unlike a “full” sample, a probe sample is not typically designed to achieve specified levels of confidence and precision. Instead, it is used to determine a net financial error rate, and thereby to indicate whether further analysis (i.e. a “full” sample) is warranted. For example, an analyst may select 30 sample units for a probe sample, then audit those 30 claims to calculate the percentage of overpayments in the sample. If the percentage exceeds some established threshold (e.g. 5%) the analyst could set aside the data for a “full” sampling. However, if the error rate does not exceed the threshold, the analyst might end their audit, finding that insufficient errors exist to justify further analysis. Such a conclusion could save significant cost and effort, while also allowing the analyst to focus on other areas of greater risk.
Benefits of a Probe Sample
While a smaller sample size is an obvious benefit of probe samples, another advantage is that the results may be “re-used” as part of the full sampling, if such follow-on analysis is necessary. In fact, OIG addresses this concern specifically stating “OIG will allow, if statistically appropriate, the discovery sample (as a whole) to be used as part of the full sample.”[i] This can save significant cost and effort. For example, if the analyst determines a sample size of 200 claims to be necessary in the “full” sample, they may use the results from each of the 50 claims in the probe sample. Therefore, the analyst only has to randomly select and review an additional 150 claims. The key here is that the probe sample be statistically appropriate for inclusion in the full sample.
How Is a Probe Sample Determined?
Much like a “full” sampling, designing and executing a probe sample should be approached methodically to ensure the conclusions of the analysis are defensible and valid. More importantly, it is also worthwhile to understand the methods by which the probe sample is designed and selected. Designing a probe sample with the mindset that it may need to be incorporated in a “full” sampling is imperative. Without this foresight, the probe sample may not be statistically appropriate for use in a “full” sampling. Similarly, a probe sample should be properly randomized and selected. Samples obtained by any method other than random selection are generally considered to be “judgment” samples. Judgment samples typically result from haphazard selection or by means of convenience (i.e. choosing charts equally from each provider). While the results of these judgement samples may be informative, they are not appropriate for use in a “full” sampling, thereby requiring analysts to duplicate efforts and ultimately sample greater numbers.
Ensuring a Statistically Valid Probe Sample
Ensuring a probe sample is designed to be statistically valid is critical. Too many analysts wait until the probe sample is reviewed before considering whether the probe results can be re-used. By then, it is often too late to avoid duplication of sampling efforts. Analysts should confer with their statistical experts in advance to identify necessary planning considerations of a probe sample. Similarly, the use of appropriate statistical software, such as RAT-STATS, can help to ensure the probe sample is properly randomized and selected. Review an in-depth discussion of RAT-STATS here.
Conclusion
Probe samples may be useful in reducing the amount of time and effort required in matters involving statistically valid sampling analyses. To best utilize probe samples, they should be appropriately designed and selected in a manner that will allow them to be incorporated into a full analysis. If properly executed, probe samples can effectively be used in healthcare audits, investigations, self-disclosure, and CIA claim reviews.
[i] https://oig.hhs.gov/faqs/corporate-integrity-agreements-faq.asp