The Benchmarking Tool

Background

The National Cancer Institute Surveillance Research Program (NCI-SRP) has developed a new quality control tool allowing Surveillance, Epidemiology, and End Results (SEER) registry managers to monitor their registry data quality. The Median/Multiple Outlier Testing Method (MMOT) is a benchmarking tool for identifying outliers using multiple hypothesis testing based on the median of the statistic of interest (e.g., proportion unknown). The goal of this method is to identify any outlier datapoint from one SEER registry for a given diagnosis year. This will allow registry managers to investigate issues in data collection, coding, or registry operations beyond characteristics attributable to the underlying patient population.

The Median/Multiple Outlier Testing Method (MMOT)

1. Define the population of interest e.g., primary site, diagnosis years
2. Calculate the statistic of interest e.g., proportion unknown for summary stage
3. Multiple hypothesis testing defines the boundaries, identifying outliers based on the median
   • If a registry for a given year falls above the upper boundary then the statistic of interest is higher when compared to the other registries
   • Similarly, if a registry for a given year falls below the lower boundary the statistic of interest is lower when compared to the other registries

Data Quality Monitoring Plan

Currently, SEER routinely evaluates the following data items from the most recent November submission using this MMOT tool. However, benchmarking analyses are not limited to these data items.