To describe the developed method of task-based sampling, we first begin with describing the associated processes of data use within MIDRC. The open data commons (approximately 80% of all MIDRC images and metadata) is published publicly and serves as a large pool from which representative samples can be drawn to develop users’ algorithms. Once a user has finalized their algorithm and is preparing for formalized evaluation or publication, they will be able to submit the algorithm to MIDRC, via secure containers, for formal performance evaluation using data from the MIDRC sequestered commons (approximately 20% of all images and metadata). An overview of this process is shown in Fig. 1. The sequestered commons serves as a large pool from which task-specific samples can be drawn to evaluate the performance of specific user-developed algorithms. Algorithms will be tested on subsets of a cohort matched to a user’s task, to reduce the probability of sampling bias in formal performance assessment. Testing on subsets of the sequestered data will also enable estimates of performance variation and an opportunity to have the algorithm re-evaluated on new data.

Data use pipeline in MIDRC, highlighting use of sequestered data in MIDRC. A subset of the sequestered commons, which is restricted from public access, will be matched to the user’s task for algorithm performance assessment, using the developed algorithm. To maintain the integrity of the sequestered data commons, MIDRC only provides the user summary performance results and not detailed case by case results

In order to evaluate, i.e., assess the performance, of an AI algorithm, one needs to use a specific independent dataset that matches the intended claim and intended clinical attribute distribution for which the AI algorithm is meant. The MIDRC sequestered commons serves as an independent dataset. However, for the testing of a specific AI algorithm, which would have been developed for a specific clinical task and specific intended population, task-based sampling of data from the sequestered commons is needed to extract the relevant dataset. Thus, this process is the focus of this paper.

The work presented here seeks to create an efficient and customizable method for generating the sample set that will subsequently be used for a machine learning algorithm performance test. As such, the sampling method is accurate if the resulting dataset includes attributes in the appropriate way to address the task. That is, the test set follows the expected distribution of multiple demographic and clinical parameters of the intended target population. The importance of the sampling algorithm’s ability to do so is evident when we consider how to test the performance of a particular AI algorithm using data from a large, clinically diverse data repository, where much of the data may not be relevant to the intended task. In such circumstances, the AI developer needs to “pull” or sample data from the larger data repository appropriately. With that need in mind, we submit that the algorithm used for the purpose of sampling from a larger dataset should be evaluated in terms of the resulting distribution of data, in terms of its ability to match that of the intended population. For example, an algorithm developed for evaluating disease severity in a minority population will need a very different sample set for evaluating its performance from an algorithm designed to perform a diagnosis on a national population. While diversity of data samples is often emphasized in manuscripts seeking to evaluate a given algorithm, this is often a byproduct of working from limited available data to build a sample large enough to become representative. However, the most general dataset may not be appropriate for the clinical use case of the algorithm [22]. The developed method seeks to approach the problem of data representativeness from an opposite perspective, where large amounts of diverse data are available from a repository, providing a method for selecting a sample that is relevant to our use case and intended population. The sample is selected to be representative of the target population and thus predictive of the performance of the algorithm in practice on that population, which is required in regulatory review.

Note that if task-based sampling is being used for algorithm performance assessment on sequestered data, the initial cohort is selected internally by the data aggregator, e.g., MIDRC. However, our task-based sampling process was also developed to be utilized for cohort building with public MIDRC data directly, or with other data. If used for cohort building in the open data commons, the initial cohort can be specified by the metadata downloadable from the MIDRC data explorer. In this way, a developer can ensure that a training cohort drawn from the open data commons is aligned with the intended population for their algorithm. For utilization in another data model, modifications by the user would be required as described on the GitHub Read Me documentation for the algorithm [23].

The purpose of the MIDRC task-based sampling algorithm is to extract relevant data from the MIDRC sequestered dataset for a specific clinical task and attribute distribution. To begin this process, the target distribution is user-defined. The definition of the target distribution may include desired proportions for patient demographics, e.g., with respect to age, sex, race, ethnicity, as well as clinical attributes, such as disease type, severity, case difficulty. The full list of demographics/clinical categories used in this paper is included in the Appendix, Table 4. The algorithm presented in this paper is not limited to demographics and clinical attributes and can in principle be used to select a dataset that matches the intended clinical claim and population for any type of categorical attribute. At the beginning of the task-based sampling algorithm, the initial cohort undergoes a preparatory quality check, to ensure all key variables are labeled uniformly, and only unique patients remain. Leaving duplicate patients in the cohort will lead to misleading results since individual patient characteristics will be represented more than once. During this process, patients with incomplete characteristics, or metadata entries left blank, are labeled “Not Reported” in the blank category. During the algorithm sampling process, the subcategory of “Not Reported” is, by default, sampled in the same manner as any other subcategory, such as “Male” or “Asian” in all categories.

Before running the task-based sampling algorithm, the user also decides the maximum allowable deviation in any attribute subcategory, or the maximum attribute deviance metric, that is acceptable to them. This will determine how “close” the user wants to get to their specified attribute distribution in their sample and will also determine the resulting size of the sample (assuming feasibility).

Once beginning the task-based sampling algorithm, in each iteration, the distributions of all attribute subcategories are compared to the target distribution. For specific attributes (such as demographics), differences in all subcategories from the target are calculated. Subsequently, the maximum attribute deviance metric is calculated as defined in Eq. (1), equal to the maximum proportion difference in any attribute subcategory, between the current sample and the target proportion.

$$\begin\mathrm\;\mathrm\;\mathrm\;\mathrm=Max\;\left(\left|Current\mathit\;proportion_i\mathit-Target\mathit\;proportion_i\right|\right)\\\mathrm\;\mathrm\;i\;\mathrm\end$$

(1)

If this value is less than the user-specified maximum allowable deviation, the process is stopped, and the current task-based samples are accepted. If the maximum attribute deviance metric is greater than the user-specified maximum allowable deviation, the process continues.

Next, each patient is assigned a patient attribute-fit deviance metric, resembling a “score” of how the clinical and demographic attributes of that patient contribute to the current sample matching the target distribution. For each patient, the patient attribute-fit deviance metric is calculated according to Eq. (2).

$$\begin\mathrm\;\mathrm-\mathrm\;\mathrm\;\mathrm=\sum\limits_^n\left(Current\mathit\;proportion_\mathit-Target\mathit\;proportion_\right)\ast\mathit}\left(\mathrm\;\mathrm=\mathrm i\right)\\\mathrm\;\mathrm\;i\;\mathrm\end$$

(2)

where I is the indicator function, for example, if a patient is male, I(patient attribute = male) = 1, and I(patient attribute = female) = 0. The patient attribute-fit deviance metric is equal to the sum of the deviances of each attribute subcategory of that patient to the target proportion, where a negative value indicates that more patients from that given subcategory is needed to reach the target proportion, and a positive value indicates fewer patients of a given subcategory are needed. By keeping the positive or negative sign, the sum of the deviances in each patient attribute-fit deviance metric jointly balances how the profile of that patient matches the target distribution.

After calculation of all patient attribute-fit deviance metrics, patients with the highest deviance metrics (positive value indicating less of a given subcategory are needed) are removed from the sample. The rate at which patients are removed from the sample is the greater of p% of the current sample, or N patients, where by default, p = 1 and N = 5. The default values of 1% and 5 patients were determined to be appropriate values by empirical observation of algorithm performance and efficiency on initial samples of approximately 1000 to 10,000 patients. The values of p and N may be modified by the user to accommodate different circumstances and initial cohort sizes. The effect of the choice of these variables will impact how quickly the sample distribution changes and approaches the target distribution. Selecting these variables to be too large compared to the initial sample size will reduce the likelihood the algorithm is able to achieve an efficient sample “matched” to the target distribution. These parameters can be interpreted similarly to an optimization algorithm learning rate hyperparameter. This is discussed further in the results and discussion.

After the patients are removed, a new iteration begins with recalculation of the difference in all attribute subcategories from the target distribution. Once the calculated maximum task-attribute deviance metric is less than the user-specified maximum allowable deviation, the process is complete, and the selected sample may be saved for future use. A diagram of the overall process is shown in Fig. 2.

Schematic of the developed task-based sampling algorithm in a simplified flowchart diagram. Boxes represent actions taken to modify the sample of data at each step while arrows represent the flow of data. Input in this use case is the MIDRC sequestered data commons, but this may be replaced by other data sources

To demonstrate the developed sampling process in simulation, an example of an initial cohort was selected from the public data commons, for the use case of demographics. For the purposes of this example, metadata from 5539 imaging studies (4193 patients) were selected from the public data commons (data.MIDRC.org) to serve as an initial cohort. Patients were identified by Logical Observation Identifiers Names and Codes (LOINC) code within the public data commons. This cohort was selected on March 11, 2023, as imaging studies with selected LOINC properties of Method (Modality) = ‘CT’ and System (Body Region) = ‘Chest’.

Specification of the target demographic distribution was completed by defining desired percentages of each demographic subcategory to be evaluated. For the example generated for this report, a target attribute distribution that approximately matched the CDC demographic distribution, with a 50:50 COVID-19 status, was defined. The full target distribution is specified in Appendix Table 4, but to simplify data within this manuscript, only the demographic categories of race and sex are shown in Table 1. For integration with the developed algorithm and the data model within data.MIDRC.org, a standardized list for the attribute categories of age, race, sex, ethnicity, and COVID-19 status was established, as listed in Appendix Table 4. This list outlines the 26 default subcategories for the five categories listed above.

After definition of the target demographic distribution, the initial cohort was filtered according to the following criteria: patients with listed age as less than 18 or not reported were removed from the sample (n = 5 studies), and multiple entries from the same patient were removed to have only one study per patient remain (n = 1318 studies).

The developed task-based sampling algorithm was applied to the remaining 4016 patients twice, with user-specified maximum allowable deviations of both 10% and 5%. Following selection of the two samples generated with each maximum allowable deviation, we calculated differences in each demographic subcategory from the specified target distribution. To illustrate the capability of using the developed task-based sampling algorithm compared to randomly sampling a database, samples matching the final cohort sizes generated with the 10% and 5% thresholds were randomly drawn from the same initial cohort.