Programmatic assessment has been framed as essential to gauge learners’ progress and inform educational strategies in competency-based curricula [1]. At the macroscopic level, programmatic assessment has been operationalized as frequent, low-stakes assessments with meaningful, constructive feedback and coaching [2]. Implementing programmatic assessment, however, has been plagued by issues with translating macro-level assessment changes into direct, in-the-moment benefits to learners [3]. In a case study of the implementation of a programmatic model at Flinders University Medical School, King et al. [4] reported that curriculum developers realized that an emphasis on the analysis of the psychometric properties of items led to a foregrounding of the assessment-of-learning function of tests versus their assessment-for-learning function. Lisk et al. [5], likewise, point to the need for educators to consider how assessments function to influence learners’ cognition to support durable retention, transfer, and preparation for future learning.

Some researchers have turned to the literature on test-enhanced learning for well-supported solutions. Test-enhanced learning refers to using testing to enhance, rather than assess or measure, learning through a variety of cognitive, metacognitive, affective and motivational mechanisms [6] The direct effect of testing can be harnessed to reinforce memory for studied information simply through retrieval practice; information that is successfully retrieved during a test is more likely to be successfully retrieved in the future [7]. But testing can have effects beyond simply enhancing the retrieval of information. Testing has indirect effects that influence both current and future learning. For example, the kinds of cognitive processing evoked when responding to tests may be important in the development of clinical reasoning [8, 9]. The foundation of clinical reasoning—and the hallmark of expertise—is a rich, interconnected network of basic science, clinical and experiential knowledge [10,11,12]. Basic science knowledge functions as a scaffold for clinical knowledge; when basic science concepts serve an explanatory, causal function in relation to clinical phenomena, learners understand, remember and apply new clinical knowledge optimally [13]. Therefore, if the power of testing can be harnessed to, first, activate prior basic science knowledge, and second, to promote the differentiation of that knowledge, learners may be better prepared to learn new, related clinical information.

Researchers often study the indirect benefits of testing by focusing on differing effects of item format (e.g. multiple-choice vs. open-ended). However, evidence suggests that item format matters less than what the item evokes or requires of the learner. In other words, researchers must clarify how the design of tests influence the learning processes of trainees [14]. Doing so can better align formative assessment practices with advances in curricula and likely deliver on the promise of test-enhanced learning in advancing assessment for learning and programmatic assessment [15, 16]. In addition to a focus on item format, researchers have, up to now, tended to focus on pre-testing novel, to-be-learned information [17] or on the effects of testing recently acquired information on new learning, for instance, in the word-list learning paradigm [18, 19]. In contrast, the proof-of-concept work we report on is a first step in a program of research that aims to investigate a novel aspect of test-potentiated new learning; that is, the use of testing to activate and elicit particular kinds of cognitive processing of prior, established knowledge in a way that may support interpreting, understanding, remembering and applying new knowledge. Such evidence could expand the validity argument for formative testing as a method of enhancing clinical reasoning [20].

In this paper, we report on the development of a multiple-choice question (MCQ) test of basic science knowledge designed to elicit a particular kind of cognitive processing—distinctive processing—that we argue may benefit future learning. We also report on the accrual of response process validity evidence to support the interpretation that the test does, in fact, elicit this type of cognitive processing. Response process validity of formative tests has generally been neglected in higher education [21], and the Health Professions Education (HPE) literatures [22,23,24].

Response processes are “the mechanisms that underlie what people do, think or feel when interacting with, and responding to, [an] item or task and are responsible for generating observed test score variation” [25]. We propose that distinctive processing is a response process that can promote development of clinical reasoning. Distinctive processing is the processing of differences in the context of similarity [26]. It consists of two different types of processing—relational and item-specific processing—carried on simultaneously. Relational processing may enhance retrieval by inducing a broad memory search of candidate items; that is, relational processing is memory retrieval at the category or superordinate level. Thus, relational processing promotes the noticing of similarities. Item-specific processing (ISP), on the other hand, entails focusing attention on individual items within a category and how their features differentiate them; item-specific processing, then, promotes noticing differences [27]. Distinctive processing may promote accurate memory, and likely characterizes expert memory [28]. Multiple-choice test items can be developed to elicit a lesser or greater degree of distinctiveness processing. Items with similar, plausible or competitive response options may elicit relational processing by virtue of their similarity. They may also elicit item-specific processing since, unlike less competitive response options, they compel the test-taker to recruit additional, relevant information from memory to differentiate among them. When response options are competitive, participants perceive some salient conceptual relation(s) that connect response options (relational processing) and then must retrieve specific information that differentiates response options (item-specific processing) in the attempt to select a correct response [29].

Reasoning through a differential diagnosis involves distinctive processing as it entails discriminating among manifestations that share some common feature. For example, the complaint of “chest pain” is common to a wide variety of conditions; clinicians must differentiate among them efficiently and accurately using their medical knowledge. As Monteiro et al. [10] state: “…the evidence demonstrates again and again that the essence of expertise is the possession of a large, organised and retrievable body of both formal and experiential knowledge.” (Italics added). We contend that testing can be designed to induce the kind of distinctive processing that organizes existing knowledge coherently in the learner’s mind, such that new learning is enhanced.

Testing prior knowledge has been found to enhance learning new, semantically or conceptually related information—a forward testing effect—through a variety of mechanisms [30], including priming. First, testing prior knowledge primes attention to important concepts presented in the new, to-be-learned content [31]. Second, testing prior knowledge enhances comprehension of new information and promotes the synthesis and integration of the old with the new [30]. Third, effortful retrieval of prior knowledge through testing with competitive response options causes learners to make more conservative judgments of what they know and thus induces them to optimize their strategies for learning new, related materials, through greater attentional focus and increased time-on-task [8].

We developed two versions of a 19-item basic science MCQ test and conducted a response process analysis using a concurrent verbal or “think-aloud” protocol. We tested the hypothesis that MCQ items designed with competitive or plausible incorrect answer options induce learners to engage in more distinctive processing than MCQ items with less competitive or less plausible incorrect response options. Our dual aim involves examining whether such tests can promote beneficial processes for learning, and demonstrating how to collect response process validity evidence for formative tests.