Study Design

This cross-sectional study assessed the efficacy of three LLM prompting techniques incorporated into five separate prompting strategies to determine the accuracy of GPT-4o in characterizing pancreatic cystic lesions, identified by four categories: (1) no reported cystic lesion, (2) ambiguously reported lesion, (3) reported cystic lesion—previously known, and (4) reported cystic lesion—not previously known.

Of these categories, ambiguously reported lesions were expected to be the most difficult to identify and least likely to be explicitly called out in the reports, as the definition of “ambiguous” can have multiple reasonable interpretations. For example: (1) it is ambiguous whether the report indicates the presence of a lesion; (2) it is ambiguous whether the patient has a lesion; (3) the patient has a lesion, but the lesion’s characteristics are ambiguous. In our study, we explicitly mean the first interpretation, a fact which would need to be communicated to both human and AI responders. This motivates the creation of an external knowledge source, which in our case would be an instructive rubric detailing the exact definition of “ambiguous” and other categories of interest. For these reasons, ambiguously reported lesions were the main category of interest for this study.

Each strategy was evaluated by comparison to a ground truth defined by a human expert reviewer provided an instructive rubric identical to the one used by the LLM as an external knowledge source.

This study is HIPAA-compliant and approved by the University of [California, San Francisco] Institutional Review Board with a waiver of individual informed consent.

Data

Free-text radiology reports were extracted from 676 abdominal CT and MRI exams performed in adults (> = 18 years) between Jan 1, 2010, and Dec 31, 2022, sampled from all abdominal CT and MRI exams performed at the University of [California, San Francisco]. Sampling was done using developmental versions of the prompting strategies assessed in this paper, pursuant to providing adequate representation of reports in the four previously defined categories.

In total, 96, 361, 87, and 132 reports were sampled respectively in each of these four categories. Of the exams sampled, 366 were used to guide prompt engineering and refinement, while the remaining 310 were used to evaluate the final prompting strategies. A full breakdown of sample sizes can be found in Table 1.

Table 1 Sample size of the training and validation sets for each answer category of each categorical question, according to the human ground truth and each assessed prompting strategy. RCI—recursive criticism and improvement. CoT—chain-of-thoughtPrompting Strategies

This study evaluated a total of five prompting strategies, each including some combination of the following three prompting techniques:

1.

External knowledge source—the instructive rubric provided to the human reviewer detailing the definitions of each category was copied into the prompt to the LLM.

2.

Chain-of-thought prompting (CoT)—the LLM was asked to provide the reasoning it took to arrive at its answer.

3.

Recursive criticism and improvement (RCI)—the LLM was re-queried after providing the initial response, asking to compare the initial response to the external knowledge source and check for discrepancies.

The five prompting strategies were defined as follows:

1.

Bare strategy—GPT-4o with no additional techniques

2.

External knowledge strategy—GPT-4o plus external knowledge source

3.

CoT strategy—GPT-4o plus external knowledge source and CoT

4.

RCI strategy—GPT-4o plus external knowledge source and RCI

5.

Full strategy—GPT-4o plus external knowledge source, RCI, and CoT

Each prompting strategy delivered either one or two prompts to the LLM, with optional components depending on whether external knowledge source, RCI, or CoT was included. The first prompt is detailed below:

1.

Start with a mission statement—“You are an expert AI physician specializing in diseases of the pancreas. The user will provide you with information about a radiological exam performed on a specific patient. Your job is to determine whether a pancreatic cystic lesion is mentioned in the report. You will also answer a collection of ancillary questions about the pancreatic cystic lesion.”

2.

For strategies that included “external knowledge source,” add the instructive rubric defining the categories.

3.

Add the full radiology report.

4.

Add the questions of interest to the study along with the acceptable values for answers.

a.

A total of five medical questions were asked, though only one is the focus of this paper—“If a pancreatic cystic lesion has been reported, was the pancreatic cystic lesion already known from previous examinations?” [Allowed values: no reported cystic lesion, ambiguously reported lesion, reported cystic lesion—previously known, reported cystic lesion—not previously known]

b.

For strategies that included CoT, add one additional question asking to provide the chain of reasoning undertaken to arrive at the answer. [Free text]

The preceding prompt was submitted to the LLM. For strategies that included RCI, a second prompt was submitted, the specific anatomy of which is as follows:

1.

Start with the full initial prompt.

2.

Add the LLM’s initial response.

3.

Add one additional prompt that includes the instruction “Compare the answers you just gave against the previously-provided rubrics and identify any discrepancies.” and thereafter provide a second response, which is treated as the final response.

The exact wording of all prompts, along with rules on acceptable answers and how unacceptable answers were handled, can be found in Supplementary File 1. A visual presentation of the structure of prompts and supplementary rules can be found in Fig. 1.

Fig. 1

Order of operations for each prompting strategy, given presence or absence of each tested technique: external knowledge source, chain-of-thought prompting (CoT), and recursive criticism and improvement (RCI)

This analysis used GPT-4o (deployment name gpt-4o-2024–08-06, API version 2024–07-01-preview), hosted in Microsoft Azure and executed via the AzureChatOpenAI function of langchain (langchain version 0.1.16, langchain-openai version 0.1.3) running under Python version 3.12.3 (packaged by conda-forge [MSC v.1938 64 bit (AMD64)]). Temperature was set to zero, while top_p and max_completion_tokens are set at 1.0 and 4,000 (effectively uncapped), respectively.

Prompt Engineering

Several rounds of prompt engineering were conducted to adjust the wording of the external knowledge source, RCI, and CoT elements of the prompt, along with the wording of the questions themselves. Special focus was given to enhancing the ability of the external knowledge source to identify exams with an ambiguously reported lesion. The process was as follows:

(1). The reviewer assessed a collection of exams with an instructive rubric.

(2). LLM assessed the same exams.

(3). LLM prompts were updated, including the core instructions, the verbiage of the questions, and the content of the external knowledge source.

(4). The instructive rubric for the reviewer is updated so that it is identical to the external knowledge source of the LLM.

(5). The reviewer assessed a new collection of exams with the updated rubric.

Ground Truth

The sampled exams (N = 676) were assessed by a radiologist human reviewer, who answered all questions for each radiology report. He was aided by the same instructions and external knowledge source as the LLM and was asked to provide answers in the same format as the LLM.

The radiologist was a resident who previously worked as a radiology technologist prior to medical school and thus had extensive experience understanding radiology reports. Further, he worked closely with a board-certified radiologist with expertise in abdominal imaging and over 10 years of clinical practice post-training. This board-certified trained the resident on how to review reports for this study and helped with interpretations of difficult cases. This training took place prior to the study and was separated into four rounds. At the end of each round of training, both individuals independently reviewed 20 randomly sampled reports and checked for agreement. By the end of the fourth round, the two agreed on the classification of 19 out of 20 reports. The 80 reports reviewed for training are not part of this study.

The radiologist reviewer was not involved with the prompt engineering or statistical analysis of this study.

Accounting for Non-Determinism

Generative AI is known to generate non-deterministic outputs even when the temperature is set to zero [11,12,13]. We accounted for this non-determinism by defining a prompting strategy’s answer to a categorical question not as the LLM output in a single run but rather as the LLM’s most common output across multiple runs.

To determine how many runs to use, we applied a simple Bayesian heuristic. For each examination and each prompting strategy, we started by running the prompt three times and measured the difference in prevalence between the most common output and the second most common output. We then continued running the prompt and recalculating this difference after each run. If this difference increased or remained the same for three consecutive runs, we stopped. The most common output across all performed runs was then taken as the final answer.

For example, if a prompt’s answer to this study’s question of interest across seven runs were ambiguous, no lesion, ambiguous, no lesion, ambiguous, ambiguous, and ambiguous, then the difference between the most prevalent and least prevalent answer would be (starting from the third run) 0.33, 0, 0.2, 0.33, and 0.5. The runs would then stop at the seventh run, since the last three runs all saw this difference increase. This example would be classified as an ambiguously reported lesion.

Statistical Analysis

To assess accuracy, the answers provided by each prompting strategy were compared to the human-provided ground truth. For each prompting strategy and each question, we computed the overall weighted accuracy and, for identification of ambiguously reported lesions, the sensitivity. The weighted accuracy was defined as the proportion of exams where AI-produced output agreed with the human-provided ground truth, with each exam weighted according to the prevalence of the corresponding answer in a broader data set of 269,907 abdomen and pelvis CT and MRI exams collected between Jan 1 st, 2010, to Dec 31 st, 2022; these reference categorizations were generated using the full prompting strategy.

To assess non-determinism, we calculated, for each prompting strategy, the mean number of runs iterated before the stopping criterion was reached (“mean iterations”). We also calculated the proportion of exams where the AI output is deterministic (“proportion unvaried”). These metrics of non-determinism were computed once for each strategy, without separate values for each question. Both measures of non-determinism were weighted in a manner identical to the weighted accuracy.

We lastly computed the mean token costs, time costs, and monetary costs for each prompting strategy, per run of the LLM. Token and time costs were directly extracted from the metadata of the LLM output. Token costs were separated between prompt tokens and output tokens. Monetary costs were calculated based on OpenAI’s pricing as of May 20, 2025 [14]—$2.50 USD per million prompt tokens and $10 USD per million output tokens.

For each metric of accuracy, non-determinism, and cost, 95% confidence intervals were computed via bootstrapping. Statistical significance in performance between the CoT, RCI, and full strategies, compared to the external knowledge strategy, was tested by bootstrapping the corresponding differences. All statistical analyses were performed by a statistician not involved in the creation of the ground truth answers. Analysis was performed in R version 4.3.2 [15].