Abstract

The integration of generative AI (GenAI) in patient communication presents benefits and challenges. This retrospective observational study analyzed EHR audit logs to assess how 75 healthcare professionals (HCPs) utilized AI-generated drafts for patient messages from October 2023 to August 2024 at a large health system in New York City. Overall utilization was low (19.4%), though prompt refinements improved usage (from 12% to 20%), particularly among physicians. GenAI drafts were generated for all messages, including 80% that received no response, adding to the review burden and potentially undermining efficiency. Text analysis showed HCPs preferred concise, information-rich drafts, with role-based differences— physicians favored shorter drafts, while clinical support staff preferred more empathetic responses. AI-generated drafts reduced message turnaround time by 6.76% despite a marginal increase in required steps (InBasket actions). These findings highlight the need for targeted GenAI deployment strategies, better aligned with clinician workflows and optimized draft generation for improved efficiency.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This work was supported by the National Science Foundation (NSF) grants 1928614 and 2129076 (PI: O.N., Co-PIs: D.M., B.M.W.).

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

IRB of New York University Langone Health gave ethical approval for this work

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Data Availability

The primary data on patient messages, HCP actions, and responses used for analysis were sourced from the electronic health record (EHR) data of the New York University (NYU) Langone Health (NYULH) system, containing protected health information (PHI); as such, they cannot be shared publicly. Access to the underlying data requires a Data Use Agreement and IRB approval from NYU Langone Health and its EHR vendor (Epic). While the full content of patient messages and HCP responses cannot be disclosed, de-identified, aggregated metadata on their characteristics can be made available from the authors upon reasonable request.

AbbreviationsEHRElectronic health recordIBInBasketHCPHealthcare providerGenAIGenerative artificial intelligenceLLMLarge language model