top of page

Closing the Loop by Operationalizing Systems Engineering and Design


Diagnostic errors in primary care are costly and can be attributable to failures or delays in follow up on diagnostic testing, referrals, and patient symptoms. The proposed research seeks to “close the loop” on this public health problem by employing system engineering methods in three diverse healthcare practices to create a system that provides highly reliable follow-up of diagnostic tests, referrals, and symptom evolution.
Results will be disseminated broadly so as to improve diagnostic safety nationally.

image 1.JPG
Specific Aims :

Aim 1:​Use systems engineering and patient engagement to design, develop, and refine a highly reliable “closed loop” system for diagnostic tests and referrals that ensures diagnostic orders and follow-up occur reliably within clinically- and patient-important time-frames.

Aim 2: Use systems engineering and patient engagement to design, develop, and refine a highly reliable “closed loop” system for symptoms that ensures clinicians receive and act on feedback about evolving symptoms and physical findings of concern to patients or clinicians. 

Aim 3: Design for generalizability across health systems more broadly so that the processes created in Aims 1 and 2 are effective in (1) a practice in an underserved community, (2) a large tele-medicine system, and (3) a representative range of simulated other health system settings and populations.

referral process.jpg


beth israel logo.png

Beth Israel Deaconess Medical Center (BIDMC) is part of Beth Israel Lahey Health, a new health care system that brings together academic medical centers and teaching hospitals, community and specialty hospitals, more than 4,000 physicians and 35,000 employees in a shared mission to expand access to great care and advance the science and practice of medicine through groundbreaking research and education.


Founded in Boston's Dorchester neighborhood in 1972 by community residents, Bowdoin Street Health Center remains focused on providing outstanding medical care while maintaining a tradition of working with residents to identify and meet their unique health needs. The mission of the health center is to provide excellent, compassionate care to our patients and support the health of the entire community.


Brigham and Women’s Hospital, part of  Mass General Brigham Health, is a world-class academic medical center based in Boston, Massachusetts that serves patients from New England, across the United States and from 120 countries around the world. BWH is a major teaching hospital of Harvard Medical School.


Center for Primary Care is part of the Harvard Medical School and is committed to uncovering, promoting, and fighting for the best and most efficient ways to deliver high-quality primary care around the world.


This research aims to close the loop on diagnostic tests, specialty referrals, and worrisome symptoms utilizing systems engineering methods. The timeline of this grant follows the AHRQ life cycle with a focus on: problem understanding in year 1, designing solutions in year 2, defining and refining solutions in year 3, and implementing and evaluating solutions in year 4. Table 1 outlines which systems engineering methods will be used within each phase of the project life cycle.


Results to Date:

For current state problem understanding, cross functional process maps were developed of loop closing processes through a series of interdisciplinary interviews and meetings (including clinicians, scheduling and administrative staff, patients), both in general and focused specifically on dermatology and gastroenterology referrals. Results then were used as input to process design characterization analyses, failure analyses, and prototype design exercises. These approaches helped to highlight the low overall system reliability of these processes, identify common and critical failures, and brainstorm potential solutions to address these failures.

Informed by the above, recent work has focused on prototyping and piloting several solution ideas to either improve existing loop closure processes or design new processes. The FMEA and FTA failure analysis results are being used to focus on the most critical and most promising areas to improve or redesign within current processes. To consider new processes altogether and avoid focusing only on improving the existing mental model of referral processes, an engineering process design exercise (structural analysis design technique, SADT) was used in this project to instead identify necessary process activities and information flows (i.e. “what” needs to occur rather than “how” it occurs). Reliability engineering design methods also are being used to stimulate new ways of thinking about closing loops more reliably and sooner, including failover designs, forcing functions, and process simplification. A broad range of resulting pilots now are being developed and tested across process flow stages (e.g. ordering, scheduling, preparation, completion, communication of findings, rescheduling, and follow-up).

Systems engineering computer or mathematical models have been used in several manners throughout the above activities to help understand process performance, inform research team discussions, evaluate potential interventions, and examine disparities. Examples include predictive, discrete event computer simulation, and cost/benefit models of engaging community health workers in closing loops (workflows, resource needs, performance); reliability models of alternate approaches for incorporating home-based and hybrid home-colonoscopy testing for GI patients; and fault tree and probability block diagram models of referral processes and symptom monitoring processes.



Benneyan J, White T, Nehls N, Yap T, Aronson M, Sternberg S, Anderson T, Goyal K, Lindenberg K, Kim H, Cohen M, Phillips R, Schiff G (2020). Systems analysis of a dedicated ambulatory respiratory unit for seeing and ensuring follow-up of patients with COVID-19 symptoms, Journal of Ambulatory Care Management, 44(4): 293-303, 2021. doi: 10.1097/JAC.0000000000000390. PMID: 34319924; PMCID: PMC8386384.

Nehls N, Yap T, Salant T, Aronson M, Schiff G, Olbricht S, Reddy S, Sternberg S, Anderson T, Phillips R, Benneyan J (2021), Systems engineering analysis of diagnostic referral closed loop processes, BMJ Open Quality 2021;10:e001603. doi: 10.1136/bmjoq-2021-001603.

Nehls N, Benneyan J, Poulsen K, Schiff G, Salant T, Aronson M, Sternberg S, Fernandez L, Phillips R5 (2022), A systems reliability view of studies to improve closing diagnostic loops, working paper.

Phillips R, Benneyan J, Bargal B, Schiff G (2022), Closing the loop: Re-engineering the assessment and tracking of symptoms in primary care, DOI: 10.1007/s11606-022-07886-7.

White T, Aronson M, Sternberg S, Shafiq U, Berkowitz S, Benneyan J, Phillips R, Schiff G (2022). Closing the loop on radiology recommendations: Developing and applying a taxonomy to track performance of recommended actions, in review.

Nehls N, Salant T, Benneyan J, Aronson M, Sternberg S, Schiff G, Phillips R, Rivera M, O’Brian D, Dreilak M (2021). CLOSED: Closing loops by operationalizing systems engineering and design, Institute for Healthcare Improvement 2021 Annual Forum.


Salant T, Benneyan J, Nehls N, Aronson M, Sternberg S, Schiff G, Phillips R, Rivera M, O’Brian D, Dreilak M (2021). CLOSED: Closing loops by operationalizing systems engineering and design, Silverman Symposium on Quality and Patient Safety.

Conference Presentations:

bottom of page