Though many medical residency programs have begun using holistic review to evaluate candidates — a process that looks at qualities such as leadership, research aptitude, and volunteerism to get a more well-rounded view of an applicant than grades and test scores can provide — some administrators are still wary that a holistic process may not result in a qualified candidate pool.
Samuel Lai, MD, a resident in the University of Colorado Department of Surgery, looked to address those concerns in a paper published in July in the Journal of Surgical Education. The study evaluates whether a holistic review process helps to eliminate bias in general surgery residency interview invitations.
Samuel Lai, MD
“Surgical residencies in general have been going toward more of a holistic review process to try to decrease or eliminate biases in recruiting to get more diversity and improve outcomes,” Lai says. “The driver for this project is to increase diversity in the CU program as well.”
Homegrown data
For his analysis, Lai drew on the holistic review process adopted by the CU surgery residency program in 2021. Previously, the department had invited candidates to interview for residency largely based on their licensing examination scores, clinical rotation grades, honor society memberships, and class rank. Nationwide, however, there is a growing recognition that such measures of academic success tend to exclude members of ethnic and socioeconomic groups that are underrepresented in medicine. A holistic review process looks at the whole candidate, including research, volunteerism, leadership, letters of recommendation, and any adversity the candidate may have experienced in their life.
“In my opinion, holistic review gives an applicant more of a fair chance,” Lai says. “Your medical school grades are not what defines you as a good physician. What we’re trying to capture with a holistic review is your aptitude for research, for leadership, how your attendings evaluate you, and how you work with your peers, which are things we can glean from letters of recommendation. The holistic review gives us a better understanding of the applicants and a more in-depth assessment of what kind of physician they might turn out to be.”
Diversity increased, academics unchanged
For the Journal of Surgical Education paper, Lai and his fellow researchers looked specifically at holistic review scores for the 586 applicants who passed preliminary screening for the school year beginning in fall 2022. The researchers found no significant differences in licensing exam scores between the 118 of those applicants who were invited to interview for residency at the CU Department of Surgery and those who were not.
“We compared the applicants who were invited to interview versus those who weren’t, looking to see if there were any differences in test scores. Were there any differences in gender or underrepresented status? It turned out there were not really any differences between the two groups,” Lai says.
“To us, this shows that, number one, we didn’t decrease the academic scores and achievements, which some program directors around the country think might happen with a holistic review that de-emphasizes test scores. And when we compared the actual applicants who were invited for an interview to the previous five years, when we did not implement this holistic review, we saw that there was a significant increase in the number of students who identify as underrepresented minorities in medicine.”
It's doable!
Lai’s paper also addressed issues around feasibility, showing that when faculty members and other residents join in the holistic review process — which they do enthusiastically at the CU Department of Surgery — it is easy to manage and complete.
“We showed that this is a feasible process, it’s doable, and it doesn’t degrade the quality of the applicants we’re getting,” Lai says. “It brings us one step closer to our overall goal of increasing diversity in the program. The applicants we have recruited this way have been phenomenal. Now more work needs to be done to spread awareness about the use of holistic review, and to show that it’s a feasible and not inferior way of assessing medical students.”
Nicole Christian, MD, an assistant professor of surgical oncology who served as Lai’s mentor on the project and is the corresponding author on the paper, says the study does a great job of showing how holistic review made for a more diverse residency pool in the CU surgery department.
“The implementation of a holistic review process allowed us, as a program, to not only better screen applicants for the personal characteristics and track record of success we are looking for in candidates to interview, but in a more rigorous and less biased manner than previous screening paradigms,” Christian says. “This is a critical strategy to help us meet our mission to train future leaders, approach DEI priorities, and recruit top candidates from all backgrounds and types of experiences. Sam’s paper elegantly demonstrates how we were able to achieve our goals of implementing a holistic approach to candidate screening that would decrease historical bias.”
