From Corkboard to Courtroom: Lessons in Flexible Research Design

Randomized controlled trials (RCT) are considered the gold standard of research design.  Thus, many researchers try to utilize them as often as possible.  It can be difficult to tease apart inference for a variety of reasons when conducting research, but using a randomized design can take away some of the confounding factors. 

For that reason, when Research Assistant Professor Kort Prince and Senior Research Analyst Derek Mueller, both with the Utah Criminal Justice Center at the College of Social Work, were designing a study to test the efficacy of a 24/7 supervision program in reducing DUI recidivism, they turned to an RCT. 

Before the study began, Dr. Prince and Mr. Mueller talked to all the partner agencies—courts, probation agencies, driver’s license division, etc.—and all opted in to an RCT design.  Courts and judges were going to have alternating weeks assigning defendants to the treatment—an intensive testing program meant to promote sobriety—or, the control group—standard DUI proceedings.

In practice, however, that’s not quite what happened; true randomization never occurred.  There are a wide variety of possible reasons the assignment didn’t go as planned, from judicial discretion to advocacy of a legal team to the cost of program participation.  Ultimately, the lack of random assignment led to large discrepancies in sample size, which were highly mathematically improbable if random assignment had occurred—given the discrepancies in sample size across the two groups, the chances that judges followed an RCT design were about one in 100,000.  So not impossible, but improbable enough that the researchers felt uncomfortable calling the process an RCT.

Rather than becoming discouraged by this, in some respects, it made the project more challenging … and more interesting. 

Dr. Prince explained, “As researchers, we enjoy the process and opportunity to use unique methods; in this case a unique design.  One of my research strengths is in quasi-experimental designs—finding other ways to analyze data that can’t be analyzed through a typical design.”  Dr. Prince and Mr. Mueller did run a standard RCT-based survival analysis, but they also ran a regression point displacement design, a design well-suited to community-wide interventions.

“This highlights an example of why it’s hard to ‘prove’ a program is effective,” said Dr. Prince.  “It can be really hard to implement an RCT in applied settings.  When you’re in a lab, and you want someone to see, say, images on a screen, that’s pretty simple.  You can make sure they get the condition they were assigned to because you’re in charge of it.  In this case, we’re dependent on the judges.  Judges are going to use their discretion and expertise for the case in front of them.”

Although information from the statistical analysis was somewhat weak in terms of both inference and statistical support, data gathered from interviews and surveys told a slightly different story. “In general, most stakeholders had pretty positive perceptions about the program and its ability to achieve desired outcomes,” said Mr. Mueller.  “Participants also generally had positive experiences in the program.  The ability to retain their license was helpful to them in their day to day lives and responsibilities.”

Dr. Prince and Mr. Mueller remain hopeful about the future of this program.  “While this is still not a definitive study in terms of evaluating the 24/7 program, I do think we improved upon previous studies,” said Mr. Mueller.  He continued, “Despite the methodological issues we were presented with, we contributed to the research base for this program.  And, as often happens with research, we identified unexpected outcomes.  For example, participants indicated that the program helped them maintain employment and improved their professional and personal relationships—that the program positively impacted participants in other life domains, beyond alcohol-related behavior.  Moving forward, I think future research should consider these additional outcomes.” 

Dr. Prince echoed these views.  “We weren’t able to draw some of the major conclusions we originally hoped for, but we moved the research forward.  And that’s what the process is really about.”