Portrait of Orna Rabinovich-Einy

Research Webinar Materials: Outcome disparities in the age of online courts

On Tuesday March 24, 2020, researchers Orna Rabinovich-Einy and Avital Mentovich of the University of Haifa presented their research on online courts in the US. View the video and read the study.

For all who attended the presentation live, thank you for your interest. Here are the materials for those who could not attend and for those who attended and wish to spend more time with or share the materials.

Outcome Disparities Research Video

Full Research Paper

Here is a pre-print of the research paper online: Are Litigation Outcome Disparities Inevitable? Courts, Technology, and the Future of Impartiality

Outcome Disparities Presentation Transcript

Dunrie Greiling, host, Matterhorn

Hello everyone, we’re here today to talk about outcome disparities in the age of online courts.

My name is Dunrie Greiling. I’m a part of the Matterhorn team. Matterhorn is software that powers court and mediation center online platforms. Our customers offer online dispute resolution for civil cases such as small claims, family and domestic cases such as separation, divorce, parenting time and child support disputes, and online adjudication for traffic citations, failure to comply warrants and online assessment of ability to pay.

First, thank you to all who have made time in an undoubtedly busy schedule to attend this event.

Thank you to our presenters Avital Mentovich and Orna Rabinovich who are taking time from their own busy schedules, which now include transitioning their teaching to this term to online.

I’m going to move to slide one…if when it behaves, come on. What if I, okay.

I’ll provide a full introduction to our topic and our presenters in a moment.

First, I want to dispense with some of the questions you might have in your head. Are we recording this session? And yes we are recording the session.

You’ll receive a link to the recording and the slides in two days, once we’ve had a moment to post it online.

Now, I want to give you just a brief introduction to how we will use Zoom. You should be able to see and hear from our panelists. All attendees are muted and your video is not shared.

There are two really useful tools on that Zoom bar which is a black bar probably at the bottom of your screen, the chat and the Q&A function.

Please use the chat to let us know about any technical glitches you’re having or when we ask to answer to quick question. And I’d like to start with that.

Would you mind testing the chat with us by dropping into the chat what you’re looking to get out of today’s session?

I see,

  • perspectives,
  • to learn about how ODR works,
  • new information…

I’m going to give you a moment to populate that. And while you’re doing that I also want to say that given our current situation, we understand that many people are seeking more logistical information about how to implement court processes online.

At the end of today, we’ll share an opportunity to engage with court leaders who have done this to get your questions answered so that’ll be a different event and today we’re going to be talking about this amazing research.

So, let me see what I’ve got here.

  • insight, perspectives,
  • a better understanding of ODR
  • Hello, from different people, people that are wonderful out there. Thank you, it is great to see you.
  • Research insights, exactly. That’s exactly what we’re going to be prepared to share with you today
  • how Matterhorn may be used. I think we’re going to focus more on the research insights
  • we’re happy to see in statistical data, oh good, I think.

Avital Mentovich, University of Haifa


Dunrie Greiling

That’s fabulous. We, I think we can help you with that, all right.

So, we’ll also have a couple more times just during our researchers presentation to stop and get your questions answered. So, please queue up questions in the Q&A widget at the bottom of the screen. It’s the, I don’t know, looks like a file folder I think with a Q on it. We’ll this the Q&A widget is great. It actually helps us keep track of what has and hasn’t been answered.

So, the chat is nice for sort of informal stuff but sometimes things can get buried in the chat and we don’t want to lose your questions and so please use the Q&A widget.

We will queue them up for the presenters and then we can mark whether they’ve been answered and we commit to getting any questions that we don’t get to online today answered via email afterwards. So, that is my Zoom housekeeping.

Now I get to introduce our panel today. Very excited to have our speakers Orna Rabinovich and Avital Mentovich.

Orna is an associate professor at the Faculty of Law at the University of Haifa. Her areas of expertise are online dispute resolution, alternative dispute resolution and civil procedure with research focusing on the impact of technology on dispute resolution, the relationship between formal and informal justice systems and dispute resolution system design. She’s published widely in these areas and recently has co-authored a book with Professor Ethan Katsch entitled Digital Justice, Technology and the Internet of Disputes, published by the very esteemed Oxford University Press.

Rabinovich is a professor, a fellow of the Haifa Forum of Law and Society, the Haifa Center for Law and Technology and the Center for Information Technology and Dispute Resolution at the University of Massachusetts in Amherst. She holds a doctorate in law from Columbia University.

I’m also so pleased to present Avital Mentovich. Tali is an assistant professor at the University of Haifa. Her areas of expertise are empirical Legal Studies, psychology of law enforcement and social justice. More specifically her research focuses on lay people’s perceptions of the legal system and the implication of such perceptions on the legitimacy of legal institutions. She’s published widely in these areas in leading law and psychology journals. Mentovich won an award of an Israel Science Foundation grant to study online courts in the US, together with Orna, and was awarded a Russell Sage Foundation Award to study perceptions of law enforcement among racial and ethnic minorities.

Mentovich holds a PhD from the New York University and she’s been a postdoctoral research fellow at the University of California, Los Angeles and Harvard University.

They are going to compare outcomes in online and in-person court cases and I’m so proud to say that their study includes cases handled on Matterhorn.

This team presented their research at the International ODR forum in October of 2019. We wanted to provide the opportunity for those who did not attend the forum to hear and participate in a discussion about their findings.

Orna and Tali are both speaking to us from Haifa, Israel where it is 8 p.m. Orna, Tali take it away. So, great to hear you, thank you.

Orna Rabinovich, University of Haifa

Wow, thank you so much. Strange times but probably one of the best times to talk about online courts if we want to find something good about these difficult times we are facing.

When this was set up I think it was before the corona crisis/epidemic and it’s probably the only engagement we had that wasn’t canceled because it was online to begin with.

Tali and I were supposed to be sitting together but because we are in semi quarantine as perhaps some of you are as well, we had to do this separately.

I have to say I feel very much at home not only because I am at home but because it’s so nice to be hosted by the Matterhorn folks who have become a real home-like environment for us. We’ve been working on this project for a while and this has really been a generous setting for us to pursue our interests.

And I should say this research is in collaboration with Professor J.J. Prescott of University of Michigan who is a full partner to this entire project. So, we can probably move a slide.

We thought we would tell you about the research in three stages. I’ll say something more general about ODR then we’ll say something of this sort of theory or framework that guides our research or can explain some of the outcomes and findings in our research and then we may take a few questions about this sort of more general part and then delve into the case study of Matterhorn where we’ll have all the data and statistics and the fun part and then we can end with questions again.

So, we thought it would be helpful to say a word, more generally about the general, about the broader context in which court ODR is located.

So, Dunrie very kindly mentioned, we can still go a slide back, Dunrie kindly mentioned that I pursued my doctorate at Columbia University of New York and this was a little embarrassing to say but some twenty years ago. And when I arrived to New York it was 1999 and it was some of you may recall it was the boom and there were two parallel developments that we’re going on there. One of them was the intranet boom, end of the 90s you know all the, all these new high tech ventures relating to the Internet. And the other parallel boom had to do with ADR. As many of you may recall, if you’ve worked with courts at the time as well, mediation was being introduced into the courts.

And as I was arriving to know and I knew I wanted to study dispute resolution, it seemed to me that these two routes were bound to intersect and at the time they already had that in a very small niche area, that area was e-commerce. It turned out that as distanced strangers were communicating over small-scale transactions from afar, they had disputes and for those disputes, it really didn’t make any sense to meet in a courtroom or in a board meeting or anywhere else. We didn’t have a Zoom at the time and they really needed some kind of avenue to find redress for their problems.

So, online dispute resolution started emerging and at the time what that meant was creating equivalence to familiar alternative dispute resolution processes but online.

Later the understanding of what constituted ODR expanding, it expanded because of several different developments. Probably one of the biggest development, the most major developments was the shift from the internet being a place we connected to through the personal computer to an arena in which we could always go in into and out of through our personal phones, right. So basically, what happened somewhere in the 2000s as the smartphones came out was that we were constantly on and offline and this distinction kind of lost its meaning.

Another thing that happened was that social media emerged and the internet became an arena in which we connected with people who were close to us, that not only distant strangers to by some kind of collectible from right. So, all our habits changed, our expectations changed and it started making sense to communicate about a lot of new stuff online and these interactions generated disputes.

And if it made sense to communicate about the interactions online, it also started making sense to communicate about those disputes online. And we saw the area of online dispute resolution expand to what we would term beforehand offline disputes. Right, these could be consumer disputes that emerged in the face-to-face settings or neighbor disputes or other types of disputes.

And finally, in the second decade of the 21st century, they enter the courts. We can move to the next slide please. And in the courts, we see now an emerging phenomenon in the U.S. Matterhorn has certainly been a leader, there are also additional platforms. We’ve seen ODR in Canada, in England, China, Singapore, more. We see them covering different types of disputes, not only those traditional disputes we would see being handled through ODR and not only small claims, also neighbor, traffic, taxes, bodily injuries and even the growth area family which might surprise many people.

We think family is so complex and requires personal interaction. It turns out a lot of family members might actually prefer to talk about their problems from afar and might actually value some of the advantages online communication gives them which we will explore further.

And we also see a lot of creativity in the types of processes that are being offered through ODR. Someone said they wanted to learn more about the types of processes. I’ll just mention here that at this point in time, ODR not only mimics what we see in the face-to-face world but people who design ODR systems like Matterhorn see this as an opportunity to reimagine what these processes can look like, taking advantages of the new technology and not seeing technology as a limitation that requires us to do things differently but actually celebrate the new opportunities that open up with this shift. We can go to the next slide.

So, as these developments are going on, it seems like a really important idea to understand what the implication of the move online is for parties and for courts and for all the different actors using or resorting to the justice system.

Well, the obvious advantage in shifting online is often measured in terms of efficiency, accessibility, perhaps even satisfaction. Indeed in many areas where you see new technologies entering, you see that there is enhanced efficiency and in courts as well. So, the processes take less time, they reduce some of the caseload of the courts, they can be accessed 24/7, right. There are many advantages in terms of efficiency.

What we wanted to do in this project and we hope we can convince you of our findings is to move beyond efficiency and to look at different parameters for evaluating the justice system.

And for us, one of the most interesting parameter, I think it’s the next slide, is impartiality or the equality of the justice system. So, this is probably one of the most basic tenets of the justice system, right. It should be impartial, similar cases should lead to similar outcomes. The system we always think, right, of justice is being blind to the particulars of the case, those that are irrelevant. We should reach the same result whether we are a woman or a man, right, whatever our religion is. So, we’re hoping that the system is blind to these results and we see we see this as an indication of the fairness and impartiality of the system.

In fact though, over the years as Tali will tell us, reality has often shown us that outcomes may actually reflect disparities. And I’ll shift over to Tali who will tell you about what outcome disparities are and why they cast a shadow on impartiality.

Avital Mentovich

So hi, more precisely to make it to continue, to build on what Orna said, I’m going to show you which type of outcome disparities we think are particularly threatening to this idea of impartiality and why.

So, it’s okay for legal outcomes even in similar cases to vary, it’s part of what we call in statistics or in human science, the human noise. It’s in legal system, it’s what we call part of judicial discretion or the discretion of any decision-making along the process. So, it’s okay for people for legal outcomes not to generate in a completely consistent manner.

What we think is problematic or the aspect of outcome disparity that is problematic from the impartiality point of view is that when these differences in outcomes, it does fall along the lines of social identities.

So, for example, when we realize that black people receive different punishment, white people will receive, that women receive different punishment or different legal outcomes than men. And this is what, we think that this threatens this concept of impartiality because more than anything it suggested that extra-legal considerations, consideration that at least per se are not relevant to the legal proceedings, penetrated the decision-making and impact legal outcomes in a way that they do. As you can imagine usually disfavoring the most marginalized groups or individuals in society.

So, these are the type of outcomes disparities that we examine in our project and not just the usual kind of like a natural variance that legal outcomes can have that are not related to groups. Now Dunrie, we can switch to the next slide.

So first, we’re going to show you again to take, we’re going to take you to a short journey about these manifestations of a outcome disparity. How much do they exist in the justice system and how much they’re pervasive. After that, after I’ll hope we will convince you a little bit that they’re pervasive enough to deserve our attention. We’ll discuss a little bit about the potential sources of such outcome disparities. So, we identify where they’re coming from and why they may be different when we shift online. And then we’re going to discuss what we think is the, what we believe would be the impact of the shift online and show you some data about whether expectation indeed met with reality. So, I’m going to start and then we can move to the next slide.

I’m going to try it will give you basically in one slide, ten years of research, maybe more of the research that was done in a social and legal science about the pervasiveness of outcome disparities. So, this research started with they’re not looking at, not starting by looking at the identity of the parties and its effect on outcomes but rather with the effect of judges identity and the effect of judges identity on outcomes. It started in the late 70s.

We did the diversification of the bench, started in the late Carter Administration and there was these expectations, I have to tell you that there were usually grounded in sort of stereotypical thinking that maybe judges from different gender or from different races would get to different legal outcomes. There was this expectation, for example, that maybe women judges will be slightly more lenient compared to male judges because you can imagine what are the stereotypes about women.

What I can tell you about this type of research is that in general, it largely proven not fulfilling in the sense that we couldn’t find any impact on judges identity, not their gender nor the race, not even a political ideology, something that you might expect to have a greater influence on their legal outcomes which was at the time great news to the, to impartiality to this principle of impartiality.

Later research started to look at, to examine the effect of parties’ identity on legal outcomes and I have to tell you that most of this research has been done on the criminal arena and the population or the group that they search for disparities in its context were usually black defendants or black suspects with comparison to non-black or white counterparts. Now, this research and again and I’m making it really short but the conclusions of the research are pretty systematic so it allows me to summarize them pretty briefly is that throughout almost every important junction in the criminal justice system, from reporting a crime, the prosecutorial decision and if we look at the end of the process, the legal outcomes, the sentencing, bailing decisions, we could see that black parties, black defendants, black suspects, they’re treated worse than similarly situated white defendants, meaning from white to the exact same situation with the same legal circumstances. Next slide.

Now, of course, the existence of these disparities were pretty, I don’t want to say shocking but at the very least surprising, to a system that is very much obligated, both the principle of it but also if I feel at the personal motivation of judges, to principle of egalitarianism to this principle of equality before the law.

So, an important question was, how come that we continue to see these disparities, illegal outcomes, despite the explicit commitment of both the legal system and specific decision makers within the legal system to this idea of impartiality and equality before the law?

Now, one explanation which I think was a convincing explanation and it was backed up by a lot of research that was done in outside of the legal arena was research in mainly in social psychology, that showed basically that even outside of the legal system, the explicit commitment of individuals or institutions to equality does not mean that they still harbor unknowingly and in the case that I described, unwillingly some implicit biases that impact their judgment even without their knowledge.

The measures of this implicit bias was basic, don’t want to get into too much detail, but they will tell you that it basically bypassed what people self report or say about what they believe in because that corresponds with their explicit attitudes and their explicit motivation but rather use very sophisticated measure of reaction time, not what people say but how they associate certain concept to uncover that implicit bias or implicit bias or biases are very pervasive.

They’re pervasive throughout almost any group or subgroup in the population and there are even two studies that were done specifically on judges and they showed them at this point, it shouldn’t be surprising what I’m going to say now. The judges, like any human beings, display the same type of biases as other people meaning they show the despite commitment to egalitarian values. They still have all kind of like unconscious schemas that suggest that they don’t see, at least on the unconscious level, they don’t judge all identities equally.

Next slide

Orna Rabinovich

We should maybe say Tali, that we’re not talking only about judges. Obviously, there are many decisions makers.

Avital Mentovich

Of course.

Orna Rabinovich

On the line.

Avital Mentovich

Of course.

Orna Rabinovich

Much of the research has been on prosecutors, right and police so they were very…

Avital Mentovich

Yes, completely but it was also found true for judges as well.

Orna Rabinovich

Of course. So, the other source that could explain why we find outcome disparities that are aligned with the group social identity is, what we termed here, structural bias. Now, I wouldn’t say we devised this but this has not received much attention in the literature, unlike implicit bias.

And what do we mean by structural bias? Here we shift the focus from decision-makers be they judges or any other decision-maker along the line to the interaction between the parties and the characteristics of the process they are undergoing.

And what we say here that some types of processes may benefit certain types of disputes but on a group level, not just as Tali said before, not just you know, I may prefer one process, and Dunrie another, and Tali feels sort of comfortable with another but this should trouble us when we see that all women feel very uncomfortable and more than feel very uncomfortable are less successful systematically in a certain type of process, okay.

So here, we say, you know up until now, we’ve assumed that the way courts have structured their proceedings is really the only way possible. We have to meet in person, we have to have real time interaction and exchanges between the parties and the courts and this has to happen orally with everyone seeing one another. This is sort of a given that we haven’t questioned up until new technology has entered the arena, right. So, what could be problematic along, why would this be problematic for members of certain groups?

Well, we know first of all that certain groups suffer from gaps in knowledge and resources and skills, many of the problems we put as barriers under access to justice, right. And clearly, this setting and the procedural mazes parties have to go through in courtrooms present more barriers than others for certain groups that require certain skills and resources they may not have.

When we meet in court in real time and there’s an interaction, there may be differing cultural expectations between the judge and the party. We as judges, if I put myself as a judge, I might expect someone who’s innocent to emphatically claim their innocence, right.

Someone else coming from a different culture, maybe a certain group would actually feel that’s not differential enough to an authority figure. A judge may think that someone lowering their eyes, maybe stuttering, sweating, is someone who’s lying, maybe that’s actually someone who is an immigrant, a minority.

So, we can think of all of these, of the setting as creating these challenges on a group wide level. Psychological impact, we know that for minority there’s actually minorities there’s a stereotype threat. So, it’s actually this vicious cycle where there were self-fulfilling prophecy where they come in front of the judge and they, actually they, they’re afraid of the stereotype that’s going to be imposed against them and they actually perform worse than they could have.

So, something about this meeting in real time, in the courtroom with the two-way exchange may pose extra barriers for members of certain groups and may advantage, may provide an advantage to others who are well versed, who are able to work quick on their feet, who are confident, who don’t feel at a disadvantage in front of the judge. Next slide please.

So, the legal system or the judicial system has not been blind to the problem of implicit bias and to a certain extent, of structural bias. And we should say these are not, they obviously can co-exist and different structural designs can actually increase the potential for implicit bias or decrease the potential so that there’s an interaction there.

And the judicial system over the years, and we’ve read several books dealing precisely with that, has taken very seriously the challenge of redressing and addressing implicit bias and there have been various tools recommended and employed by various courts and policy makers from raising awareness and self-reflection of judges and trying to have a more sort of moderate and structured and lengthier process of decision making, having self auditing, auditing by others, you know a range of possibilities for inhibiting bias.

The problem is that this again very true commitment is being employed by courts who are also facing case overload and are also required to become more and more efficient. And the need to deal with cases quickly and streamline the cases going through the courtroom, actually runs counter to the types of steps a court needs to take if they want to address implicit bias seriously. We should say also that even when these steps are taken seriously, they have had some success but not you know, this is not overwhelming success.

So, implicit bias is very difficult to deal with and the conditions under which courts operate make it almost impossible to deal with it because they have to run through cases so quickly. These are conditions that actually enhance the tendency to resort to stereotype-based decision making. So, we can go to the next slide, please.

Avital Mentovich

So, if you consider what we just told you about the sources of implicit bias, I feel it’s very easy to get a feeling why the sources are not as relevant in work or not as salient when we when we do this shift to the online setting and let me explain. So, we first talked about the implicit bias. Now, implicit bias is basically something that is triggered upon the exposure to one’s identity.

What online courts offer, they’re almost, they’re are offering the closest as we can get to this idea and Greek idea, the Greek myth of blind justice.

Now, it’s true that not all the features are blind and maybe we talk about it later as features of the design. Judges for example can maybe infer gender from names, they do get their date of birth. By the way, interestingly enough, in the Matterhorn systems, they don’t see race at all. Maybe they can infer it, probably less reliably than gender, by names but even if they do get this information, we know that the identity, when it appears on a written format, it’s much less salient than when you see somebody in person.

So, our belief was that the remoteness of the online system which makes the identity less salient will make also this all this implicit bias, the activation of implicit bias, it would be less conducive to the activation of implicit bias because you just don’t see this people and if you see them, it’s a some detail in the, in a page, it’s not a young person, a female like the or anything of that sort. It’s not as vivid.

The second aspect is what Orna mentioned with respect to structure, with respect to the structural bias. So, the only systems are way more structured. They are, I mean definitely the Matterhorn system, of course, it’s a matter of design but there is a certain inherent structure to online system which also means that there is less of a lee-room for all for implicit biases or other type of might, what we may see as irrelevant legal consideration to penetrate even inadvertently to the decision making.

And the third aspect is that online procedures are done remotely and the communication is asynchronous which means that it alleviates a lot of the stress, right that people, particularly people from marginalized group that do feel the stereotype threat about appearing in front of a judge that may think all kind of things about me because of the way I look or because of the way I talk.

So, the remoteness allows on the party side, on the participant side, the alleviation of the stress. And also, because of the asynchronous nature of the communication, to not to necessarily to think with their feet something that we know that the stereotype a threat would hurt but to calibrate their answer, maybe to get assistance from people that they trust. So, because it basically the shift to online changes, all the source and all the opportunities to these manifestations of disparities, we came to this project believing that the shift to online courts, even if we not eradicate will at the very least ameliorate the manifestation of outcome disparities compared to a face-to-face setting.

Orna Rabinovich

I’ll jump in here though. I will say we did feel that that there was a strong, we had a strong intuition that this would ameliorate implicit bias but I have to say things were not clear-cut. We also were concerned about the shift to written narratives and how that would impact parties, perhaps of marginalized group. So, it was a very sort of interest-provoking question and as scientists and you know for us we wanted to go out and study the data and luckily we found a partner. So we can.

Avital Mentovich

I know and I want to say another thing, if you jumped into my stuff so now I’m jumping into yours, that when I decided, we decided we had an idea, we decided to take the risk but part of it is because the in face-to-face setting, this phenomena of disparities in legal outcomes is so pervasive and is something that in the criminal justice system, in criminal policy making and in legal policy making, people discuss about so much and we only mentioned why they didn’t come with a solution that we didn’t, they didn’t come to a solution that managed to convincingly eradicate or eliminate these disparities. We thought that even if online system have half of a chance to do something about it, we should go for that because it will alter many of the phenomena, many of this what is I feel is disturbing phenomenon that we see in face-to-face settings. So, it was it was a high-risk but high reward project.

Orna Rabinovich

So, one more, one more ping pong here. I’ll just and then we’ll stop for questions. I’ll just say this is not only a legal problem, obviously of implicit bias.

Avital Mentovich


Orna Rabinovich

And as many of you might have heard, you know orchestras around the world have actually now have had additions with a screen for new people who are trying out for roles for various instruments and women were actually instructed not to wear high heels so they would play behind a screen and indeed the number of women in orchestras rows after that. So, we see that happening in various arenas. Now, we can take some questions perhaps.

Dunrie Greiling

I want to encourage people to drop questions in the Q&A. I haven’t seen any pop in yet and I’m wondering if the consensus out there is to move it along and talk about the research.

Orna Rabinovich

We can also take more after,

Dunrie Greiling


Orna Rabinovich

After the case study. Whatever works for people.

Dunrie Greiling

Yeah, I don’t see them popping in. I’ll just give it another minute or another moment. Yeah, it is very clear and I bet people are you prepared us very well to the main question is.

Avital Mentovich

Okay, the only question is the case study, please. I completely identify with this person. Show me the data, show me the data.

Dunrie Greiling

Yep, yep, so yeah. We have we have one saying, move us along so let’s.

Orna Rabinovich


Dunrie Greiling

Move along.

Orna Rabinovich

No problem. So, Tali has infected me with her love for data and now I’m. So, our case study is Matterhorn, we’ve as you all probably know, it’s been in operation since 2013 it’s spread out in over 70 state courts.

And with Matterhorn what was really attractive for us as researchers was that it not only introduces online mediation or online negotiation but actually online adjudication, online decision-making.

And that allowed us to compare what it meant for a judge to adjudicate a case face-to-face and adjudicate it from afar. And I think really it’s the only platform that would have allowed that this type of an examination and what we chose to study are traffic cases.

And, I have to say, you know initially, Tali and I had no familiarity with traffic cases and we pretty much fell in love with traffic cases because from a data point of view, they’re very good to work with.

What they do is they provide us with a very simple type of case that allows us to compare the online and the offline. As my research methods teacher Tali has taught me, when we want to make sure that we’re measuring the impact of the shift and medium, we want to make sure that all other aspects are similar as Tali will probably tell us in a bit. And we chose the traffic cases which were handled very similarly on and offline, the same input from the same types of parties but as we said before, instead of convening in a courtroom, what Matterhorn did here is is transform an informal hearing which takes place in a courtroom with a judge and a prosecutor to an online format but instead of meeting in the courtroom, basically, the two-way exchange in real life turns into parallel inputs from the parties and on the other side, the prosecutor and the judge. So, next slide, please.

Yeah, so what did we look at. We said we want to measure outcome disparities so we looked at two types of outcomes which can be reached in the traffic cases. They are the level of fines. So obviously, the more fines I have to pay, that’s a worse outcome for me, right and charge reduction. So, when my charge is reduced, I have less points.

So actually, the more charge reduction I get, that’s a better result for me, okay. So, we want to see whether for similar cases and Tali in a minute we’ll talk about what these similar cases, what that means. We have disparities in outcomes. Across social identity, what types of metrics did we take for social identity? We took age, gender, race and socioeconomic status on which we have some exploratory data. So, age, gender and race are all very conspicuous in a face-to-face meeting and they are either not non-observable in a written exchange or observable to varying degrees. We’ll tell you about their findings and explain that here as well. And often, not always, we might have an argument over age but often, at least gender and race should not be relevant to the outcome of the traffic case. Next slide please.

Okay, so a little bit about our sample. As you can see, we had roughly the same number, the same high numbers basically speaking of face-to-face and online cases. Now, all the controls you see here, I don’t want to get into this the methodological philosophy behind it but this was our way of guaranteeing that we are comparing apples and apples, that the face-to-face and the online cases are identical and that the changes that we do document and that we will show you in addition are only a result of the shift online and not of all these other aspects that can be intervening variables such as the identity of the judge, the date of the hearing, the time of the hearing, maybe the socioeconomic status of the neighborhood, whether it was a case that was even launched before or after Matterhorn. So, all these controls was basically to allow us to attribute all the findings that we’re going to show you to the change in medium, okay. Now Dunrie, we you can move another slide.

This is really a lot of numbers. I don’t expect you to read any of them. Let me just tell you that basically, these are the descriptive statistics for the online, for the face-to-face and the online sample. I can tell you that the samples are pretty equivalent in almost any demographic measure, perhaps the most pronounced difference is that with the participation of black users on and offline.

We see greater participation of black users in face-to-face setting compared to an online setting. We still don’t worry about it, those that care about numbers, we still had enough cases to offer meaningful statistical analysis but I’m urging you to keep that in mind when we’re going to show you what happens to black users when they’re moving online. It may be it will move, it will show that it’s under relatively, under representation of the of black users in the online system is something that we need to take care of because their, I’m going to, their situation is better off there, right. Keep that in mind, yeah.

Orna Rabinovich

Yeah. We, can go to the next one please.

Avital Mentovich

Okay, so first I’m going to show you what we found with respect to age. As you remember we look both, as Orna said, we looked both at fines and charge reduction.

This is the first, first I’m going to show you what we find with respect to fines. So here, you can see we have younger drivers, drivers who are 35 and younger and older driver, 36 and older. And as you can see in face-to-face setting, older drivers receive more favorable outcomes meaning they pay less compared to younger drivers. And again, because of the controls, we can say that these are basically similarly situated drivers meaning they’re identical in any other measure, apart of their age.

So, we do say that age is a driver of the outcomes in face-to-face settings but we see this impact of age completely disappears in the online review where a younger and older drivers receive the same penalty. By the way, it’s interesting to look at, at how we reach this equality. So, we never reach this equality by finally older drivers are paying more to the level of younger drivers, but by reducing the fines of the younger drivers

Orna Rabinovich

Everyone is paying less.

Avital Mentovich

To the level. Yes, so everybody’s paying less. So, it’s not only equality, is there everybody wins type of equality, right. Let’s move to what we find. Dunrie, next chart with respect to charge reduction.

So, with respect to charge reduction, you can see that there was no age-based disparities, face-to-face. So, the only system it nothing if you want to change in that or to fix in this sense. Exactly, they didn’t ruin anything and they didn’t fix anything because they didn’t date there was nothing to fix. So, age continued to be a non-relevant factor for the legal outcomes in online setting as it was in the face-to-face settings.

Now, let’s move to gender. In gender, that’s why I’m, we’re showing you that in one slide, in gender, we found basically no indication of gender-based disparities with respect to both fine reduction or charge reduction or fines in the face-to-face settings and again this dynamic hasn’t change in the online setting. So, we couldn’t find gender-based disparity online and offline.

I will tell you that it’s consistent with other research that perhaps surprisingly, right? It’s contrary to our stereotypes, show that women don’t receive any more lenient punishment than men when they’re compared, when they’re charged with the same offense.

Now, we’re going to move to race and I’ve already prepared you that this is probably the domain in which research suggests that we’re going to see the discrepancies in their most pronounced form and this is at least what we found in the face-to-face setting, right. No, surprises there. So, we can see that black drivers receive higher fines, compared to similarly situated white drivers. I’ll tell you two things about it.

A, the effect here of the difference, if you want, in the level of fine was larger than what, than the difference that we saw with respect to age, suggests that, if you want, the effect of race is larger. That black driver [effect] is stronger, are doing worse even compared to younger drivers.

As I can also tell you, that again the same dynamic that we found with respect to the age or any other measure that we looked at in the online setting, this difference completely was completely eliminated, not ameliorated but was eliminated in the online setting where we didn’t see any differences at the fine levels between the black and non-black drivers.

And again, this consistency, it’s the everybody wins sort of consistency in the sense that blacks were lowered to the level of fines of white people and non-blacks received slightly higher fines. We did find, Dunrie we can move now to the next slide.

With respect to charge reduction, again, we found a very similar dynamic where in the online setting, in the in-person hearings, blacks had the, black participants had significantly less chances of receiving charge reduction. It’s about by the way 20, 25% less chance of receiving charge reduction which is significant if you think about it.

And again, this dynamic changed both in the sense that it reaches equality in the online review than black and white have the exact same chance to receive charge reduction and they have a better chance of receiving charge reduction of 80%. I will say that if anything, right.

So, we see that blacks in the face-, black parties in the face-to-face settings are doing, are receiving a worse outcome, both in the charge reduction and in the level of fines, despite the fact that if anything, we’re talking about a population that is more in need of receiving some a form of lenient with respect to payment or they’re usually poor population, right.

So but the good news is that all of that disappears in the online setting.

Orna Rabinovich

We don’t have to focus so much on, I think we can.

Avital Mentovich

I’ll focus a little bit about it, about the intersectionality. We basically just found that the people that belong to these two groups, to our two implicated groups meaning young and black drivers in the face-to-face setting receive the worse outcomes compared to any other group, not surprising. They had the two categories that we showed you that are associated with inferior outcomes. And again, these differences with respect to this specific intersectional group, completely disappeared in the online setting. So, black young drivers were treated exactly like everybody else in the online setting. Next Dunrie. With respect to a SES, actually really interesting because SES if you can think about the design, it’s something that judges or other decision makers can

Orna Rabinovich

Socioeconomic status.

Avital Mentovich

Yes, socioeconomic status. Sorry, I’m so used to this acronym. So, if you think about the socioeconomic status, it’s something that they judges or other decision makers can not infer directly in the online setting, right. It may be something they can infer from ZIP. So, may perhaps because this is really preliminary results, perhaps because of this quality of the design, what we found are very strong impact of social economical status in face-to-face settings such that parties from ZIP, from basically from poor neighborhoods receive worse outcomes, both in terms of fines and in charge reduction in face-to-face setting compared to a parties from a more affluent neighborhoods. And also, this socioeconomic or socioeconomic status-based disparities completely disappeared in the online setting.

Orna Rabinovich

I do want to say two things here. First, that we hardly saw a drop in the percentage of very low socioeconomic status ZIP users with the move online, unlike the black users.

In terms of socioeconomic status, the lowest echelon actually didn’t drop much when we moved online. And also, returning to our challenge before we brought you the data, I think this is very impressive and promising, given that the shift to written communication might, we might have thought this would would create a different type of result along this side this metric. I think we can move to the next.

Yep, so we saw that in race, age and socioeconomic status, there was a significant impact in outcomes for social identity in face-to-face and the impact was practically eliminated online. With respect and you know, in here we can start, we’re not exclaiming, we get potential and hypotheses what could explain some of these dynamics.

Remember, we’re not talking about biases, we’re talking about disparate outcomes that correlate with parties’ social identity metrics and we can find different explanations.

I mean one explanation for age would be, yes, it is less salient online but it’s actually a relevant consideration. Someone might say “it’s not irrelevant. We want to give higher fines to younger drivers and deter them from making traffic violations.”

Another explanation could be that the young are more effective online, I don’t know, we were thinking about different possibilities. Or that it is some kind of and that would be a structural bias, right, that the young are less successful in face-to-face but more successful in the new medium.

Or this could be an implicit bias in the sense that judges if you ask them wouldn’t see this as a relevant consideration but it is impacting decisions by the way it might impact not necessarily the judges but the prosecutors along the way as well.

And with race, as Tali had mentioned earlier, this might be the characteristic that is most difficult to discern online. And that’s why and also we know one of the characteristics that is most detrimental to parties and face-to-face in terms of outcome disparities. So here, we see a big big difference, very interesting in our findings. Next slide please.

So, what we see here is basically the, the significance of design. As someone once said, the devil is in the design, okay. And the reduction of outcome disparities with the shift online is to a large extent, not only a product of the shift online itself but also of the specific design choices Matterhorn made, right. It could’ve made social identity more salient through video or a picture or even less salient by not having a first name, right, only an initial.

And what we see is there are inherent characteristics of the shift online that make identity less salient and also might make members of disempowered groups feel more comfortable but there are also many many elements of design that could shift the outcomes in different directions.

It’s not new to claim that process is important, process and shape outcomes. As many of you may recall, when the ADR was adopted in court, there were many fears and concerns that informal proceedings could harm minorities or women would be more collaborative. So, these are all the same questions but they’re sort of going up a level, right. We’re talking about the devil in the design online. And here what’s happening, there are many more opportunities because we’re no longer confined to the courtroom.

We have many many different options and we can very easily change these options along the way. So, if you’re a learning organization as Matterhorn is, you look at what’s happening, you look at your outcomes, you’re open to research and you can see what works and you can see how to improve what works and very easily sometimes change a feature that makes a big difference. Next slide, please. Tali, do you want to say something about the potential costs or should I go ahead?

Avital Mentovich

So, I’ll say a little bit about the procedural justice which is mine.

Orna Rabinovich

Okay, so so basically I think it and this is really I think our last slide. So, some of you might say, well, this all sounds great but sometimes, it’s really important to know who’s standing there, to see the person, to know something about their story, to know that they are a minority, a woman, a single mother or you know all this story of who we are and that’s really sort of substantive justice like concerns. Saying, we don’t really want all this formal equality. This might actually compete with our perceptions of justice.

Others might say and then Tali can jump in when you want that, you know it’s really important to have a process the parties feel is fair and that we’re not sure whether this shift from face-to-face to the online setting allows them to feel it’s fair. We know Tali.

Avital Mentovich

Particularly, so this is one of my area of that I study, particularly what we know about face-to-face setting that in order for feeling of fairness is very much contingent on the interpersonal interaction that parties had with judges or other important decision makers in the legal arena. So, it’s all about this like quality of interpersonal interaction and communication and we really, don’t know how people experience this interpersonal communication when it’s done remotely and what the price that the system can pay in terms of their satisfaction, the legitimacy, the trust they have this system. When they have a communication that is so devoid of this interpersonal touch but don’t worry, this is something that Orna and I are looking at with of course to help of Matterhorn.

Orna Rabinovich

That’s our next part.

Avital Mentovich

At this moment, this is our next research to see whether there is indeed the cost in this loss of touch or on the opposite with they lose in the interpersonal element but maybe gain higher trust in neutrality, higher trust in technology. So, we’re going to test all this consideration in face-to-face and in online courts. If you have a webinar next year maybe, maybe we have findings till then.

Orna Rabinovich

So Tali, I think we need to end because Dunrie has very politely told us.

Avital Mentovich


Orna Rabinovich

I’ll just say as a last point that when we consider all of these potential costs, a, we don’t know that they are indeed cost but we know that every time social identity is in the picture, is salient, it harms parties rather than helps them and in that respect we don’t think there is a true cost here that cannot be addressed through proper design, thank you.

Avital Mentovich


Dunrie Greiling

We had a couple questions that came in that I think are all on the research vein that I’d love to get your perspective on. Susan Yates has asked, how was race determined in your study?

Avital Mentovich

We received this information from the courts directly. Now, you can ask how the courts know it. To the best of our understanding it came from the police officer reports. So, I’m guessing it was their impression of the race and not necessarily the race as a person would self-report but we feel that at least it’s more realistic because it reflects how the decision-maker in this case, right, it started the police officer classify the race of the person.

Dunrie Greiling

Karl Branting has asked, how did you handle data confidentiality?

Avital Mentovich

Oh, we did it with the help of Matterhorn. We basically, Dunrie knows that, right, where we, Dunrie I feel that it’s a question for you.

I can tell you that with the data that we already received from Matterhorn, it’s virtually no identification, no names, no ZIP codes, nothing and whenever we needed some information, we asked them to do so. So, for example, at some point we had an idea of to run like a case of computer software that will infer the gender from names. Even at this point when we were never received the names but to ask Matterhorn to run the software for us. So whenever, the data set that were receiving have no identifiable information in it. We’re obligated to that is part of our ethical approval to do the research.

Dunrie Greiling

Thank you so much.

Orna Rabinovich

Clearly, the research was approved under IRB et cetera.

Dunrie Greiling

Thank you for that context, that’s super important to know. We have come to the end of our time so I’m just going to super quickly thank everyone for listening. Oh, let them, hold on let them know that the authors’ have generously provided a link to their paper which we will be sending you after, when we send the recording. And then I want to invite M.J. Cartwright to thank our panelists and let people know what’s next.

MJ Cartwright, Matterhorn CEO

Yeah, so Orna and Tali, thank you so much. Your research is very important for all of us and for everyone joining thank you. This is an incredibly intense time for everybody. Appreciate your time out of your busy days. I know everyone’s juggling so many things right now. We are very excited to work with some of our customers who’ve been really involved in online court processes since we started in 2014 and onwards, who are really going to help and offer suggestions, ideas and answer questions on April 7th. And for our town hall, it’ll be, Dunrie will be organizing this, not as a webinar but as a interactive session and she will have fun doing that. And I hope that you will join and ask your questions of you know staff that in the courts that are doing this, have done it and are continuing to do more and more online. So, be safe and be healthy and thank you very much for your time.

Dunrie Greiling

And thanks a million to Orna and Tali. for important research and a wonderful presentation and it’s a pleasure to work with you and spend this time together in your evening. Thank you so much.

Orna Rabinovich

Thank you very much.

Avital Mentovich

Thanks for having us.

Orna Rabinovich

We’re accessible by an email if anyone has follow-up questions, with pleasure. Okay, thank you, good day.

Outcome Disparities Presentation Slides

Outcome Disparities in the Age of Online Courts slide deck

Slides file (PDF)

Full Research Paper

Here is a pre-print of the research paper online: Are Litigation Outcome Disparities Inevitable? Courts, Technology, and the Future of Impartiality

Posted in Event.