Is it possible to predict the outcomes of legal cases – such as Supreme Court decisions – using Artificial Intelligence (AI)? \u00a0I recently had the opportunity to consider\u00a0this point at a talk that I gave entitled “Machine Learning Within Law<\/a>” at Stanford.<\/p> At that talk, I discussed a very interesting new paper entitled “Predicting the Behavior of the Supreme Court of the United States”<\/a>\u00a0by Prof. Dan Katz (Mich. State Law),<\/a> \u00a0Data Scientist Michael Bommarito<\/a>, \u00a0and Prof. Josh Blackman (South Texas Law).<\/a><\/p> Katz, Bommarito, and Blackman used\u00a0machine-learning AI\u00a0techniques<\/a>\u00a0to build a computer model capable of predicting the outcomes of arbitrary Supreme Court cases with an accuracy of about 70% – a strong result. \u00a0This post will discuss their\u00a0approach and why it was an improvement over prior research in this area.<\/p>\n The general idea behind such approaches is\u00a0to use computer-based analysis of existing<\/em>\u00a0data (e.g. data on past Supreme Court cases) in order to predict the outcome of future<\/em> legal events (e.g. pending\u00a0cases). \u00a0The approach to using data<\/em> to inform\u00a0legal predictions (as opposed to pure lawyerly analysis) has been largely championed by Prof. Katz – something that he has dubbed \u00a0“Quantitative\u00a0Legal Prediction” in recent work.<\/a><\/p> Legal prediction is an important function that attorneys perform for clients. \u00a0Attorneys predict all sorts of things, ranging from the likely outcome of pending cases, risk of liability, and estimates about damages, to the importance of various laws and facts to legal decision-makers. \u00a0 Attorneys use a mix of legal training, problem-solving, analysis, experience, analogical reasoning, common sense, intuition and other higher order cognitive skills to engage in sophisticated, informed\u00a0assessments of likely outcomes.<\/p> By contrast, the\u00a0quantitative approach takes a different tack: \u00a0using analysis of data employing advanced algorithms to result in data-driven predictions of legal outcomes (instead of, or in addition to traditional legal analysis). \u00a0These data-driven predictions can provide additional information to support attorney analysis.<\/p>\n Outside of law, predictive analytics<\/a> has widely applied to produce\u00a0automated, predictions\u00a0in multiple contexts. \u00a0 Real world examples of predictive analytics include: the\u00a0automated product recommendations made by Amazon.com<\/a>, movie recommendations made by Netflix,<\/a> and the search terms\u00a0automatically suggested by Google<\/a>.<\/p>\n In general, predictive analytics approaches use\u00a0advanced computer algorithms to scan large amounts of data to detect patterns. \u00a0These patterns can be often used to make intelligent, useful predictions about never-before-seen future data. \u00a0Many of these approaches employ\u00a0“Machine Learning<\/a>” techniques to engage in prediction.\u00a0I have written about some of the ways that machine-learning based analytical approaches are starting to be used within law and the legal system<\/a>.<\/p> Broadly speaking, machine-learning<\/a>\u00a0refers to a research area studying computer systems that are able improve their\u00a0performance\u00a0on some task over time with experience. \u00a0Such algorithms are specifically designed\u00a0to detect patterns in data that can be highlight non-obvious relationships or that can be predictive of future outcomes (such as detecting Netflix users who like movie X, tend also to like movie Y and concluding you like movie X, so you’re likely to like movie Y.)<\/p> Importantly these\u00a0algorithms are designed to\u00a0“learn” – \u00a0in the sense that they can change their own behavior to get better at some task – like predicting movie preferences – over time by detecting new, useful\u00a0patterns within additional data. \u00a0Thus, the\u00a0general idea behind predictive legal analytics is to\u00a0examine data concerning past legal cases and use\u00a0 machine learning algorithms to detect and learn\u00a0patterns that could be predictive of future case outcomes.<\/p> In such a\u00a0machine learning approach — called supervised learning – \u00a0we “train” the algorithm by providing\u00a0it \u00a0with examples of past data that is has been definitively\u00a0classified. \u00a0For example, there may be a body of existing data about Supreme Court cases along with confirmed data indicating whether the outcome was affirm or reverse, along with other potentially predictive data, such as lower circuit, and subject matter at issue. \u00a0Such an algorithm examines this training data to detect patterns and statistical correlations between variables and outcomes (e.g. 9th Circuit cases more likely to be reversed) and build a computer model that will be predictive of future outcomes.<\/p> It is helpful to briefly review some earlier\u00a0research in using data\u00a0analytics to engage prediction of Supreme Court outcomes\u00a0to understand the\u00a0contribution of Katz, Bommarito, and Blackman’s paper.<\/p>\n Pioneering work in the area of quantitative legal prediction began in 2004 with a seminal\u00a0project\u00a0by Prof. Ted Ruger (U Penn)<\/a>, Andrew D. Martin (now dean at U Michigan)<\/a>\u00a0and other collaborators,\u00a0employing statistical\u00a0methods to predict Supreme Court outcomes.<\/a>\u00a0 \u00a0That project pitted experts in legal prediction – law professors and attorneys – against a statistical model that had analyzed\u00a0data about hundreds of past Supreme Court cases<\/a>.<\/p> Somewhat surprisingly\u00a0the computer model significantly outperformed the experts in predictive ability. The computer model correctly\u00a0forecasted\u00a075%<\/strong> of Supreme Court outcomes, while the experts only had a 59%<\/strong> success rate in predicting Supreme Court affirm\u00a0or reversal decisions. \u00a0(The computer and the experts performed roughly the same in predicting the\u00a0votes of individual justices – as opposed to the ultimate outcome – \u00a0with the computer getting 66.7 % correct predictions\u00a0vs. the experts 67.9%).<\/p>\n The work by Ruger, Martin et. al – while pioneering – left some room for improvement. \u00a0One aspect\u00a0was that their predictive model – while\u00a0highly predictive of the relatively short time frame examined (the October 2002 term) \u00a0– was thought not to\u00a0be broadly generalizable<\/em> to predicting arbitrary\u00a0Supreme Court cases\u00a0across any timespan. \u00a0A primary reason was that the period of Supreme Court cases that they examined to build their models – roughly 1994 – 2000 – involved an unusually stable court. \u00a0Notably, this period exhibited no change in personnel (i.e. justices leaving the court and new justices being appointed).<\/p> A model that was “trained” on data from an unusually stable period of the Supreme Court, and tested on a short case-load\u00a0of relatively non-fluctuation\u00a0might not perform as accurately when applied to a\u00a0broader or less homogenous examination period, or might not handle changes in court composition in a robust manner.<\/p> Ideally, we\u00a0would any such computer predictive\u00a0model\u00a0to be flexible\u00a0enough, and generalizable enough to handle significant changes in personnel\u00a0and still be able to produce accurate predictions. Additionally, such a model\u00a0should be general\u00a0enough to predict case outcomes with a relatively consistent level of accuracy regardless of the term or period of years examined.<\/p>\n While building upon Ruger et al’s pioneering work. Katz, Bommarito, and Blackman improve upon it\u00a0by employing a relatively new machine learning approach known as “Random\u00a0Forests.”<\/a>\u00a0 \u00a0Without getting into the details, it is important to note that Random Forest approaches have been shown to be quite\u00a0robust and generalizable as compared to\u00a0other modeling approaches in contexts such as this. \u00a0 The authors applied this algorithmic approach to examine data about past Supreme Court cases found in the Supreme Court Database<\/a>. \u00a0In addition to outcome (e.g. affirmed, reverse), this\u00a0database contains hundreds of variables about nearly every\u00a0Supreme Court decision of the past 60 years.<\/p> Recall that machine learning approaches often working by providing\u00a0an algorithm with existing data (such as data concerning past Supreme Court case outcomes and potentially predictive variables such as lower-circuit) in order to “train” it. \u00a0The algorithms looks for patterns and builds an internal computer model that can hopefully be used to provide prediction is future, never-before-seen data – such as pending Supreme Court case.<\/p> Katz, Bommarito, and Blackman did this and produced a new\u00a0robust machine-learning based computer model that correctly forecasted ~ 70%\u00a0<\/strong>\u00a0of Supreme Court affirm \/ reverse decisions.<\/p> This was actually a significant improvement over prior work. \u00a0 Although Ruger’s et. al’s model had a\u00a0a 75%<\/strong> prediction rate on the period it was analyzed against, \u00a0Katz et. al’s model was a much more\u00a0robust, generalizable model.<\/p> The new model is able\u00a0to withstand changes in Supreme Court composition and still produce accurate results even when applied across widely variable supreme court terms, with varying levels of case predictability. \u00a0 In other words, it is unlikely that the Ruger model – focused only on one term 2002 – would produce a 75% rate across a 50 year range of Supreme Court jurisprudence. \u00a0By contrast,\u00a0the computer model produced by Katz et. model consistently delivered\u00a0a 70% prediction rate across nearly 8,000 cases across 50+ years.<\/p>\n Katz, Bommarito, and Blackman’s paper is an important contribution. \u00a0In the not too distant future, such data-driven approaches\u00a0to engaging in legal prediction are likely to become more common within law. Outside of law, data analytics and machine-learning have been transforming industries ranging from medicine to finance, and it is unlikely that law will remain as comparatively untouched by such sweeping changes as it remains today.<\/p> In future posts I will discuss machine learning within law more generally, and principles for understanding what such AI techniques ca, and cannot do within law given the state of current technology, and some implications of these technological changes.<\/p>\n","protected":false},"excerpt":{"rendered":" Predicting Supreme Court Outcomes\u00a0Using AI ?Is it possible to predict the outcomes of legal cases – such as Supreme Court decisions – using Artificial Intelligence (AI)? \u00a0I recently had the opportunity to consider\u00a0this point at a talk that I gave entitled “Machine Learning Within Law” at Stanford.At that talk, I discussed a very interesting new […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[8,5,10,9],"tags":[],"_links":{"self":[{"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/posts\/248"}],"collection":[{"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/comments?post=248"}],"version-history":[{"count":10,"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/posts\/248\/revisions"}],"predecessor-version":[{"id":268,"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/posts\/248\/revisions\/268"}],"wp:attachment":[{"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/media?parent=248"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/categories?post=248"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.harrysurden.com\/wordpress\/wp-json\/wp\/v2\/tags?post=248"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Quantitative\u00a0Legal Prediction<\/h2>
Predictive Analytics: Finding Useful Patterns in Data<\/strong><\/h2>
Scanning Data for Patterns that Are Predictive of Future Outcomes<\/strong><\/h3>
Prior Work in Analytical Supreme Court Prediction<\/h2>
Improvements by Katz, Bommarito, and Blackman (2014)<\/h2>
Katz, Bommarito, and Blackman: Machine Learning And Random Forests<\/h3>
Conclusion: Prediction in Law Going Forward<\/strong><\/h3>