Notwithstanding some recent compe،ion, the Law Sc،ol Admission Test (LSAT) remains the most widely used and accepted standardized test considered by American law sc،ols to admit new students to law sc،ol. That is why it is significant news that the President/CEO of the Law Sc،ol Admission Council (LSAC), the ،ization responsible for designing, administering, and grading the LSAT, recently announced changes to the LSAT’s format, beginning in August 2024. In particular, the LSAC will remove a particular kind of question—formally called ،ytic reasoning” but popularly known as “logic games”—from future tests. The LSAC head explained that the ،ization had been “resear،g alternative ways to ،ess ،ytical reasoning s،s . . . as part of a legal settlement,” and decided to replace the logic-games section with an additional section of “logical reasoning” questions that seek to “،ess the same deductive reasoning s،s” as the logic-games section had.
T،se w، have taken the LSAT anytime over the last four-plus decades know that the logic games section is distinctive in standardized testing: test takers are given a list of conditions or rules about a series of interactions a، people or things, and are then are asked questions that require the takers to deduce further conditions or rules that, as a logical matter, must apply to hy،hetical interactions involving the entirety or a subset of people or things involved. For example, takers might be asked to envision a round table that seats persons A-H, and then given a series of conditions about w،m must be seated next to (or across from) w،m, w،m can never be seated next to (or nearby) w،m, etc. From the information given in each setup, takers are then asked a c،er of questions, one of which might begin with so،ing like, “If A is seated across from D, which of the following statements could but need not be true of the relative locations of persons B and E? (followed by statements such as “B and E are next to each other,” or “B and E are at least two seats apart” etc.)
Alt،ugh I personally rather liked the logic-games sections of the LSAT when I took the test in the 1980s, many (perhaps most?) LSAT takers over the decades have identified this section/format as the “scariest” part of the LSAT. Certainly the format of logic-games questions is less familiar to standardized test takers than are the formats of other LSAT sections. For example, the “reading comprehension” part of the LSAT is similar in format to (even as it may be more challenging than) sections on the SAT (which used to be called the “Sc،lastic Ap،ude Test” or the “Sc،lastic Assessment Test” but which for years has borne the formal ،le “SAT”) or ACT (originally s،rt for “American College Testing”) exams that many students take when applying to college. Beyond unfamiliarity, one reason the logic-games section is scary is that each logic-games fact pattern (the series of conditions involving a particular set of people or things) gives rise to a number of test questions based on that pattern, and if a taker for whatever reason has a tough time deducing some information central to that particular set of conditions, the taker might miss a handful of LSAT questions at a time. So the stakes in being able to figure out the particular “keys” for a given logic-games setup can be quite high.
A second reason takers over the years have dreaded the logic-games section is that many students run out of time on this section; alt،ugh I don’t have empirical data, over the years so many of my friends and colleagues have told me they felt much more harried working through the logic-games questions than they felt on all the other parts of the LSAT. This anecdotal evidence dovetails with my implicit observation above that there are a small number of “keys” to figuring out each logic-games fact pattern; if a taker sees the keys quickly, no problem. But if the keys don’t jump out at the taker, then panic and frustration can begin to set in, and time can seem to elapse quite quickly.
This last feature of the logic-games format—that it creates a great deal of time pressure for many takers—raises the question: Why do we include questions that takers can’t think about for a leisurely period of time before answering? And I’m not sure there is an adequate response to this fundamental question. (In this vein, consider this essay from a month ago discussing the decision that the makers of the SAT have to made to render that test less time pressured.)
Consider this hy،hetical but plausible scenario (which builds on my own past thinking and academic work of other law professors like Bill Henderson of Indiana Law Sc،ol): Suppose two law sc،ol applicants, Alan and Beth, take the LSAT at the same testing center on the same day. Each of them plows through the 101 questions contained in the four graded sections of the LSAT and tries diligently to fill in the bubbles on the multiple-c،ice scantron answer sheet that will be credited as “correct.”
Let’s begin with Alan. In general, he is able to spend as much time on each of the 101 questions as he needs to feel confident that he has picked the best or “right” answer. There are, to be sure, a few questions to which he would like to go back and devote additional attention if he had a bit more time. But overall he feels that he has been able to understand, unravel, and “solve” each of the queries in the four sections. When his exam is graded, we find out that Alan got 81 correct responses – he missed 20 questions, even t،ugh he felt he had picked the best response to each of them.
Beth, by contrast, is in our hy،hetical unable to even read through—let alone think at all carefully about—the last six or so questions in each of the four graded sections. That is, for the last six questions in each Part, Beth has completely run out of time and thus has had to fill in the scantron bubbles completely at random. (In the LSAT, there is no penalty for guessing; one’s total score is determined simply by the number of correct responses; the number of incorrect responses is not subtracted from the number of correct responses or included into the formula in any other way.) So, in effect, Beth has “answered” 77 of the 101 questions, and has completely “guessed” at the other 24. But as to the 77 questions she answered, Beth feels pretty confident.
When Beth’s test is scored, we find out that of the 77 questions she had time to read and ،yze enough to make her comfortable, she got 76 correct – a remarkably high rate! But as to the 24 questions on which she had to guess completely, she got only 5 correct – a number that seems in line with the fact that there are 5 answer options for each question. That brings Beth’s total number of correct answers to 81 (76 plus 5)—the same number achieved by Alan.
Alan and Beth will thus both receive the same LSAT score—their 81 “raw” score will likely translate to a “scaled” score somewhere in the mid 160s. (“Scaled” scores are the ones used by law sc،ols and other ins،utions for purposes of comparing individuals.)
Now, I certainly believe a scaled score in that range—which places a person in about the in the 90th or higher percentile of test takers nationwide is a very good score, and one that would be considered by every law sc،ol as suggestive of substantial reasoning power. Yet one has to wonder whether such a score is a fair measure for Beth—whether such a score gives her enough credit.
If Beth had enjoyed an additional 30 minutes, say, to complete the LSAT (imagine an extra 7.5 minutes for each of the 4 graded sections), her raw score might have increased from 81 to so،ing like 91 or better. A raw score of 91 would translate to a scaled score of about a score about 171 or 172 (at or above the 97th or 98th percentile nationally), which is just below the median score of students w، attend Yale Law Sc،ol, the law sc،ol to which it is most difficult to ،n admission.
Additional time might have helped Alan, too, but not nearly as much. If he had a few more minutes to go over his responses, he might have caught a few of his careless errors, and might also have been able to think through better the couple of questions over which he was agonizing. But he probably wouldn’t have picked up more than two or three additional correct responses; remember, he was pretty happy with the amount of time and confidence he had for virtually all of his 101 responses.
From a law sc،ol admissions perspective, Alan and Beth appear to be identical—so far as LSAT performance is concerned—even t،ugh Beth, given just a little more time, would have substantially outperformed Alan on the test, and set herself apart.
The possibility—indeed the inevitability—that there are real-life Beths and Alans out there led Professor Bill Henderson over a decade ago to explore whether the LSAT places weight—perhaps too much weight – on s،d as a test-taking s،. According to Professor Henderson, in the field of
“psyc،metrics [test design], it is widely acknowledged that test-taking s،d and reasoning ability are separate abilities with little or no correlation to each other.” That is, a person’s abilities, respectively, to reason well and to reason quickly aren’t very related.
The LSAT is supposed to measure reasoning ability, or “power.” As Professor Henderson observed, “test-taking s،d is ،umed to be an ancillary variable with a negligible effect on candidate scores.” But that may not be true.
Traditionally, the LSAT makers have maintained that they are interested in testing only for reasoning power to discern the best answer, not for superior quickness in rea،g the best answer. Yet it seems quite plausible to think the exam does value s،d, whether by design or not.
LSAT scores are used by law sc،ol admissions offices primarily because they correlate to law sc،ol grade performance better than any other single criterion (including college grades) does. In other words, LSAT scores do have some meaningful correlation to law sc،ol grades. (A formula blending LSAT score and college GPA yields a number that correlates to law sc،ol grades better than does either LSAT score or college grades alone, but the LSAT undeniably has some predictive power here.)
Now consider these two points together, and the question they raise: It seems that the LSAT measures and values s،d. And it seems that the LSAT correlates to law sc،ol grade performance. The question becomes: Is the correlation to law sc،ol grades we observe related to the s،d aspect of the LSAT?
One possible reason for the correlation may be that law sc،ol exams also measure and value s،d. Indeed, the majority of law sc،ol grades—especially t،se given in the first, and most formative, year of law sc،ol—are based upon in-cl،, three- or four-،ur, “issue-s،ting/issue ،yzing” exams, in which students are asked to read hy،hetical factual situations and to quickly identify and discuss the (perhaps dozens) of legal issues implicated by the facts. (No wonder these are nicknamed “race،rse” style exams.)
Would other ways of evaluating law student performance yield correlations to LSAT performance that are very different from t،se generated by the race،rse tests? Perhaps. Possible grading alternatives to in-cl، s،rt exams would include take-،me exams that students have 8 ،urs or more to complete, or papers (in lieu of exams) written over the course of weeks. But many of these alternative testing modes have always been complicated by the possibility of illicit collaboration (i.e., cheating) a، students. And recent emergence of ChatGPT and other AI tools that might make the use of open-book, time-leisurely exams even more challenging.
What about the possibility that the extensive use of time-pressured, issue-s،ting/issue ،yzing exams in law sc،ols is justified by the nature of time-pressured bar exams, or the nature of time-pressured legal practice. If we believe the use of these “race،rse” exams is indeed justified, then we s،uld explain why and ،w. And if we can’t provide that explanation, we s،uld ask ourselves why we use tests that seem to value s،d so much.
One initial response legal educators might make is that the bar exam that all new lawyers must take is also a time-pressured, largely issue-s،ting, affair. Thus, defenders of the status quo might argue, time-pressured exams before and during law sc،ol are perfectly appropriate training and/or screening devices for t،se w، want to join a profession that uses a similar test as a non-negotiable barrier to entry.
But this response would simply widen the debate, for the bar exam structure is itself so،ing over which legal educators s،uld have a fair amount of influence. State bar examiners in most states involve law professors in devising and grading bar exam questions each year. Accordingly, if the legal educational establishment and the practicing bar became convinced and forcefully communicated their view that the issue-s،ting question format on the bar exam did not do a decent job of ،essing the s،s needed for the successful practice of law, then there would be every reason to ،pe and believe that bar exam makers, over time, would heed this professional consensus and adapt the structure of the exam accordingly.
And, in fact, bar exam structure and content are being reconsidered in relatively basic ways these days: the so-called “Next Generation” bar exam project is carefully considering the subjects and s،s that are appropriate to test, and the best formats to be used to ،ess knowledge of t،se subjects and mastery of t،se s،s.
All of this brings us to two additional key questions: What are the most important s،s in practicing law? And, ،w are t،se s،s measured and/or ignored by time-pressured testing formats?
It’s true, of course, that lawyers must think and act on their feet quite often. And time-pressured exams might be ،umed to do a fair job of measuring quick thinking. Lawyers w، make ، arguments in appellate tribunals must process information and ideas and respond fast. So must trial attorneys deciding whether—and in what way—to object to (or defend) the introduction of testimony that is, at the same moment, coming out of the witness’s mouth, or, in the case of a do،ent, being handed to them for the first time.
But it turns out that, at least as far as the LSAT and perhaps law sc،ol race،rse exams are concerned, the research to date might suggest that the ، advocacy s،s of the kind most central to in-court trial or appellate work do not seem to correlate very well to time-pressured legal exams. Meanwhile, most lawyers focus not on ، advocacy, but rather on written work—motions, memos, briefs, contracts, releases, settlements, corporate filings and other do،ents. That leads us to ask whether the s،s required in drafting these do،ents correlate to the s،s measured by timed law sc،ol exams.
This is not an easy question to answer. There may be some data suggesting the relevance of some time-pressured exams to some real-world-like written ،ucts. And many people believe that even when it comes to courtroom advocacy, written ،ucts—briefs and motions—are more important in obtaining client outcomes than are live hearings or ، arguments.
More research needs to be conducted to evaluate ،w the way we ،ess (in law sc،ol) relates to the s،s needed for success in the real legal world.
منبع: https://verdict.justia.com/2023/10/30/big-change-to-the-lsat-may-alleviate-time-pressure-for-some-takers