Law School Discussion

Nine Years of Discussion
;

Author Topic: The skinny on scales  (Read 1577 times)

Root Hog

  • Sr. Citizen
  • ****
  • Posts: 350
  • The USA PATRIOT Act
    • View Profile
    • LSN
The skinny on scales
« on: October 06, 2006, 06:00:11 PM »
I've done some research and here is the explaination on how scales are determined. The PERCENTILE is pre-set base on the past 11 test dates. This will set, for example, a 165 as 93%. This makes scores from any given test fairly comparable with any test given within three years. As for what raw score equals a 166, this is determined by the ACTUAL test. Your percentile is not a reflection of how you did versus takers of your teset, but rather a comparison (scaled score-wise) of how you did versus all takers in the past three years. This can explain the huge fluctuations in scales from one test to the next. To think that the OCT test would be pre-set at -10 for a 170 is wrong. There is a school of thought here that fewer takers who scored poorly will cancel than in the past and this could lead to an extra point of leniency on the test, but this may be wishful thinking.

mjb

  • Guest
Re: The skinny on scales
« Reply #1 on: October 06, 2006, 06:07:42 PM »

You're ignoring that they KNOW how people will do on this test in advance as long as their statistical samples remain accurate.  Since there is no reason to believe that, suddenly, a completely different demographic is taking the LSAT, the score is essentially predetermined by statistics.

You may have a point about the influence of increased non-cancels, but I imagine the number of high scoring retakers would balance that out quite nicely.

I think you underestimate the costs of having a "perfect" lsat if it is even possible. Can you imagine how hard itd be to prescale and have a corresponding bell curve output? The most likely scenario is they curve it afterwards with scores attached to percentile scores. I don't know where it got started that they predefine a scale because that just sounds a little off?.


gratefulDead

  • Full Member
  • ***
  • Posts: 37
    • View Profile
    • Email
Re: The skinny on scales
« Reply #2 on: October 06, 2006, 06:09:13 PM »

You're ignoring that they KNOW how people will do on this test in advance as long as their statistical samples remain accurate.  Since there is no reason to believe that, suddenly, a completely different demographic is taking the LSAT, the score is essentially predetermined by statistics.

You may have a point about the influence of increased non-cancels, but I imagine the number of high scoring retakers would balance that out quite nicely.

The fact that a test is generally difficult when presented as a whole (on test day) doesn't mean that when they tested the sections as experimentals that people scored poorly.  So I think the argument of "they had statistics about the tests" is wrong.  They had information about the questions, who missed them, how the section looked as a whole, etc.  Putting four of these sections together as one can have drastically different results IMO.

I am Penny Lane

  • Sr. Citizen
  • ****
  • Posts: 3579
  • Manager of the "Get Dotlyn to go to FSU" Campaign!
    • View Profile
    • LSN - What do you think of my choices?
Re: The skinny on scales
« Reply #3 on: October 06, 2006, 06:13:23 PM »
... This makes scores from any given test fairly comparable with any test given within three years. As for what raw score equals a 166, this is determined by the ACTUAL test. Your percentile is not a reflection of how you did versus takers of your teset, but rather a comparison (scaled score-wise) of how you did versus all takers in the past three years. ...

So if this is right, then what do you all think about tests over the last 3 years vs Sept 30?

Here is what people thought about the last 2 years: http://www.lawschooldiscussion.org/prelaw/index.php/topic,71590.0.html
LSN

Quote from: dotlyn
PennyLane invented sweet. She has the patent on it. I tried to act sweet one time and she sued me.

mjb

  • Guest
Re: The skinny on scales
« Reply #4 on: October 06, 2006, 06:16:54 PM »


Yes, I can imagine, because I know they have statistics on every question ahead of time.  That's how score correlation works.  Given those statistics and the assumption that a significant demographic shift does not occur, you know the scale of the test.


To try and gain a predictive scale to match a perfect curve would be extremely costly. There is no benefit for them to do that. If they want a curve and scores to match percentile scores they curve after the test is taken. I dont think LSAC would spend hundreds of thousands if not even millions just to have it pre-determined.

gratefulDead

  • Full Member
  • ***
  • Posts: 37
    • View Profile
    • Email
Re: The skinny on scales
« Reply #5 on: October 06, 2006, 06:17:20 PM »

I'm not really sure I understand your argument.  Are you saying a section gets easier or more difficult based on the other sections in a test?

Either really.  I'm just saying that I think they use the experimental testing as a gauge of individual question difficulty, rather than determining scales.  Sections can be scored very different depending when in a test it comes up (i.e. is it the first section?  is it the last?), is it right after a super hard games section (could result in lower scores or even higher scores??? point is, who knows).  If the scoring on a particular test as a whole is low, then it doesn't strike me as relevant what the statistics say about those sections, and I really believe that LSAC evaluates the raw score to scaled score (120 -180) after the exam.  

Root Hog

  • Sr. Citizen
  • ****
  • Posts: 350
  • The USA PATRIOT Act
    • View Profile
    • LSN
Re: The skinny on scales
« Reply #6 on: October 06, 2006, 06:17:40 PM »

Yes, I can imagine, because I know they have statistics on every question ahead of time.  That's how score correlation works.  Given those statistics and the assumption that a significant demographic shift does not occur, you know the scale of the test.




The fact that a test is generally difficult when presented as a whole (on test day) doesn't mean that when they tested the sections as experimentals that people scored poorly.  So I think the argument of "they had statistics about the tests" is wrong.  They had information about the questions, who missed them, how the section looked as a whole, etc.  Putting four of these sections together as one can have drastically different results IMO.

I'm not really sure I understand your argument.  Are you saying a section gets easier or more difficult based on the other sections in a test?
[/quote]

 You should also take into account that when they test these, they 1) don't always test them together entirely as a whole section and 2) they are TESTING, which means they will inherently throw some out and make adjustments that can have drastic effects on scoring. If they have tested everything and have such a great pre-conceived notion of how the sections fkly, why would they occasional feel the need to remove one from scoring?

Root Hog

  • Sr. Citizen
  • ****
  • Posts: 350
  • The USA PATRIOT Act
    • View Profile
    • LSN
Re: The skinny on scales
« Reply #7 on: October 06, 2006, 06:19:21 PM »
The first part of my last post was a quote and does not refelect the opinion of the OP

Root Hog

  • Sr. Citizen
  • ****
  • Posts: 350
  • The USA PATRIOT Act
    • View Profile
    • LSN
Re: The skinny on scales
« Reply #8 on: October 06, 2006, 06:24:38 PM »
Now here is the POWERSCORE explaination, which is the opposite of what I found on three different sites. Bottom line is, I don't believe LSAC gives out that information and we are at their mercy.

What is the experimental section and why is it on the test?

Each LSAT contains five sections, but only four sections are scored. The extra, unscored section is known as the experimental section. According to Law Services, this section is used to "pretest new test items and to preequate new test forms." In other words, the experimental section contains items that will appear on future LSATs, and the results of your performance are used to determine future scoring scales (for related information on the LSAT Scoring Scale, please click here).

Although the experimental section has no direct impact on your score, your performance in this section is carefully analyzed by Law Services. Sophisticated statistical analyses are performed on your answers, and these results, when combined with information from thousands of other test takers, present a clear picture of the difficulty and logical validity of each question. Then, when future tests are assembled, questions can be selected with a high degree of confidence in how test takers will perform. Thus, entire tests can be assembled with a very accurate understanding of how many questions a 170-level scorer would miss, how many questions a 160-level scorer would miss, etc. With this information, scoring scales can be determined even before a test has been administered

mjb

  • Guest
Re: The skinny on scales
« Reply #9 on: October 06, 2006, 06:24:53 PM »


To try and gain a predictive scale to match a perfect curve would be extremely costly. There is no benefit for them to do that. If they want a curve and scores to match percentile scores they curve after the test is taken. I dont think LSAC would spend hundreds of thousands if not even millions just to have it pre-determined.


Hmm... I bet a few people on this board (including me) could write a program to do exactly that in under a day, given the data.

But ok.

I think we have a miscommunication. I am saying that LSAC puts out the test, receives raw data and then puts it into their curve and generates a scale. What I believe you're saying is that LSAC crafts perfect tests where they predict how the percentiles and scores will occur. You're saying that have their scale attached to scores prior to releasing the test.

Am I wrong?