Law School Discussion

Nine Years of Discussion
;

Author Topic: US News Rankings are Bogus  (Read 19476 times)

mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
US News Rankings are Bogus
« on: September 16, 2006, 09:41:57 PM »
Take my law school, UC Hastings for example. In literally EVERY other ranking out there, UC Hastings is far better than its US News rank. You know something is wrong with your ranking system when literally EVERYONE ELSE has hastings ranked higher than you. It's not just random statistical variance with some coming in lower... US News rankings are bogus I feel seriously sorry for students who base their decisions off them... like kids who go to davis over hastings... if you look at the exterior numbers hastings has way higher bar passage rates, top firm recruitment, and median salary levels than uc davis.

UC Hastings
2007 US News Rank: 43rd

2007 Educational Quality Rank: 29th (http://www.top-law-schools.com/rankings.html)
2007 Justice Brennan's Top 20: 11th (http://www.top-law-schools.com/rankings.html)
Median Salary: tied for 18th with UC Berkeley (http://www.ilrg.com/schools/salary/)
2006 Leiter's Student Quality Ranking: 34th (http://www.leiterrankings.com/students/2006student_quality.shtml)
Leiter's Faculty Quality Rankings: 23rd (http://www.leiterrankings.com/faculty/2005faculty_impact_cites.shtml)
Bar Passage Rate vs. Average: 9th (http://www.ilrg.com/rankings/law/index.php/1/desc/SchoolvsBar)
Law School 100: 35th (http://www.lawschool100.com/)
Consensus Group 100: 26th (http://www.consusgroup.com/news/rankings/law_schools/law_schools.asp)
Leiter's Top Firms Recruiting: 20th - 200 (http://leiterlawschool.typepad.com/leiter/2006/02/national_and_re.html)

mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
Re: US News Rankings are Bogus
« Reply #1 on: September 16, 2006, 09:48:57 PM »
oh yeah and my girlfriend is a 3-L at hastings... she is in the top 25% of her class, not in moot court or law journal, and she summered at, was offered, and accepted an offer at Orrick, the #1 firm in the bay area and a top firm nationally and internationally. She will be making $135,000/year starting next year... all that from getting good but not great grades and not doing any extracurriclars...

just another reason why US News is bogus - The Firms know US News is bogus. That's why hastings is still top 20 for top firm recruitment despite being ranked 43rd in US News.

mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
Re: US News Rankings are Bogus
« Reply #2 on: September 16, 2006, 11:49:09 PM »
Look, I'm not trying to argue Hastings is better than G-town or Uc Berkeley or anything but it's definitely way better than 43rd...

"There are no other decent rankings that I'm aware of."

I think Leiter's Law Rankings are way better than US News. He also has links to other alternative rankings from his website. I agree with you about Wisconsin and Tulane also being underranked by US News. They are just other examples of why not to trust US News.

"Morrison & Foerster is the top SF firm.  Sorry.  Dude your GF is so TTT."

I think Orrick  passed them this year for Bay Area-wide (not just SF) with the strong growth at their silicon valley office. But whatever if not #1 then #2 firm in the bay area. The point is still the same - not even top 15% and no extracurricluars and still got a job at a top notch firm.

"It's fascinating that you leave out placement in academia, yet want to make the argument that the school's basically regarded as t20 or thereabouts by everybody other than USNWR."

I did not leave out their placement in academia. Leiter's faculty quality rankings put Hastings at 23rd in the country! It's there in my list! He even says this about Hastings' faculty on his website:

"Hastings recruited some leading "doctrinalists" (broadly construed), including Roger Park (evidence) from the University of Minnesota and, most recently, Geoffrey Hazard (legal ethics, civil procedure) from the University of Pennsylvania; the Hastings group in the civil procedure and evidence areas (Kane, Park, Hazard, Marcus, Faigman, among others) is now probably one of the two or three best in the country."

"Like many large, state law schools, Hastings has been treated badly by U.S. News; probably only Wisconsin has fared as badly at the hands of the U.S. News criteria that reward a school for being small and private."

http://leiterlawschool.typepad.com/leiter/2005/12/u_conn_law_dean.html

Top two or three in civil procedure and evidence! Both of which are some of the most important bar classes one will take at law school. So much for your whole "left out their place in academia" BS.

I think my point is still pretty clear: US News is bogus and should not be used as a major deciding factor. Especially if you are considering schools that it underranks like Hastings, Tulane, and Wisconsin, or schools that it overranks (private schools with huge endowments).

mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
Re: US News Rankings are Bogus
« Reply #3 on: September 17, 2006, 12:27:38 AM »
Getting some weird responses here... mistah you seem to agree that US News DOES in fact under rank schools like Hastings and Wisconsin and yet you are still highly critical of me calling out US News.

"And look, I've read basically everything Brian Leiter has to say.  He expressly does not DO "overall" rankings -- just more focused things, e.g. faculty quality, student quality, national firm placement success, SCOTUS clerkship success, and academia placement success."

OK well there are other "overall" rankings that I posted up there and all of them have Hastings ranked way higher than US News does, just like all of Leiter's rankings. Noticing any patterns there?

"Evidence is one of the most important classes you take in law school?  That's interesting.  And uninteresting, because faculty quality has nothing to do with the quality of instruction."

Evidence is the only class required for you to become bar-certified before you graduate so that you can argue motions in court during law clerkships or externships. So yes, it is pretty freakin' important.

So you have read everything Leiter has to say? Do you agree or disagree with what he has to say? Because your previous post indicates you seem to agree with him that US News does under rank schools. Which is exactly what my point is.

mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
Re: US News Rankings are Bogus
« Reply #4 on: September 17, 2006, 02:22:47 AM »
First of all I am not trying to argue Hastings should be ranked in the top 20 by US News I am just saying it is a lot better than 43rd and should be in the 20's or 30's. So you disagree with Leiter and think US News is a great ranking system. Awesome. I completely disagree with you... I agree with Leiter's critique of the US News rankings that he posted on his site:

http://www.leiterrankings.com/usnews/guide.shtml

"The U.S. News methodology for ranking law schools is confusing, comprising 12 different factors in 5 different categories.  It is crucial to recognize two things, however, at the start.  First, the relative weight of the factors varies dramatically, with some having a significant effect on the results (reputation, median numerical credentials, expenditures), while others matter hardly at all (size of the library, acceptance rates, bar passage rates).  Second, the factors vary quite a bit in their susceptibility to artificial manipulation by law schools.  However, the fact is some of the factors are highly manipulable, and, as a result, the overall ranking results are meaningless, though no less important, alas, because of that.

Note also that U.S. News has actually held its “methodology” for ranking schools constant since 1999, after making changes every year prior to that.  In 1999—a consequence of U.S. News having hired someone with expert knowledge of statistics—they made perhaps the single most dramatic change in their methodology:  they started adjusting per capita expenditures (item #6, below) to reflect differences in cost-of-living.  The results were so dramatic—Albama turned up in the top 50, Fordham and Boston College dropped out of the top 25 (BC has since returned)—that in 1999 U.S. News stopped printing the “Faculty Resources” rank (as they call this category):  it would have been too obvious how this irrelevant expenditures category had skewed the rankings.  (Of course, it still skews them, in favor of small schools like Yale and against large schools like Harvard, but more on that shortly.)

Here are the factors U.S. News employs to rank schools, in descending order of importance.  The factor in question is also described as either “Highly Manipulable,” meaning schools can exercise, through deceit or otherwise, a lot of control over this criterion; or “Not Manipulable,” meaning the criterion is basically beyond a school’s control.

Academic reputation (25% of the overall score).  Not Manipulable.  25% of the overall score is a function of academic reputation, as measured by a survey done mid-fall of some 700 law school deans and faculty (about two-thirds fill out the surveys).  Since US News switched to the new ranking scale in 1998 (1-5, where 1 is marginal and 5 is distinguished), there have been far fewer ties than there were early in the 1990s.  The scores have also stabilized in to predictable clusters of Yale, Harvard, Stanford; then Columbia and Chicago; then Michigan, NYU, Berkeley; then Virginia; then Penn; then Cornell, Duke, Georgetown, Northwestern, and Texas; then UCLA; then Vanderbilt; then Southern California, Wisconsin, Minnesota, and Iowa, and so on.  These results aren’t ridiculous, but, at least with regard to faculty quality, they are also dated:  NYU, for example, is clearly better than Michigan, and certainly on a par with Columbia; Chicago is at least on a par with Stanford; Yale is better than Harvard; and so on.  This is unsurprising given that evaluators are presented only with a list of 180 school names, and nothing more.  For a snapshot of what leading legal scholars think about faculty quality when actually presented with faculty lists, see http://www.utexas.edu/law/faculty/bleiter/rankings/rankings03.html.  Schools have little control over the results of the academic reputation survey:  even improving the quality of your school (its faculty, its student body) does not necessarily result in any increase in the academic reputation score.  (Case in point:  NYU has seen its academic reputation score decline during a period when both its faculty and student body improved.)  

15% of the overall score is based on reputation among lawyers and judges.  Not Manipulable.  These results reflect a survey of lawyers at large firms and federal and state judges.  The response rate is low:  less than one-third complete the surveys.  Because U.S. News only surveys large firms, the survey is also dramatically skewed towards the Northeast (especially New York City):  schools that have large alumni contingents in New York City perform, shall we say, suspiciously well in this survey by comparison to schools that are otherwise comparable.  Schools have little control over the results of this survey.

12.5% of the overall score is based on the median LSAT score.  Highly Manipulable.  This criterion is one of many that favors small schools.  Consider:  a school that enrolls 180 students each year, only needs to recruit 90 with an LSAT of, say, at least 164 in order to have a strong median LSAT.  A school that enrolls 450 each year, by contrast, will need to recruit 225 students (more than twice as many) with that LSAT to report the very same median.  Note also that U.S. News has no way of verifying the data reported by private schools, since the American Bar Association does not collect median LSAT data, only data about the 25th and 75th percentile.  So this factor is highly manipulable by the schools.

12% of the overall score is the employment rate 9 months after graduation. Highly Manipulable.  This data is entirely self-reported by schools, and should be treated as essentially fiction:  it may have elements of truth, but basically it’s a work of the imagination.  Schools report it, and U.S. News has no way of checking.  In addition, we know nothing about the nature of the employment—it could simply be as a research assistant, which is what Northwestern did a few years ago for its unemployed grads.

10% of the overall score is based on the median GPA of the entering class.  Highly Manipulable.  See the discussion in (3), above.  Note, too, that the feeder schools for a particular law school will have a significant effect on this criterion.  Example:  schools that draw on the “grade inflated” Ivy League have it easier than those that draw on universities with less rampant grade inflation.

9.75% of the overall score is average per capita expenditures for this year and the prior year for instruction, library, and supporting services.  Highly Manipulable.  This is the figure that is adjusted for differences in cost of living.  Once again, schools self-report the data.  This criterion, along with (3) and (5), gives a huge boost to small schools, since per capita measures penalize for economies of scale.  This explains how, in many years (including 2003), Harvard can have higher reputation scores than Yale, yet Yale will come out 1st and Harvard 3rd.  Harvard is three times the size, and that makes all the difference.

The preceding are the six major factors making up a school's overall score in US News. Together they account for 84.25% of the overall score.  Four of these factors are highly manipulable, and three favor small schools.  The remaining six factors in U.S. News (that account for just 15.75% of the overall score) are as follows:

6% of the overall score is the employment rate at graduation.  Highly Manipulable.  See the discussion in (4), above.

3% of the overall score is the student-teacher ratio.  ABA collects data on this, and so does U.S. News, but there are often discrepancies, which U.S. News appears to let slide.  So the manipulability of this category is unclear, but seems to be high.  Much depends on how schools “count” their faculty.

2.5% of the overall score is the acceptance rate for students.  Highly Manipulable.  As with (3) and (5), U.S. News has no way of verifying the data reported by private schools.  In addition, many schools inflate their “selectivity” by giving fee waivers to applicants who have no chance of getting in.  NYU is reported to have pioneered in this arena, but many others have followed suit.

2% of the overall score is the bar pass rate adjusted to reflect the avg. pass rate in the major jurisdiction where students take the exam.  Not Manipulable.

1.5% of the overall score is average per capita expenditures for the current and prior year on everything else OTHER than instruction, library & supporting services--so this includes utilities, financial aid, and the like.  Highly Manipulable.  As with (6), the criterion also favors small schools.  Stories abound about schools who, via little accounting changes here and there, boost their rank in this category astronomically.

 0.75% of the overall score is the total number of volumes in the library.  Not Manipulable.  Schools reports this data to the ABA, which means it is checkable.  (Schools that might lie to U.S. News are unlikely to lie to the ABA.)

Even putting aside the fact that this formula, with its various weightings, is impossible to rationalize in any principled way, the really striking fact about the U.S. News methodology is surely the following:

More than half the criteria—over 54%--that go in to the final score can be manipulated by the schools themselves, either through outright (and undetectable) deceit, or other devices (giving fee waivers to hopeless applicants, employing graduates in temp jobs to boost employment stats, etc.).

More than one-third of the criteria that go in to the final score favor small schools and penalize large schools.

Reread the U.S. News rankings with these two pertinent facts in mind, and a lot that looks academically indefensible about the results (Chicago behind Columbia, Penn ahead of Berkeley, Duke ahead of Georgetown and Texas) may begin to make sense."



mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
Re: US News Rankings are Bogus
« Reply #5 on: September 17, 2006, 02:27:34 AM »
I would especially like to emphasize the last part of his critique:

"Even putting aside the fact that this formula, with its various weightings, is impossible to rationalize in any principled way, the really striking fact about the U.S. News methodology is surely the following:

More than half the criteria—over 54%--that go in to the final score can be manipulated by the schools themselves, either through outright (and undetectable) deceit, or other devices (giving fee waivers to hopeless applicants, employing graduates in temp jobs to boost employment stats, etc.).

More than one-third of the criteria that go in to the final score favor small schools and penalize large schools.

Reread the U.S. News rankings with these two pertinent facts in mind, and a lot that looks academically indefensible about the results (Chicago behind Columbia, Penn ahead of Berkeley, Duke ahead of Georgetown and Texas) may begin to make sense."


So now we have way more than just a couple of fallacies in the US News rankings. We have Chicago behind Columbia, Penn ahead of Berkeley, Duke ahead of G-town and Texas, and the underrankings of Tulane, Hastings, and Wisconsin. I am sure there are way more than that. US News rankings are bogus.

mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
Re: US News Rankings are Bogus
« Reply #6 on: September 17, 2006, 02:36:28 AM »
"Let's get down to brass tacks: only 30 schools can be in the top 30.  Hastings isn't one of them, so here's the crux... which schools would you displace in order to satisfy your ego, and under what methodology would you displace them?"

I don't know how to compare hastings to schools outside of California but the best example I can give is UC Davis law school. It is ranked 34th in the most recent US News rankings. 9 spots ahead of UC Hastings. Yet Hastings has a far higher bar passage rate. Look at the bar passage rate site - http://www.ilrg.com/rankings/law/index.php/1/desc/SchoolvsBar - out of the California schools (which YOU CAN compare to) - Hastings is ranked just ahead of USC and a good 5% ahead of UC Davis.

I found a median salary report from last year. http://www.ilrg.com/rankings/law/median.php/1/desc/MSPrivate

Hastings is ranked 30th overall. UC Davis 45th overall.

I think those should be setting off big lightbulbs for everyone here that Hastings is better than US News has it and belongs at least in the top 40 if not the top 30.

mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
Re: US News Rankings are Bogus
« Reply #7 on: September 17, 2006, 05:43:34 PM »
"Indeed!  Two of the top 5 by USNWR have 500+ 1L class sizes, and one of those is likely overranked.  I would investigate the Hastings 1L class size, but LSAC is down (again)."

Hastings entering classes are a little more than 400 students with a total student body of 1,261. So it's up there in terms of size.

"Fallacy would be to say that just because some schools are underranked means that the entire ranking system is flawed.

ANY ranking system is going to leave SOME people displeased."

This isn't just being displeased. This is clear evidence of numerous schools being over ranked and numerous schools being under ranked by US News.

"Let's get down to brass tacks: only 30 schools can be in the top 30.  Hastings isn't one of them, so here's the crux... which schools would you displace in order to satisfy your ego, and under what methodology would you displace them?"

Well I'm not an expert but I like Brian Leiter's Educational Quality Rankings which are based on three factors: student quality, teacher quality, and quality of instruction. In those rankings Hastings is 29th overall. But I'm not saying that's the best alternative out there. My point is that if you look at EVERY OTHER ALTERNATIVE ranking out there, Hastings is ranked higher than it is in US News.

Of course, I also think salary and bar passage rate should be important. That is a major reason behind why people go to law school - it's a professional school that people go to to start careers.

"So is your argument that the US News rankings suck?  Or just that Hastings is underranked in these rankings?"

Both. Hastings is just one of many examples of how US News rankings are bogus.

"You're not at all clear on what your criteria for ranking schools should be.  You know that, don't you?  It's really quite pathetic.  At least Fordham trolls, while irritating, have a clear (if stupid) argument -- thank God there aren't more Hastings trolls, eh?"

Like I said, I like Leiter's rankings but I also think bar passage and salary are important. However, your point here does not really address my main argument, which is that under EVERY OTHER ALTERNATIVE RANKING out there - using numerous different criteria - Hastings is ranked higher than it is in US News. There is no statistical variance. In not one of those ranks does Hastings come in higher than the 43 spot it occupies in US News.

mitchb

  • Jr. Member
  • **
  • Posts: 14
    • View Profile
    • Email
Re: US News Rankings are Bogus
« Reply #8 on: September 17, 2006, 09:28:40 PM »
"You still have the problem of deciding which criteria are meaningful -- saying "other rankings using random criteria find differently" doesn't cut it.  You still have the problem of addressing Leiter's highly suspect methodology.  Furthermore, you've failed to cite "every other alternative ranking" out there."

Why is leiter's methodology so "suspect?" You're making a claim without any arguments or analysis behind it. I like Leiter's different rankings, they show you lots of different things about how schools stack up against each other. And in every single one of them Hastings is better than 43rd.

"The Ciolli study perhaps being the most important of these, but the ALAMAR rankings not being insignificant, nor the potentially forthcoming A-H rankings1."

I looked up the Ciolli study... the guys who did it are: "a 21-year-old kid from Queens who just finished his first year at Penn's law school, the other a college dropout and insurance salesman"

"Turns out Mr. Ciolli's delusions extend to the realm of politics.  As a Penn law student pointed out to me, Mr. Ciolli is also "very dissappointed with Bush" since "he is FAR too left wing for my tastes.""


http://leiterreports.typepad.com/blog/2006/03/anthony_ciollis.html

Real reliable source there. But even still, in the Ciolli study, UC Hastings comes in at #33 in national firm placement! That's ten spots better than the 43 that US News has it at. (http://www.autoadmit.com/studies/ciolli/draft14.pdf#search='ciolli%20study') Pg. 35, appendix B.

"I think this is the key distinction useful in interpreting ratings.  Beyond the top 20 or 30 schools, regional reputation overwhelms other differences."

I completely agree. Which is why I think it is ridiculous that UC Davis is ranked ahead of Hastings when Hastings outperforms it in all of these alternative rankings. I think Davis is the best comparison to Hastings in terms of regions. But you are right if you want to practice in the midwest it's better to go to Wisconsin or something.

"Hastings is ranked #167 in student-to-faculty ratio.  Only 13 schools in the whole country are worse."

Good point, that should definitely be a factor in rankings.

"Hastings is #72 in bar passage rate among the top 100 schools."

That is a very inaccurate measure b/c it is nationwide instead of state by state. As many people know California is one of the hardest if not the hardest Bar exams to pass. When you look at the comparison to state averages, Hastings is #9 - just ahead of USC and well ahead of UC Davis in bar passage rate!

Those last two aren't really good factors to include in rankings.

Like I said before, This isn't just being displeased. This is clear evidence of numerous schools being over ranked and numerous schools being under ranked by US News. Every other study and ranking has Hastings higher than 43. Except for this Alamar thing that has hastings at 43... but I don't know what alamar is. So you still haven't found any rankings that put hastings worse than 43. Everything else has hastings better than  43 and one is tied at 43. I think you guys are the ones with the lack of evidence here, not me. Especially considering even one of the studies YOU cited - Ciolli - has hastings way higher than 43.

So Yeah....

  • Guest
Re: US News Rankings are Bogus
« Reply #9 on: September 17, 2006, 10:16:28 PM »
remember, if you're not first you're last.