The real question is who does it hurt? It only hurts, individually, applicants at the margins who instead of going to HLS have to go to CLS. People displaced by AA aren't objectively qualified to be at that school anyways. The school would have to find some non-numerical justification to admit them as well. What's wrong with giving some person who hasn't had access to opportunity or is part of an unrepresented group a chance? You're both unqualified numerically. Or is it that you're angry because you're not qualified numerically and someone even less numerically qualified gets in over you? At that point, tough luck. I mean, there are numerically unqualified white students at even Harvard and Yale.
You see, this is actually the best point I have seen anyone make in awhile about AA (which I am against, for the record.) The vast majority of the time, it is not so much that a numerically qualified person is displaced, it's that a numerically unqualified non-URM gets rejected and an equal or less numerically qualified URM does get in. It does make for a weak argument, but you can't blame the people who get lost in the shuffle for feeling indignant about it. If you are an extremely rich or well-connected white person, you can get into HYS if you are unqualified. If you are a numerically under-qualified URM, you can get into schools well above the range of a typical applicant with your numbers. This leaves a group of people that have to hope they can write a diversity statement good enough to get them a second look and I am fairly certain these things get thrown in the trash, figuratively speaking. Schools like numbers, money and statistics. There will always be anecdotal exceptions people can point to, but overall this is the reality.
I have always said that I would be fine with race-blind, income based affirmative action, but I am not optimistic about that becoming the norm.