I don't see how those benefits would be threatened. Instead, I would worry that those who have worked hard and deserve a job/position out of MERIT wouldn't be getting it. I also think that AA based on race wrong because other factors have be used to discriminate as well. Religion, economic status, political beliefs, etc. If you look into anyone's past you can find examples of discrimination at some point. As for as "networking" and that other stuff, those are connections that most people make during college and doesn't require the government to give them a boost.
And for "waking up EVERY morning fearing the bs that other people are going to throw at you"... Wow. You must be so afraid of being black that you are probably more conscious of your skin color than most white who come in contact with you. It is sad that you think you live in a world where every white person lays awake at night thinking of ways to oppress blacks and other minorities. It sounds like you are the one who has racist views.