I've pondered this viewpoint myself.
I went to an elementary school in an inner city where I was one of TWELVE white students (no exaggeration- that was the total for all six grades).
I think it made me a better person.
But I've also come to conclude that it's not racist to want to live among certain people. If I choose to live among people who are purple, that's my choice.
We all have personal freedom.
Racism, on the other hand, is where you wish to deter another's personal freedom through intimidation or physical violence. If I want to live near white people and decide that the best way to do that is to scare off black people, that's racism. But if I go online and do some research and decide that Montana is the whitest place around, there's nothing racist about packing my bags and buying a couple plane tickets.
In fact, it would be racist to make the person stay where they don't want to be- is this not slavery?
The decision to live among people who look and think like yourself might be ignorant, stupid, and lazy- but it's not racist.