“[Liberals warned that it could make it easier to tear down poor neighborhoods.] We don't need you on this one, ... We need you to respect the right of those minorities and those poor people to hold on to what is their own.”