Former Secretary of Housing and Urban Development Julián Castro said instead of “building up a great wall around us, he has built a great wall between us.”