"The fear is ... not only would he lose the general election, he would take GOP candidates down with him," said the former national security adviser.