"I did have to say that even though the former president is entitled to his opinions, he's not entitled to his own set of facts,” Cavuto began