A former Gaetz adviser said sex-trafficking allegations are 'baseless' but gave no evidence and said he hasn't spoken to the congressman in months