Kelly reportedly said being chief of staff was the "worst job" he's ever had.