Mitt Romney (R-UT) on Wednesday took a swipe at right-wing Supreme Court Justice Clarence Thomas, arguing that Thomas is an outlier