The answer isn’t just about temperature. Our physiology, psychology and clothing choices also factor in