My limited experience has given me a healthy fear and taught me to just dress to swim and rig for epic failure... I see folks in pack rafts with critters in their hunting gear. I do not have that level of confidence hence the 5mm wetsuit, gloves, socks, etc.
If the worst were to happen the buoyancy from a suit in frigid water can't be understated as well. That and its the only attire that its not only acceptable but expected to piss your pants in.