Lynne Cox did a one mile swim (25 minutes) in Antartic waters (32°F) in 2003:

https://www.cbsnews.com/news/swimming-to-antarctica-12-02-20...

She swam in even colder Arctic waters (26.6°F) for 5m26s in 2007:

http://www.espn.com/espn/feature/story/_/page/Lynne-Cox/lynn...

Her book is a fun read:

https://www.amazon.com/Swimming-Antarctica-Long-Distance-Swi...