Not in the sea but I did swim in a lake about once a month throughout the winter. Coldest I swam in was just over 4 degrees with a wetsuit, gloves and neoprene boots. 6 degrees was the coldest I managed to swim properly at a decent pace and technique (for me). 4 degrees was a little cold and I swam around the lake mixing styles a bit in an attempt to look like I was swimming

.The other people I swam with dealt with the cold better than me but I got going in the end and managed to acclimatize to the water temp each time. One person I occasionally trained with wore only a standard swimming costume through the winter training for the 500m
Chill Swim in the Lake District. 500m suddenly becomes difficult when it's that bl**dy cold and you're not allowed a wetsuit. Very impressive and I've no idea how she did it

. It turns out there are groups of people all over the country who swim regularly outdoors all year round. I was only a dabbler but much respect to the people who go out all of the time.
To stop feeling unwell afterwards I took to wearing many layers for a few hours. It was OK getting warmed up straight after but if I didn't keep a few more layers over my standard t-shirt offering the cold started to creep back in.