The island in question is Burry Holms. The islands called Worms Head at the other end of the beach at Rhossili are little more than cliff faces and very dangerous (although we used to dive off there when we were kids).
We spent the night on Burry Holms once, intentionally though as we were fishing and had to wait for early morning for the tide to go out. There are remains of (if I remember correctly) an old monastery on the island
There is indeed a nudist beach the other side of Burry Holms which is also called Bluepool bay. It's only a nudist for a few weeks of the year though.
The campsite we used was called "Hillend" at Llangenith and the beach which stretches between Rhossili and the Holm is one of the best places to surf in the UK.