A lot depends upon the bike and it's geometry (and your trust in it). My handbuilt steel bike is incredibly stable at high speed and I've done 60 mph in N. Wales, and regularly top 50 mph round the Peaks. Much also depends upon your knowledge of a descent, and important things like junctions.
On training runs, the main hill I ride on is 40mph speed limit, and it's very easy to hit that speed - just have to take it easy near the top as there are a few junctions.
Don't think about falling off, and don't death grip the bike. One of my most fun descents was with dan_bo and Longers as we came off Saddleworth down into Greenfield - steady 43 plus mph, pedalling at points, with lots of swooping bends.
The expression 'putting ones brain in your back pocket' helps.
Descending on tight single lane descents, with blind corners is where you take things easy. Also watch out for grit washed off fields onto bends (Pyms Chair and Lamaload being typically very iffy).