I think that on the serious alpine descents it is more about controlling acceleration and getting the perfect line than pushing on - you'd be spinning out of any gear you have once you get anywhere near the speeds with which they descend.
Only to a point, FM. As the TV coverage misses much of the descents, it is not alweays obvious, but while those descents do have many long, monumentally fast sections, they also have plenty where major periods of pedalling are needed. Often it is wind-dependent.
Racing down them flat out is not a great deal less knackering than racing up them, due to the amount of pedalling, the sometimes rough surfaces and the concentration. Whenever I've been there (not since the 90s, sadly), my group always treated EVERY descent victory as a badge of honour, so they were always taken full-on - what's interesting is that we rarely encountered other cyclists able/prepared to do the same (this in July/August, when hundreds of amateur racing cyclists would be there), and we could never understand it!
Admittedly it's the sprinting out of tight bends that has the most effect on fatigue, as you identify, but the flatter wind-against sections take it out of you too, and you can spend a lot of time in the 11/12/13/14/15. The riders in the Tour, Dauphine etc often don't race down them, especially on the earlier passes. Racing down them is a major exercise in effort and concentration (even more so for recreational cyclists on open roads who have serious dangers to contend with too).