I think there is more to it than just leg fitness. I've injured my legs a couple of times, but apart from those times it has always been some other part that has caused problems on long rides. Always my lower back, often my bum and sometimes my arms, shoulders or neck; a couple of times my wrists or hands.
In my case, problems are not caused by poor bike fit or inadequate cycle clothing because my longest, hardest rides carrying panniers were trouble-free (140+ mile rides with 10,000+ ft of Pennine climbing). If I'm slim (or slimmish) and fit, I can put the work in; if I'm not, it hurts - a lot!