Johnny, firstly congrats for sticking with them for as long as possible
Fitness will come, trust me. about 10 weeks ago I used to go out on the club Sunday rides 50 odd miles with a tea stop, by the time we were close to the stop I was dropping off the back, especially if there were bumpy bits, average speed was around 15mph. Now I ride on the back of the fast group, doing my share where possible and averaging about 20mph. When I ride with the slow group (when I need to just have a light ride) I spend almost all the time on the front.
So ride fitness, strength, recovery and stamina will come, just take your time, ride as much as you want BUT factor in recovery days.
For example I never ride on a Monday, Tuesday is an easy ride, Weds hard, Thursday club ride, Friday easy, Sat easy or very hard club, Sunday hard unless I did the hard ride on Sat and then easy out hard back.