One of the biggest problems to mess your average up is when your on the open roads, having to slow down and give way at junctions etc and other traffic problems. If your unlucky and get stopped quite a lot, it can soon have an accumulative effect on your average speed and the further you get into your ride the harder it becomes to make time up. It is on the longer rides that it gets hard to alter an average speed if your slightly behind. Set your computer to display average speed and in about the last 10 miles of a long ride you try and raise it substantially, as I have found it creeps up painfully slow, sometimes about a tenth of a mile every 2 miles. In effect if you plan on doing a 60 mile ride in 4 hours at an average of 15mph, if you do the first 2 hours at 12mph average, it then means that for 36 miles you have to rocket along at 18mph just to raise that average to 15mph, and how many rural roads can you go for 36 miles without some form of delay, which means having to travel more than 18mph to make up for these delays. And also can you keep 18+ mph up for 36 miles? Imagine a 60 mile ride hoping to achieve 18mph average? the speeds that you then need to cover delays etc
It took me a long while to do a sub 3 hour, 50 mile ride, despite being able to keep up a speed that can achieve it, I would always be a minute or two out, till one day by accident, I did it with just over 10 minutes to spare but when I looked at the route, 28 miles there and back, there was only about 4 roundabouts to negotiate and luck had been with me at them.
Careful route planning should help, by minimising, towns, junctions etc and always try and plan your route anticlockwise so that when you get to a junction you only have to worry about traffic from the right as a rule and if its a good open junction you can plan ahead to filter in.