When I got to Newlands, my start time was 7:22, people were already walking and that was with 14 miles and two hills taken out.
I had a 07:20 start time and am guessing at 5:19 finish time was slower than you, when I was at Newlands there were some people starting to walk up the right hand side as well as the left. But not as many, it wasn't congested yet, but you were having to think a bit about lines. In general, aside from the walkers, it was still slow on the left, fast on the right which put me somewhere near the middle.
It's possible enough (there didn't need to be many to look like a lot of people walking) just don't do hills like it. The South East has quite a lot of flat bits and a fair number of riders are probably London based and not that dedicated. Also in some cases it was probably one walking because they had to and another because they were doing the route together.
As long as they don't block the road I don't mind as it makes me feel fitter :-)