Like Jonny I ride in London, ride fast, and very rarely have any problems.
I suspect other factors also play a part, such as whereabouts in London you ride (I find my Guardianista-heavy North London homeland noticeably less hairy than when I stray east or - horrors - go south of the river) and other aspects of the way you ride - particularly how good a feel you have for others' positions and perspectives. But yes, I'm sure that maintaining a fair speed helps.
(Think about passing a row of parked cars...if you're doing near-enough traffic speed, a quick glance back should let you slip easily into a gap, enabling you to pass those parked cars without impeding drivers....at a significantly slower speed, you're going to be moving out in such a way that drivers have to slow down to your speed and/or pass close enough to make you uncomfortable or force you to ride too close to those potentially-opening doors.)