They really don't have to pass you. Also, in a lot of urban areas in England now, the average motor traffic speed is often slower than what many cyclists will do (famously below 10mph in many at peak times) so they might not be able to, depending on road layouts.
I've mentioned before that the driver behind often doesn't seem to see a car/van/lorry in front of them and crashes into its rear, so what hope that such people will see a cyclist? I don't feel it's safe to wait to be caught in a concertina if you can avoid it.
Why wouldn't that annoy drivers too? It seems a bit like too much faith in paint.
In America the car scene is obviously very different than in England, in America the motorist has the mentality that they come first, and they will pass cyclist (and each other) no matter what because a cyclist is impeding their ability to get somewhere, and if that means they'll have to put you in danger they'll do it. Our urban streets, or streets with houses and families, the speed limit is 30 mph, which I think is too fast, motorists often do 40 to 45, main urban streets are 35 to 40 with again people going at least 10 over. So it's highly unlikely that a cyclist is going to be challenging our motorist for speed. The only time a cyclist would be able to challenge the speed of a car is in heavy traffic times.
Even with all the aggressiveness on our street not a lot of rear end accidents happen, not saying they never happen but it's not as common as someone running a stop and getting hit by the other car. If I'm in a line of traffic everyone knows that people are stopping, and I always take a look behind me to make sure the driver is aware of my presence.
Bike lanes painted on a street is only useful when cyclists are on the lane, then motorists tend to pay attention to it more, but just like riding on the street without a painted lane you still have to be aware and careful, I don't deny that, doing anything counter to that and someday it could catch up with you. It's a lot like buying a car that can self drive, even though the instructions say to be paying attention, people will get laxed and will fail to pay attention then an accident will happen, so likewise even if you're in a dedicated bike lane you still have to pay attention.