Having worn hi vis gear at work for thirty years and seen the difference, especially at night or poor light I can't see a reasoned argument for not wearing it.
Not to wear - 1 - I don't see why I should take responsibility for my own safety - it's always someone else’s fault and of course 2 - I want to prove a point [even if it kills me].
Reasons to wear 1- you have to wear something anyway [if not it can get a bit chilly]. If you ride to work cycling gear and most work gear are not the same so you can't usually use the same stuff. 2 - it just isn't expensive. 3 - it might save your life - especially at night reflective strips etc really do work. Finally there are no statistics for accidents that don't happen. To say "oh x number of motorists passed just as close" is irrelevant - the point is they saw you anyway however close they came. Only by being involved in an accident and then duplicating the exact circumstances again [impossible] with different riding gear can the theory be in anyway proved.
The closer they come, the more slight the misjudgement that would result in a collision.
Reflective strips are not hi-vis. I have subtle reflective elements even on dark jackets. I don't mind that. It's the builder-style yellow red herring jackets I no longer wear unless I'm doing something unusual, like a site inspection or marshalling.
It's fun to make up silly reasons that are easy to knock down, but not very useful. Here are some actual reasons not to wear - 1 hi-vis doesn't improve safety: there's no reduction in collisions or injury found in studies yet, 2 - yellow and orange are basically low-contrast camouflage against the low winter sun common at this time of year, 3 - IMO it's less visible than even black under the newest LED streetlights (it shows up grey - this surprised me, but it's understandable because the lights are blueish-white), 4- it doesn't matter if the driver sees you slightly earlier as long as they see you in time, 5 - it might actually be worse if the driver sees you so early that they feel they have dealt with you by the time they pass, 6 - it's antisocial because it distracts drivers from other riders/walkers/animals who may be nearer but aren't dressed like a lemon, 7 - it's a bit of an arms race - once everything is hi-vis yellow, then we're all back to square one and there'll be calls to wear something else - maybe a lit yellow star? 8 - it's ugly and spoils tourist photos (tourism is important to Norfolk), 9 - it's another thing to carry around with you all day, 10 - after a few washes (varies by jacket - check the label - I've seen as low as ten washes), it's no longer hi-vis so you have to keep buying new ones.
Calls for people to wear hi-vis are either deliberate attempts or clueless help to distract from the bigger source of danger: bad motorists.