My only argument would be for that solid colour to be different than the background to increase contrast. I am coming from the standpoint of mainly rural riding where dark clothes really do blend in well with backgrounds such as high hedgerows.
I do concede that in urban environments there may be other factors.
Possibly, yes, I agree with you (I think) that yellow, red or orange hi-viz is more noticeable than, say, dark green, black or blue. Or purple. I posted, about 18 months ago, that I prefer cyclists in hi viz as a driver as it gives me more time to plan the passing manoeuvre.
My point remains, though, is that necessary? Yellow might be more noticeable than black from 1,000m but, IMO only if the black isn't visible from 100m is it an issue.
To use an analogy, the lights on the front landing gear of a commercial airliner are visible from about 100 miles away. My Exposure bike lights are visible from about a mile. But my bike lights are all that I need on the bike and 20,000 lumens are not needed. Similarly, if I can be adequately visible at 100m wearing black, then why go for overkill with the yellow?