True, but as my bike lives indoors I clean it after mucky rides. So the rims get cleaned for other reasons. Any improvement, or no change at all, to braking is incidental.
getting confused here mr trousers, so you do clean rims, even if only "accidentally" by the by when doing other stuff.
I'll keep on cleaning mine now and again - they definitely get black oily grubby debris on them - mostly from the road I think. Definitely not imagining it. I also ride a fair old bit in certain rural northern parts where roads can be particularly mucky. Seems to me a good idea not to have an extra layer of hard gunk slime (ready to be mixed with more water) between the two braking surfaces. There's also inevitably bits of micro muck in there.
I do remember going into the famed Roberts bike builders once, somehow getting into a diversionary natter about rim brakes and being told that it was important to keep an eye on rim cleanliness. Was also by the by congratulated on the cleanliness of mine.