Your assumption is that rubber can only wear aluminium when it has some sort of abrasive embedded in it. However, in the dry, on tarmac roads, there is no grit to embed and rubber still abrades the aluminium. Obviously in the wet, the additional silica particles in the road water helps to speed up the process but in its absence it still wears away the aluminium.
Friction interfaces such as brakes are poorly understood. Typically, there is a soft and a hard compound sliding against each other. In this case, it is rubber and aluminium. When braking, the rubber heats up due to stretching of the molecules and the breaking of the covalent bonds between the molecules. This heat cannot go anywhere since rubber is a poor conductor of heat and is thus transferred to the aluminium, an excellent conductor. Eventually the rim does feel warm because of the even distribution of the heat with time. But at the time of braking there is lots of heat at the interface, enough to remove aluminium molecules and immediately oxidise them, hence the black sludge, even in the rain.
When the rubber's chemistry changes in tiny spots, that's where the larger pick-up problem arises. If you examine a rim that's been gouged by a stone, as Gobalti describes, you will see that the edges are jagged, as you would expect from a cold scrape. If you inspect the ridges caused by alu pick-up, you will see that temperature was involved and the metal has a smoother appearance at the edges of the groove. Alu starts to become soft at about 450C (IIRC, that's about the temperature at which it is pushed through dies for extrusion profiles) and easily reaches that temperature at the pad/rubber interface. All it needs at that temperature is a piece of crystalised rubber and it rolls up as we see in brake pads.
