As a gereral rule, the wider the tyre the more rolling resistance it creates, as there is more rubber in contact with the road. So going down a size or two should speed your ride up a bit, but you will notice a bit more road vibration as there is less rubber. I run 25's on my bikes, except the tourer which has 28's and they are still pretty comfortable.
The amount of rubber in contact with the road is the same for a fat tyre or a slim tyre, at the same pressure.
200lb rider
100 pounds per sq inch pressure in tyre = 2 sq inches of contact patch
Fat tyre = low deformation = low energy loss = low rolling resistance
Slim tyre = large deformation = higher energy loss = higher rolling resistance