Tyre design, construction, materials and inflation pressure have more effect on rolling resistance than cross-section. Comfort though is entirely dependent on volume. A good quality 28mm tyre might have better rolling characteristics than a cheap 19mm but a larger tyre will always be more comfy than a skinny tyre.
Unless you are racing I would err on the side of comfort, for competition the rolling resistance advantage of a few microseconds makes sense, in the real world of potholes and greasy streets a bigger tyre provides better rim protection and a larger contact area.
If you want to go faster work on the aerodynamics of your riding position, at 20mph 80% of your effort is being used just to overcome wind resistance.