I've thought about this many times. While locally, it may be true that there's a steep hill, gradual ascent, if you do a loop, you start at x feet above sea level...and end at the same x, so ascent and descent overall are the same.
I equate it to my rides...and these include rolling hills, no more than 1 mile long and not particually steep...it makes no difference whether i ride my 'hilly'..(lumpy may be a better description) route...or a flat route. My averages are the same.
No doubt, sharp steep hills will affect your times, but rolling hills dont (for me anyway)
I'm not insisting i'm right, i don't do long steep hills, but my experience tells me sometimes hills make no difference...dependent on the kind of 'hills'.