You'll be fine with 32c slicks on that, as long as you ride sensibly / somewhat cautiously.
I wouldn't bother changing tyres personally, 32c is a good all-rounder, and if you get on with them for the rest of the riding you do it's not worth the faff changing them.
To ride a pump track "properly" you really need to be able to slam your saddle way down low, which pretty much requires a MTB type frame with low stand over height. A road oriented hybrid frame won't give you enough room to pump down with your hips into the berms and rollers.
But to just tootle around the track, you can ride it on anything. I've done the local pump track on a 1950s sit up and beg bike, just for giggles (the local kids loved it lol).
If you get into MTB riding or want to get airborne on the pump track, you'd be much better off buying a cheap second hand MTB than swapping tyres on your existing bike.