@HarryTheDog I have someone similar on my commute. I usually kick his backside in the morning riding fixed, and him on a carbon Pinarello, but in the evening, he has a tactic to stay behind me, down a descent, and up a climb - so on fixed I have to power it up, but I am usually spent at the top before a set of lights.. He then rags it off up through the village. I can't respond as I need to wind the fixed up again. We are fortunately on 'oright mate' terms now he knows my nemisis, wasn't happy before that.
I have a couple of other fellas that are quick, and it's a damn battle staying in contact/passing them on fixed. Bet they all say, 'oh fark it's that git on fixed with bloody panniers !!! '