My brother started cycling recently on a hybrid, and he is slower than i am. He is also younger than me, so i get a lot of verbal abuse when i reach the peaks before he does, i get a longer breather. We have always been competitive, and i'm sure he will try to beat me one day on a roadie.
All of that means nothing to me, just great being out together,talking about stuff (mum,dad,respective wives,kids,football,hotties in cars), is something i cherish now, it is something i really look forward to ..... You are blessed with a great BIL and hubby , stop feeling inferior or a spoilsport for not having the fitness, it will come. ENJOY!