Go 75 miles. In my experience you will feel better than you expected, get caught up in the excitement of it all and wish you committed to stretch yourself. Go big, you won't regret it. Just go at your own pace. As others have said, it's not a race. You will settle into a similar speed bunch at some point, have some chats etc.
Yes there will be idiots at the front pushing on. You don't have to be one to do a sportive. Also, in all honesty the riding I've seen is a lot better than some give credit to. It's not that bad. A couple of idiots shouldn't tar everyone with the same brush.
Go on, push yourself.