If people don't like me they don't have to be in my company
Have you been listening in on my conversations and quoting me?
It sounds like something I said to my wife not long ago when she said people would think I was being funny for not going to a party.
"If that's what they think, and they've got nothing better to gossip about, then it says more about them than it does about me," was my reply.