Hmmmm, I ticked the box last night, and now it does get rid of those stubborn notifications that I'd otherwise have to cancel manually. But it's still not a complete fix, as I now have to search through all of the notifications to look for post replies.
I shall un-tick the box again and see what that does.