Sometimes you'll speak to someone that is afraid of you. Not often but occasionally.
They're not afraid as in scared, but they are too aware of you being foreign. Sometimes you can see it in their eyes but not always. The rabbit in the headlights kind of wide eyed stare. They don't know what to say to you, they'll go silent for fear of you not understanding them. They've become so aware of the possibility of miscommunication, the possibility that you've said something wrong that they'll doubt their own natural ability to understand. The situation freaks them out.
This might give you the impression that you've said something wrong, or incomprehensible (and obviously sometimes that is the case!) but it's not always. They are just too aware of your foreignness, their brains freeze and they become overly focused on the words.
Most of understanding is not in the words. The majority of it comes from other factors, context being perhaps the biggest. When people go into this state of brain freeze, they're shutting out those other factors they'd naturally use in understanding (even in the same language) So panic not and don't doubt yourself. And, no, don't just repeat yourself louder!
I've even had one overly dramatic experience where the young shop assistant who, when I asked my simple intro question, literally exclaimed 'ah, anglais!' and ran off in search of a colleague! The exchange then progressed quite happily.
The best people to talk to, and happily most people fall into this category, talk as they normally would - albeit a little more slowly. One of my club mates in the cycling club would happily chat to me as we rode along, quite normally, for ages whilst others would steadfastly stay away from me. I didn't take it personally, they were just didn't know what to say.