It's not how much wind there is per se, but what direction it's coming from.
The first time I did it, it was from the North/NE, which meant it was a full on headwind for that four and a bit mile long straight between Ramsey Heights and Pondersbridge, with nowhere to hide. It basically brought everyone to a screeching halt (especially me) and trying to push through it completely knackered me for the rest of the ride.
Second time riding two years later and we had a crosswind on that section - I was over 3mph faster on average over that section this time (14.6 vs 17.8), despite having gone into the ride less fit than in the first year. And I also wasn't pushing hard either while doing it, which meant I had more energy for the rest of the course - even the bits where I did have a headwind later weren't anywhere near as bad.
Luckily this year the wind forecast seems to be either from the SW or West/SW.