I'm of the opinion that "if you've bought it, use it" so don't tend to keep stuff for best if it can be ridden. But, although the 404s are better than some in fresher winds, they're not immune and can take a bit of handling. They could certainly be your main set of wheels on a best bike, but I'd keep a less aero backup handy for blowy days.
I'd heard about the Zipp hub recall and checked it out before I bought. Also, I was on the Spec US website yesterday checking out some helmet certifications (boo, but now a requirement unfortunately) and was surprised at the number of recall notices. Although there are clearly some feck-ups, I think it's as much a case of covering their bums in the extra-litigious American market.