Think about it this way, with each 100 miler, you are accumulating fatigue, if you do another before you are fully recovered, you will of course need to take longer and longer to recover since you you will recovering from the cumulative fatigue, i.e. the fatigue from the latest 100 plus the residual fatigue from the previous one(s).
If you are going to do this regularly, I would suggest you start taking in some protein & carbohydrate mix toward the end of the ride, then have a recovery shake or equivalent real food (I'd go for real food personally, if possible) within 20 mins of stopping and then continue eating for a few hours afterwards, little and regular. Next day, do 20-30 mins very easy. BTW, afterwards, you say you used Whey protein, you should also be consuming carbs immediately afterwards.
I would say it is not normal to not be able to ride for 3-4 days though, I would ride again the next day, nice and easy and be back training as normal within 2-3 days, although power values may be lower for a while. After having raced (which really is quite different to just riding, even a tough ride) 2 x 100 mile TT's within a 4 week period (and totally destroying myself in one of them) I wasn't right for about a month or more and have only just managed to start recovering my form in the last 2 weeks.