On flattish terrain at commuting or leisurely speeds, I find my rim braked bike to be great and no real difference to my disc braked bike.
For higher speeds and especially on fast descents, it is disc all the way for me. They do stop my bike faster, despite my rim bike being set up very well. I can brake later into corners and my descending as a result has proven to be much faster. I'm more confident too - with rim I'm far more wary I've discovered (I didn't realise it until training and racing with disc) and not as trusting.
On my MTB it was a no-brainer for racing years ago already but I am a late convert to disc for road having been perfectly happy with my rim braked road bike. Then I bought a disc bike and after a few weeks over all types of lumpy terrain in different conditions I was totally converted and now I'll not buy rim again. I've just bought my second disc road bike and have a gravel disc bike arriving next week.
I've found I can service and swop out pads just as quickly as with my rim bike too; it really is very simple. For me, there are no real downsides and they are either equal to rim or better depending upon use.
All that said, I still have my rim bike, I can stop just fine on it and I will be keeping it and riding it. I'll just not race it anymore because for me, discs just work better overall.