I appreciate you're not training for anything so you might feel this is not relevant. In my experience most coaches or coaching programmes will include rest days. The body does require time to rebuild itself and rest is as important as exercise in this instance.
Im far from a dedicated trainer. I cycle and follow a gym routine. My week is always 5 days exercise and two days rest. The intensity of the gym work is similar for each session but rides vary. If we've had a long, intense ride the next will be gentler.
Your rides appear to be fairly gentle so I doubt you're overdoing things. Listen to your body, if you feel a bit tired and achey give it a couple of days off.