I wish I could do a full week but I think I figured out part of my problem. I was riding the day before yesterday and for only the third time I saw another rider. This man came up beside me and told me he saw me from the bridge I had just passed under. We rode and chatted for about 5 miles discussing routes and he told me he has been riding for about 15 years and he was 69 years old. He looked much younger and we were doing some small hill climbing and I could tell he was a stronger rider than I am.
While discussing routes and discussing some hills in the area, he told me, "I don't like to ride to Pittsboro", the hills going there are really long climbs. This made me smile because this is the small town our shop is near and there are climbs up from two different rivers. This is my shortest route to work. I don't know the elevation change but do know that there are two climbs of about a mile each on the last 5 miles of the 11 mile trip to work.
At least now I know that I am not the only one this route is rough on.

Having a more fit rider mention it at least helps my ego a little, I was afraid it was just a mental thing with me. I will keep trying until I feel I can do the trip and do my work too, just maybe not every day both ways until then.
Congrats on your first week!