Have been there, people at work saying that its not safe for me to cycle to & from work. So on that side I can sympathise. Pressure can be applied to make it hard if not impossible to carry on cycling. But what not one of them can do, at work, is dictate to you your choice of transport to & from work. Unless its been agreed upon prior to starting there.
If your boss feels that strongly about it, let him get the second opinion for you if you still feel you need one. Invite those that say its not safe/to fat to cycle to join you one day.
I ended up, following being hit sideways by a car, being told that it wasn't safe due to this & the increase in distance that would be coming when they relocated. Having already cycled there, to prove to them that it could be done.Also to get an idea of how long it would take.
The one thing all had in common was not one of them cycled.