Muskets weren't much use away from the battlefield.
It took close to a minute to reload, a minimum of 30 seconds if you were very good and in eth rhythm of loading and firing - I use one regularly with the Sealed Knot, and can just about load in 30 seconds, but that is without the actual ball (obviously we only fire blanks).
And then if it is a matchlock, you would not normally be carrying around lit match, so you have to get that lit before you can fire.
A musket is also quite heavy, and doesn't normally have any means of carrying other than in the hand, so you wouldn't usually even have it with you around town.
Whereas of course the sword only takes a second or two to draw.