I'm 6'1" and have been various weights, when I played rugby I was rarely below 13.5 stone and normally between 14 and 14.5. When I stopped playing rugby I settled down at around 11.5 stone, I trained and swam a lot then though. My weight then sort of crept up over the years, marriage, children and work limited my training time and I got lazy/idle. For a long time I was between 14 and 15 stone without doing any exercise then I started to work out again and things were going in the right direction. Cue elbow break, ballsed up recovery due to my own stupidity/impatience, followed by long wait for surgery and hello couch potato land.
My main exercises before the break were golf and situps/chinups, doesn't sound a lot but it kept the demons at bay. I then rose to 18.5 stone before taking up cycling in desperation, that worked fantastically and I again reached 14 stone and still heading downwards. But a job change which removed my long commute and some further elbow problems set me back a bit, crept back up over 16 stone. Started to get back in to it and then had an off, entirely my own fault, and cracked a bone in hand/wrist(same arm as the elbow, both of which had been broken in childhood as well), hey presto I went back to 18.5 stone. Just about 17 stone now and still heading in the right direction, but kicking myself for all the backsliding...no excuses, greedy/depressive bastard that loves food.
But I've had it all from friends/relatives...oh you're too fat, oh you've lost too much, oh don't lose anymore, oh you don't want to over do it. Ignore them all, listen to what your body tells you, how you feel and to any professional medical advice. At 14 stone and playing rugby I was fine, at 14 stone and not playing I was overweight. When you first get down to a new weight your body needs time to settle and others need time to get used to your appearance. We don't lose weight in a uniform manner and the settling period gets longer as you get older.
I'm aiming for 12 stone and I don't expect to look skinny at that weight. Have a look at the heights and weights of boxers, cage fighters and the like, people that you wouldn't class as puny/skinny. Your weight now would put you in that sort of bracket and you'd be considered heavyish for endurance work.