You should be able to pull the leg warmers all the way up until they rub against the pad of your shorts without issues.
That'd be too high for me. Ime, and seemingly that of the OP too, that'd cause irritation.
I go for a slightly lower position; high enough to stay put yet low enough to not irritate. I also make sure there's not any gather behind the knees and that zips are not sitting on my ankles. I try to distribute any 'slack' evenly but it usually finds its own home anyway.