You asked "would it be worth doing?" The Tiagra 4700 matches the quality of the Giant Contend and (in line with
@Drago 's response) an upgrade to Ultegra would be merely for the sake of it, and whatever fun you had doing the work. Save the Ultegra groupset for a higher quality f&f or if you've decided anyway, don't waste your time asking this forum. The value of the 'upgraded' bike will not rise by as much as the groupset cost difference (maybe about half). Maybe your driver is the need for the Tiagra on another bike - only you know/can have a complete view.