I would also love to see ports grow visually - for example, if you build a defensive fort, a new building would be added to your port - visible when sailing around that island. In the same way, I would love to see buildings (including houses and other neutral, non-defensive buildings) destroyed in raids by stray cannonballs, and ports grow as new buildings are constructed in times of peace.