You could easily do that:
1) A national or clan flag at the sternpost
2) A commanders pennant at the main topmast indicating rank (capt, commodore, admiral, etc). Historically this is mostly accurate as admirals flagships flew a coloured pennant here indicating his rank.
3) A smaller pennant at the foretopmast with some kind of symbols on like lions, fleur de lys, skulls, etc with the number of them indicating rank.
4) Extra flags at the mizzentopmast for that, or smaller coloured flag at the flaghoist lines betwen the masts.
You could implement a commun ications method by flaghoists if the player had a "spyglass view" which fixed on the flag hoist. A simple "read signals from ship X" command would fix the zoomed in player's view on the signal flags for, say, 10 or 15 seconds and then the message those flags are sending is shown in a text window bottom of screen. It would certainly be something unique in a naval game.
Unfortunately clans and such will just use Teamspeak and talk to each other, there's no way round this I don't think so any historical in-game comms system would be purely for plaayer immersion. Unless you can force VOIP to not work or something.
Of course in MP fleet battles you could have an agreement not to speak over TS and all fleet signals be sent by flags. That would be truely awesome.
I'm also thinking flag hoists from forts and shore stations at harbours might be useful too. They could at least show which nation/clan owned the port and what the status of the senior player present was. Not having port or ship ownership marked in any way in the campaign map would oblige a clan or guild to send a small patrol vessel to a place to report back who owned it and what its strength was.