Excerpt from a letter from Lee to his wife dated April 19, "...next fall there will be a great change in public opinion at the North. The Republicans will be destroyed & I think the friends of peace will become so strong that the next administration will go in on that basis."
I think both of the arguments are valid, but you're both getting away from the original discussion about the removal VPs.
Which IMO wouldn't be very beneficial. I'm sure the VPs act more than just objectives but in the tech side of the gameplay they act as triggers for the choices for the next phase of battle.
Also if you notice, the VPs are all places of strategic importance. One can still win a battle with recieving more casualties than your opponent as long as you can drive them from the field or the objective.