A major reason for the Bill of Rights is to make sure the majority can't take away the rights of the minority. If the only penalty is voting, that doesn't work.