Perhaps to keep this game moving when more than half the people in the game vote, you start a countdown for how long before you stop votes entirely (obviously reminding everyone who hasn't voted to vote). If someone misses this deadline then they don't vote and you carry on. Then if someone consistently misses votes without a valid reason you kick them.