I think that can be solved if a majority of early users use the voting system in the way it was intended. Say there's a cluster of 75% "good" initial users who vote similarly on quality. Any new users that vote similarly to them on quality would also be classified as "good" and be given more weight.
Even if the majority of new users don't use the voting system correctly the system could be weighted more heavily to the group of existing and new users who agree on what quality is, even if it ends up being a minority of users.
Actually, I think even if a majority of users don't vote in the way that was intended from the beginning you might still be able to handle it because you can throw away users whose "quality" and "agreement" votes are most correlated, which I think is the most likely way users would deviate from the intended voting system.
Now, if you had a majority of users who all voted the same but were not correlated with "agreement" (e.x. "vote according to the day of the week" or something), or if only very few users were voting correctly, then it might be difficult to distinguish, but that seems unlikely.
Even if the majority of new users don't use the voting system correctly the system could be weighted more heavily to the group of existing and new users who agree on what quality is, even if it ends up being a minority of users.