You can’t get banned just for downvoting. Nobody can see someone else’s voting history. You buried the lead, you were banned for your comments not for your voting activity.