Academia and higher education have long-established ideals, institutions receive considerations from society in light of that, and they are also supposed to be regulated in some ways (e.g., accreditation).
I think a better defense than the libertarian "don't tell me what to do with my property" would've taken the angle that "university X isn't actually becoming a factory for pumping out techbro libertarian scourges of society".
