The need for affirmative action exists regardless of what you do to K-12 schools as long as college admissions count standardized test schools and extracurricular activities.
If there's a baseline level of good education and whatnot, it stands to reason that there would be roughly proportionate demographics among the applicants. Get rid of legacies, and that seems perfectly fair to me.