I'm pretty sure this is syntax highlighting. It's a known issue to be slow for large files in Vim because it is synchronous. Try starting Vim with syntax highlighting off:
This makes sense. I recently learned that VSCode is clever enough to automatically disable some features (which includes syntax highlighting among I guess other things) when it detects that the file is too big according to some heuristics (like probably, length of the longest line, or maybe just total size of the file).
So IMO I think vim is being "too dumb" here and should be able to adapt like VSCode does. But, meanwhile, if you want to test under equal conditions, you can disable VSCode's optimization by disabling this setting: