This will make it easier to run automated tests in the Android emulator in CI using regular runners. It was a pain dealing with bare-metal instances just for that.
Yes, games are a common case where your repo can be very large but building your code only requires access to a small subset of it.
For example, the League of Legends source repo is millions of files and hundreds of GB in size, because it includes things like game assets, vendored compiler toolchains for all of our target platforms, etc. But to compile the game code, only about 15,000 files and 600MB of data are needed from the repo.
That means 99% of the repo is not needed at all for building the code, and that is why we are seeing a lot of success using VFS-based tech like the one described in this blog. In this case, we built our own virtual filesystem for source code based on our existing content-defined patching tech (which we wrote about a while ago [1]). It's similar to Meta's EdenFS in that we built it on top of the ProjFS API on Windows and NFSv3 on macOS and Linux. We can mount a view into the multimillion-file repo in 3 seconds, and file data (which is compressed and deduplicated and served through a CDN) is downloaded transparently when a process requests it. We use a normal caching build system to actually run the build, in our case FASTBuild.
I recently timed it, and I can go from having nothing at all on disk to having locally built versions of the League of Legends game client and server in 20 seconds on a 32-core machine. This is with 100% cache hits, similar to the build timings mentioned in the article.
I'm the author of the GitHub issue that the blog links to, and I'd like to thank Stefan for quickly acknowledging the problem and addressing the issue! I try to keep one of our internal applications up to date with the latest libcurl version within a day or two of a release, so we sometimes hit fresh problems while running our battery of tests.
Ironically, our application has also struggled with blocking DNS resolution in the past, so I appreciate the discussion here. In case anyone is interested, here is a quick reference of the different asynchronous DNS resolution methods that you can use in a native code application for some of the most popular platforms:
- Windows/Xbox: GetAddrInfoExW / GetAddrInfoExCancel
- macOS/iOS: CFHostStartInfoResolution / CFHostCancelInfoResolution
- Linux (glibc): getaddrinfo_a / gai_cancel
- Android: android.net.DnsResolver.query (requires the use of JNI)
- PS5: proprietary DNS resolver API
If we are doing a survey. I found a few more. It feels like we need to get everyone together in a room and say "we will let you out when you decide on a standard non-blocking address lookup" What a mess.
- OpenBSD: getaddrinfo_async / asr_abort https://man.openbsd.org/asr_run
- FreeBSD: dnsres_getaddrinfo / found no way to abort https://man.freebsd.org/cgi/man.cgi?query=dnsres
Unfortunately, c-ares is not problem-free on all platforms.
On iOS, its use triggers a local network access popup (it tries to reach your DNS server, which is often on your LAN). If a user denies acess, your app will simply not work.
On Android, it's not compatible with some VPN apps. Those apps are to blame, but your users are going to blame you not them.
So, at my previous company we ended up building libcurl with a threaded DNS resolver on both iOS and Android.
I appreciate seeing this article that I wrote 4 years ago resurface :) Happy to answer any questions about it or the tech behind it.
We also use a SQLite database, and in a manner similar to the OP’s article. We use it to track which content-defined chunks are at which offsets in which files on disk. We deduplicate those chunks on the CDN so you have to download less data when updating the game, but on disk we need to recreate the files as originally uploaded because that’s how games load them.
We’ve expanded the use of this tech quite a bit since the article. For example, my team has started using the patcher internally to manage our 2-million-file vendored code and binaries repo, as a replacement for Git LFS of sorts. Cloning the repo from scratch became 10 times faster, and so did switching branches, especially for remote devs, since the files are served through CloudFront.
Some of the more interesting work we’ve done recently has been optimizing the core content-defined chunking algorithm. The GearHash rolling hash algorithm from FastCDC is decently fast, but we were able to improve chunking speeds by 5x with vectorized implementations and using AES-NI instructions in a clever way to avoid the GearHash table lookups. It can do 11GB/sec per core now using AVX-512 instructions.
Another thing we did recently was build a tool for repacking Unreal Engine Pak files so game assets maintain a consistent layout across versions, similar to what garaetjjte mentioned in a comment above about UnityFS files. This reduced the disk I/O needed to update VALORANT by over 90% and cut update times by half for players. The combination of content-defined chunking with tooling to make game data files friendlier to differential patching can make a huge difference.
I like zsync quite a bit. The chunking patcher we wrote for League of Legends is similar to zsync in some key ways.
One of the things we did to improve the efficiency of the core rolling hash based chunking logic was to add file type aware chunking. Basically, we teach the patcher about the different archive formats that our games use (say, Unreal Engine PAKs) and the algorithm ensures that chunks stop at the file boundaries within the archive, so a chunk doesn’t straddle two unrelated entries. That way when files move around in the archive we can more reliably match their chunks across versions. I think this helped improve download sizes by 5-10% in our case.
Working on Monkey Island Special Edition in 2009 while at LucasArts is one of the highlights of my career. I have really fond memories of my time there on the team (Team 3!). I wish I had worked on the original but that was way before my time.
The game is actually running the full original code alongside the new Special Edition stuff. We didn’t want to disturb the original codebase, so we ran it basically unchanged on another thread and the new game loop would surgically peek and poke some state every frame (animations, etc) to get synchronized. That’s how we essentially layered all the new art and sound on top of the original game. There was a lot of reverse engineering of data and asset formats that we had to do because all the original authors had since left, perhaps with the exception of EJ.
The F10 hotkey to transition between the old and the new art was actually a debugging feature that our lead rendering engineer wrote during development, but everyone agreed that it was too good not to ship it as an actual feature of the Special Edition.
For those who didn't experience it the first time or perhaps would like to do so again, I feel obligated to mention your version is on sale for $3.49 at Good Old Games aka gog.com. No affiliation myself other than a customer and someone a bit in awe of what you guys wrought.
