> What other API would you suggest?

MLC LLM?

I think the binary it compiles down to (Probably the Vulkan and Metal ones for yall) is seperate from the weights, so you could ship a bunch in one file.