I thought Apple MLX can do that if you convert your model using it https://mlx-framework.org/

MLX does not support the ANE.

https://github.com/ml-explore/mlx/issues/18

Yes it does.

That’s just an issue with stale and incorrect information. Here are the docs https://opensource.apple.com/projects/mlx/

No, it categorically doesn't. Not just that, it's CPU support is quite lacking (fp32 only). Currently, there are two ways to support the ANE: CoreML and MPSGraph.

Nothing in that documentation says anything about the Apple Neural Engine. MLX runs on the GPU.

None of that uses the ANE.

It does indeed, and is more modern than Core ML.

It is less about conversion and more about extending ANE support for transformer-style models or giving developers more control.

The issue is in targeting specific hardware blocks. When you convert with coremltools, Core ML takes over and doesn't provide fine-grained control - run on GPU, CPU or ANE. Also, ANE isn't really designed with transformers in mind, so most LLM inference defaults to GPU.

Neural Engine is optimized for power efficiency, not performance.

Look for Apple to add matmul acceleration into the GPU instead. Thats how to truly speed up local LLMs.