The problem with a general-purpose fuzzer is that the H.264 format is complex - you'd end up with a lot of syntactically-incorrect files (which decoders would easily reject) whereas H26Forge is a specialized fuzzer that ends up with syntactically-correct but semantically-incorrect files, and that's how it finds actual vulns before the heat death of the universe.
Re Rust: the problem here is hardware-acceleration, as far as I can tell. Even if we had a pure Rust H.264 decoder, you'd probably still want to use whatever your hardware has to use overall fewer resources. The drivers might be the place to look, and there's some progress on that front in Android for example, but as things stand fuzzing like that is extremely valuable.
Isn't the whole claim to fame for AFL that it largely mitigates or avoids that problem by tracking branch coverage so it doesn't waste time permuting the input in ways that don't change the program behavior meaningfully?
AFL is great for most file formats (e.g. ELF), but probably not suitable for video formats like H.264, which uses complex encoding even for things as simple as frame width/height in the header (see things like ue(v) and CAVLC).
It will take ages for AFL to generate a valid H.264 NALU that isn't rejected outright.
How do you know *ahead of time* which mutation of input will result in new path in code? You don’t. What you can do deduplicate possible inputs for mutation based on the branches taken/path.
AFL works by trying to modify bits and seeing what branches change direction. Arithmetic coding means this relationship desyncs almost instantly, so it’s hard to mutate into interesting test cases.
Yes, fuzzing does work on decoders. I can't remember how deep AFL managed to get but I do remember a flurry of crash bugs against our decoder when somebody first tried it
