It could be a little smarter about using the shape of the neighborhood to figure out which character to use, but it seems the lookup table is solely based on brightness, or I'm missing some subtlety in the code. It seems overly complex for what it does, but not for what it could be doing.
All of this is pulled from Sol's TextFX library. The character bitmap actually doesn't line up to Sol's table, so that's why you see weird characters popping up in places. The main goal was just to get image to text working on the GPU. I've typically done it on the CPU. Over time I hope the mess gets cleaned up.
Ah, I see what I was missing: there are four components in the key (r,g,b,a), and each one corresponds to how filled-in each quadrant of the character is. But since each component is 0..16, and the hash table lookup is 256x256, shouldn't packColor be defined as (color.r + 16.0 * color.g + 256.0 * color.b + 4096.0 * color.a)?
a) sample a 4x4 grid of 2x2 neighborhoods
b) quantize the color to 3 bits per channel (so 512 colors)
c) "hash" (not really hashing) the "key" computed from adding up all the pixel neighborhoods and index into a lookup table: http://tllabs.io/asciistreetview/fontHash.png
d) use the green channel from the hash table to index into the font: http://tllabs.io/asciistreetview/8x8.png
It could be a little smarter about using the shape of the neighborhood to figure out which character to use, but it seems the lookup table is solely based on brightness, or I'm missing some subtlety in the code. It seems overly complex for what it does, but not for what it could be doing.