Find a video of this Blackhat talk from 2005. Pretty sure it's in the iTunes store in the podcast section. The video covers pretty much everything you need to know about the theory of such systems.
As far as actual implementation, I'm not aware of a system that does this. Although honestly some researcher has probably implemented such a system in a paper somewhere. There's probably a commercial network IDS that does it too. I haven't been doing this long enough to be an expert in the field, so I can't confidently say what the true state of the art is in behavioral detection. I'm pretty sure there are no open source projects though if that's what you're asking.
If you're thinking about implementing your own, I'd say start at the network level and write a simple program that sets an alarm off when a specific host starts responding to HTTP requests. If you want to get complex, use Wireshark to log all your network traffic for a month. You can then use Bayesian learning to determine whether new traffic is out of the ordinary. SSH traffic is problematic, so if you need to allow such traffic you'll have some additional challenges to overcome.
Starting an open source project related to this is on my TODO list, but I'm focusing on demonstrating the
hopelessness of signature based detection first.
Good luck. And thanks for the links, I'll be looking into it. You mind a PM if it get's interesting (just to bounce ideas off someone)? I'm looking into machine learning stuff myself and can imagine some interesting integrations with this subject.
I'm presenting a metamorphic engine at a hacking conference in April (http://thotcon.org/). It'll be completely open source. Initial tests against modern anti-virus look very good. ;)
http://www.blackhat.com/html/bh-usa-05/bh-usa-05-speakers.ht...
As far as actual implementation, I'm not aware of a system that does this. Although honestly some researcher has probably implemented such a system in a paper somewhere. There's probably a commercial network IDS that does it too. I haven't been doing this long enough to be an expert in the field, so I can't confidently say what the true state of the art is in behavioral detection. I'm pretty sure there are no open source projects though if that's what you're asking.
If you're thinking about implementing your own, I'd say start at the network level and write a simple program that sets an alarm off when a specific host starts responding to HTTP requests. If you want to get complex, use Wireshark to log all your network traffic for a month. You can then use Bayesian learning to determine whether new traffic is out of the ordinary. SSH traffic is problematic, so if you need to allow such traffic you'll have some additional challenges to overcome.
Starting an open source project related to this is on my TODO list, but I'm focusing on demonstrating the hopelessness of signature based detection first.