Intrusion and Malware Detection Systems in Environments where an Encryption Overlay is Deployed

Encryption is used within networks with the aim of protecting confidentially and privacy, and as we move towards a more inclusive and connected world, endpoint and application encryption is more likely to be deployed. An implication of this being that malware will be masked from intrusion and malware detection systems rendering them ineffective.

Malware has been traditionally identified by looking for specific fingerprints in files or predicted traits from the traffic it generates. If this traffic is encrypted, it is obscured but as it is machine generated; it is likely to retain some predictable attributes. Recent work has utilised signal analysis techniques, looking for patterns in packet size and timing. Once packet traces were converted into a signal they were broken up using Fourier transforms. This allowed the researchers to produce profiles for a number of IP protocols. In the case of VoIP packets sufficient information was available to allow a particular voice message to be detected from within the trace. This indicates that as malware has been proven to have predictable traits in the clear; that some elements will manifest through a cryptographic overlay.

Efforts have been made to improve the confidentiality of encryption systems using different algorithms and packet padding. The problem these approaches have is that it is very difficult to mask the timing, frequency and entropy of a packet. Other approaches have been taken to address the issue associated with detecting malware within encrypted networks; but they have focused on creating an intercept point. The main issue with this approach is that it is necessary to sub-optimally route traffic through a network, so that the detection device gets to see the information and secondly it requires that the information is decrypted which may not be appropriate or in some cases possible.

My interest is to understand whether it is possible to identify the traits in the traffic that suggest whether malware is present or not, allowing it to be classified as potentially good or bad traffic. This allows the detector to remain agnostic to users and does not attempt to breach confidentially or invade their privacy.