针对网络应用存在的各种安全漏洞，种类繁多的恶意代码严重威胁信息安全，逐渐成为犯罪的工具，严重侵害各种网络服务安全和用户隐私。恶意代码的反分析和反检测技术也在不断发展，恶意攻击的复杂程度不断提高，给现有的恶意代码防范体系带来了巨大挑战。

恶意代码的机理分析和特征提取是弥补现有防范体系不足的重要手段。基于动态污点传播的自动化细粒度代码分析技术是当前恶意代码分析研究领域的热点方向。本文针对恶意代码分析的需要，对动态污点传播的关键方法和技术进行了研究，提出了基于动态污点传播的恶意代码分析技术体系，并设计实现了原型系统，在此基础上提出了网络恶意代码通信协议逆向分析和恶意攻击特征生成两个恶意代码分析应用问题的解决方案。本文的工作主要包括：

1）基于动态污点传播理论和方法，针对当前恶意代码分析面临的问题和挑战，提出了基于动态污点传播的恶意代码分析体系架构，具有不依赖源代码、对分析目标透明度高、分析粒度精细的特点，可以对采用了动态生成代码、代码混淆以及反调试等技术的恶意代码进行有效分析。

2）针对现有动态污点分析系统对目标进程执行过程信息缺乏有效利用而造成的恶意代码行为分析局限，本文提出了一种可回溯的恶意代码行为分析方法，设计了扩展污点传播流图（Extended Taint Propagation Graph， ETPG）的组织原理，实现了对恶意进程指令级和函数级行为的提取和记录。

4）在恶意攻击特征生成方面，本文基于可回溯的动态行为分析提出了一种攻击特征生成方法。通过监控指令级的动态执行流程，提取影响攻击路径的约束条件和与攻击输入直接相关的操作序列，据此重建进程执行上下文环境并添加判定语句，生成图灵机式的攻击特征。通过构造原型系统并进行测试，实验结果表明本方法可以不依赖源代码快速生成简洁高效的攻击特征。

Malwares aiming at security vulnerabilities of various internet applications have become significant threat to information security. These malwares gradually become tools of experienced crime activities, which endanger network services and invade internet users’ privacy seriously. By years of competition with security measures, the anti-analyze and anti-detect technologies used in malwares have evolved dramatically, which is a huge challenge to detect and analyze these malwares for present malware prevention system.

Malware attack mechanism analysis and signature generation are both important directions to improve existing malware prevention system. Automatic fine-grain analysis based on dynamic taint propagation technology has become hotspot in the research field of malicious code analysis. To fulfill the demands of malware analysis, this paper studied the key methodologies and technologies of dynamic taint propagation, then designed and implemented a malware analysis system based on dynamic taint propagation. On this basis, this paper proposed solutions for two typical problems of malware analysis: one is network malware communication protocol reverse engineering, the other is malicious attack signature generation. The main contributions of this paper are as follows:

1) Aiming at difficulties and challenges encountered by current malicious code analysis, this paper proposed a new analysis architecture based on dynamic taint propagation, which provided several key features, such as independence from source code, transparency for malware execution, fine-grain analysis ability, and so on. Malwares equipped with dynamic code generation technologies, code confusing technologies, anti-debug technologies and so on can be analyzed effectively by this architecture.

2) Against the limited ability of malware behavior analysis resulting from poor process execution trace usage in present dynamic taint analysis system, this paper proposed a traceable malware behavior analysis method. We proposed the principles of organizing an Extended Taint Propagation Graph (ETPG) to achieve extracting and recording both instruction level and function level behavior of malicious process.

3) On the aspect of malware communication protocol reverse engineering, recent works have limited accuracy and integrity in identifying protocol fields and are especially weak in understanding fields’ semantics. This paper proposed a behavior based analysis method. We built ETPG, from which we identified how individual protocol element was manipulated by the malicious process to divide the protocol data into different syntax fields. On this basis, we analyzed the API function calls related to each syntax field and induced the semantic information by referring the semantics contained in the functions. We implemented a prototype system and evaluated it with malware samples. The experiment results show that our method can archieve the syntax fields division and semantic extraction accurately and effectively.

4) On the aspect of malicious attack signature generation, this paper proposed a signature generation method based on traceable dynamic behavior analysis. By monitoring the instruction-lever execution of the vulnerable process, we extracted the executing trace and the constrain conditions exactly related to input data directly exploiting the vulnerability. Then, we restored the execution context and supplement the determinant statements to attain an executable Turing machine signature. We implemented a prototype system and evaluated it with different attack samples, which proved that our method was able to generate accurate attack signature fast without program source code.