Reverse engineering is the process of deconstructing software or malware to understand its structure, functionality, and the steps it went through during development. This technique is widely used in cybersecurity, antivirus research, software development, and vulnerability assessments.
Applications of Reverse Engineering
- Cybersecurity Research: Used by antivirus companies to detect malware.
- Software Development: Helps developers analyze and improve software performance.
- Vulnerability Analysis: Identifies security flaws in applications.
- Intellectual Property Violations: Some companies use reverse engineering to copy competitors' software.
Despite its importance, demand for reverse engineering professionals has declined due to the industry's shift toward web-based applications instead of desktop software.
Understanding PE (Portable Executable) Format
The PE format is the structure used by Windows executables (.exe files).
HXD is a free tool that displays the raw data inside a file.
The first four bytes of an executable file contain a signature that identifies its format.
Linux does not rely on file extensions (.exe, .dll), but rather on file headers to determine how a file should be executed.
Offset Addresses:
The left column in hex editors (such as HXD) displays offset addresses, showing the position of each byte relative to the file's beginning.
Binwalk: File Signature Analysis Tool
Binwalk identifies files within compressed or embedded data based on their header and trailer signatures.
Example: Word documents function similarly to .zip files as they can store compressed objects inside them.
Extracting Data from Files
To extract embedded files using Binwalk:
Hiding files within other files
To merge two files into one:
However, this method exposes the embedded file's signature, making it detectable.
Hiding Data in Files: Alternate Data Streams (ADS)
Windows allows hiding files inside other files without modifying the visible content using NTFS Alternate Data Streams (ADS):
This injects secret.txt into cover.jpg without visibly altering cover.jpg.
To retrieve the hidden data:
However, the hidden file can be detected by analyzing file size discrepancies (difference between Size and Size on Disk in file properties).
Online tools and steganography decoders can also detect hidden messages in images and sound waves.
Static vs. Dynamic Malware Analysis
Static Analysis: Examines a file's contents without executing it.
Dynamic Analysis: Runs the file in an isolated environment to observe behavior.
Identifying Malicious Indicators
Malware files are analyzed for patterns known as YARA Rules:
- These rules identify unique, unchangeable parts of malware.
- Even if malware is obfuscated, core components remain the same, triggering YARA alerts.
Antivirus software works through:
- Hash Comparisons
- Signature Matching
- Running the File in a Sandbox (Isolated Execution)
Common Malware Analysis Tools
- Notepad++ (Text Analysis)
- 7-Zip (Extracting Archives)
- Windows Sysinternals (Process and Memory Inspection)
- Resource Hacker (Extracting UI Components)
To download real-world malware samples for research:
Malware Investigation Process
- Upload to VirusTotal – Check if the malware is already known.
- Scan with an Antivirus – Identify additional indicators.
- Analyze Hashes – Compare MD5, SHA-1, and SHA-256 hashes to detect changes:
If the hash changes over time, the malware has been modified.
- Extract Strings from the File – Reveals hidden text, API calls, and URLs.
For example, an unexpected IP address in a non-networked application is a red flag.
Analyzing Executables
PE Format Analysis
Windows executables (.exe) begin with the MZ Header, marking them as portable executables.
The execution process:
- Checks File Extension – Determines the appropriate software to run it.
- Parses File Header – Identifies required permissions.
- WinLoader Loads the File – Creates a Process ID (PID).
- Links to DLLs – Many applications rely on precompiled DLLs for functionality.
Checking Dependencies
Use Dependency Walker to analyze which DLLs an executable requires.
Suspicious DLLs:
- kernel32.dll – Manages memory read/write operations.
- user32.dll – Handles user interface interactions.
- advapi32.dll – Manages Windows registry and system services.
- ws2_32.dll – Manages network connections.
Detecting Malware via PE Headers
Timestamps in PE Headers can be manipulated to bypass antivirus.
Suspicious indicators include unknown DLL imports or future timestamps.
Analyze executable resources using Resource Hacker to inspect icons, dialogs, and metadata.
Code Decompilation and Debugging
Decompiling .NET, Java, and Python Executables
- .NET executables can be decompiled into readable code.
- Java .class files and Python .pyc files can also be reverse-engineered.
Registry Analysis
Windows Registry manages system configurations and software settings.
Regshot takes a snapshot before and after execution to detect registry modifications.
Advanced Malware Detection Techniques
CaptureBAT logs system events and process interactions:
Running malware in a sandbox for 24 hours helps identify delayed execution techniques.
Some malware checks for virtualized environments (e.g., VMware, Sandboxie) and refuses to run.
Others verify network connectivity and delete themselves if no connection is found.
To bypass these detections, researchers set up fake DNS servers and emulate real network traffic.
← Back to Articles