Security · English
When CTF Meets Malware Analysis: My First Windows Malware Analysis
A reflection from observing malware.exe inside a Windows 7 VM, and how it changed the way I think about CTF, Malware Analysis, Windows Forensics, DFIR, and Blue Team work.
I originally thought that “network attack and defense” would mean a CTF.
Only after I sat in front of a Windows 7 VM and watched two malware.exe samples run did I realize how narrow my thinking had been.
This is not an event recap.
It is more like the first time I realized that a large part of how I had understood Cybersecurity was actually from an attacker’s perspective.
When I saw the words “network attack and defense,” what naturally came to mind were target machines, Flag, Reverse, Web, Linux, and CTF platforms. Maybe there would be a machine to scan, a vulnerability to exploit, and eventually a Flag hidden somewhere in a path.
But this time was different.
This time I was facing a Windows 7 VM and two malware.exe samples used for simulated analysis. The challenge was not whether I could break in, but whether I could understand what they did inside the system.
What left the strongest impression on me was not how difficult the challenge was. It was the first time I clearly felt that:
CTF and real Malware Analysis are two different things.
I Originally Thought It Would Be CTF
Before the activity, when I saw “network attack and defense,” I was a little excited.
The picture in my head was very typical: enter a platform, open a challenge, connect to a target machine, and start looking for vulnerabilities. Maybe it would be a Web challenge where I had to read source code, test payloads, and bypass authentication. Maybe it would be Reverse, where I had to throw the binary into tools and take it apart. Or maybe it would be a Linux target where I had to scan ports, find services, escalate privileges, and finally retrieve the Flag.
That was the security learning style I was more familiar with.
The world of CTF usually has a very clear endpoint. You know the Flag must exist. You just do not know where it is hidden. Many times, once you find the key point, the whole challenge suddenly opens up.
So I originally looked at this “attack and defense” with the same mindset.
Until I realized that what was on the desktop was not a CTF platform, nor a target machine waiting to be attacked, but a Windows 7 VM and two malware.exe samples.
That was when I realized the focus this time was not “getting in,” but “understanding what happened.”
Not Finding a Flag, But Observing Behavior
What I really needed to do this time was observe malware behavior in a Windows environment.
After malware runs, it may create a process, modify the Registry, write files, or create a network connection. The challenge was not asking me to use an exploit to get a shell. It was asking me to judge what it had done from those traces.
The analysis direction could roughly be split into several areas:
| Area | What I Needed to Observe |
|---|---|
| Process | What did it start? Were there suspicious parent-child relationships? |
| Registry | Did it write startup entries or modify system settings? |
| File | Which files did it create, modify, or delete? |
| Network | Where did it connect? Was there suspicious content in the traffic? |
This felt very different from how I usually approach CTF.
CTF feels more like “pushing forward.” I keep asking myself: Where is the entry point? Where is the vulnerability? Where is the Flag?
Malware Analysis feels more like “reconstructing backward.” I have to ask: What did this program just do? Why did it do that? What does this behavior mean in Windows? If this were a real incident, could it be an early sign of infection, persistence, a C2 connection, or data exfiltration?
This way of thinking was new to me.
It does not have the instant feedback of getting a Flag, but it feels closer to real-world security problems.
What Really Stopped Me Was Tool Familiarity
My biggest takeaway this time was not that I completely lacked security knowledge.
Instead, I felt that I did not lose because of concepts. I lost because of tool familiarity.
I knew Wireshark could inspect packets, and I knew Procmon could show process, file, registry, and network activity. I had also heard of Autoruns, Process Explorer, and TCPView, and roughly knew what each of them could do.
But knowing what a tool is for and using that tool under time pressure to find an answer are two different things.
Wireshark
It is not that I had never used Wireshark before.
But when I needed to quickly find suspicious connections, domains, IPs, HTTP requests, or determine whether the malware might have connected outward, I clearly felt that I was not familiar enough.
I knew filters could be used, but I was not yet familiar enough to see a pcap and quickly decide where to start.
Procmon
Procmon affected me even more directly.
It shows a lot of information. So much that if the filters are not set well, you get buried by events. Process creation, file operations, and Registry access are all mixed together. The real difficulty is not “is there data?” but “can I extract meaningful behavior from the data?”
This was also the first time I felt that tools do not automatically give answers just because you open them.
Other Windows Tools
This time I also encountered several tools I should become more familiar with:
- Autoruns: for checking persistence and auto-start locations.
- Process Explorer: for looking at process relationships, DLLs, handles, and more process details.
- TCPView: for watching which external connections processes establish in real time.
Each tool is not especially complicated on its own, but when analyzing Windows Malware, they do not exist separately.
You need to connect the process you see in Process Explorer to the connection shown in TCPView, then go back to Procmon to find which Registry key or file it touched, and finally use Wireshark to inspect packet contents.
I had rarely connected tools together like that before.
So the place where I truly slowed down this time was not that I completely failed to understand the concepts. It was that I was not yet familiar with turning tools into an analysis workflow.
The First Time I Stood in the Defender’s Position
When doing CTF in the past, I naturally stood from the attacker’s perspective.
Find the vulnerability, exploit it, get the Flag.
That is not a bad thing. CTF trained me to break down problems, find clues, understand vulnerabilities, and build many fundamentals.
But this time, I stood much more clearly in the defender’s position for the first time.
If a piece of malware really runs inside Windows, how do I know what it did?
If it opens a connection, how do I know whether that might be C2?
If it writes to the Registry, how do I judge whether that is persistence?
If a user’s computer is actually infected, how do I build a reasonable timeline?
These questions are different from CTF.
The goal of CTF is usually to “solve it.” Malware Analysis, Windows Forensics, and DFIR are more like answering: “What happened?” and “Where is the evidence?”
This gave me a new imagination of Blue Team work.
Blue Team is not easier, and it is not just reading logs. Real defense requires knowing how attackers may act, and also knowing what traces a system leaves after it has been touched.
The attacker only needs to succeed once.
The analyst’s job is to recover the traces left by that one success.
This sentence only started to feel real to me after this experience.
CTF Still Matters, But It Is Not Everything
I did not come away from this experience thinking CTF is unimportant.
On the contrary, I think CTF is a great entry point. It makes people willing to start touching Cybersecurity, and it turns many abstract concepts into something concrete.
But I also started to realize that if my ability stops at CTF, there is a large blank area.
I may know how to exploit a vulnerability, but not know what traces malware leaves after it really enters Windows.
I may know how to get a Flag, but not know how to read the traces of an infected endpoint.
I may know how an attack happens, but not necessarily know how to reconstruct the incident afterward.
That was the biggest shift for me this time.
I am not leaving CTF. I am starting to see that there is another road beyond CTF.
That road is called Malware Analysis, and it also includes Windows Endpoint Forensics, DFIR, Blue Team, and traffic analysis.
Next, I Want to Build My Own Malware Lab
After this experience, I am sure I need an environment where I can slowly practice.
I want to build my own Malware Lab. Not to show off, and not to start by analyzing advanced samples, but to strengthen the fundamentals.
I hope to become familiar with a more complete workflow:
- Run samples inside a safely isolated Windows VM.
- Record a baseline before execution.
- Use Process Explorer and Procmon to observe process, file, and registry behavior.
- Use Autoruns to check persistence.
- Use TCPView and Wireshark to observe network connections and packets.
- Organize observed behavior into simple analysis notes.
I still do not dare to say that I know Malware Analysis.
But at least this experience showed me that I want to move in that direction.
In the past, when I saw “network attack and defense,” the first thing I thought about was how to break in. Now I think one step further: if something really broke in, how would I know what happened?
That question should become an important thread in my security learning for a while.
CTF brought me into Cybersecurity.
And this Windows Malware Analysis experience made me seriously want to understand the world from a defender’s perspective for the first time.