Flare-On is an annual single player reverse engineering CTF that represents the skills and challenges that the Mandiant FLARE team faces. The 8-12 challenges increase in difficulty and participants have about 6 weeks to complete them all in order to win a prize.

This year’s contest included 11 challenges ranging from Windows binaries to JavaScript, .NET, Python, and even Motorola 68k Macintosh. I finished 185th out of 494 finishers and 5,345 participants which took me just over 2 weeks. I hope to finish higher/faster next year, providing I can allocate more time.

In this blogpost I will walk through my solution of my favorite challenge of this year. It is not necessarily the best, fastest or most complete solution, but I hope to show the difficulties we face during reversing. Also note that I omit all dead ends and rabbit holes I took which makes it seem I am a god, which I’m obviously not.

## The challenge

When we get to #09 we are greeted with the following encouraging message. The description does not hint us to anything except that we may be dealing with some cryptography.

Note that the description is not that diminishing as the description of challenge #05 😭:

FLARE FACT #823: Studies show that C++ Reversers have fewer friends on average than normal people do. That’s why you’re here, reversing this, instead of with them, because they don’t exist.

Anyway, if you want to follow me along, you can download the challenge archive from the Flare-On website. Let’s start.

## Initial Triage

When extracted we are presented with two files, `flareon.exe`

and `SuspiciousFile.txt.Encrypted`

. DIE tells us that we are dealing with a 64 bit PE executable (not packed). Running the binary without any arguments prints the usage: `usage: flareon path [path ...]`

.

Inspecting the encrypted file with a hexeditor reveals some interesting stuff. The file is composed of a 0x49 byte data block - that is of high entropy - concatenated with four hexstrings representing 128 bytes (split by a newline character `0x0a`

).

Looking into the strings of the binary, we find the following.

```
~*~*~* The FLARE-ON Encryptor ~*~*~*
All your files have been encrypted with a powerful combination of
symmetric and asymmetric cryptography. Do not tamper with the encrypted files.
It is of no use and will only risk corrupting your data.
To get your files decrypted send lots of cryptocurrency over Tor.
You'll need to copy and paste us these values to get your key.
```

Let’s hypothesize. The binary is used to encrypt the contents of `SuspiciousFile.txt`

which gets written to `SuspiciousFile.txt.Encrypted`

. If we are able to reverse the binary we can use that knowledge to recover the contents of the unencrypted `SuspiciousFile.txt`

(which highly likely contains the flag). We should expect to see at least one symmetric cipher and one asymmetric cipher.

## Static Analysis - Control Flow

Let’s use the ‘usage’ string to see what the top-level function could be. We check the references to the string and find `sub_3BF0`

. See control flow diagram below. This function first looks up the address of `RtlGenRandom`

function using `LoadLibraryA`

on “advapi32” and then `GetProcAddress`

on “SystemFunction036”. Next, it calls `sub_21D0`

which has a lot of nested calls that do not call WinAPIs. Together with the fact that no content has been loaded hints me that this may be some key schedule.
Next, it loops over all arguments and checks if these strings are appended with “.EncryptMe”. If so, it opens it and opens another file for writing (original filename without “.EncryptMe”, appended with “.Encrypted”) and calls `sub_22A3`

with the fileptrs of these open files. This function presumably does the encryption and reads the `.EncryptMe`

file and writes the `.Encrypted`

file. Next, the loop closes both fileptrs and when there are no more arguments left, it prints how many files it encrypted. Following, if it encrypted files, it calls `sub_239B`

which seems to write `HOW_TO_DECRYPT.txt`

with the aforementioned contents that state how to get the decrypted (or unencrypted) files back.

## Dynamic Analysis - Making assumptions

Now that we know how we can encrypt files, let’s play with the binary to see if we can make any assumptions. We create a file `test.txt.EncryptMe`

with contents “A” and supply it to the `flareon.exe`

binary. We get feedback that the file has been encrypted. We compare `SuspiciousFile.txt.Encrypted`

with `test.txt.Encrypted`

and notice that the first block of 0x49 bytes has been decreased to 1 byte, which is equal to the amount of bytes supplied in the `test.txt.EncryptMe`

file! This hints us that for the symmetric cipher we are probably dealing with a stream cipher instead of a block cipher.
Also, we notice that the first hexstring is the same, the other three differ. For the malware, this may be a hardcoded campaign ID used by the malware author to distinguish between campaigns (the value may also be used as a parameter in the encryption). In fact, the hexstring is also written to the `HOW_TO_DECRYPT.txt`

file which strengthens this assumption. Both the second and third hexstring are also written to this file. The fourth is not.

## Static Analysis - Encryption

Now that we have made some assumptions, let’s dive deeper in `sub_22A3`

which presumably does the encryption (we leave the nested calls for later). At the end of this function we see that the four hexstrings split with 0x0a that we previously saw from analyzing `SuspiciousFile.txt.Encrypted`

are written to the fileptr `.Encrypted`

. At offset 0x2321 we see the hardcoded first hexstring that starts with “9f18776b…”. At offset 0x2340 we see the hexstring `unk_9100`

. From inspecting the xrefs we find that this is probably set by the assumed key schedule function `sub_21D0`

. Same holds for the third hexstring at offset 0x235c. The fourth hexstring at offset 0x2375 is probably the result of `sub_16CC`

.

Moving back to the beginning of the function, we notice that two buffers are populated with respectively 0x20 and 0xc random bytes using `RtlGenRandom`

. Both of these buffers are passed to `sub_20F0`

along with the fileptrs to `.EncryptMe`

and `.Encrypted`

. This is highly likely the symmetric cipher which we assumed to be a stream cipher.

After, we see that `sub_16CC`

is called with an empty buffer `rbp`

, the random buffer(s) used in `sub_20F0`

(highly likely symmetric cipher) and unknown buffers `unk_4020`

and `unk_9100`

which are likely set by `sub_21D0`

. The latter of which is stored to the encrypted file as the second hexstring.

Time for some more hypotheses. The assumed symmetric cipher `sub_20F0`

uses the random buffers of length 0x20 and 0xc as key and possibly IV or nonce. The result is probably written to the `.Encrypted`

file. The function `sub_16CC`

could be the asymmetric cipher that encrypts the key (and IV or nonce) using the buffers `unk_4020`

and `unk_9100`

. The latter and the result of the encryption is written to the `.Encrypted`

file. If so, we may be able to recover the key (and IV or nonce) used in the symmetric encryption.

Let’s dive deeper into the two ciphers.

### Symmetric Cipher

When we try to reverse `sub_20F0`

we only see one instruction. There must be more. In IDA, we press `[SPACE]`

to view the inline assembly and indeed see the rest of the function one instruction further that was wrongly recognized as a function. We undefine both functions by pressing `u`

on their first instructions and press `p`

on the first instruction (offset 0x20f0) to define the entire function.

That is better. We notice two

`QWORDs`

that seem to include ASCII, we change their type to a string and notice that it says “expand 3” and “2-byte k”. A quick search on the internet shows that we may be dealing with Salsa20, which is in fact a stream cipher!
If this is indeed Salsa20, we should easily recognize its core operation, the quarter-round function

`QR(a, b, c, d)`

with the distinctive rotate left (`rol`

) instructions with operands 7, 9, 13, 18. This core operation is executed 8 times each round (20 rounds).
```
b ^= (a + d) <<< 7;
c ^= (b + a) <<< 9;
d ^= (c + b) <<< 13;
a ^= (d + c) <<< 18;
```

We inspect `sub_20F0`

further and see that for every 0x40 bytes of the `.EncryptMe`

file the function calls `sub_1F10`

and xors the results with the input buffer and writes that to the fileptr `.Encrypted`

. Diving into `sub_1F10`

we see a pattern of eight chunks that rotate some data with respectively 16, 12, 8 and 7 bytes. This looks like the eight quarter-rounds but the operands of the rotations do not match. Searching the web for these rotations, we find a variant of Salsa20: Chacha20.

We don’t need to write a proof of concept to confirm that it indeed is Chacha20 because we can just use the binary to decrypt ciphertexts, editing the input buffers dynamically (because the operations are symmetric). The most important thing is that the third and fourth argument to the function are the key and nonce, and that the second argument holds the fileptr to the plaintext (`.EncryptMe`

) and the first argument holds the fileptr to the resulting ciphertext (`.Encrypted`

).

Let’s move to the presumed asymmetric cipher in `sub_16CC`

.

### Asymmetric Cipher

The function `sub_16CC`

itself is not all that large or deep, but the exotic instructions tell me we should probably not reverse this. We’ll dynamically analyze this later. Moreover, the function is called in the previously assumed ‘key schedule function’ `sub_21D0`

which tells us that that function probably does something else.

Asymmetric cryptography is built on the fact that computations in one direction are easy, but really hard the other way around. For example, it could be built on the difficulty of factoring integers (RSA, DSA), or on the discrete logarithm problem (ECC, DSA).

RSA is often used as a text-book example of asymmetric cryptography for its ‘simplicity’. So for now, let’s assume that `sub_16CC`

is RSA because DSA and especially ECC are more complex.

## Dynamic Analysis - Asymmetric Cipher

To test our assumption we dynamically analyze `sub_16CC`

. We run `flareon.exe`

under WinDbg and break at `flareon+231c`

where the call to `sub_16CC`

is. We change the input buffers `rdx`

to all zeroes and leave the others as-is. We notice that the output buffer contains zeroes. Great news because *C = 0 ^{e} mod n = 0* and

*C = M*is undefined (function may return zeroes, we don’t know).

^{e}mod 0

Next run, we edit the buffer of the third argument `r8`

to all zeroes and leave the others as-is. The output now contains a single byte `0x01`

. Even better news because *C = M ^{0} mod n = 1*. This means that highly likely

`rdx`

points to the buffer *M*containing the Chacha20 key and nonce,

`r8`

points to buffer `e`

containing the public exponent and `r9`

points to buffer `n`

containing the modulus.

To confirm that `sub_16CC`

is indeed RSA we test it using this text-book example. We run the debugger again and edit the buffer `rdx`

to 0x02, `r8`

to 0x07 and `r9`

to 0x21 (33). The resulting buffer is 0x1d (29) which is what we expected: *C = 2 ^{7} mod 33 = 29*. We confirmed

`sub_16CC`

is indeed RSA.

## Recovering the plaintext

Now that we know which algorithms are used we can look into how we can recover the key and nonce which were used to encrypt the plaintext contents of `SuspiciousFile.txt`

. Recall that the fourth argument (`n`

, second hexstring) to the RSA function and the resulting buffer (`C`

, fourth hexstring) is written to the `.Encrypted`

file. In order to recover `M`

we have to calculate *M = C ^{d} mod n*.

`d`

, the private exponent is unknown. We could reverse the presumed ‘key schedule’ function `sub_21D0`

(which obviously is not a key schedule function anymore 😝), or we can try to attack the public exponent in the hopes of finding the private exponent. One of the most well known attacks against the public exponent of RSA is Wiener’s Attack. Let’s try that.### Wiener’s attack

We run the debugger again, break at `flareon+231c`

where the call to `sub_16CC`

is and copy the third argument buffer (`e`

) and fourth argument buffer (`n`

) for the attack. We find a nice Github repo by Nao Yonashiro that does the job for us. To our surprise they used 0x10001 as a weak private exponent! We can use this to decrypt the fourth hexstring in the `.Encrypted`

file `C`

which will give us the key and nonce used in the Chacha20 encryption.

### Decryption

Now it is time to combine all the work.

#### RSA

The RSA decryption is very straight forward now: *M = C ^{d} mod n*. We copy the fourth hexstring of the

`SuspiciousFile.txt.Encrypted`

and use it as the ciphertext `C`

. We use 0x10001 as the private exponent `d`

and copy the second hexstring to use as the modulo `n`

. We decrypt using the native Python function `pow(C, d, n)`

and see that the resulting plaintext `M`

has the format of the Chacha20 key and nonce (in reverse order).

#### Chacha20

Because Chacha20 is symmetric we’ll use the `flareon.exe`

binary to do the decryption for us. We copy the `SuspiciousFile.txt.Encrypted`

to `result.txt.EncryptMe`

and remove all bytes but the first 0x49. We run the debugger and break at `flareon+2303`

where the call to the Chacha20 function is. We modify the third argument buffer `r8`

to the recovered key and the fourth argument buffer `r9`

to the recovered nonce. We continue execution and should see the result in `result.txt.Encrypted`

.

## Solution

We inspect `result.txt.Encrypted`

and indeed, we succeeded! 🥳🎉