Flare-On 9: 09_encryptor

Flare-On is an annual single player reverse engineering CTF that represents the skills and challenges that the Mandiant FLARE team faces. The 8-12 challenges increase in difficulty and participants have about 6 weeks to complete them all in order to win a prize.

This year’s contest included 11 challenges ranging from Windows binaries to JavaScript, .NET, Python, and even Motorola 68k Macintosh. I finished 185th out of 494 finishers and 5,345 participants which took me just over 2 weeks. I hope to finish higher/faster next year, providing I can allocate more time.

In this blogpost I will walk through my solution of my favorite challenge of this year. It is not necessarily the best, fastest or most complete solution, but I hope to show the difficulties we face during reversing. Also note that I omit all dead ends and rabbit holes I took which makes it seem I am a god, which I’m obviously not.

The challenge

When we get to #09 we are greeted with the following encouraging message. The description does not hint us to anything except that we may be dealing with some cryptography.

FLARE FACT #823: Studies show that C++ Reversers have fewer friends on average than normal people do. That’s why you’re here, reversing this, instead of with them, because they don’t exist.

Anyway, if you want to follow me along, you can download the challenge archive from the Flare-On website. Let’s start.

Initial Triage

When extracted we are presented with two files, flareon.exe and SuspiciousFile.txt.Encrypted. DIE tells us that we are dealing with a 64 bit PE executable (not packed). Running the binary without any arguments prints the usage: usage: flareon path [path ...].

Inspecting the encrypted file with a hexeditor reveals some interesting stuff. The file is composed of a 0x49 byte data block - that is of high entropy - concatenated with four hexstrings representing 128 bytes (split by a newline character 0x0a).

Looking into the strings of the binary, we find the following.

~*~*~* The FLARE-ON Encryptor ~*~*~*

All your files have been encrypted with a powerful combination of
symmetric and asymmetric cryptography. Do not tamper with the encrypted files.
It is of no use and will only risk corrupting your data.

To get your files decrypted send lots of cryptocurrency over Tor.

You'll need to copy and paste us these values to get your key.

Let’s hypothesize. The binary is used to encrypt the contents of SuspiciousFile.txt which gets written to SuspiciousFile.txt.Encrypted. If we are able to reverse the binary we can use that knowledge to recover the contents of the unencrypted SuspiciousFile.txt (which highly likely contains the flag). We should expect to see at least one symmetric cipher and one asymmetric cipher.

Static Analysis - Control Flow

Let’s use the ‘usage’ string to see what the top-level function could be. We check the references to the string and find sub_3BF0. See control flow diagram below. This function first looks up the address of RtlGenRandom function using LoadLibraryA on “advapi32” and then GetProcAddress on “SystemFunction036”. Next, it calls sub_21D0 which has a lot of nested calls that do not call WinAPIs. Together with the fact that no content has been loaded hints me that this may be some key schedule. Next, it loops over all arguments and checks if these strings are appended with “.EncryptMe”. If so, it opens it and opens another file for writing (original filename without “.EncryptMe”, appended with “.Encrypted”) and calls sub_22A3 with the fileptrs of these open files. This function presumably does the encryption and reads the .EncryptMe file and writes the .Encrypted file. Next, the loop closes both fileptrs and when there are no more arguments left, it prints how many files it encrypted. Following, if it encrypted files, it calls sub_239B which seems to write HOW_TO_DECRYPT.txt with the aforementioned contents that state how to get the decrypted (or unencrypted) files back.

Dynamic Analysis - Making assumptions

Now that we know how we can encrypt files, let’s play with the binary to see if we can make any assumptions. We create a file test.txt.EncryptMe with contents “A” and supply it to the flareon.exe binary. We get feedback that the file has been encrypted. We compare SuspiciousFile.txt.Encrypted with test.txt.Encrypted and notice that the first block of 0x49 bytes has been decreased to 1 byte, which is equal to the amount of bytes supplied in the test.txt.EncryptMe file! This hints us that for the symmetric cipher we are probably dealing with a stream cipher instead of a block cipher. Also, we notice that the first hexstring is the same, the other three differ. For the malware, this may be a hardcoded campaign ID used by the malware author to distinguish between campaigns (the value may also be used as a parameter in the encryption). In fact, the hexstring is also written to the HOW_TO_DECRYPT.txt file which strengthens this assumption. Both the second and third hexstring are also written to this file. The fourth is not.

Static Analysis - Encryption

Now that we have made some assumptions, let’s dive deeper in sub_22A3 which presumably does the encryption (we leave the nested calls for later). At the end of this function we see that the four hexstrings split with 0x0a that we previously saw from analyzing SuspiciousFile.txt.Encrypted are written to the fileptr .Encrypted. At offset 0x2321 we see the hardcoded first hexstring that starts with “9f18776b…”. At offset 0x2340 we see the hexstring unk_9100. From inspecting the xrefs we find that this is probably set by the assumed key schedule function sub_21D0. Same holds for the third hexstring at offset 0x235c. The fourth hexstring at offset 0x2375 is probably the result of sub_16CC.

Moving back to the beginning of the function, we notice that two buffers are populated with respectively 0x20 and 0xc random bytes using RtlGenRandom. Both of these buffers are passed to sub_20F0 along with the fileptrs to .EncryptMe and .Encrypted. This is highly likely the symmetric cipher which we assumed to be a stream cipher.

After, we see that sub_16CC is called with an empty buffer rbp, the random buffer(s) used in sub_20F0 (highly likely symmetric cipher) and unknown buffers unk_4020 and unk_9100 which are likely set by sub_21D0. The latter of which is stored to the encrypted file as the second hexstring.

Time for some more hypotheses. The assumed symmetric cipher sub_20F0 uses the random buffers of length 0x20 and 0xc as key and possibly IV or nonce. The result is probably written to the .Encrypted file. The function sub_16CC could be the asymmetric cipher that encrypts the key (and IV or nonce) using the buffers unk_4020 and unk_9100. The latter and the result of the encryption is written to the .Encrypted file. If so, we may be able to recover the key (and IV or nonce) used in the symmetric encryption.

Let’s dive deeper into the two ciphers.

Symmetric Cipher

When we try to reverse sub_20F0 we only see one instruction. There must be more. In IDA, we press [SPACE] to view the inline assembly and indeed see the rest of the function one instruction further that was wrongly recognized as a function. We undefine both functions by pressing u on their first instructions and press p on the first instruction (offset 0x20f0) to define the entire function.

b ^= (a + d) <<< 7;
c ^= (b + a) <<< 9;
d ^= (c + b) <<< 13;
a ^= (d + c) <<< 18;

We inspect sub_20F0 further and see that for every 0x40 bytes of the .EncryptMe file the function calls sub_1F10 and xors the results with the input buffer and writes that to the fileptr .Encrypted. Diving into sub_1F10 we see a pattern of eight chunks that rotate some data with respectively 16, 12, 8 and 7 bytes. This looks like the eight quarter-rounds but the operands of the rotations do not match. Searching the web for these rotations, we find a variant of Salsa20: Chacha20.

We don’t need to write a proof of concept to confirm that it indeed is Chacha20 because we can just use the binary to decrypt ciphertexts, editing the input buffers dynamically (because the operations are symmetric). The most important thing is that the third and fourth argument to the function are the key and nonce, and that the second argument holds the fileptr to the plaintext (.EncryptMe) and the first argument holds the fileptr to the resulting ciphertext (.Encrypted).

Let’s move to the presumed asymmetric cipher in sub_16CC.

Asymmetric Cipher

The function sub_16CC itself is not all that large or deep, but the exotic instructions tell me we should probably not reverse this. We’ll dynamically analyze this later. Moreover, the function is called in the previously assumed ‘key schedule function’ sub_21D0 which tells us that that function probably does something else.

Asymmetric cryptography is built on the fact that computations in one direction are easy, but really hard the other way around. For example, it could be built on the difficulty of factoring integers (RSA, DSA), or on the discrete logarithm problem (ECC, DSA).

RSA is often used as a text-book example of asymmetric cryptography for its ‘simplicity’. So for now, let’s assume that sub_16CC is RSA because DSA and especially ECC are more complex.

Dynamic Analysis - Asymmetric Cipher

To test our assumption we dynamically analyze sub_16CC. We run flareon.exe under WinDbg and break at flareon+231c where the call to sub_16CC is. We change the input buffers rdx to all zeroes and leave the others as-is. We notice that the output buffer contains zeroes. Great news because C = 0^e mod n = 0 and C = M^e mod 0 is undefined (function may return zeroes, we don’t know).

Next run, we edit the buffer of the third argument r8 to all zeroes and leave the others as-is. The output now contains a single byte 0x01. Even better news because C = M⁰ mod n = 1. This means that highly likely rdx points to the buffer M containing the Chacha20 key and nonce, r8 points to buffer e containing the public exponent and r9 points to buffer n containing the modulus.

To confirm that sub_16CC is indeed RSA we test it using this text-book example. We run the debugger again and edit the buffer rdx to 0x02, r8 to 0x07 and r9 to 0x21 (33). The resulting buffer is 0x1d (29) which is what we expected: C = 2⁷ mod 33 = 29. We confirmed sub_16CC is indeed RSA.

Recovering the plaintext

Now that we know which algorithms are used we can look into how we can recover the key and nonce which were used to encrypt the plaintext contents of SuspiciousFile.txt. Recall that the fourth argument (n, second hexstring) to the RSA function and the resulting buffer (C, fourth hexstring) is written to the .Encrypted file. In order to recover M we have to calculate M = C^d mod n. d, the private exponent is unknown. We could reverse the presumed ‘key schedule’ function sub_21D0 (which obviously is not a key schedule function anymore 😝), or we can try to attack the public exponent in the hopes of finding the private exponent. One of the most well known attacks against the public exponent of RSA is Wiener’s Attack. Let’s try that.

Wiener’s attack

We run the debugger again, break at flareon+231c where the call to sub_16CC is and copy the third argument buffer (e) and fourth argument buffer (n) for the attack. We find a nice Github repo by Nao Yonashiro that does the job for us. To our surprise they used 0x10001 as a weak private exponent! We can use this to decrypt the fourth hexstring in the .Encrypted file C which will give us the key and nonce used in the Chacha20 encryption.

Decryption

Now it is time to combine all the work.

RSA

The RSA decryption is very straight forward now: M = C^d mod n. We copy the fourth hexstring of the SuspiciousFile.txt.Encrypted and use it as the ciphertext C. We use 0x10001 as the private exponent d and copy the second hexstring to use as the modulo n. We decrypt using the native Python function pow(C, d, n) and see that the resulting plaintext M has the format of the Chacha20 key and nonce (in reverse order).

Chacha20

Because Chacha20 is symmetric we’ll use the flareon.exe binary to do the decryption for us. We copy the SuspiciousFile.txt.Encrypted to result.txt.EncryptMe and remove all bytes but the first 0x49. We run the debugger and break at flareon+2303 where the call to the Chacha20 function is. We modify the third argument buffer r8 to the recovered key and the fourth argument buffer r9 to the recovered nonce. We continue execution and should see the result in result.txt.Encrypted.

Solution

We inspect result.txt.Encrypted and indeed, we succeeded! 🥳🎉