If you ends in a code using shift rights and lefts, xors and several arithmetic operations it’s highly possible that it’s the implementation of a cryptographic algorithm. Here it’s going to be showed some ways to identify the algorithm that it’s used without needing to reverse each step.
If this function is used, you can find which algorithm is being used checking the value of the second parameter:
Check here the table of possible algorithms and their assigned values: https://docs.microsoft.com/en-us/windows/win32/seccrypto/alg-id
Compresses and decompresses a given buffer of data.
The CryptAcquireContext function is used to acquire a handle to a particular key container within a particular cryptographic service provider (CSP). This returned handle is used in calls to CryptoAPI functions that use the selected CSP.
Initiates the hashing of a stream of data. If this function is used, you can find which algorithm is being used checking the value of the second parameter:
Check here the table of possible algorithms and their assigned values: https://docs.microsoft.com/en-us/windows/win32/seccrypto/alg-id
Sometimes it’s really easy to identify an algorithm thanks to the fact that it needs to use a special and unique value.
If you search for the first constant in Google this is what you get:
Therefore, you can assume that the decompiled function is a sha256 calculator.
You can search any of the other constants and you will obtain (probably) the same result.
If the code doesn’t have any significant constant it may be loading information from the .data section.
You can access that data, group the first dword and search for it in google as we have done in the section before:
In this case, if you look for 0xA56363C6 you can find that it’s related to the tables of the AES algorithm.
It’s composed of 3 main parts:
{% hint style=“info” %} In order to identify a RC4 in a disassembly/decompiled code you can check for 2 loops of size 0x100 (with the use of a key) and then a XOR of the input data with the 256 values created before in the 2 loops probably using a %256 (mod 256) {% endhint %}
In the following image notice how the constant 0x9E3779B9 is used (note that this constant is also used by other crypto algorithms like TEA -Tiny Encryption Algorithm).
Also note the size of the loop (132) and the number of XOR operations in the disassembly instructions and in the code example:
As it was mentioned before, this code can be visualized inside any decompiler as a very long function as there aren’t jumps inside of it. The decompiled code can look like the following:
Therefore, it’s possible to identify this algorithm checking the magic number and the initial XORs, seeing a very long function and comparing some instructions of the long function with an implementation (like the shift left by 7 and the rotate left by 22).
+7) >> 3
which is the same as in line 35 (right): +7) / 8
modulus_len < 0x040
and in line 36 (right) it’s checking if inputLen+11 > modulusLen
You can identify both of them checking the constants. Note that the sha_init has 1 constant that MD5 doesn’t have:
Note the use of more constants
Check lookup table constants:
A CRC hash algorithm looks like:
The graph is quiet large:
Check 3 comparisons to recognise it: