đź“Ź

Malware Algorithms

Encryption Algorithms

Symmetric Algorithms

  • AES
  • RC4
  • Serpent
  • Blowfish

Asymmetric Algorithms

  • RSA

Hashing and Compression

Hashing

  • MD5
  • SHA
  • CRC

Compression

  • APLib
  • LZNT
  • LZMA

Windows API calls to Functions

Encryption/Decryption

  • CryptAcquireContext()
  • CryptDeriveKey() We can identify the exact crypto algorithm used by looking at the argument passed into this function
      • notion image
  • CryptEncrypt()
  • CryptDecrypt()

Compression/Decompression

  • RtlCompressBuffer()
  • RtlDecrompressBuffer()
 

Hashing

  • CryptAcquireContext()
  • CryptCreateHash() We can identify the exact hashing algorithm used by looking at the argument passed into this function
notion image

Recognizing Custom Crypto Implementations

RC4

  1. Create a Substitution Box in memory
  1. Scramble the Substitution Box
  1. Decrypt the data byte-by-byte
  1. Uses a XOR loop
Other algorithms like AES uses this same approach as well for Encryption/Decryption
Sample code flow for RC4
Creating the Substitution Box of size 256
Creating the Substitution Box of size 256
Scrambling the Substitution Box
Scrambling the Substitution Box
Decrypting the data using the Substitution Box which XORs the values byte by byte
Decrypting the data using the Substitution Box which XORs the values byte by byte

AES

  • Similar to RC4 in that AES also uses a Substitution Box, just that AES uses several lookup tables and sub-boxes
  • We can identify AES through the use of Crypto Constants, which are constant values present in the lookup tables.
  • The decryption key must be divisibly by 16 as the keys are usually 32 bytes. RC4 on the other has keys between 1-256 bytes
Identifying sboxes. When searching for the values in the sbox, we may see them being used/reused online which will help us to identify the algorithm
Identifying sboxes. When searching for the values in the sbox, we may see them being used/reused online which will help us to identify the algorithm

Serpent

  • Simply identified based in it’s length, as the Serpent algorithm is on large function of encryption
  • Constant 9E3779B9 appears in the Serpent algorithm as a crypto constant. This also appears in Tiny Encryption Algorithm (TEA)
Locating the constant and the large loop
Locating the constant and the large loop
long chunks of operations hint that it’s Serpent
long chunks of operations hint that it’s Serpent

RSA

  • More complex than symmetric algorithms
  • No crypto constants unlike symmetric algorithms
  • An example is Ursnif which uses RSA to decrypt the Serpent key, which is then used to decrypt the rest of the payload
RSA pseudocode
RSA pseudocode

APLib

  • No constants used
  • Uses 3 compares against constants
  • The constants will not always be the same, but you can recognize the 3 compares as clue that it’s using APLib
  • Google for any values that show up and add “compression algorithm” the search and pray it shows APLib
Uses 3 compares with constants
Uses 3 compares with constants
Python code equivalent.
Python code equivalent.

MD5/SHA

  1. init()
  1. update()
  1. final()
Similar init functions in SHA and MD5
Similar init functions in SHA and MD5

CRC

  • Utilizes a lookup table containing constants
  • We can search online for these constant to verify if it’s using CRC
Sample code flow for CRC
notion image
CRC Lookup table constant which you can search online
CRC Lookup table constant which you can search online