Add detection support for newer versions of VMProtect.

[tamaroth]

Since version 2.04, every protected file has a public signature (detected with a YARA rule) and
the checksum of the first 64 DWORDS in the last executable section is always constant despite the
bytes in that part being different.

In addition, the tests were added here

[s3rvac]

Let's run TeamCity tests.

[s3rvac]

Thank you for the added support of newer versions of VMProtect! Could you please take a look at the failing tests? The code looks good, but there is something that prevents two tests from passing.

[s3rvac]

Let's run TeamCity tests.

[s3rvac]

Let's run TeamCity tests.

[s3rvac]

That macOS fail is caused by #702 (not by the present PR). However, the Linux build is unfortunately still failing. I will try to debug it. There is either another bug, or the build did not run with the fix, or there is something else going on.

[s3rvac]

The build has run for ec1cc47, which is currently the latest commit in this PR. So, the build has run with the fix.

[s3rvac]

I am able to reproduce the issue when using this Dockerfile. However, I was not yet able to determine the exact cause as when I start debugging the issue, the problems seem to go away. I will investigate it further. I just wanted to let you know about the progress.

[s3rvac]

Let's run TeamCity tests.

[s3rvac]

Alright, so, even after about 6 hours of debugging, I was unable to determine the real cause. Here is a couple of notes:

• retdec-fileinfo was crashing due to the changes in PeHeuristics::getVmProtectHeuristics() in this PR. Here is a backtrace from gdb:

  #0  0x00005555558618b8 in retdec::cpdetect::PeHeuristics::getVmProtectHeuristics() ()
  #1  0x000055555586f531 in retdec::cpdetect::PeHeuristics::getFormatSpecificCompilerHeuristics() ()
  #2  0x000055555583ec12 in retdec::cpdetect::CompilerDetector::getAllCompilers() ()
  #3  0x000055555584050b in retdec::cpdetect::CompilerDetector::getAllInformation() ()
  #4  0x0000555555666ee1 in retdec::fileinfo::FileDetector::getCompilerInformation() ()
  #5  0x00005555556677ac in retdec::fileinfo::FileDetector::getAllInformation() ()
  #6  0x0000555555614bc0 in main ()
  

  Line information is not available as the crashes are only reproducible in a release build.
• The crash was only reproducible with GCC 7 in Ubuntu (this is what the Linux TC agent runs). I was able to reproduce it by using this Dockerfile.
• The crash was only reproducible with the Release build type. If I tried a different build type (even RelWithDebInfo), it stopped crashing.
• I have tried upgrading GCC 7 from 7.4.0 to 7.5.0, but to no avail (it did not fixed the crashing).
• valgrind did not report any errors apart from retdec-fileinfo crashing with the backtrace above. I have tried several different build types with several different compilers.
• When I tried to debug the code by manually inserting debug prints, the crashing stopped. The crashing also stopped when I modified the computing of the checksum.
• The crash appeared to happen on the following line:

  checksum += *reinterpret_cast<const std::uint32_t*>(&data[i * 4]);

  However, when I simply printed the obtained integers, the code worked without any issues.
• The memory returned by section->getBytes() and stored in data seemed to be valid.

I do not know how to debug this further. I believe this to be a compiler bug. I would really like to know this for sure, but I do not have the time nor will to debug it further.

Nevertheless, I was able to modify the code so it is more readable and the Linux TC build no longer fails. @tamaroth: If the current code is OK from your point of view, we can merge the PR as all the tests pass now.

[tamaroth]

Thank you for your thorough investigation. If it crashed only on a specific version and only when placed directly in the checksum computing loop, perhaps there was a bug in optimisation? Hard to tell.

Again, thank you for your code review and updated code, it will be useful in the future.

The code looks good to me! 👍