Make VMTests pass again (both concrete and symbolic)

[feliam]

Re-re-generate tests for the official vm tests from https://github.com/ethereum/tests/tree/develop/VMTests so it matches new mcore API.
Make concrete and symbolic version of each.
Document any tests that wont pass because of mcore design limitation
Make them pass

WIP here:
https://github.com/trailofbits/manticore/blob/dev-new-auto-evmtest/tests/auto/make_evmtests.py

Related:
#727
#1040
#1157

[cryptomental]

Hi @feliam I was also looking into this today while checking #1167, most tests that do not pass are due to gas mismatch, I disassembled one example test, compared opcodes that come from pyevmasm and fees against to what I saw from my patched manticore and they match with running Manticore EVM opcodes gas consumption.

I narrowed down the reason for failures: self._get_memfee() returns incorrect gas to be consumed in some cases and in some cases SSTORE gas consumption looked suspicious.

I also added xUnit XML report generation to the report and fixed gas mismatch with SELFDESTRUCT.

I attach all xml test reports here, perhaps at some point it would be useful to hook everything into CI.

test-reports.tar.gz

[feliam]

Hi cryptometal.
VMTests are "Currently only Homestead". For VMTests to work we'll need pyevmasm to handle that particular fork, no? Also we would need different EVM emulation implementations for different forks if we expect to pass VMTests and then the new tests.
Passing Homestead First (both conc and symb) could be an excellent goal anyway.

FYI. VMTests are going to be deprecated soon and ported to GeneralStateTests like this one:
https://github.com/ethereum/tests/blob/develop/BlockchainTests/GeneralStateTests/stExample/add11_d0g0v0.json. Those tests are divided by fork flavor.

About the xml hooking it all in the CI. It sounds interesting. I'll need to investigate how to do that but you could file an issue about it if feel like it and we'll move the chat there.

[feliam]

So pyevmasm supports multiple forks and mcore can selects how to use this for example in the unittests. We are still using VMTests to test mcores. Idea is to make it pass Frontier first using VMTest and then build up from there to newer forks (and GeneralStateTest)

• Make VMTest pass in concrete mode (Frontier)
• Make VMTest pass in symbolic mode (Frontier)
• Do GeneralStateTest unittest generator Conc/Symb similar to what is generated from VMSTests
• Either add mcore the option to select different forks (support all forks) or keep building over the newest passing fork and forget about it (support only the lastest).