SC: Implement EOSVM-OC support for sync calls #1378

linh2931 · 2025-04-15T18:33:28Z

This PR implements EOSVM-OC support for sync calls. It unifies apply() and do_sync_call() into a single execute(), and in turn unifying EOSVM/EOSVM-JIT and EOSOC processing, making code simpler. Considerable refactoring is performed as a result.

In OC implementation itself, most changes are around sync_call entry point handling.

Resolves #1043.

libraries/chain/webassembly/runtimes/eos-vm-oc.cpp

linh2931 · 2025-04-15T18:38:45Z

libraries/chain/webassembly/runtimes/eos-vm-oc/executor.cpp

+               int64_t(*call_func)(uint64_t, uint64_t, uint32_t) = (int64_t(*)(uint64_t, uint64_t, uint32_t))(cb->running_code_base + *code.call_offset);
+               call_func(context.get_sender().to_uint64_t(), context.get_receiver().to_uint64_t(), static_cast<uint32_t>(static_cast<eosio::chain::sync_call_context&>(context).data.size()));
+            } else {
+               status = eosio::chain::execution_status::receiver_not_support_sync_call;


How to check sync_call entry point signature?

imo we ought to not need to. The validity of the sync_call export doesn't need to be tracked and checked by each individual wasm runtime in a bespoke way -- we ought to be able to that in a single common location.

Unfortunately it seems we cannot use the instantiation cache as is, since OC can bypass that.

We could use the code index but perhaps that is considered improper. (that said, the code index already tracks non-protocol aspects of code like a reference count!)

I don't have other ideas at the moment.

Created an issue #1392

…which start 4 nodes (controllers) in a single test process

…uce memory usage by 8 nodes to 6 nodes so tests can be run

SC: save & restore GS register to allow nested EOS VM OC executors

…e test for OC temporarily

spoonincode · 2025-04-16T01:45:06Z

libraries/chain/webassembly/runtimes/eos-vm-oc.cpp

-   : cc(data_dir, eosvmoc_config, db), exec(cc), mem(wasm_constraints::maximum_linear_memory/wasm_constraints::wasm_page_size) {
+   : cc(data_dir, eosvmoc_config, db) {
+   for (auto i = 0; i < 4; ++i) {
+      exec.emplace_back(std::make_unique<eosvmoc::executor>(cc));


Ideally every executor in a thread shares the same resources. Not really sure best approach -- an executor could expose a clone() call to return a new executor sharing the same resources, or maybe there needs to be more refactoring such as a class that holds executor's resources that is just passed to a non-member execute() call (having trouble coming up with names for these things off hand).

I don't think we need to do this executor improvement immediately though

Created an improvement issue for this #1394

spoonincode · 2025-04-16T01:46:01Z

libraries/chain/webassembly/runtimes/eos-vm-oc.cpp

-thread_local std::unique_ptr<eosvmoc::memory> eosvmoc_runtime::mem_thread_local{};
+thread_local std::vector<std::unique_ptr<eosvmoc::executor>> eosvmoc_runtime::exec_thread_local{};
+thread_local std::vector<std::unique_ptr<eosvmoc::memory>> eosvmoc_runtime::mem_thread_local{};
+thread_local uint32_t eosvmoc_runtime::ro_thread_index = 0;


I think we need the final stack approach before considering ready for review

I would like to make the wasm allocator pool as a templated class and use it here. Will retarget this PR on
#1346 to get it going.

spoonincode · 2025-04-16T01:55:22Z

unittests/sync_call_tests.cpp

+      BOOST_REQUIRE_NO_THROW(t.push_action("caller"_n, "callwithinpt"_n, "caller"_n, mvo()("input", 3)));
+      return;
+   }
+


Need to get the memory situation handled before ready for full review.

Currently the main thread memory is created with full 528 slices, and the RO thread memories are created with 10 slices.

For the simplest approach to get it floating, I'd suggest maybe just making every sync call use memory with a single slice.

Ideally we do something more complex where shallow sync calls get more slices than deeper sync calls (under the premise that shallow calls are more frequent). But we'd need to work out the math on this first.

Done with a single slice for now. Created an issue #1393.

…e_pool class

libraries/chain/include/eosio/chain/webassembly/runtime_interface.hpp

libraries/chain/webassembly/runtimes/eos-vm-oc.cpp

spoonincode · 2025-05-05T16:24:55Z

libraries/chain/include/eosio/chain/webassembly/eos-vm-oc/eos-vm-oc.hpp

   size_t code_begin;
   eosvmoc_optional_offset_or_import_t start;
   unsigned apply_offset;
+   std::optional<unsigned> call_offset;


I think this might cause a problem for existing code caches that may or may not be properly handled on start up. Changing the code_descriptor layout probably requires indicating something higher up (like at the header) that the format has been changed and the cache must be destroyed and recreated at start up.

Would the new version of current_codegen_version (line 48 below) be sufficient?

spring/libraries/chain/include/eosio/chain/webassembly/eos-vm-oc/eos-vm-oc.hpp

Line 48 in d60ed85

static constexpr uint8_t current_codegen_version = 3;

spring/libraries/chain/webassembly/runtimes/eos-vm-oc/code_cache.cpp

Lines 343 to 347 in d60ed85

if(cd.codegen_version != current_codegen_version) {

allocator->deallocate(code_mapping + cd.code_begin);

allocator->deallocate(code_mapping + cd.initdata_begin);

continue;

}

No I don't believe that will help. The serialized format of code_descriptor has changed, and it isn't versioned. I think you need to change

spring/libraries/chain/webassembly/runtimes/eos-vm-oc/code_cache.cpp

Line 21 in 8496995

static constexpr uint64_t header_id = 0x32434f4d56534f45ULL; //"EOSVMOC2" little endian

So that the cache file is completely removed and recreated

Thanks! I missed that.

…ting code cache will be destroyed and recreated at startup

spoonincode · 2025-05-05T18:42:43Z

libraries/chain/webassembly/runtimes/eos-vm-oc/code_cache.cpp

 static constexpr size_t header_size = 512u;
 static constexpr size_t total_header_size = header_offset + header_size;
-static constexpr uint64_t header_id = 0x32434f4d56534f45ULL; //"EOSVMOC2" little endian
+static constexpr uint64_t header_id = 0x33434f4d56534f45ULL; //"EOSVMOC2" little endian


change the comment too

Sorry about that.

Initial implementation of OC support for sync calls

6c9a9c1

linh2931 requested review from heifner and spoonincode April 15, 2025 18:33

Merge branch 'sync_call' into sync_call_on_oc

01d96dc

linh2931 marked this pull request as draft April 15, 2025 18:34

linh2931 commented Apr 15, 2025

View reviewed changes

libraries/chain/webassembly/runtimes/eos-vm-oc.cpp Outdated Show resolved Hide resolved

linh2931 commented Apr 15, 2025

View reviewed changes

linh2931 and others added 8 commits April 15, 2025 15:17

Reduce max_sync_call_depth temporarily to get around some unit tests …

347a9c1

…which start 4 nodes (controllers) in a single test process

save & restore GS register to allow nested executors

247a7a6

Make sure allocator and memory always return to pool after execution

2b15f44

Merge branch 'sync_call_on_oc' into sc_ocgs

8873764

Reduce exhaustive_snapshot_test's generation_count from 8 to 6 to red…

f06f5b1

…uce memory usage by 8 nodes to 6 nodes so tests can be run

Merge branch 'sync_call_on_oc' into sc_ocgs

556746a

Merge pull request #1382 from AntelopeIO/sc_ocgs

9017092

SC: save & restore GS register to allow nested EOS VM OC executors

Enable all sync call tests for OC

e570b66

linh2931 marked this pull request as ready for review April 15, 2025 22:51

linh2931 added 2 commits April 15, 2025 20:40

Reduce recursive depths for OC tests temporarily

060ac9e

Return execution status correctly; disable invalid sync_call signatur…

1c464ae

…e test for OC temporarily

spoonincode reviewed Apr 16, 2025

View reviewed changes

linh2931 marked this pull request as draft April 16, 2025 12:18

linh2931 changed the base branch from sync_call to wasm_allocs_pool April 16, 2025 12:33

linh2931 and others added 7 commits April 16, 2025 08:39

Merge branch 'wasm_allocs_pool' into sync_call_on_oc

bc8ad33

Fix a merging conflict

a146f06

rename wasm_alloc_pool.hpp to sync_call_resource_pool.hpp

f79cdb8

Remove wasm_alloc_pool.cpp

578ad96

Rename wasm_alloc_pool to call_resource_pool and make it a generic class

13b7681

Use the new call_resource_pool for eos-vm-oc

fc28a48

Use call_resource_pool for eosvmoc_tier

5d9c981

linh2931 added 3 commits April 16, 2025 20:55

Fix wrong sender

b9ca102

Restore exhaustive_snapshot_test to original number of subtesters

88dc660

Restore recursive calls related tests

110ceb8

This was referenced Apr 17, 2025

SC: OC sync_call entry point validation #1392

Closed

SC: OC -- investigate how many slices are created for sync call memory #1393

Closed

SC: OC executor improvement #1394

Open

linh2931 marked this pull request as ready for review April 17, 2025 01:19

Update verifyOcVirtualMemory test and correct VM usage

24f4689

linh2931 mentioned this pull request Apr 17, 2025

SC: A new approach to managing Wasm allocators in controller #1346

Merged

Base automatically changed from wasm_allocs_pool to sync_call April 17, 2025 17:08

linh2931 and others added 8 commits April 17, 2025 14:08

Use correct number of slices for sync calls

8b595f0

Use naked pointer for internal lockedfree statck in sync_call_resourc…

cc7f17b

…e_pool class

Always call et_num_threads_for_call_res_pools() for simplicity

dbf91ea

Merge branch 'sync_call' into sync_call_on_oc

e8e75ff

Merge branch 'sync_call' into sync_call_on_oc

d9b7094

Merge branch 'sync_call' into sync_call_on_oc

ee47110

Merge branch 'sync_call' into sync_call_on_oc

7fcfe63

fix a merging error

6416b83

heifner suggested changes Apr 29, 2025

View reviewed changes

libraries/chain/include/eosio/chain/webassembly/runtime_interface.hpp Show resolved Hide resolved

libraries/chain/webassembly/runtimes/eos-vm-oc.cpp Outdated Show resolved Hide resolved

linh2931 and others added 3 commits April 30, 2025 08:05

Merge branch 'sync_call' into sync_call_on_oc

c32d132

Remove commented out code

d3bdb5e

Use consistent if else style

d60ed85

spoonincode reviewed May 5, 2025

View reviewed changes

Change header_id of code_cache_header to a new version such that exis…

cfa20e6

…ting code cache will be destroyed and recreated at startup

spoonincode reviewed May 5, 2025

View reviewed changes

Correct the comment about header_id version

4275bc8

spoonincode approved these changes May 6, 2025

View reviewed changes

heifner approved these changes May 7, 2025

View reviewed changes

linh2931 merged commit 85d7b51 into sync_call May 7, 2025
28 checks passed

linh2931 deleted the sync_call_on_oc branch May 7, 2025 21:56

	if(cd.codegen_version != current_codegen_version) {
	allocator->deallocate(code_mapping + cd.code_begin);
	allocator->deallocate(code_mapping + cd.initdata_begin);
	continue;
	}

SC: Implement EOSVM-OC support for sync calls #1378

SC: Implement EOSVM-OC support for sync calls #1378

Uh oh!

Conversation

linh2931 commented Apr 15, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linh2931 Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

linh2931 Apr 17, 2025 •

edited

Loading