vfmt,parser: move mark used from vfmt to parser

[kbkpbot]

The original vfmt internally handled the marking of used imports and symbols by itself through a "mark used" mechanism.
This PR moves the corresponding functionality of vfmt to the parser module.
This eliminates the need for the "mark used" logic within vfmt, making import statement handling and future integration of $if { import xx } easier.

The vfmt tests for vlib have passed.

[huly-for-github]

Connected to Huly®: V_0.6-24305

[kbkpbot]

Please don't merge it now.
I found some bug related to unused import symbol:

module main

import math { max }

fn max() {
}

fn main() {
        mut max := 1
        dump(max)
}

after vfmt, it should output:

module main

import math

fn max() {
}

fn main() {
        mut max := 1
        dump(max)
}

[spytheman]

No, @kbkpbot vfmt should not try to make semantic changes, that makes the output brittle, and vfmt itself annoying to use for incomplete programs while prototyping.

For the program you posted, it should not remove the imported symbol name, and it does not on master.

[kbkpbot]

currently vfmt try to use mark used to remove unused symbols.
for example :

module main

import math { max, min }

fn main() {
}

will remove unused imported symbols.

And it also try to add missing import

os.dir()

will output :

import os

os.dir()

imho, vfmt should remove such functions such as remove unused ..., add missing....
These thing should be checked by the checker. Not let the vfmt auto-fix .

[spytheman]

imho, vfmt should remove such functions such as remove unused ..., add missing....

I agree completely. It should do only cosmetic changes, and at most updating old syntax to new one. It should not try to guess anything - that is a job, better done by other tools, that have more context.

[spytheman]

However, I think that changing it, is currently out of scope for this PR.
I do not want to review and merge PRs, that do too many unrelated things/introduce too many new bugs/features.

[spytheman]

Excellent work, that will make future improvements much easier.
Thank you @kbkpbot 🙇🏻 .

I still need to compare the performance against master, hopefully the parser has not regressed too much.

[spytheman]

The slowdown for macos is between 0.5%-1% (for short programs like hello world) and 2.2% (for v).

0[13:18:05]@m1: (move-mark-used-from-vfmt-to-parser) /opt/v $ v run .github/workflows/compare_pr_to_master.v
==================================================================================================
Current git branch: move-mark-used-from-vfmt-to-parser, commit: 7201bd9
    Compiling new V executables from PR branch: move-mark-used-from-vfmt-to-parser, commit: 7201bd9 ...
CPU: 5.60s	Real: 5.55s	Elapsed: 0:05.55	RAM: 430944KB	./v -o vnew1 cmd/v
CPU: 5.05s	Real: 4.92s	Elapsed: 0:04.92	RAM: 425728KB	./vnew1 -o vnew2 cmd/v
CPU: 4.96s	Real: 5.12s	Elapsed: 0:05.12	RAM: 400000KB	./vnew2 -no-parallel -o vnew cmd/v
CPU: 0.09s	Real: 0.10s	Elapsed: 0:00.10	RAM: 41200KB	./vnew -no-parallel -o nhw_current.c examples/hello_world.v
CPU: 0.09s	Real: 0.10s	Elapsed: 0:00.10	RAM: 41024KB	./vnew -no-parallel -o nhw_current_gcc.c -cc gcc examples/hello_world.v
CPU: 1.23s	Real: 1.34s	Elapsed: 0:01.34	RAM: 400864KB	./vnew -no-parallel -o nv_current.c cmd/v
>Size of        nhw_current.c:      32102
>Size of    nhw_current_gcc.c:      32102
>Size of         nv_current.c:    6428782
>Size of                 vnew:    3891640
Преминахте към клона „v_repo_master“
Клонът е обновен към „V_REPO/master“.
==================================================================================================
    Compiling old V executables from branch: v_repo_master, commit: 24f9128 ...
CPU: 5.63s	Real: 5.43s	Elapsed: 0:05.43	RAM: 429840KB	./v -o vold1 cmd/v
CPU: 5.03s	Real: 4.89s	Elapsed: 0:04.89	RAM: 426272KB	./vold1 -o vold2 cmd/v
CPU: 4.96s	Real: 5.11s	Elapsed: 0:05.11	RAM: 401168KB	./vold2 -no-parallel -o vold cmd/v
CPU: 0.08s	Real: 0.10s	Elapsed: 0:00.10	RAM: 41920KB	./vold -no-parallel -o ohw_master.c examples/hello_world.v
CPU: 0.08s	Real: 0.10s	Elapsed: 0:00.10	RAM: 41472KB	./vold -no-parallel -o ohw_master_gcc.c -cc gcc examples/hello_world.v
CPU: 1.22s	Real: 1.33s	Elapsed: 0:01.33	RAM: 401328KB	./vold -no-parallel -o ov_master.c cmd/v
>Size of         ohw_master.c:      32102
>Size of     ohw_master_gcc.c:      32102
>Size of          ov_master.c:    6424017
>Size of                 vold:    3875064
==================================================================================================
File sizes so far ...
>>>>>> size("    nhw_current.c") - size("     ohw_master.c") =      32102 -      32102 =          0
>>>>>> size("nhw_current_gcc.c") - size(" ohw_master_gcc.c") =      32102 -      32102 =          0
>>>>>> size("     nv_current.c") - size("      ov_master.c") =    6428782 -    6424017 =       4765
>>>>>> size("             vnew") - size("             vold") =    3891640 -    3875064 =      16576
Преминахте към клона „move-mark-used-from-vfmt-to-parser“
==================================================================================================
    Measuring at PR branch: move-mark-used-from-vfmt-to-parser, commit: 7201bd9 ...
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 1/3, took: 1640.683 ms
 >  1           base             67.6ms ± σ:    0.1ms,   67.5ms…  67.7ms `./vold -check-syntax           examples/hello_world.v`
    2     +1.0%   1.01x slower   68.3ms ± σ:    0.3ms,   67.9ms…  68.7ms `./vnew -check-syntax           examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 2/3, took: 1642.661 ms
 >  1           base             67.5ms ± σ:    0.1ms,   67.4ms…  67.6ms `./vold -check-syntax           examples/hello_world.v`
    2     +1.9%   1.02x slower   68.7ms ± σ:    0.0ms,   68.7ms…  68.7ms `./vnew -check-syntax           examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 3/3, took: 1643.798 ms
 >  1           base             67.6ms ± σ:    0.1ms,   67.6ms…  67.7ms `./vold -check-syntax           examples/hello_world.v`
    2     +1.7%   1.02x slower   68.8ms ± σ:    0.0ms,   68.7ms…  68.8ms `./vnew -check-syntax           examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 1/3, took: 2322.989 ms
 >  1           base             95.7ms ± σ:    0.1ms,   95.6ms…  95.7ms `./vold -check                  examples/hello_world.v`
    2     +1.7%   1.02x slower   97.3ms ± σ:    0.1ms,   97.1ms…  97.5ms `./vnew -check                  examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 2/3, took: 2325.047 ms
 >  1           base             95.9ms ± σ:    0.1ms,   95.8ms…  96.0ms `./vold -check                  examples/hello_world.v`
    2     +1.4%   1.01x slower   97.3ms ± σ:    0.0ms,   97.2ms…  97.3ms `./vnew -check                  examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 3/3, took: 2332.685 ms
 >  1           base             96.3ms ± σ:    0.1ms,   96.2ms…  96.5ms `./vold -check                  examples/hello_world.v`
    2     +1.1%   1.01x slower   97.4ms ± σ:    0.2ms,   97.1ms…  97.5ms `./vnew -check                  examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 1/3, took: 2609.767 ms
 >  1           base            107.7ms ± σ:    0.1ms,  107.7ms… 107.8ms `./vold -no-parallel -o ohw.c   examples/hello_world.v`
    2     +1.0%   1.01x slower  108.8ms ± σ:    0.2ms,  108.6ms… 109.0ms `./vnew -no-parallel -o nhw.c   examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 2/3, took: 2600.625 ms
 >  1           base            107.3ms ± σ:    0.1ms,  107.2ms… 107.5ms `./vold -no-parallel -o ohw.c   examples/hello_world.v`
    2     +0.9%   1.01x slower  108.3ms ± σ:    0.1ms,  108.2ms… 108.4ms `./vnew -no-parallel -o nhw.c   examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 3/3, took: 2596.041 ms
 >  1           base            107.2ms ± σ:    0.0ms,  107.1ms… 107.2ms `./vold -no-parallel -o ohw.c   examples/hello_world.v`
    2     +1.1%   1.01x slower  108.3ms ± σ:    0.2ms,  108.1ms… 108.5ms `./vnew -no-parallel -o nhw.c   examples/hello_world.v`
>>>>>> size("            nhw.c") - size("            ohw.c") =      32102 -      32102 =          0
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 1/3, took: 5450.074 ms
 >  1           base            225.9ms ± σ:    0.2ms,  225.5ms… 226.1ms `./vold -no-parallel -o ohw.exe examples/hello_world.v`
    2     +0.5%   1.00x slower  227.0ms ± σ:    0.4ms,  226.5ms… 227.2ms `./vnew -no-parallel -o nhw.exe examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 2/3, took: 5423.446 ms
 >  1           base            225.0ms ± σ:    0.1ms,  224.9ms… 225.1ms `./vold -no-parallel -o ohw.exe examples/hello_world.v`
    2     +0.1%   1.00x ~same~  225.3ms ± σ:    0.3ms,  224.9ms… 225.6ms `./vnew -no-parallel -o nhw.exe examples/hello_world.v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 3/3, took: 5408.191 ms
 >  1           base            223.7ms ± σ:    0.3ms,  223.3ms… 224.0ms `./vold -no-parallel -o ohw.exe examples/hello_world.v`
    2     +0.8%   1.01x slower  225.5ms ± σ:    0.2ms,  225.2ms… 225.6ms `./vnew -no-parallel -o nhw.exe examples/hello_world.v`
>>>>>> size("          nhw.exe") - size("          ohw.exe") =     195208 -     195208 =          0
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 1/3, took: 11820.182 ms
 >  1           base            485.5ms ± σ:    0.3ms,  485.2ms… 485.9ms `./vold -check-syntax           cmd/v`
    2     +2.2%   1.02x slower  496.1ms ± σ:    0.1ms,  496.0ms… 496.2ms `./vnew -check-syntax           cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 2/3, took: 11812.184 ms
 >  1           base            485.7ms ± σ:    0.5ms,  484.9ms… 486.1ms `./vold -check-syntax           cmd/v`
    2     +2.1%   1.02x slower  495.9ms ± σ:    0.2ms,  495.6ms… 496.2ms `./vnew -check-syntax           cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 3/3, took: 11809.001 ms
 >  1           base            485.9ms ± σ:    0.2ms,  485.6ms… 486.2ms `./vold -check-syntax           cmd/v`
    2     +2.0%   1.02x slower  495.6ms ± σ:    0.3ms,  495.2ms… 495.8ms `./vnew -check-syntax           cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 1/3, took: 19476.875 ms
 >  1           base            801.2ms ± σ:    0.8ms,  800.4ms… 802.2ms `./vold -check                  cmd/v`
    2     +1.4%   1.01x slower  812.4ms ± σ:    0.4ms,  812.1ms… 812.9ms `./vnew -check                  cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 2/3, took: 19467.010 ms
 >  1           base            802.4ms ± σ:    0.6ms,  801.6ms… 803.2ms `./vold -check                  cmd/v`
    2     +1.3%   1.01x slower  812.8ms ± σ:    0.3ms,  812.5ms… 813.2ms `./vnew -check                  cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 3/3, took: 19479.961 ms
 >  1           base            802.5ms ± σ:    0.9ms,  801.4ms… 803.6ms `./vold -check                  cmd/v`
    2     +1.2%   1.01x slower  811.9ms ± σ:    1.0ms,  810.5ms… 812.9ms `./vnew -check                  cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 1/3, took: 32335.905 ms
 >  1           base           1340.9ms ± σ:    0.4ms, 1340.5ms…1341.5ms `./vold -no-parallel -o ov.c    cmd/v`
    2     +0.7%   1.01x slower 1350.2ms ± σ:    0.3ms, 1349.7ms…1350.4ms `./vnew -no-parallel -o nv.c    cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 2/3, took: 32339.318 ms
 >  1           base           1341.4ms ± σ:    0.6ms, 1340.6ms…1342.1ms `./vold -no-parallel -o ov.c    cmd/v`
    2     +0.5%   1.01x slower 1348.1ms ± σ:    1.4ms, 1346.5ms…1349.9ms `./vnew -no-parallel -o nv.c    cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 3/3, took: 32321.069 ms
 >  1           base           1340.6ms ± σ:    0.9ms, 1339.3ms…1341.3ms `./vold -no-parallel -o ov.c    cmd/v`
    2     +0.5%   1.00x slower 1346.7ms ± σ:    4.1ms, 1340.9ms…1349.8ms `./vnew -no-parallel -o nv.c    cmd/v`
>>>>>> size("             nv.c") - size("             ov.c") =    6428782 -    6428782 =          0
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 1/3, took: 122254.017 ms
 >  1           base           5079.4ms ± σ:    6.8ms, 5069.9ms…5085.1ms `./vold -no-parallel -o ov.exe  cmd/v`
    2     +0.2%   1.00x slower 5089.0ms ± σ:    4.4ms, 5082.9ms…5093.1ms `./vnew -no-parallel -o nv.exe  cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 2/3, took: 122206.829 ms
 >  1           base           5080.2ms ± σ:    1.0ms, 5079.0ms…5081.4ms `./vold -no-parallel -o ov.exe  cmd/v`
    2     +0.2%   1.00x slower 5089.7ms ± σ:    4.3ms, 5084.1ms…5094.6ms `./vnew -no-parallel -o nv.exe  cmd/v`
Summary after 1 series x 10 runs (%s are relative to first command, or `base`), discard maxs:  7, repeat: 3/3, took: 124198.797 ms
 >  1           base           5075.3ms ± σ:   10.5ms, 5060.5ms…5083.6ms `./vold -no-parallel -o ov.exe  cmd/v`
    2     +0.4%   1.00x slower 5094.1ms ± σ:    8.7ms, 5085.3ms…5105.9ms `./vnew -no-parallel -o nv.exe  cmd/v`
>>>>>> size("           nv.exe") - size("           ov.exe") =    3891640 -    3891640 =          0
Done. Total time: 631.301634 s.
==================================================================================================
Final summary for file diff sizes on their own branches:
>>>>>> size("    nhw_current.c") - size("     ohw_master.c") =      32102 -      32102 =          0
>>>>>> size("nhw_current_gcc.c") - size(" ohw_master_gcc.c") =      32102 -      32102 =          0
>>>>>> size("     nv_current.c") - size("      ov_master.c") =    6428782 -    6424017 =       4765
>>>>>> size("             vnew") - size("             vold") =    3891640 -    3875064 =      16576
==================================================================================================
Final summary for file diff sizes for generated files on the *current* branch:
>>>>>> size("            nhw.c") - size("            ohw.c") =      32102 -      32102 =          0
>>>>>> size("             nv.c") - size("             ov.c") =    6428782 -    6428782 =          0
>>>>>> size("          nhw.exe") - size("          ohw.exe") =     195208 -     195208 =          0
>>>>>> size("           nv.exe") - size("           ov.exe") =    3891640 -    3891640 =          0
0[13:28:42]@m1: (move-mark-used-from-vfmt-to-parser) /opt/v $