fast-scan: Apply performance improvements to code: batch 3 (codebase collection and multiple inputs)

[pombredanne]

This is to apply some performance improvements to code either for archive extraction or otherwise from benchmark results.

[AyanSinhaMahapatra]

One area of improvement is ignoring files properly before scans so we can ignore large files which take a lot of time.

Scancode pre-scan plugins : --include and --ignore has some know bugs and issues, which usually causes performance issues if inputs are not specified correctly.
In some cases scancode does a file-system walk of large directories which are ignored and takes significant amount of time.
It could be useful to have some default configuration also to ignore scanning specific types of files.

There are some issues which are related, some of these could be sub-issues:

• Pre_Scan ignore takes more than two minutes for 2000 files even with configuring more processes #3745
• --ignore /math works, but --ignore /math/arm does not #3388
• Add glob matching comparable to gitignore files #3387
• --ignore does not appear to work for directories #4212
• Scanning multiple directories scans too much #3452
• How to ignore specific directorys during scan #2966
• the --ignore behaviour needs refinement  #726

[AyanSinhaMahapatra]

This was addressed by:

• Update --ignore plugin and improve multi-input performance #5055
• Merge commoncode back into scancode-toolkit #5116

Previously:

For a scancode scan with multiple input paths, we were:

• finding the common root and collecting the entire codebase with resources
• then walking the codebase again and deleting resources which were outside the scan inputs

This took an enormous amount of time and resources and the resulting --ignore/--include options were not being applied correctly.

Currently we are only walking the codebase once to collect resources for multiple input paths.

One example of performance gained:
Here aboutcode/repos/ are local checkouts of all the aboutcode repos. So scancode is essentially walking all those git repos even though they are outside the scan input scope, just because they are under the common root.

Before:

(venv)
aboutcode/repos/scancode-toolkit  f2097987fe ✔                                                                                                                                       
▶ scancode --package-only ../notebooks ../scancode-toolkit/samples --json-pp multiple-inputs.json
Setup plugins...
Collect file inventory...
Scan files for: packages with 15 process(es)...
[####################] 0
Scanning done.
Summary:        packages with 15 process(es)
Errors count:   0
Scan Speed:     0.00 files/sec.
Initial counts: 4303103 resource(s): 4146278 file(s) and 156825 directorie(s)
Final counts:   156825 resource(s): 0 file(s) and 156825 directorie(s)
Timings:
  scan_start: 2026-06-11T131708.834663
  scan_end:   2026-06-11T182843.193014
  inventory: 7306.76s
  pre-scan:ignore: 10723.32s
  pre-scan: 10723.32s
  scan: 9.68s
  output:json-pp: 12.79s
  output: 12.79s
  total: 18712.73s
Removing temporary files...done.

After (was stuck for 5 hours, but should take less than a second):

aboutcode/repos/scancode-toolkit  fix-ignore-pre-scan-plugin ✔                                                                                                                      
▶ scancode --package-only ../notebooks ../scancode-toolkit/samples --json-pp multiple-inputs.json
Setup plugins...
Collect file inventory...
Scan files for: packages with 15 process(es)...
[####################] 404
Scanning done.
Summary:        packages with 15 process(es)
Errors count:   0
Scan Speed:     1918.62 files/sec.
Initial counts: 229 resource(s): 202 file(s) and 27 directorie(s)
Final counts:   229 resource(s): 202 file(s) and 27 directorie(s)
Timings:
  scan_start: 2026-06-11T183402.090921
  scan_end:   2026-06-11T183402.236387
  scan: 0.11s
  total: 0.16s
Removing temporary files...done.

The same simple scan (with same results) is now a hundred thousand times faster. This could be less/more dependent on how many files are present from the common root of input paths, excluding the input scope for multiple inputs.

[AyanSinhaMahapatra]

This is completed, see #4065 (comment) above for details of performance improvements.