Speed up Package.getPackageConfigs

[atilaneves]

Package.getPackageConfigs was taking 3 seconds for a particular dub package, which was made worse by the fact it was called multiple times with different arguments. The changes in this PR bring it down to just under 200ms, which IMHO is still slow.

Investigating led me to realise that determineDependencyConfigs was being called over and over again for the exact same packages, especially common dependencies like mir-algorithm and unit-threaded. This PR does two things:

First, it makes the array of package dependencies smaller via .sort.uniq. This cut down the time by half to 1.5s.

Secondly, after taking a while to understand the inner workings of the function, it returns early for packages that have already been analysed. This brought it further down to the 100ms range.

[github-actions]

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5347800 bin/dub
-rough build time=64s
+executable size=5364184 bin/dub
+rough build time=63s

Full build output

DUB version 1.36.0, built on Mar  3 2024
LDC - the LLVM D compiler (1.37.0):
  based on DMD v2.107.1 and LLVM 17.0.6
  built with LDC - the LLVM D compiler (1.37.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.37.0/x64/ldc2-1.37.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.37.0-rc.1+commit.14.g998e1bcf: building configuration [application]
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5364184 bin/dub
STAT:rough build time=63s

[s-ludwig]

Looks good as far as I can see. Two additional things caught my eye:

1. getAllDependencies now performs a redundant allocation of the returned list - since getAllDependenciesRange is package, we could now just scrap it and put the implementation directly into getAllDependencies
2. Hard to tell without profiling this in some way, but it looks like it may make sense to pre-compute the result for each Package instance, as getAllDependencies is called in some places that look like they might be called often during a run

[s-ludwig]

Turns out there is actually code that mutates the dependencies of a Package through its recipe property, so the pre-computation won't work just like that, unfortunately. But on the other hand, for a certain project, this code got called more than 10 times per package on average when performing a build, so if this is something that turns up in a profiler, that should still be worthwhile to pursue.

[atilaneves]

I considered speeding up getAllDependencies but it wasn't where the time was being spent. I even cached it at one point and it made no difference.