Skip to content

Comments

PackageManager: Load packages lazily on dub build#2500

Merged
Geod24 merged 2 commits intodlang:masterfrom
Geod24:lookup-linear
Oct 20, 2022
Merged

PackageManager: Load packages lazily on dub build#2500
Geod24 merged 2 commits intodlang:masterfrom
Geod24:lookup-linear

Conversation

@Geod24
Copy link
Member

@Geod24 Geod24 commented Oct 20, 2022

Implements the solution outlined here.

A problem of the current design of dub is that all locations are scanned,
regardless of what action is being performed, often multiple times.
With this change, a regular `dub build` on a project with all dependencies
resolved will no longer loads all available package, which can substantially
speed up development.

Fixes #2338

getBestPackage is intended to find the highest local version matching a range.
Due to this, it has to eagerly load packages and iterate through them.
However, when loading a resolved dependency, we have an exact Version,
which is simply wrapped in a VersionRange, so we can call getPackage instead.
@Geod24 Geod24 requested a review from WebFreak001 October 20, 2022 13:11
@WebFreak001
Copy link
Member

CI failing though

@rikkimax
Copy link
Contributor

Error Installed ldc-1.30.0 does not comply with dubtest1531 compiler requirement: ldc >1.30.0

Possibly unrelated but still worrying.

@WebFreak001
Copy link
Member

[INFO] Running /home/runner/work/dub/dub/test/issue1040-run-with-ver.sh...
    Fetching dub 1.27.0
    Fetching dub 1.28.0
    Fetching dub 1.29.0
Error:  /opt/hostedtoolcache/dc/dmd-2.100.2/x64/dmd2/linux/bin64/dub:23 Test for doubly fetch of the specified version has failed.
[INFO] Running /home/runner/work/dub/dub/test/ddox.sh...
[...]
Error No user package found at /home/runner/work/dub/dub/test/ddox/custom-tool/
Error:  /opt/hostedtoolcache/dc/dmd-2.100.2/x64/dmd2/linux/bin64/dub:18 command failed
[INFO] Running /home/runner/work/dub/dub/test/version-spec.sh...
             Registered package: foo (version: 1.0.0)
             Registered package: foo (version: 0.1.0)
Error:  /opt/hostedtoolcache/dc/dmd-2.100.2/x64/dmd2/linux/bin64/dub:8 command failed

Seem related

@Geod24
Copy link
Member Author

Geod24 commented Oct 20, 2022

Yeah errors are related. I solved one already, looking into the others.

@Geod24 Geod24 force-pushed the lookup-linear branch 3 times, most recently from 000c80a to 74bf4f2 Compare October 20, 2022 17:30
A problem of the current design of dub is that all locations are scanned,
regardless of what action is being performed, often multiple times.
With this change, a regular `dub build` on a project with all dependencies
resolved will no longer loads all available package, which can substantially
speed up development.
@Geod24
Copy link
Member Author

Geod24 commented Oct 20, 2022

This came together quite nicely. For reference, on my machine, with ~198 packages, this shortened start-up time by 120ms.

@Geod24 Geod24 merged commit dc75f07 into dlang:master Oct 20, 2022
@Geod24 Geod24 deleted the lookup-linear branch October 20, 2022 18:14
@nordlow
Copy link
Contributor

nordlow commented Oct 20, 2022

This came together quite nicely. For reference, on my machine, with ~198 packages, this shortened start-up time by 120ms.

What was the absolute start-up time before?

@Geod24
Copy link
Member Author

Geod24 commented Oct 20, 2022

It varies but in the 6s range.

@nordlow
Copy link
Contributor

nordlow commented Oct 20, 2022

What do you mean by startup in this case?

@Geod24
Copy link
Member Author

Geod24 commented Oct 20, 2022

Right, my benchmark was on a simple dub build on an already built project, to minimize the amount of variation.

{
// See `m_initialized` documentation
if (!this.m_initialized)
this.refresh();
Copy link
Contributor

@John-Colvin John-Colvin Oct 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whenever I see this pattern I think "shouldn't refresh handle the check?"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refresh can be called multiple times and depends on the environment, so it can't decide "on its own" to skip a run.
Maybe having a simple this.checkInitialized() function would be a bit more self-contained, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A single dub build refreshes the local package cache 3 times

5 participants