Handle non-trivial frames in bundle adjustment#3214

ahojnnes · 2025-03-12T04:02:36Z

No description provided.

…camera rays (#3159) This is the first in a series of PRs to support cameras with >180deg FOV (including spherical cameras). This PR should result in no behavioral change other than changes up to numerical precision.

Currently, if we manage to initialize from one pair but get stuck after a few images and none of the other pairs satisfy the initialization thresholds, then we never try to initialize from a relaxed set of thresholds anymore for the remaining images, even though there is a high chance to succeed. In addition, we unnecessarily tried the same pairs redundantly for the same initialization constraints. This should result in more complete reconstruction results and faster initialization.

…#3167) This was a big confusion. I added some comments to explain why the fix in #3155 was correct. In the end its because that in COLMAP we use the left convention, while in GTSAM and its reference paper here https://arxiv.org/pdf/1812.01537 the right convention was employed. This is also in preparation for another PR on propagating relative pose covariance in COLMAP (with left convention) with cross-pose correlation available. This will encode the fact that the close image has lower relative pose covariance and does not get affected by Gauge ambiguity.

I found that some of my collaborators aren't aware of the pre-built Docker image. I think it would be helpful to mention it somewhere on the documentation's installation page.

A/B comparison against main branch. Changes mostly up to numerical noise except for "meadow" as the usual outlier. ``` I20250217 19:45:49.373275 1782301 compare.py:main:58] Results A - B: =====scenes===== ======AUC @ X deg (%)====== ===images=== =components= 0.5 1.0 5.0 10.0 reg all num largest ==============================eth3d=dslr============================== botanical_garden -0.03 -0.02 -0.00 -0.00 0 0 0 0 boulders -0.01 -0.00 -0.00 -0.00 0 0 0 0 bridge 0.00 0.00 0.00 0.00 0 0 0 0 courtyard 0.01 0.00 0.00 0.00 0 0 0 0 delivery_area 0.00 0.00 0.00 0.00 0 0 0 0 door -0.00 -0.00 -0.00 -0.00 0 0 0 0 electro -0.03 -0.00 0.00 0.00 0 0 0 0 exhibition_hall 0.01 -0.02 -0.00 -0.00 0 0 0 0 facade -0.01 -0.01 -0.00 -0.00 0 0 0 0 kicker -0.04 -0.02 -0.00 -0.00 0 0 0 0 lecture_room 0.00 -0.00 -0.00 -0.00 0 0 0 0 living_room 0.03 0.02 0.00 0.00 0 0 0 0 lounge -0.03 -0.01 -0.00 -0.00 0 0 0 0 meadow -3.93 -2.52 -0.50 -0.25 0 0 0 0 observatory 0.00 0.00 0.00 0.00 0 0 0 0 office -0.03 -0.03 -0.01 -0.01 0 0 0 0 old_computer -0.68 -0.35 -0.08 -0.04 0 0 0 0 pipes 1.50 0.95 0.19 0.09 0 0 0 0 playground -0.03 -0.02 -0.00 -0.00 0 0 0 0 relief 0.03 0.02 0.00 0.00 0 0 0 0 relief_2 -0.00 -0.00 -0.00 -0.00 0 0 0 0 statue 0.00 0.00 0.00 0.00 0 0 0 0 terrace -0.04 -0.02 -0.00 -0.00 0 0 0 0 terrace_2 -0.00 -0.00 -0.00 -0.00 0 0 0 0 terrains 0.00 0.00 -0.00 -0.00 0 0 0 0 ---------------------------------------------------------------------- overall -0.06 -0.03 -0.01 -0.00 0 0 0 0 ---------------------------------------------------------------------- average -0.13 -0.08 -0.02 -0.01 0 0 0 0 ```

…face (#3170) Now the name is too long. Creating this PR to initiate the thread. --------- Co-authored-by: Johannes Schönberger <jsch@demuc.de>

Towards supporting spherical / large FOV cameras for which a different logic will have to be implemented than for perspective projection models. The logic is now specialized and localized within the camera model and the user just knows about valid or invalid projections. This can also be used to more robustly deal with numerical issues in distortion computation (as we have for some of the fisheye models). The next PRs will do the equivalent for CamFromImg to return camera rays rather than image points as well as remove redundant "HasPointPositiveDepth/Cheirality" checks. --------- Co-authored-by: Paul-Edouard Sarlin <15985472+sarlinpe@users.noreply.github.com>

We got hit by the partial python version for the pycolmap CI build again: https://github.com/colmap/colmap/actions/runs/13519440475/job/37775195384 And I just realized that we are always building 3.8 rather than 3.12 on all the pycolmap ci build for pull requests. Was this intended? As it does not align with the comment here: https://github.com/colmap/colmap/blob/main/.github/workflows/build-pycolmap.yml#L35

#3193) … rays

…h/update

…ser/jsch/reconstruction-io-frames

…ser/jsch/non-trivial-frame-ba

B1ueber2y · 2025-03-15T13:57:44Z

  // Do the bundle adjustment only if there is any connected images.
  if (local_bundle.size() > 0) {
    BundleAdjustmentConfig ba_config;
+    ba_config.FixGauge(BundleAdjustmentGauge::THREE_POINTS);


This can result in different behaviors, as it was two cameras before rather than three points?

Actually, maybe the Gauge fixing is often not needed in local bundle adjustment as we mostly have some fixed 3D points in the problem.

It'll behave differently but I think it should be overall superior to the previous behavior. As you correctly point out, the previous behavior actually lead to an over-constraining of the Gauge, because typically there would already be a lot of constant points at the "boundary" of the local bundle.

I'll evaluate the impact on the ETH3D benchmark.

@B1ueber2y A/B comparison against main branch. Seems to be in the order of random noise.

I20250317 21:43:15.483952 807426 compare.py:main:56] Results A: =====scenes===== ======AUC @ X deg (%)====== ===images=== =components= 0.5 1.0 5.0 10.0 reg all num largest ==============================eth3d=dslr============================== botanical_garden 74.22 82.46 89.21 90.06 30 30 1 30 boulders 73.02 81.69 89.00 89.96 26 26 1 26 bridge 76.18 83.42 89.40 90.16 110 110 1 110 courtyard 65.30 77.45 88.19 89.55 38 38 1 38 delivery_area 65.17 70.05 73.98 74.47 40 44 1 40 door 79.23 84.38 89.60 90.26 7 7 1 7 electro 62.24 68.09 73.70 74.45 41 45 1 41 exhibition_hall 13.99 18.07 22.28 22.93 68 68 1 68 facade 74.79 82.63 89.25 90.08 76 76 1 76 kicker 69.11 76.75 83.34 84.19 30 31 1 30 lecture_room 62.49 74.69 87.36 89.14 23 23 2 23 living_room 65.91 77.34 87.95 89.43 65 65 1 65 lounge 23.17 26.74 29.59 29.95 6 10 1 6 meadow 61.53 69.76 85.62 88.27 15 15 2 15 observatory 28.58 35.66 42.12 42.95 27 27 2 14 office 30.44 39.03 49.82 51.48 20 26 1 20 old_computer 27.59 35.69 43.52 44.57 54 54 1 54 pipes 68.92 79.56 88.64 89.77 14 14 1 14 playground 53.13 58.50 62.98 63.56 32 38 2 32 relief 42.13 43.65 44.86 45.01 31 31 2 18 relief_2 43.28 45.56 47.43 47.66 31 31 2 20 statue 88.55 89.73 90.67 90.79 11 11 1 11 terrace 73.03 81.77 89.08 90.00 23 23 1 23 terrace_2 74.70 82.20 89.07 89.99 13 13 1 13 terrains 62.79 76.34 87.85 89.38 42 42 1 42 ---------------------------------------------------------------------- overall 59.67 67.19 73.92 74.81 873 898 31 836 ---------------------------------------------------------------------- average 58.38 65.65 72.58 73.52 35 36 1 33 I20250317 21:43:15.484521 807426 compare.py:main:57] Results B: =====scenes===== ======AUC @ X deg (%)====== ===images=== =components= 0.5 1.0 5.0 10.0 reg all num largest ==============================eth3d=dslr============================== botanical_garden 74.18 82.44 89.21 90.06 30 30 1 30 boulders 72.95 81.65 89.00 89.95 26 26 1 26 bridge 76.15 83.41 89.40 90.16 110 110 1 110 courtyard 64.30 76.84 88.05 89.48 38 38 1 38 delivery_area 65.16 70.05 73.97 74.47 40 44 1 40 door 79.19 84.33 89.59 90.25 7 7 1 7 electro 62.21 68.07 73.69 74.45 41 45 1 41 exhibition_hall 19.59 29.21 39.18 40.71 68 68 2 68 facade 74.79 82.63 89.25 90.08 76 76 1 76 kicker 67.86 76.11 83.21 84.13 30 31 1 30 lecture_room 62.60 74.76 87.38 89.15 23 23 2 23 living_room 66.02 77.41 87.97 89.44 65 65 1 65 lounge 23.13 26.72 29.59 29.94 6 10 1 6 meadow 63.10 70.82 85.83 88.37 15 15 2 15 observatory 28.26 35.49 42.08 42.93 27 27 2 14 office 31.56 41.25 54.49 56.61 21 26 1 21 old_computer 28.01 35.87 43.56 44.59 54 54 1 54 pipes 68.21 79.20 88.57 89.74 14 14 1 14 playground 53.33 58.59 63.00 63.57 32 38 2 32 relief 42.18 43.67 44.86 45.01 31 31 2 18 relief_2 43.42 45.62 47.44 47.67 31 31 2 20 statue 88.59 89.75 90.68 90.79 11 11 1 11 terrace 75.57 83.17 89.36 90.14 23 23 1 23 terrace_2 74.56 82.13 89.05 89.98 13 13 1 13 terrains 62.99 76.46 87.88 89.39 42 42 1 42 ---------------------------------------------------------------------- overall 60.26 68.36 75.69 76.68 874 898 32 837 ---------------------------------------------------------------------- average 58.72 66.23 73.45 74.44 35 36 1 33 I20250317 21:43:15.484681 807426 compare.py:main:58] Results A - B: =====scenes===== ======AUC @ X deg (%)====== ===images=== =components= 0.5 1.0 5.0 10.0 reg all num largest ==============================eth3d=dslr============================== botanical_garden 0.05 0.02 0.00 0.00 0 0 0 0 boulders 0.07 0.03 0.01 0.00 0 0 0 0 bridge 0.02 0.01 0.00 0.00 0 0 0 0 courtyard 1.00 0.62 0.14 0.07 0 0 0 0 delivery_area 0.01 0.01 0.00 0.00 0 0 0 0 door 0.04 0.06 0.01 0.01 0 0 0 0 electro 0.02 0.02 0.00 0.00 0 0 0 0 exhibition_hall -5.61 -11.14 -16.90 -17.78 0 0 -1 0 facade 0.00 0.00 0.00 0.00 0 0 0 0 kicker 1.25 0.63 0.13 0.06 0 0 0 0 lecture_room -0.11 -0.07 -0.02 -0.01 0 0 0 0 living_room -0.11 -0.08 -0.02 -0.01 0 0 0 0 lounge 0.04 0.02 0.00 0.00 0 0 0 0 meadow -1.57 -1.06 -0.21 -0.10 0 0 0 0 observatory 0.32 0.18 0.04 0.02 0 0 0 0 office -1.12 -2.23 -4.67 -5.13 -1 0 0 -1 old_computer -0.42 -0.19 -0.04 -0.02 0 0 0 0 pipes 0.71 0.37 0.07 0.04 0 0 0 0 playground -0.19 -0.10 -0.02 -0.01 0 0 0 0 relief -0.04 -0.02 -0.00 -0.00 0 0 0 0 relief_2 -0.14 -0.05 -0.01 -0.01 0 0 0 0 statue -0.04 -0.02 -0.00 -0.00 0 0 0 0 terrace -2.54 -1.40 -0.28 -0.14 0 0 0 0 terrace_2 0.13 0.07 0.01 0.01 0 0 0 0 terrains -0.20 -0.11 -0.02 -0.01 0 0 0 0 ---------------------------------------------------------------------- overall -0.60 -1.16 -1.77 -1.87 -1 0 -1 -1 ---------------------------------------------------------------------- average -0.34 -0.58 -0.87 -0.92 0 0 0 0

Thinking again on it, do we really need to fix the Gauge for the local bundle adjustment? Now we fix three points which can be potentially already constrained by the boundary points in the local BA. This will lead to noise at point fixing.

Also, fixing three points is more likely to introduce noise than fixing two cameras (6 + 1) as it fixed more degrees of freedom (3x3 = 9) than needed, where the three points are very likely starting at a quite noisy initialization.

If you look at the implementation, it only fixes three points, if not already three other points have been fixed (that have rank 3). In practice, this will basically never fix additional points unless the local bundle is equivalent to the global bundle.

I see. I missed that. Great then!

If we wanted to be extra safe, we could fix two cameras when the local bundle contains all images in the beginning? WDYT?

Yes but I guess it probably does not matter practically though.

IMO it might be already safe to just drop the Gauge in local bundle. I believe it may not affect much the performance there. In my experience even without Gauge fixing in global bundle it generally produces marginal difference (as long as we dont hit numerical issues).

B1ueber2y

Thanks a lot! LGTM.

ahojnnes and others added 30 commits February 17, 2025 10:50

Improved RANSAC dependency injection (#3165)

092a986

Add docker instruction link to docs (#3169)

047b677

I found that some of my collaborators aren't aware of the pre-built Docker image. I think it would be helpful to mention it somewhere on the documentation's installation page.

Add support for propagating relative pose covariance. (#3168)

877af2e

Avoid using namespace in pycolmap headers (#3173)

c0d3daf

Fix naming of cross covariance and add relative pose covariance inter…

3b87429

…face (#3170) Now the name is too long. Creating this PR to initiate the thread. --------- Co-authored-by: Johannes Schönberger <jsch@demuc.de>

Various improvements and extensions for pycolmap (#3176)

a38c640

Change CamFromImg to return optional ray (#3180)

73daf0f

Triangulation operates on camera rays (#3184)

7a6b205

Python bindings for visual index (#3185)

2e1d754

Define bindings in the correct order (#3189)

4b6f3a4

Restore CamFromImg to return normalized camera coordinates instead of… (

ced1299

#3193) … rays

Merge branch 'main' of https://github.com/colmap/colmap into user/jsc…

695b47a

…h/update

Support for serializing frames

e8282d9

d

196336d

d

c90a5e0

d

42ab839

d

ab079b7

Merge branch 'feature/rig' of https://github.com/colmap/colmap into u…

c31b7ee

…ser/jsch/reconstruction-io-frames

d

e93f5c3

d

a5e9efc

d

c2fe70b

d

2ba7c6a

d

c306d93

d

b1b4e69

ahojnnes added 5 commits March 10, 2025 13:32

d

84ba93d

d

64e7951

d

7113fa1

cr

b129842

Handle non-trivial frames in bundle adjustment

1595655

ahojnnes requested a review from B1ueber2y March 12, 2025 04:02

B1ueber2y reviewed Mar 12, 2025

View reviewed changes

Comment thread src/colmap/controllers/option_manager.cc Outdated

Comment thread src/colmap/estimators/bundle_adjustment.cc Outdated

Comment thread src/colmap/estimators/bundle_adjustment.cc Outdated

Comment thread src/colmap/estimators/bundle_adjustment.cc Outdated

ahojnnes added 2 commits March 12, 2025 13:38

w

cf8ab6c

d

6d0b9ce

Base automatically changed from user/jsch/reconstruction-io-frames to feature/rig March 13, 2025 17:49

ahojnnes added 9 commits March 13, 2025 11:08

Merge branch 'feature/rig' of https://github.com/colmap/colmap into u…

1c20526

…ser/jsch/non-trivial-frame-ba

s

60ff12e

d

9a81604

d

a307c16

d

5745e8a

d

d0e69ba

d

38ce578

d

9c18236

d

d58900b

B1ueber2y reviewed Mar 15, 2025

View reviewed changes

ahojnnes added 2 commits March 17, 2025 13:53

d

f5ecffe

d

9ef1149

B1ueber2y approved these changes Mar 17, 2025

View reviewed changes

Comment thread src/colmap/estimators/bundle_adjustment.cc Outdated

Comment thread src/colmap/estimators/bundle_adjustment.cc Outdated

ahojnnes and others added 4 commits March 17, 2025 20:10

d

2063ddd

d

f8fd101

d

02fbb81

d

3c420da

ahojnnes merged commit 34305bd into feature/rig Mar 18, 2025

ahojnnes deleted the user/jsch/non-trivial-frame-ba branch March 18, 2025 11:15

BrewTestBot mentioned this pull request Jun 30, 2025

colmap 3.12.0 Homebrew/homebrew-core#228622

Merged

Uh oh!

Conversation

ahojnnes commented Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahojnnes Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahojnnes Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

B1ueber2y left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ahojnnes Mar 17, 2025 •

edited

Loading

ahojnnes Mar 17, 2025 •

edited

Loading