Rate limit reached when pulling from a bridge

Ten minutes ago, trying to git bug bridge pull the nixpkgs project's bugs (>5k bugs, >3k PRs -- a very very large set of bugs):

import error: Something went wrong while executing your query. Please include `A8DE:54BE:87BAE:9F6E7:62140406` when reporting this issue.

I have never been able to get a pull of nixpkgs' bugs to complete, and I've been trying for weeks now (see also #740).

I am using 05d73e1, which has the fix for #585.

Is there any chance that git-bug could keep fetching other bugs when it encounters a problem like this? It appears that it just aborts as soon as any API call produces an error. The nixpkgs bugset is ginormous, so it's going to take me at least a week (due to ratelimits) anyways. But right now it keeps failing after 108 bugs out of 5000+.

Is there an environment variable I can set to cause git-bug to print a backtrace or other context when it gives up like this?

import error: Something went wrong while executing your query. Please include `96F8:8D83:295160:2A2B42:6214AE3E` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `D1D4:02E4:B8460B:160070D:6214BB7A` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `D25A:179B:23FA97:64C0C7:6214BD6F` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `B1D0:CA69:188D36:19C1B9:6214CAB9` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `DC06:4AA8:86762:96A45:6214D808` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `BCAC:8D83:653F02:675761:6214DA0A` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `CA7A:6EAD:9E8FE8:A15B35:6214E738` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `D704:58D9:193918:19A2BB:6214F484` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `9C32:41F8:BF7CB1:172FF12:621501C1` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `9C86:62D0:471B40:B08239:621503C8` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `EB3C:8E88:466FFE:47B0D0:621510E9` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `A71A:5B83:1C51E:27EA5:62151E39` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `A7B8:4750:A7461:B3EE1:62152057` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `9332:81CD:3B1F92:3CC6F8:62152D6C` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `CC06:2824:17F801:56B3B8:62153AAA` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `A6E8:CEC2:8A2504:8CDFBB:621549CE` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `CD30:3456:5C995B:5F2A3F:6215571F` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `9E18:0AF2:CF4F53:D8D530:62156454` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `EA86:4003:1E512C:681416:62156642` when reporting this issue.
imported 0 issues and 0 identities with default bridge
import error: Something went wrong while executing your query. Please include `E05E:13CE1:1049D1D:108CD2E:62157381` when reporting this issue.

[rng-dynamics]

The error is from the github server. But I can tell you what I would try:

1. You should put the nixpkg repo in a tmpfs (mount -t tmpfs) because there is a big number of issues in nixpkg and git-bug writes more data than necessary on your disk while importing. Using a tmpfs might help you to speed up the import.

2. You can try to decrease the size of the queries to the github server here: https://github.com/MichaelMure/git-bug/blob/b2f0e126a10b5f030bfc35cac4b0cabcf083b589/bridge/github/import_mediator.go#L12-L15
   These are the number of items requested from the server with one query. You could start by reducing NumIssues. If it doesn't work, try to reduce the other ones, too. This will slow the import down, but it might enable you to get a full import without error. If you have success with this method, please let me know which numbers you used.

3. You can increase the number of retries in case the github server has an error:
   https://github.com/MichaelMure/git-bug/blob/b2f0e126a10b5f030bfc35cac4b0cabcf083b589/bridge/github/client.go#L110

I hope that helps. Let us know how it is going.

[MichaelMure]

git-bug writes more data than necessary on your disk while importing.

We should adress that at some point, any idea why that is?

[rng-dynamics]

We should adress that at some point, any idea why that is?

Do you remember that discussion: #585 (comment). As far as I understand .git/git-bug/bug-cache and .git/git-bug/identity-cache are rewritten after every imported issue, comment, edit, ... .

[MichaelMure]

Ha yes, I forgot about that, thanks.

The error is from the github server. But I can tell you what I would try:

Hey, thank you for your reply. Just to clarify:

1. You should put the nixpkg repo in a tmpfs

Oh, the reason it's slow is not git-bug; it's github's API ratelimit. I did not mean to imply that git-bug was not performant.

2. You can try to decrease the size of the queries to the github server here:

3. You can increase the number of retries in case the github server has an error:

Thanks, I will give those a try tonight or tomorrow.

Thanks, I will give those a try tonight or tomorrow.

Testing now.

Is there any way to get git-bug bridge pull to print verbosely what it is doing? Right now it only prints to the console when it discovers new data. There is a long silent delay at the start of the pull where it isn't clear what it's doing.

Hrm, with latest HEAD and NumIssues=10 I get:

rate limiting: Github GraphQL API rate limit. This process will sleep until 2022-03-01 04:12:20 +0000 UTC.
rate limiting: Github GraphQL API rate limit. This process will sleep until 2022-03-01 04:12:20 +0000 UTC.
rate limiting: Github GraphQL API rate limit. This process will sleep until 2022-03-01 04:12:20 +0000 UTC.
import error: API rate limit exceeded for user ID [redacted].

Apparently git-bug's API ratelimit calculation disagrees with github's?

NumIssues=20 did not solve the problem.

2. You can try to decrease the size of the queries to the github server here:

Decreasing these (as low as 20) did not help.

3. You can increase the number of retries in case the github server has an error:

Increasing this (as high as 10) did not help.

Is there any way to get more information about what git-bug was in the middle of doing when the error happened? Like a stack trace maybe?

[rng-dynamics]

I will have look at it, but I will need a few days. In case I do not come back to this issue in about a week, please feel free to ping me.

[MichaelMure]

@rng-dynamics it might make sense to implement another ImportResult/ExportResult for simple logs that the bridges could use to spit out more detailed information on what's happening. Those could be made visible with a --verbose flag or something like that.

At this point it would make sense to rename those to importEvent/ExportEvent.

[rng-dynamics]

@a-m-joseph, thanks for the bug report. The errors are seemingly arbitrary errors by github. If you want you can use my draft in #760. Even if there are github errors with #760 the bridge will continue importing the other issues. And you can run the import again and it will try to fetch the remaining issues. Have look at the file git-bug-import after running the import. The content of the file should be self-explanatory.

Thanks, I'm trying this right now.