Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

Search backend: add support for negated predicates#40283

Merged
camdencheek merged 14 commits intomainfrom
backend-integration/cc/negated-predicates
Sep 8, 2022
Merged

Search backend: add support for negated predicates#40283
camdencheek merged 14 commits intomainfrom
backend-integration/cc/negated-predicates

Conversation

@camdencheek
Copy link
Member

@camdencheek camdencheek commented Aug 11, 2022

For most of our predicates, we can theoretically support negation. Up until now, we've explicitly disallowed negating predicates because their implementation (expanding the predicate in the query) was not performant enough to work at all unless the negation somehow excluded almost everything (-repo:contains.file(.*)?).

Now that we implement all our predicates natively, negation is actually within reach. In fact, when I rewrote our predicates, I rewrote most of them with negation in mind, so it's not a super big lift to start enabling it.

This PR removes the "predicates are not allowed to be negated" rule and enables negation for repo:contains.file(), repo:contains.content(), repo:has(key:value), and repo:has.tag(t).

Stacked on https://github.com/sourcegraph/sourcegraph/pull/40280
Closes #24325
Fixes https://github.com/sourcegraph/sourcegraph/issues/41475

Test plan

Added many integration tests. Manually tested with local repo set.

@cla-bot cla-bot bot added the cla-signed label Aug 11, 2022
@sourcegraph-bot
Copy link
Contributor

sourcegraph-bot commented Aug 11, 2022

Codenotify: Notifying subscribers in CODENOTIFY files for diff a8b4a34...edaba2d.

Notify File(s)
@beyang internal/search/query/predicate.go
internal/search/query/predicate_test.go
internal/search/query/types.go
internal/search/query/validate.go
internal/search/query/visitor.go
internal/search/query/visitor_test.go
@keegancsmith internal/search/query/predicate.go
internal/search/query/predicate_test.go
internal/search/query/types.go
internal/search/query/validate.go
internal/search/query/visitor.go
internal/search/query/visitor_test.go
@rvantonder internal/search/query/predicate.go
internal/search/query/predicate_test.go
internal/search/query/types.go
internal/search/query/validate.go
internal/search/query/visitor.go
internal/search/query/visitor_test.go
@unknwon dev/gqltest/search_test.go

@camdencheek camdencheek force-pushed the backend-integration/cc/negated-predicates branch from c1d2560 to 231576d Compare August 11, 2022 20:20
@camdencheek camdencheek changed the base branch from main to backend-integration/cc/simplify-list August 11, 2022 20:30
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This felt like the cleanest way of supporting negation in predicates. It allows each predicate to control whether it errors when negated, and we also already use this method to validate predicates during the validation step. It does somewhat blur the line between parsing, validation, and concrete representations, but I think we were actually already there. I renamed this to Unmarshal() from Parse() to reflect this.

Comment on lines 395 to 397
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the blanket "predicates do not support negation" error that is no longer needed.

@camdencheek camdencheek requested a review from a team August 11, 2022 20:38
@camdencheek camdencheek added the team/search-product Issues owned by the search product team label Aug 11, 2022
@rvantonder rvantonder requested a review from novoselrok August 12, 2022 03:20
Copy link
Contributor

@rvantonder rvantonder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs changelog entry, yes? closes https://github.com/sourcegraph/sourcegraph/issues/24325 yes?

@camdencheek
Copy link
Member Author

camdencheek commented Aug 12, 2022

needs changelog entry, yes?

ah, yep. I've got some changelog debt from predicates changes.

closes https://github.com/sourcegraph/sourcegraph/issues/24325 yes?

Sure enough. I forgot we had an issue for this

@camdencheek camdencheek force-pushed the backend-integration/cc/simplify-list branch from 98a876c to 1e478e5 Compare August 12, 2022 20:33
Base automatically changed from backend-integration/cc/simplify-list to main August 12, 2022 20:49
@camdencheek camdencheek force-pushed the backend-integration/cc/negated-predicates branch 5 times, most recently from 847b659 to afc59b8 Compare August 18, 2022 17:19
@camdencheek camdencheek force-pushed the backend-integration/cc/negated-predicates branch from afc59b8 to b33fa08 Compare September 7, 2022 20:45
@camdencheek camdencheek force-pushed the backend-integration/cc/negated-predicates branch from b33fa08 to 4176d99 Compare September 7, 2022 20:47
@camdencheek camdencheek mentioned this pull request Sep 8, 2022
@camdencheek camdencheek force-pushed the backend-integration/cc/negated-predicates branch from 43f5fab to 4176d99 Compare September 8, 2022 01:54
// Unmarshal parses the contents of the predicate arguments
// into the predicate object.
ParseParams(string) error
Unmarshal(params string, negated bool) error
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interface change: ParseParams is now Unmarshal to reflect the standard go idiom of mutating the struct being unmarshalled into. Additionally, at the point we unmarshal, we also store whether the predicate was negated (or error if not negated).

- A new [multi-version upgrade](https://docs.sourcegraph.com/admin/updates#multi-version-upgrades) process now allows Sourcegraph instances to upgrade more than a single minor version. Instances at version 3.20 or later can now jump directly to 4.0. [#40628](https://github.com/sourcegraph/sourcegraph/pull/40628)
- Matching ranges in file paths are now highlighted for path results and content results. [#41296](https://github.com/sourcegraph/sourcegraph/pull/41296) [#41385](https://github.com/sourcegraph/sourcegraph/pull/41385)
- Aggregations by repository, file, author, and capture group are now provided for search results. [#39643](https://github.com/sourcegraph/sourcegraph/issues/39643)
- Negation support for the search predicates `-repo:has.path()` and `-repo:has.content()`. [#40283](https://github.com/sourcegraph/sourcegraph/pull/40283)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the changelog does not include repo:has.tag() and repo:has(key:val) because those have not been announced/released yet, and will be tagged as experimental features for 4.0.

@camdencheek
Copy link
Member Author

Re-requesting review because this got a little stale and getting it updated required non-trivial changes.

Copy link
Contributor

@tbliu98 tbliu98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Copy link
Contributor

@rvantonder rvantonder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Curious to test on Sourcegraph.com to get a sense of how well this behaves :-)

@camdencheek camdencheek merged commit 7c30a14 into main Sep 8, 2022
@camdencheek camdencheek deleted the backend-integration/cc/negated-predicates branch September 8, 2022 20:31
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla-signed team/search-product Issues owned by the search product team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iterative Searchs Support negation for predicates

5 participants