[Cache][Enhancement] Assure sql cache only one version#5793
Merged
morningman merged 1 commit intoapache:masterfrom May 28, 2021
Merged
[Cache][Enhancement] Assure sql cache only one version#5793morningman merged 1 commit intoapache:masterfrom
morningman merged 1 commit intoapache:masterfrom
Conversation
morningman
reviewed
May 24, 2021
gensrc/proto/internal_service.proto
Outdated
Contributor
There was a problem hiding this comment.
use optional to make it compatible with old version?
be/src/runtime/cache/result_node.cpp
Outdated
Contributor
There was a problem hiding this comment.
we need to make it compatible with old version
| _data_size -= (*it)->get_data_size(); | ||
| } | ||
| // clear old cache, and create new cache node | ||
| _partition_map.clear(); |
Contributor
There was a problem hiding this comment.
The _partition_map save the pointer of PartitionRowBatch, so call clear() will not release to memory used by PartitionRowBatch.
And also I checked the origin code, it seems that the PartitionRowBatch in _partition_map is never released?
Contributor
Author
There was a problem hiding this comment.
When partition list size is bigger than config::query_cache_max_partition_count, it will call function prune_first() to release some PartitionRowBatch.
Contributor
There was a problem hiding this comment.
prune_first() is to handle _partition_list, but what ahout _partition_map?
be/src/runtime/cache/result_node.cpp
Outdated
be/src/runtime/cache/result_node.cpp
Outdated
Contributor
There was a problem hiding this comment.
using debug level maybe better.
xinghuayu007
force-pushed
the
useless_cache
branch
2 times, most recently
from
9eb6f8f to
61fdb40
Compare
xinghuayu007
commented
May 25, 2021
| PartitionRowBatch* part_node = *_partition_list.begin(); | ||
| size_t prune_size = part_node->get_data_size(); | ||
| _partition_list.erase(_partition_list.begin()); | ||
| _partition_map.erase(part_node->get_partition_key()); |
morningman
added
approved
xinghuayu007
force-pushed
the
useless_cache
branch
from
61fdb40 to
391e8a0
Compare
stdpain
pushed a commit
to stdpain/incubator-doris
that referenced
this pull request
Jul 8, 2021
For PR apache#5792. This patch add a new param `cache type` to distinguish sql cache and partition cache. When update sql cache, we make assure one sql key only has one version cache.
w41ter
pushed a commit
to w41ter/incubator-doris
that referenced
this pull request
Dec 26, 2025
## Proposed changes pick from apache#58131 Previously, since clean tablet task was a lightweight operation on the BE, it was assumed that dropping tablets would not cause task backlog. However, online observations showed that tasks were still backlogged, causing the BE to consume a lot of memory. Therefore, tablet deduplication logic was added to prevent task backlog.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
For PR #5792. This patch add a new param
cache typeto distinguish sql cache and partition cache. When update sql cache, we make assure one sql key only has one version cache.Types of changes
What types of changes does your code introduce to Doris?
Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...