[bugfix](be_metrics) update scan bytes metric correctly#52232

yiguolei · 2025-06-24T10:12:43Z

What problem does this PR solve?

unify scanbytes and scan rows definition in audit log，doris metrics，profile。 scan bytes is the uncompressed data read from the file + from the page cache. scan rows is the raw rows read from the file.
add update realtime counter interface in scanner, so other scanner type could implement this interface to get these counters realtime.
the uncompressed data read in page io is wrong.
cputimer counter is not updated correctly.

DO NOT merge this pr to 3.0 , because there is some behavior change for scan bytes. If user has config workload group policy or some monitor on scan bytes metric, there will be something wrong.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-06-24T10:13:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

yiguolei · 2025-06-24T10:14:13Z

run buildall

yiguolei · 2025-06-24T10:32:31Z

run buildall

hello-stephen · 2025-06-24T12:21:04Z

BE UT Coverage Report

Increment line coverage 4.76% (1/21) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.93% (15338/26942)
Line Coverage	46.01% (139028/302184)
Region Coverage	45.36% (70422/155261)
Branch Coverage	40.15% (37208/92678)

hello-stephen · 2025-06-24T13:48:31Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.37% (16530/26504)
Line Coverage	52.19% (157613/301996)
Region Coverage	49.67% (90506/182227)
Branch Coverage	43.31% (45670/105450)

yiguolei · 2025-06-28T08:40:02Z

run buildall

doris-robot · 2025-06-28T10:06:00Z

TPC-H: Total hot run time: 33866 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 654d5b947ddccab0c629d4273fddb467e7819204, data reload: false

------ Round 1 ----------------------------------
q1	17612	5207	5069	5069
q2	1932	275	188	188
q3	10347	1360	698	698
q4	10241	1030	516	516
q5	7809	2413	2325	2325
q6	184	155	126	126
q7	892	750	586	586
q8	9294	1301	1152	1152
q9	6965	5096	5108	5096
q10	6884	2384	1945	1945
q11	509	296	278	278
q12	336	358	214	214
q13	17769	3705	3097	3097
q14	234	221	215	215
q15	559	486	483	483
q16	416	423	374	374
q17	616	878	356	356
q18	7515	7178	7081	7081
q19	1440	953	540	540
q20	332	332	215	215
q21	3764	3160	2338	2338
q22	1025	1020	974	974
Total cold run time: 106675 ms
Total hot run time: 33866 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5113	5072	5062	5062
q2	239	319	219	219
q3	2228	2719	2311	2311
q4	1374	1799	1314	1314
q5	4203	4364	4359	4359
q6	210	166	126	126
q7	1976	1915	1772	1772
q8	2627	2564	2521	2521
q9	7099	7121	7160	7121
q10	3052	3200	2786	2786
q11	586	515	486	486
q12	697	744	619	619
q13	3594	3901	3275	3275
q14	278	290	275	275
q15	522	480	480	480
q16	434	479	428	428
q17	1150	1547	1375	1375
q18	7358	7212	6909	6909
q19	764	799	858	799
q20	1919	1957	1812	1812
q21	4810	4354	4223	4223
q22	1048	1066	991	991
Total cold run time: 51281 ms
Total hot run time: 49263 ms

doris-robot · 2025-06-28T10:17:30Z

TPC-DS: Total hot run time: 185065 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 654d5b947ddccab0c629d4273fddb467e7819204, data reload: false

query1	1012	403	406	403
query2	6548	1663	1659	1659
query3	6739	217	211	211
query4	26679	23845	23507	23507
query5	4350	601	450	450
query6	332	224	195	195
query7	4620	508	305	305
query8	262	225	211	211
query9	8614	2646	2644	2644
query10	460	329	267	267
query11	15941	15095	14935	14935
query12	159	112	101	101
query13	1636	518	409	409
query14	9291	5680	5649	5649
query15	206	187	172	172
query16	7418	644	469	469
query17	1197	797	556	556
query18	2006	407	295	295
query19	186	185	167	167
query20	114	125	113	113
query21	212	130	111	111
query22	4151	4236	3942	3942
query23	34221	33016	33120	33016
query24	8572	2357	2356	2356
query25	530	459	407	407
query26	1246	266	147	147
query27	2781	525	353	353
query28	4297	2129	2112	2112
query29	749	557	435	435
query30	289	225	199	199
query31	918	840	762	762
query32	80	64	60	60
query33	592	353	306	306
query34	784	837	523	523
query35	779	833	738	738
query36	939	971	869	869
query37	114	101	78	78
query38	4152	4192	4045	4045
query39	1506	1438	1417	1417
query40	205	119	102	102
query41	56	54	52	52
query42	125	105	102	102
query43	496	538	456	456
query44	1297	820	800	800
query45	176	172	167	167
query46	828	1027	625	625
query47	1805	1791	1685	1685
query48	390	402	313	313
query49	742	481	404	404
query50	632	684	419	419
query51	4294	4150	4113	4113
query52	106	106	104	104
query53	226	256	183	183
query54	578	573	505	505
query55	85	80	88	80
query56	316	307	272	272
query57	1205	1191	1109	1109
query58	264	252	261	252
query59	2520	2695	2457	2457
query60	351	303	316	303
query61	126	126	133	126
query62	793	720	666	666
query63	217	188	185	185
query64	4302	993	655	655
query65	4364	4231	4239	4231
query66	1098	403	313	313
query67	15796	15558	15280	15280
query68	5990	897	535	535
query69	499	315	284	284
query70	1167	1114	1031	1031
query71	400	319	286	286
query72	5852	4791	4871	4791
query73	654	642	347	347
query74	8869	9255	8651	8651
query75	3238	3204	2732	2732
query76	3139	1179	718	718
query77	489	395	281	281
query78	10442	10461	9634	9634
query79	1936	814	585	585
query80	653	514	506	506
query81	504	265	237	237
query82	211	128	99	99
query83	267	269	262	262
query84	255	112	87	87
query85	823	434	398	398
query86	327	283	284	283
query87	4463	4365	4261	4261
query88	2895	2247	2268	2247
query89	481	313	279	279
query90	1756	205	202	202
query91	134	136	112	112
query92	63	59	61	59
query93	1018	940	597	597
query94	626	407	291	291
query95	378	285	285	285
query96	493	573	281	281
query97	2688	2745	2672	2672
query98	236	214	223	214
query99	1338	1420	1265	1265
Total cold run time: 270359 ms
Total hot run time: 185065 ms

doris-robot · 2025-06-28T10:22:46Z

ClickBench: Total hot run time: 29.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 654d5b947ddccab0c629d4273fddb467e7819204, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.24	0.08	0.09
query4	1.62	0.11	0.11
query5	0.42	0.44	0.41
query6	1.18	0.65	0.65
query7	0.03	0.02	0.01
query8	0.05	0.03	0.03
query9	0.60	0.51	0.52
query10	0.57	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.62	0.62
query14	0.80	0.82	0.81
query15	0.92	0.87	0.86
query16	0.37	0.39	0.40
query17	1.08	1.06	1.07
query18	0.23	0.22	0.21
query19	1.92	1.87	1.87
query20	0.02	0.01	0.01
query21	15.39	0.91	0.55
query22	0.76	1.25	0.65
query23	14.90	1.40	0.62
query24	7.38	1.11	0.37
query25	0.38	0.24	0.09
query26	0.66	0.17	0.15
query27	0.06	0.06	0.05
query28	8.85	0.96	0.44
query29	12.57	4.04	3.33
query30	0.25	0.09	0.06
query31	2.85	0.59	0.38
query32	3.25	0.56	0.46
query33	3.10	3.10	3.05
query34	16.10	5.36	4.77
query35	4.80	4.88	4.77
query36	0.69	0.51	0.49
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.59 s
Total hot run time: 29.06 s

doris-robot · 2025-06-28T11:48:02Z

BE UT Coverage Report

Increment line coverage 9.38% (3/32) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.02% (15364/26946)
Line Coverage	46.11% (139420/302384)
Region Coverage	45.44% (70649/155471)
Branch Coverage	40.20% (37315/92816)

github-actions · 2025-07-01T02:30:45Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-01T02:30:47Z

PR approved by anyone and no changes requested.

1. unify scanbytes and scan rows definition in audit log，doris metrics，profile。 scan bytes is the uncompressed data read from the file + from the page cache. scan rows is the raw rows read from the file. 2. add update realtime counter interface in scanner, so other scanner type could implement this interface to get these counters realtime. 3. the uncompressed data read in page io is wrong. 4. cputimer counter is not updated correctly. DO NOT merge this pr to 3.0 or 3.1, because there is some behavior change for scan bytes. If user has config workload group policy or some monitor on scan bytes metric, there will be something wrong.

…52232 (#52584) Cherry-picked from #52232

### What problem does this PR solve? 1. unify scanbytes and scan rows definition in audit log，doris metrics，profile。 scan bytes is the uncompressed data read from the file + from the page cache. scan rows is the raw rows read from the file. 2. add update realtime counter interface in scanner, so other scanner type could implement this interface to get these counters realtime. 3. the uncompressed data read in page io is wrong. 4. cputimer counter is not updated correctly. DO NOT merge this pr to 3.0 or 3.1, because there is some behavior change for scan bytes. If user has config workload group policy or some monitor on scan bytes metric, there will be something wrong.

…53729) ### What problem does this PR solve? Problem Summary: ### Release note The external part is implemented according to the framework defined by the unified audit log, Doris metrics, and scanbytes and scan rows in Profile in #52232. However, - Scan bytes in the external table currently represents the bytes counted by **the top-level File Reader** called by the scan reader layer. - Scan rows represents the number of scanned rows of the underlying storage. The number of scanned rows of parquet/orc does not include the number of rows of skipped page/rowgroup. **Note: However, there is still a problem that `jni_reader` has not yet implemented the number of rows that only contain the storage to be read.**

…pache#53729) Problem Summary: The external part is implemented according to the framework defined by the unified audit log, Doris metrics, and scanbytes and scan rows in Profile in apache#52232. However, - Scan bytes in the external table currently represents the bytes counted by **the top-level File Reader** called by the scan reader layer. - Scan rows represents the number of scanned rows of the underlying storage. The number of scanned rows of parquet/orc does not include the number of rows of skipped page/rowgroup. **Note: However, there is still a problem that `jni_reader` has not yet implemented the number of rows that only contain the storage to be read.**

…pache#53729) ### What problem does this PR solve? Problem Summary: ### Release note The external part is implemented according to the framework defined by the unified audit log, Doris metrics, and scanbytes and scan rows in Profile in apache#52232. However, - Scan bytes in the external table currently represents the bytes counted by **the top-level File Reader** called by the scan reader layer. - Scan rows represents the number of scanned rows of the underlying storage. The number of scanned rows of parquet/orc does not include the number of rows of skipped page/rowgroup. **Note: However, there is still a problem that `jni_reader` has not yet implemented the number of rows that only contain the storage to be read.**

forget update profile counters in update_realtime_counters bug introduced in apache#52232 Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

forget update profile counters in update_realtime_counters. bug introduced in #52232

…55929) forget update profile counters in update_realtime_counters. bug introduced in apache#52232

yiguolei changed the title ~~[bugfix&enhancement](be_metrics) update scan bytes metric correctly~~ [bugfix](be_metrics) update scan bytes metric correctly Jun 24, 2025

[bugfix&enhancement](be_metrics) update scan bytes metric correctly

654d5b9

yiguolei force-pushed the fix_scanbytes branch from 24eeac0 to 654d5b9 Compare June 28, 2025 08:25

morningman approved these changes Jul 1, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 1, 2025

github-actions bot added the reviewed label Jul 1, 2025

zclllyybb approved these changes Jul 1, 2025

View reviewed changes

yiguolei added the dev/3.1.x label Jul 1, 2025

yiguolei merged commit af77883 into apache:master Jul 1, 2025
25 of 28 checks passed

github-actions bot added the dev/3.1.x-conflict label Jul 1, 2025

morrySnow mentioned this pull request Jul 1, 2025

branch-3.1: [bugfix](be_metrics) update scan bytes metric correctly #52232 #52584

Merged

morrySnow pushed a commit that referenced this pull request Jul 1, 2025

branch-3.1: [bugfix](be_metrics) update scan bytes metric correctly #…

28a8c76

…52232 (#52584) Cherry-picked from #52232

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jul 1, 2025

wenzhenghu mentioned this pull request Jul 11, 2025

fix scan_bytes_from_local/remote_storage issue in audit log, when enable file cache #52127

Closed

16 tasks

kaka11chen mentioned this pull request Jul 25, 2025

[enhancement](be_metrics) update scan bytes metric in file_scanner. #53729

Merged

16 tasks

wenzhenghu mentioned this pull request Aug 27, 2025

branch-3.0 [refactor](file-cache) refactor statistics collection logic in VFileScanner #53385

Merged

16 tasks

freemandealer mentioned this pull request Sep 11, 2025

[fix](filecache) fix BytesScannedFromCache/Local always zero #55929

Merged

16 tasks

dataroaring pushed a commit that referenced this pull request Sep 14, 2025

[fix](filecache) fix BytesScannedFromCache/Local always zero (#55929)

66577bb

forget update profile counters in update_realtime_counters. bug introduced in #52232

freemandealer added a commit to freemandealer/doris that referenced this pull request Oct 30, 2025

[fix](filecache) fix BytesScannedFromCache/Local always zero (apache#…

1402f28

…55929) forget update profile counters in update_realtime_counters. bug introduced in apache#52232

freemandealer mentioned this pull request Oct 30, 2025

branch-3.1: [fix](filecache) fix BytesScannedFromCache/Local always zero #55929 #57536

Merged

freemandealer added a commit to freemandealer/doris that referenced this pull request Oct 30, 2025

[fix](filecache) fix BytesScannedFromCache/Local always zero (apache#…

0704eb6

…55929) forget update profile counters in update_realtime_counters. bug introduced in apache#52232

Conversation

yiguolei commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jun 24, 2025

Uh oh!

yiguolei commented Jun 24, 2025

Uh oh!

yiguolei commented Jun 24, 2025

Uh oh!

hello-stephen commented Jun 24, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2025

BE Regression && UT Coverage Report

Uh oh!

yiguolei commented Jun 28, 2025

Uh oh!

doris-robot commented Jun 28, 2025

Uh oh!

doris-robot commented Jun 28, 2025

Uh oh!

doris-robot commented Jun 28, 2025

Uh oh!

doris-robot commented Jun 28, 2025

BE UT Coverage Report

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Comments

yiguolei commented Jun 24, 2025 •

edited

Loading