[feat][paimon] support paimon system table #52598

vinlee19 · 2025-07-01T11:21:22Z

Overview

close: #56007
This PR adds support for querying Apache Paimon system tables, enabling users to inspect metadata and operational information about Paimon tables directly through SQL queries.

For complete system table reference and advanced usage patterns, see the Apache Paimon System Tables Documentation.

Usage Examples

Basic System Table Query

-- Query snapshot information
SELECT * FROM my_table$snapshots;

-- Query manifest details
SELECT * FROM my_table$manifests;

-- Query schema evolution
SELECT * FROM my_table$schemas;

Advanced Operations

-- Filter snapshots by commit time
SELECT snapshot_id, commit_user, commit_time 
FROM my_table$snapshots 
WHERE commit_time > '2025-01-01'
ORDER BY commit_time DESC;

-- Join snapshots with schemas
SELECT s.snapshot_id, sc.fields 
FROM my_table$snapshots s 
JOIN my_table$schemas sc ON s.schema_id = sc.schema_id;

Sample Output

mysql> SELECT * FROM my_table$snapshots;
+-------------+-----------+--------------------------------------+---------------------+-------------+-------------------------+------------------------------------------------------+------------------------------------------------------+-------------------------+--------------------+--------------------+------------------------+----------------------+
| snapshot_id | schema_id | commit_user                          | commit_identifier   | commit_kind | commit_time             | base_manifest_list                                   | delta_manifest_list                                  | changelog_manifest_list | total_record_count | delta_record_count | changelog_record_count | watermark            |
+-------------+-----------+--------------------------------------+---------------------+-------------+-------------------------+------------------------------------------------------+------------------------------------------------------+-------------------------+--------------------+--------------------+------------------------+----------------------+
|           1 |         0 | d7ea4996-92c7-469f-b9ff-c76525954f1c | 9223372036854775807 | APPEND      | 2025-03-04 22:48:45.575 | manifest-list-dc5490ba-420c-445a-b6f7-6962d394935c-0 | manifest-list-dc5490ba-420c-445a-b6f7-6962d394935c-1 | NULL                    |                  1 |                  1 |                      0 | -9223372036854775808 |
+-------------+-----------+--------------------------------------+---------------------+-------------+-------------------------+------------------------------------------------------+------------------------------------------------------+-------------------------+--------------------+--------------------+------------------------+----------------------+

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes. [doc][lakehouse] add paimon system table documentation doris-website#2592

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-07-01T11:21:28Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

vinlee19 · 2025-07-01T11:35:13Z

run buildall

doris-robot · 2025-07-01T11:46:59Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.54% (1172/1403)
Line Coverage	67.47% (20661/30623)
Region Coverage	67.09% (10267/15303)
Branch Coverage	56.34% (5370/9532)

hello-stephen · 2025-07-01T12:36:30Z

FE UT Coverage Report

Increment line coverage 11.20% (14/125) 🎉
Increment coverage report
Complete coverage report

...xtensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonSysTableJniScanner.java

fe/fe-core/src/main/java/org/apache/doris/tablefunction/PaimonTableValuedFunction.java

gensrc/thrift/PlanNodes.thrift

hello-stephen · 2025-07-01T14:45:55Z

BE UT Coverage Report

Increment line coverage 0.00% (0/30) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.06% (15356/26914)
Line Coverage	46.12% (139409/302265)
Region Coverage	45.43% (70652/155534)
Branch Coverage	40.20% (37303/92802)

vinlee19 · 2025-07-02T07:59:48Z

run buildall

doris-robot · 2025-07-02T08:19:50Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	82.81% (1185/1431)
Line Coverage	67.46% (20734/30737)
Region Coverage	67.14% (10304/15347)
Branch Coverage	56.42% (5382/9540)

hello-stephen · 2025-07-02T09:11:53Z

FE UT Coverage Report

Increment line coverage 11.72% (17/145) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-02T10:11:07Z

TPC-H: Total hot run time: 34049 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9aa4f340a7ab642b94ed9925c43a3e760c1ea8c7, data reload: false

------ Round 1 ----------------------------------
q1	17613	5197	5116	5116
q2	1949	306	199	199
q3	10380	1304	735	735
q4	10226	1055	517	517
q5	7624	2460	2290	2290
q6	181	158	126	126
q7	919	750	602	602
q8	9317	1312	1079	1079
q9	7074	5078	5059	5059
q10	6895	2400	1987	1987
q11	473	285	274	274
q12	339	350	217	217
q13	17765	3711	3112	3112
q14	239	220	212	212
q15	560	490	477	477
q16	422	432	368	368
q17	606	865	360	360
q18	7669	7323	7184	7184
q19	1224	935	571	571
q20	328	349	221	221
q21	3785	3222	2377	2377
q22	1054	1001	966	966
Total cold run time: 106642 ms
Total hot run time: 34049 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5229	5090	5083	5083
q2	243	329	223	223
q3	2196	2683	2338	2338
q4	1356	1795	1360	1360
q5	4392	4539	4547	4539
q6	214	168	125	125
q7	2056	2017	1829	1829
q8	2677	2695	2647	2647
q9	7473	7321	7269	7269
q10	3100	3595	2861	2861
q11	591	514	496	496
q12	664	791	691	691
q13	3782	4066	3381	3381
q14	296	299	303	299
q15	518	465	476	465
q16	463	507	453	453
q17	1388	1578	1414	1414
q18	7923	7610	7386	7386
q19	774	829	916	829
q20	1959	1954	1829	1829
q21	4875	4346	4352	4346
q22	1081	1040	1010	1010
Total cold run time: 53250 ms
Total hot run time: 50873 ms

doris-robot · 2025-07-02T10:22:33Z

TPC-DS: Total hot run time: 184940 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9aa4f340a7ab642b94ed9925c43a3e760c1ea8c7, data reload: false

query1	1015	384	392	384
query2	6512	1707	1713	1707
query3	6744	212	211	211
query4	26249	23796	23305	23305
query5	4350	579	431	431
query6	307	220	214	214
query7	4634	494	290	290
query8	265	214	221	214
query9	8591	2608	2613	2608
query10	473	328	267	267
query11	15440	15012	14814	14814
query12	150	108	99	99
query13	1656	518	393	393
query14	8848	5678	5680	5678
query15	224	192	166	166
query16	7285	630	464	464
query17	1201	681	574	574
query18	1959	392	295	295
query19	211	186	153	153
query20	119	124	111	111
query21	206	128	103	103
query22	4059	4172	3942	3942
query23	33885	32861	33001	32861
query24	8473	2332	2366	2332
query25	553	470	395	395
query26	1245	267	147	147
query27	2752	511	343	343
query28	4305	2147	2127	2127
query29	772	558	426	426
query30	284	213	207	207
query31	923	839	718	718
query32	76	61	60	60
query33	555	366	343	343
query34	796	837	528	528
query35	791	805	769	769
query36	968	973	881	881
query37	106	105	73	73
query38	4112	4244	4247	4244
query39	1482	1414	1441	1414
query40	208	117	102	102
query41	57	64	58	58
query42	122	111	142	111
query43	505	514	494	494
query44	1291	807	811	807
query45	175	169	164	164
query46	820	1019	626	626
query47	1769	1841	1777	1777
query48	379	407	313	313
query49	729	469	384	384
query50	637	691	416	416
query51	4275	4167	4105	4105
query52	111	118	97	97
query53	223	250	188	188
query54	575	569	499	499
query55	85	79	81	79
query56	301	298	286	286
query57	1202	1206	1112	1112
query58	277	261	278	261
query59	2615	2712	2499	2499
query60	343	323	333	323
query61	124	127	119	119
query62	773	707	661	661
query63	226	184	185	184
query64	4344	1010	661	661
query65	4239	4159	4223	4159
query66	1166	409	308	308
query67	15919	15510	15451	15451
query68	5242	889	522	522
query69	484	297	277	277
query70	1215	1137	1100	1100
query71	412	328	364	328
query72	5762	4749	4746	4746
query73	638	591	350	350
query74	9202	9144	8893	8893
query75	3184	3181	2719	2719
query76	3156	1135	724	724
query77	485	362	299	299
query78	10020	10205	9300	9300
query79	1734	819	584	584
query80	754	514	446	446
query81	472	250	227	227
query82	395	129	95	95
query83	251	252	225	225
query84	251	104	82	82
query85	743	354	319	319
query86	326	308	295	295
query87	4390	4451	4331	4331
query88	2852	2244	2239	2239
query89	374	319	289	289
query90	1677	203	199	199
query91	137	143	112	112
query92	65	61	54	54
query93	1091	943	577	577
query94	661	397	303	303
query95	372	296	294	294
query96	493	573	281	281
query97	2693	2799	2670	2670
query98	222	219	211	211
query99	1346	1394	1314	1314
Total cold run time: 267165 ms
Total hot run time: 184940 ms

doris-robot · 2025-07-02T10:27:55Z

ClickBench: Total hot run time: 30.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9aa4f340a7ab642b94ed9925c43a3e760c1ea8c7, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.06	0.05
query3	0.30	0.06	0.06
query4	1.60	0.09	0.08
query5	0.45	0.41	0.41
query6	1.17	0.66	0.66
query7	0.03	0.01	0.02
query8	0.06	0.06	0.05
query9	0.64	0.52	0.52
query10	0.57	0.58	0.58
query11	0.26	0.13	0.13
query12	0.25	0.13	0.13
query13	0.65	0.63	0.62
query14	0.85	0.80	0.83
query15	1.00	0.90	0.90
query16	0.38	0.39	0.38
query17	1.06	1.09	1.11
query18	0.24	0.23	0.22
query19	1.98	1.85	1.88
query20	0.02	0.02	0.01
query21	15.38	0.96	0.66
query22	0.93	1.04	0.79
query23	14.68	1.46	0.73
query24	5.15	0.65	0.31
query25	0.17	0.10	0.09
query26	0.57	0.22	0.18
query27	0.09	0.10	0.09
query28	10.99	1.19	0.59
query29	12.53	4.17	3.45
query30	0.28	0.07	0.06
query31	2.87	0.63	0.42
query32	3.23	0.60	0.51
query33	3.16	3.10	3.14
query34	16.69	5.38	4.67
query35	4.79	4.78	4.81
query36	0.66	0.51	0.50
query37	0.20	0.18	0.17
query38	0.17	0.17	0.16
query39	0.04	0.04	0.04
query40	0.19	0.17	0.17
query41	0.10	0.05	0.05
query42	0.06	0.06	0.06
query43	0.06	0.05	0.05
Total cold run time: 104.66 s
Total hot run time: 30.21 s

hello-stephen · 2025-07-02T11:18:53Z

BE UT Coverage Report

Increment line coverage 0.00% (0/38) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.14% (15392/26938)
Line Coverage	46.17% (139640/302471)
Region Coverage	45.48% (70779/155623)
Branch Coverage	40.23% (37349/92830)

vinlee19 · 2025-07-03T00:36:04Z

run p0

vinlee19 · 2025-07-03T01:10:51Z

run nonConcurrent

suxiaogang223

LGTM

github-actions · 2025-07-03T01:48:00Z

PR approved by anyone and no changes requested.

vinlee19 · 2025-07-03T03:41:49Z

run nonConcurrent

hello-stephen · 2025-07-03T06:50:09Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	82.56% (1188/1439)
Line Coverage	67.43% (20753/30779)
Region Coverage	66.98% (10311/15393)
Branch Coverage	56.32% (5390/9570)

doris-robot · 2025-07-03T08:26:13Z

TPC-H: Total hot run time: 33847 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ddb2e703d08ddf01f19cf8ab53b67b6be6b1013, data reload: false

------ Round 1 ----------------------------------
q1	17584	5209	5072	5072
q2	1940	281	183	183
q3	10307	1305	705	705
q4	10225	1032	528	528
q5	7536	2450	2327	2327
q6	190	162	131	131
q7	907	744	609	609
q8	9326	1299	1127	1127
q9	6826	5132	5039	5039
q10	6895	2374	1951	1951
q11	466	277	267	267
q12	336	352	212	212
q13	17798	3621	3017	3017
q14	225	235	211	211
q15	540	474	490	474
q16	419	426	363	363
q17	586	872	358	358
q18	7581	7187	7201	7187
q19	1919	955	555	555
q20	331	338	219	219
q21	3771	2561	2354	2354
q22	1026	1005	958	958
Total cold run time: 106734 ms
Total hot run time: 33847 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5244	5066	5103	5066
q2	242	327	221	221
q3	2125	2678	2280	2280
q4	1364	1782	1343	1343
q5	4270	4557	4557	4557
q6	220	179	134	134
q7	2063	1952	1832	1832
q8	2647	2545	2585	2545
q9	7229	7349	7253	7253
q10	3161	3280	2914	2914
q11	577	508	478	478
q12	693	809	654	654
q13	3516	3934	3508	3508
q14	268	306	267	267
q15	516	483	478	478
q16	470	494	417	417
q17	1151	1609	1405	1405
q18	7960	7965	7633	7633
q19	841	800	837	800
q20	1875	1966	1814	1814
q21	4878	4412	4341	4341
q22	1072	1049	1012	1012
Total cold run time: 52382 ms
Total hot run time: 50952 ms

doris-robot · 2025-07-03T08:37:45Z

TPC-DS: Total hot run time: 185033 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ddb2e703d08ddf01f19cf8ab53b67b6be6b1013, data reload: false

query1	998	379	383	379
query2	6548	1688	1649	1649
query3	6735	210	215	210
query4	26580	23228	23144	23144
query5	4402	587	449	449
query6	294	210	213	210
query7	4615	481	285	285
query8	283	220	208	208
query9	8618	2652	2643	2643
query10	475	327	273	273
query11	15758	15076	14821	14821
query12	152	99	103	99
query13	1650	526	397	397
query14	9108	5655	5685	5655
query15	202	190	183	183
query16	7205	655	462	462
query17	1174	699	570	570
query18	1966	404	325	325
query19	183	185	151	151
query20	119	114	118	114
query21	215	119	100	100
query22	4068	4265	4093	4093
query23	33764	33036	33055	33036
query24	8362	2379	2361	2361
query25	539	469	392	392
query26	718	266	149	149
query27	2715	497	355	355
query28	4224	2157	2141	2141
query29	679	575	467	467
query30	286	219	202	202
query31	969	801	757	757
query32	72	66	59	59
query33	564	352	320	320
query34	777	839	510	510
query35	778	816	742	742
query36	946	975	910	910
query37	115	106	76	76
query38	4165	4166	4109	4109
query39	1481	1407	1432	1407
query40	213	117	105	105
query41	89	50	51	50
query42	122	106	106	106
query43	483	514	490	490
query44	1294	821	812	812
query45	180	168	165	165
query46	822	1017	625	625
query47	1783	1806	1726	1726
query48	372	406	316	316
query49	686	478	384	384
query50	636	701	421	421
query51	4098	4171	4075	4075
query52	118	109	101	101
query53	229	254	199	199
query54	565	582	502	502
query55	83	83	86	83
query56	308	304	283	283
query57	1197	1188	1097	1097
query58	257	268	254	254
query59	2569	2638	2538	2538
query60	329	339	337	337
query61	145	151	145	145
query62	795	720	646	646
query63	230	197	204	197
query64	3082	1105	742	742
query65	4257	4191	4167	4167
query66	1013	418	319	319
query67	15893	15528	15564	15528
query68	8012	885	520	520
query69	477	304	262	262
query70	1223	1163	1154	1154
query71	461	321	294	294
query72	5623	4682	4773	4682
query73	721	640	349	349
query74	9188	9098	8691	8691
query75	3844	3157	2711	2711
query76	3623	1155	713	713
query77	792	354	281	281
query78	10098	10294	9435	9435
query79	2203	834	596	596
query80	571	574	445	445
query81	500	263	225	225
query82	448	127	91	91
query83	242	245	237	237
query84	246	103	89	89
query85	831	404	312	312
query86	380	293	294	293
query87	4497	4531	4285	4285
query88	3585	2283	2286	2283
query89	394	317	285	285
query90	1843	219	213	213
query91	139	139	109	109
query92	79	63	52	52
query93	1800	974	593	593
query94	680	425	311	311
query95	385	297	285	285
query96	499	576	284	284
query97	2712	2797	2626	2626
query98	232	206	202	202
query99	1341	1382	1299	1299
Total cold run time: 272019 ms
Total hot run time: 185033 ms

doris-robot · 2025-07-03T08:43:06Z

ClickBench: Total hot run time: 30.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ddb2e703d08ddf01f19cf8ab53b67b6be6b1013, data reload: false

query1	0.03	0.04	0.03
query2	0.11	0.05	0.06
query3	0.29	0.06	0.07
query4	1.60	0.08	0.08
query5	0.44	0.40	0.41
query6	1.15	0.67	0.65
query7	0.03	0.01	0.01
query8	0.06	0.05	0.05
query9	0.63	0.52	0.53
query10	0.56	0.57	0.58
query11	0.25	0.13	0.12
query12	0.25	0.13	0.14
query13	0.64	0.63	0.64
query14	0.82	0.82	0.83
query15	0.97	0.88	0.90
query16	0.39	0.38	0.37
query17	1.05	1.07	1.05
query18	0.25	0.23	0.23
query19	2.04	1.84	1.87
query20	0.02	0.01	0.01
query21	15.36	0.97	0.66
query22	0.93	1.06	0.81
query23	14.72	1.51	0.78
query24	5.06	0.59	0.31
query25	0.16	0.09	0.09
query26	0.56	0.22	0.19
query27	0.09	0.09	0.09
query28	11.04	1.16	0.58
query29	12.55	4.12	3.39
query30	0.29	0.08	0.06
query31	2.82	0.66	0.43
query32	3.24	0.61	0.51
query33	3.17	3.12	3.19
query34	16.69	5.51	4.72
query35	4.68	4.78	4.78
query36	0.65	0.50	0.50
query37	0.20	0.18	0.18
query38	0.17	0.16	0.15
query39	0.06	0.05	0.05
query40	0.19	0.17	0.16
query41	0.11	0.05	0.04
query42	0.06	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 104.44 s
Total hot run time: 30.2 s

vinlee19 · 2025-07-03T09:45:57Z

run buildall

hello-stephen · 2025-07-03T09:46:18Z

BE UT Coverage Report

Increment line coverage 0.00% (0/38) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.14% (15399/26949)
Line Coverage	46.16% (139653/302542)
Region Coverage	45.47% (70769/155632)
Branch Coverage	40.23% (37343/92830)

doris-robot · 2025-07-03T09:52:55Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	82.56% (1188/1439)
Line Coverage	67.46% (20763/30779)
Region Coverage	67.00% (10314/15393)
Branch Coverage	56.35% (5393/9570)

hello-stephen · 2025-07-08T15:04:37Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.60% (1255/1557)
Line Coverage	67.87% (21444/31596)
Region Coverage	67.25% (10608/15774)
Branch Coverage	56.86% (5578/9810)

hello-stephen · 2025-07-08T15:39:56Z

FE UT Coverage Report

Increment line coverage 9.15% (15/164) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-08T16:36:22Z

TPC-H: Total hot run time: 33255 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45e892ac8ecdd62df17a0646375b612624ca110d, data reload: false

------ Round 1 ----------------------------------
q1	17593	5213	5024	5024
q2	1956	297	203	203
q3	10348	1327	739	739
q4	10252	1045	526	526
q5	7701	2389	2337	2337
q6	181	163	131	131
q7	897	742	607	607
q8	9326	1384	1047	1047
q9	7026	5156	5197	5156
q10	6938	2380	1928	1928
q11	475	293	266	266
q12	341	366	205	205
q13	17758	3636	3115	3115
q14	236	221	218	218
q15	541	496	477	477
q16	420	422	376	376
q17	605	859	375	375
q18	7597	7210	7092	7092
q19	1515	970	562	562
q20	333	346	217	217
q21	3947	2550	2367	2367
q22	368	314	287	287
Total cold run time: 106354 ms
Total hot run time: 33255 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5152	5203	5127	5127
q2	236	322	220	220
q3	2218	2676	2254	2254
q4	1407	1772	1350	1350
q5	4190	4400	4639	4400
q6	212	168	126	126
q7	2030	1993	1886	1886
q8	2699	2744	2717	2717
q9	7300	7220	7335	7220
q10	3234	3275	2878	2878
q11	597	514	530	514
q12	671	769	633	633
q13	3588	4096	3384	3384
q14	285	295	286	286
q15	541	512	500	500
q16	474	504	446	446
q17	1132	1618	1404	1404
q18	7957	7600	7517	7517
q19	808	792	905	792
q20	2021	2134	1948	1948
q21	5231	4558	4582	4558
q22	733	654	552	552
Total cold run time: 52716 ms
Total hot run time: 50712 ms

doris-robot · 2025-07-08T16:48:02Z

TPC-DS: Total hot run time: 186483 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45e892ac8ecdd62df17a0646375b612624ca110d, data reload: false

query1	1022	395	384	384
query2	6544	1679	1703	1679
query3	6740	220	216	216
query4	26586	23429	23020	23020
query5	4393	601	445	445
query6	305	212	204	204
query7	4627	494	289	289
query8	286	232	213	213
query9	8593	2605	2619	2605
query10	486	334	284	284
query11	15799	15013	14852	14852
query12	153	102	111	102
query13	1644	526	385	385
query14	9264	5988	5899	5899
query15	202	193	172	172
query16	7476	423	252	252
query17	1350	696	568	568
query18	1990	406	283	283
query19	196	181	156	156
query20	117	124	111	111
query21	209	123	101	101
query22	4122	4213	4634	4213
query23	35045	33905	33761	33761
query24	8593	2412	2393	2393
query25	523	443	383	383
query26	1236	259	143	143
query27	2777	501	335	335
query28	4312	2133	2109	2109
query29	737	542	423	423
query30	288	215	186	186
query31	914	839	757	757
query32	67	62	59	59
query33	534	335	311	311
query34	815	837	519	519
query35	587	659	541	541
query36	956	975	883	883
query37	117	101	72	72
query38	4084	4182	4033	4033
query39	1447	1582	1373	1373
query40	205	117	110	110
query41	54	54	53	53
query42	127	104	110	104
query43	514	501	484	484
query44	1296	842	832	832
query45	176	166	165	165
query46	856	1025	640	640
query47	1730	1752	1703	1703
query48	381	426	308	308
query49	722	486	389	389
query50	674	693	418	418
query51	4214	4230	4134	4134
query52	110	105	91	91
query53	231	256	183	183
query54	582	558	490	490
query55	86	78	80	78
query56	301	308	282	282
query57	1181	1194	1106	1106
query58	263	253	254	253
query59	2606	2656	2541	2541
query60	323	321	332	321
query61	152	151	151	151
query62	841	718	652	652
query63	231	193	194	193
query64	4554	1207	847	847
query65	4248	4200	4224	4200
query66	1078	406	317	317
query67	16016	15389	15548	15389
query68	7802	902	539	539
query69	494	311	271	271
query70	1287	1094	1108	1094
query71	468	346	284	284
query72	5532	4670	4767	4670
query73	648	588	346	346
query74	9115	9075	8878	8878
query75	3839	3217	2670	2670
query76	3636	1152	740	740
query77	796	423	294	294
query78	10875	10990	10413	10413
query79	2217	831	585	585
query80	584	516	444	444
query81	500	256	224	224
query82	376	124	95	95
query83	254	259	242	242
query84	241	101	86	86
query85	819	364	307	307
query86	381	315	277	277
query87	4395	4483	4391	4391
query88	3869	2327	2307	2307
query89	386	321	285	285
query90	1845	220	212	212
query91	139	155	112	112
query92	76	62	57	57
query93	1841	966	597	597
query94	642	315	210	210
query95	373	294	293	293
query96	503	580	278	278
query97	2696	2772	2634	2634
query98	238	213	204	204
query99	1392	1398	1296	1296
Total cold run time: 277147 ms
Total hot run time: 186483 ms

doris-robot · 2025-07-08T16:53:27Z

ClickBench: Total hot run time: 29.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45e892ac8ecdd62df17a0646375b612624ca110d, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.63	0.11	0.11
query5	0.42	0.42	0.41
query6	1.17	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.61	0.52	0.52
query10	0.59	0.57	0.56
query11	0.15	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.61
query14	0.79	0.84	0.80
query15	0.89	0.87	0.90
query16	0.39	0.40	0.38
query17	1.07	1.11	1.04
query18	0.22	0.21	0.21
query19	2.20	1.86	1.79
query20	0.02	0.02	0.02
query21	15.38	0.94	0.54
query22	0.74	1.16	0.64
query23	15.00	1.38	0.63
query24	7.60	1.35	0.55
query25	0.49	0.08	0.08
query26	0.64	0.17	0.13
query27	0.05	0.06	0.05
query28	9.86	0.92	0.44
query29	12.60	3.97	3.30
query30	0.25	0.10	0.06
query31	2.84	0.59	0.38
query32	3.25	0.56	0.48
query33	3.15	3.14	3.12
query34	15.99	5.38	4.85
query35	4.88	4.86	4.81
query36	0.70	0.51	0.50
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.04	0.02	0.02
query40	0.17	0.15	0.13
query41	0.07	0.02	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.25 s
Total hot run time: 29.25 s

doris-robot · 2025-07-08T17:52:55Z

BE UT Coverage Report

Increment line coverage 0.00% (0/38) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.32% (15507/27053)
Line Coverage	46.30% (140925/304370)
Region Coverage	45.55% (71251/156412)
Branch Coverage	40.27% (37542/93228)

vinlee19 · 2025-07-09T01:28:44Z

run buildall

hello-stephen · 2025-07-09T01:53:00Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.60% (1255/1557)
Line Coverage	67.74% (21404/31596)
Region Coverage	67.19% (10599/15774)
Branch Coverage	56.79% (5571/9810)

hello-stephen · 2025-07-09T02:45:09Z

FE UT Coverage Report

Increment line coverage 9.43% (15/159) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-09T02:54:47Z

TPC-H: Total hot run time: 33364 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0bd6be646cb9e998621e449bca649d4c58356ad6, data reload: false

------ Round 1 ----------------------------------
q1	17605	5223	5045	5045
q2	1926	277	209	209
q3	10305	1305	705	705
q4	10219	1027	533	533
q5	7535	2326	2401	2326
q6	178	161	128	128
q7	881	747	594	594
q8	9305	1299	1066	1066
q9	6987	5161	5114	5114
q10	6890	2393	1990	1990
q11	482	288	271	271
q12	350	343	214	214
q13	17777	3682	3067	3067
q14	231	227	220	220
q15	549	479	480	479
q16	416	425	375	375
q17	597	871	363	363
q18	7631	7301	7237	7237
q19	2164	968	560	560
q20	335	347	215	215
q21	3777	2569	2358	2358
q22	365	317	295	295
Total cold run time: 106505 ms
Total hot run time: 33364 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5224	5277	5075	5075
q2	253	321	221	221
q3	2149	2705	2313	2313
q4	1332	1769	1340	1340
q5	4224	4489	4476	4476
q6	214	174	127	127
q7	2045	1996	1838	1838
q8	2651	2514	2593	2514
q9	7301	7340	7518	7340
q10	3144	3303	2898	2898
q11	583	517	519	517
q12	676	806	685	685
q13	3669	3939	3533	3533
q14	287	310	277	277
q15	518	475	471	471
q16	436	480	468	468
q17	1226	1518	1383	1383
q18	8162	7574	7600	7574
q19	817	850	982	850
q20	2068	2045	1814	1814
q21	4742	4355	4358	4355
q22	658	578	560	560
Total cold run time: 52379 ms
Total hot run time: 50629 ms

doris-robot · 2025-07-09T03:06:16Z

TPC-DS: Total hot run time: 185710 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0bd6be646cb9e998621e449bca649d4c58356ad6, data reload: false

query1	1000	383	384	383
query2	6534	1697	1704	1697
query3	6757	211	209	209
query4	27315	23541	23347	23347
query5	4338	577	435	435
query6	305	210	211	210
query7	4621	496	281	281
query8	261	216	212	212
query9	8613	2592	2611	2592
query10	467	312	265	265
query11	15325	15039	14787	14787
query12	154	108	103	103
query13	1656	521	406	406
query14	8473	5799	5798	5798
query15	192	183	186	183
query16	7273	448	252	252
query17	1321	702	555	555
query18	1959	399	310	310
query19	197	190	158	158
query20	127	115	119	115
query21	210	125	105	105
query22	4191	4078	4225	4078
query23	34114	33198	33219	33198
query24	8466	2344	2351	2344
query25	545	471	382	382
query26	1235	262	144	144
query27	2776	499	335	335
query28	4354	2140	2131	2131
query29	757	573	435	435
query30	285	219	193	193
query31	911	842	766	766
query32	72	60	65	60
query33	539	333	298	298
query34	805	841	508	508
query35	614	668	544	544
query36	950	966	880	880
query37	106	98	80	80
query38	4182	4148	4075	4075
query39	1456	1404	1392	1392
query40	209	114	103	103
query41	62	54	51	51
query42	125	113	104	104
query43	508	493	474	474
query44	1294	831	819	819
query45	187	172	163	163
query46	826	1009	626	626
query47	1770	1803	1731	1731
query48	380	409	315	315
query49	726	473	392	392
query50	635	683	419	419
query51	4160	4275	4114	4114
query52	108	111	99	99
query53	222	288	182	182
query54	564	556	494	494
query55	81	78	81	78
query56	314	289	286	286
query57	1173	1204	1141	1141
query58	267	249	257	249
query59	2685	2632	2554	2554
query60	327	314	327	314
query61	127	125	121	121
query62	814	700	653	653
query63	219	189	189	189
query64	4529	1189	827	827
query65	4246	4174	4199	4174
query66	1136	413	318	318
query67	16212	15678	15287	15287
query68	5643	873	529	529
query69	520	303	281	281
query70	1229	1116	1108	1108
query71	421	317	310	310
query72	5764	4700	4793	4700
query73	634	578	352	352
query74	8924	9143	8822	8822
query75	3210	3205	2693	2693
query76	3133	1136	717	717
query77	478	397	285	285
query78	10939	11074	10285	10285
query79	1074	813	586	586
query80	652	550	493	493
query81	471	254	218	218
query82	184	127	97	97
query83	253	253	239	239
query84	245	116	93	93
query85	765	358	322	322
query86	322	319	271	271
query87	4365	4468	4308	4308
query88	2861	2273	2291	2273
query89	367	311	281	281
query90	1775	211	203	203
query91	152	140	108	108
query92	63	59	60	59
query93	1026	943	605	605
query94	616	323	205	205
query95	385	290	286	286
query96	498	566	278	278
query97	2682	2749	2669	2669
query98	227	212	205	205
query99	1336	1413	1306	1306
Total cold run time: 268576 ms
Total hot run time: 185710 ms

doris-robot · 2025-07-09T03:11:37Z

ClickBench: Total hot run time: 30.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0bd6be646cb9e998621e449bca649d4c58356ad6, data reload: false

query1	0.04	0.03	0.04
query2	0.11	0.05	0.06
query3	0.28	0.06	0.07
query4	1.60	0.08	0.08
query5	0.43	0.41	0.41
query6	1.18	0.66	0.67
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.63	0.52	0.52
query10	0.59	0.58	0.57
query11	0.25	0.14	0.12
query12	0.26	0.13	0.14
query13	0.65	0.63	0.63
query14	0.82	0.84	0.83
query15	0.98	0.90	0.89
query16	0.38	0.37	0.39
query17	1.09	1.06	1.03
query18	0.25	0.22	0.23
query19	1.99	1.84	1.90
query20	0.02	0.02	0.01
query21	15.38	0.93	0.67
query22	0.93	1.05	0.89
query23	14.66	1.49	0.74
query24	5.12	0.63	0.31
query25	0.18	0.10	0.09
query26	0.57	0.22	0.18
query27	0.09	0.08	0.08
query28	11.06	1.17	0.59
query29	12.57	4.06	3.32
query30	0.28	0.09	0.06
query31	2.86	0.63	0.44
query32	3.24	0.60	0.50
query33	3.08	3.14	3.18
query34	17.09	5.45	4.72
query35	4.82	4.78	4.79
query36	0.64	0.50	0.50
query37	0.22	0.17	0.18
query38	0.17	0.16	0.17
query39	0.05	0.05	0.04
query40	0.19	0.18	0.16
query41	0.11	0.06	0.06
query42	0.07	0.06	0.06
query43	0.05	0.05	0.04
Total cold run time: 105.05 s
Total hot run time: 30.21 s

doris-robot · 2025-07-09T04:02:50Z

BE UT Coverage Report

Increment line coverage 0.00% (0/38) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.32% (15507/27053)
Line Coverage	46.30% (140913/304370)
Region Coverage	45.55% (71250/156412)
Branch Coverage	40.27% (37544/93228)

morningman

LGTM

github-actions · 2025-07-09T06:13:35Z

PR approved by at least one committer and no changes requested.

This PR adds support for querying Apache Paimon system tables, enabling users to inspect metadata and operational information about Paimon tables directly through SQL queries. For complete system table reference and advanced usage patterns, see the [Apache Paimon System Tables Documentation](https://paimon.apache.org/docs/master/concepts/system-tables/). ```sql -- Query snapshot information SELECT * FROM my_table$snapshots; ``` ```sql -- Query manifest details SELECT * FROM my_table$manifests; ``` ```sql -- Query schema evolution SELECT * FROM my_table$schemas; ``` ```sql -- Filter snapshots by commit time SELECT snapshot_id, commit_user, commit_time FROM my_table$snapshots WHERE commit_time > '2025-01-01' ORDER BY commit_time DESC; ``` ```sql -- Join snapshots with schemas SELECT s.snapshot_id, sc.fields FROM my_table$snapshots s JOIN my_table$schemas sc ON s.schema_id = sc.schema_id; ``` Sample Output ```sql mysql> SELECT * FROM my_table$snapshots; +-------------+-----------+--------------------------------------+---------------------+-------------+-------------------------+------------------------------------------------------+------------------------------------------------------+-------------------------+--------------------+--------------------+------------------------+----------------------+ | snapshot_id | schema_id | commit_user | commit_identifier | commit_kind | commit_time | base_manifest_list | delta_manifest_list | changelog_manifest_list | total_record_count | delta_record_count | changelog_record_count | watermark | +-------------+-----------+--------------------------------------+---------------------+-------------+-------------------------+------------------------------------------------------+------------------------------------------------------+-------------------------+--------------------+--------------------+------------------------+----------------------+ | 1 | 0 | d7ea4996-92c7-469f-b9ff-c76525954f1c | 9223372036854775807 | APPEND | 2025-03-04 22:48:45.575 | manifest-list-dc5490ba-420c-445a-b6f7-6962d394935c-0 | manifest-list-dc5490ba-420c-445a-b6f7-6962d394935c-1 | NULL | 1 | 1 | 0 | -9223372036854775808 | +-------------+-----------+--------------------------------------+---------------------+-------------+-------------------------+------------------------------------------------------+------------------------------------------------------+-------------------------+--------------------+--------------------+------------------------+----------------------+ ```

picked from #52598

suxiaogang223 reviewed Jul 1, 2025

View reviewed changes

...xtensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonSysTableJniScanner.java Outdated Show resolved Hide resolved

suxiaogang223 reviewed Jul 1, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/tablefunction/PaimonTableValuedFunction.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/tablefunction/PaimonTableValuedFunction.java Outdated Show resolved Hide resolved

suxiaogang223 reviewed Jul 1, 2025

View reviewed changes

gensrc/thrift/PlanNodes.thrift Outdated Show resolved Hide resolved

vinlee19 requested a review from suxiaogang223 July 2, 2025 08:02

vinlee19 mentioned this pull request Jul 2, 2025

[doc][lakehouse] add paimon system table documentation apache/doris-website#2592

Merged

8 tasks

suxiaogang223 approved these changes Jul 3, 2025

View reviewed changes

github-actions bot added the reviewed label Jul 3, 2025

vinlee19 added 3 commits July 8, 2025 21:14

remove ro binlog audit_log

857caf6

optimize print error info

8d01a17

format code

45e892a

vinlee19 requested a review from morningman July 8, 2025 14:42

revert print info

0bd6be6

morningman approved these changes Jul 9, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 9, 2025

suxiaogang223 approved these changes Jul 9, 2025

View reviewed changes

morningman merged commit 7713596 into apache:master Jul 9, 2025
26 of 27 checks passed

github-actions bot added the dev/3.1.x-conflict label Jul 9, 2025

morningman mentioned this pull request Jul 14, 2025

branch-3.1: [feat](paimon) support paimon system table (#52598) #53239

Closed

morrySnow pushed a commit that referenced this pull request Jul 15, 2025

branch-3.1: [feat][paimon] support paimon system table #52598 (#53187)

e5ec387

picked from #52598

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jul 15, 2025

Comments

Conversation

vinlee19 commented Jul 1, 2025 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Usage Examples

Basic System Table Query

Advanced Operations

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 1, 2025

Uh oh!

vinlee19 commented Jul 1, 2025

Uh oh!

doris-robot commented Jul 1, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jul 1, 2025

FE UT Coverage Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Jul 1, 2025

BE UT Coverage Report

Uh oh!

vinlee19 commented Jul 2, 2025

Uh oh!

doris-robot commented Jul 2, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jul 2, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jul 2, 2025

Uh oh!

doris-robot commented Jul 2, 2025

Uh oh!

doris-robot commented Jul 2, 2025

Uh oh!

hello-stephen commented Jul 2, 2025

BE UT Coverage Report

Uh oh!

vinlee19 commented Jul 3, 2025

Uh oh!

vinlee19 commented Jul 3, 2025

Uh oh!

suxiaogang223 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 3, 2025

Uh oh!

vinlee19 commented Jul 3, 2025

Uh oh!

hello-stephen commented Jul 3, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Jul 3, 2025

Uh oh!

doris-robot commented Jul 3, 2025

Uh oh!

doris-robot commented Jul 3, 2025

Uh oh!

vinlee19 commented Jul 3, 2025

Uh oh!

hello-stephen commented Jul 3, 2025

BE UT Coverage Report

Uh oh!

doris-robot commented Jul 3, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jul 8, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jul 8, 2025

FE UT Coverage Report

vinlee19 commented Jul 1, 2025 •

edited by morningman

Loading