Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[branch-3.0]count tablet meta's static memory load from disk (#41429) #45788

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Dec 23, 2024

pick #41429

Count segment and tableMeta's static memory which load from disk, and
add the memory usage to bvar.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wangbo
Copy link
Contributor Author

wangbo commented Dec 23, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40657 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f182228f4ff63b97f9a7cf0908ba0b4f77eb7c6, data reload: false

------ Round 1 ----------------------------------
q1	17601	7459	7344	7344
q2	2057	165	167	165
q3	10556	1126	1151	1126
q4	10223	799	691	691
q5	7739	2772	2741	2741
q6	233	145	145	145
q7	953	609	598	598
q8	9351	1929	1997	1929
q9	6624	6425	6460	6425
q10	7064	2271	2334	2271
q11	465	261	262	261
q12	400	215	211	211
q13	17825	2996	2967	2967
q14	236	207	206	206
q15	566	518	523	518
q16	699	598	614	598
q17	985	541	567	541
q18	7364	6739	6503	6503
q19	2136	1025	1004	1004
q20	489	208	203	203
q21	3875	3243	3224	3224
q22	1075	986	1009	986
Total cold run time: 108516 ms
Total hot run time: 40657 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7331	7226	7263	7226
q2	328	234	226	226
q3	2858	2707	2733	2707
q4	1883	1624	1632	1624
q5	5410	5429	5429	5429
q6	214	140	133	133
q7	2019	1642	1672	1642
q8	3184	3346	3393	3346
q9	8515	8494	8475	8475
q10	3446	3399	3393	3393
q11	595	490	491	490
q12	761	602	563	563
q13	16886	2939	2980	2939
q14	294	254	254	254
q15	557	505	511	505
q16	719	660	685	660
q17	1821	1593	1558	1558
q18	7771	7584	7141	7141
q19	1660	1523	1538	1523
q20	2017	1812	1766	1766
q21	5169	5001	4987	4987
q22	1053	994	972	972
Total cold run time: 74491 ms
Total hot run time: 57559 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189832 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f182228f4ff63b97f9a7cf0908ba0b4f77eb7c6, data reload: false

query1	1005	376	369	369
query2	6526	2054	1987	1987
query3	6705	210	217	210
query4	40653	23453	23481	23453
query5	4340	451	450	450
query6	250	173	172	172
query7	4065	306	305	305
query8	282	224	232	224
query9	6178	2698	2689	2689
query10	432	258	260	258
query11	15441	15172	15234	15172
query12	159	101	106	101
query13	984	421	404	404
query14	9769	6669	7131	6669
query15	215	170	172	170
query16	7765	477	511	477
query17	1591	572	551	551
query18	2009	314	315	314
query19	233	155	157	155
query20	118	107	110	107
query21	67	47	49	47
query22	4522	4171	4262	4171
query23	34582	33788	34722	33788
query24	12088	2877	2902	2877
query25	572	384	405	384
query26	1126	169	165	165
query27	2798	292	298	292
query28	7913	2480	2421	2421
query29	676	469	434	434
query30	334	171	167	167
query31	1006	783	834	783
query32	71	60	56	56
query33	651	289	285	285
query34	887	488	498	488
query35	854	734	720	720
query36	1068	934	932	932
query37	201	76	72	72
query38	3880	3835	3897	3835
query39	1472	1416	1434	1416
query40	233	87	88	87
query41	52	52	50	50
query42	113	104	98	98
query43	524	459	484	459
query44	1268	787	779	779
query45	181	170	169	169
query46	1144	733	718	718
query47	1883	1800	1801	1800
query48	467	375	365	365
query49	1287	420	388	388
query50	811	413	411	411
query51	7241	7114	6912	6912
query52	106	92	91	91
query53	267	188	183	183
query54	586	465	462	462
query55	81	77	77	77
query56	277	254	253	253
query57	1187	1112	1079	1079
query58	254	210	206	206
query59	3238	2844	2831	2831
query60	289	263	266	263
query61	108	105	106	105
query62	875	646	652	646
query63	212	186	184	184
query64	4559	649	608	608
query65	3311	3174	3164	3164
query66	1207	302	324	302
query67	15943	15400	15401	15400
query68	6189	564	552	552
query69	456	250	256	250
query70	1158	1057	1117	1057
query71	484	252	252	252
query72	6248	3902	3953	3902
query73	994	330	335	330
query74	10100	8884	8903	8884
query75	3975	2593	2609	2593
query76	3929	1111	1078	1078
query77	759	289	276	276
query78	10668	9829	9639	9639
query79	7485	578	575	575
query80	1907	423	413	413
query81	576	241	242	241
query82	1632	132	115	115
query83	306	142	145	142
query84	286	84	80	80
query85	1698	315	281	281
query86	481	297	301	297
query87	4490	4274	4239	4239
query88	5285	2379	2370	2370
query89	482	291	293	291
query90	2134	184	189	184
query91	187	146	149	146
query92	61	48	50	48
query93	6391	551	535	535
query94	1001	289	288	288
query95	346	258	253	253
query96	623	277	277	277
query97	3331	3144	3123	3123
query98	222	211	200	200
query99	1607	1286	1261	1261
Total cold run time: 316683 ms
Total hot run time: 189832 ms

@doris-robot
Copy link

TPC-H: Total hot run time: 40808 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f182228f4ff63b97f9a7cf0908ba0b4f77eb7c6, data reload: false

------ Round 1 ----------------------------------
q1	17567	7386	7269	7269
q2	2055	168	162	162
q3	10741	1879	1311	1311
q4	10998	763	766	763
q5	7740	2794	2778	2778
q6	235	148	146	146
q7	984	610	599	599
q8	9586	1955	2028	1955
q9	7956	6338	6390	6338
q10	7022	2273	2278	2273
q11	454	279	267	267
q12	404	218	215	215
q13	17780	2984	3010	2984
q14	252	206	205	205
q15	562	514	523	514
q16	681	616	596	596
q17	987	551	533	533
q18	7238	6595	6568	6568
q19	2447	1038	1013	1013
q20	487	201	198	198
q21	4025	3275	3131	3131
q22	1087	990	999	990
Total cold run time: 111288 ms
Total hot run time: 40808 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7390	7269	7317	7269
q2	326	250	251	250
q3	3038	2884	2903	2884
q4	2017	1776	1750	1750
q5	5654	5740	5703	5703
q6	226	144	145	144
q7	2218	1792	1767	1767
q8	3332	3482	3384	3384
q9	8981	8915	8869	8869
q10	3558	3529	3499	3499
q11	614	535	494	494
q12	819	642	551	551
q13	16565	3161	3131	3131
q14	314	266	265	265
q15	574	533	527	527
q16	719	663	671	663
q17	1856	1616	1618	1616
q18	8245	7749	7483	7483
q19	1690	1494	1528	1494
q20	2071	1858	1849	1849
q21	5408	5290	5255	5255
q22	1113	1005	1028	1005
Total cold run time: 76728 ms
Total hot run time: 59852 ms

@doris-robot
Copy link

TPC-H: Total hot run time: 40181 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f182228f4ff63b97f9a7cf0908ba0b4f77eb7c6, data reload: false

------ Round 1 ----------------------------------
q1	17555	7306	7228	7228
q2	2057	172	161	161
q3	10750	1034	1139	1034
q4	10555	711	676	676
q5	7741	2797	2671	2671
q6	237	147	145	145
q7	956	613	602	602
q8	9368	1934	1987	1934
q9	6520	6386	6319	6319
q10	7004	2304	2286	2286
q11	462	260	263	260
q12	408	223	210	210
q13	17781	2972	2965	2965
q14	229	204	206	204
q15	553	502	520	502
q16	682	600	614	600
q17	956	530	497	497
q18	7137	6483	6563	6483
q19	3790	1074	976	976
q20	493	216	203	203
q21	4324	3348	3275	3275
q22	1125	1040	950	950
Total cold run time: 110683 ms
Total hot run time: 40181 ms

----- Round 2, with runtime_filter_mode=off -----
q1	8067	7229	7251	7229
q2	328	233	225	225
q3	2970	2872	2801	2801
q4	2009	1785	1727	1727
q5	5632	5666	5674	5666
q6	241	146	141	141
q7	2191	1772	1747	1747
q8	3277	3545	3431	3431
q9	8785	8803	8855	8803
q10	3548	3485	3501	3485
q11	594	505	492	492
q12	784	590	589	589
q13	16459	3109	3066	3066
q14	307	264	270	264
q15	565	517	507	507
q16	705	673	663	663
q17	1835	1596	1601	1596
q18	8199	7842	7576	7576
q19	2547	1416	1463	1416
q20	2106	1840	1836	1836
q21	5347	5276	5294	5276
q22	1164	1004	993	993
Total cold run time: 77660 ms
Total hot run time: 59529 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195585 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f182228f4ff63b97f9a7cf0908ba0b4f77eb7c6, data reload: false

query1	1262	943	912	912
query2	6252	2106	2083	2083
query3	10800	4159	4228	4159
query4	68596	29011	23429	23429
query5	5186	458	429	429
query6	476	166	172	166
query7	5670	317	311	311
query8	322	224	220	220
query9	9432	2664	2652	2652
query10	502	256	251	251
query11	17936	15335	15896	15335
query12	152	107	103	103
query13	1536	425	420	420
query14	11453	7023	6670	6670
query15	203	171	178	171
query16	7259	487	490	487
query17	1099	562	560	560
query18	1882	309	294	294
query19	222	153	147	147
query20	113	111	111	111
query21	70	43	44	43
query22	4755	4373	4519	4373
query23	34655	34020	33978	33978
query24	6110	2877	2844	2844
query25	515	401	381	381
query26	669	171	165	165
query27	2001	292	296	292
query28	4314	2512	2449	2449
query29	678	435	445	435
query30	243	171	159	159
query31	985	810	840	810
query32	65	59	56	56
query33	452	279	282	279
query34	899	508	497	497
query35	825	754	752	752
query36	1097	957	937	937
query37	122	76	75	75
query38	4053	4079	4011	4011
query39	1503	1472	1492	1472
query40	144	85	85	85
query41	51	53	50	50
query42	115	96	102	96
query43	522	495	507	495
query44	1147	801	800	800
query45	190	179	176	176
query46	1143	750	737	737
query47	1980	1912	1896	1896
query48	460	397	372	372
query49	753	400	399	399
query50	859	417	420	417
query51	7347	7161	7100	7100
query52	113	86	91	86
query53	260	192	189	189
query54	579	453	445	445
query55	74	75	75	75
query56	261	253	244	244
query57	1210	1120	1116	1116
query58	229	206	208	206
query59	3383	3102	3002	3002
query60	292	261	257	257
query61	129	125	128	125
query62	766	675	670	670
query63	214	196	196	196
query64	1809	691	625	625
query65	3242	3153	3157	3153
query66	708	296	297	296
query67	15584	15252	15254	15252
query68	4599	562	550	550
query69	467	249	250	249
query70	1150	1075	1135	1075
query71	418	264	249	249
query72	6660	3938	3909	3909
query73	763	340	338	338
query74	10357	8992	9026	8992
query75	3333	2624	2606	2606
query76	2236	1019	1027	1019
query77	572	265	264	264
query78	10700	9700	9457	9457
query79	5943	589	590	589
query80	2254	414	417	414
query81	553	242	239	239
query82	1764	116	125	116
query83	279	141	140	140
query84	289	78	77	77
query85	1185	298	293	293
query86	472	302	298	298
query87	4435	4248	4358	4248
query88	5652	2377	2363	2363
query89	432	292	289	289
query90	2129	190	183	183
query91	184	145	143	143
query92	67	47	47	47
query93	5695	538	532	532
query94	823	272	297	272
query95	353	247	266	247
query96	636	277	277	277
query97	3356	3149	3140	3140
query98	216	201	196	196
query99	1843	1312	1296	1296
Total cold run time: 337743 ms
Total hot run time: 195585 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195389 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f182228f4ff63b97f9a7cf0908ba0b4f77eb7c6, data reload: false

query1	1322	940	906	906
query2	6257	2088	2079	2079
query3	10825	4135	4172	4135
query4	67802	29298	23393	23393
query5	5165	439	436	436
query6	446	171	162	162
query7	5643	310	307	307
query8	317	240	238	238
query9	9471	2685	2673	2673
query10	510	265	257	257
query11	17669	15288	15733	15288
query12	154	101	100	100
query13	1540	437	416	416
query14	10357	6816	7423	6816
query15	209	171	180	171
query16	7268	524	502	502
query17	1049	570	575	570
query18	1881	338	322	322
query19	216	154	162	154
query20	123	109	117	109
query21	66	46	47	46
query22	4625	4517	4385	4385
query23	34648	33862	34503	33862
query24	6104	2924	2912	2912
query25	555	411	428	411
query26	675	176	168	168
query27	1959	298	309	298
query28	4462	2541	2521	2521
query29	731	468	462	462
query30	258	160	190	160
query31	968	791	851	791
query32	65	55	54	54
query33	458	272	274	272
query34	895	504	514	504
query35	836	755	751	751
query36	1071	953	970	953
query37	119	72	70	70
query38	3992	3939	3960	3939
query39	1554	1480	1454	1454
query40	146	88	83	83
query41	48	48	48	48
query42	119	106	99	99
query43	540	500	510	500
query44	1138	806	783	783
query45	184	166	164	164
query46	1130	722	739	722
query47	2006	1913	1883	1883
query48	474	390	357	357
query49	722	386	399	386
query50	833	424	404	404
query51	7316	7263	7191	7191
query52	92	95	84	84
query53	257	180	182	180
query54	558	444	434	434
query55	75	74	75	74
query56	255	250	234	234
query57	1146	1092	1068	1068
query58	206	197	199	197
query59	3283	3405	3028	3028
query60	269	242	241	241
query61	128	113	107	107
query62	774	666	688	666
query63	209	179	178	178
query64	1766	659	611	611
query65	3252	3183	3176	3176
query66	710	294	293	293
query67	15772	15341	15500	15341
query68	4966	548	565	548
query69	412	244	285	244
query70	1161	1140	1065	1065
query71	429	263	240	240
query72	6511	3977	3934	3934
query73	764	336	334	334
query74	10155	8889	8844	8844
query75	3350	2612	2590	2590
query76	2256	1107	981	981
query77	485	267	275	267
query78	10835	9665	9432	9432
query79	8344	566	580	566
query80	2297	410	417	410
query81	566	238	238	238
query82	1192	120	115	115
query83	249	145	140	140
query84	288	89	85	85
query85	1350	300	291	291
query86	488	300	289	289
query87	4579	4226	4267	4226
query88	5584	2359	2410	2359
query89	547	286	288	286
query90	2084	185	190	185
query91	184	146	144	144
query92	66	47	49	47
query93	6974	535	534	534
query94	919	282	286	282
query95	352	251	250	250
query96	630	282	276	276
query97	3362	3115	3168	3115
query98	237	198	197	197
query99	1614	1296	1315	1296
Total cold run time: 338903 ms
Total hot run time: 195389 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants