[2024-11-19 12:44:33.313697] Starting SPDK v24.09.1-pre git sha1 b18e1bd62 / DPDK 22.11.4 initialization... [2024-11-19 12:44:33.313964] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3516399 ] Set seed to 1732016673307001 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-19 12:44:33.412387] app.c: 917:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-19 12:44:33.454227] app.c: 610:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-19 12:44:33.454278] app.c: 611:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 3516399' to capture a snapshot of events at runtime. [2024-11-19 12:44:33.454289] app.c: 616:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-19 12:44:33.454299] app.c: 617:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-19 12:44:33.454308] app.c: 618:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid3516399 for offline analysis/debug. [2024-11-19 12:44:33.454391] reactor.c: 990:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Tue Nov 19 12:44:37 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/19-12:44:42 ... thread 0: (1000000,1000000) ops and (199465.0,199465.0) ops/second in (5.013411,5.013411) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/18 3.07 GB 37.7 0.4 0.0 0.4 1.2 0.9 0.0 1.4 62.2 207.2 6.08 3.97 13 0.468 383K 536 0.0 0.0 L3 17/0 880.72 MB 0.9 0.2 0.1 0.1 0.2 0.1 0.0 1.5 204.7 204.5 0.93 0.85 1 0.929 130K 0 0.0 0.0 L4 164/0 9.41 GB 1.0 3.0 1.6 1.4 3.0 1.6 0.2 1.9 113.0 112.0 27.08 14.39 13 2.083 261K 0 0.0 0.0 L5 1120/0 92.77 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1320/18 106.10 GB 0.0 3.5 1.7 1.8 4.4 2.5 4.6 5.1 106.4 131.5 34.09 19.21 27 1.263 775K 536 0.0 0.0 Int 0/0 0.00 KB 0.0 3.5 1.7 1.8 4.4 2.5 4.6 5.1 106.4 131.5 34.09 19.21 27 1.263 775K 536 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 3.5 1.7 1.8 3.5 1.7 0.0 0.0 122.5 121.6 29.61 16.74 15 1.974 775K 536 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 196.7 4.48 2.47 12 0.374 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 5.2 total, 5.0 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 4.38 GB write, 868.14 MB/s write, 3.54 GB read, 702.74 MB/s read, 34.1 seconds Interval compaction: 4.38 GB write, 893.75 MB/s write, 3.54 GB read, 723.46 MB/s read, 34.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 1, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 1, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 4, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 1, total-stops: 4, interval: 5 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 311742 Average: 11.9481 StdDev: 134.23 Min: 1 Median: 1.7693 Max: 9679 Percentiles: P50: 1.77 P75: 2.59 P99: 98.34 P99.9: 1459.68 P99.99: 2349.87 ------------------------------------------------------ [ 0, 1 ] 19925 6.392% 6.392% # ( 1, 2 ] 176711 56.685% 63.077% ########### ( 2, 3 ] 62741 20.126% 83.202% #### ( 3, 4 ] 22011 7.061% 90.263% # ( 4, 6 ] 20604 6.609% 96.872% # ( 6, 10 ] 5537 1.776% 98.649% ( 10, 15 ] 913 0.293% 98.941% ( 15, 22 ] 104 0.033% 98.975% ( 22, 34 ] 13 0.004% 98.979% ( 34, 51 ] 8 0.003% 98.982% ( 51, 76 ] 7 0.002% 98.984% ( 76, 110 ] 80 0.026% 99.009% ( 110, 170 ] 386 0.124% 99.133% ( 170, 250 ] 439 0.141% 99.274% ( 250, 380 ] 444 0.142% 99.417% ( 380, 580 ] 521 0.167% 99.584% ( 580, 870 ] 511 0.164% 99.748% ( 870, 1300 ] 407 0.131% 99.878% ( 1300, 1900 ] 264 0.085% 99.963% ( 1900, 2900 ] 193 0.062% 100.025% ( 2900, 4400 ] 138 0.044% 100.069% ( 4400, 6600 ] 38 0.012% 100.081% ( 6600, 9900 ] 4 0.001% 100.082% ** Level 2 read latency histogram (micros): Count: 32721 Average: 4.8866 StdDev: 66.71 Min: 1 Median: 1.7796 Max: 7146 Percentiles: P50: 1.78 P75: 2.57 P99: 9.23 P99.9: 627.21 P99.99: 3445.92 ------------------------------------------------------ [ 0, 1 ] 898 2.744% 2.744% # ( 1, 2 ] 19833 60.612% 63.357% ############ ( 2, 3 ] 6692 20.452% 83.809% #### ( 3, 4 ] 2291 7.002% 90.810% # ( 4, 6 ] 2087 6.378% 97.188% # ( 6, 10 ] 734 2.243% 99.432% ( 10, 15 ] 71 0.217% 99.649% ( 15, 22 ] 5 0.015% 99.664% ( 34, 51 ] 1 0.003% 99.667% ( 51, 76 ] 1 0.003% 99.670% ( 110, 170 ] 13 0.040% 99.710% ( 170, 250 ] 16 0.049% 99.759% ( 250, 380 ] 23 0.070% 99.829% ( 380, 580 ] 21 0.064% 99.893% ( 580, 870 ] 14 0.043% 99.936% ( 870, 1300 ] 12 0.037% 99.972% ( 1300, 1900 ] 3 0.009% 99.982% ( 1900, 2900 ] 2 0.006% 99.988% ( 2900, 4400 ] 2 0.006% 99.994% ( 4400, 6600 ] 1 0.003% 99.997% ( 6600, 9900 ] 1 0.003% 100.000% ** Level 3 read latency histogram (micros): Count: 232490 Average: 11.6749 StdDev: 138.18 Min: 1 Median: 1.9114 Max: 9042 Percentiles: P50: 1.91 P75: 2.88 P99: 11.76 P99.9: 1393.52 P99.99: 2067.17 ------------------------------------------------------ [ 0, 1 ] 6313 2.715% 2.715% # ( 1, 2 ] 120614 51.879% 54.595% ########## ( 2, 3 ] 53852 23.163% 77.758% ##### ( 3, 4 ] 21690 9.329% 87.087% ## ( 4, 6 ] 21023 9.043% 96.130% ## ( 6, 10 ] 6304 2.712% 98.841% # ( 10, 15 ] 1046 0.450% 99.291% ( 15, 22 ] 160 0.069% 99.360% ( 22, 34 ] 14 0.006% 99.366% ( 34, 51 ] 16 0.007% 99.373% ( 51, 76 ] 6 0.003% 99.375% ( 76, 110 ] 12 0.005% 99.381% ( 110, 170 ] 56 0.024% 99.405% ( 170, 250 ] 91 0.039% 99.444% ( 250, 380 ] 162 0.070% 99.514% ( 380, 580 ] 252 0.108% 99.622% ( 580, 870 ] 297 0.128% 99.750% ( 870, 1300 ] 316 0.136% 99.886% ( 1300, 1900 ] 215 0.092% 99.978% ( 1900, 2900 ] 166 0.071% 100.049% ( 2900, 4400 ] 112 0.048% 100.098% ( 4400, 6600 ] 23 0.010% 100.108% ( 6600, 9900 ] 8 0.003% 100.111% ** Level 4 read latency histogram (micros): Count: 255609 Average: 23.9396 StdDev: 223.96 Min: 1 Median: 1.9364 Max: 9722 Percentiles: P50: 1.94 P75: 2.95 P99: 481.41 P99.9: 2182.07 P99.99: 2703.72 ------------------------------------------------------ [ 0, 1 ] 7281 2.848% 2.848% # ( 1, 2 ] 128706 50.353% 53.201% ########## ( 2, 3 ] 58645 22.943% 76.144% ##### ( 3, 4 ] 25039 9.796% 85.940% ## ( 4, 6 ] 23917 9.357% 95.297% ## ( 6, 10 ] 6824 2.670% 97.967% # ( 10, 15 ] 1259 0.493% 98.459% ( 15, 22 ] 185 0.072% 98.532% ( 22, 34 ] 31 0.012% 98.544% ( 34, 51 ] 12 0.005% 98.549% ( 51, 76 ] 11 0.004% 98.553% ( 76, 110 ] 63 0.025% 98.578% ( 110, 170 ] 194 0.076% 98.653% ( 170, 250 ] 282 0.110% 98.764% ( 250, 380 ] 323 0.126% 98.890% ( 380, 580 ] 554 0.217% 99.107% ( 580, 870 ] 673 0.263% 99.370% ( 870, 1300 ] 692 0.271% 99.641% ( 1300, 1900 ] 538 0.210% 99.851% ( 1900, 2900 ] 441 0.173% 100.024% ( 2900, 4400 ] 353 0.138% 100.162% ( 4400, 6600 ] 93 0.036% 100.198% ( 6600, 9900 ] 17 0.007% 100.205% ** Level 5 read latency histogram (micros): Count: 13202 Average: 1067.2442 StdDev: 1301.29 Min: 76 Median: 528.3487 Max: 11495 Percentiles: P50: 528.35 P75: 1282.63 P99: 6200.40 P99.9: 9367.25 P99.99: 11495.00 ------------------------------------------------------ ( 51, 76 ] 1 0.008% 0.008% ( 76, 110 ] 324 2.454% 2.462% ( 110, 170 ] 1508 11.423% 13.884% ## ( 170, 250 ] 1657 12.551% 26.435% ### ( 250, 380 ] 1742 13.195% 39.630% ### ( 380, 580 ] 1847 13.990% 53.621% ### ( 580, 870 ] 1600 12.119% 65.740% ## ( 870, 1300 ] 1275 9.658% 75.398% ## ( 1300, 1900 ] 889 6.734% 82.131% # ( 1900, 2900 ] 924 6.999% 89.130% # ( 2900, 4400 ] 984 7.453% 96.584% # ( 4400, 6600 ] 391 2.962% 99.546% # ( 6600, 9900 ] 57 0.432% 99.977% ( 9900, 14000 ] 4 0.030% 100.008% ** DB Stats ** Uptime(secs): 5.2 total, 5.0 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 190.62 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:1.275 H:M:S, 24.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 196.21 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.275 H:M:S, 25.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 3 num-running-flushes: 1 2024/11/19-12:44:48 ... thread 0: (1000000,2000000) ops and (169704.0,183384.9) ops/second in (5.892612,10.906023) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/17 4.11 GB 37.7 1.8 0.0 1.8 3.8 1.9 0.0 2.0 123.4 250.7 15.32 12.61 36 0.425 1919K 2025 0.0 0.0 L3 17/0 880.72 MB 0.9 0.2 0.1 0.1 0.2 0.1 0.0 1.5 204.7 204.5 0.93 0.85 1 0.929 130K 0 0.0 0.0 L4 164/0 9.41 GB 1.0 3.0 1.6 1.4 3.0 1.6 0.2 1.9 113.0 112.0 27.08 14.39 13 2.083 261K 0 0.0 0.0 L5 1120/0 92.77 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1319/17 107.15 GB 0.0 5.0 1.7 3.3 6.9 3.6 4.6 3.6 118.6 163.0 43.33 27.84 50 0.867 2311K 2025 0.0 0.0 Int 0/0 0.00 KB 0.0 5.0 1.7 3.3 6.9 3.6 4.6 3.6 118.6 163.0 43.33 27.84 50 0.867 2311K 2025 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 5.0 1.7 3.3 5.0 1.7 0.0 0.0 145.0 144.2 35.45 22.44 21 1.688 2311K 2025 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 247.7 7.88 5.40 29 0.272 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 11.1 total, 10.9 interval Flush(GB): cumulative 1.907, interval 1.907 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.90 GB write, 638.93 MB/s write, 5.02 GB read, 464.95 MB/s read, 43.3 seconds Interval compaction: 6.90 GB write, 647.59 MB/s write, 5.02 GB read, 471.26 MB/s read, 43.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 10, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 10, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 4, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 10, total-stops: 4, interval: 14 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 689187 Average: 16.6885 StdDev: 104.01 Min: 0 Median: 1.5997 Max: 9679 Percentiles: P50: 1.60 P75: 2.29 P99: 247.79 P99.9: 847.45 P99.99: 1449.10 ------------------------------------------------------ [ 0, 1 ] 125493 18.209% 18.209% #### ( 1, 2 ] 365325 53.008% 71.217% ########### ( 2, 3 ] 88947 12.906% 84.123% ### ( 3, 4 ] 31823 4.617% 88.741% # ( 4, 6 ] 26103 3.788% 92.528% # ( 6, 10 ] 6886 0.999% 93.527% ( 10, 15 ] 1271 0.184% 93.712% ( 15, 22 ] 181 0.026% 93.738% ( 22, 34 ] 26 0.004% 93.742% ( 34, 51 ] 10 0.001% 93.743% ( 51, 76 ] 23 0.003% 93.746% ( 76, 110 ] 6770 0.982% 94.729% ( 110, 170 ] 21931 3.182% 97.911% # ( 170, 250 ] 7719 1.120% 99.031% ( 250, 380 ] 2797 0.406% 99.437% ( 380, 580 ] 2289 0.332% 99.769% ( 580, 870 ] 980 0.142% 99.911% ( 870, 1300 ] 475 0.069% 99.980% ( 1300, 1900 ] 278 0.040% 100.020% ( 1900, 2900 ] 218 0.032% 100.052% ( 2900, 4400 ] 152 0.022% 100.074% ( 4400, 6600 ] 38 0.006% 100.080% ( 6600, 9900 ] 4 0.001% 100.080% ** Level 2 read latency histogram (micros): Count: 32721 Average: 4.8866 StdDev: 66.71 Min: 1 Median: 1.7796 Max: 7146 Percentiles: P50: 1.78 P75: 2.57 P99: 9.23 P99.9: 627.21 P99.99: 3445.92 ------------------------------------------------------ [ 0, 1 ] 898 2.744% 2.744% # ( 1, 2 ] 19833 60.612% 63.357% ############ ( 2, 3 ] 6692 20.452% 83.809% #### ( 3, 4 ] 2291 7.002% 90.810% # ( 4, 6 ] 2087 6.378% 97.188% # ( 6, 10 ] 734 2.243% 99.432% ( 10, 15 ] 71 0.217% 99.649% ( 15, 22 ] 5 0.015% 99.664% ( 34, 51 ] 1 0.003% 99.667% ( 51, 76 ] 1 0.003% 99.670% ( 110, 170 ] 13 0.040% 99.710% ( 170, 250 ] 16 0.049% 99.759% ( 250, 380 ] 23 0.070% 99.829% ( 380, 580 ] 21 0.064% 99.893% ( 580, 870 ] 14 0.043% 99.936% ( 870, 1300 ] 12 0.037% 99.972% ( 1300, 1900 ] 3 0.009% 99.982% ( 1900, 2900 ] 2 0.006% 99.988% ( 2900, 4400 ] 2 0.006% 99.994% ( 4400, 6600 ] 1 0.003% 99.997% ( 6600, 9900 ] 1 0.003% 100.000% ** Level 3 read latency histogram (micros): Count: 233877 Average: 12.4954 StdDev: 138.49 Min: 1 Median: 1.9170 Max: 9042 Percentiles: P50: 1.92 P75: 2.90 P99: 117.78 P99.9: 1392.01 P99.99: 2066.34 ------------------------------------------------------ [ 0, 1 ] 6313 2.699% 2.699% # ( 1, 2 ] 120638 51.582% 54.281% ########## ( 2, 3 ] 53899 23.046% 77.327% ##### ( 3, 4 ] 21758 9.303% 86.630% ## ( 4, 6 ] 21083 9.015% 95.645% ## ( 6, 10 ] 6314 2.700% 98.344% # ( 10, 15 ] 1050 0.449% 98.793% ( 15, 22 ] 160 0.068% 98.862% ( 22, 34 ] 14 0.006% 98.868% ( 34, 51 ] 16 0.007% 98.875% ( 51, 76 ] 6 0.003% 98.877% ( 76, 110 ] 200 0.086% 98.963% ( 110, 170 ] 673 0.288% 99.250% ( 170, 250 ] 328 0.140% 99.391% ( 250, 380 ] 230 0.098% 99.489% ( 380, 580 ] 295 0.126% 99.615% ( 580, 870 ] 316 0.135% 99.750% ( 870, 1300 ] 317 0.136% 99.886% ( 1300, 1900 ] 216 0.092% 99.978% ( 1900, 2900 ] 166 0.071% 100.049% ( 2900, 4400 ] 112 0.048% 100.097% ( 4400, 6600 ] 23 0.010% 100.107% ( 6600, 9900 ] 8 0.003% 100.110% ** Level 4 read latency histogram (micros): Count: 267660 Average: 30.1750 StdDev: 222.23 Min: 1 Median: 1.9827 Max: 9722 Percentiles: P50: 1.98 P75: 3.23 P99: 534.24 P99.9: 2165.34 P99.99: 2705.46 ------------------------------------------------------ [ 0, 1 ] 7281 2.720% 2.720% # ( 1, 2 ] 128780 48.113% 50.834% ########## ( 2, 3 ] 58923 22.014% 72.848% #### ( 3, 4 ] 25229 9.426% 82.273% ## ( 4, 6 ] 24203 9.042% 91.316% ## ( 6, 10 ] 6852 2.560% 93.876% # ( 10, 15 ] 1273 0.476% 94.351% ( 15, 22 ] 188 0.070% 94.422% ( 22, 34 ] 33 0.012% 94.434% ( 34, 51 ] 12 0.004% 94.438% ( 51, 76 ] 11 0.004% 94.443% ( 76, 110 ] 1909 0.713% 95.156% ( 110, 170 ] 6355 2.374% 97.530% ( 170, 250 ] 2209 0.825% 98.355% ( 250, 380 ] 938 0.350% 98.706% ( 380, 580 ] 1021 0.381% 99.087% ( 580, 870 ] 811 0.303% 99.390% ( 870, 1300 ] 703 0.263% 99.653% ( 1300, 1900 ] 543 0.203% 99.856% ( 1900, 2900 ] 446 0.167% 100.022% ( 2900, 4400 ] 354 0.132% 100.155% ( 4400, 6600 ] 93 0.035% 100.189% ( 6600, 9900 ] 17 0.006% 100.196% ** Level 5 read latency histogram (micros): Count: 97435 Average: 295.7684 StdDev: 578.51 Min: 75 Median: 153.1641 Max: 11495 Percentiles: P50: 153.16 P75: 224.43 P99: 3605.44 P99.9: 6389.37 P99.99: 9509.59 ------------------------------------------------------ ( 51, 76 ] 12 0.012% 0.012% ( 76, 110 ] 14490 14.871% 14.884% ### ( 110, 170 ] 47561 48.813% 63.697% ########## ( 170, 250 ] 16187 16.613% 80.310% ### ( 250, 380 ] 6414 6.583% 86.893% # ( 380, 580 ] 5478 5.622% 92.515% # ( 580, 870 ] 2564 2.631% 95.147% # ( 870, 1300 ] 1412 1.449% 96.596% ( 1300, 1900 ] 917 0.941% 97.537% ( 1900, 2900 ] 961 0.986% 98.523% ( 2900, 4400 ] 988 1.014% 99.537% ( 4400, 6600 ] 391 0.401% 99.938% ( 6600, 9900 ] 57 0.059% 99.997% ( 9900, 14000 ] 4 0.004% 100.001% ** DB Stats ** Uptime(secs): 11.1 total, 5.9 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 178.04 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.662 H:M:S, 33.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 167.02 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.387 H:M:S, 40.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 0 2024/11/19-12:44:58 ... thread 0: (1000000,3000000) ops and (105410.7,147111.3) ops/second in (9.486702,20.392725) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/10 5.02 GB 288.5 4.1 0.0 4.1 6.9 2.8 0.0 2.5 155.9 262.4 27.07 23.82 55 0.492 4284K 13K 0.0 0.0 L3 17/0 880.72 MB 0.9 0.2 0.1 0.1 0.2 0.1 0.0 1.5 204.7 204.5 0.93 0.85 1 0.929 130K 0 0.0 0.0 L4 164/0 9.41 GB 1.0 3.0 1.6 1.4 3.0 1.6 0.2 1.9 113.0 112.0 27.08 14.39 13 2.083 261K 0 0.0 0.0 L5 1120/0 92.77 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1319/10 108.06 GB 0.0 7.3 1.7 5.6 10.1 4.5 4.6 3.6 135.6 187.5 55.09 39.06 69 0.798 4676K 13K 0.0 0.0 Int 0/0 0.00 KB 0.0 7.3 1.7 5.6 10.1 4.5 4.6 3.6 135.6 187.5 55.09 39.06 69 0.798 4676K 13K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 7.3 1.7 5.6 7.3 1.7 0.0 0.0 169.0 168.0 44.21 31.06 25 1.768 4676K 13K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 266.4 10.88 8.00 44 0.247 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 20.5 total, 20.4 interval Flush(GB): cumulative 2.830, interval 2.830 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 10.08 GB write, 502.72 MB/s write, 7.29 GB read, 363.63 MB/s read, 55.1 seconds Interval compaction: 10.08 GB write, 506.37 MB/s write, 7.29 GB read, 366.27 MB/s read, 55.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 17, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 17, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 4, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 17, total-stops: 4, interval: 21 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1370419 Average: 18.1549 StdDev: 86.56 Min: 0 Median: 1.4911 Max: 9679 Percentiles: P50: 1.49 P75: 1.98 P99: 247.62 P99.9: 665.44 P99.99: 911.73 ------------------------------------------------------ [ 0, 1 ] 341792 24.941% 24.941% ##### ( 1, 2 ] 699222 51.022% 75.963% ########## ( 2, 3 ] 124536 9.087% 85.051% ## ( 3, 4 ] 46835 3.418% 88.468% # ( 4, 6 ] 33056 2.412% 90.880% ( 6, 10 ] 8339 0.609% 91.489% ( 10, 15 ] 1792 0.131% 91.620% ( 15, 22 ] 270 0.020% 91.639% ( 22, 34 ] 46 0.003% 91.643% ( 34, 51 ] 14 0.001% 91.644% ( 51, 76 ] 32 0.002% 91.646% ( 76, 110 ] 18241 1.331% 92.977% ( 110, 170 ] 62555 4.565% 97.542% # ( 170, 250 ] 20598 1.503% 99.045% ( 250, 380 ] 6366 0.465% 99.509% ( 380, 580 ] 4862 0.355% 99.864% ( 580, 870 ] 1672 0.122% 99.986% ( 870, 1300 ] 556 0.041% 100.027% ( 1300, 1900 ] 301 0.022% 100.049% ( 1900, 2900 ] 239 0.017% 100.066% ( 2900, 4400 ] 165 0.012% 100.078% ( 4400, 6600 ] 39 0.003% 100.081% ( 6600, 9900 ] 4 0.000% 100.081% ** Level 2 read latency histogram (micros): Count: 32721 Average: 4.8866 StdDev: 66.71 Min: 1 Median: 1.7796 Max: 7146 Percentiles: P50: 1.78 P75: 2.57 P99: 9.23 P99.9: 627.21 P99.99: 3445.92 ------------------------------------------------------ [ 0, 1 ] 898 2.744% 2.744% # ( 1, 2 ] 19833 60.612% 63.357% ############ ( 2, 3 ] 6692 20.452% 83.809% #### ( 3, 4 ] 2291 7.002% 90.810% # ( 4, 6 ] 2087 6.378% 97.188% # ( 6, 10 ] 734 2.243% 99.432% ( 10, 15 ] 71 0.217% 99.649% ( 15, 22 ] 5 0.015% 99.664% ( 34, 51 ] 1 0.003% 99.667% ( 51, 76 ] 1 0.003% 99.670% ( 110, 170 ] 13 0.040% 99.710% ( 170, 250 ] 16 0.049% 99.759% ( 250, 380 ] 23 0.070% 99.829% ( 380, 580 ] 21 0.064% 99.893% ( 580, 870 ] 14 0.043% 99.936% ( 870, 1300 ] 12 0.037% 99.972% ( 1300, 1900 ] 3 0.009% 99.982% ( 1900, 2900 ] 2 0.006% 99.988% ( 2900, 4400 ] 2 0.006% 99.994% ( 4400, 6600 ] 1 0.003% 99.997% ( 6600, 9900 ] 1 0.003% 100.000% ** Level 3 read latency histogram (micros): Count: 236098 Average: 13.7013 StdDev: 138.78 Min: 1 Median: 1.9258 Max: 9042 Percentiles: P50: 1.93 P75: 2.93 P99: 165.68 P99.9: 1385.84 P99.99: 2065.00 ------------------------------------------------------ [ 0, 1 ] 6313 2.674% 2.674% # ( 1, 2 ] 120686 51.117% 53.791% ########## ( 2, 3 ] 53998 22.871% 76.662% ##### ( 3, 4 ] 21833 9.247% 85.909% ## ( 4, 6 ] 21205 8.981% 94.891% ## ( 6, 10 ] 6326 2.679% 97.570% # ( 10, 15 ] 1051 0.445% 98.015% ( 15, 22 ] 161 0.068% 98.083% ( 22, 34 ] 14 0.006% 98.089% ( 34, 51 ] 16 0.007% 98.096% ( 51, 76 ] 6 0.003% 98.099% ( 76, 110 ] 517 0.219% 98.318% ( 110, 170 ] 1736 0.735% 99.053% ( 170, 250 ] 642 0.272% 99.325% ( 250, 380 ] 320 0.136% 99.460% ( 380, 580 ] 355 0.150% 99.611% ( 580, 870 ] 330 0.140% 99.751% ( 870, 1300 ] 322 0.136% 99.887% ( 1300, 1900 ] 216 0.091% 99.978% ( 1900, 2900 ] 166 0.070% 100.049% ( 2900, 4400 ] 112 0.047% 100.096% ( 4400, 6600 ] 23 0.010% 100.106% ( 6600, 9900 ] 8 0.003% 100.109% ** Level 4 read latency histogram (micros): Count: 287362 Average: 39.4520 StdDev: 219.76 Min: 1 Median: 2.1278 Max: 9722 Percentiles: P50: 2.13 P75: 3.80 P99: 555.38 P99.9: 2128.59 P99.99: 2704.60 ------------------------------------------------------ [ 0, 1 ] 7281 2.534% 2.534% # ( 1, 2 ] 128851 44.839% 47.373% ######### ( 2, 3 ] 59069 20.556% 67.929% #### ( 3, 4 ] 25378 8.831% 76.760% ## ( 4, 6 ] 24376 8.483% 85.243% ## ( 6, 10 ] 6878 2.393% 87.636% ( 10, 15 ] 1275 0.444% 88.080% ( 15, 22 ] 189 0.066% 88.146% ( 22, 34 ] 33 0.011% 88.157% ( 34, 51 ] 12 0.004% 88.161% ( 51, 76 ] 13 0.005% 88.166% ( 76, 110 ] 5250 1.827% 89.993% ( 110, 170 ] 17077 5.943% 95.935% # ( 170, 250 ] 5378 1.872% 97.807% ( 250, 380 ] 1879 0.654% 98.461% ( 380, 580 ] 1768 0.615% 99.076% ( 580, 870 ] 990 0.345% 99.421% ( 870, 1300 ] 727 0.253% 99.674% ( 1300, 1900 ] 549 0.191% 99.865% ( 1900, 2900 ] 449 0.156% 100.021% ( 2900, 4400 ] 355 0.124% 100.144% ( 4400, 6600 ] 93 0.032% 100.177% ( 6600, 9900 ] 18 0.006% 100.183% ** Level 5 read latency histogram (micros): Count: 237218 Average: 220.3716 StdDev: 384.36 Min: 72 Median: 147.6690 Max: 11495 Percentiles: P50: 147.67 P75: 194.96 P99: 1953.46 P99.9: 5569.11 P99.99: 8352.95 ------------------------------------------------------ ( 51, 76 ] 38 0.016% 0.016% ( 76, 110 ] 39225 16.535% 16.551% ### ( 110, 170 ] 126384 53.278% 69.829% ########### ( 170, 250 ] 39314 16.573% 86.402% ### ( 250, 380 ] 12845 5.415% 91.817% # ( 380, 580 ] 10692 4.507% 96.324% # ( 580, 870 ] 3784 1.595% 97.919% ( 870, 1300 ] 1556 0.656% 98.575% ( 1300, 1900 ] 955 0.403% 98.978% ( 1900, 2900 ] 988 0.416% 99.394% ( 2900, 4400 ] 992 0.418% 99.812% ( 4400, 6600 ] 391 0.165% 99.977% ( 6600, 9900 ] 57 0.024% 100.001% ( 9900, 14000 ] 4 0.002% 100.003% ** DB Stats ** Uptime(secs): 20.5 total, 9.5 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 143.73 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:9.491 H:M:S, 46.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 103.74 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:5.829 H:M:S, 61.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 2024/11/19-12:45:22 ... thread 0: (1000000,4000000) ops and (41757.4,90210.9) ops/second in (23.947839,44.340564) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/10 6.00 GB 388.5 6.9 0.0 6.9 10.7 3.8 0.0 2.8 172.4 266.6 41.26 37.44 77 0.536 7225K 21K 0.0 0.0 L3 17/0 880.72 MB 0.9 0.2 0.1 0.1 0.2 0.1 0.0 1.5 204.7 204.5 0.93 0.85 1 0.929 130K 0 0.0 0.0 L4 164/0 9.41 GB 1.0 3.0 1.6 1.4 3.0 1.6 0.2 1.9 113.0 112.0 27.08 14.39 13 2.083 261K 0 0.0 0.0 L5 1120/0 92.77 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1316/10 109.04 GB 0.0 10.1 1.7 8.4 13.9 5.5 4.6 3.6 149.6 205.3 69.28 52.67 91 0.761 7617K 21K 0.0 0.0 Int 0/0 0.00 KB 0.0 10.1 1.7 8.4 13.9 5.5 4.6 3.6 149.6 205.3 69.28 52.67 91 0.761 7617K 21K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 10.1 1.7 8.4 10.1 1.7 0.0 0.0 188.0 187.1 55.13 41.82 31 1.778 7617K 21K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 276.1 14.15 10.85 60 0.236 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 44.5 total, 44.3 interval Flush(GB): cumulative 3.815, interval 3.815 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 13.89 GB write, 319.69 MB/s write, 10.12 GB read, 232.99 MB/s read, 69.3 seconds Interval compaction: 13.89 GB write, 320.76 MB/s write, 10.12 GB read, 233.77 MB/s read, 69.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 27, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 27, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 4, pending-compaction-bytes-delays: 6, pending-compaction-bytes-stops: 0, total-delays: 33, total-stops: 4, interval: 37 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 2360010 Average: 24.0932 StdDev: 82.21 Min: 0 Median: 1.5257 Max: 9679 Percentiles: P50: 1.53 P75: 2.26 P99: 247.45 P99.9: 577.21 P99.99: 802.95 ------------------------------------------------------ [ 0, 1 ] 579250 24.544% 24.544% ##### ( 1, 2 ] 1142813 48.424% 72.968% ########## ( 2, 3 ] 184702 7.826% 80.795% ## ( 3, 4 ] 77276 3.274% 84.069% # ( 4, 6 ] 47771 2.024% 86.093% ( 6, 10 ] 10896 0.462% 86.555% ( 10, 15 ] 2650 0.112% 86.667% ( 15, 22 ] 458 0.019% 86.687% ( 22, 34 ] 102 0.004% 86.691% ( 34, 51 ] 18 0.001% 86.692% ( 51, 76 ] 94 0.004% 86.696% ( 76, 110 ] 58105 2.462% 89.158% ( 110, 170 ] 187858 7.960% 97.118% ## ( 170, 250 ] 45880 1.944% 99.062% ( 250, 380 ] 11373 0.482% 99.544% ( 380, 580 ] 8524 0.361% 99.905% ( 580, 870 ] 2608 0.111% 100.016% ( 870, 1300 ] 682 0.029% 100.044% ( 1300, 1900 ] 327 0.014% 100.058% ( 1900, 2900 ] 295 0.012% 100.071% ( 2900, 4400 ] 209 0.009% 100.080% ( 4400, 6600 ] 40 0.002% 100.081% ( 6600, 9900 ] 4 0.000% 100.082% ** Level 2 read latency histogram (micros): Count: 32721 Average: 4.8866 StdDev: 66.71 Min: 1 Median: 1.7796 Max: 7146 Percentiles: P50: 1.78 P75: 2.57 P99: 9.23 P99.9: 627.21 P99.99: 3445.92 ------------------------------------------------------ [ 0, 1 ] 898 2.744% 2.744% # ( 1, 2 ] 19833 60.612% 63.357% ############ ( 2, 3 ] 6692 20.452% 83.809% #### ( 3, 4 ] 2291 7.002% 90.810% # ( 4, 6 ] 2087 6.378% 97.188% # ( 6, 10 ] 734 2.243% 99.432% ( 10, 15 ] 71 0.217% 99.649% ( 15, 22 ] 5 0.015% 99.664% ( 34, 51 ] 1 0.003% 99.667% ( 51, 76 ] 1 0.003% 99.670% ( 110, 170 ] 13 0.040% 99.710% ( 170, 250 ] 16 0.049% 99.759% ( 250, 380 ] 23 0.070% 99.829% ( 380, 580 ] 21 0.064% 99.893% ( 580, 870 ] 14 0.043% 99.936% ( 870, 1300 ] 12 0.037% 99.972% ( 1300, 1900 ] 3 0.009% 99.982% ( 1900, 2900 ] 2 0.006% 99.988% ( 2900, 4400 ] 2 0.006% 99.994% ( 4400, 6600 ] 1 0.003% 99.997% ( 6600, 9900 ] 1 0.003% 100.000% ** Level 3 read latency histogram (micros): Count: 242577 Average: 17.2870 StdDev: 139.17 Min: 1 Median: 1.9526 Max: 9042 Percentiles: P50: 1.95 P75: 3.04 P99: 218.12 P99.9: 1370.29 P99.99: 2061.10 ------------------------------------------------------ [ 0, 1 ] 6313 2.602% 2.602% # ( 1, 2 ] 120692 49.754% 52.357% ########## ( 2, 3 ] 54005 22.263% 74.620% #### ( 3, 4 ] 21843 9.005% 83.624% ## ( 4, 6 ] 21213 8.745% 92.369% ## ( 6, 10 ] 6327 2.608% 94.977% # ( 10, 15 ] 1051 0.433% 95.411% ( 15, 22 ] 161 0.066% 95.477% ( 22, 34 ] 14 0.006% 95.483% ( 34, 51 ] 16 0.007% 95.489% ( 51, 76 ] 7 0.003% 95.492% ( 76, 110 ] 1931 0.796% 96.288% ( 110, 170 ] 5765 2.377% 98.665% ( 170, 250 ] 1352 0.557% 99.222% ( 250, 380 ] 466 0.192% 99.414% ( 380, 580 ] 462 0.190% 99.605% ( 580, 870 ] 365 0.150% 99.755% ( 870, 1300 ] 326 0.134% 99.890% ( 1300, 1900 ] 217 0.089% 99.979% ( 1900, 2900 ] 166 0.068% 100.047% ( 2900, 4400 ] 112 0.046% 100.094% ( 4400, 6600 ] 23 0.009% 100.103% ( 6600, 9900 ] 8 0.003% 100.106% ** Level 4 read latency histogram (micros): Count: 343972 Average: 57.2044 StdDev: 207.22 Min: 1 Median: 2.6070 Max: 9722 Percentiles: P50: 2.61 P75: 84.56 P99: 542.46 P99.9: 2013.14 P99.99: 2699.56 ------------------------------------------------------ [ 0, 1 ] 7281 2.117% 2.117% ( 1, 2 ] 128851 37.460% 39.576% ####### ( 2, 3 ] 59069 17.173% 56.749% ### ( 3, 4 ] 25378 7.378% 64.127% # ( 4, 6 ] 24376 7.087% 71.214% # ( 6, 10 ] 6878 2.000% 73.213% ( 10, 15 ] 1275 0.371% 73.584% ( 15, 22 ] 189 0.055% 73.639% ( 22, 34 ] 33 0.010% 73.648% ( 34, 51 ] 12 0.003% 73.652% ( 51, 76 ] 38 0.011% 73.663% ( 76, 110 ] 18263 5.309% 78.972% # ( 110, 170 ] 51909 15.091% 94.063% ### ( 170, 250 ] 11642 3.385% 97.448% # ( 250, 380 ] 3137 0.912% 98.360% ( 380, 580 ] 2710 0.788% 99.148% ( 580, 870 ] 1223 0.356% 99.503% ( 870, 1300 ] 761 0.221% 99.725% ( 1300, 1900 ] 552 0.160% 99.885% ( 1900, 2900 ] 451 0.131% 100.016% ( 2900, 4400 ] 360 0.105% 100.121% ( 4400, 6600 ] 93 0.027% 100.148% ( 6600, 9900 ] 18 0.005% 100.153% ** Level 5 read latency histogram (micros): Count: 640766 Average: 174.4074 StdDev: 245.12 Min: 71 Median: 140.0340 Max: 11495 Percentiles: P50: 140.03 P75: 165.77 P99: 809.51 P99.9: 4091.52 P99.99: 6459.26 ------------------------------------------------------ ( 51, 76 ] 237 0.037% 0.037% ( 76, 110 ] 133170 20.783% 20.820% #### ( 110, 170 ] 373529 58.294% 79.114% ############ ( 170, 250 ] 83913 13.096% 92.210% ### ( 250, 380 ] 21773 3.398% 95.608% # ( 380, 580 ] 17420 2.719% 98.326% # ( 580, 870 ] 5454 0.851% 99.178% ( 870, 1300 ] 1778 0.277% 99.455% ( 1300, 1900 ] 1005 0.157% 99.612% ( 1900, 2900 ] 1036 0.162% 99.774% ( 2900, 4400 ] 1020 0.159% 99.933% ( 4400, 6600 ] 392 0.061% 99.994% ( 6600, 9900 ] 57 0.009% 100.003% ( 9900, 14000 ] 4 0.001% 100.003% ** DB Stats ** Uptime(secs): 44.5 total, 23.9 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 88.48 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:29.172 H:M:S, 65.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 41.10 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:19.681 H:M:S, 82.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 213.463 micros/op 18710 ops/sec 60.179 seconds 1125996 operations; 18.1 MB/s (271999 of 271999 found) Microseconds per read: Count: 1125996 Average: 213.4650 StdDev: 536.27 Min: 2 Median: 167.1419 Max: 53417 Percentiles: P50: 167.14 P75: 229.19 P99: 743.32 P99.9: 5893.99 P99.99: 25740.04 ------------------------------------------------------ ( 1, 2 ] 18 0.002% 0.002% ( 2, 3 ] 74 0.007% 0.008% ( 3, 4 ] 63 0.006% 0.014% ( 4, 6 ] 97 0.009% 0.022% ( 6, 10 ] 899 0.080% 0.102% ( 10, 15 ] 6537 0.581% 0.683% ( 15, 22 ] 3289 0.292% 0.975% ( 22, 34 ] 8612 0.765% 1.740% ( 34, 51 ] 1819 0.162% 1.901% ( 51, 76 ] 240 0.021% 1.923% ( 76, 110 ] 34328 3.049% 4.971% # ( 110, 170 ] 532382 47.281% 52.252% ######### ( 170, 250 ] 346220 30.748% 83.000% ###### ( 250, 380 ] 127966 11.365% 94.365% ## ( 380, 580 ] 44771 3.976% 98.341% # ( 580, 870 ] 13177 1.170% 99.511% ( 870, 1300 ] 2963 0.263% 99.774% ( 1300, 1900 ] 567 0.050% 99.825% ( 1900, 2900 ] 419 0.037% 99.862% ( 2900, 4400 ] 283 0.025% 99.887% ( 4400, 6600 ] 215 0.019% 99.906% ( 6600, 9900 ] 300 0.027% 99.933% ( 9900, 14000 ] 260 0.023% 99.956% ( 14000, 22000 ] 347 0.031% 99.987% ( 22000, 33000 ] 110 0.010% 99.996% ( 33000, 50000 ] 35 0.003% 100.000% ( 50000, 75000 ] 5 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 19350889 rocksdb.bloom.filter.full.positive COUNT : 1313041 rocksdb.bloom.filter.full.true.positive COUNT : 1125533 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 463 rocksdb.memtable.miss COUNT : 1125533 rocksdb.l0.hit COUNT : 62673 rocksdb.l1.hit COUNT : 0 rocksdb.l2andup.hit COUNT : 1062860 rocksdb.compaction.key.drop.new COUNT : 63842 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 4058880 rocksdb.number.keys.read COUNT : 1125996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 4188764160 rocksdb.bytes.read COUNT : 1125996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1443 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 44467778 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 12889919590 rocksdb.compact.write.bytes COUNT : 12824853766 rocksdb.flush.write.bytes COUNT : 4161899676 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 68 rocksdb.number.superversion_acquires COUNT : 678 rocksdb.number.superversion_releases COUNT : 643 rocksdb.number.superversion_cleanups COUNT : 168 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 4128065536 rocksdb.memtable.garbage.bytes.at.flush COUNT : 1355776 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 17751181701 rocksdb.non.last.level.read.count COUNT : 4144193 rocksdb.block.checksum.compute.count COUNT : 4447049 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 166.281968 P95 : 410.192637 P99 : 742.018001 P100 : 53416.000000 COUNT : 1125996 SUM : 239392641 rocksdb.db.write.micros P50 : 1.530542 P95 : 5.065108 P99 : 14.281088 P100 : 789239.000000 COUNT : 4058880 SUM : 54873296 rocksdb.compaction.times.micros P50 : 1410000.000000 P95 : 4043333.333333 P99 : 5818000.000000 P100 : 5889331.000000 COUNT : 31 SUM : 55127382 rocksdb.compaction.times.cpu_micros P50 : 1059565.217391 P95 : 3915000.000000 P99 : 5796331.000000 P100 : 5796331.000000 COUNT : 31 SUM : 41822628 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 793.472222 P95 : 3612.500000 P99 : 59750.000000 P100 : 72608.000000 COUNT : 61 SUM : 138950 rocksdb.compaction.outfile.sync.micros P50 : 4400.000000 P95 : 160000.000000 P99 : 206704.000000 P100 : 206704.000000 COUNT : 90 SUM : 2247200 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 143.157895 P95 : 3950.000000 P99 : 7406.000000 P100 : 7406.000000 COUNT : 108 SUM : 77545 rocksdb.table.open.io.micros P50 : 7842.296512 P95 : 19512.299465 P99 : 21981.604278 P100 : 27899.000000 COUNT : 1443 SUM : 12420684 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.684632 P95 : 141.361239 P99 : 235.803138 P100 : 9727.000000 COUNT : 3121935 SUM : 67916926 rocksdb.read.block.get.micros P50 : 138.608989 P95 : 262.949057 P99 : 572.752988 P100 : 15620.000000 COUNT : 1323620 SUM : 204881949 rocksdb.write.raw.block.micros P50 : 0.547285 P95 : 1.561395 P99 : 3.659272 P100 : 12308.000000 COUNT : 4114086 SUM : 10021511 rocksdb.numfiles.in.singlecompaction P50 : 3.500000 P95 : 5.650000 P99 : 9.060000 P100 : 10.000000 COUNT : 47 SUM : 192 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.573765 P95 : 1275.954122 P99 : 2680.025257 P100 : 789086.000000 COUNT : 235753 SUM : 44491368 rocksdb.sst.read.micros P50 : 2.199005 P95 : 198.171444 P99 : 440.120637 P100 : 11495.000000 COUNT : 4144218 SUM : 257744118 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1125996 SUM : 1125996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 4058880 SUM : 4188764160 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 216037.735849 P95 : 440166.666667 P99 : 992600.000000 P100 : 1065063.000000 COUNT : 61 SUM : 14381823 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 126 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0