[2024-12-16 11:00:56.235707] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 23.11.0 initialization... [2024-12-16 11:00:56.235849] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid977865 ] Set seed to 1734343256232163 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-16 11:00:56.336825] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-16 11:00:56.376994] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-16 11:00:56.377041] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 977865' to capture a snapshot of events at runtime. [2024-12-16 11:00:56.377054] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid977865 for offline analysis/debug. [2024-12-16 11:00:56.377077] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Mon Dec 16 11:01:00 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/16-11:01:06 ... thread 0: (1000000,1000000) ops and (168062.1,168062.1) ops/second in (5.950181,5.950181) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/15 2.35 GB 37.8 0.2 0.0 0.2 1.1 0.9 0.0 1.3 41.1 184.9 6.13 3.40 14 0.438 255K 224 0.0 0.0 L2 8/8 976.98 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 14/1 967.73 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 168/0 9.76 GB 1.0 3.4 2.0 1.4 3.4 2.0 0.2 1.7 105.5 104.2 33.05 16.00 19 1.740 511K 0 0.0 0.0 L5 1112/0 91.60 GB 0.9 0.0 0.0 0.0 0.0 0.0 3.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1319/24 105.61 GB 0.0 3.7 2.0 1.6 4.5 2.8 4.2 5.2 95.4 116.8 39.18 19.40 33 1.187 767K 224 0.0 0.0 Int 0/0 0.00 KB 0.0 3.7 2.0 1.6 4.5 2.8 4.1 5.2 95.4 116.8 39.18 19.40 33 1.187 767K 224 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 3.7 2.0 1.6 3.6 2.0 0.0 0.0 109.4 108.1 34.19 16.98 20 1.709 767K 224 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 176.5 5.00 2.42 13 0.384 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 6.1 total, 6.0 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 4.47 GB write, 749.18 MB/s write, 3.65 GB read, 611.87 MB/s read, 39.2 seconds Interval compaction: 4.47 GB write, 769.14 MB/s write, 3.65 GB read, 628.18 MB/s read, 39.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 1, pending-compaction-bytes-delays: 12, pending-compaction-bytes-stops: 0, total-delays: 12, total-stops: 1, interval: 13 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 320182 Average: 10.1624 StdDev: 117.05 Min: 1 Median: 1.7017 Max: 9097 Percentiles: P50: 1.70 P75: 2.52 P99: 112.31 P99.9: 1313.70 P99.99: 2535.52 ------------------------------------------------------ [ 0, 1 ] 32737 10.224% 10.224% ## ( 1, 2 ] 181506 56.688% 66.913% ########### ( 2, 3 ] 49596 15.490% 82.403% ### ( 3, 4 ] 24263 7.578% 89.981% ## ( 4, 6 ] 21719 6.783% 96.764% # ( 6, 10 ] 5816 1.816% 98.580% ( 10, 15 ] 1002 0.313% 98.893% ( 15, 22 ] 155 0.048% 98.942% ( 22, 34 ] 31 0.010% 98.952% ( 34, 51 ] 9 0.003% 98.954% ( 51, 76 ] 10 0.003% 98.957% ( 76, 110 ] 115 0.036% 98.993% ( 110, 170 ] 576 0.180% 99.173% ( 170, 250 ] 554 0.173% 99.346% ( 250, 380 ] 440 0.137% 99.484% ( 380, 580 ] 512 0.160% 99.644% ( 580, 870 ] 472 0.147% 99.791% ( 870, 1300 ] 345 0.108% 99.899% ( 1300, 1900 ] 211 0.066% 99.965% ( 1900, 2900 ] 129 0.040% 100.005% ( 2900, 4400 ] 88 0.027% 100.032% ( 4400, 6600 ] 28 0.009% 100.041% ( 6600, 9900 ] 9 0.003% 100.044% ** Level 2 read latency histogram (micros): Count: 199495 Average: 10.9094 StdDev: 130.48 Min: 1 Median: 1.8203 Max: 8737 Percentiles: P50: 1.82 P75: 2.80 P99: 13.99 P99.9: 2073.77 P99.99: 5189.68 ------------------------------------------------------ [ 0, 1 ] 11480 5.755% 5.755% # ( 1, 2 ] 107603 53.938% 59.692% ########### ( 2, 3 ] 38049 19.073% 78.765% #### ( 3, 4 ] 17429 8.737% 87.501% ## ( 4, 6 ] 16968 8.505% 96.007% ## ( 6, 10 ] 5115 2.564% 98.571% # ( 10, 15 ] 1072 0.537% 99.108% ( 15, 22 ] 177 0.089% 99.197% ( 22, 34 ] 27 0.014% 99.211% ( 34, 51 ] 14 0.007% 99.218% ( 51, 76 ] 14 0.007% 99.225% ( 76, 110 ] 21 0.011% 99.235% ( 110, 170 ] 90 0.045% 99.280% ( 170, 250 ] 129 0.065% 99.345% ( 250, 380 ] 179 0.090% 99.435% ( 380, 580 ] 232 0.116% 99.551% ( 580, 870 ] 279 0.140% 99.691% ( 870, 1300 ] 235 0.118% 99.809% ( 1300, 1900 ] 162 0.081% 99.890% ( 1900, 2900 ] 118 0.059% 99.949% ( 2900, 4400 ] 72 0.036% 99.985% ( 4400, 6600 ] 28 0.014% 99.999% ( 6600, 9900 ] 3 0.002% 100.001% ** Level 3 read latency histogram (micros): Count: 336974 Average: 11.1077 StdDev: 133.94 Min: 1 Median: 1.8237 Max: 9074 Percentiles: P50: 1.82 P75: 2.86 P99: 10.84 P99.9: 1171.90 P99.99: 1625.26 ------------------------------------------------------ [ 0, 1 ] 20184 5.990% 5.990% # ( 1, 2 ] 180053 53.432% 59.422% ########### ( 2, 3 ] 60700 18.013% 77.435% #### ( 3, 4 ] 31541 9.360% 86.795% ## ( 4, 6 ] 32047 9.510% 96.306% ## ( 6, 10 ] 8797 2.611% 98.916% # ( 10, 15 ] 1686 0.500% 99.417% ( 15, 22 ] 239 0.071% 99.487% ( 22, 34 ] 34 0.010% 99.498% ( 34, 51 ] 13 0.004% 99.501% ( 51, 76 ] 7 0.002% 99.504% ( 76, 110 ] 16 0.005% 99.508% ( 110, 170 ] 74 0.022% 99.530% ( 170, 250 ] 112 0.033% 99.563% ( 250, 380 ] 152 0.045% 99.609% ( 380, 580 ] 246 0.073% 99.682% ( 580, 870 ] 425 0.126% 99.808% ( 870, 1300 ] 443 0.131% 99.939% ( 1300, 1900 ] 316 0.094% 100.033% ( 1900, 2900 ] 243 0.072% 100.105% ( 2900, 4400 ] 153 0.045% 100.150% ( 4400, 6600 ] 32 0.009% 100.160% ( 6600, 9900 ] 6 0.002% 100.162% ** Level 4 read latency histogram (micros): Count: 265638 Average: 24.9377 StdDev: 219.27 Min: 1 Median: 1.8098 Max: 12095 Percentiles: P50: 1.81 P75: 2.86 P99: 695.60 P99.9: 2702.22 P99.99: 3619.23 ------------------------------------------------------ [ 0, 1 ] 17922 6.747% 6.747% # ( 1, 2 ] 141888 53.414% 60.161% ########### ( 2, 3 ] 45716 17.210% 77.371% ### ( 3, 4 ] 24549 9.242% 86.612% ## ( 4, 6 ] 22684 8.539% 95.152% ## ( 6, 10 ] 6295 2.370% 97.521% ( 10, 15 ] 1189 0.448% 97.969% ( 15, 22 ] 169 0.064% 98.033% ( 22, 34 ] 32 0.012% 98.045% ( 34, 51 ] 9 0.003% 98.048% ( 51, 76 ] 11 0.004% 98.052% ( 76, 110 ] 73 0.027% 98.080% ( 110, 170 ] 391 0.147% 98.227% ( 170, 250 ] 480 0.181% 98.408% ( 250, 380 ] 467 0.176% 98.583% ( 380, 580 ] 765 0.288% 98.871% ( 580, 870 ] 857 0.323% 99.194% ( 870, 1300 ] 881 0.332% 99.526% ( 1300, 1900 ] 647 0.244% 99.769% ( 1900, 2900 ] 433 0.163% 99.932% ( 2900, 4400 ] 320 0.120% 100.053% ( 4400, 6600 ] 79 0.030% 100.082% ( 6600, 9900 ] 13 0.005% 100.087% ( 9900, 14000 ] 3 0.001% 100.088% ** Level 5 read latency histogram (micros): Count: 16836 Average: 991.9534 StdDev: 1268.96 Min: 80 Median: 495.5210 Max: 13232 Percentiles: P50: 495.48 P75: 1164.10 P99: 6185.80 P99.9: 9442.79 P99.99: 13013.83 ------------------------------------------------------ ( 76, 110 ] 473 2.809% 2.809% # ( 110, 170 ] 2283 13.560% 16.370% ### ( 170, 250 ] 2283 13.560% 29.930% ### ( 250, 380 ] 2045 12.147% 42.077% ## ( 380, 580 ] 2313 13.738% 55.815% ### ( 580, 870 ] 2134 12.675% 68.490% ### ( 870, 1300 ] 1605 9.533% 78.023% ## ( 1300, 1900 ] 1101 6.540% 84.563% # ( 1900, 2900 ] 1017 6.041% 90.603% # ( 2900, 4400 ] 1026 6.094% 96.698% # ( 4400, 6600 ] 480 2.851% 99.549% # ( 6600, 9900 ] 71 0.422% 99.970% ( 9900, 14000 ] 7 0.042% 100.012% ** DB Stats ** Uptime(secs): 6.1 total, 6.0 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 161.06 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:2.221 H:M:S, 36.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 165.32 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.221 H:M:S, 37.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 4 num-running-flushes: 1 2024/12/16-11:01:10 ... thread 0: (1000000,2000000) ops and (247481.1,200182.3) ops/second in (4.040712,9.990893) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/7 3.33 GB 188.8 1.5 0.0 1.5 3.4 1.8 0.0 1.8 105.6 232.2 14.92 11.17 34 0.439 1599K 1589 0.0 0.0 L3 12/0 917.63 MB 0.9 1.0 1.0 0.1 1.0 0.9 0.1 1.1 150.8 150.2 6.89 4.79 1 6.889 992K 0 0.0 0.0 L4 162/0 9.41 GB 1.0 4.3 2.9 1.4 4.3 2.9 0.3 1.5 111.1 110.0 39.71 20.21 27 1.471 1431K 0 0.0 0.0 L5 1128/0 92.95 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1319/7 106.59 GB 0.0 6.9 3.9 3.0 8.7 5.7 5.6 4.7 114.2 144.1 61.52 36.17 62 0.992 4023K 1589 0.0 0.0 Int 0/0 0.00 KB 0.0 6.9 3.9 3.0 8.7 5.7 5.6 4.7 114.2 144.1 61.52 36.17 62 0.992 4023K 1589 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.9 3.9 3.0 6.8 3.8 0.0 0.0 132.8 131.8 52.91 30.95 34 1.556 4023K 1589 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 219.6 8.61 5.22 28 0.307 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 10.2 total, 10.0 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 8.66 GB write, 873.28 MB/s write, 6.86 GB read, 692.01 MB/s read, 61.5 seconds Interval compaction: 8.66 GB write, 887.14 MB/s write, 6.86 GB read, 703.00 MB/s read, 61.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 1, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 1, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 7, pending-compaction-bytes-delays: 12, pending-compaction-bytes-stops: 0, total-delays: 13, total-stops: 7, interval: 20 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 624045 Average: 14.0333 StdDev: 115.87 Min: 0 Median: 1.6072 Max: 9097 Percentiles: P50: 1.61 P75: 2.29 P99: 267.54 P99.9: 1134.01 P99.99: 1995.56 ------------------------------------------------------ [ 0, 1 ] 105413 16.892% 16.892% ### ( 1, 2 ] 340245 54.523% 71.414% ########### ( 2, 3 ] 76972 12.334% 83.749% ## ( 3, 4 ] 38956 6.242% 89.991% # ( 4, 6 ] 31892 5.111% 95.102% # ( 6, 10 ] 7624 1.222% 96.324% ( 10, 15 ] 1358 0.218% 96.541% ( 15, 22 ] 284 0.046% 96.587% ( 22, 34 ] 52 0.008% 96.595% ( 34, 51 ] 15 0.002% 96.597% ( 51, 76 ] 13 0.002% 96.599% ( 76, 110 ] 1997 0.320% 96.919% ( 110, 170 ] 8221 1.317% 98.237% ( 170, 250 ] 4450 0.713% 98.950% ( 250, 380 ] 2316 0.371% 99.321% ( 380, 580 ] 2080 0.333% 99.654% ( 580, 870 ] 1175 0.188% 99.843% ( 870, 1300 ] 583 0.093% 99.936% ( 1300, 1900 ] 315 0.050% 99.987% ( 1900, 2900 ] 226 0.036% 100.023% ( 2900, 4400 ] 171 0.027% 100.050% ( 4400, 6600 ] 49 0.008% 100.058% ( 6600, 9900 ] 12 0.002% 100.060% ** Level 2 read latency histogram (micros): Count: 248481 Average: 9.4455 StdDev: 117.29 Min: 1 Median: 1.7759 Max: 8737 Percentiles: P50: 1.78 P75: 2.68 P99: 13.02 P99.9: 1795.16 P99.99: 4804.79 ------------------------------------------------------ [ 0, 1 ] 15644 6.296% 6.296% # ( 1, 2 ] 139956 56.325% 62.620% ########### ( 2, 3 ] 45466 18.298% 80.918% #### ( 3, 4 ] 19612 7.893% 88.811% ## ( 4, 6 ] 18612 7.490% 96.301% # ( 6, 10 ] 6000 2.415% 98.716% ( 10, 15 ] 1171 0.471% 99.187% ( 15, 22 ] 190 0.076% 99.264% ( 22, 34 ] 29 0.012% 99.275% ( 34, 51 ] 14 0.006% 99.281% ( 51, 76 ] 14 0.006% 99.286% ( 76, 110 ] 38 0.015% 99.302% ( 110, 170 ] 183 0.074% 99.375% ( 170, 250 ] 193 0.078% 99.453% ( 250, 380 ] 199 0.080% 99.533% ( 380, 580 ] 253 0.102% 99.635% ( 580, 870 ] 286 0.115% 99.750% ( 870, 1300 ] 238 0.096% 99.846% ( 1300, 1900 ] 163 0.066% 99.911% ( 1900, 2900 ] 118 0.047% 99.959% ( 2900, 4400 ] 72 0.029% 99.988% ( 4400, 6600 ] 28 0.011% 99.999% ( 6600, 9900 ] 3 0.001% 100.000% ** Level 3 read latency histogram (micros): Count: 565440 Average: 9.4349 StdDev: 125.78 Min: 1 Median: 1.8475 Max: 9558 Percentiles: P50: 1.85 P75: 2.89 P99: 9.70 P99.9: 530.61 P99.99: 826.50 ------------------------------------------------------ [ 0, 1 ] 27332 4.834% 4.834% # ( 1, 2 ] 301356 53.296% 58.130% ########### ( 2, 3 ] 107029 18.928% 77.058% #### ( 3, 4 ] 56296 9.956% 87.014% ## ( 4, 6 ] 56268 9.951% 96.965% ## ( 6, 10 ] 12439 2.200% 99.165% ( 10, 15 ] 2469 0.437% 99.602% ( 15, 22 ] 298 0.053% 99.655% ( 22, 34 ] 42 0.007% 99.662% ( 34, 51 ] 49 0.009% 99.671% ( 51, 76 ] 38 0.007% 99.677% ( 76, 110 ] 110 0.019% 99.697% ( 110, 170 ] 350 0.062% 99.759% ( 170, 250 ] 284 0.050% 99.809% ( 250, 380 ] 251 0.044% 99.853% ( 380, 580 ] 350 0.062% 99.915% ( 580, 870 ] 497 0.088% 100.003% ( 870, 1300 ] 490 0.087% 100.090% ( 1300, 1900 ] 347 0.061% 100.151% ( 1900, 2900 ] 297 0.053% 100.204% ( 2900, 4400 ] 237 0.042% 100.246% ( 4400, 6600 ] 55 0.010% 100.255% ( 6600, 9900 ] 16 0.003% 100.258% ** Level 4 read latency histogram (micros): Count: 276777 Average: 31.7319 StdDev: 233.53 Min: 1 Median: 1.8355 Max: 12095 Percentiles: P50: 1.84 P75: 2.97 P99: 817.81 P99.9: 2844.78 P99.99: 3773.64 ------------------------------------------------------ [ 0, 1 ] 18053 6.523% 6.523% # ( 1, 2 ] 144030 52.038% 58.561% ########## ( 2, 3 ] 46951 16.963% 75.524% ### ( 3, 4 ] 25116 9.074% 84.599% ## ( 4, 6 ] 23195 8.380% 92.979% ## ( 6, 10 ] 6414 2.317% 95.297% ( 10, 15 ] 1209 0.437% 95.733% ( 15, 22 ] 173 0.063% 95.796% ( 22, 34 ] 32 0.012% 95.807% ( 34, 51 ] 10 0.004% 95.811% ( 51, 76 ] 12 0.004% 95.815% ( 76, 110 ] 756 0.273% 96.089% ( 110, 170 ] 2883 1.042% 97.130% ( 170, 250 ] 1772 0.640% 97.770% ( 250, 380 ] 1093 0.395% 98.165% ( 380, 580 ] 1318 0.476% 98.642% ( 580, 870 ] 1210 0.437% 99.079% ( 870, 1300 ] 1059 0.383% 99.461% ( 1300, 1900 ] 739 0.267% 99.728% ( 1900, 2900 ] 503 0.182% 99.910% ( 2900, 4400 ] 380 0.137% 100.047% ( 4400, 6600 ] 86 0.031% 100.078% ( 6600, 9900 ] 16 0.006% 100.084% ( 9900, 14000 ] 3 0.001% 100.085% ** Level 5 read latency histogram (micros): Count: 57430 Average: 475.8107 StdDev: 822.61 Min: 75 Median: 199.5556 Max: 13232 Percentiles: P50: 199.56 P75: 431.15 P99: 4492.34 P99.9: 7588.63 P99.99: 10415.37 ------------------------------------------------------ ( 51, 76 ] 1 0.002% 0.002% ( 76, 110 ] 5229 9.105% 9.107% ## ( 110, 170 ] 19553 34.047% 43.153% ####### ( 170, 250 ] 10643 18.532% 61.686% #### ( 250, 380 ] 6192 10.782% 72.467% ## ( 380, 580 ] 5687 9.902% 82.370% ## ( 580, 870 ] 3675 6.399% 88.769% # ( 870, 1300 ] 2150 3.744% 92.513% # ( 1300, 1900 ] 1300 2.264% 94.776% ( 1900, 2900 ] 1223 2.130% 96.906% ( 2900, 4400 ] 1181 2.056% 98.962% ( 4400, 6600 ] 517 0.900% 99.862% ( 6600, 9900 ] 72 0.125% 99.988% ( 9900, 14000 ] 10 0.017% 100.005% ** DB Stats ** Uptime(secs): 10.2 total, 4.0 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 193.90 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:2.703 H:M:S, 26.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 243.58 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.482 H:M:S, 11.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/12/16-11:01:20 ... thread 0: (1000000,3000000) ops and (100496.4,150440.0) ops/second in (9.950609,19.941502) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/7 4.31 GB 288.5 4.1 0.0 4.1 6.9 2.8 0.0 2.4 150.4 254.8 27.63 23.34 55 0.502 4220K 13K 0.0 0.0 L3 12/0 917.63 MB 0.9 1.0 1.0 0.1 1.0 0.9 0.1 1.1 150.8 150.2 6.89 4.79 1 6.889 992K 0 0.0 0.0 L4 162/0 9.41 GB 1.0 4.3 2.9 1.4 4.3 2.9 0.3 1.5 111.1 110.0 39.71 20.21 27 1.471 1431K 0 0.0 0.0 L5 1128/0 92.95 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1318/7 107.56 GB 0.0 9.4 3.9 5.5 12.2 6.6 5.6 4.3 129.4 167.6 74.23 48.34 83 0.894 6644K 13K 0.0 0.0 Int 0/0 0.00 KB 0.0 9.4 3.9 5.5 12.2 6.6 5.6 4.3 129.4 167.6 74.23 48.34 83 0.894 6644K 13K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 9.4 3.9 5.5 9.3 3.8 0.0 0.0 153.8 152.8 62.48 40.36 39 1.602 6644K 13K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 246.5 11.76 7.98 44 0.267 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 20.1 total, 19.9 interval Flush(GB): cumulative 2.830, interval 2.830 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 12.15 GB write, 618.98 MB/s write, 9.38 GB read, 477.87 MB/s read, 74.2 seconds Interval compaction: 12.15 GB write, 623.90 MB/s write, 9.38 GB read, 481.67 MB/s read, 74.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 14, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 14, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 7, pending-compaction-bytes-delays: 12, pending-compaction-bytes-stops: 0, total-delays: 26, total-stops: 7, interval: 33 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1408676 Average: 15.4879 StdDev: 89.61 Min: 0 Median: 1.4870 Max: 9097 Percentiles: P50: 1.49 P75: 1.98 P99: 246.68 P99.9: 723.50 P99.99: 1042.97 ------------------------------------------------------ [ 0, 1 ] 359523 25.522% 25.522% ##### ( 1, 2 ] 708011 50.261% 75.783% ########## ( 2, 3 ] 126041 8.947% 84.730% ## ( 3, 4 ] 68326 4.850% 89.581% # ( 4, 6 ] 46793 3.322% 92.902% # ( 6, 10 ] 10818 0.768% 93.670% ( 10, 15 ] 1951 0.138% 93.809% ( 15, 22 ] 563 0.040% 93.849% ( 22, 34 ] 117 0.008% 93.857% ( 34, 51 ] 19 0.001% 93.858% ( 51, 76 ] 24 0.002% 93.860% ( 76, 110 ] 12094 0.859% 94.719% ( 110, 170 ] 44923 3.189% 97.908% # ( 170, 250 ] 16052 1.140% 99.047% ( 250, 380 ] 5996 0.426% 99.473% ( 380, 580 ] 5042 0.358% 99.831% ( 580, 870 ] 1969 0.140% 99.971% ( 870, 1300 ] 679 0.048% 100.019% ( 1300, 1900 ] 331 0.023% 100.042% ( 1900, 2900 ] 254 0.018% 100.060% ( 2900, 4400 ] 187 0.013% 100.074% ( 4400, 6600 ] 50 0.004% 100.077% ( 6600, 9900 ] 12 0.001% 100.078% ** Level 2 read latency histogram (micros): Count: 248618 Average: 9.5369 StdDev: 117.34 Min: 1 Median: 1.7764 Max: 8737 Percentiles: P50: 1.78 P75: 2.68 P99: 13.59 P99.9: 1794.66 P99.99: 4803.72 ------------------------------------------------------ [ 0, 1 ] 15644 6.292% 6.292% # ( 1, 2 ] 139956 56.294% 62.586% ########### ( 2, 3 ] 45466 18.287% 80.873% #### ( 3, 4 ] 19612 7.888% 88.762% ## ( 4, 6 ] 18612 7.486% 96.248% # ( 6, 10 ] 6000 2.413% 98.661% ( 10, 15 ] 1171 0.471% 99.132% ( 15, 22 ] 190 0.076% 99.209% ( 22, 34 ] 29 0.012% 99.220% ( 34, 51 ] 14 0.006% 99.226% ( 51, 76 ] 14 0.006% 99.232% ( 76, 110 ] 61 0.025% 99.256% ( 110, 170 ] 256 0.103% 99.359% ( 170, 250 ] 219 0.088% 99.447% ( 250, 380 ] 207 0.083% 99.531% ( 380, 580 ] 260 0.105% 99.635% ( 580, 870 ] 286 0.115% 99.750% ( 870, 1300 ] 238 0.096% 99.846% ( 1300, 1900 ] 163 0.066% 99.912% ( 1900, 2900 ] 118 0.047% 99.959% ( 2900, 4400 ] 72 0.029% 99.988% ( 4400, 6600 ] 28 0.011% 99.999% ( 6600, 9900 ] 3 0.001% 100.000% ** Level 3 read latency histogram (micros): Count: 567626 Average: 10.0647 StdDev: 126.16 Min: 1 Median: 1.8511 Max: 9558 Percentiles: P50: 1.85 P75: 2.91 P99: 12.49 P99.9: 553.57 P99.99: 831.80 ------------------------------------------------------ [ 0, 1 ] 27332 4.815% 4.815% # ( 1, 2 ] 301356 53.091% 57.906% ########### ( 2, 3 ] 107029 18.856% 76.761% #### ( 3, 4 ] 56296 9.918% 86.679% ## ( 4, 6 ] 56268 9.913% 96.592% ## ( 6, 10 ] 12439 2.191% 98.783% ( 10, 15 ] 2469 0.435% 99.218% ( 15, 22 ] 298 0.052% 99.271% ( 22, 34 ] 42 0.007% 99.278% ( 34, 51 ] 49 0.009% 99.287% ( 51, 76 ] 38 0.007% 99.294% ( 76, 110 ] 482 0.085% 99.378% ( 110, 170 ] 1574 0.277% 99.656% ( 170, 250 ] 652 0.115% 99.771% ( 250, 380 ] 356 0.063% 99.833% ( 380, 580 ] 436 0.077% 99.910% ( 580, 870 ] 522 0.092% 100.002% ( 870, 1300 ] 494 0.087% 100.089% ( 1300, 1900 ] 348 0.061% 100.150% ( 1900, 2900 ] 298 0.052% 100.203% ( 2900, 4400 ] 237 0.042% 100.245% ( 4400, 6600 ] 55 0.010% 100.254% ( 6600, 9900 ] 16 0.003% 100.257% ** Level 4 read latency histogram (micros): Count: 299706 Average: 41.4660 StdDev: 229.15 Min: 1 Median: 1.9141 Max: 12095 Percentiles: P50: 1.91 P75: 3.60 P99: 788.29 P99.9: 2799.79 P99.99: 3762.32 ------------------------------------------------------ [ 0, 1 ] 18055 6.024% 6.024% # ( 1, 2 ] 144182 48.108% 54.132% ########## ( 2, 3 ] 47351 15.799% 69.931% ### ( 3, 4 ] 25511 8.512% 78.443% ## ( 4, 6 ] 23894 7.972% 86.416% ## ( 6, 10 ] 6508 2.171% 88.587% ( 10, 15 ] 1230 0.410% 88.998% ( 15, 22 ] 174 0.058% 89.056% ( 22, 34 ] 32 0.011% 89.066% ( 34, 51 ] 10 0.003% 89.070% ( 51, 76 ] 16 0.005% 89.075% ( 76, 110 ] 4355 1.453% 90.528% ( 110, 170 ] 14712 4.909% 95.437% # ( 170, 250 ] 5256 1.754% 97.191% ( 250, 380 ] 2233 0.745% 97.936% ( 380, 580 ] 2165 0.722% 98.658% ( 580, 870 ] 1427 0.476% 99.134% ( 870, 1300 ] 1092 0.364% 99.499% ( 1300, 1900 ] 748 0.250% 99.748% ( 1900, 2900 ] 506 0.169% 99.917% ( 2900, 4400 ] 381 0.127% 100.044% ( 4400, 6600 ] 86 0.029% 100.073% ( 6600, 9900 ] 16 0.005% 100.078% ( 9900, 14000 ] 3 0.001% 100.079% ** Level 5 read latency histogram (micros): Count: 198140 Average: 260.5466 StdDev: 473.49 Min: 74 Median: 152.8092 Max: 13232 Percentiles: P50: 152.81 P75: 220.47 P99: 2738.70 P99.9: 6071.74 P99.99: 9083.52 ------------------------------------------------------ ( 51, 76 ] 14 0.007% 0.007% ( 76, 110 ] 29029 14.651% 14.658% ### ( 110, 170 ] 98149 49.535% 64.193% ########## ( 170, 250 ] 33946 17.132% 81.325% ### ( 250, 380 ] 13600 6.864% 88.189% # ( 380, 580 ] 11443 5.775% 93.964% # ( 580, 870 ] 5208 2.628% 96.593% # ( 870, 1300 ] 2356 1.189% 97.782% ( 1300, 1900 ] 1357 0.685% 98.467% ( 1900, 2900 ] 1261 0.636% 99.103% ( 2900, 4400 ] 1187 0.599% 99.702% ( 4400, 6600 ] 517 0.261% 99.963% ( 6600, 9900 ] 72 0.036% 99.999% ( 9900, 14000 ] 10 0.005% 100.005% ** DB Stats ** Uptime(secs): 20.1 total, 10.0 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 146.88 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:8.990 H:M:S, 44.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 98.91 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:6.287 H:M:S, 63.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 2024/12/16-11:01:26 ... thread 0: (1000000,4000000) ops and (181078.7,157084.7) ops/second in (5.522461,25.463963) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/12 5.23 GB 257.0 5.0 0.0 5.0 8.8 3.7 0.0 2.3 150.2 261.5 34.39 29.63 74 0.465 5243K 14K 0.0 0.0 L3 12/0 917.63 MB 0.9 1.0 1.0 0.1 1.0 0.9 0.1 1.1 150.8 150.2 6.89 4.79 1 6.889 992K 0 0.0 0.0 L4 162/0 9.41 GB 1.0 4.3 2.9 1.4 4.3 2.9 0.3 1.5 111.1 110.0 39.71 20.21 27 1.471 1431K 0 0.0 0.0 L5 1128/0 92.95 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1321/12 108.48 GB 0.0 10.4 3.9 6.5 14.1 7.5 5.6 3.7 131.1 177.7 80.99 54.62 102 0.794 7667K 14K 0.0 0.0 Int 0/0 0.00 KB 0.0 10.4 3.9 6.5 14.1 7.5 5.6 3.7 131.1 177.7 80.99 54.62 102 0.794 7667K 14K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 10.4 3.9 6.5 10.3 3.8 0.0 0.0 160.2 159.2 66.26 44.08 43 1.541 7667K 14K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 261.0 14.73 10.54 59 0.250 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 25.6 total, 25.5 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 14.06 GB write, 561.76 MB/s write, 10.37 GB read, 414.22 MB/s read, 81.0 seconds Interval compaction: 14.06 GB write, 565.26 MB/s write, 10.37 GB read, 416.80 MB/s read, 81.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 22, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 22, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 7, pending-compaction-bytes-delays: 12, pending-compaction-bytes-stops: 0, total-delays: 34, total-stops: 7, interval: 41 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1769498 Average: 16.4618 StdDev: 86.01 Min: 0 Median: 1.4779 Max: 9097 Percentiles: P50: 1.48 P75: 1.98 P99: 249.74 P99.9: 687.10 P99.99: 903.87 ------------------------------------------------------ [ 0, 1 ] 463263 26.180% 26.180% ##### ( 1, 2 ] 881927 49.841% 76.021% ########## ( 2, 3 ] 151247 8.547% 84.568% ## ( 3, 4 ] 81013 4.578% 89.147% # ( 4, 6 ] 53187 3.006% 92.153% # ( 6, 10 ] 12213 0.690% 92.843% ( 10, 15 ] 2253 0.127% 92.970% ( 15, 22 ] 672 0.038% 93.008% ( 22, 34 ] 143 0.008% 93.016% ( 34, 51 ] 22 0.001% 93.017% ( 51, 76 ] 28 0.002% 93.019% ( 76, 110 ] 18087 1.022% 94.041% ( 110, 170 ] 65108 3.679% 97.721% # ( 170, 250 ] 22716 1.284% 99.004% ( 250, 380 ] 8188 0.463% 99.467% ( 380, 580 ] 6764 0.382% 99.849% ( 580, 870 ] 2433 0.137% 99.987% ( 870, 1300 ] 737 0.042% 100.028% ( 1300, 1900 ] 345 0.019% 100.048% ( 1900, 2900 ] 281 0.016% 100.064% ( 2900, 4400 ] 199 0.011% 100.075% ( 4400, 6600 ] 50 0.003% 100.078% ( 6600, 9900 ] 12 0.001% 100.079% ** Level 2 read latency histogram (micros): Count: 248707 Average: 9.5980 StdDev: 117.39 Min: 1 Median: 1.7767 Max: 8737 Percentiles: P50: 1.78 P75: 2.68 P99: 13.97 P99.9: 1794.33 P99.99: 4803.02 ------------------------------------------------------ [ 0, 1 ] 15644 6.290% 6.290% # ( 1, 2 ] 139956 56.273% 62.564% ########### ( 2, 3 ] 45466 18.281% 80.845% #### ( 3, 4 ] 19612 7.886% 88.730% ## ( 4, 6 ] 18612 7.484% 96.214% # ( 6, 10 ] 6000 2.412% 98.626% ( 10, 15 ] 1171 0.471% 99.097% ( 15, 22 ] 190 0.076% 99.173% ( 22, 34 ] 29 0.012% 99.185% ( 34, 51 ] 14 0.006% 99.191% ( 51, 76 ] 14 0.006% 99.196% ( 76, 110 ] 77 0.031% 99.227% ( 110, 170 ] 306 0.123% 99.350% ( 170, 250 ] 233 0.094% 99.444% ( 250, 380 ] 210 0.084% 99.528% ( 380, 580 ] 264 0.106% 99.635% ( 580, 870 ] 288 0.116% 99.750% ( 870, 1300 ] 238 0.096% 99.846% ( 1300, 1900 ] 163 0.066% 99.912% ( 1900, 2900 ] 118 0.047% 99.959% ( 2900, 4400 ] 72 0.029% 99.988% ( 4400, 6600 ] 28 0.011% 99.999% ( 6600, 9900 ] 3 0.001% 100.000% ** Level 3 read latency histogram (micros): Count: 568747 Average: 10.3946 StdDev: 126.37 Min: 1 Median: 1.8529 Max: 9558 Percentiles: P50: 1.85 P75: 2.91 P99: 14.74 P99.9: 562.41 P99.99: 835.25 ------------------------------------------------------ [ 0, 1 ] 27332 4.806% 4.806% # ( 1, 2 ] 301356 52.986% 57.792% ########### ( 2, 3 ] 107029 18.818% 76.610% #### ( 3, 4 ] 56296 9.898% 86.508% ## ( 4, 6 ] 56268 9.893% 96.402% ## ( 6, 10 ] 12439 2.187% 98.589% ( 10, 15 ] 2469 0.434% 99.023% ( 15, 22 ] 298 0.052% 99.075% ( 22, 34 ] 42 0.007% 99.083% ( 34, 51 ] 49 0.009% 99.091% ( 51, 76 ] 38 0.007% 99.098% ( 76, 110 ] 670 0.118% 99.216% ( 110, 170 ] 2183 0.384% 99.599% ( 170, 250 ] 855 0.150% 99.750% ( 250, 380 ] 411 0.072% 99.822% ( 380, 580 ] 486 0.085% 99.908% ( 580, 870 ] 533 0.094% 100.001% ( 870, 1300 ] 498 0.088% 100.089% ( 1300, 1900 ] 348 0.061% 100.150% ( 1900, 2900 ] 299 0.053% 100.203% ( 2900, 4400 ] 237 0.042% 100.244% ( 4400, 6600 ] 55 0.010% 100.254% ( 6600, 9900 ] 16 0.003% 100.257% ** Level 4 read latency histogram (micros): Count: 312310 Average: 46.7213 StdDev: 227.32 Min: 1 Median: 1.9577 Max: 12095 Percentiles: P50: 1.96 P75: 3.96 P99: 776.04 P99.9: 2779.55 P99.99: 3760.71 ------------------------------------------------------ [ 0, 1 ] 18055 5.781% 5.781% # ( 1, 2 ] 144193 46.170% 51.951% ######### ( 2, 3 ] 47430 15.187% 67.138% ### ( 3, 4 ] 25561 8.184% 75.322% ## ( 4, 6 ] 23986 7.680% 83.002% ## ( 6, 10 ] 6517 2.087% 85.089% ( 10, 15 ] 1232 0.394% 85.484% ( 15, 22 ] 174 0.056% 85.539% ( 22, 34 ] 32 0.010% 85.550% ( 34, 51 ] 10 0.003% 85.553% ( 51, 76 ] 17 0.005% 85.558% ( 76, 110 ] 6461 2.069% 87.627% ( 110, 170 ] 21478 6.877% 94.504% # ( 170, 250 ] 7345 2.352% 96.856% ( 250, 380 ] 2908 0.931% 97.787% ( 380, 580 ] 2734 0.875% 98.663% ( 580, 870 ] 1559 0.499% 99.162% ( 870, 1300 ] 1105 0.354% 99.516% ( 1300, 1900 ] 753 0.241% 99.757% ( 1900, 2900 ] 509 0.163% 99.920% ( 2900, 4400 ] 383 0.123% 100.042% ( 4400, 6600 ] 86 0.028% 100.070% ( 6600, 9900 ] 16 0.005% 100.075% ( 9900, 14000 ] 3 0.001% 100.076% ** Level 5 read latency histogram (micros): Count: 274267 Average: 236.7749 StdDev: 409.41 Min: 74 Median: 150.8656 Max: 13232 Percentiles: P50: 150.87 P75: 210.72 P99: 2161.59 P99.9: 5728.21 P99.99: 8505.44 ------------------------------------------------------ ( 51, 76 ] 17 0.006% 0.006% ( 76, 110 ] 41747 15.221% 15.227% ### ( 110, 170 ] 140024 51.054% 66.281% ########## ( 170, 250 ] 46984 17.131% 83.412% ### ( 250, 380 ] 17734 6.466% 89.878% # ( 380, 580 ] 14764 5.383% 95.261% # ( 580, 870 ] 6069 2.213% 97.474% ( 870, 1300 ] 2461 0.897% 98.371% ( 1300, 1900 ] 1384 0.505% 98.876% ( 1900, 2900 ] 1301 0.474% 99.350% ( 2900, 4400 ] 1195 0.436% 99.786% ( 4400, 6600 ] 518 0.189% 99.975% ( 6600, 9900 ] 72 0.026% 100.001% ( 9900, 14000 ] 10 0.004% 100.005% ** DB Stats ** Uptime(secs): 25.6 total, 5.5 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 153.63 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:11.039 H:M:S, 43.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 178.22 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.049 H:M:S, 37.1 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 226.410 micros/op 17659 ops/sec 60.081 seconds 1060996 operations; 17.1 MB/s (259999 of 259999 found) Microseconds per read: Count: 1060996 Average: 226.4131 StdDev: 581.39 Min: 2 Median: 176.6633 Max: 71345 Percentiles: P50: 176.66 P75: 232.38 P99: 824.74 P99.9: 8815.34 P99.99: 25828.72 ------------------------------------------------------ ( 1, 2 ] 1 0.000% 0.000% ( 2, 3 ] 21 0.002% 0.002% ( 3, 4 ] 40 0.004% 0.006% ( 4, 6 ] 92 0.009% 0.015% ( 6, 10 ] 414 0.039% 0.054% ( 10, 15 ] 2374 0.224% 0.277% ( 15, 22 ] 4076 0.384% 0.661% ( 22, 34 ] 12816 1.208% 1.869% ( 34, 51 ] 3292 0.310% 2.180% ( 51, 76 ] 389 0.037% 2.216% ( 76, 110 ] 15093 1.423% 3.639% ( 110, 170 ] 460169 43.371% 47.010% ######### ( 170, 250 ] 380843 35.895% 82.905% ####### ( 250, 380 ] 110571 10.421% 93.327% ## ( 380, 580 ] 46462 4.379% 97.706% # ( 580, 870 ] 16273 1.534% 99.239% ( 870, 1300 ] 4351 0.410% 99.649% ( 1300, 1900 ] 1043 0.098% 99.748% ( 1900, 2900 ] 731 0.069% 99.817% ( 2900, 4400 ] 534 0.050% 99.867% ( 4400, 6600 ] 205 0.019% 99.886% ( 6600, 9900 ] 216 0.020% 99.907% ( 9900, 14000 ] 367 0.035% 99.941% ( 14000, 22000 ] 472 0.044% 99.986% ( 22000, 33000 ] 129 0.012% 99.998% ( 33000, 50000 ] 20 0.002% 100.000% ( 50000, 75000 ] 2 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 17647073 rocksdb.bloom.filter.full.positive COUNT : 1232802 rocksdb.bloom.filter.full.true.positive COUNT : 1060545 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 451 rocksdb.memtable.miss COUNT : 1060545 rocksdb.l0.hit COUNT : 47623 rocksdb.l1.hit COUNT : 9740 rocksdb.l2andup.hit COUNT : 1003182 rocksdb.compaction.key.drop.new COUNT : 113631 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 4750869 rocksdb.number.keys.read COUNT : 1060996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 4902896808 rocksdb.bytes.read COUNT : 1060996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1519 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 42607458 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 16474908373 rocksdb.compact.write.bytes COUNT : 16356357814 rocksdb.flush.write.bytes COUNT : 4888577080 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 55 rocksdb.number.superversion_acquires COUNT : 850 rocksdb.number.superversion_releases COUNT : 820 rocksdb.number.superversion_cleanups COUNT : 208 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 183 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 4848859136 rocksdb.memtable.garbage.bytes.at.flush COUNT : 1619968 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 21027385150 rocksdb.non.last.level.read.count COUNT : 4936664 rocksdb.block.checksum.compute.count COUNT : 5234174 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 175.083971 P95 : 454.987312 P99 : 824.116910 P100 : 71342.000000 COUNT : 1060996 SUM : 239220264 rocksdb.db.write.micros P50 : 1.496461 P95 : 4.279300 P99 : 10.119865 P100 : 134302.000000 COUNT : 4750869 SUM : 54154151 rocksdb.compaction.times.micros P50 : 1140869.565217 P95 : 6060000.000000 P99 : 27720000.000000 P100 : 29412896.000000 COUNT : 48 SUM : 103295843 rocksdb.compaction.times.cpu_micros P50 : 938461.538462 P95 : 5473333.333333 P99 : 10579893.000000 P100 : 10579893.000000 COUNT : 48 SUM : 62092867 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 775.581395 P95 : 39800.000000 P99 : 262198.000000 P100 : 262198.000000 COUNT : 72 SUM : 733976 rocksdb.compaction.outfile.sync.micros P50 : 4292.857143 P95 : 238333.333333 P99 : 344900.000000 P100 : 372273.000000 COUNT : 135 SUM : 6512880 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 154.634146 P95 : 4466.000000 P99 : 7064.000000 P100 : 7064.000000 COUNT : 137 SUM : 104365 rocksdb.table.open.io.micros P50 : 8725.886525 P95 : 19160.655738 P99 : 21816.830601 P100 : 26089.000000 COUNT : 1519 SUM : 13427220 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.702031 P95 : 112.647736 P99 : 222.118811 P100 : 24469.000000 COUNT : 3989744 SUM : 71215000 rocksdb.read.block.get.micros P50 : 140.347687 P95 : 330.025532 P99 : 730.802624 P100 : 26503.000000 COUNT : 1242951 SUM : 201500619 rocksdb.write.raw.block.micros P50 : 0.534342 P95 : 1.325403 P99 : 3.595979 P100 : 14163.000000 COUNT : 5145582 SUM : 12387809 rocksdb.numfiles.in.singlecompaction P50 : 3.412500 P95 : 5.887500 P99 : 25.000000 P100 : 25.000000 COUNT : 69 SUM : 285 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.566489 P95 : 1967.938748 P99 : 2718.782971 P100 : 134134.000000 COUNT : 234132 SUM : 42625304 rocksdb.sst.read.micros P50 : 1.938066 P95 : 187.082674 P99 : 464.318397 P100 : 13232.000000 COUNT : 4936798 SUM : 255531465 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1060996 SUM : 1060996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 4750869 SUM : 4902896808 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 216451.612903 P95 : 494000.000000 P99 : 955200.000000 P100 : 1113231.000000 COUNT : 72 SUM : 17294499 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 193 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0