[2024-12-02 08:44:13.012391] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 23.11.0 initialization... [2024-12-02 08:44:13.012537] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1930071 ] Set seed to 1733125453008917 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-02 08:44:13.123750] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-02 08:44:13.332621] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-02 08:44:13.332667] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1930071' to capture a snapshot of events at runtime. [2024-12-02 08:44:13.332678] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1930071 for offline analysis/debug. [2024-12-02 08:44:13.332703] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Mon Dec 2 08:44:22 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/02-08:44:29 ... thread 0: (1000000,1000000) ops and (136510.8,136510.8) ops/second in (7.325426,7.325426) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 8/7 1.29 GB 12.6 0.5 0.0 0.5 1.3 0.8 0.0 1.6 48.4 126.7 10.44 4.37 11 0.949 512K 493 0.0 0.0 L2 4/1 445.80 MB 3.2 0.6 0.5 0.1 0.6 0.5 0.0 1.1 108.8 108.3 5.28 2.68 1 5.282 523K 0 0.0 0.0 L3 64/9 4.28 GB 3.6 4.8 3.2 1.6 4.8 3.1 0.4 1.5 101.1 100.5 48.52 22.90 22 2.206 2085K 1694 0.0 0.0 L4 163/8 10.57 GB 1.0 1.1 0.9 0.2 1.0 0.9 0.1 1.2 117.6 116.6 9.20 4.90 10 0.920 754K 0 0.0 0.0 L5 1076/0 89.94 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1315/25 106.51 GB 0.0 6.9 4.6 2.3 7.7 5.3 0.5 9.6 96.2 106.8 73.44 34.84 44 1.669 3875K 2187 0.0 0.0 Int 0/0 0.00 KB 0.0 6.9 4.6 2.3 7.7 5.3 0.5 9.6 96.2 106.8 73.44 34.84 44 1.669 3875K 2187 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.9 4.6 2.3 6.9 4.5 0.0 0.0 105.6 105.0 66.89 32.57 35 1.911 3875K 2187 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 125.0 6.55 2.28 9 0.728 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.5 total, 7.3 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.66 GB write, 1041.83 MB/s write, 6.90 GB read, 938.53 MB/s read, 73.4 seconds Interval compaction: 7.66 GB write, 1070.68 MB/s write, 6.90 GB read, 964.52 MB/s read, 73.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 29, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 29, interval: 29 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 129071 Average: 16.7793 StdDev: 197.64 Min: 1 Median: 1.9048 Max: 11661 Percentiles: P50: 1.90 P75: 2.94 P99: 12.87 P99.9: 3488.10 P99.99: 6229.38 ------------------------------------------------------ [ 0, 1 ] 4972 3.852% 3.852% # ( 1, 2 ] 65830 51.003% 54.855% ########## ( 2, 3 ] 27729 21.484% 76.339% #### ( 3, 4 ] 11308 8.761% 85.100% ## ( 4, 6 ] 14775 11.447% 96.547% ## ( 6, 10 ] 2787 2.159% 98.706% ( 10, 15 ] 660 0.511% 99.217% ( 15, 22 ] 78 0.060% 99.278% ( 22, 34 ] 1 0.001% 99.279% ( 34, 51 ] 2 0.002% 99.280% ( 76, 110 ] 2 0.002% 99.282% ( 110, 170 ] 3 0.002% 99.284% ( 170, 250 ] 9 0.007% 99.291% ( 250, 380 ] 21 0.016% 99.307% ( 380, 580 ] 54 0.042% 99.349% ( 580, 870 ] 108 0.084% 99.433% ( 870, 1300 ] 168 0.130% 99.563% ( 1300, 1900 ] 189 0.146% 99.709% ( 1900, 2900 ] 193 0.150% 99.859% ( 2900, 4400 ] 135 0.105% 99.964% ( 4400, 6600 ] 41 0.032% 99.995% ( 6600, 9900 ] 6 0.005% 100.000% ( 9900, 14000 ] 1 0.001% 100.001% ** Level 1 read latency histogram (micros): Count: 130847 Average: 20.0512 StdDev: 219.99 Min: 1 Median: 2.0736 Max: 9518 Percentiles: P50: 2.07 P75: 3.19 P99: 14.84 P99.9: 3743.97 P99.99: 6357.13 ------------------------------------------------------ [ 0, 1 ] 3009 2.300% 2.300% ( 1, 2 ] 60007 45.860% 48.160% ######### ( 2, 3 ] 32712 25.000% 73.160% ##### ( 3, 4 ] 12983 9.922% 83.083% ## ( 4, 6 ] 15839 12.105% 95.188% ## ( 6, 10 ] 3685 2.816% 98.004% # ( 10, 15 ] 1346 1.029% 99.032% ( 15, 22 ] 203 0.155% 99.188% ( 22, 34 ] 16 0.012% 99.200% ( 34, 51 ] 4 0.003% 99.203% ( 51, 76 ] 2 0.002% 99.204% ( 170, 250 ] 8 0.006% 99.211% ( 250, 380 ] 14 0.011% 99.221% ( 380, 580 ] 30 0.023% 99.244% ( 580, 870 ] 95 0.073% 99.317% ( 870, 1300 ] 144 0.110% 99.427% ( 1300, 1900 ] 228 0.174% 99.601% ( 1900, 2900 ] 282 0.216% 99.817% ( 2900, 4400 ] 194 0.148% 99.965% ( 4400, 6600 ] 37 0.028% 99.993% ( 6600, 9900 ] 9 0.007% 100.000% ** Level 2 read latency histogram (micros): Count: 687973 Average: 18.2510 StdDev: 209.41 Min: 1 Median: 2.0399 Max: 11998 Percentiles: P50: 2.04 P75: 3.11 P99: 13.36 P99.9: 2424.41 P99.99: 2896.34 ------------------------------------------------------ [ 0, 1 ] 15932 2.316% 2.316% ( 1, 2 ] 321198 46.688% 49.003% ######### ( 2, 3 ] 171817 24.974% 73.978% ##### ( 3, 4 ] 66935 9.729% 83.707% ## ( 4, 6 ] 81867 11.900% 95.607% ## ( 6, 10 ] 19156 2.784% 98.391% # ( 10, 15 ] 6239 0.907% 99.298% ( 15, 22 ] 838 0.122% 99.420% ( 22, 34 ] 77 0.011% 99.431% ( 34, 51 ] 34 0.005% 99.436% ( 51, 76 ] 22 0.003% 99.439% ( 76, 110 ] 14 0.002% 99.441% ( 110, 170 ] 18 0.003% 99.444% ( 170, 250 ] 35 0.005% 99.449% ( 250, 380 ] 74 0.011% 99.460% ( 380, 580 ] 195 0.028% 99.488% ( 580, 870 ] 384 0.056% 99.544% ( 870, 1300 ] 748 0.109% 99.653% ( 1300, 1900 ] 1014 0.147% 99.800% ( 1900, 2900 ] 1312 0.191% 99.991% ( 2900, 4400 ] 889 0.129% 100.120% ( 4400, 6600 ] 224 0.033% 100.152% ( 6600, 9900 ] 37 0.005% 100.158% ( 9900, 14000 ] 3 0.000% 100.158% ** Level 3 read latency histogram (micros): Count: 650780 Average: 14.7355 StdDev: 179.81 Min: 1 Median: 2.0541 Max: 10567 Percentiles: P50: 2.05 P75: 3.04 P99: 11.61 P99.9: 1875.14 P99.99: 2503.40 ------------------------------------------------------ [ 0, 1 ] 11848 1.821% 1.821% ( 1, 2 ] 304368 46.770% 48.590% ######### ( 2, 3 ] 169607 26.062% 74.652% ##### ( 3, 4 ] 64049 9.842% 84.494% ## ( 4, 6 ] 77312 11.880% 96.374% ## ( 6, 10 ] 15663 2.407% 98.781% ( 10, 15 ] 4422 0.679% 99.460% ( 15, 22 ] 524 0.081% 99.541% ( 22, 34 ] 51 0.008% 99.549% ( 34, 51 ] 25 0.004% 99.553% ( 51, 76 ] 33 0.005% 99.558% ( 76, 110 ] 19 0.003% 99.561% ( 110, 170 ] 23 0.004% 99.564% ( 170, 250 ] 74 0.011% 99.576% ( 250, 380 ] 112 0.017% 99.593% ( 380, 580 ] 211 0.032% 99.625% ( 580, 870 ] 426 0.065% 99.691% ( 870, 1300 ] 628 0.096% 99.787% ( 1300, 1900 ] 767 0.118% 99.905% ( 1900, 2900 ] 918 0.141% 100.046% ( 2900, 4400 ] 599 0.092% 100.138% ( 4400, 6600 ] 170 0.026% 100.164% ( 6600, 9900 ] 18 0.003% 100.167% ( 9900, 14000 ] 1 0.000% 100.167% ** Level 4 read latency histogram (micros): Count: 65542 Average: 70.8664 StdDev: 445.36 Min: 1 Median: 1.9751 Max: 9513 Percentiles: P50: 1.98 P75: 3.04 P99: 2473.35 P99.9: 5508.26 P99.99: 6571.97 ------------------------------------------------------ [ 0, 1 ] 1544 2.356% 2.356% ( 1, 2 ] 32023 48.859% 51.214% ########## ( 2, 3 ] 15327 23.385% 74.599% ##### ( 3, 4 ] 5933 9.052% 83.652% ## ( 4, 6 ] 6360 9.704% 93.355% ## ( 6, 10 ] 1603 2.446% 95.801% ( 10, 15 ] 452 0.690% 96.491% ( 15, 22 ] 38 0.058% 96.549% ( 22, 34 ] 4 0.006% 96.555% ( 34, 51 ] 3 0.005% 96.559% ( 51, 76 ] 4 0.006% 96.566% ( 76, 110 ] 1 0.002% 96.567% ( 110, 170 ] 17 0.026% 96.593% ( 170, 250 ] 40 0.061% 96.654% ( 250, 380 ] 59 0.090% 96.744% ( 380, 580 ] 141 0.215% 96.959% ( 580, 870 ] 273 0.417% 97.376% ( 870, 1300 ] 391 0.597% 97.972% ( 1300, 1900 ] 416 0.635% 98.607% ( 1900, 2900 ] 451 0.688% 99.295% ( 2900, 4400 ] 336 0.513% 99.808% ( 4400, 6600 ] 122 0.186% 99.994% ( 6600, 9900 ] 26 0.040% 100.034% ** Level 5 read latency histogram (micros): Count: 8768 Average: 2385.7444 StdDev: 1707.03 Min: 91 Median: 1896.0759 Max: 11787 Percentiles: P50: 1896.08 P75: 3447.45 P99: 8585.71 P99.9: 11432.23 P99.99: 11787.00 ------------------------------------------------------ ( 76, 110 ] 4 0.046% 0.046% ( 110, 170 ] 16 0.182% 0.228% ( 170, 250 ] 68 0.776% 1.004% ( 250, 380 ] 165 1.882% 2.885% ( 380, 580 ] 444 5.064% 7.949% # ( 580, 870 ] 873 9.957% 17.906% ## ( 870, 1300 ] 1295 14.770% 32.676% ### ( 1300, 1900 ] 1529 17.438% 50.114% ### ( 1900, 2900 ] 1582 18.043% 68.157% #### ( 2900, 4400 ] 1644 18.750% 86.907% #### ( 4400, 6600 ] 949 10.823% 97.730% ## ( 6600, 9900 ] 185 2.110% 99.840% ( 9900, 14000 ] 14 0.160% 100.000% ** DB Stats ** Uptime(secs): 7.5 total, 7.3 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 130.70 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.525 H:M:S, 46.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 134.30 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.525 H:M:S, 48.1 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/12/02-08:44:36 ... thread 0: (1000000,2000000) ops and (132467.8,134458.9) ops/second in (7.549005,14.874431) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/11 2.34 GB 81.8 1.2 0.0 1.2 3.0 1.8 0.0 1.6 53.9 138.9 22.22 10.63 26 0.854 1216K 1449 0.0 0.0 L2 0/0 0.00 KB 0.0 0.6 0.5 0.1 0.6 0.5 0.0 1.1 108.8 108.3 5.28 2.68 1 5.282 523K 0 0.0 0.0 L3 16/1 1.09 GB 1.0 5.3 3.4 1.9 5.3 3.4 0.6 1.5 104.4 103.7 51.87 25.16 24 2.161 2471K 2618 0.0 0.0 L4 173/5 9.63 GB 1.0 7.3 4.1 3.2 7.2 3.9 0.5 1.8 113.9 111.7 65.54 38.12 53 1.237 4082K 6804 0.0 0.0 L5 1128/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1332/17 107.42 GB 0.0 14.3 8.0 6.3 16.0 9.7 5.4 8.7 101.1 112.9 144.91 76.59 104 1.393 8293K 10K 0.0 0.0 Int 0/0 0.00 KB 0.0 14.3 8.0 6.3 16.0 9.7 5.4 8.7 101.1 112.9 144.91 76.59 104 1.393 8293K 10K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 14.3 8.0 6.3 14.1 7.8 0.0 0.0 112.2 110.8 130.65 71.02 82 1.593 8293K 10K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 132.5 14.26 5.57 22 0.648 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 15.1 total, 14.9 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 15.98 GB write, 1085.17 MB/s write, 14.31 GB read, 971.76 MB/s read, 144.9 seconds Interval compaction: 15.98 GB write, 1099.98 MB/s write, 14.31 GB read, 985.02 MB/s read, 144.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 63, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 63, interval: 63 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 343370 Average: 38.4289 StdDev: 426.28 Min: 1 Median: 1.9168 Max: 62243 Percentiles: P50: 1.92 P75: 2.97 P99: 1099.07 P99.9: 4296.58 P99.99: 9770.23 ------------------------------------------------------ [ 0, 1 ] 13740 4.002% 4.002% # ( 1, 2 ] 172279 50.173% 54.175% ########## ( 2, 3 ] 73369 21.367% 75.542% #### ( 3, 4 ] 29655 8.636% 84.178% ## ( 4, 6 ] 36909 10.749% 94.927% ## ( 6, 10 ] 6717 1.956% 96.884% ( 10, 15 ] 1500 0.437% 97.320% ( 15, 22 ] 179 0.052% 97.373% ( 22, 34 ] 2 0.001% 97.373% ( 34, 51 ] 4 0.001% 97.374% ( 51, 76 ] 11 0.003% 97.377% ( 76, 110 ] 23 0.007% 97.384% ( 110, 170 ] 266 0.077% 97.462% ( 170, 250 ] 534 0.156% 97.617% ( 250, 380 ] 771 0.225% 97.842% ( 380, 580 ] 1265 0.368% 98.210% ( 580, 870 ] 1804 0.525% 98.735% ( 870, 1300 ] 1705 0.497% 99.232% ( 1300, 1900 ] 1051 0.306% 99.538% ( 1900, 2900 ] 684 0.199% 99.737% ( 2900, 4400 ] 600 0.175% 99.912% ( 4400, 6600 ] 235 0.068% 99.980% ( 6600, 9900 ] 34 0.010% 99.990% ( 9900, 14000 ] 6 0.002% 99.992% ( 14000, 22000 ] 13 0.004% 99.996% ( 22000, 33000 ] 25 0.007% 100.003% ( 33000, 50000 ] 6 0.002% 100.005% ( 50000, 75000 ] 1 0.000% 100.005% ** Level 1 read latency histogram (micros): Count: 130847 Average: 20.0512 StdDev: 219.99 Min: 1 Median: 2.0736 Max: 9518 Percentiles: P50: 2.07 P75: 3.19 P99: 14.84 P99.9: 3743.97 P99.99: 6357.13 ------------------------------------------------------ [ 0, 1 ] 3009 2.300% 2.300% ( 1, 2 ] 60007 45.860% 48.160% ######### ( 2, 3 ] 32712 25.000% 73.160% ##### ( 3, 4 ] 12983 9.922% 83.083% ## ( 4, 6 ] 15839 12.105% 95.188% ## ( 6, 10 ] 3685 2.816% 98.004% # ( 10, 15 ] 1346 1.029% 99.032% ( 15, 22 ] 203 0.155% 99.188% ( 22, 34 ] 16 0.012% 99.200% ( 34, 51 ] 4 0.003% 99.203% ( 51, 76 ] 2 0.002% 99.204% ( 170, 250 ] 8 0.006% 99.211% ( 250, 380 ] 14 0.011% 99.221% ( 380, 580 ] 30 0.023% 99.244% ( 580, 870 ] 95 0.073% 99.317% ( 870, 1300 ] 144 0.110% 99.427% ( 1300, 1900 ] 228 0.174% 99.601% ( 1900, 2900 ] 282 0.216% 99.817% ( 2900, 4400 ] 194 0.148% 99.965% ( 4400, 6600 ] 37 0.028% 99.993% ( 6600, 9900 ] 9 0.007% 100.000% ** Level 2 read latency histogram (micros): Count: 812192 Average: 16.7615 StdDev: 201.59 Min: 1 Median: 2.0265 Max: 37249 Percentiles: P50: 2.03 P75: 3.07 P99: 13.13 P99.9: 2334.90 P99.99: 2877.57 ------------------------------------------------------ [ 0, 1 ] 18607 2.291% 2.291% ( 1, 2 ] 382104 47.046% 49.337% ######### ( 2, 3 ] 203118 25.009% 74.346% ##### ( 3, 4 ] 77636 9.559% 83.904% ## ( 4, 6 ] 95311 11.735% 95.639% ## ( 6, 10 ] 22772 2.804% 98.443% # ( 10, 15 ] 7229 0.890% 99.333% ( 15, 22 ] 937 0.115% 99.449% ( 22, 34 ] 84 0.010% 99.459% ( 34, 51 ] 40 0.005% 99.464% ( 51, 76 ] 27 0.003% 99.467% ( 76, 110 ] 18 0.002% 99.469% ( 110, 170 ] 23 0.003% 99.472% ( 170, 250 ] 44 0.005% 99.478% ( 250, 380 ] 120 0.015% 99.492% ( 380, 580 ] 283 0.035% 99.527% ( 580, 870 ] 518 0.064% 99.591% ( 870, 1300 ] 860 0.106% 99.697% ( 1300, 1900 ] 1064 0.131% 99.828% ( 1900, 2900 ] 1347 0.166% 99.994% ( 2900, 4400 ] 923 0.114% 100.107% ( 4400, 6600 ] 239 0.029% 100.137% ( 6600, 9900 ] 38 0.005% 100.142% ( 9900, 14000 ] 3 0.000% 100.142% ( 33000, 50000 ] 1 0.000% 100.142% ** Level 3 read latency histogram (micros): Count: 1414999 Average: 13.8347 StdDev: 231.80 Min: 1 Median: 2.0588 Max: 41313 Percentiles: P50: 2.06 P75: 3.08 P99: 11.39 P99.9: 1407.87 P99.99: 2120.11 ------------------------------------------------------ [ 0, 1 ] 26747 1.890% 1.890% ( 1, 2 ] 659328 46.596% 48.486% ######### ( 2, 3 ] 364252 25.742% 74.228% ##### ( 3, 4 ] 137948 9.749% 83.977% ## ( 4, 6 ] 171810 12.142% 96.119% ## ( 6, 10 ] 37898 2.678% 98.797% # ( 10, 15 ] 10292 0.727% 99.525% ( 15, 22 ] 1137 0.080% 99.605% ( 22, 34 ] 104 0.007% 99.613% ( 34, 51 ] 68 0.005% 99.617% ( 51, 76 ] 125 0.009% 99.626% ( 76, 110 ] 41 0.003% 99.629% ( 110, 170 ] 45 0.003% 99.632% ( 170, 250 ] 128 0.009% 99.641% ( 250, 380 ] 269 0.019% 99.660% ( 380, 580 ] 607 0.043% 99.703% ( 580, 870 ] 1132 0.080% 99.783% ( 870, 1300 ] 1436 0.101% 99.885% ( 1300, 1900 ] 1207 0.085% 99.970% ( 1900, 2900 ] 1288 0.091% 100.061% ( 2900, 4400 ] 1025 0.072% 100.133% ( 4400, 6600 ] 316 0.022% 100.156% ( 6600, 9900 ] 75 0.005% 100.161% ( 9900, 14000 ] 28 0.002% 100.163% ( 14000, 22000 ] 35 0.002% 100.166% ( 22000, 33000 ] 27 0.002% 100.167% ( 33000, 50000 ] 3 0.000% 100.168% ** Level 4 read latency histogram (micros): Count: 871260 Average: 22.2347 StdDev: 334.21 Min: 1 Median: 2.0565 Max: 50090 Percentiles: P50: 2.06 P75: 3.15 P99: 18.96 P99.9: 2128.82 P99.99: 2915.72 ------------------------------------------------------ [ 0, 1 ] 17107 1.963% 1.963% ( 1, 2 ] 406210 46.623% 48.587% ######### ( 2, 3 ] 217742 24.992% 73.578% ##### ( 3, 4 ] 83630 9.599% 83.177% ## ( 4, 6 ] 104874 12.037% 95.214% ## ( 6, 10 ] 25563 2.934% 98.148% # ( 10, 15 ] 7030 0.807% 98.955% ( 15, 22 ] 691 0.079% 99.034% ( 22, 34 ] 66 0.008% 99.042% ( 34, 51 ] 67 0.008% 99.050% ( 51, 76 ] 106 0.012% 99.062% ( 76, 110 ] 58 0.007% 99.068% ( 110, 170 ] 140 0.016% 99.085% ( 170, 250 ] 403 0.046% 99.131% ( 250, 380 ] 546 0.063% 99.193% ( 380, 580 ] 1013 0.116% 99.310% ( 580, 870 ] 1778 0.204% 99.514% ( 870, 1300 ] 1922 0.221% 99.734% ( 1300, 1900 ] 1213 0.139% 99.874% ( 1900, 2900 ] 1004 0.115% 99.989% ( 2900, 4400 ] 942 0.108% 100.097% ( 4400, 6600 ] 338 0.039% 100.136% ( 6600, 9900 ] 87 0.010% 100.146% ( 9900, 14000 ] 27 0.003% 100.149% ( 14000, 22000 ] 68 0.008% 100.157% ( 22000, 33000 ] 30 0.003% 100.160% ( 33000, 50000 ] 6 0.001% 100.161% ( 50000, 75000 ] 1 0.000% 100.161% ** Level 5 read latency histogram (micros): Count: 25915 Average: 1665.7172 StdDev: 2167.67 Min: 87 Median: 1067.2129 Max: 62192 Percentiles: P50: 1067.21 P75: 2129.43 P99: 7998.80 P99.9: 30349.59 P99.99: 58803.12 ------------------------------------------------------ ( 76, 110 ] 45 0.174% 0.174% ( 110, 170 ] 550 2.122% 2.296% ( 170, 250 ] 1149 4.434% 6.730% # ( 250, 380 ] 1764 6.807% 13.537% # ( 380, 580 ] 2932 11.314% 24.850% ## ( 580, 870 ] 4361 16.828% 41.679% ### ( 870, 1300 ] 4702 18.144% 59.822% #### ( 1300, 1900 ] 3317 12.800% 72.622% ### ( 1900, 2900 ] 2686 10.365% 82.987% ## ( 2900, 4400 ] 2712 10.465% 93.452% ## ( 4400, 6600 ] 1334 5.148% 98.599% # ( 6600, 9900 ] 245 0.945% 99.545% ( 9900, 14000 ] 30 0.116% 99.660% ( 14000, 22000 ] 34 0.131% 99.792% ( 22000, 33000 ] 37 0.143% 99.934% ( 33000, 50000 ] 13 0.050% 99.985% ( 50000, 75000 ] 4 0.015% 100.000% ** DB Stats ** Uptime(secs): 15.1 total, 7.5 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 130.54 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.145 H:M:S, 47.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 130.37 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.620 H:M:S, 48.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 3 num-running-flushes: 1 2024/12/02-08:44:40 ... thread 0: (1000000,3000000) ops and (285115.9,163205.1) ops/second in (3.507346,18.381777) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/11 3.26 GB 188.8 2.3 0.0 2.3 5.0 2.8 0.0 1.8 78.1 172.9 29.86 17.66 45 0.664 2367K 2723 0.0 0.0 L2 0/0 0.00 KB 0.0 0.6 0.5 0.1 0.6 0.5 0.0 1.1 108.8 108.3 5.28 2.68 1 5.282 523K 0 0.0 0.0 L3 15/0 990.28 MB 1.0 5.3 3.4 1.9 5.3 3.4 0.6 1.5 104.4 103.7 51.87 25.16 24 2.161 2471K 2618 0.0 0.0 L4 174/0 9.75 GB 1.0 7.6 4.2 3.4 7.5 4.1 0.5 1.8 115.7 113.5 67.51 39.67 54 1.250 4275K 6804 0.0 0.0 L5 1128/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1335/11 108.34 GB 0.0 15.8 8.1 7.6 18.3 10.7 5.4 6.6 104.4 121.5 154.52 85.18 124 1.246 9637K 12K 0.0 0.0 Int 0/0 0.00 KB 0.0 15.8 8.1 7.6 18.3 10.7 5.4 6.6 104.4 121.5 154.52 85.18 124 1.246 9637K 12K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 15.8 8.1 7.6 15.6 7.9 0.0 0.0 117.7 116.3 137.09 76.91 87 1.576 9637K 12K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 162.6 17.43 8.26 37 0.471 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.6 total, 18.4 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 18.34 GB write, 1010.37 MB/s write, 15.75 GB read, 867.91 MB/s read, 154.5 seconds Interval compaction: 18.34 GB write, 1021.52 MB/s write, 15.75 GB read, 877.49 MB/s read, 154.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 1, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 1, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 63, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 1, total-stops: 63, interval: 64 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 658776 Average: 27.8682 StdDev: 311.74 Min: 1 Median: 1.7179 Max: 62243 Percentiles: P50: 1.72 P75: 2.59 P99: 590.98 P99.9: 2903.00 P99.99: 4358.57 ------------------------------------------------------ [ 0, 1 ] 68493 10.397% 10.397% ## ( 1, 2 ] 363400 55.163% 65.560% ########### ( 2, 3 ] 105724 16.049% 81.608% ### ( 3, 4 ] 38620 5.862% 87.471% # ( 4, 6 ] 43780 6.646% 94.117% # ( 6, 10 ] 8264 1.254% 95.371% ( 10, 15 ] 1960 0.298% 95.668% ( 15, 22 ] 265 0.040% 95.709% ( 22, 34 ] 14 0.002% 95.711% ( 34, 51 ] 7 0.001% 95.712% ( 51, 76 ] 14 0.002% 95.714% ( 76, 110 ] 847 0.129% 95.843% ( 110, 170 ] 7756 1.177% 97.020% ( 170, 250 ] 6941 1.054% 98.074% ( 250, 380 ] 3102 0.471% 98.544% ( 380, 580 ] 2911 0.442% 98.986% ( 580, 870 ] 2436 0.370% 99.356% ( 870, 1300 ] 1824 0.277% 99.633% ( 1300, 1900 ] 1067 0.162% 99.795% ( 1900, 2900 ] 693 0.105% 99.900% ( 2900, 4400 ] 611 0.093% 99.993% ( 4400, 6600 ] 235 0.036% 100.029% ( 6600, 9900 ] 34 0.005% 100.034% ( 9900, 14000 ] 6 0.001% 100.035% ( 14000, 22000 ] 13 0.002% 100.037% ( 22000, 33000 ] 25 0.004% 100.040% ( 33000, 50000 ] 6 0.001% 100.041% ( 50000, 75000 ] 1 0.000% 100.041% ** Level 1 read latency histogram (micros): Count: 130847 Average: 20.0512 StdDev: 219.99 Min: 1 Median: 2.0736 Max: 9518 Percentiles: P50: 2.07 P75: 3.19 P99: 14.84 P99.9: 3743.97 P99.99: 6357.13 ------------------------------------------------------ [ 0, 1 ] 3009 2.300% 2.300% ( 1, 2 ] 60007 45.860% 48.160% ######### ( 2, 3 ] 32712 25.000% 73.160% ##### ( 3, 4 ] 12983 9.922% 83.083% ## ( 4, 6 ] 15839 12.105% 95.188% ## ( 6, 10 ] 3685 2.816% 98.004% # ( 10, 15 ] 1346 1.029% 99.032% ( 15, 22 ] 203 0.155% 99.188% ( 22, 34 ] 16 0.012% 99.200% ( 34, 51 ] 4 0.003% 99.203% ( 51, 76 ] 2 0.002% 99.204% ( 170, 250 ] 8 0.006% 99.211% ( 250, 380 ] 14 0.011% 99.221% ( 380, 580 ] 30 0.023% 99.244% ( 580, 870 ] 95 0.073% 99.317% ( 870, 1300 ] 144 0.110% 99.427% ( 1300, 1900 ] 228 0.174% 99.601% ( 1900, 2900 ] 282 0.216% 99.817% ( 2900, 4400 ] 194 0.148% 99.965% ( 4400, 6600 ] 37 0.028% 99.993% ( 6600, 9900 ] 9 0.007% 100.000% ** Level 2 read latency histogram (micros): Count: 818916 Average: 16.6813 StdDev: 200.79 Min: 1 Median: 2.0211 Max: 37249 Percentiles: P50: 2.02 P75: 3.06 P99: 13.18 P99.9: 2329.91 P99.99: 2877.07 ------------------------------------------------------ [ 0, 1 ] 18999 2.320% 2.320% ( 1, 2 ] 386137 47.152% 49.472% ######### ( 2, 3 ] 204398 24.960% 74.432% ##### ( 3, 4 ] 77972 9.521% 83.953% ## ( 4, 6 ] 95675 11.683% 95.636% ## ( 6, 10 ] 22921 2.799% 98.435% # ( 10, 15 ] 7262 0.887% 99.322% ( 15, 22 ] 947 0.116% 99.438% ( 22, 34 ] 85 0.010% 99.448% ( 34, 51 ] 40 0.005% 99.453% ( 51, 76 ] 27 0.003% 99.456% ( 76, 110 ] 19 0.002% 99.459% ( 110, 170 ] 62 0.008% 99.466% ( 170, 250 ] 97 0.012% 99.478% ( 250, 380 ] 139 0.017% 99.495% ( 380, 580 ] 293 0.036% 99.531% ( 580, 870 ] 519 0.063% 99.594% ( 870, 1300 ] 862 0.105% 99.699% ( 1300, 1900 ] 1064 0.130% 99.829% ( 1900, 2900 ] 1347 0.164% 99.994% ( 2900, 4400 ] 923 0.113% 100.106% ( 4400, 6600 ] 239 0.029% 100.136% ( 6600, 9900 ] 38 0.005% 100.140% ( 9900, 14000 ] 3 0.000% 100.141% ( 33000, 50000 ] 1 0.000% 100.141% ** Level 3 read latency histogram (micros): Count: 1415659 Average: 13.9178 StdDev: 231.80 Min: 1 Median: 2.0597 Max: 41313 Percentiles: P50: 2.06 P75: 3.08 P99: 11.65 P99.9: 1407.54 P99.99: 2120.06 ------------------------------------------------------ [ 0, 1 ] 26747 1.889% 1.889% ( 1, 2 ] 659333 46.574% 48.464% ######### ( 2, 3 ] 364294 25.733% 74.197% ##### ( 3, 4 ] 137966 9.746% 83.943% ## ( 4, 6 ] 171858 12.140% 96.082% ## ( 6, 10 ] 37907 2.678% 98.760% # ( 10, 15 ] 10293 0.727% 99.487% ( 15, 22 ] 1138 0.080% 99.567% ( 22, 34 ] 104 0.007% 99.575% ( 34, 51 ] 68 0.005% 99.580% ( 51, 76 ] 125 0.009% 99.588% ( 76, 110 ] 59 0.004% 99.593% ( 110, 170 ] 240 0.017% 99.610% ( 170, 250 ] 317 0.022% 99.632% ( 250, 380 ] 330 0.023% 99.655% ( 380, 580 ] 660 0.047% 99.702% ( 580, 870 ] 1149 0.081% 99.783% ( 870, 1300 ] 1439 0.102% 99.885% ( 1300, 1900 ] 1207 0.085% 99.970% ( 1900, 2900 ] 1288 0.091% 100.061% ( 2900, 4400 ] 1025 0.072% 100.133% ( 4400, 6600 ] 316 0.022% 100.156% ( 6600, 9900 ] 75 0.005% 100.161% ( 9900, 14000 ] 28 0.002% 100.163% ( 14000, 22000 ] 35 0.002% 100.165% ( 22000, 33000 ] 27 0.002% 100.167% ( 33000, 50000 ] 3 0.000% 100.168% ** Level 4 read latency histogram (micros): Count: 897129 Average: 23.1429 StdDev: 330.05 Min: 1 Median: 2.0582 Max: 50090 Percentiles: P50: 2.06 P75: 3.18 P99: 251.31 P99.9: 2103.85 P99.99: 2911.58 ------------------------------------------------------ [ 0, 1 ] 18038 2.011% 2.011% ( 1, 2 ] 417593 46.548% 48.558% ######### ( 2, 3 ] 222264 24.775% 73.333% ##### ( 3, 4 ] 84908 9.464% 82.798% ## ( 4, 6 ] 106326 11.852% 94.650% ## ( 6, 10 ] 26003 2.898% 97.548% # ( 10, 15 ] 7152 0.797% 98.345% ( 15, 22 ] 716 0.080% 98.425% ( 22, 34 ] 71 0.008% 98.433% ( 34, 51 ] 67 0.007% 98.440% ( 51, 76 ] 106 0.012% 98.452% ( 76, 110 ] 258 0.029% 98.481% ( 110, 170 ] 2326 0.259% 98.740% ( 170, 250 ] 2317 0.258% 98.999% ( 250, 380 ] 1260 0.140% 99.139% ( 380, 580 ] 1485 0.166% 99.305% ( 580, 870 ] 1963 0.219% 99.523% ( 870, 1300 ] 1956 0.218% 99.741% ( 1300, 1900 ] 1218 0.136% 99.877% ( 1900, 2900 ] 1005 0.112% 99.989% ( 2900, 4400 ] 944 0.105% 100.094% ( 4400, 6600 ] 338 0.038% 100.132% ( 6600, 9900 ] 87 0.010% 100.142% ( 9900, 14000 ] 27 0.003% 100.145% ( 14000, 22000 ] 68 0.008% 100.152% ( 22000, 33000 ] 30 0.003% 100.156% ( 33000, 50000 ] 6 0.001% 100.156% ( 50000, 75000 ] 1 0.000% 100.156% ** Level 5 read latency histogram (micros): Count: 64292 Average: 809.1080 StdDev: 1550.50 Min: 83 Median: 272.8727 Max: 62192 Percentiles: P50: 272.87 P75: 832.87 P99: 6138.36 P99.9: 19578.35 P99.99: 46823.35 ------------------------------------------------------ ( 76, 110 ] 1362 2.118% 2.118% ( 110, 170 ] 15447 24.026% 26.145% ##### ( 170, 250 ] 14237 22.144% 48.289% #### ( 250, 380 ] 6252 9.724% 58.013% ## ( 380, 580 ] 6092 9.476% 67.489% ## ( 580, 870 ] 5538 8.614% 76.103% ## ( 870, 1300 ] 4899 7.620% 83.723% ## ( 1300, 1900 ] 3343 5.200% 88.922% # ( 1900, 2900 ] 2706 4.209% 93.131% # ( 2900, 4400 ] 2719 4.229% 97.360% # ( 4400, 6600 ] 1334 2.075% 99.435% ( 6600, 9900 ] 245 0.381% 99.816% ( 9900, 14000 ] 30 0.047% 99.863% ( 14000, 22000 ] 34 0.053% 99.916% ( 22000, 33000 ] 37 0.058% 99.974% ( 33000, 50000 ] 13 0.020% 99.994% ( 50000, 75000 ] 4 0.006% 100.000% ** DB Stats ** Uptime(secs): 18.6 total, 3.5 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 158.86 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.245 H:M:S, 39.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 280.61 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.100 H:M:S, 2.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/02-08:44:51 ... thread 0: (1000000,4000000) ops and (92510.2,137026.7) ops/second in (10.809618,29.191395) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/11 4.23 GB 288.4 4.8 0.0 4.8 8.5 3.7 0.0 2.3 114.9 204.4 42.77 29.93 66 0.648 4988K 14K 0.0 0.0 L2 0/0 0.00 KB 0.0 0.6 0.5 0.1 0.6 0.5 0.0 1.1 108.8 108.3 5.28 2.68 1 5.282 523K 0 0.0 0.0 L3 15/0 990.28 MB 1.0 5.3 3.4 1.9 5.3 3.4 0.6 1.5 104.4 103.7 51.87 25.16 24 2.161 2471K 2618 0.0 0.0 L4 174/0 9.75 GB 1.0 7.6 4.2 3.4 7.5 4.1 0.5 1.8 115.7 113.5 67.51 39.67 54 1.250 4275K 6804 0.0 0.0 L5 1128/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1333/11 109.31 GB 0.0 18.3 8.1 10.2 21.8 11.7 5.4 5.8 111.8 133.5 167.43 97.45 145 1.155 12M 23K 0.0 0.0 Int 0/0 0.00 KB 0.0 18.3 8.1 10.2 21.8 11.7 5.4 5.8 111.8 133.5 167.43 97.45 145 1.155 12M 23K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 18.3 8.1 10.2 18.1 7.9 0.0 0.0 127.6 126.2 146.70 86.41 92 1.595 12M 23K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 185.4 20.72 11.04 53 0.391 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 29.4 total, 29.2 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 21.83 GB write, 760.54 MB/s write, 18.27 GB read, 636.58 MB/s read, 167.4 seconds Interval compaction: 21.83 GB write, 765.83 MB/s write, 18.27 GB read, 641.00 MB/s read, 167.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 12, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 12, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 63, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 12, total-stops: 63, interval: 75 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1381511 Average: 24.0601 StdDev: 220.69 Min: 1 Median: 1.5879 Max: 62243 Percentiles: P50: 1.59 P75: 2.21 P99: 363.15 P99.9: 1670.78 P99.99: 3169.78 ------------------------------------------------------ [ 0, 1 ] 243046 17.593% 17.593% #### ( 1, 2 ] 761565 55.126% 72.718% ########### ( 2, 3 ] 151844 10.991% 83.709% ## ( 3, 4 ] 54932 3.976% 87.686% # ( 4, 6 ] 56110 4.061% 91.747% # ( 6, 10 ] 10262 0.743% 92.490% ( 10, 15 ] 2343 0.170% 92.660% ( 15, 22 ] 316 0.023% 92.682% ( 22, 34 ] 18 0.001% 92.684% ( 34, 51 ] 10 0.001% 92.684% ( 51, 76 ] 15 0.001% 92.686% ( 76, 110 ] 5085 0.368% 93.054% ( 110, 170 ] 46719 3.382% 96.435% # ( 170, 250 ] 28965 2.097% 98.532% ( 250, 380 ] 7430 0.538% 99.070% ( 380, 580 ] 5713 0.414% 99.483% ( 580, 870 ] 3186 0.231% 99.714% ( 870, 1300 ] 1901 0.138% 99.852% ( 1300, 1900 ] 1085 0.079% 99.930% ( 1900, 2900 ] 715 0.052% 99.982% ( 2900, 4400 ] 633 0.046% 100.028% ( 4400, 6600 ] 236 0.017% 100.045% ( 6600, 9900 ] 34 0.002% 100.047% ( 9900, 14000 ] 6 0.000% 100.048% ( 14000, 22000 ] 13 0.001% 100.049% ( 22000, 33000 ] 25 0.002% 100.050% ( 33000, 50000 ] 6 0.000% 100.051% ( 50000, 75000 ] 1 0.000% 100.051% ** Level 1 read latency histogram (micros): Count: 130847 Average: 20.0512 StdDev: 219.99 Min: 1 Median: 2.0736 Max: 9518 Percentiles: P50: 2.07 P75: 3.19 P99: 14.84 P99.9: 3743.97 P99.99: 6357.13 ------------------------------------------------------ [ 0, 1 ] 3009 2.300% 2.300% ( 1, 2 ] 60007 45.860% 48.160% ######### ( 2, 3 ] 32712 25.000% 73.160% ##### ( 3, 4 ] 12983 9.922% 83.083% ## ( 4, 6 ] 15839 12.105% 95.188% ## ( 6, 10 ] 3685 2.816% 98.004% # ( 10, 15 ] 1346 1.029% 99.032% ( 15, 22 ] 203 0.155% 99.188% ( 22, 34 ] 16 0.012% 99.200% ( 34, 51 ] 4 0.003% 99.203% ( 51, 76 ] 2 0.002% 99.204% ( 170, 250 ] 8 0.006% 99.211% ( 250, 380 ] 14 0.011% 99.221% ( 380, 580 ] 30 0.023% 99.244% ( 580, 870 ] 95 0.073% 99.317% ( 870, 1300 ] 144 0.110% 99.427% ( 1300, 1900 ] 228 0.174% 99.601% ( 1900, 2900 ] 282 0.216% 99.817% ( 2900, 4400 ] 194 0.148% 99.965% ( 4400, 6600 ] 37 0.028% 99.993% ( 6600, 9900 ] 9 0.007% 100.000% ** Level 2 read latency histogram (micros): Count: 819276 Average: 16.7540 StdDev: 200.78 Min: 1 Median: 2.0220 Max: 37249 Percentiles: P50: 2.02 P75: 3.06 P99: 13.43 P99.9: 2329.64 P99.99: 2877.04 ------------------------------------------------------ [ 0, 1 ] 18999 2.319% 2.319% ( 1, 2 ] 386137 47.131% 49.450% ######### ( 2, 3 ] 204398 24.949% 74.399% ##### ( 3, 4 ] 77972 9.517% 83.916% ## ( 4, 6 ] 95675 11.678% 95.594% ## ( 6, 10 ] 22921 2.798% 98.392% # ( 10, 15 ] 7262 0.886% 99.278% ( 15, 22 ] 947 0.116% 99.394% ( 22, 34 ] 85 0.010% 99.404% ( 34, 51 ] 40 0.005% 99.409% ( 51, 76 ] 27 0.003% 99.413% ( 76, 110 ] 34 0.004% 99.417% ( 110, 170 ] 249 0.030% 99.447% ( 170, 250 ] 215 0.026% 99.473% ( 250, 380 ] 165 0.020% 99.493% ( 380, 580 ] 304 0.037% 99.531% ( 580, 870 ] 522 0.064% 99.594% ( 870, 1300 ] 862 0.105% 99.699% ( 1300, 1900 ] 1064 0.130% 99.829% ( 1900, 2900 ] 1347 0.164% 99.994% ( 2900, 4400 ] 923 0.113% 100.106% ( 4400, 6600 ] 239 0.029% 100.136% ( 6600, 9900 ] 38 0.005% 100.140% ( 9900, 14000 ] 3 0.000% 100.141% ( 33000, 50000 ] 1 0.000% 100.141% ** Level 3 read latency histogram (micros): Count: 1418203 Average: 14.1863 StdDev: 231.76 Min: 1 Median: 2.0631 Max: 41313 Percentiles: P50: 2.06 P75: 3.10 P99: 12.71 P99.9: 1408.76 P99.99: 2122.88 ------------------------------------------------------ [ 0, 1 ] 26747 1.886% 1.886% ( 1, 2 ] 659358 46.492% 48.378% ######### ( 2, 3 ] 364372 25.693% 74.071% ##### ( 3, 4 ] 138025 9.732% 83.803% ## ( 4, 6 ] 172009 12.129% 95.932% ## ( 6, 10 ] 37923 2.674% 98.606% # ( 10, 15 ] 10299 0.726% 99.332% ( 15, 22 ] 1140 0.080% 99.413% ( 22, 34 ] 104 0.007% 99.420% ( 34, 51 ] 68 0.005% 99.425% ( 51, 76 ] 125 0.009% 99.434% ( 76, 110 ] 166 0.012% 99.445% ( 110, 170 ] 1407 0.099% 99.544% ( 170, 250 ] 1035 0.073% 99.617% ( 250, 380 ] 440 0.031% 99.648% ( 380, 580 ] 741 0.052% 99.701% ( 580, 870 ] 1166 0.082% 99.783% ( 870, 1300 ] 1441 0.102% 99.885% ( 1300, 1900 ] 1207 0.085% 99.970% ( 1900, 2900 ] 1293 0.091% 100.061% ( 2900, 4400 ] 1025 0.072% 100.133% ( 4400, 6600 ] 316 0.022% 100.155% ( 6600, 9900 ] 75 0.005% 100.161% ( 9900, 14000 ] 28 0.002% 100.163% ( 14000, 22000 ] 35 0.002% 100.165% ( 22000, 33000 ] 27 0.002% 100.167% ( 33000, 50000 ] 3 0.000% 100.167% ** Level 4 read latency histogram (micros): Count: 921438 Average: 27.1701 StdDev: 327.04 Min: 1 Median: 2.1123 Max: 50090 Percentiles: P50: 2.11 P75: 3.38 P99: 363.72 P99.9: 2083.28 P99.99: 2910.87 ------------------------------------------------------ [ 0, 1 ] 18038 1.958% 1.958% ( 1, 2 ] 417687 45.330% 47.288% ######### ( 2, 3 ] 222596 24.157% 71.445% ##### ( 3, 4 ] 85189 9.245% 80.690% ## ( 4, 6 ] 106914 11.603% 92.293% ## ( 6, 10 ] 26069 2.829% 95.122% # ( 10, 15 ] 7173 0.778% 95.901% ( 15, 22 ] 719 0.078% 95.979% ( 22, 34 ] 71 0.008% 95.986% ( 34, 51 ] 67 0.007% 95.994% ( 51, 76 ] 106 0.012% 96.005% ( 76, 110 ] 1354 0.147% 96.152% ( 110, 170 ] 14509 1.575% 97.727% ( 170, 250 ] 9431 1.024% 98.750% ( 250, 380 ] 2630 0.285% 99.036% ( 380, 580 ] 2408 0.261% 99.297% ( 580, 870 ] 2166 0.235% 99.532% ( 870, 1300 ] 1986 0.216% 99.748% ( 1300, 1900 ] 1219 0.132% 99.880% ( 1900, 2900 ] 1007 0.109% 99.989% ( 2900, 4400 ] 946 0.103% 100.092% ( 4400, 6600 ] 338 0.037% 100.129% ( 6600, 9900 ] 87 0.009% 100.138% ( 9900, 14000 ] 27 0.003% 100.141% ( 14000, 22000 ] 68 0.007% 100.148% ( 22000, 33000 ] 30 0.003% 100.152% ( 33000, 50000 ] 6 0.001% 100.152% ( 50000, 75000 ] 1 0.000% 100.152% ** Level 5 read latency histogram (micros): Count: 207744 Average: 379.2704 StdDev: 913.25 Min: 80 Median: 175.7400 Max: 62192 Percentiles: P50: 175.74 P75: 246.33 P99: 4189.96 P99.9: 8677.73 P99.99: 31580.58 ------------------------------------------------------ ( 76, 110 ] 8434 4.060% 4.060% # ( 110, 170 ] 91215 43.907% 47.967% ######### ( 170, 250 ] 58857 28.332% 76.299% ###### ( 250, 380 ] 14947 7.195% 83.494% # ( 380, 580 ] 11760 5.661% 89.154% # ( 580, 870 ] 6965 3.353% 92.507% # ( 870, 1300 ] 5041 2.427% 94.934% ( 1300, 1900 ] 3372 1.623% 96.557% ( 1900, 2900 ] 2733 1.316% 97.872% ( 2900, 4400 ] 2724 1.311% 99.184% ( 4400, 6600 ] 1334 0.642% 99.826% ( 6600, 9900 ] 245 0.118% 99.944% ( 9900, 14000 ] 30 0.014% 99.958% ( 14000, 22000 ] 34 0.016% 99.974% ( 22000, 33000 ] 37 0.018% 99.992% ( 33000, 50000 ] 13 0.006% 99.999% ( 50000, 75000 ] 4 0.002% 100.000% ** DB Stats ** Uptime(secs): 29.4 total, 10.8 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 133.92 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:14.342 H:M:S, 48.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 91.05 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:7.098 H:M:S, 65.7 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/02-08:44:56 ... thread 0: (1000000,5000000) ops and (177795.4,143612.8) ops/second in (5.624444,34.815839) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/13 5.28 GB 288.4 6.0 0.0 6.0 10.8 4.8 0.0 2.3 121.2 217.3 50.94 37.48 88 0.579 6268K 15K 0.0 0.0 L2 0/0 0.00 KB 0.0 0.6 0.5 0.1 0.6 0.5 0.0 1.1 108.8 108.3 5.28 2.68 1 5.282 523K 0 0.0 0.0 L3 15/0 990.28 MB 1.0 5.3 3.4 1.9 5.3 3.4 0.6 1.5 104.4 103.7 51.87 25.16 24 2.161 2471K 2618 0.0 0.0 L4 174/0 9.75 GB 1.0 7.6 4.2 3.4 7.5 4.1 0.5 1.8 115.7 113.5 67.51 39.67 54 1.250 4275K 6804 0.0 0.0 L5 1128/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1335/13 110.36 GB 0.0 19.5 8.1 11.4 24.1 12.7 5.4 5.0 113.7 140.6 175.60 104.99 167 1.052 13M 25K 0.0 0.0 Int 0/0 0.00 KB 0.0 19.5 8.1 11.4 24.1 12.7 5.4 5.0 113.7 140.6 175.60 104.99 167 1.052 13M 25K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 19.5 8.1 11.4 19.3 7.9 0.0 0.0 131.9 130.5 151.47 91.09 97 1.562 13M 25K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.8 4.8 0.0 0.0 0.0 203.6 24.14 13.90 70 0.345 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 35.0 total, 34.8 interval Flush(GB): cumulative 4.799, interval 4.799 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 24.11 GB write, 704.91 MB/s write, 19.50 GB read, 570.31 MB/s read, 175.6 seconds Interval compaction: 24.11 GB write, 709.02 MB/s write, 19.50 GB read, 573.64 MB/s read, 175.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 21, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 21, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 63, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 21, total-stops: 63, interval: 84 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1739862 Average: 23.6500 StdDev: 199.59 Min: 1 Median: 1.5680 Max: 62243 Percentiles: P50: 1.57 P75: 2.11 P99: 350.47 P99.9: 1386.18 P99.99: 2728.21 ------------------------------------------------------ [ 0, 1 ] 322752 18.550% 18.550% #### ( 1, 2 ] 963415 55.373% 73.924% ########### ( 2, 3 ] 177088 10.178% 84.102% ## ( 3, 4 ] 62757 3.607% 87.709% # ( 4, 6 ] 62097 3.569% 91.278% # ( 6, 10 ] 11348 0.652% 91.930% ( 10, 15 ] 2640 0.152% 92.082% ( 15, 22 ] 353 0.020% 92.102% ( 22, 34 ] 24 0.001% 92.104% ( 34, 51 ] 12 0.001% 92.104% ( 51, 76 ] 18 0.001% 92.105% ( 76, 110 ] 6953 0.400% 92.505% ( 110, 170 ] 64440 3.704% 96.209% # ( 170, 250 ] 40567 2.332% 98.540% ( 250, 380 ] 10352 0.595% 99.135% ( 380, 580 ] 7480 0.430% 99.565% ( 580, 870 ] 3693 0.212% 99.777% ( 870, 1300 ] 1975 0.114% 99.891% ( 1300, 1900 ] 1108 0.064% 99.955% ( 1900, 2900 ] 745 0.043% 99.997% ( 2900, 4400 ] 645 0.037% 100.034% ( 4400, 6600 ] 236 0.014% 100.048% ( 6600, 9900 ] 34 0.002% 100.050% ( 9900, 14000 ] 6 0.000% 100.050% ( 14000, 22000 ] 13 0.001% 100.051% ( 22000, 33000 ] 25 0.001% 100.053% ( 33000, 50000 ] 6 0.000% 100.053% ( 50000, 75000 ] 1 0.000% 100.053% ** Level 1 read latency histogram (micros): Count: 130847 Average: 20.0512 StdDev: 219.99 Min: 1 Median: 2.0736 Max: 9518 Percentiles: P50: 2.07 P75: 3.19 P99: 14.84 P99.9: 3743.97 P99.99: 6357.13 ------------------------------------------------------ [ 0, 1 ] 3009 2.300% 2.300% ( 1, 2 ] 60007 45.860% 48.160% ######### ( 2, 3 ] 32712 25.000% 73.160% ##### ( 3, 4 ] 12983 9.922% 83.083% ## ( 4, 6 ] 15839 12.105% 95.188% ## ( 6, 10 ] 3685 2.816% 98.004% # ( 10, 15 ] 1346 1.029% 99.032% ( 15, 22 ] 203 0.155% 99.188% ( 22, 34 ] 16 0.012% 99.200% ( 34, 51 ] 4 0.003% 99.203% ( 51, 76 ] 2 0.002% 99.204% ( 170, 250 ] 8 0.006% 99.211% ( 250, 380 ] 14 0.011% 99.221% ( 380, 580 ] 30 0.023% 99.244% ( 580, 870 ] 95 0.073% 99.317% ( 870, 1300 ] 144 0.110% 99.427% ( 1300, 1900 ] 228 0.174% 99.601% ( 1900, 2900 ] 282 0.216% 99.817% ( 2900, 4400 ] 194 0.148% 99.965% ( 4400, 6600 ] 37 0.028% 99.993% ( 6600, 9900 ] 9 0.007% 100.000% ** Level 2 read latency histogram (micros): Count: 819446 Average: 16.7940 StdDev: 200.80 Min: 1 Median: 2.0224 Max: 37249 Percentiles: P50: 2.02 P75: 3.06 P99: 13.55 P99.9: 2329.94 P99.99: 2877.04 ------------------------------------------------------ [ 0, 1 ] 18999 2.319% 2.319% ( 1, 2 ] 386137 47.122% 49.440% ######### ( 2, 3 ] 204398 24.943% 74.384% ##### ( 3, 4 ] 77972 9.515% 83.899% ## ( 4, 6 ] 95675 11.676% 95.574% ## ( 6, 10 ] 22921 2.797% 98.372% # ( 10, 15 ] 7262 0.886% 99.258% ( 15, 22 ] 947 0.116% 99.373% ( 22, 34 ] 85 0.010% 99.384% ( 34, 51 ] 40 0.005% 99.389% ( 51, 76 ] 27 0.003% 99.392% ( 76, 110 ] 42 0.005% 99.397% ( 110, 170 ] 327 0.040% 99.437% ( 170, 250 ] 268 0.033% 99.470% ( 250, 380 ] 186 0.023% 99.492% ( 380, 580 ] 311 0.038% 99.530% ( 580, 870 ] 524 0.064% 99.594% ( 870, 1300 ] 862 0.105% 99.699% ( 1300, 1900 ] 1064 0.130% 99.829% ( 1900, 2900 ] 1348 0.165% 99.994% ( 2900, 4400 ] 923 0.113% 100.106% ( 4400, 6600 ] 239 0.029% 100.136% ( 6600, 9900 ] 38 0.005% 100.140% ( 9900, 14000 ] 3 0.000% 100.141% ( 33000, 50000 ] 1 0.000% 100.141% ** Level 3 read latency histogram (micros): Count: 1419427 Average: 14.3391 StdDev: 231.75 Min: 1 Median: 2.0648 Max: 41313 Percentiles: P50: 2.06 P75: 3.10 P99: 13.28 P99.9: 1409.06 P99.99: 2123.38 ------------------------------------------------------ [ 0, 1 ] 26747 1.884% 1.884% ( 1, 2 ] 659360 46.453% 48.337% ######### ( 2, 3 ] 364384 25.671% 74.008% ##### ( 3, 4 ] 138028 9.724% 83.732% ## ( 4, 6 ] 172022 12.119% 95.851% ## ( 6, 10 ] 37926 2.672% 98.523% # ( 10, 15 ] 10299 0.726% 99.249% ( 15, 22 ] 1140 0.080% 99.329% ( 22, 34 ] 104 0.007% 99.337% ( 34, 51 ] 68 0.005% 99.341% ( 51, 76 ] 125 0.009% 99.350% ( 76, 110 ] 230 0.016% 99.366% ( 110, 170 ] 1971 0.139% 99.505% ( 170, 250 ] 1433 0.101% 99.606% ( 250, 380 ] 537 0.038% 99.644% ( 380, 580 ] 788 0.056% 99.700% ( 580, 870 ] 1183 0.083% 99.783% ( 870, 1300 ] 1443 0.102% 99.885% ( 1300, 1900 ] 1208 0.085% 99.970% ( 1900, 2900 ] 1294 0.091% 100.061% ( 2900, 4400 ] 1025 0.072% 100.133% ( 4400, 6600 ] 316 0.022% 100.155% ( 6600, 9900 ] 75 0.005% 100.161% ( 9900, 14000 ] 28 0.002% 100.163% ( 14000, 22000 ] 35 0.002% 100.165% ( 22000, 33000 ] 27 0.002% 100.167% ( 33000, 50000 ] 3 0.000% 100.167% ** Level 4 read latency histogram (micros): Count: 933476 Average: 29.3485 StdDev: 325.79 Min: 1 Median: 2.1393 Max: 50090 Percentiles: P50: 2.14 P75: 3.49 P99: 403.52 P99.9: 2076.41 P99.99: 2910.54 ------------------------------------------------------ [ 0, 1 ] 18038 1.932% 1.932% ( 1, 2 ] 417689 44.746% 46.678% ######### ( 2, 3 ] 222623 23.849% 70.527% ##### ( 3, 4 ] 85207 9.128% 79.655% ## ( 4, 6 ] 106957 11.458% 91.113% ## ( 6, 10 ] 26075 2.793% 93.906% # ( 10, 15 ] 7175 0.769% 94.675% ( 15, 22 ] 720 0.077% 94.752% ( 22, 34 ] 71 0.008% 94.759% ( 34, 51 ] 67 0.007% 94.766% ( 51, 76 ] 106 0.011% 94.778% ( 76, 110 ] 1954 0.209% 94.987% ( 110, 170 ] 20395 2.185% 97.172% ( 170, 250 ] 13198 1.414% 98.586% ( 250, 380 ] 3516 0.377% 98.962% ( 380, 580 ] 2978 0.319% 99.282% ( 580, 870 ] 2347 0.251% 99.533% ( 870, 1300 ] 2023 0.217% 99.750% ( 1300, 1900 ] 1225 0.131% 99.881% ( 1900, 2900 ] 1012 0.108% 99.989% ( 2900, 4400 ] 947 0.101% 100.091% ( 4400, 6600 ] 338 0.036% 100.127% ( 6600, 9900 ] 87 0.009% 100.136% ( 9900, 14000 ] 27 0.003% 100.139% ( 14000, 22000 ] 68 0.007% 100.146% ( 22000, 33000 ] 30 0.003% 100.150% ( 33000, 50000 ] 6 0.001% 100.150% ( 50000, 75000 ] 1 0.000% 100.150% ** Level 5 read latency histogram (micros): Count: 277727 Average: 333.4869 StdDev: 796.22 Min: 80 Median: 171.5577 Max: 62192 Percentiles: P50: 171.56 P75: 239.94 P99: 3806.33 P99.9: 7735.11 P99.99: 29500.01 ------------------------------------------------------ ( 76, 110 ] 11772 4.239% 4.239% # ( 110, 170 ] 125510 45.192% 49.431% ######### ( 170, 250 ] 81224 29.246% 78.677% ###### ( 250, 380 ] 20188 7.269% 85.946% # ( 380, 580 ] 15383 5.539% 91.484% # ( 580, 870 ] 7890 2.841% 94.325% # ( 870, 1300 ] 5172 1.862% 96.188% ( 1300, 1900 ] 3399 1.224% 97.411% ( 1900, 2900 ] 2761 0.994% 98.406% ( 2900, 4400 ] 2732 0.984% 99.389% ( 4400, 6600 ] 1334 0.480% 99.870% ( 6600, 9900 ] 245 0.088% 99.958% ( 9900, 14000 ] 30 0.011% 99.969% ( 14000, 22000 ] 34 0.012% 99.981% ( 22000, 33000 ] 37 0.013% 99.994% ( 33000, 50000 ] 13 0.005% 99.999% ( 50000, 75000 ] 4 0.001% 100.000% ** DB Stats ** Uptime(secs): 35.0 total, 5.6 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 140.51 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:16.472 H:M:S, 47.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 174.95 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.129 H:M:S, 37.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 0 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 293.590 micros/op 13614 ops/sec 60.155 seconds 818996 operations; 13.2 MB/s (198999 of 198999 found) Microseconds per read: Count: 818996 Average: 293.5924 StdDev: 1025.08 Min: 1 Median: 203.9610 Max: 212017 Percentiles: P50: 203.96 P75: 248.33 P99: 1601.90 P99.9: 18136.88 P99.99: 37936.23 ------------------------------------------------------ [ 0, 1 ] 1 0.000% 0.000% ( 1, 2 ] 3 0.000% 0.000% ( 2, 3 ] 27 0.003% 0.004% ( 3, 4 ] 30 0.004% 0.007% ( 4, 6 ] 77 0.009% 0.017% ( 6, 10 ] 306 0.037% 0.054% ( 10, 15 ] 3470 0.424% 0.478% ( 15, 22 ] 3234 0.395% 0.873% ( 22, 34 ] 6182 0.755% 1.628% ( 34, 51 ] 2495 0.305% 1.932% ( 51, 76 ] 340 0.042% 1.974% ( 76, 110 ] 2633 0.321% 2.295% ( 110, 170 ] 233985 28.570% 30.865% ###### ( 170, 250 ] 369165 45.075% 75.940% ######### ( 250, 380 ] 108546 13.254% 89.194% ### ( 380, 580 ] 53001 6.471% 95.665% # ( 580, 870 ] 18142 2.215% 97.880% ( 870, 1300 ] 7498 0.916% 98.796% ( 1300, 1900 ] 3321 0.405% 99.201% ( 1900, 2900 ] 2337 0.285% 99.487% ( 2900, 4400 ] 1951 0.238% 99.725% ( 4400, 6600 ] 710 0.087% 99.812% ( 6600, 9900 ] 178 0.022% 99.833% ( 9900, 14000 ] 152 0.019% 99.852% ( 14000, 22000 ] 760 0.093% 99.945% ( 22000, 33000 ] 346 0.042% 99.987% ( 33000, 50000 ] 83 0.010% 99.997% ( 50000, 75000 ] 18 0.002% 99.999% ( 75000, 110000 ] 2 0.000% 100.000% ( 110000, 170000 ] 2 0.000% 100.000% ( 170000, 250000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 14387291 rocksdb.bloom.filter.full.positive COUNT : 959327 rocksdb.bloom.filter.full.true.positive COUNT : 818638 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 358 rocksdb.memtable.miss COUNT : 818638 rocksdb.l0.hit COUNT : 40631 rocksdb.l1.hit COUNT : 13 rocksdb.l2andup.hit COUNT : 777994 rocksdb.compaction.key.drop.new COUNT : 214648 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5395114 rocksdb.number.keys.read COUNT : 818996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5567757648 rocksdb.bytes.read COUNT : 818996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1625 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 39445640 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 23545772876 rocksdb.compact.write.bytes COUNT : 23321157379 rocksdb.flush.write.bytes COUNT : 5548899679 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 32 rocksdb.number.superversion_acquires COUNT : 1101 rocksdb.number.superversion_releases COUNT : 1064 rocksdb.number.superversion_cleanups COUNT : 274 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 94 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5504044032 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2031616 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 27218280270 rocksdb.non.last.level.read.count COUNT : 6431710 rocksdb.block.checksum.compute.count COUNT : 6672213 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 203.099155 P95 : 558.807725 P99 : 1600.550060 P100 : 212016.000000 COUNT : 818996 SUM : 239735286 rocksdb.db.write.micros P50 : 1.570932 P95 : 4.654839 P99 : 9.924091 P100 : 1033460.000000 COUNT : 5395114 SUM : 53313738 rocksdb.compaction.times.micros P50 : 1342592.592593 P95 : 3635000.000000 P99 : 5823079.000000 P100 : 5823079.000000 COUNT : 99 SUM : 157357650 rocksdb.compaction.times.cpu_micros P50 : 855606.060606 P95 : 1855666.666667 P99 : 5411000.000000 P100 : 5762525.000000 COUNT : 99 SUM : 96882465 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 847.105263 P95 : 34700.000000 P99 : 179081.000000 P100 : 179081.000000 COUNT : 76 SUM : 526097 rocksdb.compaction.outfile.sync.micros P50 : 24860.000000 P95 : 139785.714286 P99 : 404700.000000 P100 : 521819.000000 COUNT : 261 SUM : 10925630 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 503.076923 P95 : 4473.333333 P99 : 6370.222222 P100 : 7312.000000 COUNT : 194 SUM : 221069 rocksdb.table.open.io.micros P50 : 12317.641326 P95 : 21602.189781 P99 : 30474.489796 P100 : 165376.000000 COUNT : 1625 SUM : 19239522 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.920220 P95 : 13.483578 P99 : 235.308452 P100 : 62249.000000 COUNT : 5701577 SUM : 117314541 rocksdb.read.block.get.micros P50 : 157.472794 P95 : 430.970851 P99 : 1586.641068 P100 : 62197.000000 COUNT : 969002 SUM : 206265556 rocksdb.write.raw.block.micros P50 : 0.532089 P95 : 1.268565 P99 : 3.605379 P100 : 89646.000000 COUNT : 6992527 SUM : 24500528 rocksdb.numfiles.in.singlecompaction P50 : 2.285714 P95 : 4.666667 P99 : 7.000000 P100 : 7.000000 COUNT : 120 SUM : 364 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.575970 P95 : 1223.786134 P99 : 2593.465626 P100 : 1033296.000000 COUNT : 186195 SUM : 39465358 rocksdb.sst.read.micros P50 : 1.989249 P95 : 202.239801 P99 : 524.712851 P100 : 62243.000000 COUNT : 6431746 SUM : 306340210 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 818996 SUM : 818996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5395114 SUM : 5567757648 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 224285.714286 P95 : 1009600.000000 P99 : 1294047.000000 P100 : 1294047.000000 COUNT : 76 SUM : 25396186 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 292 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0