[2024-12-05 09:56:17.717322] Starting SPDK v24.09.1-pre git sha1 b18e1bd62 / DPDK 23.11.0 initialization... [2024-12-05 09:56:17.717588] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1464059 ] Set seed to 1733388977710650 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-05 09:56:17.824226] app.c: 917:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-05 09:56:17.862623] app.c: 610:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-05 09:56:17.862669] app.c: 611:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1464059' to capture a snapshot of events at runtime. [2024-12-05 09:56:17.862680] app.c: 616:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-12-05 09:56:17.862690] app.c: 617:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-12-05 09:56:17.862698] app.c: 618:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1464059 for offline analysis/debug. [2024-12-05 09:56:17.862782] reactor.c: 990:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Thu Dec 5 09:56:22 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/05-09:56:29 ... thread 0: (1000000,1000000) ops and (131986.7,131986.7) ops/second in (7.576523,7.576523) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 6/4 566.87 MB 18.9 0.0 0.0 0.0 0.8 0.8 0.0 1.0 0.0 121.4 6.74 2.33 9 0.749 0 0 0.0 0.0 L1 2/2 209.80 MB 0.0 0.7 0.4 0.3 0.7 0.4 0.0 1.7 119.0 118.8 5.92 3.24 2 2.959 714K 916 0.0 0.0 L2 7/3 501.08 MB 2.5 0.2 0.1 0.0 0.2 0.1 0.1 1.1 127.4 127.4 1.29 0.76 2 0.647 167K 50 0.0 0.0 L3 34/7 3.21 GB 2.4 2.2 1.6 0.6 2.2 1.5 0.0 1.4 100.9 99.5 22.36 10.38 11 2.032 1436K 592 0.0 0.0 L4 213/20 12.82 GB 1.2 3.8 3.1 0.7 3.8 3.1 0.0 1.2 110.8 110.4 35.10 17.97 28 1.253 2045K 1552 0.0 0.0 L5 1068/0 87.85 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1330/36 105.13 GB 0.0 6.8 5.3 1.6 7.6 6.0 0.2 9.5 98.2 109.0 71.41 34.68 52 1.373 4364K 3110 0.0 0.0 Int 0/0 0.00 KB 0.0 6.8 5.3 1.6 7.6 6.0 0.2 9.5 98.2 109.0 71.41 34.68 52 1.373 4364K 3110 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.8 5.3 1.6 6.8 5.2 0.0 0.0 108.4 107.7 64.66 32.35 43 1.504 4364K 3110 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 121.4 6.74 2.33 9 0.749 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.8 total, 7.6 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.60 GB write, 1001.87 MB/s write, 6.85 GB read, 902.33 MB/s read, 71.4 seconds Interval compaction: 7.60 GB write, 1027.43 MB/s write, 6.85 GB read, 925.35 MB/s read, 71.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 31, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 31, interval: 31 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 200141 Average: 7.5970 StdDev: 101.44 Min: 1 Median: 1.8991 Max: 8842 Percentiles: P50: 1.90 P75: 2.92 P99: 9.59 P99.9: 1576.11 P99.99: 4202.93 ------------------------------------------------------ [ 0, 1 ] 7679 3.837% 3.837% # ( 1, 2 ] 102765 51.346% 55.183% ########## ( 2, 3 ] 43032 21.501% 76.684% #### ( 3, 4 ] 20280 10.133% 86.817% ## ( 4, 6 ] 20637 10.311% 97.128% ## ( 6, 10 ] 4177 2.087% 99.215% ( 10, 15 ] 687 0.343% 99.558% ( 15, 22 ] 85 0.042% 99.601% ( 22, 34 ] 9 0.004% 99.605% ( 34, 51 ] 12 0.006% 99.611% ( 51, 76 ] 11 0.005% 99.617% ( 76, 110 ] 1 0.000% 99.617% ( 110, 170 ] 12 0.006% 99.623% ( 170, 250 ] 31 0.015% 99.639% ( 250, 380 ] 73 0.036% 99.675% ( 380, 580 ] 119 0.059% 99.735% ( 580, 870 ] 165 0.082% 99.817% ( 870, 1300 ] 118 0.059% 99.876% ( 1300, 1900 ] 104 0.052% 99.928% ( 1900, 2900 ] 71 0.035% 99.964% ( 2900, 4400 ] 61 0.030% 99.994% ( 4400, 6600 ] 11 0.005% 100.000% ( 6600, 9900 ] 1 0.000% 100.000% ** Level 1 read latency histogram (micros): Count: 133601 Average: 11.8760 StdDev: 146.65 Min: 1 Median: 1.9844 Max: 7936 Percentiles: P50: 1.98 P75: 3.01 P99: 10.85 P99.9: 2490.38 P99.99: 4337.78 ------------------------------------------------------ [ 0, 1 ] 2634 1.972% 1.972% ( 1, 2 ] 65185 48.791% 50.762% ########## ( 2, 3 ] 32228 24.123% 74.885% ##### ( 3, 4 ] 14063 10.526% 85.411% ## ( 4, 6 ] 14625 10.947% 96.358% ## ( 6, 10 ] 3433 2.570% 98.927% # ( 10, 15 ] 577 0.432% 99.359% ( 15, 22 ] 77 0.058% 99.417% ( 22, 34 ] 8 0.006% 99.423% ( 34, 51 ] 8 0.006% 99.429% ( 51, 76 ] 4 0.003% 99.432% ( 76, 110 ] 2 0.001% 99.433% ( 110, 170 ] 4 0.003% 99.436% ( 170, 250 ] 10 0.007% 99.444% ( 250, 380 ] 33 0.025% 99.469% ( 380, 580 ] 80 0.060% 99.528% ( 580, 870 ] 145 0.109% 99.637% ( 870, 1300 ] 160 0.120% 99.757% ( 1300, 1900 ] 131 0.098% 99.855% ( 1900, 2900 ] 104 0.078% 99.933% ( 2900, 4400 ] 81 0.061% 99.993% ( 4400, 6600 ] 19 0.014% 100.007% ( 6600, 9900 ] 3 0.002% 100.010% ** Level 2 read latency histogram (micros): Count: 398475 Average: 14.6574 StdDev: 175.98 Min: 1 Median: 1.9868 Max: 9058 Percentiles: P50: 1.99 P75: 3.08 P99: 12.24 P99.9: 1547.62 P99.99: 2153.45 ------------------------------------------------------ [ 0, 1 ] 11410 2.863% 2.863% # ( 1, 2 ] 190346 47.769% 50.632% ########## ( 2, 3 ] 93914 23.568% 74.200% ##### ( 3, 4 ] 42228 10.597% 84.798% ## ( 4, 6 ] 43573 10.935% 95.733% ## ( 6, 10 ] 11975 3.005% 98.738% # ( 10, 15 ] 2336 0.586% 99.324% ( 15, 22 ] 323 0.081% 99.405% ( 22, 34 ] 87 0.022% 99.427% ( 34, 51 ] 38 0.010% 99.437% ( 51, 76 ] 21 0.005% 99.442% ( 76, 110 ] 18 0.005% 99.446% ( 110, 170 ] 50 0.013% 99.459% ( 170, 250 ] 86 0.022% 99.481% ( 250, 380 ] 197 0.049% 99.530% ( 380, 580 ] 351 0.088% 99.618% ( 580, 870 ] 444 0.111% 99.729% ( 870, 1300 ] 500 0.125% 99.855% ( 1300, 1900 ] 435 0.109% 99.964% ( 1900, 2900 ] 407 0.102% 100.066% ( 2900, 4400 ] 350 0.088% 100.154% ( 4400, 6600 ] 108 0.027% 100.181% ( 6600, 9900 ] 17 0.004% 100.185% ** Level 3 read latency histogram (micros): Count: 947932 Average: 12.1960 StdDev: 159.34 Min: 1 Median: 2.0207 Max: 13726 Percentiles: P50: 2.02 P75: 3.06 P99: 9.65 P99.9: 923.52 P99.99: 1282.82 ------------------------------------------------------ [ 0, 1 ] 14653 1.546% 1.546% ( 1, 2 ] 454443 47.940% 49.486% ########## ( 2, 3 ] 235629 24.857% 74.343% ##### ( 3, 4 ] 106495 11.234% 85.578% ## ( 4, 6 ] 106380 11.222% 96.800% ## ( 6, 10 ] 22883 2.414% 99.214% ( 10, 15 ] 3887 0.410% 99.624% ( 15, 22 ] 545 0.057% 99.682% ( 22, 34 ] 55 0.006% 99.688% ( 34, 51 ] 51 0.005% 99.693% ( 51, 76 ] 61 0.006% 99.699% ( 76, 110 ] 26 0.003% 99.702% ( 110, 170 ] 38 0.004% 99.706% ( 170, 250 ] 95 0.010% 99.716% ( 250, 380 ] 217 0.023% 99.739% ( 380, 580 ] 535 0.056% 99.795% ( 580, 870 ] 864 0.091% 99.887% ( 870, 1300 ] 1021 0.108% 99.994% ( 1300, 1900 ] 746 0.079% 100.073% ( 1900, 2900 ] 756 0.080% 100.153% ( 2900, 4400 ] 692 0.073% 100.226% ( 4400, 6600 ] 199 0.021% 100.247% ( 6600, 9900 ] 36 0.004% 100.251% ( 9900, 14000 ] 4 0.000% 100.251% ** Level 4 read latency histogram (micros): Count: 241661 Average: 35.7646 StdDev: 297.33 Min: 1 Median: 2.0093 Max: 21698 Percentiles: P50: 2.01 P75: 3.10 P99: 1110.42 P99.9: 3883.91 P99.99: 4453.72 ------------------------------------------------------ [ 0, 1 ] 4266 1.765% 1.765% ( 1, 2 ] 116024 48.011% 49.776% ########## ( 2, 3 ] 58289 24.120% 73.896% ##### ( 3, 4 ] 27207 11.258% 85.155% ## ( 4, 6 ] 24771 10.250% 95.405% ## ( 6, 10 ] 5093 2.107% 97.513% ( 10, 15 ] 980 0.406% 97.918% ( 15, 22 ] 128 0.053% 97.971% ( 22, 34 ] 17 0.007% 97.978% ( 34, 51 ] 17 0.007% 97.985% ( 51, 76 ] 22 0.009% 97.994% ( 76, 110 ] 13 0.005% 98.000% ( 110, 170 ] 47 0.019% 98.019% ( 170, 250 ] 138 0.057% 98.076% ( 250, 380 ] 233 0.096% 98.173% ( 380, 580 ] 551 0.228% 98.401% ( 580, 870 ] 902 0.373% 98.774% ( 870, 1300 ] 979 0.405% 99.179% ( 1300, 1900 ] 712 0.295% 99.474% ( 1900, 2900 ] 624 0.258% 99.732% ( 2900, 4400 ] 621 0.257% 99.989% ( 4400, 6600 ] 157 0.065% 100.054% ( 6600, 9900 ] 34 0.014% 100.068% ( 9900, 14000 ] 1 0.000% 100.068% ( 14000, 22000 ] 2 0.001% 100.069% ** Level 5 read latency histogram (micros): Count: 12561 Average: 1733.5937 StdDev: 1481.10 Min: 85 Median: 1195.3673 Max: 20074 Percentiles: P50: 1195.37 P75: 2504.88 P99: 6594.41 P99.9: 9716.51 P99.99: 15983.73 ------------------------------------------------------ ( 76, 110 ] 6 0.048% 0.048% ( 110, 170 ] 85 0.677% 0.724% ( 170, 250 ] 235 1.871% 2.595% ( 250, 380 ] 676 5.382% 7.977% # ( 380, 580 ] 1400 11.146% 19.123% ## ( 580, 870 ] 2148 17.101% 36.223% ### ( 870, 1300 ] 2287 18.207% 54.430% #### ( 1300, 1900 ] 1648 13.120% 67.550% ### ( 1900, 2900 ] 1547 12.316% 79.866% ## ( 2900, 4400 ] 1771 14.099% 93.965% ### ( 4400, 6600 ] 634 5.047% 99.013% # ( 6600, 9900 ] 118 0.939% 99.952% ( 9900, 14000 ] 4 0.032% 99.984% ( 14000, 22000 ] 3 0.024% 100.008% ** DB Stats ** Uptime(secs): 7.8 total, 7.6 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 126.64 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.593 H:M:S, 46.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 129.85 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.593 H:M:S, 47.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/12/05-09:56:35 ... thread 0: (1000000,2000000) ops and (188697.5,155327.6) ops/second in (5.299487,12.876010) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 10/4 1.54 GB 132.1 1.0 0.0 1.0 2.8 1.8 0.0 1.6 60.2 162.8 17.78 9.37 26 0.684 1087K 1499 0.0 0.0 L1 0/0 0.00 KB 0.0 0.7 0.4 0.3 0.7 0.4 0.0 1.7 119.0 118.8 5.92 3.24 2 2.959 714K 916 0.0 0.0 L2 2/1 195.12 MB 0.7 0.6 0.4 0.3 0.6 0.3 0.1 1.7 122.4 122.2 5.12 2.95 4 1.279 636K 1017 0.0 0.0 L3 35/20 2.24 GB 1.0 3.4 2.1 1.3 3.3 2.0 0.0 1.6 116.9 115.5 29.37 15.75 16 1.836 1979K 718 0.0 0.0 L4 167/10 9.63 GB 1.0 6.7 4.6 2.1 6.6 4.5 0.0 1.4 115.3 114.4 59.07 31.73 40 1.477 2499K 1552 0.0 0.0 L5 1124/0 92.46 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1338/35 106.06 GB 0.0 12.3 7.4 4.9 14.0 9.1 4.8 7.9 107.8 122.6 117.27 63.05 88 1.333 6917K 5702 0.0 0.0 Int 0/0 0.00 KB 0.0 12.3 7.4 4.9 14.0 9.1 4.8 7.9 107.8 122.6 117.27 63.05 88 1.333 6917K 5702 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.3 7.4 4.9 12.3 7.3 0.0 0.0 120.4 119.5 105.02 57.94 65 1.616 6917K 5702 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 149.2 12.24 5.11 23 0.532 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 13.1 total, 12.9 interval Flush(GB): cumulative 1.784, interval 1.784 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 14.04 GB write, 1099.57 MB/s write, 12.35 GB read, 967.47 MB/s read, 117.3 seconds Interval compaction: 14.04 GB write, 1116.08 MB/s write, 12.35 GB read, 982.00 MB/s read, 117.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 44, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 44, interval: 44 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 415877 Average: 7.8563 StdDev: 102.15 Min: 1 Median: 1.8380 Max: 8842 Percentiles: P50: 1.84 P75: 2.79 P99: 9.58 P99.9: 1435.07 P99.99: 3936.08 ------------------------------------------------------ [ 0, 1 ] 19678 4.732% 4.732% # ( 1, 2 ] 224642 54.016% 58.748% ########### ( 2, 3 ] 86047 20.690% 79.439% #### ( 3, 4 ] 37740 9.075% 88.513% ## ( 4, 6 ] 37304 8.970% 97.483% ## ( 6, 10 ] 7047 1.694% 99.178% ( 10, 15 ] 1212 0.291% 99.469% ( 15, 22 ] 157 0.038% 99.507% ( 22, 34 ] 12 0.003% 99.510% ( 34, 51 ] 25 0.006% 99.516% ( 51, 76 ] 19 0.005% 99.521% ( 76, 110 ] 10 0.002% 99.523% ( 110, 170 ] 72 0.017% 99.540% ( 170, 250 ] 141 0.034% 99.574% ( 250, 380 ] 247 0.059% 99.634% ( 380, 580 ] 362 0.087% 99.721% ( 580, 870 ] 419 0.101% 99.821% ( 870, 1300 ] 284 0.068% 99.890% ( 1300, 1900 ] 196 0.047% 99.937% ( 1900, 2900 ] 134 0.032% 99.969% ( 2900, 4400 ] 128 0.031% 100.000% ( 4400, 6600 ] 26 0.006% 100.006% ( 6600, 9900 ] 2 0.000% 100.006% ** Level 1 read latency histogram (micros): Count: 181827 Average: 12.4060 StdDev: 154.81 Min: 1 Median: 1.9905 Max: 12011 Percentiles: P50: 1.99 P75: 3.00 P99: 11.33 P99.9: 2596.51 P99.99: 4336.80 ------------------------------------------------------ [ 0, 1 ] 3321 1.826% 1.826% ( 1, 2 ] 88434 48.636% 50.463% ########## ( 2, 3 ] 44787 24.632% 75.094% ##### ( 3, 4 ] 18795 10.337% 85.431% ## ( 4, 6 ] 19760 10.867% 96.299% ## ( 6, 10 ] 4709 2.590% 98.889% # ( 10, 15 ] 764 0.420% 99.309% ( 15, 22 ] 104 0.057% 99.366% ( 22, 34 ] 10 0.005% 99.371% ( 34, 51 ] 13 0.007% 99.379% ( 51, 76 ] 9 0.005% 99.383% ( 76, 110 ] 7 0.004% 99.387% ( 110, 170 ] 17 0.009% 99.397% ( 170, 250 ] 32 0.018% 99.414% ( 250, 380 ] 64 0.035% 99.449% ( 380, 580 ] 115 0.063% 99.513% ( 580, 870 ] 217 0.119% 99.632% ( 870, 1300 ] 217 0.119% 99.751% ( 1300, 1900 ] 165 0.091% 99.842% ( 1900, 2900 ] 151 0.083% 99.925% ( 2900, 4400 ] 123 0.068% 99.993% ( 4400, 6600 ] 26 0.014% 100.007% ( 6600, 9900 ] 5 0.003% 100.010% ( 9900, 14000 ] 1 0.001% 100.010% ** Level 2 read latency histogram (micros): Count: 564144 Average: 12.4929 StdDev: 164.55 Min: 1 Median: 1.9768 Max: 34199 Percentiles: P50: 1.98 P75: 2.99 P99: 10.98 P99.9: 1487.48 P99.99: 2279.84 ------------------------------------------------------ [ 0, 1 ] 13659 2.421% 2.421% ( 1, 2 ] 274782 48.708% 51.129% ########## ( 2, 3 ] 135883 24.087% 75.216% ##### ( 3, 4 ] 58526 10.374% 85.590% ## ( 4, 6 ] 59794 10.599% 96.189% ## ( 6, 10 ] 15300 2.712% 98.901% # ( 10, 15 ] 2838 0.503% 99.404% ( 15, 22 ] 390 0.069% 99.473% ( 22, 34 ] 92 0.016% 99.489% ( 34, 51 ] 53 0.009% 99.499% ( 51, 76 ] 37 0.007% 99.505% ( 76, 110 ] 21 0.004% 99.509% ( 110, 170 ] 69 0.012% 99.521% ( 170, 250 ] 128 0.023% 99.544% ( 250, 380 ] 255 0.045% 99.589% ( 380, 580 ] 449 0.080% 99.669% ( 580, 870 ] 563 0.100% 99.769% ( 870, 1300 ] 589 0.104% 99.873% ( 1300, 1900 ] 486 0.086% 99.959% ( 1900, 2900 ] 457 0.081% 100.040% ( 2900, 4400 ] 391 0.069% 100.110% ( 4400, 6600 ] 117 0.021% 100.130% ( 6600, 9900 ] 21 0.004% 100.134% ( 9900, 14000 ] 1 0.000% 100.134% ( 33000, 50000 ] 2 0.000% 100.135% ** Level 3 read latency histogram (micros): Count: 1599499 Average: 10.9933 StdDev: 155.52 Min: 1 Median: 1.9871 Max: 31712 Percentiles: P50: 1.99 P75: 2.98 P99: 9.62 P99.9: 837.97 P99.99: 1252.97 ------------------------------------------------------ [ 0, 1 ] 23550 1.472% 1.472% ( 1, 2 ] 786361 49.163% 50.635% ########## ( 2, 3 ] 396913 24.815% 75.450% ##### ( 3, 4 ] 169888 10.621% 86.071% ## ( 4, 6 ] 171496 10.722% 96.793% ## ( 6, 10 ] 39024 2.440% 99.233% ( 10, 15 ] 6183 0.387% 99.620% ( 15, 22 ] 842 0.053% 99.672% ( 22, 34 ] 88 0.006% 99.678% ( 34, 51 ] 116 0.007% 99.685% ( 51, 76 ] 104 0.007% 99.692% ( 76, 110 ] 64 0.004% 99.696% ( 110, 170 ] 168 0.011% 99.706% ( 170, 250 ] 304 0.019% 99.725% ( 250, 380 ] 507 0.032% 99.757% ( 380, 580 ] 993 0.062% 99.819% ( 580, 870 ] 1462 0.091% 99.910% ( 870, 1300 ] 1435 0.090% 100.000% ( 1300, 1900 ] 1012 0.063% 100.063% ( 1900, 2900 ] 1010 0.063% 100.126% ( 2900, 4400 ] 978 0.061% 100.187% ( 4400, 6600 ] 293 0.018% 100.206% ( 6600, 9900 ] 52 0.003% 100.209% ( 9900, 14000 ] 7 0.000% 100.210% ( 14000, 22000 ] 8 0.001% 100.210% ( 22000, 33000 ] 3 0.000% 100.210% ** Level 4 read latency histogram (micros): Count: 581381 Average: 26.1117 StdDev: 256.78 Min: 1 Median: 1.9985 Max: 40256 Percentiles: P50: 2.00 P75: 3.07 P99: 682.19 P99.9: 2894.39 P99.99: 3801.15 ------------------------------------------------------ [ 0, 1 ] 9637 1.658% 1.658% ( 1, 2 ] 281472 48.414% 50.072% ########## ( 2, 3 ] 140592 24.182% 74.254% ##### ( 3, 4 ] 62502 10.751% 85.005% ## ( 4, 6 ] 60940 10.482% 95.487% ## ( 6, 10 ] 13520 2.325% 97.812% ( 10, 15 ] 2208 0.380% 98.192% ( 15, 22 ] 288 0.050% 98.242% ( 22, 34 ] 36 0.006% 98.248% ( 34, 51 ] 51 0.009% 98.257% ( 51, 76 ] 55 0.009% 98.266% ( 76, 110 ] 57 0.010% 98.276% ( 110, 170 ] 320 0.055% 98.331% ( 170, 250 ] 643 0.111% 98.442% ( 250, 380 ] 940 0.162% 98.603% ( 380, 580 ] 1576 0.271% 98.874% ( 580, 870 ] 2075 0.357% 99.231% ( 870, 1300 ] 1846 0.318% 99.549% ( 1300, 1900 ] 1088 0.187% 99.736% ( 1900, 2900 ] 960 0.165% 99.901% ( 2900, 4400 ] 862 0.148% 100.049% ( 4400, 6600 ] 213 0.037% 100.086% ( 6600, 9900 ] 51 0.009% 100.095% ( 9900, 14000 ] 7 0.001% 100.096% ( 14000, 22000 ] 5 0.001% 100.097% ( 22000, 33000 ] 6 0.001% 100.098% ( 33000, 50000 ] 1 0.000% 100.098% ** Level 5 read latency histogram (micros): Count: 30767 Average: 1230.3459 StdDev: 1461.91 Min: 78 Median: 766.1831 Max: 47713 Percentiles: P50: 766.18 P75: 1502.85 P99: 6222.35 P99.9: 9725.64 P99.99: 44188.46 ------------------------------------------------------ ( 76, 110 ] 181 0.588% 0.588% ( 110, 170 ] 1218 3.959% 4.547% # ( 170, 250 ] 2054 6.676% 11.223% # ( 250, 380 ] 3287 10.684% 21.907% ## ( 380, 580 ] 4959 16.118% 38.025% ### ( 580, 870 ] 5739 18.653% 56.678% #### ( 870, 1300 ] 4694 15.257% 71.934% ### ( 1300, 1900 ] 2790 9.068% 81.002% ## ( 1900, 2900 ] 2357 7.661% 88.663% ## ( 2900, 4400 ] 2516 8.178% 96.841% ## ( 4400, 6600 ] 802 2.607% 99.447% # ( 6600, 9900 ] 147 0.478% 99.925% ( 9900, 14000 ] 5 0.016% 99.941% ( 14000, 22000 ] 6 0.020% 99.961% ( 22000, 33000 ] 3 0.010% 99.971% ( 33000, 50000 ] 9 0.029% 100.000% ** DB Stats ** Uptime(secs): 13.1 total, 5.3 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 150.59 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.101 H:M:S, 39.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 185.71 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.509 H:M:S, 28.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 8 num-running-flushes: 1 2024/12/05-09:56:39 ... thread 0: (1000000,3000000) ops and (235605.7,175229.7) ops/second in (4.244380,17.120390) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/12 2.52 GB 62.9 1.8 0.0 1.8 4.6 2.8 0.0 1.6 73.6 187.7 24.83 15.12 43 0.577 1854K 2200 0.0 0.0 L1 0/0 0.00 KB 0.0 0.7 0.4 0.3 0.7 0.4 0.0 1.7 119.0 118.8 5.92 3.24 2 2.959 714K 916 0.0 0.0 L2 1/0 66.34 MB 0.7 0.6 0.4 0.3 0.6 0.3 0.1 1.7 122.4 122.2 5.12 2.95 4 1.279 636K 1017 0.0 0.0 L3 17/1 1.03 GB 0.9 4.3 2.2 2.1 4.3 2.1 0.0 1.9 125.0 123.4 35.35 20.17 17 2.079 2850K 718 0.0 0.0 L4 174/5 9.93 GB 1.0 8.4 5.5 2.9 8.3 5.4 0.5 1.5 122.5 121.3 69.94 39.88 50 1.399 3571K 6855 0.0 0.0 L5 1136/0 93.44 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1343/18 106.99 GB 0.0 15.8 8.4 7.4 18.4 11.0 6.2 6.6 114.4 133.4 141.15 81.36 116 1.217 9627K 11K 0.0 0.0 Int 0/0 0.00 KB 0.0 15.8 8.4 7.4 18.4 11.0 6.2 6.6 114.4 133.4 141.15 81.36 116 1.217 9627K 11K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 15.8 8.4 7.4 15.6 8.3 0.0 0.0 129.1 127.9 125.05 73.50 79 1.583 9627K 11K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 176.1 16.10 7.85 37 0.435 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 17.3 total, 17.1 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 18.39 GB write, 1087.46 MB/s write, 15.77 GB read, 932.34 MB/s read, 141.2 seconds Interval compaction: 18.39 GB write, 1099.74 MB/s write, 15.77 GB read, 942.87 MB/s read, 141.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 48, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 48, interval: 48 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 770269 Average: 7.4162 StdDev: 89.07 Min: 1 Median: 1.7746 Max: 8842 Percentiles: P50: 1.77 P75: 2.63 P99: 9.88 P99.9: 1074.19 P99.99: 2734.86 ------------------------------------------------------ [ 0, 1 ] 49705 6.453% 6.453% # ( 1, 2 ] 433060 56.222% 62.675% ########### ( 2, 3 ] 149935 19.465% 82.140% #### ( 3, 4 ] 60909 7.907% 90.048% ## ( 4, 6 ] 57866 7.512% 97.560% ## ( 6, 10 ] 11422 1.483% 99.043% ( 10, 15 ] 1811 0.235% 99.278% ( 15, 22 ] 218 0.028% 99.306% ( 22, 34 ] 22 0.003% 99.309% ( 34, 51 ] 33 0.004% 99.313% ( 51, 76 ] 29 0.004% 99.317% ( 76, 110 ] 171 0.022% 99.339% ( 110, 170 ] 846 0.110% 99.449% ( 170, 250 ] 835 0.108% 99.558% ( 250, 380 ] 747 0.097% 99.655% ( 380, 580 ] 871 0.113% 99.768% ( 580, 870 ] 789 0.102% 99.870% ( 870, 1300 ] 488 0.063% 99.934% ( 1300, 1900 ] 270 0.035% 99.969% ( 1900, 2900 ] 200 0.026% 99.995% ( 2900, 4400 ] 162 0.021% 100.016% ( 4400, 6600 ] 32 0.004% 100.020% ( 6600, 9900 ] 2 0.000% 100.020% ** Level 1 read latency histogram (micros): Count: 204070 Average: 11.3992 StdDev: 146.26 Min: 1 Median: 1.9480 Max: 12011 Percentiles: P50: 1.95 P75: 2.93 P99: 10.49 P99.9: 2449.21 P99.99: 4309.67 ------------------------------------------------------ [ 0, 1 ] 5528 2.709% 2.709% # ( 1, 2 ] 101802 49.886% 52.595% ########## ( 2, 3 ] 49116 24.068% 76.663% ##### ( 3, 4 ] 19864 9.734% 86.397% ## ( 4, 6 ] 20522 10.056% 96.453% ## ( 6, 10 ] 5119 2.508% 98.962% # ( 10, 15 ] 796 0.390% 99.352% ( 15, 22 ] 111 0.054% 99.406% ( 22, 34 ] 10 0.005% 99.411% ( 34, 51 ] 14 0.007% 99.418% ( 51, 76 ] 9 0.004% 99.422% ( 76, 110 ] 11 0.005% 99.428% ( 110, 170 ] 36 0.018% 99.445% ( 170, 250 ] 42 0.021% 99.466% ( 250, 380 ] 75 0.037% 99.503% ( 380, 580 ] 124 0.061% 99.563% ( 580, 870 ] 222 0.109% 99.672% ( 870, 1300 ] 217 0.106% 99.779% ( 1300, 1900 ] 165 0.081% 99.859% ( 1900, 2900 ] 151 0.074% 99.933% ( 2900, 4400 ] 123 0.060% 99.994% ( 4400, 6600 ] 26 0.013% 100.006% ( 6600, 9900 ] 5 0.002% 100.009% ( 9900, 14000 ] 1 0.000% 100.009% ** Level 2 read latency histogram (micros): Count: 564262 Average: 12.5528 StdDev: 164.65 Min: 1 Median: 1.9770 Max: 34199 Percentiles: P50: 1.98 P75: 2.99 P99: 11.19 P99.9: 1488.57 P99.99: 2281.17 ------------------------------------------------------ [ 0, 1 ] 13659 2.421% 2.421% ( 1, 2 ] 274783 48.698% 51.118% ########## ( 2, 3 ] 135884 24.082% 75.200% ##### ( 3, 4 ] 58526 10.372% 85.572% ## ( 4, 6 ] 59794 10.597% 96.169% ## ( 6, 10 ] 15300 2.712% 98.881% # ( 10, 15 ] 2838 0.503% 99.384% ( 15, 22 ] 390 0.069% 99.453% ( 22, 34 ] 92 0.016% 99.469% ( 34, 51 ] 53 0.009% 99.478% ( 51, 76 ] 37 0.007% 99.485% ( 76, 110 ] 33 0.006% 99.491% ( 110, 170 ] 104 0.018% 99.509% ( 170, 250 ] 159 0.028% 99.537% ( 250, 380 ] 268 0.047% 99.585% ( 380, 580 ] 458 0.081% 99.666% ( 580, 870 ] 573 0.102% 99.768% ( 870, 1300 ] 594 0.105% 99.873% ( 1300, 1900 ] 486 0.086% 99.959% ( 1900, 2900 ] 458 0.081% 100.040% ( 2900, 4400 ] 391 0.069% 100.110% ( 4400, 6600 ] 117 0.021% 100.130% ( 6600, 9900 ] 21 0.004% 100.134% ( 9900, 14000 ] 1 0.000% 100.134% ( 33000, 50000 ] 2 0.000% 100.135% ** Level 3 read latency histogram (micros): Count: 1888254 Average: 10.1618 StdDev: 145.52 Min: 1 Median: 1.9616 Max: 31712 Percentiles: P50: 1.96 P75: 2.94 P99: 9.58 P99.9: 778.13 P99.99: 1201.75 ------------------------------------------------------ [ 0, 1 ] 33977 1.799% 1.799% ( 1, 2 ] 946470 50.124% 51.923% ########## ( 2, 3 ] 461235 24.427% 76.350% ##### ( 3, 4 ] 192697 10.205% 86.555% ## ( 4, 6 ] 194467 10.299% 96.854% ## ( 6, 10 ] 45282 2.398% 99.252% ( 10, 15 ] 6957 0.368% 99.620% ( 15, 22 ] 919 0.049% 99.669% ( 22, 34 ] 93 0.005% 99.674% ( 34, 51 ] 122 0.006% 99.680% ( 51, 76 ] 127 0.007% 99.687% ( 76, 110 ] 131 0.007% 99.694% ( 110, 170 ] 400 0.021% 99.715% ( 170, 250 ] 497 0.026% 99.742% ( 250, 380 ] 695 0.037% 99.778% ( 380, 580 ] 1179 0.062% 99.841% ( 580, 870 ] 1636 0.087% 99.927% ( 870, 1300 ] 1531 0.081% 100.009% ( 1300, 1900 ] 1062 0.056% 100.065% ( 1900, 2900 ] 1073 0.057% 100.122% ( 2900, 4400 ] 1005 0.053% 100.175% ( 4400, 6600 ] 301 0.016% 100.191% ( 6600, 9900 ] 52 0.003% 100.194% ( 9900, 14000 ] 7 0.000% 100.194% ( 14000, 22000 ] 8 0.000% 100.194% ( 22000, 33000 ] 3 0.000% 100.194% ** Level 4 read latency histogram (micros): Count: 811075 Average: 23.4166 StdDev: 228.04 Min: 0 Median: 1.9837 Max: 40256 Percentiles: P50: 1.98 P75: 3.01 P99: 554.20 P99.9: 2496.05 P99.99: 3361.98 ------------------------------------------------------ [ 0, 1 ] 13436 1.657% 1.657% ( 1, 2 ] 398592 49.144% 50.800% ########## ( 2, 3 ] 195184 24.065% 74.865% ##### ( 3, 4 ] 83409 10.284% 85.149% ## ( 4, 6 ] 81304 10.024% 95.173% ## ( 6, 10 ] 19236 2.372% 97.545% ( 10, 15 ] 3011 0.371% 97.916% ( 15, 22 ] 380 0.047% 97.963% ( 22, 34 ] 46 0.006% 97.968% ( 34, 51 ] 80 0.010% 97.978% ( 51, 76 ] 67 0.008% 97.987% ( 76, 110 ] 399 0.049% 98.036% ( 110, 170 ] 1713 0.211% 98.247% ( 170, 250 ] 1865 0.230% 98.477% ( 250, 380 ] 1969 0.243% 98.720% ( 380, 580 ] 2610 0.322% 99.042% ( 580, 870 ] 2811 0.347% 99.388% ( 870, 1300 ] 2262 0.279% 99.667% ( 1300, 1900 ] 1245 0.153% 99.820% ( 1900, 2900 ] 1082 0.133% 99.954% ( 2900, 4400 ] 951 0.117% 100.071% ( 4400, 6600 ] 225 0.028% 100.099% ( 6600, 9900 ] 54 0.007% 100.106% ( 9900, 14000 ] 7 0.001% 100.106% ( 14000, 22000 ] 5 0.001% 100.107% ( 22000, 33000 ] 6 0.001% 100.108% ( 33000, 50000 ] 1 0.000% 100.108% ** Level 5 read latency histogram (micros): Count: 60540 Average: 821.9572 StdDev: 1176.94 Min: 75 Median: 446.3499 Max: 47713 Percentiles: P50: 446.35 P75: 913.55 P99: 5484.75 P99.9: 9079.59 P99.99: 38564.67 ------------------------------------------------------ ( 51, 76 ] 1 0.002% 0.002% ( 76, 110 ] 2140 3.535% 3.537% # ( 110, 170 ] 9093 15.020% 18.556% ### ( 170, 250 ] 8187 13.523% 32.080% ### ( 250, 380 ] 7847 12.962% 45.041% ### ( 380, 580 ] 9049 14.947% 59.988% ### ( 580, 870 ] 8482 14.011% 73.999% ### ( 870, 1300 ] 5984 9.884% 83.883% ## ( 1300, 1900 ] 3241 5.353% 89.237% # ( 1900, 2900 ] 2701 4.462% 93.698% # ( 2900, 4400 ] 2790 4.609% 98.307% # ( 4400, 6600 ] 851 1.406% 99.713% ( 6600, 9900 ] 151 0.249% 99.962% ( 9900, 14000 ] 5 0.008% 99.970% ( 14000, 22000 ] 6 0.010% 99.980% ( 22000, 33000 ] 3 0.005% 99.985% ( 33000, 50000 ] 9 0.015% 100.000% ** DB Stats ** Uptime(secs): 17.3 total, 4.2 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 170.52 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.632 H:M:S, 32.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 231.88 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.531 H:M:S, 12.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 3 num-running-flushes: 1 2024/12/05-09:56:43 ... thread 0: (1000000,4000000) ops and (224329.4,185373.0) ops/second in (4.457730,21.578120) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/12 3.50 GB 169.9 2.8 0.0 2.8 6.6 3.7 0.0 1.8 90.4 210.1 32.07 21.86 63 0.509 2942K 3279 0.0 0.0 L1 0/0 0.00 KB 0.0 0.7 0.4 0.3 0.7 0.4 0.0 1.7 119.0 118.8 5.92 3.24 2 2.959 714K 916 0.0 0.0 L2 1/0 66.34 MB 0.7 0.6 0.4 0.3 0.6 0.3 0.1 1.7 122.4 122.2 5.12 2.95 4 1.279 636K 1017 0.0 0.0 L3 16/0 928.21 MB 0.9 4.3 2.2 2.1 4.3 2.1 0.0 1.9 125.0 123.4 35.35 20.17 17 2.079 2850K 718 0.0 0.0 L4 171/0 9.69 GB 1.0 8.7 5.6 3.1 8.6 5.5 0.5 1.5 123.7 122.5 72.02 41.47 51 1.412 3731K 6855 0.0 0.0 L5 1140/0 93.81 GB 1.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1347/12 107.97 GB 0.0 17.1 8.5 8.6 20.7 12.1 6.6 5.5 116.7 141.2 150.48 89.70 137 1.098 10M 12K 0.0 0.0 Int 0/0 0.00 KB 0.0 17.1 8.5 8.6 20.7 12.1 6.6 5.5 116.7 141.2 150.48 89.70 137 1.098 10M 12K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 17.1 8.5 8.6 17.0 8.4 0.0 0.0 133.8 132.6 131.21 79.09 84 1.562 10M 12K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 199.4 19.27 10.61 53 0.364 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 21.8 total, 21.6 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 20.75 GB write, 975.79 MB/s write, 17.15 GB read, 806.39 MB/s read, 150.5 seconds Interval compaction: 20.75 GB write, 984.53 MB/s write, 17.15 GB read, 813.62 MB/s read, 150.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 5, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 5, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 48, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 48, interval: 53 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1082272 Average: 10.7759 StdDev: 83.66 Min: 0 Median: 1.6820 Max: 8842 Percentiles: P50: 1.68 P75: 2.46 P99: 204.57 P99.9: 819.27 P99.99: 1787.82 ------------------------------------------------------ [ 0, 1 ] 132060 12.202% 12.202% ## ( 1, 2 ] 599779 55.419% 67.621% ########### ( 2, 3 ] 171954 15.888% 83.509% ### ( 3, 4 ] 67757 6.261% 89.769% # ( 4, 6 ] 61658 5.697% 95.467% # ( 6, 10 ] 12205 1.128% 96.594% ( 10, 15 ] 2041 0.189% 96.783% ( 15, 22 ] 259 0.024% 96.807% ( 22, 34 ] 28 0.003% 96.809% ( 34, 51 ] 36 0.003% 96.813% ( 51, 76 ] 32 0.003% 96.816% ( 76, 110 ] 4459 0.412% 97.228% ( 110, 170 ] 16395 1.515% 98.743% ( 170, 250 ] 6450 0.596% 99.339% ( 250, 380 ] 2753 0.254% 99.593% ( 380, 580 ] 2300 0.213% 99.805% ( 580, 870 ] 1242 0.115% 99.920% ( 870, 1300 ] 534 0.049% 99.970% ( 1300, 1900 ] 274 0.025% 99.995% ( 1900, 2900 ] 216 0.020% 100.015% ( 2900, 4400 ] 171 0.016% 100.031% ( 4400, 6600 ] 32 0.003% 100.034% ( 6600, 9900 ] 2 0.000% 100.034% ** Level 1 read latency histogram (micros): Count: 204070 Average: 11.3992 StdDev: 146.26 Min: 1 Median: 1.9480 Max: 12011 Percentiles: P50: 1.95 P75: 2.93 P99: 10.49 P99.9: 2449.21 P99.99: 4309.67 ------------------------------------------------------ [ 0, 1 ] 5528 2.709% 2.709% # ( 1, 2 ] 101802 49.886% 52.595% ########## ( 2, 3 ] 49116 24.068% 76.663% ##### ( 3, 4 ] 19864 9.734% 86.397% ## ( 4, 6 ] 20522 10.056% 96.453% ## ( 6, 10 ] 5119 2.508% 98.962% # ( 10, 15 ] 796 0.390% 99.352% ( 15, 22 ] 111 0.054% 99.406% ( 22, 34 ] 10 0.005% 99.411% ( 34, 51 ] 14 0.007% 99.418% ( 51, 76 ] 9 0.004% 99.422% ( 76, 110 ] 11 0.005% 99.428% ( 110, 170 ] 36 0.018% 99.445% ( 170, 250 ] 42 0.021% 99.466% ( 250, 380 ] 75 0.037% 99.503% ( 380, 580 ] 124 0.061% 99.563% ( 580, 870 ] 222 0.109% 99.672% ( 870, 1300 ] 217 0.106% 99.779% ( 1300, 1900 ] 165 0.081% 99.859% ( 1900, 2900 ] 151 0.074% 99.933% ( 2900, 4400 ] 123 0.060% 99.994% ( 4400, 6600 ] 26 0.013% 100.006% ( 6600, 9900 ] 5 0.002% 100.009% ( 9900, 14000 ] 1 0.000% 100.009% ** Level 2 read latency histogram (micros): Count: 564449 Average: 12.6071 StdDev: 164.66 Min: 1 Median: 1.9774 Max: 34199 Percentiles: P50: 1.98 P75: 2.99 P99: 11.51 P99.9: 1488.33 P99.99: 2281.12 ------------------------------------------------------ [ 0, 1 ] 13659 2.420% 2.420% ( 1, 2 ] 274783 48.682% 51.102% ########## ( 2, 3 ] 135884 24.074% 75.175% ##### ( 3, 4 ] 58526 10.369% 85.544% ## ( 4, 6 ] 59794 10.593% 96.137% ## ( 6, 10 ] 15300 2.711% 98.848% # ( 10, 15 ] 2838 0.503% 99.351% ( 15, 22 ] 390 0.069% 99.420% ( 22, 34 ] 92 0.016% 99.436% ( 34, 51 ] 53 0.009% 99.445% ( 51, 76 ] 37 0.007% 99.452% ( 76, 110 ] 69 0.012% 99.464% ( 110, 170 ] 200 0.035% 99.500% ( 170, 250 ] 189 0.033% 99.533% ( 250, 380 ] 281 0.050% 99.583% ( 380, 580 ] 469 0.083% 99.666% ( 580, 870 ] 574 0.102% 99.768% ( 870, 1300 ] 594 0.105% 99.873% ( 1300, 1900 ] 486 0.086% 99.959% ( 1900, 2900 ] 458 0.081% 100.040% ( 2900, 4400 ] 391 0.069% 100.109% ( 4400, 6600 ] 117 0.021% 100.130% ( 6600, 9900 ] 21 0.004% 100.134% ( 9900, 14000 ] 1 0.000% 100.134% ( 33000, 50000 ] 2 0.000% 100.134% ** Level 3 read latency histogram (micros): Count: 1889728 Average: 10.2938 StdDev: 145.58 Min: 1 Median: 1.9624 Max: 31712 Percentiles: P50: 1.96 P75: 2.95 P99: 9.71 P99.9: 779.34 P99.99: 1202.27 ------------------------------------------------------ [ 0, 1 ] 33977 1.798% 1.798% ( 1, 2 ] 946474 50.085% 51.883% ########## ( 2, 3 ] 461240 24.408% 76.291% ##### ( 3, 4 ] 192700 10.197% 86.488% ## ( 4, 6 ] 194476 10.291% 96.779% ## ( 6, 10 ] 45284 2.396% 99.176% ( 10, 15 ] 6957 0.368% 99.544% ( 15, 22 ] 919 0.049% 99.592% ( 22, 34 ] 93 0.005% 99.597% ( 34, 51 ] 122 0.006% 99.604% ( 51, 76 ] 127 0.007% 99.611% ( 76, 110 ] 349 0.018% 99.629% ( 110, 170 ] 1185 0.063% 99.692% ( 170, 250 ] 760 0.040% 99.732% ( 250, 380 ] 790 0.042% 99.774% ( 380, 580 ] 1247 0.066% 99.840% ( 580, 870 ] 1656 0.088% 99.927% ( 870, 1300 ] 1531 0.081% 100.008% ( 1300, 1900 ] 1062 0.056% 100.065% ( 1900, 2900 ] 1075 0.057% 100.121% ( 2900, 4400 ] 1005 0.053% 100.175% ( 4400, 6600 ] 301 0.016% 100.191% ( 6600, 9900 ] 52 0.003% 100.193% ( 9900, 14000 ] 7 0.000% 100.194% ( 14000, 22000 ] 8 0.000% 100.194% ( 22000, 33000 ] 3 0.000% 100.194% ** Level 4 read latency histogram (micros): Count: 825664 Average: 25.2526 StdDev: 227.12 Min: 0 Median: 1.9957 Max: 40256 Percentiles: P50: 2.00 P75: 3.10 P99: 559.13 P99.9: 2484.64 P99.99: 3360.77 ------------------------------------------------------ [ 0, 1 ] 13584 1.645% 1.645% ( 1, 2 ] 400972 48.564% 50.209% ########## ( 2, 3 ] 196281 23.773% 73.981% ##### ( 3, 4 ] 83770 10.146% 84.127% ## ( 4, 6 ] 81654 9.889% 94.017% ## ( 6, 10 ] 19345 2.343% 96.360% ( 10, 15 ] 3026 0.366% 96.726% ( 15, 22 ] 383 0.046% 96.772% ( 22, 34 ] 46 0.006% 96.778% ( 34, 51 ] 80 0.010% 96.788% ( 51, 76 ] 69 0.008% 96.796% ( 76, 110 ] 1980 0.240% 97.036% ( 110, 170 ] 7059 0.855% 97.891% ( 170, 250 ] 3704 0.449% 98.339% ( 250, 380 ] 2642 0.320% 98.659% ( 380, 580 ] 3140 0.380% 99.040% ( 580, 870 ] 2940 0.356% 99.396% ( 870, 1300 ] 2280 0.276% 99.672% ( 1300, 1900 ] 1249 0.151% 99.823% ( 1900, 2900 ] 1085 0.131% 99.955% ( 2900, 4400 ] 952 0.115% 100.070% ( 4400, 6600 ] 225 0.027% 100.097% ( 6600, 9900 ] 54 0.007% 100.104% ( 9900, 14000 ] 7 0.001% 100.105% ( 14000, 22000 ] 5 0.001% 100.105% ( 22000, 33000 ] 6 0.001% 100.106% ( 33000, 50000 ] 1 0.000% 100.106% ** Level 5 read latency histogram (micros): Count: 118923 Average: 508.3071 StdDev: 903.07 Min: 74 Median: 205.2317 Max: 47713 Percentiles: P50: 205.23 P75: 505.62 P99: 4308.20 P99.9: 7650.69 P99.99: 13268.31 ------------------------------------------------------ ( 51, 76 ] 6 0.005% 0.005% ( 76, 110 ] 11075 9.313% 9.318% ## ( 110, 170 ] 40035 33.665% 42.982% ####### ( 170, 250 ] 18950 15.935% 58.917% ### ( 250, 380 ] 11587 9.743% 68.660% ## ( 380, 580 ] 12003 10.093% 78.753% ## ( 580, 870 ] 9370 7.879% 86.633% ## ( 870, 1300 ] 6098 5.128% 91.760% # ( 1300, 1900 ] 3252 2.735% 94.495% # ( 1900, 2900 ] 2731 2.296% 96.791% ( 2900, 4400 ] 2798 2.353% 99.144% ( 4400, 6600 ] 851 0.716% 99.860% ( 6600, 9900 ] 151 0.127% 99.987% ( 9900, 14000 ] 5 0.004% 99.991% ( 14000, 22000 ] 6 0.005% 99.996% ( 22000, 33000 ] 3 0.003% 99.998% ( 33000, 50000 ] 9 0.008% 100.006% ** DB Stats ** Uptime(secs): 21.8 total, 4.5 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 180.81 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.466 H:M:S, 29.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 220.78 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.834 H:M:S, 18.7 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/05-09:56:51 ... thread 0: (1000000,5000000) ops and (125190.7,169113.6) ops/second in (7.987811,29.565931) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/12 4.48 GB 213.5 4.9 0.0 4.9 9.6 4.7 0.0 2.0 116.1 229.0 42.87 32.12 84 0.510 5052K 8825 0.0 0.0 L1 0/0 0.00 KB 0.0 0.7 0.4 0.3 0.7 0.4 0.0 1.7 119.0 118.8 5.92 3.24 2 2.959 714K 916 0.0 0.0 L2 1/0 66.34 MB 0.7 0.6 0.4 0.3 0.6 0.3 0.1 1.7 122.4 122.2 5.12 2.95 4 1.279 636K 1017 0.0 0.0 L3 16/0 928.21 MB 0.9 4.3 2.2 2.1 4.3 2.1 0.0 1.9 125.0 123.4 35.35 20.17 17 2.079 2850K 718 0.0 0.0 L4 171/0 9.69 GB 1.0 8.7 5.6 3.1 8.6 5.5 0.5 1.5 123.7 122.5 72.02 41.47 51 1.412 3731K 6855 0.0 0.0 L5 1140/0 93.81 GB 1.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1347/12 108.95 GB 0.0 19.2 8.5 10.6 23.8 13.1 6.6 5.0 121.7 150.8 161.28 99.96 158 1.021 12M 18K 0.0 0.0 Int 0/0 0.00 KB 0.0 19.2 8.5 10.6 23.8 13.1 6.6 5.0 121.7 150.8 161.28 99.96 158 1.021 12M 18K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 19.2 8.5 10.6 19.0 8.4 0.0 0.0 141.4 140.2 138.89 86.65 89 1.561 12M 18K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 216.7 22.39 13.30 69 0.324 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 29.8 total, 29.6 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 23.76 GB write, 817.39 MB/s write, 19.18 GB read, 659.78 MB/s read, 161.3 seconds Interval compaction: 23.76 GB write, 822.74 MB/s write, 19.18 GB read, 664.09 MB/s read, 161.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 16, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 16, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 48, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 16, total-stops: 48, interval: 64 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1704641 Average: 13.5714 StdDev: 77.47 Min: 0 Median: 1.5941 Max: 8842 Percentiles: P50: 1.59 P75: 2.22 P99: 232.25 P99.9: 704.02 P99.99: 1174.11 ------------------------------------------------------ [ 0, 1 ] 298789 17.528% 17.528% #### ( 1, 2 ] 931691 54.656% 72.184% ########### ( 2, 3 ] 214321 12.573% 84.757% ### ( 3, 4 ] 81429 4.777% 89.534% # ( 4, 6 ] 68549 4.021% 93.555% # ( 6, 10 ] 13468 0.790% 94.345% ( 10, 15 ] 2557 0.150% 94.495% ( 15, 22 ] 316 0.019% 94.514% ( 22, 34 ] 30 0.002% 94.516% ( 34, 51 ] 38 0.002% 94.518% ( 51, 76 ] 42 0.002% 94.520% ( 76, 110 ] 14427 0.846% 95.367% ( 110, 170 ] 48516 2.846% 98.213% # ( 170, 250 ] 17250 1.012% 99.225% ( 250, 380 ] 5944 0.349% 99.573% ( 380, 580 ] 4748 0.279% 99.852% ( 580, 870 ] 1923 0.113% 99.965% ( 870, 1300 ] 613 0.036% 100.001% ( 1300, 1900 ] 293 0.017% 100.018% ( 1900, 2900 ] 241 0.014% 100.032% ( 2900, 4400 ] 194 0.011% 100.043% ( 4400, 6600 ] 32 0.002% 100.045% ( 6600, 9900 ] 2 0.000% 100.045% ** Level 1 read latency histogram (micros): Count: 204070 Average: 11.3992 StdDev: 146.26 Min: 1 Median: 1.9480 Max: 12011 Percentiles: P50: 1.95 P75: 2.93 P99: 10.49 P99.9: 2449.21 P99.99: 4309.67 ------------------------------------------------------ [ 0, 1 ] 5528 2.709% 2.709% # ( 1, 2 ] 101802 49.886% 52.595% ########## ( 2, 3 ] 49116 24.068% 76.663% ##### ( 3, 4 ] 19864 9.734% 86.397% ## ( 4, 6 ] 20522 10.056% 96.453% ## ( 6, 10 ] 5119 2.508% 98.962% # ( 10, 15 ] 796 0.390% 99.352% ( 15, 22 ] 111 0.054% 99.406% ( 22, 34 ] 10 0.005% 99.411% ( 34, 51 ] 14 0.007% 99.418% ( 51, 76 ] 9 0.004% 99.422% ( 76, 110 ] 11 0.005% 99.428% ( 110, 170 ] 36 0.018% 99.445% ( 170, 250 ] 42 0.021% 99.466% ( 250, 380 ] 75 0.037% 99.503% ( 380, 580 ] 124 0.061% 99.563% ( 580, 870 ] 222 0.109% 99.672% ( 870, 1300 ] 217 0.106% 99.779% ( 1300, 1900 ] 165 0.081% 99.859% ( 1900, 2900 ] 151 0.074% 99.933% ( 2900, 4400 ] 123 0.060% 99.994% ( 4400, 6600 ] 26 0.013% 100.006% ( 6600, 9900 ] 5 0.002% 100.009% ( 9900, 14000 ] 1 0.000% 100.009% ** Level 2 read latency histogram (micros): Count: 564800 Average: 12.7076 StdDev: 164.69 Min: 1 Median: 1.9780 Max: 34199 Percentiles: P50: 1.98 P75: 2.99 P99: 12.12 P99.9: 1489.14 P99.99: 2282.40 ------------------------------------------------------ [ 0, 1 ] 13659 2.418% 2.418% ( 1, 2 ] 274783 48.651% 51.070% ########## ( 2, 3 ] 135884 24.059% 75.129% ##### ( 3, 4 ] 58526 10.362% 85.491% ## ( 4, 6 ] 59794 10.587% 96.078% ## ( 6, 10 ] 15300 2.709% 98.786% # ( 10, 15 ] 2838 0.502% 99.289% ( 15, 22 ] 390 0.069% 99.358% ( 22, 34 ] 92 0.016% 99.374% ( 34, 51 ] 53 0.009% 99.384% ( 51, 76 ] 37 0.007% 99.390% ( 76, 110 ] 126 0.022% 99.413% ( 110, 170 ] 397 0.070% 99.483% ( 170, 250 ] 252 0.045% 99.527% ( 250, 380 ] 298 0.053% 99.580% ( 380, 580 ] 483 0.086% 99.666% ( 580, 870 ] 576 0.102% 99.768% ( 870, 1300 ] 594 0.105% 99.873% ( 1300, 1900 ] 486 0.086% 99.959% ( 1900, 2900 ] 459 0.081% 100.040% ( 2900, 4400 ] 391 0.069% 100.109% ( 4400, 6600 ] 117 0.021% 100.130% ( 6600, 9900 ] 21 0.004% 100.134% ( 9900, 14000 ] 1 0.000% 100.134% ( 33000, 50000 ] 2 0.000% 100.134% ** Level 3 read latency histogram (micros): Count: 1892386 Average: 10.5226 StdDev: 145.68 Min: 1 Median: 1.9638 Max: 31712 Percentiles: P50: 1.96 P75: 2.95 P99: 9.94 P99.9: 781.15 P99.99: 1203.23 ------------------------------------------------------ [ 0, 1 ] 33977 1.795% 1.795% ( 1, 2 ] 946474 50.015% 51.810% ########## ( 2, 3 ] 461240 24.373% 76.184% ##### ( 3, 4 ] 192700 10.183% 86.367% ## ( 4, 6 ] 194476 10.277% 96.643% ## ( 6, 10 ] 45284 2.393% 99.036% ( 10, 15 ] 6957 0.368% 99.404% ( 15, 22 ] 919 0.049% 99.453% ( 22, 34 ] 93 0.005% 99.458% ( 34, 51 ] 122 0.006% 99.464% ( 51, 76 ] 127 0.007% 99.471% ( 76, 110 ] 764 0.040% 99.511% ( 110, 170 ] 2676 0.141% 99.652% ( 170, 250 ] 1243 0.066% 99.718% ( 250, 380 ] 923 0.049% 99.767% ( 380, 580 ] 1354 0.072% 99.838% ( 580, 870 ] 1679 0.089% 99.927% ( 870, 1300 ] 1534 0.081% 100.008% ( 1300, 1900 ] 1063 0.056% 100.064% ( 1900, 2900 ] 1077 0.057% 100.121% ( 2900, 4400 ] 1005 0.053% 100.174% ( 4400, 6600 ] 301 0.016% 100.190% ( 6600, 9900 ] 52 0.003% 100.193% ( 9900, 14000 ] 7 0.000% 100.193% ( 14000, 22000 ] 8 0.000% 100.194% ( 22000, 33000 ] 3 0.000% 100.194% ** Level 4 read latency histogram (micros): Count: 846551 Average: 28.7664 StdDev: 226.16 Min: 0 Median: 2.0439 Max: 40256 Percentiles: P50: 2.04 P75: 3.28 P99: 568.33 P99.9: 2469.09 P99.99: 3358.57 ------------------------------------------------------ [ 0, 1 ] 13584 1.605% 1.605% ( 1, 2 ] 401059 47.376% 48.980% ######### ( 2, 3 ] 196444 23.205% 72.185% ##### ( 3, 4 ] 84008 9.924% 82.109% ## ( 4, 6 ] 81938 9.679% 91.788% ## ( 6, 10 ] 19360 2.287% 94.075% ( 10, 15 ] 3036 0.359% 94.434% ( 15, 22 ] 384 0.045% 94.479% ( 22, 34 ] 46 0.005% 94.484% ( 34, 51 ] 80 0.009% 94.494% ( 51, 76 ] 69 0.008% 94.502% ( 76, 110 ] 5351 0.632% 95.134% ( 110, 170 ] 18111 2.139% 97.274% ( 170, 250 ] 7229 0.854% 98.127% ( 250, 380 ] 3698 0.437% 98.564% ( 380, 580 ] 3917 0.463% 99.027% ( 580, 870 ] 3203 0.378% 99.405% ( 870, 1300 ] 2309 0.273% 99.678% ( 1300, 1900 ] 1257 0.148% 99.827% ( 1900, 2900 ] 1092 0.129% 99.956% ( 2900, 4400 ] 953 0.113% 100.068% ( 4400, 6600 ] 225 0.027% 100.095% ( 6600, 9900 ] 54 0.006% 100.101% ( 9900, 14000 ] 7 0.001% 100.102% ( 14000, 22000 ] 5 0.001% 100.103% ( 22000, 33000 ] 6 0.001% 100.103% ( 33000, 50000 ] 1 0.000% 100.103% ** Level 5 read latency histogram (micros): Count: 226943 Average: 348.9187 StdDev: 679.53 Min: 74 Median: 160.6081 Max: 47713 Percentiles: P50: 160.61 P75: 278.54 P99: 3730.14 P99.9: 6437.28 P99.99: 9688.14 ------------------------------------------------------ ( 51, 76 ] 16 0.007% 0.007% ( 76, 110 ] 29199 12.866% 12.873% ### ( 110, 170 ] 99893 44.017% 56.890% ######### ( 170, 250 ] 37294 16.433% 73.323% ### ( 250, 380 ] 17332 7.637% 80.960% ## ( 380, 580 ] 16504 7.272% 88.233% # ( 580, 870 ] 10585 4.664% 92.897% # ( 870, 1300 ] 6249 2.754% 95.650% # ( 1300, 1900 ] 3287 1.448% 97.099% ( 1900, 2900 ] 2760 1.216% 98.315% ( 2900, 4400 ] 2809 1.238% 99.553% ( 4400, 6600 ] 851 0.375% 99.928% ( 6600, 9900 ] 151 0.067% 99.994% ( 9900, 14000 ] 5 0.002% 99.996% ( 14000, 22000 ] 6 0.003% 99.999% ( 22000, 33000 ] 3 0.001% 100.000% ( 33000, 50000 ] 9 0.004% 100.004% ** DB Stats ** Uptime(secs): 29.8 total, 8.0 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 165.35 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:10.789 H:M:S, 36.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 123.21 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:4.323 H:M:S, 54.1 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 260.600 micros/op 15344 ops/sec 60.152 seconds 922996 operations; 14.9 MB/s (226999 of 226999 found) Microseconds per read: Count: 922996 Average: 260.6019 StdDev: 735.42 Min: 2 Median: 176.9361 Max: 67504 Percentiles: P50: 176.94 P75: 239.85 P99: 1544.45 P99.9: 11688.14 P99.99: 30121.46 ------------------------------------------------------ ( 1, 2 ] 9 0.001% 0.001% ( 2, 3 ] 34 0.004% 0.005% ( 3, 4 ] 39 0.004% 0.009% ( 4, 6 ] 88 0.010% 0.018% ( 6, 10 ] 422 0.046% 0.064% ( 10, 15 ] 3654 0.396% 0.460% ( 15, 22 ] 2787 0.302% 0.762% ( 22, 34 ] 7335 0.795% 1.557% ( 34, 51 ] 2046 0.222% 1.778% ( 51, 76 ] 240 0.026% 1.804% ( 76, 110 ] 21300 2.308% 4.112% ( 110, 170 ] 398104 43.132% 47.244% ######### ( 170, 250 ] 293422 31.790% 79.034% ###### ( 250, 380 ] 110155 11.935% 90.968% ## ( 380, 580 ] 44575 4.829% 95.798% # ( 580, 870 ] 19300 2.091% 97.889% ( 870, 1300 ] 8749 0.948% 98.837% ( 1300, 1900 ] 3699 0.401% 99.237% ( 1900, 2900 ] 2692 0.292% 99.529% ( 2900, 4400 ] 2327 0.252% 99.781% ( 4400, 6600 ] 620 0.067% 99.848% ( 6600, 9900 ] 278 0.030% 99.879% ( 9900, 14000 ] 454 0.049% 99.928% ( 14000, 22000 ] 461 0.050% 99.978% ( 22000, 33000 ] 154 0.017% 99.994% ( 33000, 50000 ] 42 0.005% 99.999% ( 50000, 75000 ] 10 0.001% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 17075276 rocksdb.bloom.filter.full.positive COUNT : 1089346 rocksdb.bloom.filter.full.true.positive COUNT : 922593 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 403 rocksdb.memtable.miss COUNT : 922593 rocksdb.l0.hit COUNT : 39933 rocksdb.l1.hit COUNT : 1683 rocksdb.l2andup.hit COUNT : 880977 rocksdb.compaction.key.drop.new COUNT : 179613 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5575404 rocksdb.number.keys.read COUNT : 922996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5753816928 rocksdb.bytes.read COUNT : 922996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1678 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 38559358 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 23870607544 rocksdb.compact.write.bytes COUNT : 23680044799 rocksdb.flush.write.bytes COUNT : 5747009931 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 18 rocksdb.number.superversion_acquires COUNT : 1133 rocksdb.number.superversion_releases COUNT : 1098 rocksdb.number.superversion_cleanups COUNT : 282 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 324 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5700628480 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2182144 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 28462204353 rocksdb.non.last.level.read.count COUNT : 6734795 rocksdb.block.checksum.compute.count COUNT : 6881489 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 175.255673 P95 : 546.223972 P99 : 1542.898457 P100 : 67502.000000 COUNT : 922996 SUM : 239706484 rocksdb.db.write.micros P50 : 1.548334 P95 : 4.441091 P99 : 9.203401 P100 : 839009.000000 COUNT : 5575404 SUM : 52659756 rocksdb.compaction.times.micros P50 : 1368518.518519 P95 : 3681666.666667 P99 : 34120000.000000 P100 : 42328794.000000 COUNT : 93 SUM : 186160779 rocksdb.compaction.times.cpu_micros P50 : 904074.074074 P95 : 2250000.000000 P99 : 10094000.000000 P100 : 12457676.000000 COUNT : 93 SUM : 103967549 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 831.617647 P95 : 50625.000000 P99 : 277300.000000 P100 : 286279.000000 COUNT : 79 SUM : 701495 rocksdb.compaction.outfile.sync.micros P50 : 12145.238095 P95 : 270681.818182 P99 : 482125.000000 P100 : 587200.000000 COUNT : 285 SUM : 15502500 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 228.333333 P95 : 3736.538462 P99 : 5903.333333 P100 : 8148.000000 COUNT : 195 SUM : 171595 rocksdb.table.open.io.micros P50 : 8455.828221 P95 : 19892.765957 P99 : 24734.285714 P100 : 38439.000000 COUNT : 1678 SUM : 14594704 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.910489 P95 : 13.126745 P99 : 185.719440 P100 : 34206.000000 COUNT : 5779791 SUM : 92561924 rocksdb.read.block.get.micros P50 : 140.476029 P95 : 422.812742 P99 : 1394.646218 P100 : 47717.000000 COUNT : 1100016 SUM : 205943959 rocksdb.write.raw.block.micros P50 : 0.529076 P95 : 1.143110 P99 : 3.473241 P100 : 70498.000000 COUNT : 7127556 SUM : 21617159 rocksdb.numfiles.in.singlecompaction P50 : 2.919355 P95 : 5.645455 P99 : 14.050000 P100 : 25.000000 COUNT : 119 SUM : 433 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.568500 P95 : 1291.495062 P99 : 2695.368033 P100 : 838839.000000 COUNT : 184807 SUM : 38575701 rocksdb.sst.read.micros P50 : 2.055814 P95 : 164.745996 P99 : 489.569455 P100 : 47713.000000 COUNT : 6734814 SUM : 282923980 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 922996 SUM : 922996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5575404 SUM : 5753816928 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 225438.596491 P95 : 976166.666667 P99 : 1090714.000000 P100 : 1090714.000000 COUNT : 79 SUM : 24457019 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 324 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0