[2024-11-29 07:57:01.609225] Starting SPDK v24.09.1-pre git sha1 b18e1bd62 / DPDK 23.11.0 initialization... [2024-11-29 07:57:01.609337] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3366382 ] Set seed to 1732863421606760 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-29 07:57:01.710913] app.c: 917:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-29 07:57:01.752537] app.c: 610:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-29 07:57:01.752597] app.c: 611:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 3366382' to capture a snapshot of events at runtime. [2024-11-29 07:57:01.752609] app.c: 616:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-29 07:57:01.752620] app.c: 617:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-29 07:57:01.752630] app.c: 618:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid3366382 for offline analysis/debug. [2024-11-29 07:57:01.752713] reactor.c: 990:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Fri Nov 29 07:57:05 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/29-07:57:14 ... thread 0: (1000000,1000000) ops and (123849.4,123849.4) ops/second in (8.074323,8.074323) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 11/8 1.35 GB 56.6 0.4 0.0 0.4 1.2 0.9 0.0 1.4 37.9 126.4 9.96 4.02 11 0.905 383K 541 0.0 0.0 L1 5/5 433.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 2/0 171.17 MB 1.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 31/8 2.63 GB 1.7 2.1 1.3 0.8 2.0 1.2 0.0 1.6 96.6 95.1 21.89 10.15 11 1.990 822K 0 0.0 0.0 L4 200/27 12.57 GB 1.1 4.2 3.0 1.1 4.1 3.0 0.0 1.4 109.2 108.0 39.03 19.91 28 1.394 2219K 19 0.0 0.0 L5 1096/0 90.18 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1345/48 107.32 GB 0.0 6.6 4.3 2.3 7.4 5.1 0.0 8.6 95.3 106.6 70.89 34.08 50 1.418 3426K 560 0.0 0.0 Int 0/0 0.00 KB 0.0 6.6 4.3 2.3 7.4 5.1 0.0 8.6 95.3 106.6 70.89 34.08 50 1.418 3426K 560 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.6 4.3 2.3 6.5 4.2 0.0 0.0 106.6 105.3 63.40 31.57 40 1.585 3426K 560 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 117.7 7.49 2.51 10 0.749 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 8.2 total, 8.1 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.38 GB write, 921.33 MB/s write, 6.60 GB read, 823.80 MB/s read, 70.9 seconds Interval compaction: 7.38 GB write, 935.66 MB/s write, 6.60 GB read, 836.62 MB/s read, 70.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 29, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 29, interval: 29 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 300845 Average: 15.2751 StdDev: 182.22 Min: 1 Median: 1.8679 Max: 11640 Percentiles: P50: 1.87 P75: 2.89 P99: 12.97 P99.9: 3245.31 P99.99: 5888.32 ------------------------------------------------------ [ 0, 1 ] 13154 4.372% 4.372% # ( 1, 2 ] 158169 52.575% 56.947% ########### ( 2, 3 ] 61325 20.384% 77.332% #### ( 3, 4 ] 28794 9.571% 86.903% ## ( 4, 6 ] 29145 9.688% 96.590% ## ( 6, 10 ] 6614 2.198% 98.789% ( 10, 15 ] 1071 0.356% 99.145% ( 15, 22 ] 103 0.034% 99.179% ( 22, 34 ] 25 0.008% 99.187% ( 34, 51 ] 14 0.005% 99.192% ( 51, 76 ] 20 0.007% 99.199% ( 76, 110 ] 11 0.004% 99.202% ( 110, 170 ] 25 0.008% 99.211% ( 170, 250 ] 55 0.018% 99.229% ( 250, 380 ] 142 0.047% 99.276% ( 380, 580 ] 279 0.093% 99.369% ( 580, 870 ] 462 0.154% 99.522% ( 870, 1300 ] 460 0.153% 99.675% ( 1300, 1900 ] 359 0.119% 99.795% ( 1900, 2900 ] 255 0.085% 99.879% ( 2900, 4400 ] 270 0.090% 99.969% ( 4400, 6600 ] 93 0.031% 100.000% ( 6600, 9900 ] 15 0.005% 100.005% ( 9900, 14000 ] 1 0.000% 100.005% ** Level 1 read latency histogram (micros): Count: 71892 Average: 13.6218 StdDev: 179.99 Min: 1 Median: 2.1509 Max: 8338 Percentiles: P50: 2.15 P75: 3.22 P99: 12.34 P99.9: 3197.54 P99.99: 6579.19 ------------------------------------------------------ [ 0, 1 ] 667 0.928% 0.928% ( 1, 2 ] 32394 45.059% 45.987% ######### ( 2, 3 ] 19116 26.590% 72.577% ##### ( 3, 4 ] 7861 10.934% 83.511% ## ( 4, 6 ] 8652 12.035% 95.546% ## ( 6, 10 ] 2281 3.173% 98.719% # ( 10, 15 ] 431 0.600% 99.318% ( 15, 22 ] 57 0.079% 99.398% ( 22, 34 ] 7 0.010% 99.407% ( 34, 51 ] 3 0.004% 99.412% ( 51, 76 ] 6 0.008% 99.420% ( 76, 110 ] 6 0.008% 99.428% ( 110, 170 ] 5 0.007% 99.435% ( 170, 250 ] 8 0.011% 99.446% ( 250, 380 ] 16 0.022% 99.469% ( 380, 580 ] 36 0.050% 99.519% ( 580, 870 ] 59 0.082% 99.601% ( 870, 1300 ] 86 0.120% 99.720% ( 1300, 1900 ] 61 0.085% 99.805% ( 1900, 2900 ] 57 0.079% 99.885% ( 2900, 4400 ] 56 0.078% 99.962% ( 4400, 6600 ] 20 0.028% 99.990% ( 6600, 9900 ] 7 0.010% 100.000% ** Level 2 read latency histogram (micros): Count: 218231 Average: 16.6181 StdDev: 226.07 Min: 1 Median: 1.9810 Max: 15830 Percentiles: P50: 1.98 P75: 3.05 P99: 12.97 P99.9: 2724.72 P99.99: 4062.33 ------------------------------------------------------ [ 0, 1 ] 6800 3.116% 3.116% # ( 1, 2 ] 104301 47.794% 50.910% ########## ( 2, 3 ] 51352 23.531% 74.441% ##### ( 3, 4 ] 22553 10.334% 84.775% ## ( 4, 6 ] 23411 10.728% 95.503% ## ( 6, 10 ] 6805 3.118% 98.621% # ( 10, 15 ] 1392 0.638% 99.259% ( 15, 22 ] 222 0.102% 99.361% ( 22, 34 ] 48 0.022% 99.383% ( 34, 51 ] 10 0.005% 99.387% ( 51, 76 ] 11 0.005% 99.392% ( 76, 110 ] 8 0.004% 99.396% ( 110, 170 ] 27 0.012% 99.408% ( 170, 250 ] 49 0.022% 99.431% ( 250, 380 ] 97 0.044% 99.475% ( 380, 580 ] 157 0.072% 99.547% ( 580, 870 ] 216 0.099% 99.646% ( 870, 1300 ] 207 0.095% 99.741% ( 1300, 1900 ] 181 0.083% 99.824% ( 1900, 2900 ] 201 0.092% 99.916% ( 2900, 4400 ] 208 0.095% 100.011% ( 4400, 6600 ] 125 0.057% 100.069% ( 6600, 9900 ] 28 0.013% 100.082% ( 9900, 14000 ] 2 0.001% 100.082% ( 14000, 22000 ] 5 0.002% 100.085% ** Level 3 read latency histogram (micros): Count: 913015 Average: 12.8006 StdDev: 176.14 Min: 1 Median: 1.9605 Max: 19226 Percentiles: P50: 1.96 P75: 2.97 P99: 9.55 P99.9: 728.12 P99.99: 1049.62 ------------------------------------------------------ [ 0, 1 ] 17703 1.939% 1.939% ( 1, 2 ] 456835 50.036% 51.975% ########## ( 2, 3 ] 217474 23.819% 75.794% ##### ( 3, 4 ] 96062 10.521% 86.316% ## ( 4, 6 ] 96391 10.557% 96.873% ## ( 6, 10 ] 21884 2.397% 99.270% ( 10, 15 ] 3781 0.414% 99.684% ( 15, 22 ] 514 0.056% 99.740% ( 22, 34 ] 56 0.006% 99.746% ( 34, 51 ] 39 0.004% 99.751% ( 51, 76 ] 58 0.006% 99.757% ( 76, 110 ] 21 0.002% 99.759% ( 110, 170 ] 48 0.005% 99.765% ( 170, 250 ] 75 0.008% 99.773% ( 250, 380 ] 228 0.025% 99.798% ( 380, 580 ] 507 0.056% 99.853% ( 580, 870 ] 836 0.092% 99.945% ( 870, 1300 ] 988 0.108% 100.053% ( 1300, 1900 ] 706 0.077% 100.130% ( 1900, 2900 ] 657 0.072% 100.202% ( 2900, 4400 ] 698 0.076% 100.279% ( 4400, 6600 ] 255 0.028% 100.307% ( 6600, 9900 ] 70 0.008% 100.314% ( 9900, 14000 ] 3 0.000% 100.315% ( 14000, 22000 ] 4 0.000% 100.315% ** Level 4 read latency histogram (micros): Count: 400883 Average: 23.8182 StdDev: 245.44 Min: 1 Median: 1.9643 Max: 39406 Percentiles: P50: 1.96 P75: 3.01 P99: 420.22 P99.9: 2451.53 P99.99: 3106.38 ------------------------------------------------------ [ 0, 1 ] 7765 1.937% 1.937% ( 1, 2 ] 199800 49.840% 51.777% ########## ( 2, 3 ] 92815 23.153% 74.930% ##### ( 3, 4 ] 42875 10.695% 85.625% ## ( 4, 6 ] 41154 10.266% 95.891% ## ( 6, 10 ] 10034 2.503% 98.394% # ( 10, 15 ] 1604 0.400% 98.794% ( 15, 22 ] 202 0.050% 98.844% ( 22, 34 ] 21 0.005% 98.849% ( 34, 51 ] 17 0.004% 98.854% ( 51, 76 ] 37 0.009% 98.863% ( 76, 110 ] 11 0.003% 98.866% ( 110, 170 ] 54 0.013% 98.879% ( 170, 250 ] 138 0.034% 98.913% ( 250, 380 ] 245 0.061% 98.975% ( 380, 580 ] 513 0.128% 99.102% ( 580, 870 ] 942 0.235% 99.337% ( 870, 1300 ] 1066 0.266% 99.603% ( 1300, 1900 ] 841 0.210% 99.813% ( 1900, 2900 ] 633 0.158% 99.971% ( 2900, 4400 ] 559 0.139% 100.111% ( 4400, 6600 ] 193 0.048% 100.159% ( 6600, 9900 ] 45 0.011% 100.170% ( 9900, 14000 ] 3 0.001% 100.171% ( 33000, 50000 ] 1 0.000% 100.171% ** Level 5 read latency histogram (micros): Count: 12182 Average: 1891.4155 StdDev: 1823.11 Min: 83 Median: 1244.2380 Max: 55345 Percentiles: P50: 1244.24 P75: 2718.91 P99: 8101.50 P99.9: 10950.35 P99.99: 46290.60 ------------------------------------------------------ ( 76, 110 ] 12 0.099% 0.099% ( 110, 170 ] 62 0.509% 0.607% ( 170, 250 ] 172 1.412% 2.019% ( 250, 380 ] 583 4.786% 6.805% # ( 380, 580 ] 1276 10.474% 17.280% ## ( 580, 870 ] 2033 16.689% 33.968% ### ( 870, 1300 ] 2244 18.421% 52.389% #### ( 1300, 1900 ] 1581 12.978% 65.367% ### ( 1900, 2900 ] 1433 11.763% 77.130% ## ( 2900, 4400 ] 1711 14.045% 91.176% ### ( 4400, 6600 ] 864 7.092% 98.268% # ( 6600, 9900 ] 196 1.609% 99.877% ( 9900, 14000 ] 11 0.090% 99.967% ( 14000, 22000 ] 1 0.008% 99.975% ( 22000, 33000 ] 1 0.008% 99.984% ( 33000, 50000 ] 1 0.008% 99.992% ( 50000, 75000 ] 2 0.016% 100.008% ** DB Stats ** Uptime(secs): 8.2 total, 8.1 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 119.99 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:4.101 H:M:S, 50.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 121.85 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:4.101 H:M:S, 50.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/11/29-07:57:20 ... thread 0: (1000000,2000000) ops and (165723.9,141759.0) ops/second in (6.034131,14.108454) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 11/4 1.84 GB 163.6 1.4 0.0 1.4 3.2 1.8 0.0 1.7 63.1 149.1 21.96 10.86 27 0.813 1407K 1849 0.0 0.0 L1 3/0 301.87 MB 30.2 0.9 0.5 0.4 0.9 0.5 0.0 1.9 92.3 92.0 10.07 4.53 1 10.073 944K 2994 0.0 0.0 L2 2/1 257.53 MB 1.3 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 13/4 1015.95 MB 0.6 2.6 1.6 0.9 2.5 1.6 0.2 1.5 98.6 97.2 26.64 12.56 15 1.776 1206K 0 0.0 0.0 L4 171/4 9.70 GB 1.0 9.1 4.9 4.1 8.9 4.8 0.3 1.8 119.8 117.8 77.51 42.93 43 1.802 3040K 7649 0.0 0.0 L5 1156/0 95.11 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1356/13 108.19 GB 0.0 13.9 7.1 6.8 15.6 8.7 6.0 8.4 104.5 116.9 136.18 70.89 86 1.584 6598K 12K 0.0 0.0 Int 0/0 0.00 KB 0.0 13.9 7.1 6.8 15.6 8.7 6.0 8.4 104.5 116.9 136.18 70.89 86 1.584 6598K 12K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 13.9 7.1 6.8 13.7 6.9 0.0 0.0 115.8 114.2 122.86 65.57 63 1.950 6598K 12K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 141.8 13.33 5.32 23 0.579 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.2 total, 14.1 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 15.55 GB write, 1118.51 MB/s write, 13.89 GB read, 999.40 MB/s read, 136.2 seconds Interval compaction: 15.55 GB write, 1128.47 MB/s write, 13.89 GB read, 1008.30 MB/s read, 136.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 45, pending-compaction-bytes-delays: 2, pending-compaction-bytes-stops: 0, total-delays: 2, total-stops: 45, interval: 47 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 544381 Average: 15.0429 StdDev: 166.72 Min: 1 Median: 1.8181 Max: 11640 Percentiles: P50: 1.82 P75: 2.78 P99: 171.25 P99.9: 2692.53 P99.99: 4711.42 ------------------------------------------------------ [ 0, 1 ] 28517 5.238% 5.238% # ( 1, 2 ] 297836 54.711% 59.949% ########### ( 2, 3 ] 105684 19.414% 79.363% #### ( 3, 4 ] 47762 8.774% 88.137% ## ( 4, 6 ] 47258 8.681% 96.818% ## ( 6, 10 ] 9843 1.808% 98.626% ( 10, 15 ] 1550 0.285% 98.911% ( 15, 22 ] 146 0.027% 98.937% ( 22, 34 ] 32 0.006% 98.943% ( 34, 51 ] 19 0.003% 98.947% ( 51, 76 ] 23 0.004% 98.951% ( 76, 110 ] 45 0.008% 98.959% ( 110, 170 ] 217 0.040% 98.999% ( 170, 250 ] 332 0.061% 99.060% ( 250, 380 ] 510 0.094% 99.154% ( 380, 580 ] 897 0.165% 99.318% ( 580, 870 ] 1231 0.226% 99.545% ( 870, 1300 ] 1002 0.184% 99.729% ( 1300, 1900 ] 595 0.109% 99.838% ( 1900, 2900 ] 426 0.078% 99.916% ( 2900, 4400 ] 385 0.071% 99.987% ( 4400, 6600 ] 117 0.021% 100.008% ( 6600, 9900 ] 17 0.003% 100.012% ( 9900, 14000 ] 3 0.001% 100.012% ** Level 1 read latency histogram (micros): Count: 208980 Average: 12.7394 StdDev: 158.41 Min: 1 Median: 2.1815 Max: 8338 Percentiles: P50: 2.18 P75: 3.25 P99: 13.38 P99.9: 2520.31 P99.99: 4369.39 ------------------------------------------------------ [ 0, 1 ] 1739 0.832% 0.832% ( 1, 2 ] 92455 44.241% 45.073% ######### ( 2, 3 ] 56734 27.148% 72.221% ##### ( 3, 4 ] 23491 11.241% 83.462% ## ( 4, 6 ] 24899 11.915% 95.377% ## ( 6, 10 ] 6723 3.217% 98.594% # ( 10, 15 ] 1255 0.601% 99.194% ( 15, 22 ] 182 0.087% 99.281% ( 22, 34 ] 13 0.006% 99.287% ( 34, 51 ] 23 0.011% 99.298% ( 51, 76 ] 22 0.011% 99.309% ( 76, 110 ] 18 0.009% 99.318% ( 110, 170 ] 47 0.022% 99.340% ( 170, 250 ] 74 0.035% 99.376% ( 250, 380 ] 114 0.055% 99.430% ( 380, 580 ] 184 0.088% 99.518% ( 580, 870 ] 297 0.142% 99.660% ( 870, 1300 ] 266 0.127% 99.788% ( 1300, 1900 ] 155 0.074% 99.862% ( 1900, 2900 ] 129 0.062% 99.923% ( 2900, 4400 ] 142 0.068% 99.991% ( 4400, 6600 ] 41 0.020% 100.011% ( 6600, 9900 ] 12 0.006% 100.017% ** Level 2 read latency histogram (micros): Count: 250980 Average: 15.1656 StdDev: 211.76 Min: 1 Median: 1.9678 Max: 15830 Percentiles: P50: 1.97 P75: 3.00 P99: 12.43 P99.9: 2573.27 P99.99: 4040.44 ------------------------------------------------------ [ 0, 1 ] 7589 3.024% 3.024% # ( 1, 2 ] 121820 48.538% 51.561% ########## ( 2, 3 ] 58704 23.390% 74.951% ##### ( 3, 4 ] 25554 10.182% 85.133% ## ( 4, 6 ] 26392 10.516% 95.649% ## ( 6, 10 ] 7691 3.064% 98.713% # ( 10, 15 ] 1480 0.590% 99.303% ( 15, 22 ] 232 0.092% 99.395% ( 22, 34 ] 48 0.019% 99.414% ( 34, 51 ] 15 0.006% 99.420% ( 51, 76 ] 15 0.006% 99.426% ( 76, 110 ] 8 0.003% 99.429% ( 110, 170 ] 30 0.012% 99.441% ( 170, 250 ] 56 0.022% 99.464% ( 250, 380 ] 106 0.042% 99.506% ( 380, 580 ] 195 0.078% 99.584% ( 580, 870 ] 244 0.097% 99.681% ( 870, 1300 ] 223 0.089% 99.770% ( 1300, 1900 ] 189 0.075% 99.845% ( 1900, 2900 ] 205 0.082% 99.927% ( 2900, 4400 ] 209 0.083% 100.010% ( 4400, 6600 ] 125 0.050% 100.060% ( 6600, 9900 ] 28 0.011% 100.071% ( 9900, 14000 ] 2 0.001% 100.072% ( 14000, 22000 ] 5 0.002% 100.074% ** Level 3 read latency histogram (micros): Count: 1369743 Average: 10.9867 StdDev: 154.82 Min: 1 Median: 1.9411 Max: 19226 Percentiles: P50: 1.94 P75: 2.94 P99: 9.50 P99.9: 695.30 P99.99: 1034.43 ------------------------------------------------------ [ 0, 1 ] 28011 2.045% 2.045% ( 1, 2 ] 697986 50.957% 53.002% ########## ( 2, 3 ] 320813 23.421% 76.424% ##### ( 3, 4 ] 139786 10.205% 86.629% ## ( 4, 6 ] 139902 10.214% 96.843% ## ( 6, 10 ] 33727 2.462% 99.305% ( 10, 15 ] 5294 0.386% 99.692% ( 15, 22 ] 688 0.050% 99.742% ( 22, 34 ] 70 0.005% 99.747% ( 34, 51 ] 68 0.005% 99.752% ( 51, 76 ] 94 0.007% 99.759% ( 76, 110 ] 36 0.003% 99.761% ( 110, 170 ] 119 0.009% 99.770% ( 170, 250 ] 160 0.012% 99.782% ( 250, 380 ] 369 0.027% 99.809% ( 380, 580 ] 766 0.056% 99.865% ( 580, 870 ] 1218 0.089% 99.954% ( 870, 1300 ] 1305 0.095% 100.049% ( 1300, 1900 ] 896 0.065% 100.114% ( 1900, 2900 ] 790 0.058% 100.172% ( 2900, 4400 ] 824 0.060% 100.232% ( 4400, 6600 ] 284 0.021% 100.253% ( 6600, 9900 ] 75 0.005% 100.258% ( 9900, 14000 ] 4 0.000% 100.259% ( 14000, 22000 ] 4 0.000% 100.259% ** Level 4 read latency histogram (micros): Count: 1130378 Average: 16.2180 StdDev: 179.60 Min: 0 Median: 1.9285 Max: 39406 Percentiles: P50: 1.93 P75: 2.93 P99: 13.82 P99.9: 1305.32 P99.99: 1734.28 ------------------------------------------------------ [ 0, 1 ] 22509 1.991% 1.991% ( 1, 2 ] 584486 51.707% 53.698% ########## ( 2, 3 ] 259355 22.944% 76.643% ##### ( 3, 4 ] 111909 9.900% 86.543% ## ( 4, 6 ] 109484 9.686% 96.228% ## ( 6, 10 ] 28200 2.495% 98.723% ( 10, 15 ] 4103 0.363% 99.086% ( 15, 22 ] 474 0.042% 99.128% ( 22, 34 ] 53 0.005% 99.133% ( 34, 51 ] 60 0.005% 99.138% ( 51, 76 ] 100 0.009% 99.147% ( 76, 110 ] 108 0.010% 99.156% ( 110, 170 ] 377 0.033% 99.190% ( 170, 250 ] 626 0.055% 99.245% ( 250, 380 ] 960 0.085% 99.330% ( 380, 580 ] 1733 0.153% 99.483% ( 580, 870 ] 2433 0.215% 99.699% ( 870, 1300 ] 2265 0.200% 99.899% ( 1300, 1900 ] 1423 0.126% 100.025% ( 1900, 2900 ] 1072 0.095% 100.120% ( 2900, 4400 ] 888 0.079% 100.198% ( 4400, 6600 ] 252 0.022% 100.220% ( 6600, 9900 ] 50 0.004% 100.225% ( 9900, 14000 ] 3 0.000% 100.225% ( 33000, 50000 ] 1 0.000% 100.225% ** Level 5 read latency histogram (micros): Count: 33237 Average: 1231.7533 StdDev: 1397.21 Min: 77 Median: 778.5038 Max: 55345 Percentiles: P50: 778.50 P75: 1466.52 P99: 6386.15 P99.9: 9667.21 P99.99: 19410.40 ------------------------------------------------------ ( 76, 110 ] 370 1.113% 1.113% ( 110, 170 ] 1713 5.154% 6.267% # ( 170, 250 ] 2130 6.409% 12.676% # ( 250, 380 ] 3041 9.149% 21.825% ## ( 380, 580 ] 5007 15.065% 36.890% ### ( 580, 870 ] 6366 19.153% 56.043% #### ( 870, 1300 ] 5454 16.409% 72.452% ### ( 1300, 1900 ] 3051 9.180% 81.632% ## ( 1900, 2900 ] 2372 7.137% 88.769% # ( 2900, 4400 ] 2487 7.483% 96.251% # ( 4400, 6600 ] 1012 3.045% 99.296% # ( 6600, 9900 ] 216 0.650% 99.946% ( 9900, 14000 ] 14 0.042% 99.988% ( 14000, 22000 ] 1 0.003% 99.991% ( 22000, 33000 ] 1 0.003% 99.994% ( 33000, 50000 ] 1 0.003% 99.997% ( 50000, 75000 ] 2 0.006% 100.003% ** DB Stats ** Uptime(secs): 14.2 total, 6.0 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 138.26 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.375 H:M:S, 44.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 163.11 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.274 H:M:S, 37.7 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 3 num-running-flushes: 1 2024/11/29-07:57:23 ... thread 0: (1000000,3000000) ops and (282222.6,169954.8) ops/second in (3.543302,17.651756) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/12 2.77 GB 69.2 2.1 0.0 2.1 4.9 2.8 0.0 1.7 75.3 177.1 28.44 16.79 46 0.618 2174K 2588 0.0 0.0 L1 3/3 301.87 MB 0.0 0.9 0.5 0.4 0.9 0.5 0.0 1.9 92.3 92.0 10.07 4.53 1 10.073 944K 2994 0.0 0.0 L3 18/2 1.07 GB 1.0 3.3 1.9 1.4 3.3 1.9 0.2 1.7 111.7 110.5 30.38 16.02 17 1.787 1551K 0 0.0 0.0 L4 171/3 9.56 GB 1.0 9.4 5.1 4.3 9.2 4.9 0.3 1.8 121.4 119.4 79.31 44.45 44 1.802 3140K 7649 0.0 0.0 L5 1160/0 95.41 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1369/20 109.10 GB 0.0 15.7 7.4 8.3 18.3 10.1 5.7 6.5 108.6 126.8 148.20 81.80 108 1.372 7810K 13K 0.0 0.0 Int 0/0 0.00 KB 0.0 15.7 7.4 8.3 18.3 10.1 5.7 6.5 108.6 126.8 148.20 81.80 108 1.372 7810K 13K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 15.7 7.4 8.3 15.5 7.2 0.0 0.0 122.5 120.9 131.45 73.52 69 1.905 7810K 13K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 173.0 16.75 8.29 39 0.429 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 17.8 total, 17.7 interval Flush(GB): cumulative 2.830, interval 2.830 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 18.35 GB write, 1056.82 MB/s write, 15.72 GB read, 905.34 MB/s read, 148.2 seconds Interval compaction: 18.35 GB write, 1064.34 MB/s write, 15.72 GB read, 911.78 MB/s read, 148.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 45, pending-compaction-bytes-delays: 2, pending-compaction-bytes-stops: 0, total-delays: 2, total-stops: 45, interval: 47 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 867505 Average: 12.1693 StdDev: 135.96 Min: 1 Median: 1.7350 Max: 14579 Percentiles: P50: 1.73 P75: 2.55 P99: 172.93 P99.9: 1655.17 P99.99: 3277.88 ------------------------------------------------------ [ 0, 1 ] 65912 7.598% 7.598% ## ( 1, 2 ] 500496 57.694% 65.292% ############ ( 2, 3 ] 153099 17.648% 82.940% #### ( 3, 4 ] 63532 7.324% 90.263% # ( 4, 6 ] 57862 6.670% 96.933% # ( 6, 10 ] 12415 1.431% 98.364% ( 10, 15 ] 1935 0.223% 98.587% ( 15, 22 ] 213 0.025% 98.612% ( 22, 34 ] 38 0.004% 98.616% ( 34, 51 ] 21 0.002% 98.619% ( 51, 76 ] 25 0.003% 98.622% ( 76, 110 ] 570 0.066% 98.687% ( 110, 170 ] 2642 0.305% 98.992% ( 170, 250 ] 1963 0.226% 99.218% ( 250, 380 ] 1438 0.166% 99.384% ( 380, 580 ] 1542 0.178% 99.562% ( 580, 870 ] 1506 0.174% 99.735% ( 870, 1300 ] 1070 0.123% 99.859% ( 1300, 1900 ] 609 0.070% 99.929% ( 1900, 2900 ] 434 0.050% 99.979% ( 2900, 4400 ] 390 0.045% 100.024% ( 4400, 6600 ] 117 0.013% 100.037% ( 6600, 9900 ] 17 0.002% 100.039% ( 9900, 14000 ] 3 0.000% 100.040% ( 14000, 22000 ] 1 0.000% 100.040% ** Level 1 read latency histogram (micros): Count: 275953 Average: 10.7722 StdDev: 138.69 Min: 1 Median: 1.9556 Max: 8338 Percentiles: P50: 1.96 P75: 2.93 P99: 13.17 P99.9: 1962.38 P99.99: 4245.82 ------------------------------------------------------ [ 0, 1 ] 8342 3.023% 3.023% # ( 1, 2 ] 135663 49.162% 52.185% ########## ( 2, 3 ] 67665 24.520% 76.705% ##### ( 3, 4 ] 26009 9.425% 86.130% ## ( 4, 6 ] 26741 9.690% 95.821% ## ( 6, 10 ] 7912 2.867% 98.688% # ( 10, 15 ] 1357 0.492% 99.180% ( 15, 22 ] 194 0.070% 99.250% ( 22, 34 ] 15 0.005% 99.255% ( 34, 51 ] 24 0.009% 99.264% ( 51, 76 ] 22 0.008% 99.272% ( 76, 110 ] 57 0.021% 99.293% ( 110, 170 ] 238 0.086% 99.379% ( 170, 250 ] 209 0.076% 99.455% ( 250, 380 ] 201 0.073% 99.527% ( 380, 580 ] 261 0.095% 99.622% ( 580, 870 ] 330 0.120% 99.742% ( 870, 1300 ] 274 0.099% 99.841% ( 1300, 1900 ] 155 0.056% 99.897% ( 1900, 2900 ] 129 0.047% 99.944% ( 2900, 4400 ] 142 0.051% 99.995% ( 4400, 6600 ] 41 0.015% 100.010% ( 6600, 9900 ] 12 0.004% 100.014% ** Level 2 read latency histogram (micros): Count: 251007 Average: 15.1910 StdDev: 211.77 Min: 1 Median: 1.9679 Max: 15830 Percentiles: P50: 1.97 P75: 3.01 P99: 12.52 P99.9: 2573.14 P99.99: 4040.43 ------------------------------------------------------ [ 0, 1 ] 7589 3.023% 3.023% # ( 1, 2 ] 121820 48.533% 51.556% ########## ( 2, 3 ] 58704 23.387% 74.943% ##### ( 3, 4 ] 25554 10.181% 85.124% ## ( 4, 6 ] 26392 10.514% 95.638% ## ( 6, 10 ] 7691 3.064% 98.702% # ( 10, 15 ] 1480 0.590% 99.292% ( 15, 22 ] 232 0.092% 99.384% ( 22, 34 ] 48 0.019% 99.404% ( 34, 51 ] 15 0.006% 99.410% ( 51, 76 ] 15 0.006% 99.416% ( 76, 110 ] 10 0.004% 99.420% ( 110, 170 ] 38 0.015% 99.435% ( 170, 250 ] 65 0.026% 99.461% ( 250, 380 ] 111 0.044% 99.505% ( 380, 580 ] 197 0.078% 99.583% ( 580, 870 ] 244 0.097% 99.680% ( 870, 1300 ] 224 0.089% 99.770% ( 1300, 1900 ] 189 0.075% 99.845% ( 1900, 2900 ] 205 0.082% 99.927% ( 2900, 4400 ] 209 0.083% 100.010% ( 4400, 6600 ] 125 0.050% 100.060% ( 6600, 9900 ] 28 0.011% 100.071% ( 9900, 14000 ] 2 0.001% 100.072% ( 14000, 22000 ] 5 0.002% 100.074% ** Level 3 read latency histogram (micros): Count: 1494781 Average: 10.4174 StdDev: 148.83 Min: 1 Median: 1.9129 Max: 19226 Percentiles: P50: 1.91 P75: 2.89 P99: 9.51 P99.9: 674.73 P99.99: 1031.22 ------------------------------------------------------ [ 0, 1 ] 37938 2.538% 2.538% # ( 1, 2 ] 777101 51.988% 54.526% ########## ( 2, 3 ] 343503 22.980% 77.506% ##### ( 3, 4 ] 145568 9.738% 87.244% ## ( 4, 6 ] 144301 9.654% 96.898% ## ( 6, 10 ] 35801 2.395% 99.293% ( 10, 15 ] 5508 0.368% 99.661% ( 15, 22 ] 723 0.048% 99.710% ( 22, 34 ] 73 0.005% 99.715% ( 34, 51 ] 68 0.005% 99.719% ( 51, 76 ] 96 0.006% 99.726% ( 76, 110 ] 112 0.007% 99.733% ( 110, 170 ] 413 0.028% 99.761% ( 170, 250 ] 346 0.023% 99.784% ( 250, 380 ] 476 0.032% 99.816% ( 380, 580 ] 845 0.057% 99.872% ( 580, 870 ] 1268 0.085% 99.957% ( 870, 1300 ] 1311 0.088% 100.045% ( 1300, 1900 ] 897 0.060% 100.105% ( 1900, 2900 ] 791 0.053% 100.158% ( 2900, 4400 ] 825 0.055% 100.213% ( 4400, 6600 ] 284 0.019% 100.232% ( 6600, 9900 ] 75 0.005% 100.237% ( 9900, 14000 ] 5 0.000% 100.237% ( 14000, 22000 ] 4 0.000% 100.238% ** Level 4 read latency histogram (micros): Count: 1143461 Average: 17.4586 StdDev: 180.24 Min: 0 Median: 1.9333 Max: 39406 Percentiles: P50: 1.93 P75: 2.94 P99: 215.91 P99.9: 1311.87 P99.99: 1739.77 ------------------------------------------------------ [ 0, 1 ] 22743 1.989% 1.989% ( 1, 2 ] 588235 51.443% 53.432% ########## ( 2, 3 ] 261007 22.826% 76.258% ##### ( 3, 4 ] 112585 9.846% 86.104% ## ( 4, 6 ] 110014 9.621% 95.726% ## ( 6, 10 ] 28378 2.482% 98.207% ( 10, 15 ] 4131 0.361% 98.569% ( 15, 22 ] 477 0.042% 98.610% ( 22, 34 ] 53 0.005% 98.615% ( 34, 51 ] 60 0.005% 98.620% ( 51, 76 ] 101 0.009% 98.629% ( 76, 110 ] 556 0.049% 98.678% ( 110, 170 ] 2515 0.220% 98.898% ( 170, 250 ] 2041 0.178% 99.076% ( 250, 380 ] 1845 0.161% 99.237% ( 380, 580 ] 2446 0.214% 99.451% ( 580, 870 ] 2749 0.240% 99.692% ( 870, 1300 ] 2353 0.206% 99.898% ( 1300, 1900 ] 1443 0.126% 100.024% ( 1900, 2900 ] 1080 0.094% 100.118% ( 2900, 4400 ] 892 0.078% 100.196% ( 4400, 6600 ] 252 0.022% 100.218% ( 6600, 9900 ] 50 0.004% 100.223% ( 9900, 14000 ] 3 0.000% 100.223% ( 33000, 50000 ] 1 0.000% 100.223% ** Level 5 read latency histogram (micros): Count: 69055 Average: 730.6342 StdDev: 1099.81 Min: 74 Median: 351.3337 Max: 55345 Percentiles: P50: 351.33 P75: 798.84 P99: 5624.64 P99.9: 9190.95 P99.99: 13338.19 ------------------------------------------------------ ( 51, 76 ] 2 0.003% 0.003% ( 76, 110 ] 3290 4.764% 4.767% # ( 110, 170 ] 14004 20.279% 25.047% #### ( 170, 250 ] 10782 15.614% 40.660% ### ( 250, 380 ] 8274 11.982% 52.642% ## ( 380, 580 ] 9249 13.394% 66.036% ### ( 580, 870 ] 8203 11.879% 77.915% ## ( 870, 1300 ] 5956 8.625% 86.540% ## ( 1300, 1900 ] 3128 4.530% 91.069% # ( 1900, 2900 ] 2409 3.489% 94.558% # ( 2900, 4400 ] 2503 3.625% 98.183% # ( 4400, 6600 ] 1014 1.468% 99.651% ( 6600, 9900 ] 219 0.317% 99.968% ( 9900, 14000 ] 18 0.026% 99.994% ( 14000, 22000 ] 2 0.003% 99.997% ( 22000, 33000 ] 1 0.001% 99.999% ( 33000, 50000 ] 1 0.001% 100.000% ( 50000, 75000 ] 2 0.003% 100.003% ** DB Stats ** Uptime(secs): 17.8 total, 3.5 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 166.06 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.375 H:M:S, 35.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 277.76 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 4 num-running-flushes: 1 2024/11/29-07:57:27 ... thread 0: (1000000,4000000) ops and (239742.4,183293.7) ops/second in (4.171143,21.822899) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/8 3.75 GB 195.1 3.3 0.0 3.3 7.1 3.7 0.0 1.9 93.6 199.2 36.36 24.20 66 0.551 3454K 3836 0.0 0.0 L1 3/3 301.87 MB 0.0 0.9 0.5 0.4 0.9 0.5 0.0 1.9 92.3 92.0 10.07 4.53 1 10.073 944K 2994 0.0 0.0 L3 16/0 971.88 MB 1.0 3.3 1.9 1.4 3.3 1.9 0.2 1.7 111.7 110.5 30.38 16.02 17 1.787 1551K 0 0.0 0.0 L4 173/0 9.67 GB 1.0 9.7 5.2 4.5 9.5 5.0 0.3 1.8 122.8 120.7 80.62 45.66 46 1.753 3264K 7649 0.0 0.0 L5 1160/0 95.41 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1370/11 110.08 GB 0.0 17.2 7.6 9.7 20.8 11.1 5.7 5.5 111.9 135.0 157.43 90.43 130 1.211 9214K 14K 0.0 0.0 Int 0/0 0.00 KB 0.0 17.2 7.6 9.7 20.8 11.1 5.7 5.5 111.9 135.0 157.43 90.43 130 1.211 9214K 14K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 17.2 7.6 9.7 17.0 7.3 0.0 0.0 128.0 126.5 137.65 79.51 76 1.811 9214K 14K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 194.2 19.78 10.92 54 0.366 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 22.0 total, 21.8 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 20.75 GB write, 968.19 MB/s write, 17.21 GB read, 802.83 MB/s read, 157.4 seconds Interval compaction: 20.75 GB write, 973.77 MB/s write, 17.21 GB read, 807.45 MB/s read, 157.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 4, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 4, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 45, pending-compaction-bytes-delays: 2, pending-compaction-bytes-stops: 0, total-delays: 6, total-stops: 45, interval: 51 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1186503 Average: 13.0895 StdDev: 121.06 Min: 0 Median: 1.6689 Max: 14579 Percentiles: P50: 1.67 P75: 2.41 P99: 225.92 P99.9: 1233.65 P99.99: 2502.97 ------------------------------------------------------ [ 0, 1 ] 143207 12.070% 12.070% ## ( 1, 2 ] 672851 56.709% 68.778% ########### ( 2, 3 ] 181385 15.287% 84.066% ### ( 3, 4 ] 73562 6.200% 90.266% # ( 4, 6 ] 64392 5.427% 95.693% # ( 6, 10 ] 13719 1.156% 96.849% ( 10, 15 ] 2237 0.189% 97.038% ( 15, 22 ] 244 0.021% 97.058% ( 22, 34 ] 43 0.004% 97.062% ( 34, 51 ] 26 0.002% 97.064% ( 51, 76 ] 46 0.004% 97.068% ( 76, 110 ] 4056 0.342% 97.410% ( 110, 170 ] 14317 1.207% 98.616% ( 170, 250 ] 6515 0.549% 99.165% ( 250, 380 ] 3020 0.255% 99.420% ( 380, 580 ] 2823 0.238% 99.658% ( 580, 870 ] 1918 0.162% 99.819% ( 870, 1300 ] 1131 0.095% 99.915% ( 1300, 1900 ] 619 0.052% 99.967% ( 1900, 2900 ] 455 0.038% 100.005% ( 2900, 4400 ] 391 0.033% 100.038% ( 4400, 6600 ] 117 0.010% 100.048% ( 6600, 9900 ] 19 0.002% 100.050% ( 9900, 14000 ] 3 0.000% 100.050% ( 14000, 22000 ] 1 0.000% 100.050% ** Level 1 read latency histogram (micros): Count: 285132 Average: 10.7332 StdDev: 136.71 Min: 1 Median: 1.9601 Max: 8338 Percentiles: P50: 1.96 P75: 2.94 P99: 14.00 P99.9: 1899.49 P99.99: 4236.13 ------------------------------------------------------ [ 0, 1 ] 8537 2.994% 2.994% # ( 1, 2 ] 139598 48.959% 51.953% ########## ( 2, 3 ] 69866 24.503% 76.456% ##### ( 3, 4 ] 27069 9.493% 85.950% ## ( 4, 6 ] 27888 9.781% 95.730% ## ( 6, 10 ] 8196 2.874% 98.605% # ( 10, 15 ] 1409 0.494% 99.099% ( 15, 22 ] 214 0.075% 99.174% ( 22, 34 ] 18 0.006% 99.180% ( 34, 51 ] 24 0.008% 99.189% ( 51, 76 ] 22 0.008% 99.197% ( 76, 110 ] 95 0.033% 99.230% ( 110, 170 ] 369 0.129% 99.359% ( 170, 250 ] 275 0.096% 99.456% ( 250, 380 ] 217 0.076% 99.532% ( 380, 580 ] 281 0.099% 99.630% ( 580, 870 ] 339 0.119% 99.749% ( 870, 1300 ] 275 0.096% 99.846% ( 1300, 1900 ] 155 0.054% 99.900% ( 1900, 2900 ] 130 0.046% 99.946% ( 2900, 4400 ] 142 0.050% 99.995% ( 4400, 6600 ] 41 0.014% 100.010% ( 6600, 9900 ] 12 0.004% 100.014% ** Level 2 read latency histogram (micros): Count: 251048 Average: 15.2220 StdDev: 211.78 Min: 1 Median: 1.9681 Max: 15830 Percentiles: P50: 1.97 P75: 3.01 P99: 12.66 P99.9: 2572.94 P99.99: 4040.40 ------------------------------------------------------ [ 0, 1 ] 7589 3.023% 3.023% # ( 1, 2 ] 121820 48.525% 51.548% ########## ( 2, 3 ] 58704 23.384% 74.931% ##### ( 3, 4 ] 25554 10.179% 85.110% ## ( 4, 6 ] 26392 10.513% 95.623% ## ( 6, 10 ] 7691 3.064% 98.686% # ( 10, 15 ] 1480 0.590% 99.276% ( 15, 22 ] 232 0.092% 99.368% ( 22, 34 ] 48 0.019% 99.387% ( 34, 51 ] 15 0.006% 99.393% ( 51, 76 ] 15 0.006% 99.399% ( 76, 110 ] 12 0.005% 99.404% ( 110, 170 ] 63 0.025% 99.429% ( 170, 250 ] 69 0.027% 99.457% ( 250, 380 ] 117 0.047% 99.503% ( 380, 580 ] 200 0.080% 99.583% ( 580, 870 ] 245 0.098% 99.681% ( 870, 1300 ] 224 0.089% 99.770% ( 1300, 1900 ] 189 0.075% 99.845% ( 1900, 2900 ] 205 0.082% 99.927% ( 2900, 4400 ] 209 0.083% 100.010% ( 4400, 6600 ] 125 0.050% 100.060% ( 6600, 9900 ] 28 0.011% 100.071% ( 9900, 14000 ] 2 0.001% 100.072% ( 14000, 22000 ] 5 0.002% 100.074% ** Level 3 read latency histogram (micros): Count: 1522078 Average: 10.3752 StdDev: 147.60 Min: 1 Median: 1.9105 Max: 19226 Percentiles: P50: 1.91 P75: 2.89 P99: 9.57 P99.9: 671.24 P99.99: 1029.75 ------------------------------------------------------ [ 0, 1 ] 39027 2.564% 2.564% # ( 1, 2 ] 792965 52.098% 54.662% ########## ( 2, 3 ] 349228 22.944% 77.606% ##### ( 3, 4 ] 147486 9.690% 87.296% ## ( 4, 6 ] 146031 9.594% 96.890% ## ( 6, 10 ] 36000 2.365% 99.255% ( 10, 15 ] 5523 0.363% 99.618% ( 15, 22 ] 726 0.048% 99.665% ( 22, 34 ] 73 0.005% 99.670% ( 34, 51 ] 70 0.005% 99.675% ( 51, 76 ] 96 0.006% 99.681% ( 76, 110 ] 224 0.015% 99.696% ( 110, 170 ] 789 0.052% 99.748% ( 170, 250 ] 480 0.032% 99.779% ( 250, 380 ] 542 0.036% 99.815% ( 380, 580 ] 891 0.059% 99.873% ( 580, 870 ] 1287 0.085% 99.958% ( 870, 1300 ] 1313 0.086% 100.044% ( 1300, 1900 ] 897 0.059% 100.103% ( 1900, 2900 ] 793 0.052% 100.155% ( 2900, 4400 ] 825 0.054% 100.209% ( 4400, 6600 ] 284 0.019% 100.228% ( 6600, 9900 ] 75 0.005% 100.233% ( 9900, 14000 ] 5 0.000% 100.233% ( 14000, 22000 ] 4 0.000% 100.234% ** Level 4 read latency histogram (micros): Count: 1183223 Average: 18.5160 StdDev: 179.81 Min: 0 Median: 1.9357 Max: 39406 Percentiles: P50: 1.94 P75: 2.96 P99: 263.04 P99.9: 1299.24 P99.99: 1738.90 ------------------------------------------------------ [ 0, 1 ] 23995 2.028% 2.028% ( 1, 2 ] 606607 51.267% 53.295% ########## ( 2, 3 ] 267370 22.597% 75.892% ##### ( 3, 4 ] 114640 9.689% 85.581% ## ( 4, 6 ] 111790 9.448% 95.029% ## ( 6, 10 ] 29021 2.453% 97.481% ( 10, 15 ] 4176 0.353% 97.834% ( 15, 22 ] 483 0.041% 97.875% ( 22, 34 ] 54 0.005% 97.880% ( 34, 51 ] 60 0.005% 97.885% ( 51, 76 ] 102 0.009% 97.893% ( 76, 110 ] 1912 0.162% 98.055% ( 110, 170 ] 7143 0.604% 98.659% ( 170, 250 ] 3784 0.320% 98.979% ( 250, 380 ] 2530 0.214% 99.192% ( 380, 580 ] 3052 0.258% 99.450% ( 580, 870 ] 2929 0.248% 99.698% ( 870, 1300 ] 2396 0.202% 99.900% ( 1300, 1900 ] 1450 0.123% 100.023% ( 1900, 2900 ] 1087 0.092% 100.115% ( 2900, 4400 ] 892 0.075% 100.190% ( 4400, 6600 ] 253 0.021% 100.212% ( 6600, 9900 ] 50 0.004% 100.216% ( 9900, 14000 ] 4 0.000% 100.216% ( 14000, 22000 ] 2 0.000% 100.216% ( 33000, 50000 ] 1 0.000% 100.216% ** Level 5 read latency histogram (micros): Count: 121379 Average: 500.8570 StdDev: 878.65 Min: 73 Median: 207.4458 Max: 55345 Percentiles: P50: 207.45 P75: 492.08 P99: 4489.32 P99.9: 8387.44 P99.99: 11812.35 ------------------------------------------------------ ( 51, 76 ] 5 0.004% 0.004% ( 76, 110 ] 10835 8.927% 8.931% ## ( 110, 170 ] 40094 33.032% 41.963% ####### ( 170, 250 ] 20844 17.173% 59.135% ### ( 250, 380 ] 12226 10.073% 69.208% ## ( 380, 580 ] 12547 10.337% 79.545% ## ( 580, 870 ] 9279 7.645% 87.190% ## ( 870, 1300 ] 6159 5.074% 92.264% # ( 1300, 1900 ] 3161 2.604% 94.868% # ( 1900, 2900 ] 2462 2.028% 96.896% ( 2900, 4400 ] 2513 2.070% 98.967% ( 4400, 6600 ] 1015 0.836% 99.803% ( 6600, 9900 ] 219 0.180% 99.984% ( 9900, 14000 ] 19 0.016% 99.999% ( 14000, 22000 ] 3 0.002% 100.002% ( 22000, 33000 ] 1 0.001% 100.002% ( 33000, 50000 ] 1 0.001% 100.003% ( 50000, 75000 ] 2 0.002% 100.005% ** DB Stats ** Uptime(secs): 22.0 total, 4.2 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 179.34 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.039 H:M:S, 32.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 235.95 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.664 H:M:S, 15.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/11/29-07:57:36 ... thread 0: (1000000,5000000) ops and (114578.2,163663.1) ops/second in (8.727660,30.550559) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/12 4.72 GB 269.6 5.6 0.0 5.6 10.3 4.7 0.0 2.2 118.8 219.1 48.24 35.52 86 0.561 5819K 15K 0.0 0.0 L1 3/3 301.87 MB 0.0 0.9 0.5 0.4 0.9 0.5 0.0 1.9 92.3 92.0 10.07 4.53 1 10.073 944K 2994 0.0 0.0 L3 16/0 971.88 MB 1.0 3.3 1.9 1.4 3.3 1.9 0.2 1.7 111.7 110.5 30.38 16.02 17 1.787 1551K 0 0.0 0.0 L4 173/0 9.67 GB 1.0 9.7 5.2 4.5 9.5 5.0 0.3 1.8 122.8 120.7 80.62 45.66 46 1.753 3264K 7649 0.0 0.0 L5 1160/0 95.41 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1371/15 111.05 GB 0.0 19.5 7.6 11.9 24.0 12.1 5.7 5.1 117.8 145.2 169.31 101.74 150 1.129 11M 25K 0.0 0.0 Int 0/0 0.00 KB 0.0 19.5 7.6 11.9 24.0 12.1 5.7 5.1 117.8 145.2 169.31 101.74 150 1.129 11M 25K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 19.5 7.6 11.9 19.3 7.3 0.0 0.0 136.3 134.8 146.40 88.15 80 1.830 11M 25K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 211.7 22.91 13.59 70 0.327 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 30.7 total, 30.6 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 24.00 GB write, 801.17 MB/s write, 19.48 GB read, 650.36 MB/s read, 169.3 seconds Interval compaction: 24.00 GB write, 804.46 MB/s write, 19.48 GB read, 653.04 MB/s read, 169.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 11, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 11, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 45, pending-compaction-bytes-delays: 2, pending-compaction-bytes-stops: 0, total-delays: 13, total-stops: 45, interval: 58 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1891668 Average: 14.8620 StdDev: 103.35 Min: 0 Median: 1.5683 Max: 14579 Percentiles: P50: 1.57 P75: 2.11 P99: 238.58 P99.9: 853.36 P99.99: 1601.26 ------------------------------------------------------ [ 0, 1 ] 355906 18.814% 18.814% #### ( 1, 2 ] 1037979 54.871% 73.685% ########### ( 2, 3 ] 221448 11.706% 85.392% ## ( 3, 4 ] 87034 4.601% 89.993% # ( 4, 6 ] 70995 3.753% 93.746% # ( 6, 10 ] 15130 0.800% 94.546% ( 10, 15 ] 2536 0.134% 94.680% ( 15, 22 ] 282 0.015% 94.695% ( 22, 34 ] 50 0.003% 94.697% ( 34, 51 ] 26 0.001% 94.699% ( 51, 76 ] 79 0.004% 94.703% ( 76, 110 ] 15582 0.824% 95.527% ( 110, 170 ] 50465 2.668% 98.194% # ( 170, 250 ] 17780 0.940% 99.134% ( 250, 380 ] 6496 0.343% 99.478% ( 380, 580 ] 5500 0.291% 99.768% ( 580, 870 ] 2643 0.140% 99.908% ( 870, 1300 ] 1233 0.065% 99.973% ( 1300, 1900 ] 633 0.033% 100.007% ( 1900, 2900 ] 495 0.026% 100.033% ( 2900, 4400 ] 403 0.021% 100.054% ( 4400, 6600 ] 118 0.006% 100.061% ( 6600, 9900 ] 19 0.001% 100.062% ( 9900, 14000 ] 3 0.000% 100.062% ( 14000, 22000 ] 1 0.000% 100.062% ** Level 1 read latency histogram (micros): Count: 285777 Average: 11.1019 StdDev: 136.85 Min: 1 Median: 1.9624 Max: 8338 Percentiles: P50: 1.96 P75: 2.95 P99: 89.78 P99.9: 1896.99 P99.99: 4235.45 ------------------------------------------------------ [ 0, 1 ] 8537 2.987% 2.987% # ( 1, 2 ] 139598 48.849% 51.836% ########## ( 2, 3 ] 69866 24.448% 76.284% ##### ( 3, 4 ] 27069 9.472% 85.756% ## ( 4, 6 ] 27888 9.759% 95.514% ## ( 6, 10 ] 8196 2.868% 98.382% # ( 10, 15 ] 1409 0.493% 98.875% ( 15, 22 ] 214 0.075% 98.950% ( 22, 34 ] 18 0.006% 98.957% ( 34, 51 ] 24 0.008% 98.965% ( 51, 76 ] 22 0.008% 98.973% ( 76, 110 ] 193 0.068% 99.040% ( 110, 170 ] 714 0.250% 99.290% ( 170, 250 ] 407 0.142% 99.432% ( 250, 380 ] 255 0.089% 99.522% ( 380, 580 ] 302 0.106% 99.627% ( 580, 870 ] 350 0.122% 99.750% ( 870, 1300 ] 275 0.096% 99.846% ( 1300, 1900 ] 155 0.054% 99.900% ( 1900, 2900 ] 130 0.045% 99.946% ( 2900, 4400 ] 142 0.050% 99.995% ( 4400, 6600 ] 41 0.014% 100.010% ( 6600, 9900 ] 12 0.004% 100.014% ** Level 2 read latency histogram (micros): Count: 251116 Average: 15.2648 StdDev: 211.77 Min: 1 Median: 1.9684 Max: 15830 Percentiles: P50: 1.97 P75: 3.01 P99: 12.89 P99.9: 2572.60 P99.99: 4040.35 ------------------------------------------------------ [ 0, 1 ] 7589 3.022% 3.022% # ( 1, 2 ] 121820 48.511% 51.534% ########## ( 2, 3 ] 58704 23.377% 74.911% ##### ( 3, 4 ] 25554 10.176% 85.087% ## ( 4, 6 ] 26392 10.510% 95.597% ## ( 6, 10 ] 7691 3.063% 98.660% # ( 10, 15 ] 1480 0.589% 99.249% ( 15, 22 ] 232 0.092% 99.341% ( 22, 34 ] 48 0.019% 99.360% ( 34, 51 ] 15 0.006% 99.366% ( 51, 76 ] 15 0.006% 99.372% ( 76, 110 ] 23 0.009% 99.382% ( 110, 170 ] 101 0.040% 99.422% ( 170, 250 ] 81 0.032% 99.454% ( 250, 380 ] 120 0.048% 99.502% ( 380, 580 ] 203 0.081% 99.583% ( 580, 870 ] 246 0.098% 99.681% ( 870, 1300 ] 224 0.089% 99.770% ( 1300, 1900 ] 189 0.075% 99.845% ( 1900, 2900 ] 205 0.082% 99.927% ( 2900, 4400 ] 209 0.083% 100.010% ( 4400, 6600 ] 125 0.050% 100.060% ( 6600, 9900 ] 28 0.011% 100.071% ( 9900, 14000 ] 2 0.001% 100.072% ( 14000, 22000 ] 5 0.002% 100.074% ** Level 3 read latency histogram (micros): Count: 1525113 Average: 10.6432 StdDev: 147.67 Min: 1 Median: 1.9124 Max: 19226 Percentiles: P50: 1.91 P75: 2.89 P99: 9.85 P99.9: 677.44 P99.99: 1030.06 ------------------------------------------------------ [ 0, 1 ] 39028 2.559% 2.559% # ( 1, 2 ] 793037 51.999% 54.558% ########## ( 2, 3 ] 349374 22.908% 77.466% ##### ( 3, 4 ] 147628 9.680% 87.145% ## ( 4, 6 ] 146169 9.584% 96.730% ## ( 6, 10 ] 36006 2.361% 99.090% ( 10, 15 ] 5527 0.362% 99.453% ( 15, 22 ] 726 0.048% 99.500% ( 22, 34 ] 73 0.005% 99.505% ( 34, 51 ] 70 0.005% 99.510% ( 51, 76 ] 96 0.006% 99.516% ( 76, 110 ] 655 0.043% 99.559% ( 110, 170 ] 2171 0.142% 99.701% ( 170, 250 ] 916 0.060% 99.762% ( 250, 380 ] 683 0.045% 99.806% ( 380, 580 ] 982 0.064% 99.871% ( 580, 870 ] 1330 0.087% 99.958% ( 870, 1300 ] 1315 0.086% 100.044% ( 1300, 1900 ] 897 0.059% 100.103% ( 1900, 2900 ] 793 0.052% 100.155% ( 2900, 4400 ] 825 0.054% 100.209% ( 4400, 6600 ] 284 0.019% 100.228% ( 6600, 9900 ] 75 0.005% 100.233% ( 9900, 14000 ] 5 0.000% 100.233% ( 14000, 22000 ] 4 0.000% 100.233% ** Level 4 read latency histogram (micros): Count: 1204857 Average: 21.2318 StdDev: 179.96 Min: 0 Median: 1.9535 Max: 39406 Percentiles: P50: 1.95 P75: 3.05 P99: 332.37 P99.9: 1297.90 P99.99: 1742.27 ------------------------------------------------------ [ 0, 1 ] 23995 1.992% 1.992% ( 1, 2 ] 606652 50.351% 52.342% ########## ( 2, 3 ] 267486 22.201% 74.543% #### ( 3, 4 ] 114752 9.524% 84.067% ## ( 4, 6 ] 111880 9.286% 93.353% ## ( 6, 10 ] 29031 2.409% 95.762% ( 10, 15 ] 4179 0.347% 96.109% ( 15, 22 ] 484 0.040% 96.149% ( 22, 34 ] 54 0.004% 96.154% ( 34, 51 ] 60 0.005% 96.159% ( 51, 76 ] 105 0.009% 96.167% ( 76, 110 ] 5511 0.457% 96.625% ( 110, 170 ] 18960 1.574% 98.198% ( 170, 250 ] 7365 0.611% 98.810% ( 250, 380 ] 3621 0.301% 99.110% ( 380, 580 ] 3945 0.327% 99.438% ( 580, 870 ] 3159 0.262% 99.700% ( 870, 1300 ] 2425 0.201% 99.901% ( 1300, 1900 ] 1455 0.121% 100.022% ( 1900, 2900 ] 1095 0.091% 100.113% ( 2900, 4400 ] 893 0.074% 100.187% ( 4400, 6600 ] 253 0.021% 100.208% ( 6600, 9900 ] 50 0.004% 100.212% ( 9900, 14000 ] 4 0.000% 100.212% ( 14000, 22000 ] 2 0.000% 100.212% ( 33000, 50000 ] 1 0.000% 100.212% ** Level 5 read latency histogram (micros): Count: 241872 Average: 337.4033 StdDev: 648.50 Min: 71 Median: 160.3861 Max: 55345 Percentiles: P50: 160.39 P75: 264.32 P99: 3706.48 P99.9: 6595.94 P99.99: 9851.97 ------------------------------------------------------ ( 51, 76 ] 18 0.007% 0.007% ( 76, 110 ] 31533 13.037% 13.045% ### ( 110, 170 ] 106440 44.007% 57.051% ######### ( 170, 250 ] 41371 17.105% 74.156% ### ( 250, 380 ] 18544 7.667% 81.823% ## ( 380, 580 ] 17545 7.254% 89.076% # ( 580, 870 ] 10622 4.392% 93.468% # ( 870, 1300 ] 6347 2.624% 96.092% # ( 1300, 1900 ] 3189 1.318% 97.411% ( 1900, 2900 ] 2492 1.030% 98.441% ( 2900, 4400 ] 2517 1.041% 99.482% ( 4400, 6600 ] 1015 0.420% 99.901% ( 6600, 9900 ] 219 0.091% 99.992% ( 9900, 14000 ] 19 0.008% 100.000% ( 14000, 22000 ] 3 0.001% 100.001% ( 22000, 33000 ] 1 0.000% 100.001% ( 33000, 50000 ] 1 0.000% 100.002% ( 50000, 75000 ] 2 0.001% 100.002% ** DB Stats ** Uptime(secs): 30.7 total, 8.7 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 160.40 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:12.181 H:M:S, 39.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 112.77 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:5.143 H:M:S, 58.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 263.380 micros/op 15169 ops/sec 60.186 seconds 912996 operations; 14.7 MB/s (229999 of 229999 found) Microseconds per read: Count: 912996 Average: 263.3826 StdDev: 817.05 Min: 2 Median: 179.9399 Max: 94375 Percentiles: P50: 179.94 P75: 240.46 P99: 1452.38 P99.9: 12831.83 P99.99: 32396.00 ------------------------------------------------------ ( 1, 2 ] 5 0.001% 0.001% ( 2, 3 ] 39 0.004% 0.005% ( 3, 4 ] 43 0.005% 0.010% ( 4, 6 ] 92 0.010% 0.020% ( 6, 10 ] 374 0.041% 0.061% ( 10, 15 ] 2789 0.305% 0.366% ( 15, 22 ] 2471 0.271% 0.637% ( 22, 34 ] 6392 0.700% 1.337% ( 34, 51 ] 1730 0.189% 1.526% ( 51, 76 ] 225 0.025% 1.551% ( 76, 110 ] 16079 1.761% 3.312% ( 110, 170 ] 388770 42.582% 45.894% ######### ( 170, 250 ] 301726 33.048% 78.942% ####### ( 250, 380 ] 109349 11.977% 90.919% ## ( 380, 580 ] 44809 4.908% 95.827% # ( 580, 870 ] 19222 2.105% 97.932% ( 870, 1300 ] 8806 0.965% 98.896% ( 1300, 1900 ] 3721 0.408% 99.304% ( 1900, 2900 ] 2496 0.273% 99.577% ( 2900, 4400 ] 1914 0.210% 99.787% ( 4400, 6600 ] 576 0.063% 99.850% ( 6600, 9900 ] 194 0.021% 99.871% ( 9900, 14000 ] 365 0.040% 99.911% ( 14000, 22000 ] 506 0.055% 99.967% ( 22000, 33000 ] 224 0.025% 99.991% ( 33000, 50000 ] 59 0.006% 99.998% ( 50000, 75000 ] 18 0.002% 100.000% ( 75000, 110000 ] 2 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 17633633 rocksdb.bloom.filter.full.positive COUNT : 1084209 rocksdb.bloom.filter.full.true.positive COUNT : 912558 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 438 rocksdb.memtable.miss COUNT : 912558 rocksdb.l0.hit COUNT : 42740 rocksdb.l1.hit COUNT : 2723 rocksdb.l2andup.hit COUNT : 867095 rocksdb.compaction.key.drop.new COUNT : 243937 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5573627 rocksdb.number.keys.read COUNT : 912996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5751983064 rocksdb.bytes.read COUNT : 912996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1661 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 39168350 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 23958723583 rocksdb.compact.write.bytes COUNT : 23707303706 rocksdb.flush.write.bytes COUNT : 5747116938 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 22 rocksdb.number.superversion_acquires COUNT : 1098 rocksdb.number.superversion_releases COUNT : 1058 rocksdb.number.superversion_cleanups COUNT : 272 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 217 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5700658176 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2110464 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 28032882432 rocksdb.non.last.level.read.count COUNT : 6629972 rocksdb.block.checksum.compute.count COUNT : 6900262 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 178.455109 P95 : 545.587416 P99 : 1450.451335 P100 : 94372.000000 COUNT : 912996 SUM : 239670896 rocksdb.db.write.micros P50 : 1.545722 P95 : 4.410520 P99 : 9.062604 P100 : 909311.000000 COUNT : 5573627 SUM : 53127869 rocksdb.compaction.times.micros P50 : 1500000.000000 P95 : 2890909.090909 P99 : 10073291.000000 P100 : 10073291.000000 COUNT : 84 SUM : 150225044 rocksdb.compaction.times.cpu_micros P50 : 949473.684211 P95 : 1866400.000000 P99 : 5576000.000000 P100 : 5860097.000000 COUNT : 84 SUM : 91884867 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 803.076923 P95 : 86666.666667 P99 : 272354.000000 P100 : 272354.000000 COUNT : 80 SUM : 979764 rocksdb.compaction.outfile.sync.micros P50 : 29333.333333 P95 : 269500.000000 P99 : 489250.000000 P100 : 903539.000000 COUNT : 270 SUM : 20088743 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 225.000000 P95 : 3740.000000 P99 : 6864.000000 P100 : 9565.000000 COUNT : 184 SUM : 173503 rocksdb.table.open.io.micros P50 : 9477.906977 P95 : 21703.355705 P99 : 31231.666667 P100 : 69744.000000 COUNT : 1661 SUM : 16606810 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.892845 P95 : 13.589981 P99 : 183.970304 P100 : 39411.000000 COUNT : 5803561 SUM : 94211598 rocksdb.read.block.get.micros P50 : 140.568223 P95 : 426.386918 P99 : 1297.336638 P100 : 55485.000000 COUNT : 1095386 SUM : 203726095 rocksdb.write.raw.block.micros P50 : 0.527637 P95 : 1.072408 P99 : 3.474579 P100 : 38033.000000 COUNT : 7135272 SUM : 21279301 rocksdb.numfiles.in.singlecompaction P50 : 3.108696 P95 : 5.545455 P99 : 9.800000 P100 : 11.000000 COUNT : 110 SUM : 388 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.566146 P95 : 1965.418278 P99 : 2715.363322 P100 : 909075.000000 COUNT : 184224 SUM : 39184395 rocksdb.sst.read.micros P50 : 1.966012 P95 : 165.212052 P99 : 494.806968 P100 : 55345.000000 COUNT : 6630533 SUM : 281275176 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 912996 SUM : 912996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5573627 SUM : 5751983064 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 226140.350877 P95 : 973333.333333 P99 : 1154666.666667 P100 : 1176650.000000 COUNT : 80 SUM : 24983101 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 293 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0