[2024-12-14 22:23:17.278341] Starting SPDK v25.01-pre git sha1 e01cb43b8 / DPDK 22.11.4 initialization... [2024-12-14 22:23:17.278615] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid196262 ] Set seed to 1734211397271623 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-14 22:23:17.391757] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-14 22:23:17.415009] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-14 22:23:17.415050] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 196262' to capture a snapshot of events at runtime. [2024-12-14 22:23:17.415061] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-12-14 22:23:17.415071] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-12-14 22:23:17.415079] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid196262 for offline analysis/debug. [2024-12-14 22:23:17.415636] reactor.c: 995:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sat Dec 14 22:23:21 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/14-22:23:29 ... thread 0: (1000000,1000000) ops and (128244.7,128244.7) ops/second in (7.797592,7.797592) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 9/7 818.85 MB 18.9 0.0 0.0 0.0 0.9 0.9 0.0 1.0 0.0 122.7 7.19 2.59 10 0.719 0 0 0.0 0.0 L1 4/4 263.43 MB 0.0 0.3 0.2 0.1 0.3 0.2 0.0 1.5 106.2 106.1 2.48 1.23 1 2.484 267K 289 0.0 0.0 L2 2/0 221.38 MB 2.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 45/10 3.00 GB 2.1 2.9 1.8 1.1 2.9 1.8 0.0 1.6 97.9 97.1 30.61 14.21 17 1.800 1884K 809 0.0 0.0 L4 194/16 12.32 GB 1.2 3.4 2.7 0.7 3.4 2.6 0.1 1.3 109.6 108.9 31.82 16.22 24 1.326 1097K 0 0.0 0.0 L5 1080/0 88.83 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1334/37 105.43 GB 0.0 6.6 4.6 1.9 7.4 5.5 0.1 8.6 93.6 105.1 72.09 34.26 52 1.386 3249K 1098 0.0 0.0 Int 0/0 0.00 KB 0.0 6.6 4.6 1.9 7.4 5.5 0.1 8.6 93.6 105.1 72.09 34.26 52 1.386 3249K 1098 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.6 4.6 1.9 6.5 4.6 0.0 0.0 104.0 103.2 64.91 31.67 42 1.545 3249K 1098 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 122.7 7.19 2.59 10 0.719 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 8.0 total, 7.8 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.40 GB write, 944.51 MB/s write, 6.59 GB read, 840.75 MB/s read, 72.1 seconds Interval compaction: 7.40 GB write, 971.90 MB/s write, 6.59 GB read, 865.13 MB/s read, 72.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 37, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 37, interval: 37 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 198861 Average: 9.0471 StdDev: 119.16 Min: 1 Median: 1.9103 Max: 7837 Percentiles: P50: 1.91 P75: 2.91 P99: 9.77 P99.9: 1631.04 P99.99: 3544.89 ------------------------------------------------------ [ 0, 1 ] 6978 3.509% 3.509% # ( 1, 2 ] 101566 51.074% 54.583% ########## ( 2, 3 ] 44493 22.374% 76.957% #### ( 3, 4 ] 20156 10.136% 87.092% ## ( 4, 6 ] 19434 9.773% 96.865% ## ( 6, 10 ] 4509 2.267% 99.133% ( 10, 15 ] 778 0.391% 99.524% ( 15, 22 ] 83 0.042% 99.566% ( 22, 34 ] 15 0.008% 99.573% ( 34, 51 ] 3 0.002% 99.575% ( 51, 76 ] 15 0.008% 99.582% ( 110, 170 ] 11 0.006% 99.588% ( 170, 250 ] 24 0.012% 99.600% ( 250, 380 ] 41 0.021% 99.620% ( 380, 580 ] 99 0.050% 99.670% ( 580, 870 ] 183 0.092% 99.762% ( 870, 1300 ] 198 0.100% 99.862% ( 1300, 1900 ] 138 0.069% 99.931% ( 1900, 2900 ] 81 0.041% 99.972% ( 2900, 4400 ] 84 0.042% 100.014% ( 4400, 6600 ] 20 0.010% 100.024% ( 6600, 9900 ] 1 0.001% 100.025% ** Level 1 read latency histogram (micros): Count: 87910 Average: 11.4451 StdDev: 143.30 Min: 1 Median: 1.9735 Max: 7942 Percentiles: P50: 1.97 P75: 2.98 P99: 11.82 P99.9: 2433.83 P99.99: 5641.80 ------------------------------------------------------ [ 0, 1 ] 2068 2.352% 2.352% ( 1, 2 ] 43026 48.943% 51.296% ########## ( 2, 3 ] 21281 24.208% 75.503% ##### ( 3, 4 ] 9189 10.453% 85.956% ## ( 4, 6 ] 8844 10.060% 96.016% ## ( 6, 10 ] 2462 2.801% 98.817% # ( 10, 15 ] 443 0.504% 99.321% ( 15, 22 ] 50 0.057% 99.378% ( 22, 34 ] 6 0.007% 99.385% ( 34, 51 ] 2 0.002% 99.387% ( 51, 76 ] 4 0.005% 99.391% ( 76, 110 ] 1 0.001% 99.393% ( 110, 170 ] 4 0.005% 99.397% ( 170, 250 ] 11 0.013% 99.410% ( 250, 380 ] 35 0.040% 99.449% ( 380, 580 ] 61 0.069% 99.519% ( 580, 870 ] 119 0.135% 99.654% ( 870, 1300 ] 123 0.140% 99.794% ( 1300, 1900 ] 68 0.077% 99.871% ( 1900, 2900 ] 47 0.053% 99.925% ( 2900, 4400 ] 51 0.058% 99.983% ( 4400, 6600 ] 11 0.013% 99.995% ( 6600, 9900 ] 4 0.005% 100.000% ** Level 2 read latency histogram (micros): Count: 353646 Average: 13.2824 StdDev: 157.43 Min: 1 Median: 2.0209 Max: 8025 Percentiles: P50: 2.02 P75: 3.07 P99: 12.88 P99.9: 1594.68 P99.99: 2341.14 ------------------------------------------------------ [ 0, 1 ] 8129 2.299% 2.299% ( 1, 2 ] 166865 47.184% 49.483% ######### ( 2, 3 ] 87556 24.758% 74.241% ##### ( 3, 4 ] 37927 10.725% 84.965% ## ( 4, 6 ] 37155 10.506% 95.472% ## ( 6, 10 ] 11201 3.167% 98.639% # ( 10, 15 ] 2218 0.627% 99.266% ( 15, 22 ] 354 0.100% 99.366% ( 22, 34 ] 67 0.019% 99.385% ( 34, 51 ] 28 0.008% 99.393% ( 51, 76 ] 22 0.006% 99.399% ( 76, 110 ] 14 0.004% 99.403% ( 110, 170 ] 46 0.013% 99.416% ( 170, 250 ] 95 0.027% 99.443% ( 250, 380 ] 153 0.043% 99.486% ( 380, 580 ] 356 0.101% 99.587% ( 580, 870 ] 456 0.129% 99.716% ( 870, 1300 ] 476 0.135% 99.851% ( 1300, 1900 ] 355 0.100% 99.951% ( 1900, 2900 ] 312 0.088% 100.039% ( 2900, 4400 ] 230 0.065% 100.104% ( 4400, 6600 ] 78 0.022% 100.126% ( 6600, 9900 ] 6 0.002% 100.128% ** Level 3 read latency histogram (micros): Count: 955190 Average: 13.1935 StdDev: 162.37 Min: 1 Median: 2.0461 Max: 9420 Percentiles: P50: 2.05 P75: 3.02 P99: 10.45 P99.9: 1167.14 P99.99: 1618.87 ------------------------------------------------------ [ 0, 1 ] 14122 1.478% 1.478% ( 1, 2 ] 452024 47.323% 48.801% ######### ( 2, 3 ] 248183 25.983% 74.784% ##### ( 3, 4 ] 104247 10.914% 85.698% ## ( 4, 6 ] 99846 10.453% 96.151% ## ( 6, 10 ] 26764 2.802% 98.953% # ( 10, 15 ] 5035 0.527% 99.480% ( 15, 22 ] 670 0.070% 99.550% ( 22, 34 ] 71 0.007% 99.557% ( 34, 51 ] 53 0.006% 99.563% ( 51, 76 ] 82 0.009% 99.571% ( 76, 110 ] 21 0.002% 99.574% ( 110, 170 ] 54 0.006% 99.579% ( 170, 250 ] 129 0.014% 99.593% ( 250, 380 ] 313 0.033% 99.626% ( 380, 580 ] 680 0.071% 99.697% ( 580, 870 ] 1059 0.111% 99.808% ( 870, 1300 ] 1279 0.134% 99.942% ( 1300, 1900 ] 874 0.092% 100.033% ( 1900, 2900 ] 810 0.085% 100.118% ( 2900, 4400 ] 749 0.078% 100.196% ( 4400, 6600 ] 204 0.021% 100.218% ( 6600, 9900 ] 30 0.003% 100.221% ** Level 4 read latency histogram (micros): Count: 246838 Average: 31.3035 StdDev: 264.86 Min: 1 Median: 2.0857 Max: 10148 Percentiles: P50: 2.09 P75: 3.13 P99: 933.14 P99.9: 3479.42 P99.99: 4128.99 ------------------------------------------------------ [ 0, 1 ] 3705 1.501% 1.501% ( 1, 2 ] 114282 46.298% 47.799% ######### ( 2, 3 ] 63409 25.689% 73.488% ##### ( 3, 4 ] 28458 11.529% 85.017% ## ( 4, 6 ] 25035 10.142% 95.159% ## ( 6, 10 ] 6015 2.437% 97.596% ( 10, 15 ] 1231 0.499% 98.095% ( 15, 22 ] 147 0.060% 98.154% ( 22, 34 ] 16 0.006% 98.161% ( 34, 51 ] 17 0.007% 98.168% ( 51, 76 ] 26 0.011% 98.178% ( 76, 110 ] 8 0.003% 98.181% ( 110, 170 ] 45 0.018% 98.200% ( 170, 250 ] 131 0.053% 98.253% ( 250, 380 ] 216 0.088% 98.340% ( 380, 580 ] 552 0.224% 98.564% ( 580, 870 ] 938 0.380% 98.944% ( 870, 1300 ] 944 0.382% 99.326% ( 1300, 1900 ] 717 0.290% 99.617% ( 1900, 2900 ] 501 0.203% 99.820% ( 2900, 4400 ] 513 0.208% 100.028% ( 4400, 6600 ] 136 0.055% 100.083% ( 6600, 9900 ] 13 0.005% 100.088% ( 9900, 14000 ] 2 0.001% 100.089% ** Level 5 read latency histogram (micros): Count: 13197 Average: 1719.7230 StdDev: 1407.01 Min: 89 Median: 1202.9433 Max: 12413 Percentiles: P50: 1202.94 P75: 2451.72 P99: 6437.49 P99.9: 9583.33 P99.99: 12413.00 ------------------------------------------------------ ( 76, 110 ] 10 0.076% 0.076% ( 110, 170 ] 44 0.333% 0.409% ( 170, 250 ] 171 1.296% 1.705% ( 250, 380 ] 574 4.349% 6.054% # ( 380, 580 ] 1426 10.805% 16.860% ## ( 580, 870 ] 2365 17.921% 34.781% #### ( 870, 1300 ] 2594 19.656% 54.437% #### ( 1300, 1900 ] 1815 13.753% 68.190% ### ( 1900, 2900 ] 1629 12.344% 80.533% ## ( 2900, 4400 ] 1798 13.624% 94.158% ### ( 4400, 6600 ] 690 5.228% 99.386% # ( 6600, 9900 ] 75 0.568% 99.955% ( 9900, 14000 ] 6 0.045% 100.000% ** DB Stats ** Uptime(secs): 8.0 total, 7.8 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 122.62 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.874 H:M:S, 48.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 126.17 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.874 H:M:S, 49.7 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/12/14-22:23:34 ... thread 0: (1000000,2000000) ops and (187737.0,152390.3) ops/second in (5.326600,13.124192) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 13/8 1.48 GB 69.2 0.7 0.0 0.7 2.5 1.8 0.0 1.4 43.0 160.2 16.11 8.37 25 0.645 703K 878 0.0 0.0 L1 5/5 526.75 MB 0.0 0.8 0.5 0.3 0.8 0.5 0.0 1.7 127.2 126.9 6.62 3.88 2 3.311 855K 1489 0.0 0.0 L2 1/0 50.33 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 17/0 948.15 MB 0.9 3.3 2.0 1.2 3.2 2.0 0.0 1.6 100.4 99.6 33.35 15.89 19 1.755 2051K 809 0.0 0.0 L4 169/0 9.58 GB 1.0 7.0 4.6 2.4 6.9 4.5 0.4 1.5 116.7 115.5 61.35 33.32 47 1.305 2795K 2261 0.0 0.0 L5 1140/0 93.81 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1345/13 106.36 GB 0.0 11.8 7.1 4.7 13.5 8.9 5.5 7.3 102.6 117.8 117.44 61.46 93 1.263 6405K 5437 0.0 0.0 Int 0/0 0.00 KB 0.0 11.8 7.1 4.7 13.5 8.9 5.5 7.3 102.6 117.8 117.44 61.46 93 1.263 6405K 5437 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 11.8 7.1 4.7 11.7 7.0 0.0 0.0 114.7 113.7 104.98 55.87 70 1.500 6405K 5437 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 151.7 12.45 5.59 23 0.542 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 13.4 total, 13.1 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 13.51 GB write, 1035.78 MB/s write, 11.76 GB read, 901.98 MB/s read, 117.4 seconds Interval compaction: 13.51 GB write, 1053.62 MB/s write, 11.76 GB read, 917.53 MB/s read, 117.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 65, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 65, interval: 65 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 412229 Average: 11.0762 StdDev: 115.23 Min: 1 Median: 1.7865 Max: 7837 Percentiles: P50: 1.79 P75: 2.69 P99: 185.89 P99.9: 1406.90 P99.99: 3477.77 ------------------------------------------------------ [ 0, 1 ] 32679 7.927% 7.927% ## ( 1, 2 ] 220516 53.494% 61.421% ########### ( 2, 3 ] 80982 19.645% 81.066% #### ( 3, 4 ] 32895 7.980% 89.046% ## ( 4, 6 ] 29968 7.270% 96.315% # ( 6, 10 ] 7196 1.746% 98.061% ( 10, 15 ] 1175 0.285% 98.346% ( 15, 22 ] 129 0.031% 98.377% ( 22, 34 ] 17 0.004% 98.381% ( 34, 51 ] 3 0.001% 98.382% ( 51, 76 ] 19 0.005% 98.387% ( 76, 110 ] 378 0.092% 98.479% ( 110, 170 ] 1899 0.461% 98.939% ( 170, 250 ] 1263 0.306% 99.246% ( 250, 380 ] 795 0.193% 99.438% ( 380, 580 ] 797 0.193% 99.632% ( 580, 870 ] 635 0.154% 99.786% ( 870, 1300 ] 434 0.105% 99.891% ( 1300, 1900 ] 212 0.051% 99.943% ( 1900, 2900 ] 139 0.034% 99.976% ( 2900, 4400 ] 150 0.036% 100.013% ( 4400, 6600 ] 35 0.008% 100.021% ( 6600, 9900 ] 4 0.001% 100.022% ** Level 1 read latency histogram (micros): Count: 154607 Average: 11.3917 StdDev: 144.96 Min: 1 Median: 2.1619 Max: 24261 Percentiles: P50: 2.16 P75: 3.43 P99: 14.92 P99.9: 2139.42 P99.99: 5113.32 ------------------------------------------------------ [ 0, 1 ] 3078 1.991% 1.991% ( 1, 2 ] 68247 44.142% 46.133% ######### ( 2, 3 ] 36933 23.888% 70.021% ##### ( 3, 4 ] 17724 11.464% 81.485% ## ( 4, 6 ] 20352 13.164% 94.649% ### ( 6, 10 ] 5769 3.731% 98.380% # ( 10, 15 ] 973 0.629% 99.010% ( 15, 22 ] 260 0.168% 99.178% ( 22, 34 ] 17 0.011% 99.189% ( 34, 51 ] 4 0.003% 99.191% ( 51, 76 ] 7 0.005% 99.196% ( 76, 110 ] 37 0.024% 99.220% ( 110, 170 ] 150 0.097% 99.317% ( 170, 250 ] 96 0.062% 99.379% ( 250, 380 ] 112 0.072% 99.452% ( 380, 580 ] 152 0.098% 99.550% ( 580, 870 ] 217 0.140% 99.690% ( 870, 1300 ] 196 0.127% 99.817% ( 1300, 1900 ] 109 0.071% 99.887% ( 1900, 2900 ] 81 0.052% 99.940% ( 2900, 4400 ] 73 0.047% 99.987% ( 4400, 6600 ] 14 0.009% 99.996% ( 6600, 9900 ] 5 0.003% 99.999% ( 22000, 33000 ] 1 0.001% 100.000% ** Level 2 read latency histogram (micros): Count: 409886 Average: 13.0425 StdDev: 156.65 Min: 1 Median: 2.0230 Max: 8025 Percentiles: P50: 2.02 P75: 3.06 P99: 12.60 P99.9: 1681.30 P99.99: 2542.08 ------------------------------------------------------ [ 0, 1 ] 8909 2.174% 2.174% ( 1, 2 ] 193675 47.251% 49.424% ######### ( 2, 3 ] 102422 24.988% 74.412% ##### ( 3, 4 ] 43667 10.653% 85.066% ## ( 4, 6 ] 43052 10.503% 95.569% ## ( 6, 10 ] 12781 3.118% 98.687% # ( 10, 15 ] 2463 0.601% 99.288% ( 15, 22 ] 375 0.091% 99.380% ( 22, 34 ] 68 0.017% 99.396% ( 34, 51 ] 29 0.007% 99.403% ( 51, 76 ] 26 0.006% 99.410% ( 76, 110 ] 14 0.003% 99.413% ( 110, 170 ] 48 0.012% 99.425% ( 170, 250 ] 103 0.025% 99.450% ( 250, 380 ] 173 0.042% 99.492% ( 380, 580 ] 386 0.094% 99.586% ( 580, 870 ] 513 0.125% 99.712% ( 870, 1300 ] 523 0.128% 99.839% ( 1300, 1900 ] 392 0.096% 99.935% ( 1900, 2900 ] 352 0.086% 100.021% ( 2900, 4400 ] 270 0.066% 100.087% ( 4400, 6600 ] 90 0.022% 100.109% ( 6600, 9900 ] 8 0.002% 100.111% ** Level 3 read latency histogram (micros): Count: 1404834 Average: 12.3232 StdDev: 156.53 Min: 1 Median: 2.0130 Max: 19021 Percentiles: P50: 2.01 P75: 2.98 P99: 10.16 P99.9: 1113.09 P99.99: 1581.49 ------------------------------------------------------ [ 0, 1 ] 21983 1.565% 1.565% ( 1, 2 ] 675726 48.100% 49.665% ########## ( 2, 3 ] 362678 25.816% 75.481% ##### ( 3, 4 ] 148317 10.558% 86.039% ## ( 4, 6 ] 142709 10.158% 96.197% ## ( 6, 10 ] 39141 2.786% 98.984% # ( 10, 15 ] 7045 0.501% 99.485% ( 15, 22 ] 927 0.066% 99.551% ( 22, 34 ] 88 0.006% 99.557% ( 34, 51 ] 65 0.005% 99.562% ( 51, 76 ] 111 0.008% 99.570% ( 76, 110 ] 60 0.004% 99.574% ( 110, 170 ] 208 0.015% 99.589% ( 170, 250 ] 263 0.019% 99.608% ( 250, 380 ] 529 0.038% 99.645% ( 380, 580 ] 1094 0.078% 99.723% ( 580, 870 ] 1523 0.108% 99.832% ( 870, 1300 ] 1702 0.121% 99.953% ( 1300, 1900 ] 1118 0.080% 100.032% ( 1900, 2900 ] 1031 0.073% 100.106% ( 2900, 4400 ] 1013 0.072% 100.178% ( 4400, 6600 ] 271 0.019% 100.197% ( 6600, 9900 ] 38 0.003% 100.200% ( 9900, 14000 ] 2 0.000% 100.200% ( 14000, 22000 ] 3 0.000% 100.200% ** Level 4 read latency histogram (micros): Count: 627446 Average: 22.2546 StdDev: 213.32 Min: 1 Median: 2.0016 Max: 20290 Percentiles: P50: 2.00 P75: 3.00 P99: 495.84 P99.9: 2576.04 P99.99: 3460.95 ------------------------------------------------------ [ 0, 1 ] 10869 1.732% 1.732% ( 1, 2 ] 302596 48.227% 49.959% ########## ( 2, 3 ] 157335 25.075% 75.034% ##### ( 3, 4 ] 65702 10.471% 85.506% ## ( 4, 6 ] 60715 9.677% 95.182% ## ( 6, 10 ] 16471 2.625% 97.807% # ( 10, 15 ] 2850 0.454% 98.262% ( 15, 22 ] 343 0.055% 98.316% ( 22, 34 ] 27 0.004% 98.320% ( 34, 51 ] 33 0.005% 98.326% ( 51, 76 ] 42 0.007% 98.332% ( 76, 110 ] 280 0.045% 98.377% ( 110, 170 ] 1208 0.193% 98.570% ( 170, 250 ] 995 0.159% 98.728% ( 250, 380 ] 890 0.142% 98.870% ( 380, 580 ] 1408 0.224% 99.094% ( 580, 870 ] 1814 0.289% 99.384% ( 870, 1300 ] 1646 0.262% 99.646% ( 1300, 1900 ] 1074 0.171% 99.817% ( 1900, 2900 ] 770 0.123% 99.940% ( 2900, 4400 ] 843 0.134% 100.074% ( 4400, 6600 ] 197 0.031% 100.106% ( 6600, 9900 ] 21 0.003% 100.109% ( 9900, 14000 ] 3 0.000% 100.109% ( 14000, 22000 ] 3 0.000% 100.110% ** Level 5 read latency histogram (micros): Count: 41635 Average: 920.8652 StdDev: 1161.19 Min: 77 Median: 490.7825 Max: 24442 Percentiles: P50: 490.78 P75: 1124.09 P99: 5762.87 P99.9: 8703.38 P99.99: 12732.81 ------------------------------------------------------ ( 76, 110 ] 1604 3.853% 3.853% # ( 110, 170 ] 7501 18.016% 21.869% #### ( 170, 250 ] 5135 12.333% 34.202% ## ( 250, 380 ] 3824 9.185% 43.387% ## ( 380, 580 ] 4971 11.939% 55.326% ## ( 580, 870 ] 5424 13.028% 68.354% ### ( 870, 1300 ] 4683 11.248% 79.601% ## ( 1300, 1900 ] 2836 6.812% 86.413% # ( 1900, 2900 ] 2267 5.445% 91.858% # ( 2900, 4400 ] 2457 5.901% 97.759% # ( 4400, 6600 ] 834 2.003% 99.762% ( 6600, 9900 ] 90 0.216% 99.978% ( 9900, 14000 ] 7 0.017% 99.995% ( 14000, 22000 ] 1 0.002% 99.998% ( 22000, 33000 ] 2 0.005% 100.002% ** DB Stats ** Uptime(secs): 13.4 total, 5.3 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 147.42 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.449 H:M:S, 40.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 184.77 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.575 H:M:S, 29.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/14-22:23:38 ... thread 0: (1000000,3000000) ops and (280754.9,179791.2) ops/second in (3.561826,16.686018) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/8 2.40 GB 163.6 1.7 0.0 1.7 4.4 2.8 0.0 1.6 73.3 195.3 23.21 14.98 44 0.527 1727K 1855 0.0 0.0 L1 5/5 526.75 MB 0.0 0.8 0.5 0.3 0.8 0.5 0.0 1.7 127.2 126.9 6.62 3.88 2 3.311 855K 1489 0.0 0.0 L2 1/0 50.33 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 17/0 948.15 MB 0.9 3.3 2.0 1.2 3.2 2.0 0.0 1.6 100.4 99.6 33.35 15.89 19 1.755 2051K 809 0.0 0.0 L4 169/0 9.58 GB 1.0 7.0 4.6 2.4 6.9 4.5 0.4 1.5 116.7 115.5 61.35 33.32 47 1.305 2795K 2261 0.0 0.0 L5 1140/0 93.81 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1348/13 107.28 GB 0.0 12.7 7.1 5.6 15.4 9.8 5.5 5.6 104.8 126.7 124.53 68.06 112 1.112 7429K 6414 0.0 0.0 Int 0/0 0.00 KB 0.0 12.7 7.1 5.6 15.4 9.8 5.5 5.6 104.8 126.7 124.53 68.06 112 1.112 7429K 6414 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.7 7.1 5.6 12.6 7.0 0.0 0.0 119.9 118.9 108.87 59.65 74 1.471 7429K 6414 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 181.0 15.67 8.41 38 0.412 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 16.9 total, 16.7 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 15.41 GB write, 933.07 MB/s write, 12.75 GB read, 771.64 MB/s read, 124.5 seconds Interval compaction: 15.41 GB write, 945.71 MB/s write, 12.75 GB read, 782.10 MB/s read, 124.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 65, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 65, interval: 65 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 694383 Average: 11.3055 StdDev: 96.33 Min: 1 Median: 1.6605 Max: 7837 Percentiles: P50: 1.66 P75: 2.44 P99: 224.12 P99.9: 946.76 P99.99: 1885.03 ------------------------------------------------------ [ 0, 1 ] 101508 14.618% 14.618% ### ( 1, 2 ] 371988 53.571% 68.189% ########### ( 2, 3 ] 107894 15.538% 83.728% ### ( 3, 4 ] 45853 6.603% 90.331% # ( 4, 6 ] 37623 5.418% 95.749% # ( 6, 10 ] 8773 1.263% 97.013% ( 10, 15 ] 1446 0.208% 97.221% ( 15, 22 ] 243 0.035% 97.256% ( 22, 34 ] 48 0.007% 97.263% ( 34, 51 ] 7 0.001% 97.264% ( 51, 76 ] 20 0.003% 97.267% ( 76, 110 ] 1650 0.238% 97.504% ( 110, 170 ] 7496 1.080% 98.584% ( 170, 250 ] 4272 0.615% 99.199% ( 250, 380 ] 2044 0.294% 99.493% ( 380, 580 ] 1738 0.250% 99.744% ( 580, 870 ] 996 0.143% 99.887% ( 870, 1300 ] 502 0.072% 99.959% ( 1300, 1900 ] 218 0.031% 99.991% ( 1900, 2900 ] 151 0.022% 100.013% ( 2900, 4400 ] 156 0.022% 100.035% ( 4400, 6600 ] 36 0.005% 100.040% ( 6600, 9900 ] 4 0.001% 100.041% ** Level 1 read latency histogram (micros): Count: 174341 Average: 11.7088 StdDev: 137.79 Min: 1 Median: 2.3731 Max: 24261 Percentiles: P50: 2.37 P75: 3.83 P99: 135.76 P99.9: 1908.04 P99.99: 4803.21 ------------------------------------------------------ [ 0, 1 ] 3162 1.814% 1.814% ( 1, 2 ] 69413 39.815% 41.628% ######## ( 2, 3 ] 39115 22.436% 64.064% #### ( 3, 4 ] 22912 13.142% 77.206% ### ( 4, 6 ] 28343 16.257% 93.463% ### ( 6, 10 ] 7402 4.246% 97.709% # ( 10, 15 ] 1314 0.754% 98.463% ( 15, 22 ] 492 0.282% 98.745% ( 22, 34 ] 47 0.027% 98.772% ( 34, 51 ] 5 0.003% 98.775% ( 51, 76 ] 7 0.004% 98.779% ( 76, 110 ] 137 0.079% 98.857% ( 110, 170 ] 579 0.332% 99.190% ( 170, 250 ] 274 0.157% 99.347% ( 250, 380 ] 200 0.115% 99.461% ( 380, 580 ] 217 0.124% 99.586% ( 580, 870 ] 235 0.135% 99.721% ( 870, 1300 ] 202 0.116% 99.837% ( 1300, 1900 ] 110 0.063% 99.900% ( 1900, 2900 ] 82 0.047% 99.947% ( 2900, 4400 ] 73 0.042% 99.989% ( 4400, 6600 ] 14 0.008% 99.997% ( 6600, 9900 ] 5 0.003% 99.999% ( 22000, 33000 ] 1 0.001% 100.000% ** Level 2 read latency histogram (micros): Count: 409886 Average: 13.0425 StdDev: 156.65 Min: 1 Median: 2.0230 Max: 8025 Percentiles: P50: 2.02 P75: 3.06 P99: 12.60 P99.9: 1681.30 P99.99: 2542.08 ------------------------------------------------------ [ 0, 1 ] 8909 2.174% 2.174% ( 1, 2 ] 193675 47.251% 49.424% ######### ( 2, 3 ] 102422 24.988% 74.412% ##### ( 3, 4 ] 43667 10.653% 85.066% ## ( 4, 6 ] 43052 10.503% 95.569% ## ( 6, 10 ] 12781 3.118% 98.687% # ( 10, 15 ] 2463 0.601% 99.288% ( 15, 22 ] 375 0.091% 99.380% ( 22, 34 ] 68 0.017% 99.396% ( 34, 51 ] 29 0.007% 99.403% ( 51, 76 ] 26 0.006% 99.410% ( 76, 110 ] 14 0.003% 99.413% ( 110, 170 ] 48 0.012% 99.425% ( 170, 250 ] 103 0.025% 99.450% ( 250, 380 ] 173 0.042% 99.492% ( 380, 580 ] 386 0.094% 99.586% ( 580, 870 ] 513 0.125% 99.712% ( 870, 1300 ] 523 0.128% 99.839% ( 1300, 1900 ] 392 0.096% 99.935% ( 1900, 2900 ] 352 0.086% 100.021% ( 2900, 4400 ] 270 0.066% 100.087% ( 4400, 6600 ] 90 0.022% 100.109% ( 6600, 9900 ] 8 0.002% 100.111% ** Level 3 read latency histogram (micros): Count: 1405738 Average: 12.4341 StdDev: 156.58 Min: 1 Median: 2.0142 Max: 19021 Percentiles: P50: 2.01 P75: 2.98 P99: 10.76 P99.9: 1113.30 P99.99: 1581.45 ------------------------------------------------------ [ 0, 1 ] 21983 1.564% 1.564% ( 1, 2 ] 675728 48.069% 49.633% ########## ( 2, 3 ] 362688 25.801% 75.434% ##### ( 3, 4 ] 148332 10.552% 85.986% ## ( 4, 6 ] 142738 10.154% 96.139% ## ( 6, 10 ] 39146 2.785% 98.924% # ( 10, 15 ] 7048 0.501% 99.426% ( 15, 22 ] 927 0.066% 99.492% ( 22, 34 ] 88 0.006% 99.498% ( 34, 51 ] 65 0.005% 99.502% ( 51, 76 ] 111 0.008% 99.510% ( 76, 110 ] 155 0.011% 99.521% ( 110, 170 ] 621 0.044% 99.565% ( 170, 250 ] 439 0.031% 99.597% ( 250, 380 ] 620 0.044% 99.641% ( 380, 580 ] 1139 0.081% 99.722% ( 580, 870 ] 1539 0.109% 99.831% ( 870, 1300 ] 1706 0.121% 99.953% ( 1300, 1900 ] 1118 0.080% 100.032% ( 1900, 2900 ] 1031 0.073% 100.106% ( 2900, 4400 ] 1013 0.072% 100.178% ( 4400, 6600 ] 271 0.019% 100.197% ( 6600, 9900 ] 38 0.003% 100.200% ( 9900, 14000 ] 2 0.000% 100.200% ( 14000, 22000 ] 3 0.000% 100.200% ** Level 4 read latency histogram (micros): Count: 635131 Average: 24.2392 StdDev: 213.36 Min: 1 Median: 2.0259 Max: 20290 Percentiles: P50: 2.03 P75: 3.08 P99: 530.84 P99.9: 2566.50 P99.99: 3459.59 ------------------------------------------------------ [ 0, 1 ] 10869 1.711% 1.711% ( 1, 2 ] 302617 47.646% 49.358% ########## ( 2, 3 ] 157418 24.785% 74.143% ##### ( 3, 4 ] 65897 10.375% 84.518% ## ( 4, 6 ] 61023 9.608% 94.126% ## ( 6, 10 ] 16507 2.599% 96.725% # ( 10, 15 ] 2860 0.450% 97.175% ( 15, 22 ] 343 0.054% 97.229% ( 22, 34 ] 27 0.004% 97.234% ( 34, 51 ] 33 0.005% 97.239% ( 51, 76 ] 42 0.007% 97.245% ( 76, 110 ] 1068 0.168% 97.414% ( 110, 170 ] 4647 0.732% 98.145% ( 170, 250 ] 2491 0.392% 98.537% ( 250, 380 ] 1510 0.238% 98.775% ( 380, 580 ] 1893 0.298% 99.073% ( 580, 870 ] 1993 0.314% 99.387% ( 870, 1300 ] 1667 0.262% 99.650% ( 1300, 1900 ] 1077 0.170% 99.819% ( 1900, 2900 ] 771 0.121% 99.940% ( 2900, 4400 ] 843 0.133% 100.073% ( 4400, 6600 ] 197 0.031% 100.104% ( 6600, 9900 ] 21 0.003% 100.108% ( 9900, 14000 ] 3 0.000% 100.108% ( 14000, 22000 ] 3 0.000% 100.108% ** Level 5 read latency histogram (micros): Count: 85436 Average: 552.2561 StdDev: 892.40 Min: 77 Median: 209.5692 Max: 24442 Percentiles: P50: 209.57 P75: 539.59 P99: 4602.17 P99.9: 7024.01 P99.99: 9843.40 ------------------------------------------------------ ( 76, 110 ] 6744 7.894% 7.894% ## ( 110, 170 ] 28718 33.613% 41.507% ####### ( 170, 250 ] 14670 17.171% 58.678% ### ( 250, 380 ] 7515 8.796% 67.474% ## ( 380, 580 ] 8058 9.432% 76.906% ## ( 580, 870 ] 6387 7.476% 84.381% # ( 870, 1300 ] 4818 5.639% 90.021% # ( 1300, 1900 ] 2849 3.335% 93.355% # ( 1900, 2900 ] 2287 2.677% 96.032% # ( 2900, 4400 ] 2459 2.878% 98.910% # ( 4400, 6600 ] 834 0.976% 99.886% ( 6600, 9900 ] 90 0.105% 99.992% ( 9900, 14000 ] 7 0.008% 100.000% ( 14000, 22000 ] 1 0.001% 100.001% ( 22000, 33000 ] 2 0.002% 100.004% ** DB Stats ** Uptime(secs): 16.9 total, 3.6 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 174.56 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.449 H:M:S, 32.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 276.31 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/14-22:23:42 ... thread 0: (1000000,4000000) ops and (242666.4,192243.9) ops/second in (4.120883,20.806901) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/4 3.44 GB 295.8 2.9 0.0 2.9 6.7 3.8 0.0 1.8 93.4 216.5 31.70 22.93 66 0.480 3006K 3096 0.0 0.0 L1 5/5 526.75 MB 0.0 0.8 0.5 0.3 0.8 0.5 0.0 1.7 127.2 126.9 6.62 3.88 2 3.311 855K 1489 0.0 0.0 L2 1/0 50.33 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 17/0 948.15 MB 0.9 3.3 2.0 1.2 3.2 2.0 0.0 1.6 100.4 99.6 33.35 15.89 19 1.755 2051K 809 0.0 0.0 L4 169/0 9.58 GB 1.0 7.0 4.6 2.4 6.9 4.5 0.4 1.5 116.7 115.5 61.35 33.32 47 1.305 2795K 2261 0.0 0.0 L5 1140/0 93.81 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1350/9 108.32 GB 0.0 14.0 7.1 6.9 17.7 10.8 5.5 4.6 107.6 136.2 133.02 76.02 134 0.993 8708K 7655 0.0 0.0 Int 0/0 0.00 KB 0.0 14.0 7.1 6.9 17.7 10.8 5.5 4.6 107.6 136.2 133.02 76.02 134 0.993 8708K 7655 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 14.0 7.1 6.9 13.9 7.0 0.0 0.0 125.9 125.0 113.68 64.38 79 1.439 8708K 7655 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 202.0 19.33 11.64 55 0.351 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 21.0 total, 20.8 interval Flush(GB): cumulative 3.814, interval 3.814 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 17.69 GB write, 861.03 MB/s write, 13.98 GB read, 680.37 MB/s read, 133.0 seconds Interval compaction: 17.69 GB write, 870.39 MB/s write, 13.98 GB read, 687.77 MB/s read, 133.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 5, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 5, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 65, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 65, interval: 70 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1031240 Average: 11.0797 StdDev: 84.55 Min: 0 Median: 1.6095 Max: 7837 Percentiles: P50: 1.61 P75: 2.27 P99: 226.61 P99.9: 778.82 P99.99: 1255.69 ------------------------------------------------------ [ 0, 1 ] 172790 16.756% 16.756% ### ( 1, 2 ] 562508 54.547% 71.302% ########### ( 2, 3 ] 143473 13.913% 85.215% ### ( 3, 4 ] 60183 5.836% 91.051% # ( 4, 6 ] 46670 4.526% 95.577% # ( 6, 10 ] 10972 1.064% 96.641% ( 10, 15 ] 1984 0.192% 96.833% ( 15, 22 ] 362 0.035% 96.868% ( 22, 34 ] 81 0.008% 96.876% ( 34, 51 ] 10 0.001% 96.877% ( 51, 76 ] 22 0.002% 96.879% ( 76, 110 ] 2906 0.282% 97.161% ( 110, 170 ] 13613 1.320% 98.481% ( 170, 250 ] 7566 0.734% 99.215% ( 250, 380 ] 3327 0.323% 99.537% ( 380, 580 ] 2821 0.274% 99.811% ( 580, 870 ] 1343 0.130% 99.941% ( 870, 1300 ] 564 0.055% 99.996% ( 1300, 1900 ] 229 0.022% 100.018% ( 1900, 2900 ] 161 0.016% 100.033% ( 2900, 4400 ] 158 0.015% 100.049% ( 4400, 6600 ] 36 0.003% 100.052% ( 6600, 9900 ] 4 0.000% 100.053% ** Level 1 read latency histogram (micros): Count: 216562 Average: 10.9861 StdDev: 124.68 Min: 1 Median: 2.3488 Max: 24261 Percentiles: P50: 2.35 P75: 3.86 P99: 149.24 P99.9: 1669.94 P99.99: 4324.87 ------------------------------------------------------ [ 0, 1 ] 5711 2.637% 2.637% # ( 1, 2 ] 86609 39.993% 42.630% ######## ( 2, 3 ] 45764 21.132% 63.762% #### ( 3, 4 ] 28259 13.049% 76.811% ### ( 4, 6 ] 35377 16.336% 93.147% ### ( 6, 10 ] 9240 4.267% 97.413% # ( 10, 15 ] 1681 0.776% 98.189% ( 15, 22 ] 691 0.319% 98.509% ( 22, 34 ] 65 0.030% 98.539% ( 34, 51 ] 5 0.002% 98.541% ( 51, 76 ] 7 0.003% 98.544% ( 76, 110 ] 270 0.125% 98.669% ( 110, 170 ] 1097 0.507% 99.175% ( 170, 250 ] 482 0.223% 99.398% ( 250, 380 ] 278 0.128% 99.526% ( 380, 580 ] 284 0.131% 99.657% ( 580, 870 ] 254 0.117% 99.775% ( 870, 1300 ] 203 0.094% 99.868% ( 1300, 1900 ] 111 0.051% 99.920% ( 1900, 2900 ] 83 0.038% 99.958% ( 2900, 4400 ] 73 0.034% 99.992% ( 4400, 6600 ] 14 0.006% 99.998% ( 6600, 9900 ] 5 0.002% 100.000% ( 22000, 33000 ] 1 0.000% 100.001% ** Level 2 read latency histogram (micros): Count: 409886 Average: 13.0425 StdDev: 156.65 Min: 1 Median: 2.0230 Max: 8025 Percentiles: P50: 2.02 P75: 3.06 P99: 12.60 P99.9: 1681.30 P99.99: 2542.08 ------------------------------------------------------ [ 0, 1 ] 8909 2.174% 2.174% ( 1, 2 ] 193675 47.251% 49.424% ######### ( 2, 3 ] 102422 24.988% 74.412% ##### ( 3, 4 ] 43667 10.653% 85.066% ## ( 4, 6 ] 43052 10.503% 95.569% ## ( 6, 10 ] 12781 3.118% 98.687% # ( 10, 15 ] 2463 0.601% 99.288% ( 15, 22 ] 375 0.091% 99.380% ( 22, 34 ] 68 0.017% 99.396% ( 34, 51 ] 29 0.007% 99.403% ( 51, 76 ] 26 0.006% 99.410% ( 76, 110 ] 14 0.003% 99.413% ( 110, 170 ] 48 0.012% 99.425% ( 170, 250 ] 103 0.025% 99.450% ( 250, 380 ] 173 0.042% 99.492% ( 380, 580 ] 386 0.094% 99.586% ( 580, 870 ] 513 0.125% 99.712% ( 870, 1300 ] 523 0.128% 99.839% ( 1300, 1900 ] 392 0.096% 99.935% ( 1900, 2900 ] 352 0.086% 100.021% ( 2900, 4400 ] 270 0.066% 100.087% ( 4400, 6600 ] 90 0.022% 100.109% ( 6600, 9900 ] 8 0.002% 100.111% ** Level 3 read latency histogram (micros): Count: 1406691 Average: 12.5599 StdDev: 156.64 Min: 1 Median: 2.0155 Max: 19021 Percentiles: P50: 2.02 P75: 2.99 P99: 11.42 P99.9: 1113.60 P99.99: 1581.39 ------------------------------------------------------ [ 0, 1 ] 21983 1.563% 1.563% ( 1, 2 ] 675728 48.037% 49.599% ########## ( 2, 3 ] 362689 25.783% 75.383% ##### ( 3, 4 ] 148333 10.545% 85.927% ## ( 4, 6 ] 142740 10.147% 96.075% ## ( 6, 10 ] 39146 2.783% 98.857% # ( 10, 15 ] 7048 0.501% 99.358% ( 15, 22 ] 927 0.066% 99.424% ( 22, 34 ] 88 0.006% 99.431% ( 34, 51 ] 65 0.005% 99.435% ( 51, 76 ] 112 0.008% 99.443% ( 76, 110 ] 271 0.019% 99.462% ( 110, 170 ] 1090 0.077% 99.540% ( 170, 250 ] 635 0.045% 99.585% ( 250, 380 ] 698 0.050% 99.635% ( 380, 580 ] 1203 0.086% 99.720% ( 580, 870 ] 1559 0.111% 99.831% ( 870, 1300 ] 1711 0.122% 99.953% ( 1300, 1900 ] 1118 0.079% 100.032% ( 1900, 2900 ] 1031 0.073% 100.105% ( 2900, 4400 ] 1013 0.072% 100.178% ( 4400, 6600 ] 271 0.019% 100.197% ( 6600, 9900 ] 38 0.003% 100.199% ( 9900, 14000 ] 2 0.000% 100.200% ( 14000, 22000 ] 3 0.000% 100.200% ** Level 4 read latency histogram (micros): Count: 643709 Average: 26.5739 StdDev: 213.57 Min: 1 Median: 2.0531 Max: 20290 Percentiles: P50: 2.05 P75: 3.18 P99: 555.19 P99.9: 2558.85 P99.99: 3460.29 ------------------------------------------------------ [ 0, 1 ] 10869 1.688% 1.688% ( 1, 2 ] 302629 47.013% 48.702% ######### ( 2, 3 ] 157444 24.459% 73.161% ##### ( 3, 4 ] 65971 10.249% 83.409% ## ( 4, 6 ] 61131 9.497% 92.906% ## ( 6, 10 ] 16521 2.567% 95.472% # ( 10, 15 ] 2865 0.445% 95.918% ( 15, 22 ] 344 0.053% 95.971% ( 22, 34 ] 27 0.004% 95.975% ( 34, 51 ] 33 0.005% 95.980% ( 51, 76 ] 42 0.007% 95.987% ( 76, 110 ] 1995 0.310% 96.297% ( 110, 170 ] 8640 1.342% 97.639% ( 170, 250 ] 4324 0.672% 98.311% ( 250, 380 ] 2283 0.355% 98.665% ( 380, 580 ] 2459 0.382% 99.047% ( 580, 870 ] 2197 0.341% 99.389% ( 870, 1300 ] 1701 0.264% 99.653% ( 1300, 1900 ] 1081 0.168% 99.821% ( 1900, 2900 ] 773 0.120% 99.941% ( 2900, 4400 ] 845 0.131% 100.072% ( 4400, 6600 ] 197 0.031% 100.103% ( 6600, 9900 ] 21 0.003% 100.106% ( 9900, 14000 ] 3 0.000% 100.107% ( 14000, 22000 ] 3 0.000% 100.107% ** Level 5 read latency histogram (micros): Count: 134283 Average: 425.4417 StdDev: 736.71 Min: 76 Median: 177.2392 Max: 24442 Percentiles: P50: 177.24 P75: 368.95 P99: 4148.68 P99.9: 6499.01 P99.99: 9627.63 ------------------------------------------------------ ( 51, 76 ] 4 0.003% 0.003% ( 76, 110 ] 12735 9.484% 9.487% ## ( 110, 170 ] 52136 38.825% 48.312% ######## ( 170, 250 ] 25047 18.652% 66.965% #### ( 250, 380 ] 11793 8.782% 75.747% ## ( 380, 580 ] 11518 8.577% 84.324% ## ( 580, 870 ] 7460 5.555% 89.880% # ( 870, 1300 ] 5005 3.727% 93.607% # ( 1300, 1900 ] 2876 2.142% 95.749% ( 1900, 2900 ] 2315 1.724% 97.473% ( 2900, 4400 ] 2464 1.835% 99.307% ( 4400, 6600 ] 834 0.621% 99.929% ( 6600, 9900 ] 90 0.067% 99.996% ( 9900, 14000 ] 7 0.005% 100.001% ( 14000, 22000 ] 1 0.001% 100.001% ( 22000, 33000 ] 2 0.001% 100.003% ** DB Stats ** Uptime(secs): 21.0 total, 4.1 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 187.15 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.935 H:M:S, 28.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 238.83 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.486 H:M:S, 11.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 2024/12/14-22:23:48 ... thread 0: (1000000,5000000) ops and (182633.6,190241.8) ops/second in (5.475444,26.282345) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/17 3.87 GB 25.2 3.9 0.0 3.9 8.7 4.8 0.0 1.8 102.5 229.3 38.72 29.45 86 0.450 4029K 4080 0.0 0.0 L1 11/11 1.06 GB 0.0 1.9 1.0 0.9 1.9 1.0 0.0 1.8 99.6 99.3 19.43 11.07 3 6.478 1965K 5729 0.0 0.0 L2 1/0 50.33 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 17/0 948.15 MB 0.9 3.3 2.0 1.2 3.2 2.0 0.0 1.6 100.4 99.6 33.35 15.89 19 1.755 2051K 809 0.0 0.0 L4 169/0 9.58 GB 1.0 7.0 4.6 2.4 6.9 4.5 0.4 1.5 116.7 115.5 61.35 33.32 47 1.305 2795K 2261 0.0 0.0 L5 1140/0 93.81 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1356/28 109.30 GB 0.0 16.0 7.7 8.4 20.7 12.3 5.5 4.3 107.4 138.8 152.85 89.72 155 0.986 10M 12K 0.0 0.0 Int 0/0 0.00 KB 0.0 16.0 7.7 8.4 20.7 12.3 5.5 4.3 107.4 138.8 152.85 89.72 155 0.986 10M 12K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.0 7.7 8.4 15.9 7.6 0.0 0.0 125.9 125.0 130.39 75.39 84 1.552 10M 12K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.8 4.8 0.0 0.0 0.0 218.8 22.46 14.33 71 0.316 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 26.5 total, 26.3 interval Flush(GB): cumulative 4.799, interval 4.799 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 20.72 GB write, 800.27 MB/s write, 16.03 GB read, 619.11 MB/s read, 152.8 seconds Interval compaction: 20.72 GB write, 807.16 MB/s write, 16.03 GB read, 624.44 MB/s read, 152.8 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 10, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 10, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 65, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 10, total-stops: 65, interval: 75 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1342897 Average: 10.9045 StdDev: 77.84 Min: 0 Median: 1.5793 Max: 7837 Percentiles: P50: 1.58 P75: 2.20 P99: 218.88 P99.9: 712.24 P99.99: 1155.96 ------------------------------------------------------ [ 0, 1 ] 254081 18.920% 18.920% #### ( 1, 2 ] 720521 53.654% 72.575% ########### ( 2, 3 ] 166975 12.434% 85.009% ## ( 3, 4 ] 81050 6.035% 91.044% # ( 4, 6 ] 57457 4.279% 95.323% # ( 6, 10 ] 13078 0.974% 96.296% ( 10, 15 ] 2496 0.186% 96.482% ( 15, 22 ] 423 0.031% 96.514% ( 22, 34 ] 88 0.007% 96.520% ( 34, 51 ] 15 0.001% 96.521% ( 51, 76 ] 40 0.003% 96.524% ( 76, 110 ] 5121 0.381% 96.906% ( 110, 170 ] 21912 1.632% 98.537% ( 170, 250 ] 10168 0.757% 99.295% ( 250, 380 ] 4070 0.303% 99.598% ( 380, 580 ] 3372 0.251% 99.849% ( 580, 870 ] 1509 0.112% 99.961% ( 870, 1300 ] 583 0.043% 100.005% ( 1300, 1900 ] 232 0.017% 100.022% ( 1900, 2900 ] 178 0.013% 100.035% ( 2900, 4400 ] 168 0.013% 100.048% ( 4400, 6600 ] 36 0.003% 100.050% ( 6600, 9900 ] 4 0.000% 100.051% ** Level 1 read latency histogram (micros): Count: 232249 Average: 11.3153 StdDev: 121.68 Min: 1 Median: 2.3305 Max: 24261 Percentiles: P50: 2.33 P75: 3.87 P99: 161.31 P99.9: 1608.90 P99.99: 4292.64 ------------------------------------------------------ [ 0, 1 ] 7810 3.363% 3.363% # ( 1, 2 ] 92709 39.918% 43.281% ######## ( 2, 3 ] 47224 20.333% 63.614% #### ( 3, 4 ] 30241 13.021% 76.635% ### ( 4, 6 ] 37777 16.266% 92.901% ### ( 6, 10 ] 9645 4.153% 97.054% # ( 10, 15 ] 1756 0.756% 97.810% ( 15, 22 ] 752 0.324% 98.133% ( 22, 34 ] 69 0.030% 98.163% ( 34, 51 ] 5 0.002% 98.165% ( 51, 76 ] 7 0.003% 98.168% ( 76, 110 ] 458 0.197% 98.366% ( 110, 170 ] 1723 0.742% 99.107% ( 170, 250 ] 647 0.279% 99.386% ( 250, 380 ] 326 0.140% 99.526% ( 380, 580 ] 330 0.142% 99.668% ( 580, 870 ] 268 0.115% 99.784% ( 870, 1300 ] 209 0.090% 99.874% ( 1300, 1900 ] 118 0.051% 99.925% ( 1900, 2900 ] 84 0.036% 99.961% ( 2900, 4400 ] 73 0.031% 99.992% ( 4400, 6600 ] 14 0.006% 99.998% ( 6600, 9900 ] 5 0.002% 100.000% ( 22000, 33000 ] 1 0.000% 100.001% ** Level 2 read latency histogram (micros): Count: 409886 Average: 13.0425 StdDev: 156.65 Min: 1 Median: 2.0230 Max: 8025 Percentiles: P50: 2.02 P75: 3.06 P99: 12.60 P99.9: 1681.30 P99.99: 2542.08 ------------------------------------------------------ [ 0, 1 ] 8909 2.174% 2.174% ( 1, 2 ] 193675 47.251% 49.424% ######### ( 2, 3 ] 102422 24.988% 74.412% ##### ( 3, 4 ] 43667 10.653% 85.066% ## ( 4, 6 ] 43052 10.503% 95.569% ## ( 6, 10 ] 12781 3.118% 98.687% # ( 10, 15 ] 2463 0.601% 99.288% ( 15, 22 ] 375 0.091% 99.380% ( 22, 34 ] 68 0.017% 99.396% ( 34, 51 ] 29 0.007% 99.403% ( 51, 76 ] 26 0.006% 99.410% ( 76, 110 ] 14 0.003% 99.413% ( 110, 170 ] 48 0.012% 99.425% ( 170, 250 ] 103 0.025% 99.450% ( 250, 380 ] 173 0.042% 99.492% ( 380, 580 ] 386 0.094% 99.586% ( 580, 870 ] 513 0.125% 99.712% ( 870, 1300 ] 523 0.128% 99.839% ( 1300, 1900 ] 392 0.096% 99.935% ( 1900, 2900 ] 352 0.086% 100.021% ( 2900, 4400 ] 270 0.066% 100.087% ( 4400, 6600 ] 90 0.022% 100.109% ( 6600, 9900 ] 8 0.002% 100.111% ** Level 3 read latency histogram (micros): Count: 1408178 Average: 12.7372 StdDev: 156.74 Min: 1 Median: 2.0176 Max: 19021 Percentiles: P50: 2.02 P75: 2.99 P99: 12.47 P99.9: 1114.70 P99.99: 1583.49 ------------------------------------------------------ [ 0, 1 ] 21983 1.561% 1.561% ( 1, 2 ] 675728 47.986% 49.547% ########## ( 2, 3 ] 362689 25.756% 75.303% ##### ( 3, 4 ] 148333 10.534% 85.837% ## ( 4, 6 ] 142740 10.137% 95.973% ## ( 6, 10 ] 39146 2.780% 98.753% # ( 10, 15 ] 7048 0.501% 99.254% ( 15, 22 ] 927 0.066% 99.319% ( 22, 34 ] 88 0.006% 99.326% ( 34, 51 ] 65 0.005% 99.330% ( 51, 76 ] 112 0.008% 99.338% ( 76, 110 ] 522 0.037% 99.375% ( 110, 170 ] 1913 0.136% 99.511% ( 170, 250 ] 876 0.062% 99.573% ( 250, 380 ] 778 0.055% 99.629% ( 380, 580 ] 1268 0.090% 99.719% ( 580, 870 ] 1579 0.112% 99.831% ( 870, 1300 ] 1713 0.122% 99.952% ( 1300, 1900 ] 1120 0.080% 100.032% ( 1900, 2900 ] 1033 0.073% 100.105% ( 2900, 4400 ] 1014 0.072% 100.177% ( 4400, 6600 ] 271 0.019% 100.197% ( 6600, 9900 ] 38 0.003% 100.199% ( 9900, 14000 ] 2 0.000% 100.199% ( 14000, 22000 ] 3 0.000% 100.200% ** Level 4 read latency histogram (micros): Count: 656557 Average: 29.5196 StdDev: 213.13 Min: 1 Median: 2.0939 Max: 20290 Percentiles: P50: 2.09 P75: 3.33 P99: 565.94 P99.9: 2543.15 P99.99: 3458.01 ------------------------------------------------------ [ 0, 1 ] 10869 1.655% 1.655% ( 1, 2 ] 302629 46.093% 47.749% ######### ( 2, 3 ] 157444 23.980% 71.729% ##### ( 3, 4 ] 65971 10.048% 81.777% ## ( 4, 6 ] 61131 9.311% 91.088% ## ( 6, 10 ] 16521 2.516% 93.604% # ( 10, 15 ] 2865 0.436% 94.041% ( 15, 22 ] 344 0.052% 94.093% ( 22, 34 ] 27 0.004% 94.097% ( 34, 51 ] 33 0.005% 94.102% ( 51, 76 ] 42 0.006% 94.109% ( 76, 110 ] 4087 0.622% 94.731% ( 110, 170 ] 15765 2.401% 97.132% ( 170, 250 ] 6444 0.981% 98.114% ( 250, 380 ] 3009 0.458% 98.572% ( 380, 580 ] 3023 0.460% 99.032% ( 580, 870 ] 2389 0.364% 99.396% ( 870, 1300 ] 1718 0.262% 99.658% ( 1300, 1900 ] 1091 0.166% 99.824% ( 1900, 2900 ] 775 0.118% 99.942% ( 2900, 4400 ] 845 0.129% 100.071% ( 4400, 6600 ] 197 0.030% 100.101% ( 6600, 9900 ] 21 0.003% 100.104% ( 9900, 14000 ] 3 0.000% 100.104% ( 14000, 22000 ] 3 0.000% 100.105% ** Level 5 read latency histogram (micros): Count: 208066 Average: 337.3392 StdDev: 608.15 Min: 76 Median: 160.8903 Max: 24442 Percentiles: P50: 160.89 P75: 259.02 P99: 3700.85 P99.9: 6299.11 P99.99: 9283.76 ------------------------------------------------------ ( 51, 76 ] 4 0.002% 0.002% ( 76, 110 ] 25345 12.181% 12.183% ## ( 110, 170 ] 92769 44.586% 56.769% ######### ( 170, 250 ] 36814 17.693% 74.463% #### ( 250, 380 ] 16098 7.737% 82.200% ## ( 380, 580 ] 14773 7.100% 89.300% # ( 580, 870 ] 8473 4.072% 93.372% # ( 870, 1300 ] 5147 2.474% 95.846% ( 1300, 1900 ] 2903 1.395% 97.241% ( 1900, 2900 ] 2339 1.124% 98.365% ( 2900, 4400 ] 2473 1.189% 99.554% ( 4400, 6600 ] 834 0.401% 99.955% ( 6600, 9900 ] 90 0.043% 99.998% ( 9900, 14000 ] 7 0.003% 100.001% ( 14000, 22000 ] 1 0.000% 100.002% ( 22000, 33000 ] 2 0.001% 100.003% ** DB Stats ** Uptime(secs): 26.5 total, 5.5 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 185.62 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.809 H:M:S, 29.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 179.73 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.874 H:M:S, 34.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 254.254 micros/op 15721 ops/sec 60.171 seconds 945996 operations; 15.2 MB/s (235999 of 235999 found) Microseconds per read: Count: 945996 Average: 254.2584 StdDev: 694.09 Min: 3 Median: 190.9483 Max: 75712 Percentiles: P50: 190.95 P75: 238.06 P99: 1242.04 P99.9: 11646.99 P99.99: 28464.65 ------------------------------------------------------ ( 2, 3 ] 2 0.000% 0.000% ( 3, 4 ] 3 0.000% 0.001% ( 4, 6 ] 38 0.004% 0.005% ( 6, 10 ] 287 0.030% 0.035% ( 10, 15 ] 137 0.014% 0.049% ( 15, 22 ] 484 0.051% 0.101% ( 22, 34 ] 14284 1.510% 1.610% ( 34, 51 ] 6649 0.703% 2.313% ( 51, 76 ] 942 0.100% 2.413% ( 76, 110 ] 502 0.053% 2.466% ( 110, 170 ] 344506 36.417% 38.883% ####### ( 170, 250 ] 401614 42.454% 81.337% ######## ( 250, 380 ] 106101 11.216% 92.553% ## ( 380, 580 ] 40193 4.249% 96.802% # ( 580, 870 ] 15114 1.598% 98.400% ( 870, 1300 ] 6565 0.694% 99.094% ( 1300, 1900 ] 2866 0.303% 99.397% ( 1900, 2900 ] 1897 0.201% 99.597% ( 2900, 4400 ] 1870 0.198% 99.795% ( 4400, 6600 ] 590 0.062% 99.857% ( 6600, 9900 ] 210 0.022% 99.879% ( 9900, 14000 ] 460 0.049% 99.928% ( 14000, 22000 ] 511 0.054% 99.982% ( 22000, 33000 ] 130 0.014% 99.996% ( 33000, 50000 ] 32 0.003% 99.999% ( 50000, 75000 ] 8 0.001% 100.000% ( 75000, 110000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 19287588 rocksdb.bloom.filter.full.positive COUNT : 1133755 rocksdb.bloom.filter.full.true.positive COUNT : 945547 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 449 rocksdb.memtable.miss COUNT : 945547 rocksdb.l0.hit COUNT : 35581 rocksdb.l1.hit COUNT : 9219 rocksdb.l2andup.hit COUNT : 900747 rocksdb.compaction.key.drop.new COUNT : 144397 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5315524 rocksdb.number.keys.read COUNT : 945996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5485620768 rocksdb.bytes.read COUNT : 945996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1624 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 39453786 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 19024882277 rocksdb.compact.write.bytes COUNT : 18876697570 rocksdb.flush.write.bytes COUNT : 5482894196 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 27 rocksdb.number.superversion_acquires COUNT : 1064 rocksdb.number.superversion_releases COUNT : 1017 rocksdb.number.superversion_cleanups COUNT : 264 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 258 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5438562304 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2003968 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 23483703580 rocksdb.non.last.level.read.count COUNT : 5531230 rocksdb.block.checksum.compute.count COUNT : 5755110 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 189.609046 P95 : 493.694849 P99 : 1241.070619 P100 : 75710.000000 COUNT : 945996 SUM : 239382841 rocksdb.db.write.micros P50 : 1.558834 P95 : 4.783471 P99 : 10.169116 P100 : 869892.000000 COUNT : 5315524 SUM : 53153477 rocksdb.compaction.times.micros P50 : 1345121.951220 P95 : 2721428.571429 P99 : 10430000.000000 P100 : 12811876.000000 COUNT : 85 SUM : 131349384 rocksdb.compaction.times.cpu_micros P50 : 835000.000000 P95 : 1585000.000000 P99 : 6995000.000000 P100 : 7186623.000000 COUNT : 85 SUM : 76340090 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 761.250000 P95 : 122000.000000 P99 : 281200.000000 P100 : 338265.000000 COUNT : 76 SUM : 1175336 rocksdb.compaction.outfile.sync.micros P50 : 35833.333333 P95 : 225833.333333 P99 : 332875.000000 P100 : 571117.000000 COUNT : 245 SUM : 16599586 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 256.500000 P95 : 4607.777778 P99 : 6397.111111 P100 : 8285.000000 COUNT : 183 SUM : 209998 rocksdb.table.open.io.micros P50 : 8925.779626 P95 : 19120.000000 P99 : 21785.025641 P100 : 29311.000000 COUNT : 1624 SUM : 14350685 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 4.103621 P95 : 13.561509 P99 : 190.490201 P100 : 24267.000000 COUNT : 4607892 SUM : 77116712 rocksdb.read.block.get.micros P50 : 138.849642 P95 : 344.077674 P99 : 1168.669340 P100 : 24445.000000 COUNT : 1145622 SUM : 195835883 rocksdb.write.raw.block.micros P50 : 0.556536 P95 : 1.663287 P99 : 4.353782 P100 : 25284.000000 COUNT : 5900364 SUM : 18103676 rocksdb.numfiles.in.singlecompaction P50 : 2.648649 P95 : 3.962162 P99 : 9.840000 P100 : 28.000000 COUNT : 108 SUM : 362 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.583926 P95 : 2150.309704 P99 : 2751.163111 P100 : 869724.000000 COUNT : 152782 SUM : 39472549 rocksdb.sst.read.micros P50 : 2.477206 P95 : 166.602274 P99 : 493.132977 P100 : 24442.000000 COUNT : 5531310 SUM : 256151434 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 945996 SUM : 945996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5315524 SUM : 5485620768 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 221525.423729 P95 : 877000.000000 P99 : 1089622.000000 P100 : 1089622.000000 COUNT : 76 SUM : 23536501 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 269 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0