[2024-10-16 12:07:07.140161] Starting SPDK v24.09.1-pre git sha1 b18e1bd62 / DPDK 22.11.4 initialization... [2024-10-16 12:07:07.140279] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1498220 ] Set seed to 1729073227137738 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-10-16 12:07:07.226406] app.c: 917:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-10-16 12:07:07.268129] app.c: 610:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-10-16 12:07:07.268185] app.c: 611:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1498220' to capture a snapshot of events at runtime. [2024-10-16 12:07:07.268196] app.c: 616:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-10-16 12:07:07.268206] app.c: 617:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-10-16 12:07:07.268214] app.c: 618:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1498220 for offline analysis/debug. [2024-10-16 12:07:07.268302] reactor.c: 990:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Wed Oct 16 12:07:11 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/10/16-12:07:20 ... thread 0: (1000000,1000000) ops and (107252.7,107252.7) ops/second in (9.323778,9.323778) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/11 1.96 GB 56.6 0.4 0.0 0.4 1.2 0.9 0.0 1.4 39.0 129.9 9.69 4.18 11 0.881 383K 517 0.0 0.0 L1 7/7 836.70 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 25/6 1.63 GB 1.0 2.1 1.2 0.9 2.1 1.2 0.3 1.7 99.2 98.1 22.10 10.30 10 2.210 751K 0 0.0 0.0 L4 184/18 10.23 GB 1.0 4.8 3.3 1.5 4.7 3.2 0.0 1.4 107.5 106.1 45.81 23.23 34 1.347 1988K 0 0.0 0.0 L5 1112/0 91.48 GB 0.9 0.0 0.0 0.0 0.0 0.0 2.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1342/42 106.11 GB 0.0 7.3 4.5 2.8 8.1 5.3 2.9 9.4 96.6 106.8 77.60 37.71 55 1.411 3124K 517 0.0 0.0 Int 0/0 0.00 KB 0.0 7.3 4.5 2.8 8.1 5.3 2.9 9.4 96.6 106.8 77.60 37.71 55 1.411 3124K 517 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 7.3 4.5 2.8 7.2 4.4 0.0 0.0 106.2 105.0 70.57 35.12 45 1.568 3124K 517 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 125.4 7.03 2.59 10 0.703 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 9.5 total, 9.3 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 8.09 GB write, 874.23 MB/s write, 7.32 GB read, 790.58 MB/s read, 77.6 seconds Interval compaction: 8.09 GB write, 888.86 MB/s write, 7.32 GB read, 803.81 MB/s read, 77.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 26, pending-compaction-bytes-delays: 4, pending-compaction-bytes-stops: 0, total-delays: 4, total-stops: 26, interval: 29 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 430210 Average: 12.0932 StdDev: 151.15 Min: 1 Median: 1.8876 Max: 21325 Percentiles: P50: 1.89 P75: 2.87 P99: 11.25 P99.9: 1938.55 P99.99: 3954.26 ------------------------------------------------------ [ 0, 1 ] 15860 3.687% 3.687% # ( 1, 2 ] 224471 52.177% 55.864% ########## ( 2, 3 ] 94999 22.082% 77.946% #### ( 3, 4 ] 39379 9.153% 87.099% ## ( 4, 6 ] 40993 9.529% 96.628% ## ( 6, 10 ] 9846 2.289% 98.916% ( 10, 15 ] 1443 0.335% 99.252% ( 15, 22 ] 206 0.048% 99.300% ( 22, 34 ] 22 0.005% 99.305% ( 34, 51 ] 13 0.003% 99.308% ( 51, 76 ] 21 0.005% 99.313% ( 76, 110 ] 9 0.002% 99.315% ( 110, 170 ] 30 0.007% 99.322% ( 170, 250 ] 87 0.020% 99.342% ( 250, 380 ] 223 0.052% 99.394% ( 380, 580 ] 456 0.106% 99.500% ( 580, 870 ] 698 0.162% 99.662% ( 870, 1300 ] 625 0.145% 99.807% ( 1300, 1900 ] 391 0.091% 99.898% ( 1900, 2900 ] 228 0.053% 99.951% ( 2900, 4400 ] 239 0.056% 100.007% ( 4400, 6600 ] 70 0.016% 100.023% ( 6600, 9900 ] 12 0.003% 100.026% ( 9900, 14000 ] 2 0.000% 100.026% ( 14000, 22000 ] 1 0.000% 100.026% ** Level 1 read latency histogram (micros): Count: 20930 Average: 21.2364 StdDev: 223.05 Min: 1 Median: 2.1239 Max: 7139 Percentiles: P50: 2.12 P75: 3.21 P99: 546.18 P99.9: 3946.54 P99.99: 7139.00 ------------------------------------------------------ [ 0, 1 ] 137 0.655% 0.655% ( 1, 2 ] 9656 46.135% 46.789% ######### ( 2, 3 ] 5424 25.915% 72.704% ##### ( 3, 4 ] 2278 10.884% 83.588% ## ( 4, 6 ] 2485 11.873% 95.461% ## ( 6, 10 ] 581 2.776% 98.237% # ( 10, 15 ] 67 0.320% 98.557% ( 15, 22 ] 8 0.038% 98.595% ( 22, 34 ] 1 0.005% 98.600% ( 51, 76 ] 6 0.029% 98.629% ( 110, 170 ] 4 0.019% 98.648% ( 170, 250 ] 6 0.029% 98.677% ( 250, 380 ] 22 0.105% 98.782% ( 380, 580 ] 55 0.263% 99.044% ( 580, 870 ] 67 0.320% 99.365% ( 870, 1300 ] 51 0.244% 99.608% ( 1300, 1900 ] 27 0.129% 99.737% ( 1900, 2900 ] 25 0.119% 99.857% ( 2900, 4400 ] 13 0.062% 99.919% ( 4400, 6600 ] 14 0.067% 99.986% ( 6600, 9900 ] 3 0.014% 100.000% ** Level 2 read latency histogram (micros): Count: 159215 Average: 9.8994 StdDev: 131.36 Min: 1 Median: 2.0444 Max: 13246 Percentiles: P50: 2.04 P75: 3.18 P99: 11.16 P99.9: 1819.91 P99.99: 3894.12 ------------------------------------------------------ [ 0, 1 ] 3440 2.161% 2.161% ( 1, 2 ] 74460 46.767% 48.928% ######### ( 2, 3 ] 38497 24.179% 73.107% ##### ( 3, 4 ] 17056 10.713% 83.819% ## ( 4, 6 ] 18801 11.809% 95.628% ## ( 6, 10 ] 5180 3.253% 98.881% # ( 10, 15 ] 811 0.509% 99.391% ( 15, 22 ] 133 0.084% 99.474% ( 22, 34 ] 21 0.013% 99.487% ( 34, 51 ] 11 0.007% 99.494% ( 51, 76 ] 24 0.015% 99.509% ( 76, 110 ] 7 0.004% 99.514% ( 110, 170 ] 9 0.006% 99.520% ( 170, 250 ] 41 0.026% 99.545% ( 250, 380 ] 80 0.050% 99.596% ( 380, 580 ] 118 0.074% 99.670% ( 580, 870 ] 138 0.087% 99.756% ( 870, 1300 ] 143 0.090% 99.846% ( 1300, 1900 ] 99 0.062% 99.908% ( 1900, 2900 ] 87 0.055% 99.963% ( 2900, 4400 ] 65 0.041% 100.004% ( 4400, 6600 ] 23 0.014% 100.018% ( 6600, 9900 ] 3 0.002% 100.020% ( 9900, 14000 ] 1 0.001% 100.021% ** Level 3 read latency histogram (micros): Count: 990221 Average: 11.6513 StdDev: 158.36 Min: 1 Median: 1.9968 Max: 30490 Percentiles: P50: 2.00 P75: 3.00 P99: 9.78 P99.9: 926.96 P99.99: 1367.35 ------------------------------------------------------ [ 0, 1 ] 16896 1.706% 1.706% ( 1, 2 ] 479770 48.451% 50.157% ########## ( 2, 3 ] 245605 24.803% 74.960% ##### ( 3, 4 ] 102478 10.349% 85.309% ## ( 4, 6 ] 110330 11.142% 96.451% ## ( 6, 10 ] 26706 2.697% 99.148% # ( 10, 15 ] 4221 0.426% 99.574% ( 15, 22 ] 550 0.056% 99.630% ( 22, 34 ] 67 0.007% 99.637% ( 34, 51 ] 67 0.007% 99.643% ( 51, 76 ] 87 0.009% 99.652% ( 76, 110 ] 38 0.004% 99.656% ( 110, 170 ] 100 0.010% 99.666% ( 170, 250 ] 149 0.015% 99.681% ( 250, 380 ] 369 0.037% 99.718% ( 380, 580 ] 658 0.066% 99.785% ( 580, 870 ] 1016 0.103% 99.887% ( 870, 1300 ] 942 0.095% 99.983% ( 1300, 1900 ] 659 0.067% 100.049% ( 1900, 2900 ] 621 0.063% 100.112% ( 2900, 4400 ] 712 0.072% 100.184% ( 4400, 6600 ] 203 0.021% 100.204% ( 6600, 9900 ] 32 0.003% 100.208% ( 14000, 22000 ] 1 0.000% 100.208% ( 22000, 33000 ] 2 0.000% 100.208% ** Level 4 read latency histogram (micros): Count: 503225 Average: 24.6440 StdDev: 249.87 Min: 1 Median: 2.0191 Max: 36680 Percentiles: P50: 2.02 P75: 3.07 P99: 603.79 P99.9: 2632.57 P99.99: 3421.61 ------------------------------------------------------ [ 0, 1 ] 8375 1.664% 1.664% ( 1, 2 ] 240859 47.863% 49.527% ########## ( 2, 3 ] 124612 24.763% 74.290% ##### ( 3, 4 ] 52036 10.341% 84.631% ## ( 4, 6 ] 54035 10.738% 95.368% ## ( 6, 10 ] 13781 2.739% 98.107% # ( 10, 15 ] 2073 0.412% 98.519% ( 15, 22 ] 307 0.061% 98.580% ( 22, 34 ] 30 0.006% 98.586% ( 34, 51 ] 31 0.006% 98.592% ( 51, 76 ] 65 0.013% 98.605% ( 76, 110 ] 18 0.004% 98.608% ( 110, 170 ] 100 0.020% 98.628% ( 170, 250 ] 211 0.042% 98.670% ( 250, 380 ] 515 0.102% 98.773% ( 380, 580 ] 1021 0.203% 98.975% ( 580, 870 ] 1521 0.302% 99.278% ( 870, 1300 ] 1583 0.315% 99.592% ( 1300, 1900 ] 1015 0.202% 99.794% ( 1900, 2900 ] 730 0.145% 99.939% ( 2900, 4400 ] 741 0.147% 100.086% ( 4400, 6600 ] 204 0.041% 100.127% ( 6600, 9900 ] 37 0.007% 100.134% ( 9900, 14000 ] 2 0.000% 100.135% ( 14000, 22000 ] 2 0.000% 100.135% ( 22000, 33000 ] 5 0.001% 100.136% ( 33000, 50000 ] 1 0.000% 100.136% ** Level 5 read latency histogram (micros): Count: 17812 Average: 1499.0670 StdDev: 1456.89 Min: 85 Median: 1027.1457 Max: 36911 Percentiles: P50: 1027.15 P75: 1876.43 P99: 6314.66 P99.9: 10715.53 P99.99: 30851.70 ------------------------------------------------------ ( 76, 110 ] 17 0.095% 0.095% ( 110, 170 ] 113 0.634% 0.730% ( 170, 250 ] 352 1.976% 2.706% ( 250, 380 ] 1102 6.187% 8.893% # ( 380, 580 ] 2414 13.553% 22.446% ### ( 580, 870 ] 3611 20.273% 42.718% #### ( 870, 1300 ] 3549 19.925% 62.643% #### ( 1300, 1900 ] 2291 12.862% 75.505% ### ( 1900, 2900 ] 1761 9.887% 85.392% ## ( 2900, 4400 ] 1846 10.364% 95.756% ## ( 4400, 6600 ] 664 3.728% 99.483% # ( 6600, 9900 ] 72 0.404% 99.888% ( 9900, 14000 ] 11 0.062% 99.949% ( 14000, 22000 ] 4 0.022% 99.972% ( 22000, 33000 ] 4 0.022% 99.994% ( 33000, 50000 ] 3 0.017% 100.011% ** DB Stats ** Uptime(secs): 9.5 total, 9.3 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 103.80 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.400 H:M:S, 57.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 105.53 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:5.400 H:M:S, 57.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 7 num-running-flushes: 1 2024/10/16-12:07:24 ... thread 0: (1000000,2000000) ops and (290803.3,156708.8) ops/second in (3.438751,12.762529) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/11 2.94 GB 163.6 1.5 0.0 1.5 3.3 1.8 0.0 1.8 85.9 193.2 17.60 11.32 31 0.568 1535K 1740 0.0 0.0 L1 7/7 836.70 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 19/0 988.16 MB 1.0 2.1 1.2 0.9 2.1 1.2 0.3 1.7 99.2 98.1 22.10 10.30 10 2.210 751K 0 0.0 0.0 L4 172/0 9.58 GB 1.0 6.1 4.0 2.1 6.0 3.9 0.0 1.5 112.8 111.4 55.15 29.24 39 1.414 2761K 6698 0.0 0.0 L5 1128/0 92.77 GB 0.9 0.0 0.0 0.0 0.0 0.0 3.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1344/18 107.07 GB 0.0 9.7 5.2 4.5 11.4 6.9 4.2 6.2 104.7 123.5 94.84 50.86 80 1.186 5048K 8438 0.0 0.0 Int 0/0 0.00 KB 0.0 9.7 5.2 4.5 11.4 6.9 4.2 6.2 104.7 123.5 94.84 50.86 80 1.186 5048K 8438 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 9.7 5.2 4.5 9.6 5.1 0.0 0.0 117.4 116.2 84.53 45.50 54 1.565 5048K 8438 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 183.3 10.31 5.37 26 0.397 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 12.9 total, 12.8 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 11.44 GB write, 906.37 MB/s write, 9.69 GB read, 768.34 MB/s read, 94.8 seconds Interval compaction: 11.44 GB write, 917.45 MB/s write, 9.69 GB read, 777.73 MB/s read, 94.8 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 26, pending-compaction-bytes-delays: 4, pending-compaction-bytes-stops: 0, total-delays: 4, total-stops: 26, interval: 29 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 692226 Average: 12.5476 StdDev: 126.39 Min: 0 Median: 1.7303 Max: 21325 Percentiles: P50: 1.73 P75: 2.58 P99: 223.69 P99.9: 1368.79 P99.99: 2970.95 ------------------------------------------------------ [ 0, 1 ] 72507 10.474% 10.474% ## ( 1, 2 ] 374642 54.121% 64.596% ########### ( 2, 3 ] 123221 17.801% 82.397% #### ( 3, 4 ] 47259 6.827% 89.224% # ( 4, 6 ] 46652 6.739% 95.963% # ( 6, 10 ] 10809 1.561% 97.525% ( 10, 15 ] 1604 0.232% 97.756% ( 15, 22 ] 236 0.034% 97.790% ( 22, 34 ] 23 0.003% 97.794% ( 34, 51 ] 17 0.002% 97.796% ( 51, 76 ] 22 0.003% 97.799% ( 76, 110 ] 1301 0.188% 97.987% ( 110, 170 ] 5035 0.727% 98.715% ( 170, 250 ] 2944 0.425% 99.140% ( 250, 380 ] 1657 0.239% 99.379% ( 380, 580 ] 1618 0.234% 99.613% ( 580, 870 ] 1177 0.170% 99.783% ( 870, 1300 ] 763 0.110% 99.893% ( 1300, 1900 ] 408 0.059% 99.952% ( 1900, 2900 ] 250 0.036% 99.988% ( 2900, 4400 ] 249 0.036% 100.024% ( 4400, 6600 ] 70 0.010% 100.034% ( 6600, 9900 ] 12 0.002% 100.036% ( 9900, 14000 ] 2 0.000% 100.036% ( 14000, 22000 ] 1 0.000% 100.037% ** Level 1 read latency histogram (micros): Count: 70171 Average: 10.9882 StdDev: 125.83 Min: 1 Median: 2.8511 Max: 7139 Percentiles: P50: 2.85 P75: 4.46 P99: 128.07 P99.9: 1585.09 P99.99: 5811.60 ------------------------------------------------------ [ 0, 1 ] 335 0.477% 0.477% ( 1, 2 ] 17793 25.357% 25.834% ##### ( 2, 3 ] 19924 28.393% 54.228% ###### ( 3, 4 ] 10928 15.573% 69.801% ### ( 4, 6 ] 15866 22.610% 92.411% ##### ( 6, 10 ] 3543 5.049% 97.460% # ( 10, 15 ] 624 0.889% 98.350% ( 15, 22 ] 295 0.420% 98.770% ( 22, 34 ] 19 0.027% 98.797% ( 34, 51 ] 1 0.001% 98.799% ( 51, 76 ] 6 0.009% 98.807% ( 76, 110 ] 63 0.090% 98.897% ( 110, 170 ] 240 0.342% 99.239% ( 170, 250 ] 125 0.178% 99.417% ( 250, 380 ] 92 0.131% 99.548% ( 380, 580 ] 99 0.141% 99.689% ( 580, 870 ] 80 0.114% 99.803% ( 870, 1300 ] 55 0.078% 99.882% ( 1300, 1900 ] 27 0.038% 99.920% ( 1900, 2900 ] 25 0.036% 99.956% ( 2900, 4400 ] 15 0.021% 99.977% ( 4400, 6600 ] 14 0.020% 99.997% ( 6600, 9900 ] 3 0.004% 100.001% ** Level 2 read latency histogram (micros): Count: 159217 Average: 9.9138 StdDev: 131.42 Min: 1 Median: 2.0444 Max: 13246 Percentiles: P50: 2.04 P75: 3.18 P99: 11.18 P99.9: 1820.70 P99.99: 3894.11 ------------------------------------------------------ [ 0, 1 ] 3440 2.161% 2.161% ( 1, 2 ] 74460 46.766% 48.927% ######### ( 2, 3 ] 38497 24.179% 73.106% ##### ( 3, 4 ] 17056 10.712% 83.818% ## ( 4, 6 ] 18801 11.808% 95.627% ## ( 6, 10 ] 5180 3.253% 98.880% # ( 10, 15 ] 811 0.509% 99.390% ( 15, 22 ] 133 0.084% 99.473% ( 22, 34 ] 21 0.013% 99.486% ( 34, 51 ] 11 0.007% 99.493% ( 51, 76 ] 24 0.015% 99.508% ( 76, 110 ] 7 0.004% 99.513% ( 110, 170 ] 9 0.006% 99.518% ( 170, 250 ] 41 0.026% 99.544% ( 250, 380 ] 80 0.050% 99.594% ( 380, 580 ] 118 0.074% 99.668% ( 580, 870 ] 138 0.087% 99.755% ( 870, 1300 ] 144 0.090% 99.845% ( 1300, 1900 ] 100 0.063% 99.908% ( 1900, 2900 ] 87 0.055% 99.963% ( 2900, 4400 ] 65 0.041% 100.004% ( 4400, 6600 ] 23 0.014% 100.018% ( 6600, 9900 ] 3 0.002% 100.020% ( 9900, 14000 ] 1 0.001% 100.021% ** Level 3 read latency histogram (micros): Count: 1012610 Average: 11.6201 StdDev: 156.90 Min: 1 Median: 1.9918 Max: 30490 Percentiles: P50: 1.99 P75: 3.00 P99: 9.86 P99.9: 921.32 P99.99: 1367.84 ------------------------------------------------------ [ 0, 1 ] 17553 1.733% 1.733% ( 1, 2 ] 492809 48.667% 50.401% ########## ( 2, 3 ] 250301 24.718% 75.119% ##### ( 3, 4 ] 103953 10.266% 85.385% ## ( 4, 6 ] 111657 11.027% 96.412% ## ( 6, 10 ] 27167 2.683% 99.094% # ( 10, 15 ] 4249 0.420% 99.514% ( 15, 22 ] 555 0.055% 99.569% ( 22, 34 ] 68 0.007% 99.576% ( 34, 51 ] 67 0.007% 99.582% ( 51, 76 ] 87 0.009% 99.591% ( 76, 110 ] 126 0.012% 99.603% ( 110, 170 ] 385 0.038% 99.641% ( 170, 250 ] 318 0.031% 99.673% ( 250, 380 ] 430 0.042% 99.715% ( 380, 580 ] 716 0.071% 99.786% ( 580, 870 ] 1043 0.103% 99.889% ( 870, 1300 ] 950 0.094% 99.983% ( 1300, 1900 ] 661 0.065% 100.048% ( 1900, 2900 ] 625 0.062% 100.110% ( 2900, 4400 ] 713 0.070% 100.180% ( 4400, 6600 ] 203 0.020% 100.200% ( 6600, 9900 ] 32 0.003% 100.203% ( 14000, 22000 ] 1 0.000% 100.203% ( 22000, 33000 ] 2 0.000% 100.204% ** Level 4 read latency histogram (micros): Count: 559370 Average: 24.9281 StdDev: 239.20 Min: 1 Median: 1.9989 Max: 36680 Percentiles: P50: 2.00 P75: 3.05 P99: 561.60 P99.9: 2489.73 P99.99: 3304.28 ------------------------------------------------------ [ 0, 1 ] 9528 1.703% 1.703% ( 1, 2 ] 270468 48.352% 50.056% ########## ( 2, 3 ] 136573 24.416% 74.471% ##### ( 3, 4 ] 55475 9.917% 84.389% ## ( 4, 6 ] 56973 10.185% 94.574% ## ( 6, 10 ] 14850 2.655% 97.228% # ( 10, 15 ] 2185 0.391% 97.619% ( 15, 22 ] 317 0.057% 97.676% ( 22, 34 ] 32 0.006% 97.681% ( 34, 51 ] 31 0.006% 97.687% ( 51, 76 ] 68 0.012% 97.699% ( 76, 110 ] 794 0.142% 97.841% ( 110, 170 ] 2492 0.446% 98.287% ( 170, 250 ] 1472 0.263% 98.550% ( 250, 380 ] 1120 0.200% 98.750% ( 380, 580 ] 1540 0.275% 99.025% ( 580, 870 ] 1767 0.316% 99.341% ( 870, 1300 ] 1658 0.296% 99.638% ( 1300, 1900 ] 1033 0.185% 99.822% ( 1900, 2900 ] 737 0.132% 99.954% ( 2900, 4400 ] 746 0.133% 100.087% ( 4400, 6600 ] 204 0.036% 100.124% ( 6600, 9900 ] 37 0.007% 100.131% ( 9900, 14000 ] 2 0.000% 100.131% ( 14000, 22000 ] 2 0.000% 100.131% ( 22000, 33000 ] 5 0.001% 100.132% ( 33000, 50000 ] 1 0.000% 100.132% ** Level 5 read latency histogram (micros): Count: 56605 Average: 626.2255 StdDev: 1021.91 Min: 75 Median: 231.5824 Max: 36911 Percentiles: P50: 231.58 P75: 661.75 P99: 5031.71 P99.9: 8222.27 P99.99: 20679.00 ------------------------------------------------------ ( 51, 76 ] 2 0.004% 0.004% ( 76, 110 ] 4776 8.437% 8.441% ## ( 110, 170 ] 16889 29.837% 38.278% ###### ( 170, 250 ] 8620 15.228% 53.506% ### ( 250, 380 ] 5023 8.874% 62.380% ## ( 380, 580 ] 5760 10.176% 72.555% ## ( 580, 870 ] 4909 8.672% 81.228% ## ( 870, 1300 ] 3845 6.793% 88.020% # ( 1300, 1900 ] 2347 4.146% 92.167% # ( 1900, 2900 ] 1806 3.191% 95.357% # ( 2900, 4400 ] 1871 3.305% 98.663% # ( 4400, 6600 ] 665 1.175% 99.837% ( 6600, 9900 ] 72 0.127% 99.965% ( 9900, 14000 ] 11 0.019% 99.984% ( 14000, 22000 ] 4 0.007% 99.991% ( 22000, 33000 ] 4 0.007% 99.998% ( 33000, 50000 ] 3 0.005% 100.004% ** DB Stats ** Uptime(secs): 12.9 total, 3.4 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 152.35 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.400 H:M:S, 41.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 286.21 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/10/16-12:07:30 ... thread 0: (1000000,3000000) ops and (170197.7,160961.1) ops/second in (5.875520,18.638049) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/14 3.86 GB 125.8 2.7 0.0 2.7 5.5 2.8 0.0 2.0 109.2 220.8 25.37 18.64 51 0.498 2814K 2983 0.0 0.0 L1 7/7 836.70 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 19/0 988.16 MB 1.0 2.1 1.2 0.9 2.1 1.2 0.3 1.7 99.2 98.1 22.10 10.30 10 2.210 751K 0 0.0 0.0 L4 172/0 9.58 GB 1.0 6.1 4.0 2.1 6.0 3.9 0.0 1.5 112.8 111.4 55.15 29.24 39 1.414 2761K 6698 0.0 0.0 L5 1128/0 92.77 GB 0.9 0.0 0.0 0.0 0.0 0.0 3.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1344/21 107.99 GB 0.0 10.9 5.2 5.8 13.6 7.8 4.2 4.9 109.0 135.6 102.62 58.19 100 1.026 6327K 9681 0.0 0.0 Int 0/0 0.00 KB 0.0 10.9 5.2 5.8 13.6 7.8 4.2 4.9 109.0 135.6 102.62 58.19 100 1.026 6327K 9681 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 10.9 5.2 5.8 10.8 5.1 0.0 0.0 125.2 124.0 89.32 50.20 59 1.514 6327K 9681 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 213.1 13.30 7.99 41 0.324 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.8 total, 18.6 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 13.59 GB write, 740.29 MB/s write, 10.92 GB read, 595.20 MB/s read, 102.6 seconds Interval compaction: 13.59 GB write, 746.49 MB/s write, 10.92 GB read, 600.18 MB/s read, 102.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 9, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 9, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 26, pending-compaction-bytes-delays: 4, pending-compaction-bytes-stops: 0, total-delays: 13, total-stops: 26, interval: 38 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1040280 Average: 13.6502 StdDev: 109.06 Min: 0 Median: 1.6411 Max: 21325 Percentiles: P50: 1.64 P75: 2.38 P99: 242.43 P99.9: 1057.31 P99.99: 2122.11 ------------------------------------------------------ [ 0, 1 ] 159147 15.298% 15.298% ### ( 1, 2 ] 563087 54.128% 69.427% ########### ( 2, 3 ] 153503 14.756% 84.183% ### ( 3, 4 ] 56255 5.408% 89.590% # ( 4, 6 ] 53388 5.132% 94.723% # ( 6, 10 ] 11813 1.136% 95.858% ( 10, 15 ] 1941 0.187% 96.045% ( 15, 22 ] 286 0.027% 96.072% ( 22, 34 ] 25 0.002% 96.075% ( 34, 51 ] 19 0.002% 96.076% ( 51, 76 ] 26 0.002% 96.079% ( 76, 110 ] 5000 0.481% 96.560% ( 110, 170 ] 18028 1.733% 98.293% ( 170, 250 ] 8130 0.782% 99.074% ( 250, 380 ] 3555 0.342% 99.416% ( 380, 580 ] 3082 0.296% 99.712% ( 580, 870 ] 1602 0.154% 99.866% ( 870, 1300 ] 812 0.078% 99.944% ( 1300, 1900 ] 420 0.040% 99.985% ( 1900, 2900 ] 261 0.025% 100.010% ( 2900, 4400 ] 253 0.024% 100.034% ( 4400, 6600 ] 70 0.007% 100.041% ( 6600, 9900 ] 12 0.001% 100.042% ( 9900, 14000 ] 2 0.000% 100.042% ( 14000, 22000 ] 1 0.000% 100.042% ** Level 1 read latency histogram (micros): Count: 161801 Average: 8.2450 StdDev: 84.99 Min: 1 Median: 3.2067 Max: 7139 Percentiles: P50: 3.21 P75: 4.42 P99: 110.30 P99.9: 808.45 P99.99: 4181.99 ------------------------------------------------------ [ 0, 1 ] 607 0.375% 0.375% ( 1, 2 ] 26693 16.497% 16.873% ### ( 2, 3 ] 45001 27.813% 44.685% ###### ( 3, 4 ] 41598 25.709% 70.394% ##### ( 4, 6 ] 35654 22.036% 92.430% #### ( 6, 10 ] 7701 4.760% 97.190% # ( 10, 15 ] 1918 1.185% 98.375% ( 15, 22 ] 688 0.425% 98.800% ( 22, 34 ] 48 0.030% 98.830% ( 34, 51 ] 4 0.002% 98.833% ( 51, 76 ] 6 0.004% 98.836% ( 76, 110 ] 261 0.161% 98.998% ( 110, 170 ] 808 0.499% 99.497% ( 170, 250 ] 279 0.172% 99.669% ( 250, 380 ] 153 0.095% 99.764% ( 380, 580 ] 143 0.088% 99.852% ( 580, 870 ] 98 0.061% 99.913% ( 870, 1300 ] 58 0.036% 99.949% ( 1300, 1900 ] 27 0.017% 99.965% ( 1900, 2900 ] 27 0.017% 99.982% ( 2900, 4400 ] 15 0.009% 99.991% ( 4400, 6600 ] 14 0.009% 100.000% ( 6600, 9900 ] 3 0.002% 100.002% ** Level 2 read latency histogram (micros): Count: 159217 Average: 9.9138 StdDev: 131.42 Min: 1 Median: 2.0444 Max: 13246 Percentiles: P50: 2.04 P75: 3.18 P99: 11.18 P99.9: 1820.70 P99.99: 3894.11 ------------------------------------------------------ [ 0, 1 ] 3440 2.161% 2.161% ( 1, 2 ] 74460 46.766% 48.927% ######### ( 2, 3 ] 38497 24.179% 73.106% ##### ( 3, 4 ] 17056 10.712% 83.818% ## ( 4, 6 ] 18801 11.808% 95.627% ## ( 6, 10 ] 5180 3.253% 98.880% # ( 10, 15 ] 811 0.509% 99.390% ( 15, 22 ] 133 0.084% 99.473% ( 22, 34 ] 21 0.013% 99.486% ( 34, 51 ] 11 0.007% 99.493% ( 51, 76 ] 24 0.015% 99.508% ( 76, 110 ] 7 0.004% 99.513% ( 110, 170 ] 9 0.006% 99.518% ( 170, 250 ] 41 0.026% 99.544% ( 250, 380 ] 80 0.050% 99.594% ( 380, 580 ] 118 0.074% 99.668% ( 580, 870 ] 138 0.087% 99.755% ( 870, 1300 ] 144 0.090% 99.845% ( 1300, 1900 ] 100 0.063% 99.908% ( 1900, 2900 ] 87 0.055% 99.963% ( 2900, 4400 ] 65 0.041% 100.004% ( 4400, 6600 ] 23 0.014% 100.018% ( 6600, 9900 ] 3 0.002% 100.020% ( 9900, 14000 ] 1 0.001% 100.021% ** Level 3 read latency histogram (micros): Count: 1013937 Average: 11.8397 StdDev: 156.99 Min: 1 Median: 1.9931 Max: 30490 Percentiles: P50: 1.99 P75: 3.00 P99: 10.42 P99.9: 923.27 P99.99: 1370.23 ------------------------------------------------------ [ 0, 1 ] 17553 1.731% 1.731% ( 1, 2 ] 492809 48.604% 50.335% ########## ( 2, 3 ] 250301 24.686% 75.021% ##### ( 3, 4 ] 103953 10.252% 85.273% ## ( 4, 6 ] 111657 11.012% 96.285% ## ( 6, 10 ] 27167 2.679% 98.965% # ( 10, 15 ] 4249 0.419% 99.384% ( 15, 22 ] 555 0.055% 99.439% ( 22, 34 ] 68 0.007% 99.445% ( 34, 51 ] 67 0.007% 99.452% ( 51, 76 ] 88 0.009% 99.461% ( 76, 110 ] 386 0.038% 99.499% ( 110, 170 ] 1067 0.105% 99.604% ( 170, 250 ] 548 0.054% 99.658% ( 250, 380 ] 503 0.050% 99.707% ( 380, 580 ] 768 0.076% 99.783% ( 580, 870 ] 1066 0.105% 99.888% ( 870, 1300 ] 953 0.094% 99.982% ( 1300, 1900 ] 663 0.065% 100.048% ( 1900, 2900 ] 626 0.062% 100.109% ( 2900, 4400 ] 713 0.070% 100.180% ( 4400, 6600 ] 203 0.020% 100.200% ( 6600, 9900 ] 32 0.003% 100.203% ( 14000, 22000 ] 1 0.000% 100.203% ( 22000, 33000 ] 2 0.000% 100.203% ** Level 4 read latency histogram (micros): Count: 572918 Average: 28.2722 StdDev: 238.03 Min: 1 Median: 2.0467 Max: 36680 Percentiles: P50: 2.05 P75: 3.23 P99: 570.15 P99.9: 2474.44 P99.99: 3303.03 ------------------------------------------------------ [ 0, 1 ] 9528 1.663% 1.663% ( 1, 2 ] 270541 47.222% 48.885% ######### ( 2, 3 ] 136810 23.880% 72.764% ##### ( 3, 4 ] 55646 9.713% 82.477% ## ( 4, 6 ] 57163 9.978% 92.454% ## ( 6, 10 ] 14876 2.597% 95.051% # ( 10, 15 ] 2194 0.383% 95.434% ( 15, 22 ] 318 0.056% 95.489% ( 22, 34 ] 32 0.006% 95.495% ( 34, 51 ] 31 0.005% 95.500% ( 51, 76 ] 70 0.012% 95.513% ( 76, 110 ] 3129 0.546% 96.059% ( 110, 170 ] 9241 1.613% 97.672% ( 170, 250 ] 3690 0.644% 98.316% ( 250, 380 ] 1890 0.330% 98.646% ( 380, 580 ] 2135 0.373% 99.018% ( 580, 870 ] 1915 0.334% 99.353% ( 870, 1300 ] 1676 0.293% 99.645% ( 1300, 1900 ] 1035 0.181% 99.826% ( 1900, 2900 ] 740 0.129% 99.955% ( 2900, 4400 ] 747 0.130% 100.085% ( 4400, 6600 ] 204 0.036% 100.121% ( 6600, 9900 ] 37 0.006% 100.127% ( 9900, 14000 ] 2 0.000% 100.128% ( 14000, 22000 ] 2 0.000% 100.128% ( 22000, 33000 ] 5 0.001% 100.129% ( 33000, 50000 ] 1 0.000% 100.129% ** Level 5 read latency histogram (micros): Count: 138390 Average: 360.6490 StdDev: 696.20 Min: 71 Median: 159.5934 Max: 36911 Percentiles: P50: 159.59 P75: 272.71 P99: 3896.62 P99.9: 6436.60 P99.99: 11078.19 ------------------------------------------------------ ( 51, 76 ] 11 0.008% 0.008% ( 76, 110 ] 19903 14.382% 14.390% ### ( 110, 170 ] 59622 43.083% 57.472% ######### ( 170, 250 ] 22497 16.256% 73.729% ### ( 250, 380 ] 10071 7.277% 81.006% # ( 380, 580 ] 9523 6.881% 87.887% # ( 580, 870 ] 5930 4.285% 92.172% # ( 870, 1300 ] 3997 2.888% 95.060% # ( 1300, 1900 ] 2368 1.711% 96.771% ( 1900, 2900 ] 1837 1.327% 98.099% ( 2900, 4400 ] 1877 1.356% 99.455% ( 4400, 6600 ] 665 0.481% 99.936% ( 6600, 9900 ] 72 0.052% 99.988% ( 9900, 14000 ] 11 0.008% 99.996% ( 14000, 22000 ] 4 0.003% 99.999% ( 22000, 33000 ] 4 0.003% 100.001% ( 33000, 50000 ] 3 0.002% 100.004% ** DB Stats ** Uptime(secs): 18.8 total, 5.9 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 157.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.714 H:M:S, 41.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 167.51 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.314 H:M:S, 39.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/10/16-12:07:38 ... thread 0: (1000000,4000000) ops and (121595.2,148908.9) ops/second in (8.224011,26.862060) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/12 3.74 GB 12.6 4.8 0.0 4.8 8.5 3.7 0.0 2.3 128.8 229.2 38.13 29.82 69 0.553 4987K 14K 0.0 0.0 L1 10/10 1.14 GB 0.0 1.9 1.1 0.8 1.9 1.1 0.0 1.7 89.0 88.3 22.00 12.22 1 22.004 1988K 14K 0.0 0.0 L3 22/2 1.09 GB 1.0 3.7 1.8 1.8 3.6 1.8 0.4 2.0 118.8 117.8 31.66 17.57 19 1.666 2083K 6176 0.0 0.0 L4 173/5 9.51 GB 1.0 7.4 4.6 2.8 7.3 4.5 0.0 1.6 119.4 117.7 63.26 35.36 46 1.375 3258K 6698 0.0 0.0 L5 1136/0 93.44 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1355/29 108.92 GB 0.0 17.8 7.5 10.3 21.3 11.1 5.0 5.7 117.3 141.0 155.05 94.97 135 1.149 12M 41K 0.0 0.0 Int 0/0 0.00 KB 0.0 17.8 7.5 10.3 21.3 11.1 5.0 5.7 117.3 141.0 155.05 94.97 135 1.149 12M 41K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 17.8 7.5 10.3 17.6 7.3 0.0 0.0 132.0 130.8 137.73 84.13 79 1.743 12M 41K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 221.9 17.32 10.84 56 0.309 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 27.0 total, 26.9 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 21.35 GB write, 808.98 MB/s write, 17.76 GB read, 672.91 MB/s read, 155.1 seconds Interval compaction: 21.35 GB write, 813.68 MB/s write, 17.76 GB read, 676.82 MB/s read, 155.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 12, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 12, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 28, pending-compaction-bytes-delays: 15, pending-compaction-bytes-stops: 0, total-delays: 27, total-stops: 28, interval: 54 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1626811 Average: 13.3288 StdDev: 100.15 Min: 0 Median: 1.6418 Max: 21325 Percentiles: P50: 1.64 P75: 2.38 P99: 241.25 P99.9: 914.92 P99.99: 1765.68 ------------------------------------------------------ [ 0, 1 ] 249253 15.322% 15.322% ### ( 1, 2 ] 879039 54.034% 69.356% ########### ( 2, 3 ] 239433 14.718% 84.074% ### ( 3, 4 ] 88860 5.462% 89.536% # ( 4, 6 ] 83997 5.163% 94.700% # ( 6, 10 ] 16431 1.010% 95.710% ( 10, 15 ] 2744 0.169% 95.878% ( 15, 22 ] 382 0.023% 95.902% ( 22, 34 ] 31 0.002% 95.904% ( 34, 51 ] 29 0.002% 95.905% ( 51, 76 ] 34 0.002% 95.907% ( 76, 110 ] 8437 0.519% 96.426% ( 110, 170 ] 30264 1.860% 98.286% ( 170, 250 ] 13037 0.801% 99.088% ( 250, 380 ] 5724 0.352% 99.440% ( 380, 580 ] 4887 0.300% 99.740% ( 580, 870 ] 2483 0.153% 99.893% ( 870, 1300 ] 1160 0.071% 99.964% ( 1300, 1900 ] 548 0.034% 99.998% ( 1900, 2900 ] 338 0.021% 100.018% ( 2900, 4400 ] 301 0.019% 100.037% ( 4400, 6600 ] 83 0.005% 100.042% ( 6600, 9900 ] 14 0.001% 100.043% ( 9900, 14000 ] 4 0.000% 100.043% ( 14000, 22000 ] 1 0.000% 100.043% ** Level 1 read latency histogram (micros): Count: 374079 Average: 8.4958 StdDev: 88.13 Min: 1 Median: 2.7544 Max: 7256 Percentiles: P50: 2.75 P75: 3.87 P99: 95.81 P99.9: 988.11 P99.99: 2621.33 ------------------------------------------------------ [ 0, 1 ] 7595 2.030% 2.030% ( 1, 2 ] 110009 29.408% 31.438% ###### ( 2, 3 ] 92036 24.603% 56.042% ##### ( 3, 4 ] 81548 21.800% 77.841% #### ( 4, 6 ] 60725 16.233% 94.075% ### ( 6, 10 ] 13865 3.706% 97.781% # ( 10, 15 ] 3298 0.882% 98.663% ( 15, 22 ] 890 0.238% 98.900% ( 22, 34 ] 69 0.018% 98.919% ( 34, 51 ] 18 0.005% 98.924% ( 51, 76 ] 23 0.006% 98.930% ( 76, 110 ] 450 0.120% 99.050% ( 110, 170 ] 1422 0.380% 99.430% ( 170, 250 ] 579 0.155% 99.585% ( 250, 380 ] 402 0.107% 99.693% ( 380, 580 ] 383 0.102% 99.795% ( 580, 870 ] 327 0.087% 99.882% ( 870, 1300 ] 240 0.064% 99.947% ( 1300, 1900 ] 102 0.027% 99.974% ( 1900, 2900 ] 84 0.022% 99.996% ( 2900, 4400 ] 57 0.015% 100.011% ( 4400, 6600 ] 24 0.006% 100.018% ( 6600, 9900 ] 4 0.001% 100.019% ** Level 2 read latency histogram (micros): Count: 159217 Average: 9.9138 StdDev: 131.42 Min: 1 Median: 2.0444 Max: 13246 Percentiles: P50: 2.04 P75: 3.18 P99: 11.18 P99.9: 1820.70 P99.99: 3894.11 ------------------------------------------------------ [ 0, 1 ] 3440 2.161% 2.161% ( 1, 2 ] 74460 46.766% 48.927% ######### ( 2, 3 ] 38497 24.179% 73.106% ##### ( 3, 4 ] 17056 10.712% 83.818% ## ( 4, 6 ] 18801 11.808% 95.627% ## ( 6, 10 ] 5180 3.253% 98.880% # ( 10, 15 ] 811 0.509% 99.390% ( 15, 22 ] 133 0.084% 99.473% ( 22, 34 ] 21 0.013% 99.486% ( 34, 51 ] 11 0.007% 99.493% ( 51, 76 ] 24 0.015% 99.508% ( 76, 110 ] 7 0.004% 99.513% ( 110, 170 ] 9 0.006% 99.518% ( 170, 250 ] 41 0.026% 99.544% ( 250, 380 ] 80 0.050% 99.594% ( 380, 580 ] 118 0.074% 99.668% ( 580, 870 ] 138 0.087% 99.755% ( 870, 1300 ] 144 0.090% 99.845% ( 1300, 1900 ] 100 0.063% 99.908% ( 1900, 2900 ] 87 0.055% 99.963% ( 2900, 4400 ] 65 0.041% 100.004% ( 4400, 6600 ] 23 0.014% 100.018% ( 6600, 9900 ] 3 0.002% 100.020% ( 9900, 14000 ] 1 0.001% 100.021% ** Level 3 read latency histogram (micros): Count: 1414350 Average: 10.3536 StdDev: 140.41 Min: 0 Median: 1.9724 Max: 30490 Percentiles: P50: 1.97 P75: 2.96 P99: 9.95 P99.9: 818.98 P99.99: 1269.97 ------------------------------------------------------ [ 0, 1 ] 26804 1.895% 1.895% ( 1, 2 ] 699718 49.473% 51.368% ########## ( 2, 3 ] 346729 24.515% 75.883% ##### ( 3, 4 ] 141661 10.016% 85.899% ## ( 4, 6 ] 149594 10.577% 96.476% ## ( 6, 10 ] 36194 2.559% 99.035% # ( 10, 15 ] 5355 0.379% 99.414% ( 15, 22 ] 694 0.049% 99.463% ( 22, 34 ] 78 0.006% 99.468% ( 34, 51 ] 121 0.009% 99.477% ( 51, 76 ] 135 0.010% 99.486% ( 76, 110 ] 567 0.040% 99.526% ( 110, 170 ] 1566 0.111% 99.637% ( 170, 250 ] 813 0.057% 99.694% ( 250, 380 ] 774 0.055% 99.749% ( 380, 580 ] 1064 0.075% 99.824% ( 580, 870 ] 1298 0.092% 99.916% ( 870, 1300 ] 1123 0.079% 99.996% ( 1300, 1900 ] 765 0.054% 100.050% ( 1900, 2900 ] 735 0.052% 100.102% ( 2900, 4400 ] 789 0.056% 100.157% ( 4400, 6600 ] 214 0.015% 100.173% ( 6600, 9900 ] 35 0.002% 100.175% ( 9900, 14000 ] 1 0.000% 100.175% ( 14000, 22000 ] 1 0.000% 100.175% ( 22000, 33000 ] 2 0.000% 100.175% ** Level 4 read latency histogram (micros): Count: 768423 Average: 27.8831 StdDev: 220.97 Min: 0 Median: 2.0349 Max: 36680 Percentiles: P50: 2.03 P75: 3.26 P99: 549.90 P99.9: 2156.72 P99.99: 2984.01 ------------------------------------------------------ [ 0, 1 ] 14100 1.835% 1.835% ( 1, 2 ] 363850 47.350% 49.185% ######### ( 2, 3 ] 179288 23.332% 72.517% ##### ( 3, 4 ] 73196 9.525% 82.043% ## ( 4, 6 ] 74771 9.730% 91.773% ## ( 6, 10 ] 19896 2.589% 94.362% # ( 10, 15 ] 2774 0.361% 94.723% ( 15, 22 ] 400 0.052% 94.775% ( 22, 34 ] 41 0.005% 94.781% ( 34, 51 ] 49 0.006% 94.787% ( 51, 76 ] 88 0.011% 94.798% ( 76, 110 ] 4986 0.649% 95.447% ( 110, 170 ] 15196 1.978% 97.425% ( 170, 250 ] 6055 0.788% 98.213% ( 250, 380 ] 3209 0.418% 98.630% ( 380, 580 ] 3344 0.435% 99.066% ( 580, 870 ] 2781 0.362% 99.428% ( 870, 1300 ] 2168 0.282% 99.710% ( 1300, 1900 ] 1241 0.161% 99.871% ( 1900, 2900 ] 867 0.113% 99.984% ( 2900, 4400 ] 842 0.110% 100.094% ( 4400, 6600 ] 222 0.029% 100.122% ( 6600, 9900 ] 39 0.005% 100.128% ( 9900, 14000 ] 6 0.001% 100.128% ( 14000, 22000 ] 2 0.000% 100.129% ( 22000, 33000 ] 5 0.001% 100.129% ( 33000, 50000 ] 1 0.000% 100.129% ** Level 5 read latency histogram (micros): Count: 217722 Average: 327.0931 StdDev: 602.43 Min: 71 Median: 158.9255 Max: 36911 Percentiles: P50: 158.93 P75: 256.76 P99: 3480.67 P99.9: 6235.15 P99.99: 9620.65 ------------------------------------------------------ ( 51, 76 ] 16 0.007% 0.007% ( 76, 110 ] 31378 14.412% 14.419% ### ( 110, 170 ] 95002 43.635% 58.054% ######### ( 170, 250 ] 36036 16.551% 74.605% ### ( 250, 380 ] 16531 7.593% 82.198% ## ( 380, 580 ] 15321 7.037% 89.235% # ( 580, 870 ] 9353 4.296% 93.531% # ( 870, 1300 ] 5790 2.659% 96.190% # ( 1300, 1900 ] 3033 1.393% 97.583% ( 1900, 2900 ] 2237 1.027% 98.611% ( 2900, 4400 ] 2190 1.006% 99.616% ( 4400, 6600 ] 740 0.340% 99.956% ( 6600, 9900 ] 80 0.037% 99.993% ( 9900, 14000 ] 11 0.005% 99.998% ( 14000, 22000 ] 4 0.002% 100.000% ( 22000, 33000 ] 4 0.002% 100.002% ( 33000, 50000 ] 3 0.001% 100.003% ** DB Stats ** Uptime(secs): 27.0 total, 8.2 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 145.70 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:12.192 H:M:S, 45.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 119.67 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:4.478 H:M:S, 54.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 4 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 244.118 micros/op 16381 ops/sec 60.128 seconds 984996 operations; 15.9 MB/s (241999 of 241999 found) Microseconds per read: Count: 984996 Average: 244.1213 StdDev: 654.41 Min: 2 Median: 169.2582 Max: 69446 Percentiles: P50: 169.26 P75: 231.84 P99: 1285.24 P99.9: 10651.30 P99.99: 26744.31 ------------------------------------------------------ ( 1, 2 ] 7 0.001% 0.001% ( 2, 3 ] 48 0.005% 0.006% ( 3, 4 ] 58 0.006% 0.011% ( 4, 6 ] 95 0.010% 0.021% ( 6, 10 ] 454 0.046% 0.067% ( 10, 15 ] 4108 0.417% 0.484% ( 15, 22 ] 5700 0.579% 1.063% ( 22, 34 ] 9353 0.950% 2.012% ( 34, 51 ] 3762 0.382% 2.394% ( 51, 76 ] 361 0.037% 2.431% ( 76, 110 ] 22896 2.324% 4.756% ( 110, 170 ] 451235 45.811% 50.566% ######### ( 170, 250 ] 311347 31.609% 82.175% ###### ( 250, 380 ] 97746 9.923% 92.099% ## ( 380, 580 ] 41980 4.262% 96.361% # ( 580, 870 ] 18174 1.845% 98.206% ( 870, 1300 ] 8100 0.822% 99.028% ( 1300, 1900 ] 3500 0.355% 99.384% ( 1900, 2900 ] 2222 0.226% 99.609% ( 2900, 4400 ] 1794 0.182% 99.791% ( 4400, 6600 ] 637 0.065% 99.856% ( 6600, 9900 ] 340 0.035% 99.890% ( 9900, 14000 ] 513 0.052% 99.943% ( 14000, 22000 ] 411 0.042% 99.984% ( 22000, 33000 ] 131 0.013% 99.998% ( 33000, 50000 ] 16 0.002% 99.999% ( 50000, 75000 ] 8 0.001% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 14585622 rocksdb.bloom.filter.full.positive COUNT : 1127105 rocksdb.bloom.filter.full.true.positive COUNT : 984580 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 416 rocksdb.memtable.miss COUNT : 984580 rocksdb.l0.hit COUNT : 42406 rocksdb.l1.hit COUNT : 10339 rocksdb.l2andup.hit COUNT : 931835 rocksdb.compaction.key.drop.new COUNT : 192243 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 4902565 rocksdb.number.keys.read COUNT : 984996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5059447080 rocksdb.bytes.read COUNT : 984996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1651 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 41412097 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 22887581519 rocksdb.compact.write.bytes COUNT : 22681955919 rocksdb.flush.write.bytes COUNT : 5020581815 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 25 rocksdb.number.superversion_acquires COUNT : 1036 rocksdb.number.superversion_releases COUNT : 1001 rocksdb.number.superversion_cleanups COUNT : 257 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 248 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 4979854336 rocksdb.memtable.garbage.bytes.at.flush COUNT : 1709056 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 26547944031 rocksdb.non.last.level.read.count COUNT : 6271711 rocksdb.block.checksum.compute.count COUNT : 6679553 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 168.326465 P95 : 515.173009 P99 : 1284.753528 P100 : 69444.000000 COUNT : 984996 SUM : 239577575 rocksdb.db.write.micros P50 : 1.545909 P95 : 4.727048 P99 : 11.004189 P100 : 840023.000000 COUNT : 4902565 SUM : 53894875 rocksdb.compaction.times.micros P50 : 1332758.620690 P95 : 2878125.000000 P99 : 22004231.000000 P100 : 22004231.000000 COUNT : 87 SUM : 148847212 rocksdb.compaction.times.cpu_micros P50 : 885757.575758 P95 : 1805500.000000 P99 : 10346000.000000 P100 : 12215463.000000 COUNT : 87 SUM : 94961178 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 824.024390 P95 : 90750.000000 P99 : 384696.000000 P100 : 384696.000000 COUNT : 71 SUM : 1049444 rocksdb.compaction.outfile.sync.micros P50 : 21142.857143 P95 : 312833.333333 P99 : 528471.428571 P100 : 789050.000000 COUNT : 253 SUM : 18337637 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 214.000000 P95 : 2842.105263 P99 : 5698.000000 P100 : 7400.000000 COUNT : 182 SUM : 138615 rocksdb.table.open.io.micros P50 : 8220.652174 P95 : 18345.341615 P99 : 21626.832298 P100 : 58058.000000 COUNT : 1651 SUM : 13704177 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.920261 P95 : 13.626932 P99 : 188.666953 P100 : 36684.000000 COUNT : 5540905 SUM : 85852256 rocksdb.read.block.get.micros P50 : 140.629902 P95 : 403.678101 P99 : 1218.271815 P100 : 36944.000000 COUNT : 1136985 SUM : 206606251 rocksdb.write.raw.block.micros P50 : 0.532345 P95 : 1.260534 P99 : 3.713255 P100 : 41027.000000 COUNT : 6709508 SUM : 17909606 rocksdb.numfiles.in.singlecompaction P50 : 2.937500 P95 : 5.360000 P99 : 14.400000 P100 : 18.000000 COUNT : 112 SUM : 396 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.564704 P95 : 1244.023488 P99 : 2658.288466 P100 : 839853.000000 COUNT : 229176 SUM : 41432418 rocksdb.sst.read.micros P50 : 2.104960 P95 : 167.187040 P99 : 490.737837 P100 : 36911.000000 COUNT : 6271728 SUM : 273984879 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 984996 SUM : 984996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 4902565 SUM : 5059447080 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 222592.592593 P95 : 806833.333333 P99 : 1082329.000000 P100 : 1082329.000000 COUNT : 71 SUM : 20406879 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 301 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0