[2024-12-10 14:12:13.223193] Starting SPDK v25.01-pre git sha1 02d0d9b38 / DPDK 24.03.0 initialization... [2024-12-10 14:12:13.223331] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1957580 ] Set seed to 1733836333220634 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-10 14:12:13.361461] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-10 14:12:13.463975] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-10 14:12:13.464033] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1957580' to capture a snapshot of events at runtime. [2024-12-10 14:12:13.464046] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-12-10 14:12:13.464060] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-12-10 14:12:13.464070] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1957580 for offline analysis/debug. [2024-12-10 14:12:13.465342] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Tue Dec 10 14:12:16 2024 CPU: 72 * Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz CPUCache: 25344 KB 2024/12/10-14:12:22 ... thread 0: (1000000,1000000) ops and (156090.7,156090.7) ops/second in (6.406532,6.406532) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/13 1.32 GB 25.1 0.0 0.0 0.0 0.9 0.9 0.0 1.0 0.0 154.6 5.69 2.65 11 0.517 0 0 0.0 0.0 L1 6/6 418.33 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 20/1 1020.05 MB 1.0 2.0 1.4 0.6 2.0 1.3 0.0 1.4 118.4 116.3 17.48 10.45 10 1.748 1628K 11K 0.0 0.0 L4 140/4 9.97 GB 1.0 3.4 1.7 1.7 3.2 1.5 0.4 1.8 123.2 115.3 28.34 18.17 15 1.890 728K 0 0.0 0.0 L5 144/0 11.38 GB 0.1 0.0 0.0 0.0 0.0 0.0 2.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 326/24 24.08 GB 0.0 5.4 3.1 2.3 6.0 3.7 2.7 7.0 107.9 120.0 51.52 31.26 36 1.431 2356K 11K 0.0 0.0 Int 0/0 0.00 KB 0.0 5.4 3.1 2.3 6.0 3.7 2.7 7.0 107.9 120.0 51.52 31.26 36 1.431 2356K 11K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 5.4 3.1 2.3 5.2 2.9 0.0 0.0 121.3 115.7 45.83 28.62 25 1.833 2356K 11K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 154.6 5.69 2.65 11 0.517 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 6.5 total, 6.4 interval Flush(GB): cumulative 0.860, interval 0.860 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.04 GB write, 954.58 MB/s write, 5.43 GB read, 858.74 MB/s read, 51.5 seconds Interval compaction: 6.04 GB write, 964.42 MB/s write, 5.43 GB read, 867.60 MB/s read, 51.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 14, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 14, interval: 14 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 108381 Average: 105.5479 StdDev: 770.26 Min: 1 Median: 2.3995 Max: 55031 Percentiles: P50: 2.40 P75: 4.04 P99: 1882.11 P99.9: 6314.19 P99.99: 30993.15 ------------------------------------------------------ [ 0, 1 ] 1814 1.674% 1.674% ( 1, 2 ] 41181 37.997% 39.670% ######## ( 2, 3 ] 28022 25.855% 65.525% ##### ( 3, 4 ] 10035 9.259% 74.784% ## ( 4, 6 ] 10455 9.647% 84.431% ## ( 6, 10 ] 2834 2.615% 87.046% # ( 10, 15 ] 680 0.627% 87.673% ( 15, 22 ] 123 0.113% 87.787% ( 22, 34 ] 10 0.009% 87.796% ( 34, 51 ] 6 0.006% 87.801% ( 51, 76 ] 15 0.014% 87.815% ( 76, 110 ] 12 0.011% 87.826% ( 110, 170 ] 245 0.226% 88.052% ( 170, 250 ] 1720 1.587% 89.639% ( 250, 380 ] 3109 2.869% 92.508% # ( 380, 580 ] 3509 3.238% 95.746% # ( 580, 870 ] 2445 2.256% 98.001% ( 870, 1300 ] 828 0.764% 98.765% ( 1300, 1900 ] 262 0.242% 99.007% ( 1900, 2900 ] 395 0.364% 99.372% ( 2900, 4400 ] 483 0.446% 99.817% ( 4400, 6600 ] 103 0.095% 99.912% ( 6600, 9900 ] 31 0.029% 99.941% ( 9900, 14000 ] 7 0.006% 99.947% ( 14000, 22000 ] 20 0.018% 99.966% ( 22000, 33000 ] 32 0.030% 99.995% ( 33000, 50000 ] 9 0.008% 100.004% ( 50000, 75000 ] 1 0.001% 100.005% ** Level 1 read latency histogram (micros): Count: 502 Average: 990.1853 StdDev: 1916.97 Min: 120 Median: 532.3810 Max: 35016 Percentiles: P50: 532.38 P75: 823.01 P99: 6258.15 P99.9: 35016.00 P99.99: 35016.00 ------------------------------------------------------ ( 110, 170 ] 8 1.594% 1.594% ( 170, 250 ] 42 8.367% 9.960% ## ( 250, 380 ] 89 17.729% 27.689% #### ( 380, 580 ] 147 29.283% 56.972% ###### ( 580, 870 ] 108 21.514% 78.486% #### ( 870, 1300 ] 34 6.773% 85.259% # ( 1300, 1900 ] 13 2.590% 87.849% # ( 1900, 2900 ] 17 3.386% 91.235% # ( 2900, 4400 ] 28 5.578% 96.813% # ( 4400, 6600 ] 13 2.590% 99.402% # ( 6600, 9900 ] 1 0.199% 99.602% ( 9900, 14000 ] 1 0.199% 99.801% ( 33000, 50000 ] 1 0.199% 100.000% ** Level 2 read latency histogram (micros): Count: 261515 Average: 12.7125 StdDev: 239.23 Min: 1 Median: 2.2079 Max: 63053 Percentiles: P50: 2.21 P75: 3.16 P99: 12.71 P99.9: 1281.21 P99.99: 2674.05 ------------------------------------------------------ [ 0, 1 ] 2933 1.122% 1.122% ( 1, 2 ] 111912 42.794% 43.915% ######### ( 2, 3 ] 76539 29.268% 73.183% ###### ( 3, 4 ] 28916 11.057% 84.240% ## ( 4, 6 ] 30027 11.482% 95.722% ## ( 6, 10 ] 7563 2.892% 98.614% # ( 10, 15 ] 1864 0.713% 99.327% ( 15, 22 ] 310 0.119% 99.445% ( 22, 34 ] 40 0.015% 99.460% ( 34, 51 ] 61 0.023% 99.484% ( 51, 76 ] 34 0.013% 99.497% ( 76, 110 ] 14 0.005% 99.502% ( 110, 170 ] 24 0.009% 99.511% ( 170, 250 ] 39 0.015% 99.526% ( 250, 380 ] 149 0.057% 99.583% ( 380, 580 ] 343 0.131% 99.714% ( 580, 870 ] 321 0.123% 99.837% ( 870, 1300 ] 172 0.066% 99.903% ( 1300, 1900 ] 104 0.040% 99.943% ( 1900, 2900 ] 160 0.061% 100.004% ( 2900, 4400 ] 182 0.070% 100.073% ( 4400, 6600 ] 44 0.017% 100.090% ( 6600, 9900 ] 11 0.004% 100.094% ( 9900, 14000 ] 7 0.003% 100.097% ( 14000, 22000 ] 9 0.003% 100.101% ( 22000, 33000 ] 2 0.001% 100.101% ( 50000, 75000 ] 1 0.000% 100.102% ** Level 3 read latency histogram (micros): Count: 463132 Average: 16.2945 StdDev: 336.44 Min: 1 Median: 2.2760 Max: 51209 Percentiles: P50: 2.28 P75: 3.27 P99: 14.06 P99.9: 868.07 P99.99: 1505.35 ------------------------------------------------------ [ 0, 1 ] 4753 1.026% 1.026% ( 1, 2 ] 187979 40.589% 41.615% ######## ( 2, 3 ] 140728 30.386% 72.001% ###### ( 3, 4 ] 52263 11.285% 83.286% ## ( 4, 6 ] 56750 12.254% 95.539% ## ( 6, 10 ] 13591 2.935% 98.474% # ( 10, 15 ] 2998 0.647% 99.121% ( 15, 22 ] 443 0.096% 99.217% ( 22, 34 ] 116 0.025% 99.242% ( 34, 51 ] 200 0.043% 99.285% ( 51, 76 ] 124 0.027% 99.312% ( 76, 110 ] 38 0.008% 99.320% ( 110, 170 ] 62 0.013% 99.333% ( 170, 250 ] 245 0.053% 99.386% ( 250, 380 ] 607 0.131% 99.517% ( 380, 580 ] 1007 0.217% 99.735% ( 580, 870 ] 770 0.166% 99.901% ( 870, 1300 ] 347 0.075% 99.976% ( 1300, 1900 ] 189 0.041% 100.017% ( 1900, 2900 ] 238 0.051% 100.068% ( 2900, 4400 ] 295 0.064% 100.132% ( 4400, 6600 ] 87 0.019% 100.151% ( 6600, 9900 ] 36 0.008% 100.158% ( 9900, 14000 ] 20 0.004% 100.163% ( 14000, 22000 ] 46 0.010% 100.173% ( 22000, 33000 ] 20 0.004% 100.177% ( 33000, 50000 ] 6 0.001% 100.178% ( 50000, 75000 ] 1 0.000% 100.179% ** Level 4 read latency histogram (micros): Count: 482287 Average: 32.6759 StdDev: 446.85 Min: 0 Median: 2.2392 Max: 47912 Percentiles: P50: 2.24 P75: 3.27 P99: 568.82 P99.9: 1794.13 P99.99: 2588.27 ------------------------------------------------------ [ 0, 1 ] 6521 1.352% 1.352% ( 1, 2 ] 201184 41.715% 43.067% ######## ( 2, 3 ] 139785 28.984% 72.050% ###### ( 3, 4 ] 51747 10.730% 82.780% ## ( 4, 6 ] 53934 11.183% 93.963% ## ( 6, 10 ] 13367 2.772% 96.735% # ( 10, 15 ] 3052 0.633% 97.367% ( 15, 22 ] 454 0.094% 97.461% ( 22, 34 ] 104 0.022% 97.483% ( 34, 51 ] 194 0.040% 97.523% ( 51, 76 ] 106 0.022% 97.545% ( 76, 110 ] 25 0.005% 97.550% ( 110, 170 ] 159 0.033% 97.583% ( 170, 250 ] 1088 0.226% 97.809% ( 250, 380 ] 2387 0.495% 98.304% ( 380, 580 ] 3557 0.738% 99.041% ( 580, 870 ] 2677 0.555% 99.597% ( 870, 1300 ] 1090 0.226% 99.823% ( 1300, 1900 ] 455 0.094% 99.917% ( 1900, 2900 ] 514 0.107% 100.023% ( 2900, 4400 ] 685 0.142% 100.165% ( 4400, 6600 ] 168 0.035% 100.200% ( 6600, 9900 ] 56 0.012% 100.212% ( 9900, 14000 ] 30 0.006% 100.218% ( 14000, 22000 ] 56 0.012% 100.230% ( 22000, 33000 ] 39 0.008% 100.238% ( 33000, 50000 ] 16 0.003% 100.241% ** Level 5 read latency histogram (micros): Count: 8629 Average: 963.4076 StdDev: 2024.31 Min: 99 Median: 527.4192 Max: 47973 Percentiles: P50: 527.42 P75: 829.39 P99: 6201.32 P99.9: 31298.88 P99.99: 47555.12 ------------------------------------------------------ ( 76, 110 ] 2 0.023% 0.023% ( 110, 170 ] 116 1.344% 1.367% ( 170, 250 ] 789 9.144% 10.511% ## ( 250, 380 ] 1651 19.133% 29.644% #### ( 380, 580 ] 2383 27.616% 57.260% ###### ( 580, 870 ] 1780 20.628% 77.889% #### ( 870, 1300 ] 731 8.471% 86.360% ## ( 1300, 1900 ] 255 2.955% 89.315% # ( 1900, 2900 ] 309 3.581% 92.896% # ( 2900, 4400 ] 426 4.937% 97.833% # ( 4400, 6600 ] 123 1.425% 99.258% ( 6600, 9900 ] 23 0.267% 99.525% ( 9900, 14000 ] 5 0.058% 99.583% ( 14000, 22000 ] 13 0.151% 99.733% ( 22000, 33000 ] 17 0.197% 99.930% ( 33000, 50000 ] 6 0.070% 100.000% ** DB Stats ** Uptime(secs): 6.5 total, 6.4 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 152.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:1.649 H:M:S, 25.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 153.54 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.649 H:M:S, 25.7 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 4 num-running-flushes: 1 2024/12/10-14:12:27 ... thread 0: (1000000,2000000) ops and (207116.9,178019.5) ops/second in (4.828191,11.234723) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/9 2.30 GB 156.5 1.4 0.0 1.4 3.2 1.8 0.0 1.7 69.7 164.3 19.86 11.40 32 0.621 1405K 7219 0.0 0.0 L1 6/6 418.33 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 19/0 958.99 MB 1.0 2.0 1.4 0.6 2.0 1.3 0.0 1.4 118.4 116.3 17.48 10.45 10 1.748 1628K 11K 0.0 0.0 L4 137/0 9.72 GB 1.0 3.8 1.8 2.0 3.5 1.5 0.4 1.9 126.5 117.2 30.56 19.84 16 1.910 790K 0 0.0 0.0 L5 148/0 11.63 GB 0.1 0.0 0.0 0.0 0.0 0.0 2.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 327/15 24.99 GB 0.0 7.1 3.2 4.0 8.7 4.7 3.0 4.7 107.8 130.7 67.91 41.68 58 1.171 3824K 18K 0.0 0.0 Int 0/0 0.00 KB 0.0 7.1 3.2 4.0 8.7 4.7 3.0 4.7 107.8 130.7 67.91 41.68 58 1.171 3824K 18K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 7.1 3.2 4.0 6.8 2.9 0.0 0.0 124.6 119.0 58.77 36.04 31 1.896 3824K 18K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 206.4 9.14 5.64 27 0.339 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 11.3 total, 11.2 interval Flush(GB): cumulative 1.843, interval 1.843 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 8.67 GB write, 785.51 MB/s write, 7.15 GB read, 647.61 MB/s read, 67.9 seconds Interval compaction: 8.67 GB write, 790.13 MB/s write, 7.15 GB read, 651.41 MB/s read, 67.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 5, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 5, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 14, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 14, interval: 19 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 514861 Average: 45.0572 StdDev: 361.59 Min: 0 Median: 1.7038 Max: 55031 Percentiles: P50: 1.70 P75: 2.74 P99: 560.27 P99.9: 2336.44 P99.99: 3640.73 ------------------------------------------------------ [ 0, 1 ] 75032 14.573% 14.573% ### ( 1, 2 ] 259169 50.338% 64.911% ########## ( 2, 3 ] 70378 13.669% 78.580% ### ( 3, 4 ] 23424 4.550% 83.130% # ( 4, 6 ] 18423 3.578% 86.708% # ( 6, 10 ] 5750 1.117% 87.825% ( 10, 15 ] 1227 0.238% 88.063% ( 15, 22 ] 239 0.046% 88.110% ( 22, 34 ] 39 0.008% 88.117% ( 34, 51 ] 13 0.003% 88.120% ( 51, 76 ] 15 0.003% 88.123% ( 76, 110 ] 241 0.047% 88.169% ( 110, 170 ] 8412 1.634% 89.803% ( 170, 250 ] 29959 5.819% 95.622% # ( 250, 380 ] 12233 2.376% 97.998% ( 380, 580 ] 5723 1.112% 99.110% ( 580, 870 ] 2767 0.537% 99.647% ( 870, 1300 ] 863 0.168% 99.815% ( 1300, 1900 ] 265 0.051% 99.866% ( 1900, 2900 ] 399 0.077% 99.944% ( 2900, 4400 ] 483 0.094% 100.037% ( 4400, 6600 ] 103 0.020% 100.057% ( 6600, 9900 ] 31 0.006% 100.064% ( 9900, 14000 ] 7 0.001% 100.065% ( 14000, 22000 ] 20 0.004% 100.069% ( 22000, 33000 ] 32 0.006% 100.075% ( 33000, 50000 ] 9 0.002% 100.077% ( 50000, 75000 ] 1 0.000% 100.077% ** Level 1 read latency histogram (micros): Count: 70975 Average: 15.7938 StdDev: 184.44 Min: 1 Median: 2.8624 Max: 35016 Percentiles: P50: 2.86 P75: 4.49 P99: 294.39 P99.9: 1347.31 P99.99: 5568.12 ------------------------------------------------------ [ 0, 1 ] 1131 1.594% 1.594% ( 1, 2 ] 18947 26.695% 28.289% ##### ( 2, 3 ] 17869 25.176% 53.465% ##### ( 3, 4 ] 11806 16.634% 70.099% ### ( 4, 6 ] 14324 20.182% 90.281% #### ( 6, 10 ] 3365 4.741% 95.022% # ( 10, 15 ] 857 1.207% 96.230% ( 15, 22 ] 494 0.696% 96.926% ( 22, 34 ] 92 0.130% 97.055% ( 34, 51 ] 12 0.017% 97.072% ( 51, 76 ] 1 0.001% 97.074% ( 76, 110 ] 9 0.013% 97.086% ( 110, 170 ] 273 0.385% 97.471% ( 170, 250 ] 949 1.337% 98.808% ( 250, 380 ] 399 0.562% 99.370% ( 380, 580 ] 220 0.310% 99.680% ( 580, 870 ] 120 0.169% 99.849% ( 870, 1300 ] 35 0.049% 99.899% ( 1300, 1900 ] 13 0.018% 99.917% ( 1900, 2900 ] 17 0.024% 99.941% ( 2900, 4400 ] 28 0.039% 99.980% ( 4400, 6600 ] 13 0.018% 99.999% ( 6600, 9900 ] 1 0.001% 100.000% ( 9900, 14000 ] 1 0.001% 100.001% ( 33000, 50000 ] 1 0.001% 100.003% ** Level 2 read latency histogram (micros): Count: 261515 Average: 12.7125 StdDev: 239.23 Min: 1 Median: 2.2079 Max: 63053 Percentiles: P50: 2.21 P75: 3.16 P99: 12.71 P99.9: 1281.21 P99.99: 2674.05 ------------------------------------------------------ [ 0, 1 ] 2933 1.122% 1.122% ( 1, 2 ] 111912 42.794% 43.915% ######### ( 2, 3 ] 76539 29.268% 73.183% ###### ( 3, 4 ] 28916 11.057% 84.240% ## ( 4, 6 ] 30027 11.482% 95.722% ## ( 6, 10 ] 7563 2.892% 98.614% # ( 10, 15 ] 1864 0.713% 99.327% ( 15, 22 ] 310 0.119% 99.445% ( 22, 34 ] 40 0.015% 99.460% ( 34, 51 ] 61 0.023% 99.484% ( 51, 76 ] 34 0.013% 99.497% ( 76, 110 ] 14 0.005% 99.502% ( 110, 170 ] 24 0.009% 99.511% ( 170, 250 ] 39 0.015% 99.526% ( 250, 380 ] 149 0.057% 99.583% ( 380, 580 ] 343 0.131% 99.714% ( 580, 870 ] 321 0.123% 99.837% ( 870, 1300 ] 172 0.066% 99.903% ( 1300, 1900 ] 104 0.040% 99.943% ( 1900, 2900 ] 160 0.061% 100.004% ( 2900, 4400 ] 182 0.070% 100.073% ( 4400, 6600 ] 44 0.017% 100.090% ( 6600, 9900 ] 11 0.004% 100.094% ( 9900, 14000 ] 7 0.003% 100.097% ( 14000, 22000 ] 9 0.003% 100.101% ( 22000, 33000 ] 2 0.001% 100.101% ( 50000, 75000 ] 1 0.000% 100.102% ** Level 3 read latency histogram (micros): Count: 468126 Average: 18.2282 StdDev: 335.34 Min: 1 Median: 2.2898 Max: 51209 Percentiles: P50: 2.29 P75: 3.33 P99: 226.28 P99.9: 866.68 P99.99: 1503.77 ------------------------------------------------------ [ 0, 1 ] 4895 1.046% 1.046% ( 1, 2 ] 188378 40.241% 41.287% ######## ( 2, 3 ] 140774 30.072% 71.358% ###### ( 3, 4 ] 52277 11.167% 82.526% ## ( 4, 6 ] 56757 12.124% 94.650% ## ( 6, 10 ] 13600 2.905% 97.555% # ( 10, 15 ] 2998 0.640% 98.196% ( 15, 22 ] 443 0.095% 98.290% ( 22, 34 ] 116 0.025% 98.315% ( 34, 51 ] 200 0.043% 98.358% ( 51, 76 ] 124 0.026% 98.384% ( 76, 110 ] 64 0.014% 98.398% ( 110, 170 ] 854 0.182% 98.580% ( 170, 250 ] 2793 0.597% 99.177% ( 250, 380 ] 1394 0.298% 99.475% ( 380, 580 ] 1202 0.257% 99.731% ( 580, 870 ] 798 0.170% 99.902% ( 870, 1300 ] 348 0.074% 99.976% ( 1300, 1900 ] 189 0.040% 100.017% ( 1900, 2900 ] 238 0.051% 100.068% ( 2900, 4400 ] 295 0.063% 100.131% ( 4400, 6600 ] 87 0.019% 100.149% ( 6600, 9900 ] 36 0.008% 100.157% ( 9900, 14000 ] 20 0.004% 100.161% ( 14000, 22000 ] 46 0.010% 100.171% ( 22000, 33000 ] 20 0.004% 100.175% ( 33000, 50000 ] 6 0.001% 100.176% ( 50000, 75000 ] 1 0.000% 100.177% ** Level 4 read latency histogram (micros): Count: 518224 Average: 43.8579 StdDev: 433.97 Min: 0 Median: 2.3438 Max: 47912 Percentiles: P50: 2.34 P75: 3.71 P99: 565.78 P99.9: 1747.41 P99.99: 2577.33 ------------------------------------------------------ [ 0, 1 ] 7192 1.388% 1.388% ( 1, 2 ] 203539 39.276% 40.664% ######## ( 2, 3 ] 140734 27.157% 67.821% ##### ( 3, 4 ] 52145 10.062% 77.883% ## ( 4, 6 ] 54342 10.486% 88.370% ## ( 6, 10 ] 13469 2.599% 90.969% # ( 10, 15 ] 3071 0.593% 91.561% ( 15, 22 ] 465 0.090% 91.651% ( 22, 34 ] 107 0.021% 91.672% ( 34, 51 ] 194 0.037% 91.709% ( 51, 76 ] 106 0.020% 91.729% ( 76, 110 ] 189 0.036% 91.766% ( 110, 170 ] 5923 1.143% 92.909% ( 170, 250 ] 19298 3.724% 96.633% # ( 250, 380 ] 7731 1.492% 98.125% ( 380, 580 ] 4884 0.942% 99.067% ( 580, 870 ] 2868 0.553% 99.620% ( 870, 1300 ] 1108 0.214% 99.834% ( 1300, 1900 ] 457 0.088% 99.922% ( 1900, 2900 ] 517 0.100% 100.022% ( 2900, 4400 ] 686 0.132% 100.155% ( 4400, 6600 ] 168 0.032% 100.187% ( 6600, 9900 ] 56 0.011% 100.198% ( 9900, 14000 ] 30 0.006% 100.204% ( 14000, 22000 ] 56 0.011% 100.214% ( 22000, 33000 ] 39 0.008% 100.222% ( 33000, 50000 ] 16 0.003% 100.225% ** Level 5 read latency histogram (micros): Count: 32191 Average: 421.6222 StdDev: 1100.22 Min: 94 Median: 232.4267 Max: 47973 Percentiles: P50: 232.43 P75: 360.80 P99: 3924.96 P99.9: 16344.00 P99.99: 40879.22 ------------------------------------------------------ ( 76, 110 ] 142 0.441% 0.441% ( 110, 170 ] 4538 14.097% 14.538% ### ( 170, 250 ] 14629 45.444% 59.983% ######### ( 250, 380 ] 5672 17.620% 77.602% #### ( 380, 580 ] 3377 10.491% 88.093% ## ( 580, 870 ] 1910 5.933% 94.026% # ( 870, 1300 ] 746 2.317% 96.344% ( 1300, 1900 ] 255 0.792% 97.136% ( 1900, 2900 ] 309 0.960% 98.096% ( 2900, 4400 ] 426 1.323% 99.419% ( 4400, 6600 ] 123 0.382% 99.801% ( 6600, 9900 ] 23 0.071% 99.873% ( 9900, 14000 ] 5 0.016% 99.888% ( 14000, 22000 ] 13 0.040% 99.929% ( 22000, 33000 ] 17 0.053% 99.981% ( 33000, 50000 ] 6 0.019% 100.000% ** DB Stats ** Uptime(secs): 11.3 total, 4.8 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 174.15 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:2.725 H:M:S, 24.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 203.85 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.076 H:M:S, 22.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/10-14:12:32 ... thread 0: (1000000,3000000) ops and (219107.9,189889.2) ops/second in (4.563962,15.798685) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/8 2.81 GB 231.6 2.1 0.0 2.1 4.9 2.8 0.0 1.7 82.6 193.8 25.90 16.97 51 0.508 2171K 10K 0.0 0.0 L1 9/0 875.94 MB 87.6 0.9 0.5 0.4 0.9 0.4 0.0 1.9 66.1 65.1 13.45 5.79 1 13.452 903K 13K 0.0 0.0 L3 19/0 958.99 MB 1.0 2.0 1.4 0.6 2.0 1.3 0.0 1.4 118.4 116.3 17.48 10.45 10 1.748 1628K 11K 0.0 0.0 L4 137/0 9.72 GB 1.0 3.8 1.8 2.0 3.5 1.5 0.4 1.9 126.5 117.2 30.56 19.84 16 1.910 790K 0 0.0 0.0 L5 148/0 11.63 GB 0.1 0.0 0.0 0.0 0.0 0.0 2.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 332/8 25.96 GB 0.0 8.8 3.6 5.1 11.2 6.1 3.0 4.0 102.6 131.7 87.40 53.05 78 1.120 5494K 36K 0.0 0.0 Int 0/0 0.00 KB 0.0 8.8 3.6 5.1 11.2 6.1 3.0 4.0 102.6 131.7 87.40 53.05 78 1.120 5494K 36K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 8.8 3.6 5.1 8.4 3.3 0.0 0.0 119.3 114.7 75.14 44.68 35 2.147 5494K 36K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 236.0 12.26 8.37 43 0.285 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 15.9 total, 15.8 interval Flush(GB): cumulative 2.826, interval 2.826 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 11.24 GB write, 725.55 MB/s write, 8.75 GB read, 564.98 MB/s read, 87.4 seconds Interval compaction: 11.24 GB write, 728.59 MB/s write, 8.75 GB read, 567.34 MB/s read, 87.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 8, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 8, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 14, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 8, total-stops: 14, interval: 22 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 829205 Average: 37.2341 StdDev: 288.17 Min: 0 Median: 1.6489 Max: 55031 Percentiles: P50: 1.65 P75: 2.56 P99: 466.19 P99.9: 1242.24 P99.99: 2794.94 ------------------------------------------------------ [ 0, 1 ] 135293 16.316% 16.316% ### ( 1, 2 ] 430412 51.907% 68.223% ########## ( 2, 3 ] 100809 12.157% 80.380% ## ( 3, 4 ] 33054 3.986% 84.366% # ( 4, 6 ] 24565 2.962% 87.329% # ( 6, 10 ] 7772 0.937% 88.266% ( 10, 15 ] 1577 0.190% 88.456% ( 15, 22 ] 313 0.038% 88.494% ( 22, 34 ] 70 0.008% 88.502% ( 34, 51 ] 15 0.002% 88.504% ( 51, 76 ] 17 0.002% 88.506% ( 76, 110 ] 609 0.073% 88.580% ( 110, 170 ] 17530 2.114% 90.694% ( 170, 250 ] 50023 6.033% 96.726% # ( 250, 380 ] 15968 1.926% 98.652% ( 380, 580 ] 6701 0.808% 99.460% ( 580, 870 ] 2888 0.348% 99.808% ( 870, 1300 ] 880 0.106% 99.914% ( 1300, 1900 ] 271 0.033% 99.947% ( 1900, 2900 ] 399 0.048% 99.995% ( 2900, 4400 ] 483 0.058% 100.054% ( 4400, 6600 ] 103 0.012% 100.066% ( 6600, 9900 ] 31 0.004% 100.070% ( 9900, 14000 ] 7 0.001% 100.071% ( 14000, 22000 ] 20 0.002% 100.073% ( 22000, 33000 ] 32 0.004% 100.077% ( 33000, 50000 ] 9 0.001% 100.078% ( 50000, 75000 ] 1 0.000% 100.078% ** Level 1 read latency histogram (micros): Count: 110519 Average: 15.3223 StdDev: 151.03 Min: 1 Median: 2.8634 Max: 35016 Percentiles: P50: 2.86 P75: 4.41 P99: 259.32 P99.9: 924.81 P99.99: 4729.68 ------------------------------------------------------ [ 0, 1 ] 1692 1.531% 1.531% ( 1, 2 ] 27597 24.970% 26.501% ##### ( 2, 3 ] 30078 27.215% 53.717% ##### ( 3, 4 ] 19221 17.392% 71.108% ### ( 4, 6 ] 20753 18.778% 89.886% #### ( 6, 10 ] 4933 4.463% 94.349% # ( 10, 15 ] 1310 1.185% 95.535% ( 15, 22 ] 676 0.612% 96.146% ( 22, 34 ] 130 0.118% 96.264% ( 34, 51 ] 15 0.014% 96.278% ( 51, 76 ] 2 0.002% 96.279% ( 76, 110 ] 26 0.024% 96.303% ( 110, 170 ] 824 0.746% 97.048% ( 170, 250 ] 2112 1.911% 98.959% ( 250, 380 ] 625 0.566% 99.525% ( 380, 580 ] 275 0.249% 99.774% ( 580, 870 ] 134 0.121% 99.895% ( 870, 1300 ] 43 0.039% 99.934% ( 1300, 1900 ] 14 0.013% 99.947% ( 1900, 2900 ] 18 0.016% 99.963% ( 2900, 4400 ] 28 0.025% 99.988% ( 4400, 6600 ] 13 0.012% 100.000% ( 6600, 9900 ] 1 0.001% 100.001% ( 9900, 14000 ] 1 0.001% 100.002% ( 33000, 50000 ] 1 0.001% 100.003% ** Level 2 read latency histogram (micros): Count: 261515 Average: 12.7125 StdDev: 239.23 Min: 1 Median: 2.2079 Max: 63053 Percentiles: P50: 2.21 P75: 3.16 P99: 12.71 P99.9: 1281.21 P99.99: 2674.05 ------------------------------------------------------ [ 0, 1 ] 2933 1.122% 1.122% ( 1, 2 ] 111912 42.794% 43.915% ######### ( 2, 3 ] 76539 29.268% 73.183% ###### ( 3, 4 ] 28916 11.057% 84.240% ## ( 4, 6 ] 30027 11.482% 95.722% ## ( 6, 10 ] 7563 2.892% 98.614% # ( 10, 15 ] 1864 0.713% 99.327% ( 15, 22 ] 310 0.119% 99.445% ( 22, 34 ] 40 0.015% 99.460% ( 34, 51 ] 61 0.023% 99.484% ( 51, 76 ] 34 0.013% 99.497% ( 76, 110 ] 14 0.005% 99.502% ( 110, 170 ] 24 0.009% 99.511% ( 170, 250 ] 39 0.015% 99.526% ( 250, 380 ] 149 0.057% 99.583% ( 380, 580 ] 343 0.131% 99.714% ( 580, 870 ] 321 0.123% 99.837% ( 870, 1300 ] 172 0.066% 99.903% ( 1300, 1900 ] 104 0.040% 99.943% ( 1900, 2900 ] 160 0.061% 100.004% ( 2900, 4400 ] 182 0.070% 100.073% ( 4400, 6600 ] 44 0.017% 100.090% ( 6600, 9900 ] 11 0.004% 100.094% ( 9900, 14000 ] 7 0.003% 100.097% ( 14000, 22000 ] 9 0.003% 100.101% ( 22000, 33000 ] 2 0.001% 100.101% ( 50000, 75000 ] 1 0.000% 100.102% ** Level 3 read latency histogram (micros): Count: 472354 Average: 19.9150 StdDev: 334.39 Min: 1 Median: 2.3048 Max: 51209 Percentiles: P50: 2.30 P75: 3.39 P99: 245.77 P99.9: 865.98 P99.99: 1505.60 ------------------------------------------------------ [ 0, 1 ] 4895 1.036% 1.036% ( 1, 2 ] 188378 39.881% 40.917% ######## ( 2, 3 ] 140774 29.803% 70.720% ###### ( 3, 4 ] 52277 11.067% 81.787% ## ( 4, 6 ] 56757 12.016% 93.803% ## ( 6, 10 ] 13600 2.879% 96.682% # ( 10, 15 ] 2998 0.635% 97.317% ( 15, 22 ] 443 0.094% 97.410% ( 22, 34 ] 116 0.025% 97.435% ( 34, 51 ] 200 0.042% 97.477% ( 51, 76 ] 124 0.026% 97.504% ( 76, 110 ] 104 0.022% 97.526% ( 110, 170 ] 1993 0.422% 97.948% ( 170, 250 ] 5249 1.111% 99.059% ( 250, 380 ] 1835 0.388% 99.447% ( 380, 580 ] 1330 0.282% 99.729% ( 580, 870 ] 820 0.174% 99.902% ( 870, 1300 ] 349 0.074% 99.976% ( 1300, 1900 ] 189 0.040% 100.016% ( 1900, 2900 ] 239 0.051% 100.067% ( 2900, 4400 ] 295 0.062% 100.129% ( 4400, 6600 ] 87 0.018% 100.148% ( 6600, 9900 ] 36 0.008% 100.155% ( 9900, 14000 ] 20 0.004% 100.160% ( 14000, 22000 ] 46 0.010% 100.169% ( 22000, 33000 ] 20 0.004% 100.174% ( 33000, 50000 ] 6 0.001% 100.175% ( 50000, 75000 ] 1 0.000% 100.175% ** Level 4 read latency histogram (micros): Count: 550434 Average: 53.3371 StdDev: 423.16 Min: 0 Median: 2.4582 Max: 47912 Percentiles: P50: 2.46 P75: 4.34 P99: 562.04 P99.9: 1707.69 P99.99: 2573.03 ------------------------------------------------------ [ 0, 1 ] 7192 1.307% 1.307% ( 1, 2 ] 203539 36.978% 38.285% ####### ( 2, 3 ] 140734 25.568% 63.852% ##### ( 3, 4 ] 52145 9.473% 73.326% ## ( 4, 6 ] 54342 9.873% 83.198% ## ( 6, 10 ] 13469 2.447% 85.645% ( 10, 15 ] 3071 0.558% 86.203% ( 15, 22 ] 465 0.084% 86.288% ( 22, 34 ] 107 0.019% 86.307% ( 34, 51 ] 194 0.035% 86.342% ( 51, 76 ] 106 0.019% 86.362% ( 76, 110 ] 523 0.095% 86.457% ( 110, 170 ] 14792 2.687% 89.144% # ( 170, 250 ] 37812 6.869% 96.014% # ( 250, 380 ] 11105 2.017% 98.031% ( 380, 580 ] 5860 1.065% 99.096% ( 580, 870 ] 2996 0.544% 99.640% ( 870, 1300 ] 1119 0.203% 99.843% ( 1300, 1900 ] 460 0.084% 99.927% ( 1900, 2900 ] 517 0.094% 100.021% ( 2900, 4400 ] 686 0.125% 100.145% ( 4400, 6600 ] 168 0.031% 100.176% ( 6600, 9900 ] 56 0.010% 100.186% ( 9900, 14000 ] 30 0.005% 100.191% ( 14000, 22000 ] 56 0.010% 100.202% ( 22000, 33000 ] 39 0.007% 100.209% ( 33000, 50000 ] 16 0.003% 100.212% ** Level 5 read latency histogram (micros): Count: 55114 Average: 331.2154 StdDev: 848.82 Min: 92 Median: 216.7561 Max: 47973 Percentiles: P50: 216.76 P75: 285.00 P99: 3117.82 P99.9: 7874.95 P99.99: 34384.37 ------------------------------------------------------ ( 76, 110 ] 379 0.688% 0.688% ( 110, 170 ] 10842 19.672% 20.360% #### ( 170, 250 ] 27951 50.715% 71.075% ########## ( 250, 380 ] 8036 14.581% 85.655% ### ( 380, 580 ] 3994 7.247% 92.902% # ( 580, 870 ] 1982 3.596% 96.498% # ( 870, 1300 ] 751 1.363% 97.861% ( 1300, 1900 ] 257 0.466% 98.327% ( 1900, 2900 ] 309 0.561% 98.888% ( 2900, 4400 ] 426 0.773% 99.661% ( 4400, 6600 ] 123 0.223% 99.884% ( 6600, 9900 ] 23 0.042% 99.926% ( 9900, 14000 ] 5 0.009% 99.935% ( 14000, 22000 ] 13 0.024% 99.958% ( 22000, 33000 ] 17 0.031% 99.989% ( 33000, 50000 ] 6 0.011% 100.000% ** DB Stats ** Uptime(secs): 15.9 total, 4.6 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 186.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.547 H:M:S, 22.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 215.64 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.821 H:M:S, 18.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 0 2024/12/10-14:12:35 ... thread 0: (1000000,4000000) ops and (272768.6,205499.2) ops/second in (3.666111,19.464796) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/4 3.73 GB 356.7 3.4 0.0 3.4 7.1 3.7 0.0 1.9 79.9 168.1 43.31 25.76 71 0.610 3513K 17K 0.0 0.0 L1 9/0 875.94 MB 87.6 0.9 0.5 0.4 0.9 0.4 0.0 1.9 66.1 65.1 13.45 5.79 1 13.452 903K 13K 0.0 0.0 L3 19/0 958.99 MB 1.0 2.0 1.4 0.6 2.0 1.3 0.0 1.4 118.4 116.3 17.48 10.45 10 1.748 1628K 11K 0.0 0.0 L4 137/0 9.72 GB 1.0 3.8 1.8 2.0 3.5 1.5 0.4 1.9 126.5 117.2 30.56 19.84 16 1.910 790K 0 0.0 0.0 L5 148/0 11.63 GB 0.1 0.0 0.0 0.0 0.0 0.0 2.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 332/4 26.87 GB 0.0 10.0 3.6 6.4 13.4 7.0 3.0 3.6 98.1 131.4 104.81 61.84 98 1.069 6835K 43K 0.0 0.0 Int 0/0 0.00 KB 0.0 10.0 3.6 6.4 13.4 7.0 3.0 3.6 98.1 131.4 104.81 61.84 98 1.069 6835K 43K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 10.0 3.6 6.4 9.7 3.3 0.0 0.0 114.7 110.8 89.68 50.91 40 2.242 6835K 43K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.7 3.7 0.0 0.0 0.0 253.6 15.13 10.93 58 0.261 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 19.5 total, 19.5 interval Flush(GB): cumulative 3.747, interval 3.747 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 13.45 GB write, 704.97 MB/s write, 10.04 GB read, 526.57 MB/s read, 104.8 seconds Interval compaction: 13.45 GB write, 707.36 MB/s write, 10.04 GB read, 528.35 MB/s read, 104.8 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 9, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 9, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 14, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 9, total-stops: 14, interval: 23 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1101170 Average: 30.3964 StdDev: 251.13 Min: 0 Median: 1.5865 Max: 55031 Percentiles: P50: 1.59 P75: 2.26 P99: 390.49 P99.9: 1057.11 P99.99: 2444.71 ------------------------------------------------------ [ 0, 1 ] 205204 18.635% 18.635% #### ( 1, 2 ] 588860 53.476% 72.111% ########### ( 2, 3 ] 121220 11.008% 83.119% ## ( 3, 4 ] 39694 3.605% 86.724% # ( 4, 6 ] 28919 2.626% 89.350% # ( 6, 10 ] 8814 0.800% 90.151% ( 10, 15 ] 1883 0.171% 90.322% ( 15, 22 ] 352 0.032% 90.354% ( 22, 34 ] 88 0.008% 90.362% ( 34, 51 ] 21 0.002% 90.363% ( 51, 76 ] 17 0.002% 90.365% ( 76, 110 ] 895 0.081% 90.446% ( 110, 170 ] 21721 1.973% 92.419% ( 170, 250 ] 55350 5.026% 97.445% # ( 250, 380 ] 16756 1.522% 98.967% ( 380, 580 ] 6964 0.632% 99.599% ( 580, 870 ] 2925 0.266% 99.865% ( 870, 1300 ] 889 0.081% 99.946% ( 1300, 1900 ] 271 0.025% 99.970% ( 1900, 2900 ] 400 0.036% 100.007% ( 2900, 4400 ] 483 0.044% 100.050% ( 4400, 6600 ] 103 0.009% 100.060% ( 6600, 9900 ] 31 0.003% 100.063% ( 9900, 14000 ] 7 0.001% 100.063% ( 14000, 22000 ] 20 0.002% 100.065% ( 22000, 33000 ] 32 0.003% 100.068% ( 33000, 50000 ] 9 0.001% 100.069% ( 50000, 75000 ] 1 0.000% 100.069% ** Level 1 read latency histogram (micros): Count: 113214 Average: 19.2127 StdDev: 151.76 Min: 1 Median: 2.9057 Max: 35016 Percentiles: P50: 2.91 P75: 4.60 P99: 293.42 P99.9: 907.00 P99.99: 4684.07 ------------------------------------------------------ [ 0, 1 ] 1692 1.495% 1.495% ( 1, 2 ] 27619 24.395% 25.890% ##### ( 2, 3 ] 30138 26.620% 52.510% ##### ( 3, 4 ] 19258 17.010% 69.521% ### ( 4, 6 ] 20785 18.359% 87.880% #### ( 6, 10 ] 4939 4.363% 92.242% # ( 10, 15 ] 1311 1.158% 93.400% ( 15, 22 ] 676 0.597% 93.997% ( 22, 34 ] 131 0.116% 94.113% ( 34, 51 ] 15 0.013% 94.126% ( 51, 76 ] 2 0.002% 94.128% ( 76, 110 ] 89 0.079% 94.207% ( 110, 170 ] 1855 1.638% 95.845% ( 170, 250 ] 3303 2.917% 98.763% # ( 250, 380 ] 805 0.711% 99.474% ( 380, 580 ] 338 0.299% 99.772% ( 580, 870 ] 141 0.125% 99.897% ( 870, 1300 ] 44 0.039% 99.936% ( 1300, 1900 ] 14 0.012% 99.948% ( 1900, 2900 ] 18 0.016% 99.964% ( 2900, 4400 ] 28 0.025% 99.989% ( 4400, 6600 ] 13 0.011% 100.000% ( 6600, 9900 ] 1 0.001% 100.001% ( 9900, 14000 ] 1 0.001% 100.002% ( 33000, 50000 ] 1 0.001% 100.003% ** Level 2 read latency histogram (micros): Count: 261515 Average: 12.7125 StdDev: 239.23 Min: 1 Median: 2.2079 Max: 63053 Percentiles: P50: 2.21 P75: 3.16 P99: 12.71 P99.9: 1281.21 P99.99: 2674.05 ------------------------------------------------------ [ 0, 1 ] 2933 1.122% 1.122% ( 1, 2 ] 111912 42.794% 43.915% ######### ( 2, 3 ] 76539 29.268% 73.183% ###### ( 3, 4 ] 28916 11.057% 84.240% ## ( 4, 6 ] 30027 11.482% 95.722% ## ( 6, 10 ] 7563 2.892% 98.614% # ( 10, 15 ] 1864 0.713% 99.327% ( 15, 22 ] 310 0.119% 99.445% ( 22, 34 ] 40 0.015% 99.460% ( 34, 51 ] 61 0.023% 99.484% ( 51, 76 ] 34 0.013% 99.497% ( 76, 110 ] 14 0.005% 99.502% ( 110, 170 ] 24 0.009% 99.511% ( 170, 250 ] 39 0.015% 99.526% ( 250, 380 ] 149 0.057% 99.583% ( 380, 580 ] 343 0.131% 99.714% ( 580, 870 ] 321 0.123% 99.837% ( 870, 1300 ] 172 0.066% 99.903% ( 1300, 1900 ] 104 0.040% 99.943% ( 1900, 2900 ] 160 0.061% 100.004% ( 2900, 4400 ] 182 0.070% 100.073% ( 4400, 6600 ] 44 0.017% 100.090% ( 6600, 9900 ] 11 0.004% 100.094% ( 9900, 14000 ] 7 0.003% 100.097% ( 14000, 22000 ] 9 0.003% 100.101% ( 22000, 33000 ] 2 0.001% 100.101% ( 50000, 75000 ] 1 0.000% 100.102% ** Level 3 read latency histogram (micros): Count: 475911 Average: 21.1798 StdDev: 333.50 Min: 1 Median: 2.3174 Max: 51209 Percentiles: P50: 2.32 P75: 3.44 P99: 251.18 P99.9: 865.47 P99.99: 1504.47 ------------------------------------------------------ [ 0, 1 ] 4895 1.029% 1.029% ( 1, 2 ] 188378 39.583% 40.611% ######## ( 2, 3 ] 140774 29.580% 70.191% ###### ( 3, 4 ] 52277 10.985% 81.176% ## ( 4, 6 ] 56757 11.926% 93.102% ## ( 6, 10 ] 13600 2.858% 95.959% # ( 10, 15 ] 2998 0.630% 96.589% ( 15, 22 ] 443 0.093% 96.682% ( 22, 34 ] 116 0.024% 96.707% ( 34, 51 ] 200 0.042% 96.749% ( 51, 76 ] 124 0.026% 96.775% ( 76, 110 ] 203 0.043% 96.817% ( 110, 170 ] 3388 0.712% 97.529% ( 170, 250 ] 6980 1.467% 98.996% ( 250, 380 ] 2088 0.439% 99.435% ( 380, 580 ] 1401 0.294% 99.729% ( 580, 870 ] 826 0.174% 99.903% ( 870, 1300 ] 351 0.074% 99.976% ( 1300, 1900 ] 189 0.040% 100.016% ( 1900, 2900 ] 239 0.050% 100.066% ( 2900, 4400 ] 295 0.062% 100.128% ( 4400, 6600 ] 87 0.018% 100.147% ( 6600, 9900 ] 36 0.008% 100.154% ( 9900, 14000 ] 20 0.004% 100.158% ( 14000, 22000 ] 46 0.010% 100.168% ( 22000, 33000 ] 20 0.004% 100.172% ( 33000, 50000 ] 6 0.001% 100.174% ( 50000, 75000 ] 1 0.000% 100.174% ** Level 4 read latency histogram (micros): Count: 577496 Average: 59.7617 StdDev: 414.40 Min: 0 Median: 2.5544 Max: 47912 Percentiles: P50: 2.55 P75: 5.09 P99: 558.40 P99.9: 1672.89 P99.99: 2567.80 ------------------------------------------------------ [ 0, 1 ] 7192 1.245% 1.245% ( 1, 2 ] 203539 35.245% 36.490% ####### ( 2, 3 ] 140734 24.370% 60.860% ##### ( 3, 4 ] 52145 9.029% 69.890% ## ( 4, 6 ] 54342 9.410% 79.300% ## ( 6, 10 ] 13469 2.332% 81.632% ( 10, 15 ] 3071 0.532% 82.164% ( 15, 22 ] 465 0.081% 82.244% ( 22, 34 ] 107 0.019% 82.263% ( 34, 51 ] 194 0.034% 82.296% ( 51, 76 ] 106 0.018% 82.315% ( 76, 110 ] 1275 0.221% 82.535% ( 110, 170 ] 25268 4.375% 86.911% # ( 170, 250 ] 51050 8.840% 95.751% ## ( 250, 380 ] 12991 2.250% 98.000% ( 380, 580 ] 6472 1.121% 99.121% ( 580, 870 ] 3087 0.535% 99.656% ( 870, 1300 ] 1125 0.195% 99.850% ( 1300, 1900 ] 461 0.080% 99.930% ( 1900, 2900 ] 517 0.090% 100.020% ( 2900, 4400 ] 686 0.119% 100.139% ( 4400, 6600 ] 168 0.029% 100.168% ( 6600, 9900 ] 56 0.010% 100.177% ( 9900, 14000 ] 30 0.005% 100.183% ( 14000, 22000 ] 56 0.010% 100.192% ( 22000, 33000 ] 39 0.007% 100.199% ( 33000, 50000 ] 16 0.003% 100.202% ** Level 5 read latency histogram (micros): Count: 74117 Average: 295.1998 StdDev: 735.28 Min: 91 Median: 208.9063 Max: 47973 Percentiles: P50: 208.91 P75: 248.45 P99: 2485.21 P99.9: 6419.05 P99.99: 32086.55 ------------------------------------------------------ ( 76, 110 ] 908 1.225% 1.225% ( 110, 170 ] 17919 24.177% 25.402% ##### ( 170, 250 ] 37488 50.579% 75.981% ########## ( 250, 380 ] 9421 12.711% 88.692% ### ( 380, 580 ] 4417 5.959% 94.652% # ( 580, 870 ] 2030 2.739% 97.391% # ( 870, 1300 ] 754 1.017% 98.408% ( 1300, 1900 ] 258 0.348% 98.756% ( 1900, 2900 ] 309 0.417% 99.173% ( 2900, 4400 ] 426 0.575% 99.748% ( 4400, 6600 ] 123 0.166% 99.914% ( 6600, 9900 ] 23 0.031% 99.945% ( 9900, 14000 ] 5 0.007% 99.951% ( 14000, 22000 ] 13 0.018% 99.969% ( 22000, 33000 ] 17 0.023% 99.992% ( 33000, 50000 ] 6 0.008% 100.000% ** DB Stats ** Uptime(secs): 19.5 total, 3.7 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 201.54 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.567 H:M:S, 18.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 268.46 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.020 H:M:S, 0.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 20000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 19378.7 MB (estimated) FileSize: 19378.7 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 346.010 micros/op 11539 ops/sec 20.104 seconds 231996 operations; 11.2 MB/s (57999 of 57999 found) Microseconds per read: Count: 231996 Average: 346.0275 StdDev: 1012.70 Min: 2 Median: 236.1225 Max: 88172 Percentiles: P50: 236.12 P75: 345.71 P99: 2008.96 P99.9: 15274.62 P99.99: 42761.51 ------------------------------------------------------ ( 1, 2 ] 9 0.004% 0.004% ( 2, 3 ] 58 0.025% 0.029% ( 3, 4 ] 73 0.031% 0.060% ( 4, 6 ] 195 0.084% 0.144% ( 6, 10 ] 355 0.153% 0.297% ( 10, 15 ] 1478 0.637% 0.934% ( 15, 22 ] 3322 1.432% 2.366% ( 22, 34 ] 4619 1.991% 4.357% ( 34, 51 ] 2695 1.162% 5.519% ( 51, 76 ] 473 0.204% 5.723% ( 76, 110 ] 121 0.052% 5.775% ( 110, 170 ] 16792 7.238% 13.013% # ( 170, 250 ] 103817 44.749% 57.763% ######### ( 250, 380 ] 54317 23.413% 81.176% ##### ( 380, 580 ] 27356 11.792% 92.967% ## ( 580, 870 ] 9943 4.286% 97.253% # ( 870, 1300 ] 3089 1.331% 98.584% ( 1300, 1900 ] 893 0.385% 98.969% ( 1900, 2900 ] 652 0.281% 99.250% ( 2900, 4400 ] 944 0.407% 99.657% ( 4400, 6600 ] 304 0.131% 99.788% ( 6600, 9900 ] 131 0.056% 99.845% ( 9900, 14000 ] 110 0.047% 99.892% ( 14000, 22000 ] 113 0.049% 99.941% ( 22000, 33000 ] 96 0.041% 99.982% ( 33000, 50000 ] 31 0.013% 99.996% ( 50000, 75000 ] 9 0.004% 100.000% ( 75000, 110000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 3921081 rocksdb.bloom.filter.full.positive COUNT : 269300 rocksdb.bloom.filter.full.true.positive COUNT : 231029 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 967 rocksdb.memtable.miss COUNT : 231029 rocksdb.l0.hit COUNT : 27203 rocksdb.l1.hit COUNT : 6412 rocksdb.l2andup.hit COUNT : 197414 rocksdb.compaction.key.drop.new COUNT : 359232 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 4095278 rocksdb.number.keys.read COUNT : 231996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 4226326896 rocksdb.bytes.read COUNT : 231996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 476 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 3830912 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 11062964782 rocksdb.compact.write.bytes COUNT : 10685359335 rocksdb.flush.write.bytes COUNT : 4155399223 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 23 rocksdb.number.superversion_acquires COUNT : 713 rocksdb.number.superversion_releases COUNT : 677 rocksdb.number.superversion_cleanups COUNT : 174 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 145 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 4128040960 rocksdb.memtable.garbage.bytes.at.flush COUNT : 7233536 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 11084226845 rocksdb.non.last.level.read.count COUNT : 2645631 rocksdb.block.checksum.compute.count COUNT : 2950723 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 235.348370 P95 : 716.553995 P99 : 2007.258806 P100 : 88171.000000 COUNT : 231996 SUM : 80063280 rocksdb.db.write.micros P50 : 1.537044 P95 : 4.995501 P99 : 7.227850 P100 : 548702.000000 COUNT : 4095278 SUM : 14793368 rocksdb.compaction.times.micros P50 : 1530555.555556 P95 : 9635000.000000 P99 : 13139000.000000 P100 : 13451595.000000 COUNT : 41 SUM : 90665063 rocksdb.compaction.times.cpu_micros P50 : 1115000.000000 P95 : 1897500.000000 P99 : 5598000.000000 P100 : 5792463.000000 COUNT : 41 SUM : 51883734 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 821.666667 P95 : 6600.000000 P99 : 86862.000000 P100 : 86862.000000 COUNT : 60 SUM : 170124 rocksdb.compaction.outfile.sync.micros P50 : 1840.000000 P95 : 6592.142857 P99 : 30690.000000 P100 : 37203.000000 COUNT : 121 SUM : 358616 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 189.259259 P95 : 3605.000000 P99 : 40395.000000 P100 : 44387.000000 COUNT : 113 SUM : 138125 rocksdb.table.open.io.micros P50 : 6729.921260 P95 : 18930.612245 P99 : 22528.000000 P100 : 30189.000000 COUNT : 476 SUM : 3338876 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 4.065346 P95 : 10.949989 P99 : 275.463115 P100 : 55035.000000 COUNT : 2677657 SUM : 50203436 rocksdb.read.block.get.micros P50 : 202.788498 P95 : 550.949008 P99 : 1698.758815 P100 : 63059.000000 COUNT : 272580 SUM : 70546655 rocksdb.write.raw.block.micros P50 : 0.538489 P95 : 1.460220 P99 : 4.668085 P100 : 70164.000000 COUNT : 3594654 SUM : 12377744 rocksdb.numfiles.in.singlecompaction P50 : 3.108696 P95 : 5.675000 P99 : 26.000000 P100 : 26.000000 COUNT : 53 SUM : 209 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.559067 P95 : 1120.477603 P99 : 2044.137931 P100 : 548521.000000 COUNT : 17974 SUM : 3832440 rocksdb.sst.read.micros P50 : 1.964614 P95 : 224.639434 P99 : 503.655646 P100 : 63053.000000 COUNT : 2645650 SUM : 107939055 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 231996 SUM : 231996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 4095278 SUM : 4226326896 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 218333.333333 P95 : 570000.000000 P99 : 802000.000000 P100 : 834068.000000 COUNT : 60 SUM : 15511937 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 145 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0