[2024-11-21 01:50:43.517648] Starting SPDK v24.09.1-pre git sha1 b18e1bd62 / DPDK 23.11.0 initialization... [2024-11-21 01:50:43.517773] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1404264 ] Set seed to 1732150243515197 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-21 01:50:43.618019] app.c: 917:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-21 01:50:43.660080] app.c: 610:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-21 01:50:43.660129] app.c: 611:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1404264' to capture a snapshot of events at runtime. [2024-11-21 01:50:43.660139] app.c: 616:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-21 01:50:43.660149] app.c: 617:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-21 01:50:43.660157] app.c: 618:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1404264 for offline analysis/debug. [2024-11-21 01:50:43.660251] reactor.c: 990:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Thu Nov 21 01:50:47 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/21-01:50:55 ... thread 0: (1000000,1000000) ops and (137172.5,137172.5) ops/second in (7.290089,7.290089) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 6/4 818.25 MB 18.9 0.4 0.0 0.4 1.2 0.8 0.0 1.5 40.4 127.9 9.35 3.86 10 0.935 383K 537 0.0 0.0 L1 2/2 174.92 MB 0.0 0.6 0.4 0.2 0.6 0.4 0.0 1.5 101.4 101.2 6.21 2.88 1 6.210 639K 1472 0.0 0.0 L2 5/3 453.49 MB 2.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 48/36 3.61 GB 1.4 2.7 1.9 0.8 2.6 1.9 0.3 1.4 101.0 100.2 27.03 12.96 14 1.931 1698K 2447 0.0 0.0 L4 194/12 12.21 GB 1.2 2.8 2.4 0.4 2.8 2.4 0.1 1.2 103.1 103.1 27.38 13.42 20 1.369 1000K 0 0.0 0.0 L5 1076/0 88.46 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1331/57 105.69 GB 0.0 6.4 4.7 1.7 7.2 5.5 0.8 9.0 93.8 105.1 69.97 33.13 45 1.555 3722K 4456 0.0 0.0 Int 0/0 0.00 KB 0.0 6.4 4.7 1.7 7.2 5.5 0.8 9.0 93.8 105.1 69.97 33.13 45 1.555 3722K 4456 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.4 4.7 1.7 6.4 4.7 0.0 0.0 103.5 103.1 63.39 30.79 36 1.761 3722K 4456 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 124.4 6.58 2.34 9 0.731 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.5 total, 7.3 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.18 GB write, 985.08 MB/s write, 6.41 GB read, 878.96 MB/s read, 70.0 seconds Interval compaction: 7.18 GB write, 1008.71 MB/s write, 6.41 GB read, 900.04 MB/s read, 70.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 39, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 39, interval: 39 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 230857 Average: 13.1952 StdDev: 160.70 Min: 1 Median: 1.8716 Max: 18913 Percentiles: P50: 1.87 P75: 2.87 P99: 12.95 P99.9: 2675.09 P99.99: 5205.56 ------------------------------------------------------ [ 0, 1 ] 8819 3.820% 3.820% # ( 1, 2 ] 122317 52.984% 56.804% ########### ( 2, 3 ] 48293 20.919% 77.723% #### ( 3, 4 ] 22087 9.567% 87.290% ## ( 4, 6 ] 21281 9.218% 96.509% ## ( 6, 10 ] 5140 2.226% 98.735% ( 10, 15 ] 1036 0.449% 99.184% ( 15, 22 ] 146 0.063% 99.247% ( 22, 34 ] 16 0.007% 99.254% ( 34, 51 ] 18 0.008% 99.262% ( 51, 76 ] 7 0.003% 99.265% ( 76, 110 ] 2 0.001% 99.266% ( 110, 170 ] 12 0.005% 99.271% ( 170, 250 ] 57 0.025% 99.296% ( 250, 380 ] 130 0.056% 99.352% ( 380, 580 ] 202 0.088% 99.439% ( 580, 870 ] 313 0.136% 99.575% ( 870, 1300 ] 347 0.150% 99.725% ( 1300, 1900 ] 252 0.109% 99.835% ( 1900, 2900 ] 195 0.084% 99.919% ( 2900, 4400 ] 150 0.065% 99.984% ( 4400, 6600 ] 38 0.016% 100.000% ( 6600, 9900 ] 5 0.002% 100.003% ( 9900, 14000 ] 1 0.000% 100.003% ( 14000, 22000 ] 1 0.000% 100.003% ** Level 1 read latency histogram (micros): Count: 56909 Average: 20.8373 StdDev: 220.60 Min: 1 Median: 2.0753 Max: 10865 Percentiles: P50: 2.08 P75: 3.13 P99: 259.20 P99.9: 3901.82 P99.99: 6329.33 ------------------------------------------------------ [ 0, 1 ] 621 1.091% 1.091% ( 1, 2 ] 26742 46.991% 48.082% ######### ( 2, 3 ] 14508 25.493% 73.575% ##### ( 3, 4 ] 6225 10.939% 84.514% ## ( 4, 6 ] 6309 11.086% 95.600% ## ( 6, 10 ] 1535 2.697% 98.297% # ( 10, 15 ] 336 0.590% 98.888% ( 15, 22 ] 38 0.067% 98.954% ( 22, 34 ] 4 0.007% 98.961% ( 34, 51 ] 3 0.005% 98.967% ( 51, 76 ] 5 0.009% 98.976% ( 76, 110 ] 2 0.004% 98.979% ( 110, 170 ] 2 0.004% 98.983% ( 170, 250 ] 9 0.016% 98.998% ( 250, 380 ] 27 0.047% 99.046% ( 380, 580 ] 49 0.086% 99.132% ( 580, 870 ] 82 0.144% 99.276% ( 870, 1300 ] 121 0.213% 99.489% ( 1300, 1900 ] 96 0.169% 99.657% ( 1900, 2900 ] 89 0.156% 99.814% ( 2900, 4400 ] 75 0.132% 99.946% ( 4400, 6600 ] 30 0.053% 99.998% ( 6600, 9900 ] 1 0.002% 100.000% ( 9900, 14000 ] 1 0.002% 100.002% ** Level 2 read latency histogram (micros): Count: 421752 Average: 12.7084 StdDev: 154.39 Min: 1 Median: 1.9729 Max: 8858 Percentiles: P50: 1.97 P75: 3.02 P99: 12.46 P99.9: 1301.92 P99.99: 1884.39 ------------------------------------------------------ [ 0, 1 ] 11704 2.775% 2.775% # ( 1, 2 ] 204724 48.541% 51.316% ########## ( 2, 3 ] 99091 23.495% 74.812% ##### ( 3, 4 ] 44091 10.454% 85.266% ## ( 4, 6 ] 43635 10.346% 95.612% ## ( 6, 10 ] 12879 3.054% 98.666% # ( 10, 15 ] 2872 0.681% 99.347% ( 15, 22 ] 472 0.112% 99.458% ( 22, 34 ] 95 0.023% 99.481% ( 34, 51 ] 49 0.012% 99.493% ( 51, 76 ] 32 0.008% 99.500% ( 76, 110 ] 16 0.004% 99.504% ( 110, 170 ] 57 0.014% 99.517% ( 170, 250 ] 94 0.022% 99.540% ( 250, 380 ] 197 0.047% 99.586% ( 380, 580 ] 355 0.084% 99.671% ( 580, 870 ] 458 0.109% 99.779% ( 870, 1300 ] 508 0.120% 99.900% ( 1300, 1900 ] 391 0.093% 99.992% ( 1900, 2900 ] 377 0.089% 100.082% ( 2900, 4400 ] 280 0.066% 100.148% ( 4400, 6600 ] 82 0.019% 100.168% ( 6600, 9900 ] 8 0.002% 100.170% ** Level 3 read latency histogram (micros): Count: 885635 Average: 12.0545 StdDev: 152.36 Min: 1 Median: 1.9756 Max: 10042 Percentiles: P50: 1.98 P75: 2.97 P99: 9.85 P99.9: 704.35 P99.99: 970.63 ------------------------------------------------------ [ 0, 1 ] 13595 1.535% 1.535% ( 1, 2 ] 439969 49.678% 51.213% ########## ( 2, 3 ] 218065 24.622% 75.836% ##### ( 3, 4 ] 93339 10.539% 86.375% ## ( 4, 6 ] 89935 10.155% 96.530% ## ( 6, 10 ] 22719 2.565% 99.095% # ( 10, 15 ] 4720 0.533% 99.628% ( 15, 22 ] 690 0.078% 99.706% ( 22, 34 ] 53 0.006% 99.712% ( 34, 51 ] 45 0.005% 99.717% ( 51, 76 ] 55 0.006% 99.723% ( 76, 110 ] 24 0.003% 99.726% ( 110, 170 ] 74 0.008% 99.734% ( 170, 250 ] 142 0.016% 99.750% ( 250, 380 ] 306 0.035% 99.785% ( 380, 580 ] 603 0.068% 99.853% ( 580, 870 ] 971 0.110% 99.963% ( 870, 1300 ] 1036 0.117% 100.080% ( 1300, 1900 ] 791 0.089% 100.169% ( 1900, 2900 ] 702 0.079% 100.248% ( 2900, 4400 ] 584 0.066% 100.314% ( 4400, 6600 ] 130 0.015% 100.329% ( 6600, 9900 ] 38 0.004% 100.333% ( 9900, 14000 ] 1 0.000% 100.333% ** Level 4 read latency histogram (micros): Count: 129919 Average: 57.5931 StdDev: 375.00 Min: 1 Median: 1.9752 Max: 15942 Percentiles: P50: 1.98 P75: 3.02 P99: 1785.64 P99.9: 3950.79 P99.99: 4279.24 ------------------------------------------------------ [ 0, 1 ] 1730 1.332% 1.332% ( 1, 2 ] 64838 49.906% 51.238% ########## ( 2, 3 ] 30597 23.551% 74.789% ##### ( 3, 4 ] 13790 10.614% 85.403% ## ( 4, 6 ] 11611 8.937% 94.340% ## ( 6, 10 ] 2453 1.888% 96.228% ( 10, 15 ] 561 0.432% 96.660% ( 15, 22 ] 79 0.061% 96.721% ( 22, 34 ] 7 0.005% 96.726% ( 34, 51 ] 6 0.005% 96.731% ( 51, 76 ] 6 0.005% 96.736% ( 76, 110 ] 8 0.006% 96.742% ( 110, 170 ] 39 0.030% 96.772% ( 170, 250 ] 122 0.094% 96.866% ( 250, 380 ] 207 0.159% 97.025% ( 380, 580 ] 426 0.328% 97.353% ( 580, 870 ] 691 0.532% 97.885% ( 870, 1300 ] 862 0.663% 98.548% ( 1300, 1900 ] 725 0.558% 99.106% ( 1900, 2900 ] 657 0.506% 99.612% ( 2900, 4400 ] 534 0.411% 100.023% ( 4400, 6600 ] 126 0.097% 100.120% ( 6600, 9900 ] 23 0.018% 100.138% ( 14000, 22000 ] 2 0.002% 100.139% ** Level 5 read latency histogram (micros): Count: 12502 Average: 1658.5503 StdDev: 1440.56 Min: 89 Median: 1162.0109 Max: 27728 Percentiles: P50: 1162.01 P75: 2290.30 P99: 6520.32 P99.9: 9811.22 P99.99: 18666.13 ------------------------------------------------------ ( 76, 110 ] 22 0.176% 0.176% ( 110, 170 ] 98 0.784% 0.960% ( 170, 250 ] 234 1.872% 2.832% ( 250, 380 ] 619 4.951% 7.783% # ( 380, 580 ] 1439 11.510% 19.293% ## ( 580, 870 ] 2218 17.741% 37.034% #### ( 870, 1300 ] 2387 19.093% 56.127% #### ( 1300, 1900 ] 1776 14.206% 70.333% ### ( 1900, 2900 ] 1495 11.958% 82.291% ## ( 2900, 4400 ] 1503 12.022% 94.313% ## ( 4400, 6600 ] 608 4.863% 99.176% # ( 6600, 9900 ] 93 0.744% 99.920% ( 9900, 14000 ] 7 0.056% 99.976% ( 14000, 22000 ] 3 0.024% 100.000% ( 22000, 33000 ] 1 0.008% 100.008% ** DB Stats ** Uptime(secs): 7.5 total, 7.3 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 131.80 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.377 H:M:S, 45.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 134.95 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.377 H:M:S, 46.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/11/21-01:51:01 ... thread 0: (1000000,2000000) ops and (169076.2,151462.6) ops/second in (5.914494,13.204583) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 7/6 1.23 GB 6.3 1.3 0.0 1.3 3.1 1.8 0.0 1.7 67.2 163.2 19.68 10.60 27 0.729 1343K 1646 0.0 0.0 L1 6/6 561.43 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.3 120.9 120.6 11.86 6.63 2 5.930 1456K 3817 0.0 0.0 L2 3/3 369.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.7 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 27/13 2.12 GB 0.9 4.5 2.1 2.4 4.5 2.1 0.3 2.1 110.5 109.8 41.82 21.81 17 2.460 3485K 6944 0.0 0.0 L4 168/0 9.73 GB 1.0 5.4 3.9 1.5 5.4 3.8 0.5 1.4 114.4 113.7 48.34 25.93 35 1.381 2121K 0 0.0 0.0 L5 1128/0 92.71 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1339/28 106.69 GB 0.0 12.6 7.0 5.6 14.4 8.8 5.7 7.8 106.1 121.0 121.69 64.97 81 1.502 8406K 12K 0.0 0.0 Int 0/0 0.00 KB 0.0 12.6 7.0 5.6 14.4 8.8 5.7 7.8 106.1 121.0 121.69 64.97 81 1.502 8406K 12K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.6 7.0 5.6 12.5 7.0 0.0 0.0 118.2 117.6 109.20 59.65 58 1.883 8406K 12K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 151.3 12.49 5.33 23 0.543 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 13.4 total, 13.2 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 14.38 GB write, 1100.56 MB/s write, 12.61 GB read, 964.68 MB/s read, 121.7 seconds Interval compaction: 14.38 GB write, 1115.13 MB/s write, 12.61 GB read, 977.46 MB/s read, 121.7 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 62, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 62, interval: 62 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 609516 Average: 10.2186 StdDev: 130.11 Min: 1 Median: 1.8307 Max: 18913 Percentiles: P50: 1.83 P75: 2.77 P99: 10.39 P99.9: 1835.33 P99.99: 3770.29 ------------------------------------------------------ [ 0, 1 ] 26238 4.305% 4.305% # ( 1, 2 ] 335289 55.009% 59.314% ########### ( 2, 3 ] 123580 20.275% 79.589% #### ( 3, 4 ] 55577 9.118% 88.707% ## ( 4, 6 ] 51272 8.412% 97.119% ## ( 6, 10 ] 11292 1.853% 98.972% ( 10, 15 ] 2206 0.362% 99.334% ( 15, 22 ] 263 0.043% 99.377% ( 22, 34 ] 23 0.004% 99.380% ( 34, 51 ] 33 0.005% 99.386% ( 51, 76 ] 33 0.005% 99.391% ( 76, 110 ] 17 0.003% 99.394% ( 110, 170 ] 119 0.020% 99.414% ( 170, 250 ] 229 0.038% 99.451% ( 250, 380 ] 350 0.057% 99.509% ( 380, 580 ] 569 0.093% 99.602% ( 580, 870 ] 769 0.126% 99.728% ( 870, 1300 ] 680 0.112% 99.840% ( 1300, 1900 ] 413 0.068% 99.907% ( 1900, 2900 ] 359 0.059% 99.966% ( 2900, 4400 ] 250 0.041% 100.007% ( 4400, 6600 ] 68 0.011% 100.019% ( 6600, 9900 ] 9 0.001% 100.020% ( 9900, 14000 ] 1 0.000% 100.020% ( 14000, 22000 ] 1 0.000% 100.020% ** Level 1 read latency histogram (micros): Count: 192399 Average: 13.6874 StdDev: 159.61 Min: 1 Median: 1.9771 Max: 10865 Percentiles: P50: 1.98 P75: 2.96 P99: 14.30 P99.9: 2689.61 P99.99: 4289.08 ------------------------------------------------------ [ 0, 1 ] 2793 1.452% 1.452% ( 1, 2 ] 95601 49.689% 51.141% ########## ( 2, 3 ] 47749 24.818% 75.958% ##### ( 3, 4 ] 19340 10.052% 86.010% ## ( 4, 6 ] 19152 9.954% 95.965% ## ( 6, 10 ] 5057 2.628% 98.593% # ( 10, 15 ] 913 0.475% 99.068% ( 15, 22 ] 102 0.053% 99.121% ( 22, 34 ] 10 0.005% 99.126% ( 34, 51 ] 21 0.011% 99.137% ( 51, 76 ] 27 0.014% 99.151% ( 76, 110 ] 23 0.012% 99.163% ( 110, 170 ] 89 0.046% 99.209% ( 170, 250 ] 112 0.058% 99.267% ( 250, 380 ] 160 0.083% 99.350% ( 380, 580 ] 228 0.119% 99.469% ( 580, 870 ] 267 0.139% 99.608% ( 870, 1300 ] 270 0.140% 99.748% ( 1300, 1900 ] 173 0.090% 99.838% ( 1900, 2900 ] 154 0.080% 99.918% ( 2900, 4400 ] 152 0.079% 99.997% ( 4400, 6600 ] 40 0.021% 100.018% ( 6600, 9900 ] 2 0.001% 100.019% ( 9900, 14000 ] 1 0.001% 100.019% ** Level 2 read latency histogram (micros): Count: 454510 Average: 12.3340 StdDev: 150.98 Min: 1 Median: 1.9714 Max: 8858 Percentiles: P50: 1.97 P75: 3.01 P99: 12.32 P99.9: 1303.68 P99.99: 1914.41 ------------------------------------------------------ [ 0, 1 ] 12385 2.725% 2.725% # ( 1, 2 ] 221195 48.667% 51.392% ########## ( 2, 3 ] 106664 23.468% 74.860% ##### ( 3, 4 ] 47314 10.410% 85.269% ## ( 4, 6 ] 47201 10.385% 95.654% ## ( 6, 10 ] 13804 3.037% 98.692% # ( 10, 15 ] 3022 0.665% 99.356% ( 15, 22 ] 499 0.110% 99.466% ( 22, 34 ] 97 0.021% 99.488% ( 34, 51 ] 49 0.011% 99.498% ( 51, 76 ] 33 0.007% 99.506% ( 76, 110 ] 18 0.004% 99.510% ( 110, 170 ] 62 0.014% 99.523% ( 170, 250 ] 107 0.024% 99.547% ( 250, 380 ] 211 0.046% 99.593% ( 380, 580 ] 379 0.083% 99.677% ( 580, 870 ] 485 0.107% 99.783% ( 870, 1300 ] 528 0.116% 99.899% ( 1300, 1900 ] 406 0.089% 99.989% ( 1900, 2900 ] 385 0.085% 100.073% ( 2900, 4400 ] 288 0.063% 100.137% ( 4400, 6600 ] 85 0.019% 100.156% ( 6600, 9900 ] 8 0.002% 100.157% ** Level 3 read latency histogram (micros): Count: 1694932 Average: 10.8288 StdDev: 138.56 Min: 1 Median: 1.9558 Max: 10042 Percentiles: P50: 1.96 P75: 2.94 P99: 9.76 P99.9: 647.85 P99.99: 899.33 ------------------------------------------------------ [ 0, 1 ] 23749 1.401% 1.401% ( 1, 2 ] 861808 50.846% 52.247% ########## ( 2, 3 ] 412124 24.315% 76.562% ##### ( 3, 4 ] 173761 10.252% 86.814% ## ( 4, 6 ] 166494 9.823% 96.637% ## ( 6, 10 ] 42555 2.511% 99.148% # ( 10, 15 ] 8215 0.485% 99.633% ( 15, 22 ] 1122 0.066% 99.699% ( 22, 34 ] 86 0.005% 99.704% ( 34, 51 ] 97 0.006% 99.710% ( 51, 76 ] 132 0.008% 99.717% ( 76, 110 ] 57 0.003% 99.721% ( 110, 170 ] 210 0.012% 99.733% ( 170, 250 ] 379 0.022% 99.756% ( 250, 380 ] 716 0.042% 99.798% ( 380, 580 ] 1305 0.077% 99.875% ( 580, 870 ] 1834 0.108% 99.983% ( 870, 1300 ] 1767 0.104% 100.087% ( 1300, 1900 ] 1229 0.073% 100.160% ( 1900, 2900 ] 1103 0.065% 100.225% ( 2900, 4400 ] 980 0.058% 100.283% ( 4400, 6600 ] 192 0.011% 100.294% ( 6600, 9900 ] 49 0.003% 100.297% ( 9900, 14000 ] 1 0.000% 100.297% ** Level 4 read latency histogram (micros): Count: 403074 Average: 32.3617 StdDev: 264.41 Min: 1 Median: 1.9799 Max: 15942 Percentiles: P50: 1.98 P75: 3.04 P99: 951.46 P99.9: 3406.30 P99.99: 4041.98 ------------------------------------------------------ [ 0, 1 ] 6038 1.498% 1.498% ( 1, 2 ] 199514 49.498% 50.996% ########## ( 2, 3 ] 95089 23.591% 74.587% ##### ( 3, 4 ] 42279 10.489% 85.076% ## ( 4, 6 ] 39618 9.829% 94.905% ## ( 6, 10 ] 9769 2.424% 97.329% ( 10, 15 ] 1934 0.480% 97.809% ( 15, 22 ] 224 0.056% 97.864% ( 22, 34 ] 25 0.006% 97.870% ( 34, 51 ] 30 0.007% 97.878% ( 51, 76 ] 50 0.012% 97.890% ( 76, 110 ] 49 0.012% 97.902% ( 110, 170 ] 227 0.056% 97.959% ( 170, 250 ] 452 0.112% 98.071% ( 250, 380 ] 692 0.172% 98.243% ( 380, 580 ] 1168 0.290% 98.532% ( 580, 870 ] 1585 0.393% 98.926% ( 870, 1300 ] 1585 0.393% 99.319% ( 1300, 1900 ] 1059 0.263% 99.581% ( 1900, 2900 ] 995 0.247% 99.828% ( 2900, 4400 ] 856 0.212% 100.041% ( 4400, 6600 ] 175 0.043% 100.084% ( 6600, 9900 ] 28 0.007% 100.091% ( 14000, 22000 ] 2 0.000% 100.092% ** Level 5 read latency histogram (micros): Count: 31415 Average: 1215.3812 StdDev: 1209.31 Min: 77 Median: 803.4539 Max: 27728 Percentiles: P50: 803.45 P75: 1513.29 P99: 6045.31 P99.9: 9251.66 P99.99: 13917.12 ------------------------------------------------------ ( 76, 110 ] 200 0.637% 0.637% ( 110, 170 ] 1161 3.696% 4.332% # ( 170, 250 ] 1845 5.873% 10.205% # ( 250, 380 ] 2974 9.467% 19.672% ## ( 380, 580 ] 4808 15.305% 34.977% ### ( 580, 870 ] 6125 19.497% 54.474% #### ( 870, 1300 ] 5337 16.989% 71.463% ### ( 1300, 1900 ] 3126 9.951% 81.413% ## ( 1900, 2900 ] 2541 8.088% 89.502% ## ( 2900, 4400 ] 2405 7.656% 97.157% ## ( 4400, 6600 ] 774 2.464% 99.621% ( 6600, 9900 ] 109 0.347% 99.968% ( 9900, 14000 ] 7 0.022% 99.990% ( 14000, 22000 ] 3 0.010% 100.000% ( 22000, 33000 ] 1 0.003% 100.003% ** DB Stats ** Uptime(secs): 13.4 total, 5.9 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 147.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.388 H:M:S, 40.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 166.39 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.011 H:M:S, 34.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 4 num-running-flushes: 1 2024/11/21-01:51:06 ... thread 0: (1000000,3000000) ops and (189240.9,162259.9) ops/second in (5.284270,18.488853) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/10 2.15 GB 75.5 1.8 0.0 1.8 4.6 2.8 0.0 1.7 69.1 172.7 27.35 15.58 42 0.651 1918K 2202 0.0 0.0 L1 6/6 561.43 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.3 120.9 120.6 11.86 6.63 2 5.930 1456K 3817 0.0 0.0 L3 16/1 1.06 GB 1.0 6.1 2.5 3.6 6.1 2.5 0.3 2.4 122.8 122.1 51.00 29.28 20 2.550 4371K 8676 0.0 0.0 L4 174/4 9.73 GB 1.0 8.1 5.3 2.8 8.0 5.2 0.5 1.5 117.2 115.9 70.46 38.67 55 1.281 3838K 8299 0.0 0.0 L5 1144/0 94.06 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1354/21 107.55 GB 0.0 17.4 8.8 8.6 20.1 11.4 6.4 7.2 111.1 127.9 160.66 90.16 119 1.350 11M 22K 0.0 0.0 Int 0/0 0.00 KB 0.0 17.4 8.8 8.6 20.1 11.4 6.4 7.2 111.1 127.9 160.66 90.16 119 1.350 11M 22K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 17.4 8.8 8.6 17.3 8.7 0.0 0.0 124.7 123.7 143.18 82.11 83 1.725 11M 22K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 162.2 17.48 8.05 36 0.486 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.7 total, 18.5 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 20.06 GB write, 1100.59 MB/s write, 17.43 GB read, 956.27 MB/s read, 160.7 seconds Interval compaction: 20.06 GB write, 1111.00 MB/s write, 17.43 GB read, 965.31 MB/s read, 160.7 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 82, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 82, interval: 82 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 937656 Average: 10.0392 StdDev: 127.35 Min: 1 Median: 1.8218 Max: 23743 Percentiles: P50: 1.82 P75: 2.75 P99: 13.68 P99.9: 1523.79 P99.99: 3381.42 ------------------------------------------------------ [ 0, 1 ] 43403 4.629% 4.629% # ( 1, 2 ] 517686 55.211% 59.840% ########### ( 2, 3 ] 189342 20.193% 80.033% #### ( 3, 4 ] 82831 8.834% 88.866% ## ( 4, 6 ] 76402 8.148% 97.015% ## ( 6, 10 ] 16362 1.745% 98.760% ( 10, 15 ] 3064 0.327% 99.086% ( 15, 22 ] 366 0.039% 99.125% ( 22, 34 ] 34 0.004% 99.129% ( 34, 51 ] 47 0.005% 99.134% ( 51, 76 ] 59 0.006% 99.140% ( 76, 110 ] 345 0.037% 99.177% ( 110, 170 ] 1294 0.138% 99.315% ( 170, 250 ] 1117 0.119% 99.434% ( 250, 380 ] 917 0.098% 99.532% ( 380, 580 ] 1126 0.120% 99.652% ( 580, 870 ] 1184 0.126% 99.778% ( 870, 1300 ] 948 0.101% 99.880% ( 1300, 1900 ] 513 0.055% 99.934% ( 1900, 2900 ] 416 0.044% 99.979% ( 2900, 4400 ] 331 0.035% 100.014% ( 4400, 6600 ] 102 0.011% 100.025% ( 6600, 9900 ] 27 0.003% 100.028% ( 9900, 14000 ] 1 0.000% 100.028% ( 14000, 22000 ] 1 0.000% 100.028% ( 22000, 33000 ] 1 0.000% 100.028% ** Level 1 read latency histogram (micros): Count: 294284 Average: 11.6743 StdDev: 143.48 Min: 1 Median: 1.9251 Max: 10865 Percentiles: P50: 1.93 P75: 2.88 P99: 12.72 P99.9: 2184.09 P99.99: 4001.88 ------------------------------------------------------ [ 0, 1 ] 5112 1.737% 1.737% ( 1, 2 ] 153524 52.169% 53.906% ########## ( 2, 3 ] 70350 23.905% 77.811% ##### ( 3, 4 ] 27539 9.358% 87.169% ## ( 4, 6 ] 26774 9.098% 96.267% ## ( 6, 10 ] 7424 2.523% 98.790% # ( 10, 15 ] 1137 0.386% 99.176% ( 15, 22 ] 138 0.047% 99.223% ( 22, 34 ] 12 0.004% 99.227% ( 34, 51 ] 24 0.008% 99.235% ( 51, 76 ] 33 0.011% 99.247% ( 76, 110 ] 45 0.015% 99.262% ( 110, 170 ] 176 0.060% 99.322% ( 170, 250 ] 193 0.066% 99.387% ( 250, 380 ] 231 0.078% 99.466% ( 380, 580 ] 324 0.110% 99.576% ( 580, 870 ] 367 0.125% 99.701% ( 870, 1300 ] 336 0.114% 99.815% ( 1300, 1900 ] 201 0.068% 99.883% ( 1900, 2900 ] 175 0.059% 99.943% ( 2900, 4400 ] 190 0.065% 100.007% ( 4400, 6600 ] 47 0.016% 100.023% ( 6600, 9900 ] 4 0.001% 100.024% ( 9900, 14000 ] 1 0.000% 100.025% ** Level 2 read latency histogram (micros): Count: 487231 Average: 12.3170 StdDev: 156.27 Min: 1 Median: 1.9746 Max: 15928 Percentiles: P50: 1.97 P75: 3.01 P99: 12.26 P99.9: 1343.04 P99.99: 2033.53 ------------------------------------------------------ [ 0, 1 ] 12747 2.616% 2.616% # ( 1, 2 ] 236888 48.619% 51.235% ########## ( 2, 3 ] 115317 23.668% 74.903% ##### ( 3, 4 ] 50666 10.399% 85.302% ## ( 4, 6 ] 50576 10.380% 95.682% ## ( 6, 10 ] 14722 3.022% 98.704% # ( 10, 15 ] 3186 0.654% 99.358% ( 15, 22 ] 523 0.107% 99.465% ( 22, 34 ] 98 0.020% 99.485% ( 34, 51 ] 51 0.010% 99.496% ( 51, 76 ] 45 0.009% 99.505% ( 76, 110 ] 28 0.006% 99.511% ( 110, 170 ] 74 0.015% 99.526% ( 170, 250 ] 123 0.025% 99.551% ( 250, 380 ] 230 0.047% 99.598% ( 380, 580 ] 400 0.082% 99.680% ( 580, 870 ] 500 0.103% 99.783% ( 870, 1300 ] 540 0.111% 99.894% ( 1300, 1900 ] 415 0.085% 99.979% ( 1900, 2900 ] 399 0.082% 100.061% ( 2900, 4400 ] 304 0.062% 100.123% ( 4400, 6600 ] 98 0.020% 100.143% ( 6600, 9900 ] 12 0.002% 100.146% ( 9900, 14000 ] 3 0.001% 100.147% ( 14000, 22000 ] 1 0.000% 100.147% ** Level 3 read latency histogram (micros): Count: 2184224 Average: 10.3828 StdDev: 140.12 Min: 1 Median: 1.9389 Max: 31431 Percentiles: P50: 1.94 P75: 2.92 P99: 9.70 P99.9: 632.22 P99.99: 899.64 ------------------------------------------------------ [ 0, 1 ] 35605 1.630% 1.630% ( 1, 2 ] 1125230 51.516% 53.146% ########## ( 2, 3 ] 521648 23.883% 77.029% ##### ( 3, 4 ] 219463 10.048% 87.077% ## ( 4, 6 ] 210732 9.648% 96.724% ## ( 6, 10 ] 53761 2.461% 99.186% ( 10, 15 ] 9881 0.452% 99.638% ( 15, 22 ] 1331 0.061% 99.699% ( 22, 34 ] 108 0.005% 99.704% ( 34, 51 ] 123 0.006% 99.710% ( 51, 76 ] 169 0.008% 99.717% ( 76, 110 ] 93 0.004% 99.722% ( 110, 170 ] 392 0.018% 99.740% ( 170, 250 ] 563 0.026% 99.765% ( 250, 380 ] 929 0.043% 99.808% ( 380, 580 ] 1612 0.074% 99.882% ( 580, 870 ] 2220 0.102% 99.983% ( 870, 1300 ] 2112 0.097% 100.080% ( 1300, 1900 ] 1407 0.064% 100.144% ( 1900, 2900 ] 1232 0.056% 100.201% ( 2900, 4400 ] 1168 0.053% 100.254% ( 4400, 6600 ] 278 0.013% 100.267% ( 6600, 9900 ] 76 0.003% 100.271% ( 9900, 14000 ] 3 0.000% 100.271% ( 14000, 22000 ] 2 0.000% 100.271% ( 22000, 33000 ] 1 0.000% 100.271% ** Level 4 read latency histogram (micros): Count: 778262 Average: 24.6339 StdDev: 228.89 Min: 1 Median: 1.9517 Max: 27569 Percentiles: P50: 1.95 P75: 2.98 P99: 593.62 P99.9: 2561.27 P99.99: 3293.10 ------------------------------------------------------ [ 0, 1 ] 15009 1.929% 1.929% ( 1, 2 ] 393099 50.510% 52.438% ########## ( 2, 3 ] 179678 23.087% 75.525% ##### ( 3, 4 ] 78198 10.048% 85.573% ## ( 4, 6 ] 75041 9.642% 95.215% ## ( 6, 10 ] 19495 2.505% 97.720% # ( 10, 15 ] 3521 0.452% 98.173% ( 15, 22 ] 393 0.050% 98.223% ( 22, 34 ] 42 0.005% 98.229% ( 34, 51 ] 71 0.009% 98.238% ( 51, 76 ] 102 0.013% 98.251% ( 76, 110 ] 202 0.026% 98.277% ( 110, 170 ] 947 0.122% 98.398% ( 170, 250 ] 1147 0.147% 98.546% ( 250, 380 ] 1388 0.178% 98.724% ( 380, 580 ] 2027 0.260% 98.985% ( 580, 870 ] 2541 0.326% 99.311% ( 870, 1300 ] 2365 0.304% 99.615% ( 1300, 1900 ] 1409 0.181% 99.796% ( 1900, 2900 ] 1223 0.157% 99.953% ( 2900, 4400 ] 1092 0.140% 100.094% ( 4400, 6600 ] 259 0.033% 100.127% ( 6600, 9900 ] 55 0.007% 100.134% ( 9900, 14000 ] 4 0.001% 100.134% ( 14000, 22000 ] 4 0.001% 100.135% ( 22000, 33000 ] 1 0.000% 100.135% ** Level 5 read latency histogram (micros): Count: 54604 Average: 979.2521 StdDev: 1142.04 Min: 75 Median: 604.3559 Max: 31268 Percentiles: P50: 604.36 P75: 1165.71 P99: 5830.24 P99.9: 9196.45 P99.99: 15079.20 ------------------------------------------------------ ( 51, 76 ] 2 0.004% 0.004% ( 76, 110 ] 1198 2.194% 2.198% ( 110, 170 ] 5183 9.492% 11.690% ## ( 170, 250 ] 5606 10.267% 21.956% ## ( 250, 380 ] 6164 11.289% 33.245% ## ( 380, 580 ] 8366 15.321% 48.566% ### ( 580, 870 ] 9323 17.074% 65.640% ### ( 870, 1300 ] 7432 13.611% 79.251% ### ( 1300, 1900 ] 4048 7.413% 86.664% # ( 1900, 2900 ] 3054 5.593% 92.257% # ( 2900, 4400 ] 3013 5.518% 97.775% # ( 4400, 6600 ] 1029 1.884% 99.659% ( 6600, 9900 ] 167 0.306% 99.965% ( 9900, 14000 ] 13 0.024% 99.989% ( 14000, 22000 ] 4 0.007% 99.996% ( 22000, 33000 ] 3 0.005% 100.002% ** DB Stats ** Uptime(secs): 18.7 total, 5.3 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 158.18 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.923 H:M:S, 37.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 186.26 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.535 H:M:S, 29.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 3 num-running-flushes: 1 2024/11/21-01:51:09 ... thread 0: (1000000,4000000) ops and (286304.1,181970.0) ops/second in (3.492790,21.981643) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/10 3.13 GB 176.2 2.8 0.0 2.8 6.6 3.7 0.0 1.8 84.1 195.6 34.44 22.17 62 0.555 2942K 3218 0.0 0.0 L1 6/6 561.43 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.3 120.9 120.6 11.86 6.63 2 5.930 1456K 3817 0.0 0.0 L3 15/0 956.16 MB 1.0 6.1 2.5 3.6 6.1 2.5 0.3 2.4 122.8 122.1 51.00 29.28 20 2.550 4371K 8676 0.0 0.0 L4 172/0 9.54 GB 1.0 8.4 5.4 3.0 8.3 5.3 0.5 1.5 118.8 117.4 72.27 40.11 56 1.290 4107K 8299 0.0 0.0 L5 1148/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1359/16 108.53 GB 0.0 18.7 8.9 9.8 22.3 12.5 6.7 6.0 113.1 134.9 169.57 98.19 140 1.211 12M 24K 0.0 0.0 Int 0/0 0.00 KB 0.0 18.7 8.9 9.8 22.3 12.5 6.7 6.0 113.1 134.9 169.57 98.19 140 1.211 12M 24K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 18.7 8.9 9.8 18.6 8.8 0.0 0.0 128.8 127.9 148.87 87.35 88 1.692 12M 24K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 185.7 20.70 10.84 52 0.398 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 22.2 total, 22.0 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 22.34 GB write, 1032.44 MB/s write, 18.73 GB read, 865.64 MB/s read, 169.6 seconds Interval compaction: 22.34 GB write, 1040.65 MB/s write, 18.73 GB read, 872.53 MB/s read, 169.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 1, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 1, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 82, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 1, total-stops: 82, interval: 83 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1224311 Average: 11.3981 StdDev: 115.89 Min: 0 Median: 1.7231 Max: 23743 Percentiles: P50: 1.72 P75: 2.57 P99: 199.12 P99.9: 1194.15 P99.99: 2659.46 ------------------------------------------------------ [ 0, 1 ] 122247 9.985% 9.985% ## ( 1, 2 ] 677531 55.340% 65.325% ########### ( 2, 3 ] 207679 16.963% 82.288% ### ( 3, 4 ] 88871 7.259% 89.547% # ( 4, 6 ] 79337 6.480% 96.027% # ( 6, 10 ] 17125 1.399% 97.425% ( 10, 15 ] 3298 0.269% 97.695% ( 15, 22 ] 407 0.033% 97.728% ( 22, 34 ] 34 0.003% 97.731% ( 34, 51 ] 50 0.004% 97.735% ( 51, 76 ] 59 0.005% 97.740% ( 76, 110 ] 2946 0.241% 97.980% ( 110, 170 ] 10534 0.860% 98.841% ( 170, 250 ] 5359 0.438% 99.278% ( 250, 380 ] 2647 0.216% 99.495% ( 380, 580 ] 2551 0.208% 99.703% ( 580, 870 ] 1640 0.134% 99.837% ( 870, 1300 ] 1025 0.084% 99.921% ( 1300, 1900 ] 523 0.043% 99.963% ( 1900, 2900 ] 430 0.035% 99.999% ( 2900, 4400 ] 335 0.027% 100.026% ( 4400, 6600 ] 102 0.008% 100.034% ( 6600, 9900 ] 27 0.002% 100.036% ( 9900, 14000 ] 1 0.000% 100.037% ( 14000, 22000 ] 1 0.000% 100.037% ( 22000, 33000 ] 1 0.000% 100.037% ** Level 1 read latency histogram (micros): Count: 303282 Average: 12.0265 StdDev: 141.95 Min: 1 Median: 1.9399 Max: 10865 Percentiles: P50: 1.94 P75: 2.91 P99: 88.36 P99.9: 2137.03 P99.99: 3994.78 ------------------------------------------------------ [ 0, 1 ] 5233 1.725% 1.725% ( 1, 2 ] 155770 51.361% 53.087% ########## ( 2, 3 ] 72635 23.950% 77.037% ##### ( 3, 4 ] 28970 9.552% 86.589% ## ( 4, 6 ] 28306 9.333% 95.922% ## ( 6, 10 ] 7791 2.569% 98.491% # ( 10, 15 ] 1231 0.406% 98.897% ( 15, 22 ] 183 0.060% 98.957% ( 22, 34 ] 19 0.006% 98.963% ( 34, 51 ] 24 0.008% 98.971% ( 51, 76 ] 33 0.011% 98.982% ( 76, 110 ] 149 0.049% 99.031% ( 110, 170 ] 554 0.183% 99.214% ( 170, 250 ] 400 0.132% 99.346% ( 250, 380 ] 324 0.107% 99.453% ( 380, 580 ] 387 0.128% 99.580% ( 580, 870 ] 391 0.129% 99.709% ( 870, 1300 ] 336 0.111% 99.820% ( 1300, 1900 ] 201 0.066% 99.886% ( 1900, 2900 ] 176 0.058% 99.944% ( 2900, 4400 ] 190 0.063% 100.007% ( 4400, 6600 ] 47 0.015% 100.022% ( 6600, 9900 ] 4 0.001% 100.024% ( 9900, 14000 ] 1 0.000% 100.024% ** Level 2 read latency histogram (micros): Count: 487231 Average: 12.3170 StdDev: 156.27 Min: 1 Median: 1.9746 Max: 15928 Percentiles: P50: 1.97 P75: 3.01 P99: 12.26 P99.9: 1343.04 P99.99: 2033.53 ------------------------------------------------------ [ 0, 1 ] 12747 2.616% 2.616% # ( 1, 2 ] 236888 48.619% 51.235% ########## ( 2, 3 ] 115317 23.668% 74.903% ##### ( 3, 4 ] 50666 10.399% 85.302% ## ( 4, 6 ] 50576 10.380% 95.682% ## ( 6, 10 ] 14722 3.022% 98.704% # ( 10, 15 ] 3186 0.654% 99.358% ( 15, 22 ] 523 0.107% 99.465% ( 22, 34 ] 98 0.020% 99.485% ( 34, 51 ] 51 0.010% 99.496% ( 51, 76 ] 45 0.009% 99.505% ( 76, 110 ] 28 0.006% 99.511% ( 110, 170 ] 74 0.015% 99.526% ( 170, 250 ] 123 0.025% 99.551% ( 250, 380 ] 230 0.047% 99.598% ( 380, 580 ] 400 0.082% 99.680% ( 580, 870 ] 500 0.103% 99.783% ( 870, 1300 ] 540 0.111% 99.894% ( 1300, 1900 ] 415 0.085% 99.979% ( 1900, 2900 ] 399 0.082% 100.061% ( 2900, 4400 ] 304 0.062% 100.123% ( 4400, 6600 ] 98 0.020% 100.143% ( 6600, 9900 ] 12 0.002% 100.146% ( 9900, 14000 ] 3 0.001% 100.147% ( 14000, 22000 ] 1 0.000% 100.147% ** Level 3 read latency histogram (micros): Count: 2189897 Average: 10.4473 StdDev: 140.03 Min: 1 Median: 1.9382 Max: 31431 Percentiles: P50: 1.94 P75: 2.91 P99: 9.76 P99.9: 635.15 P99.99: 900.88 ------------------------------------------------------ [ 0, 1 ] 36084 1.648% 1.648% ( 1, 2 ] 1128594 51.536% 53.184% ########## ( 2, 3 ] 522225 23.847% 77.031% ##### ( 3, 4 ] 219597 10.028% 87.059% ## ( 4, 6 ] 210840 9.628% 96.687% ## ( 6, 10 ] 53842 2.459% 99.145% ( 10, 15 ] 9886 0.451% 99.597% ( 15, 22 ] 1332 0.061% 99.658% ( 22, 34 ] 108 0.005% 99.663% ( 34, 51 ] 123 0.006% 99.668% ( 51, 76 ] 169 0.008% 99.676% ( 76, 110 ] 228 0.010% 99.686% ( 110, 170 ] 824 0.038% 99.724% ( 170, 250 ] 739 0.034% 99.758% ( 250, 380 ] 1016 0.046% 99.804% ( 380, 580 ] 1673 0.076% 99.880% ( 580, 870 ] 2246 0.103% 99.983% ( 870, 1300 ] 2117 0.097% 100.080% ( 1300, 1900 ] 1409 0.064% 100.144% ( 1900, 2900 ] 1232 0.056% 100.200% ( 2900, 4400 ] 1168 0.053% 100.254% ( 4400, 6600 ] 278 0.013% 100.266% ( 6600, 9900 ] 76 0.003% 100.270% ( 9900, 14000 ] 3 0.000% 100.270% ( 14000, 22000 ] 2 0.000% 100.270% ( 22000, 33000 ] 1 0.000% 100.270% ** Level 4 read latency histogram (micros): Count: 790148 Average: 26.1293 StdDev: 228.20 Min: 0 Median: 1.9593 Max: 27569 Percentiles: P50: 1.96 P75: 3.01 P99: 599.08 P99.9: 2551.88 P99.99: 3290.09 ------------------------------------------------------ [ 0, 1 ] 15459 1.956% 1.956% ( 1, 2 ] 395728 50.083% 52.039% ########## ( 2, 3 ] 180534 22.848% 74.887% ##### ( 3, 4 ] 78462 9.930% 84.817% ## ( 4, 6 ] 75270 9.526% 94.343% ## ( 6, 10 ] 19576 2.478% 96.821% ( 10, 15 ] 3532 0.447% 97.268% ( 15, 22 ] 395 0.050% 97.318% ( 22, 34 ] 42 0.005% 97.323% ( 34, 51 ] 71 0.009% 97.332% ( 51, 76 ] 104 0.013% 97.345% ( 76, 110 ] 1193 0.151% 97.496% ( 110, 170 ] 4496 0.569% 98.065% ( 170, 250 ] 2677 0.339% 98.404% ( 250, 380 ] 2004 0.254% 98.658% ( 380, 580 ] 2527 0.320% 98.978% ( 580, 870 ] 2683 0.340% 99.317% ( 870, 1300 ] 2395 0.303% 99.620% ( 1300, 1900 ] 1410 0.178% 99.799% ( 1900, 2900 ] 1227 0.155% 99.954% ( 2900, 4400 ] 1092 0.138% 100.092% ( 4400, 6600 ] 259 0.033% 100.125% ( 6600, 9900 ] 55 0.007% 100.132% ( 9900, 14000 ] 4 0.001% 100.133% ( 14000, 22000 ] 4 0.001% 100.133% ( 22000, 33000 ] 1 0.000% 100.133% ** Level 5 read latency histogram (micros): Count: 97943 Average: 633.7029 StdDev: 941.36 Min: 74 Median: 267.2088 Max: 31268 Percentiles: P50: 267.21 P75: 712.64 P99: 4899.37 P99.9: 8300.53 P99.99: 12172.57 ------------------------------------------------------ ( 51, 76 ] 5 0.005% 0.005% ( 76, 110 ] 7339 7.493% 7.498% # ( 110, 170 ] 25744 26.285% 33.783% ##### ( 170, 250 ] 14589 14.895% 48.678% ### ( 250, 380 ] 9779 9.984% 58.663% ## ( 380, 580 ] 11309 11.547% 70.209% ## ( 580, 870 ] 10259 10.474% 80.684% ## ( 870, 1300 ] 7553 7.712% 88.395% ## ( 1300, 1900 ] 4064 4.149% 92.545% # ( 1900, 2900 ] 3074 3.139% 95.683% # ( 2900, 4400 ] 3015 3.078% 98.762% # ( 4400, 6600 ] 1029 1.051% 99.812% ( 6600, 9900 ] 167 0.171% 99.983% ( 9900, 14000 ] 13 0.013% 99.996% ( 14000, 22000 ] 4 0.004% 100.000% ( 22000, 33000 ] 3 0.003% 100.003% ** DB Stats ** Uptime(secs): 22.2 total, 3.5 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 177.66 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.999 H:M:S, 31.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 281.78 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.076 H:M:S, 2.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/21-01:51:21 ... thread 0: (1000000,5000000) ops and (89714.1,150929.0) ops/second in (11.146522,33.128165) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/10 4.11 GB 275.9 5.4 0.0 5.4 10.1 4.7 0.0 2.1 115.8 218.0 47.32 34.44 83 0.570 5563K 14K 0.0 0.0 L1 6/6 561.43 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.3 120.9 120.6 11.86 6.63 2 5.930 1456K 3817 0.0 0.0 L3 15/0 956.16 MB 1.0 6.1 2.5 3.6 6.1 2.5 0.3 2.4 122.8 122.1 51.00 29.28 20 2.550 4371K 8676 0.0 0.0 L4 172/0 9.54 GB 1.0 8.4 5.4 3.0 8.3 5.3 0.5 1.5 118.8 117.4 72.27 40.11 56 1.290 4107K 8299 0.0 0.0 L5 1148/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1357/16 109.50 GB 0.0 21.3 8.9 12.3 25.8 13.5 6.7 5.5 119.3 145.0 182.45 110.46 161 1.133 15M 35K 0.0 0.0 Int 0/0 0.00 KB 0.0 21.3 8.9 12.3 25.8 13.5 6.7 5.5 119.3 145.0 182.45 110.46 161 1.133 15M 35K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 21.3 8.9 12.3 21.1 8.8 0.0 0.0 137.2 136.2 158.57 96.91 93 1.705 15M 35K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 203.1 23.88 13.55 68 0.351 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 33.3 total, 33.1 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 25.83 GB write, 794.33 MB/s write, 21.25 GB read, 653.44 MB/s read, 182.5 seconds Interval compaction: 25.83 GB write, 798.52 MB/s write, 21.25 GB read, 656.89 MB/s read, 182.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 10, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 10, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 82, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 10, total-stops: 82, interval: 92 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1960279 Average: 14.3965 StdDev: 99.97 Min: 0 Median: 1.6021 Max: 23743 Percentiles: P50: 1.60 P75: 2.31 P99: 232.11 P99.9: 836.32 P99.99: 1716.30 ------------------------------------------------------ [ 0, 1 ] 353405 18.028% 18.028% #### ( 1, 2 ] 1040970 53.103% 71.131% ########### ( 2, 3 ] 245750 12.536% 83.668% ### ( 3, 4 ] 105181 5.366% 89.034% # ( 4, 6 ] 86911 4.434% 93.467% # ( 6, 10 ] 18778 0.958% 94.425% ( 10, 15 ] 3845 0.196% 94.621% ( 15, 22 ] 490 0.025% 94.646% ( 22, 34 ] 40 0.002% 94.648% ( 34, 51 ] 52 0.003% 94.651% ( 51, 76 ] 68 0.003% 94.654% ( 76, 110 ] 17033 0.869% 95.523% ( 110, 170 ] 54219 2.766% 98.289% # ( 170, 250 ] 17948 0.916% 99.205% ( 250, 380 ] 6214 0.317% 99.522% ( 380, 580 ] 5359 0.273% 99.795% ( 580, 870 ] 2327 0.119% 99.914% ( 870, 1300 ] 1120 0.057% 99.971% ( 1300, 1900 ] 539 0.027% 99.998% ( 1900, 2900 ] 464 0.024% 100.022% ( 2900, 4400 ] 358 0.018% 100.040% ( 4400, 6600 ] 103 0.005% 100.046% ( 6600, 9900 ] 27 0.001% 100.047% ( 9900, 14000 ] 1 0.000% 100.047% ( 14000, 22000 ] 1 0.000% 100.047% ( 22000, 33000 ] 1 0.000% 100.047% ** Level 1 read latency histogram (micros): Count: 336654 Average: 12.7141 StdDev: 135.84 Min: 1 Median: 1.9963 Max: 10865 Percentiles: P50: 2.00 P75: 3.08 P99: 165.04 P99.9: 1947.42 P99.99: 3968.43 ------------------------------------------------------ [ 0, 1 ] 5786 1.719% 1.719% ( 1, 2 ] 163139 48.459% 50.178% ########## ( 2, 3 ] 80859 24.018% 74.196% ##### ( 3, 4 ] 34025 10.107% 84.303% ## ( 4, 6 ] 35503 10.546% 94.849% ## ( 6, 10 ] 9148 2.717% 97.566% # ( 10, 15 ] 1497 0.445% 98.011% ( 15, 22 ] 364 0.108% 98.119% ( 22, 34 ] 33 0.010% 98.129% ( 34, 51 ] 25 0.007% 98.136% ( 51, 76 ] 33 0.010% 98.146% ( 76, 110 ] 715 0.212% 98.358% ( 110, 170 ] 2355 0.700% 99.058% ( 170, 250 ] 918 0.273% 99.330% ( 250, 380 ] 468 0.139% 99.469% ( 380, 580 ] 490 0.146% 99.615% ( 580, 870 ] 411 0.122% 99.737% ( 870, 1300 ] 338 0.100% 99.838% ( 1300, 1900 ] 202 0.060% 99.898% ( 1900, 2900 ] 176 0.052% 99.950% ( 2900, 4400 ] 190 0.056% 100.006% ( 4400, 6600 ] 47 0.014% 100.020% ( 6600, 9900 ] 4 0.001% 100.021% ( 9900, 14000 ] 1 0.000% 100.022% ** Level 2 read latency histogram (micros): Count: 487231 Average: 12.3170 StdDev: 156.27 Min: 1 Median: 1.9746 Max: 15928 Percentiles: P50: 1.97 P75: 3.01 P99: 12.26 P99.9: 1343.04 P99.99: 2033.53 ------------------------------------------------------ [ 0, 1 ] 12747 2.616% 2.616% # ( 1, 2 ] 236888 48.619% 51.235% ########## ( 2, 3 ] 115317 23.668% 74.903% ##### ( 3, 4 ] 50666 10.399% 85.302% ## ( 4, 6 ] 50576 10.380% 95.682% ## ( 6, 10 ] 14722 3.022% 98.704% # ( 10, 15 ] 3186 0.654% 99.358% ( 15, 22 ] 523 0.107% 99.465% ( 22, 34 ] 98 0.020% 99.485% ( 34, 51 ] 51 0.010% 99.496% ( 51, 76 ] 45 0.009% 99.505% ( 76, 110 ] 28 0.006% 99.511% ( 110, 170 ] 74 0.015% 99.526% ( 170, 250 ] 123 0.025% 99.551% ( 250, 380 ] 230 0.047% 99.598% ( 380, 580 ] 400 0.082% 99.680% ( 580, 870 ] 500 0.103% 99.783% ( 870, 1300 ] 540 0.111% 99.894% ( 1300, 1900 ] 415 0.085% 99.979% ( 1900, 2900 ] 399 0.082% 100.061% ( 2900, 4400 ] 304 0.062% 100.123% ( 4400, 6600 ] 98 0.020% 100.143% ( 6600, 9900 ] 12 0.002% 100.146% ( 9900, 14000 ] 3 0.001% 100.147% ( 14000, 22000 ] 1 0.000% 100.147% ** Level 3 read latency histogram (micros): Count: 2193457 Average: 10.6935 StdDev: 140.10 Min: 1 Median: 1.9398 Max: 31431 Percentiles: P50: 1.94 P75: 2.92 P99: 10.17 P99.9: 637.81 P99.99: 901.76 ------------------------------------------------------ [ 0, 1 ] 36084 1.645% 1.645% ( 1, 2 ] 1128594 51.453% 53.098% ########## ( 2, 3 ] 522225 23.808% 76.906% ##### ( 3, 4 ] 219597 10.011% 86.918% ## ( 4, 6 ] 210840 9.612% 96.530% ## ( 6, 10 ] 53842 2.455% 98.984% ( 10, 15 ] 9886 0.451% 99.435% ( 15, 22 ] 1332 0.061% 99.496% ( 22, 34 ] 108 0.005% 99.501% ( 34, 51 ] 123 0.006% 99.506% ( 51, 76 ] 172 0.008% 99.514% ( 76, 110 ] 921 0.042% 99.556% ( 110, 170 ] 2848 0.130% 99.686% ( 170, 250 ] 1285 0.059% 99.745% ( 250, 380 ] 1167 0.053% 99.798% ( 380, 580 ] 1787 0.081% 99.879% ( 580, 870 ] 2270 0.103% 99.983% ( 870, 1300 ] 2121 0.097% 100.080% ( 1300, 1900 ] 1410 0.064% 100.144% ( 1900, 2900 ] 1232 0.056% 100.200% ( 2900, 4400 ] 1168 0.053% 100.253% ( 4400, 6600 ] 278 0.013% 100.266% ( 6600, 9900 ] 76 0.003% 100.269% ( 9900, 14000 ] 3 0.000% 100.270% ( 14000, 22000 ] 2 0.000% 100.270% ( 22000, 33000 ] 1 0.000% 100.270% ** Level 4 read latency histogram (micros): Count: 820017 Average: 30.9742 StdDev: 226.27 Min: 0 Median: 1.9967 Max: 27569 Percentiles: P50: 2.00 P75: 3.29 P99: 596.86 P99.9: 2529.86 P99.99: 3286.65 ------------------------------------------------------ [ 0, 1 ] 15459 1.885% 1.885% ( 1, 2 ] 395863 48.275% 50.160% ########## ( 2, 3 ] 180897 22.060% 72.220% #### ( 3, 4 ] 78793 9.609% 81.829% ## ( 4, 6 ] 75563 9.215% 91.044% ## ( 6, 10 ] 19605 2.391% 93.435% ( 10, 15 ] 3539 0.432% 93.866% ( 15, 22 ] 397 0.048% 93.915% ( 22, 34 ] 42 0.005% 93.920% ( 34, 51 ] 71 0.009% 93.928% ( 51, 76 ] 107 0.013% 93.941% ( 76, 110 ] 6568 0.801% 94.742% ( 110, 170 ] 20742 2.529% 97.272% # ( 170, 250 ] 7190 0.877% 98.149% ( 250, 380 ] 3302 0.403% 98.551% ( 380, 580 ] 3508 0.428% 98.979% ( 580, 870 ] 2939 0.358% 99.338% ( 870, 1300 ] 2419 0.295% 99.633% ( 1300, 1900 ] 1417 0.173% 99.805% ( 1900, 2900 ] 1232 0.150% 99.956% ( 2900, 4400 ] 1094 0.133% 100.089% ( 4400, 6600 ] 259 0.032% 100.121% ( 6600, 9900 ] 55 0.007% 100.127% ( 9900, 14000 ] 4 0.000% 100.128% ( 14000, 22000 ] 4 0.000% 100.128% ( 22000, 33000 ] 1 0.000% 100.128% ** Level 5 read latency histogram (micros): Count: 256733 Average: 344.2654 StdDev: 629.66 Min: 72 Median: 157.2947 Max: 31268 Percentiles: P50: 157.29 P75: 253.13 P99: 3725.01 P99.9: 6431.67 P99.99: 9610.05 ------------------------------------------------------ ( 51, 76 ] 35 0.014% 0.014% ( 76, 110 ] 37573 14.635% 14.649% ### ( 110, 170 ] 115140 44.848% 59.497% ######### ( 170, 250 ] 39393 15.344% 74.841% ### ( 250, 380 ] 16995 6.620% 81.461% # ( 380, 580 ] 16783 6.537% 87.998% # ( 580, 870 ] 11610 4.522% 92.520% # ( 870, 1300 ] 7762 3.023% 95.543% # ( 1300, 1900 ] 4106 1.599% 97.143% ( 1900, 2900 ] 3106 1.210% 98.352% ( 2900, 4400 ] 3023 1.177% 99.530% ( 4400, 6600 ] 1029 0.401% 99.931% ( 6600, 9900 ] 167 0.065% 99.996% ( 9900, 14000 ] 13 0.005% 100.001% ( 14000, 22000 ] 4 0.002% 100.002% ( 22000, 33000 ] 3 0.001% 100.004% ** DB Stats ** Uptime(secs): 33.3 total, 11.1 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 147.75 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:14.417 H:M:S, 43.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 88.30 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:7.418 H:M:S, 66.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/21-01:51:25 ... thread 0: (1000000,6000000) ops and (201365.9,157504.1) ops/second in (4.966085,38.094250) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/6 5.03 GB 395.5 6.3 0.0 6.3 12.0 5.6 0.0 2.1 119.5 226.0 54.27 40.88 102 0.532 6587K 15K 0.0 0.0 L1 6/6 561.43 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.3 120.9 120.6 11.86 6.63 2 5.930 1456K 3817 0.0 0.0 L3 15/0 956.16 MB 1.0 6.1 2.5 3.6 6.1 2.5 0.3 2.4 122.8 122.1 51.00 29.28 20 2.550 4371K 8676 0.0 0.0 L4 172/0 9.54 GB 1.0 8.4 5.4 3.0 8.3 5.3 0.5 1.5 118.8 117.4 72.27 40.11 56 1.290 4107K 8299 0.0 0.0 L5 1148/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1360/12 110.42 GB 0.0 22.2 8.9 13.3 27.7 14.4 6.7 4.9 120.2 150.0 189.40 116.90 180 1.052 16M 36K 0.0 0.0 Int 0/0 0.00 KB 0.0 22.2 8.9 13.3 27.7 14.4 6.7 4.9 120.2 150.0 189.40 116.90 180 1.052 16M 36K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 22.2 8.9 13.3 22.1 8.8 0.0 0.0 140.1 139.1 162.52 100.76 97 1.676 16M 36K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 5.7 5.7 0.0 0.0 0.0 215.7 26.87 16.14 83 0.324 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 38.3 total, 38.1 interval Flush(GB): cumulative 5.660, interval 5.660 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 27.74 GB write, 742.26 MB/s write, 22.24 GB read, 594.99 MB/s read, 189.4 seconds Interval compaction: 27.74 GB write, 745.67 MB/s write, 22.24 GB read, 597.72 MB/s read, 189.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 18, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 18, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 82, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 18, total-stops: 82, interval: 100 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 2258555 Average: 15.3415 StdDev: 96.78 Min: 0 Median: 1.5801 Max: 23743 Percentiles: P50: 1.58 P75: 2.24 P99: 240.04 P99.9: 802.18 P99.99: 1528.16 ------------------------------------------------------ [ 0, 1 ] 437931 19.390% 19.390% #### ( 1, 2 ] 1191761 52.767% 72.156% ########### ( 2, 3 ] 263727 11.677% 83.833% ## ( 3, 4 ] 112699 4.990% 88.823% # ( 4, 6 ] 90712 4.016% 92.839% # ( 6, 10 ] 19622 0.869% 93.708% ( 10, 15 ] 4056 0.180% 93.888% ( 15, 22 ] 526 0.023% 93.911% ( 22, 34 ] 46 0.002% 93.913% ( 34, 51 ] 54 0.002% 93.916% ( 51, 76 ] 73 0.003% 93.919% ( 76, 110 ] 22246 0.985% 94.904% ( 110, 170 ] 71466 3.164% 98.068% # ( 170, 250 ] 24045 1.065% 99.133% ( 250, 380 ] 8300 0.367% 99.500% ( 380, 580 ] 6945 0.307% 99.808% ( 580, 870 ] 2726 0.121% 99.928% ( 870, 1300 ] 1186 0.053% 99.981% ( 1300, 1900 ] 550 0.024% 100.005% ( 1900, 2900 ] 483 0.021% 100.027% ( 2900, 4400 ] 367 0.016% 100.043% ( 4400, 6600 ] 103 0.005% 100.047% ( 6600, 9900 ] 27 0.001% 100.049% ( 9900, 14000 ] 1 0.000% 100.049% ( 14000, 22000 ] 1 0.000% 100.049% ( 22000, 33000 ] 1 0.000% 100.049% ** Level 1 read latency histogram (micros): Count: 351643 Average: 12.9291 StdDev: 133.51 Min: 1 Median: 2.0492 Max: 10865 Percentiles: P50: 2.05 P75: 3.20 P99: 169.99 P99.9: 1883.32 P99.99: 3958.92 ------------------------------------------------------ [ 0, 1 ] 5899 1.678% 1.678% ( 1, 2 ] 165763 47.140% 48.817% ######### ( 2, 3 ] 84589 24.055% 72.872% ##### ( 3, 4 ] 36853 10.480% 83.353% ## ( 4, 6 ] 38962 11.080% 94.433% ## ( 6, 10 ] 9909 2.818% 97.251% # ( 10, 15 ] 1656 0.471% 97.722% ( 15, 22 ] 471 0.134% 97.855% ( 22, 34 ] 40 0.011% 97.867% ( 34, 51 ] 25 0.007% 97.874% ( 51, 76 ] 33 0.009% 97.883% ( 76, 110 ] 907 0.258% 98.141% ( 110, 170 ] 3020 0.859% 99.000% ( 170, 250 ] 1125 0.320% 99.320% ( 250, 380 ] 539 0.153% 99.473% ( 380, 580 ] 546 0.155% 99.629% ( 580, 870 ] 419 0.119% 99.748% ( 870, 1300 ] 338 0.096% 99.844% ( 1300, 1900 ] 203 0.058% 99.902% ( 1900, 2900 ] 176 0.050% 99.952% ( 2900, 4400 ] 191 0.054% 100.006% ( 4400, 6600 ] 47 0.013% 100.019% ( 6600, 9900 ] 4 0.001% 100.020% ( 9900, 14000 ] 1 0.000% 100.021% ** Level 2 read latency histogram (micros): Count: 487231 Average: 12.3170 StdDev: 156.27 Min: 1 Median: 1.9746 Max: 15928 Percentiles: P50: 1.97 P75: 3.01 P99: 12.26 P99.9: 1343.04 P99.99: 2033.53 ------------------------------------------------------ [ 0, 1 ] 12747 2.616% 2.616% # ( 1, 2 ] 236888 48.619% 51.235% ########## ( 2, 3 ] 115317 23.668% 74.903% ##### ( 3, 4 ] 50666 10.399% 85.302% ## ( 4, 6 ] 50576 10.380% 95.682% ## ( 6, 10 ] 14722 3.022% 98.704% # ( 10, 15 ] 3186 0.654% 99.358% ( 15, 22 ] 523 0.107% 99.465% ( 22, 34 ] 98 0.020% 99.485% ( 34, 51 ] 51 0.010% 99.496% ( 51, 76 ] 45 0.009% 99.505% ( 76, 110 ] 28 0.006% 99.511% ( 110, 170 ] 74 0.015% 99.526% ( 170, 250 ] 123 0.025% 99.551% ( 250, 380 ] 230 0.047% 99.598% ( 380, 580 ] 400 0.082% 99.680% ( 580, 870 ] 500 0.103% 99.783% ( 870, 1300 ] 540 0.111% 99.894% ( 1300, 1900 ] 415 0.085% 99.979% ( 1900, 2900 ] 399 0.082% 100.061% ( 2900, 4400 ] 304 0.062% 100.123% ( 4400, 6600 ] 98 0.020% 100.143% ( 6600, 9900 ] 12 0.002% 100.146% ( 9900, 14000 ] 3 0.001% 100.147% ( 14000, 22000 ] 1 0.000% 100.147% ** Level 3 read latency histogram (micros): Count: 2194886 Average: 10.8052 StdDev: 140.17 Min: 1 Median: 1.9404 Max: 31431 Percentiles: P50: 1.94 P75: 2.92 P99: 10.89 P99.9: 640.87 P99.99: 903.26 ------------------------------------------------------ [ 0, 1 ] 36084 1.644% 1.644% ( 1, 2 ] 1128594 51.419% 53.063% ########## ( 2, 3 ] 522225 23.793% 76.856% ##### ( 3, 4 ] 219597 10.005% 86.861% ## ( 4, 6 ] 210840 9.606% 96.467% ## ( 6, 10 ] 53842 2.453% 98.920% ( 10, 15 ] 9886 0.450% 99.370% ( 15, 22 ] 1332 0.061% 99.431% ( 22, 34 ] 108 0.005% 99.436% ( 34, 51 ] 123 0.006% 99.442% ( 51, 76 ] 172 0.008% 99.449% ( 76, 110 ] 1162 0.053% 99.502% ( 110, 170 ] 3615 0.165% 99.667% ( 170, 250 ] 1523 0.069% 99.737% ( 250, 380 ] 1254 0.057% 99.794% ( 380, 580 ] 1853 0.084% 99.878% ( 580, 870 ] 2292 0.104% 99.983% ( 870, 1300 ] 2127 0.097% 100.079% ( 1300, 1900 ] 1410 0.064% 100.144% ( 1900, 2900 ] 1234 0.056% 100.200% ( 2900, 4400 ] 1168 0.053% 100.253% ( 4400, 6600 ] 278 0.013% 100.266% ( 6600, 9900 ] 76 0.003% 100.269% ( 9900, 14000 ] 3 0.000% 100.269% ( 14000, 22000 ] 2 0.000% 100.269% ( 22000, 33000 ] 1 0.000% 100.269% ** Level 4 read latency histogram (micros): Count: 832107 Average: 33.1304 StdDev: 225.91 Min: 0 Median: 2.0261 Max: 27569 Percentiles: P50: 2.03 P75: 3.40 P99: 602.55 P99.9: 2524.00 P99.99: 3288.04 ------------------------------------------------------ [ 0, 1 ] 15459 1.858% 1.858% ( 1, 2 ] 395864 47.574% 49.432% ########## ( 2, 3 ] 180900 21.740% 71.171% #### ( 3, 4 ] 78796 9.469% 80.641% ## ( 4, 6 ] 75565 9.081% 89.722% ## ( 6, 10 ] 19605 2.356% 92.078% ( 10, 15 ] 3539 0.425% 92.503% ( 15, 22 ] 397 0.048% 92.551% ( 22, 34 ] 42 0.005% 92.556% ( 34, 51 ] 71 0.009% 92.565% ( 51, 76 ] 108 0.013% 92.578% ( 76, 110 ] 8617 1.036% 93.613% ( 110, 170 ] 27205 3.269% 96.883% # ( 170, 250 ] 9315 1.119% 98.002% ( 250, 380 ] 3986 0.479% 98.481% ( 380, 580 ] 4078 0.490% 98.971% ( 580, 870 ] 3086 0.371% 99.342% ( 870, 1300 ] 2443 0.294% 99.636% ( 1300, 1900 ] 1428 0.172% 99.807% ( 1900, 2900 ] 1237 0.149% 99.956% ( 2900, 4400 ] 1097 0.132% 100.088% ( 4400, 6600 ] 259 0.031% 100.119% ( 6600, 9900 ] 55 0.007% 100.126% ( 9900, 14000 ] 4 0.000% 100.126% ( 14000, 22000 ] 4 0.000% 100.127% ( 22000, 33000 ] 1 0.000% 100.127% ** Level 5 read latency histogram (micros): Count: 320632 Average: 310.9100 StdDev: 570.12 Min: 72 Median: 154.9370 Max: 31268 Percentiles: P50: 154.94 P75: 237.30 P99: 3410.56 P99.9: 6289.54 P99.99: 9424.50 ------------------------------------------------------ ( 51, 76 ] 45 0.014% 0.014% ( 76, 110 ] 48121 15.008% 15.022% ### ( 110, 170 ] 149743 46.702% 61.725% ######### ( 170, 250 ] 50601 15.782% 77.506% ### ( 250, 380 ] 20751 6.472% 83.978% # ( 380, 580 ] 19644 6.127% 90.105% # ( 580, 870 ] 12356 3.854% 93.958% # ( 870, 1300 ] 7878 2.457% 96.416% ( 1300, 1900 ] 4127 1.287% 97.703% ( 1900, 2900 ] 3128 0.976% 98.678% ( 2900, 4400 ] 3031 0.945% 99.624% ( 4400, 6600 ] 1032 0.322% 99.945% ( 6600, 9900 ] 167 0.052% 99.998% ( 9900, 14000 ] 13 0.004% 100.002% ( 14000, 22000 ] 4 0.001% 100.003% ( 22000, 33000 ] 3 0.001% 100.004% ** DB Stats ** Uptime(secs): 38.3 total, 5.0 interval Cumulative writes: 0 writes, 6000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 5.77 GB, 154.30 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:15.933 H:M:S, 41.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 198.18 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.516 H:M:S, 30.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 272.810 micros/op 14655 ops/sec 60.184 seconds 881996 operations; 14.2 MB/s (225999 of 225999 found) Microseconds per read: Count: 881996 Average: 272.8131 StdDev: 759.56 Min: 1 Median: 174.0189 Max: 77430 Percentiles: P50: 174.02 P75: 241.00 P99: 1830.59 P99.9: 12419.91 P99.99: 29465.36 ------------------------------------------------------ [ 0, 1 ] 1 0.000% 0.000% ( 1, 2 ] 3 0.000% 0.000% ( 2, 3 ] 35 0.004% 0.004% ( 3, 4 ] 48 0.005% 0.010% ( 4, 6 ] 99 0.011% 0.021% ( 6, 10 ] 437 0.050% 0.071% ( 10, 15 ] 4212 0.478% 0.548% ( 15, 22 ] 4042 0.458% 1.006% ( 22, 34 ] 6577 0.746% 1.752% ( 34, 51 ] 1889 0.214% 1.966% ( 51, 76 ] 218 0.025% 1.991% ( 76, 110 ] 24431 2.770% 4.761% # ( 110, 170 ] 385775 43.739% 48.500% ######### ( 170, 250 ] 263376 29.861% 78.361% ###### ( 250, 380 ] 98023 11.114% 89.475% ## ( 380, 580 ] 46786 5.305% 94.780% # ( 580, 870 ] 22397 2.539% 97.319% # ( 870, 1300 ] 10730 1.217% 98.535% ( 1300, 1900 ] 4633 0.525% 99.061% ( 1900, 2900 ] 3172 0.360% 99.420% ( 2900, 4400 ] 2713 0.308% 99.728% ( 4400, 6600 ] 879 0.100% 99.828% ( 6600, 9900 ] 335 0.038% 99.866% ( 9900, 14000 ] 493 0.056% 99.922% ( 14000, 22000 ] 502 0.057% 99.978% ( 22000, 33000 ] 150 0.017% 99.995% ( 33000, 50000 ] 31 0.004% 99.999% ( 50000, 75000 ] 8 0.001% 100.000% ( 75000, 110000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 13819676 rocksdb.bloom.filter.full.positive COUNT : 1015775 rocksdb.bloom.filter.full.true.positive COUNT : 881547 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 449 rocksdb.memtable.miss COUNT : 881547 rocksdb.l0.hit COUNT : 35816 rocksdb.l1.hit COUNT : 8282 rocksdb.l2andup.hit COUNT : 837449 rocksdb.compaction.key.drop.new COUNT : 192674 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 6143908 rocksdb.number.keys.read COUNT : 881996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 6340513056 rocksdb.bytes.read COUNT : 881996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1705 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 36513607 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 27884701680 rocksdb.compact.write.bytes COUNT : 27682230582 rocksdb.flush.write.bytes COUNT : 6341528013 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 23 rocksdb.number.superversion_acquires COUNT : 1235 rocksdb.number.superversion_releases COUNT : 1189 rocksdb.number.superversion_cleanups COUNT : 307 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 346 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 6290325504 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2383872 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 32174052113 rocksdb.non.last.level.read.count COUNT : 7631950 rocksdb.block.checksum.compute.count COUNT : 7779057 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 172.266200 P95 : 603.677659 P99 : 1829.157439 P100 : 77427.000000 COUNT : 881996 SUM : 239856179 rocksdb.db.write.micros P50 : 1.547006 P95 : 4.622403 P99 : 9.626042 P100 : 821415.000000 COUNT : 6143908 SUM : 52260021 rocksdb.compaction.times.micros P50 : 1379487.179487 P95 : 5400000.000000 P99 : 6500000.000000 P100 : 33928616.000000 COUNT : 100 SUM : 202288166 rocksdb.compaction.times.cpu_micros P50 : 880606.060606 P95 : 3366666.666667 P99 : 6500000.000000 P100 : 12602101.000000 COUNT : 100 SUM : 119110683 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 852.317073 P95 : 94250.000000 P99 : 266900.000000 P100 : 373326.000000 COUNT : 87 SUM : 1263565 rocksdb.compaction.outfile.sync.micros P50 : 22458.333333 P95 : 235142.857143 P99 : 435733.333333 P100 : 735489.000000 COUNT : 312 SUM : 18567266 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 241.578947 P95 : 4187.500000 P99 : 7158.000000 P100 : 7158.000000 COUNT : 214 SUM : 213964 rocksdb.table.open.io.micros P50 : 8798.788546 P95 : 19098.591549 P99 : 21660.093897 P100 : 43115.000000 COUNT : 1705 SUM : 14689760 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.912082 P95 : 9.910635 P99 : 176.170331 P100 : 31451.000000 COUNT : 6752100 SUM : 97696433 rocksdb.read.block.get.micros P50 : 142.698984 P95 : 511.781063 P99 : 1720.344900 P100 : 31273.000000 COUNT : 1025245 SUM : 208587805 rocksdb.write.raw.block.micros P50 : 0.532253 P95 : 1.266502 P99 : 3.748969 P100 : 33670.000000 COUNT : 8240663 SUM : 24364806 rocksdb.numfiles.in.singlecompaction P50 : 2.757576 P95 : 5.885714 P99 : 14.533333 P100 : 22.000000 COUNT : 128 SUM : 457 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.570579 P95 : 1182.800987 P99 : 2459.921205 P100 : 821232.000000 COUNT : 191694 SUM : 36533039 rocksdb.sst.read.micros P50 : 1.934438 P95 : 162.201039 P99 : 506.509629 P100 : 31431.000000 COUNT : 7631975 SUM : 287545116 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 881996 SUM : 881996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 6143908 SUM : 6340513056 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 225238.095238 P95 : 988714.285714 P99 : 1063484.000000 P100 : 1063484.000000 COUNT : 87 SUM : 27715423 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 347 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0