[2024-12-15 08:22:01.528572] Starting SPDK v24.09.1-pre git sha1 b18e1bd62 / DPDK 22.11.4 initialization... [2024-12-15 08:22:01.528710] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1220302 ] Set seed to 1734247321525183 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-15 08:22:01.611310] app.c: 917:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-15 08:22:01.650099] app.c: 610:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-15 08:22:01.650144] app.c: 611:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1220302' to capture a snapshot of events at runtime. [2024-12-15 08:22:01.650154] app.c: 616:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-12-15 08:22:01.650164] app.c: 617:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-12-15 08:22:01.650172] app.c: 618:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1220302 for offline analysis/debug. [2024-12-15 08:22:01.650254] reactor.c: 990:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sun Dec 15 08:22:05 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/15-08:22:13 ... thread 0: (1000000,1000000) ops and (135128.3,135128.3) ops/second in (7.400377,7.400377) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 6/5 818.31 MB 37.7 0.4 0.0 0.4 1.2 0.8 0.0 1.5 39.1 123.9 9.66 3.86 10 0.966 383K 488 0.0 0.0 L1 4/1 308.86 MB 18.0 0.5 0.4 0.1 0.5 0.4 0.0 1.2 100.5 100.2 5.51 2.56 1 5.507 561K 1108 0.0 0.0 L2 6/3 483.72 MB 2.4 0.0 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 46/8 3.06 GB 2.6 2.4 1.5 0.8 2.3 1.5 0.0 1.5 99.2 98.5 24.27 11.26 16 1.517 997K 0 0.0 0.0 L4 185/7 11.98 GB 1.2 3.0 2.1 0.9 3.0 2.1 0.1 1.4 102.6 101.9 30.05 14.18 19 1.582 868K 0 0.0 0.0 L5 1082/0 89.63 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1329/24 106.24 GB 0.0 6.3 4.1 2.2 7.0 4.9 0.3 8.8 92.4 103.7 69.48 31.86 46 1.510 2811K 1596 0.0 0.0 Int 0/0 0.00 KB 0.0 6.3 4.1 2.2 7.0 4.9 0.3 8.8 92.4 103.7 69.48 31.86 46 1.510 2811K 1596 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.3 4.1 2.2 6.2 4.1 0.0 0.0 102.5 101.9 62.67 29.52 37 1.694 2811K 1596 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 120.2 6.81 2.34 9 0.757 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.5 total, 7.4 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.03 GB write, 954.30 MB/s write, 6.27 GB read, 850.84 MB/s read, 69.5 seconds Interval compaction: 7.03 GB write, 973.04 MB/s write, 6.27 GB read, 867.54 MB/s read, 69.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 34, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 34, interval: 34 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 212009 Average: 12.5471 StdDev: 147.71 Min: 1 Median: 1.9015 Max: 7383 Percentiles: P50: 1.90 P75: 2.89 P99: 12.49 P99.9: 2350.27 P99.99: 4306.08 ------------------------------------------------------ [ 0, 1 ] 7090 3.344% 3.344% # ( 1, 2 ] 109728 51.756% 55.100% ########## ( 2, 3 ] 47360 22.339% 77.439% #### ( 3, 4 ] 20235 9.544% 86.984% ## ( 4, 6 ] 20260 9.556% 96.540% ## ( 6, 10 ] 4830 2.278% 98.818% ( 10, 15 ] 776 0.366% 99.184% ( 15, 22 ] 106 0.050% 99.234% ( 22, 34 ] 17 0.008% 99.242% ( 34, 51 ] 6 0.003% 99.245% ( 51, 76 ] 7 0.003% 99.248% ( 76, 110 ] 7 0.003% 99.251% ( 110, 170 ] 30 0.014% 99.266% ( 170, 250 ] 60 0.028% 99.294% ( 250, 380 ] 118 0.056% 99.350% ( 380, 580 ] 246 0.116% 99.466% ( 580, 870 ] 339 0.160% 99.625% ( 870, 1300 ] 295 0.139% 99.765% ( 1300, 1900 ] 219 0.103% 99.868% ( 1900, 2900 ] 151 0.071% 99.939% ( 2900, 4400 ] 115 0.054% 99.993% ( 4400, 6600 ] 41 0.019% 100.013% ( 6600, 9900 ] 2 0.001% 100.014% ** Level 1 read latency histogram (micros): Count: 46696 Average: 18.0142 StdDev: 204.84 Min: 1 Median: 1.9671 Max: 8521 Percentiles: P50: 1.97 P75: 2.92 P99: 21.85 P99.9: 3558.09 P99.99: 7331.72 ------------------------------------------------------ [ 0, 1 ] 948 2.030% 2.030% ( 1, 2 ] 23163 49.604% 51.634% ########## ( 2, 3 ] 11856 25.390% 77.024% ##### ( 3, 4 ] 4519 9.677% 86.701% ## ( 4, 6 ] 4340 9.294% 95.995% ## ( 6, 10 ] 1135 2.431% 98.426% ( 10, 15 ] 223 0.478% 98.904% ( 15, 22 ] 46 0.099% 99.002% ( 22, 34 ] 6 0.013% 99.015% ( 34, 51 ] 4 0.009% 99.023% ( 51, 76 ] 2 0.004% 99.028% ( 76, 110 ] 2 0.004% 99.032% ( 110, 170 ] 4 0.009% 99.041% ( 170, 250 ] 12 0.026% 99.066% ( 250, 380 ] 24 0.051% 99.118% ( 380, 580 ] 59 0.126% 99.244% ( 580, 870 ] 78 0.167% 99.411% ( 870, 1300 ] 101 0.216% 99.627% ( 1300, 1900 ] 57 0.122% 99.749% ( 1900, 2900 ] 51 0.109% 99.859% ( 2900, 4400 ] 44 0.094% 99.953% ( 4400, 6600 ] 16 0.034% 99.987% ( 6600, 9900 ] 6 0.013% 100.000% ** Level 2 read latency histogram (micros): Count: 372357 Average: 12.3651 StdDev: 155.80 Min: 1 Median: 1.9834 Max: 11110 Percentiles: P50: 1.98 P75: 2.98 P99: 11.25 P99.9: 1607.29 P99.99: 2477.96 ------------------------------------------------------ [ 0, 1 ] 9013 2.421% 2.421% ( 1, 2 ] 180155 48.382% 50.803% ########## ( 2, 3 ] 91716 24.631% 75.434% ##### ( 3, 4 ] 39069 10.492% 85.926% ## ( 4, 6 ] 37560 10.087% 96.014% ## ( 6, 10 ] 10656 2.862% 98.875% # ( 10, 15 ] 1862 0.500% 99.375% ( 15, 22 ] 300 0.081% 99.456% ( 22, 34 ] 65 0.017% 99.473% ( 34, 51 ] 25 0.007% 99.480% ( 51, 76 ] 28 0.008% 99.488% ( 76, 110 ] 11 0.003% 99.491% ( 110, 170 ] 48 0.013% 99.503% ( 170, 250 ] 71 0.019% 99.523% ( 250, 380 ] 140 0.038% 99.560% ( 380, 580 ] 305 0.082% 99.642% ( 580, 870 ] 345 0.093% 99.735% ( 870, 1300 ] 441 0.118% 99.853% ( 1300, 1900 ] 341 0.092% 99.945% ( 1900, 2900 ] 292 0.078% 100.023% ( 2900, 4400 ] 242 0.065% 100.088% ( 4400, 6600 ] 82 0.022% 100.110% ( 6600, 9900 ] 6 0.002% 100.112% ( 9900, 14000 ] 2 0.001% 100.112% ** Level 3 read latency histogram (micros): Count: 827830 Average: 12.0876 StdDev: 151.70 Min: 1 Median: 1.9786 Max: 11115 Percentiles: P50: 1.98 P75: 2.95 P99: 9.84 P99.9: 1060.90 P99.99: 1446.55 ------------------------------------------------------ [ 0, 1 ] 15569 1.881% 1.881% ( 1, 2 ] 407044 49.170% 51.051% ########## ( 2, 3 ] 208421 25.177% 76.227% ##### ( 3, 4 ] 85358 10.311% 86.539% ## ( 4, 6 ] 83266 10.058% 96.597% ## ( 6, 10 ] 20723 2.503% 99.100% # ( 10, 15 ] 3340 0.403% 99.504% ( 15, 22 ] 480 0.058% 99.562% ( 22, 34 ] 47 0.006% 99.567% ( 34, 51 ] 33 0.004% 99.571% ( 51, 76 ] 55 0.007% 99.578% ( 76, 110 ] 30 0.004% 99.582% ( 110, 170 ] 86 0.010% 99.592% ( 170, 250 ] 167 0.020% 99.612% ( 250, 380 ] 337 0.041% 99.653% ( 380, 580 ] 623 0.075% 99.728% ( 580, 870 ] 965 0.117% 99.845% ( 870, 1300 ] 1032 0.125% 99.969% ( 1300, 1900 ] 701 0.085% 100.054% ( 1900, 2900 ] 607 0.073% 100.127% ( 2900, 4400 ] 522 0.063% 100.190% ( 4400, 6600 ] 160 0.019% 100.210% ( 6600, 9900 ] 21 0.003% 100.212% ( 9900, 14000 ] 2 0.000% 100.212% ** Level 4 read latency histogram (micros): Count: 295732 Average: 23.1316 StdDev: 214.06 Min: 1 Median: 1.9419 Max: 7641 Percentiles: P50: 1.94 P75: 2.91 P99: 621.37 P99.9: 2696.43 P99.99: 3601.51 ------------------------------------------------------ [ 0, 1 ] 6405 2.166% 2.166% ( 1, 2 ] 150192 50.787% 52.952% ########## ( 2, 3 ] 71367 24.132% 77.085% ##### ( 3, 4 ] 29281 9.901% 86.986% ## ( 4, 6 ] 26322 8.901% 95.886% ## ( 6, 10 ] 6637 2.244% 98.131% ( 10, 15 ] 1118 0.378% 98.509% ( 15, 22 ] 152 0.051% 98.560% ( 22, 34 ] 15 0.005% 98.565% ( 34, 51 ] 17 0.006% 98.571% ( 51, 76 ] 28 0.009% 98.580% ( 76, 110 ] 18 0.006% 98.587% ( 110, 170 ] 64 0.022% 98.608% ( 170, 250 ] 159 0.054% 98.662% ( 250, 380 ] 300 0.101% 98.763% ( 380, 580 ] 577 0.195% 98.959% ( 580, 870 ] 860 0.291% 99.249% ( 870, 1300 ] 897 0.303% 99.553% ( 1300, 1900 ] 641 0.217% 99.769% ( 1900, 2900 ] 485 0.164% 99.933% ( 2900, 4400 ] 358 0.121% 100.054% ( 4400, 6600 ] 112 0.038% 100.092% ( 6600, 9900 ] 9 0.003% 100.095% ** Level 5 read latency histogram (micros): Count: 13247 Average: 1598.7866 StdDev: 1417.71 Min: 84 Median: 1083.9422 Max: 11954 Percentiles: P50: 1083.94 P75: 2195.22 P99: 6494.60 P99.9: 9650.88 P99.99: 11954.00 ------------------------------------------------------ ( 76, 110 ] 23 0.174% 0.174% ( 110, 170 ] 152 1.147% 1.321% ( 170, 250 ] 318 2.401% 3.722% ( 250, 380 ] 830 6.266% 9.987% # ( 380, 580 ] 1631 12.312% 22.299% ## ( 580, 870 ] 2456 18.540% 40.839% #### ( 870, 1300 ] 2439 18.412% 59.251% #### ( 1300, 1900 ] 1657 12.508% 71.760% ### ( 1900, 2900 ] 1454 10.976% 82.736% ## ( 2900, 4400 ] 1549 11.693% 94.429% ## ( 4400, 6600 ] 636 4.801% 99.230% # ( 6600, 9900 ] 96 0.725% 99.955% ( 9900, 14000 ] 7 0.053% 100.008% ** DB Stats ** Uptime(secs): 7.5 total, 7.4 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 130.39 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.497 H:M:S, 46.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 132.94 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.497 H:M:S, 47.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/12/15-08:22:19 ... thread 0: (1000000,2000000) ops and (151473.1,142834.6) ops/second in (6.601831,14.002208) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 12/9 1.84 GB 44.1 1.0 0.0 1.0 2.9 1.8 0.0 1.6 53.4 147.6 20.05 9.68 25 0.802 1087K 1490 0.0 0.0 L1 3/3 180.09 MB 0.0 0.5 0.4 0.1 0.5 0.4 0.0 1.2 100.5 100.2 5.51 2.56 1 5.507 561K 1108 0.0 0.0 L2 0/0 0.00 KB 0.0 0.2 0.1 0.1 0.2 0.1 0.2 1.6 143.6 143.3 1.46 0.95 1 1.457 212K 396 0.0 0.0 L3 19/3 1.16 GB 0.9 4.3 2.0 2.3 4.2 2.0 0.1 2.1 115.6 114.9 37.85 20.27 22 1.720 1766K 0 0.0 0.0 L4 172/5 9.70 GB 1.0 7.4 4.2 3.2 7.3 4.1 0.5 1.7 114.1 112.5 66.09 34.77 52 1.271 3434K 3297 0.0 0.0 L5 1138/0 94.30 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.7 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1344/20 107.19 GB 0.0 13.4 6.8 6.7 15.1 8.5 5.6 8.2 105.0 118.4 130.95 68.23 101 1.297 7062K 6291 0.0 0.0 Int 0/0 0.00 KB 0.0 13.4 6.8 6.7 15.1 8.5 5.6 8.2 105.0 118.4 130.95 68.23 101 1.297 7062K 6291 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 13.4 6.8 6.7 13.3 6.6 0.0 0.0 116.9 115.7 117.58 62.88 79 1.488 7062K 6291 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 141.4 13.37 5.35 22 0.608 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.1 total, 14.0 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 15.14 GB write, 1095.39 MB/s write, 13.43 GB read, 971.70 MB/s read, 130.9 seconds Interval compaction: 15.14 GB write, 1106.76 MB/s write, 13.43 GB read, 981.78 MB/s read, 130.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 64, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 64, interval: 64 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 613387 Average: 9.9995 StdDev: 129.27 Min: 1 Median: 1.8756 Max: 10491 Percentiles: P50: 1.88 P75: 2.83 P99: 9.81 P99.9: 1710.19 P99.99: 3813.35 ------------------------------------------------------ [ 0, 1 ] 22083 3.600% 3.600% # ( 1, 2 ] 325038 52.991% 56.591% ########### ( 2, 3 ] 135485 22.088% 78.679% #### ( 3, 4 ] 57633 9.396% 88.075% ## ( 4, 6 ] 55798 9.097% 97.171% ## ( 6, 10 ] 11766 1.918% 99.090% ( 10, 15 ] 1739 0.284% 99.373% ( 15, 22 ] 208 0.034% 99.407% ( 22, 34 ] 24 0.004% 99.411% ( 34, 51 ] 19 0.003% 99.414% ( 51, 76 ] 26 0.004% 99.418% ( 76, 110 ] 19 0.003% 99.421% ( 110, 170 ] 83 0.014% 99.435% ( 170, 250 ] 200 0.033% 99.468% ( 250, 380 ] 311 0.051% 99.518% ( 380, 580 ] 604 0.098% 99.617% ( 580, 870 ] 795 0.130% 99.746% ( 870, 1300 ] 665 0.108% 99.855% ( 1300, 1900 ] 409 0.067% 99.921% ( 1900, 2900 ] 268 0.044% 99.965% ( 2900, 4400 ] 254 0.041% 100.007% ( 4400, 6600 ] 81 0.013% 100.020% ( 6600, 9900 ] 12 0.002% 100.022% ( 9900, 14000 ] 1 0.000% 100.022% ** Level 1 read latency histogram (micros): Count: 150618 Average: 13.5545 StdDev: 166.68 Min: 1 Median: 1.9128 Max: 8521 Percentiles: P50: 1.91 P75: 2.85 P99: 12.31 P99.9: 2849.84 P99.99: 5740.67 ------------------------------------------------------ [ 0, 1 ] 3371 2.238% 2.238% ( 1, 2 ] 78813 52.326% 54.565% ########## ( 2, 3 ] 36073 23.950% 78.515% ##### ( 3, 4 ] 14068 9.340% 87.855% ## ( 4, 6 ] 12953 8.600% 96.455% ## ( 6, 10 ] 3592 2.385% 98.839% ( 10, 15 ] 523 0.347% 99.187% ( 15, 22 ] 66 0.044% 99.231% ( 22, 34 ] 10 0.007% 99.237% ( 34, 51 ] 8 0.005% 99.242% ( 51, 76 ] 5 0.003% 99.246% ( 76, 110 ] 6 0.004% 99.250% ( 110, 170 ] 17 0.011% 99.261% ( 170, 250 ] 48 0.032% 99.293% ( 250, 380 ] 87 0.058% 99.351% ( 380, 580 ] 149 0.099% 99.450% ( 580, 870 ] 209 0.139% 99.588% ( 870, 1300 ] 236 0.157% 99.745% ( 1300, 1900 ] 127 0.084% 99.829% ( 1900, 2900 ] 112 0.074% 99.904% ( 2900, 4400 ] 108 0.072% 99.975% ( 4400, 6600 ] 36 0.024% 99.999% ( 6600, 9900 ] 9 0.006% 100.005% ** Level 2 read latency histogram (micros): Count: 394513 Average: 12.0963 StdDev: 153.29 Min: 1 Median: 1.9802 Max: 11110 Percentiles: P50: 1.98 P75: 2.98 P99: 11.03 P99.9: 1601.70 P99.99: 2507.19 ------------------------------------------------------ [ 0, 1 ] 9498 2.408% 2.408% ( 1, 2 ] 191548 48.553% 50.961% ########## ( 2, 3 ] 97261 24.653% 75.614% ##### ( 3, 4 ] 41184 10.439% 86.053% ## ( 4, 6 ] 39572 10.031% 96.084% ## ( 6, 10 ] 11110 2.816% 98.900% # ( 10, 15 ] 1910 0.484% 99.384% ( 15, 22 ] 307 0.078% 99.462% ( 22, 34 ] 65 0.016% 99.478% ( 34, 51 ] 26 0.007% 99.485% ( 51, 76 ] 29 0.007% 99.492% ( 76, 110 ] 12 0.003% 99.495% ( 110, 170 ] 51 0.013% 99.508% ( 170, 250 ] 79 0.020% 99.528% ( 250, 380 ] 149 0.038% 99.566% ( 380, 580 ] 322 0.082% 99.648% ( 580, 870 ] 364 0.092% 99.740% ( 870, 1300 ] 456 0.116% 99.856% ( 1300, 1900 ] 349 0.088% 99.944% ( 1900, 2900 ] 299 0.076% 100.020% ( 2900, 4400 ] 248 0.063% 100.083% ( 4400, 6600 ] 84 0.021% 100.104% ( 6600, 9900 ] 6 0.002% 100.105% ( 9900, 14000 ] 2 0.001% 100.106% ** Level 3 read latency histogram (micros): Count: 1642681 Average: 10.5232 StdDev: 142.13 Min: 0 Median: 1.9750 Max: 19105 Percentiles: P50: 1.97 P75: 2.94 P99: 9.60 P99.9: 894.07 P99.99: 1275.43 ------------------------------------------------------ [ 0, 1 ] 26005 1.583% 1.583% ( 1, 2 ] 815756 49.660% 51.243% ########## ( 2, 3 ] 414366 25.225% 76.468% ##### ( 3, 4 ] 169685 10.330% 86.798% ## ( 4, 6 ] 166009 10.106% 96.904% ## ( 6, 10 ] 38208 2.326% 99.230% ( 10, 15 ] 5807 0.354% 99.583% ( 15, 22 ] 764 0.047% 99.630% ( 22, 34 ] 69 0.004% 99.634% ( 34, 51 ] 93 0.006% 99.640% ( 51, 76 ] 137 0.008% 99.648% ( 76, 110 ] 76 0.005% 99.653% ( 110, 170 ] 192 0.012% 99.664% ( 170, 250 ] 347 0.021% 99.685% ( 250, 380 ] 654 0.040% 99.725% ( 380, 580 ] 1127 0.069% 99.794% ( 580, 870 ] 1651 0.101% 99.894% ( 870, 1300 ] 1667 0.101% 99.996% ( 1300, 1900 ] 1040 0.063% 100.059% ( 1900, 2900 ] 883 0.054% 100.113% ( 2900, 4400 ] 877 0.053% 100.166% ( 4400, 6600 ] 274 0.017% 100.183% ( 6600, 9900 ] 48 0.003% 100.186% ( 9900, 14000 ] 6 0.000% 100.186% ( 14000, 22000 ] 2 0.000% 100.186% ** Level 4 read latency histogram (micros): Count: 851950 Average: 19.2346 StdDev: 202.91 Min: 1 Median: 1.9759 Max: 19667 Percentiles: P50: 1.98 P75: 2.97 P99: 292.44 P99.9: 2173.91 P99.99: 3103.75 ------------------------------------------------------ [ 0, 1 ] 14158 1.662% 1.662% ( 1, 2 ] 421968 49.530% 51.192% ########## ( 2, 3 ] 208603 24.485% 75.677% ##### ( 3, 4 ] 88780 10.421% 86.098% ## ( 4, 6 ] 84297 9.895% 95.992% ## ( 6, 10 ] 21008 2.466% 98.458% ( 10, 15 ] 3057 0.359% 98.817% ( 15, 22 ] 381 0.045% 98.862% ( 22, 34 ] 33 0.004% 98.866% ( 34, 51 ] 59 0.007% 98.872% ( 51, 76 ] 110 0.013% 98.885% ( 76, 110 ] 57 0.007% 98.892% ( 110, 170 ] 179 0.021% 98.913% ( 170, 250 ] 464 0.054% 98.968% ( 250, 380 ] 847 0.099% 99.067% ( 380, 580 ] 1495 0.175% 99.242% ( 580, 870 ] 2086 0.245% 99.487% ( 870, 1300 ] 1982 0.233% 99.720% ( 1300, 1900 ] 1290 0.151% 99.871% ( 1900, 2900 ] 891 0.105% 99.976% ( 2900, 4400 ] 882 0.104% 100.079% ( 4400, 6600 ] 286 0.034% 100.113% ( 6600, 9900 ] 55 0.006% 100.119% ( 9900, 14000 ] 5 0.001% 100.120% ( 14000, 22000 ] 2 0.000% 100.120% ** Level 5 read latency histogram (micros): Count: 30725 Average: 1335.9939 StdDev: 1294.14 Min: 84 Median: 868.6049 Max: 21161 Percentiles: P50: 868.60 P75: 1659.84 P99: 6319.35 P99.9: 9609.03 P99.99: 13685.91 ------------------------------------------------------ ( 76, 110 ] 97 0.316% 0.316% ( 110, 170 ] 626 2.037% 2.353% ( 170, 250 ] 1229 4.000% 6.353% # ( 250, 380 ] 2524 8.215% 14.568% ## ( 380, 580 ] 4577 14.897% 29.465% ### ( 580, 870 ] 6340 20.635% 50.099% #### ( 870, 1300 ] 5689 18.516% 68.615% #### ( 1300, 1900 ] 3271 10.646% 79.261% ## ( 1900, 2900 ] 2570 8.365% 87.626% ## ( 2900, 4400 ] 2645 8.609% 96.234% ## ( 4400, 6600 ] 974 3.170% 99.404% # ( 6600, 9900 ] 167 0.544% 99.948% ( 9900, 14000 ] 14 0.046% 99.993% ( 14000, 22000 ] 3 0.010% 100.003% ** DB Stats ** Uptime(secs): 14.1 total, 6.6 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 139.11 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.134 H:M:S, 43.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 149.08 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.637 H:M:S, 39.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 5 num-running-flushes: 1 2024/12/15-08:22:23 ... thread 0: (1000000,3000000) ops and (284776.8,171294.2) ops/second in (3.511522,17.513730) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/9 2.77 GB 144.7 2.2 0.0 2.2 4.9 2.8 0.0 1.8 78.7 179.7 28.02 16.81 44 0.637 2238K 2700 0.0 0.0 L1 3/3 180.09 MB 0.0 0.5 0.4 0.1 0.5 0.4 0.0 1.2 100.5 100.2 5.51 2.56 1 5.507 561K 1108 0.0 0.0 L2 0/0 0.00 KB 0.0 0.2 0.1 0.1 0.2 0.1 0.2 1.6 143.6 143.3 1.46 0.95 1 1.457 212K 396 0.0 0.0 L3 16/0 919.89 MB 0.9 4.3 2.0 2.3 4.2 2.0 0.1 2.1 115.6 114.9 37.85 20.27 22 1.720 1766K 0 0.0 0.0 L4 173/0 9.65 GB 1.0 7.9 4.5 3.4 7.7 4.3 0.5 1.7 116.1 114.4 69.39 37.10 55 1.262 3706K 3297 0.0 0.0 L5 1142/0 94.61 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1349/12 108.10 GB 0.0 15.0 7.0 8.0 17.7 9.6 5.9 6.4 108.3 127.1 142.22 77.68 123 1.156 8485K 7501 0.0 0.0 Int 0/0 0.00 KB 0.0 15.0 7.0 8.0 17.7 9.6 5.9 6.4 108.3 127.1 142.22 77.68 123 1.156 8485K 7501 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 15.0 7.0 8.0 14.9 6.9 0.0 0.0 122.5 121.2 125.74 69.64 86 1.462 8485K 7501 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 172.0 16.48 8.04 37 0.445 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 17.7 total, 17.5 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 17.66 GB write, 1023.75 MB/s write, 15.04 GB read, 871.89 MB/s read, 142.2 seconds Interval compaction: 17.66 GB write, 1032.24 MB/s write, 15.04 GB read, 879.13 MB/s read, 142.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 64, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 64, interval: 64 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 975464 Average: 9.3789 StdDev: 106.80 Min: 0 Median: 1.7511 Max: 10491 Percentiles: P50: 1.75 P75: 2.60 P99: 140.69 P99.9: 1101.84 P99.99: 2280.19 ------------------------------------------------------ [ 0, 1 ] 81507 8.356% 8.356% ## ( 1, 2 ] 540808 55.441% 63.797% ########### ( 2, 3 ] 183508 18.812% 82.609% #### ( 3, 4 ] 73750 7.561% 90.170% ## ( 4, 6 ] 65410 6.706% 96.875% # ( 6, 10 ] 14789 1.516% 98.391% ( 10, 15 ] 2198 0.225% 98.617% ( 15, 22 ] 348 0.036% 98.652% ( 22, 34 ] 47 0.005% 98.657% ( 34, 51 ] 23 0.002% 98.660% ( 51, 76 ] 30 0.003% 98.663% ( 76, 110 ] 1115 0.114% 98.777% ( 110, 170 ] 4259 0.437% 99.214% ( 170, 250 ] 2341 0.240% 99.453% ( 250, 380 ] 1305 0.134% 99.587% ( 380, 580 ] 1520 0.156% 99.743% ( 580, 870 ] 1126 0.115% 99.859% ( 870, 1300 ] 754 0.077% 99.936% ( 1300, 1900 ] 424 0.043% 99.979% ( 1900, 2900 ] 280 0.029% 100.008% ( 2900, 4400 ] 256 0.026% 100.034% ( 4400, 6600 ] 82 0.008% 100.043% ( 6600, 9900 ] 12 0.001% 100.044% ( 9900, 14000 ] 1 0.000% 100.044% ** Level 1 read latency histogram (micros): Count: 150878 Average: 13.8935 StdDev: 166.85 Min: 1 Median: 1.9144 Max: 8521 Percentiles: P50: 1.91 P75: 2.86 P99: 14.76 P99.9: 2847.52 P99.99: 5739.08 ------------------------------------------------------ [ 0, 1 ] 3371 2.234% 2.234% ( 1, 2 ] 78813 52.236% 54.470% ########## ( 2, 3 ] 36073 23.909% 78.379% ##### ( 3, 4 ] 14068 9.324% 87.703% ## ( 4, 6 ] 12954 8.586% 96.289% ## ( 6, 10 ] 3592 2.381% 98.670% ( 10, 15 ] 523 0.347% 99.016% ( 15, 22 ] 66 0.044% 99.060% ( 22, 34 ] 10 0.007% 99.067% ( 34, 51 ] 8 0.005% 99.072% ( 51, 76 ] 5 0.003% 99.075% ( 76, 110 ] 36 0.024% 99.099% ( 110, 170 ] 131 0.087% 99.186% ( 170, 250 ] 113 0.075% 99.261% ( 250, 380 ] 108 0.072% 99.333% ( 380, 580 ] 170 0.113% 99.445% ( 580, 870 ] 214 0.142% 99.587% ( 870, 1300 ] 239 0.158% 99.745% ( 1300, 1900 ] 127 0.084% 99.830% ( 1900, 2900 ] 112 0.074% 99.904% ( 2900, 4400 ] 108 0.072% 99.975% ( 4400, 6600 ] 36 0.024% 99.999% ( 6600, 9900 ] 9 0.006% 100.005% ** Level 2 read latency histogram (micros): Count: 394629 Average: 12.1586 StdDev: 153.34 Min: 1 Median: 1.9805 Max: 11110 Percentiles: P50: 1.98 P75: 2.98 P99: 11.33 P99.9: 1601.50 P99.99: 2507.15 ------------------------------------------------------ [ 0, 1 ] 9498 2.407% 2.407% ( 1, 2 ] 191548 48.539% 50.946% ########## ( 2, 3 ] 97261 24.646% 75.592% ##### ( 3, 4 ] 41184 10.436% 86.028% ## ( 4, 6 ] 39572 10.028% 96.056% ## ( 6, 10 ] 11110 2.815% 98.871% # ( 10, 15 ] 1910 0.484% 99.355% ( 15, 22 ] 307 0.078% 99.433% ( 22, 34 ] 65 0.016% 99.449% ( 34, 51 ] 26 0.007% 99.456% ( 51, 76 ] 29 0.007% 99.463% ( 76, 110 ] 27 0.007% 99.470% ( 110, 170 ] 103 0.026% 99.496% ( 170, 250 ] 99 0.025% 99.521% ( 250, 380 ] 162 0.041% 99.562% ( 380, 580 ] 330 0.084% 99.646% ( 580, 870 ] 372 0.094% 99.740% ( 870, 1300 ] 456 0.116% 99.856% ( 1300, 1900 ] 349 0.088% 99.944% ( 1900, 2900 ] 299 0.076% 100.020% ( 2900, 4400 ] 248 0.063% 100.083% ( 4400, 6600 ] 84 0.021% 100.104% ( 6600, 9900 ] 6 0.002% 100.105% ( 9900, 14000 ] 2 0.001% 100.106% ** Level 3 read latency histogram (micros): Count: 1684374 Average: 10.4290 StdDev: 140.50 Min: 0 Median: 1.9677 Max: 19105 Percentiles: P50: 1.97 P75: 2.93 P99: 9.65 P99.9: 882.99 P99.99: 1271.92 ------------------------------------------------------ [ 0, 1 ] 27730 1.646% 1.646% ( 1, 2 ] 841633 49.967% 51.613% ########## ( 2, 3 ] 422627 25.091% 76.704% ##### ( 3, 4 ] 172047 10.214% 86.919% ## ( 4, 6 ] 167846 9.965% 96.884% ## ( 6, 10 ] 39025 2.317% 99.201% ( 10, 15 ] 5855 0.348% 99.548% ( 15, 22 ] 773 0.046% 99.594% ( 22, 34 ] 71 0.004% 99.598% ( 34, 51 ] 94 0.006% 99.604% ( 51, 76 ] 137 0.008% 99.612% ( 76, 110 ] 163 0.010% 99.622% ( 110, 170 ] 505 0.030% 99.652% ( 170, 250 ] 514 0.031% 99.682% ( 250, 380 ] 746 0.044% 99.726% ( 380, 580 ] 1201 0.071% 99.798% ( 580, 870 ] 1672 0.099% 99.897% ( 870, 1300 ] 1676 0.100% 99.996% ( 1300, 1900 ] 1040 0.062% 100.058% ( 1900, 2900 ] 884 0.052% 100.111% ( 2900, 4400 ] 877 0.052% 100.163% ( 4400, 6600 ] 274 0.016% 100.179% ( 6600, 9900 ] 48 0.003% 100.182% ( 9900, 14000 ] 6 0.000% 100.182% ( 14000, 22000 ] 2 0.000% 100.182% ** Level 4 read latency histogram (micros): Count: 894744 Average: 20.0238 StdDev: 199.41 Min: 1 Median: 1.9706 Max: 19667 Percentiles: P50: 1.97 P75: 2.97 P99: 371.27 P99.9: 2112.81 P99.99: 3072.27 ------------------------------------------------------ [ 0, 1 ] 15729 1.758% 1.758% ( 1, 2 ] 444719 49.703% 51.461% ########## ( 2, 3 ] 216027 24.144% 75.605% ##### ( 3, 4 ] 91064 10.178% 85.783% ## ( 4, 6 ] 86164 9.630% 95.413% ## ( 6, 10 ] 21756 2.432% 97.845% ( 10, 15 ] 3111 0.348% 98.192% ( 15, 22 ] 396 0.044% 98.237% ( 22, 34 ] 35 0.004% 98.241% ( 34, 51 ] 60 0.007% 98.247% ( 51, 76 ] 110 0.012% 98.260% ( 76, 110 ] 712 0.080% 98.339% ( 110, 170 ] 2644 0.296% 98.635% ( 170, 250 ] 1833 0.205% 98.839% ( 250, 380 ] 1540 0.172% 99.012% ( 380, 580 ] 2076 0.232% 99.244% ( 580, 870 ] 2338 0.261% 99.505% ( 870, 1300 ] 2052 0.229% 99.734% ( 1300, 1900 ] 1293 0.145% 99.879% ( 1900, 2900 ] 894 0.100% 99.979% ( 2900, 4400 ] 884 0.099% 100.077% ( 4400, 6600 ] 286 0.032% 100.109% ( 6600, 9900 ] 55 0.006% 100.116% ( 9900, 14000 ] 5 0.001% 100.116% ( 14000, 22000 ] 2 0.000% 100.116% ** Level 5 read latency histogram (micros): Count: 69128 Average: 724.4353 StdDev: 1030.78 Min: 77 Median: 330.6884 Max: 21161 Percentiles: P50: 330.69 P75: 817.21 P99: 5447.42 P99.9: 8810.64 P99.99: 11975.54 ------------------------------------------------------ ( 76, 110 ] 4433 6.413% 6.413% # ( 110, 170 ] 16145 23.355% 29.768% ##### ( 170, 250 ] 9712 14.049% 43.817% ### ( 250, 380 ] 6886 9.961% 53.778% ## ( 380, 580 ] 8272 11.966% 65.745% ## ( 580, 870 ] 7822 11.315% 77.060% ## ( 870, 1300 ] 6129 8.866% 85.926% ## ( 1300, 1900 ] 3332 4.820% 90.746% # ( 1900, 2900 ] 2593 3.751% 94.497% # ( 2900, 4400 ] 2649 3.832% 98.329% # ( 4400, 6600 ] 974 1.409% 99.738% ( 6600, 9900 ] 167 0.242% 99.980% ( 9900, 14000 ] 14 0.020% 100.000% ( 14000, 22000 ] 3 0.004% 100.004% ** DB Stats ** Uptime(secs): 17.7 total, 3.5 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 167.18 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.134 H:M:S, 34.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 280.28 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/15-08:22:26 ... thread 0: (1000000,4000000) ops and (291095.6,190939.7) ops/second in (3.435297,20.949027) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/4 2.64 GB 245.4 3.1 0.0 3.1 6.9 3.7 0.0 1.8 91.6 201.2 35.06 23.32 64 0.548 3262K 3677 0.0 0.0 L1 12/0 1.27 GB 130.6 1.8 1.5 0.3 1.8 1.5 0.0 1.2 132.1 131.5 14.12 8.86 2 7.061 1894K 7907 0.0 0.0 L2 0/0 0.00 KB 0.0 0.2 0.1 0.1 0.2 0.1 0.2 1.6 143.6 143.3 1.46 0.95 1 1.457 212K 396 0.0 0.0 L3 16/0 919.89 MB 0.9 4.3 2.0 2.3 4.2 2.0 0.1 2.1 115.6 114.9 37.85 20.27 22 1.720 1766K 0 0.0 0.0 L4 173/0 9.65 GB 1.0 7.9 4.5 3.4 7.7 4.3 0.5 1.7 116.1 114.4 69.39 37.10 55 1.262 3706K 3297 0.0 0.0 L5 1142/0 94.61 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1357/4 109.07 GB 0.0 17.3 8.1 9.2 20.9 11.7 5.9 5.6 112.2 135.6 157.87 90.49 144 1.096 10M 15K 0.0 0.0 Int 0/0 0.00 KB 0.0 17.3 8.1 9.2 20.9 11.7 5.9 5.6 112.2 135.6 157.87 90.49 144 1.096 10M 15K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 17.3 8.1 9.2 17.1 8.0 0.0 0.0 128.2 127.1 138.18 79.67 91 1.519 10M 15K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 195.2 19.69 10.82 53 0.371 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 21.1 total, 21.0 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 20.90 GB write, 1014.44 MB/s write, 17.30 GB read, 839.92 MB/s read, 157.9 seconds Interval compaction: 20.90 GB write, 1021.48 MB/s write, 17.30 GB read, 845.74 MB/s read, 157.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 64, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 64, interval: 64 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1256250 Average: 8.3643 StdDev: 95.30 Min: 0 Median: 1.6665 Max: 10491 Percentiles: P50: 1.67 P75: 2.38 P99: 136.94 P99.9: 902.33 P99.99: 1886.57 ------------------------------------------------------ [ 0, 1 ] 155659 12.391% 12.391% ## ( 1, 2 ] 708885 56.429% 68.819% ########### ( 2, 3 ] 206382 16.428% 85.248% ### ( 3, 4 ] 80553 6.412% 91.660% # ( 4, 6 ] 69181 5.507% 97.167% # ( 6, 10 ] 15947 1.269% 98.436% ( 10, 15 ] 2444 0.195% 98.631% ( 15, 22 ] 388 0.031% 98.662% ( 22, 34 ] 49 0.004% 98.666% ( 34, 51 ] 25 0.002% 98.668% ( 51, 76 ] 32 0.003% 98.670% ( 76, 110 ] 1524 0.121% 98.792% ( 110, 170 ] 5832 0.464% 99.256% ( 170, 250 ] 3242 0.258% 99.514% ( 250, 380 ] 1728 0.138% 99.651% ( 380, 580 ] 1837 0.146% 99.798% ( 580, 870 ] 1228 0.098% 99.895% ( 870, 1300 ] 768 0.061% 99.957% ( 1300, 1900 ] 430 0.034% 99.991% ( 1900, 2900 ] 281 0.022% 100.013% ( 2900, 4400 ] 258 0.021% 100.034% ( 4400, 6600 ] 82 0.007% 100.040% ( 6600, 9900 ] 12 0.001% 100.041% ( 9900, 14000 ] 1 0.000% 100.041% ** Level 1 read latency histogram (micros): Count: 167129 Average: 13.5455 StdDev: 159.56 Min: 1 Median: 1.8910 Max: 8521 Percentiles: P50: 1.89 P75: 2.82 P99: 137.33 P99.9: 2704.17 P99.99: 5639.77 ------------------------------------------------------ [ 0, 1 ] 4702 2.813% 2.813% # ( 1, 2 ] 88506 52.957% 55.770% ########### ( 2, 3 ] 38977 23.322% 79.092% ##### ( 3, 4 ] 14843 8.881% 87.973% ## ( 4, 6 ] 13593 8.133% 96.106% ## ( 6, 10 ] 3873 2.317% 98.423% ( 10, 15 ] 556 0.333% 98.756% ( 15, 22 ] 76 0.045% 98.802% ( 22, 34 ] 11 0.007% 98.808% ( 34, 51 ] 9 0.005% 98.813% ( 51, 76 ] 5 0.003% 98.816% ( 76, 110 ] 130 0.078% 98.894% ( 110, 170 ] 388 0.232% 99.126% ( 170, 250 ] 240 0.144% 99.270% ( 250, 380 ] 148 0.089% 99.359% ( 380, 580 ] 202 0.121% 99.479% ( 580, 870 ] 235 0.141% 99.620% ( 870, 1300 ] 246 0.147% 99.767% ( 1300, 1900 ] 131 0.078% 99.846% ( 1900, 2900 ] 113 0.068% 99.913% ( 2900, 4400 ] 108 0.065% 99.978% ( 4400, 6600 ] 36 0.022% 99.999% ( 6600, 9900 ] 9 0.005% 100.005% ** Level 2 read latency histogram (micros): Count: 394790 Average: 12.2430 StdDev: 153.39 Min: 1 Median: 1.9809 Max: 11110 Percentiles: P50: 1.98 P75: 2.98 P99: 11.75 P99.9: 1601.22 P99.99: 2507.09 ------------------------------------------------------ [ 0, 1 ] 9498 2.406% 2.406% ( 1, 2 ] 191548 48.519% 50.925% ########## ( 2, 3 ] 97261 24.636% 75.561% ##### ( 3, 4 ] 41184 10.432% 85.993% ## ( 4, 6 ] 39572 10.024% 96.016% ## ( 6, 10 ] 11110 2.814% 98.831% # ( 10, 15 ] 1910 0.484% 99.314% ( 15, 22 ] 307 0.078% 99.392% ( 22, 34 ] 65 0.016% 99.409% ( 34, 51 ] 26 0.007% 99.415% ( 51, 76 ] 29 0.007% 99.422% ( 76, 110 ] 51 0.013% 99.435% ( 110, 170 ] 165 0.042% 99.477% ( 170, 250 ] 132 0.033% 99.511% ( 250, 380 ] 184 0.047% 99.557% ( 380, 580 ] 345 0.087% 99.645% ( 580, 870 ] 377 0.095% 99.740% ( 870, 1300 ] 456 0.116% 99.856% ( 1300, 1900 ] 349 0.088% 99.944% ( 1900, 2900 ] 299 0.076% 100.020% ( 2900, 4400 ] 248 0.063% 100.083% ( 4400, 6600 ] 84 0.021% 100.104% ( 6600, 9900 ] 6 0.002% 100.105% ( 9900, 14000 ] 2 0.001% 100.106% ** Level 3 read latency histogram (micros): Count: 1685318 Average: 10.5384 StdDev: 140.58 Min: 0 Median: 1.9683 Max: 19105 Percentiles: P50: 1.97 P75: 2.93 P99: 9.75 P99.9: 883.76 P99.99: 1272.44 ------------------------------------------------------ [ 0, 1 ] 27730 1.645% 1.645% ( 1, 2 ] 841633 49.939% 51.585% ########## ( 2, 3 ] 422627 25.077% 76.661% ##### ( 3, 4 ] 172047 10.209% 86.870% ## ( 4, 6 ] 167846 9.959% 96.829% ## ( 6, 10 ] 39025 2.316% 99.145% ( 10, 15 ] 5855 0.347% 99.492% ( 15, 22 ] 773 0.046% 99.538% ( 22, 34 ] 71 0.004% 99.542% ( 34, 51 ] 94 0.006% 99.548% ( 51, 76 ] 137 0.008% 99.556% ( 76, 110 ] 304 0.018% 99.574% ( 110, 170 ] 937 0.056% 99.630% ( 170, 250 ] 700 0.042% 99.671% ( 250, 380 ] 830 0.049% 99.721% ( 380, 580 ] 1282 0.076% 99.797% ( 580, 870 ] 1688 0.100% 99.897% ( 870, 1300 ] 1678 0.100% 99.996% ( 1300, 1900 ] 1041 0.062% 100.058% ( 1900, 2900 ] 885 0.053% 100.111% ( 2900, 4400 ] 877 0.052% 100.163% ( 4400, 6600 ] 274 0.016% 100.179% ( 6600, 9900 ] 48 0.003% 100.182% ( 9900, 14000 ] 6 0.000% 100.182% ( 14000, 22000 ] 2 0.000% 100.182% ** Level 4 read latency histogram (micros): Count: 902591 Average: 21.4913 StdDev: 199.67 Min: 1 Median: 1.9793 Max: 19667 Percentiles: P50: 1.98 P75: 3.00 P99: 424.24 P99.9: 2107.81 P99.99: 3072.44 ------------------------------------------------------ [ 0, 1 ] 15729 1.743% 1.743% ( 1, 2 ] 444753 49.275% 51.018% ########## ( 2, 3 ] 216203 23.954% 74.971% ##### ( 3, 4 ] 91184 10.102% 85.074% ## ( 4, 6 ] 86298 9.561% 94.635% ## ( 6, 10 ] 21773 2.412% 97.047% ( 10, 15 ] 3114 0.345% 97.392% ( 15, 22 ] 396 0.044% 97.436% ( 22, 34 ] 35 0.004% 97.440% ( 34, 51 ] 60 0.007% 97.447% ( 51, 76 ] 111 0.012% 97.459% ( 76, 110 ] 1920 0.213% 97.672% ( 110, 170 ] 6023 0.667% 98.339% ( 170, 250 ] 3252 0.360% 98.699% ( 250, 380 ] 2135 0.237% 98.936% ( 380, 580 ] 2618 0.290% 99.226% ( 580, 870 ] 2523 0.280% 99.505% ( 870, 1300 ] 2079 0.230% 99.736% ( 1300, 1900 ] 1296 0.144% 99.879% ( 1900, 2900 ] 897 0.099% 99.979% ( 2900, 4400 ] 885 0.098% 100.077% ( 4400, 6600 ] 286 0.032% 100.108% ( 6600, 9900 ] 55 0.006% 100.115% ( 9900, 14000 ] 5 0.001% 100.115% ( 14000, 22000 ] 2 0.000% 100.115% ** Level 5 read latency histogram (micros): Count: 112754 Average: 522.3018 StdDev: 851.38 Min: 74 Median: 208.7706 Max: 21161 Percentiles: P50: 208.77 P75: 524.38 P99: 4455.25 P99.9: 7889.29 P99.99: 9894.56 ------------------------------------------------------ ( 51, 76 ] 8 0.007% 0.007% ( 76, 110 ] 11631 10.315% 10.322% ## ( 110, 170 ] 36081 32.000% 42.322% ###### ( 170, 250 ] 17863 15.842% 58.165% ### ( 250, 380 ] 10697 9.487% 67.652% ## ( 380, 580 ] 11477 10.179% 77.830% ## ( 580, 870 ] 8928 7.918% 85.749% ## ( 870, 1300 ] 6295 5.583% 91.332% # ( 1300, 1900 ] 3355 2.976% 94.307% # ( 1900, 2900 ] 2616 2.320% 96.627% ( 2900, 4400 ] 2651 2.351% 98.978% ( 4400, 6600 ] 974 0.864% 99.842% ( 6600, 9900 ] 167 0.148% 99.990% ( 9900, 14000 ] 14 0.012% 100.003% ( 14000, 22000 ] 3 0.003% 100.005% ** DB Stats ** Uptime(secs): 21.1 total, 3.4 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 186.61 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.134 H:M:S, 29.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 286.49 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/12/15-08:22:30 ... thread 0: (1000000,5000000) ops and (284148.3,204345.9) ops/second in (3.519289,24.468316) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/4 3.63 GB 346.1 4.1 0.0 4.1 8.9 4.7 0.0 1.9 100.5 216.0 41.98 29.75 84 0.500 4285K 4669 0.0 0.0 L1 12/0 1.27 GB 130.6 1.8 1.5 0.3 1.8 1.5 0.0 1.2 132.1 131.5 14.12 8.86 2 7.061 1894K 7907 0.0 0.0 L2 0/0 0.00 KB 0.0 0.2 0.1 0.1 0.2 0.1 0.2 1.6 143.6 143.3 1.46 0.95 1 1.457 212K 396 0.0 0.0 L3 16/0 919.89 MB 0.9 4.3 2.0 2.3 4.2 2.0 0.1 2.1 115.6 114.9 37.85 20.27 22 1.720 1766K 0 0.0 0.0 L4 173/0 9.65 GB 1.0 7.9 4.5 3.4 7.7 4.3 0.5 1.7 116.1 114.4 69.39 37.10 55 1.262 3706K 3297 0.0 0.0 L5 1142/0 94.61 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1361/4 110.06 GB 0.0 18.3 8.1 10.2 22.9 12.7 5.9 4.8 113.6 142.1 164.80 96.92 164 1.005 11M 16K 0.0 0.0 Int 0/0 0.00 KB 0.0 18.3 8.1 10.2 22.9 12.7 5.9 4.8 113.6 142.1 164.80 96.92 164 1.005 11M 16K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 18.3 8.1 10.2 18.1 8.0 0.0 0.0 131.8 130.7 142.06 83.45 95 1.495 11M 16K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 213.3 22.74 13.47 69 0.330 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 24.6 total, 24.5 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 22.87 GB write, 951.26 MB/s write, 18.29 GB read, 760.78 MB/s read, 164.8 seconds Interval compaction: 22.87 GB write, 956.91 MB/s write, 18.29 GB read, 765.30 MB/s read, 164.8 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 1, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 1, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 64, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 1, total-stops: 64, interval: 65 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1526343 Average: 7.6616 StdDev: 87.59 Min: 0 Median: 1.6097 Max: 10491 Percentiles: P50: 1.61 P75: 2.19 P99: 131.84 P99.9: 811.66 P99.99: 1761.94 ------------------------------------------------------ [ 0, 1 ] 235075 15.401% 15.401% ### ( 1, 2 ] 866181 56.749% 72.150% ########### ( 2, 3 ] 225098 14.748% 86.898% ### ( 3, 4 ] 87001 5.700% 92.597% # ( 4, 6 ] 73117 4.790% 97.388% # ( 6, 10 ] 16758 1.098% 98.486% ( 10, 15 ] 2718 0.178% 98.664% ( 15, 22 ] 417 0.027% 98.691% ( 22, 34 ] 53 0.003% 98.695% ( 34, 51 ] 27 0.002% 98.696% ( 51, 76 ] 32 0.002% 98.698% ( 76, 110 ] 1951 0.128% 98.826% ( 110, 170 ] 7284 0.477% 99.303% ( 170, 250 ] 4028 0.264% 99.567% ( 250, 380 ] 2000 0.131% 99.698% ( 380, 580 ] 2039 0.134% 99.832% ( 580, 870 ] 1299 0.085% 99.917% ( 870, 1300 ] 779 0.051% 99.968% ( 1300, 1900 ] 433 0.028% 99.997% ( 1900, 2900 ] 289 0.019% 100.015% ( 2900, 4400 ] 260 0.017% 100.032% ( 4400, 6600 ] 83 0.005% 100.038% ( 6600, 9900 ] 12 0.001% 100.039% ( 9900, 14000 ] 1 0.000% 100.039% ** Level 1 read latency histogram (micros): Count: 168451 Average: 14.2992 StdDev: 159.57 Min: 1 Median: 1.8980 Max: 8521 Percentiles: P50: 1.90 P75: 2.85 P99: 174.34 P99.9: 2692.47 P99.99: 5631.69 ------------------------------------------------------ [ 0, 1 ] 4702 2.791% 2.791% # ( 1, 2 ] 88559 52.573% 55.364% ########### ( 2, 3 ] 39139 23.235% 78.599% ##### ( 3, 4 ] 14977 8.891% 87.490% ## ( 4, 6 ] 13754 8.165% 95.655% ## ( 6, 10 ] 3905 2.318% 97.973% ( 10, 15 ] 561 0.333% 98.306% ( 15, 22 ] 77 0.046% 98.351% ( 22, 34 ] 11 0.007% 98.358% ( 34, 51 ] 9 0.005% 98.363% ( 51, 76 ] 5 0.003% 98.366% ( 76, 110 ] 275 0.163% 98.530% ( 110, 170 ] 773 0.459% 98.988% ( 170, 250 ] 359 0.213% 99.202% ( 250, 380 ] 214 0.127% 99.329% ( 380, 580 ] 245 0.145% 99.474% ( 580, 870 ] 250 0.148% 99.622% ( 870, 1300 ] 247 0.147% 99.769% ( 1300, 1900 ] 131 0.078% 99.847% ( 1900, 2900 ] 113 0.067% 99.914% ( 2900, 4400 ] 108 0.064% 99.978% ( 4400, 6600 ] 36 0.021% 99.999% ( 6600, 9900 ] 9 0.005% 100.005% ** Level 2 read latency histogram (micros): Count: 394972 Average: 12.3158 StdDev: 153.41 Min: 1 Median: 1.9814 Max: 11110 Percentiles: P50: 1.98 P75: 2.98 P99: 12.22 P99.9: 1600.91 P99.99: 2507.03 ------------------------------------------------------ [ 0, 1 ] 9498 2.405% 2.405% ( 1, 2 ] 191548 48.497% 50.901% ########## ( 2, 3 ] 97261 24.625% 75.526% ##### ( 3, 4 ] 41184 10.427% 85.953% ## ( 4, 6 ] 39572 10.019% 95.972% ## ( 6, 10 ] 11110 2.813% 98.785% # ( 10, 15 ] 1910 0.484% 99.269% ( 15, 22 ] 307 0.078% 99.346% ( 22, 34 ] 65 0.016% 99.363% ( 34, 51 ] 26 0.007% 99.369% ( 51, 76 ] 29 0.007% 99.377% ( 76, 110 ] 100 0.025% 99.402% ( 110, 170 ] 247 0.063% 99.465% ( 170, 250 ] 158 0.040% 99.505% ( 250, 380 ] 198 0.050% 99.555% ( 380, 580 ] 354 0.090% 99.644% ( 580, 870 ] 379 0.096% 99.740% ( 870, 1300 ] 456 0.115% 99.856% ( 1300, 1900 ] 349 0.088% 99.944% ( 1900, 2900 ] 299 0.076% 100.020% ( 2900, 4400 ] 248 0.063% 100.083% ( 4400, 6600 ] 84 0.021% 100.104% ( 6600, 9900 ] 6 0.002% 100.105% ( 9900, 14000 ] 2 0.001% 100.106% ** Level 3 read latency histogram (micros): Count: 1686359 Average: 10.6424 StdDev: 140.63 Min: 0 Median: 1.9689 Max: 19105 Percentiles: P50: 1.97 P75: 2.94 P99: 9.86 P99.9: 883.99 P99.99: 1272.69 ------------------------------------------------------ [ 0, 1 ] 27730 1.644% 1.644% ( 1, 2 ] 841633 49.908% 51.553% ########## ( 2, 3 ] 422627 25.062% 76.614% ##### ( 3, 4 ] 172047 10.202% 86.816% ## ( 4, 6 ] 167846 9.953% 96.770% ## ( 6, 10 ] 39025 2.314% 99.084% ( 10, 15 ] 5855 0.347% 99.431% ( 15, 22 ] 773 0.046% 99.477% ( 22, 34 ] 71 0.004% 99.481% ( 34, 51 ] 94 0.006% 99.487% ( 51, 76 ] 137 0.008% 99.495% ( 76, 110 ] 497 0.029% 99.524% ( 110, 170 ] 1470 0.087% 99.611% ( 170, 250 ] 874 0.052% 99.663% ( 250, 380 ] 902 0.053% 99.717% ( 380, 580 ] 1334 0.079% 99.796% ( 580, 870 ] 1703 0.101% 99.897% ( 870, 1300 ] 1679 0.100% 99.996% ( 1300, 1900 ] 1042 0.062% 100.058% ( 1900, 2900 ] 885 0.052% 100.111% ( 2900, 4400 ] 877 0.052% 100.163% ( 4400, 6600 ] 274 0.016% 100.179% ( 6600, 9900 ] 48 0.003% 100.182% ( 9900, 14000 ] 6 0.000% 100.182% ( 14000, 22000 ] 2 0.000% 100.182% ** Level 4 read latency histogram (micros): Count: 910960 Average: 22.9859 StdDev: 199.84 Min: 1 Median: 1.9887 Max: 19667 Percentiles: P50: 1.99 P75: 3.07 P99: 453.72 P99.9: 2105.14 P99.99: 3074.02 ------------------------------------------------------ [ 0, 1 ] 15729 1.727% 1.727% ( 1, 2 ] 444761 48.823% 50.550% ########## ( 2, 3 ] 216239 23.737% 74.287% ##### ( 3, 4 ] 91201 10.012% 84.299% ## ( 4, 6 ] 86334 9.477% 93.776% ## ( 6, 10 ] 21780 2.391% 96.167% ( 10, 15 ] 3115 0.342% 96.509% ( 15, 22 ] 396 0.043% 96.553% ( 22, 34 ] 35 0.004% 96.556% ( 34, 51 ] 60 0.007% 96.563% ( 51, 76 ] 112 0.012% 96.575% ( 76, 110 ] 3462 0.380% 96.955% ( 110, 170 ] 10164 1.116% 98.071% ( 170, 250 ] 4615 0.507% 98.578% ( 250, 380 ] 2707 0.297% 98.875% ( 380, 580 ] 3094 0.340% 99.214% ( 580, 870 ] 2654 0.291% 99.506% ( 870, 1300 ] 2108 0.231% 99.737% ( 1300, 1900 ] 1298 0.142% 99.880% ( 1900, 2900 ] 902 0.099% 99.979% ( 2900, 4400 ] 887 0.097% 100.076% ( 4400, 6600 ] 286 0.031% 100.107% ( 6600, 9900 ] 55 0.006% 100.114% ( 9900, 14000 ] 5 0.001% 100.114% ( 14000, 22000 ] 2 0.000% 100.114% ** Level 5 read latency histogram (micros): Count: 159803 Average: 423.8876 StdDev: 735.65 Min: 74 Median: 170.4604 Max: 21161 Percentiles: P50: 170.46 P75: 386.36 P99: 4148.58 P99.9: 6939.82 P99.99: 9781.83 ------------------------------------------------------ ( 51, 76 ] 17 0.011% 0.011% ( 76, 110 ] 20534 12.850% 12.860% ### ( 110, 170 ] 59202 37.047% 49.907% ####### ( 170, 250 ] 25802 16.146% 66.053% ### ( 250, 380 ] 13839 8.660% 74.713% ## ( 380, 580 ] 14408 9.016% 83.729% ## ( 580, 870 ] 9752 6.103% 89.832% # ( 870, 1300 ] 6407 4.009% 93.841% # ( 1300, 1900 ] 3388 2.120% 95.961% ( 1900, 2900 ] 2636 1.650% 97.611% ( 2900, 4400 ] 2667 1.669% 99.280% ( 4400, 6600 ] 974 0.610% 99.889% ( 6600, 9900 ] 167 0.105% 99.994% ( 9900, 14000 ] 14 0.009% 100.003% ( 14000, 22000 ] 3 0.002% 100.004% ** DB Stats ** Uptime(secs): 24.6 total, 3.5 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 199.91 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.208 H:M:S, 25.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 279.66 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.074 H:M:S, 2.1 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 247.162 micros/op 16170 ops/sec 60.171 seconds 972996 operations; 15.7 MB/s (253999 of 253999 found) Microseconds per read: Count: 972996 Average: 247.1653 StdDev: 703.66 Min: 2 Median: 168.2450 Max: 76292 Percentiles: P50: 168.24 P75: 230.66 P99: 1461.23 P99.9: 11151.04 P99.99: 28607.83 ------------------------------------------------------ ( 1, 2 ] 4 0.000% 0.000% ( 2, 3 ] 39 0.004% 0.004% ( 3, 4 ] 70 0.007% 0.012% ( 4, 6 ] 76 0.008% 0.019% ( 6, 10 ] 513 0.053% 0.072% ( 10, 15 ] 4670 0.480% 0.552% ( 15, 22 ] 5721 0.588% 1.140% ( 22, 34 ] 8931 0.918% 2.058% ( 34, 51 ] 3057 0.314% 2.372% ( 51, 76 ] 454 0.047% 2.419% ( 76, 110 ] 25982 2.670% 5.089% # ( 110, 170 ] 450148 46.264% 51.353% ######### ( 170, 250 ] 303430 31.185% 82.538% ###### ( 250, 380 ] 97433 10.014% 92.552% ## ( 380, 580 ] 37489 3.853% 96.405% # ( 580, 870 ] 16299 1.675% 98.080% ( 870, 1300 ] 7966 0.819% 98.899% ( 1300, 1900 ] 3662 0.376% 99.275% ( 1900, 2900 ] 2552 0.262% 99.538% ( 2900, 4400 ] 2232 0.229% 99.767% ( 4400, 6600 ] 792 0.081% 99.848% ( 6600, 9900 ] 366 0.038% 99.886% ( 9900, 14000 ] 449 0.046% 99.932% ( 14000, 22000 ] 479 0.049% 99.981% ( 22000, 33000 ] 141 0.014% 99.996% ( 33000, 50000 ] 30 0.003% 99.999% ( 50000, 75000 ] 10 0.001% 100.000% ( 75000, 110000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 20539327 rocksdb.bloom.filter.full.positive COUNT : 1173591 rocksdb.bloom.filter.full.true.positive COUNT : 972577 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 419 rocksdb.memtable.miss COUNT : 972577 rocksdb.l0.hit COUNT : 39004 rocksdb.l1.hit COUNT : 11405 rocksdb.l2andup.hit COUNT : 922168 rocksdb.compaction.key.drop.new COUNT : 202237 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5743266 rocksdb.number.keys.read COUNT : 972996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5927050512 rocksdb.bytes.read COUNT : 972996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1658 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 38437699 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 22119211655 rocksdb.compact.write.bytes COUNT : 21914850260 rocksdb.flush.write.bytes COUNT : 5879153286 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 26 rocksdb.number.superversion_acquires COUNT : 1167 rocksdb.number.superversion_releases COUNT : 1125 rocksdb.number.superversion_cleanups COUNT : 291 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 287 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5831676928 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2204672 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 27082947324 rocksdb.non.last.level.read.count COUNT : 6401572 rocksdb.block.checksum.compute.count COUNT : 6545473 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 167.401182 P95 : 506.078815 P99 : 1459.491924 P100 : 76289.000000 COUNT : 972996 SUM : 239667902 rocksdb.db.write.micros P50 : 1.527946 P95 : 4.234050 P99 : 8.751753 P100 : 802309.000000 COUNT : 5743266 SUM : 52720677 rocksdb.compaction.times.micros P50 : 1271794.871795 P95 : 2810000.000000 P99 : 6566000.000000 P100 : 8615554.000000 COUNT : 98 SUM : 144952165 rocksdb.compaction.times.cpu_micros P50 : 790952.380952 P95 : 1774000.000000 P99 : 4344000.000000 P100 : 6299556.000000 COUNT : 98 SUM : 86266536 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 797.500000 P95 : 100812.500000 P99 : 416100.000000 P100 : 447981.000000 COUNT : 81 SUM : 1610697 rocksdb.compaction.outfile.sync.micros P50 : 40140.000000 P95 : 232434.782609 P99 : 364957.142857 P100 : 1022474.000000 COUNT : 281 SUM : 19997064 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 218.571429 P95 : 4388.157895 P99 : 6153.400000 P100 : 6556.000000 COUNT : 203 SUM : 215637 rocksdb.table.open.io.micros P50 : 8552.783505 P95 : 19444.000000 P99 : 23401.052632 P100 : 28282.000000 COUNT : 1658 SUM : 14272337 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 4.028854 P95 : 9.918399 P99 : 164.879628 P100 : 19670.000000 COUNT : 5357794 SUM : 76874065 rocksdb.read.block.get.micros P50 : 134.206324 P95 : 368.559900 P99 : 1281.131557 P100 : 21165.000000 COUNT : 1186011 SUM : 203908150 rocksdb.write.raw.block.micros P50 : 0.538415 P95 : 1.416156 P99 : 3.904776 P100 : 23498.000000 COUNT : 6731899 SUM : 20256937 rocksdb.numfiles.in.singlecompaction P50 : 2.350000 P95 : 3.993243 P99 : 9.666667 P100 : 29.000000 COUNT : 125 SUM : 396 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.584252 P95 : 2187.253133 P99 : 2758.814035 P100 : 802133.000000 COUNT : 142533 SUM : 38451150 rocksdb.sst.read.micros P50 : 2.236413 P95 : 162.094010 P99 : 474.604180 P100 : 21161.000000 COUNT : 6401600 SUM : 264145410 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 972996 SUM : 972996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5743266 SUM : 5927050512 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 222258.064516 P95 : 924600.000000 P99 : 1028280.000000 P100 : 1028280.000000 COUNT : 81 SUM : 25222897 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 297 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0