[2024-12-15 09:58:56.922049] Starting SPDK v25.01-pre git sha1 e01cb43b8 / DPDK 24.03.0 initialization... [2024-12-15 09:58:56.922315] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1719388 ] Set seed to 1734253136915266 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-15 09:58:57.049110] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-15 09:58:57.158771] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-15 09:58:57.158810] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1719388' to capture a snapshot of events at runtime. [2024-12-15 09:58:57.158820] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-12-15 09:58:57.158847] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-12-15 09:58:57.158855] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1719388 for offline analysis/debug. [2024-12-15 09:58:57.160237] reactor.c: 995:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sun Dec 15 09:59:05 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/15-09:59:13 ... thread 0: (1000000,1000000) ops and (132819.4,132819.4) ops/second in (7.529021,7.529021) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 6/4 566.89 MB 18.9 0.0 0.0 0.0 0.8 0.8 0.0 1.0 0.0 118.1 6.93 2.33 9 0.770 0 0 0.0 0.0 L1 1/1 128.76 MB 0.0 0.4 0.3 0.1 0.4 0.3 0.0 1.4 119.9 119.8 3.62 2.00 2 1.812 441K 335 0.0 0.0 L2 4/1 243.09 MB 1.9 0.2 0.1 0.1 0.2 0.1 0.3 3.1 116.3 116.1 1.63 0.86 1 1.630 192K 386 0.0 0.0 L3 48/22 4.05 GB 2.6 4.1 2.9 1.2 4.1 2.9 0.2 1.4 98.7 97.8 42.56 19.94 24 1.773 2041K 469 0.0 0.0 L4 198/15 12.14 GB 1.2 2.0 1.5 0.5 2.0 1.5 1.1 1.3 116.3 115.4 17.55 9.43 17 1.033 1494K 163 0.0 0.0 L5 1088/0 89.45 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1345/43 106.55 GB 0.0 6.7 4.8 1.9 7.5 5.6 1.6 9.3 95.0 105.5 72.30 34.55 53 1.364 4170K 1353 0.0 0.0 Int 0/0 0.00 KB 0.0 6.7 4.8 1.9 7.5 5.6 1.6 9.3 95.0 105.5 72.30 34.55 53 1.364 4170K 1353 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.7 4.8 1.9 6.7 4.8 0.0 0.0 105.1 104.2 65.37 32.22 44 1.486 4170K 1353 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 118.1 6.93 2.33 9 0.770 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.7 total, 7.5 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.45 GB write, 987.73 MB/s write, 6.71 GB read, 889.13 MB/s read, 72.3 seconds Interval compaction: 7.45 GB write, 1013.06 MB/s write, 6.71 GB read, 911.93 MB/s read, 72.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 46, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 46, interval: 46 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 90387 Average: 12.8667 StdDev: 170.15 Min: 1 Median: 2.0276 Max: 22313 Percentiles: P50: 2.03 P75: 2.95 P99: 11.07 P99.9: 2802.48 P99.99: 5239.60 ------------------------------------------------------ [ 0, 1 ] 1595 1.765% 1.765% ( 1, 2 ] 42924 47.489% 49.254% ######### ( 2, 3 ] 24467 27.069% 76.323% ##### ( 3, 4 ] 8809 9.746% 86.069% ## ( 4, 6 ] 9801 10.843% 96.912% ## ( 6, 10 ] 1795 1.986% 98.898% ( 10, 15 ] 436 0.482% 99.380% ( 15, 22 ] 50 0.055% 99.436% ( 22, 34 ] 4 0.004% 99.440% ( 34, 51 ] 7 0.008% 99.448% ( 110, 170 ] 1 0.001% 99.449% ( 170, 250 ] 9 0.010% 99.459% ( 250, 380 ] 13 0.014% 99.473% ( 380, 580 ] 31 0.034% 99.508% ( 580, 870 ] 60 0.066% 99.574% ( 870, 1300 ] 91 0.101% 99.675% ( 1300, 1900 ] 127 0.141% 99.815% ( 1900, 2900 ] 86 0.095% 99.910% ( 2900, 4400 ] 68 0.075% 99.986% ( 4400, 6600 ] 13 0.014% 100.000% ( 22000, 33000 ] 1 0.001% 100.001% ** Level 1 read latency histogram (micros): Count: 88223 Average: 12.2425 StdDev: 179.55 Min: 1 Median: 2.1678 Max: 23267 Percentiles: P50: 2.17 P75: 3.00 P99: 10.57 P99.9: 2877.77 P99.99: 6010.61 ------------------------------------------------------ [ 0, 1 ] 1064 1.206% 1.206% ( 1, 2 ] 38604 43.757% 44.963% ######### ( 2, 3 ] 26487 30.023% 74.986% ###### ( 3, 4 ] 8956 10.152% 85.138% ## ( 4, 6 ] 10185 11.545% 96.682% ## ( 6, 10 ] 1989 2.255% 98.937% ( 10, 15 ] 492 0.558% 99.494% ( 15, 22 ] 64 0.073% 99.567% ( 22, 34 ] 5 0.006% 99.573% ( 34, 51 ] 2 0.002% 99.575% ( 51, 76 ] 3 0.003% 99.578% ( 110, 170 ] 1 0.001% 99.579% ( 170, 250 ] 4 0.005% 99.584% ( 250, 380 ] 11 0.012% 99.596% ( 380, 580 ] 16 0.018% 99.615% ( 580, 870 ] 27 0.031% 99.645% ( 870, 1300 ] 61 0.069% 99.714% ( 1300, 1900 ] 66 0.075% 99.789% ( 1900, 2900 ] 100 0.113% 99.903% ( 2900, 4400 ] 64 0.073% 99.975% ( 4400, 6600 ] 18 0.020% 99.995% ( 6600, 9900 ] 4 0.005% 100.000% ( 22000, 33000 ] 1 0.001% 100.001% ** Level 2 read latency histogram (micros): Count: 717479 Average: 19.4079 StdDev: 222.00 Min: 1 Median: 2.1480 Max: 25106 Percentiles: P50: 2.15 P75: 3.05 P99: 12.66 P99.9: 2026.04 P99.99: 2469.54 ------------------------------------------------------ [ 0, 1 ] 10493 1.462% 1.462% ( 1, 2 ] 317728 44.284% 45.746% ######### ( 2, 3 ] 206207 28.740% 74.487% ###### ( 3, 4 ] 70842 9.874% 84.361% ## ( 4, 6 ] 80889 11.274% 95.635% ## ( 6, 10 ] 21438 2.988% 98.623% # ( 10, 15 ] 5095 0.710% 99.333% ( 15, 22 ] 769 0.107% 99.440% ( 22, 34 ] 58 0.008% 99.448% ( 34, 51 ] 26 0.004% 99.452% ( 51, 76 ] 20 0.003% 99.454% ( 76, 110 ] 9 0.001% 99.456% ( 110, 170 ] 31 0.004% 99.460% ( 170, 250 ] 68 0.009% 99.470% ( 250, 380 ] 154 0.021% 99.491% ( 380, 580 ] 254 0.035% 99.526% ( 580, 870 ] 484 0.067% 99.594% ( 870, 1300 ] 844 0.118% 99.711% ( 1300, 1900 ] 1169 0.163% 99.874% ( 1900, 2900 ] 1456 0.203% 100.077% ( 2900, 4400 ] 865 0.121% 100.198% ( 4400, 6600 ] 305 0.043% 100.240% ( 6600, 9900 ] 47 0.007% 100.247% ( 9900, 14000 ] 1 0.000% 100.247% ( 14000, 22000 ] 1 0.000% 100.247% ( 22000, 33000 ] 2 0.000% 100.248% ** Level 3 read latency histogram (micros): Count: 819379 Average: 12.6924 StdDev: 168.73 Min: 1 Median: 2.1950 Max: 23941 Percentiles: P50: 2.20 P75: 3.03 P99: 9.64 P99.9: 855.40 P99.99: 1386.45 ------------------------------------------------------ [ 0, 1 ] 7032 0.858% 0.858% ( 1, 2 ] 353569 43.151% 44.009% ######### ( 2, 3 ] 251697 30.718% 74.727% ###### ( 3, 4 ] 83589 10.202% 84.929% ## ( 4, 6 ] 96608 11.790% 96.719% ## ( 6, 10 ] 20536 2.506% 99.225% # ( 10, 15 ] 4130 0.504% 99.729% ( 15, 22 ] 497 0.061% 99.790% ( 22, 34 ] 54 0.007% 99.797% ( 34, 51 ] 31 0.004% 99.800% ( 51, 76 ] 54 0.007% 99.807% ( 76, 110 ] 13 0.002% 99.809% ( 110, 170 ] 38 0.005% 99.813% ( 170, 250 ] 64 0.008% 99.821% ( 250, 380 ] 89 0.011% 99.832% ( 380, 580 ] 195 0.024% 99.856% ( 580, 870 ] 385 0.047% 99.903% ( 870, 1300 ] 610 0.074% 99.977% ( 1300, 1900 ] 750 0.092% 100.069% ( 1900, 2900 ] 886 0.108% 100.177% ( 2900, 4400 ] 670 0.082% 100.258% ( 4400, 6600 ] 195 0.024% 100.282% ( 6600, 9900 ] 24 0.003% 100.285% ( 9900, 14000 ] 1 0.000% 100.285% ( 22000, 33000 ] 1 0.000% 100.285% ** Level 4 read latency histogram (micros): Count: 175594 Average: 43.1046 StdDev: 354.17 Min: 1 Median: 2.1937 Max: 28437 Percentiles: P50: 2.19 P75: 3.09 P99: 1523.25 P99.9: 4076.86 P99.99: 4764.13 ------------------------------------------------------ [ 0, 1 ] 1365 0.777% 0.777% ( 1, 2 ] 76297 43.451% 44.228% ######### ( 2, 3 ] 52347 29.811% 74.040% ###### ( 3, 4 ] 18220 10.376% 84.416% ## ( 4, 6 ] 18992 10.816% 95.232% ## ( 6, 10 ] 4220 2.403% 97.635% ( 10, 15 ] 860 0.490% 98.125% ( 15, 22 ] 101 0.058% 98.182% ( 22, 34 ] 11 0.006% 98.188% ( 34, 51 ] 8 0.005% 98.193% ( 51, 76 ] 9 0.005% 98.198% ( 76, 110 ] 12 0.007% 98.205% ( 110, 170 ] 14 0.008% 98.213% ( 170, 250 ] 49 0.028% 98.241% ( 250, 380 ] 62 0.035% 98.276% ( 380, 580 ] 182 0.104% 98.380% ( 580, 870 ] 336 0.191% 98.571% ( 870, 1300 ] 524 0.298% 98.870% ( 1300, 1900 ] 621 0.354% 99.223% ( 1900, 2900 ] 733 0.417% 99.641% ( 2900, 4400 ] 583 0.332% 99.973% ( 4400, 6600 ] 196 0.112% 100.084% ( 6600, 9900 ] 37 0.021% 100.105% ( 9900, 14000 ] 2 0.001% 100.106% ( 22000, 33000 ] 1 0.001% 100.107% ** Level 5 read latency histogram (micros): Count: 8943 Average: 2390.1034 StdDev: 1758.79 Min: 119 Median: 1879.1287 Max: 25698 Percentiles: P50: 1879.13 P75: 3474.50 P99: 8698.90 P99.9: 12441.05 P99.99: 23162.70 ------------------------------------------------------ ( 110, 170 ] 16 0.179% 0.179% ( 170, 250 ] 68 0.760% 0.939% ( 250, 380 ] 212 2.371% 3.310% ( 380, 580 ] 473 5.289% 8.599% # ( 580, 870 ] 908 10.153% 18.752% ## ( 870, 1300 ] 1310 14.648% 33.400% ### ( 1300, 1900 ] 1538 17.198% 50.598% ### ( 1900, 2900 ] 1548 17.310% 67.908% ### ( 2900, 4400 ] 1656 18.517% 86.425% #### ( 4400, 6600 ] 998 11.160% 97.585% ## ( 6600, 9900 ] 199 2.225% 99.810% ( 9900, 14000 ] 13 0.145% 99.955% ( 14000, 22000 ] 3 0.034% 99.989% ( 22000, 33000 ] 1 0.011% 100.000% ** DB Stats ** Uptime(secs): 7.7 total, 7.5 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 127.42 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.733 H:M:S, 48.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 130.67 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.733 H:M:S, 49.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/12/15-09:59:20 ... thread 0: (1000000,2000000) ops and (144784.9,138544.3) ops/second in (6.906796,14.435817) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 9/4 1.23 GB 94.4 0.7 0.0 0.7 2.6 1.8 0.0 1.4 41.0 143.6 18.42 8.33 24 0.767 767K 1121 0.0 0.0 L1 3/0 194.34 MB 19.4 0.9 0.7 0.2 0.9 0.7 0.0 1.4 113.2 113.0 8.32 4.38 3 2.772 955K 1334 0.0 0.0 L2 4/3 378.34 MB 0.5 0.2 0.1 0.1 0.2 0.1 0.6 3.1 116.3 116.1 1.63 0.86 1 1.630 192K 386 0.0 0.0 L3 21/10 1.62 GB 0.8 5.1 3.1 2.0 5.0 3.1 0.2 1.6 100.7 99.8 51.61 24.62 27 1.911 3010K 2800 0.0 0.0 L4 173/0 9.72 GB 1.0 6.8 3.9 2.9 6.7 3.8 1.4 1.7 119.1 117.2 58.36 32.07 37 1.577 2642K 163 0.0 0.0 L5 1148/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1358/17 107.50 GB 0.0 13.7 7.7 6.0 15.4 9.4 7.1 8.3 101.4 114.0 138.33 70.26 92 1.504 7568K 5804 0.0 0.0 Int 0/0 0.00 KB 0.0 13.7 7.7 6.0 15.4 9.4 7.1 8.3 101.4 114.0 138.33 70.26 92 1.504 7568K 5804 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 13.7 7.7 6.0 13.6 7.6 0.0 0.0 112.6 111.4 124.58 64.95 70 1.780 7568K 5804 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 137.4 13.76 5.31 22 0.625 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.6 total, 14.4 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 15.40 GB write, 1077.73 MB/s write, 13.70 GB read, 959.07 MB/s read, 138.3 seconds Interval compaction: 15.40 GB write, 1092.15 MB/s write, 13.70 GB read, 971.90 MB/s read, 138.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 79, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 79, interval: 79 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 418470 Average: 10.6034 StdDev: 137.90 Min: 1 Median: 1.9263 Max: 22313 Percentiles: P50: 1.93 P75: 2.83 P99: 9.90 P99.9: 1823.27 P99.99: 3778.30 ------------------------------------------------------ [ 0, 1 ] 11715 2.799% 2.799% # ( 1, 2 ] 213244 50.958% 53.757% ########## ( 2, 3 ] 106787 25.518% 79.276% ##### ( 3, 4 ] 35796 8.554% 87.830% ## ( 4, 6 ] 38451 9.188% 97.018% ## ( 6, 10 ] 8509 2.033% 99.052% ( 10, 15 ] 1432 0.342% 99.394% ( 15, 22 ] 184 0.044% 99.438% ( 22, 34 ] 6 0.001% 99.439% ( 34, 51 ] 17 0.004% 99.443% ( 51, 76 ] 12 0.003% 99.446% ( 76, 110 ] 2 0.000% 99.447% ( 110, 170 ] 29 0.007% 99.454% ( 170, 250 ] 103 0.025% 99.478% ( 250, 380 ] 167 0.040% 99.518% ( 380, 580 ] 304 0.073% 99.591% ( 580, 870 ] 487 0.116% 99.707% ( 870, 1300 ] 477 0.114% 99.821% ( 1300, 1900 ] 379 0.091% 99.912% ( 1900, 2900 ] 221 0.053% 99.965% ( 2900, 4400 ] 183 0.044% 100.008% ( 4400, 6600 ] 63 0.015% 100.023% ( 6600, 9900 ] 9 0.002% 100.026% ( 22000, 33000 ] 1 0.000% 100.026% ** Level 1 read latency histogram (micros): Count: 190494 Average: 11.9912 StdDev: 158.14 Min: 1 Median: 2.0948 Max: 23267 Percentiles: P50: 2.09 P75: 2.93 P99: 11.78 P99.9: 2530.76 P99.99: 4863.37 ------------------------------------------------------ [ 0, 1 ] 2128 1.117% 1.117% ( 1, 2 ] 87735 46.057% 47.174% ######### ( 2, 3 ] 56809 29.822% 76.996% ###### ( 3, 4 ] 17517 9.196% 86.191% ## ( 4, 6 ] 19416 10.192% 96.384% ## ( 6, 10 ] 4679 2.456% 98.840% ( 10, 15 ] 857 0.450% 99.290% ( 15, 22 ] 112 0.059% 99.349% ( 22, 34 ] 12 0.006% 99.355% ( 34, 51 ] 7 0.004% 99.359% ( 51, 76 ] 8 0.004% 99.363% ( 76, 110 ] 3 0.002% 99.364% ( 110, 170 ] 25 0.013% 99.377% ( 170, 250 ] 62 0.033% 99.410% ( 250, 380 ] 122 0.064% 99.474% ( 380, 580 ] 179 0.094% 99.568% ( 580, 870 ] 184 0.097% 99.665% ( 870, 1300 ] 201 0.106% 99.770% ( 1300, 1900 ] 151 0.079% 99.849% ( 1900, 2900 ] 153 0.080% 99.930% ( 2900, 4400 ] 108 0.057% 99.986% ( 4400, 6600 ] 33 0.017% 100.004% ( 6600, 9900 ] 7 0.004% 100.007% ( 22000, 33000 ] 1 0.001% 100.008% ** Level 2 read latency histogram (micros): Count: 798300 Average: 18.5120 StdDev: 214.42 Min: 1 Median: 2.1498 Max: 25106 Percentiles: P50: 2.15 P75: 3.03 P99: 12.58 P99.9: 2014.70 P99.99: 2494.64 ------------------------------------------------------ [ 0, 1 ] 10962 1.373% 1.373% ( 1, 2 ] 353505 44.282% 45.655% ######### ( 2, 3 ] 231518 29.001% 74.657% ###### ( 3, 4 ] 78399 9.821% 84.478% ## ( 4, 6 ] 89405 11.199% 95.677% ## ( 6, 10 ] 23710 2.970% 98.647% # ( 10, 15 ] 5465 0.685% 99.332% ( 15, 22 ] 832 0.104% 99.436% ( 22, 34 ] 60 0.008% 99.443% ( 34, 51 ] 28 0.004% 99.447% ( 51, 76 ] 28 0.004% 99.450% ( 76, 110 ] 11 0.001% 99.452% ( 110, 170 ] 37 0.005% 99.456% ( 170, 250 ] 89 0.011% 99.467% ( 250, 380 ] 198 0.025% 99.492% ( 380, 580 ] 338 0.042% 99.535% ( 580, 870 ] 579 0.073% 99.607% ( 870, 1300 ] 942 0.118% 99.725% ( 1300, 1900 ] 1224 0.153% 99.878% ( 1900, 2900 ] 1497 0.188% 100.066% ( 2900, 4400 ] 900 0.113% 100.179% ( 4400, 6600 ] 320 0.040% 100.219% ( 6600, 9900 ] 47 0.006% 100.225% ( 9900, 14000 ] 1 0.000% 100.225% ( 14000, 22000 ] 1 0.000% 100.225% ( 22000, 33000 ] 2 0.000% 100.225% ** Level 3 read latency histogram (micros): Count: 1615840 Average: 12.0331 StdDev: 159.83 Min: 1 Median: 2.1979 Max: 23941 Percentiles: P50: 2.20 P75: 3.03 P99: 9.81 P99.9: 847.60 P99.99: 1283.96 ------------------------------------------------------ [ 0, 1 ] 12490 0.773% 0.773% ( 1, 2 ] 696805 43.123% 43.896% ######### ( 2, 3 ] 498432 30.847% 74.743% ###### ( 3, 4 ] 162323 10.046% 84.789% ## ( 4, 6 ] 187241 11.588% 96.377% ## ( 6, 10 ] 44542 2.757% 99.133% # ( 10, 15 ] 8179 0.506% 99.639% ( 15, 22 ] 1035 0.064% 99.703% ( 22, 34 ] 98 0.006% 99.709% ( 34, 51 ] 80 0.005% 99.714% ( 51, 76 ] 128 0.008% 99.722% ( 76, 110 ] 34 0.002% 99.724% ( 110, 170 ] 130 0.008% 99.732% ( 170, 250 ] 286 0.018% 99.750% ( 250, 380 ] 453 0.028% 99.778% ( 380, 580 ] 811 0.050% 99.828% ( 580, 870 ] 1254 0.078% 99.906% ( 870, 1300 ] 1410 0.087% 99.993% ( 1300, 1900 ] 1263 0.078% 100.071% ( 1900, 2900 ] 1277 0.079% 100.150% ( 2900, 4400 ] 1173 0.073% 100.223% ( 4400, 6600 ] 381 0.024% 100.247% ( 6600, 9900 ] 46 0.003% 100.249% ( 9900, 14000 ] 2 0.000% 100.250% ( 22000, 33000 ] 1 0.000% 100.250% ** Level 4 read latency histogram (micros): Count: 759456 Average: 22.4249 StdDev: 229.75 Min: 0 Median: 2.2006 Max: 28437 Percentiles: P50: 2.20 P75: 3.09 P99: 211.78 P99.9: 2386.77 P99.99: 2961.38 ------------------------------------------------------ [ 0, 1 ] 6158 0.811% 0.811% ( 1, 2 ] 327745 43.155% 43.966% ######### ( 2, 3 ] 228435 30.079% 74.045% ###### ( 3, 4 ] 76586 10.084% 84.129% ## ( 4, 6 ] 86729 11.420% 95.549% ## ( 6, 10 ] 21346 2.811% 98.360% # ( 10, 15 ] 3878 0.511% 98.870% ( 15, 22 ] 490 0.065% 98.935% ( 22, 34 ] 31 0.004% 98.939% ( 34, 51 ] 43 0.006% 98.945% ( 51, 76 ] 64 0.008% 98.953% ( 76, 110 ] 29 0.004% 98.957% ( 110, 170 ] 141 0.019% 98.975% ( 170, 250 ] 357 0.047% 99.022% ( 250, 380 ] 520 0.068% 99.091% ( 380, 580 ] 969 0.128% 99.219% ( 580, 870 ] 1554 0.205% 99.423% ( 870, 1300 ] 1706 0.225% 99.648% ( 1300, 1900 ] 1310 0.172% 99.820% ( 1900, 2900 ] 1244 0.164% 99.984% ( 2900, 4400 ] 1101 0.145% 100.129% ( 4400, 6600 ] 329 0.043% 100.172% ( 6600, 9900 ] 60 0.008% 100.180% ( 9900, 14000 ] 2 0.000% 100.181% ( 22000, 33000 ] 1 0.000% 100.181% ** Level 5 read latency histogram (micros): Count: 28801 Average: 1457.0397 StdDev: 1452.21 Min: 82 Median: 926.6502 Max: 25698 Percentiles: P50: 926.65 P75: 1860.80 P99: 6555.82 P99.9: 9766.35 P99.99: 14319.73 ------------------------------------------------------ ( 76, 110 ] 63 0.219% 0.219% ( 110, 170 ] 739 2.566% 2.785% # ( 170, 250 ] 1569 5.448% 8.232% # ( 250, 380 ] 2523 8.760% 16.992% ## ( 380, 580 ] 3842 13.340% 30.332% ### ( 580, 870 ] 5037 17.489% 47.821% ### ( 870, 1300 ] 4763 16.538% 64.359% ### ( 1300, 1900 ] 3279 11.385% 75.744% ## ( 1900, 2900 ] 2703 9.385% 85.129% ## ( 2900, 4400 ] 2677 9.295% 94.424% ## ( 4400, 6600 ] 1345 4.670% 99.094% # ( 6600, 9900 ] 242 0.840% 99.934% ( 9900, 14000 ] 16 0.056% 99.990% ( 14000, 22000 ] 3 0.010% 100.000% ( 22000, 33000 ] 1 0.003% 100.003% ** DB Stats ** Uptime(secs): 14.6 total, 6.9 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 134.54 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.793 H:M:S, 46.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 142.50 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.060 H:M:S, 44.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 4 num-running-flushes: 1 2024/12/15-09:59:24 ... thread 0: (1000000,3000000) ops and (241541.8,161499.7) ops/second in (4.140071,18.575888) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/11 2.15 GB 44.1 1.6 0.0 1.6 4.4 2.8 0.0 1.6 63.5 173.3 25.79 14.39 41 0.629 1662K 2076 0.0 0.0 L1 3/3 194.34 MB 0.0 0.9 0.7 0.2 0.9 0.7 0.0 1.4 113.2 113.0 8.32 4.38 3 2.772 955K 1334 0.0 0.0 L2 1/0 53.76 MB 0.5 0.2 0.1 0.1 0.2 0.1 0.6 3.1 116.3 116.1 1.63 0.86 1 1.630 192K 386 0.0 0.0 L3 15/0 908.59 MB 0.9 6.3 3.4 2.8 6.2 3.4 0.2 1.8 109.6 108.8 58.53 30.25 30 1.951 3683K 2800 0.0 0.0 L4 173/0 9.68 GB 1.0 8.9 4.7 4.2 8.7 4.5 1.6 1.9 128.0 125.9 70.85 41.71 48 1.476 4337K 19K 0.0 0.0 L5 1164/0 95.41 GB 1.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1370/14 108.37 GB 0.0 17.8 8.8 9.0 20.4 11.4 8.4 7.4 110.5 126.5 165.12 91.58 123 1.342 10M 25K 0.0 0.0 Int 0/0 0.00 KB 0.0 17.8 8.8 9.0 20.4 11.4 8.4 7.4 110.5 126.5 165.12 91.58 123 1.342 10M 25K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 17.8 8.8 9.0 17.6 8.6 0.0 0.0 123.7 122.3 147.58 83.64 87 1.696 10M 25K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 161.6 17.54 7.94 36 0.487 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.8 total, 18.6 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 20.40 GB write, 1112.63 MB/s write, 17.82 GB read, 972.32 MB/s read, 165.1 seconds Interval compaction: 20.40 GB write, 1124.20 MB/s write, 17.82 GB read, 982.43 MB/s read, 165.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 85, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 85, interval: 85 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 753964 Average: 8.7491 StdDev: 112.98 Min: 1 Median: 1.8530 Max: 22313 Percentiles: P50: 1.85 P75: 2.70 P99: 9.89 P99.9: 1187.25 P99.99: 2199.61 ------------------------------------------------------ [ 0, 1 ] 30344 4.025% 4.025% # ( 1, 2 ] 406388 53.900% 57.925% ########### ( 2, 3 ] 182756 24.239% 82.164% ##### ( 3, 4 ] 56367 7.476% 89.640% # ( 4, 6 ] 58328 7.736% 97.376% ## ( 6, 10 ] 12597 1.671% 99.047% ( 10, 15 ] 2139 0.284% 99.331% ( 15, 22 ] 258 0.034% 99.365% ( 22, 34 ] 12 0.002% 99.367% ( 34, 51 ] 21 0.003% 99.369% ( 51, 76 ] 16 0.002% 99.372% ( 76, 110 ] 46 0.006% 99.378% ( 110, 170 ] 479 0.064% 99.441% ( 170, 250 ] 711 0.094% 99.536% ( 250, 380 ] 648 0.086% 99.621% ( 380, 580 ] 787 0.104% 99.726% ( 580, 870 ] 834 0.111% 99.836% ( 870, 1300 ] 652 0.086% 99.923% ( 1300, 1900 ] 433 0.057% 99.980% ( 1900, 2900 ] 249 0.033% 100.013% ( 2900, 4400 ] 208 0.028% 100.041% ( 4400, 6600 ] 69 0.009% 100.050% ( 6600, 9900 ] 11 0.001% 100.052% ( 9900, 14000 ] 2 0.000% 100.052% ( 22000, 33000 ] 1 0.000% 100.052% ** Level 1 read latency histogram (micros): Count: 227140 Average: 11.6029 StdDev: 151.84 Min: 1 Median: 2.0772 Max: 23267 Percentiles: P50: 2.08 P75: 2.91 P99: 12.41 P99.9: 2411.63 P99.99: 4732.26 ------------------------------------------------------ [ 0, 1 ] 2695 1.186% 1.186% ( 1, 2 ] 105628 46.503% 47.690% ######### ( 2, 3 ] 67946 29.914% 77.604% ###### ( 3, 4 ] 20174 8.882% 86.485% ## ( 4, 6 ] 22398 9.861% 96.346% ## ( 6, 10 ] 5558 2.447% 98.793% ( 10, 15 ] 974 0.429% 99.222% ( 15, 22 ] 118 0.052% 99.274% ( 22, 34 ] 14 0.006% 99.280% ( 34, 51 ] 11 0.005% 99.285% ( 51, 76 ] 11 0.005% 99.290% ( 76, 110 ] 4 0.002% 99.292% ( 110, 170 ] 85 0.037% 99.329% ( 170, 250 ] 159 0.070% 99.399% ( 250, 380 ] 190 0.084% 99.483% ( 380, 580 ] 250 0.110% 99.593% ( 580, 870 ] 227 0.100% 99.693% ( 870, 1300 ] 226 0.099% 99.792% ( 1300, 1900 ] 163 0.072% 99.864% ( 1900, 2900 ] 160 0.070% 99.934% ( 2900, 4400 ] 121 0.053% 99.988% ( 4400, 6600 ] 35 0.015% 100.003% ( 6600, 9900 ] 9 0.004% 100.007% ( 22000, 33000 ] 1 0.000% 100.007% ** Level 2 read latency histogram (micros): Count: 798306 Average: 18.5143 StdDev: 214.42 Min: 1 Median: 2.1498 Max: 25106 Percentiles: P50: 2.15 P75: 3.04 P99: 12.58 P99.9: 2014.69 P99.99: 2494.64 ------------------------------------------------------ [ 0, 1 ] 10962 1.373% 1.373% ( 1, 2 ] 353505 44.282% 45.655% ######### ( 2, 3 ] 231518 29.001% 74.656% ###### ( 3, 4 ] 78399 9.821% 84.477% ## ( 4, 6 ] 89405 11.199% 95.676% ## ( 6, 10 ] 23710 2.970% 98.646% # ( 10, 15 ] 5465 0.685% 99.331% ( 15, 22 ] 832 0.104% 99.435% ( 22, 34 ] 60 0.008% 99.443% ( 34, 51 ] 28 0.004% 99.446% ( 51, 76 ] 28 0.004% 99.450% ( 76, 110 ] 11 0.001% 99.451% ( 110, 170 ] 37 0.005% 99.456% ( 170, 250 ] 92 0.012% 99.467% ( 250, 380 ] 200 0.025% 99.492% ( 380, 580 ] 338 0.042% 99.535% ( 580, 870 ] 580 0.073% 99.607% ( 870, 1300 ] 942 0.118% 99.725% ( 1300, 1900 ] 1224 0.153% 99.878% ( 1900, 2900 ] 1497 0.188% 100.066% ( 2900, 4400 ] 900 0.113% 100.179% ( 4400, 6600 ] 320 0.040% 100.219% ( 6600, 9900 ] 47 0.006% 100.225% ( 9900, 14000 ] 1 0.000% 100.225% ( 14000, 22000 ] 1 0.000% 100.225% ( 22000, 33000 ] 2 0.000% 100.225% ** Level 3 read latency histogram (micros): Count: 1879040 Average: 11.2306 StdDev: 151.61 Min: 1 Median: 2.1789 Max: 23941 Percentiles: P50: 2.18 P75: 2.99 P99: 9.76 P99.9: 752.22 P99.99: 1183.60 ------------------------------------------------------ [ 0, 1 ] 15345 0.817% 0.817% ( 1, 2 ] 820897 43.687% 44.504% ######### ( 2, 3 ] 577409 30.729% 75.233% ###### ( 3, 4 ] 185979 9.898% 85.130% ## ( 4, 6 ] 213393 11.356% 96.487% ## ( 6, 10 ] 50179 2.670% 99.157% # ( 10, 15 ] 9029 0.481% 99.638% ( 15, 22 ] 1137 0.061% 99.698% ( 22, 34 ] 109 0.006% 99.704% ( 34, 51 ] 114 0.006% 99.710% ( 51, 76 ] 165 0.009% 99.719% ( 76, 110 ] 52 0.003% 99.722% ( 110, 170 ] 275 0.015% 99.736% ( 170, 250 ] 507 0.027% 99.763% ( 250, 380 ] 671 0.036% 99.799% ( 380, 580 ] 1046 0.056% 99.855% ( 580, 870 ] 1438 0.077% 99.931% ( 870, 1300 ] 1518 0.081% 100.012% ( 1300, 1900 ] 1315 0.070% 100.082% ( 1900, 2900 ] 1315 0.070% 100.152% ( 2900, 4400 ] 1209 0.064% 100.216% ( 4400, 6600 ] 386 0.021% 100.237% ( 6600, 9900 ] 48 0.003% 100.239% ( 9900, 14000 ] 6 0.000% 100.240% ( 22000, 33000 ] 1 0.000% 100.240% ** Level 4 read latency histogram (micros): Count: 1092039 Average: 19.5720 StdDev: 203.40 Min: 0 Median: 2.1639 Max: 28437 Percentiles: P50: 2.16 P75: 3.00 P99: 230.25 P99.9: 1858.55 P99.99: 2537.80 ------------------------------------------------------ [ 0, 1 ] 9340 0.855% 0.855% ( 1, 2 ] 483157 44.244% 45.099% ######### ( 2, 3 ] 326498 29.898% 74.997% ###### ( 3, 4 ] 106035 9.710% 84.707% ## ( 4, 6 ] 118278 10.831% 95.538% ## ( 6, 10 ] 29990 2.746% 98.284% # ( 10, 15 ] 5072 0.464% 98.748% ( 15, 22 ] 618 0.057% 98.805% ( 22, 34 ] 44 0.004% 98.809% ( 34, 51 ] 81 0.007% 98.816% ( 51, 76 ] 111 0.010% 98.827% ( 76, 110 ] 103 0.009% 98.836% ( 110, 170 ] 805 0.074% 98.910% ( 170, 250 ] 1310 0.120% 99.030% ( 250, 380 ] 1535 0.141% 99.170% ( 380, 580 ] 1987 0.182% 99.352% ( 580, 870 ] 2440 0.223% 99.576% ( 870, 1300 ] 2168 0.199% 99.774% ( 1300, 1900 ] 1477 0.135% 99.909% ( 1900, 2900 ] 1381 0.126% 100.036% ( 2900, 4400 ] 1213 0.111% 100.147% ( 4400, 6600 ] 355 0.033% 100.179% ( 6600, 9900 ] 64 0.006% 100.185% ( 9900, 14000 ] 4 0.000% 100.186% ( 22000, 33000 ] 1 0.000% 100.186% ** Level 5 read latency histogram (micros): Count: 53274 Average: 1011.6532 StdDev: 1230.01 Min: 82 Median: 565.7058 Max: 25698 Percentiles: P50: 565.71 P75: 1172.91 P99: 6186.25 P99.9: 9432.51 P99.99: 13147.35 ------------------------------------------------------ ( 76, 110 ] 476 0.893% 0.893% ( 110, 170 ] 4457 8.366% 9.260% ## ( 170, 250 ] 6624 12.434% 21.694% ## ( 250, 380 ] 7259 13.626% 35.319% ### ( 380, 580 ] 8423 15.811% 51.130% ### ( 580, 870 ] 8220 15.430% 66.560% ### ( 870, 1300 ] 6383 11.981% 78.541% ## ( 1300, 1900 ] 3846 7.219% 85.760% # ( 1900, 2900 ] 3021 5.671% 91.431% # ( 2900, 4400 ] 2885 5.415% 96.846% # ( 4400, 6600 ] 1413 2.652% 99.499% # ( 6600, 9900 ] 249 0.467% 99.966% ( 9900, 14000 ] 16 0.030% 99.996% ( 14000, 22000 ] 3 0.006% 100.002% ( 22000, 33000 ] 1 0.002% 100.004% ** DB Stats ** Uptime(secs): 18.8 total, 4.1 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 157.29 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.135 H:M:S, 38.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 237.73 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.342 H:M:S, 8.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/15-09:59:27 ... thread 0: (1000000,4000000) ops and (292052.2,181818.7) ops/second in (3.424045,21.999933) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/11 3.13 GB 144.8 2.6 0.0 2.6 6.3 3.7 0.0 1.7 80.0 196.0 33.09 21.07 61 0.542 2686K 2988 0.0 0.0 L1 3/3 194.34 MB 0.0 0.9 0.7 0.2 0.9 0.7 0.0 1.4 113.2 113.0 8.32 4.38 3 2.772 955K 1334 0.0 0.0 L2 1/0 53.76 MB 0.5 0.2 0.1 0.1 0.2 0.1 0.6 3.1 116.3 116.1 1.63 0.86 1 1.630 192K 386 0.0 0.0 L3 15/0 908.59 MB 0.9 6.3 3.4 2.8 6.2 3.4 0.2 1.8 109.6 108.8 58.53 30.25 30 1.951 3683K 2800 0.0 0.0 L4 173/0 9.68 GB 1.0 8.9 4.7 4.2 8.7 4.5 1.6 1.9 128.0 125.9 70.85 41.71 48 1.476 4337K 19K 0.0 0.0 L5 1164/0 95.41 GB 1.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1374/14 109.36 GB 0.0 18.8 8.8 10.0 22.4 12.4 8.4 6.0 111.7 132.8 172.42 98.27 143 1.206 11M 26K 0.0 0.0 Int 0/0 0.00 KB 0.0 18.8 8.8 10.0 22.4 12.4 8.4 6.0 111.7 132.8 172.42 98.27 143 1.206 11M 26K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 18.8 8.8 10.0 18.6 8.6 0.0 0.0 127.1 125.7 151.55 87.49 91 1.665 11M 26K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 184.1 20.87 10.77 52 0.401 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 22.2 total, 22.0 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 22.36 GB write, 1031.77 MB/s write, 18.81 GB read, 867.73 MB/s read, 172.4 seconds Interval compaction: 22.36 GB write, 1040.83 MB/s write, 18.81 GB read, 875.35 MB/s read, 172.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 85, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 85, interval: 85 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1050048 Average: 10.3266 StdDev: 101.78 Min: 1 Median: 1.7484 Max: 22313 Percentiles: P50: 1.75 P75: 2.54 P99: 203.87 P99.9: 901.41 P99.99: 1672.27 ------------------------------------------------------ [ 0, 1 ] 87282 8.312% 8.312% ## ( 1, 2 ] 584896 55.702% 64.014% ########### ( 2, 3 ] 213432 20.326% 84.340% #### ( 3, 4 ] 64200 6.114% 90.454% # ( 4, 6 ] 62867 5.987% 96.441% # ( 6, 10 ] 13797 1.314% 97.755% ( 10, 15 ] 2541 0.242% 97.997% ( 15, 22 ] 309 0.029% 98.026% ( 22, 34 ] 22 0.002% 98.028% ( 34, 51 ] 24 0.002% 98.031% ( 51, 76 ] 19 0.002% 98.033% ( 76, 110 ] 747 0.071% 98.104% ( 110, 170 ] 6840 0.651% 98.755% ( 170, 250 ] 6077 0.579% 99.334% ( 250, 380 ] 2572 0.245% 99.579% ( 380, 580 ] 2057 0.196% 99.775% ( 580, 870 ] 1264 0.120% 99.895% ( 870, 1300 ] 725 0.069% 99.964% ( 1300, 1900 ] 440 0.042% 100.006% ( 1900, 2900 ] 261 0.025% 100.031% ( 2900, 4400 ] 220 0.021% 100.052% ( 4400, 6600 ] 69 0.007% 100.058% ( 6600, 9900 ] 11 0.001% 100.059% ( 9900, 14000 ] 2 0.000% 100.060% ( 22000, 33000 ] 1 0.000% 100.060% ** Level 1 read latency histogram (micros): Count: 227566 Average: 12.0011 StdDev: 152.11 Min: 1 Median: 2.0804 Max: 23267 Percentiles: P50: 2.08 P75: 2.92 P99: 14.58 P99.9: 2408.96 P99.99: 4729.59 ------------------------------------------------------ [ 0, 1 ] 2695 1.184% 1.184% ( 1, 2 ] 105628 46.416% 47.601% ######### ( 2, 3 ] 67946 29.858% 77.458% ###### ( 3, 4 ] 20174 8.865% 86.324% ## ( 4, 6 ] 22398 9.842% 96.166% ## ( 6, 10 ] 5558 2.442% 98.608% ( 10, 15 ] 974 0.428% 99.036% ( 15, 22 ] 118 0.052% 99.088% ( 22, 34 ] 14 0.006% 99.094% ( 34, 51 ] 11 0.005% 99.099% ( 51, 76 ] 11 0.005% 99.104% ( 76, 110 ] 25 0.011% 99.115% ( 110, 170 ] 248 0.109% 99.224% ( 170, 250 ] 302 0.133% 99.357% ( 250, 380 ] 243 0.107% 99.463% ( 380, 580 ] 283 0.124% 99.588% ( 580, 870 ] 238 0.105% 99.692% ( 870, 1300 ] 227 0.100% 99.792% ( 1300, 1900 ] 164 0.072% 99.864% ( 1900, 2900 ] 160 0.070% 99.935% ( 2900, 4400 ] 121 0.053% 99.988% ( 4400, 6600 ] 35 0.015% 100.003% ( 6600, 9900 ] 9 0.004% 100.007% ( 22000, 33000 ] 1 0.000% 100.007% ** Level 2 read latency histogram (micros): Count: 798306 Average: 18.5143 StdDev: 214.42 Min: 1 Median: 2.1498 Max: 25106 Percentiles: P50: 2.15 P75: 3.04 P99: 12.58 P99.9: 2014.69 P99.99: 2494.64 ------------------------------------------------------ [ 0, 1 ] 10962 1.373% 1.373% ( 1, 2 ] 353505 44.282% 45.655% ######### ( 2, 3 ] 231518 29.001% 74.656% ###### ( 3, 4 ] 78399 9.821% 84.477% ## ( 4, 6 ] 89405 11.199% 95.676% ## ( 6, 10 ] 23710 2.970% 98.646% # ( 10, 15 ] 5465 0.685% 99.331% ( 15, 22 ] 832 0.104% 99.435% ( 22, 34 ] 60 0.008% 99.443% ( 34, 51 ] 28 0.004% 99.446% ( 51, 76 ] 28 0.004% 99.450% ( 76, 110 ] 11 0.001% 99.451% ( 110, 170 ] 37 0.005% 99.456% ( 170, 250 ] 92 0.012% 99.467% ( 250, 380 ] 200 0.025% 99.492% ( 380, 580 ] 338 0.042% 99.535% ( 580, 870 ] 580 0.073% 99.607% ( 870, 1300 ] 942 0.118% 99.725% ( 1300, 1900 ] 1224 0.153% 99.878% ( 1900, 2900 ] 1497 0.188% 100.066% ( 2900, 4400 ] 900 0.113% 100.179% ( 4400, 6600 ] 320 0.040% 100.219% ( 6600, 9900 ] 47 0.006% 100.225% ( 9900, 14000 ] 1 0.000% 100.225% ( 14000, 22000 ] 1 0.000% 100.225% ( 22000, 33000 ] 2 0.000% 100.225% ** Level 3 read latency histogram (micros): Count: 1880689 Average: 11.4171 StdDev: 151.75 Min: 1 Median: 2.1803 Max: 23941 Percentiles: P50: 2.18 P75: 2.99 P99: 9.89 P99.9: 757.85 P99.99: 1184.81 ------------------------------------------------------ [ 0, 1 ] 15346 0.816% 0.816% ( 1, 2 ] 820901 43.649% 44.465% ######### ( 2, 3 ] 577418 30.702% 75.167% ###### ( 3, 4 ] 185984 9.889% 85.057% ## ( 4, 6 ] 213395 11.347% 96.403% ## ( 6, 10 ] 50179 2.668% 99.071% # ( 10, 15 ] 9029 0.480% 99.551% ( 15, 22 ] 1137 0.060% 99.612% ( 22, 34 ] 109 0.006% 99.618% ( 34, 51 ] 114 0.006% 99.624% ( 51, 76 ] 165 0.009% 99.632% ( 76, 110 ] 118 0.006% 99.639% ( 110, 170 ] 917 0.049% 99.688% ( 170, 250 ] 1053 0.056% 99.743% ( 250, 380 ] 871 0.046% 99.790% ( 380, 580 ] 1161 0.062% 99.852% ( 580, 870 ] 1486 0.079% 99.931% ( 870, 1300 ] 1527 0.081% 100.012% ( 1300, 1900 ] 1316 0.070% 100.082% ( 1900, 2900 ] 1316 0.070% 100.152% ( 2900, 4400 ] 1209 0.064% 100.216% ( 4400, 6600 ] 386 0.021% 100.237% ( 6600, 9900 ] 48 0.003% 100.239% ( 9900, 14000 ] 6 0.000% 100.239% ( 22000, 33000 ] 1 0.000% 100.239% ** Level 4 read latency histogram (micros): Count: 1098682 Average: 20.6620 StdDev: 203.61 Min: 0 Median: 2.1737 Max: 28437 Percentiles: P50: 2.17 P75: 3.04 P99: 304.45 P99.9: 1856.75 P99.99: 2537.84 ------------------------------------------------------ [ 0, 1 ] 9340 0.850% 0.850% ( 1, 2 ] 483216 43.981% 44.832% ######### ( 2, 3 ] 326829 29.747% 74.579% ###### ( 3, 4 ] 106178 9.664% 84.243% ## ( 4, 6 ] 118411 10.778% 95.021% ## ( 6, 10 ] 30007 2.731% 97.752% # ( 10, 15 ] 5076 0.462% 98.214% ( 15, 22 ] 618 0.056% 98.270% ( 22, 34 ] 44 0.004% 98.274% ( 34, 51 ] 81 0.007% 98.281% ( 51, 76 ] 111 0.010% 98.291% ( 76, 110 ] 354 0.032% 98.324% ( 110, 170 ] 3260 0.297% 98.620% ( 170, 250 ] 3250 0.296% 98.916% ( 250, 380 ] 2197 0.200% 99.116% ( 380, 580 ] 2457 0.224% 99.340% ( 580, 870 ] 2593 0.236% 99.576% ( 870, 1300 ] 2188 0.199% 99.775% ( 1300, 1900 ] 1480 0.135% 99.910% ( 1900, 2900 ] 1383 0.126% 100.036% ( 2900, 4400 ] 1213 0.110% 100.146% ( 4400, 6600 ] 355 0.032% 100.178% ( 6600, 9900 ] 64 0.006% 100.184% ( 9900, 14000 ] 4 0.000% 100.184% ( 22000, 33000 ] 1 0.000% 100.185% ** Level 5 read latency histogram (micros): Count: 92071 Average: 679.4591 StdDev: 1018.09 Min: 82 Median: 297.6461 Max: 25698 Percentiles: P50: 297.65 P75: 713.40 P99: 5580.64 P99.9: 8905.08 P99.99: 11896.93 ------------------------------------------------------ ( 76, 110 ] 2126 2.309% 2.309% ( 110, 170 ] 20490 22.255% 24.564% #### ( 170, 250 ] 19168 20.819% 45.382% #### ( 250, 380 ] 11600 12.599% 57.981% ### ( 380, 580 ] 11430 12.414% 70.396% ## ( 580, 870 ] 9216 10.010% 80.405% ## ( 870, 1300 ] 6575 7.141% 87.547% # ( 1300, 1900 ] 3860 4.192% 91.739% # ( 1900, 2900 ] 3034 3.295% 95.034% # ( 2900, 4400 ] 2893 3.142% 98.176% # ( 4400, 6600 ] 1413 1.535% 99.711% ( 6600, 9900 ] 249 0.270% 99.982% ( 9900, 14000 ] 16 0.017% 99.999% ( 14000, 22000 ] 3 0.003% 100.002% ( 22000, 33000 ] 1 0.001% 100.003% ** DB Stats ** Uptime(secs): 22.2 total, 3.4 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 177.37 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.135 H:M:S, 32.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 287.44 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/15-09:59:33 ... thread 0: (1000000,5000000) ops and (173862.0,180169.6) ops/second in (5.751689,27.751622) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/14 4.12 GB 113.3 3.8 0.0 3.8 8.5 4.7 0.0 1.8 95.0 212.8 41.12 28.51 82 0.501 3965K 4259 0.0 0.0 L1 3/3 194.34 MB 0.0 0.9 0.7 0.2 0.9 0.7 0.0 1.4 113.2 113.0 8.32 4.38 3 2.772 955K 1334 0.0 0.0 L2 1/0 53.76 MB 0.5 0.2 0.1 0.1 0.2 0.1 0.6 3.1 116.3 116.1 1.63 0.86 1 1.630 192K 386 0.0 0.0 L3 15/0 908.59 MB 0.9 6.3 3.4 2.8 6.2 3.4 0.2 1.8 109.6 108.8 58.53 30.25 30 1.951 3683K 2800 0.0 0.0 L4 173/0 9.68 GB 1.0 8.9 4.7 4.2 8.7 4.5 1.6 1.9 128.0 125.9 70.85 41.71 48 1.476 4337K 19K 0.0 0.0 L5 1164/0 95.41 GB 1.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1375/17 110.34 GB 0.0 20.0 8.8 11.2 24.6 13.4 8.4 5.2 113.7 139.5 180.45 105.70 164 1.100 13M 27K 0.0 0.0 Int 0/0 0.00 KB 0.0 20.0 8.8 11.2 24.6 13.4 8.4 5.2 113.7 139.5 180.45 105.70 164 1.100 13M 27K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 20.0 8.8 11.2 19.8 8.6 0.0 0.0 131.3 130.0 156.32 92.17 96 1.628 13M 27K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 201.0 24.13 13.53 68 0.355 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 27.9 total, 27.8 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 24.58 GB write, 900.53 MB/s write, 20.04 GB read, 734.23 MB/s read, 180.4 seconds Interval compaction: 24.58 GB write, 906.79 MB/s write, 20.04 GB read, 739.34 MB/s read, 180.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 10, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 10, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 85, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 10, total-stops: 85, interval: 95 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1409337 Average: 12.9812 StdDev: 94.89 Min: 1 Median: 1.6791 Max: 22313 Percentiles: P50: 1.68 P75: 2.42 P99: 238.62 P99.9: 795.57 P99.99: 1354.88 ------------------------------------------------------ [ 0, 1 ] 170112 12.070% 12.070% ## ( 1, 2 ] 787212 55.857% 67.927% ########### ( 2, 3 ] 238948 16.955% 84.882% ### ( 3, 4 ] 72059 5.113% 89.995% # ( 4, 6 ] 67999 4.825% 94.820% # ( 6, 10 ] 14722 1.045% 95.864% ( 10, 15 ] 2864 0.203% 96.068% ( 15, 22 ] 343 0.024% 96.092% ( 22, 34 ] 23 0.002% 96.094% ( 34, 51 ] 25 0.002% 96.095% ( 51, 76 ] 21 0.001% 96.097% ( 76, 110 ] 2319 0.165% 96.261% ( 110, 170 ] 23736 1.684% 97.946% ( 170, 250 ] 17325 1.229% 99.175% ( 250, 380 ] 5241 0.372% 99.547% ( 380, 580 ] 3712 0.263% 99.810% ( 580, 870 ] 1704 0.121% 99.931% ( 870, 1300 ] 790 0.056% 99.987% ( 1300, 1900 ] 449 0.032% 100.019% ( 1900, 2900 ] 283 0.020% 100.039% ( 2900, 4400 ] 230 0.016% 100.055% ( 4400, 6600 ] 69 0.005% 100.060% ( 6600, 9900 ] 11 0.001% 100.061% ( 9900, 14000 ] 2 0.000% 100.061% ( 22000, 33000 ] 1 0.000% 100.061% ** Level 1 read latency histogram (micros): Count: 228304 Average: 12.5878 StdDev: 152.33 Min: 1 Median: 2.0858 Max: 23267 Percentiles: P50: 2.09 P75: 2.93 P99: 151.71 P99.9: 2404.35 P99.99: 4724.95 ------------------------------------------------------ [ 0, 1 ] 2695 1.180% 1.180% ( 1, 2 ] 105628 46.266% 47.447% ######### ( 2, 3 ] 67946 29.761% 77.208% ###### ( 3, 4 ] 20174 8.836% 86.044% ## ( 4, 6 ] 22398 9.811% 95.855% ## ( 6, 10 ] 5558 2.434% 98.290% ( 10, 15 ] 974 0.427% 98.716% ( 15, 22 ] 118 0.052% 98.768% ( 22, 34 ] 14 0.006% 98.774% ( 34, 51 ] 11 0.005% 98.779% ( 51, 76 ] 11 0.005% 98.784% ( 76, 110 ] 65 0.028% 98.812% ( 110, 170 ] 617 0.270% 99.082% ( 170, 250 ] 526 0.230% 99.313% ( 250, 380 ] 300 0.131% 99.444% ( 380, 580 ] 316 0.138% 99.583% ( 580, 870 ] 252 0.110% 99.693% ( 870, 1300 ] 228 0.100% 99.793% ( 1300, 1900 ] 164 0.072% 99.865% ( 1900, 2900 ] 160 0.070% 99.935% ( 2900, 4400 ] 121 0.053% 99.988% ( 4400, 6600 ] 35 0.015% 100.003% ( 6600, 9900 ] 9 0.004% 100.007% ( 22000, 33000 ] 1 0.000% 100.007% ** Level 2 read latency histogram (micros): Count: 798306 Average: 18.5143 StdDev: 214.42 Min: 1 Median: 2.1498 Max: 25106 Percentiles: P50: 2.15 P75: 3.04 P99: 12.58 P99.9: 2014.69 P99.99: 2494.64 ------------------------------------------------------ [ 0, 1 ] 10962 1.373% 1.373% ( 1, 2 ] 353505 44.282% 45.655% ######### ( 2, 3 ] 231518 29.001% 74.656% ###### ( 3, 4 ] 78399 9.821% 84.477% ## ( 4, 6 ] 89405 11.199% 95.676% ## ( 6, 10 ] 23710 2.970% 98.646% # ( 10, 15 ] 5465 0.685% 99.331% ( 15, 22 ] 832 0.104% 99.435% ( 22, 34 ] 60 0.008% 99.443% ( 34, 51 ] 28 0.004% 99.446% ( 51, 76 ] 28 0.004% 99.450% ( 76, 110 ] 11 0.001% 99.451% ( 110, 170 ] 37 0.005% 99.456% ( 170, 250 ] 92 0.012% 99.467% ( 250, 380 ] 200 0.025% 99.492% ( 380, 580 ] 338 0.042% 99.535% ( 580, 870 ] 580 0.073% 99.607% ( 870, 1300 ] 942 0.118% 99.725% ( 1300, 1900 ] 1224 0.153% 99.878% ( 1900, 2900 ] 1497 0.188% 100.066% ( 2900, 4400 ] 900 0.113% 100.179% ( 4400, 6600 ] 320 0.040% 100.219% ( 6600, 9900 ] 47 0.006% 100.225% ( 9900, 14000 ] 1 0.000% 100.225% ( 14000, 22000 ] 1 0.000% 100.225% ( 22000, 33000 ] 2 0.000% 100.225% ** Level 3 read latency histogram (micros): Count: 1883718 Average: 11.7110 StdDev: 151.87 Min: 1 Median: 2.1829 Max: 23941 Percentiles: P50: 2.18 P75: 3.00 P99: 10.92 P99.9: 760.99 P99.99: 1185.38 ------------------------------------------------------ [ 0, 1 ] 15346 0.815% 0.815% ( 1, 2 ] 820901 43.579% 44.393% ######### ( 2, 3 ] 577418 30.653% 75.047% ###### ( 3, 4 ] 185984 9.873% 84.920% ## ( 4, 6 ] 213395 11.328% 96.248% ## ( 6, 10 ] 50179 2.664% 98.912% # ( 10, 15 ] 9029 0.479% 99.391% ( 15, 22 ] 1137 0.060% 99.452% ( 22, 34 ] 109 0.006% 99.457% ( 34, 51 ] 114 0.006% 99.464% ( 51, 76 ] 165 0.009% 99.472% ( 76, 110 ] 260 0.014% 99.486% ( 110, 170 ] 2454 0.130% 99.616% ( 170, 250 ] 1989 0.106% 99.722% ( 250, 380 ] 1103 0.059% 99.780% ( 380, 580 ] 1302 0.069% 99.850% ( 580, 870 ] 1521 0.081% 99.930% ( 870, 1300 ] 1532 0.081% 100.012% ( 1300, 1900 ] 1316 0.070% 100.082% ( 1900, 2900 ] 1316 0.070% 100.151% ( 2900, 4400 ] 1210 0.064% 100.216% ( 4400, 6600 ] 386 0.020% 100.236% ( 6600, 9900 ] 48 0.003% 100.239% ( 9900, 14000 ] 6 0.000% 100.239% ( 22000, 33000 ] 1 0.000% 100.239% ** Level 4 read latency histogram (micros): Count: 1111270 Average: 22.5525 StdDev: 203.56 Min: 0 Median: 2.1929 Max: 28437 Percentiles: P50: 2.19 P75: 3.13 P99: 352.01 P99.9: 1853.67 P99.99: 2538.24 ------------------------------------------------------ [ 0, 1 ] 9340 0.840% 0.840% ( 1, 2 ] 483240 43.485% 44.326% ######### ( 2, 3 ] 326955 29.422% 73.748% ###### ( 3, 4 ] 106252 9.561% 83.309% ## ( 4, 6 ] 118524 10.666% 93.975% ## ( 6, 10 ] 30027 2.702% 96.677% # ( 10, 15 ] 5085 0.458% 97.134% ( 15, 22 ] 619 0.056% 97.190% ( 22, 34 ] 44 0.004% 97.194% ( 34, 51 ] 81 0.007% 97.201% ( 51, 76 ] 111 0.010% 97.211% ( 76, 110 ] 937 0.084% 97.295% ( 110, 170 ] 9315 0.838% 98.134% ( 170, 250 ] 7208 0.649% 98.782% ( 250, 380 ] 3083 0.277% 99.060% ( 380, 580 ] 3030 0.273% 99.332% ( 580, 870 ] 2739 0.246% 99.579% ( 870, 1300 ] 2203 0.198% 99.777% ( 1300, 1900 ] 1480 0.133% 99.910% ( 1900, 2900 ] 1388 0.125% 100.035% ( 2900, 4400 ] 1213 0.109% 100.144% ( 4400, 6600 ] 355 0.032% 100.176% ( 6600, 9900 ] 64 0.006% 100.182% ( 9900, 14000 ] 4 0.000% 100.182% ( 22000, 33000 ] 1 0.000% 100.182% ** Level 5 read latency histogram (micros): Count: 165346 Average: 464.5008 StdDev: 800.81 Min: 80 Median: 207.2860 Max: 25698 Percentiles: P50: 207.29 P75: 405.69 P99: 4441.32 P99.9: 7947.22 P99.99: 10275.51 ------------------------------------------------------ ( 76, 110 ] 5838 3.531% 3.531% # ( 110, 170 ] 57107 34.538% 38.069% ####### ( 170, 250 ] 42328 25.600% 63.668% ##### ( 250, 380 ] 16836 10.182% 73.851% ## ( 380, 580 ] 14795 8.948% 82.798% ## ( 580, 870 ] 10219 6.180% 88.979% # ( 870, 1300 ] 6705 4.055% 93.034% # ( 1300, 1900 ] 3880 2.347% 95.381% ( 1900, 2900 ] 3060 1.851% 97.231% ( 2900, 4400 ] 2898 1.753% 98.984% ( 4400, 6600 ] 1413 0.855% 99.839% ( 6600, 9900 ] 249 0.151% 99.989% ( 9900, 14000 ] 16 0.010% 99.999% ( 14000, 22000 ] 3 0.002% 100.001% ( 22000, 33000 ] 1 0.001% 100.001% ** DB Stats ** Uptime(secs): 27.9 total, 5.8 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 176.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:9.295 H:M:S, 33.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 171.11 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.160 H:M:S, 37.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 307.031 micros/op 13016 ops/sec 60.156 seconds 782996 operations; 12.6 MB/s (193999 of 193999 found) Microseconds per read: Count: 782996 Average: 307.0349 StdDev: 958.29 Min: 2 Median: 208.0407 Max: 92677 Percentiles: P50: 208.04 P75: 263.08 P99: 1804.89 P99.9: 17677.28 P99.99: 33212.62 ------------------------------------------------------ ( 1, 2 ] 6 0.001% 0.001% ( 2, 3 ] 23 0.003% 0.004% ( 3, 4 ] 41 0.005% 0.009% ( 4, 6 ] 79 0.010% 0.019% ( 6, 10 ] 355 0.045% 0.064% ( 10, 15 ] 2406 0.307% 0.372% ( 15, 22 ] 1836 0.234% 0.606% ( 22, 34 ] 5600 0.715% 1.321% ( 34, 51 ] 1486 0.190% 1.511% ( 51, 76 ] 181 0.023% 1.534% ( 76, 110 ] 2373 0.303% 1.837% ( 110, 170 ] 209351 26.737% 28.574% ##### ( 170, 250 ] 352803 45.058% 73.633% ######### ( 250, 380 ] 106415 13.591% 87.223% ### ( 380, 580 ] 56676 7.238% 94.462% # ( 580, 870 ] 22098 2.822% 97.284% # ( 870, 1300 ] 9785 1.250% 98.534% ( 1300, 1900 ] 4340 0.554% 99.088% ( 1900, 2900 ] 2802 0.358% 99.446% ( 2900, 4400 ] 2129 0.272% 99.718% ( 4400, 6600 ] 767 0.098% 99.816% ( 6600, 9900 ] 152 0.019% 99.835% ( 9900, 14000 ] 190 0.024% 99.859% ( 14000, 22000 ] 694 0.089% 99.948% ( 22000, 33000 ] 329 0.042% 99.990% ( 33000, 50000 ] 56 0.007% 99.997% ( 50000, 75000 ] 20 0.003% 100.000% ( 75000, 110000 ] 3 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 13277683 rocksdb.bloom.filter.full.positive COUNT : 912272 rocksdb.bloom.filter.full.true.positive COUNT : 782628 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 368 rocksdb.memtable.miss COUNT : 782628 rocksdb.l0.hit COUNT : 33804 rocksdb.l1.hit COUNT : 1507 rocksdb.l2andup.hit COUNT : 747317 rocksdb.compaction.key.drop.new COUNT : 231314 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5834357 rocksdb.number.keys.read COUNT : 782996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 6021056424 rocksdb.bytes.read COUNT : 782996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1707 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 38072764 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 26467364192 rocksdb.compact.write.bytes COUNT : 26234308830 rocksdb.flush.write.bytes COUNT : 6011271478 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 29 rocksdb.number.superversion_acquires COUNT : 1212 rocksdb.number.superversion_releases COUNT : 1163 rocksdb.number.superversion_cleanups COUNT : 306 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 241 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5962741760 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2263040 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 30727881137 rocksdb.non.last.level.read.count COUNT : 7282050 rocksdb.block.checksum.compute.count COUNT : 7334659 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 207.248284 P95 : 633.783926 P99 : 1803.794824 P100 : 92674.000000 COUNT : 782996 SUM : 239742697 rocksdb.db.write.micros P50 : 1.561122 P95 : 4.260598 P99 : 8.515646 P100 : 921520.000000 COUNT : 5834357 SUM : 52697616 rocksdb.compaction.times.micros P50 : 1420370.370370 P95 : 3726000.000000 P99 : 5759333.333333 P100 : 6434194.000000 COUNT : 101 SUM : 167952660 rocksdb.compaction.times.cpu_micros P50 : 909583.333333 P95 : 2387500.000000 P99 : 4293000.000000 P100 : 5723842.000000 COUNT : 101 SUM : 103672714 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 1018.275862 P95 : 61250.000000 P99 : 273400.000000 P100 : 311701.000000 COUNT : 82 SUM : 959036 rocksdb.compaction.outfile.sync.micros P50 : 14640.000000 P95 : 168285.714286 P99 : 354866.666667 P100 : 847325.000000 COUNT : 316 SUM : 14916181 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 420.000000 P95 : 4702.500000 P99 : 8101.500000 P100 : 8382.000000 COUNT : 218 SUM : 278362 rocksdb.table.open.io.micros P50 : 11727.003817 P95 : 21536.883629 P99 : 29930.185185 P100 : 40037.000000 COUNT : 1707 SUM : 19003118 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 4.062777 P95 : 10.027219 P99 : 217.700949 P100 : 25110.000000 COUNT : 6411373 SUM : 106073178 rocksdb.read.block.get.micros P50 : 159.892600 P95 : 514.974914 P99 : 1794.785265 P100 : 25701.000000 COUNT : 921611 SUM : 206661568 rocksdb.write.raw.block.micros P50 : 0.527543 P95 : 1.067887 P99 : 3.488478 P100 : 31218.000000 COUNT : 7810349 SUM : 24218995 rocksdb.numfiles.in.singlecompaction P50 : 2.978261 P95 : 5.075000 P99 : 9.630000 P100 : 14.000000 COUNT : 137 SUM : 461 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.568477 P95 : 1221.612650 P99 : 2611.241740 P100 : 921352.000000 COUNT : 189695 SUM : 38087125 rocksdb.sst.read.micros P50 : 2.096215 P95 : 190.697008 P99 : 505.919573 P100 : 28437.000000 COUNT : 7282137 SUM : 297474925 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 782996 SUM : 782996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5834357 SUM : 6021056424 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 226551.724138 P95 : 1000857.142857 P99 : 1139196.000000 P100 : 1139196.000000 COUNT : 82 SUM : 26984794 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 338 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0