[2024-12-07 21:25:08.050282] Starting SPDK v25.01-pre git sha1 a2f5e1c2d / DPDK 23.11.0 initialization... [2024-12-07 21:25:08.050549] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid201386 ] Set seed to 1733603108043555 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-07 21:25:08.181797] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-07 21:25:08.203040] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-07 21:25:08.203083] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 201386' to capture a snapshot of events at runtime. [2024-12-07 21:25:08.203094] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-12-07 21:25:08.203105] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-12-07 21:25:08.203113] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid201386 for offline analysis/debug. [2024-12-07 21:25:08.203651] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sat Dec 7 21:25:12 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/07-21:25:20 ... thread 0: (1000000,1000000) ops and (128988.7,128988.7) ops/second in (7.752618,7.752618) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 13/8 1.78 GB 88.1 0.6 0.0 0.6 1.5 0.9 0.0 1.7 61.6 147.7 10.23 5.10 13 0.787 639K 674 0.0 0.0 L1 5/5 517.20 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 13/1 1023.07 MB 0.9 1.1 0.7 0.4 1.1 0.7 0.2 1.5 109.7 108.4 10.52 5.40 4 2.631 629K 0 0.0 0.0 L4 175/3 9.70 GB 1.0 5.3 3.8 1.4 5.2 3.8 0.1 1.4 107.5 106.3 50.18 25.69 36 1.394 1310K 0 0.0 0.0 L5 1124/0 92.52 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1330/17 105.51 GB 0.0 7.0 4.6 2.4 7.8 5.4 5.0 9.1 101.2 112.6 70.93 36.19 53 1.338 2579K 674 0.0 0.0 Int 0/0 0.00 KB 0.0 7.0 4.6 2.4 7.8 5.4 5.0 9.1 101.2 112.6 70.93 36.19 53 1.338 2579K 674 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 7.0 4.6 2.4 6.9 4.5 0.0 0.0 112.0 110.8 64.10 33.61 42 1.526 2579K 674 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 129.1 6.83 2.58 11 0.621 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.9 total, 7.8 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.80 GB write, 1005.32 MB/s write, 7.01 GB read, 903.64 MB/s read, 70.9 seconds Interval compaction: 7.80 GB write, 1029.86 MB/s write, 7.01 GB read, 925.71 MB/s read, 70.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 30, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 30, interval: 35 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 383944 Average: 11.4859 StdDev: 158.00 Min: 1 Median: 1.9104 Max: 23580 Percentiles: P50: 1.91 P75: 2.90 P99: 10.85 P99.9: 1892.24 P99.99: 3784.72 ------------------------------------------------------ [ 0, 1 ] 13933 3.629% 3.629% # ( 1, 2 ] 195557 50.934% 54.563% ########## ( 2, 3 ] 87447 22.776% 77.339% ##### ( 3, 4 ] 35882 9.346% 86.684% ## ( 4, 6 ] 37846 9.857% 96.541% ## ( 6, 10 ] 9174 2.389% 98.931% ( 10, 15 ] 1553 0.404% 99.335% ( 15, 22 ] 196 0.051% 99.386% ( 22, 34 ] 14 0.004% 99.390% ( 34, 51 ] 14 0.004% 99.394% ( 51, 76 ] 16 0.004% 99.398% ( 76, 110 ] 33 0.009% 99.406% ( 110, 170 ] 85 0.022% 99.429% ( 170, 250 ] 113 0.029% 99.458% ( 250, 380 ] 195 0.051% 99.509% ( 380, 580 ] 315 0.082% 99.591% ( 580, 870 ] 439 0.114% 99.705% ( 870, 1300 ] 447 0.116% 99.822% ( 1300, 1900 ] 305 0.079% 99.901% ( 1900, 2900 ] 203 0.053% 99.954% ( 2900, 4400 ] 235 0.061% 100.015% ( 4400, 6600 ] 60 0.016% 100.031% ( 6600, 9900 ] 12 0.003% 100.034% ( 9900, 14000 ] 1 0.000% 100.034% ( 14000, 22000 ] 2 0.001% 100.035% ( 22000, 33000 ] 1 0.000% 100.035% ** Level 1 read latency histogram (micros): Count: 98 Average: 1232.1122 StdDev: 1353.98 Min: 110 Median: 708.8889 Max: 6996 Percentiles: P50: 708.89 P75: 1787.50 P99: 6666.00 P99.9: 6996.00 P99.99: 6996.00 ------------------------------------------------------ ( 76, 110 ] 1 1.020% 1.020% ( 110, 170 ] 15 15.306% 16.327% ### ( 170, 250 ] 9 9.184% 25.510% ## ( 250, 380 ] 6 6.122% 31.633% # ( 380, 580 ] 14 14.286% 45.918% ### ( 580, 870 ] 9 9.184% 55.102% ## ( 870, 1300 ] 13 13.265% 68.367% ### ( 1300, 1900 ] 8 8.163% 76.531% ## ( 1900, 2900 ] 11 11.224% 87.755% ## ( 2900, 4400 ] 7 7.143% 94.898% # ( 4400, 6600 ] 4 4.082% 98.980% # ( 6600, 9900 ] 1 1.020% 100.000% ** Level 2 read latency histogram (micros): Count: 150854 Average: 13.5260 StdDev: 167.41 Min: 1 Median: 1.9788 Max: 10877 Percentiles: P50: 1.98 P75: 2.97 P99: 13.00 P99.9: 2381.46 P99.99: 3862.13 ------------------------------------------------------ [ 0, 1 ] 3829 2.538% 2.538% # ( 1, 2 ] 73149 48.490% 51.028% ########## ( 2, 3 ] 37096 24.591% 75.619% ##### ( 3, 4 ] 15075 9.993% 85.612% ## ( 4, 6 ] 15326 10.159% 95.771% ## ( 6, 10 ] 4332 2.872% 98.643% # ( 10, 15 ] 898 0.595% 99.238% ( 15, 22 ] 152 0.101% 99.339% ( 22, 34 ] 21 0.014% 99.353% ( 34, 51 ] 9 0.006% 99.359% ( 51, 76 ] 7 0.005% 99.364% ( 76, 110 ] 3 0.002% 99.366% ( 110, 170 ] 13 0.009% 99.374% ( 170, 250 ] 25 0.017% 99.391% ( 250, 380 ] 60 0.040% 99.431% ( 380, 580 ] 108 0.072% 99.502% ( 580, 870 ] 169 0.112% 99.614% ( 870, 1300 ] 195 0.129% 99.743% ( 1300, 1900 ] 175 0.116% 99.859% ( 1900, 2900 ] 127 0.084% 99.944% ( 2900, 4400 ] 109 0.072% 100.016% ( 4400, 6600 ] 31 0.021% 100.036% ( 6600, 9900 ] 4 0.003% 100.039% ( 9900, 14000 ] 2 0.001% 100.040% ** Level 3 read latency histogram (micros): Count: 886611 Average: 11.4711 StdDev: 169.10 Min: 1 Median: 2.0502 Max: 28015 Percentiles: P50: 2.05 P75: 3.08 P99: 9.76 P99.9: 796.77 P99.99: 1228.05 ------------------------------------------------------ [ 0, 1 ] 14058 1.586% 1.586% ( 1, 2 ] 417930 47.138% 48.724% ######### ( 2, 3 ] 225568 25.442% 74.165% ##### ( 3, 4 ] 95943 10.821% 84.986% ## ( 4, 6 ] 101430 11.440% 96.427% ## ( 6, 10 ] 24274 2.738% 99.164% # ( 10, 15 ] 4483 0.506% 99.670% ( 15, 22 ] 651 0.073% 99.744% ( 22, 34 ] 73 0.008% 99.752% ( 34, 51 ] 37 0.004% 99.756% ( 51, 76 ] 75 0.008% 99.764% ( 76, 110 ] 34 0.004% 99.768% ( 110, 170 ] 47 0.005% 99.774% ( 170, 250 ] 72 0.008% 99.782% ( 250, 380 ] 182 0.021% 99.802% ( 380, 580 ] 386 0.044% 99.846% ( 580, 870 ] 644 0.073% 99.918% ( 870, 1300 ] 763 0.086% 100.004% ( 1300, 1900 ] 631 0.071% 100.076% ( 1900, 2900 ] 541 0.061% 100.137% ( 2900, 4400 ] 595 0.067% 100.204% ( 4400, 6600 ] 200 0.023% 100.226% ( 6600, 9900 ] 41 0.005% 100.231% ( 9900, 14000 ] 10 0.001% 100.232% ( 14000, 22000 ] 3 0.000% 100.232% ( 22000, 33000 ] 3 0.000% 100.233% ** Level 4 read latency histogram (micros): Count: 381041 Average: 29.0946 StdDev: 294.64 Min: 0 Median: 2.0210 Max: 51009 Percentiles: P50: 2.02 P75: 3.07 P99: 795.39 P99.9: 3369.90 P99.99: 4123.05 ------------------------------------------------------ [ 0, 1 ] 6816 1.789% 1.789% ( 1, 2 ] 181717 47.690% 49.478% ########## ( 2, 3 ] 94482 24.796% 74.274% ##### ( 3, 4 ] 39038 10.245% 84.519% ## ( 4, 6 ] 40216 10.554% 95.073% ## ( 6, 10 ] 10398 2.729% 97.802% # ( 10, 15 ] 1834 0.481% 98.284% ( 15, 22 ] 260 0.068% 98.352% ( 22, 34 ] 25 0.007% 98.358% ( 34, 51 ] 27 0.007% 98.366% ( 51, 76 ] 28 0.007% 98.373% ( 76, 110 ] 40 0.010% 98.383% ( 110, 170 ] 104 0.027% 98.411% ( 170, 250 ] 245 0.064% 98.475% ( 250, 380 ] 361 0.095% 98.570% ( 380, 580 ] 755 0.198% 98.768% ( 580, 870 ] 1191 0.313% 99.080% ( 870, 1300 ] 1340 0.352% 99.432% ( 1300, 1900 ] 911 0.239% 99.671% ( 1900, 2900 ] 658 0.173% 99.844% ( 2900, 4400 ] 683 0.179% 100.023% ( 4400, 6600 ] 229 0.060% 100.083% ( 6600, 9900 ] 39 0.010% 100.093% ( 9900, 14000 ] 3 0.001% 100.094% ( 14000, 22000 ] 2 0.001% 100.095% ( 22000, 33000 ] 4 0.001% 100.096% ( 33000, 50000 ] 1 0.000% 100.096% ( 50000, 75000 ] 1 0.000% 100.096% ** Level 5 read latency histogram (micros): Count: 12790 Average: 1673.3646 StdDev: 1676.23 Min: 77 Median: 1081.2018 Max: 47022 Percentiles: P50: 1081.20 P75: 2337.96 P99: 7816.47 P99.9: 9995.21 P99.99: 19441.80 ------------------------------------------------------ ( 76, 110 ] 68 0.532% 0.532% ( 110, 170 ] 388 3.034% 3.565% # ( 170, 250 ] 480 3.753% 7.318% # ( 250, 380 ] 828 6.474% 13.792% # ( 380, 580 ] 1449 11.329% 25.121% ## ( 580, 870 ] 2126 16.622% 41.744% ### ( 870, 1300 ] 2151 16.818% 58.561% ### ( 1300, 1900 ] 1512 11.822% 70.383% ## ( 1900, 2900 ] 1350 10.555% 80.938% ## ( 2900, 4400 ] 1502 11.744% 92.682% ## ( 4400, 6600 ] 742 5.801% 98.483% # ( 6600, 9900 ] 182 1.423% 99.906% ( 9900, 14000 ] 9 0.070% 99.977% ( 14000, 22000 ] 4 0.031% 100.008% ( 33000, 50000 ] 2 0.016% 100.023% ** DB Stats ** Uptime(secs): 7.9 total, 7.8 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 123.89 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.550 H:M:S, 44.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 126.90 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.550 H:M:S, 45.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/07-21:25:24 ... thread 0: (1000000,2000000) ops and (252065.4,170650.8) ops/second in (3.967225,11.719843) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 21/12 2.83 GB 169.9 1.4 0.0 1.4 3.3 1.9 0.0 1.7 82.2 198.0 16.85 11.19 33 0.511 1407K 1435 0.0 0.0 L1 5/5 517.20 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 12/0 894.29 MB 0.9 1.1 0.7 0.4 1.1 0.7 0.2 1.5 109.7 108.4 10.52 5.40 4 2.631 629K 0 0.0 0.0 L4 173/0 9.51 GB 1.0 5.5 4.0 1.5 5.5 3.9 0.1 1.4 109.0 107.8 51.84 26.82 37 1.401 1569K 3502 0.0 0.0 L5 1128/0 92.83 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1339/17 106.55 GB 0.0 8.0 4.7 3.3 9.8 6.5 5.3 5.2 103.4 127.1 79.21 43.41 74 1.070 3606K 4937 0.0 0.0 Int 0/0 0.00 KB 0.0 8.0 4.7 3.3 9.8 6.5 5.3 5.2 103.4 127.1 79.21 43.41 74 1.070 3606K 4937 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 8.0 4.7 3.3 7.9 4.6 0.0 0.0 119.2 118.1 68.68 37.59 46 1.493 3606K 4937 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 185.4 10.53 5.82 28 0.376 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 11.9 total, 11.7 interval Flush(GB): cumulative 1.907, interval 1.907 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 9.83 GB write, 844.91 MB/s write, 8.00 GB read, 687.55 MB/s read, 79.2 seconds Interval compaction: 9.83 GB write, 858.56 MB/s write, 8.00 GB read, 698.65 MB/s read, 79.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 4, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 4, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 30, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 9, total-stops: 30, interval: 39 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 654687 Average: 12.0262 StdDev: 126.95 Min: 1 Median: 1.7165 Max: 23580 Percentiles: P50: 1.72 P75: 2.58 P99: 210.83 P99.9: 1245.12 P99.99: 2865.58 ------------------------------------------------------ [ 0, 1 ] 79651 12.166% 12.166% ## ( 1, 2 ] 345700 52.804% 64.970% ########### ( 2, 3 ] 113559 17.346% 82.316% ### ( 3, 4 ] 43792 6.689% 89.005% # ( 4, 6 ] 44081 6.733% 95.738% # ( 6, 10 ] 10187 1.556% 97.294% ( 10, 15 ] 1682 0.257% 97.551% ( 15, 22 ] 226 0.035% 97.585% ( 22, 34 ] 18 0.003% 97.588% ( 34, 51 ] 18 0.003% 97.591% ( 51, 76 ] 17 0.003% 97.593% ( 76, 110 ] 1316 0.201% 97.794% ( 110, 170 ] 6186 0.945% 98.739% ( 170, 250 ] 3355 0.512% 99.252% ( 250, 380 ] 1553 0.237% 99.489% ( 380, 580 ] 1460 0.223% 99.712% ( 580, 870 ] 801 0.122% 99.834% ( 870, 1300 ] 499 0.076% 99.910% ( 1300, 1900 ] 316 0.048% 99.959% ( 1900, 2900 ] 217 0.033% 99.992% ( 2900, 4400 ] 239 0.037% 100.028% ( 4400, 6600 ] 60 0.009% 100.038% ( 6600, 9900 ] 12 0.002% 100.039% ( 9900, 14000 ] 1 0.000% 100.040% ( 14000, 22000 ] 2 0.000% 100.040% ( 22000, 33000 ] 1 0.000% 100.040% ** Level 1 read latency histogram (micros): Count: 61099 Average: 8.5221 StdDev: 77.45 Min: 1 Median: 3.9833 Max: 6996 Percentiles: P50: 3.98 P75: 5.31 P99: 87.57 P99.9: 694.85 P99.99: 4162.16 ------------------------------------------------------ [ 0, 1 ] 473 0.774% 0.774% ( 1, 2 ] 5692 9.316% 10.090% ## ( 2, 3 ] 7931 12.981% 23.071% ### ( 3, 4 ] 16733 27.387% 50.457% ##### ( 4, 6 ] 22892 37.467% 87.925% ####### ( 6, 10 ] 5098 8.344% 96.268% ## ( 10, 15 ] 990 1.620% 97.889% ( 15, 22 ] 600 0.982% 98.871% ( 22, 34 ] 61 0.100% 98.971% ( 34, 51 ] 1 0.002% 98.972% ( 76, 110 ] 50 0.082% 99.054% ( 110, 170 ] 270 0.442% 99.496% ( 170, 250 ] 124 0.203% 99.699% ( 250, 380 ] 52 0.085% 99.784% ( 380, 580 ] 61 0.100% 99.884% ( 580, 870 ] 25 0.041% 99.925% ( 870, 1300 ] 14 0.023% 99.948% ( 1300, 1900 ] 8 0.013% 99.961% ( 1900, 2900 ] 12 0.020% 99.980% ( 2900, 4400 ] 7 0.011% 99.992% ( 4400, 6600 ] 4 0.007% 99.998% ( 6600, 9900 ] 1 0.002% 100.000% ** Level 2 read latency histogram (micros): Count: 160167 Average: 13.0037 StdDev: 162.58 Min: 1 Median: 1.9486 Max: 10877 Percentiles: P50: 1.95 P75: 2.93 P99: 13.10 P99.9: 2308.13 P99.99: 3849.31 ------------------------------------------------------ [ 0, 1 ] 5204 3.249% 3.249% # ( 1, 2 ] 78939 49.285% 52.535% ########## ( 2, 3 ] 38520 24.050% 76.584% ##### ( 3, 4 ] 15316 9.563% 86.147% ## ( 4, 6 ] 15533 9.698% 95.845% ## ( 6, 10 ] 4492 2.805% 98.650% # ( 10, 15 ] 906 0.566% 99.215% ( 15, 22 ] 152 0.095% 99.310% ( 22, 34 ] 21 0.013% 99.323% ( 34, 51 ] 9 0.006% 99.329% ( 51, 76 ] 7 0.004% 99.333% ( 76, 110 ] 13 0.008% 99.341% ( 110, 170 ] 58 0.036% 99.378% ( 170, 250 ] 53 0.033% 99.411% ( 250, 380 ] 77 0.048% 99.459% ( 380, 580 ] 116 0.072% 99.531% ( 580, 870 ] 169 0.106% 99.637% ( 870, 1300 ] 195 0.122% 99.758% ( 1300, 1900 ] 175 0.109% 99.868% ( 1900, 2900 ] 127 0.079% 99.947% ( 2900, 4400 ] 109 0.068% 100.015% ( 4400, 6600 ] 31 0.019% 100.034% ( 6600, 9900 ] 4 0.002% 100.037% ( 9900, 14000 ] 2 0.001% 100.038% ** Level 3 read latency histogram (micros): Count: 887273 Average: 11.6118 StdDev: 169.15 Min: 1 Median: 2.0516 Max: 28015 Percentiles: P50: 2.05 P75: 3.08 P99: 9.87 P99.9: 799.14 P99.99: 1228.21 ------------------------------------------------------ [ 0, 1 ] 14058 1.584% 1.584% ( 1, 2 ] 417930 47.103% 48.687% ######### ( 2, 3 ] 225568 25.423% 74.110% ##### ( 3, 4 ] 95943 10.813% 84.923% ## ( 4, 6 ] 101430 11.432% 96.355% ## ( 6, 10 ] 24274 2.736% 99.090% # ( 10, 15 ] 4483 0.505% 99.596% ( 15, 22 ] 651 0.073% 99.669% ( 22, 34 ] 73 0.008% 99.677% ( 34, 51 ] 37 0.004% 99.681% ( 51, 76 ] 75 0.008% 99.690% ( 76, 110 ] 111 0.013% 99.702% ( 110, 170 ] 369 0.042% 99.744% ( 170, 250 ] 201 0.023% 99.767% ( 250, 380 ] 256 0.029% 99.796% ( 380, 580 ] 428 0.048% 99.844% ( 580, 870 ] 660 0.074% 99.918% ( 870, 1300 ] 765 0.086% 100.004% ( 1300, 1900 ] 631 0.071% 100.076% ( 1900, 2900 ] 541 0.061% 100.136% ( 2900, 4400 ] 595 0.067% 100.204% ( 4400, 6600 ] 200 0.023% 100.226% ( 6600, 9900 ] 41 0.005% 100.231% ( 9900, 14000 ] 10 0.001% 100.232% ( 14000, 22000 ] 3 0.000% 100.232% ( 22000, 33000 ] 3 0.000% 100.233% ** Level 4 read latency histogram (micros): Count: 405837 Average: 31.0329 StdDev: 287.16 Min: 0 Median: 2.0200 Max: 51009 Percentiles: P50: 2.02 P75: 3.15 P99: 755.79 P99.9: 3312.04 P99.99: 4111.86 ------------------------------------------------------ [ 0, 1 ] 9128 2.249% 2.249% ( 1, 2 ] 191837 47.269% 49.519% ######### ( 2, 3 ] 97462 24.015% 73.534% ##### ( 3, 4 ] 39906 9.833% 83.367% ## ( 4, 6 ] 41136 10.136% 93.503% ## ( 6, 10 ] 10761 2.652% 96.154% # ( 10, 15 ] 1863 0.459% 96.613% ( 15, 22 ] 267 0.066% 96.679% ( 22, 34 ] 26 0.006% 96.686% ( 34, 51 ] 27 0.007% 96.692% ( 51, 76 ] 29 0.007% 96.699% ( 76, 110 ] 860 0.212% 96.911% ( 110, 170 ] 3547 0.874% 97.785% ( 170, 250 ] 1826 0.450% 98.235% ( 250, 380 ] 1042 0.257% 98.492% ( 380, 580 ] 1253 0.309% 98.801% ( 580, 870 ] 1334 0.329% 99.129% ( 870, 1300 ] 1362 0.336% 99.465% ( 1300, 1900 ] 916 0.226% 99.691% ( 1900, 2900 ] 661 0.163% 99.854% ( 2900, 4400 ] 685 0.169% 100.022% ( 4400, 6600 ] 229 0.056% 100.079% ( 6600, 9900 ] 39 0.010% 100.088% ( 9900, 14000 ] 3 0.001% 100.089% ( 14000, 22000 ] 2 0.000% 100.090% ( 22000, 33000 ] 4 0.001% 100.091% ( 33000, 50000 ] 1 0.000% 100.091% ( 50000, 75000 ] 1 0.000% 100.091% ** Level 5 read latency histogram (micros): Count: 59393 Average: 520.5418 StdDev: 993.24 Min: 75 Median: 181.1096 Max: 47022 Percentiles: P50: 181.11 P75: 414.58 P99: 5405.33 P99.9: 8986.28 P99.99: 11294.32 ------------------------------------------------------ ( 51, 76 ] 1 0.002% 0.002% ( 76, 110 ] 5840 9.833% 9.834% ## ( 110, 170 ] 22401 37.717% 47.551% ######## ( 170, 250 ] 10481 17.647% 65.198% #### ( 250, 380 ] 4986 8.395% 73.593% ## ( 380, 580 ] 4839 8.147% 81.740% ## ( 580, 870 ] 3175 5.346% 87.086% # ( 870, 1300 ] 2329 3.921% 91.007% # ( 1300, 1900 ] 1533 2.581% 93.588% # ( 1900, 2900 ] 1369 2.305% 95.893% ( 2900, 4400 ] 1507 2.537% 98.431% # ( 4400, 6600 ] 742 1.249% 99.680% ( 6600, 9900 ] 182 0.306% 99.987% ( 9900, 14000 ] 9 0.015% 100.002% ( 14000, 22000 ] 4 0.007% 100.008% ( 33000, 50000 ] 2 0.003% 100.012% ** DB Stats ** Uptime(secs): 11.9 total, 4.0 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 165.23 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:4.027 H:M:S, 33.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 247.97 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.477 H:M:S, 12.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 0 2024/12/07-21:25:30 ... thread 0: (1000000,3000000) ops and (155152.0,165151.5) ops/second in (6.445294,18.165137) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/10 2.77 GB 100.7 2.3 0.0 2.3 5.1 2.8 0.0 1.8 99.4 217.1 24.08 17.44 50 0.482 2430K 2418 0.0 0.0 L1 3/0 351.45 MB 35.1 1.4 0.9 0.5 1.4 0.9 0.0 1.5 91.1 90.5 16.06 9.25 1 16.058 1484K 8768 0.0 0.0 L2 2/2 257.55 MB 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 25/8 1.69 GB 0.9 2.0 1.2 0.8 2.0 1.2 0.6 1.6 122.9 121.8 16.76 9.61 9 1.862 1278K 1326 0.0 0.0 L4 174/4 9.52 GB 1.0 5.5 4.0 1.5 5.5 3.9 0.1 1.4 109.0 107.8 51.84 26.82 37 1.401 1569K 3502 0.0 0.0 L5 1128/0 92.83 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1346/24 107.40 GB 0.0 11.3 6.1 5.2 14.0 8.8 6.7 5.0 106.4 131.6 108.73 63.11 97 1.121 6763K 16K 0.0 0.0 Int 0/0 0.00 KB 0.0 11.3 6.1 5.2 14.0 8.8 6.7 5.0 106.4 131.6 108.73 63.11 97 1.121 6763K 16K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 11.3 6.1 5.2 11.2 6.0 0.0 0.0 121.9 120.9 94.88 54.82 56 1.694 6763K 16K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 204.6 13.86 8.29 41 0.338 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.4 total, 18.2 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 13.97 GB write, 779.39 MB/s write, 11.29 GB read, 630.00 MB/s read, 108.7 seconds Interval compaction: 13.97 GB write, 787.51 MB/s write, 11.29 GB read, 636.57 MB/s read, 108.7 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 12, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 12, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 32, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 17, total-stops: 32, interval: 49 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1101907 Average: 11.8392 StdDev: 107.92 Min: 0 Median: 1.6782 Max: 23580 Percentiles: P50: 1.68 P75: 2.49 P99: 216.02 P99.9: 919.42 P99.99: 2047.69 ------------------------------------------------------ [ 0, 1 ] 151816 13.778% 13.778% ### ( 1, 2 ] 588504 53.408% 67.185% ########### ( 2, 3 ] 176313 16.001% 83.186% ### ( 3, 4 ] 68153 6.185% 89.371% # ( 4, 6 ] 66026 5.992% 95.363% # ( 6, 10 ] 13555 1.230% 96.593% ( 10, 15 ] 2377 0.216% 96.809% ( 15, 22 ] 303 0.027% 96.836% ( 22, 34 ] 21 0.002% 96.838% ( 34, 51 ] 30 0.003% 96.841% ( 51, 76 ] 26 0.002% 96.843% ( 76, 110 ] 3986 0.362% 97.205% ( 110, 170 ] 15794 1.433% 98.638% ( 170, 250 ] 6932 0.629% 99.268% ( 250, 380 ] 3060 0.278% 99.545% ( 380, 580 ] 2634 0.239% 99.784% ( 580, 870 ] 1203 0.109% 99.893% ( 870, 1300 ] 662 0.060% 99.954% ( 1300, 1900 ] 367 0.033% 99.987% ( 1900, 2900 ] 256 0.023% 100.010% ( 2900, 4400 ] 292 0.026% 100.037% ( 4400, 6600 ] 68 0.006% 100.043% ( 6600, 9900 ] 12 0.001% 100.044% ( 9900, 14000 ] 1 0.000% 100.044% ( 14000, 22000 ] 2 0.000% 100.044% ( 22000, 33000 ] 1 0.000% 100.044% ** Level 1 read latency histogram (micros): Count: 315684 Average: 7.4403 StdDev: 92.78 Min: 1 Median: 2.7833 Max: 9654 Percentiles: P50: 2.78 P75: 4.00 P99: 14.57 P99.9: 521.20 P99.99: 1107.32 ------------------------------------------------------ [ 0, 1 ] 2002 0.634% 0.634% ( 1, 2 ] 96128 30.451% 31.085% ###### ( 2, 3 ] 76234 24.149% 55.234% ##### ( 3, 4 ] 62466 19.788% 75.021% #### ( 4, 6 ] 59371 18.807% 93.828% #### ( 6, 10 ] 13585 4.303% 98.132% # ( 10, 15 ] 3002 0.951% 99.083% ( 15, 22 ] 1036 0.328% 99.411% ( 22, 34 ] 109 0.035% 99.445% ( 34, 51 ] 42 0.013% 99.459% ( 51, 76 ] 30 0.010% 99.468% ( 76, 110 ] 150 0.048% 99.516% ( 110, 170 ] 610 0.193% 99.709% ( 170, 250 ] 268 0.085% 99.794% ( 250, 380 ] 180 0.057% 99.851% ( 380, 580 ] 220 0.070% 99.920% ( 580, 870 ] 167 0.053% 99.973% ( 870, 1300 ] 95 0.030% 100.003% ( 1300, 1900 ] 53 0.017% 100.020% ( 1900, 2900 ] 60 0.019% 100.039% ( 2900, 4400 ] 90 0.029% 100.068% ( 4400, 6600 ] 15 0.005% 100.073% ( 6600, 9900 ] 7 0.002% 100.075% ** Level 2 read latency histogram (micros): Count: 190088 Average: 12.6146 StdDev: 159.96 Min: 1 Median: 1.9633 Max: 10877 Percentiles: P50: 1.96 P75: 2.95 P99: 13.25 P99.9: 2339.94 P99.99: 3958.04 ------------------------------------------------------ [ 0, 1 ] 5464 2.874% 2.874% # ( 1, 2 ] 92989 48.919% 51.793% ########## ( 2, 3 ] 46502 24.463% 76.257% ##### ( 3, 4 ] 18401 9.680% 85.937% ## ( 4, 6 ] 18820 9.901% 95.838% ## ( 6, 10 ] 5303 2.790% 98.627% # ( 10, 15 ] 1088 0.572% 99.200% ( 15, 22 ] 173 0.091% 99.291% ( 22, 34 ] 23 0.012% 99.303% ( 34, 51 ] 10 0.005% 99.308% ( 51, 76 ] 10 0.005% 99.313% ( 76, 110 ] 27 0.014% 99.328% ( 110, 170 ] 112 0.059% 99.387% ( 170, 250 ] 82 0.043% 99.430% ( 250, 380 ] 101 0.053% 99.483% ( 380, 580 ] 148 0.078% 99.561% ( 580, 870 ] 188 0.099% 99.660% ( 870, 1300 ] 211 0.111% 99.771% ( 1300, 1900 ] 183 0.096% 99.867% ( 1900, 2900 ] 143 0.075% 99.942% ( 2900, 4400 ] 129 0.068% 100.010% ( 4400, 6600 ] 34 0.018% 100.028% ( 6600, 9900 ] 6 0.003% 100.031% ( 9900, 14000 ] 2 0.001% 100.032% ** Level 3 read latency histogram (micros): Count: 1080039 Average: 11.5419 StdDev: 164.87 Min: 1 Median: 2.0389 Max: 28015 Percentiles: P50: 2.04 P75: 3.04 P99: 9.97 P99.9: 839.57 P99.99: 1269.22 ------------------------------------------------------ [ 0, 1 ] 15983 1.480% 1.480% ( 1, 2 ] 513277 47.524% 49.004% ########## ( 2, 3 ] 276699 25.619% 74.623% ##### ( 3, 4 ] 114590 10.610% 85.233% ## ( 4, 6 ] 119999 11.111% 96.344% ## ( 6, 10 ] 28900 2.676% 99.019% # ( 10, 15 ] 5274 0.488% 99.508% ( 15, 22 ] 740 0.069% 99.576% ( 22, 34 ] 81 0.007% 99.584% ( 34, 51 ] 66 0.006% 99.590% ( 51, 76 ] 91 0.008% 99.598% ( 76, 110 ] 260 0.024% 99.622% ( 110, 170 ] 786 0.073% 99.695% ( 170, 250 ] 408 0.038% 99.733% ( 250, 380 ] 416 0.039% 99.771% ( 380, 580 ] 623 0.058% 99.829% ( 580, 870 ] 858 0.079% 99.909% ( 870, 1300 ] 950 0.088% 99.996% ( 1300, 1900 ] 735 0.068% 100.065% ( 1900, 2900 ] 628 0.058% 100.123% ( 2900, 4400 ] 712 0.066% 100.189% ( 4400, 6600 ] 229 0.021% 100.210% ( 6600, 9900 ] 47 0.004% 100.214% ( 9900, 14000 ] 11 0.001% 100.215% ( 14000, 22000 ] 3 0.000% 100.215% ( 22000, 33000 ] 3 0.000% 100.216% ** Level 4 read latency histogram (micros): Count: 422906 Average: 36.5265 StdDev: 290.38 Min: 0 Median: 2.0762 Max: 51009 Percentiles: P50: 2.08 P75: 3.35 P99: 821.04 P99.9: 3382.55 P99.99: 4157.21 ------------------------------------------------------ [ 0, 1 ] 9240 2.185% 2.185% ( 1, 2 ] 194681 46.034% 48.219% ######### ( 2, 3 ] 98814 23.365% 71.584% ##### ( 3, 4 ] 40779 9.643% 81.227% ## ( 4, 6 ] 41970 9.924% 91.151% ## ( 6, 10 ] 10940 2.587% 93.738% # ( 10, 15 ] 1884 0.445% 94.184% ( 15, 22 ] 269 0.064% 94.247% ( 22, 34 ] 26 0.006% 94.253% ( 34, 51 ] 27 0.006% 94.260% ( 51, 76 ] 29 0.007% 94.267% ( 76, 110 ] 2297 0.543% 94.810% ( 110, 170 ] 8748 2.069% 96.878% ( 170, 250 ] 3697 0.874% 97.752% ( 250, 380 ] 1869 0.442% 98.194% ( 380, 580 ] 1969 0.466% 98.660% ( 580, 870 ] 1730 0.409% 99.069% ( 870, 1300 ] 1563 0.370% 99.439% ( 1300, 1900 ] 998 0.236% 99.675% ( 1900, 2900 ] 716 0.169% 99.844% ( 2900, 4400 ] 737 0.174% 100.018% ( 4400, 6600 ] 243 0.057% 100.076% ( 6600, 9900 ] 39 0.009% 100.085% ( 9900, 14000 ] 3 0.001% 100.086% ( 14000, 22000 ] 2 0.000% 100.086% ( 22000, 33000 ] 4 0.001% 100.087% ( 33000, 50000 ] 1 0.000% 100.087% ( 50000, 75000 ] 1 0.000% 100.087% ** Level 5 read latency histogram (micros): Count: 123833 Average: 382.1696 StdDev: 756.79 Min: 75 Median: 161.9116 Max: 47022 Percentiles: P50: 161.91 P75: 285.31 P99: 4219.76 P99.9: 7868.14 P99.99: 9773.76 ------------------------------------------------------ ( 51, 76 ] 1 0.001% 0.001% ( 76, 110 ] 15061 12.162% 12.163% ## ( 110, 170 ] 54155 43.732% 55.895% ######### ( 170, 250 ] 21063 17.009% 72.905% ### ( 250, 380 ] 9556 7.717% 80.621% ## ( 380, 580 ] 8823 7.125% 87.746% # ( 580, 870 ] 5199 4.198% 91.945% # ( 870, 1300 ] 3453 2.788% 94.733% # ( 1300, 1900 ] 2005 1.619% 96.352% ( 1900, 2900 ] 1659 1.340% 97.692% ( 2900, 4400 ] 1842 1.487% 99.180% ( 4400, 6600 ] 819 0.661% 99.841% ( 6600, 9900 ] 193 0.156% 99.997% ( 9900, 14000 ] 9 0.007% 100.004% ( 14000, 22000 ] 4 0.003% 100.007% ( 33000, 50000 ] 2 0.002% 100.009% ** DB Stats ** Uptime(secs): 18.4 total, 6.4 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 160.85 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.600 H:M:S, 36.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 152.74 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.573 H:M:S, 39.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 7 num-running-flushes: 1 2024/12/07-21:25:35 ... thread 0: (1000000,4000000) ops and (225774.4,177035.5) ops/second in (4.429200,22.594337) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/4 3.74 GB 357.8 4.7 0.0 4.7 8.4 3.7 0.0 2.2 124.0 223.3 38.59 29.82 69 0.559 4859K 13K 0.0 0.0 L1 3/0 351.45 MB 35.1 1.4 0.9 0.5 1.4 0.9 0.0 1.5 91.1 90.5 16.06 9.25 1 16.058 1484K 8768 0.0 0.0 L3 18/0 992.95 MB 1.0 2.7 1.5 1.3 2.7 1.4 0.6 1.8 127.8 126.7 21.85 13.01 11 1.986 1540K 1326 0.0 0.0 L4 175/0 9.42 GB 1.0 7.3 4.9 2.3 7.2 4.8 0.1 1.5 116.4 115.1 64.04 35.10 49 1.307 3192K 13K 0.0 0.0 L5 1140/0 93.87 GB 1.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1351/4 108.34 GB 0.0 16.1 7.3 8.8 19.7 11.0 6.7 5.3 117.4 143.8 140.53 87.17 130 1.081 11M 36K 0.0 0.0 Int 0/0 0.00 KB 0.0 16.1 7.3 8.8 19.7 11.0 6.7 5.3 117.4 143.8 140.53 87.17 130 1.081 11M 36K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.1 7.3 8.8 16.0 7.2 0.0 0.0 134.9 133.8 122.27 75.88 74 1.652 11M 36K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 210.4 18.26 11.29 56 0.326 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 22.8 total, 22.6 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 19.73 GB write, 886.87 MB/s write, 16.11 GB read, 724.01 MB/s read, 140.5 seconds Interval compaction: 19.73 GB write, 894.30 MB/s write, 16.11 GB read, 730.07 MB/s read, 140.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 12, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 12, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 40, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 17, total-stops: 40, interval: 57 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1541248 Average: 10.4242 StdDev: 100.37 Min: 0 Median: 1.6740 Max: 23580 Percentiles: P50: 1.67 P75: 2.45 P99: 193.22 P99.9: 841.83 P99.99: 1782.50 ------------------------------------------------------ [ 0, 1 ] 202517 13.140% 13.140% ### ( 1, 2 ] 842850 54.686% 67.826% ########### ( 2, 3 ] 248046 16.094% 83.920% ### ( 3, 4 ] 95220 6.178% 90.098% # ( 4, 6 ] 90432 5.867% 95.965% # ( 6, 10 ] 19065 1.237% 97.202% ( 10, 15 ] 3227 0.209% 97.412% ( 15, 22 ] 410 0.027% 97.438% ( 22, 34 ] 28 0.002% 97.440% ( 34, 51 ] 37 0.002% 97.443% ( 51, 76 ] 45 0.003% 97.446% ( 76, 110 ] 4360 0.283% 97.728% ( 110, 170 ] 17303 1.123% 98.851% ( 170, 250 ] 7913 0.513% 99.364% ( 250, 380 ] 3664 0.238% 99.602% ( 380, 580 ] 3185 0.207% 99.809% ( 580, 870 ] 1557 0.101% 99.910% ( 870, 1300 ] 874 0.057% 99.967% ( 1300, 1900 ] 450 0.029% 99.996% ( 1900, 2900 ] 319 0.021% 100.016% ( 2900, 4400 ] 344 0.022% 100.039% ( 4400, 6600 ] 89 0.006% 100.045% ( 6600, 9900 ] 15 0.001% 100.046% ( 9900, 14000 ] 2 0.000% 100.046% ( 14000, 22000 ] 2 0.000% 100.046% ( 22000, 33000 ] 1 0.000% 100.046% ** Level 1 read latency histogram (micros): Count: 357519 Average: 7.5793 StdDev: 92.74 Min: 1 Median: 2.6461 Max: 9654 Percentiles: P50: 2.65 P75: 3.90 P99: 14.71 P99.9: 606.28 P99.99: 1284.40 ------------------------------------------------------ [ 0, 1 ] 3535 0.989% 0.989% ( 1, 2 ] 119756 33.496% 34.485% ####### ( 2, 3 ] 85853 24.014% 58.499% ##### ( 3, 4 ] 65310 18.268% 76.766% #### ( 4, 6 ] 61979 17.336% 94.102% ### ( 6, 10 ] 14576 4.077% 98.179% # ( 10, 15 ] 3114 0.871% 99.050% ( 15, 22 ] 1067 0.298% 99.349% ( 22, 34 ] 111 0.031% 99.380% ( 34, 51 ] 44 0.012% 99.392% ( 51, 76 ] 32 0.009% 99.401% ( 76, 110 ] 181 0.051% 99.451% ( 110, 170 ] 729 0.204% 99.655% ( 170, 250 ] 336 0.094% 99.749% ( 250, 380 ] 240 0.067% 99.817% ( 380, 580 ] 279 0.078% 99.895% ( 580, 870 ] 215 0.060% 99.955% ( 870, 1300 ] 131 0.037% 99.991% ( 1300, 1900 ] 70 0.020% 100.011% ( 1900, 2900 ] 78 0.022% 100.033% ( 2900, 4400 ] 95 0.027% 100.059% ( 4400, 6600 ] 16 0.004% 100.064% ( 6600, 9900 ] 8 0.002% 100.066% ** Level 2 read latency histogram (micros): Count: 190088 Average: 12.6146 StdDev: 159.96 Min: 1 Median: 1.9633 Max: 10877 Percentiles: P50: 1.96 P75: 2.95 P99: 13.25 P99.9: 2339.94 P99.99: 3958.04 ------------------------------------------------------ [ 0, 1 ] 5464 2.874% 2.874% # ( 1, 2 ] 92989 48.919% 51.793% ########## ( 2, 3 ] 46502 24.463% 76.257% ##### ( 3, 4 ] 18401 9.680% 85.937% ## ( 4, 6 ] 18820 9.901% 95.838% ## ( 6, 10 ] 5303 2.790% 98.627% # ( 10, 15 ] 1088 0.572% 99.200% ( 15, 22 ] 173 0.091% 99.291% ( 22, 34 ] 23 0.012% 99.303% ( 34, 51 ] 10 0.005% 99.308% ( 51, 76 ] 10 0.005% 99.313% ( 76, 110 ] 27 0.014% 99.328% ( 110, 170 ] 112 0.059% 99.387% ( 170, 250 ] 82 0.043% 99.430% ( 250, 380 ] 101 0.053% 99.483% ( 380, 580 ] 148 0.078% 99.561% ( 580, 870 ] 188 0.099% 99.660% ( 870, 1300 ] 211 0.111% 99.771% ( 1300, 1900 ] 183 0.096% 99.867% ( 1900, 2900 ] 143 0.075% 99.942% ( 2900, 4400 ] 129 0.068% 100.010% ( 4400, 6600 ] 34 0.018% 100.028% ( 6600, 9900 ] 6 0.003% 100.031% ( 9900, 14000 ] 2 0.001% 100.032% ** Level 3 read latency histogram (micros): Count: 1330030 Average: 10.2563 StdDev: 151.53 Min: 1 Median: 2.0319 Max: 28015 Percentiles: P50: 2.03 P75: 3.02 P99: 9.82 P99.9: 686.78 P99.99: 1152.23 ------------------------------------------------------ [ 0, 1 ] 18868 1.419% 1.419% ( 1, 2 ] 635257 47.763% 49.181% ########## ( 2, 3 ] 341092 25.645% 74.827% ##### ( 3, 4 ] 141465 10.636% 85.463% ## ( 4, 6 ] 148389 11.157% 96.620% ## ( 6, 10 ] 33151 2.493% 99.112% ( 10, 15 ] 5930 0.446% 99.558% ( 15, 22 ] 827 0.062% 99.620% ( 22, 34 ] 90 0.007% 99.627% ( 34, 51 ] 80 0.006% 99.633% ( 51, 76 ] 118 0.009% 99.642% ( 76, 110 ] 325 0.024% 99.666% ( 110, 170 ] 1034 0.078% 99.744% ( 170, 250 ] 568 0.043% 99.787% ( 250, 380 ] 496 0.037% 99.824% ( 380, 580 ] 683 0.051% 99.875% ( 580, 870 ] 888 0.067% 99.942% ( 870, 1300 ] 969 0.073% 100.015% ( 1300, 1900 ] 750 0.056% 100.071% ( 1900, 2900 ] 647 0.049% 100.120% ( 2900, 4400 ] 745 0.056% 100.176% ( 4400, 6600 ] 237 0.018% 100.194% ( 6600, 9900 ] 51 0.004% 100.198% ( 9900, 14000 ] 11 0.001% 100.199% ( 14000, 22000 ] 3 0.000% 100.199% ( 22000, 33000 ] 3 0.000% 100.199% ** Level 4 read latency histogram (micros): Count: 600111 Average: 32.9032 StdDev: 268.84 Min: 0 Median: 2.0611 Max: 51009 Percentiles: P50: 2.06 P75: 3.30 P99: 729.90 P99.9: 3179.59 P99.99: 4089.87 ------------------------------------------------------ [ 0, 1 ] 12097 2.016% 2.016% ( 1, 2 ] 279324 46.545% 48.561% ######### ( 2, 3 ] 141303 23.546% 72.107% ##### ( 3, 4 ] 58213 9.700% 81.808% ## ( 4, 6 ] 59887 9.979% 91.787% ## ( 6, 10 ] 15592 2.598% 94.385% # ( 10, 15 ] 2549 0.425% 94.810% ( 15, 22 ] 344 0.057% 94.867% ( 22, 34 ] 30 0.005% 94.872% ( 34, 51 ] 32 0.005% 94.878% ( 51, 76 ] 48 0.008% 94.886% ( 76, 110 ] 2808 0.468% 95.354% ( 110, 170 ] 10601 1.767% 97.120% ( 170, 250 ] 4773 0.795% 97.915% ( 250, 380 ] 2559 0.426% 98.342% ( 380, 580 ] 2745 0.457% 98.799% ( 580, 870 ] 2331 0.388% 99.188% ( 870, 1300 ] 1973 0.329% 99.516% ( 1300, 1900 ] 1241 0.207% 99.723% ( 1900, 2900 ] 895 0.149% 99.872% ( 2900, 4400 ] 890 0.148% 100.021% ( 4400, 6600 ] 304 0.051% 100.071% ( 6600, 9900 ] 54 0.009% 100.080% ( 9900, 14000 ] 8 0.001% 100.082% ( 14000, 22000 ] 2 0.000% 100.082% ( 22000, 33000 ] 4 0.001% 100.083% ( 33000, 50000 ] 1 0.000% 100.083% ( 50000, 75000 ] 1 0.000% 100.083% ** Level 5 read latency histogram (micros): Count: 155441 Average: 377.6697 StdDev: 719.65 Min: 75 Median: 164.9097 Max: 47022 Percentiles: P50: 164.91 P75: 311.50 P99: 4094.31 P99.9: 7622.68 P99.99: 9740.39 ------------------------------------------------------ ( 51, 76 ] 2 0.001% 0.001% ( 76, 110 ] 18093 11.640% 11.641% ## ( 110, 170 ] 65153 41.915% 53.556% ######## ( 170, 250 ] 27075 17.418% 70.974% ### ( 250, 380 ] 13229 8.511% 79.485% ## ( 380, 580 ] 12455 8.013% 87.498% ## ( 580, 870 ] 7368 4.740% 92.238% # ( 870, 1300 ] 4472 2.877% 95.115% # ( 1300, 1900 ] 2438 1.568% 96.683% ( 1900, 2900 ] 1921 1.236% 97.919% ( 2900, 4400 ] 2112 1.359% 99.278% ( 4400, 6600 ] 901 0.580% 99.857% ( 6600, 9900 ] 218 0.140% 99.997% ( 9900, 14000 ] 10 0.006% 100.004% ( 14000, 22000 ] 4 0.003% 100.006% ( 33000, 50000 ] 2 0.001% 100.008% ** DB Stats ** Uptime(secs): 22.8 total, 4.4 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 172.77 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.336 H:M:S, 32.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 222.21 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.736 H:M:S, 16.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/12/07-21:25:38 ... thread 0: (1000000,5000000) ops and (292659.6,192224.3) ops/second in (3.416939,26.011276) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/4 4.72 GB 458.5 5.7 0.0 5.7 10.4 4.7 0.0 2.2 126.9 232.9 45.65 36.40 89 0.513 5883K 13K 0.0 0.0 L1 3/0 351.45 MB 35.1 1.4 0.9 0.5 1.4 0.9 0.0 1.5 91.1 90.5 16.06 9.25 1 16.058 1484K 8768 0.0 0.0 L3 18/0 992.95 MB 1.0 2.7 1.5 1.3 2.7 1.4 0.6 1.8 127.8 126.7 21.85 13.01 11 1.986 1540K 1326 0.0 0.0 L4 175/0 9.42 GB 1.0 7.3 4.9 2.3 7.2 4.8 0.1 1.5 116.4 115.1 64.04 35.10 49 1.307 3192K 13K 0.0 0.0 L5 1140/0 93.87 GB 1.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1355/4 109.33 GB 0.0 17.1 7.3 9.8 21.7 11.9 6.7 4.6 118.6 150.6 147.59 93.76 150 0.984 12M 37K 0.0 0.0 Int 0/0 0.00 KB 0.0 17.1 7.3 9.8 21.7 11.9 6.7 4.6 118.6 150.6 147.59 93.76 150 0.984 12M 37K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 17.1 7.3 9.8 17.0 7.2 0.0 0.0 138.8 137.7 126.13 79.66 78 1.617 12M 37K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 226.1 21.46 14.09 72 0.298 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 26.2 total, 26.0 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 21.70 GB write, 848.12 MB/s write, 17.09 GB read, 668.06 MB/s read, 147.6 seconds Interval compaction: 21.70 GB write, 854.29 MB/s write, 17.09 GB read, 672.93 MB/s read, 147.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 12, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 12, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 40, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 17, total-stops: 40, interval: 57 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1801872 Average: 9.7973 StdDev: 93.92 Min: 0 Median: 1.6291 Max: 23580 Percentiles: P50: 1.63 P75: 2.31 P99: 185.36 P99.9: 788.72 P99.99: 1647.06 ------------------------------------------------------ [ 0, 1 ] 277640 15.408% 15.408% ### ( 1, 2 ] 990777 54.986% 70.394% ########### ( 2, 3 ] 267620 14.852% 85.247% ### ( 3, 4 ] 102102 5.666% 90.913% # ( 4, 6 ] 94976 5.271% 96.184% # ( 6, 10 ] 19974 1.109% 97.293% ( 10, 15 ] 3519 0.195% 97.488% ( 15, 22 ] 457 0.025% 97.513% ( 22, 34 ] 32 0.002% 97.515% ( 34, 51 ] 39 0.002% 97.517% ( 51, 76 ] 48 0.003% 97.520% ( 76, 110 ] 5061 0.281% 97.801% ( 110, 170 ] 19856 1.102% 98.903% ( 170, 250 ] 9134 0.507% 99.410% ( 250, 380 ] 4111 0.228% 99.638% ( 380, 580 ] 3534 0.196% 99.834% ( 580, 870 ] 1655 0.092% 99.926% ( 870, 1300 ] 890 0.049% 99.975% ( 1300, 1900 ] 463 0.026% 100.001% ( 1900, 2900 ] 323 0.018% 100.019% ( 2900, 4400 ] 347 0.019% 100.038% ( 4400, 6600 ] 89 0.005% 100.043% ( 6600, 9900 ] 15 0.001% 100.044% ( 9900, 14000 ] 2 0.000% 100.044% ( 14000, 22000 ] 2 0.000% 100.044% ( 22000, 33000 ] 1 0.000% 100.044% ** Level 1 read latency histogram (micros): Count: 357952 Average: 7.8004 StdDev: 93.00 Min: 1 Median: 2.6486 Max: 9654 Percentiles: P50: 2.65 P75: 3.91 P99: 16.64 P99.9: 621.12 P99.99: 1284.50 ------------------------------------------------------ [ 0, 1 ] 3535 0.988% 0.988% ( 1, 2 ] 119756 33.456% 34.443% ####### ( 2, 3 ] 85853 23.985% 58.428% ##### ( 3, 4 ] 65310 18.245% 76.673% #### ( 4, 6 ] 61979 17.315% 93.988% ### ( 6, 10 ] 14576 4.072% 98.060% # ( 10, 15 ] 3114 0.870% 98.930% ( 15, 22 ] 1067 0.298% 99.228% ( 22, 34 ] 111 0.031% 99.259% ( 34, 51 ] 44 0.012% 99.272% ( 51, 76 ] 32 0.009% 99.281% ( 76, 110 ] 245 0.068% 99.349% ( 110, 170 ] 942 0.263% 99.612% ( 170, 250 ] 427 0.119% 99.732% ( 250, 380 ] 273 0.076% 99.808% ( 380, 580 ] 298 0.083% 99.891% ( 580, 870 ] 226 0.063% 99.954% ( 870, 1300 ] 133 0.037% 99.991% ( 1300, 1900 ] 70 0.020% 100.011% ( 1900, 2900 ] 78 0.022% 100.033% ( 2900, 4400 ] 95 0.027% 100.059% ( 4400, 6600 ] 16 0.004% 100.064% ( 6600, 9900 ] 8 0.002% 100.066% ** Level 2 read latency histogram (micros): Count: 190088 Average: 12.6146 StdDev: 159.96 Min: 1 Median: 1.9633 Max: 10877 Percentiles: P50: 1.96 P75: 2.95 P99: 13.25 P99.9: 2339.94 P99.99: 3958.04 ------------------------------------------------------ [ 0, 1 ] 5464 2.874% 2.874% # ( 1, 2 ] 92989 48.919% 51.793% ########## ( 2, 3 ] 46502 24.463% 76.257% ##### ( 3, 4 ] 18401 9.680% 85.937% ## ( 4, 6 ] 18820 9.901% 95.838% ## ( 6, 10 ] 5303 2.790% 98.627% # ( 10, 15 ] 1088 0.572% 99.200% ( 15, 22 ] 173 0.091% 99.291% ( 22, 34 ] 23 0.012% 99.303% ( 34, 51 ] 10 0.005% 99.308% ( 51, 76 ] 10 0.005% 99.313% ( 76, 110 ] 27 0.014% 99.328% ( 110, 170 ] 112 0.059% 99.387% ( 170, 250 ] 82 0.043% 99.430% ( 250, 380 ] 101 0.053% 99.483% ( 380, 580 ] 148 0.078% 99.561% ( 580, 870 ] 188 0.099% 99.660% ( 870, 1300 ] 211 0.111% 99.771% ( 1300, 1900 ] 183 0.096% 99.867% ( 1900, 2900 ] 143 0.075% 99.942% ( 2900, 4400 ] 129 0.068% 100.010% ( 4400, 6600 ] 34 0.018% 100.028% ( 6600, 9900 ] 6 0.003% 100.031% ( 9900, 14000 ] 2 0.001% 100.032% ** Level 3 read latency histogram (micros): Count: 1330702 Average: 10.3484 StdDev: 151.59 Min: 1 Median: 2.0329 Max: 28015 Percentiles: P50: 2.03 P75: 3.02 P99: 9.90 P99.9: 689.70 P99.99: 1153.39 ------------------------------------------------------ [ 0, 1 ] 18868 1.418% 1.418% ( 1, 2 ] 635257 47.738% 49.156% ########## ( 2, 3 ] 341092 25.632% 74.789% ##### ( 3, 4 ] 141465 10.631% 85.420% ## ( 4, 6 ] 148389 11.151% 96.571% ## ( 6, 10 ] 33151 2.491% 99.062% ( 10, 15 ] 5930 0.446% 99.508% ( 15, 22 ] 827 0.062% 99.570% ( 22, 34 ] 90 0.007% 99.577% ( 34, 51 ] 80 0.006% 99.583% ( 51, 76 ] 118 0.009% 99.592% ( 76, 110 ] 429 0.032% 99.624% ( 110, 170 ] 1385 0.104% 99.728% ( 170, 250 ] 675 0.051% 99.779% ( 250, 380 ] 556 0.042% 99.820% ( 380, 580 ] 720 0.054% 99.875% ( 580, 870 ] 897 0.067% 99.942% ( 870, 1300 ] 971 0.073% 100.015% ( 1300, 1900 ] 751 0.056% 100.071% ( 1900, 2900 ] 648 0.049% 100.120% ( 2900, 4400 ] 745 0.056% 100.176% ( 4400, 6600 ] 237 0.018% 100.194% ( 6600, 9900 ] 51 0.004% 100.198% ( 9900, 14000 ] 11 0.001% 100.198% ( 14000, 22000 ] 3 0.000% 100.199% ( 22000, 33000 ] 3 0.000% 100.199% ** Level 4 read latency histogram (micros): Count: 607857 Average: 34.6843 StdDev: 268.05 Min: 0 Median: 2.0884 Max: 51009 Percentiles: P50: 2.09 P75: 3.40 P99: 728.75 P99.9: 3167.92 P99.99: 4088.91 ------------------------------------------------------ [ 0, 1 ] 12097 1.990% 1.990% ( 1, 2 ] 279331 45.953% 47.944% ######### ( 2, 3 ] 141334 23.251% 71.195% ##### ( 3, 4 ] 58487 9.622% 80.817% ## ( 4, 6 ] 60256 9.913% 90.729% ## ( 6, 10 ] 15629 2.571% 93.301% # ( 10, 15 ] 2561 0.421% 93.722% ( 15, 22 ] 344 0.057% 93.778% ( 22, 34 ] 32 0.005% 93.784% ( 34, 51 ] 32 0.005% 93.789% ( 51, 76 ] 48 0.008% 93.797% ( 76, 110 ] 3871 0.637% 94.434% ( 110, 170 ] 14142 2.327% 96.760% ( 170, 250 ] 6113 1.006% 97.766% ( 250, 380 ] 3047 0.501% 98.267% ( 380, 580 ] 3207 0.528% 98.795% ( 580, 870 ] 2432 0.400% 99.195% ( 870, 1300 ] 1984 0.326% 99.521% ( 1300, 1900 ] 1245 0.205% 99.726% ( 1900, 2900 ] 898 0.148% 99.874% ( 2900, 4400 ] 891 0.147% 100.020% ( 4400, 6600 ] 304 0.050% 100.070% ( 6600, 9900 ] 54 0.009% 100.079% ( 9900, 14000 ] 8 0.001% 100.081% ( 14000, 22000 ] 2 0.000% 100.081% ( 22000, 33000 ] 4 0.001% 100.082% ( 33000, 50000 ] 1 0.000% 100.082% ( 50000, 75000 ] 1 0.000% 100.082% ** Level 5 read latency histogram (micros): Count: 197100 Average: 338.6274 StdDev: 647.00 Min: 75 Median: 161.5973 Max: 47022 Percentiles: P50: 161.60 P75: 270.49 P99: 3799.86 P99.9: 6992.06 P99.99: 9677.33 ------------------------------------------------------ ( 51, 76 ] 2 0.001% 0.001% ( 76, 110 ] 24616 12.489% 12.490% ## ( 110, 170 ] 85972 43.618% 56.109% ######### ( 170, 250 ] 34650 17.580% 73.688% #### ( 250, 380 ] 16404 8.323% 82.011% ## ( 380, 580 ] 15114 7.668% 89.679% ## ( 580, 870 ] 8124 4.122% 93.801% # ( 870, 1300 ] 4566 2.317% 96.118% ( 1300, 1900 ] 2459 1.248% 97.365% ( 1900, 2900 ] 1952 0.990% 98.356% ( 2900, 4400 ] 2117 1.074% 99.430% ( 4400, 6600 ] 901 0.457% 99.887% ( 6600, 9900 ] 218 0.111% 99.997% ( 9900, 14000 ] 10 0.005% 100.003% ( 14000, 22000 ] 4 0.002% 100.005% ( 33000, 50000 ] 2 0.001% 100.006% ** DB Stats ** Uptime(secs): 26.2 total, 3.4 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 187.80 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.336 H:M:S, 28.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 288.03 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 256.035 micros/op 15605 ops/sec 60.172 seconds 938996 operations; 15.1 MB/s (234999 of 234999 found) Microseconds per read: Count: 938996 Average: 256.0381 StdDev: 767.22 Min: 4 Median: 190.7022 Max: 75505 Percentiles: P50: 190.70 P75: 237.96 P99: 1202.48 P99.9: 12604.04 P99.99: 31193.19 ------------------------------------------------------ ( 3, 4 ] 5 0.001% 0.001% ( 4, 6 ] 52 0.006% 0.006% ( 6, 10 ] 279 0.030% 0.036% ( 10, 15 ] 157 0.017% 0.053% ( 15, 22 ] 503 0.054% 0.106% ( 22, 34 ] 15330 1.633% 1.739% ( 34, 51 ] 7260 0.773% 2.512% ( 51, 76 ] 1069 0.114% 2.626% ( 76, 110 ] 526 0.056% 2.682% ( 110, 170 ] 341482 36.367% 39.048% ####### ( 170, 250 ] 397387 42.320% 81.369% ######## ( 250, 380 ] 102592 10.926% 92.295% ## ( 380, 580 ] 41895 4.462% 96.756% # ( 580, 870 ] 16129 1.718% 98.474% ( 870, 1300 ] 6389 0.680% 99.154% ( 1300, 1900 ] 2717 0.289% 99.444% ( 1900, 2900 ] 1707 0.182% 99.625% ( 2900, 4400 ] 1512 0.161% 99.786% ( 4400, 6600 ] 581 0.062% 99.848% ( 6600, 9900 ] 270 0.029% 99.877% ( 9900, 14000 ] 326 0.035% 99.912% ( 14000, 22000 ] 526 0.056% 99.968% ( 22000, 33000 ] 249 0.027% 99.994% ( 33000, 50000 ] 44 0.005% 99.999% ( 50000, 75000 ] 8 0.001% 100.000% ( 75000, 110000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 18890603 rocksdb.bloom.filter.full.positive COUNT : 1121983 rocksdb.bloom.filter.full.true.positive COUNT : 938518 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 478 rocksdb.memtable.miss COUNT : 938518 rocksdb.l0.hit COUNT : 46232 rocksdb.l1.hit COUNT : 3443 rocksdb.l2andup.hit COUNT : 888843 rocksdb.compaction.key.drop.new COUNT : 159482 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5775915 rocksdb.number.keys.read COUNT : 938996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5960744280 rocksdb.bytes.read COUNT : 938996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1634 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 37564607 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 20524830548 rocksdb.compact.write.bytes COUNT : 20357633822 rocksdb.flush.write.bytes COUNT : 5945352851 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 34 rocksdb.number.superversion_acquires COUNT : 1142 rocksdb.number.superversion_releases COUNT : 1095 rocksdb.number.superversion_cleanups COUNT : 285 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 270 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5897221120 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2124800 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 24714387319 rocksdb.non.last.level.read.count COUNT : 5827573 rocksdb.block.checksum.compute.count COUNT : 6105450 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 189.425663 P95 : 499.989410 P99 : 1201.437109 P100 : 75504.000000 COUNT : 938996 SUM : 239361260 rocksdb.db.write.micros P50 : 1.552132 P95 : 4.700504 P99 : 9.951124 P100 : 818101.000000 COUNT : 5775915 SUM : 52572137 rocksdb.compaction.times.micros P50 : 1230434.782609 P95 : 2776470.588235 P99 : 15440000.000000 P100 : 16057689.000000 COUNT : 82 SUM : 129996629 rocksdb.compaction.times.cpu_micros P50 : 881935.483871 P95 : 1814444.444444 P99 : 7094000.000000 P100 : 9245089.000000 COUNT : 82 SUM : 83431803 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 776.354167 P95 : 45750.000000 P99 : 177101.000000 P100 : 177101.000000 COUNT : 85 SUM : 615411 rocksdb.compaction.outfile.sync.micros P50 : 14000.000000 P95 : 186000.000000 P99 : 367866.666667 P100 : 477053.000000 COUNT : 228 SUM : 9658784 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 186.000000 P95 : 4693.333333 P99 : 6732.000000 P100 : 51318.000000 COUNT : 196 SUM : 238249 rocksdb.table.open.io.micros P50 : 9482.865169 P95 : 20733.559322 P99 : 28220.000000 P100 : 38903.000000 COUNT : 1634 SUM : 15749146 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 4.005034 P95 : 12.375638 P99 : 168.928778 P100 : 39970.000000 COUNT : 4970062 SUM : 77369535 rocksdb.read.block.get.micros P50 : 139.443366 P95 : 357.923270 P99 : 1110.028703 P100 : 51014.000000 COUNT : 1133777 SUM : 195522202 rocksdb.write.raw.block.micros P50 : 0.545569 P95 : 1.545793 P99 : 4.414962 P100 : 25720.000000 COUNT : 6370671 SUM : 18553586 rocksdb.numfiles.in.singlecompaction P50 : 3.051020 P95 : 5.242857 P99 : 14.350000 P100 : 20.000000 COUNT : 113 SUM : 394 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.574078 P95 : 1286.928473 P99 : 2693.224327 P100 : 817921.000000 COUNT : 180180 SUM : 37586278 rocksdb.sst.read.micros P50 : 2.315074 P95 : 165.385660 P99 : 454.018463 P100 : 51009.000000 COUNT : 5827657 SUM : 253368242 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 938996 SUM : 938996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5775915 SUM : 5960744280 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 220746.268657 P95 : 787500.000000 P99 : 1092589.000000 P100 : 1092589.000000 COUNT : 85 SUM : 24158990 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 278 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0