[2024-12-15 20:35:11.785355] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 22.11.4 initialization... [2024-12-15 20:35:11.785620] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid704980 ] Set seed to 1734291311778705 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-15 20:35:11.883417] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-15 20:35:11.919498] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-15 20:35:11.919547] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 704980' to capture a snapshot of events at runtime. [2024-12-15 20:35:11.919559] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid704980 for offline analysis/debug. [2024-12-15 20:35:11.919587] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sun Dec 15 20:35:16 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/15-20:35:22 ... thread 0: (1000000,1000000) ops and (150906.0,150906.0) ops/second in (6.626641,6.626641) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/12 1.47 GB 56.7 0.3 0.0 0.3 1.1 0.8 0.0 1.4 51.4 185.2 6.12 3.57 12 0.510 319K 370 0.0 0.0 L1 5/5 324.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/1 128.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 21/8 1.74 GB 1.0 1.1 0.8 0.2 1.1 0.8 0.0 1.3 130.5 130.1 8.54 5.11 2 4.269 784K 0 0.0 0.0 L4 173/7 9.76 GB 1.0 3.4 3.3 0.1 3.4 3.3 0.3 1.0 97.9 97.8 35.40 16.42 33 1.073 105K 0 0.0 0.0 L5 1104/0 90.98 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1320/33 104.39 GB 0.0 4.8 4.1 0.7 5.6 4.9 4.9 7.0 97.8 114.0 50.06 25.11 47 1.065 1210K 370 0.0 0.0 Int 0/0 0.00 KB 0.0 4.8 4.1 0.7 5.6 4.9 4.9 7.0 97.8 114.0 50.06 25.11 47 1.065 1210K 370 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 4.8 4.1 0.7 4.8 4.1 0.0 0.0 108.0 107.8 45.34 22.80 36 1.259 1210K 370 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 173.4 4.72 2.30 11 0.429 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 6.8 total, 6.6 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 5.57 GB write, 834.74 MB/s write, 4.78 GB read, 716.18 MB/s read, 50.1 seconds Interval compaction: 5.57 GB write, 860.76 MB/s write, 4.78 GB read, 738.50 MB/s read, 50.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 22, pending-compaction-bytes-delays: 1, pending-compaction-bytes-stops: 0, total-delays: 1, total-stops: 22, interval: 22 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 271177 Average: 13.0630 StdDev: 165.88 Min: 1 Median: 1.7513 Max: 25087 Percentiles: P50: 1.75 P75: 2.66 P99: 136.94 P99.9: 2170.59 P99.99: 3827.37 ------------------------------------------------------ [ 0, 1 ] 23334 8.605% 8.605% ## ( 1, 2 ] 149422 55.101% 63.706% ########### ( 2, 3 ] 46650 17.203% 80.909% ### ( 3, 4 ] 21159 7.803% 88.711% ## ( 4, 6 ] 21246 7.835% 96.546% ## ( 6, 10 ] 5113 1.885% 98.432% ( 10, 15 ] 1137 0.419% 98.851% ( 15, 22 ] 126 0.046% 98.897% ( 22, 34 ] 12 0.004% 98.902% ( 34, 51 ] 11 0.004% 98.906% ( 51, 76 ] 10 0.004% 98.910% ( 76, 110 ] 80 0.030% 98.939% ( 110, 170 ] 368 0.136% 99.075% ( 170, 250 ] 330 0.122% 99.196% ( 250, 380 ] 328 0.121% 99.317% ( 380, 580 ] 473 0.174% 99.492% ( 580, 870 ] 426 0.157% 99.649% ( 870, 1300 ] 358 0.132% 99.781% ( 1300, 1900 ] 266 0.098% 99.879% ( 1900, 2900 ] 210 0.077% 99.956% ( 2900, 4400 ] 147 0.054% 100.011% ( 4400, 6600 ] 26 0.010% 100.020% ( 6600, 9900 ] 6 0.002% 100.022% ( 9900, 14000 ] 1 0.000% 100.023% ( 14000, 22000 ] 2 0.001% 100.024% ( 22000, 33000 ] 2 0.001% 100.024% ** Level 1 read latency histogram (micros): Count: 65463 Average: 11.2707 StdDev: 142.97 Min: 1 Median: 1.8225 Max: 10355 Percentiles: P50: 1.82 P75: 2.85 P99: 14.18 P99.9: 2372.49 P99.99: 5899.77 ------------------------------------------------------ [ 0, 1 ] 4055 6.194% 6.194% # ( 1, 2 ] 34865 53.259% 59.453% ########### ( 2, 3 ] 11942 18.242% 77.696% #### ( 3, 4 ] 5700 8.707% 86.403% ## ( 4, 6 ] 6195 9.463% 95.866% ## ( 6, 10 ] 1756 2.682% 98.549% # ( 10, 15 ] 353 0.539% 99.088% ( 15, 22 ] 46 0.070% 99.158% ( 22, 34 ] 6 0.009% 99.167% ( 34, 51 ] 7 0.011% 99.178% ( 51, 76 ] 4 0.006% 99.184% ( 76, 110 ] 9 0.014% 99.198% ( 110, 170 ] 43 0.066% 99.264% ( 170, 250 ] 55 0.084% 99.348% ( 250, 380 ] 69 0.105% 99.453% ( 380, 580 ] 89 0.136% 99.589% ( 580, 870 ] 72 0.110% 99.699% ( 870, 1300 ] 75 0.115% 99.814% ( 1300, 1900 ] 40 0.061% 99.875% ( 1900, 2900 ] 35 0.053% 99.928% ( 2900, 4400 ] 35 0.053% 99.982% ( 4400, 6600 ] 8 0.012% 99.994% ( 6600, 9900 ] 3 0.005% 99.998% ( 9900, 14000 ] 1 0.002% 100.000% ** Level 2 read latency histogram (micros): Count: 229795 Average: 13.2934 StdDev: 172.46 Min: 1 Median: 1.8433 Max: 14903 Percentiles: P50: 1.84 P75: 2.87 P99: 13.42 P99.9: 2733.52 P99.99: 5295.68 ------------------------------------------------------ [ 0, 1 ] 13159 5.726% 5.726% # ( 1, 2 ] 120648 52.502% 58.229% ########### ( 2, 3 ] 44251 19.257% 77.486% #### ( 3, 4 ] 21410 9.317% 86.803% ## ( 4, 6 ] 21105 9.184% 95.987% ## ( 6, 10 ] 5970 2.598% 98.585% # ( 10, 15 ] 1396 0.607% 99.192% ( 15, 22 ] 204 0.089% 99.281% ( 22, 34 ] 29 0.013% 99.294% ( 34, 51 ] 9 0.004% 99.298% ( 51, 76 ] 9 0.004% 99.302% ( 76, 110 ] 20 0.009% 99.310% ( 110, 170 ] 57 0.025% 99.335% ( 170, 250 ] 70 0.030% 99.366% ( 250, 380 ] 124 0.054% 99.419% ( 380, 580 ] 189 0.082% 99.502% ( 580, 870 ] 243 0.106% 99.607% ( 870, 1300 ] 279 0.121% 99.729% ( 1300, 1900 ] 224 0.097% 99.826% ( 1900, 2900 ] 203 0.088% 99.915% ( 2900, 4400 ] 149 0.065% 99.980% ( 4400, 6600 ] 59 0.026% 100.005% ( 6600, 9900 ] 11 0.005% 100.010% ( 9900, 14000 ] 1 0.000% 100.010% ( 14000, 22000 ] 2 0.001% 100.011% ** Level 3 read latency histogram (micros): Count: 803945 Average: 15.4197 StdDev: 190.13 Min: 1 Median: 1.9171 Max: 16489 Percentiles: P50: 1.92 P75: 3.01 P99: 12.51 P99.9: 1234.86 P99.99: 1741.96 ------------------------------------------------------ [ 0, 1 ] 30229 3.760% 3.760% # ( 1, 2 ] 405355 50.421% 54.181% ########## ( 2, 3 ] 166229 20.677% 74.857% #### ( 3, 4 ] 86142 10.715% 85.572% ## ( 4, 6 ] 85765 10.668% 96.240% ## ( 6, 10 ] 19439 2.418% 98.658% ( 10, 15 ] 5469 0.680% 99.339% ( 15, 22 ] 813 0.101% 99.440% ( 22, 34 ] 98 0.012% 99.452% ( 34, 51 ] 55 0.007% 99.459% ( 51, 76 ] 42 0.005% 99.464% ( 76, 110 ] 55 0.007% 99.471% ( 110, 170 ] 161 0.020% 99.491% ( 170, 250 ] 277 0.034% 99.525% ( 250, 380 ] 482 0.060% 99.585% ( 380, 580 ] 742 0.092% 99.678% ( 580, 870 ] 940 0.117% 99.795% ( 870, 1300 ] 1003 0.125% 99.919% ( 1300, 1900 ] 776 0.097% 100.016% ( 1900, 2900 ] 842 0.105% 100.121% ( 2900, 4400 ] 802 0.100% 100.220% ( 4400, 6600 ] 233 0.029% 100.249% ( 6600, 9900 ] 58 0.007% 100.256% ( 9900, 14000 ] 2 0.000% 100.257% ( 14000, 22000 ] 5 0.001% 100.257% ** Level 4 read latency histogram (micros): Count: 28432 Average: 196.5336 StdDev: 653.48 Min: 1 Median: 2.3908 Max: 28161 Percentiles: P50: 2.39 P75: 4.48 P99: 3461.44 P99.9: 6374.45 P99.99: 9562.08 ------------------------------------------------------ [ 0, 1 ] 595 2.093% 2.093% ( 1, 2 ] 11178 39.315% 41.408% ######## ( 2, 3 ] 6251 21.986% 63.393% #### ( 3, 4 ] 2726 9.588% 72.981% ## ( 4, 6 ] 2369 8.332% 81.313% ## ( 6, 10 ] 582 2.047% 83.360% ( 10, 15 ] 146 0.514% 83.874% ( 15, 22 ] 15 0.053% 83.927% ( 22, 34 ] 5 0.018% 83.944% ( 34, 51 ] 5 0.018% 83.962% ( 51, 76 ] 5 0.018% 83.979% ( 76, 110 ] 45 0.158% 84.138% ( 110, 170 ] 246 0.865% 85.003% ( 170, 250 ] 342 1.203% 86.206% ( 250, 380 ] 374 1.315% 87.521% ( 380, 580 ] 540 1.899% 89.420% ( 580, 870 ] 690 2.427% 91.847% ( 870, 1300 ] 870 3.060% 94.907% # ( 1300, 1900 ] 620 2.181% 97.088% ( 1900, 2900 ] 440 1.548% 98.635% ( 2900, 4400 ] 277 0.974% 99.610% ( 4400, 6600 ] 92 0.324% 99.933% ( 6600, 9900 ] 18 0.063% 99.996% ( 22000, 33000 ] 1 0.004% 100.000% ** Level 5 read latency histogram (micros): Count: 14709 Average: 1242.3228 StdDev: 1502.78 Min: 77 Median: 671.9753 Max: 37966 Percentiles: P50: 671.98 P75: 1645.68 P99: 6453.24 P99.9: 9842.45 P99.99: 31273.37 ------------------------------------------------------ ( 76, 110 ] 293 1.992% 1.992% ( 110, 170 ] 1457 9.906% 11.897% ## ( 170, 250 ] 1536 10.443% 22.340% ## ( 250, 380 ] 1606 10.918% 33.259% ## ( 380, 580 ] 1858 12.632% 45.890% ### ( 580, 870 ] 1906 12.958% 58.848% ### ( 870, 1300 ] 1655 11.252% 70.100% ## ( 1300, 1900 ] 1251 8.505% 78.605% ## ( 1900, 2900 ] 1220 8.294% 86.899% ## ( 2900, 4400 ] 1275 8.668% 95.567% ## ( 4400, 6600 ] 541 3.678% 99.245% # ( 6600, 9900 ] 98 0.666% 99.912% ( 9900, 14000 ] 6 0.041% 99.952% ( 14000, 22000 ] 3 0.020% 99.973% ( 22000, 33000 ] 3 0.020% 99.993% ( 33000, 50000 ] 1 0.007% 100.000% ** DB Stats ** Uptime(secs): 6.8 total, 6.6 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 143.99 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:2.752 H:M:S, 40.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 148.46 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.752 H:M:S, 41.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/12/15-20:35:27 ... thread 0: (1000000,2000000) ops and (214080.1,177025.8) ops/second in (4.671148,11.297789) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/8 2.57 GB 195.1 1.6 0.0 1.6 3.5 1.9 0.0 1.8 105.5 231.3 15.52 11.86 34 0.456 1663K 1775 0.0 0.0 L1 5/5 324.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 12/0 927.10 MB 0.9 1.3 1.0 0.3 1.3 1.0 0.0 1.3 134.1 133.6 9.90 6.08 3 3.300 1001K 1220 0.0 0.0 L4 171/0 9.73 GB 1.0 4.3 4.2 0.1 4.3 4.2 0.3 1.0 104.9 104.8 41.97 20.91 42 0.999 1041K 0 0.0 0.0 L5 1116/0 91.97 GB 0.9 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1322/13 105.50 GB 0.0 7.2 5.2 2.0 9.1 7.0 5.9 4.8 109.4 138.2 67.39 38.85 79 0.853 3706K 2995 0.0 0.0 Int 0/0 0.00 KB 0.0 7.2 5.2 2.0 9.1 7.0 5.9 4.8 109.4 138.2 67.39 38.85 79 0.853 3706K 2995 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 7.2 5.2 2.0 7.2 5.1 0.0 0.0 126.0 125.7 58.52 33.28 51 1.147 3706K 2995 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 220.0 8.88 5.57 28 0.317 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 11.5 total, 11.3 interval Flush(GB): cumulative 1.907, interval 1.907 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 9.09 GB write, 809.22 MB/s write, 7.20 GB read, 640.57 MB/s read, 67.4 seconds Interval compaction: 9.09 GB write, 824.01 MB/s write, 7.20 GB read, 652.28 MB/s read, 67.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 5, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 5, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 30, pending-compaction-bytes-delays: 1, pending-compaction-bytes-stops: 0, total-delays: 6, total-stops: 30, interval: 35 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 590228 Average: 16.3696 StdDev: 130.77 Min: 0 Median: 1.5955 Max: 25087 Percentiles: P50: 1.60 P75: 2.24 P99: 302.75 P99.9: 1275.88 P99.99: 2612.51 ------------------------------------------------------ [ 0, 1 ] 102324 17.336% 17.336% ### ( 1, 2 ] 323770 54.855% 72.191% ########### ( 2, 3 ] 69933 11.848% 84.040% ## ( 3, 4 ] 30684 5.199% 89.239% # ( 4, 6 ] 27664 4.687% 93.926% # ( 6, 10 ] 6286 1.065% 94.991% ( 10, 15 ] 1440 0.244% 95.235% ( 15, 22 ] 172 0.029% 95.264% ( 22, 34 ] 17 0.003% 95.267% ( 34, 51 ] 17 0.003% 95.269% ( 51, 76 ] 17 0.003% 95.272% ( 76, 110 ] 3020 0.512% 95.784% ( 110, 170 ] 12223 2.071% 97.855% ( 170, 250 ] 5745 0.973% 98.828% ( 250, 380 ] 2503 0.424% 99.252% ( 380, 580 ] 2225 0.377% 99.629% ( 580, 870 ] 1091 0.185% 99.814% ( 870, 1300 ] 539 0.091% 99.905% ( 1300, 1900 ] 320 0.054% 99.960% ( 1900, 2900 ] 254 0.043% 100.003% ( 2900, 4400 ] 187 0.032% 100.034% ( 4400, 6600 ] 34 0.006% 100.040% ( 6600, 9900 ] 7 0.001% 100.041% ( 9900, 14000 ] 1 0.000% 100.042% ( 14000, 22000 ] 2 0.000% 100.042% ( 22000, 33000 ] 2 0.000% 100.042% ** Level 1 read latency histogram (micros): Count: 74385 Average: 13.0547 StdDev: 139.47 Min: 1 Median: 1.9072 Max: 10355 Percentiles: P50: 1.91 P75: 3.21 P99: 201.84 P99.9: 2186.89 P99.99: 5759.48 ------------------------------------------------------ [ 0, 1 ] 4153 5.583% 5.583% # ( 1, 2 ] 36419 48.960% 54.543% ########## ( 2, 3 ] 13607 18.293% 72.836% #### ( 3, 4 ] 7620 10.244% 83.080% ## ( 4, 6 ] 8255 11.098% 94.178% ## ( 6, 10 ] 2270 3.052% 97.229% # ( 10, 15 ] 495 0.665% 97.895% ( 15, 22 ] 130 0.175% 98.070% ( 22, 34 ] 19 0.026% 98.095% ( 34, 51 ] 10 0.013% 98.108% ( 51, 76 ] 4 0.005% 98.114% ( 76, 110 ] 119 0.160% 98.274% ( 110, 170 ] 454 0.610% 98.884% ( 170, 250 ] 219 0.294% 99.179% ( 250, 380 ] 149 0.200% 99.379% ( 380, 580 ] 153 0.206% 99.585% ( 580, 870 ] 101 0.136% 99.720% ( 870, 1300 ] 82 0.110% 99.831% ( 1300, 1900 ] 42 0.056% 99.887% ( 1900, 2900 ] 37 0.050% 99.937% ( 2900, 4400 ] 35 0.047% 99.984% ( 4400, 6600 ] 9 0.012% 99.996% ( 6600, 9900 ] 3 0.004% 100.000% ( 9900, 14000 ] 1 0.001% 100.001% ** Level 2 read latency histogram (micros): Count: 253759 Average: 12.5899 StdDev: 168.21 Min: 1 Median: 1.8243 Max: 14903 Percentiles: P50: 1.82 P75: 2.83 P99: 12.84 P99.9: 2686.54 P99.99: 5364.65 ------------------------------------------------------ [ 0, 1 ] 15831 6.239% 6.239% # ( 1, 2 ] 134718 53.089% 59.328% ########### ( 2, 3 ] 47749 18.817% 78.144% #### ( 3, 4 ] 23007 9.066% 87.211% ## ( 4, 6 ] 22623 8.915% 96.126% ## ( 6, 10 ] 6462 2.547% 98.672% # ( 10, 15 ] 1463 0.577% 99.249% ( 15, 22 ] 217 0.086% 99.334% ( 22, 34 ] 30 0.012% 99.346% ( 34, 51 ] 10 0.004% 99.350% ( 51, 76 ] 10 0.004% 99.354% ( 76, 110 ] 21 0.008% 99.362% ( 110, 170 ] 60 0.024% 99.386% ( 170, 250 ] 70 0.028% 99.414% ( 250, 380 ] 126 0.050% 99.463% ( 380, 580 ] 193 0.076% 99.539% ( 580, 870 ] 246 0.097% 99.636% ( 870, 1300 ] 282 0.111% 99.747% ( 1300, 1900 ] 226 0.089% 99.836% ( 1900, 2900 ] 205 0.081% 99.917% ( 2900, 4400 ] 157 0.062% 99.979% ( 4400, 6600 ] 63 0.025% 100.004% ( 6600, 9900 ] 13 0.005% 100.009% ( 9900, 14000 ] 1 0.000% 100.009% ( 14000, 22000 ] 2 0.001% 100.010% ** Level 3 read latency histogram (micros): Count: 956540 Average: 13.8300 StdDev: 177.82 Min: 1 Median: 1.8943 Max: 16489 Percentiles: P50: 1.89 P75: 2.96 P99: 12.17 P99.9: 1114.23 P99.99: 1617.82 ------------------------------------------------------ [ 0, 1 ] 36663 3.833% 3.833% # ( 1, 2 ] 493825 51.626% 55.459% ########## ( 2, 3 ] 194901 20.376% 75.835% #### ( 3, 4 ] 99264 10.377% 86.212% ## ( 4, 6 ] 97587 10.202% 96.414% ## ( 6, 10 ] 22146 2.315% 98.729% ( 10, 15 ] 5965 0.624% 99.353% ( 15, 22 ] 888 0.093% 99.446% ( 22, 34 ] 104 0.011% 99.457% ( 34, 51 ] 61 0.006% 99.463% ( 51, 76 ] 60 0.006% 99.469% ( 76, 110 ] 163 0.017% 99.486% ( 110, 170 ] 570 0.060% 99.546% ( 170, 250 ] 445 0.047% 99.592% ( 250, 380 ] 572 0.060% 99.652% ( 380, 580 ] 813 0.085% 99.737% ( 580, 870 ] 972 0.102% 99.839% ( 870, 1300 ] 1029 0.108% 99.946% ( 1300, 1900 ] 786 0.082% 100.029% ( 1900, 2900 ] 849 0.089% 100.117% ( 2900, 4400 ] 821 0.086% 100.203% ( 4400, 6600 ] 238 0.025% 100.228% ( 6600, 9900 ] 69 0.007% 100.235% ( 9900, 14000 ] 2 0.000% 100.236% ( 14000, 22000 ] 5 0.001% 100.236% ** Level 4 read latency histogram (micros): Count: 40638 Average: 184.7549 StdDev: 564.39 Min: 1 Median: 2.9786 Max: 28161 Percentiles: P50: 2.98 P75: 136.70 P99: 2897.00 P99.9: 6172.88 P99.99: 9418.55 ------------------------------------------------------ [ 0, 1 ] 705 1.735% 1.735% ( 1, 2 ] 12622 31.060% 32.794% ###### ( 2, 3 ] 7145 17.582% 50.376% #### ( 3, 4 ] 3147 7.744% 58.120% ## ( 4, 6 ] 2860 7.038% 65.158% # ( 6, 10 ] 663 1.631% 66.790% ( 10, 15 ] 170 0.418% 67.208% ( 15, 22 ] 20 0.049% 67.257% ( 22, 34 ] 6 0.015% 67.272% ( 34, 51 ] 5 0.012% 67.284% ( 51, 76 ] 5 0.012% 67.297% ( 76, 110 ] 1180 2.904% 70.200% # ( 110, 170 ] 4383 10.785% 80.986% ## ( 170, 250 ] 2123 5.224% 86.210% # ( 250, 380 ] 1150 2.830% 89.040% # ( 380, 580 ] 1133 2.788% 91.828% # ( 580, 870 ] 895 2.202% 94.030% ( 870, 1300 ] 925 2.276% 96.306% ( 1300, 1900 ] 636 1.565% 97.871% ( 1900, 2900 ] 460 1.132% 99.003% ( 2900, 4400 ] 287 0.706% 99.710% ( 4400, 6600 ] 96 0.236% 99.946% ( 6600, 9900 ] 21 0.052% 99.998% ( 22000, 33000 ] 1 0.002% 100.000% ** Level 5 read latency histogram (micros): Count: 68725 Average: 436.6411 StdDev: 839.92 Min: 73 Median: 169.8929 Max: 37966 Percentiles: P50: 169.89 P75: 347.79 P99: 4375.04 P99.9: 8064.78 P99.99: 12903.25 ------------------------------------------------------ ( 51, 76 ] 3 0.004% 0.004% ( 76, 110 ] 7232 10.523% 10.527% ## ( 110, 170 ] 27176 39.543% 50.071% ######## ( 170, 250 ] 12421 18.073% 68.144% #### ( 250, 380 ] 6264 9.115% 77.259% ## ( 380, 580 ] 5607 8.159% 85.417% ## ( 580, 870 ] 3298 4.799% 90.216% # ( 870, 1300 ] 2025 2.947% 93.163% # ( 1300, 1900 ] 1363 1.983% 95.146% ( 1900, 2900 ] 1334 1.941% 97.087% ( 2900, 4400 ] 1337 1.945% 99.032% ( 4400, 6600 ] 551 0.802% 99.834% ( 6600, 9900 ] 102 0.148% 99.983% ( 9900, 14000 ] 7 0.010% 99.993% ( 14000, 22000 ] 3 0.004% 99.997% ( 22000, 33000 ] 3 0.004% 100.001% ( 33000, 50000 ] 1 0.001% 100.003% ** DB Stats ** Uptime(secs): 11.5 total, 4.7 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 171.07 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.894 H:M:S, 33.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 210.70 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.142 H:M:S, 24.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 2024/12/15-20:35:32 ... thread 0: (1000000,3000000) ops and (198970.4,183782.3) ops/second in (5.025874,16.323663) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/10 2.77 GB 125.9 2.3 0.0 2.3 5.1 2.8 0.0 1.8 113.2 247.1 21.15 17.00 51 0.415 2430K 2506 0.0 0.0 L1 11/0 1005.82 MB 100.6 1.0 0.7 0.3 1.0 0.7 0.0 1.5 68.9 68.6 14.66 6.01 1 14.663 1025K 4357 0.0 0.0 L3 12/0 927.10 MB 0.9 1.3 1.0 0.3 1.3 1.0 0.0 1.3 134.1 133.6 9.90 6.08 3 3.300 1001K 1220 0.0 0.0 L4 171/0 9.73 GB 1.0 4.3 4.2 0.1 4.3 4.2 0.3 1.0 104.9 104.8 41.97 20.91 42 0.999 1041K 0 0.0 0.0 L5 1116/0 91.97 GB 0.9 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1325/10 106.35 GB 0.0 8.9 5.8 3.1 11.7 8.6 5.9 4.2 104.2 136.3 87.68 50.00 97 0.904 5499K 8083 0.0 0.0 Int 0/0 0.00 KB 0.0 8.9 5.8 3.1 11.7 8.6 5.9 4.2 104.2 136.3 87.68 50.00 97 0.904 5499K 8083 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 8.9 5.8 3.1 8.9 5.8 0.0 0.0 120.1 119.9 76.06 42.09 55 1.383 5499K 8083 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 243.9 11.62 7.91 42 0.277 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 16.5 total, 16.3 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 11.67 GB write, 723.08 MB/s write, 8.92 GB read, 552.67 MB/s read, 87.7 seconds Interval compaction: 11.67 GB write, 732.23 MB/s write, 8.92 GB read, 559.67 MB/s read, 87.7 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 11, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 11, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 30, pending-compaction-bytes-delays: 1, pending-compaction-bytes-stops: 0, total-delays: 12, total-stops: 30, interval: 41 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 970981 Average: 14.8725 StdDev: 107.31 Min: 0 Median: 1.5289 Max: 25087 Percentiles: P50: 1.53 P75: 1.98 P99: 246.31 P99.9: 859.08 P99.99: 1763.79 ------------------------------------------------------ [ 0, 1 ] 201935 20.797% 20.797% #### ( 1, 2 ] 536130 55.215% 76.012% ########### ( 2, 3 ] 97420 10.033% 86.045% ## ( 3, 4 ] 40904 4.213% 90.258% # ( 4, 6 ] 34357 3.538% 93.796% # ( 6, 10 ] 7488 0.771% 94.568% ( 10, 15 ] 1825 0.188% 94.756% ( 15, 22 ] 206 0.021% 94.777% ( 22, 34 ] 20 0.002% 94.779% ( 34, 51 ] 17 0.002% 94.781% ( 51, 76 ] 22 0.002% 94.783% ( 76, 110 ] 6255 0.644% 95.427% ( 110, 170 ] 25172 2.592% 98.020% # ( 170, 250 ] 9982 1.028% 99.048% ( 250, 380 ] 3753 0.387% 99.434% ( 380, 580 ] 3249 0.335% 99.769% ( 580, 870 ] 1327 0.137% 99.905% ( 870, 1300 ] 575 0.059% 99.965% ( 1300, 1900 ] 322 0.033% 99.998% ( 1900, 2900 ] 268 0.028% 100.025% ( 2900, 4400 ] 195 0.020% 100.045% ( 4400, 6600 ] 35 0.004% 100.049% ( 6600, 9900 ] 7 0.001% 100.050% ( 9900, 14000 ] 1 0.000% 100.050% ( 14000, 22000 ] 2 0.000% 100.050% ( 22000, 33000 ] 2 0.000% 100.050% ** Level 1 read latency histogram (micros): Count: 84937 Average: 15.1206 StdDev: 134.50 Min: 1 Median: 2.0104 Max: 10355 Percentiles: P50: 2.01 P75: 3.60 P99: 247.77 P99.9: 1976.58 P99.99: 5501.54 ------------------------------------------------------ [ 0, 1 ] 4303 5.066% 5.066% # ( 1, 2 ] 38009 44.750% 49.816% ######### ( 2, 3 ] 15023 17.687% 67.503% #### ( 3, 4 ] 10529 12.396% 79.899% ## ( 4, 6 ] 10648 12.536% 92.436% ### ( 6, 10 ] 2746 3.233% 95.669% # ( 10, 15 ] 623 0.733% 96.402% ( 15, 22 ] 222 0.261% 96.663% ( 22, 34 ] 35 0.041% 96.705% ( 34, 51 ] 12 0.014% 96.719% ( 51, 76 ] 4 0.005% 96.723% ( 76, 110 ] 317 0.373% 97.097% ( 110, 170 ] 1186 1.396% 98.493% ( 170, 250 ] 443 0.522% 99.015% ( 250, 380 ] 257 0.303% 99.317% ( 380, 580 ] 225 0.265% 99.582% ( 580, 870 ] 131 0.154% 99.736% ( 870, 1300 ] 91 0.107% 99.843% ( 1300, 1900 ] 45 0.053% 99.896% ( 1900, 2900 ] 40 0.047% 99.943% ( 2900, 4400 ] 35 0.041% 99.985% ( 4400, 6600 ] 9 0.011% 99.995% ( 6600, 9900 ] 3 0.004% 99.999% ( 9900, 14000 ] 1 0.001% 100.000% ** Level 2 read latency histogram (micros): Count: 253759 Average: 12.5899 StdDev: 168.21 Min: 1 Median: 1.8243 Max: 14903 Percentiles: P50: 1.82 P75: 2.83 P99: 12.84 P99.9: 2686.54 P99.99: 5364.65 ------------------------------------------------------ [ 0, 1 ] 15831 6.239% 6.239% # ( 1, 2 ] 134718 53.089% 59.328% ########### ( 2, 3 ] 47749 18.817% 78.144% #### ( 3, 4 ] 23007 9.066% 87.211% ## ( 4, 6 ] 22623 8.915% 96.126% ## ( 6, 10 ] 6462 2.547% 98.672% # ( 10, 15 ] 1463 0.577% 99.249% ( 15, 22 ] 217 0.086% 99.334% ( 22, 34 ] 30 0.012% 99.346% ( 34, 51 ] 10 0.004% 99.350% ( 51, 76 ] 10 0.004% 99.354% ( 76, 110 ] 21 0.008% 99.362% ( 110, 170 ] 60 0.024% 99.386% ( 170, 250 ] 70 0.028% 99.414% ( 250, 380 ] 126 0.050% 99.463% ( 380, 580 ] 193 0.076% 99.539% ( 580, 870 ] 246 0.097% 99.636% ( 870, 1300 ] 282 0.111% 99.747% ( 1300, 1900 ] 226 0.089% 99.836% ( 1900, 2900 ] 205 0.081% 99.917% ( 2900, 4400 ] 157 0.062% 99.979% ( 4400, 6600 ] 63 0.025% 100.004% ( 6600, 9900 ] 13 0.005% 100.009% ( 9900, 14000 ] 1 0.000% 100.009% ( 14000, 22000 ] 2 0.001% 100.010% ** Level 3 read latency histogram (micros): Count: 957739 Average: 14.0198 StdDev: 177.88 Min: 1 Median: 1.8955 Max: 16489 Percentiles: P50: 1.90 P75: 2.96 P99: 13.04 P99.9: 1115.11 P99.99: 1618.85 ------------------------------------------------------ [ 0, 1 ] 36663 3.828% 3.828% # ( 1, 2 ] 493831 51.562% 55.390% ########## ( 2, 3 ] 194946 20.355% 75.745% #### ( 3, 4 ] 99299 10.368% 86.113% ## ( 4, 6 ] 97644 10.195% 96.308% ## ( 6, 10 ] 22154 2.313% 98.622% ( 10, 15 ] 5965 0.623% 99.244% ( 15, 22 ] 888 0.093% 99.337% ( 22, 34 ] 104 0.011% 99.348% ( 34, 51 ] 61 0.006% 99.354% ( 51, 76 ] 60 0.006% 99.361% ( 76, 110 ] 343 0.036% 99.396% ( 110, 170 ] 1097 0.115% 99.511% ( 170, 250 ] 636 0.066% 99.577% ( 250, 380 ] 636 0.066% 99.644% ( 380, 580 ] 876 0.091% 99.735% ( 580, 870 ] 990 0.103% 99.839% ( 870, 1300 ] 1032 0.108% 99.946% ( 1300, 1900 ] 787 0.082% 100.029% ( 1900, 2900 ] 849 0.089% 100.117% ( 2900, 4400 ] 822 0.086% 100.203% ( 4400, 6600 ] 238 0.025% 100.228% ( 6600, 9900 ] 69 0.007% 100.235% ( 9900, 14000 ] 2 0.000% 100.235% ( 14000, 22000 ] 5 0.001% 100.236% ** Level 4 read latency histogram (micros): Count: 52957 Average: 181.9566 StdDev: 499.42 Min: 1 Median: 5.4473 Max: 28161 Percentiles: P50: 5.45 P75: 158.66 P99: 2639.68 P99.9: 5890.57 P99.99: 9224.96 ------------------------------------------------------ [ 0, 1 ] 705 1.331% 1.331% ( 1, 2 ] 12692 23.967% 25.298% ##### ( 2, 3 ] 7460 14.087% 39.385% ### ( 3, 4 ] 3350 6.326% 45.711% # ( 4, 6 ] 3139 5.927% 51.638% # ( 6, 10 ] 701 1.324% 52.962% ( 10, 15 ] 181 0.342% 53.304% ( 15, 22 ] 20 0.038% 53.341% ( 22, 34 ] 6 0.011% 53.353% ( 34, 51 ] 5 0.009% 53.362% ( 51, 76 ] 7 0.013% 53.375% ( 76, 110 ] 3066 5.790% 59.165% # ( 110, 170 ] 10341 19.527% 78.692% #### ( 170, 250 ] 4071 7.687% 86.380% ## ( 250, 380 ] 1898 3.584% 89.964% # ( 380, 580 ] 1742 3.289% 93.253% # ( 580, 870 ] 1101 2.079% 95.332% ( 870, 1300 ] 958 1.809% 97.141% ( 1300, 1900 ] 639 1.207% 98.348% ( 1900, 2900 ] 467 0.882% 99.230% ( 2900, 4400 ] 290 0.548% 99.777% ( 4400, 6600 ] 96 0.181% 99.958% ( 6600, 9900 ] 21 0.040% 99.998% ( 22000, 33000 ] 1 0.002% 100.000% ** Level 5 read latency histogram (micros): Count: 136606 Average: 311.4397 StdDev: 616.85 Min: 73 Median: 157.4729 Max: 37966 Percentiles: P50: 157.47 P75: 240.20 P99: 3617.23 P99.9: 6489.78 P99.99: 9684.51 ------------------------------------------------------ ( 51, 76 ] 4 0.003% 0.003% ( 76, 110 ] 18678 13.673% 13.676% ### ( 110, 170 ] 62715 45.909% 59.585% ######### ( 170, 250 ] 23997 17.567% 77.152% #### ( 250, 380 ] 10600 7.760% 84.911% ## ( 380, 580 ] 9126 6.681% 91.592% # ( 580, 870 ] 4495 3.290% 94.882% # ( 870, 1300 ] 2220 1.625% 96.507% ( 1300, 1900 ] 1388 1.016% 97.524% ( 1900, 2900 ] 1370 1.003% 98.526% ( 2900, 4400 ] 1353 0.990% 99.517% ( 4400, 6600 ] 551 0.403% 99.920% ( 6600, 9900 ] 102 0.075% 99.995% ( 9900, 14000 ] 7 0.005% 100.000% ( 14000, 22000 ] 3 0.002% 100.002% ( 22000, 33000 ] 3 0.002% 100.004% ( 33000, 50000 ] 1 0.001% 100.005% ** DB Stats ** Uptime(secs): 16.5 total, 5.0 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 178.60 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.471 H:M:S, 33.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 195.83 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.578 H:M:S, 31.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/15-20:35:36 ... thread 0: (1000000,4000000) ops and (266689.9,199269.3) ops/second in (3.749673,20.073336) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/4 3.74 GB 358.0 4.6 0.0 4.6 8.4 3.7 0.0 2.2 142.1 257.4 33.23 28.54 72 0.462 4796K 10K 0.0 0.0 L1 11/0 1005.82 MB 100.6 1.0 0.7 0.3 1.0 0.7 0.0 1.5 68.9 68.6 14.66 6.01 1 14.663 1025K 4357 0.0 0.0 L3 12/0 927.10 MB 0.9 1.3 1.0 0.3 1.3 1.0 0.0 1.3 134.1 133.6 9.90 6.08 3 3.300 1001K 1220 0.0 0.0 L4 171/0 9.73 GB 1.0 4.3 4.2 0.1 4.3 4.2 0.3 1.0 104.9 104.8 41.97 20.91 42 0.999 1041K 0 0.0 0.0 L5 1116/0 91.97 GB 0.9 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1324/4 107.33 GB 0.0 11.2 5.8 5.4 14.9 9.5 5.9 4.0 114.9 153.2 99.77 61.53 118 0.846 7865K 16K 0.0 0.0 Int 0/0 0.00 KB 0.0 11.2 5.8 5.4 14.9 9.5 5.9 4.0 114.9 153.2 99.77 61.53 118 0.846 7865K 16K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 11.2 5.8 5.4 11.2 5.8 0.0 0.0 135.0 134.7 84.91 50.81 60 1.415 7865K 16K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 258.7 14.86 10.73 58 0.256 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 20.3 total, 20.1 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 14.92 GB write, 753.55 MB/s write, 11.20 GB read, 565.36 MB/s read, 99.8 seconds Interval compaction: 14.92 GB write, 761.30 MB/s write, 11.20 GB read, 571.18 MB/s read, 99.8 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 13, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 13, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 30, pending-compaction-bytes-delays: 1, pending-compaction-bytes-stops: 0, total-delays: 14, total-stops: 30, interval: 43 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1422779 Average: 11.6199 StdDev: 90.33 Min: 0 Median: 1.4957 Max: 25087 Percentiles: P50: 1.50 P75: 1.93 P99: 220.62 P99.9: 718.63 P99.99: 1226.01 ------------------------------------------------------ [ 0, 1 ] 306925 21.572% 21.572% #### ( 1, 2 ] 815920 57.347% 78.919% ########### ( 2, 3 ] 135843 9.548% 88.467% ## ( 3, 4 ] 54256 3.813% 92.280% # ( 4, 6 ] 42302 2.973% 95.253% # ( 6, 10 ] 9025 0.634% 95.888% ( 10, 15 ] 2165 0.152% 96.040% ( 15, 22 ] 239 0.017% 96.057% ( 22, 34 ] 24 0.002% 96.058% ( 34, 51 ] 18 0.001% 96.060% ( 51, 76 ] 23 0.002% 96.061% ( 76, 110 ] 6978 0.490% 96.552% ( 110, 170 ] 27765 1.951% 98.503% ( 170, 250 ] 11173 0.785% 99.289% ( 250, 380 ] 4279 0.301% 99.589% ( 380, 580 ] 3709 0.261% 99.850% ( 580, 870 ] 1492 0.105% 99.955% ( 870, 1300 ] 606 0.043% 99.997% ( 1300, 1900 ] 324 0.023% 100.020% ( 1900, 2900 ] 269 0.019% 100.039% ( 2900, 4400 ] 197 0.014% 100.053% ( 4400, 6600 ] 35 0.002% 100.055% ( 6600, 9900 ] 7 0.000% 100.056% ( 9900, 14000 ] 1 0.000% 100.056% ( 14000, 22000 ] 2 0.000% 100.056% ( 22000, 33000 ] 2 0.000% 100.056% ** Level 1 read latency histogram (micros): Count: 86069 Average: 17.2309 StdDev: 135.98 Min: 1 Median: 2.0468 Max: 10355 Percentiles: P50: 2.05 P75: 3.68 P99: 314.93 P99.9: 1948.28 P99.99: 5473.87 ------------------------------------------------------ [ 0, 1 ] 4304 5.001% 5.001% # ( 1, 2 ] 38024 44.179% 49.179% ######### ( 2, 3 ] 15090 17.532% 66.712% #### ( 3, 4 ] 10561 12.270% 78.982% ## ( 4, 6 ] 10697 12.428% 91.410% ## ( 6, 10 ] 2754 3.200% 94.610% # ( 10, 15 ] 625 0.726% 95.336% ( 15, 22 ] 222 0.258% 95.594% ( 22, 34 ] 35 0.041% 95.635% ( 34, 51 ] 12 0.014% 95.649% ( 51, 76 ] 4 0.005% 95.653% ( 76, 110 ] 459 0.533% 96.187% ( 110, 170 ] 1635 1.900% 98.086% ( 170, 250 ] 611 0.710% 98.796% ( 250, 380 ] 351 0.408% 99.204% ( 380, 580 ] 300 0.349% 99.553% ( 580, 870 ] 155 0.180% 99.733% ( 870, 1300 ] 96 0.112% 99.844% ( 1300, 1900 ] 46 0.053% 99.898% ( 1900, 2900 ] 40 0.046% 99.944% ( 2900, 4400 ] 35 0.041% 99.985% ( 4400, 6600 ] 9 0.010% 99.995% ( 6600, 9900 ] 3 0.003% 99.999% ( 9900, 14000 ] 1 0.001% 100.000% ** Level 2 read latency histogram (micros): Count: 253759 Average: 12.5899 StdDev: 168.21 Min: 1 Median: 1.8243 Max: 14903 Percentiles: P50: 1.82 P75: 2.83 P99: 12.84 P99.9: 2686.54 P99.99: 5364.65 ------------------------------------------------------ [ 0, 1 ] 15831 6.239% 6.239% # ( 1, 2 ] 134718 53.089% 59.328% ########### ( 2, 3 ] 47749 18.817% 78.144% #### ( 3, 4 ] 23007 9.066% 87.211% ## ( 4, 6 ] 22623 8.915% 96.126% ## ( 6, 10 ] 6462 2.547% 98.672% # ( 10, 15 ] 1463 0.577% 99.249% ( 15, 22 ] 217 0.086% 99.334% ( 22, 34 ] 30 0.012% 99.346% ( 34, 51 ] 10 0.004% 99.350% ( 51, 76 ] 10 0.004% 99.354% ( 76, 110 ] 21 0.008% 99.362% ( 110, 170 ] 60 0.024% 99.386% ( 170, 250 ] 70 0.028% 99.414% ( 250, 380 ] 126 0.050% 99.463% ( 380, 580 ] 193 0.076% 99.539% ( 580, 870 ] 246 0.097% 99.636% ( 870, 1300 ] 282 0.111% 99.747% ( 1300, 1900 ] 226 0.089% 99.836% ( 1900, 2900 ] 205 0.081% 99.917% ( 2900, 4400 ] 157 0.062% 99.979% ( 4400, 6600 ] 63 0.025% 100.004% ( 6600, 9900 ] 13 0.005% 100.009% ( 9900, 14000 ] 1 0.000% 100.009% ( 14000, 22000 ] 2 0.001% 100.010% ** Level 3 read latency histogram (micros): Count: 958518 Average: 14.1730 StdDev: 177.95 Min: 1 Median: 1.8962 Max: 16489 Percentiles: P50: 1.90 P75: 2.97 P99: 13.68 P99.9: 1116.33 P99.99: 1619.91 ------------------------------------------------------ [ 0, 1 ] 36663 3.825% 3.825% # ( 1, 2 ] 493832 51.520% 55.345% ########## ( 2, 3 ] 194949 20.339% 75.684% #### ( 3, 4 ] 99302 10.360% 86.044% ## ( 4, 6 ] 97648 10.187% 96.231% ## ( 6, 10 ] 22154 2.311% 98.543% ( 10, 15 ] 5965 0.622% 99.165% ( 15, 22 ] 888 0.093% 99.257% ( 22, 34 ] 104 0.011% 99.268% ( 34, 51 ] 61 0.006% 99.275% ( 51, 76 ] 60 0.006% 99.281% ( 76, 110 ] 481 0.050% 99.331% ( 110, 170 ] 1448 0.151% 99.482% ( 170, 250 ] 765 0.080% 99.562% ( 250, 380 ] 698 0.073% 99.635% ( 380, 580 ] 941 0.098% 99.733% ( 580, 870 ] 1007 0.105% 99.838% ( 870, 1300 ] 1036 0.108% 99.946% ( 1300, 1900 ] 788 0.082% 100.028% ( 1900, 2900 ] 850 0.089% 100.117% ( 2900, 4400 ] 822 0.086% 100.203% ( 4400, 6600 ] 238 0.025% 100.228% ( 6600, 9900 ] 69 0.007% 100.235% ( 9900, 14000 ] 2 0.000% 100.235% ( 14000, 22000 ] 5 0.001% 100.236% ** Level 4 read latency histogram (micros): Count: 61125 Average: 183.3339 StdDev: 467.90 Min: 1 Median: 92.0365 Max: 28161 Percentiles: P50: 92.04 P75: 165.09 P99: 2467.55 P99.9: 5703.39 P99.99: 9096.61 ------------------------------------------------------ [ 0, 1 ] 705 1.153% 1.153% ( 1, 2 ] 12707 20.789% 21.942% #### ( 2, 3 ] 7546 12.345% 34.287% ## ( 3, 4 ] 3411 5.580% 39.867% # ( 4, 6 ] 3223 5.273% 45.140% # ( 6, 10 ] 719 1.176% 46.317% ( 10, 15 ] 181 0.296% 46.613% ( 15, 22 ] 21 0.034% 46.647% ( 22, 34 ] 6 0.010% 46.657% ( 34, 51 ] 5 0.008% 46.665% ( 51, 76 ] 8 0.013% 46.678% ( 76, 110 ] 4305 7.043% 53.721% # ( 110, 170 ] 14167 23.177% 76.898% ##### ( 170, 250 ] 5480 8.965% 85.863% ## ( 250, 380 ] 2541 4.157% 90.020% # ( 380, 580 ] 2299 3.761% 93.782% # ( 580, 870 ] 1298 2.124% 95.905% ( 870, 1300 ] 978 1.600% 97.505% ( 1300, 1900 ] 647 1.058% 98.564% ( 1900, 2900 ] 470 0.769% 99.333% ( 2900, 4400 ] 290 0.474% 99.807% ( 4400, 6600 ] 96 0.157% 99.964% ( 6600, 9900 ] 21 0.034% 99.998% ( 22000, 33000 ] 1 0.002% 100.000% ** Level 5 read latency histogram (micros): Count: 181997 Average: 284.8437 StdDev: 541.58 Min: 73 Median: 156.8994 Max: 37966 Percentiles: P50: 156.90 P75: 236.93 P99: 3115.74 P99.9: 6304.55 P99.99: 9505.30 ------------------------------------------------------ ( 51, 76 ] 4 0.002% 0.002% ( 76, 110 ] 25655 14.096% 14.099% ### ( 110, 170 ] 83591 45.930% 60.028% ######### ( 170, 250 ] 32570 17.896% 77.924% #### ( 250, 380 ] 14622 8.034% 85.959% ## ( 380, 580 ] 12656 6.954% 92.913% # ( 580, 870 ] 5684 3.123% 96.036% # ( 870, 1300 ] 2401 1.319% 97.355% ( 1300, 1900 ] 1411 0.775% 98.130% ( 1900, 2900 ] 1388 0.763% 98.893% ( 2900, 4400 ] 1356 0.745% 99.638% ( 4400, 6600 ] 551 0.303% 99.941% ( 6600, 9900 ] 102 0.056% 99.997% ( 9900, 14000 ] 7 0.004% 100.001% ( 14000, 22000 ] 3 0.002% 100.002% ( 22000, 33000 ] 3 0.002% 100.004% ( 33000, 50000 ] 1 0.001% 100.004% ** DB Stats ** Uptime(secs): 20.3 total, 3.7 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 194.10 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.784 H:M:S, 28.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 262.47 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.312 H:M:S, 8.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/12/15-20:35:55 ... thread 0: (1000000,5000000) ops and (52470.0,127773.2) ops/second in (19.058500,39.131836) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/17 4.79 GB 12.6 5.6 0.0 5.6 10.4 4.8 0.0 2.2 142.4 264.2 40.25 35.08 93 0.433 5819K 11K 0.0 0.0 L1 11/11 1005.82 MB 0.0 1.0 0.7 0.3 1.0 0.7 0.0 1.5 68.9 68.6 14.66 6.01 1 14.663 1025K 4357 0.0 0.0 L3 12/0 927.10 MB 0.9 1.3 1.0 0.3 1.3 1.0 0.0 1.3 134.1 133.6 9.90 6.08 3 3.300 1001K 1220 0.0 0.0 L4 171/0 9.73 GB 1.0 4.3 4.2 0.1 4.3 4.2 0.3 1.0 104.9 104.8 41.97 20.91 42 0.999 1041K 0 0.0 0.0 L5 1116/0 91.97 GB 0.9 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1329/28 108.37 GB 0.0 12.2 5.8 6.4 17.0 10.6 5.9 3.5 116.8 162.6 106.79 68.08 139 0.768 8888K 17K 0.0 0.0 Int 0/0 0.00 KB 0.0 12.2 5.8 6.4 17.0 10.6 5.9 3.5 116.8 162.6 106.79 68.08 139 0.768 8888K 17K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.2 5.8 6.4 12.2 5.8 0.0 0.0 140.6 140.3 88.73 54.53 64 1.386 8888K 17K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.8 4.8 0.0 0.0 0.0 272.0 18.06 13.55 75 0.241 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 39.3 total, 39.1 interval Flush(GB): cumulative 4.799, interval 4.799 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 16.95 GB write, 441.30 MB/s write, 12.18 GB read, 317.09 MB/s read, 106.8 seconds Interval compaction: 16.95 GB write, 443.63 MB/s write, 12.18 GB read, 318.76 MB/s read, 106.8 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 15, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 15, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 30, pending-compaction-bytes-delays: 6, pending-compaction-bytes-stops: 0, total-delays: 21, total-stops: 30, interval: 50 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1937733 Average: 17.4930 StdDev: 87.22 Min: 0 Median: 1.5518 Max: 25087 Percentiles: P50: 1.55 P75: 2.13 P99: 226.01 P99.9: 655.54 P99.99: 1178.56 ------------------------------------------------------ [ 0, 1 ] 402714 20.783% 20.783% #### ( 1, 2 ] 1025923 52.944% 73.727% ########### ( 2, 3 ] 183282 9.459% 83.186% ## ( 3, 4 ] 76731 3.960% 87.146% # ( 4, 6 ] 62509 3.226% 90.372% # ( 6, 10 ] 11161 0.576% 90.948% ( 10, 15 ] 2704 0.140% 91.087% ( 15, 22 ] 282 0.015% 91.102% ( 22, 34 ] 27 0.001% 91.103% ( 34, 51 ] 20 0.001% 91.104% ( 51, 76 ] 50 0.003% 91.107% ( 76, 110 ] 37354 1.928% 93.034% ( 110, 170 ] 100463 5.185% 98.219% # ( 170, 250 ] 21620 1.116% 99.335% ( 250, 380 ] 5860 0.302% 99.637% ( 380, 580 ] 4642 0.240% 99.877% ( 580, 870 ] 1744 0.090% 99.967% ( 870, 1300 ] 633 0.033% 99.999% ( 1300, 1900 ] 334 0.017% 100.017% ( 1900, 2900 ] 361 0.019% 100.035% ( 2900, 4400 ] 225 0.012% 100.047% ( 4400, 6600 ] 35 0.002% 100.049% ( 6600, 9900 ] 7 0.000% 100.049% ( 9900, 14000 ] 1 0.000% 100.049% ( 14000, 22000 ] 2 0.000% 100.049% ( 22000, 33000 ] 2 0.000% 100.049% ** Level 1 read latency histogram (micros): Count: 130804 Average: 20.8981 StdDev: 116.55 Min: 1 Median: 1.9932 Max: 10355 Percentiles: P50: 1.99 P75: 3.54 P99: 255.69 P99.9: 1367.77 P99.99: 4353.70 ------------------------------------------------------ [ 0, 1 ] 7701 5.887% 5.887% # ( 1, 2 ] 58095 44.414% 50.301% ######### ( 2, 3 ] 26162 20.001% 70.302% #### ( 3, 4 ] 11389 8.707% 79.009% ## ( 4, 6 ] 11395 8.712% 87.721% ## ( 6, 10 ] 3220 2.462% 90.182% ( 10, 15 ] 806 0.616% 90.798% ( 15, 22 ] 240 0.183% 90.982% ( 22, 34 ] 35 0.027% 91.009% ( 34, 51 ] 12 0.009% 91.018% ( 51, 76 ] 4 0.003% 91.021% ( 76, 110 ] 2359 1.803% 92.824% ( 110, 170 ] 6637 5.074% 97.898% # ( 170, 250 ] 1419 1.085% 98.983% ( 250, 380 ] 502 0.384% 99.367% ( 380, 580 ] 404 0.309% 99.676% ( 580, 870 ] 188 0.144% 99.820% ( 870, 1300 ] 100 0.076% 99.896% ( 1300, 1900 ] 46 0.035% 99.931% ( 1900, 2900 ] 43 0.033% 99.964% ( 2900, 4400 ] 35 0.027% 99.991% ( 4400, 6600 ] 9 0.007% 99.998% ( 6600, 9900 ] 3 0.002% 100.000% ( 9900, 14000 ] 1 0.001% 100.001% ** Level 2 read latency histogram (micros): Count: 253759 Average: 12.5899 StdDev: 168.21 Min: 1 Median: 1.8243 Max: 14903 Percentiles: P50: 1.82 P75: 2.83 P99: 12.84 P99.9: 2686.54 P99.99: 5364.65 ------------------------------------------------------ [ 0, 1 ] 15831 6.239% 6.239% # ( 1, 2 ] 134718 53.089% 59.328% ########### ( 2, 3 ] 47749 18.817% 78.144% #### ( 3, 4 ] 23007 9.066% 87.211% ## ( 4, 6 ] 22623 8.915% 96.126% ## ( 6, 10 ] 6462 2.547% 98.672% # ( 10, 15 ] 1463 0.577% 99.249% ( 15, 22 ] 217 0.086% 99.334% ( 22, 34 ] 30 0.012% 99.346% ( 34, 51 ] 10 0.004% 99.350% ( 51, 76 ] 10 0.004% 99.354% ( 76, 110 ] 21 0.008% 99.362% ( 110, 170 ] 60 0.024% 99.386% ( 170, 250 ] 70 0.028% 99.414% ( 250, 380 ] 126 0.050% 99.463% ( 380, 580 ] 193 0.076% 99.539% ( 580, 870 ] 246 0.097% 99.636% ( 870, 1300 ] 282 0.111% 99.747% ( 1300, 1900 ] 226 0.089% 99.836% ( 1900, 2900 ] 205 0.081% 99.917% ( 2900, 4400 ] 157 0.062% 99.979% ( 4400, 6600 ] 63 0.025% 100.004% ( 6600, 9900 ] 13 0.005% 100.009% ( 9900, 14000 ] 1 0.000% 100.009% ( 14000, 22000 ] 2 0.001% 100.010% ** Level 3 read latency histogram (micros): Count: 964132 Average: 14.9263 StdDev: 177.78 Min: 1 Median: 1.9019 Max: 16489 Percentiles: P50: 1.90 P75: 2.99 P99: 122.69 P99.9: 1115.54 P99.99: 1620.19 ------------------------------------------------------ [ 0, 1 ] 36663 3.803% 3.803% # ( 1, 2 ] 493832 51.220% 55.023% ########## ( 2, 3 ] 194949 20.220% 75.243% #### ( 3, 4 ] 99302 10.300% 85.543% ## ( 4, 6 ] 97648 10.128% 95.671% ## ( 6, 10 ] 22154 2.298% 97.969% ( 10, 15 ] 5965 0.619% 98.587% ( 15, 22 ] 888 0.092% 98.680% ( 22, 34 ] 104 0.011% 98.690% ( 34, 51 ] 61 0.006% 98.697% ( 51, 76 ] 60 0.006% 98.703% ( 76, 110 ] 1800 0.187% 98.890% ( 110, 170 ] 5034 0.522% 99.412% ( 170, 250 ] 1282 0.133% 99.545% ( 250, 380 ] 793 0.082% 99.627% ( 380, 580 ] 1015 0.105% 99.732% ( 580, 870 ] 1024 0.106% 99.838% ( 870, 1300 ] 1040 0.108% 99.946% ( 1300, 1900 ] 790 0.082% 100.028% ( 1900, 2900 ] 850 0.088% 100.116% ( 2900, 4400 ] 822 0.085% 100.202% ( 4400, 6600 ] 238 0.025% 100.226% ( 6600, 9900 ] 69 0.007% 100.233% ( 9900, 14000 ] 2 0.000% 100.234% ( 14000, 22000 ] 5 0.001% 100.234% ** Level 4 read latency histogram (micros): Count: 117466 Average: 164.7798 StdDev: 342.56 Min: 1 Median: 125.2195 Max: 28161 Percentiles: P50: 125.22 P75: 160.69 P99: 1645.90 P99.9: 4412.24 P99.99: 8211.25 ------------------------------------------------------ [ 0, 1 ] 705 0.600% 0.600% ( 1, 2 ] 12707 10.818% 11.418% ## ( 2, 3 ] 7549 6.427% 17.844% # ( 3, 4 ] 3419 2.911% 20.755% # ( 4, 6 ] 3232 2.751% 23.506% # ( 6, 10 ] 720 0.613% 24.119% ( 10, 15 ] 181 0.154% 24.273% ( 15, 22 ] 21 0.018% 24.291% ( 22, 34 ] 6 0.005% 24.296% ( 34, 51 ] 5 0.004% 24.301% ( 51, 76 ] 21 0.018% 24.319% ( 76, 110 ] 17566 14.954% 39.273% ### ( 110, 170 ] 49677 42.291% 81.563% ######## ( 170, 250 ] 11036 9.395% 90.958% ## ( 250, 380 ] 3524 3.000% 93.958% # ( 380, 580 ] 3066 2.610% 96.568% # ( 580, 870 ] 1463 1.245% 97.814% ( 870, 1300 ] 1014 0.863% 98.677% ( 1300, 1900 ] 658 0.560% 99.237% ( 1900, 2900 ] 487 0.415% 99.652% ( 2900, 4400 ] 291 0.248% 99.900% ( 4400, 6600 ] 96 0.082% 99.981% ( 6600, 9900 ] 21 0.018% 99.999% ( 22000, 33000 ] 1 0.001% 100.000% ** Level 5 read latency histogram (micros): Count: 501228 Average: 195.4679 StdDev: 338.71 Min: 73 Median: 141.7131 Max: 37966 Percentiles: P50: 141.71 P75: 168.00 P99: 1287.22 P99.9: 5021.96 P99.99: 8407.79 ------------------------------------------------------ ( 51, 76 ] 40 0.008% 0.008% ( 76, 110 ] 99392 19.830% 19.838% #### ( 110, 170 ] 286031 57.066% 76.904% ########### ( 170, 250 ] 64366 12.842% 89.745% ### ( 250, 380 ] 20447 4.079% 93.825% # ( 380, 580 ] 16801 3.352% 97.177% # ( 580, 870 ] 6681 1.333% 98.510% ( 870, 1300 ] 2533 0.505% 99.015% ( 1300, 1900 ] 1457 0.291% 99.306% ( 1900, 2900 ] 1454 0.290% 99.596% ( 2900, 4400 ] 1369 0.273% 99.869% ( 4400, 6600 ] 551 0.110% 99.979% ( 6600, 9900 ] 102 0.020% 99.999% ( 9900, 14000 ] 7 0.001% 100.001% ( 14000, 22000 ] 3 0.001% 100.001% ( 22000, 33000 ] 3 0.001% 100.002% ( 33000, 50000 ] 1 0.000% 100.002% ** DB Stats ** Uptime(secs): 39.3 total, 19.1 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 125.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:20.812 H:M:S, 52.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 51.64 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:15.029 H:M:S, 78.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 223.613 micros/op 17873 ops/sec 60.088 seconds 1073996 operations; 17.3 MB/s (261999 of 261999 found) Microseconds per read: Count: 1073996 Average: 223.6143 StdDev: 611.77 Min: 2 Median: 172.4892 Max: 61795 Percentiles: P50: 172.49 P75: 229.56 P99: 812.88 P99.9: 8930.26 P99.99: 28033.56 ------------------------------------------------------ ( 1, 2 ] 12 0.001% 0.001% ( 2, 3 ] 49 0.005% 0.006% ( 3, 4 ] 69 0.006% 0.012% ( 4, 6 ] 108 0.010% 0.022% ( 6, 10 ] 386 0.036% 0.058% ( 10, 15 ] 3136 0.292% 0.350% ( 15, 22 ] 4874 0.454% 0.804% ( 22, 34 ] 10962 1.021% 1.825% ( 34, 51 ] 2918 0.272% 2.096% ( 51, 76 ] 470 0.044% 2.140% ( 76, 110 ] 19107 1.779% 3.919% ( 110, 170 ] 483195 44.990% 48.909% ######### ( 170, 250 ] 376403 35.047% 83.956% ####### ( 250, 380 ] 109467 10.192% 94.149% ## ( 380, 580 ] 41071 3.824% 97.973% # ( 580, 870 ] 13734 1.279% 99.252% ( 870, 1300 ] 3904 0.364% 99.615% ( 1300, 1900 ] 1094 0.102% 99.717% ( 1900, 2900 ] 927 0.086% 99.804% ( 2900, 4400 ] 613 0.057% 99.861% ( 4400, 6600 ] 250 0.023% 99.884% ( 6600, 9900 ] 245 0.023% 99.907% ( 9900, 14000 ] 332 0.031% 99.938% ( 14000, 22000 ] 471 0.044% 99.981% ( 22000, 33000 ] 167 0.016% 99.997% ( 33000, 50000 ] 27 0.003% 100.000% ( 50000, 75000 ] 5 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 21261120 rocksdb.bloom.filter.full.positive COUNT : 1280562 rocksdb.bloom.filter.full.true.positive COUNT : 1073515 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 481 rocksdb.memtable.miss COUNT : 1073515 rocksdb.l0.hit COUNT : 47067 rocksdb.l1.hit COUNT : 9651 rocksdb.l2andup.hit COUNT : 1016797 rocksdb.compaction.key.drop.new COUNT : 73624 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5064630 rocksdb.number.keys.read COUNT : 1073996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5226698160 rocksdb.bytes.read COUNT : 1073996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1515 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 41068960 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 14945214113 rocksdb.compact.write.bytes COUNT : 14872277403 rocksdb.flush.write.bytes COUNT : 5218735429 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 40 rocksdb.number.superversion_acquires COUNT : 870 rocksdb.number.superversion_releases COUNT : 840 rocksdb.number.superversion_cleanups COUNT : 225 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 185 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5176419328 rocksdb.memtable.garbage.bytes.at.flush COUNT : 1808384 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 20058842512 rocksdb.non.last.level.read.count COUNT : 4704286 rocksdb.block.checksum.compute.count COUNT : 4914420 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 170.879734 P95 : 422.915901 P99 : 812.090140 P100 : 61792.000000 COUNT : 1073996 SUM : 239215115 rocksdb.db.write.micros P50 : 1.570563 P95 : 4.685751 P99 : 11.192623 P100 : 732650.000000 COUNT : 5064630 SUM : 54054712 rocksdb.compaction.times.micros P50 : 1023200.000000 P95 : 2700000.000000 P99 : 14663433.000000 P100 : 14663433.000000 COUNT : 64 SUM : 88726405 rocksdb.compaction.times.cpu_micros P50 : 528292.682927 P95 : 1830000.000000 P99 : 5796000.000000 P100 : 6006106.000000 COUNT : 64 SUM : 54528668 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 731.904762 P95 : 3200.000000 P99 : 132941.000000 P100 : 132941.000000 COUNT : 76 SUM : 325887 rocksdb.compaction.outfile.sync.micros P50 : 11950.000000 P95 : 98100.000000 P99 : 287700.000000 P100 : 291772.000000 COUNT : 142 SUM : 4381148 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 151.020408 P95 : 1813.750000 P99 : 4077.500000 P100 : 6839.000000 COUNT : 143 SUM : 65865 rocksdb.table.open.io.micros P50 : 8832.106599 P95 : 20118.181818 P99 : 26056.250000 P100 : 28293.000000 COUNT : 1515 SUM : 13805886 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.807726 P95 : 112.779948 P99 : 206.188770 P100 : 25092.000000 COUNT : 3620042 SUM : 69296882 rocksdb.read.block.get.micros P50 : 136.694117 P95 : 286.966282 P99 : 711.807603 P100 : 37975.000000 COUNT : 1292841 SUM : 200091401 rocksdb.write.raw.block.micros P50 : 0.530803 P95 : 1.206986 P99 : 3.980126 P100 : 39086.000000 COUNT : 4865844 SUM : 12635601 rocksdb.numfiles.in.singlecompaction P50 : 1.988095 P95 : 3.992857 P99 : 23.800000 P100 : 28.000000 COUNT : 85 SUM : 288 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.590715 P95 : 2043.657934 P99 : 2732.132477 P100 : 732475.000000 COUNT : 174098 SUM : 41087724 rocksdb.sst.read.micros P50 : 2.173370 P95 : 174.473081 P99 : 455.093357 P100 : 37966.000000 COUNT : 4704314 SUM : 254931916 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1073996 SUM : 1073996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5064630 SUM : 5226698160 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 215373.134328 P95 : 589333.333333 P99 : 941600.000000 P100 : 970449.000000 COUNT : 76 SUM : 18301646 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 187 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0