[2024-12-14 03:45:38.904327] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 22.11.4 initialization... [2024-12-14 03:45:38.904601] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid713213 ] Set seed to 1734144338897702 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-14 03:45:39.000815] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-14 03:45:39.040878] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-14 03:45:39.040921] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 713213' to capture a snapshot of events at runtime. [2024-12-14 03:45:39.040933] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid713213 for offline analysis/debug. [2024-12-14 03:45:39.040963] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sat Dec 14 03:45:43 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/14-03:45:50 ... thread 0: (1000000,1000000) ops and (131844.5,131844.5) ops/second in (7.584692,7.584692) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 11/9 1.32 GB 50.3 0.4 0.0 0.4 1.2 0.8 0.0 1.5 41.1 130.2 9.19 3.88 10 0.919 383K 524 0.0 0.0 L1 2/2 257.56 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 2/1 257.56 MB 1.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 44/9 3.08 GB 2.4 3.3 2.1 1.2 3.2 2.0 0.0 1.6 100.9 99.9 33.04 15.41 14 2.360 1762K 0 0.0 0.0 L4 188/14 11.64 GB 1.1 2.3 1.9 0.4 2.3 1.9 0.0 1.2 113.6 112.8 21.12 11.01 18 1.173 1517K 0 0.0 0.0 L5 1096/0 90.12 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1343/35 106.67 GB 0.0 6.0 4.0 2.0 6.7 4.7 0.0 8.4 96.5 108.6 63.35 30.30 42 1.508 3663K 524 0.0 0.0 Int 0/0 0.00 KB 0.0 6.0 4.0 2.0 6.7 4.7 0.0 8.4 96.5 108.6 63.35 30.30 42 1.508 3663K 524 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.0 4.0 2.0 5.9 3.9 0.0 0.0 108.1 107.1 56.58 27.94 33 1.714 3663K 524 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 120.9 6.77 2.36 9 0.752 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.8 total, 7.6 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.72 GB write, 881.23 MB/s write, 5.97 GB read, 783.24 MB/s read, 63.3 seconds Interval compaction: 6.72 GB write, 906.67 MB/s write, 5.97 GB read, 805.85 MB/s read, 63.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 26, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 26, interval: 26 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 301875 Average: 12.9889 StdDev: 184.42 Min: 1 Median: 1.8473 Max: 26979 Percentiles: P50: 1.85 P75: 2.91 P99: 11.54 P99.9: 2425.92 P99.99: 4191.93 ------------------------------------------------------ [ 0, 1 ] 17920 5.936% 5.936% # ( 1, 2 ] 156985 52.003% 57.940% ########## ( 2, 3 ] 56602 18.750% 76.690% #### ( 3, 4 ] 30080 9.964% 86.654% ## ( 4, 6 ] 30514 10.108% 96.762% ## ( 6, 10 ] 6308 2.090% 98.852% ( 10, 15 ] 1450 0.480% 99.332% ( 15, 22 ] 158 0.052% 99.385% ( 22, 34 ] 15 0.005% 99.389% ( 34, 51 ] 11 0.004% 99.393% ( 51, 76 ] 21 0.007% 99.400% ( 76, 110 ] 9 0.003% 99.403% ( 110, 170 ] 18 0.006% 99.409% ( 170, 250 ] 41 0.014% 99.423% ( 250, 380 ] 94 0.031% 99.454% ( 380, 580 ] 214 0.071% 99.525% ( 580, 870 ] 345 0.114% 99.639% ( 870, 1300 ] 404 0.134% 99.773% ( 1300, 1900 ] 270 0.089% 99.862% ( 1900, 2900 ] 217 0.072% 99.934% ( 2900, 4400 ] 196 0.065% 99.999% ( 4400, 6600 ] 77 0.026% 100.025% ( 6600, 9900 ] 14 0.005% 100.029% ( 14000, 22000 ] 5 0.002% 100.031% ( 22000, 33000 ] 2 0.001% 100.031% ** Level 1 read latency histogram (micros): Count: 65441 Average: 12.8176 StdDev: 153.63 Min: 1 Median: 1.9346 Max: 7580 Percentiles: P50: 1.93 P75: 3.01 P99: 13.83 P99.9: 2449.21 P99.99: 6169.46 ------------------------------------------------------ [ 0, 1 ] 2447 3.739% 3.739% # ( 1, 2 ] 32392 49.498% 53.237% ########## ( 2, 3 ] 14201 21.700% 74.938% #### ( 3, 4 ] 6832 10.440% 85.378% ## ( 4, 6 ] 6893 10.533% 95.911% ## ( 6, 10 ] 1692 2.586% 98.496% # ( 10, 15 ] 430 0.657% 99.153% ( 15, 22 ] 66 0.101% 99.254% ( 22, 34 ] 9 0.014% 99.268% ( 34, 51 ] 1 0.002% 99.270% ( 51, 76 ] 2 0.003% 99.273% ( 76, 110 ] 5 0.008% 99.280% ( 110, 170 ] 7 0.011% 99.291% ( 170, 250 ] 16 0.024% 99.315% ( 250, 380 ] 31 0.047% 99.363% ( 380, 580 ] 56 0.086% 99.448% ( 580, 870 ] 85 0.130% 99.578% ( 870, 1300 ] 106 0.162% 99.740% ( 1300, 1900 ] 76 0.116% 99.856% ( 1900, 2900 ] 52 0.079% 99.936% ( 2900, 4400 ] 25 0.038% 99.974% ( 4400, 6600 ] 13 0.020% 99.994% ( 6600, 9900 ] 4 0.006% 100.000% ** Level 2 read latency histogram (micros): Count: 369911 Average: 14.0516 StdDev: 168.47 Min: 1 Median: 1.9150 Max: 8430 Percentiles: P50: 1.92 P75: 3.02 P99: 13.28 P99.9: 1778.18 P99.99: 2622.74 ------------------------------------------------------ [ 0, 1 ] 17401 4.704% 4.704% # ( 1, 2 ] 183114 49.502% 54.206% ########## ( 2, 3 ] 76036 20.555% 74.761% #### ( 3, 4 ] 39229 10.605% 85.366% ## ( 4, 6 ] 38645 10.447% 95.814% ## ( 6, 10 ] 10070 2.722% 98.536% # ( 10, 15 ] 2620 0.708% 99.244% ( 15, 22 ] 433 0.117% 99.361% ( 22, 34 ] 54 0.015% 99.376% ( 34, 51 ] 28 0.008% 99.383% ( 51, 76 ] 13 0.004% 99.387% ( 76, 110 ] 16 0.004% 99.391% ( 110, 170 ] 36 0.010% 99.401% ( 170, 250 ] 50 0.014% 99.414% ( 250, 380 ] 164 0.044% 99.459% ( 380, 580 ] 280 0.076% 99.534% ( 580, 870 ] 432 0.117% 99.651% ( 870, 1300 ] 575 0.155% 99.807% ( 1300, 1900 ] 433 0.117% 99.924% ( 1900, 2900 ] 339 0.092% 100.015% ( 2900, 4400 ] 257 0.069% 100.085% ( 4400, 6600 ] 91 0.025% 100.109% ( 6600, 9900 ] 15 0.004% 100.114% ** Level 3 read latency histogram (micros): Count: 787821 Average: 11.9930 StdDev: 183.28 Min: 1 Median: 1.9175 Max: 42164 Percentiles: P50: 1.92 P75: 2.97 P99: 9.93 P99.9: 1031.67 P99.99: 1493.25 ------------------------------------------------------ [ 0, 1 ] 25784 3.273% 3.273% # ( 1, 2 ] 401210 50.927% 54.199% ########## ( 2, 3 ] 168124 21.340% 75.540% #### ( 3, 4 ] 85139 10.807% 86.347% ## ( 4, 6 ] 83208 10.562% 96.908% ## ( 6, 10 ] 16783 2.130% 99.039% ( 10, 15 ] 4353 0.553% 99.591% ( 15, 22 ] 499 0.063% 99.655% ( 22, 34 ] 48 0.006% 99.661% ( 34, 51 ] 27 0.003% 99.664% ( 51, 76 ] 56 0.007% 99.671% ( 76, 110 ] 19 0.002% 99.674% ( 110, 170 ] 37 0.005% 99.678% ( 170, 250 ] 115 0.015% 99.693% ( 250, 380 ] 176 0.022% 99.715% ( 380, 580 ] 428 0.054% 99.770% ( 580, 870 ] 719 0.091% 99.861% ( 870, 1300 ] 825 0.105% 99.966% ( 1300, 1900 ] 603 0.077% 100.042% ( 1900, 2900 ] 545 0.069% 100.111% ( 2900, 4400 ] 529 0.067% 100.178% ( 4400, 6600 ] 152 0.019% 100.198% ( 6600, 9900 ] 39 0.005% 100.203% ( 9900, 14000 ] 2 0.000% 100.203% ( 14000, 22000 ] 6 0.001% 100.204% ( 22000, 33000 ] 5 0.001% 100.204% ( 33000, 50000 ] 1 0.000% 100.204% ** Level 4 read latency histogram (micros): Count: 203454 Average: 33.5010 StdDev: 319.11 Min: 1 Median: 1.9378 Max: 37309 Percentiles: P50: 1.94 P75: 3.10 P99: 982.55 P99.9: 3653.57 P99.99: 4279.23 ------------------------------------------------------ [ 0, 1 ] 6889 3.386% 3.386% # ( 1, 2 ] 101130 49.707% 53.093% ########## ( 2, 3 ] 42433 20.856% 73.949% #### ( 3, 4 ] 22307 10.964% 84.913% ## ( 4, 6 ] 21325 10.481% 95.395% ## ( 6, 10 ] 4482 2.203% 97.597% ( 10, 15 ] 1080 0.531% 98.128% ( 15, 22 ] 120 0.059% 98.187% ( 22, 34 ] 13 0.006% 98.194% ( 34, 51 ] 15 0.007% 98.201% ( 51, 76 ] 26 0.013% 98.214% ( 76, 110 ] 4 0.002% 98.216% ( 110, 170 ] 26 0.013% 98.229% ( 170, 250 ] 100 0.049% 98.278% ( 250, 380 ] 158 0.078% 98.355% ( 380, 580 ] 389 0.191% 98.547% ( 580, 870 ] 724 0.356% 98.902% ( 870, 1300 ] 762 0.375% 99.277% ( 1300, 1900 ] 588 0.289% 99.566% ( 1900, 2900 ] 460 0.226% 99.792% ( 2900, 4400 ] 439 0.216% 100.008% ( 4400, 6600 ] 136 0.067% 100.075% ( 6600, 9900 ] 20 0.010% 100.085% ( 9900, 14000 ] 3 0.001% 100.086% ( 14000, 22000 ] 4 0.002% 100.088% ( 22000, 33000 ] 3 0.001% 100.089% ( 33000, 50000 ] 1 0.000% 100.090% ** Level 5 read latency histogram (micros): Count: 11881 Average: 1813.4525 StdDev: 1671.85 Min: 84 Median: 1236.3068 Max: 36244 Percentiles: P50: 1236.31 P75: 2586.63 P99: 7061.20 P99.9: 10664.65 P99.99: 30821.82 ------------------------------------------------------ ( 76, 110 ] 13 0.109% 0.109% ( 110, 170 ] 63 0.530% 0.640% ( 170, 250 ] 197 1.658% 2.298% ( 250, 380 ] 519 4.368% 6.666% # ( 380, 580 ] 1185 9.974% 16.640% ## ( 580, 870 ] 2034 17.120% 33.760% ### ( 870, 1300 ] 2265 19.064% 52.824% #### ( 1300, 1900 ] 1600 13.467% 66.291% ### ( 1900, 2900 ] 1507 12.684% 78.975% ### ( 2900, 4400 ] 1655 13.930% 92.905% ### ( 4400, 6600 ] 707 5.951% 98.855% # ( 6600, 9900 ] 123 1.035% 99.891% ( 9900, 14000 ] 6 0.051% 99.941% ( 14000, 22000 ] 1 0.008% 99.949% ( 22000, 33000 ] 6 0.051% 100.000% ( 33000, 50000 ] 1 0.008% 100.008% ** DB Stats ** Uptime(secs): 7.8 total, 7.6 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 126.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.704 H:M:S, 47.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 129.71 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.704 H:M:S, 48.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/12/14-03:45:57 ... thread 0: (1000000,2000000) ops and (145020.9,138119.1) ops/second in (6.895560,14.480252) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 10/5 1.84 GB 88.1 1.3 0.0 1.3 3.1 1.8 0.0 1.7 62.0 150.5 21.33 10.79 26 0.820 1343K 1611 0.0 0.0 L1 6/6 363.33 MB 0.0 0.8 0.5 0.3 0.8 0.5 0.0 1.5 103.3 103.0 7.64 3.70 1 7.642 801K 2349 0.0 0.0 L3 21/2 1.10 GB 0.9 4.8 2.7 2.0 4.7 2.7 0.0 1.7 108.1 107.2 45.21 22.83 20 2.260 2585K 805 0.0 0.0 L4 181/6 9.88 GB 1.0 7.6 4.2 3.4 7.5 4.1 0.3 1.8 118.2 116.4 65.81 36.51 51 1.290 3777K 0 0.0 0.0 L5 1148/0 94.43 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1366/19 107.61 GB 0.0 14.4 7.5 6.9 16.1 9.2 4.6 8.7 105.6 117.9 139.99 73.83 98 1.428 8507K 4765 0.0 0.0 Int 0/0 0.00 KB 0.0 14.4 7.5 6.9 16.1 9.2 4.6 8.7 105.6 117.9 139.99 73.83 98 1.428 8507K 4765 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 14.4 7.5 6.9 14.3 7.3 0.0 0.0 117.1 115.8 126.21 68.40 76 1.661 8507K 4765 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 137.1 13.78 5.43 22 0.627 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.7 total, 14.5 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 16.12 GB write, 1122.73 MB/s write, 14.43 GB read, 1005.29 MB/s read, 140.0 seconds Interval compaction: 16.12 GB write, 1139.71 MB/s write, 14.43 GB read, 1020.49 MB/s read, 140.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 70, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 70, interval: 70 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 560971 Average: 16.8572 StdDev: 201.30 Min: 1 Median: 1.8405 Max: 31921 Percentiles: P50: 1.84 P75: 2.90 P99: 300.01 P99.9: 2587.00 P99.99: 4222.54 ------------------------------------------------------ [ 0, 1 ] 34308 6.116% 6.116% # ( 1, 2 ] 292905 52.214% 58.330% ########## ( 2, 3 ] 103399 18.432% 76.762% #### ( 3, 4 ] 54739 9.758% 86.520% ## ( 4, 6 ] 54775 9.764% 96.284% ## ( 6, 10 ] 10762 1.918% 98.203% ( 10, 15 ] 2411 0.430% 98.632% ( 15, 22 ] 250 0.045% 98.677% ( 22, 34 ] 19 0.003% 98.680% ( 34, 51 ] 18 0.003% 98.684% ( 51, 76 ] 40 0.007% 98.691% ( 76, 110 ] 114 0.020% 98.711% ( 110, 170 ] 630 0.112% 98.823% ( 170, 250 ] 677 0.121% 98.944% ( 250, 380 ] 817 0.146% 99.090% ( 380, 580 ] 1118 0.199% 99.289% ( 580, 870 ] 1287 0.229% 99.518% ( 870, 1300 ] 1169 0.208% 99.727% ( 1300, 1900 ] 667 0.119% 99.846% ( 1900, 2900 ] 444 0.079% 99.925% ( 2900, 4400 ] 415 0.074% 99.999% ( 4400, 6600 ] 148 0.026% 100.025% ( 6600, 9900 ] 22 0.004% 100.029% ( 14000, 22000 ] 6 0.001% 100.030% ( 22000, 33000 ] 6 0.001% 100.031% ** Level 1 read latency histogram (micros): Count: 176819 Average: 12.6713 StdDev: 188.33 Min: 1 Median: 1.8949 Max: 27942 Percentiles: P50: 1.89 P75: 2.91 P99: 13.67 P99.9: 2028.92 P99.99: 4296.72 ------------------------------------------------------ [ 0, 1 ] 5906 3.340% 3.340% # ( 1, 2 ] 92190 52.138% 55.478% ########## ( 2, 3 ] 37804 21.380% 76.858% #### ( 3, 4 ] 17296 9.782% 86.640% ## ( 4, 6 ] 16536 9.352% 95.992% ## ( 6, 10 ] 4558 2.578% 98.570% # ( 10, 15 ] 1036 0.586% 99.156% ( 15, 22 ] 150 0.085% 99.240% ( 22, 34 ] 16 0.009% 99.250% ( 34, 51 ] 10 0.006% 99.255% ( 51, 76 ] 11 0.006% 99.261% ( 76, 110 ] 17 0.010% 99.271% ( 110, 170 ] 30 0.017% 99.288% ( 170, 250 ] 60 0.034% 99.322% ( 250, 380 ] 116 0.066% 99.388% ( 380, 580 ] 201 0.114% 99.501% ( 580, 870 ] 285 0.161% 99.662% ( 870, 1300 ] 239 0.135% 99.798% ( 1300, 1900 ] 167 0.094% 99.892% ( 1900, 2900 ] 110 0.062% 99.954% ( 2900, 4400 ] 68 0.038% 99.993% ( 4400, 6600 ] 32 0.018% 100.011% ( 6600, 9900 ] 5 0.003% 100.014% ( 9900, 14000 ] 3 0.002% 100.015% ( 14000, 22000 ] 5 0.003% 100.018% ( 22000, 33000 ] 1 0.001% 100.019% ** Level 2 read latency histogram (micros): Count: 415279 Average: 13.3897 StdDev: 162.65 Min: 1 Median: 1.9161 Max: 8430 Percentiles: P50: 1.92 P75: 3.01 P99: 13.06 P99.9: 1768.34 P99.99: 2668.18 ------------------------------------------------------ [ 0, 1 ] 18379 4.426% 4.426% # ( 1, 2 ] 206602 49.750% 54.176% ########## ( 2, 3 ] 86067 20.725% 74.901% #### ( 3, 4 ] 43984 10.591% 85.492% ## ( 4, 6 ] 43079 10.374% 95.866% ## ( 6, 10 ] 11224 2.703% 98.569% # ( 10, 15 ] 2923 0.704% 99.273% ( 15, 22 ] 462 0.111% 99.384% ( 22, 34 ] 59 0.014% 99.398% ( 34, 51 ] 29 0.007% 99.405% ( 51, 76 ] 18 0.004% 99.409% ( 76, 110 ] 16 0.004% 99.413% ( 110, 170 ] 39 0.009% 99.423% ( 170, 250 ] 59 0.014% 99.437% ( 250, 380 ] 175 0.042% 99.479% ( 380, 580 ] 312 0.075% 99.554% ( 580, 870 ] 475 0.114% 99.668% ( 870, 1300 ] 605 0.146% 99.814% ( 1300, 1900 ] 457 0.110% 99.924% ( 1900, 2900 ] 356 0.086% 100.010% ( 2900, 4400 ] 270 0.065% 100.075% ( 4400, 6600 ] 95 0.023% 100.098% ( 6600, 9900 ] 15 0.004% 100.101% ** Level 3 read latency histogram (micros): Count: 1493027 Average: 10.8691 StdDev: 175.27 Min: 1 Median: 1.9081 Max: 42164 Percentiles: P50: 1.91 P75: 2.96 P99: 9.89 P99.9: 930.23 P99.99: 1367.41 ------------------------------------------------------ [ 0, 1 ] 48996 3.282% 3.282% # ( 1, 2 ] 768085 51.445% 54.726% ########## ( 2, 3 ] 316129 21.174% 75.900% #### ( 3, 4 ] 157020 10.517% 86.417% ## ( 4, 6 ] 155572 10.420% 96.837% ## ( 6, 10 ] 33198 2.224% 99.060% ( 10, 15 ] 8011 0.537% 99.597% ( 15, 22 ] 953 0.064% 99.661% ( 22, 34 ] 84 0.006% 99.667% ( 34, 51 ] 84 0.006% 99.672% ( 51, 76 ] 151 0.010% 99.682% ( 76, 110 ] 44 0.003% 99.685% ( 110, 170 ] 119 0.008% 99.693% ( 170, 250 ] 235 0.016% 99.709% ( 250, 380 ] 410 0.027% 99.736% ( 380, 580 ] 872 0.058% 99.795% ( 580, 870 ] 1369 0.092% 99.886% ( 870, 1300 ] 1442 0.097% 99.983% ( 1300, 1900 ] 923 0.062% 100.045% ( 1900, 2900 ] 816 0.055% 100.100% ( 2900, 4400 ] 893 0.060% 100.159% ( 4400, 6600 ] 222 0.015% 100.174% ( 6600, 9900 ] 50 0.003% 100.178% ( 9900, 14000 ] 13 0.001% 100.178% ( 14000, 22000 ] 15 0.001% 100.179% ( 22000, 33000 ] 7 0.000% 100.180% ( 33000, 50000 ] 3 0.000% 100.180% ** Level 4 read latency histogram (micros): Count: 918901 Average: 20.2649 StdDev: 225.83 Min: 1 Median: 1.9132 Max: 37309 Percentiles: P50: 1.91 P75: 3.01 P99: 242.92 P99.9: 2143.08 P99.99: 2922.92 ------------------------------------------------------ [ 0, 1 ] 31178 3.393% 3.393% # ( 1, 2 ] 468988 51.038% 54.431% ########## ( 2, 3 ] 188484 20.512% 74.943% #### ( 3, 4 ] 95102 10.350% 85.292% ## ( 4, 6 ] 95954 10.442% 95.735% ## ( 6, 10 ] 22675 2.468% 98.202% ( 10, 15 ] 5523 0.601% 98.803% ( 15, 22 ] 573 0.062% 98.866% ( 22, 34 ] 50 0.005% 98.871% ( 34, 51 ] 81 0.009% 98.880% ( 51, 76 ] 109 0.012% 98.892% ( 76, 110 ] 91 0.010% 98.902% ( 110, 170 ] 380 0.041% 98.943% ( 170, 250 ] 576 0.063% 99.006% ( 250, 380 ] 812 0.088% 99.094% ( 380, 580 ] 1473 0.160% 99.254% ( 580, 870 ] 2207 0.240% 99.495% ( 870, 1300 ] 2124 0.231% 99.726% ( 1300, 1900 ] 1343 0.146% 99.872% ( 1900, 2900 ] 1070 0.116% 99.988% ( 2900, 4400 ] 1120 0.122% 100.110% ( 4400, 6600 ] 317 0.034% 100.145% ( 6600, 9900 ] 55 0.006% 100.151% ( 9900, 14000 ] 7 0.001% 100.151% ( 14000, 22000 ] 10 0.001% 100.152% ( 22000, 33000 ] 4 0.000% 100.153% ( 33000, 50000 ] 1 0.000% 100.153% ** Level 5 read latency histogram (micros): Count: 33918 Average: 1245.8253 StdDev: 1445.40 Min: 80 Median: 801.5259 Max: 41698 Percentiles: P50: 801.53 P75: 1517.38 P99: 6250.99 P99.9: 9879.11 P99.99: 35067.88 ------------------------------------------------------ ( 76, 110 ] 391 1.153% 1.153% ( 110, 170 ] 1929 5.687% 6.840% # ( 170, 250 ] 2246 6.622% 13.462% # ( 250, 380 ] 2895 8.535% 21.997% ## ( 380, 580 ] 4697 13.848% 35.845% ### ( 580, 870 ] 6285 18.530% 54.375% #### ( 870, 1300 ] 5801 17.103% 71.478% ### ( 1300, 1900 ] 3297 9.721% 81.199% ## ( 1900, 2900 ] 2514 7.412% 88.611% # ( 2900, 4400 ] 2669 7.869% 96.480% ## ( 4400, 6600 ] 1016 2.995% 99.475% # ( 6600, 9900 ] 145 0.428% 99.903% ( 9900, 14000 ] 11 0.032% 99.935% ( 14000, 22000 ] 7 0.021% 99.956% ( 22000, 33000 ] 11 0.032% 99.988% ( 33000, 50000 ] 5 0.015% 100.003% ** DB Stats ** Uptime(secs): 14.7 total, 6.9 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 133.89 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.733 H:M:S, 45.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 142.73 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.029 H:M:S, 43.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 3 num-running-flushes: 1 2024/12/14-03:46:01 ... thread 0: (1000000,3000000) ops and (288676.9,167183.7) ops/second in (3.464080,17.944332) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/9 2.77 GB 157.3 2.1 0.0 2.1 4.9 2.8 0.0 1.8 77.7 180.4 27.57 16.51 44 0.627 2174K 2504 0.0 0.0 L1 6/6 363.33 MB 0.0 0.8 0.5 0.3 0.8 0.5 0.0 1.5 103.3 103.0 7.64 3.70 1 7.642 801K 2349 0.0 0.0 L3 19/0 937.09 MB 0.9 4.8 2.7 2.0 4.7 2.7 0.0 1.7 108.1 107.2 45.21 22.83 20 2.260 2585K 805 0.0 0.0 L4 180/0 9.75 GB 1.0 8.1 4.4 3.7 8.0 4.3 0.3 1.8 120.9 119.1 68.41 38.66 53 1.291 3931K 0 0.0 0.0 L5 1152/0 94.74 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1373/15 108.52 GB 0.0 15.7 7.7 8.0 18.3 10.3 4.9 6.6 108.1 126.0 148.82 81.70 118 1.261 9492K 5658 0.0 0.0 Int 0/0 0.00 KB 0.0 15.7 7.7 8.0 18.3 10.3 4.9 6.6 108.1 126.0 148.82 81.70 118 1.261 9492K 5658 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 15.7 7.7 8.0 15.5 7.5 0.0 0.0 121.9 120.6 131.97 73.63 81 1.629 9492K 5658 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 168.2 16.85 8.08 37 0.456 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.2 total, 17.9 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 18.31 GB write, 1032.27 MB/s write, 15.71 GB read, 885.79 MB/s read, 148.8 seconds Interval compaction: 18.31 GB write, 1044.87 MB/s write, 15.71 GB read, 896.60 MB/s read, 148.8 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 70, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 70, interval: 70 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 839579 Average: 16.3990 StdDev: 169.28 Min: 0 Median: 1.7058 Max: 31921 Percentiles: P50: 1.71 P75: 2.62 P99: 312.34 P99.9: 1811.25 P99.99: 3604.16 ------------------------------------------------------ [ 0, 1 ] 102703 12.233% 12.233% ## ( 1, 2 ] 449266 53.511% 65.744% ########### ( 2, 3 ] 125465 14.944% 80.687% ### ( 3, 4 ] 63101 7.516% 88.203% ## ( 4, 6 ] 59667 7.107% 95.310% # ( 6, 10 ] 11682 1.391% 96.701% ( 10, 15 ] 2724 0.324% 97.026% ( 15, 22 ] 291 0.035% 97.060% ( 22, 34 ] 23 0.003% 97.063% ( 34, 51 ] 21 0.003% 97.066% ( 51, 76 ] 47 0.006% 97.071% ( 76, 110 ] 2170 0.258% 97.330% ( 110, 170 ] 8122 0.967% 98.297% ( 170, 250 ] 4620 0.550% 98.847% ( 250, 380 ] 2674 0.318% 99.166% ( 380, 580 ] 2546 0.303% 99.469% ( 580, 870 ] 1785 0.213% 99.682% ( 870, 1300 ] 1254 0.149% 99.831% ( 1300, 1900 ] 680 0.081% 99.912% ( 1900, 2900 ] 456 0.054% 99.966% ( 2900, 4400 ] 424 0.051% 100.017% ( 4400, 6600 ] 150 0.018% 100.035% ( 6600, 9900 ] 22 0.003% 100.037% ( 14000, 22000 ] 6 0.001% 100.038% ( 22000, 33000 ] 6 0.001% 100.039% ** Level 1 read latency histogram (micros): Count: 185700 Average: 12.9031 StdDev: 184.32 Min: 1 Median: 1.9253 Max: 27942 Percentiles: P50: 1.93 P75: 2.99 P99: 83.51 P99.9: 1948.18 P99.99: 4277.13 ------------------------------------------------------ [ 0, 1 ] 6007 3.235% 3.235% # ( 1, 2 ] 93849 50.538% 53.773% ########## ( 2, 3 ] 39890 21.481% 75.254% #### ( 3, 4 ] 18794 10.121% 85.374% ## ( 4, 6 ] 18762 10.103% 95.478% ## ( 6, 10 ] 5097 2.745% 98.222% # ( 10, 15 ] 1168 0.629% 98.851% ( 15, 22 ] 209 0.113% 98.964% ( 22, 34 ] 27 0.015% 98.978% ( 34, 51 ] 10 0.005% 98.984% ( 51, 76 ] 11 0.006% 98.990% ( 76, 110 ] 86 0.046% 99.036% ( 110, 170 ] 277 0.149% 99.185% ( 170, 250 ] 191 0.103% 99.288% ( 250, 380 ] 183 0.099% 99.387% ( 380, 580 ] 237 0.128% 99.514% ( 580, 870 ] 300 0.162% 99.676% ( 870, 1300 ] 243 0.131% 99.807% ( 1300, 1900 ] 168 0.090% 99.897% ( 1900, 2900 ] 110 0.059% 99.956% ( 2900, 4400 ] 68 0.037% 99.993% ( 4400, 6600 ] 32 0.017% 100.010% ( 6600, 9900 ] 5 0.003% 100.013% ( 9900, 14000 ] 3 0.002% 100.015% ( 14000, 22000 ] 5 0.003% 100.017% ( 22000, 33000 ] 1 0.001% 100.018% ** Level 2 read latency histogram (micros): Count: 415279 Average: 13.3897 StdDev: 162.65 Min: 1 Median: 1.9161 Max: 8430 Percentiles: P50: 1.92 P75: 3.01 P99: 13.06 P99.9: 1768.34 P99.99: 2668.18 ------------------------------------------------------ [ 0, 1 ] 18379 4.426% 4.426% # ( 1, 2 ] 206602 49.750% 54.176% ########## ( 2, 3 ] 86067 20.725% 74.901% #### ( 3, 4 ] 43984 10.591% 85.492% ## ( 4, 6 ] 43079 10.374% 95.866% ## ( 6, 10 ] 11224 2.703% 98.569% # ( 10, 15 ] 2923 0.704% 99.273% ( 15, 22 ] 462 0.111% 99.384% ( 22, 34 ] 59 0.014% 99.398% ( 34, 51 ] 29 0.007% 99.405% ( 51, 76 ] 18 0.004% 99.409% ( 76, 110 ] 16 0.004% 99.413% ( 110, 170 ] 39 0.009% 99.423% ( 170, 250 ] 59 0.014% 99.437% ( 250, 380 ] 175 0.042% 99.479% ( 380, 580 ] 312 0.075% 99.554% ( 580, 870 ] 475 0.114% 99.668% ( 870, 1300 ] 605 0.146% 99.814% ( 1300, 1900 ] 457 0.110% 99.924% ( 1900, 2900 ] 356 0.086% 100.010% ( 2900, 4400 ] 270 0.065% 100.075% ( 4400, 6600 ] 95 0.023% 100.098% ( 6600, 9900 ] 15 0.004% 100.101% ** Level 3 read latency histogram (micros): Count: 1521629 Average: 10.8069 StdDev: 173.72 Min: 1 Median: 1.9009 Max: 42164 Percentiles: P50: 1.90 P75: 2.95 P99: 9.95 P99.9: 922.22 P99.99: 1365.55 ------------------------------------------------------ [ 0, 1 ] 51962 3.415% 3.415% # ( 1, 2 ] 786821 51.709% 55.124% ########## ( 2, 3 ] 320016 21.031% 76.155% #### ( 3, 4 ] 158094 10.390% 86.545% ## ( 4, 6 ] 156270 10.270% 96.815% ## ( 6, 10 ] 33665 2.212% 99.027% ( 10, 15 ] 8083 0.531% 99.558% ( 15, 22 ] 962 0.063% 99.622% ( 22, 34 ] 85 0.006% 99.627% ( 34, 51 ] 85 0.006% 99.633% ( 51, 76 ] 151 0.010% 99.643% ( 76, 110 ] 120 0.008% 99.651% ( 110, 170 ] 420 0.028% 99.678% ( 170, 250 ] 397 0.026% 99.704% ( 250, 380 ] 480 0.032% 99.736% ( 380, 580 ] 932 0.061% 99.797% ( 580, 870 ] 1389 0.091% 99.888% ( 870, 1300 ] 1444 0.095% 99.983% ( 1300, 1900 ] 923 0.061% 100.044% ( 1900, 2900 ] 816 0.054% 100.098% ( 2900, 4400 ] 894 0.059% 100.156% ( 4400, 6600 ] 222 0.015% 100.171% ( 6600, 9900 ] 50 0.003% 100.174% ( 9900, 14000 ] 13 0.001% 100.175% ( 14000, 22000 ] 15 0.001% 100.176% ( 22000, 33000 ] 7 0.000% 100.177% ( 33000, 50000 ] 3 0.000% 100.177% ** Level 4 read latency histogram (micros): Count: 963693 Average: 20.8808 StdDev: 221.54 Min: 1 Median: 1.9045 Max: 37309 Percentiles: P50: 1.90 P75: 3.00 P99: 334.08 P99.9: 2099.54 P99.99: 2910.22 ------------------------------------------------------ [ 0, 1 ] 34963 3.628% 3.628% # ( 1, 2 ] 494085 51.270% 54.898% ########## ( 2, 3 ] 194200 20.152% 75.050% #### ( 3, 4 ] 96760 10.041% 85.090% ## ( 4, 6 ] 97256 10.092% 95.182% ## ( 6, 10 ] 23368 2.425% 97.607% ( 10, 15 ] 5638 0.585% 98.192% ( 15, 22 ] 584 0.061% 98.253% ( 22, 34 ] 52 0.005% 98.258% ( 34, 51 ] 81 0.008% 98.266% ( 51, 76 ] 109 0.011% 98.278% ( 76, 110 ] 784 0.081% 98.359% ( 110, 170 ] 3166 0.329% 98.688% ( 170, 250 ] 2047 0.212% 98.900% ( 250, 380 ] 1489 0.155% 99.055% ( 380, 580 ] 2026 0.210% 99.265% ( 580, 870 ] 2388 0.248% 99.513% ( 870, 1300 ] 2169 0.225% 99.738% ( 1300, 1900 ] 1350 0.140% 99.878% ( 1900, 2900 ] 1074 0.111% 99.989% ( 2900, 4400 ] 1120 0.116% 100.105% ( 4400, 6600 ] 317 0.033% 100.138% ( 6600, 9900 ] 55 0.006% 100.144% ( 9900, 14000 ] 7 0.001% 100.145% ( 14000, 22000 ] 10 0.001% 100.146% ( 22000, 33000 ] 4 0.000% 100.146% ( 33000, 50000 ] 1 0.000% 100.146% ** Level 5 read latency histogram (micros): Count: 75515 Average: 677.7776 StdDev: 1102.54 Min: 77 Median: 258.6807 Max: 41698 Percentiles: P50: 258.68 P75: 748.31 P99: 5348.10 P99.9: 8909.66 P99.99: 28448.50 ------------------------------------------------------ ( 76, 110 ] 4849 6.421% 6.421% # ( 110, 170 ] 20605 27.286% 33.707% ##### ( 170, 250 ] 11829 15.664% 49.372% ### ( 250, 380 ] 7106 9.410% 58.782% ## ( 380, 580 ] 7939 10.513% 69.295% ## ( 580, 870 ] 7423 9.830% 79.125% ## ( 870, 1300 ] 6031 7.986% 87.111% ## ( 1300, 1900 ] 3325 4.403% 91.514% # ( 1900, 2900 ] 2538 3.361% 94.875% # ( 2900, 4400 ] 2677 3.545% 98.420% # ( 4400, 6600 ] 1016 1.345% 99.766% ( 6600, 9900 ] 145 0.192% 99.958% ( 9900, 14000 ] 11 0.015% 99.972% ( 14000, 22000 ] 7 0.009% 99.981% ( 22000, 33000 ] 11 0.015% 99.996% ( 33000, 50000 ] 5 0.007% 100.003% ** DB Stats ** Uptime(secs): 18.2 total, 3.5 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 162.54 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.733 H:M:S, 37.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 284.11 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/14-03:46:05 ... thread 0: (1000000,4000000) ops and (245961.3,181735.5) ops/second in (4.065680,22.010012) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/5 3.81 GB 289.5 3.3 0.0 3.3 7.1 3.8 0.0 1.9 95.3 204.6 35.70 24.10 66 0.541 3454K 3772 0.0 0.0 L1 6/6 363.33 MB 0.0 0.8 0.5 0.3 0.8 0.5 0.0 1.5 103.3 103.0 7.64 3.70 1 7.642 801K 2349 0.0 0.0 L3 19/0 937.09 MB 0.9 4.8 2.7 2.0 4.7 2.7 0.0 1.7 108.1 107.2 45.21 22.83 20 2.260 2585K 805 0.0 0.0 L4 180/0 9.75 GB 1.0 8.1 4.4 3.7 8.0 4.3 0.3 1.8 120.9 119.1 68.41 38.66 53 1.291 3931K 0 0.0 0.0 L5 1152/0 94.74 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1375/11 109.57 GB 0.0 16.9 7.7 9.3 20.6 11.3 4.9 5.4 110.5 134.3 156.96 89.29 140 1.121 10M 6926 0.0 0.0 Int 0/0 0.00 KB 0.0 16.9 7.7 9.3 20.6 11.3 4.9 5.4 110.5 134.3 156.96 89.29 140 1.121 10M 6926 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.9 7.7 9.3 16.8 7.5 0.0 0.0 126.9 125.6 136.75 78.29 86 1.590 10M 6926 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 193.3 20.21 11.00 54 0.374 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 22.2 total, 22.0 interval Flush(GB): cumulative 3.814, interval 3.814 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 20.59 GB write, 948.28 MB/s write, 16.94 GB read, 780.47 MB/s read, 157.0 seconds Interval compaction: 20.59 GB write, 957.72 MB/s write, 16.94 GB read, 788.24 MB/s read, 157.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 4, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 4, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 70, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 4, total-stops: 70, interval: 74 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1154162 Average: 16.0963 StdDev: 147.95 Min: 0 Median: 1.6280 Max: 31921 Percentiles: P50: 1.63 P75: 2.40 P99: 296.61 P99.9: 1376.16 P99.99: 2838.63 ------------------------------------------------------ [ 0, 1 ] 190408 16.498% 16.498% ### ( 1, 2 ] 615677 53.344% 69.842% ########### ( 2, 3 ] 148680 12.882% 82.724% ### ( 3, 4 ] 72131 6.250% 88.973% # ( 4, 6 ] 64942 5.627% 94.600% # ( 6, 10 ] 12854 1.114% 95.714% ( 10, 15 ] 3042 0.264% 95.977% ( 15, 22 ] 366 0.032% 96.009% ( 22, 34 ] 36 0.003% 96.012% ( 34, 51 ] 23 0.002% 96.014% ( 51, 76 ] 52 0.005% 96.019% ( 76, 110 ] 5023 0.435% 96.454% ( 110, 170 ] 18636 1.615% 98.069% ( 170, 250 ] 9157 0.793% 98.862% ( 250, 380 ] 4444 0.385% 99.247% ( 380, 580 ] 3937 0.341% 99.588% ( 580, 870 ] 2205 0.191% 99.779% ( 870, 1300 ] 1307 0.113% 99.892% ( 1300, 1900 ] 692 0.060% 99.952% ( 1900, 2900 ] 463 0.040% 99.992% ( 2900, 4400 ] 430 0.037% 100.030% ( 4400, 6600 ] 150 0.013% 100.043% ( 6600, 9900 ] 22 0.002% 100.045% ( 14000, 22000 ] 6 0.001% 100.045% ( 22000, 33000 ] 6 0.001% 100.046% ** Level 1 read latency histogram (micros): Count: 200890 Average: 12.9159 StdDev: 177.75 Min: 1 Median: 1.9636 Max: 27942 Percentiles: P50: 1.96 P75: 3.11 P99: 149.14 P99.9: 1864.68 P99.99: 4243.62 ------------------------------------------------------ [ 0, 1 ] 6295 3.134% 3.134% # ( 1, 2 ] 97703 48.635% 51.769% ########## ( 2, 3 ] 44361 22.082% 73.851% #### ( 3, 4 ] 21113 10.510% 84.361% ## ( 4, 6 ] 21352 10.629% 94.989% ## ( 6, 10 ] 5767 2.871% 97.860% # ( 10, 15 ] 1323 0.659% 98.519% ( 15, 22 ] 290 0.144% 98.663% ( 22, 34 ] 37 0.018% 98.681% ( 34, 51 ] 11 0.005% 98.687% ( 51, 76 ] 11 0.005% 98.692% ( 76, 110 ] 196 0.098% 98.790% ( 110, 170 ] 647 0.322% 99.112% ( 170, 250 ] 341 0.170% 99.282% ( 250, 380 ] 238 0.118% 99.400% ( 380, 580 ] 288 0.143% 99.544% ( 580, 870 ] 311 0.155% 99.698% ( 870, 1300 ] 247 0.123% 99.821% ( 1300, 1900 ] 168 0.084% 99.905% ( 1900, 2900 ] 110 0.055% 99.960% ( 2900, 4400 ] 68 0.034% 99.994% ( 4400, 6600 ] 32 0.016% 100.009% ( 6600, 9900 ] 5 0.002% 100.012% ( 9900, 14000 ] 3 0.001% 100.013% ( 14000, 22000 ] 5 0.002% 100.016% ( 22000, 33000 ] 1 0.000% 100.016% ** Level 2 read latency histogram (micros): Count: 415279 Average: 13.3897 StdDev: 162.65 Min: 1 Median: 1.9161 Max: 8430 Percentiles: P50: 1.92 P75: 3.01 P99: 13.06 P99.9: 1768.34 P99.99: 2668.18 ------------------------------------------------------ [ 0, 1 ] 18379 4.426% 4.426% # ( 1, 2 ] 206602 49.750% 54.176% ########## ( 2, 3 ] 86067 20.725% 74.901% #### ( 3, 4 ] 43984 10.591% 85.492% ## ( 4, 6 ] 43079 10.374% 95.866% ## ( 6, 10 ] 11224 2.703% 98.569% # ( 10, 15 ] 2923 0.704% 99.273% ( 15, 22 ] 462 0.111% 99.384% ( 22, 34 ] 59 0.014% 99.398% ( 34, 51 ] 29 0.007% 99.405% ( 51, 76 ] 18 0.004% 99.409% ( 76, 110 ] 16 0.004% 99.413% ( 110, 170 ] 39 0.009% 99.423% ( 170, 250 ] 59 0.014% 99.437% ( 250, 380 ] 175 0.042% 99.479% ( 380, 580 ] 312 0.075% 99.554% ( 580, 870 ] 475 0.114% 99.668% ( 870, 1300 ] 605 0.146% 99.814% ( 1300, 1900 ] 457 0.110% 99.924% ( 1900, 2900 ] 356 0.086% 100.010% ( 2900, 4400 ] 270 0.065% 100.075% ( 4400, 6600 ] 95 0.023% 100.098% ( 6600, 9900 ] 15 0.004% 100.101% ** Level 3 read latency histogram (micros): Count: 1522609 Average: 10.9277 StdDev: 173.77 Min: 1 Median: 1.9015 Max: 42164 Percentiles: P50: 1.90 P75: 2.95 P99: 10.34 P99.9: 923.05 P99.99: 1366.71 ------------------------------------------------------ [ 0, 1 ] 51962 3.413% 3.413% # ( 1, 2 ] 786821 51.676% 55.089% ########## ( 2, 3 ] 320016 21.018% 76.106% #### ( 3, 4 ] 158094 10.383% 86.489% ## ( 4, 6 ] 156270 10.263% 96.753% ## ( 6, 10 ] 33665 2.211% 98.964% ( 10, 15 ] 8083 0.531% 99.494% ( 15, 22 ] 962 0.063% 99.558% ( 22, 34 ] 85 0.006% 99.563% ( 34, 51 ] 85 0.006% 99.569% ( 51, 76 ] 151 0.010% 99.579% ( 76, 110 ] 257 0.017% 99.596% ( 110, 170 ] 925 0.061% 99.656% ( 170, 250 ] 565 0.037% 99.693% ( 250, 380 ] 560 0.037% 99.730% ( 380, 580 ] 995 0.065% 99.796% ( 580, 870 ] 1412 0.093% 99.888% ( 870, 1300 ] 1446 0.095% 99.983% ( 1300, 1900 ] 924 0.061% 100.044% ( 1900, 2900 ] 817 0.054% 100.098% ( 2900, 4400 ] 894 0.059% 100.156% ( 4400, 6600 ] 222 0.015% 100.171% ( 6600, 9900 ] 50 0.003% 100.174% ( 9900, 14000 ] 13 0.001% 100.175% ( 14000, 22000 ] 15 0.001% 100.176% ( 22000, 33000 ] 7 0.000% 100.176% ( 33000, 50000 ] 3 0.000% 100.177% ** Level 4 read latency histogram (micros): Count: 972942 Average: 22.3815 StdDev: 221.44 Min: 1 Median: 1.9137 Max: 37309 Percentiles: P50: 1.91 P75: 3.06 P99: 390.86 P99.9: 2093.18 P99.99: 2908.98 ------------------------------------------------------ [ 0, 1 ] 34963 3.594% 3.594% # ( 1, 2 ] 494133 50.788% 54.381% ########## ( 2, 3 ] 194469 19.988% 74.369% #### ( 3, 4 ] 96927 9.962% 84.331% ## ( 4, 6 ] 97475 10.019% 94.350% ## ( 6, 10 ] 23407 2.406% 96.755% ( 10, 15 ] 5646 0.580% 97.336% ( 15, 22 ] 585 0.060% 97.396% ( 22, 34 ] 52 0.005% 97.401% ( 34, 51 ] 82 0.008% 97.410% ( 51, 76 ] 111 0.011% 97.421% ( 76, 110 ] 2014 0.207% 97.628% ( 110, 170 ] 7373 0.758% 98.386% ( 170, 250 ] 3693 0.380% 98.765% ( 250, 380 ] 2143 0.220% 98.986% ( 380, 580 ] 2589 0.266% 99.252% ( 580, 870 ] 2553 0.262% 99.514% ( 870, 1300 ] 2191 0.225% 99.739% ( 1300, 1900 ] 1356 0.139% 99.879% ( 1900, 2900 ] 1077 0.111% 99.989% ( 2900, 4400 ] 1120 0.115% 100.105% ( 4400, 6600 ] 317 0.033% 100.137% ( 6600, 9900 ] 55 0.006% 100.143% ( 9900, 14000 ] 7 0.001% 100.143% ( 14000, 22000 ] 10 0.001% 100.145% ( 22000, 33000 ] 4 0.000% 100.145% ( 33000, 50000 ] 1 0.000% 100.145% ** Level 5 read latency histogram (micros): Count: 127910 Average: 478.6474 StdDev: 884.48 Min: 75 Median: 187.1864 Max: 41698 Percentiles: P50: 187.19 P75: 440.51 P99: 4350.71 P99.9: 7671.70 P99.99: 21096.00 ------------------------------------------------------ ( 51, 76 ] 4 0.003% 0.003% ( 76, 110 ] 12200 9.538% 9.541% ## ( 110, 170 ] 46956 36.710% 46.251% ####### ( 170, 250 ] 22320 17.450% 63.701% ### ( 250, 380 ] 11105 8.682% 72.383% ## ( 380, 580 ] 11064 8.650% 81.033% ## ( 580, 870 ] 8314 6.500% 87.533% # ( 870, 1300 ] 6163 4.818% 92.351% # ( 1300, 1900 ] 3350 2.619% 94.970% # ( 1900, 2900 ] 2562 2.003% 96.973% ( 2900, 4400 ] 2681 2.096% 99.069% ( 4400, 6600 ] 1016 0.794% 99.863% ( 6600, 9900 ] 145 0.113% 99.977% ( 9900, 14000 ] 11 0.009% 99.985% ( 14000, 22000 ] 7 0.005% 99.991% ( 22000, 33000 ] 11 0.009% 99.999% ( 33000, 50000 ] 5 0.004% 100.003% ** DB Stats ** Uptime(secs): 22.2 total, 4.1 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 177.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.337 H:M:S, 33.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 242.07 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.605 H:M:S, 14.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 2024/12/14-03:46:14 ... thread 0: (1000000,5000000) ops and (114625.6,162685.9) ops/second in (8.724053,30.734065) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/9 4.72 GB 357.6 5.6 0.0 5.6 10.3 4.7 0.0 2.2 121.2 223.5 47.27 35.13 85 0.556 5819K 15K 0.0 0.0 L1 6/6 363.33 MB 0.0 0.8 0.5 0.3 0.8 0.5 0.0 1.5 103.3 103.0 7.64 3.70 1 7.642 801K 2349 0.0 0.0 L3 19/0 937.09 MB 0.9 4.8 2.7 2.0 4.7 2.7 0.0 1.7 108.1 107.2 45.21 22.83 20 2.260 2585K 805 0.0 0.0 L4 180/0 9.75 GB 1.0 8.1 4.4 3.7 8.0 4.3 0.3 1.8 120.9 119.1 68.41 38.66 53 1.291 3931K 0 0.0 0.0 L5 1152/0 94.74 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1375/15 110.48 GB 0.0 19.2 7.7 11.5 23.8 12.2 4.9 5.0 116.8 144.4 168.53 100.32 159 1.060 13M 18K 0.0 0.0 Int 0/0 0.00 KB 0.0 19.2 7.7 11.5 23.8 12.2 4.9 5.0 116.8 144.4 168.53 100.32 159 1.060 13M 18K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 19.2 7.7 11.5 19.0 7.5 0.0 0.0 135.3 134.1 145.40 86.78 90 1.616 13M 18K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 209.7 23.13 13.54 69 0.335 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 31.0 total, 30.7 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 23.77 GB write, 786.43 MB/s write, 19.22 GB read, 635.76 MB/s read, 168.5 seconds Interval compaction: 23.77 GB write, 792.04 MB/s write, 19.22 GB read, 640.29 MB/s read, 168.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 11, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 11, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 70, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 11, total-stops: 70, interval: 81 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1857889 Average: 15.8601 StdDev: 122.20 Min: 0 Median: 1.5449 Max: 31921 Percentiles: P50: 1.54 P75: 2.09 P99: 258.60 P99.9: 964.43 P99.99: 1790.55 ------------------------------------------------------ [ 0, 1 ] 394553 21.237% 21.237% #### ( 1, 2 ] 980729 52.787% 74.024% ########### ( 2, 3 ] 196407 10.572% 84.595% ## ( 3, 4 ] 91302 4.914% 89.510% # ( 4, 6 ] 76533 4.119% 93.629% # ( 6, 10 ] 15066 0.811% 94.440% ( 10, 15 ] 3580 0.193% 94.633% ( 15, 22 ] 445 0.024% 94.657% ( 22, 34 ] 47 0.003% 94.659% ( 34, 51 ] 26 0.001% 94.661% ( 51, 76 ] 67 0.004% 94.664% ( 76, 110 ] 12732 0.685% 95.349% ( 110, 170 ] 48127 2.590% 97.940% # ( 170, 250 ] 19190 1.033% 98.973% ( 250, 380 ] 7663 0.412% 99.385% ( 380, 580 ] 6400 0.344% 99.730% ( 580, 870 ] 2857 0.154% 99.883% ( 870, 1300 ] 1403 0.076% 99.959% ( 1300, 1900 ] 706 0.038% 99.997% ( 1900, 2900 ] 492 0.026% 100.023% ( 2900, 4400 ] 438 0.024% 100.047% ( 4400, 6600 ] 151 0.008% 100.055% ( 6600, 9900 ] 22 0.001% 100.056% ( 14000, 22000 ] 6 0.000% 100.057% ( 22000, 33000 ] 6 0.000% 100.057% ** Level 1 read latency histogram (micros): Count: 232530 Average: 12.9017 StdDev: 166.05 Min: 1 Median: 2.0695 Max: 27942 Percentiles: P50: 2.07 P75: 3.42 P99: 167.29 P99.9: 1751.68 P99.99: 4173.83 ------------------------------------------------------ [ 0, 1 ] 7306 3.142% 3.142% # ( 1, 2 ] 105448 45.348% 48.490% ######### ( 2, 3 ] 50511 21.722% 70.212% #### ( 3, 4 ] 26354 11.334% 81.546% ## ( 4, 6 ] 29026 12.483% 94.029% ## ( 6, 10 ] 7359 3.165% 97.193% # ( 10, 15 ] 1642 0.706% 97.900% ( 15, 22 ] 535 0.230% 98.130% ( 22, 34 ] 52 0.022% 98.152% ( 34, 51 ] 11 0.005% 98.157% ( 51, 76 ] 12 0.005% 98.162% ( 76, 110 ] 482 0.207% 98.369% ( 110, 170 ] 1536 0.661% 99.030% ( 170, 250 ] 616 0.265% 99.295% ( 250, 380 ] 344 0.148% 99.443% ( 380, 580 ] 353 0.152% 99.594% ( 580, 870 ] 336 0.144% 99.739% ( 870, 1300 ] 248 0.107% 99.846% ( 1300, 1900 ] 168 0.072% 99.918% ( 1900, 2900 ] 110 0.047% 99.965% ( 2900, 4400 ] 68 0.029% 99.994% ( 4400, 6600 ] 32 0.014% 100.008% ( 6600, 9900 ] 5 0.002% 100.010% ( 9900, 14000 ] 3 0.001% 100.012% ( 14000, 22000 ] 5 0.002% 100.014% ( 22000, 33000 ] 1 0.000% 100.014% ** Level 2 read latency histogram (micros): Count: 415279 Average: 13.3897 StdDev: 162.65 Min: 1 Median: 1.9161 Max: 8430 Percentiles: P50: 1.92 P75: 3.01 P99: 13.06 P99.9: 1768.34 P99.99: 2668.18 ------------------------------------------------------ [ 0, 1 ] 18379 4.426% 4.426% # ( 1, 2 ] 206602 49.750% 54.176% ########## ( 2, 3 ] 86067 20.725% 74.901% #### ( 3, 4 ] 43984 10.591% 85.492% ## ( 4, 6 ] 43079 10.374% 95.866% ## ( 6, 10 ] 11224 2.703% 98.569% # ( 10, 15 ] 2923 0.704% 99.273% ( 15, 22 ] 462 0.111% 99.384% ( 22, 34 ] 59 0.014% 99.398% ( 34, 51 ] 29 0.007% 99.405% ( 51, 76 ] 18 0.004% 99.409% ( 76, 110 ] 16 0.004% 99.413% ( 110, 170 ] 39 0.009% 99.423% ( 170, 250 ] 59 0.014% 99.437% ( 250, 380 ] 175 0.042% 99.479% ( 380, 580 ] 312 0.075% 99.554% ( 580, 870 ] 475 0.114% 99.668% ( 870, 1300 ] 605 0.146% 99.814% ( 1300, 1900 ] 457 0.110% 99.924% ( 1900, 2900 ] 356 0.086% 100.010% ( 2900, 4400 ] 270 0.065% 100.075% ( 4400, 6600 ] 95 0.023% 100.098% ( 6600, 9900 ] 15 0.004% 100.101% ** Level 3 read latency histogram (micros): Count: 1524835 Average: 11.1631 StdDev: 173.80 Min: 1 Median: 1.9029 Max: 42164 Percentiles: P50: 1.90 P75: 2.95 P99: 11.71 P99.9: 923.43 P99.99: 1366.57 ------------------------------------------------------ [ 0, 1 ] 51962 3.408% 3.408% # ( 1, 2 ] 786821 51.600% 55.008% ########## ( 2, 3 ] 320016 20.987% 75.995% #### ( 3, 4 ] 158094 10.368% 86.363% ## ( 4, 6 ] 156270 10.248% 96.611% ## ( 6, 10 ] 33665 2.208% 98.819% ( 10, 15 ] 8083 0.530% 99.349% ( 15, 22 ] 962 0.063% 99.412% ( 22, 34 ] 85 0.006% 99.418% ( 34, 51 ] 85 0.006% 99.423% ( 51, 76 ] 151 0.010% 99.433% ( 76, 110 ] 608 0.040% 99.473% ( 110, 170 ] 2165 0.142% 99.615% ( 170, 250 ] 964 0.063% 99.678% ( 250, 380 ] 677 0.044% 99.723% ( 380, 580 ] 1090 0.071% 99.794% ( 580, 870 ] 1432 0.094% 99.888% ( 870, 1300 ] 1450 0.095% 99.983% ( 1300, 1900 ] 924 0.061% 100.044% ( 1900, 2900 ] 817 0.054% 100.097% ( 2900, 4400 ] 894 0.059% 100.156% ( 4400, 6600 ] 222 0.015% 100.171% ( 6600, 9900 ] 50 0.003% 100.174% ( 9900, 14000 ] 13 0.001% 100.175% ( 14000, 22000 ] 15 0.001% 100.176% ( 22000, 33000 ] 7 0.000% 100.176% ( 33000, 50000 ] 3 0.000% 100.176% ** Level 4 read latency histogram (micros): Count: 994162 Average: 25.5797 StdDev: 220.79 Min: 1 Median: 1.9351 Max: 37309 Percentiles: P50: 1.94 P75: 3.23 P99: 443.72 P99.9: 2079.74 P99.99: 2907.47 ------------------------------------------------------ [ 0, 1 ] 34963 3.517% 3.517% # ( 1, 2 ] 494165 49.707% 53.224% ########## ( 2, 3 ] 194560 19.570% 72.794% #### ( 3, 4 ] 96993 9.756% 82.550% ## ( 4, 6 ] 97617 9.819% 92.369% ## ( 6, 10 ] 23420 2.356% 94.725% ( 10, 15 ] 5652 0.569% 95.293% ( 15, 22 ] 585 0.059% 95.352% ( 22, 34 ] 52 0.005% 95.357% ( 34, 51 ] 82 0.008% 95.366% ( 51, 76 ] 112 0.011% 95.377% ( 76, 110 ] 5473 0.551% 95.927% ( 110, 170 ] 18777 1.889% 97.816% ( 170, 250 ] 7336 0.738% 98.554% ( 250, 380 ] 3323 0.334% 98.888% ( 380, 580 ] 3485 0.351% 99.239% ( 580, 870 ] 2787 0.280% 99.519% ( 870, 1300 ] 2229 0.224% 99.743% ( 1300, 1900 ] 1362 0.137% 99.880% ( 1900, 2900 ] 1084 0.109% 99.989% ( 2900, 4400 ] 1121 0.113% 100.102% ( 4400, 6600 ] 317 0.032% 100.134% ( 6600, 9900 ] 55 0.006% 100.140% ( 9900, 14000 ] 7 0.001% 100.140% ( 14000, 22000 ] 10 0.001% 100.141% ( 22000, 33000 ] 4 0.000% 100.142% ( 33000, 50000 ] 1 0.000% 100.142% ** Level 5 read latency histogram (micros): Count: 249826 Average: 330.0709 StdDev: 655.60 Min: 73 Median: 158.6247 Max: 41698 Percentiles: P50: 158.62 P75: 245.49 P99: 3667.71 P99.9: 6427.15 P99.99: 9906.49 ------------------------------------------------------ ( 51, 76 ] 16 0.006% 0.006% ( 76, 110 ] 32536 13.023% 13.030% ### ( 110, 170 ] 113968 45.619% 58.649% ######### ( 170, 250 ] 43289 17.328% 75.976% ### ( 250, 380 ] 17843 7.142% 83.119% # ( 380, 580 ] 16275 6.515% 89.633% # ( 580, 870 ] 9706 3.885% 93.518% # ( 870, 1300 ] 6362 2.547% 96.065% # ( 1300, 1900 ] 3375 1.351% 97.416% ( 1900, 2900 ] 2582 1.034% 98.449% ( 2900, 4400 ] 2688 1.076% 99.525% ( 4400, 6600 ] 1016 0.407% 99.932% ( 6600, 9900 ] 145 0.058% 99.990% ( 9900, 14000 ] 11 0.004% 99.994% ( 14000, 22000 ] 7 0.003% 99.997% ( 22000, 33000 ] 11 0.004% 100.002% ( 33000, 50000 ] 5 0.002% 100.004% ** DB Stats ** Uptime(secs): 31.0 total, 8.7 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 158.97 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:12.402 H:M:S, 40.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 112.81 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:5.065 H:M:S, 58.1 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 255.688 micros/op 15634 ops/sec 60.123 seconds 939996 operations; 15.1 MB/s (238999 of 238999 found) Microseconds per read: Count: 939996 Average: 255.6895 StdDev: 756.49 Min: 2 Median: 170.8666 Max: 115075 Percentiles: P50: 170.87 P75: 234.07 P99: 1481.06 P99.9: 12176.84 P99.99: 30702.56 ------------------------------------------------------ ( 1, 2 ] 6 0.001% 0.001% ( 2, 3 ] 39 0.004% 0.005% ( 3, 4 ] 44 0.005% 0.009% ( 4, 6 ] 94 0.010% 0.019% ( 6, 10 ] 404 0.043% 0.062% ( 10, 15 ] 3954 0.421% 0.483% ( 15, 22 ] 3781 0.402% 0.885% ( 22, 34 ] 9003 0.958% 1.843% ( 34, 51 ] 2021 0.215% 2.058% ( 51, 76 ] 269 0.029% 2.087% ( 76, 110 ] 21701 2.309% 4.395% ( 110, 170 ] 425460 45.262% 49.657% ######### ( 170, 250 ] 297448 31.644% 81.301% ###### ( 250, 380 ] 93792 9.978% 91.279% ## ( 380, 580 ] 43298 4.606% 95.885% # ( 580, 870 ] 19318 2.055% 97.940% ( 870, 1300 ] 8833 0.940% 98.880% ( 1300, 1900 ] 3748 0.399% 99.278% ( 1900, 2900 ] 2482 0.264% 99.542% ( 2900, 4400 ] 2068 0.220% 99.762% ( 4400, 6600 ] 770 0.082% 99.844% ( 6600, 9900 ] 262 0.028% 99.872% ( 9900, 14000 ] 470 0.050% 99.922% ( 14000, 22000 ] 512 0.054% 99.977% ( 22000, 33000 ] 158 0.017% 99.994% ( 33000, 50000 ] 50 0.005% 99.999% ( 50000, 75000 ] 10 0.001% 100.000% ( 110000, 170000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 12986845 rocksdb.bloom.filter.full.positive COUNT : 1066231 rocksdb.bloom.filter.full.true.positive COUNT : 939602 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 394 rocksdb.memtable.miss COUNT : 939602 rocksdb.l0.hit COUNT : 39512 rocksdb.l1.hit COUNT : 8209 rocksdb.l2andup.hit COUNT : 891881 rocksdb.compaction.key.drop.new COUNT : 221044 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5586278 rocksdb.number.keys.read COUNT : 939996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5765038896 rocksdb.bytes.read COUNT : 939996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1677 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 39175267 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 25044817628 rocksdb.compact.write.bytes COUNT : 24810036314 rocksdb.flush.write.bytes COUNT : 5747089535 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 29 rocksdb.number.superversion_acquires COUNT : 1125 rocksdb.number.superversion_releases COUNT : 1093 rocksdb.number.superversion_cleanups COUNT : 280 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 304 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5700668416 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2142208 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 28760939578 rocksdb.non.last.level.read.count COUNT : 6805310 rocksdb.block.checksum.compute.count COUNT : 7140921 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 169.522171 P95 : 540.784696 P99 : 1479.477145 P100 : 115075.000000 COUNT : 939996 SUM : 239539322 rocksdb.db.write.micros P50 : 1.533977 P95 : 4.326094 P99 : 8.985377 P100 : 959200.000000 COUNT : 5586278 SUM : 53068564 rocksdb.compaction.times.micros P50 : 1359090.909091 P95 : 4483333.333333 P99 : 22550000.000000 P100 : 29243746.000000 COUNT : 95 SUM : 182348838 rocksdb.compaction.times.cpu_micros P50 : 840405.405405 P95 : 2150000.000000 P99 : 10010000.000000 P100 : 10353548.000000 COUNT : 95 SUM : 104727900 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 793.875000 P95 : 93375.000000 P99 : 220787.000000 P100 : 220787.000000 COUNT : 79 SUM : 1215428 rocksdb.compaction.outfile.sync.micros P50 : 29017.241379 P95 : 258937.500000 P99 : 510150.000000 P100 : 806532.000000 COUNT : 289 SUM : 19041679 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 227.142857 P95 : 3821.428571 P99 : 6255.333333 P100 : 7972.000000 COUNT : 194 SUM : 170357 rocksdb.table.open.io.micros P50 : 8803.548387 P95 : 19740.639269 P99 : 24739.523810 P100 : 55841.000000 COUNT : 1677 SUM : 14913723 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.838407 P95 : 11.452266 P99 : 194.309817 P100 : 42169.000000 COUNT : 6063630 SUM : 96600134 rocksdb.read.block.get.micros P50 : 142.542256 P95 : 444.257698 P99 : 1417.609340 P100 : 41702.000000 COUNT : 1075627 SUM : 206973550 rocksdb.write.raw.block.micros P50 : 0.532372 P95 : 1.266155 P99 : 3.549932 P100 : 74913.000000 COUNT : 7401084 SUM : 22263877 rocksdb.numfiles.in.singlecompaction P50 : 2.661290 P95 : 5.625000 P99 : 14.250000 P100 : 20.000000 COUNT : 115 SUM : 390 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.568482 P95 : 1240.500382 P99 : 2643.428380 P100 : 959031.000000 COUNT : 193359 SUM : 39189744 rocksdb.sst.read.micros P50 : 1.925223 P95 : 166.002611 P99 : 503.254317 P100 : 42164.000000 COUNT : 6805369 SUM : 283798738 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 939996 SUM : 939996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5586278 SUM : 5765038896 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 223559.322034 P95 : 999400.000000 P99 : 1200304.000000 P100 : 1200304.000000 COUNT : 79 SUM : 25147400 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 305 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0