[2024-07-25 06:29:55.952252] Starting SPDK v24.09-pre git sha1 d005e023b / DPDK 23.11.0 initialization... [2024-07-25 06:29:55.952361] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid80227 ] Set seed to 1721881795949937 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-07-25 06:29:56.048251] app.c: 909:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-07-25 06:29:56.089830] app.c: 603:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-07-25 06:29:56.089881] app.c: 604:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 80227' to capture a snapshot of events at runtime. [2024-07-25 06:29:56.089893] app.c: 609:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-07-25 06:29:56.089909] app.c: 610:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-07-25 06:29:56.089918] app.c: 611:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid80227 for offline analysis/debug. [2024-07-25 06:29:56.090016] reactor.c: 941:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Thu Jul 25 06:30:00 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/07/25-06:30:08 ... thread 0: (1000000,1000000) ops and (118204.9,118204.9) ops/second in (8.459883,8.459883) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 5/4 440.88 MB 12.6 0.0 0.0 0.0 0.8 0.8 0.0 1.0 0.0 115.5 7.09 2.29 9 0.788 0 0 0.0 0.0 L1 3/0 192.78 MB 19.3 0.7 0.5 0.2 0.7 0.5 0.0 1.4 108.7 108.5 6.65 3.26 2 3.326 734K 1078 0.0 0.0 L2 19/14 1.95 GB 4.2 0.0 0.0 0.0 0.0 0.0 0.5 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 43/22 3.33 GB 1.5 5.2 3.5 1.7 5.2 3.5 0.0 1.5 106.7 105.9 49.84 24.54 19 2.623 2195K 2341 0.0 0.0 L4 195/5 11.07 GB 1.1 1.5 1.3 0.2 1.5 1.3 0.0 1.2 106.4 105.8 14.36 7.06 12 1.197 1317K 0 0.0 0.0 L5 1064/0 87.60 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1329/45 104.57 GB 0.0 7.4 5.3 2.1 8.1 6.0 0.5 10.2 97.1 107.0 77.94 37.15 42 1.856 4246K 3419 0.0 0.0 Int 0/0 0.00 KB 0.0 7.4 5.3 2.1 8.1 6.0 0.5 10.2 97.1 107.0 77.94 37.15 42 1.856 4246K 3419 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 7.4 5.3 2.1 7.3 5.2 0.0 0.0 106.9 106.1 70.85 34.86 33 2.147 4246K 3419 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 115.5 7.09 2.29 9 0.788 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 8.6 total, 8.5 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 8.14 GB write, 967.51 MB/s write, 7.39 GB read, 878.35 MB/s read, 77.9 seconds Interval compaction: 8.14 GB write, 985.48 MB/s write, 7.39 GB read, 894.66 MB/s read, 77.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 19, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 19, interval: 24 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 188545 Average: 11.5649 StdDev: 140.76 Min: 1 Median: 1.8754 Max: 7169 Percentiles: P50: 1.88 P75: 2.86 P99: 9.99 P99.9: 2375.03 P99.99: 4209.34 ------------------------------------------------------ [ 0, 1 ] 7174 3.805% 3.805% # ( 1, 2 ] 99495 52.770% 56.575% ########### ( 2, 3 ] 40501 21.481% 78.056% #### ( 3, 4 ] 17566 9.317% 87.372% ## ( 4, 6 ] 17945 9.518% 96.890% ## ( 6, 10 ] 3995 2.119% 99.009% ( 10, 15 ] 668 0.354% 99.363% ( 15, 22 ] 63 0.033% 99.396% ( 22, 34 ] 1 0.001% 99.397% ( 34, 51 ] 2 0.001% 99.398% ( 51, 76 ] 15 0.008% 99.406% ( 76, 110 ] 2 0.001% 99.407% ( 110, 170 ] 5 0.003% 99.410% ( 170, 250 ] 14 0.007% 99.417% ( 250, 380 ] 59 0.031% 99.448% ( 380, 580 ] 114 0.060% 99.509% ( 580, 870 ] 248 0.132% 99.640% ( 870, 1300 ] 237 0.126% 99.766% ( 1300, 1900 ] 187 0.099% 99.865% ( 1900, 2900 ] 142 0.075% 99.941% ( 2900, 4400 ] 109 0.058% 99.998% ( 4400, 6600 ] 23 0.012% 100.011% ( 6600, 9900 ] 2 0.001% 100.012% ** Level 1 read latency histogram (micros): Count: 96497 Average: 13.4478 StdDev: 167.27 Min: 1 Median: 1.9528 Max: 6507 Percentiles: P50: 1.95 P75: 3.00 P99: 10.41 P99.9: 3008.38 P99.99: 4683.49 ------------------------------------------------------ [ 0, 1 ] 2323 2.407% 2.407% ( 1, 2 ] 48199 49.949% 52.356% ########## ( 2, 3 ] 21923 22.719% 75.075% ##### ( 3, 4 ] 9782 10.137% 85.212% ## ( 4, 6 ] 10706 11.095% 96.307% ## ( 6, 10 ] 2569 2.662% 98.969% # ( 10, 15 ] 364 0.377% 99.346% ( 15, 22 ] 41 0.042% 99.389% ( 22, 34 ] 1 0.001% 99.390% ( 34, 51 ] 1 0.001% 99.391% ( 51, 76 ] 6 0.006% 99.397% ( 76, 110 ] 4 0.004% 99.401% ( 110, 170 ] 4 0.004% 99.405% ( 170, 250 ] 19 0.020% 99.425% ( 250, 380 ] 27 0.028% 99.453% ( 380, 580 ] 53 0.055% 99.508% ( 580, 870 ] 87 0.090% 99.598% ( 870, 1300 ] 112 0.116% 99.714% ( 1300, 1900 ] 79 0.082% 99.796% ( 1900, 2900 ] 94 0.097% 99.893% ( 2900, 4400 ] 90 0.093% 99.987% ( 4400, 6600 ] 26 0.027% 100.013% ** Level 2 read latency histogram (micros): Count: 926675 Average: 13.9994 StdDev: 166.79 Min: 1 Median: 2.0331 Max: 8759 Percentiles: P50: 2.03 P75: 3.08 P99: 12.02 P99.9: 1397.17 P99.99: 2003.74 ------------------------------------------------------ [ 0, 1 ] 17362 1.874% 1.874% ( 1, 2 ] 438319 47.300% 49.174% ######### ( 2, 3 ] 231019 24.930% 74.104% ##### ( 3, 4 ] 99242 10.709% 84.813% ## ( 4, 6 ] 101944 11.001% 95.814% ## ( 6, 10 ] 27309 2.947% 98.761% # ( 10, 15 ] 5471 0.590% 99.352% ( 15, 22 ] 729 0.079% 99.430% ( 22, 34 ] 89 0.010% 99.440% ( 34, 51 ] 66 0.007% 99.447% ( 51, 76 ] 85 0.009% 99.456% ( 76, 110 ] 51 0.006% 99.462% ( 110, 170 ] 111 0.012% 99.474% ( 170, 250 ] 187 0.020% 99.494% ( 250, 380 ] 411 0.044% 99.538% ( 380, 580 ] 804 0.087% 99.625% ( 580, 870 ] 1179 0.127% 99.752% ( 870, 1300 ] 1227 0.132% 99.885% ( 1300, 1900 ] 885 0.096% 99.980% ( 1900, 2900 ] 890 0.096% 100.076% ( 2900, 4400 ] 776 0.084% 100.160% ( 4400, 6600 ] 206 0.022% 100.182% ( 6600, 9900 ] 22 0.002% 100.184% ** Level 3 read latency histogram (micros): Count: 757566 Average: 11.3346 StdDev: 146.70 Min: 1 Median: 2.0191 Max: 8282 Percentiles: P50: 2.02 P75: 3.02 P99: 9.85 P99.9: 1202.06 P99.99: 1839.73 ------------------------------------------------------ [ 0, 1 ] 10787 1.424% 1.424% ( 1, 2 ] 364337 48.093% 49.517% ########## ( 2, 3 ] 191120 25.228% 74.745% ##### ( 3, 4 ] 82620 10.906% 85.651% ## ( 4, 6 ] 84008 11.089% 96.740% ## ( 6, 10 ] 17803 2.350% 99.090% ( 10, 15 ] 3521 0.465% 99.555% ( 15, 22 ] 399 0.053% 99.608% ( 22, 34 ] 25 0.003% 99.611% ( 34, 51 ] 40 0.005% 99.616% ( 51, 76 ] 57 0.008% 99.624% ( 76, 110 ] 27 0.004% 99.627% ( 110, 170 ] 50 0.007% 99.634% ( 170, 250 ] 100 0.013% 99.647% ( 250, 380 ] 221 0.029% 99.676% ( 380, 580 ] 420 0.055% 99.732% ( 580, 870 ] 685 0.090% 99.822% ( 870, 1300 ] 762 0.101% 99.923% ( 1300, 1900 ] 565 0.075% 99.997% ( 1900, 2900 ] 511 0.067% 100.065% ( 2900, 4400 ] 543 0.072% 100.137% ( 4400, 6600 ] 120 0.016% 100.152% ( 6600, 9900 ] 17 0.002% 100.155% ** Level 4 read latency histogram (micros): Count: 61695 Average: 96.4197 StdDev: 495.76 Min: 1 Median: 2.0959 Max: 11690 Percentiles: P50: 2.10 P75: 3.45 P99: 2805.28 P99.9: 5583.36 P99.99: 6455.91 ------------------------------------------------------ [ 0, 1 ] 1152 1.867% 1.867% ( 1, 2 ] 28367 45.979% 47.847% ######### ( 2, 3 ] 13858 22.462% 70.309% #### ( 3, 4 ] 6448 10.451% 80.760% ## ( 4, 6 ] 6345 10.284% 91.045% ## ( 6, 10 ] 1693 2.744% 93.789% # ( 10, 15 ] 254 0.412% 94.201% ( 15, 22 ] 15 0.024% 94.225% ( 22, 34 ] 5 0.008% 94.233% ( 34, 51 ] 7 0.011% 94.244% ( 51, 76 ] 3 0.005% 94.249% ( 76, 110 ] 9 0.015% 94.264% ( 110, 170 ] 25 0.041% 94.304% ( 170, 250 ] 86 0.139% 94.444% ( 250, 380 ] 173 0.280% 94.724% ( 380, 580 ] 385 0.624% 95.348% ( 580, 870 ] 639 1.036% 96.384% ( 870, 1300 ] 676 1.096% 97.480% ( 1300, 1900 ] 518 0.840% 98.319% ( 1900, 2900 ] 464 0.752% 99.071% ( 2900, 4400 ] 436 0.707% 99.778% ( 4400, 6600 ] 140 0.227% 100.005% ( 6600, 9900 ] 17 0.028% 100.032% ( 9900, 14000 ] 3 0.005% 100.037% ** Level 5 read latency histogram (micros): Count: 14306 Average: 1714.5317 StdDev: 1429.17 Min: 89 Median: 1203.1799 Max: 12124 Percentiles: P50: 1203.18 P75: 2430.91 P99: 6540.36 P99.9: 9614.20 P99.99: 12124.00 ------------------------------------------------------ ( 76, 110 ] 13 0.091% 0.091% ( 110, 170 ] 78 0.545% 0.636% ( 170, 250 ] 239 1.671% 2.307% ( 250, 380 ] 684 4.781% 7.088% # ( 380, 580 ] 1579 11.037% 18.125% ## ( 580, 870 ] 2540 17.755% 35.880% #### ( 870, 1300 ] 2607 18.223% 54.103% #### ( 1300, 1900 ] 2019 14.113% 68.216% ### ( 1900, 2900 ] 1828 12.778% 80.994% ### ( 2900, 4400 ] 1856 12.974% 93.968% ### ( 4400, 6600 ] 740 5.173% 99.140% # ( 6600, 9900 ] 119 0.832% 99.972% ( 9900, 14000 ] 4 0.028% 100.000% ** DB Stats ** Uptime(secs): 8.6 total, 8.5 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 114.19 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:4.514 H:M:S, 52.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 116.30 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:4.514 H:M:S, 53.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/07/25-06:30:13 ... thread 0: (1000000,2000000) ops and (190338.6,145839.8) ops/second in (5.253794,13.713677) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 7/6 1007.37 MB 12.6 0.8 0.0 0.8 2.6 1.8 0.0 1.4 49.5 163.8 16.53 8.39 26 0.636 831K 820 0.0 0.0 L1 12/12 694.47 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.4 135.5 135.2 10.48 6.41 3 3.492 1441K 2840 0.0 0.0 L2 2/1 183.88 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.5 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 42/8 3.21 GB 2.5 8.6 5.2 3.3 8.5 5.2 0.1 1.6 116.4 115.7 75.30 40.41 25 3.012 4391K 6876 0.0 0.0 L4 182/6 9.99 GB 1.0 3.3 2.9 0.3 3.3 2.9 0.3 1.1 117.2 116.7 28.55 15.48 27 1.057 3101K 85 0.0 0.0 L5 1100/0 90.55 GB 0.9 0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1345/33 105.59 GB 0.0 14.0 9.2 4.9 15.8 10.9 3.8 8.6 109.7 123.6 130.86 70.69 81 1.616 9766K 10K 0.0 0.0 Int 0/0 0.00 KB 0.0 14.0 9.2 4.9 15.8 10.9 3.8 8.6 109.7 123.6 130.86 70.69 81 1.616 9766K 10K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 14.0 9.2 4.9 13.9 9.1 0.0 0.0 121.3 120.7 118.33 65.55 58 2.040 9766K 10K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 150.8 12.53 5.14 23 0.545 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 13.9 total, 13.7 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 15.79 GB write, 1165.55 MB/s write, 14.02 GB read, 1034.55 MB/s read, 130.9 seconds Interval compaction: 15.79 GB write, 1178.91 MB/s write, 14.02 GB read, 1046.40 MB/s read, 130.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 46, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 46, interval: 51 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 477194 Average: 9.0977 StdDev: 117.21 Min: 1 Median: 1.8531 Max: 7682 Percentiles: P50: 1.85 P75: 2.79 P99: 9.58 P99.9: 1716.39 P99.99: 3667.45 ------------------------------------------------------ [ 0, 1 ] 18544 3.886% 3.886% # ( 1, 2 ] 257949 54.055% 57.941% ########### ( 2, 3 ] 102644 21.510% 79.451% #### ( 3, 4 ] 43008 9.013% 88.464% ## ( 4, 6 ] 42802 8.970% 97.434% ## ( 6, 10 ] 8352 1.750% 99.184% ( 10, 15 ] 1386 0.290% 99.474% ( 15, 22 ] 156 0.033% 99.507% ( 22, 34 ] 9 0.002% 99.509% ( 34, 51 ] 33 0.007% 99.516% ( 51, 76 ] 44 0.009% 99.525% ( 76, 110 ] 7 0.001% 99.526% ( 110, 170 ] 32 0.007% 99.533% ( 170, 250 ] 81 0.017% 99.550% ( 250, 380 ] 175 0.037% 99.587% ( 380, 580 ] 317 0.066% 99.653% ( 580, 870 ] 506 0.106% 99.759% ( 870, 1300 ] 465 0.097% 99.857% ( 1300, 1900 ] 298 0.062% 99.919% ( 1900, 2900 ] 238 0.050% 99.969% ( 2900, 4400 ] 196 0.041% 100.010% ( 4400, 6600 ] 36 0.008% 100.018% ( 6600, 9900 ] 4 0.001% 100.018% ** Level 1 read latency histogram (micros): Count: 201809 Average: 12.1846 StdDev: 151.05 Min: 1 Median: 1.9502 Max: 9223 Percentiles: P50: 1.95 P75: 2.94 P99: 10.65 P99.9: 2545.14 P99.99: 4242.96 ------------------------------------------------------ [ 0, 1 ] 4146 2.054% 2.054% ( 1, 2 ] 101831 50.459% 52.514% ########## ( 2, 3 ] 48159 23.864% 76.377% ##### ( 3, 4 ] 19778 9.800% 86.178% ## ( 4, 6 ] 20711 10.263% 96.440% ## ( 6, 10 ] 5077 2.516% 98.956% # ( 10, 15 ] 681 0.337% 99.293% ( 15, 22 ] 78 0.039% 99.332% ( 22, 34 ] 4 0.002% 99.334% ( 34, 51 ] 17 0.008% 99.342% ( 51, 76 ] 14 0.007% 99.349% ( 76, 110 ] 7 0.003% 99.353% ( 110, 170 ] 19 0.009% 99.362% ( 170, 250 ] 62 0.031% 99.393% ( 250, 380 ] 122 0.060% 99.453% ( 380, 580 ] 173 0.086% 99.539% ( 580, 870 ] 243 0.120% 99.660% ( 870, 1300 ] 232 0.115% 99.775% ( 1300, 1900 ] 159 0.079% 99.853% ( 1900, 2900 ] 146 0.072% 99.926% ( 2900, 4400 ] 145 0.072% 99.998% ( 4400, 6600 ] 36 0.018% 100.015% ( 6600, 9900 ] 3 0.001% 100.017% ** Level 2 read latency histogram (micros): Count: 1098142 Average: 13.5405 StdDev: 163.98 Min: 1 Median: 2.0168 Max: 15614 Percentiles: P50: 2.02 P75: 3.04 P99: 11.82 P99.9: 1503.95 P99.99: 2228.92 ------------------------------------------------------ [ 0, 1 ] 18928 1.724% 1.724% ( 1, 2 ] 525526 47.856% 49.580% ########## ( 2, 3 ] 274259 24.975% 74.554% ##### ( 3, 4 ] 116052 10.568% 85.122% ## ( 4, 6 ] 118481 10.789% 95.912% ## ( 6, 10 ] 31692 2.886% 98.798% # ( 10, 15 ] 6101 0.556% 99.353% ( 15, 22 ] 807 0.073% 99.427% ( 22, 34 ] 97 0.009% 99.436% ( 34, 51 ] 81 0.007% 99.443% ( 51, 76 ] 92 0.008% 99.451% ( 76, 110 ] 60 0.005% 99.457% ( 110, 170 ] 128 0.012% 99.468% ( 170, 250 ] 233 0.021% 99.490% ( 250, 380 ] 501 0.046% 99.535% ( 380, 580 ] 930 0.085% 99.620% ( 580, 870 ] 1357 0.124% 99.743% ( 870, 1300 ] 1383 0.126% 99.869% ( 1300, 1900 ] 991 0.090% 99.960% ( 1900, 2900 ] 1016 0.093% 100.052% ( 2900, 4400 ] 884 0.080% 100.133% ( 4400, 6600 ] 226 0.021% 100.153% ( 6600, 9900 ] 25 0.002% 100.156% ( 9900, 14000 ] 1 0.000% 100.156% ( 14000, 22000 ] 1 0.000% 100.156% ** Level 3 read latency histogram (micros): Count: 1626519 Average: 10.4869 StdDev: 137.24 Min: 1 Median: 2.0108 Max: 14292 Percentiles: P50: 2.01 P75: 2.99 P99: 9.74 P99.9: 857.42 P99.99: 1273.82 ------------------------------------------------------ [ 0, 1 ] 18312 1.126% 1.126% ( 1, 2 ] 790443 48.597% 49.723% ########## ( 2, 3 ] 415604 25.552% 75.275% ##### ( 3, 4 ] 175781 10.807% 86.082% ## ( 4, 6 ] 173900 10.692% 96.774% ## ( 6, 10 ] 38748 2.382% 99.156% ( 10, 15 ] 7128 0.438% 99.594% ( 15, 22 ] 803 0.049% 99.643% ( 22, 34 ] 73 0.004% 99.648% ( 34, 51 ] 154 0.009% 99.657% ( 51, 76 ] 146 0.009% 99.666% ( 76, 110 ] 56 0.003% 99.670% ( 110, 170 ] 170 0.010% 99.680% ( 170, 250 ] 335 0.021% 99.701% ( 250, 380 ] 661 0.041% 99.741% ( 380, 580 ] 1135 0.070% 99.811% ( 580, 870 ] 1510 0.093% 99.904% ( 870, 1300 ] 1489 0.092% 99.996% ( 1300, 1900 ] 1014 0.062% 100.058% ( 1900, 2900 ] 1036 0.064% 100.122% ( 2900, 4400 ] 969 0.060% 100.181% ( 4400, 6600 ] 200 0.012% 100.194% ( 6600, 9900 ] 32 0.002% 100.196% ( 9900, 14000 ] 1 0.000% 100.196% ( 14000, 22000 ] 2 0.000% 100.196% ** Level 4 read latency histogram (micros): Count: 90137 Average: 111.9208 StdDev: 509.65 Min: 1 Median: 2.1321 Max: 11690 Percentiles: P50: 2.13 P75: 3.55 P99: 2885.66 P99.9: 5586.62 P99.99: 6541.02 ------------------------------------------------------ [ 0, 1 ] 1512 1.677% 1.677% ( 1, 2 ] 40880 45.353% 47.031% ######### ( 2, 3 ] 20267 22.485% 69.515% #### ( 3, 4 ] 9049 10.039% 79.554% ## ( 4, 6 ] 8796 9.758% 89.313% ## ( 6, 10 ] 2317 2.571% 91.883% # ( 10, 15 ] 330 0.366% 92.250% ( 15, 22 ] 26 0.029% 92.278% ( 22, 34 ] 7 0.008% 92.286% ( 34, 51 ] 16 0.018% 92.304% ( 51, 76 ] 12 0.013% 92.317% ( 76, 110 ] 27 0.030% 92.347% ( 110, 170 ] 92 0.102% 92.449% ( 170, 250 ] 261 0.290% 92.739% ( 250, 380 ] 451 0.500% 93.239% ( 380, 580 ] 914 1.014% 94.253% ( 580, 870 ] 1364 1.513% 95.766% ( 870, 1300 ] 1287 1.428% 97.194% ( 1300, 1900 ] 846 0.939% 98.133% ( 1900, 2900 ] 793 0.880% 99.013% ( 2900, 4400 ] 699 0.775% 99.788% ( 4400, 6600 ] 187 0.207% 99.996% ( 6600, 9900 ] 24 0.027% 100.022% ( 9900, 14000 ] 3 0.003% 100.026% ** Level 5 read latency histogram (micros): Count: 29445 Average: 1372.3345 StdDev: 1267.59 Min: 82 Median: 907.8350 Max: 15151 Percentiles: P50: 907.83 P75: 1786.90 P99: 6203.38 P99.9: 9295.56 P99.99: 11987.92 ------------------------------------------------------ ( 76, 110 ] 67 0.228% 0.228% ( 110, 170 ] 495 1.681% 1.909% ( 170, 250 ] 1113 3.780% 5.689% # ( 250, 380 ] 2448 8.314% 14.002% ## ( 380, 580 ] 4385 14.892% 28.895% ### ( 580, 870 ] 5764 19.575% 48.470% #### ( 870, 1300 ] 5120 17.388% 65.858% ### ( 1300, 1900 ] 3317 11.265% 77.123% ## ( 1900, 2900 ] 2923 9.927% 87.050% ## ( 2900, 4400 ] 2789 9.472% 96.522% ## ( 4400, 6600 ] 890 3.023% 99.545% # ( 6600, 9900 ] 128 0.435% 99.980% ( 9900, 14000 ] 6 0.020% 100.000% ( 14000, 22000 ] 2 0.007% 100.007% ** DB Stats ** Uptime(secs): 13.9 total, 5.3 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 141.89 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.889 H:M:S, 42.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 187.33 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.375 H:M:S, 26.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/07/25-06:30:19 ... thread 0: (1000000,3000000) ops and (191612.7,158457.4) ops/second in (5.218861,18.932538) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/10 1.91 GB 81.8 1.5 0.0 1.5 4.2 2.8 0.0 1.5 62.1 178.6 24.33 13.82 41 0.593 1535K 1732 0.0 0.0 L1 12/12 694.47 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.4 135.5 135.2 10.48 6.41 3 3.492 1441K 2840 0.0 0.0 L2 1/0 55.10 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.5 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 15/0 994.43 MB 1.0 8.8 5.3 3.5 8.7 5.3 0.1 1.6 116.8 116.1 77.14 41.53 26 2.967 4638K 7564 0.0 0.0 L4 171/0 9.67 GB 1.0 6.5 5.2 1.3 6.5 5.2 0.3 1.2 120.0 119.1 55.58 31.01 53 1.049 5728K 658 0.0 0.0 L5 1132/0 93.20 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1345/22 106.48 GB 0.0 18.2 11.6 6.6 20.8 14.2 6.5 7.5 111.1 127.4 167.53 92.77 123 1.362 13M 12K 0.0 0.0 Int 0/0 0.00 KB 0.0 18.2 11.6 6.6 20.8 14.2 6.5 7.5 111.1 127.4 167.53 92.77 123 1.362 13M 12K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 18.2 11.6 6.6 18.1 11.5 0.0 0.0 124.0 123.2 150.15 84.94 87 1.726 13M 12K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 163.1 17.38 7.82 36 0.483 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 19.1 total, 18.9 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 20.84 GB write, 1117.57 MB/s write, 18.18 GB read, 974.85 MB/s read, 167.5 seconds Interval compaction: 20.84 GB write, 1126.85 MB/s write, 18.18 GB read, 982.95 MB/s read, 167.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 68, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 68, interval: 73 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 788107 Average: 10.3635 StdDev: 119.77 Min: 0 Median: 1.7884 Max: 25785 Percentiles: P50: 1.79 P75: 2.69 P99: 147.36 P99.9: 1423.99 P99.99: 3143.96 ------------------------------------------------------ [ 0, 1 ] 54783 6.951% 6.951% # ( 1, 2 ] 430310 54.600% 61.552% ########### ( 2, 3 ] 154580 19.614% 81.166% #### ( 3, 4 ] 62468 7.926% 89.092% ## ( 4, 6 ] 60496 7.676% 96.768% ## ( 6, 10 ] 12216 1.550% 98.318% ( 10, 15 ] 2099 0.266% 98.585% ( 15, 22 ] 230 0.029% 98.614% ( 22, 34 ] 17 0.002% 98.616% ( 34, 51 ] 40 0.005% 98.621% ( 51, 76 ] 48 0.006% 98.627% ( 76, 110 ] 919 0.117% 98.744% ( 110, 170 ] 3246 0.412% 99.156% ( 170, 250 ] 1868 0.237% 99.393% ( 250, 380 ] 1057 0.134% 99.527% ( 380, 580 ] 1125 0.143% 99.669% ( 580, 870 ] 986 0.125% 99.795% ( 870, 1300 ] 742 0.094% 99.889% ( 1300, 1900 ] 435 0.055% 99.944% ( 1900, 2900 ] 318 0.040% 99.984% ( 2900, 4400 ] 284 0.036% 100.020% ( 4400, 6600 ] 57 0.007% 100.028% ( 6600, 9900 ] 6 0.001% 100.028% ( 14000, 22000 ] 2 0.000% 100.029% ( 22000, 33000 ] 1 0.000% 100.029% ** Level 1 read latency histogram (micros): Count: 257436 Average: 12.2208 StdDev: 147.85 Min: 1 Median: 1.9689 Max: 9223 Percentiles: P50: 1.97 P75: 2.99 P99: 14.25 P99.9: 2433.12 P99.99: 4310.33 ------------------------------------------------------ [ 0, 1 ] 5252 2.040% 2.040% ( 1, 2 ] 127429 49.499% 51.539% ########## ( 2, 3 ] 61161 23.758% 75.297% ##### ( 3, 4 ] 25581 9.937% 85.234% ## ( 4, 6 ] 27650 10.741% 95.975% ## ( 6, 10 ] 6976 2.710% 98.684% # ( 10, 15 ] 956 0.371% 99.056% ( 15, 22 ] 148 0.057% 99.113% ( 22, 34 ] 11 0.004% 99.117% ( 34, 51 ] 20 0.008% 99.125% ( 51, 76 ] 19 0.007% 99.133% ( 76, 110 ] 81 0.031% 99.164% ( 110, 170 ] 279 0.108% 99.272% ( 170, 250 ] 225 0.087% 99.360% ( 250, 380 ] 212 0.082% 99.442% ( 380, 580 ] 270 0.105% 99.547% ( 580, 870 ] 335 0.130% 99.677% ( 870, 1300 ] 295 0.115% 99.792% ( 1300, 1900 ] 189 0.073% 99.865% ( 1900, 2900 ] 168 0.065% 99.930% ( 2900, 4400 ] 163 0.063% 99.994% ( 4400, 6600 ] 44 0.017% 100.011% ( 6600, 9900 ] 7 0.003% 100.014% ** Level 2 read latency histogram (micros): Count: 1127131 Average: 13.4151 StdDev: 163.45 Min: 1 Median: 2.0135 Max: 15614 Percentiles: P50: 2.01 P75: 3.03 P99: 11.72 P99.9: 1512.18 P99.99: 2259.77 ------------------------------------------------------ [ 0, 1 ] 19222 1.705% 1.705% ( 1, 2 ] 540534 47.957% 49.662% ########## ( 2, 3 ] 281447 24.970% 74.632% ##### ( 3, 4 ] 118889 10.548% 85.180% ## ( 4, 6 ] 121192 10.752% 95.932% ## ( 6, 10 ] 32435 2.878% 98.810% # ( 10, 15 ] 6211 0.551% 99.361% ( 15, 22 ] 818 0.073% 99.434% ( 22, 34 ] 97 0.009% 99.442% ( 34, 51 ] 87 0.008% 99.450% ( 51, 76 ] 97 0.009% 99.459% ( 76, 110 ] 60 0.005% 99.464% ( 110, 170 ] 128 0.011% 99.475% ( 170, 250 ] 233 0.021% 99.496% ( 250, 380 ] 505 0.045% 99.541% ( 380, 580 ] 938 0.083% 99.624% ( 580, 870 ] 1369 0.121% 99.745% ( 870, 1300 ] 1390 0.123% 99.869% ( 1300, 1900 ] 995 0.088% 99.957% ( 1900, 2900 ] 1032 0.092% 100.049% ( 2900, 4400 ] 903 0.080% 100.129% ( 4400, 6600 ] 230 0.020% 100.149% ( 6600, 9900 ] 25 0.002% 100.151% ( 9900, 14000 ] 2 0.000% 100.152% ( 14000, 22000 ] 1 0.000% 100.152% ** Level 3 read latency histogram (micros): Count: 2145970 Average: 10.6268 StdDev: 152.02 Min: 1 Median: 2.0024 Max: 51968 Percentiles: P50: 2.00 P75: 2.99 P99: 9.73 P99.9: 838.97 P99.99: 1270.90 ------------------------------------------------------ [ 0, 1 ] 25463 1.187% 1.187% ( 1, 2 ] 1046235 48.753% 49.940% ########## ( 2, 3 ] 544937 25.394% 75.334% ##### ( 3, 4 ] 230142 10.724% 86.058% ## ( 4, 6 ] 229697 10.704% 96.762% ## ( 6, 10 ] 51522 2.401% 99.162% ( 10, 15 ] 9459 0.441% 99.603% ( 15, 22 ] 1070 0.050% 99.653% ( 22, 34 ] 91 0.004% 99.657% ( 34, 51 ] 196 0.009% 99.666% ( 51, 76 ] 192 0.009% 99.675% ( 76, 110 ] 112 0.005% 99.681% ( 110, 170 ] 339 0.016% 99.696% ( 170, 250 ] 459 0.021% 99.718% ( 250, 380 ] 822 0.038% 99.756% ( 380, 580 ] 1411 0.066% 99.822% ( 580, 870 ] 1878 0.088% 99.909% ( 870, 1300 ] 1856 0.086% 99.996% ( 1300, 1900 ] 1295 0.060% 100.056% ( 1900, 2900 ] 1329 0.062% 100.118% ( 2900, 4400 ] 1312 0.061% 100.179% ( 4400, 6600 ] 299 0.014% 100.193% ( 6600, 9900 ] 57 0.003% 100.196% ( 9900, 14000 ] 6 0.000% 100.196% ( 14000, 22000 ] 6 0.000% 100.196% ( 22000, 33000 ] 3 0.000% 100.197% ( 33000, 50000 ] 1 0.000% 100.197% ( 50000, 75000 ] 1 0.000% 100.197% ** Level 4 read latency histogram (micros): Count: 339506 Average: 48.1991 StdDev: 340.62 Min: 1 Median: 2.0108 Max: 37833 Percentiles: P50: 2.01 P75: 3.21 P99: 1294.20 P99.9: 3887.93 P99.99: 4358.49 ------------------------------------------------------ [ 0, 1 ] 6265 1.845% 1.845% ( 1, 2 ] 162636 47.904% 49.749% ########## ( 2, 3 ] 78616 23.156% 72.905% ##### ( 3, 4 ] 33802 9.956% 82.861% ## ( 4, 6 ] 33679 9.920% 92.781% ## ( 6, 10 ] 9043 2.664% 95.445% # ( 10, 15 ] 1370 0.404% 95.848% ( 15, 22 ] 165 0.049% 95.897% ( 22, 34 ] 16 0.005% 95.902% ( 34, 51 ] 29 0.009% 95.910% ( 51, 76 ] 38 0.011% 95.921% ( 76, 110 ] 495 0.146% 96.067% ( 110, 170 ] 1694 0.499% 96.566% ( 170, 250 ] 1191 0.351% 96.917% ( 250, 380 ] 1035 0.305% 97.222% ( 380, 580 ] 1736 0.511% 97.733% ( 580, 870 ] 2250 0.663% 98.396% ( 870, 1300 ] 2079 0.612% 99.008% ( 1300, 1900 ] 1304 0.384% 99.392% ( 1900, 2900 ] 1082 0.319% 99.711% ( 2900, 4400 ] 974 0.287% 99.998% ( 4400, 6600 ] 262 0.077% 100.075% ( 6600, 9900 ] 43 0.013% 100.088% ( 9900, 14000 ] 7 0.002% 100.090% ( 14000, 22000 ] 1 0.000% 100.090% ( 22000, 33000 ] 3 0.001% 100.091% ( 33000, 50000 ] 1 0.000% 100.091% ** Level 5 read latency histogram (micros): Count: 59557 Average: 926.9064 StdDev: 1166.34 Min: 75 Median: 519.0371 Max: 38246 Percentiles: P50: 519.04 P75: 1141.24 P99: 5699.01 P99.9: 8802.43 P99.99: 11972.70 ------------------------------------------------------ ( 51, 76 ] 3 0.005% 0.005% ( 76, 110 ] 2787 4.680% 4.685% # ( 110, 170 ] 10097 16.954% 21.638% ### ( 170, 250 ] 6326 10.622% 32.260% ## ( 250, 380 ] 5353 8.988% 41.248% ## ( 380, 580 ] 7498 12.590% 53.838% ### ( 580, 870 ] 8257 13.864% 67.702% ### ( 870, 1300 ] 6891 11.570% 79.272% ## ( 1300, 1900 ] 4263 7.158% 86.430% # ( 1900, 2900 ] 3491 5.862% 92.291% # ( 2900, 4400 ] 3366 5.652% 97.943% # ( 4400, 6600 ] 1066 1.790% 99.733% ( 6600, 9900 ] 149 0.250% 99.983% ( 9900, 14000 ] 8 0.013% 99.997% ( 14000, 22000 ] 2 0.003% 100.000% ( 22000, 33000 ] 2 0.003% 100.003% ( 33000, 50000 ] 3 0.005% 100.008% ** DB Stats ** Uptime(secs): 19.1 total, 5.2 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 154.65 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.393 H:M:S, 38.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 188.58 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.504 H:M:S, 28.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/07/25-06:30:22 ... thread 0: (1000000,4000000) ops and (291379.3,178855.0) ops/second in (3.431953,22.364491) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/10 2.89 GB 182.5 2.5 0.0 2.5 6.2 3.7 0.0 1.7 80.4 202.8 31.36 20.33 61 0.514 2558K 2735 0.0 0.0 L1 12/12 694.47 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.4 135.5 135.2 10.48 6.41 3 3.492 1441K 2840 0.0 0.0 L2 1/0 55.10 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.5 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 15/0 994.43 MB 1.0 8.8 5.3 3.5 8.7 5.3 0.1 1.6 116.8 116.1 77.14 41.53 26 2.967 4638K 7564 0.0 0.0 L4 171/0 9.67 GB 1.0 6.5 5.2 1.3 6.5 5.2 0.3 1.2 120.0 119.1 55.58 31.01 53 1.049 5728K 658 0.0 0.0 L5 1132/0 93.20 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1349/22 107.46 GB 0.0 19.2 11.6 7.6 22.8 15.2 6.5 6.1 112.4 133.8 174.56 99.28 143 1.221 14M 13K 0.0 0.0 Int 0/0 0.00 KB 0.0 19.2 11.6 7.6 22.8 15.2 6.5 6.1 112.4 133.8 174.56 99.28 143 1.221 14M 13K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 19.2 11.6 7.6 19.1 11.5 0.0 0.0 127.4 126.7 154.00 88.71 91 1.692 14M 13K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 186.9 20.56 10.58 52 0.395 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 22.5 total, 22.4 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 22.80 GB write, 1036.75 MB/s write, 19.16 GB read, 871.07 MB/s read, 174.6 seconds Interval compaction: 22.80 GB write, 1044.03 MB/s write, 19.16 GB read, 877.19 MB/s read, 174.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 68, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 68, interval: 73 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1063064 Average: 11.4790 StdDev: 108.05 Min: 0 Median: 1.6913 Max: 25785 Percentiles: P50: 1.69 P75: 2.48 P99: 211.41 P99.9: 1140.84 P99.99: 2528.60 ------------------------------------------------------ [ 0, 1 ] 126360 11.886% 11.886% ## ( 1, 2 ] 586112 55.134% 67.021% ########### ( 2, 3 ] 174998 16.462% 83.482% ### ( 3, 4 ] 68284 6.423% 89.906% # ( 4, 6 ] 63640 5.986% 95.892% # ( 6, 10 ] 12890 1.213% 97.105% ( 10, 15 ] 2408 0.227% 97.331% ( 15, 22 ] 273 0.026% 97.357% ( 22, 34 ] 21 0.002% 97.359% ( 34, 51 ] 41 0.004% 97.363% ( 51, 76 ] 50 0.005% 97.367% ( 76, 110 ] 3044 0.286% 97.654% ( 110, 170 ] 11360 1.069% 98.722% ( 170, 250 ] 5703 0.536% 99.259% ( 250, 380 ] 2630 0.247% 99.506% ( 380, 580 ] 2300 0.216% 99.723% ( 580, 870 ] 1378 0.130% 99.852% ( 870, 1300 ] 808 0.076% 99.928% ( 1300, 1900 ] 449 0.042% 99.970% ( 1900, 2900 ] 332 0.031% 100.002% ( 2900, 4400 ] 288 0.027% 100.029% ( 4400, 6600 ] 57 0.005% 100.034% ( 6600, 9900 ] 6 0.001% 100.035% ( 14000, 22000 ] 2 0.000% 100.035% ( 22000, 33000 ] 1 0.000% 100.035% ** Level 1 read latency histogram (micros): Count: 273939 Average: 12.5740 StdDev: 144.34 Min: 1 Median: 2.0158 Max: 9223 Percentiles: P50: 2.02 P75: 3.12 P99: 140.54 P99.9: 2338.95 P99.99: 4285.95 ------------------------------------------------------ [ 0, 1 ] 5387 1.966% 1.966% ( 1, 2 ] 130541 47.653% 49.620% ########## ( 2, 3 ] 66123 24.138% 73.758% ##### ( 3, 4 ] 27511 10.043% 83.800% ## ( 4, 6 ] 31620 11.543% 95.343% ## ( 6, 10 ] 7886 2.879% 98.222% # ( 10, 15 ] 1166 0.426% 98.648% ( 15, 22 ] 249 0.091% 98.738% ( 22, 34 ] 17 0.006% 98.745% ( 34, 51 ] 20 0.007% 98.752% ( 51, 76 ] 19 0.007% 98.759% ( 76, 110 ] 228 0.083% 98.842% ( 110, 170 ] 850 0.310% 99.152% ( 170, 250 ] 470 0.172% 99.324% ( 250, 380 ] 310 0.113% 99.437% ( 380, 580 ] 342 0.125% 99.562% ( 580, 870 ] 365 0.133% 99.695% ( 870, 1300 ] 297 0.108% 99.804% ( 1300, 1900 ] 189 0.069% 99.873% ( 1900, 2900 ] 171 0.062% 99.935% ( 2900, 4400 ] 163 0.060% 99.995% ( 4400, 6600 ] 44 0.016% 100.011% ( 6600, 9900 ] 7 0.003% 100.013% ** Level 2 read latency histogram (micros): Count: 1127131 Average: 13.4151 StdDev: 163.45 Min: 1 Median: 2.0135 Max: 15614 Percentiles: P50: 2.01 P75: 3.03 P99: 11.72 P99.9: 1512.18 P99.99: 2259.77 ------------------------------------------------------ [ 0, 1 ] 19222 1.705% 1.705% ( 1, 2 ] 540534 47.957% 49.662% ########## ( 2, 3 ] 281447 24.970% 74.632% ##### ( 3, 4 ] 118889 10.548% 85.180% ## ( 4, 6 ] 121192 10.752% 95.932% ## ( 6, 10 ] 32435 2.878% 98.810% # ( 10, 15 ] 6211 0.551% 99.361% ( 15, 22 ] 818 0.073% 99.434% ( 22, 34 ] 97 0.009% 99.442% ( 34, 51 ] 87 0.008% 99.450% ( 51, 76 ] 97 0.009% 99.459% ( 76, 110 ] 60 0.005% 99.464% ( 110, 170 ] 128 0.011% 99.475% ( 170, 250 ] 233 0.021% 99.496% ( 250, 380 ] 505 0.045% 99.541% ( 380, 580 ] 938 0.083% 99.624% ( 580, 870 ] 1369 0.121% 99.745% ( 870, 1300 ] 1390 0.123% 99.869% ( 1300, 1900 ] 995 0.088% 99.957% ( 1900, 2900 ] 1032 0.092% 100.049% ( 2900, 4400 ] 903 0.080% 100.129% ( 4400, 6600 ] 230 0.020% 100.149% ( 6600, 9900 ] 25 0.002% 100.151% ( 9900, 14000 ] 2 0.000% 100.152% ( 14000, 22000 ] 1 0.000% 100.152% ** Level 3 read latency histogram (micros): Count: 2146806 Average: 10.6994 StdDev: 152.06 Min: 1 Median: 2.0031 Max: 51968 Percentiles: P50: 2.00 P75: 2.99 P99: 9.79 P99.9: 840.05 P99.99: 1271.64 ------------------------------------------------------ [ 0, 1 ] 25463 1.186% 1.186% ( 1, 2 ] 1046235 48.734% 49.921% ########## ( 2, 3 ] 544937 25.384% 75.304% ##### ( 3, 4 ] 230142 10.720% 86.024% ## ( 4, 6 ] 229697 10.699% 96.724% ## ( 6, 10 ] 51522 2.400% 99.124% ( 10, 15 ] 9459 0.441% 99.564% ( 15, 22 ] 1070 0.050% 99.614% ( 22, 34 ] 91 0.004% 99.619% ( 34, 51 ] 196 0.009% 99.628% ( 51, 76 ] 193 0.009% 99.637% ( 76, 110 ] 214 0.010% 99.647% ( 110, 170 ] 743 0.035% 99.681% ( 170, 250 ] 651 0.030% 99.712% ( 250, 380 ] 889 0.041% 99.753% ( 380, 580 ] 1466 0.068% 99.821% ( 580, 870 ] 1886 0.088% 99.909% ( 870, 1300 ] 1860 0.087% 99.996% ( 1300, 1900 ] 1298 0.060% 100.056% ( 1900, 2900 ] 1329 0.062% 100.118% ( 2900, 4400 ] 1312 0.061% 100.179% ( 4400, 6600 ] 299 0.014% 100.193% ( 6600, 9900 ] 57 0.003% 100.196% ( 9900, 14000 ] 6 0.000% 100.196% ( 14000, 22000 ] 6 0.000% 100.196% ( 22000, 33000 ] 3 0.000% 100.196% ( 33000, 50000 ] 1 0.000% 100.197% ( 50000, 75000 ] 1 0.000% 100.197% ** Level 4 read latency histogram (micros): Count: 347652 Average: 51.2430 StdDev: 337.89 Min: 1 Median: 2.0613 Max: 37833 Percentiles: P50: 2.06 P75: 3.38 P99: 1278.83 P99.9: 3875.92 P99.99: 4357.28 ------------------------------------------------------ [ 0, 1 ] 6265 1.802% 1.802% ( 1, 2 ] 162723 46.806% 48.608% ######### ( 2, 3 ] 78938 22.706% 71.314% ##### ( 3, 4 ] 33958 9.768% 81.082% ## ( 4, 6 ] 33855 9.738% 90.820% ## ( 6, 10 ] 9064 2.607% 93.428% # ( 10, 15 ] 1378 0.396% 93.824% ( 15, 22 ] 167 0.048% 93.872% ( 22, 34 ] 16 0.005% 93.877% ( 34, 51 ] 29 0.008% 93.885% ( 51, 76 ] 39 0.011% 93.896% ( 76, 110 ] 1507 0.433% 94.330% ( 110, 170 ] 5275 1.517% 95.847% ( 170, 250 ] 2694 0.775% 96.622% ( 250, 380 ] 1663 0.478% 97.100% ( 380, 580 ] 2206 0.635% 97.735% ( 580, 870 ] 2399 0.690% 98.425% ( 870, 1300 ] 2103 0.605% 99.030% ( 1300, 1900 ] 1308 0.376% 99.406% ( 1900, 2900 ] 1083 0.312% 99.718% ( 2900, 4400 ] 975 0.280% 99.998% ( 4400, 6600 ] 262 0.075% 100.073% ( 6600, 9900 ] 43 0.012% 100.086% ( 9900, 14000 ] 7 0.002% 100.088% ( 14000, 22000 ] 1 0.000% 100.088% ( 22000, 33000 ] 3 0.001% 100.089% ( 33000, 50000 ] 1 0.000% 100.089% ** Level 5 read latency histogram (micros): Count: 103366 Average: 617.4765 StdDev: 960.35 Min: 74 Median: 230.1531 Max: 38246 Percentiles: P50: 230.15 P75: 665.44 P99: 4784.57 P99.9: 7721.42 P99.99: 9781.81 ------------------------------------------------------ ( 51, 76 ] 7 0.007% 0.007% ( 76, 110 ] 9084 8.788% 8.795% ## ( 110, 170 ] 31102 30.089% 38.884% ###### ( 170, 250 ] 15281 14.783% 53.668% ### ( 250, 380 ] 8959 8.667% 62.335% ## ( 380, 580 ] 10403 10.064% 72.399% ## ( 580, 870 ] 9125 8.828% 81.227% ## ( 870, 1300 ] 7015 6.787% 88.013% # ( 1300, 1900 ] 4293 4.153% 92.167% # ( 1900, 2900 ] 3509 3.395% 95.561% # ( 2900, 4400 ] 3368 3.258% 98.820% # ( 4400, 6600 ] 1066 1.031% 99.851% ( 6600, 9900 ] 149 0.144% 99.995% ( 9900, 14000 ] 8 0.008% 100.003% ( 14000, 22000 ] 2 0.002% 100.005% ( 22000, 33000 ] 2 0.002% 100.007% ( 33000, 50000 ] 3 0.003% 100.010% ** DB Stats ** Uptime(secs): 22.5 total, 3.4 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 174.78 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.393 H:M:S, 32.8 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 286.77 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/07/25-06:30:29 ... thread 0: (1000000,5000000) ops and (153428.1,173117.0) ops/second in (6.517710,28.882201) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/13 3.87 GB 151.0 3.7 0.0 3.7 8.4 4.7 0.0 1.8 96.5 220.3 39.15 27.62 82 0.477 3837K 3989 0.0 0.0 L1 12/12 694.47 MB 0.0 1.4 1.0 0.4 1.4 1.0 0.0 1.4 135.5 135.2 10.48 6.41 3 3.492 1441K 2840 0.0 0.0 L2 1/0 55.10 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.5 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 15/0 994.43 MB 1.0 8.8 5.3 3.5 8.7 5.3 0.1 1.6 116.8 116.1 77.14 41.53 26 2.967 4638K 7564 0.0 0.0 L4 171/0 9.67 GB 1.0 6.5 5.2 1.3 6.5 5.2 0.3 1.2 120.0 119.1 55.58 31.01 53 1.049 5728K 658 0.0 0.0 L5 1132/0 93.20 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1350/25 108.44 GB 0.0 20.4 11.6 8.8 25.0 16.2 6.5 5.3 114.5 140.5 182.35 106.57 164 1.112 15M 15K 0.0 0.0 Int 0/0 0.00 KB 0.0 20.4 11.6 8.8 25.0 16.2 6.5 5.3 114.5 140.5 182.35 106.57 164 1.112 15M 15K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 20.4 11.6 8.8 20.3 11.5 0.0 0.0 131.6 130.8 158.71 93.33 96 1.653 15M 15K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 205.2 23.64 13.23 68 0.348 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 29.0 total, 28.9 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 25.02 GB write, 882.12 MB/s write, 20.39 GB read, 718.96 MB/s read, 182.4 seconds Interval compaction: 25.02 GB write, 886.92 MB/s write, 20.39 GB read, 722.87 MB/s read, 182.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 9, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 9, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 68, pending-compaction-bytes-delays: 5, pending-compaction-bytes-stops: 0, total-delays: 14, total-stops: 68, interval: 82 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1582079 Average: 12.5269 StdDev: 94.86 Min: 0 Median: 1.6090 Max: 25785 Percentiles: P50: 1.61 P75: 2.25 P99: 229.31 P99.9: 827.39 P99.99: 1652.67 ------------------------------------------------------ [ 0, 1 ] 259322 16.391% 16.391% ### ( 1, 2 ] 873069 55.185% 71.576% ########### ( 2, 3 ] 215638 13.630% 85.206% ### ( 3, 4 ] 81557 5.155% 90.361% # ( 4, 6 ] 71497 4.519% 94.880% # ( 6, 10 ] 14190 0.897% 95.777% ( 10, 15 ] 2659 0.168% 95.945% ( 15, 22 ] 303 0.019% 95.965% ( 22, 34 ] 23 0.001% 95.966% ( 34, 51 ] 47 0.003% 95.969% ( 51, 76 ] 81 0.005% 95.974% ( 76, 110 ] 8420 0.532% 96.506% ( 110, 170 ] 30067 1.900% 98.407% ( 170, 250 ] 12659 0.800% 99.207% ( 250, 380 ] 5181 0.327% 99.534% ( 380, 580 ] 4152 0.262% 99.797% ( 580, 870 ] 1913 0.121% 99.918% ( 870, 1300 ] 873 0.055% 99.973% ( 1300, 1900 ] 459 0.029% 100.002% ( 1900, 2900 ] 341 0.022% 100.024% ( 2900, 4400 ] 303 0.019% 100.043% ( 4400, 6600 ] 57 0.004% 100.046% ( 6600, 9900 ] 6 0.000% 100.047% ( 14000, 22000 ] 2 0.000% 100.047% ( 22000, 33000 ] 1 0.000% 100.047% ** Level 1 read latency histogram (micros): Count: 306302 Average: 13.1418 StdDev: 137.91 Min: 1 Median: 2.1753 Max: 9223 Percentiles: P50: 2.18 P75: 3.57 P99: 171.39 P99.9: 2148.24 P99.99: 4238.75 ------------------------------------------------------ [ 0, 1 ] 5610 1.832% 1.832% ( 1, 2 ] 135267 44.161% 45.993% ######### ( 2, 3 ] 70034 22.864% 68.857% ##### ( 3, 4 ] 32765 10.697% 79.554% ## ( 4, 6 ] 44339 14.476% 94.030% ### ( 6, 10 ] 10239 3.343% 97.373% # ( 10, 15 ] 1606 0.524% 97.897% ( 15, 22 ] 521 0.170% 98.067% ( 22, 34 ] 50 0.016% 98.083% ( 34, 51 ] 22 0.007% 98.090% ( 51, 76 ] 20 0.007% 98.097% ( 76, 110 ] 620 0.202% 98.299% ( 110, 170 ] 2130 0.695% 98.995% ( 170, 250 ] 918 0.300% 99.294% ( 250, 380 ] 465 0.152% 99.446% ( 380, 580 ] 453 0.148% 99.594% ( 580, 870 ] 403 0.132% 99.726% ( 870, 1300 ] 302 0.099% 99.824% ( 1300, 1900 ] 189 0.062% 99.886% ( 1900, 2900 ] 172 0.056% 99.942% ( 2900, 4400 ] 164 0.054% 99.996% ( 4400, 6600 ] 44 0.014% 100.010% ( 6600, 9900 ] 7 0.002% 100.012% ** Level 2 read latency histogram (micros): Count: 1127131 Average: 13.4151 StdDev: 163.45 Min: 1 Median: 2.0135 Max: 15614 Percentiles: P50: 2.01 P75: 3.03 P99: 11.72 P99.9: 1512.18 P99.99: 2259.77 ------------------------------------------------------ [ 0, 1 ] 19222 1.705% 1.705% ( 1, 2 ] 540534 47.957% 49.662% ########## ( 2, 3 ] 281447 24.970% 74.632% ##### ( 3, 4 ] 118889 10.548% 85.180% ## ( 4, 6 ] 121192 10.752% 95.932% ## ( 6, 10 ] 32435 2.878% 98.810% # ( 10, 15 ] 6211 0.551% 99.361% ( 15, 22 ] 818 0.073% 99.434% ( 22, 34 ] 97 0.009% 99.442% ( 34, 51 ] 87 0.008% 99.450% ( 51, 76 ] 97 0.009% 99.459% ( 76, 110 ] 60 0.005% 99.464% ( 110, 170 ] 128 0.011% 99.475% ( 170, 250 ] 233 0.021% 99.496% ( 250, 380 ] 505 0.045% 99.541% ( 380, 580 ] 938 0.083% 99.624% ( 580, 870 ] 1369 0.121% 99.745% ( 870, 1300 ] 1390 0.123% 99.869% ( 1300, 1900 ] 995 0.088% 99.957% ( 1900, 2900 ] 1032 0.092% 100.049% ( 2900, 4400 ] 903 0.080% 100.129% ( 4400, 6600 ] 230 0.020% 100.149% ( 6600, 9900 ] 25 0.002% 100.151% ( 9900, 14000 ] 2 0.000% 100.152% ( 14000, 22000 ] 1 0.000% 100.152% ** Level 3 read latency histogram (micros): Count: 2148498 Average: 10.8386 StdDev: 152.13 Min: 1 Median: 2.0047 Max: 51968 Percentiles: P50: 2.00 P75: 2.99 P99: 9.92 P99.9: 841.59 P99.99: 1272.17 ------------------------------------------------------ [ 0, 1 ] 25463 1.185% 1.185% ( 1, 2 ] 1046235 48.696% 49.881% ########## ( 2, 3 ] 544937 25.364% 75.245% ##### ( 3, 4 ] 230142 10.712% 85.957% ## ( 4, 6 ] 229697 10.691% 96.648% ## ( 6, 10 ] 51522 2.398% 99.046% ( 10, 15 ] 9459 0.440% 99.486% ( 15, 22 ] 1070 0.050% 99.536% ( 22, 34 ] 91 0.004% 99.540% ( 34, 51 ] 196 0.009% 99.549% ( 51, 76 ] 193 0.009% 99.558% ( 76, 110 ] 470 0.022% 99.580% ( 110, 170 ] 1640 0.076% 99.656% ( 170, 250 ] 949 0.044% 99.701% ( 250, 380 ] 1010 0.047% 99.748% ( 380, 580 ] 1549 0.072% 99.820% ( 580, 870 ] 1914 0.089% 99.909% ( 870, 1300 ] 1867 0.087% 99.996% ( 1300, 1900 ] 1298 0.060% 100.056% ( 1900, 2900 ] 1331 0.062% 100.118% ( 2900, 4400 ] 1312 0.061% 100.179% ( 4400, 6600 ] 299 0.014% 100.193% ( 6600, 9900 ] 57 0.003% 100.196% ( 9900, 14000 ] 6 0.000% 100.196% ( 14000, 22000 ] 6 0.000% 100.196% ( 22000, 33000 ] 3 0.000% 100.196% ( 33000, 50000 ] 1 0.000% 100.196% ( 50000, 75000 ] 1 0.000% 100.196% ** Level 4 read latency histogram (micros): Count: 364279 Average: 56.7785 StdDev: 332.10 Min: 1 Median: 2.1651 Max: 37833 Percentiles: P50: 2.17 P75: 3.73 P99: 1247.25 P99.9: 3851.47 P99.99: 4354.82 ------------------------------------------------------ [ 0, 1 ] 6265 1.720% 1.720% ( 1, 2 ] 162811 44.694% 46.414% ######### ( 2, 3 ] 79126 21.721% 68.135% #### ( 3, 4 ] 34086 9.357% 77.492% ## ( 4, 6 ] 33988 9.330% 86.822% ## ( 6, 10 ] 9083 2.493% 89.316% ( 10, 15 ] 1381 0.379% 89.695% ( 15, 22 ] 167 0.046% 89.741% ( 22, 34 ] 16 0.004% 89.745% ( 34, 51 ] 29 0.008% 89.753% ( 51, 76 ] 42 0.012% 89.765% ( 76, 110 ] 4194 1.151% 90.916% ( 110, 170 ] 13886 3.812% 94.728% # ( 170, 250 ] 5438 1.493% 96.221% ( 250, 380 ] 2650 0.727% 96.948% ( 380, 580 ] 3011 0.827% 97.775% ( 580, 870 ] 2598 0.713% 98.488% ( 870, 1300 ] 2126 0.584% 99.072% ( 1300, 1900 ] 1311 0.360% 99.431% ( 1900, 2900 ] 1087 0.298% 99.730% ( 2900, 4400 ] 977 0.268% 99.998% ( 4400, 6600 ] 262 0.072% 100.070% ( 6600, 9900 ] 43 0.012% 100.082% ( 9900, 14000 ] 7 0.002% 100.084% ( 14000, 22000 ] 1 0.000% 100.084% ( 22000, 33000 ] 3 0.001% 100.085% ( 33000, 50000 ] 1 0.000% 100.085% ** Level 5 read latency histogram (micros): Count: 192553 Average: 414.2929 StdDev: 741.52 Min: 74 Median: 165.2425 Max: 38246 Percentiles: P50: 165.24 P75: 348.32 P99: 4081.99 P99.9: 6499.80 P99.99: 9362.80 ------------------------------------------------------ ( 51, 76 ] 20 0.010% 0.010% ( 76, 110 ] 24444 12.695% 12.705% ### ( 110, 170 ] 77997 40.507% 53.212% ######## ( 170, 250 ] 31056 16.129% 69.340% ### ( 250, 380 ] 14409 7.483% 76.824% # ( 380, 580 ] 14708 7.638% 84.462% ## ( 580, 870 ] 10279 5.338% 89.800% # ( 870, 1300 ] 7193 3.736% 93.536% # ( 1300, 1900 ] 4332 2.250% 95.786% ( 1900, 2900 ] 3530 1.833% 97.619% ( 2900, 4400 ] 3375 1.753% 99.372% ( 4400, 6600 ] 1066 0.554% 99.925% ( 6600, 9900 ] 149 0.077% 100.003% ( 9900, 14000 ] 8 0.004% 100.007% ( 14000, 22000 ] 2 0.001% 100.008% ( 22000, 33000 ] 2 0.001% 100.009% ( 33000, 50000 ] 3 0.002% 100.010% ** DB Stats ** Uptime(secs): 29.0 total, 6.5 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 169.45 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:10.286 H:M:S, 35.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 151.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.893 H:M:S, 44.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 265.141 micros/op 15077 ops/sec 60.156 seconds 906996 operations; 14.6 MB/s (226999 of 226999 found) Microseconds per read: Count: 906996 Average: 265.1439 StdDev: 748.33 Min: 2 Median: 169.1967 Max: 74961 Percentiles: P50: 169.20 P75: 234.82 P99: 1902.21 P99.9: 12091.09 P99.99: 29452.35 ------------------------------------------------------ ( 1, 2 ] 5 0.001% 0.001% ( 2, 3 ] 40 0.004% 0.005% ( 3, 4 ] 47 0.005% 0.010% ( 4, 6 ] 97 0.011% 0.021% ( 6, 10 ] 338 0.037% 0.058% ( 10, 15 ] 3386 0.373% 0.431% ( 15, 22 ] 3642 0.402% 0.833% ( 22, 34 ] 7492 0.826% 1.659% ( 34, 51 ] 1578 0.174% 1.833% ( 51, 76 ] 215 0.024% 1.857% ( 76, 110 ] 22349 2.464% 4.321% ( 110, 170 ] 419931 46.299% 50.620% ######### ( 170, 250 ] 272927 30.091% 80.711% ###### ( 250, 380 ] 92471 10.195% 90.906% ## ( 380, 580 ] 40331 4.447% 95.353% # ( 580, 870 ] 18700 2.062% 97.415% ( 870, 1300 ] 9531 1.051% 98.466% ( 1300, 1900 ] 4838 0.533% 98.999% ( 1900, 2900 ] 3641 0.401% 99.401% ( 2900, 4400 ] 3119 0.344% 99.744% ( 4400, 6600 ] 902 0.099% 99.844% ( 6600, 9900 ] 276 0.030% 99.874% ( 9900, 14000 ] 436 0.048% 99.922% ( 14000, 22000 ] 511 0.056% 99.979% ( 22000, 33000 ] 151 0.017% 99.995% ( 33000, 50000 ] 32 0.004% 99.999% ( 50000, 75000 ] 10 0.001% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 13089470 rocksdb.bloom.filter.full.positive COUNT : 1033654 rocksdb.bloom.filter.full.true.positive COUNT : 906588 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 408 rocksdb.memtable.miss COUNT : 906588 rocksdb.l0.hit COUNT : 33858 rocksdb.l1.hit COUNT : 8210 rocksdb.l2andup.hit COUNT : 864520 rocksdb.compaction.key.drop.new COUNT : 145156 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5596555 rocksdb.number.keys.read COUNT : 906996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5775644760 rocksdb.bytes.read COUNT : 906996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1691 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 38908445 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 27029031035 rocksdb.compact.write.bytes COUNT : 26872346029 rocksdb.flush.write.bytes COUNT : 5747004031 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 52 rocksdb.number.superversion_acquires COUNT : 1169 rocksdb.number.superversion_releases COUNT : 1136 rocksdb.number.superversion_cleanups COUNT : 292 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 350 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5700683776 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2236416 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 30869714551 rocksdb.non.last.level.read.count COUNT : 7318676 rocksdb.block.checksum.compute.count COUNT : 7588626 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 168.374911 P95 : 563.553006 P99 : 1900.285950 P100 : 74959.000000 COUNT : 906996 SUM : 239752532 rocksdb.db.write.micros P50 : 1.559099 P95 : 3.990509 P99 : 8.565929 P100 : 958971.000000 COUNT : 5596555 SUM : 52793408 rocksdb.compaction.times.micros P50 : 1139056.603774 P95 : 4822500.000000 P99 : 13958000.000000 P100 : 34711409.000000 COUNT : 101 SUM : 204119731 rocksdb.compaction.times.cpu_micros P50 : 779444.444444 P95 : 3726000.000000 P99 : 9767000.000000 P100 : 13569838.000000 COUNT : 101 SUM : 117469913 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 861.212121 P95 : 132000.000000 P99 : 421800.000000 P100 : 534187.000000 COUNT : 78 SUM : 1367458 rocksdb.compaction.outfile.sync.micros P50 : 20105.263158 P95 : 240888.888889 P99 : 368063.636364 P100 : 504557.000000 COUNT : 301 SUM : 18386305 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 324.285714 P95 : 3989.473684 P99 : 6028.000000 P100 : 14732.000000 COUNT : 204 SUM : 204951 rocksdb.table.open.io.micros P50 : 8741.562500 P95 : 19809.053498 P99 : 22666.111111 P100 : 30149.000000 COUNT : 1691 SUM : 14926071 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.974794 P95 : 9.983835 P99 : 184.981905 P100 : 51973.000000 COUNT : 6544042 SUM : 102460482 rocksdb.read.block.get.micros P50 : 141.599697 P95 : 479.592045 P99 : 1815.627742 P100 : 38251.000000 COUNT : 1042868 SUM : 208840747 rocksdb.write.raw.block.micros P50 : 0.534143 P95 : 1.311956 P99 : 3.476137 P100 : 41599.000000 COUNT : 7900480 SUM : 23518768 rocksdb.numfiles.in.singlecompaction P50 : 2.648148 P95 : 6.550000 P99 : 17.485000 P100 : 21.000000 COUNT : 129 SUM : 454 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.562834 P95 : 1231.901745 P99 : 2647.774243 P100 : 958801.000000 COUNT : 208099 SUM : 38921140 rocksdb.sst.read.micros P50 : 2.055709 P95 : 162.266480 P99 : 510.343764 P100 : 51968.000000 COUNT : 7318720 SUM : 291377331 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 906996 SUM : 906996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5596555 SUM : 5775644760 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 227777.777778 P95 : 979000.000000 P99 : 1155800.000000 P100 : 1197001.000000 COUNT : 78 SUM : 25664548 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 350 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0