[2024-10-17 14:05:03.247206] Starting SPDK v25.01-pre git sha1 ffd9f7465 / DPDK 24.11.0-rc0 initialization... [2024-10-17 14:05:03.247471] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1352388 ] Set seed to 1729166703240489 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-10-17 14:05:03.418112] pci_dpdk.c: 37:dpdk_pci_init: *NOTICE*: In-development DPDK 24.11.0-rc0 is used. There is no support for it in SPDK. Enabled only for validation. [2024-10-17 14:05:03.460031] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-10-17 14:05:03.483727] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-10-17 14:05:03.483768] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1352388' to capture a snapshot of events at runtime. [2024-10-17 14:05:03.483780] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-10-17 14:05:03.483789] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-10-17 14:05:03.483799] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1352388 for offline analysis/debug. [2024-10-17 14:05:03.484354] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Thu Oct 17 14:05:07 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/10/17-14:05:16 ... thread 0: (1000000,1000000) ops and (110802.4,110802.4) ops/second in (9.025075,9.025075) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/12 1.97 GB 50.4 0.3 0.0 0.3 1.2 0.9 0.0 1.4 32.5 123.4 9.69 3.84 11 0.881 319K 355 0.0 0.0 L1 7/7 856.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 81.42 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 16/4 1.42 GB 0.9 1.2 0.6 0.6 1.2 0.6 0.1 2.0 105.8 103.5 12.01 5.76 5 2.402 509K 0 0.0 0.0 L4 173/15 10.41 GB 1.0 6.0 3.3 2.7 5.9 3.2 0.0 1.8 111.3 109.8 55.23 28.70 28 1.973 2014K 853 0.0 0.0 L5 1120/0 92.15 GB 0.9 0.0 0.0 0.0 0.0 0.0 2.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1332/38 106.87 GB 0.0 7.6 3.9 3.7 8.3 4.6 2.5 9.6 100.5 110.6 76.94 38.30 44 1.749 2843K 1208 0.0 0.0 Int 0/0 0.00 KB 0.0 7.6 3.9 3.7 8.3 4.6 2.5 9.6 100.5 110.6 76.94 38.30 44 1.749 2843K 1208 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 7.6 3.9 3.7 7.4 3.8 0.0 0.0 111.5 109.9 69.40 35.75 34 2.041 2843K 1208 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 117.0 7.54 2.55 10 0.754 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 9.2 total, 9.0 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 8.31 GB write, 927.32 MB/s write, 7.55 GB read, 843.21 MB/s read, 76.9 seconds Interval compaction: 8.31 GB write, 942.35 MB/s write, 7.55 GB read, 856.88 MB/s read, 76.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 31, pending-compaction-bytes-delays: 6, pending-compaction-bytes-stops: 0, total-delays: 6, total-stops: 31, interval: 37 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 411501 Average: 15.6266 StdDev: 179.53 Min: 1 Median: 1.8838 Max: 11691 Percentiles: P50: 1.88 P75: 2.89 P99: 14.28 P99.9: 2769.11 P99.99: 4513.31 ------------------------------------------------------ [ 0, 1 ] 15028 3.652% 3.652% # ( 1, 2 ] 215787 52.439% 56.091% ########## ( 2, 3 ] 87800 21.337% 77.428% #### ( 3, 4 ] 39971 9.713% 87.141% ## ( 4, 6 ] 38159 9.273% 96.414% ## ( 6, 10 ] 9367 2.276% 98.690% ( 10, 15 ] 1489 0.362% 99.052% ( 15, 22 ] 181 0.044% 99.096% ( 22, 34 ] 16 0.004% 99.100% ( 34, 51 ] 10 0.002% 99.103% ( 51, 76 ] 11 0.003% 99.105% ( 76, 110 ] 15 0.004% 99.109% ( 110, 170 ] 39 0.009% 99.118% ( 170, 250 ] 118 0.029% 99.147% ( 250, 380 ] 263 0.064% 99.211% ( 380, 580 ] 510 0.124% 99.335% ( 580, 870 ] 743 0.181% 99.515% ( 870, 1300 ] 806 0.196% 99.711% ( 1300, 1900 ] 481 0.117% 99.828% ( 1900, 2900 ] 340 0.083% 99.911% ( 2900, 4400 ] 319 0.078% 99.988% ( 4400, 6600 ] 133 0.032% 100.021% ( 6600, 9900 ] 13 0.003% 100.024% ( 9900, 14000 ] 3 0.001% 100.025% ** Level 1 read latency histogram (micros): Count: 126 Average: 1557.3810 StdDev: 1324.95 Min: 222 Median: 1006.8182 Max: 6296 Percentiles: P50: 1006.82 P75: 2108.33 P99: 6204.00 P99.9: 6296.00 P99.99: 6296.00 ------------------------------------------------------ ( 170, 250 ] 1 0.794% 0.794% ( 250, 380 ] 8 6.349% 7.143% # ( 380, 580 ] 15 11.905% 19.048% ## ( 580, 870 ] 32 25.397% 44.444% ##### ( 870, 1300 ] 22 17.460% 61.905% ### ( 1300, 1900 ] 14 11.111% 73.016% ## ( 1900, 2900 ] 12 9.524% 82.540% ## ( 2900, 4400 ] 15 11.905% 94.444% ## ( 4400, 6600 ] 7 5.556% 100.000% # ** Level 2 read latency histogram (micros): Count: 40 Average: 1431.4250 StdDev: 1533.80 Min: 198 Median: 913.0000 Max: 8900 Percentiles: P50: 913.00 P75: 1500.00 P99: 8580.00 P99.9: 8900.00 P99.99: 8900.00 ------------------------------------------------------ ( 170, 250 ] 1 2.500% 2.500% # ( 250, 380 ] 6 15.000% 17.500% ### ( 380, 580 ] 2 5.000% 22.500% # ( 580, 870 ] 10 25.000% 47.500% ##### ( 870, 1300 ] 10 25.000% 72.500% ##### ( 1300, 1900 ] 3 7.500% 80.000% ## ( 1900, 2900 ] 4 10.000% 90.000% ## ( 2900, 4400 ] 2 5.000% 95.000% # ( 4400, 6600 ] 1 2.500% 97.500% # ( 6600, 9900 ] 1 2.500% 100.000% # ** Level 3 read latency histogram (micros): Count: 819893 Average: 8.3712 StdDev: 121.96 Min: 1 Median: 1.9524 Max: 10604 Percentiles: P50: 1.95 P75: 2.96 P99: 9.40 P99.9: 598.04 P99.99: 1076.56 ------------------------------------------------------ [ 0, 1 ] 15935 1.944% 1.944% ( 1, 2 ] 413723 50.461% 52.404% ########## ( 2, 3 ] 193052 23.546% 75.950% ##### ( 3, 4 ] 88400 10.782% 86.732% ## ( 4, 6 ] 84089 10.256% 96.988% ## ( 6, 10 ] 19398 2.366% 99.354% ( 10, 15 ] 3261 0.398% 99.752% ( 15, 22 ] 436 0.053% 99.805% ( 22, 34 ] 48 0.006% 99.811% ( 34, 51 ] 44 0.005% 99.816% ( 51, 76 ] 59 0.007% 99.823% ( 76, 110 ] 22 0.003% 99.826% ( 110, 170 ] 22 0.003% 99.829% ( 170, 250 ] 62 0.008% 99.836% ( 250, 380 ] 154 0.019% 99.855% ( 380, 580 ] 337 0.041% 99.896% ( 580, 870 ] 500 0.061% 99.957% ( 870, 1300 ] 560 0.068% 100.025% ( 1300, 1900 ] 392 0.048% 100.073% ( 1900, 2900 ] 297 0.036% 100.110% ( 2900, 4400 ] 342 0.042% 100.151% ( 4400, 6600 ] 121 0.015% 100.166% ( 6600, 9900 ] 14 0.002% 100.168% ( 9900, 14000 ] 2 0.000% 100.168% ** Level 4 read latency histogram (micros): Count: 800360 Average: 18.6929 StdDev: 211.93 Min: 1 Median: 1.9698 Max: 12738 Percentiles: P50: 1.97 P75: 3.00 P99: 14.12 P99.9: 1865.41 P99.99: 2724.67 ------------------------------------------------------ [ 0, 1 ] 16293 2.036% 2.036% ( 1, 2 ] 395849 49.459% 51.495% ########## ( 2, 3 ] 188601 23.565% 75.059% ##### ( 3, 4 ] 86095 10.757% 85.816% ## ( 4, 6 ] 81021 10.123% 95.939% ## ( 6, 10 ] 21451 2.680% 98.619% # ( 10, 15 ] 3693 0.461% 99.081% ( 15, 22 ] 563 0.070% 99.151% ( 22, 34 ] 47 0.006% 99.157% ( 34, 51 ] 58 0.007% 99.164% ( 51, 76 ] 74 0.009% 99.173% ( 76, 110 ] 32 0.004% 99.177% ( 110, 170 ] 100 0.012% 99.190% ( 170, 250 ] 223 0.028% 99.218% ( 250, 380 ] 422 0.053% 99.271% ( 380, 580 ] 934 0.117% 99.387% ( 580, 870 ] 1424 0.178% 99.565% ( 870, 1300 ] 1595 0.199% 99.764% ( 1300, 1900 ] 1151 0.144% 99.908% ( 1900, 2900 ] 793 0.099% 100.007% ( 2900, 4400 ] 898 0.112% 100.120% ( 4400, 6600 ] 338 0.042% 100.162% ( 6600, 9900 ] 69 0.009% 100.170% ( 9900, 14000 ] 9 0.001% 100.172% ** Level 5 read latency histogram (micros): Count: 16326 Average: 1570.6587 StdDev: 1472.54 Min: 86 Median: 1038.9237 Max: 12892 Percentiles: P50: 1038.92 P75: 1960.19 P99: 6822.91 P99.9: 9872.48 P99.99: 12892.00 ------------------------------------------------------ ( 76, 110 ] 17 0.104% 0.104% ( 110, 170 ] 139 0.851% 0.956% ( 170, 250 ] 372 2.279% 3.234% ( 250, 380 ] 1069 6.548% 9.782% # ( 380, 580 ] 2140 13.108% 22.890% ### ( 580, 870 ] 3185 19.509% 42.399% #### ( 870, 1300 ] 3159 19.350% 61.748% #### ( 1300, 1900 ] 2074 12.704% 74.452% ### ( 1900, 2900 ] 1487 9.108% 83.560% ## ( 2900, 4400 ] 1660 10.168% 93.728% ## ( 4400, 6600 ] 850 5.206% 98.934% # ( 6600, 9900 ] 159 0.974% 99.908% ( 9900, 14000 ] 15 0.092% 100.000% ** DB Stats ** Uptime(secs): 9.2 total, 9.0 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 107.29 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.009 H:M:S, 54.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 109.01 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:5.009 H:M:S, 55.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 6 num-running-flushes: 1 2024/10/17-14:05:20 ... thread 0: (1000000,2000000) ops and (277972.4,158446.5) ops/second in (3.597480,12.622555) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/12 2.95 GB 163.6 1.5 0.0 1.5 3.3 1.8 0.0 1.8 79.5 178.8 19.01 11.46 31 0.613 1535K 1731 0.0 0.0 L1 7/7 856.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 81.42 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 12/0 947.77 MB 0.9 1.2 0.6 0.6 1.2 0.6 0.1 2.0 105.8 103.5 12.01 5.76 5 2.402 509K 0 0.0 0.0 L4 163/0 9.59 GB 1.0 7.3 3.8 3.5 7.1 3.7 0.0 1.9 115.7 114.0 64.20 34.54 32 2.006 2278K 853 0.0 0.0 L5 1136/0 93.44 GB 1.0 0.0 0.0 0.0 0.0 0.0 3.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1338/19 107.83 GB 0.0 10.0 4.4 5.6 11.7 6.1 3.8 6.3 107.2 125.6 95.22 51.77 68 1.400 4323K 2584 0.0 0.0 Int 0/0 0.00 KB 0.0 10.0 4.4 5.6 11.7 6.1 3.8 6.3 107.2 125.6 95.22 51.77 68 1.400 4323K 2584 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 10.0 4.4 5.6 9.8 4.2 0.0 0.0 121.1 119.5 84.30 46.33 42 2.007 4323K 2584 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 173.0 10.93 5.44 26 0.420 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 12.8 total, 12.6 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 11.68 GB write, 936.76 MB/s write, 9.97 GB read, 799.52 MB/s read, 95.2 seconds Interval compaction: 11.68 GB write, 947.62 MB/s write, 9.97 GB read, 808.79 MB/s read, 95.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 2, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 2, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 31, pending-compaction-bytes-delays: 6, pending-compaction-bytes-stops: 0, total-delays: 8, total-stops: 31, interval: 39 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 696894 Average: 16.3078 StdDev: 145.88 Min: 0 Median: 1.7162 Max: 11691 Percentiles: P50: 1.72 P75: 2.59 P99: 293.32 P99.9: 1680.87 P99.99: 3288.95 ------------------------------------------------------ [ 0, 1 ] 78318 11.238% 11.238% ## ( 1, 2 ] 377191 54.125% 65.363% ########### ( 2, 3 ] 113417 16.275% 81.637% ### ( 3, 4 ] 48748 6.995% 88.632% # ( 4, 6 ] 43212 6.201% 94.833% # ( 6, 10 ] 10487 1.505% 96.338% ( 10, 15 ] 1767 0.254% 96.591% ( 15, 22 ] 239 0.034% 96.626% ( 22, 34 ] 26 0.004% 96.629% ( 34, 51 ] 14 0.002% 96.631% ( 51, 76 ] 22 0.003% 96.635% ( 76, 110 ] 2501 0.359% 96.994% ( 110, 170 ] 8649 1.241% 98.235% ( 170, 250 ] 4563 0.655% 98.889% ( 250, 380 ] 2314 0.332% 99.221% ( 380, 580 ] 2094 0.300% 99.522% ( 580, 870 ] 1350 0.194% 99.716% ( 870, 1300 ] 962 0.138% 99.854% ( 1300, 1900 ] 509 0.073% 99.927% ( 1900, 2900 ] 356 0.051% 99.978% ( 2900, 4400 ] 329 0.047% 100.025% ( 4400, 6600 ] 133 0.019% 100.044% ( 6600, 9900 ] 13 0.002% 100.046% ( 9900, 14000 ] 3 0.000% 100.046% ** Level 1 read latency histogram (micros): Count: 611 Average: 499.8920 StdDev: 822.71 Min: 75 Median: 197.9646 Max: 6296 Percentiles: P50: 197.96 P75: 482.73 P99: 4679.71 P99.9: 6296.00 P99.99: 6296.00 ------------------------------------------------------ ( 51, 76 ] 1 0.164% 0.164% ( 76, 110 ] 58 9.493% 9.656% ## ( 110, 170 ] 207 33.879% 43.535% ####### ( 170, 250 ] 113 18.494% 62.029% #### ( 250, 380 ] 51 8.347% 70.376% ## ( 380, 580 ] 55 9.002% 79.378% ## ( 580, 870 ] 52 8.511% 87.889% ## ( 870, 1300 ] 24 3.928% 91.817% # ( 1300, 1900 ] 16 2.619% 94.435% # ( 1900, 2900 ] 12 1.964% 96.399% ( 2900, 4400 ] 15 2.455% 98.854% ( 4400, 6600 ] 7 1.146% 100.000% ** Level 2 read latency histogram (micros): Count: 147 Average: 553.5510 StdDev: 975.58 Min: 91 Median: 210.0000 Max: 8900 Percentiles: P50: 210.00 P75: 585.18 P99: 5566.00 P99.9: 8900.00 P99.99: 8900.00 ------------------------------------------------------ ( 76, 110 ] 12 8.163% 8.163% ## ( 110, 170 ] 48 32.653% 40.816% ####### ( 170, 250 ] 27 18.367% 59.184% #### ( 250, 380 ] 13 8.844% 68.027% ## ( 380, 580 ] 10 6.803% 74.830% # ( 580, 870 ] 14 9.524% 84.354% ## ( 870, 1300 ] 12 8.163% 92.517% ## ( 1300, 1900 ] 3 2.041% 94.558% ( 1900, 2900 ] 4 2.721% 97.279% # ( 2900, 4400 ] 2 1.361% 98.639% ( 4400, 6600 ] 1 0.680% 99.320% ( 6600, 9900 ] 1 0.680% 100.000% ** Level 3 read latency histogram (micros): Count: 857340 Average: 8.2541 StdDev: 119.46 Min: 1 Median: 1.9399 Max: 10604 Percentiles: P50: 1.94 P75: 2.94 P99: 9.47 P99.9: 585.97 P99.99: 1070.77 ------------------------------------------------------ [ 0, 1 ] 17751 2.070% 2.070% ( 1, 2 ] 437188 50.994% 53.064% ########## ( 2, 3 ] 199935 23.320% 76.384% ##### ( 3, 4 ] 90683 10.577% 86.962% ## ( 4, 6 ] 85701 9.996% 96.958% ## ( 6, 10 ] 20165 2.352% 99.310% ( 10, 15 ] 3344 0.390% 99.700% ( 15, 22 ] 447 0.052% 99.752% ( 22, 34 ] 49 0.006% 99.758% ( 34, 51 ] 45 0.005% 99.763% ( 51, 76 ] 59 0.007% 99.770% ( 76, 110 ] 89 0.010% 99.780% ( 110, 170 ] 246 0.029% 99.809% ( 170, 250 ] 178 0.021% 99.830% ( 250, 380 ] 207 0.024% 99.854% ( 380, 580 ] 385 0.045% 99.899% ( 580, 870 ] 518 0.060% 99.959% ( 870, 1300 ] 566 0.066% 100.025% ( 1300, 1900 ] 392 0.046% 100.071% ( 1900, 2900 ] 297 0.035% 100.106% ( 2900, 4400 ] 342 0.040% 100.145% ( 4400, 6600 ] 121 0.014% 100.160% ( 6600, 9900 ] 14 0.002% 100.161% ( 9900, 14000 ] 2 0.000% 100.161% ** Level 4 read latency histogram (micros): Count: 881311 Average: 18.7174 StdDev: 203.36 Min: 1 Median: 1.9525 Max: 12738 Percentiles: P50: 1.95 P75: 2.98 P99: 186.99 P99.9: 1807.69 P99.99: 2669.55 ------------------------------------------------------ [ 0, 1 ] 19359 2.197% 2.197% ( 1, 2 ] 442302 50.187% 52.383% ########## ( 2, 3 ] 204073 23.156% 75.539% ##### ( 3, 4 ] 90810 10.304% 85.843% ## ( 4, 6 ] 84706 9.611% 95.454% ## ( 6, 10 ] 23080 2.619% 98.073% # ( 10, 15 ] 3886 0.441% 98.514% ( 15, 22 ] 583 0.066% 98.580% ( 22, 34 ] 52 0.006% 98.586% ( 34, 51 ] 58 0.007% 98.593% ( 51, 76 ] 76 0.009% 98.601% ( 76, 110 ] 613 0.070% 98.671% ( 110, 170 ] 2580 0.293% 98.964% ( 170, 250 ] 1506 0.171% 99.135% ( 250, 380 ] 1019 0.116% 99.250% ( 380, 580 ] 1428 0.162% 99.412% ( 580, 870 ] 1655 0.188% 99.600% ( 870, 1300 ] 1663 0.189% 99.789% ( 1300, 1900 ] 1159 0.132% 99.920% ( 1900, 2900 ] 799 0.091% 100.011% ( 2900, 4400 ] 900 0.102% 100.113% ( 4400, 6600 ] 338 0.038% 100.151% ( 6600, 9900 ] 69 0.008% 100.159% ( 9900, 14000 ] 9 0.001% 100.160% ** Level 5 read latency histogram (micros): Count: 56468 Average: 614.3035 StdDev: 1011.55 Min: 75 Median: 223.3155 Max: 12892 Percentiles: P50: 223.32 P75: 598.71 P99: 5586.24 P99.9: 9018.59 P99.99: 12183.21 ------------------------------------------------------ ( 51, 76 ] 6 0.011% 0.011% ( 76, 110 ] 4548 8.054% 8.065% ## ( 110, 170 ] 17690 31.327% 39.392% ###### ( 170, 250 ] 8988 15.917% 55.309% ### ( 250, 380 ] 5219 9.242% 64.552% ## ( 380, 580 ] 5604 9.924% 74.476% ## ( 580, 870 ] 4589 8.127% 82.603% ## ( 870, 1300 ] 3476 6.156% 88.758% # ( 1300, 1900 ] 2127 3.767% 92.525% # ( 1900, 2900 ] 1526 2.702% 95.227% # ( 2900, 4400 ] 1672 2.961% 98.188% # ( 4400, 6600 ] 850 1.505% 99.694% ( 6600, 9900 ] 159 0.282% 99.975% ( 9900, 14000 ] 15 0.027% 100.002% ** DB Stats ** Uptime(secs): 12.8 total, 3.6 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 154.13 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.186 H:M:S, 40.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 273.58 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.177 H:M:S, 4.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/10/17-14:05:31 ... thread 0: (1000000,3000000) ops and (89754.0,126240.8) ops/second in (11.141559,23.764114) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/12 3.87 GB 256.9 4.0 0.0 4.0 6.8 2.8 0.0 2.4 128.9 217.7 31.76 23.69 51 0.623 4156K 13K 0.0 0.0 L1 7/7 856.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 81.42 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 12/0 947.77 MB 0.9 1.2 0.6 0.6 1.2 0.6 0.1 2.0 105.8 103.5 12.01 5.76 5 2.402 509K 0 0.0 0.0 L4 163/0 9.59 GB 1.0 7.3 3.8 3.5 7.1 3.7 0.0 1.9 115.7 114.0 64.20 34.54 32 2.006 2278K 853 0.0 0.0 L5 1136/0 93.44 GB 1.0 0.0 0.0 0.0 0.0 0.0 3.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1335/19 108.74 GB 0.0 12.5 4.4 8.1 15.1 7.0 3.8 5.5 118.5 143.3 107.97 63.99 88 1.227 6944K 14K 0.0 0.0 Int 0/0 0.00 KB 0.0 12.5 4.4 8.1 15.1 7.0 3.8 5.5 118.5 143.3 107.97 63.99 88 1.227 6944K 14K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.5 4.4 8.1 12.3 4.2 0.0 0.0 135.9 134.3 94.11 56.00 47 2.002 6944K 14K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 204.6 13.86 7.99 41 0.338 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 23.9 total, 23.8 interval Flush(GB): cumulative 2.769, interval 2.769 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 15.12 GB write, 647.28 MB/s write, 12.49 GB read, 534.95 MB/s read, 108.0 seconds Interval compaction: 15.12 GB write, 651.26 MB/s write, 12.49 GB read, 538.24 MB/s read, 108.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 13, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 13, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 31, pending-compaction-bytes-delays: 6, pending-compaction-bytes-stops: 0, total-delays: 19, total-stops: 31, interval: 50 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1393625 Average: 16.0132 StdDev: 111.12 Min: 0 Median: 1.5701 Max: 11691 Percentiles: P50: 1.57 P75: 2.17 P99: 251.62 P99.9: 1019.31 P99.99: 2053.07 ------------------------------------------------------ [ 0, 1 ] 272215 19.533% 19.533% #### ( 1, 2 ] 744786 53.442% 72.975% ########### ( 2, 3 ] 164334 11.792% 84.767% ## ( 3, 4 ] 64370 4.619% 89.386% # ( 4, 6 ] 54449 3.907% 93.293% # ( 6, 10 ] 12264 0.880% 94.173% ( 10, 15 ] 2286 0.164% 94.337% ( 15, 22 ] 304 0.022% 94.359% ( 22, 34 ] 30 0.002% 94.361% ( 34, 51 ] 15 0.001% 94.362% ( 51, 76 ] 33 0.002% 94.364% ( 76, 110 ] 12612 0.905% 95.269% ( 110, 170 ] 37767 2.710% 97.979% # ( 170, 250 ] 14153 1.016% 98.995% ( 250, 380 ] 5689 0.408% 99.403% ( 380, 580 ] 4577 0.328% 99.732% ( 580, 870 ] 1988 0.143% 99.874% ( 870, 1300 ] 1035 0.074% 99.948% ( 1300, 1900 ] 522 0.037% 99.986% ( 1900, 2900 ] 370 0.027% 100.012% ( 2900, 4400 ] 336 0.024% 100.037% ( 4400, 6600 ] 134 0.010% 100.046% ( 6600, 9900 ] 14 0.001% 100.047% ( 9900, 14000 ] 3 0.000% 100.047% ** Level 1 read latency histogram (micros): Count: 183385 Average: 7.6672 StdDev: 59.50 Min: 1 Median: 3.3140 Max: 6296 Percentiles: P50: 3.31 P75: 3.93 P99: 130.97 P99.9: 528.82 P99.99: 2720.12 ------------------------------------------------------ [ 0, 1 ] 880 0.480% 0.480% ( 1, 2 ] 15015 8.188% 8.668% ## ( 2, 3 ] 52283 28.510% 37.178% ###### ( 3, 4 ] 74880 40.832% 78.010% ######## ( 4, 6 ] 27822 15.171% 93.181% ### ( 6, 10 ] 6400 3.490% 96.671% # ( 10, 15 ] 2795 1.524% 98.195% ( 15, 22 ] 481 0.262% 98.457% ( 22, 34 ] 49 0.027% 98.484% ( 51, 76 ] 1 0.001% 98.485% ( 76, 110 ] 447 0.244% 98.728% ( 110, 170 ] 1425 0.777% 99.505% ( 170, 250 ] 448 0.244% 99.750% ( 250, 380 ] 164 0.089% 99.839% ( 380, 580 ] 150 0.082% 99.921% ( 580, 870 ] 73 0.040% 99.961% ( 870, 1300 ] 27 0.015% 99.975% ( 1300, 1900 ] 16 0.009% 99.984% ( 1900, 2900 ] 13 0.007% 99.991% ( 2900, 4400 ] 15 0.008% 99.999% ( 4400, 6600 ] 7 0.004% 100.003% ** Level 2 read latency histogram (micros): Count: 603 Average: 261.3184 StdDev: 515.91 Min: 74 Median: 151.6388 Max: 8900 Percentiles: P50: 151.64 P75: 216.02 P99: 2392.50 P99.9: 7910.10 P99.99: 8900.00 ------------------------------------------------------ ( 51, 76 ] 1 0.166% 0.166% ( 76, 110 ] 93 15.423% 15.589% ### ( 110, 170 ] 299 49.585% 65.174% ########## ( 170, 250 ] 103 17.081% 82.255% ### ( 250, 380 ] 42 6.965% 89.221% # ( 380, 580 ] 24 3.980% 93.201% # ( 580, 870 ] 17 2.819% 96.020% # ( 870, 1300 ] 13 2.156% 98.176% ( 1300, 1900 ] 3 0.498% 98.673% ( 1900, 2900 ] 4 0.663% 99.337% ( 2900, 4400 ] 2 0.332% 99.668% ( 4400, 6600 ] 1 0.166% 99.834% ( 6600, 9900 ] 1 0.166% 100.000% ** Level 3 read latency histogram (micros): Count: 859388 Average: 8.6381 StdDev: 119.67 Min: 1 Median: 1.9423 Max: 10604 Percentiles: P50: 1.94 P75: 2.95 P99: 9.88 P99.9: 595.96 P99.99: 1071.02 ------------------------------------------------------ [ 0, 1 ] 17751 2.066% 2.066% ( 1, 2 ] 437188 50.872% 52.938% ########## ( 2, 3 ] 199935 23.265% 76.202% ##### ( 3, 4 ] 90683 10.552% 86.754% ## ( 4, 6 ] 85701 9.972% 96.727% ## ( 6, 10 ] 20165 2.346% 99.073% ( 10, 15 ] 3344 0.389% 99.462% ( 15, 22 ] 447 0.052% 99.514% ( 22, 34 ] 49 0.006% 99.520% ( 34, 51 ] 45 0.005% 99.525% ( 51, 76 ] 59 0.007% 99.532% ( 76, 110 ] 471 0.055% 99.587% ( 110, 170 ] 1368 0.159% 99.746% ( 170, 250 ] 484 0.056% 99.802% ( 250, 380 ] 336 0.039% 99.842% ( 380, 580 ] 473 0.055% 99.897% ( 580, 870 ] 538 0.063% 99.959% ( 870, 1300 ] 567 0.066% 100.025% ( 1300, 1900 ] 392 0.046% 100.071% ( 1900, 2900 ] 297 0.035% 100.105% ( 2900, 4400 ] 342 0.040% 100.145% ( 4400, 6600 ] 121 0.014% 100.159% ( 6600, 9900 ] 14 0.002% 100.161% ( 9900, 14000 ] 2 0.000% 100.161% ** Level 4 read latency histogram (micros): Count: 907590 Average: 22.8258 StdDev: 202.68 Min: 1 Median: 1.9819 Max: 12738 Percentiles: P50: 1.98 P75: 3.16 P99: 315.03 P99.9: 1795.66 P99.99: 2668.76 ------------------------------------------------------ [ 0, 1 ] 19359 2.133% 2.133% ( 1, 2 ] 442438 48.749% 50.882% ########## ( 2, 3 ] 204562 22.539% 73.421% ##### ( 3, 4 ] 91120 10.040% 83.460% ## ( 4, 6 ] 85124 9.379% 92.840% ## ( 6, 10 ] 23146 2.550% 95.390% # ( 10, 15 ] 3899 0.430% 95.819% ( 15, 22 ] 585 0.064% 95.884% ( 22, 34 ] 52 0.006% 95.890% ( 34, 51 ] 58 0.006% 95.896% ( 51, 76 ] 79 0.009% 95.905% ( 76, 110 ] 5398 0.595% 96.500% ( 110, 170 ] 16193 1.784% 98.284% ( 170, 250 ] 5299 0.584% 98.868% ( 250, 380 ] 2403 0.265% 99.132% ( 380, 580 ] 2403 0.265% 99.397% ( 580, 870 ] 1907 0.210% 99.607% ( 870, 1300 ] 1695 0.187% 99.794% ( 1300, 1900 ] 1165 0.128% 99.922% ( 1900, 2900 ] 799 0.088% 100.010% ( 2900, 4400 ] 902 0.099% 100.110% ( 4400, 6600 ] 338 0.037% 100.147% ( 6600, 9900 ] 69 0.008% 100.155% ( 9900, 14000 ] 9 0.001% 100.156% ** Level 5 read latency histogram (micros): Count: 217094 Average: 284.9085 StdDev: 559.28 Min: 73 Median: 151.5494 Max: 12892 Percentiles: P50: 151.55 P75: 221.06 P99: 3370.31 P99.9: 6470.34 P99.99: 9615.47 ------------------------------------------------------ ( 51, 76 ] 29 0.013% 0.013% ( 76, 110 ] 35542 16.372% 16.385% ### ( 110, 170 ] 105382 48.542% 64.927% ########## ( 170, 250 ] 34259 15.781% 80.708% ### ( 250, 380 ] 13788 6.351% 87.059% # ( 380, 580 ] 11892 5.478% 92.537% # ( 580, 870 ] 6124 2.821% 95.358% # ( 870, 1300 ] 3671 1.691% 97.049% ( 1300, 1900 ] 2151 0.991% 98.040% ( 1900, 2900 ] 1558 0.718% 98.757% ( 2900, 4400 ] 1681 0.774% 99.532% ( 4400, 6600 ] 850 0.392% 99.923% ( 6600, 9900 ] 159 0.073% 99.996% ( 9900, 14000 ] 15 0.007% 100.003% ** DB Stats ** Uptime(secs): 23.9 total, 11.1 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 123.47 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:12.543 H:M:S, 52.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 88.34 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:7.357 H:M:S, 66.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 220.984 micros/op 18081 ops/sec 60.117 seconds 1086996 operations; 17.5 MB/s (271999 of 271999 found) Microseconds per read: Count: 1086996 Average: 220.9858 StdDev: 648.35 Min: 2 Median: 164.2190 Max: 65328 Percentiles: P50: 164.22 P75: 221.42 P99: 883.73 P99.9: 10147.90 P99.99: 28286.74 ------------------------------------------------------ ( 1, 2 ] 16 0.001% 0.001% ( 2, 3 ] 58 0.005% 0.007% ( 3, 4 ] 54 0.005% 0.012% ( 4, 6 ] 92 0.008% 0.020% ( 6, 10 ] 482 0.044% 0.065% ( 10, 15 ] 3695 0.340% 0.405% ( 15, 22 ] 3611 0.332% 0.737% ( 22, 34 ] 8622 0.793% 1.530% ( 34, 51 ] 1953 0.180% 1.710% ( 51, 76 ] 248 0.023% 1.732% ( 76, 110 ] 33483 3.080% 4.813% # ( 110, 170 ] 543556 50.005% 54.818% ########## ( 170, 250 ] 341323 31.401% 86.219% ###### ( 250, 380 ] 91721 8.438% 94.657% ## ( 380, 580 ] 34221 3.148% 97.805% # ( 580, 870 ] 12836 1.181% 98.986% ( 870, 1300 ] 4857 0.447% 99.433% ( 1300, 1900 ] 2114 0.194% 99.627% ( 1900, 2900 ] 1258 0.116% 99.743% ( 2900, 4400 ] 991 0.091% 99.834% ( 4400, 6600 ] 459 0.042% 99.876% ( 6600, 9900 ] 235 0.022% 99.898% ( 9900, 14000 ] 397 0.037% 99.934% ( 14000, 22000 ] 515 0.047% 99.982% ( 22000, 33000 ] 158 0.015% 99.996% ( 33000, 50000 ] 32 0.003% 99.999% ( 50000, 75000 ] 9 0.001% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 14430525 rocksdb.bloom.filter.full.positive COUNT : 1227004 rocksdb.bloom.filter.full.true.positive COUNT : 1086588 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 408 rocksdb.memtable.miss COUNT : 1086588 rocksdb.l0.hit COUNT : 36085 rocksdb.l1.hit COUNT : 17916 rocksdb.l2andup.hit COUNT : 1032587 rocksdb.compaction.key.drop.new COUNT : 187974 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 3440031 rocksdb.number.keys.read COUNT : 1086996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 3550111992 rocksdb.bytes.read COUNT : 1086996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1531 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 46275840 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 17348402615 rocksdb.compact.write.bytes COUNT : 17147886237 rocksdb.flush.write.bytes COUNT : 3501189554 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 30 rocksdb.number.superversion_acquires COUNT : 672 rocksdb.number.superversion_releases COUNT : 636 rocksdb.number.superversion_cleanups COUNT : 165 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 191 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 3472788480 rocksdb.memtable.garbage.bytes.at.flush COUNT : 1189888 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 21149291102 rocksdb.non.last.level.read.count COUNT : 4966203 rocksdb.block.checksum.compute.count COUNT : 5438028 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 163.292555 P95 : 399.802050 P99 : 881.987448 P100 : 65325.000000 COUNT : 1086996 SUM : 239186293 rocksdb.db.write.micros P50 : 1.555084 P95 : 5.238790 P99 : 76.156008 P100 : 964276.000000 COUNT : 3440031 SUM : 55404061 rocksdb.compaction.times.micros P50 : 2015384.615385 P95 : 2880769.230769 P99 : 25933950.000000 P100 : 25933950.000000 COUNT : 50 SUM : 121948837 rocksdb.compaction.times.cpu_micros P50 : 1120000.000000 P95 : 1881578.947368 P99 : 11900000.000000 P100 : 12976680.000000 COUNT : 50 SUM : 70847693 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 775.192308 P95 : 232000.000000 P99 : 440304.000000 P100 : 440304.000000 COUNT : 49 SUM : 1371520 rocksdb.compaction.outfile.sync.micros P50 : 52272.727273 P95 : 399000.000000 P99 : 552688.888889 P100 : 582667.000000 COUNT : 182 SUM : 16718604 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 206.666667 P95 : 4291.666667 P99 : 8373.000000 P100 : 8373.000000 COUNT : 113 SUM : 124655 rocksdb.table.open.io.micros P50 : 9462.995050 P95 : 20484.462151 P99 : 27378.214286 P100 : 34055.000000 COUNT : 1531 SUM : 14921216 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.996418 P95 : 95.728528 P99 : 201.344693 P100 : 15691.000000 COUNT : 4200192 SUM : 74594699 rocksdb.read.block.get.micros P50 : 138.680870 P95 : 291.650014 P99 : 844.722561 P100 : 19451.000000 COUNT : 1236300 SUM : 204702919 rocksdb.write.raw.block.micros P50 : 0.543168 P95 : 1.499698 P99 : 3.856278 P100 : 19038.000000 COUNT : 5001691 SUM : 14098216 rocksdb.numfiles.in.singlecompaction P50 : 3.303030 P95 : 5.933333 P99 : 26.000000 P100 : 26.000000 COUNT : 64 SUM : 263 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.575929 P95 : 1247.775411 P99 : 2646.192009 P100 : 964095.000000 COUNT : 234881 SUM : 46300021 rocksdb.sst.read.micros P50 : 2.537916 P95 : 169.676685 P99 : 450.164161 P100 : 12892.000000 COUNT : 4966240 SUM : 261967461 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1086996 SUM : 1086996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 3440031 SUM : 3550111992 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 221578.947368 P95 : 1035666.666667 P99 : 1211095.000000 P100 : 1211095.000000 COUNT : 49 SUM : 15499051 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 195 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0