[2024-11-29 03:10:12.652042] Starting SPDK v25.01-pre git sha1 35cd3e84d / DPDK 24.11.0-rc4 initialization... [2024-11-29 03:10:12.652307] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1910610 ] Set seed to 1732846212645386 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-29 03:10:12.825476] pci_dpdk.c: 37:dpdk_pci_init: *NOTICE*: In-development DPDK 24.11.0-rc4 is used. There is no support for it in SPDK. Enabled only for validation. [2024-11-29 03:10:12.873763] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-29 03:10:12.895337] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-29 03:10:12.895378] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1910610' to capture a snapshot of events at runtime. [2024-11-29 03:10:12.895390] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-29 03:10:12.895399] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-29 03:10:12.895409] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1910610 for offline analysis/debug. [2024-11-29 03:10:12.895933] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Fri Nov 29 03:10:17 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/29-03:10:24 ... thread 0: (1000000,1000000) ops and (135592.0,135592.0) ops/second in (7.375064,7.375064) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 13/11 1.87 GB 44.0 0.4 0.0 0.4 1.2 0.8 0.0 1.5 39.0 123.4 9.69 3.96 10 0.969 383K 495 0.0 0.0 L1 9/9 966.73 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/1 128.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 20/7 1.71 GB 1.0 0.9 0.5 0.4 0.9 0.5 0.0 1.7 104.7 103.7 9.26 4.55 5 1.852 461K 0 0.0 0.0 L4 190/25 10.77 GB 1.0 4.8 2.2 2.6 4.7 2.1 0.0 2.2 108.5 106.5 45.11 22.56 22 2.051 959K 0 0.0 0.0 L5 1112/0 91.60 GB 0.9 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1345/53 107.02 GB 0.0 6.1 2.7 3.4 6.8 3.4 1.0 8.5 97.4 108.6 64.06 31.07 37 1.731 1804K 495 0.0 0.0 Int 0/0 0.00 KB 0.0 6.1 2.7 3.4 6.8 3.4 1.0 8.5 97.4 108.6 64.06 31.07 37 1.731 1804K 495 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.1 2.7 3.4 6.0 2.6 0.0 0.0 108.9 107.2 57.30 28.63 28 2.046 1804K 495 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 121.1 6.76 2.44 9 0.751 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.5 total, 7.4 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.80 GB write, 925.62 MB/s write, 6.10 GB read, 830.36 MB/s read, 64.1 seconds Interval compaction: 6.80 GB write, 943.38 MB/s write, 6.10 GB read, 846.29 MB/s read, 64.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 22, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 22, interval: 22 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 285105 Average: 16.3915 StdDev: 169.43 Min: 1 Median: 1.9033 Max: 16029 Percentiles: P50: 1.90 P75: 2.91 P99: 306.01 P99.9: 2401.63 P99.99: 3960.80 ------------------------------------------------------ [ 0, 1 ] 9599 3.367% 3.367% # ( 1, 2 ] 147180 51.623% 54.990% ########## ( 2, 3 ] 62614 21.962% 76.952% #### ( 3, 4 ] 27263 9.562% 86.514% ## ( 4, 6 ] 27130 9.516% 96.030% ## ( 6, 10 ] 6737 2.363% 98.393% ( 10, 15 ] 1311 0.460% 98.853% ( 15, 22 ] 158 0.055% 98.908% ( 22, 34 ] 15 0.005% 98.913% ( 34, 51 ] 13 0.005% 98.918% ( 51, 76 ] 21 0.007% 98.925% ( 76, 110 ] 5 0.002% 98.927% ( 110, 170 ] 31 0.011% 98.938% ( 170, 250 ] 81 0.028% 98.966% ( 250, 380 ] 225 0.079% 99.045% ( 380, 580 ] 459 0.161% 99.206% ( 580, 870 ] 703 0.247% 99.453% ( 870, 1300 ] 699 0.245% 99.698% ( 1300, 1900 ] 456 0.160% 99.858% ( 1900, 2900 ] 241 0.085% 99.942% ( 2900, 4400 ] 193 0.068% 100.010% ( 4400, 6600 ] 47 0.016% 100.027% ( 6600, 9900 ] 10 0.004% 100.030% ( 14000, 22000 ] 1 0.000% 100.031% ** Level 1 read latency histogram (micros): Count: 148 Average: 1466.4257 StdDev: 1542.64 Min: 126 Median: 958.9655 Max: 7596 Percentiles: P50: 958.97 P75: 1665.22 P99: 7596.00 P99.9: 7596.00 P99.99: 7596.00 ------------------------------------------------------ ( 110, 170 ] 6 4.054% 4.054% # ( 170, 250 ] 4 2.703% 6.757% # ( 250, 380 ] 15 10.135% 16.892% ## ( 380, 580 ] 18 12.162% 29.054% ## ( 580, 870 ] 25 16.892% 45.946% ### ( 870, 1300 ] 29 19.595% 65.541% #### ( 1300, 1900 ] 23 15.541% 81.081% ### ( 1900, 2900 ] 10 6.757% 87.838% # ( 2900, 4400 ] 7 4.730% 92.568% # ( 4400, 6600 ] 7 4.730% 97.297% # ( 6600, 9900 ] 4 2.703% 100.000% # ** Level 2 read latency histogram (micros): Count: 50427 Average: 10.6498 StdDev: 141.07 Min: 1 Median: 1.9432 Max: 7481 Percentiles: P50: 1.94 P75: 2.93 P99: 10.79 P99.9: 2505.09 P99.99: 5484.34 ------------------------------------------------------ [ 0, 1 ] 1050 2.082% 2.082% ( 1, 2 ] 25619 50.804% 52.886% ########## ( 2, 3 ] 12035 23.866% 76.753% ##### ( 3, 4 ] 4913 9.743% 86.495% ## ( 4, 6 ] 4873 9.663% 96.159% ## ( 6, 10 ] 1397 2.770% 98.929% # ( 10, 15 ] 233 0.462% 99.391% ( 15, 22 ] 27 0.054% 99.445% ( 22, 34 ] 3 0.006% 99.451% ( 34, 51 ] 7 0.014% 99.465% ( 51, 76 ] 9 0.018% 99.482% ( 76, 110 ] 4 0.008% 99.490% ( 110, 170 ] 5 0.010% 99.500% ( 170, 250 ] 7 0.014% 99.514% ( 250, 380 ] 17 0.034% 99.548% ( 380, 580 ] 34 0.067% 99.615% ( 580, 870 ] 48 0.095% 99.710% ( 870, 1300 ] 40 0.079% 99.790% ( 1300, 1900 ] 36 0.071% 99.861% ( 1900, 2900 ] 34 0.067% 99.929% ( 2900, 4400 ] 29 0.058% 99.986% ( 4400, 6600 ] 6 0.012% 99.998% ( 6600, 9900 ] 2 0.004% 100.002% ** Level 3 read latency histogram (micros): Count: 545345 Average: 8.7212 StdDev: 121.03 Min: 1 Median: 1.9594 Max: 21777 Percentiles: P50: 1.96 P75: 2.96 P99: 9.59 P99.9: 667.93 P99.99: 1048.38 ------------------------------------------------------ [ 0, 1 ] 11603 2.128% 2.128% ( 1, 2 ] 272124 49.899% 52.027% ########## ( 2, 3 ] 130764 23.978% 76.005% ##### ( 3, 4 ] 55303 10.141% 86.146% ## ( 4, 6 ] 57355 10.517% 96.663% ## ( 6, 10 ] 14203 2.604% 99.268% # ( 10, 15 ] 2365 0.434% 99.701% ( 15, 22 ] 314 0.058% 99.759% ( 22, 34 ] 48 0.009% 99.768% ( 34, 51 ] 50 0.009% 99.777% ( 51, 76 ] 56 0.010% 99.787% ( 76, 110 ] 19 0.003% 99.791% ( 110, 170 ] 31 0.006% 99.796% ( 170, 250 ] 61 0.011% 99.808% ( 250, 380 ] 121 0.022% 99.830% ( 380, 580 ] 256 0.047% 99.877% ( 580, 870 ] 421 0.077% 99.954% ( 870, 1300 ] 476 0.087% 100.041% ( 1300, 1900 ] 345 0.063% 100.105% ( 1900, 2900 ] 227 0.042% 100.146% ( 2900, 4400 ] 209 0.038% 100.184% ( 4400, 6600 ] 46 0.008% 100.193% ( 6600, 9900 ] 10 0.002% 100.195% ( 9900, 14000 ] 1 0.000% 100.195% ( 14000, 22000 ] 1 0.000% 100.195% ** Level 4 read latency histogram (micros): Count: 716445 Average: 19.5231 StdDev: 215.06 Min: 1 Median: 1.9519 Max: 34622 Percentiles: P50: 1.95 P75: 2.96 P99: 13.62 P99.9: 1451.54 P99.99: 1775.56 ------------------------------------------------------ [ 0, 1 ] 16428 2.293% 2.293% ( 1, 2 ] 359051 50.116% 52.409% ########## ( 2, 3 ] 168125 23.467% 75.875% ##### ( 3, 4 ] 72771 10.157% 86.032% ## ( 4, 6 ] 71035 9.915% 95.947% ## ( 6, 10 ] 19469 2.717% 98.665% # ( 10, 15 ] 3316 0.463% 99.128% ( 15, 22 ] 460 0.064% 99.192% ( 22, 34 ] 64 0.009% 99.201% ( 34, 51 ] 69 0.010% 99.210% ( 51, 76 ] 69 0.010% 99.220% ( 76, 110 ] 32 0.004% 99.225% ( 110, 170 ] 78 0.011% 99.235% ( 170, 250 ] 188 0.026% 99.262% ( 250, 380 ] 361 0.050% 99.312% ( 380, 580 ] 885 0.124% 99.436% ( 580, 870 ] 1446 0.202% 99.637% ( 870, 1300 ] 1580 0.221% 99.858% ( 1300, 1900 ] 1194 0.167% 100.025% ( 1900, 2900 ] 857 0.120% 100.144% ( 2900, 4400 ] 796 0.111% 100.255% ( 4400, 6600 ] 306 0.043% 100.298% ( 6600, 9900 ] 41 0.006% 100.304% ( 9900, 14000 ] 2 0.000% 100.304% ( 14000, 22000 ] 1 0.000% 100.304% ( 22000, 33000 ] 1 0.000% 100.304% ( 33000, 50000 ] 1 0.000% 100.304% ** Level 5 read latency histogram (micros): Count: 12394 Average: 1684.9000 StdDev: 1531.14 Min: 78 Median: 1129.0694 Max: 31485 Percentiles: P50: 1129.07 P75: 2301.15 P99: 6735.76 P99.9: 9728.45 P99.99: 21042.40 ------------------------------------------------------ ( 76, 110 ] 6 0.048% 0.048% ( 110, 170 ] 72 0.581% 0.629% ( 170, 250 ] 213 1.719% 2.348% ( 250, 380 ] 622 5.019% 7.366% # ( 380, 580 ] 1450 11.699% 19.066% ## ( 580, 870 ] 2332 18.816% 37.881% #### ( 870, 1300 ] 2493 20.115% 57.996% #### ( 1300, 1900 ] 1586 12.797% 70.792% ### ( 1900, 2900 ] 1300 10.489% 81.281% ## ( 2900, 4400 ] 1495 12.062% 93.344% ## ( 4400, 6600 ] 696 5.616% 98.959% # ( 6600, 9900 ] 123 0.992% 99.952% ( 9900, 14000 ] 3 0.024% 99.976% ( 14000, 22000 ] 2 0.016% 99.992% ( 22000, 33000 ] 2 0.016% 100.008% ** DB Stats ** Uptime(secs): 7.5 total, 7.4 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 130.90 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.451 H:M:S, 45.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 133.40 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.451 H:M:S, 46.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 8 num-running-flushes: 1 2024/11/29-03:10:28 ... thread 0: (1000000,2000000) ops and (226135.9,169532.0) ops/second in (4.422119,11.797183) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/11 2.91 GB 163.6 1.6 0.0 1.6 3.4 1.8 0.0 1.9 84.5 181.9 19.38 11.81 29 0.668 1662K 2019 0.0 0.0 L1 9/9 966.73 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 12/0 953.25 MB 1.0 1.2 0.7 0.5 1.2 0.7 0.0 1.8 113.4 112.4 10.90 5.77 6 1.817 733K 2127 0.0 0.0 L4 176/0 9.66 GB 1.0 6.9 3.0 3.9 6.8 2.9 0.1 2.3 117.6 115.3 60.06 32.45 30 2.002 1510K 0 0.0 0.0 L5 1136/0 93.57 GB 1.0 0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1349/20 108.02 GB 0.0 9.7 3.7 6.0 11.4 5.4 3.0 6.2 110.0 129.3 90.34 50.04 65 1.390 3906K 4146 0.0 0.0 Int 0/0 0.00 KB 0.0 9.7 3.7 6.0 11.4 5.4 3.0 6.2 110.0 129.3 90.34 50.04 65 1.390 3906K 4146 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 9.7 3.7 6.0 9.6 3.5 0.0 0.0 125.4 123.5 79.26 44.63 41 1.933 3906K 4146 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 170.5 11.08 5.41 24 0.462 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 11.9 total, 11.8 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 11.41 GB write, 978.11 MB/s write, 9.71 GB read, 832.39 MB/s read, 90.3 seconds Interval compaction: 11.41 GB write, 989.84 MB/s write, 9.71 GB read, 842.37 MB/s read, 90.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 28, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 28, interval: 28 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 653432 Average: 14.6912 StdDev: 134.72 Min: 1 Median: 1.7367 Max: 16029 Percentiles: P50: 1.74 P75: 2.61 P99: 269.96 P99.9: 1671.56 P99.99: 2863.44 ------------------------------------------------------ [ 0, 1 ] 66217 10.134% 10.134% ## ( 1, 2 ] 353627 54.118% 64.252% ########### ( 2, 3 ] 114554 17.531% 81.783% #### ( 3, 4 ] 46601 7.132% 88.915% # ( 4, 6 ] 43608 6.674% 95.589% # ( 6, 10 ] 10449 1.599% 97.188% ( 10, 15 ] 2104 0.322% 97.510% ( 15, 22 ] 255 0.039% 97.549% ( 22, 34 ] 18 0.003% 97.552% ( 34, 51 ] 19 0.003% 97.554% ( 51, 76 ] 30 0.005% 97.559% ( 76, 110 ] 1451 0.222% 97.781% ( 110, 170 ] 5126 0.784% 98.566% ( 170, 250 ] 2608 0.399% 98.965% ( 250, 380 ] 1509 0.231% 99.196% ( 380, 580 ] 1792 0.274% 99.470% ( 580, 870 ] 1349 0.206% 99.676% ( 870, 1300 ] 1078 0.165% 99.841% ( 1300, 1900 ] 621 0.095% 99.936% ( 1900, 2900 ] 365 0.056% 99.992% ( 2900, 4400 ] 261 0.040% 100.032% ( 4400, 6600 ] 59 0.009% 100.041% ( 6600, 9900 ] 11 0.002% 100.043% ( 14000, 22000 ] 1 0.000% 100.043% ** Level 1 read latency histogram (micros): Count: 34119 Average: 16.3264 StdDev: 158.24 Min: 1 Median: 2.7199 Max: 7596 Percentiles: P50: 2.72 P75: 4.49 P99: 271.62 P99.9: 1898.02 P99.99: 6486.73 ------------------------------------------------------ [ 0, 1 ] 490 1.436% 1.436% ( 1, 2 ] 10182 29.843% 31.279% ###### ( 2, 3 ] 8873 26.006% 57.285% ##### ( 3, 4 ] 4275 12.530% 69.814% ### ( 4, 6 ] 7272 21.314% 91.128% #### ( 6, 10 ] 1732 5.076% 96.204% # ( 10, 15 ] 319 0.935% 97.139% ( 15, 22 ] 160 0.469% 97.608% ( 22, 34 ] 22 0.064% 97.673% ( 51, 76 ] 1 0.003% 97.676% ( 76, 110 ] 83 0.243% 97.919% ( 110, 170 ] 251 0.736% 98.655% ( 170, 250 ] 107 0.314% 98.968% ( 250, 380 ] 71 0.208% 99.176% ( 380, 580 ] 79 0.232% 99.408% ( 580, 870 ] 79 0.232% 99.639% ( 870, 1300 ] 54 0.158% 99.798% ( 1300, 1900 ] 36 0.106% 99.903% ( 1900, 2900 ] 13 0.038% 99.941% ( 2900, 4400 ] 10 0.029% 99.971% ( 4400, 6600 ] 8 0.023% 99.994% ( 6600, 9900 ] 4 0.012% 100.006% ** Level 2 read latency histogram (micros): Count: 65430 Average: 9.3356 StdDev: 127.54 Min: 1 Median: 1.9529 Max: 7481 Percentiles: P50: 1.95 P75: 2.94 P99: 9.93 P99.9: 2202.00 P99.99: 4934.23 ------------------------------------------------------ [ 0, 1 ] 1269 1.939% 1.939% ( 1, 2 ] 33000 50.436% 52.375% ########## ( 2, 3 ] 15794 24.139% 76.514% ##### ( 3, 4 ] 6482 9.907% 86.421% ## ( 4, 6 ] 6451 9.859% 96.280% ## ( 6, 10 ] 1810 2.766% 99.046% # ( 10, 15 ] 288 0.440% 99.486% ( 15, 22 ] 32 0.049% 99.535% ( 22, 34 ] 4 0.006% 99.541% ( 34, 51 ] 7 0.011% 99.552% ( 51, 76 ] 9 0.014% 99.566% ( 76, 110 ] 4 0.006% 99.572% ( 110, 170 ] 6 0.009% 99.581% ( 170, 250 ] 8 0.012% 99.593% ( 250, 380 ] 17 0.026% 99.619% ( 380, 580 ] 37 0.057% 99.676% ( 580, 870 ] 52 0.079% 99.755% ( 870, 1300 ] 46 0.070% 99.826% ( 1300, 1900 ] 38 0.058% 99.884% ( 1900, 2900 ] 35 0.053% 99.937% ( 2900, 4400 ] 33 0.050% 99.988% ( 4400, 6600 ] 6 0.009% 99.997% ( 6600, 9900 ] 2 0.003% 100.000% ** Level 3 read latency histogram (micros): Count: 709040 Average: 7.8423 StdDev: 109.24 Min: 1 Median: 1.9614 Max: 21777 Percentiles: P50: 1.96 P75: 2.96 P99: 9.57 P99.9: 488.00 P99.99: 879.28 ------------------------------------------------------ [ 0, 1 ] 14210 2.004% 2.004% ( 1, 2 ] 353978 49.924% 51.928% ########## ( 2, 3 ] 171263 24.154% 76.082% ##### ( 3, 4 ] 71261 10.050% 86.132% ## ( 4, 6 ] 74980 10.575% 96.707% ## ( 6, 10 ] 18220 2.570% 99.277% # ( 10, 15 ] 3016 0.425% 99.702% ( 15, 22 ] 374 0.053% 99.755% ( 22, 34 ] 53 0.007% 99.762% ( 34, 51 ] 65 0.009% 99.772% ( 51, 76 ] 69 0.010% 99.781% ( 76, 110 ] 90 0.013% 99.794% ( 110, 170 ] 237 0.033% 99.827% ( 170, 250 ] 157 0.022% 99.850% ( 250, 380 ] 183 0.026% 99.875% ( 380, 580 ] 324 0.046% 99.921% ( 580, 870 ] 478 0.067% 99.988% ( 870, 1300 ] 514 0.072% 100.061% ( 1300, 1900 ] 370 0.052% 100.113% ( 1900, 2900 ] 243 0.034% 100.147% ( 2900, 4400 ] 223 0.031% 100.179% ( 4400, 6600 ] 48 0.007% 100.186% ( 6600, 9900 ] 10 0.001% 100.187% ( 9900, 14000 ] 1 0.000% 100.187% ( 14000, 22000 ] 1 0.000% 100.187% ** Level 4 read latency histogram (micros): Count: 935240 Average: 18.9297 StdDev: 200.85 Min: 1 Median: 1.9586 Max: 34622 Percentiles: P50: 1.96 P75: 2.97 P99: 150.08 P99.9: 1328.73 P99.99: 1691.02 ------------------------------------------------------ [ 0, 1 ] 19863 2.124% 2.124% ( 1, 2 ] 467090 49.943% 52.067% ########## ( 2, 3 ] 220296 23.555% 75.622% ##### ( 3, 4 ] 94174 10.070% 85.692% ## ( 4, 6 ] 92133 9.851% 95.543% ## ( 6, 10 ] 25320 2.707% 98.250% # ( 10, 15 ] 4200 0.449% 98.699% ( 15, 22 ] 553 0.059% 98.759% ( 22, 34 ] 73 0.008% 98.766% ( 34, 51 ] 107 0.011% 98.778% ( 51, 76 ] 105 0.011% 98.789% ( 76, 110 ] 605 0.065% 98.854% ( 110, 170 ] 2049 0.219% 99.073% ( 170, 250 ] 1175 0.126% 99.198% ( 250, 380 ] 951 0.102% 99.300% ( 380, 580 ] 1539 0.165% 99.465% ( 580, 870 ] 2020 0.216% 99.681% ( 870, 1300 ] 1985 0.212% 99.893% ( 1300, 1900 ] 1394 0.149% 100.042% ( 1900, 2900 ] 1005 0.107% 100.149% ( 2900, 4400 ] 924 0.099% 100.248% ( 4400, 6600 ] 330 0.035% 100.283% ( 6600, 9900 ] 42 0.004% 100.288% ( 9900, 14000 ] 2 0.000% 100.288% ( 14000, 22000 ] 1 0.000% 100.288% ( 22000, 33000 ] 1 0.000% 100.288% ( 33000, 50000 ] 1 0.000% 100.288% ** Level 5 read latency histogram (micros): Count: 49461 Average: 661.8969 StdDev: 1042.10 Min: 74 Median: 233.5021 Max: 31485 Percentiles: P50: 233.50 P75: 720.85 P99: 5517.68 P99.9: 8779.52 P99.99: 11340.33 ------------------------------------------------------ ( 51, 76 ] 2 0.004% 0.004% ( 76, 110 ] 4371 8.837% 8.841% ## ( 110, 170 ] 14759 29.840% 38.681% ###### ( 170, 250 ] 7053 14.260% 52.941% ### ( 250, 380 ] 3980 8.047% 60.987% ## ( 380, 580 ] 4801 9.707% 70.694% ## ( 580, 870 ] 4385 8.866% 79.560% ## ( 870, 1300 ] 3737 7.555% 87.115% ## ( 1300, 1900 ] 2211 4.470% 91.585% # ( 1900, 2900 ] 1589 3.213% 94.798% # ( 2900, 4400 ] 1706 3.449% 98.247% # ( 4400, 6600 ] 733 1.482% 99.729% ( 6600, 9900 ] 128 0.259% 99.988% ( 9900, 14000 ] 3 0.006% 99.994% ( 14000, 22000 ] 2 0.004% 99.998% ( 22000, 33000 ] 2 0.004% 100.002% ** DB Stats ** Uptime(secs): 11.9 total, 4.4 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 164.85 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:4.248 H:M:S, 35.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 222.57 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.797 H:M:S, 18.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/29-03:10:33 ... thread 0: (1000000,3000000) ops and (201806.6,179078.6) ops/second in (4.955240,16.752423) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/7 3.83 GB 283.2 2.6 0.0 2.6 5.3 2.8 0.0 1.9 101.5 210.2 26.06 18.04 48 0.543 2686K 2978 0.0 0.0 L1 9/9 966.73 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 12/0 953.25 MB 1.0 1.2 0.7 0.5 1.2 0.7 0.0 1.8 113.4 112.4 10.90 5.77 6 1.817 733K 2127 0.0 0.0 L4 176/0 9.66 GB 1.0 6.9 3.0 3.9 6.8 2.9 0.1 2.3 117.6 115.3 60.06 32.45 30 2.002 1510K 0 0.0 0.0 L5 1136/0 93.57 GB 1.0 0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1352/16 108.94 GB 0.0 10.7 3.7 7.0 13.3 6.3 3.0 4.8 112.8 140.5 97.02 56.27 84 1.155 4930K 5105 0.0 0.0 Int 0/0 0.00 KB 0.0 10.7 3.7 7.0 13.3 6.3 3.0 4.8 112.8 140.5 97.02 56.27 84 1.155 4930K 5105 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 10.7 3.7 7.0 10.5 3.5 0.0 0.0 131.9 130.1 83.01 48.31 45 1.845 4930K 5105 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 202.4 14.01 7.95 39 0.359 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 16.9 total, 16.8 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 13.31 GB write, 806.78 MB/s write, 10.69 GB read, 647.92 MB/s read, 97.0 seconds Interval compaction: 13.31 GB write, 813.60 MB/s write, 10.69 GB read, 653.39 MB/s read, 97.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 5, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 5, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 28, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 5, total-stops: 28, interval: 33 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 936679 Average: 15.5449 StdDev: 118.23 Min: 1 Median: 1.6511 Max: 16029 Percentiles: P50: 1.65 P75: 2.42 P99: 277.24 P99.9: 1293.25 P99.99: 2418.22 ------------------------------------------------------ [ 0, 1 ] 139457 14.888% 14.888% ### ( 1, 2 ] 505137 53.929% 68.817% ########### ( 2, 3 ] 136419 14.564% 83.381% ### ( 3, 4 ] 53456 5.707% 89.088% # ( 4, 6 ] 48661 5.195% 94.283% # ( 6, 10 ] 11286 1.205% 95.488% ( 10, 15 ] 2340 0.250% 95.738% ( 15, 22 ] 285 0.030% 95.768% ( 22, 34 ] 20 0.002% 95.770% ( 34, 51 ] 21 0.002% 95.773% ( 51, 76 ] 37 0.004% 95.777% ( 76, 110 ] 5068 0.541% 96.318% ( 110, 170 ] 17071 1.823% 98.140% ( 170, 250 ] 7394 0.789% 98.930% ( 250, 380 ] 3151 0.336% 99.266% ( 380, 580 ] 3103 0.331% 99.597% ( 580, 870 ] 1727 0.184% 99.782% ( 870, 1300 ] 1127 0.120% 99.902% ( 1300, 1900 ] 631 0.067% 99.969% ( 1900, 2900 ] 375 0.040% 100.009% ( 2900, 4400 ] 271 0.029% 100.038% ( 4400, 6600 ] 59 0.006% 100.045% ( 6600, 9900 ] 11 0.001% 100.046% ( 14000, 22000 ] 1 0.000% 100.046% ** Level 1 read latency histogram (micros): Count: 88418 Average: 11.3224 StdDev: 101.23 Min: 1 Median: 3.6474 Max: 7596 Percentiles: P50: 3.65 P75: 5.09 P99: 163.70 P99.9: 1163.82 P99.99: 4993.50 ------------------------------------------------------ [ 0, 1 ] 757 0.856% 0.856% ( 1, 2 ] 15309 17.314% 18.171% ### ( 2, 3 ] 16195 18.316% 36.487% #### ( 3, 4 ] 18454 20.871% 57.358% #### ( 4, 6 ] 28696 32.455% 89.813% ###### ( 6, 10 ] 5327 6.025% 95.838% # ( 10, 15 ] 1142 1.292% 97.130% ( 15, 22 ] 587 0.664% 97.793% ( 22, 34 ] 61 0.069% 97.862% ( 34, 51 ] 5 0.006% 97.868% ( 51, 76 ] 1 0.001% 97.869% ( 76, 110 ] 282 0.319% 98.188% ( 110, 170 ] 802 0.907% 99.095% ( 170, 250 ] 314 0.355% 99.450% ( 250, 380 ] 132 0.149% 99.600% ( 380, 580 ] 124 0.140% 99.740% ( 580, 870 ] 104 0.118% 99.857% ( 870, 1300 ] 55 0.062% 99.920% ( 1300, 1900 ] 37 0.042% 99.962% ( 1900, 2900 ] 13 0.015% 99.976% ( 2900, 4400 ] 10 0.011% 99.988% ( 4400, 6600 ] 8 0.009% 99.997% ( 6600, 9900 ] 4 0.005% 100.001% ** Level 2 read latency histogram (micros): Count: 65430 Average: 9.3356 StdDev: 127.54 Min: 1 Median: 1.9529 Max: 7481 Percentiles: P50: 1.95 P75: 2.94 P99: 9.93 P99.9: 2202.00 P99.99: 4934.23 ------------------------------------------------------ [ 0, 1 ] 1269 1.939% 1.939% ( 1, 2 ] 33000 50.436% 52.375% ########## ( 2, 3 ] 15794 24.139% 76.514% ##### ( 3, 4 ] 6482 9.907% 86.421% ## ( 4, 6 ] 6451 9.859% 96.280% ## ( 6, 10 ] 1810 2.766% 99.046% # ( 10, 15 ] 288 0.440% 99.486% ( 15, 22 ] 32 0.049% 99.535% ( 22, 34 ] 4 0.006% 99.541% ( 34, 51 ] 7 0.011% 99.552% ( 51, 76 ] 9 0.014% 99.566% ( 76, 110 ] 4 0.006% 99.572% ( 110, 170 ] 6 0.009% 99.581% ( 170, 250 ] 8 0.012% 99.593% ( 250, 380 ] 17 0.026% 99.619% ( 380, 580 ] 37 0.057% 99.676% ( 580, 870 ] 52 0.079% 99.755% ( 870, 1300 ] 46 0.070% 99.826% ( 1300, 1900 ] 38 0.058% 99.884% ( 1900, 2900 ] 35 0.053% 99.937% ( 2900, 4400 ] 33 0.050% 99.988% ( 4400, 6600 ] 6 0.009% 99.997% ( 6600, 9900 ] 2 0.003% 100.000% ** Level 3 read latency histogram (micros): Count: 710037 Average: 8.0811 StdDev: 109.48 Min: 1 Median: 1.9628 Max: 21777 Percentiles: P50: 1.96 P75: 2.96 P99: 9.78 P99.9: 506.18 P99.99: 882.50 ------------------------------------------------------ [ 0, 1 ] 14210 2.001% 2.001% ( 1, 2 ] 353980 49.854% 51.855% ########## ( 2, 3 ] 171267 24.121% 75.976% ##### ( 3, 4 ] 71262 10.036% 86.012% ## ( 4, 6 ] 74982 10.560% 96.573% ## ( 6, 10 ] 18220 2.566% 99.139% # ( 10, 15 ] 3016 0.425% 99.563% ( 15, 22 ] 374 0.053% 99.616% ( 22, 34 ] 53 0.007% 99.624% ( 34, 51 ] 65 0.009% 99.633% ( 51, 76 ] 69 0.010% 99.642% ( 76, 110 ] 269 0.038% 99.680% ( 110, 170 ] 759 0.107% 99.787% ( 170, 250 ] 318 0.045% 99.832% ( 250, 380 ] 247 0.035% 99.867% ( 380, 580 ] 374 0.053% 99.919% ( 580, 870 ] 486 0.068% 99.988% ( 870, 1300 ] 516 0.073% 100.061% ( 1300, 1900 ] 371 0.052% 100.113% ( 1900, 2900 ] 244 0.034% 100.147% ( 2900, 4400 ] 223 0.031% 100.179% ( 4400, 6600 ] 48 0.007% 100.185% ( 6600, 9900 ] 10 0.001% 100.187% ( 9900, 14000 ] 1 0.000% 100.187% ( 14000, 22000 ] 1 0.000% 100.187% ** Level 4 read latency histogram (micros): Count: 946102 Average: 20.5763 StdDev: 200.73 Min: 1 Median: 1.9700 Max: 34622 Percentiles: P50: 1.97 P75: 3.02 P99: 230.29 P99.9: 1325.76 P99.99: 1692.00 ------------------------------------------------------ [ 0, 1 ] 19863 2.099% 2.099% ( 1, 2 ] 467200 49.382% 51.481% ########## ( 2, 3 ] 220646 23.322% 74.803% ##### ( 3, 4 ] 94356 9.973% 84.776% ## ( 4, 6 ] 92421 9.769% 94.544% ## ( 6, 10 ] 25359 2.680% 97.225% # ( 10, 15 ] 4207 0.445% 97.669% ( 15, 22 ] 555 0.059% 97.728% ( 22, 34 ] 73 0.008% 97.736% ( 34, 51 ] 107 0.011% 97.747% ( 51, 76 ] 107 0.011% 97.758% ( 76, 110 ] 2395 0.253% 98.012% ( 110, 170 ] 7195 0.760% 98.772% ( 170, 250 ] 2862 0.303% 99.075% ( 250, 380 ] 1574 0.166% 99.241% ( 380, 580 ] 2025 0.214% 99.455% ( 580, 870 ] 2148 0.227% 99.682% ( 870, 1300 ] 2003 0.212% 99.894% ( 1300, 1900 ] 1395 0.147% 100.041% ( 1900, 2900 ] 1008 0.107% 100.148% ( 2900, 4400 ] 925 0.098% 100.245% ( 4400, 6600 ] 330 0.035% 100.280% ( 6600, 9900 ] 42 0.004% 100.285% ( 9900, 14000 ] 2 0.000% 100.285% ( 14000, 22000 ] 1 0.000% 100.285% ( 22000, 33000 ] 1 0.000% 100.285% ( 33000, 50000 ] 1 0.000% 100.285% ** Level 5 read latency histogram (micros): Count: 118860 Average: 379.5735 StdDev: 719.96 Min: 74 Median: 159.7986 Max: 31485 Percentiles: P50: 159.80 P75: 284.63 P99: 4115.22 P99.9: 6887.20 P99.99: 9645.13 ------------------------------------------------------ ( 51, 76 ] 17 0.014% 0.014% ( 76, 110 ] 17095 14.382% 14.397% ### ( 110, 170 ] 50987 42.897% 57.293% ######### ( 170, 250 ] 18859 15.867% 73.160% ### ( 250, 380 ] 8211 6.908% 80.068% # ( 380, 580 ] 8056 6.778% 86.846% # ( 580, 870 ] 5320 4.476% 91.322% # ( 870, 1300 ] 3871 3.257% 94.578% # ( 1300, 1900 ] 2245 1.889% 96.467% ( 1900, 2900 ] 1621 1.364% 97.831% ( 2900, 4400 ] 1715 1.443% 99.274% ( 4400, 6600 ] 733 0.617% 99.891% ( 6600, 9900 ] 128 0.108% 99.998% ( 9900, 14000 ] 3 0.003% 100.001% ( 14000, 22000 ] 2 0.002% 100.003% ( 22000, 33000 ] 2 0.002% 100.004% ** DB Stats ** Uptime(secs): 16.9 total, 5.0 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 174.75 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.663 H:M:S, 33.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 198.62 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.416 H:M:S, 28.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 222.172 micros/op 17983 ops/sec 60.165 seconds 1081996 operations; 17.4 MB/s (276999 of 276999 found) Microseconds per read: Count: 1081996 Average: 222.1737 StdDev: 648.82 Min: 2 Median: 164.5438 Max: 62560 Percentiles: P50: 164.54 P75: 222.80 P99: 912.60 P99.9: 10185.35 P99.99: 28572.00 ------------------------------------------------------ ( 1, 2 ] 9 0.001% 0.001% ( 2, 3 ] 50 0.005% 0.005% ( 3, 4 ] 72 0.007% 0.012% ( 4, 6 ] 86 0.008% 0.020% ( 6, 10 ] 513 0.047% 0.067% ( 10, 15 ] 4035 0.373% 0.440% ( 15, 22 ] 3964 0.366% 0.807% ( 22, 34 ] 8830 0.816% 1.623% ( 34, 51 ] 1995 0.184% 1.807% ( 51, 76 ] 317 0.029% 1.837% ( 76, 110 ] 32613 3.014% 4.851% # ( 110, 170 ] 537382 49.666% 54.516% ########## ( 170, 250 ] 335812 31.036% 85.553% ###### ( 250, 380 ] 95902 8.863% 94.416% ## ( 380, 580 ] 35827 3.311% 97.727% # ( 580, 870 ] 13267 1.226% 98.954% ( 870, 1300 ] 5067 0.468% 99.422% ( 1300, 1900 ] 2185 0.202% 99.624% ( 1900, 2900 ] 1425 0.132% 99.756% ( 2900, 4400 ] 980 0.091% 99.846% ( 4400, 6600 ] 368 0.034% 99.880% ( 6600, 9900 ] 188 0.017% 99.898% ( 9900, 14000 ] 388 0.036% 99.933% ( 14000, 22000 ] 519 0.048% 99.981% ( 22000, 33000 ] 157 0.015% 99.996% ( 33000, 50000 ] 38 0.004% 99.999% ( 50000, 75000 ] 7 0.001% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 14844990 rocksdb.bloom.filter.full.positive COUNT : 1226780 rocksdb.bloom.filter.full.true.positive COUNT : 1081531 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 465 rocksdb.memtable.miss COUNT : 1081531 rocksdb.l0.hit COUNT : 41859 rocksdb.l1.hit COUNT : 17575 rocksdb.l2andup.hit COUNT : 1022097 rocksdb.compaction.key.drop.new COUNT : 199971 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 3957288 rocksdb.number.keys.read COUNT : 1081996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 4083921216 rocksdb.bytes.read COUNT : 1081996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1577 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 44358639 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 17389450472 rocksdb.compact.write.bytes COUNT : 17177996889 rocksdb.flush.write.bytes COUNT : 4029564365 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 39 rocksdb.number.superversion_acquires COUNT : 705 rocksdb.number.superversion_releases COUNT : 685 rocksdb.number.superversion_cleanups COUNT : 177 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 223 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 3997028352 rocksdb.memtable.garbage.bytes.at.flush COUNT : 1511424 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 21505062940 rocksdb.non.last.level.read.count COUNT : 5051063 rocksdb.block.checksum.compute.count COUNT : 5448430 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 163.713366 P95 : 413.491121 P99 : 910.996282 P100 : 62556.000000 COUNT : 1081996 SUM : 239440480 rocksdb.db.write.micros P50 : 1.572071 P95 : 5.154203 P99 : 14.297674 P100 : 936966.000000 COUNT : 3957288 SUM : 54798089 rocksdb.compaction.times.micros P50 : 1861111.111111 P95 : 4171666.666667 P99 : 27390000.000000 P100 : 29508948.000000 COUNT : 51 SUM : 122144458 rocksdb.compaction.times.cpu_micros P50 : 1139285.714286 P95 : 1881666.666667 P99 : 14035624.000000 P100 : 14035624.000000 COUNT : 51 SUM : 71795977 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 795.303030 P95 : 83750.000000 P99 : 389044.000000 P100 : 389044.000000 COUNT : 55 SUM : 858349 rocksdb.compaction.outfile.sync.micros P50 : 45466.666667 P95 : 232000.000000 P99 : 482600.000000 P100 : 1104766.000000 COUNT : 192 SUM : 14535947 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 210.000000 P95 : 5555.000000 P99 : 7742.000000 P100 : 7742.000000 COUNT : 117 SUM : 130233 rocksdb.table.open.io.micros P50 : 9413.175676 P95 : 20174.169742 P99 : 23040.769231 P100 : 48067.000000 COUNT : 1577 SUM : 15193206 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.978443 P95 : 98.721530 P99 : 208.856971 P100 : 34628.000000 COUNT : 4210087 SUM : 74977894 rocksdb.read.block.get.micros P50 : 138.408920 P95 : 298.696378 P99 : 856.198855 P100 : 21558.000000 COUNT : 1236773 SUM : 204400310 rocksdb.write.raw.block.micros P50 : 0.545391 P95 : 1.542081 P99 : 3.763521 P100 : 39403.000000 COUNT : 5136549 SUM : 14922227 rocksdb.numfiles.in.singlecompaction P50 : 3.480000 P95 : 5.986667 P99 : 26.560000 P100 : 28.000000 COUNT : 62 SUM : 289 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.578137 P95 : 1253.455665 P99 : 2649.312348 P100 : 936801.000000 COUNT : 217117 SUM : 44381216 rocksdb.sst.read.micros P50 : 2.478054 P95 : 169.743833 P99 : 466.382015 P100 : 34622.000000 COUNT : 5051129 SUM : 263167400 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1081996 SUM : 1081996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 3957288 SUM : 4083921216 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 223658.536585 P95 : 966250.000000 P99 : 1153250.000000 P100 : 1194682.000000 COUNT : 55 SUM : 17293821 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 227 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0