[2024-11-17 01:44:57.293049] Starting SPDK v24.09.1-pre git sha1 b18e1bd62 / DPDK 23.11.0 initialization... [2024-11-17 01:44:57.293168] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1460099 ] Set seed to 1731804297290644 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-17 01:44:57.393769] app.c: 917:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-17 01:44:57.436362] app.c: 610:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-17 01:44:57.436406] app.c: 611:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1460099' to capture a snapshot of events at runtime. [2024-11-17 01:44:57.436416] app.c: 616:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-17 01:44:57.436443] app.c: 617:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-17 01:44:57.436451] app.c: 618:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1460099 for offline analysis/debug. [2024-11-17 01:44:57.436537] reactor.c: 990:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sun Nov 17 01:45:01 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/17-01:45:05 ... thread 0: (1000000,1000000) ops and (236682.6,236682.6) ops/second in (4.225067,4.225067) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/15 3.74 GB 62.9 0.6 0.0 0.6 1.4 0.9 0.0 1.6 98.1 250.7 5.78 4.53 15 0.385 575K 654 0.0 0.0 L1 7/7 728.59 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 17/0 907.69 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 157/0 9.47 GB 1.0 1.7 0.5 1.2 1.6 0.5 0.0 3.2 151.3 148.9 11.33 8.33 5 2.266 0 0 0.0 0.0 L5 1108/0 92.27 GB 0.9 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1307/22 107.08 GB 0.0 2.2 0.5 1.7 3.1 1.3 2.0 3.6 133.4 183.3 17.11 12.86 20 0.855 575K 654 0.0 0.0 Int 0/0 0.00 KB 0.0 2.2 0.5 1.7 3.1 1.3 2.0 3.6 133.4 183.3 17.11 12.86 20 0.855 575K 654 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 2.2 0.5 1.7 2.2 0.5 0.0 0.0 168.1 166.1 13.57 10.52 7 1.938 575K 654 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 249.3 3.54 2.35 13 0.272 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 4.4 total, 4.2 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 3.06 GB write, 709.14 MB/s write, 2.23 GB read, 516.00 MB/s read, 17.1 seconds Interval compaction: 3.06 GB write, 741.82 MB/s write, 2.23 GB read, 539.78 MB/s read, 17.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 2, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 2, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 5, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 2, total-stops: 5, interval: 7 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 292252 Average: 10.6904 StdDev: 113.94 Min: 1 Median: 1.6997 Max: 13062 Percentiles: P50: 1.70 P75: 2.48 P99: 205.42 P99.9: 865.76 P99.99: 1496.55 ------------------------------------------------------ [ 0, 1 ] 34565 11.827% 11.827% ## ( 1, 2 ] 159446 54.558% 66.385% ########### ( 2, 3 ] 52256 17.880% 84.265% #### ( 3, 4 ] 17944 6.140% 90.405% # ( 4, 6 ] 16031 5.485% 95.891% # ( 6, 10 ] 4276 1.463% 97.354% ( 10, 15 ] 794 0.272% 97.625% ( 15, 22 ] 133 0.046% 97.671% ( 22, 34 ] 34 0.012% 97.682% ( 34, 51 ] 13 0.004% 97.687% ( 51, 76 ] 7 0.002% 97.689% ( 76, 110 ] 762 0.261% 97.950% ( 110, 170 ] 2474 0.847% 98.797% ( 170, 250 ] 1345 0.460% 99.257% ( 250, 380 ] 805 0.275% 99.532% ( 380, 580 ] 654 0.224% 99.756% ( 580, 870 ] 428 0.146% 99.902% ( 870, 1300 ] 225 0.077% 99.979% ( 1300, 1900 ] 97 0.033% 100.013% ( 1900, 2900 ] 63 0.022% 100.034% ( 2900, 4400 ] 34 0.012% 100.046% ( 4400, 6600 ] 5 0.002% 100.048% ( 6600, 9900 ] 10 0.003% 100.051% ( 9900, 14000 ] 8 0.003% 100.054% ** Level 1 read latency histogram (micros): Count: 136 Average: 652.6324 StdDev: 1637.71 Min: 84 Median: 224.1935 Max: 14428 Percentiles: P50: 224.19 P75: 446.67 P99: 12524.00 P99.9: 14428.00 P99.99: 14428.00 ------------------------------------------------------ ( 76, 110 ] 10 7.353% 7.353% # ( 110, 170 ] 37 27.206% 34.559% ##### ( 170, 250 ] 31 22.794% 57.353% ##### ( 250, 380 ] 20 14.706% 72.059% ### ( 380, 580 ] 12 8.824% 80.882% ## ( 580, 870 ] 7 5.147% 86.029% # ( 870, 1300 ] 5 3.676% 89.706% # ( 1300, 1900 ] 5 3.676% 93.382% # ( 1900, 2900 ] 4 2.941% 96.324% # ( 2900, 4400 ] 2 1.471% 97.794% ( 4400, 6600 ] 1 0.735% 98.529% ( 9900, 14000 ] 1 0.735% 99.265% ( 14000, 22000 ] 1 0.735% 100.000% ** Level 3 read latency histogram (micros): Count: 84273 Average: 6.4284 StdDev: 79.71 Min: 1 Median: 1.9338 Max: 9551 Percentiles: P50: 1.93 P75: 2.89 P99: 12.11 P99.9: 753.66 P99.99: 1769.74 ------------------------------------------------------ [ 0, 1 ] 3044 3.612% 3.612% # ( 1, 2 ] 41864 49.677% 53.289% ########## ( 2, 3 ] 20444 24.259% 77.548% ##### ( 3, 4 ] 7793 9.247% 86.795% ## ( 4, 6 ] 7656 9.085% 95.880% ## ( 6, 10 ] 2467 2.927% 98.807% # ( 10, 15 ] 385 0.457% 99.264% ( 15, 22 ] 66 0.078% 99.343% ( 22, 34 ] 37 0.044% 99.387% ( 34, 51 ] 9 0.011% 99.397% ( 51, 76 ] 4 0.005% 99.402% ( 76, 110 ] 28 0.033% 99.435% ( 110, 170 ] 121 0.144% 99.579% ( 170, 250 ] 69 0.082% 99.661% ( 250, 380 ] 66 0.078% 99.739% ( 380, 580 ] 98 0.116% 99.855% ( 580, 870 ] 63 0.075% 99.930% ( 870, 1300 ] 31 0.037% 99.967% ( 1300, 1900 ] 25 0.030% 99.996% ( 1900, 2900 ] 14 0.017% 100.013% ( 2900, 4400 ] 7 0.008% 100.021% ( 6600, 9900 ] 3 0.004% 100.025% ** Level 4 read latency histogram (micros): Count: 244649 Average: 15.4870 StdDev: 150.02 Min: 1 Median: 1.9253 Max: 14455 Percentiles: P50: 1.93 P75: 2.89 P99: 283.29 P99.9: 1364.35 P99.99: 1831.17 ------------------------------------------------------ [ 0, 1 ] 7746 3.166% 3.166% # ( 1, 2 ] 123833 50.617% 53.783% ########## ( 2, 3 ] 58560 23.936% 77.719% ##### ( 3, 4 ] 21721 8.878% 86.598% ## ( 4, 6 ] 20699 8.461% 95.058% ## ( 6, 10 ] 6764 2.765% 97.823% # ( 10, 15 ] 1117 0.457% 98.280% ( 15, 22 ] 204 0.083% 98.363% ( 22, 34 ] 62 0.025% 98.388% ( 34, 51 ] 30 0.012% 98.401% ( 51, 76 ] 20 0.008% 98.409% ( 76, 110 ] 164 0.067% 98.476% ( 110, 170 ] 657 0.269% 98.744% ( 170, 250 ] 499 0.204% 98.948% ( 250, 380 ] 494 0.202% 99.150% ( 380, 580 ] 646 0.264% 99.414% ( 580, 870 ] 649 0.265% 99.680% ( 870, 1300 ] 509 0.208% 99.888% ( 1300, 1900 ] 283 0.116% 100.003% ( 1900, 2900 ] 156 0.064% 100.067% ( 2900, 4400 ] 115 0.047% 100.114% ( 4400, 6600 ] 27 0.011% 100.125% ( 6600, 9900 ] 6 0.002% 100.128% ( 9900, 14000 ] 1 0.000% 100.128% ( 14000, 22000 ] 1 0.000% 100.128% ** Level 5 read latency histogram (micros): Count: 17473 Average: 663.2158 StdDev: 1067.50 Min: 72 Median: 232.5794 Max: 33778 Percentiles: P50: 232.58 P75: 638.84 P99: 5215.29 P99.9: 8352.70 P99.99: 10936.07 ------------------------------------------------------ ( 51, 76 ] 4 0.023% 0.023% ( 76, 110 ] 1431 8.190% 8.213% ## ( 110, 170 ] 5209 29.812% 38.024% ###### ( 170, 250 ] 2675 15.309% 53.334% ### ( 250, 380 ] 1735 9.930% 63.263% ## ( 380, 580 ] 1787 10.227% 73.491% ## ( 580, 870 ] 1300 7.440% 80.931% # ( 870, 1300 ] 889 5.088% 86.018% # ( 1300, 1900 ] 658 3.766% 89.784% # ( 1900, 2900 ] 833 4.767% 94.552% # ( 2900, 4400 ] 695 3.978% 98.529% # ( 4400, 6600 ] 222 1.271% 99.800% ( 6600, 9900 ] 33 0.189% 99.989% ( 9900, 14000 ] 1 0.006% 99.994% ( 14000, 22000 ] 1 0.006% 100.000% ( 22000, 33000 ] 1 0.006% 100.006% ( 33000, 50000 ] 1 0.006% 100.011% ** DB Stats ** Uptime(secs): 4.4 total, 4.2 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 222.59 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.557 H:M:S, 12.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 232.80 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.557 H:M:S, 13.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/17-01:45:13 ... thread 0: (1000000,2000000) ops and (131310.1,168909.9) ops/second in (7.615562,11.840629) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/15 4.72 GB 163.2 2.6 0.0 2.6 4.4 1.8 0.0 2.4 159.3 272.7 16.61 14.84 36 0.461 2685K 6277 0.0 0.0 L1 7/7 728.59 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 17/0 907.69 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 157/0 9.47 GB 1.0 1.7 0.5 1.2 1.6 0.5 0.0 3.2 151.3 148.9 11.33 8.33 5 2.266 0 0 0.0 0.0 L5 1108/0 92.27 GB 0.9 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1307/22 108.06 GB 0.0 4.3 0.5 3.7 6.1 2.3 2.0 3.3 156.1 222.5 27.93 23.17 41 0.681 2685K 6277 0.0 0.0 Int 0/0 0.00 KB 0.0 4.3 0.5 3.7 6.1 2.3 2.0 3.3 156.1 222.5 27.93 23.17 41 0.681 2685K 6277 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 4.3 0.5 3.7 4.2 0.5 0.0 0.0 204.4 202.8 21.33 18.18 12 1.777 2685K 6277 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 286.0 6.61 4.99 29 0.228 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 12.0 total, 11.8 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.07 GB write, 516.39 MB/s write, 4.26 GB read, 362.17 MB/s read, 27.9 seconds Interval compaction: 6.07 GB write, 524.88 MB/s write, 4.26 GB read, 368.12 MB/s read, 27.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 11, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 11, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 5, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 11, total-stops: 5, interval: 16 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 864091 Average: 16.2139 StdDev: 85.00 Min: 0 Median: 1.5252 Max: 13062 Percentiles: P50: 1.53 P75: 1.99 P99: 246.56 P99.9: 639.91 P99.99: 843.46 ------------------------------------------------------ [ 0, 1 ] 188241 21.785% 21.785% #### ( 1, 2 ] 464245 53.726% 75.511% ########### ( 2, 3 ] 89308 10.335% 85.847% ## ( 3, 4 ] 30409 3.519% 89.366% # ( 4, 6 ] 22818 2.641% 92.007% # ( 6, 10 ] 5677 0.657% 92.664% ( 10, 15 ] 1029 0.119% 92.783% ( 15, 22 ] 169 0.020% 92.802% ( 22, 34 ] 40 0.005% 92.807% ( 34, 51 ] 17 0.002% 92.809% ( 51, 76 ] 14 0.002% 92.810% ( 76, 110 ] 8949 1.036% 93.846% ( 110, 170 ] 33302 3.854% 97.700% # ( 170, 250 ] 11738 1.358% 99.059% ( 250, 380 ] 3965 0.459% 99.517% ( 380, 580 ] 3078 0.356% 99.874% ( 580, 870 ] 1108 0.128% 100.002% ( 870, 1300 ] 315 0.036% 100.038% ( 1300, 1900 ] 111 0.013% 100.051% ( 1900, 2900 ] 84 0.010% 100.061% ( 2900, 4400 ] 47 0.005% 100.066% ( 4400, 6600 ] 5 0.001% 100.067% ( 6600, 9900 ] 10 0.001% 100.068% ( 9900, 14000 ] 8 0.001% 100.069% ** Level 1 read latency histogram (micros): Count: 1054 Average: 233.1366 StdDev: 619.32 Min: 79 Median: 149.1771 Max: 14428 Percentiles: P50: 149.18 P75: 200.89 P99: 1835.20 P99.9: 13778.60 P99.99: 14428.00 ------------------------------------------------------ ( 76, 110 ] 162 15.370% 15.370% ### ( 110, 170 ] 559 53.036% 68.406% ########### ( 170, 250 ] 180 17.078% 85.484% ### ( 250, 380 ] 71 6.736% 92.220% # ( 380, 580 ] 46 4.364% 96.584% # ( 580, 870 ] 13 1.233% 97.818% ( 870, 1300 ] 8 0.759% 98.577% ( 1300, 1900 ] 5 0.474% 99.051% ( 1900, 2900 ] 5 0.474% 99.526% ( 2900, 4400 ] 2 0.190% 99.715% ( 4400, 6600 ] 1 0.095% 99.810% ( 9900, 14000 ] 1 0.095% 99.905% ( 14000, 22000 ] 1 0.095% 100.000% ** Level 3 read latency histogram (micros): Count: 85815 Average: 9.3896 StdDev: 83.36 Min: 1 Median: 1.9522 Max: 9551 Percentiles: P50: 1.95 P75: 2.95 P99: 164.67 P99.9: 788.32 P99.99: 1775.97 ------------------------------------------------------ [ 0, 1 ] 3044 3.547% 3.547% # ( 1, 2 ] 41864 48.784% 52.331% ########## ( 2, 3 ] 20444 23.823% 76.155% ##### ( 3, 4 ] 7793 9.081% 85.236% ## ( 4, 6 ] 7656 8.922% 94.157% ## ( 6, 10 ] 2467 2.875% 97.032% # ( 10, 15 ] 385 0.449% 97.481% ( 15, 22 ] 66 0.077% 97.558% ( 22, 34 ] 37 0.043% 97.601% ( 34, 51 ] 9 0.010% 97.611% ( 51, 76 ] 4 0.005% 97.616% ( 76, 110 ] 304 0.354% 97.970% ( 110, 170 ] 970 1.130% 99.100% ( 170, 250 ] 339 0.395% 99.495% ( 250, 380 ] 135 0.157% 99.653% ( 380, 580 ] 154 0.179% 99.832% ( 580, 870 ] 81 0.094% 99.927% ( 870, 1300 ] 33 0.038% 99.965% ( 1300, 1900 ] 27 0.031% 99.997% ( 1900, 2900 ] 14 0.016% 100.013% ( 2900, 4400 ] 7 0.008% 100.021% ( 6600, 9900 ] 3 0.003% 100.024% ** Level 4 read latency histogram (micros): Count: 259237 Average: 23.4787 StdDev: 152.06 Min: 1 Median: 1.9833 Max: 14455 Percentiles: P50: 1.98 P75: 3.17 P99: 412.53 P99.9: 1349.68 P99.99: 1837.44 ------------------------------------------------------ [ 0, 1 ] 7747 2.988% 2.988% # ( 1, 2 ] 123942 47.810% 50.799% ########## ( 2, 3 ] 58957 22.743% 73.541% ##### ( 3, 4 ] 21998 8.486% 82.027% ## ( 4, 6 ] 21164 8.164% 90.191% ## ( 6, 10 ] 6816 2.629% 92.820% # ( 10, 15 ] 1131 0.436% 93.256% ( 15, 22 ] 206 0.079% 93.336% ( 22, 34 ] 62 0.024% 93.360% ( 34, 51 ] 30 0.012% 93.371% ( 51, 76 ] 21 0.008% 93.379% ( 76, 110 ] 2429 0.937% 94.316% ( 110, 170 ] 7943 3.064% 97.380% # ( 170, 250 ] 2793 1.077% 98.458% ( 250, 380 ] 1214 0.468% 98.926% ( 380, 580 ] 1178 0.454% 99.380% ( 580, 870 ] 799 0.308% 99.689% ( 870, 1300 ] 524 0.202% 99.891% ( 1300, 1900 ] 287 0.111% 100.002% ( 1900, 2900 ] 159 0.061% 100.063% ( 2900, 4400 ] 117 0.045% 100.108% ( 4400, 6600 ] 27 0.010% 100.118% ( 6600, 9900 ] 6 0.002% 100.121% ( 9900, 14000 ] 1 0.000% 100.121% ( 14000, 22000 ] 1 0.000% 100.122% ** Level 5 read latency histogram (micros): Count: 125964 Average: 240.3059 StdDev: 444.26 Min: 72 Median: 149.6306 Max: 33778 Percentiles: P50: 149.63 P75: 205.37 P99: 2541.03 P99.9: 5629.19 P99.99: 8140.36 ------------------------------------------------------ ( 51, 76 ] 18 0.014% 0.014% ( 76, 110 ] 20050 15.917% 15.932% ### ( 110, 170 ] 64971 51.579% 67.511% ########## ( 170, 250 ] 21337 16.939% 84.450% ### ( 250, 380 ] 7386 5.864% 90.313% # ( 380, 580 ] 6237 4.951% 95.265% # ( 580, 870 ] 2439 1.936% 97.201% ( 870, 1300 ] 1038 0.824% 98.025% ( 1300, 1900 ] 679 0.539% 98.564% ( 1900, 2900 ] 857 0.680% 99.244% ( 2900, 4400 ] 702 0.557% 99.802% ( 4400, 6600 ] 222 0.176% 99.978% ( 6600, 9900 ] 33 0.026% 100.004% ( 9900, 14000 ] 1 0.001% 100.005% ( 14000, 22000 ] 1 0.001% 100.006% ( 22000, 33000 ] 1 0.001% 100.006% ( 33000, 50000 ] 1 0.001% 100.007% ** DB Stats ** Uptime(secs): 12.0 total, 7.6 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 163.53 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:4.486 H:M:S, 37.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 129.24 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.929 H:M:S, 51.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 202.489 micros/op 19731 ops/sec 60.156 seconds 1186996 operations; 19.1 MB/s (300999 of 300999 found) Microseconds per read: Count: 1186996 Average: 202.4909 StdDev: 408.87 Min: 2 Median: 166.2809 Max: 61715 Percentiles: P50: 166.28 P75: 225.12 P99: 607.88 P99.9: 4293.21 P99.99: 19815.94 ------------------------------------------------------ ( 1, 2 ] 13 0.001% 0.001% ( 2, 3 ] 54 0.005% 0.006% ( 3, 4 ] 72 0.006% 0.012% ( 4, 6 ] 115 0.010% 0.021% ( 6, 10 ] 651 0.055% 0.076% ( 10, 15 ] 6896 0.581% 0.657% ( 15, 22 ] 3564 0.300% 0.957% ( 22, 34 ] 11961 1.008% 1.965% ( 34, 51 ] 2143 0.181% 2.146% ( 51, 76 ] 232 0.020% 2.165% ( 76, 110 ] 29945 2.523% 4.688% # ( 110, 170 ] 573394 48.306% 52.994% ########## ( 170, 250 ] 379076 31.936% 84.930% ###### ( 250, 380 ] 130085 10.959% 95.889% ## ( 380, 580 ] 36089 3.040% 98.930% # ( 580, 870 ] 8697 0.733% 99.662% ( 870, 1300 ] 1787 0.151% 99.813% ( 1300, 1900 ] 353 0.030% 99.843% ( 1900, 2900 ] 382 0.032% 99.875% ( 2900, 4400 ] 323 0.027% 99.902% ( 4400, 6600 ] 246 0.021% 99.923% ( 6600, 9900 ] 310 0.026% 99.949% ( 9900, 14000 ] 325 0.027% 99.976% ( 14000, 22000 ] 226 0.019% 99.995% ( 22000, 33000 ] 51 0.004% 99.999% ( 33000, 50000 ] 3 0.000% 100.000% ( 50000, 75000 ] 3 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 20560271 rocksdb.bloom.filter.full.positive COUNT : 1387431 rocksdb.bloom.filter.full.true.positive COUNT : 1186532 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 464 rocksdb.memtable.miss COUNT : 1186532 rocksdb.l0.hit COUNT : 61921 rocksdb.l1.hit COUNT : 8202 rocksdb.l2andup.hit COUNT : 1116409 rocksdb.compaction.key.drop.new COUNT : 59079 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2630590 rocksdb.number.keys.read COUNT : 1186996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2714768880 rocksdb.bytes.read COUNT : 1186996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1384 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 49149155 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 8152924266 rocksdb.compact.write.bytes COUNT : 8098770471 rocksdb.flush.write.bytes COUNT : 2708508313 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 46 rocksdb.number.superversion_acquires COUNT : 418 rocksdb.number.superversion_releases COUNT : 393 rocksdb.number.superversion_cleanups COUNT : 103 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2686516224 rocksdb.memtable.garbage.bytes.at.flush COUNT : 910336 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 14079017569 rocksdb.non.last.level.read.count COUNT : 3255864 rocksdb.block.checksum.compute.count COUNT : 3376219 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 165.415311 P95 : 368.972563 P99 : 605.699247 P100 : 61713.000000 COUNT : 1186996 SUM : 239308936 rocksdb.db.write.micros P50 : 1.595600 P95 : 5.939613 P99 : 1024.219386 P100 : 287748.000000 COUNT : 2630590 SUM : 56469523 rocksdb.compaction.times.micros P50 : 1200000.000000 P95 : 3740000.000000 P99 : 3972173.000000 P100 : 3972173.000000 COUNT : 16 SUM : 27260516 rocksdb.compaction.times.cpu_micros P50 : 1132000.000000 P95 : 3740000.000000 P99 : 3935727.000000 P100 : 3935727.000000 COUNT : 16 SUM : 24024618 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 737.083333 P95 : 4400.000000 P99 : 60410.000000 P100 : 60410.000000 COUNT : 40 SUM : 99299 rocksdb.compaction.outfile.sync.micros P50 : 1480.000000 P95 : 27912.500000 P99 : 34599.000000 P100 : 34599.000000 COUNT : 57 SUM : 304958 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 142.608696 P95 : 1670.000000 P99 : 4504.000000 P100 : 4504.000000 COUNT : 63 SUM : 25327 rocksdb.table.open.io.micros P50 : 6459.200000 P95 : 13834.384236 P99 : 20618.947368 P100 : 50787.000000 COUNT : 1384 SUM : 9854936 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.756436 P95 : 157.725517 P99 : 238.837464 P100 : 14458.000000 COUNT : 1975996 SUM : 57610003 rocksdb.read.block.get.micros P50 : 136.732213 P95 : 238.317550 P99 : 509.980275 P100 : 38832.000000 COUNT : 1398829 SUM : 203698571 rocksdb.write.raw.block.micros P50 : 0.550179 P95 : 1.604288 P99 : 3.690431 P100 : 27336.000000 COUNT : 2617287 SUM : 6105158 rocksdb.numfiles.in.singlecompaction P50 : 4.000000 P95 : 9.800000 P99 : 18.000000 P100 : 18.000000 COUNT : 23 SUM : 120 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.583128 P95 : 1282.669586 P99 : 2683.259325 P100 : 287570.000000 COUNT : 244250 SUM : 49176062 rocksdb.sst.read.micros P50 : 77.294875 P95 : 199.157267 P99 : 380.546461 P100 : 33778.000000 COUNT : 3255922 SUM : 250988199 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1186996 SUM : 1186996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2630590 SUM : 2714768880 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 215714.285714 P95 : 380000.000000 P99 : 532000.000000 P100 : 533540.000000 COUNT : 40 SUM : 8847278 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 79 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0