[2024-12-16 17:53:21.991987] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 23.11.0 initialization... [2024-12-16 17:53:21.992196] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid966052 ] Set seed to 1734368001986766 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-12-16 17:53:22.097141] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-12-16 17:53:22.133141] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-12-16 17:53:22.133183] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 966052' to capture a snapshot of events at runtime. [2024-12-16 17:53:22.133195] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid966052 for offline analysis/debug. [2024-12-16 17:53:22.133233] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Mon Dec 16 17:53:26 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/12/16-17:53:31 ... thread 0: (1000000,1000000) ops and (180053.9,180053.9) ops/second in (5.553891,5.553891) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/18 2.83 GB 6.3 0.7 0.0 0.7 1.6 0.9 0.0 1.9 134.2 290.7 5.63 5.15 17 0.331 767K 685 0.0 0.0 L1 10/10 561.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 14.90 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 15/0 991.41 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 172/0 9.68 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L5 1112/0 91.35 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1329/28 105.40 GB 0.0 0.7 0.0 0.7 1.6 0.9 0.0 1.9 134.2 290.7 5.63 5.15 17 0.331 767K 685 0.0 0.0 Int 0/0 0.00 KB 0.0 0.7 0.0 0.7 1.6 0.9 0.0 1.9 134.2 290.7 5.63 5.15 17 0.331 767K 685 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.7 0.0 0.7 0.7 -0.0 0.0 0.0 272.9 272.7 2.77 2.72 3 0.923 767K 685 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 308.2 2.86 2.43 14 0.204 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 5.8 total, 5.6 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 1.60 GB write, 284.01 MB/s write, 0.74 GB read, 131.14 MB/s read, 5.6 seconds Interval compaction: 1.60 GB write, 294.72 MB/s write, 0.74 GB read, 136.09 MB/s read, 5.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 7, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 7, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 7, total-stops: 0, interval: 7 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 318753 Average: 17.9817 StdDev: 81.11 Min: 0 Median: 1.4061 Max: 5985 Percentiles: P50: 1.41 P75: 1.90 P99: 270.16 P99.9: 715.07 P99.99: 1177.62 ------------------------------------------------------ [ 0, 1 ] 93320 29.277% 29.277% ###### ( 1, 2 ] 162654 51.028% 80.305% ########## ( 2, 3 ] 20561 6.450% 86.755% # ( 3, 4 ] 7866 2.468% 89.223% ( 4, 6 ] 4962 1.557% 90.780% ( 6, 10 ] 2139 0.671% 91.451% ( 10, 15 ] 380 0.119% 91.570% ( 15, 22 ] 104 0.033% 91.603% ( 22, 34 ] 29 0.009% 91.612% ( 34, 51 ] 17 0.005% 91.617% ( 51, 76 ] 7 0.002% 91.619% ( 76, 110 ] 4263 1.337% 92.957% ( 110, 170 ] 14221 4.461% 97.418% # ( 170, 250 ] 4787 1.502% 98.920% ( 250, 380 ] 1654 0.519% 99.439% ( 380, 580 ] 1291 0.405% 99.844% ( 580, 870 ] 387 0.121% 99.965% ( 870, 1300 ] 112 0.035% 100.000% ( 1300, 1900 ] 63 0.020% 100.020% ( 1900, 2900 ] 48 0.015% 100.035% ( 2900, 4400 ] 27 0.008% 100.044% ( 4400, 6600 ] 8 0.003% 100.046% ** Level 1 read latency histogram (micros): Count: 15312 Average: 16.7130 StdDev: 96.57 Min: 1 Median: 1.9636 Max: 5287 Percentiles: P50: 1.96 P75: 3.96 P99: 253.94 P99.9: 1205.97 P99.99: 4001.60 ------------------------------------------------------ [ 0, 1 ] 1493 9.751% 9.751% ## ( 1, 2 ] 6396 41.771% 51.522% ######## ( 2, 3 ] 2173 14.191% 65.713% ### ( 3, 4 ] 1489 9.724% 75.438% ## ( 4, 6 ] 1851 12.089% 87.526% ## ( 6, 10 ] 747 4.879% 92.405% # ( 10, 15 ] 121 0.790% 93.195% ( 15, 22 ] 74 0.483% 93.678% ( 22, 34 ] 10 0.065% 93.743% ( 34, 51 ] 3 0.020% 93.763% ( 76, 110 ] 173 1.130% 94.893% ( 110, 170 ] 474 3.096% 97.989% # ( 170, 250 ] 153 0.999% 98.988% ( 250, 380 ] 62 0.405% 99.393% ( 380, 580 ] 55 0.359% 99.752% ( 580, 870 ] 18 0.118% 99.869% ( 870, 1300 ] 6 0.039% 99.909% ( 1300, 1900 ] 6 0.039% 99.948% ( 1900, 2900 ] 5 0.033% 99.980% ( 2900, 4400 ] 2 0.013% 99.993% ( 4400, 6600 ] 1 0.007% 100.000% ** Level 2 read latency histogram (micros): Count: 16 Average: 205.3750 StdDev: 112.51 Min: 110 Median: 156.6667 Max: 521 Percentiles: P50: 156.67 P75: 250.00 P99: 521.00 P99.9: 521.00 P99.99: 521.00 ------------------------------------------------------ ( 76, 110 ] 1 6.250% 6.250% # ( 110, 170 ] 9 56.250% 62.500% ########### ( 170, 250 ] 2 12.500% 75.000% ### ( 250, 380 ] 3 18.750% 93.750% #### ( 380, 580 ] 1 6.250% 100.000% # ** Level 3 read latency histogram (micros): Count: 784 Average: 229.9681 StdDev: 356.92 Min: 57 Median: 148.4496 Max: 4610 Percentiles: P50: 148.45 P75: 206.19 P99: 1710.67 P99.9: 4610.00 P99.99: 4610.00 ------------------------------------------------------ ( 51, 76 ] 2 0.255% 0.255% ( 76, 110 ] 142 18.112% 18.367% #### ( 110, 170 ] 387 49.362% 67.730% ########## ( 170, 250 ] 126 16.071% 83.801% ### ( 250, 380 ] 47 5.995% 89.796% # ( 380, 580 ] 43 5.485% 95.281% # ( 580, 870 ] 14 1.786% 97.066% ( 870, 1300 ] 9 1.148% 98.214% ( 1300, 1900 ] 9 1.148% 99.362% ( 2900, 4400 ] 4 0.510% 99.872% ( 4400, 6600 ] 1 0.128% 100.000% ** Level 4 read latency histogram (micros): Count: 7159 Average: 232.8747 StdDev: 314.51 Min: 48 Median: 150.7269 Max: 5686 Percentiles: P50: 150.73 P75: 214.89 P99: 1795.08 P99.9: 3540.17 P99.99: 5341.67 ------------------------------------------------------ ( 34, 51 ] 1 0.014% 0.014% ( 51, 76 ] 6 0.084% 0.098% ( 76, 110 ] 1154 16.120% 16.217% ### ( 110, 170 ] 3563 49.770% 65.987% ########## ( 170, 250 ] 1150 16.064% 82.051% ### ( 250, 380 ] 520 7.264% 89.314% # ( 380, 580 ] 358 5.001% 94.315% # ( 580, 870 ] 180 2.514% 96.829% # ( 870, 1300 ] 96 1.341% 98.170% ( 1300, 1900 ] 72 1.006% 99.176% ( 1900, 2900 ] 48 0.670% 99.846% ( 2900, 4400 ] 9 0.126% 99.972% ( 4400, 6600 ] 3 0.042% 100.014% ** Level 5 read latency histogram (micros): Count: 58391 Average: 245.8426 StdDev: 387.47 Min: 68 Median: 150.0280 Max: 6553 Percentiles: P50: 150.03 P75: 210.49 P99: 2486.35 P99.9: 4221.26 P99.99: 5910.16 ------------------------------------------------------ ( 51, 76 ] 16 0.027% 0.027% ( 76, 110 ] 9433 16.155% 16.182% ### ( 110, 170 ] 29599 50.691% 66.873% ########## ( 170, 250 ] 9375 16.056% 82.929% ### ( 250, 380 ] 3754 6.429% 89.358% # ( 380, 580 ] 3000 5.138% 94.496% # ( 580, 870 ] 1135 1.944% 96.440% ( 870, 1300 ] 562 0.962% 97.402% ( 1300, 1900 ] 559 0.957% 98.359% ( 1900, 2900 ] 638 1.093% 99.452% ( 2900, 4400 ] 297 0.509% 99.961% ( 4400, 6600 ] 25 0.043% 100.003% ** DB Stats ** Uptime(secs): 5.8 total, 5.6 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 170.71 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:2.013 H:M:S, 34.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 177.12 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.013 H:M:S, 36.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/12/16-17:53:44 ... thread 0: (1000000,2000000) ops and (77335.2,108198.1) ops/second in (12.930727,18.484618) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/14 3.87 GB 194.6 3.3 0.0 3.3 5.2 1.9 0.0 2.7 183.0 287.8 18.58 17.52 40 0.464 3452K 6643 0.0 0.0 L1 10/10 561.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 14.90 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 15/0 991.41 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 172/0 9.68 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L5 1112/0 91.35 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1328/24 106.44 GB 0.0 3.3 0.0 3.3 5.2 1.9 0.0 2.7 183.0 287.8 18.58 17.52 40 0.464 3452K 6643 0.0 0.0 Int 0/0 0.00 KB 0.0 3.3 0.0 3.3 5.2 1.9 0.0 2.7 183.0 287.8 18.58 17.52 40 0.464 3452K 6643 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 3.3 0.0 3.3 3.3 -0.0 0.0 0.0 274.6 274.1 12.38 12.21 9 1.376 3452K 6643 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 315.2 6.20 5.31 31 0.200 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.7 total, 18.5 interval Flush(GB): cumulative 1.907, interval 1.907 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 5.22 GB write, 286.00 MB/s write, 3.32 GB read, 181.89 MB/s read, 18.6 seconds Interval compaction: 5.22 GB write, 289.24 MB/s write, 3.32 GB read, 183.95 MB/s read, 18.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 23, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 23, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 23, total-stops: 0, interval: 23 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1069119 Average: 22.2010 StdDev: 73.05 Min: 0 Median: 1.3967 Max: 5985 Percentiles: P50: 1.40 P75: 1.91 P99: 260.53 P99.9: 575.29 P99.99: 797.46 ------------------------------------------------------ [ 0, 1 ] 329271 30.798% 30.798% ###### ( 1, 2 ] 517545 48.409% 79.207% ########## ( 2, 3 ] 53257 4.981% 84.188% # ( 3, 4 ] 21690 2.029% 86.217% ( 4, 6 ] 12862 1.203% 87.420% ( 6, 10 ] 3981 0.372% 87.792% ( 10, 15 ] 646 0.060% 87.853% ( 15, 22 ] 137 0.013% 87.866% ( 22, 34 ] 34 0.003% 87.869% ( 34, 51 ] 20 0.002% 87.871% ( 51, 76 ] 20 0.002% 87.873% ( 76, 110 ] 23560 2.204% 90.076% ( 110, 170 ] 74345 6.954% 97.030% # ( 170, 250 ] 20585 1.925% 98.956% ( 250, 380 ] 5864 0.548% 99.504% ( 380, 580 ] 4335 0.405% 99.910% ( 580, 870 ] 1147 0.107% 100.017% ( 870, 1300 ] 191 0.018% 100.035% ( 1300, 1900 ] 85 0.008% 100.043% ( 1900, 2900 ] 86 0.008% 100.051% ( 2900, 4400 ] 53 0.005% 100.056% ( 4400, 6600 ] 8 0.001% 100.056% ** Level 1 read latency histogram (micros): Count: 41918 Average: 18.5140 StdDev: 72.66 Min: 1 Median: 2.8214 Max: 5287 Percentiles: P50: 2.82 P75: 5.00 P99: 240.88 P99.9: 700.11 P99.99: 2661.64 ------------------------------------------------------ [ 0, 1 ] 2083 4.969% 4.969% # ( 1, 2 ] 13518 32.249% 37.218% ###### ( 2, 3 ] 6523 15.561% 52.779% ### ( 3, 4 ] 5407 12.899% 65.678% ### ( 4, 6 ] 7822 18.660% 84.338% #### ( 6, 10 ] 2243 5.351% 89.689% # ( 10, 15 ] 392 0.935% 90.625% ( 15, 22 ] 254 0.606% 91.230% ( 22, 34 ] 31 0.074% 91.304% ( 34, 51 ] 5 0.012% 91.316% ( 76, 110 ] 708 1.689% 93.005% ( 110, 170 ] 2045 4.879% 97.884% # ( 170, 250 ] 528 1.260% 99.144% ( 250, 380 ] 167 0.398% 99.542% ( 380, 580 ] 136 0.324% 99.866% ( 580, 870 ] 34 0.081% 99.948% ( 870, 1300 ] 8 0.019% 99.967% ( 1300, 1900 ] 6 0.014% 99.981% ( 1900, 2900 ] 5 0.012% 99.993% ( 2900, 4400 ] 2 0.005% 99.998% ( 4400, 6600 ] 1 0.002% 100.000% ** Level 2 read latency histogram (micros): Count: 43 Average: 176.0000 StdDev: 101.86 Min: 92 Median: 147.2000 Max: 583 Percentiles: P50: 147.20 P75: 186.67 P99: 583.00 P99.9: 583.00 P99.99: 583.00 ------------------------------------------------------ ( 76, 110 ] 6 13.953% 13.953% ### ( 110, 170 ] 25 58.140% 72.093% ############ ( 170, 250 ] 6 13.953% 86.047% ### ( 250, 380 ] 4 9.302% 95.349% ## ( 380, 580 ] 1 2.326% 97.674% ( 580, 870 ] 1 2.326% 100.000% ** Level 3 read latency histogram (micros): Count: 3593 Average: 176.5057 StdDev: 188.51 Min: 57 Median: 143.8662 Max: 4610 Percentiles: P50: 143.87 P75: 171.79 P99: 798.23 P99.9: 3427.63 P99.99: 4610.00 ------------------------------------------------------ ( 51, 76 ] 2 0.056% 0.056% ( 76, 110 ] 647 18.007% 18.063% #### ( 110, 170 ] 2033 56.582% 74.645% ########### ( 170, 250 ] 569 15.836% 90.481% ### ( 250, 380 ] 168 4.676% 95.157% # ( 380, 580 ] 117 3.256% 98.414% # ( 580, 870 ] 28 0.779% 99.193% ( 870, 1300 ] 12 0.334% 99.527% ( 1300, 1900 ] 12 0.334% 99.861% ( 2900, 4400 ] 4 0.111% 99.972% ( 4400, 6600 ] 1 0.028% 100.000% ** Level 4 read latency histogram (micros): Count: 30605 Average: 177.4818 StdDev: 177.06 Min: 48 Median: 143.6114 Max: 5686 Percentiles: P50: 143.61 P75: 170.77 P99: 828.65 P99.9: 2555.46 P99.99: 4240.92 ------------------------------------------------------ ( 34, 51 ] 1 0.003% 0.003% ( 51, 76 ] 6 0.020% 0.023% ( 76, 110 ] 5607 18.321% 18.343% #### ( 110, 170 ] 17296 56.514% 74.857% ########### ( 170, 250 ] 4647 15.184% 90.041% ### ( 250, 380 ] 1442 4.712% 94.752% # ( 380, 580 ] 1024 3.346% 98.098% # ( 580, 870 ] 323 1.055% 99.154% ( 870, 1300 ] 116 0.379% 99.533% ( 1300, 1900 ] 78 0.255% 99.788% ( 1900, 2900 ] 54 0.176% 99.964% ( 2900, 4400 ] 10 0.033% 99.997% ( 4400, 6600 ] 3 0.010% 100.007% ** Level 5 read latency histogram (micros): Count: 254521 Average: 180.5094 StdDev: 206.36 Min: 68 Median: 143.6817 Max: 6553 Percentiles: P50: 143.68 P75: 169.90 P99: 843.23 P99.9: 3207.27 P99.99: 4333.84 ------------------------------------------------------ ( 51, 76 ] 35 0.014% 0.014% ( 76, 110 ] 45485 17.871% 17.885% #### ( 110, 170 ] 145611 57.210% 75.094% ########### ( 170, 250 ] 38410 15.091% 90.185% ### ( 250, 380 ] 11469 4.506% 94.692% # ( 380, 580 ] 8751 3.438% 98.130% # ( 580, 870 ] 2440 0.959% 99.088% ( 870, 1300 ] 748 0.294% 99.382% ( 1300, 1900 ] 593 0.233% 99.615% ( 1900, 2900 ] 662 0.260% 99.875% ( 2900, 4400 ] 305 0.120% 99.995% ( 4400, 6600 ] 25 0.010% 100.005% ** DB Stats ** Uptime(secs): 18.7 total, 12.9 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 105.28 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:11.095 H:M:S, 59.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 76.11 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:9.082 H:M:S, 70.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 207.424 micros/op 19256 ops/sec 60.187 seconds 1158996 operations; 18.7 MB/s (288999 of 288999 found) Microseconds per read: Count: 1158996 Average: 207.4259 StdDev: 268.78 Min: 2 Median: 171.8676 Max: 28966 Percentiles: P50: 171.87 P75: 233.14 P99: 674.32 P99.9: 4176.07 P99.99: 9852.72 ------------------------------------------------------ ( 1, 2 ] 16 0.001% 0.001% ( 2, 3 ] 57 0.005% 0.006% ( 3, 4 ] 72 0.006% 0.013% ( 4, 6 ] 90 0.008% 0.020% ( 6, 10 ] 546 0.047% 0.067% ( 10, 15 ] 6093 0.526% 0.593% ( 15, 22 ] 3524 0.304% 0.897% ( 22, 34 ] 9377 0.809% 1.706% ( 34, 51 ] 931 0.080% 1.787% ( 51, 76 ] 84 0.007% 1.794% ( 76, 110 ] 24892 2.148% 3.942% ( 110, 170 ] 524984 45.296% 49.238% ######### ( 170, 250 ] 378333 32.643% 81.881% ####### ( 250, 380 ] 148591 12.821% 94.702% ### ( 380, 580 ] 46224 3.988% 98.690% # ( 580, 870 ] 11044 0.953% 99.643% ( 870, 1300 ] 2052 0.177% 99.820% ( 1300, 1900 ] 315 0.027% 99.847% ( 1900, 2900 ] 310 0.027% 99.874% ( 2900, 4400 ] 355 0.031% 99.905% ( 4400, 6600 ] 653 0.056% 99.961% ( 6600, 9900 ] 342 0.030% 99.990% ( 9900, 14000 ] 61 0.005% 99.996% ( 14000, 22000 ] 32 0.003% 99.998% ( 22000, 33000 ] 18 0.002% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 24778819 rocksdb.bloom.filter.full.positive COUNT : 1400412 rocksdb.bloom.filter.full.true.positive COUNT : 1158556 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 440 rocksdb.memtable.miss COUNT : 1158556 rocksdb.l0.hit COUNT : 47907 rocksdb.l1.hit COUNT : 6329 rocksdb.l2andup.hit COUNT : 1104320 rocksdb.compaction.key.drop.new COUNT : 31327 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2721547 rocksdb.number.keys.read COUNT : 1158996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2808636504 rocksdb.bytes.read COUNT : 1158996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1378 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 49699146 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 6869886365 rocksdb.compact.write.bytes COUNT : 6846163870 rocksdb.flush.write.bytes COUNT : 2774664159 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 14 rocksdb.number.superversion_acquires COUNT : 408 rocksdb.number.superversion_releases COUNT : 382 rocksdb.number.superversion_cleanups COUNT : 96 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2752082944 rocksdb.memtable.garbage.bytes.at.flush COUNT : 887808 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 13291735101 rocksdb.non.last.level.read.count COUNT : 3067768 rocksdb.block.checksum.compute.count COUNT : 3080560 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 170.195316 P95 : 393.036481 P99 : 672.371233 P100 : 28963.000000 COUNT : 1158996 SUM : 239453620 rocksdb.db.write.micros P50 : 1.476099 P95 : 4.907806 P99 : 975.347640 P100 : 2528.000000 COUNT : 2721547 SUM : 56363387 rocksdb.compaction.times.micros P50 : 1086666.666667 P95 : 2996587.000000 P99 : 2996587.000000 P100 : 2996587.000000 COUNT : 12 SUM : 17233056 rocksdb.compaction.times.cpu_micros P50 : 1086666.666667 P95 : 2960237.000000 P99 : 2960237.000000 P100 : 2960237.000000 COUNT : 12 SUM : 16986395 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 736.153846 P95 : 1240.875000 P99 : 1306.000000 P100 : 1306.000000 COUNT : 42 SUM : 31934 rocksdb.compaction.outfile.sync.micros P50 : 977.500000 P95 : 2466.666667 P99 : 2625.000000 P100 : 2625.000000 COUNT : 26 SUM : 30532 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 153.928571 P95 : 396.666667 P99 : 543.333333 P100 : 570.000000 COUNT : 55 SUM : 9470 rocksdb.table.open.io.micros P50 : 4361.914062 P95 : 8345.040000 P99 : 9800.208000 P100 : 13692.000000 COUNT : 1378 SUM : 6186686 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.540554 P95 : 162.447204 P99 : 242.929469 P100 : 14315.000000 COUNT : 1665730 SUM : 52343903 rocksdb.read.block.get.micros P50 : 137.041643 P95 : 241.917525 P99 : 513.934383 P100 : 14320.000000 COUNT : 1413459 SUM : 204314046 rocksdb.write.raw.block.micros P50 : 0.573075 P95 : 1.775123 P99 : 3.871261 P100 : 3473.000000 COUNT : 2329853 SUM : 4986711 rocksdb.numfiles.in.singlecompaction P50 : 4.000000 P95 : 24.000000 P99 : 24.000000 P100 : 24.000000 COUNT : 13 SUM : 72 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.591999 P95 : 2286.573845 P99 : 2414.000000 P100 : 2414.000000 COUNT : 198701 SUM : 49715841 rocksdb.sst.read.micros P50 : 85.292052 P95 : 208.949357 P99 : 395.030316 P100 : 6553.000000 COUNT : 3067795 SUM : 246234595 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1158996 SUM : 1158996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2721547 SUM : 2808636504 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 212000.000000 P95 : 249800.000000 P99 : 334005.000000 P100 : 334005.000000 COUNT : 42 SUM : 8359948 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 55 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0