[2024-11-28 07:39:26.028487] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 22.11.4 initialization... [2024-11-28 07:39:26.028769] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid4076022 ] Set seed to 1732775966021767 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-28 07:39:26.129950] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-28 07:39:26.170033] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-28 07:39:26.170081] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 4076022' to capture a snapshot of events at runtime. [2024-11-28 07:39:26.170096] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid4076022 for offline analysis/debug. [2024-11-28 07:39:26.170123] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Thu Nov 28 07:39:30 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/28-07:39:35 ... thread 0: (1000000,1000000) ops and (204494.5,204494.5) ops/second in (4.890108,4.890108) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/13 2.95 GB 88.1 0.7 0.0 0.7 1.6 0.9 0.0 1.9 127.4 275.9 5.94 5.26 17 0.349 767K 713 0.0 0.0 L1 9/9 1.02 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 2/0 82.63 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 19/0 986.57 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 168/0 9.55 GB 1.0 0.7 0.4 0.4 0.7 0.4 0.0 2.0 163.6 161.9 4.68 3.71 3 1.560 0 0 0.0 0.0 L5 1116/0 91.84 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.7 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1332/22 106.41 GB 0.0 1.5 0.4 1.1 2.3 1.2 0.7 2.7 143.3 225.6 10.62 8.96 20 0.531 767K 713 0.0 0.0 Int 0/0 0.00 KB 0.0 1.5 0.4 1.1 2.3 1.2 0.7 2.7 143.3 225.6 10.62 8.96 20 0.531 767K 713 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 1.5 0.4 1.1 1.5 0.4 0.0 0.0 201.0 199.9 7.57 6.54 6 1.262 767K 713 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 289.6 3.05 2.43 14 0.218 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 5.1 total, 4.9 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 2.34 GB write, 470.67 MB/s write, 1.49 GB read, 299.04 MB/s read, 10.6 seconds Interval compaction: 2.34 GB write, 489.64 MB/s write, 1.49 GB read, 311.09 MB/s read, 10.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 6, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 6, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 6, total-stops: 0, interval: 6 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 336044 Average: 13.2254 StdDev: 81.98 Min: 0 Median: 1.5353 Max: 8294 Percentiles: P50: 1.54 P75: 1.98 P99: 231.28 P99.9: 787.75 P99.99: 1252.36 ------------------------------------------------------ [ 0, 1 ] 66713 19.852% 19.852% #### ( 1, 2 ] 189251 56.317% 76.170% ########### ( 2, 3 ] 35823 10.660% 86.830% ## ( 3, 4 ] 12465 3.709% 90.539% # ( 4, 6 ] 10235 3.046% 93.585% # ( 6, 10 ] 3945 1.174% 94.759% ( 10, 15 ] 573 0.171% 94.930% ( 15, 22 ] 149 0.044% 94.974% ( 22, 34 ] 53 0.016% 94.990% ( 34, 51 ] 21 0.006% 94.996% ( 51, 76 ] 9 0.003% 94.999% ( 76, 110 ] 2340 0.696% 95.695% ( 110, 170 ] 8892 2.646% 98.341% # ( 170, 250 ] 2891 0.860% 99.201% ( 250, 380 ] 1131 0.337% 99.538% ( 380, 580 ] 891 0.265% 99.803% ( 580, 870 ] 455 0.135% 99.938% ( 870, 1300 ] 195 0.058% 99.996% ( 1300, 1900 ] 81 0.024% 100.021% ( 1900, 2900 ] 55 0.016% 100.037% ( 2900, 4400 ] 25 0.007% 100.044% ( 4400, 6600 ] 8 0.002% 100.047% ( 6600, 9900 ] 3 0.001% 100.048% ** Level 1 read latency histogram (micros): Count: 467 Average: 318.4775 StdDev: 661.99 Min: 77 Median: 148.3182 Max: 6808 Percentiles: P50: 148.32 P75: 216.56 P99: 3565.00 P99.9: 6808.00 P99.99: 6808.00 ------------------------------------------------------ ( 76, 110 ] 93 19.914% 19.914% #### ( 110, 170 ] 220 47.109% 67.024% ######### ( 170, 250 ] 64 13.704% 80.728% ### ( 250, 380 ] 22 4.711% 85.439% # ( 380, 580 ] 29 6.210% 91.649% # ( 580, 870 ] 9 1.927% 93.576% ( 870, 1300 ] 7 1.499% 95.075% ( 1300, 1900 ] 12 2.570% 97.645% # ( 1900, 2900 ] 5 1.071% 98.715% ( 2900, 4400 ] 3 0.642% 99.358% ( 4400, 6600 ] 2 0.428% 99.786% ( 6600, 9900 ] 1 0.214% 100.000% ** Level 2 read latency histogram (micros): Count: 51 Average: 278.0000 StdDev: 306.87 Min: 82 Median: 156.8000 Max: 1431 Percentiles: P50: 156.80 P75: 256.50 P99: 1431.00 P99.9: 1431.00 P99.99: 1431.00 ------------------------------------------------------ ( 76, 110 ] 6 11.765% 11.765% ## ( 110, 170 ] 25 49.020% 60.784% ########## ( 170, 250 ] 7 13.725% 74.510% ### ( 250, 380 ] 5 9.804% 84.314% ## ( 380, 580 ] 3 5.882% 90.196% # ( 580, 870 ] 1 1.961% 92.157% ( 870, 1300 ] 3 5.882% 98.039% # ( 1300, 1900 ] 1 1.961% 100.000% ** Level 3 read latency histogram (micros): Count: 632 Average: 269.7389 StdDev: 426.70 Min: 78 Median: 152.9452 Max: 5591 Percentiles: P50: 152.95 P75: 228.25 P99: 2513.33 P99.9: 5209.60 P99.99: 5591.00 ------------------------------------------------------ ( 76, 110 ] 107 16.930% 16.930% ### ( 110, 170 ] 292 46.203% 63.133% ######### ( 170, 250 ] 103 16.297% 79.430% ### ( 250, 380 ] 42 6.646% 86.076% # ( 380, 580 ] 36 5.696% 91.772% # ( 580, 870 ] 22 3.481% 95.253% # ( 870, 1300 ] 15 2.373% 97.627% ( 1300, 1900 ] 5 0.791% 98.418% ( 1900, 2900 ] 6 0.949% 99.367% ( 2900, 4400 ] 3 0.475% 99.842% ( 4400, 6600 ] 1 0.158% 100.000% ** Level 4 read latency histogram (micros): Count: 60856 Average: 31.6401 StdDev: 170.03 Min: 1 Median: 1.8847 Max: 6272 Percentiles: P50: 1.88 P75: 3.21 P99: 641.23 P99.9: 2204.53 P99.99: 2995.72 ------------------------------------------------------ [ 0, 1 ] 4903 8.057% 8.057% ## ( 1, 2 ] 28853 47.412% 55.469% ######### ( 2, 3 ] 10908 17.924% 73.393% #### ( 3, 4 ] 4593 7.547% 80.940% ## ( 4, 6 ] 4407 7.242% 88.182% # ( 6, 10 ] 1495 2.457% 90.639% ( 10, 15 ] 315 0.518% 91.156% ( 15, 22 ] 54 0.089% 91.245% ( 22, 34 ] 16 0.026% 91.271% ( 34, 51 ] 8 0.013% 91.284% ( 51, 76 ] 5 0.008% 91.293% ( 76, 110 ] 746 1.226% 92.518% ( 110, 170 ] 2164 3.556% 96.074% # ( 170, 250 ] 836 1.374% 97.448% ( 250, 380 ] 471 0.774% 98.222% ( 380, 580 ] 412 0.677% 98.899% ( 580, 870 ] 291 0.478% 99.377% ( 870, 1300 ] 175 0.288% 99.665% ( 1300, 1900 ] 120 0.197% 99.862% ( 1900, 2900 ] 76 0.125% 99.987% ( 2900, 4400 ] 30 0.049% 100.036% ( 4400, 6600 ] 7 0.012% 100.048% ** Level 5 read latency histogram (micros): Count: 40377 Average: 325.5129 StdDev: 581.47 Min: 58 Median: 154.9817 Max: 9379 Percentiles: P50: 154.98 P75: 239.39 P99: 3468.28 P99.9: 5850.72 P99.99: 7686.18 ------------------------------------------------------ ( 51, 76 ] 7 0.017% 0.017% ( 76, 110 ] 6037 14.952% 14.969% ### ( 110, 170 ] 18867 46.727% 61.696% ######### ( 170, 250 ] 6193 15.338% 77.034% ### ( 250, 380 ] 2866 7.098% 84.132% # ( 380, 580 ] 2470 6.117% 90.249% # ( 580, 870 ] 1298 3.215% 93.464% # ( 870, 1300 ] 703 1.741% 95.205% ( 1300, 1900 ] 608 1.506% 96.711% ( 1900, 2900 ] 742 1.838% 98.549% ( 2900, 4400 ] 481 1.191% 99.740% ( 4400, 6600 ] 98 0.243% 99.983% ( 6600, 9900 ] 9 0.022% 100.005% ** DB Stats ** Uptime(secs): 5.1 total, 4.9 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 193.39 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:1.268 H:M:S, 24.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 201.14 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.268 H:M:S, 25.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/11/28-07:39:45 ... thread 0: (1000000,2000000) ops and (102585.9,136630.3) ops/second in (9.747931,14.638039) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/17 3.93 GB 106.6 2.8 0.0 2.8 4.6 1.8 0.0 2.5 168.1 279.8 16.86 15.56 38 0.444 2877K 6180 0.0 0.0 L1 9/9 1.02 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 2/0 82.63 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 19/0 986.57 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 168/0 9.55 GB 1.0 0.7 0.4 0.4 0.7 0.4 0.0 2.0 163.6 161.9 4.68 3.71 3 1.560 0 0 0.0 0.0 L5 1116/0 91.84 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.7 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1332/26 107.39 GB 0.0 3.5 0.4 3.1 5.3 2.2 0.7 2.9 167.1 254.2 21.54 19.26 41 0.525 2877K 6180 0.0 0.0 Int 0/0 0.00 KB 0.0 3.5 0.4 3.1 5.3 2.2 0.7 2.9 167.1 254.2 21.54 19.26 41 0.525 2877K 6180 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 3.5 0.4 3.1 3.5 0.4 0.0 0.0 235.1 234.2 15.31 14.11 11 1.392 2877K 6180 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 303.4 6.23 5.15 30 0.208 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.8 total, 14.6 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 5.35 GB write, 369.06 MB/s write, 3.52 GB read, 242.63 MB/s read, 21.5 seconds Interval compaction: 5.35 GB write, 374.04 MB/s write, 3.52 GB read, 245.90 MB/s read, 21.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 20, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 20, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 20, total-stops: 0, interval: 20 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 967747 Average: 19.0609 StdDev: 73.08 Min: 0 Median: 1.4555 Max: 8294 Percentiles: P50: 1.46 P75: 1.94 P99: 247.68 P99.9: 601.97 P99.99: 831.80 ------------------------------------------------------ [ 0, 1 ] 254526 26.301% 26.301% ##### ( 1, 2 ] 503474 52.025% 78.326% ########## ( 2, 3 ] 67725 6.998% 85.324% # ( 3, 4 ] 24207 2.501% 87.826% # ( 4, 6 ] 17167 1.774% 89.600% ( 6, 10 ] 5578 0.576% 90.176% ( 10, 15 ] 1075 0.111% 90.287% ( 15, 22 ] 219 0.023% 90.310% ( 22, 34 ] 59 0.006% 90.316% ( 34, 51 ] 22 0.002% 90.318% ( 51, 76 ] 21 0.002% 90.320% ( 76, 110 ] 15014 1.551% 91.872% ( 110, 170 ] 53648 5.544% 97.415% # ( 170, 250 ] 15792 1.632% 99.047% ( 250, 380 ] 4641 0.480% 99.527% ( 380, 580 ] 3528 0.365% 99.891% ( 580, 870 ] 1099 0.114% 100.005% ( 870, 1300 ] 273 0.028% 100.033% ( 1300, 1900 ] 94 0.010% 100.043% ( 1900, 2900 ] 91 0.009% 100.052% ( 2900, 4400 ] 37 0.004% 100.056% ( 4400, 6600 ] 8 0.001% 100.057% ( 6600, 9900 ] 3 0.000% 100.057% ** Level 1 read latency histogram (micros): Count: 2178 Average: 199.4793 StdDev: 329.19 Min: 76 Median: 144.0260 Max: 6808 Percentiles: P50: 144.03 P75: 180.56 P99: 1480.86 P99.9: 5304.20 P99.99: 6808.00 ------------------------------------------------------ ( 51, 76 ] 1 0.046% 0.046% ( 76, 110 ] 433 19.881% 19.927% #### ( 110, 170 ] 1155 53.030% 72.957% ########### ( 170, 250 ] 337 15.473% 88.430% ### ( 250, 380 ] 102 4.683% 93.113% # ( 380, 580 ] 94 4.316% 97.429% # ( 580, 870 ] 19 0.872% 98.301% ( 870, 1300 ] 11 0.505% 98.806% ( 1300, 1900 ] 14 0.643% 99.449% ( 1900, 2900 ] 6 0.275% 99.725% ( 2900, 4400 ] 3 0.138% 99.862% ( 4400, 6600 ] 2 0.092% 99.954% ( 6600, 9900 ] 1 0.046% 100.000% ** Level 2 read latency histogram (micros): Count: 233 Average: 180.8326 StdDev: 161.79 Min: 82 Median: 145.1799 Max: 1431 Percentiles: P50: 145.18 P75: 171.71 P99: 1109.37 P99.9: 1431.00 P99.99: 1431.00 ------------------------------------------------------ ( 76, 110 ] 35 15.021% 15.021% ### ( 110, 170 ] 139 59.657% 74.678% ############ ( 170, 250 ] 35 15.021% 89.700% ### ( 250, 380 ] 12 5.150% 94.850% # ( 380, 580 ] 7 3.004% 97.854% # ( 580, 870 ] 1 0.429% 98.283% ( 870, 1300 ] 3 1.288% 99.571% ( 1300, 1900 ] 1 0.429% 100.000% ** Level 3 read latency histogram (micros): Count: 2871 Average: 191.2522 StdDev: 222.88 Min: 78 Median: 147.0644 Max: 5591 Percentiles: P50: 147.06 P75: 188.29 P99: 1012.35 P99.9: 3464.50 P99.99: 5591.00 ------------------------------------------------------ ( 76, 110 ] 457 15.918% 15.918% ### ( 110, 170 ] 1584 55.172% 71.090% ########### ( 170, 250 ] 491 17.102% 88.192% ### ( 250, 380 ] 150 5.225% 93.417% # ( 380, 580 ] 113 3.936% 97.353% # ( 580, 870 ] 41 1.428% 98.781% ( 870, 1300 ] 19 0.662% 99.443% ( 1300, 1900 ] 6 0.209% 99.652% ( 1900, 2900 ] 6 0.209% 99.861% ( 2900, 4400 ] 3 0.104% 99.965% ( 4400, 6600 ] 1 0.035% 100.000% ** Level 4 read latency histogram (micros): Count: 78684 Average: 59.4866 StdDev: 166.36 Min: 1 Median: 2.4866 Max: 6272 Percentiles: P50: 2.49 P75: 96.11 P99: 614.03 P99.9: 1992.61 P99.99: 2889.01 ------------------------------------------------------ [ 0, 1 ] 4903 6.231% 6.231% # ( 1, 2 ] 28952 36.795% 43.027% ####### ( 2, 3 ] 11277 14.332% 57.359% ### ( 3, 4 ] 4826 6.133% 63.492% # ( 4, 6 ] 4927 6.262% 69.754% # ( 6, 10 ] 1549 1.969% 71.722% ( 10, 15 ] 337 0.428% 72.151% ( 15, 22 ] 54 0.069% 72.219% ( 22, 34 ] 16 0.020% 72.240% ( 34, 51 ] 8 0.010% 72.250% ( 51, 76 ] 6 0.008% 72.257% ( 76, 110 ] 3648 4.636% 76.894% # ( 110, 170 ] 11680 14.844% 91.738% ### ( 170, 250 ] 3493 4.439% 96.177% # ( 250, 380 ] 1173 1.491% 97.668% ( 380, 580 ] 997 1.267% 98.935% ( 580, 870 ] 436 0.554% 99.489% ( 870, 1300 ] 190 0.241% 99.731% ( 1300, 1900 ] 126 0.160% 99.891% ( 1900, 2900 ] 79 0.100% 99.991% ( 2900, 4400 ] 31 0.039% 100.031% ( 4400, 6600 ] 7 0.009% 100.039% ** Level 5 read latency histogram (micros): Count: 185383 Average: 200.7105 StdDev: 293.93 Min: 58 Median: 145.9816 Max: 9379 Percentiles: P50: 145.98 P75: 184.31 P99: 1447.10 P99.9: 4137.17 P99.99: 6206.28 ------------------------------------------------------ ( 51, 76 ] 18 0.010% 0.010% ( 76, 110 ] 31190 16.825% 16.834% ### ( 110, 170 ] 102525 55.304% 72.139% ########### ( 170, 250 ] 29658 15.998% 88.137% ### ( 250, 380 ] 9403 5.072% 93.209% # ( 380, 580 ] 7298 3.937% 97.146% # ( 580, 870 ] 2460 1.327% 98.473% ( 870, 1300 ] 821 0.443% 98.916% ( 1300, 1900 ] 637 0.344% 99.259% ( 1900, 2900 ] 781 0.421% 99.681% ( 2900, 4400 ] 493 0.266% 99.947% ( 4400, 6600 ] 98 0.053% 99.999% ( 6600, 9900 ] 9 0.005% 100.004% ** DB Stats ** Uptime(secs): 14.8 total, 9.7 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 132.67 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.432 H:M:S, 50.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 100.96 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:6.164 H:M:S, 63.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 202.233 micros/op 19759 ops/sec 60.173 seconds 1188996 operations; 19.1 MB/s (293999 of 293999 found) Microseconds per read: Count: 1188996 Average: 202.2338 StdDev: 321.83 Min: 2 Median: 168.1809 Max: 34087 Percentiles: P50: 168.18 P75: 227.51 P99: 588.82 P99.9: 4405.09 P99.99: 14008.20 ------------------------------------------------------ ( 1, 2 ] 17 0.001% 0.001% ( 2, 3 ] 56 0.005% 0.006% ( 3, 4 ] 60 0.005% 0.011% ( 4, 6 ] 86 0.007% 0.018% ( 6, 10 ] 778 0.065% 0.084% ( 10, 15 ] 6188 0.520% 0.604% ( 15, 22 ] 2902 0.244% 0.848% ( 22, 34 ] 11866 0.998% 1.846% ( 34, 51 ] 1649 0.139% 1.985% ( 51, 76 ] 166 0.014% 1.999% ( 76, 110 ] 26162 2.200% 4.199% ( 110, 170 ] 561595 47.233% 51.432% ######### ( 170, 250 ] 389784 32.783% 84.215% ####### ( 250, 380 ] 137680 11.580% 95.794% ## ( 380, 580 ] 37858 3.184% 98.978% # ( 580, 870 ] 8515 0.716% 99.694% ( 870, 1300 ] 1549 0.130% 99.825% ( 1300, 1900 ] 310 0.026% 99.851% ( 1900, 2900 ] 317 0.027% 99.877% ( 2900, 4400 ] 268 0.023% 99.900% ( 4400, 6600 ] 434 0.037% 99.936% ( 6600, 9900 ] 441 0.037% 99.974% ( 9900, 14000 ] 196 0.016% 99.990% ( 14000, 22000 ] 98 0.008% 99.998% ( 22000, 33000 ] 20 0.002% 100.000% ( 33000, 50000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 21749176 rocksdb.bloom.filter.full.positive COUNT : 1400102 rocksdb.bloom.filter.full.true.positive COUNT : 1188615 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 381 rocksdb.memtable.miss COUNT : 1188615 rocksdb.l0.hit COUNT : 48646 rocksdb.l1.hit COUNT : 11925 rocksdb.l2andup.hit COUNT : 1128044 rocksdb.compaction.key.drop.new COUNT : 34318 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2260609 rocksdb.number.keys.read COUNT : 1188996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2332948488 rocksdb.bytes.read COUNT : 1188996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1384 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 51488840 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 6771122943 rocksdb.compact.write.bytes COUNT : 6740927370 rocksdb.flush.write.bytes COUNT : 2312171282 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 28 rocksdb.number.superversion_acquires COUNT : 356 rocksdb.number.superversion_releases COUNT : 339 rocksdb.number.superversion_cleanups COUNT : 86 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2293348352 rocksdb.memtable.garbage.bytes.at.flush COUNT : 734208 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 12670744594 rocksdb.non.last.level.read.count COUNT : 2916646 rocksdb.block.checksum.compute.count COUNT : 3054423 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 167.321512 P95 : 370.563161 P99 : 586.685768 P100 : 34084.000000 COUNT : 1188996 SUM : 239445912 rocksdb.db.write.micros P50 : 1.540107 P95 : 5.075576 P99 : 1150.275502 P100 : 2761.000000 COUNT : 2260609 SUM : 57170139 rocksdb.compaction.times.micros P50 : 1136250.000000 P95 : 3845000.000000 P99 : 3923824.000000 P100 : 3923824.000000 COUNT : 13 SUM : 19255990 rocksdb.compaction.times.cpu_micros P50 : 1105555.555556 P95 : 3845000.000000 P99 : 3883115.000000 P100 : 3883115.000000 COUNT : 13 SUM : 17987982 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 786.052632 P95 : 1550.000000 P99 : 1736.000000 P100 : 1736.000000 COUNT : 35 SUM : 29695 rocksdb.compaction.outfile.sync.micros P50 : 1085.000000 P95 : 6765.000000 P99 : 8076.000000 P100 : 8076.000000 COUNT : 38 SUM : 69885 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 154.444444 P95 : 812.000000 P99 : 1588.000000 P100 : 1864.000000 COUNT : 52 SUM : 12860 rocksdb.table.open.io.micros P50 : 5435.781544 P95 : 10639.154930 P99 : 13836.000000 P100 : 17141.000000 COUNT : 1384 SUM : 7892362 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.635887 P95 : 162.250122 P99 : 240.197438 P100 : 4280.000000 COUNT : 1641046 SUM : 55534367 rocksdb.read.block.get.micros P50 : 137.473742 P95 : 237.242836 P99 : 491.164381 P100 : 26966.000000 COUNT : 1411973 SUM : 204298529 rocksdb.write.raw.block.micros P50 : 0.554220 P95 : 1.653924 P99 : 4.146197 P100 : 4703.000000 COUNT : 2192478 SUM : 4752161 rocksdb.numfiles.in.singlecompaction P50 : 3.535714 P95 : 16.050000 P99 : 20.810000 P100 : 22.000000 COUNT : 17 SUM : 84 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.591726 P95 : 2376.408242 P99 : 2683.000000 P100 : 2683.000000 COUNT : 194640 SUM : 51504670 rocksdb.sst.read.micros P50 : 95.210075 P95 : 206.023621 P99 : 376.126606 P100 : 9379.000000 COUNT : 2916658 SUM : 249320141 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1188996 SUM : 1188996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2260609 SUM : 2332948488 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 211250.000000 P95 : 266250.000000 P99 : 272268.000000 P100 : 272268.000000 COUNT : 35 SUM : 7268010 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 53 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0