[2024-11-19 12:00:48.164689] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 23.11.0 initialization... [2024-11-19 12:00:48.164953] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3272378 ] Set seed to 1732014048158055 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-19 12:00:48.276011] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-19 12:00:48.313467] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-19 12:00:48.313513] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 3272378' to capture a snapshot of events at runtime. [2024-11-19 12:00:48.313525] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid3272378 for offline analysis/debug. [2024-11-19 12:00:48.313549] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Tue Nov 19 12:00:52 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/19-12:00:58 ... thread 0: (1000000,1000000) ops and (174080.6,174080.6) ops/second in (5.744464,5.744464) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/18 2.77 GB 6.3 0.7 0.0 0.7 1.6 0.9 0.0 1.9 126.1 273.2 5.99 5.16 17 0.353 767K 742 0.0 0.0 L1 10/10 1.19 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 19/0 912.17 MB 0.9 0.2 0.1 0.1 0.2 0.1 0.0 1.8 164.2 163.6 1.44 1.08 1 1.442 0 0 0.0 0.0 L4 168/0 9.66 GB 1.0 1.2 0.7 0.5 1.2 0.7 0.0 1.7 157.5 156.1 7.70 5.75 6 1.283 74K 0 0.0 0.0 L5 1112/0 91.48 GB 0.9 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1328/28 105.99 GB 0.0 2.2 0.8 1.3 3.0 1.7 1.0 3.5 145.7 203.2 15.14 11.99 24 0.631 842K 742 0.0 0.0 Int 0/0 0.00 KB 0.0 2.2 0.8 1.3 3.0 1.7 1.0 3.5 145.7 203.2 15.14 11.99 24 0.631 842K 742 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 2.2 0.8 1.3 2.1 0.8 0.0 0.0 183.9 182.9 11.99 9.62 10 1.199 842K 742 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 280.6 3.14 2.38 14 0.225 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 5.9 total, 5.7 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 3.00 GB write, 520.51 MB/s write, 2.15 GB read, 373.34 MB/s read, 15.1 seconds Interval compaction: 3.00 GB write, 535.17 MB/s write, 2.15 GB read, 383.85 MB/s read, 15.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 9, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 9, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 9, total-stops: 0, interval: 9 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 364165 Average: 15.9396 StdDev: 97.41 Min: 0 Median: 1.5110 Max: 8314 Percentiles: P50: 1.51 P75: 1.97 P99: 246.21 P99.9: 975.95 P99.99: 1629.93 ------------------------------------------------------ [ 0, 1 ] 80180 22.017% 22.017% #### ( 1, 2 ] 199403 54.756% 76.774% ########### ( 2, 3 ] 34045 9.349% 86.122% ## ( 3, 4 ] 13871 3.809% 89.931% # ( 4, 6 ] 10638 2.921% 92.853% # ( 6, 10 ] 3262 0.896% 93.748% ( 10, 15 ] 481 0.132% 93.881% ( 15, 22 ] 86 0.024% 93.904% ( 22, 34 ] 22 0.006% 93.910% ( 34, 51 ] 7 0.002% 93.912% ( 51, 76 ] 8 0.002% 93.914% ( 76, 110 ] 3203 0.880% 94.794% ( 110, 170 ] 11787 3.237% 98.031% # ( 170, 250 ] 3708 1.018% 99.049% ( 250, 380 ] 1309 0.359% 99.408% ( 380, 580 ] 1131 0.311% 99.719% ( 580, 870 ] 582 0.160% 99.879% ( 870, 1300 ] 324 0.089% 99.968% ( 1300, 1900 ] 152 0.042% 100.009% ( 1900, 2900 ] 88 0.024% 100.034% ( 2900, 4400 ] 43 0.012% 100.045% ( 4400, 6600 ] 12 0.003% 100.049% ( 6600, 9900 ] 7 0.002% 100.051% ** Level 1 read latency histogram (micros): Count: 776 Average: 278.0812 StdDev: 548.31 Min: 80 Median: 151.2214 Max: 7548 Percentiles: P50: 151.22 P75: 214.72 P99: 2727.50 P99.9: 7339.20 P99.99: 7548.00 ------------------------------------------------------ ( 76, 110 ] 118 15.206% 15.206% ### ( 110, 170 ] 393 50.644% 65.851% ########## ( 170, 250 ] 127 16.366% 82.216% ### ( 250, 380 ] 43 5.541% 87.758% # ( 380, 580 ] 44 5.670% 93.428% # ( 580, 870 ] 18 2.320% 95.747% ( 870, 1300 ] 8 1.031% 96.778% ( 1300, 1900 ] 4 0.515% 97.294% ( 1900, 2900 ] 16 2.062% 99.356% ( 2900, 4400 ] 2 0.258% 99.613% ( 4400, 6600 ] 2 0.258% 99.871% ( 6600, 9900 ] 1 0.129% 100.000% ** Level 3 read latency histogram (micros): Count: 112114 Average: 7.6760 StdDev: 74.26 Min: 1 Median: 1.7363 Max: 4538 Percentiles: P50: 1.74 P75: 2.62 P99: 27.14 P99.9: 992.99 P99.99: 1649.54 ------------------------------------------------------ [ 0, 1 ] 10607 9.461% 9.461% ## ( 1, 2 ] 61730 55.060% 64.521% ########### ( 2, 3 ] 18862 16.824% 81.345% ### ( 3, 4 ] 9033 8.057% 89.402% ## ( 4, 6 ] 7631 6.806% 96.208% # ( 6, 10 ] 2518 2.246% 98.454% ( 10, 15 ] 511 0.456% 98.910% ( 15, 22 ] 91 0.081% 98.991% ( 22, 34 ] 23 0.021% 99.012% ( 34, 51 ] 14 0.012% 99.024% ( 51, 76 ] 3 0.003% 99.027% ( 76, 110 ] 77 0.069% 99.096% ( 110, 170 ] 309 0.276% 99.371% ( 170, 250 ] 154 0.137% 99.509% ( 250, 380 ] 115 0.103% 99.611% ( 380, 580 ] 157 0.140% 99.751% ( 580, 870 ] 140 0.125% 99.876% ( 870, 1300 ] 94 0.084% 99.960% ( 1300, 1900 ] 58 0.052% 100.012% ( 1900, 2900 ] 18 0.016% 100.028% ( 2900, 4400 ] 10 0.009% 100.037% ( 4400, 6600 ] 1 0.001% 100.037% ** Level 4 read latency histogram (micros): Count: 89546 Average: 29.4896 StdDev: 174.07 Min: 0 Median: 1.8362 Max: 7790 Percentiles: P50: 1.84 P75: 3.00 P99: 635.12 P99.9: 2273.20 P99.99: 3393.25 ------------------------------------------------------ [ 0, 1 ] 9052 10.109% 10.109% ## ( 1, 2 ] 42716 47.703% 57.812% ########## ( 2, 3 ] 15429 17.230% 75.042% ### ( 3, 4 ] 7055 7.879% 82.921% ## ( 4, 6 ] 5693 6.358% 89.278% # ( 6, 10 ] 2306 2.575% 91.853% # ( 10, 15 ] 542 0.605% 92.459% ( 15, 22 ] 138 0.154% 92.613% ( 22, 34 ] 46 0.051% 92.664% ( 34, 51 ] 10 0.011% 92.675% ( 51, 76 ] 3 0.003% 92.679% ( 76, 110 ] 792 0.884% 93.563% ( 110, 170 ] 2679 2.992% 96.555% # ( 170, 250 ] 991 1.107% 97.662% ( 250, 380 ] 518 0.578% 98.240% ( 380, 580 ] 599 0.669% 98.909% ( 580, 870 ] 429 0.479% 99.388% ( 870, 1300 ] 275 0.307% 99.695% ( 1300, 1900 ] 148 0.165% 99.860% ( 1900, 2900 ] 95 0.106% 99.966% ( 2900, 4400 ] 64 0.071% 100.038% ( 4400, 6600 ] 7 0.008% 100.046% ( 6600, 9900 ] 2 0.002% 100.048% ** Level 5 read latency histogram (micros): Count: 48186 Average: 314.8657 StdDev: 575.78 Min: 75 Median: 153.8398 Max: 12432 Percentiles: P50: 153.84 P75: 232.34 P99: 3508.12 P99.9: 5880.23 P99.99: 6666.51 ------------------------------------------------------ ( 51, 76 ] 3 0.006% 0.006% ( 76, 110 ] 7143 14.824% 14.830% ### ( 110, 170 ] 23195 48.136% 62.966% ########## ( 170, 250 ] 7442 15.444% 78.411% ### ( 250, 380 ] 3225 6.693% 85.104% # ( 380, 580 ] 2775 5.759% 90.862% # ( 580, 870 ] 1474 3.059% 93.921% # ( 870, 1300 ] 854 1.772% 95.694% ( 1300, 1900 ] 618 1.283% 96.976% ( 1900, 2900 ] 741 1.538% 98.514% ( 2900, 4400 ] 580 1.204% 99.718% ( 4400, 6600 ] 132 0.274% 99.992% ( 6600, 9900 ] 9 0.019% 100.010% ( 9900, 14000 ] 2 0.004% 100.015% ** DB Stats ** Uptime(secs): 5.9 total, 5.7 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 166.58 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:2.029 H:M:S, 34.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 171.25 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.029 H:M:S, 35.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/19-12:01:11 ... thread 0: (1000000,2000000) ops and (78220.9,107940.2) ops/second in (12.784309,18.528773) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/14 3.75 GB 188.4 3.3 0.0 3.3 5.2 1.8 0.0 2.8 180.7 280.8 18.82 17.45 39 0.482 3452K 6490 0.0 0.0 L1 10/10 1.19 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 19/0 912.17 MB 0.9 0.2 0.1 0.1 0.2 0.1 0.0 1.8 164.2 163.6 1.44 1.08 1 1.442 0 0 0.0 0.0 L4 168/0 9.66 GB 1.0 1.2 0.7 0.5 1.2 0.7 0.0 1.7 157.5 156.1 7.70 5.75 6 1.283 74K 0 0.0 0.0 L5 1112/0 91.48 GB 0.9 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1326/24 106.97 GB 0.0 4.7 0.8 3.9 6.6 2.6 1.0 3.6 173.5 240.4 27.96 24.28 46 0.608 3527K 6490 0.0 0.0 Int 0/0 0.00 KB 0.0 4.7 0.8 3.9 6.6 2.6 1.0 3.6 173.5 240.4 27.96 24.28 46 0.608 3527K 6490 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 4.7 0.8 3.9 4.7 0.8 0.0 0.0 223.2 222.4 21.73 19.21 16 1.358 3527K 6490 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 303.3 6.23 5.07 30 0.208 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.7 total, 18.5 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.56 GB write, 359.61 MB/s write, 4.74 GB read, 259.48 MB/s read, 28.0 seconds Interval compaction: 6.56 GB write, 362.75 MB/s write, 4.74 GB read, 261.74 MB/s read, 28.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 24, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 24, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 24, total-stops: 0, interval: 24 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1116603 Average: 21.4271 StdDev: 79.06 Min: 0 Median: 1.4362 Max: 8314 Percentiles: P50: 1.44 P75: 1.94 P99: 249.61 P99.9: 642.35 P99.99: 892.77 ------------------------------------------------------ [ 0, 1 ] 315274 28.235% 28.235% ###### ( 1, 2 ] 557143 49.896% 78.131% ########## ( 2, 3 ] 67072 6.007% 84.138% # ( 3, 4 ] 27590 2.471% 86.609% ( 4, 6 ] 17898 1.603% 88.212% ( 6, 10 ] 4842 0.434% 88.646% ( 10, 15 ] 790 0.071% 88.716% ( 15, 22 ] 135 0.012% 88.728% ( 22, 34 ] 24 0.002% 88.731% ( 34, 51 ] 8 0.001% 88.731% ( 51, 76 ] 27 0.002% 88.734% ( 76, 110 ] 20916 1.873% 90.607% ( 110, 170 ] 73780 6.608% 97.214% # ( 170, 250 ] 20035 1.794% 99.009% ( 250, 380 ] 5492 0.492% 99.501% ( 380, 580 ] 4191 0.375% 99.876% ( 580, 870 ] 1253 0.112% 99.988% ( 870, 1300 ] 403 0.036% 100.024% ( 1300, 1900 ] 174 0.016% 100.040% ( 1900, 2900 ] 116 0.010% 100.050% ( 2900, 4400 ] 65 0.006% 100.056% ( 4400, 6600 ] 12 0.001% 100.057% ( 6600, 9900 ] 7 0.001% 100.058% ** Level 1 read latency histogram (micros): Count: 3758 Average: 186.3151 StdDev: 265.74 Min: 79 Median: 144.0315 Max: 7548 Percentiles: P50: 144.03 P75: 170.77 P99: 857.94 P99.9: 3831.50 P99.99: 7548.00 ------------------------------------------------------ ( 76, 110 ] 655 17.429% 17.429% ### ( 110, 170 ] 2158 57.424% 74.854% ########### ( 170, 250 ] 572 15.221% 90.075% ### ( 250, 380 ] 157 4.178% 94.252% # ( 380, 580 ] 142 3.779% 98.031% # ( 580, 870 ] 38 1.011% 99.042% ( 870, 1300 ] 11 0.293% 99.335% ( 1300, 1900 ] 4 0.106% 99.441% ( 1900, 2900 ] 16 0.426% 99.867% ( 2900, 4400 ] 2 0.053% 99.920% ( 4400, 6600 ] 2 0.053% 99.973% ( 6600, 9900 ] 1 0.027% 100.000% ** Level 3 read latency histogram (micros): Count: 114853 Average: 11.2387 StdDev: 78.79 Min: 1 Median: 1.7583 Max: 4538 Percentiles: P50: 1.76 P75: 2.73 P99: 194.71 P99.9: 1003.64 P99.99: 1665.15 ------------------------------------------------------ [ 0, 1 ] 10608 9.236% 9.236% ## ( 1, 2 ] 61741 53.757% 62.993% ########### ( 2, 3 ] 18909 16.464% 79.456% ### ( 3, 4 ] 9060 7.888% 87.345% ## ( 4, 6 ] 7651 6.662% 94.006% # ( 6, 10 ] 2522 2.196% 96.202% ( 10, 15 ] 512 0.446% 96.648% ( 15, 22 ] 91 0.079% 96.727% ( 22, 34 ] 23 0.020% 96.747% ( 34, 51 ] 14 0.012% 96.759% ( 51, 76 ] 3 0.003% 96.762% ( 76, 110 ] 559 0.487% 97.249% ( 110, 170 ] 1836 1.599% 98.847% ( 170, 250 ] 568 0.495% 99.342% ( 250, 380 ] 213 0.185% 99.527% ( 380, 580 ] 243 0.212% 99.739% ( 580, 870 ] 155 0.135% 99.874% ( 870, 1300 ] 97 0.084% 99.958% ( 1300, 1900 ] 60 0.052% 100.010% ( 1900, 2900 ] 18 0.016% 100.026% ( 2900, 4400 ] 11 0.010% 100.036% ( 4400, 6600 ] 1 0.001% 100.037% ** Level 4 read latency histogram (micros): Count: 114096 Average: 54.8390 StdDev: 168.29 Min: 0 Median: 2.3192 Max: 7790 Percentiles: P50: 2.32 P75: 79.27 P99: 590.47 P99.9: 2044.83 P99.99: 3367.96 ------------------------------------------------------ [ 0, 1 ] 9052 7.934% 7.934% ## ( 1, 2 ] 42851 37.557% 45.491% ######## ( 2, 3 ] 16118 14.127% 59.617% ### ( 3, 4 ] 7476 6.552% 66.170% # ( 4, 6 ] 6456 5.658% 71.828% # ( 6, 10 ] 2392 2.096% 73.925% ( 10, 15 ] 561 0.492% 74.416% ( 15, 22 ] 141 0.124% 74.540% ( 22, 34 ] 46 0.040% 74.580% ( 34, 51 ] 10 0.009% 74.589% ( 51, 76 ] 4 0.004% 74.592% ( 76, 110 ] 4833 4.236% 78.828% # ( 110, 170 ] 16039 14.057% 92.886% ### ( 170, 250 ] 4352 3.814% 96.700% # ( 250, 380 ] 1408 1.234% 97.934% ( 380, 580 ] 1195 1.047% 98.982% ( 580, 870 ] 583 0.511% 99.493% ( 870, 1300 ] 297 0.260% 99.753% ( 1300, 1900 ] 154 0.135% 99.888% ( 1900, 2900 ] 96 0.084% 99.972% ( 2900, 4400 ] 66 0.058% 100.030% ( 4400, 6600 ] 7 0.006% 100.036% ( 6600, 9900 ] 2 0.002% 100.038% ** Level 5 read latency histogram (micros): Count: 243099 Average: 191.4106 StdDev: 277.60 Min: 74 Median: 143.8954 Max: 12432 Percentiles: P50: 143.90 P75: 171.19 P99: 1185.73 P99.9: 4105.83 P99.99: 6111.50 ------------------------------------------------------ ( 51, 76 ] 21 0.009% 0.009% ( 76, 110 ] 43316 17.818% 17.827% #### ( 110, 170 ] 138448 56.951% 74.778% ########### ( 170, 250 ] 36344 14.950% 89.728% ### ( 250, 380 ] 10627 4.371% 94.100% # ( 380, 580 ] 8402 3.456% 97.556% # ( 580, 870 ] 2764 1.137% 98.693% ( 870, 1300 ] 1016 0.418% 99.111% ( 1300, 1900 ] 665 0.274% 99.385% ( 1900, 2900 ] 777 0.320% 99.704% ( 2900, 4400 ] 592 0.244% 99.948% ( 4400, 6600 ] 132 0.054% 100.002% ( 6600, 9900 ] 9 0.004% 100.006% ( 9900, 14000 ] 2 0.001% 100.007% ** DB Stats ** Uptime(secs): 18.7 total, 12.8 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 105.30 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:11.044 H:M:S, 59.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 76.98 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:9.015 H:M:S, 70.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 211.095 micros/op 18932 ops/sec 60.160 seconds 1138996 operations; 18.3 MB/s (280999 of 280999 found) Microseconds per read: Count: 1138996 Average: 211.0972 StdDev: 368.33 Min: 2 Median: 174.3288 Max: 35152 Percentiles: P50: 174.33 P75: 233.35 P99: 671.02 P99.9: 4723.19 P99.99: 17921.70 ------------------------------------------------------ ( 1, 2 ] 4 0.000% 0.000% ( 2, 3 ] 29 0.003% 0.003% ( 3, 4 ] 43 0.004% 0.007% ( 4, 6 ] 94 0.008% 0.015% ( 6, 10 ] 428 0.038% 0.053% ( 10, 15 ] 4857 0.426% 0.479% ( 15, 22 ] 4471 0.393% 0.871% ( 22, 34 ] 11193 0.983% 1.854% ( 34, 51 ] 2396 0.210% 2.065% ( 51, 76 ] 194 0.017% 2.082% ( 76, 110 ] 19643 1.725% 3.806% ( 110, 170 ] 505261 44.360% 48.166% ######### ( 170, 250 ] 385972 33.887% 82.053% ####### ( 250, 380 ] 144252 12.665% 94.718% ### ( 380, 580 ] 45441 3.990% 98.708% # ( 580, 870 ] 10603 0.931% 99.639% ( 870, 1300 ] 1947 0.171% 99.810% ( 1300, 1900 ] 354 0.031% 99.841% ( 1900, 2900 ] 342 0.030% 99.871% ( 2900, 4400 ] 281 0.025% 99.895% ( 4400, 6600 ] 354 0.031% 99.927% ( 6600, 9900 ] 413 0.036% 99.963% ( 9900, 14000 ] 240 0.021% 99.984% ( 14000, 22000 ] 143 0.013% 99.996% ( 22000, 33000 ] 39 0.003% 100.000% ( 33000, 50000 ] 2 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 23110876 rocksdb.bloom.filter.full.positive COUNT : 1364015 rocksdb.bloom.filter.full.true.positive COUNT : 1138590 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 406 rocksdb.memtable.miss COUNT : 1138590 rocksdb.l0.hit COUNT : 46730 rocksdb.l1.hit COUNT : 13421 rocksdb.l2andup.hit COUNT : 1078439 rocksdb.compaction.key.drop.new COUNT : 40026 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2794361 rocksdb.number.keys.read COUNT : 1138996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2883780552 rocksdb.bytes.read COUNT : 1138996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1404 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 49006059 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 8656910784 rocksdb.compact.write.bytes COUNT : 8619619827 rocksdb.flush.write.bytes COUNT : 2840673890 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 40 rocksdb.number.superversion_acquires COUNT : 455 rocksdb.number.superversion_releases COUNT : 427 rocksdb.number.superversion_cleanups COUNT : 109 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2817571840 rocksdb.memtable.garbage.bytes.at.flush COUNT : 924672 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 14272291578 rocksdb.non.last.level.read.count COUNT : 3304060 rocksdb.block.checksum.compute.count COUNT : 3474708 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 172.600497 P95 : 391.831525 P99 : 669.485872 P100 : 35150.000000 COUNT : 1138996 SUM : 239424560 rocksdb.db.write.micros P50 : 1.484548 P95 : 5.084069 P99 : 945.808065 P100 : 4996.000000 COUNT : 2794361 SUM : 55974704 rocksdb.compaction.times.micros P50 : 1166000.000000 P95 : 3016567.000000 P99 : 3016567.000000 P100 : 3016567.000000 COUNT : 20 SUM : 27496878 rocksdb.compaction.times.cpu_micros P50 : 1051250.000000 P95 : 2989013.000000 P99 : 2989013.000000 P100 : 2989013.000000 COUNT : 20 SUM : 24889794 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 743.709677 P95 : 2750.000000 P99 : 23184.000000 P100 : 23184.000000 COUNT : 43 SUM : 61462 rocksdb.compaction.outfile.sync.micros P50 : 1249.411765 P95 : 5896.000000 P99 : 10643.000000 P100 : 10643.000000 COUNT : 52 SUM : 114066 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 145.806452 P95 : 1009.750000 P99 : 2998.000000 P100 : 2998.000000 COUNT : 67 SUM : 19920 rocksdb.table.open.io.micros P50 : 5828.865979 P95 : 12302.456140 P99 : 17116.521739 P100 : 19175.000000 COUNT : 1404 SUM : 8678643 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.588796 P95 : 156.409296 P99 : 239.296206 P100 : 18037.000000 COUNT : 2097616 SUM : 57026116 rocksdb.read.block.get.micros P50 : 138.018373 P95 : 242.850602 P99 : 523.281456 P100 : 17753.000000 COUNT : 1375664 SUM : 202685641 rocksdb.write.raw.block.micros P50 : 0.554235 P95 : 1.662840 P99 : 3.540558 P100 : 5334.000000 COUNT : 2775649 SUM : 5969954 rocksdb.numfiles.in.singlecompaction P50 : 3.475000 P95 : 5.500000 P99 : 24.000000 P100 : 24.000000 COUNT : 25 SUM : 114 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.585601 P95 : 2234.907037 P99 : 2767.032172 P100 : 4381.000000 COUNT : 209644 SUM : 49023599 rocksdb.sst.read.micros P50 : 5.251150 P95 : 204.811569 P99 : 400.875497 P100 : 12432.000000 COUNT : 3304087 SUM : 248437750 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1138996 SUM : 1138996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2794361 SUM : 2883780552 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 212051.282051 P95 : 286833.333333 P99 : 290187.000000 P100 : 290187.000000 COUNT : 43 SUM : 8741682 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 78 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0