00:00:00.000  Started by upstream project "nightly-trigger" build number 3087
00:00:00.000  originally caused by:
00:00:00.000   Started by timer
00:00:00.007  Checking out git https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool into /var/jenkins_home/workspace/autotest-spdk-master-vs-dpdk-v23.11_script/33b20b30f0a51e6b52980845e0f6aa336787973ad45e341fbbf98d1b65b265d4 to read jbp/jenkins/jjb-config/jobs/autotest-upstream/autotest.groovy
00:00:00.008  The recommended git tool is: git
00:00:00.008  using credential 00000000-0000-0000-0000-000000000002
00:00:00.010   > git rev-parse --resolve-git-dir /var/jenkins_home/workspace/autotest-spdk-master-vs-dpdk-v23.11_script/33b20b30f0a51e6b52980845e0f6aa336787973ad45e341fbbf98d1b65b265d4/jbp/.git # timeout=10
00:00:00.022  Fetching changes from the remote Git repository
00:00:00.024   > git config remote.origin.url https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool # timeout=10
00:00:00.035  Using shallow fetch with depth 1
00:00:00.035  Fetching upstream changes from https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool
00:00:00.035   > git --version # timeout=10
00:00:00.044   > git --version # 'git version 2.39.2'
00:00:00.044  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:00.045  Setting http proxy: proxy-dmz.intel.com:911
00:00:00.045   > git fetch --tags --force --progress --depth=1 -- https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool refs/heads/master # timeout=5
00:00:02.187   > git rev-parse origin/FETCH_HEAD^{commit} # timeout=10
00:00:02.199   > git rev-parse FETCH_HEAD^{commit} # timeout=10
00:00:02.209  Checking out Revision 10da8f6d99838e411e4e94523ded0bfebf3e7100 (FETCH_HEAD)
00:00:02.209   > git config core.sparsecheckout # timeout=10
00:00:02.218   > git read-tree -mu HEAD # timeout=10
00:00:02.231   > git checkout -f 10da8f6d99838e411e4e94523ded0bfebf3e7100 # timeout=5
00:00:02.251  Commit message: "scripts/create_git_mirror: Update path to xnvme submodule"
00:00:02.251   > git rev-list --no-walk 10da8f6d99838e411e4e94523ded0bfebf3e7100 # timeout=10
00:00:02.392  [Pipeline] Start of Pipeline
00:00:02.402  [Pipeline] library
00:00:02.404  Loading library shm_lib@master
00:00:02.404  Library shm_lib@master is cached. Copying from home.
00:00:02.424  [Pipeline] node
00:00:02.433  Running on ME3 in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11
00:00:02.435  [Pipeline] {
00:00:02.450  [Pipeline] cleanWs
00:00:02.460  [WS-CLEANUP] Deleting project workspace...
00:00:02.460  [WS-CLEANUP] Deferred wipeout is used...
00:00:02.467  [WS-CLEANUP] done
00:00:02.474  [Pipeline] stage
00:00:02.479  [Pipeline] { (Prologue)
00:00:02.591  [Pipeline] withCredentials
00:00:02.599   > git --version # timeout=10
00:00:02.609   > git --version # 'git version 2.39.2'
00:00:02.624  Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD or $GIT_ASKPASS
00:00:02.626  [Pipeline] {
00:00:02.631  [Pipeline] retry
00:00:02.632  [Pipeline] {
00:00:02.780  [Pipeline] sh
00:00:03.057  + git ls-remote https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool refs/heads/master
00:00:04.973  [Pipeline] }
00:00:04.999  [Pipeline] // retry
00:00:05.005  [Pipeline] }
00:00:05.030  [Pipeline] // withCredentials
00:00:05.039  [Pipeline] nodesByLabel
00:00:05.040  Found a total of 1 nodes with the 'sorcerer' label
00:00:05.051  [Pipeline] httpRequest
00:00:05.056  HttpMethod: GET
00:00:05.056  URL: http://10.211.164.101/packages/jbp_10da8f6d99838e411e4e94523ded0bfebf3e7100.tar.gz
00:00:05.057  Sending request to url: http://10.211.164.101/packages/jbp_10da8f6d99838e411e4e94523ded0bfebf3e7100.tar.gz
00:00:05.059  Response Code: HTTP/1.1 200 OK
00:00:05.059  Success: Status code 200 is in the accepted range: 200,404
00:00:05.060  Saving response body to /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/jbp_10da8f6d99838e411e4e94523ded0bfebf3e7100.tar.gz
00:00:05.199  [Pipeline] sh
00:00:05.477  + tar --no-same-owner -xf jbp_10da8f6d99838e411e4e94523ded0bfebf3e7100.tar.gz
00:00:05.497  [Pipeline] withCredentials
00:00:05.508   > git --version # timeout=10
00:00:05.521   > git --version # 'git version 2.39.2'
00:00:05.538  Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD or $GIT_ASKPASS
00:00:05.541  [Pipeline] {
00:00:05.549  [Pipeline] retry
00:00:05.551  [Pipeline] {
00:00:05.567  [Pipeline] sh
00:00:05.847  + git ls-remote https://review.spdk.io/gerrit/a/spdk/spdk refs/heads/master
00:00:09.148  [Pipeline] }
00:00:09.172  [Pipeline] // retry
00:00:09.179  [Pipeline] }
00:00:09.204  [Pipeline] // withCredentials
00:00:09.218  [Pipeline] httpRequest
00:00:09.222  HttpMethod: GET
00:00:09.223  URL: http://10.211.164.101/packages/spdk_4506c0c368f63ba9b9b013ecff216cef6ee8d0a4.tar.gz
00:00:09.224  Sending request to url: http://10.211.164.101/packages/spdk_4506c0c368f63ba9b9b013ecff216cef6ee8d0a4.tar.gz
00:00:09.225  Response Code: HTTP/1.1 200 OK
00:00:09.225  Success: Status code 200 is in the accepted range: 200,404
00:00:09.226  Saving response body to /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk_4506c0c368f63ba9b9b013ecff216cef6ee8d0a4.tar.gz
00:00:11.348  [Pipeline] sh
00:00:11.625  + tar --no-same-owner -xf spdk_4506c0c368f63ba9b9b013ecff216cef6ee8d0a4.tar.gz
00:00:13.533  [Pipeline] sh
00:00:13.813  + git -C spdk log --oneline -n5
00:00:13.814  4506c0c36 test/common: Enable inherit_errexit
00:00:13.814  b24df7cfa test: Drop superfluous calls to print_backtrace()
00:00:13.814  7b52e4c17 test/scheduler: Meassure utime of $spdk_pid threads as a fallback
00:00:13.814  1dc065205 test/scheduler: Calculate median of the cpu load samples
00:00:13.814  b22f1b34d test/scheduler: Enhance lookup of the $old_cgroup in move_proc()
00:00:13.830  [Pipeline] sh
00:00:14.114  + git -C spdk show -s --format=%s 4506c0c368f63ba9b9b013ecff216cef6ee8d0a4
00:00:14.131  [Pipeline] setCustomBuildProperty
00:00:14.140  [Pipeline] setCustomBuildProperty
00:00:14.160  [Pipeline] withCredentials
00:00:14.171   > git --version # timeout=10
00:00:14.184   > git --version # 'git version 2.39.2'
00:00:14.203  Masking supported pattern matches of $GIT_PASSWORD or $GIT_ASKPASS
00:00:14.205  [Pipeline] {
00:00:14.215  [Pipeline] retry
00:00:14.217  [Pipeline] {
00:00:14.234  [Pipeline] sh
00:00:14.519  + git ls-remote http://dpdk.org/git/dpdk-stable v23.11
00:00:15.472  [Pipeline] }
00:00:15.493  [Pipeline] // retry
00:00:15.498  [Pipeline] }
00:00:15.520  [Pipeline] // withCredentials
00:00:15.534  [Pipeline] httpRequest
00:00:15.539  HttpMethod: GET
00:00:15.540  URL: http://10.211.164.101/packages/dpdk_d15625009dced269fcec27fc81dd74fd58d54cdb.tar.gz
00:00:15.540  Sending request to url: http://10.211.164.101/packages/dpdk_d15625009dced269fcec27fc81dd74fd58d54cdb.tar.gz
00:00:15.542  Response Code: HTTP/1.1 200 OK
00:00:15.543  Success: Status code 200 is in the accepted range: 200,404
00:00:15.543  Saving response body to /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dpdk_d15625009dced269fcec27fc81dd74fd58d54cdb.tar.gz
00:00:16.772  [Pipeline] sh
00:00:17.050  + tar --no-same-owner -xf dpdk_d15625009dced269fcec27fc81dd74fd58d54cdb.tar.gz
00:00:18.433  [Pipeline] sh
00:00:18.719  + git -C dpdk log --oneline -n5
00:00:18.719  eeb0605f11 version: 23.11.0
00:00:18.719  238778122a doc: update release notes for 23.11
00:00:18.719  46aa6b3cfc doc: fix description of RSS features
00:00:18.719  dd88f51a57 devtools: forbid DPDK API in cnxk base driver
00:00:18.719  7e421ae345 devtools: support skipping forbid rule check
00:00:18.732  [Pipeline] catchError
00:00:18.734  [Pipeline] {
00:00:18.757  [Pipeline] readJSON
00:00:18.773  [Pipeline] }
00:00:18.801  [Pipeline] // catchError
00:00:18.806  [Pipeline] }
00:00:18.835  [Pipeline] // stage
00:00:18.853  [Pipeline] catchError
00:00:18.855  [Pipeline] {
00:00:18.876  [Pipeline] stage
00:00:18.878  [Pipeline] { (Tests)
00:00:19.089  [Pipeline] parallel
00:00:19.182  [Pipeline] { (Branch: unittest-vg-autotest)
00:00:19.183  [Pipeline] { (Branch: valgrind-vg-autotest)
00:00:19.185  [Pipeline] { (Branch: autobuild-vg-autotest)
00:00:19.186  [Pipeline] { (Branch: scanbuild-vg-autotest)
00:00:19.188  [Pipeline] { (Branch: lvol-vg-autotest)
00:00:19.189  [Pipeline] { (Branch: vhost-initiator-vg-autotest)
00:00:19.191  [Pipeline] { (Branch: nvmf-tcp-vg-autotest)
00:00:19.192  [Pipeline] { (Branch: nvmf-tcp-uring-vg-autotest)
00:00:19.194  [Pipeline] { (Branch: nvme-vg-autotest)
00:00:19.195  [Pipeline] { (Branch: centos7-vg-autotest)
00:00:19.196  [Pipeline] { (Branch: rocky9-vg-autotest)
00:00:19.197  [Pipeline] { (Branch: ubuntu20-vg-autotest)
00:00:19.199  [Pipeline] { (Branch: ubuntu22-vg-autotest)
00:00:19.200  [Pipeline] { (Branch: iscsi-vg-autotest)
00:00:19.202  [Pipeline] { (Branch: iscsi-uring-vg-autotest)
00:00:19.203  [Pipeline] { (Branch: zns-vg-autotest)
00:00:19.204  [Pipeline] { (Branch: clang-vg-autotest)
00:00:19.205  [Pipeline] { (Branch: nvme-cmb-pmr-vg-autotest)
00:00:19.207  [Pipeline] { (Branch: BlobFS-phy-autotest)
00:00:19.208  [Pipeline] { (Branch: crypto-phy-autotest)
00:00:19.210  [Pipeline] { (Branch: vhost-phy-autotest)
00:00:19.211  [Pipeline] { (Branch: nvmf-phy-autotest)
00:00:19.212  [Pipeline] { (Branch: nvmf-tcp-phy-autotest)
00:00:19.214  [Pipeline] { (Branch: nvme-phy-autotest)
00:00:19.215  [Pipeline] { (Branch: vfio-user-phy-autotest)
00:00:19.216  [Pipeline] { (Branch: short-fuzz-phy-autotest)
00:00:19.217  [Pipeline] { (Branch: dsa-phy-autotest)
00:00:19.218  [Pipeline] { (Branch: vmd-phy-autotest)
00:00:19.248  [Pipeline] retry
00:00:19.251  [Pipeline] {
00:00:19.257  [Pipeline] retry
00:00:19.259  [Pipeline] {
00:00:19.266  [Pipeline] retry
00:00:19.268  [Pipeline] {
00:00:19.274  [Pipeline] retry
00:00:19.275  [Pipeline] {
00:00:19.281  [Pipeline] retry
00:00:19.283  [Pipeline] {
00:00:19.289  [Pipeline] retry
00:00:19.291  [Pipeline] {
00:00:19.297  [Pipeline] retry
00:00:19.299  [Pipeline] {
00:00:19.304  [Pipeline] retry
00:00:19.305  [Pipeline] {
00:00:19.310  [Pipeline] retry
00:00:19.312  [Pipeline] {
00:00:19.317  [Pipeline] retry
00:00:19.318  [Pipeline] {
00:00:19.323  [Pipeline] retry
00:00:19.324  [Pipeline] {
00:00:19.329  [Pipeline] retry
00:00:19.331  [Pipeline] {
00:00:19.336  [Pipeline] retry
00:00:19.338  [Pipeline] {
00:00:19.343  [Pipeline] retry
00:00:19.345  [Pipeline] {
00:00:19.350  [Pipeline] retry
00:00:19.351  [Pipeline] {
00:00:19.356  [Pipeline] retry
00:00:19.358  [Pipeline] {
00:00:19.363  [Pipeline] retry
00:00:19.365  [Pipeline] {
00:00:19.369  [Pipeline] retry
00:00:19.371  [Pipeline] {
00:00:19.376  [Pipeline] retry
00:00:19.378  [Pipeline] {
00:00:19.383  [Pipeline] retry
00:00:19.385  [Pipeline] {
00:00:19.391  [Pipeline] retry
00:00:19.393  [Pipeline] {
00:00:19.398  [Pipeline] retry
00:00:19.400  [Pipeline] {
00:00:19.404  [Pipeline] retry
00:00:19.406  [Pipeline] {
00:00:19.410  [Pipeline] retry
00:00:19.412  [Pipeline] {
00:00:19.417  [Pipeline] retry
00:00:19.418  [Pipeline] {
00:00:19.423  [Pipeline] retry
00:00:19.424  [Pipeline] {
00:00:19.428  [Pipeline] retry
00:00:19.429  [Pipeline] {
00:00:19.432  [Pipeline] retry
00:00:19.433  [Pipeline] {
00:00:19.454  [Pipeline] build
00:00:19.456  Scheduling project: unittest-vg-autotest
00:00:19.461  [Pipeline] build
00:00:19.463  Scheduling project: valgrind-vg-autotest
00:00:19.470  [Pipeline] build
00:00:19.472  Scheduling project: autobuild-vg-autotest
00:00:19.481  [Pipeline] build
00:00:19.483  Scheduling project: scanbuild-vg-autotest
00:00:19.489  [Pipeline] build
00:00:19.491  Scheduling project: lvol-vg-autotest
00:00:19.498  [Pipeline] build
00:00:19.499  Scheduling project: vhost-initiator-vg-autotest
00:00:19.507  [Pipeline] build
00:00:19.509  Scheduling project: nvmf-tcp-vg-autotest
00:00:19.516  [Pipeline] build
00:00:19.519  Scheduling project: nvmf-tcp-uring-vg-autotest
00:00:19.524  [Pipeline] build
00:00:19.526  Scheduling project: nvme-vg-autotest
00:00:19.531  [Pipeline] build
00:00:19.533  Scheduling project: centos7-vg-autotest
00:00:19.539  [Pipeline] build
00:00:19.541  Scheduling project: rocky9-vg-autotest
00:00:19.546  [Pipeline] build
00:00:19.548  Scheduling project: ubuntu20-vg-autotest
00:00:19.553  [Pipeline] build
00:00:19.555  Scheduling project: ubuntu22-vg-autotest
00:00:19.560  [Pipeline] build
00:00:19.562  Scheduling project: iscsi-vg-autotest
00:00:19.567  [Pipeline] build
00:00:19.569  Scheduling project: iscsi-uring-vg-autotest
00:00:19.576  [Pipeline] build
00:00:19.578  Scheduling project: zns-vg-autotest
00:00:19.585  [Pipeline] build
00:00:19.586  Scheduling project: clang-vg-autotest
00:00:19.592  [Pipeline] build
00:00:19.594  Scheduling project: nvme-cmb-pmr-vg-autotest
00:00:19.599  [Pipeline] build
00:00:19.601  Scheduling project: BlobFS-phy-autotest
00:00:19.607  [Pipeline] build
00:00:19.609  Scheduling project: crypto-phy-autotest
00:00:19.613  [Pipeline] build
00:00:19.615  Scheduling project: vhost-phy-autotest
00:00:19.621  [Pipeline] build
00:00:19.622  Scheduling project: nvmf-phy-autotest
00:00:19.628  [Pipeline] build
00:00:19.631  Scheduling project: nvmf-tcp-phy-autotest
00:00:19.636  [Pipeline] build
00:00:19.638  Scheduling project: nvme-phy-autotest
00:00:19.643  [Pipeline] build
00:00:19.644  Scheduling project: vfio-user-phy-autotest
00:00:19.649  [Pipeline] build
00:00:19.651  Scheduling project: short-fuzz-phy-autotest
00:00:19.656  [Pipeline] build
00:00:19.658  Scheduling project: dsa-phy-autotest
00:00:19.663  [Pipeline] build
00:00:19.665  Scheduling project: vmd-phy-autotest
00:00:25.125  Starting building: dsa-phy-autotest #4898
00:00:25.130  Starting building: BlobFS-phy-autotest #26204
00:00:25.133  Starting building: short-fuzz-phy-autotest #26195
00:00:25.137  Starting building: vhost-phy-autotest #26223
00:00:25.141  Starting building: vmd-phy-autotest #2620
00:00:25.144  Starting building: valgrind-vg-autotest #49192
00:00:25.147  Starting building: iscsi-uring-vg-autotest #77383
00:00:25.150  Starting building: lvol-vg-autotest #77427
00:00:25.155  Starting building: nvme-phy-autotest #72370
00:00:25.158  Starting building: vfio-user-phy-autotest #26206
00:00:25.161  Starting building: crypto-phy-autotest #26198
00:00:25.164  Starting building: zns-vg-autotest #76584
00:00:25.168  Starting building: autobuild-vg-autotest #63418
00:00:25.172  Starting building: rocky9-vg-autotest #16088
00:00:25.177  Starting building: nvmf-phy-autotest #72442
00:00:25.181  Starting building: clang-vg-autotest #57626
00:00:25.184  Starting building: nvmf-tcp-phy-autotest #72329
00:00:25.187  Starting building: vhost-initiator-vg-autotest #77498
00:00:25.195  Starting building: nvmf-tcp-uring-vg-autotest #77403
00:00:25.203  Starting building: ubuntu20-vg-autotest #73797
00:00:25.210  Starting building: ubuntu22-vg-autotest #37811
00:00:25.215  Starting building: nvme-cmb-pmr-vg-autotest #78601
00:00:25.219  Starting building: unittest-vg-autotest #63375
00:00:25.223  Starting building: nvmf-tcp-vg-autotest #77469
00:00:25.227  Starting building: scanbuild-vg-autotest #63377
00:00:25.229  Starting building: nvme-vg-autotest #77478
00:00:25.230  Starting building: iscsi-vg-autotest #77516
00:00:25.232  Starting building: centos7-vg-autotest #73657
00:08:15.218  Build vmd-phy-autotest #2620 completed: SUCCESS
00:08:15.221  [Pipeline] }
00:08:15.262  [Pipeline] // retry
00:08:15.269  [Pipeline] }
00:08:32.512  Build short-fuzz-phy-autotest #26195 completed: SUCCESS
00:08:32.514  [Pipeline] }
00:08:32.565  [Pipeline] // retry
00:08:32.572  [Pipeline] }
00:10:53.706  Build dsa-phy-autotest #4898 completed: SUCCESS
00:10:53.709  [Pipeline] }
00:10:53.758  [Pipeline] // retry
00:10:53.763  [Pipeline] }
00:17:39.399  Build nvme-phy-autotest #72370 completed: FAILURE
00:17:40.385  [Pipeline] echo
00:17:40.386  No retry patterns found.
00:17:40.387  [Pipeline] }
00:17:40.420  [Pipeline] // retry
00:17:40.428  [Pipeline] error
00:17:40.434  [Pipeline] }
00:17:40.439  Failed in branch nvme-phy-autotest
00:18:21.310  Build BlobFS-phy-autotest #26204 completed: SUCCESS
00:18:21.312  [Pipeline] }
00:18:21.351  [Pipeline] // retry
00:18:21.357  [Pipeline] }
00:19:55.672  Build vfio-user-phy-autotest #26206 completed: SUCCESS
00:19:55.674  [Pipeline] }
00:19:55.705  [Pipeline] // retry
00:19:55.710  [Pipeline] }
00:21:54.921  Build zns-vg-autotest #76584 completed: SUCCESS
00:21:54.923  [Pipeline] }
00:21:54.968  [Pipeline] // retry
00:21:54.973  [Pipeline] }
00:22:07.471  Build vhost-initiator-vg-autotest #77498 completed: SUCCESS
00:22:07.473  [Pipeline] }
00:22:07.521  [Pipeline] // retry
00:22:07.527  [Pipeline] }
00:24:47.582  Build lvol-vg-autotest #77427 completed: SUCCESS
00:24:47.585  [Pipeline] }
00:24:47.631  [Pipeline] // retry
00:24:47.636  [Pipeline] }
00:29:44.012  Build nvme-cmb-pmr-vg-autotest #78601 completed: SUCCESS
00:29:44.015  [Pipeline] }
00:29:44.056  [Pipeline] // retry
00:29:44.063  [Pipeline] }
00:30:16.657  Build nvme-vg-autotest #77478 completed: FAILURE
00:30:17.421  [Pipeline] echo
00:30:17.422  No retry patterns found.
00:30:17.423  [Pipeline] }
00:30:17.454  [Pipeline] // retry
00:30:17.460  [Pipeline] error
00:30:17.465  [Pipeline] }
00:30:17.469  Failed in branch nvme-vg-autotest
00:31:58.695  Build scanbuild-vg-autotest #63377 completed: SUCCESS
00:31:58.697  [Pipeline] }
00:31:58.753  [Pipeline] // retry
00:31:58.763  [Pipeline] }
00:32:25.078  Build vhost-phy-autotest #26223 completed: SUCCESS
00:32:25.081  [Pipeline] }
00:32:25.123  [Pipeline] // retry
00:32:25.129  [Pipeline] }
00:33:45.151  Build autobuild-vg-autotest #63418 completed: SUCCESS
00:33:45.153  [Pipeline] }
00:33:45.201  [Pipeline] // retry
00:33:45.207  [Pipeline] }
00:37:56.889  Build crypto-phy-autotest #26198 completed: SUCCESS
00:37:56.894  [Pipeline] }
00:37:56.938  [Pipeline] // retry
00:37:56.944  [Pipeline] }
00:38:17.831  Build iscsi-uring-vg-autotest #77383 completed: SUCCESS
00:38:17.833  [Pipeline] }
00:38:17.892  [Pipeline] // retry
00:38:17.899  [Pipeline] }
00:40:54.487  Build nvmf-tcp-uring-vg-autotest #77403 completed: SUCCESS
00:40:54.491  [Pipeline] }
00:40:54.536  [Pipeline] // retry
00:40:54.541  [Pipeline] }
00:41:21.816  Build nvmf-tcp-phy-autotest #72329 completed: ABORTED
00:41:23.645  [Pipeline] echo
00:41:23.646  No retry patterns found.
00:41:23.647  [Pipeline] }
00:41:23.682  [Pipeline] // retry
00:41:23.689  [Pipeline] error
00:41:23.692  [Pipeline] }
00:41:23.695  Failed in branch nvmf-tcp-phy-autotest
00:41:24.860  Build unittest-vg-autotest #63375 completed: SUCCESS
00:41:24.862  [Pipeline] }
00:41:24.904  [Pipeline] // retry
00:41:24.910  [Pipeline] }
00:41:30.256  Build nvmf-phy-autotest #72442 completed: ABORTED
00:41:31.690  [Pipeline] echo
00:41:31.691  No retry patterns found.
00:41:31.692  [Pipeline] }
00:41:31.712  [Pipeline] // retry
00:41:31.717  [Pipeline] error
00:41:31.720  [Pipeline] }
00:41:31.723  Failed in branch nvmf-phy-autotest
00:45:02.330  Build nvmf-tcp-vg-autotest #77469 completed: SUCCESS
00:45:02.333  [Pipeline] }
00:45:02.371  [Pipeline] // retry
00:45:02.376  [Pipeline] }
00:46:01.702  Build centos7-vg-autotest #73657 completed: SUCCESS
00:46:01.706  [Pipeline] }
00:46:01.736  [Pipeline] // retry
00:46:01.740  [Pipeline] }
00:47:30.957  Build clang-vg-autotest #57626 completed: SUCCESS
00:47:30.959  [Pipeline] }
00:47:31.002  [Pipeline] // retry
00:47:31.008  [Pipeline] }
00:47:37.177  Build iscsi-vg-autotest #77516 completed: SUCCESS
00:47:37.179  [Pipeline] }
00:47:37.220  [Pipeline] // retry
00:47:37.226  [Pipeline] }
00:49:40.572  Build ubuntu20-vg-autotest #73797 completed: FAILURE
00:49:43.442  [Pipeline] echo
00:49:43.443  No retry patterns found.
00:49:43.444  [Pipeline] }
00:49:43.476  [Pipeline] // retry
00:49:43.482  [Pipeline] error
00:49:43.486  [Pipeline] }
00:49:43.488  Failed in branch ubuntu20-vg-autotest
00:53:57.624  Build valgrind-vg-autotest #49192 completed: SUCCESS
00:53:57.626  [Pipeline] }
00:53:57.672  [Pipeline] // retry
00:53:57.679  [Pipeline] }
01:00:58.374  Build rocky9-vg-autotest #16088 completed: SUCCESS
01:00:58.377  [Pipeline] }
01:00:58.412  [Pipeline] // retry
01:00:58.417  [Pipeline] }
01:07:52.813  Build ubuntu22-vg-autotest #37811 completed: SUCCESS
01:07:52.816  [Pipeline] }
01:07:52.842  [Pipeline] // retry
01:07:52.848  [Pipeline] }
01:07:53.054  [Pipeline] // parallel
01:07:53.064  [Pipeline] }
01:07:53.103  [Pipeline] // stage
01:07:53.114  [Pipeline] }
01:07:53.128  ERROR: Build nvme-phy-autotest #72370 failed
01:07:53.157  [Pipeline] // catchError
01:07:53.167  [Pipeline] stage
01:07:53.169  [Pipeline] { (Autorun Post and Coverage)
01:07:53.201  [Pipeline] setCustomBuildProperty
01:07:53.505  [Pipeline] dir
01:07:53.505  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620
01:07:53.506  [Pipeline] {
01:07:53.546  [Pipeline] copyArtifacts
01:07:53.753  Copied 12 artifacts from "vmd-phy-autotest" build number 2620
01:07:53.757  [Pipeline] writeFile
01:07:53.886  [Pipeline] }
01:07:53.979  [Pipeline] // dir
01:07:54.377  [Pipeline] dir
01:07:54.377  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195
01:07:54.379  [Pipeline] {
01:07:54.424  [Pipeline] copyArtifacts
01:07:54.530  Copied 10 artifacts from "short-fuzz-phy-autotest" build number 26195
01:07:54.534  [Pipeline] writeFile
01:07:54.720  [Pipeline] }
01:07:54.872  [Pipeline] // dir
01:07:55.188  [Pipeline] dir
01:07:55.188  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898
01:07:55.189  [Pipeline] {
01:07:55.225  [Pipeline] copyArtifacts
01:07:55.446  Copied 12 artifacts from "dsa-phy-autotest" build number 4898
01:07:55.451  [Pipeline] writeFile
01:07:55.574  [Pipeline] }
01:07:55.677  [Pipeline] // dir
01:07:56.386  [Pipeline] dir
01:07:56.386  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370
01:07:56.387  [Pipeline] {
01:07:56.446  [Pipeline] copyArtifacts
01:07:56.812  Copied 11 artifacts from "nvme-phy-autotest" build number 72370
01:07:56.816  [Pipeline] writeFile
01:07:57.011  [Pipeline] }
01:07:57.129  [Pipeline] // dir
01:07:57.431  [Pipeline] dir
01:07:57.431  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204
01:07:57.433  [Pipeline] {
01:07:57.474  [Pipeline] copyArtifacts
01:07:57.824  Copied 28 artifacts from "BlobFS-phy-autotest" build number 26204
01:07:57.828  [Pipeline] writeFile
01:07:57.950  [Pipeline] }
01:07:58.023  [Pipeline] // dir
01:07:58.575  [Pipeline] dir
01:07:58.576  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206
01:07:58.577  [Pipeline] {
01:07:58.626  [Pipeline] copyArtifacts
01:07:58.834  Copied 12 artifacts from "vfio-user-phy-autotest" build number 26206
01:07:58.837  [Pipeline] writeFile
01:07:59.050  [Pipeline] }
01:07:59.157  [Pipeline] // dir
01:07:59.584  [Pipeline] dir
01:07:59.584  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584
01:07:59.585  [Pipeline] {
01:07:59.621  [Pipeline] copyArtifacts
01:07:59.810  Copied 12 artifacts from "zns-vg-autotest" build number 76584
01:07:59.817  [Pipeline] writeFile
01:07:59.994  [Pipeline] }
01:08:00.090  [Pipeline] // dir
01:08:00.486  [Pipeline] dir
01:08:00.487  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498
01:08:00.488  [Pipeline] {
01:08:00.526  [Pipeline] copyArtifacts
01:08:00.749  Copied 15 artifacts from "vhost-initiator-vg-autotest" build number 77498
01:08:00.753  [Pipeline] writeFile
01:08:00.885  [Pipeline] }
01:08:00.965  [Pipeline] // dir
01:08:01.600  [Pipeline] dir
01:08:01.600  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427
01:08:01.602  [Pipeline] {
01:08:01.658  [Pipeline] copyArtifacts
01:08:01.840  Copied 12 artifacts from "lvol-vg-autotest" build number 77427
01:08:01.844  [Pipeline] writeFile
01:08:02.078  [Pipeline] }
01:08:02.225  [Pipeline] // dir
01:08:02.603  [Pipeline] dir
01:08:02.603  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601
01:08:02.605  [Pipeline] {
01:08:02.645  [Pipeline] copyArtifacts
01:08:02.837  Copied 12 artifacts from "nvme-cmb-pmr-vg-autotest" build number 78601
01:08:02.841  [Pipeline] writeFile
01:08:02.989  [Pipeline] }
01:08:03.076  [Pipeline] // dir
01:08:03.541  [Pipeline] dir
01:08:03.541  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478
01:08:03.542  [Pipeline] {
01:08:03.590  [Pipeline] copyArtifacts
01:08:03.831  Copied 11 artifacts from "nvme-vg-autotest" build number 77478
01:08:03.836  [Pipeline] writeFile
01:08:04.008  [Pipeline] }
01:08:04.155  [Pipeline] // dir
01:08:04.260  [Pipeline] dir
01:08:04.261  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377
01:08:04.262  [Pipeline] {
01:08:04.286  [Pipeline] copyArtifacts
01:08:04.392  Copied 12 artifacts from "scanbuild-vg-autotest" build number 63377
01:08:04.395  [Pipeline] writeFile
01:08:04.444  [Pipeline] }
01:08:04.497  [Pipeline] // dir
01:08:05.158  [Pipeline] dir
01:08:05.158  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223
01:08:05.159  [Pipeline] {
01:08:05.214  [Pipeline] copyArtifacts
01:08:05.441  Copied 13 artifacts from "vhost-phy-autotest" build number 26223
01:08:05.446  [Pipeline] writeFile
01:08:05.671  [Pipeline] }
01:08:05.815  [Pipeline] // dir
01:08:06.007  [Pipeline] dir
01:08:06.007  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418
01:08:06.009  [Pipeline] {
01:08:06.037  [Pipeline] copyArtifacts
01:08:06.318  Copied 9 artifacts from "autobuild-vg-autotest" build number 63418
01:08:06.323  [Pipeline] writeFile
01:08:06.389  [Pipeline] }
01:08:06.454  [Pipeline] // dir
01:08:07.822  [Pipeline] dir
01:08:07.822  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198
01:08:07.823  [Pipeline] {
01:08:07.915  [Pipeline] copyArtifacts
01:08:08.112  Copied 12 artifacts from "crypto-phy-autotest" build number 26198
01:08:08.116  [Pipeline] writeFile
01:08:08.694  [Pipeline] }
01:08:08.970  [Pipeline] // dir
01:08:11.048  [Pipeline] dir
01:08:11.048  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383
01:08:11.050  [Pipeline] {
01:08:11.231  [Pipeline] copyArtifacts
01:08:13.540  Copied 16 artifacts from "iscsi-uring-vg-autotest" build number 77383
01:08:13.544  [Pipeline] writeFile
01:08:14.819  [Pipeline] }
01:08:15.180  [Pipeline] // dir
01:08:15.883  [Pipeline] dir
01:08:15.884  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403
01:08:15.886  [Pipeline] {
01:08:15.988  [Pipeline] copyArtifacts
01:08:16.170  Copied 14 artifacts from "nvmf-tcp-uring-vg-autotest" build number 77403
01:08:16.175  [Pipeline] writeFile
01:08:16.501  [Pipeline] }
01:08:16.675  [Pipeline] // dir
01:08:17.872  [Pipeline] dir
01:08:17.872  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329
01:08:17.873  [Pipeline] {
01:08:17.960  [Pipeline] copyArtifacts
01:08:18.341  Copied 13 artifacts from "nvmf-tcp-phy-autotest" build number 72329
01:08:18.345  [Pipeline] writeFile
01:08:18.830  [Pipeline] }
01:08:19.116  [Pipeline] // dir
01:08:19.397  [Pipeline] dir
01:08:19.397  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375
01:08:19.398  [Pipeline] {
01:08:19.437  [Pipeline] copyArtifacts
01:08:19.814  Copied 13 artifacts from "unittest-vg-autotest" build number 63375
01:08:19.819  [Pipeline] writeFile
01:08:19.941  [Pipeline] }
01:08:20.016  [Pipeline] // dir
01:08:20.831  [Pipeline] dir
01:08:20.831  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442
01:08:20.832  [Pipeline] {
01:08:20.901  [Pipeline] copyArtifacts
01:08:21.313  Copied 12 artifacts from "nvmf-phy-autotest" build number 72442
01:08:21.317  [Pipeline] writeFile
01:08:21.608  [Pipeline] }
01:08:21.816  [Pipeline] // dir
01:08:22.744  [Pipeline] dir
01:08:22.744  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469
01:08:22.745  [Pipeline] {
01:08:22.811  [Pipeline] copyArtifacts
01:08:22.996  Copied 14 artifacts from "nvmf-tcp-vg-autotest" build number 77469
01:08:22.999  [Pipeline] writeFile
01:08:23.406  [Pipeline] }
01:08:23.623  [Pipeline] // dir
01:08:24.508  [Pipeline] dir
01:08:24.508  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657
01:08:24.509  [Pipeline] {
01:08:24.568  [Pipeline] copyArtifacts
01:08:24.916  Copied 13 artifacts from "centos7-vg-autotest" build number 73657
01:08:24.920  [Pipeline] writeFile
01:08:25.252  [Pipeline] }
01:08:25.463  [Pipeline] // dir
01:08:25.730  [Pipeline] dir
01:08:25.730  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626
01:08:25.731  [Pipeline] {
01:08:25.767  [Pipeline] copyArtifacts
01:08:25.874  Copied 10 artifacts from "clang-vg-autotest" build number 57626
01:08:25.878  [Pipeline] writeFile
01:08:26.008  [Pipeline] }
01:08:26.096  [Pipeline] // dir
01:08:28.717  [Pipeline] dir
01:08:28.717  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516
01:08:28.718  [Pipeline] {
01:08:28.989  [Pipeline] copyArtifacts
01:08:31.094  Copied 16 artifacts from "iscsi-vg-autotest" build number 77516
01:08:31.099  [Pipeline] writeFile
01:08:32.404  [Pipeline] }
01:08:33.062  [Pipeline] // dir
01:08:34.274  [Pipeline] dir
01:08:34.275  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797
01:08:34.276  [Pipeline] {
01:08:34.374  [Pipeline] copyArtifacts
01:08:34.934  Copied 12 artifacts from "ubuntu20-vg-autotest" build number 73797
01:08:34.938  [Pipeline] writeFile
01:08:35.369  [Pipeline] }
01:08:35.587  [Pipeline] // dir
01:08:36.407  [Pipeline] dir
01:08:36.408  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192
01:08:36.409  [Pipeline] {
01:08:36.475  [Pipeline] copyArtifacts
01:08:36.764  Copied 13 artifacts from "valgrind-vg-autotest" build number 49192
01:08:36.769  [Pipeline] writeFile
01:08:37.069  [Pipeline] }
01:08:37.249  [Pipeline] // dir
01:08:38.546  [Pipeline] dir
01:08:38.546  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088
01:08:38.548  [Pipeline] {
01:08:38.654  [Pipeline] copyArtifacts
01:08:39.015  Copied 13 artifacts from "rocky9-vg-autotest" build number 16088
01:08:39.019  [Pipeline] writeFile
01:08:39.532  [Pipeline] }
01:08:39.807  [Pipeline] // dir
01:08:41.168  [Pipeline] dir
01:08:41.168  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811
01:08:41.169  [Pipeline] {
01:08:41.254  [Pipeline] copyArtifacts
01:08:41.630  Copied 13 artifacts from "ubuntu22-vg-autotest" build number 37811
01:08:41.634  [Pipeline] writeFile
01:08:42.093  [Pipeline] }
01:08:42.335  [Pipeline] // dir
01:08:42.343  [Pipeline] catchError
01:08:42.345  [Pipeline] {
01:08:42.364  [Pipeline] sh
01:08:42.655  + jbp/jenkins/jjb-config/jobs/scripts/post_gen_coverage.sh
01:08:42.655  + shopt -s globstar nullglob
01:08:42.655  + echo 'Start stage post_gen_coverage.sh'
01:08:42.655  Start stage post_gen_coverage.sh
01:08:42.655  + cd /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11
01:08:42.655  + rm -rf /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/doc
01:08:42.655  + trap 'compress_coverage_and_docs; remove_partial_coverage_files && echo '\''End stage post_gen_coverage.sh'\''' EXIT
01:08:42.655  + move_artifacts
01:08:42.655  + local out_dirs
01:08:42.655  + out_dirs=(./**/output/)
01:08:42.655  + for dir in "${out_dirs[@]}"
01:08:42.655  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:42.655  + [[ -f ./autobuild-vg-autotest_63418/output//doc.tar.xz ]]
01:08:42.655  + tar -C ./autobuild-vg-autotest_63418/output/ -xf ./autobuild-vg-autotest_63418/output//doc.tar.xz
01:08:42.912  + rm ./autobuild-vg-autotest_63418/output//doc.tar.xz
01:08:42.912  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:42.912  + [[ -f ./autobuild-vg-autotest_63418/output//ut_coverage.tar.xz ]]
01:08:42.912  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:42.912  + [[ -f ./autobuild-vg-autotest_63418/output//llvm.tar.xz ]]
01:08:42.912  + mv ./autobuild-vg-autotest_63418/output//build-pkg-manifest.txt ./autobuild-vg-autotest_63418/output//build-repo-manifest.txt ./autobuild-vg-autotest_63418/output//dmesg.init.log ./autobuild-vg-autotest_63418/output//dmesg.log ./autobuild-vg-autotest_63418/output//doc ./autobuild-vg-autotest_63418/output//power.tar.xz ./autobuild-vg-autotest_63418/output//test_completions.txt ./autobuild-vg-autotest_63418/output//timing.svg ./autobuild-vg-autotest_63418/output//timing.txt ./autobuild-vg-autotest_63418/output//..
01:08:42.912  + rmdir ./autobuild-vg-autotest_63418/output/
01:08:42.912  + for dir in "${out_dirs[@]}"
01:08:42.912  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:42.912  + [[ -f ./BlobFS-phy-autotest_26204/output//doc.tar.xz ]]
01:08:42.912  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:42.912  + [[ -f ./BlobFS-phy-autotest_26204/output//ut_coverage.tar.xz ]]
01:08:42.912  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:42.912  + [[ -f ./BlobFS-phy-autotest_26204/output//llvm.tar.xz ]]
01:08:42.912  + mv ./BlobFS-phy-autotest_26204/output//all_tests.txt ./BlobFS-phy-autotest_26204/output//build-pkg-manifest.txt ./BlobFS-phy-autotest_26204/output//build-repo-manifest.txt ./BlobFS-phy-autotest_26204/output//cov_base.info.xz ./BlobFS-phy-autotest_26204/output//cov_test.info.xz ./BlobFS-phy-autotest_26204/output//cov_total.info.xz ./BlobFS-phy-autotest_26204/output//dmesg.init.log ./BlobFS-phy-autotest_26204/output//dmesg.log ./BlobFS-phy-autotest_26204/output//power.tar.xz ./BlobFS-phy-autotest_26204/output//rocksdb ./BlobFS-phy-autotest_26204/output//test_completions.txt ./BlobFS-phy-autotest_26204/output//timing.svg ./BlobFS-phy-autotest_26204/output//timing.txt ./BlobFS-phy-autotest_26204/output//..
01:08:42.912  + rmdir ./BlobFS-phy-autotest_26204/output/
01:08:42.912  + for dir in "${out_dirs[@]}"
01:08:42.912  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:42.912  + [[ -f ./centos7-vg-autotest_73657/output//doc.tar.xz ]]
01:08:42.912  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:42.912  + [[ -f ./centos7-vg-autotest_73657/output//ut_coverage.tar.xz ]]
01:08:42.912  + tar -C ./centos7-vg-autotest_73657/output/ -xf ./centos7-vg-autotest_73657/output//ut_coverage.tar.xz
01:08:43.170  + rm ./centos7-vg-autotest_73657/output//ut_coverage.tar.xz
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./centos7-vg-autotest_73657/output//llvm.tar.xz ]]
01:08:43.170  + mv ./centos7-vg-autotest_73657/output//all_tests.txt ./centos7-vg-autotest_73657/output//build-pkg-manifest.txt ./centos7-vg-autotest_73657/output//build-repo-manifest.txt ./centos7-vg-autotest_73657/output//cov_base.info.xz ./centos7-vg-autotest_73657/output//cov_test.info.xz ./centos7-vg-autotest_73657/output//cov_total.info.xz ./centos7-vg-autotest_73657/output//dmesg.init.log ./centos7-vg-autotest_73657/output//dmesg.log ./centos7-vg-autotest_73657/output//power.tar.xz ./centos7-vg-autotest_73657/output//test_completions.txt ./centos7-vg-autotest_73657/output//timing.svg ./centos7-vg-autotest_73657/output//timing.txt ./centos7-vg-autotest_73657/output//ut_coverage ./centos7-vg-autotest_73657/output//..
01:08:43.170  + rmdir ./centos7-vg-autotest_73657/output/
01:08:43.170  + for dir in "${out_dirs[@]}"
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./clang-vg-autotest_57626/output//doc.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./clang-vg-autotest_57626/output//ut_coverage.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./clang-vg-autotest_57626/output//llvm.tar.xz ]]
01:08:43.170  + mv ./clang-vg-autotest_57626/output//all_tests.txt ./clang-vg-autotest_57626/output//build-kernel-manifest.txt ./clang-vg-autotest_57626/output//build-pkg-manifest.txt ./clang-vg-autotest_57626/output//build-repo-manifest.txt ./clang-vg-autotest_57626/output//dmesg.init.log ./clang-vg-autotest_57626/output//dmesg.log ./clang-vg-autotest_57626/output//power.tar.xz ./clang-vg-autotest_57626/output//test_completions.txt ./clang-vg-autotest_57626/output//timing.svg ./clang-vg-autotest_57626/output//timing.txt ./clang-vg-autotest_57626/output//..
01:08:43.170  + rmdir ./clang-vg-autotest_57626/output/
01:08:43.170  + for dir in "${out_dirs[@]}"
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./crypto-phy-autotest_26198/output//doc.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./crypto-phy-autotest_26198/output//ut_coverage.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./crypto-phy-autotest_26198/output//llvm.tar.xz ]]
01:08:43.170  + mv ./crypto-phy-autotest_26198/output//all_tests.txt ./crypto-phy-autotest_26198/output//build-pkg-manifest.txt ./crypto-phy-autotest_26198/output//build-repo-manifest.txt ./crypto-phy-autotest_26198/output//cov_base.info.xz ./crypto-phy-autotest_26198/output//cov_test.info.xz ./crypto-phy-autotest_26198/output//cov_total.info.xz ./crypto-phy-autotest_26198/output//dmesg.init.log ./crypto-phy-autotest_26198/output//dmesg.log ./crypto-phy-autotest_26198/output//power.tar.xz ./crypto-phy-autotest_26198/output//test_completions.txt ./crypto-phy-autotest_26198/output//timing.svg ./crypto-phy-autotest_26198/output//timing.txt ./crypto-phy-autotest_26198/output//..
01:08:43.170  + rmdir ./crypto-phy-autotest_26198/output/
01:08:43.170  + for dir in "${out_dirs[@]}"
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./dsa-phy-autotest_4898/output//doc.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./dsa-phy-autotest_4898/output//ut_coverage.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./dsa-phy-autotest_4898/output//llvm.tar.xz ]]
01:08:43.170  + mv ./dsa-phy-autotest_4898/output//all_tests.txt ./dsa-phy-autotest_4898/output//build-pkg-manifest.txt ./dsa-phy-autotest_4898/output//build-repo-manifest.txt ./dsa-phy-autotest_4898/output//cov_base.info.xz ./dsa-phy-autotest_4898/output//cov_test.info.xz ./dsa-phy-autotest_4898/output//cov_total.info.xz ./dsa-phy-autotest_4898/output//dmesg.init.log ./dsa-phy-autotest_4898/output//dmesg.log ./dsa-phy-autotest_4898/output//power.tar.xz ./dsa-phy-autotest_4898/output//test_completions.txt ./dsa-phy-autotest_4898/output//timing.svg ./dsa-phy-autotest_4898/output//timing.txt ./dsa-phy-autotest_4898/output//..
01:08:43.170  + rmdir ./dsa-phy-autotest_4898/output/
01:08:43.170  + for dir in "${out_dirs[@]}"
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./iscsi-uring-vg-autotest_77383/output//doc.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./iscsi-uring-vg-autotest_77383/output//ut_coverage.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./iscsi-uring-vg-autotest_77383/output//llvm.tar.xz ]]
01:08:43.170  + mv ./iscsi-uring-vg-autotest_77383/output//all_tests.txt ./iscsi-uring-vg-autotest_77383/output//build-pkg-manifest.txt ./iscsi-uring-vg-autotest_77383/output//build-repo-manifest.txt ./iscsi-uring-vg-autotest_77383/output//calsoft.json ./iscsi-uring-vg-autotest_77383/output//calsoft.tar.xz ./iscsi-uring-vg-autotest_77383/output//cov_base.info.xz ./iscsi-uring-vg-autotest_77383/output//cov_test.info.xz ./iscsi-uring-vg-autotest_77383/output//cov_total.info.xz ./iscsi-uring-vg-autotest_77383/output//dmesg.init.log ./iscsi-uring-vg-autotest_77383/output//dmesg.log ./iscsi-uring-vg-autotest_77383/output//iscsi_autofuzz_logs.txt ./iscsi-uring-vg-autotest_77383/output//iscsi_autofuzz_tgt_output.txt ./iscsi-uring-vg-autotest_77383/output//power.tar.xz ./iscsi-uring-vg-autotest_77383/output//test_completions.txt ./iscsi-uring-vg-autotest_77383/output//timing.svg ./iscsi-uring-vg-autotest_77383/output//timing.txt ./iscsi-uring-vg-autotest_77383/output//..
01:08:43.170  + rmdir ./iscsi-uring-vg-autotest_77383/output/
01:08:43.170  + for dir in "${out_dirs[@]}"
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./iscsi-vg-autotest_77516/output//doc.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./iscsi-vg-autotest_77516/output//ut_coverage.tar.xz ]]
01:08:43.170  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.170  + [[ -f ./iscsi-vg-autotest_77516/output//llvm.tar.xz ]]
01:08:43.171  + mv ./iscsi-vg-autotest_77516/output//all_tests.txt ./iscsi-vg-autotest_77516/output//build-pkg-manifest.txt ./iscsi-vg-autotest_77516/output//build-repo-manifest.txt ./iscsi-vg-autotest_77516/output//calsoft.json ./iscsi-vg-autotest_77516/output//calsoft.tar.xz ./iscsi-vg-autotest_77516/output//cov_base.info.xz ./iscsi-vg-autotest_77516/output//cov_test.info.xz ./iscsi-vg-autotest_77516/output//cov_total.info.xz ./iscsi-vg-autotest_77516/output//dmesg.init.log ./iscsi-vg-autotest_77516/output//dmesg.log ./iscsi-vg-autotest_77516/output//iscsi_autofuzz_logs.txt ./iscsi-vg-autotest_77516/output//iscsi_autofuzz_tgt_output.txt ./iscsi-vg-autotest_77516/output//power.tar.xz ./iscsi-vg-autotest_77516/output//test_completions.txt ./iscsi-vg-autotest_77516/output//timing.svg ./iscsi-vg-autotest_77516/output//timing.txt ./iscsi-vg-autotest_77516/output//..
01:08:43.171  + rmdir ./iscsi-vg-autotest_77516/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./lvol-vg-autotest_77427/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./lvol-vg-autotest_77427/output//ut_coverage.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./lvol-vg-autotest_77427/output//llvm.tar.xz ]]
01:08:43.171  + mv ./lvol-vg-autotest_77427/output//all_tests.txt ./lvol-vg-autotest_77427/output//build-pkg-manifest.txt ./lvol-vg-autotest_77427/output//build-repo-manifest.txt ./lvol-vg-autotest_77427/output//cov_base.info.xz ./lvol-vg-autotest_77427/output//cov_test.info.xz ./lvol-vg-autotest_77427/output//cov_total.info.xz ./lvol-vg-autotest_77427/output//dmesg.init.log ./lvol-vg-autotest_77427/output//dmesg.log ./lvol-vg-autotest_77427/output//power.tar.xz ./lvol-vg-autotest_77427/output//test_completions.txt ./lvol-vg-autotest_77427/output//timing.svg ./lvol-vg-autotest_77427/output//timing.txt ./lvol-vg-autotest_77427/output//..
01:08:43.171  + rmdir ./lvol-vg-autotest_77427/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-cmb-pmr-vg-autotest_78601/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-cmb-pmr-vg-autotest_78601/output//ut_coverage.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-cmb-pmr-vg-autotest_78601/output//llvm.tar.xz ]]
01:08:43.171  + mv ./nvme-cmb-pmr-vg-autotest_78601/output//all_tests.txt ./nvme-cmb-pmr-vg-autotest_78601/output//build-pkg-manifest.txt ./nvme-cmb-pmr-vg-autotest_78601/output//build-repo-manifest.txt ./nvme-cmb-pmr-vg-autotest_78601/output//cov_base.info.xz ./nvme-cmb-pmr-vg-autotest_78601/output//cov_test.info.xz ./nvme-cmb-pmr-vg-autotest_78601/output//cov_total.info.xz ./nvme-cmb-pmr-vg-autotest_78601/output//dmesg.init.log ./nvme-cmb-pmr-vg-autotest_78601/output//dmesg.log ./nvme-cmb-pmr-vg-autotest_78601/output//power.tar.xz ./nvme-cmb-pmr-vg-autotest_78601/output//test_completions.txt ./nvme-cmb-pmr-vg-autotest_78601/output//timing.svg ./nvme-cmb-pmr-vg-autotest_78601/output//timing.txt ./nvme-cmb-pmr-vg-autotest_78601/output//..
01:08:43.171  + rmdir ./nvme-cmb-pmr-vg-autotest_78601/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-phy-autotest_72370/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-phy-autotest_72370/output//ut_coverage.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-phy-autotest_72370/output//llvm.tar.xz ]]
01:08:43.171  + mv ./nvme-phy-autotest_72370/output//all_tests.txt ./nvme-phy-autotest_72370/output//build-pkg-manifest.txt ./nvme-phy-autotest_72370/output//build-repo-manifest.txt ./nvme-phy-autotest_72370/output//cov_base.info.xz ./nvme-phy-autotest_72370/output//dmesg.init.log ./nvme-phy-autotest_72370/output//dmesg.log ./nvme-phy-autotest_72370/output//power.tar.xz ./nvme-phy-autotest_72370/output//proc_list.txt ./nvme-phy-autotest_72370/output//test_completions.txt ./nvme-phy-autotest_72370/output//timing.txt ./nvme-phy-autotest_72370/output//udev.log ./nvme-phy-autotest_72370/output//..
01:08:43.171  + rmdir ./nvme-phy-autotest_72370/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-vg-autotest_77478/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-vg-autotest_77478/output//ut_coverage.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvme-vg-autotest_77478/output//llvm.tar.xz ]]
01:08:43.171  + mv ./nvme-vg-autotest_77478/output//all_tests.txt ./nvme-vg-autotest_77478/output//build-pkg-manifest.txt ./nvme-vg-autotest_77478/output//build-repo-manifest.txt ./nvme-vg-autotest_77478/output//cov_base.info.xz ./nvme-vg-autotest_77478/output//dmesg.init.log ./nvme-vg-autotest_77478/output//dmesg.log ./nvme-vg-autotest_77478/output//power.tar.xz ./nvme-vg-autotest_77478/output//proc_list.txt ./nvme-vg-autotest_77478/output//test_completions.txt ./nvme-vg-autotest_77478/output//timing.txt ./nvme-vg-autotest_77478/output//udev.log ./nvme-vg-autotest_77478/output//..
01:08:43.171  + rmdir ./nvme-vg-autotest_77478/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-phy-autotest_72442/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-phy-autotest_72442/output//ut_coverage.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-phy-autotest_72442/output//llvm.tar.xz ]]
01:08:43.171  + mv ./nvmf-phy-autotest_72442/output//all_tests.txt ./nvmf-phy-autotest_72442/output//build-pkg-manifest.txt ./nvmf-phy-autotest_72442/output//build-repo-manifest.txt ./nvmf-phy-autotest_72442/output//cov_base.info.xz ./nvmf-phy-autotest_72442/output//dmesg.init.log ./nvmf-phy-autotest_72442/output//dmesg.log ./nvmf-phy-autotest_72442/output//nvmf_fuzz_tgt_output.txt ./nvmf-phy-autotest_72442/output//nvmf_trace.0_shm.tar.gz ./nvmf-phy-autotest_72442/output//power.tar.xz ./nvmf-phy-autotest_72442/output//test_completions.txt ./nvmf-phy-autotest_72442/output//timing.txt ./nvmf-phy-autotest_72442/output//udev.log ./nvmf-phy-autotest_72442/output//..
01:08:43.171  + rmdir ./nvmf-phy-autotest_72442/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-phy-autotest_72329/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-phy-autotest_72329/output//ut_coverage.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-phy-autotest_72329/output//llvm.tar.xz ]]
01:08:43.171  + mv ./nvmf-tcp-phy-autotest_72329/output//all_tests.txt ./nvmf-tcp-phy-autotest_72329/output//build-pkg-manifest.txt ./nvmf-tcp-phy-autotest_72329/output//build-repo-manifest.txt ./nvmf-tcp-phy-autotest_72329/output//cov_base.info.xz ./nvmf-tcp-phy-autotest_72329/output//cov_test.info.xz ./nvmf-tcp-phy-autotest_72329/output//cov_total.info.xz ./nvmf-tcp-phy-autotest_72329/output//dmesg.init.log ./nvmf-tcp-phy-autotest_72329/output//dmesg.log ./nvmf-tcp-phy-autotest_72329/output//nvmf_fuzz_tgt_output.txt ./nvmf-tcp-phy-autotest_72329/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-phy-autotest_72329/output//power.tar.xz ./nvmf-tcp-phy-autotest_72329/output//test_completions.txt ./nvmf-tcp-phy-autotest_72329/output//timing.txt ./nvmf-tcp-phy-autotest_72329/output//..
01:08:43.171  + rmdir ./nvmf-tcp-phy-autotest_72329/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-uring-vg-autotest_77403/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-uring-vg-autotest_77403/output//ut_coverage.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-uring-vg-autotest_77403/output//llvm.tar.xz ]]
01:08:43.171  + mv ./nvmf-tcp-uring-vg-autotest_77403/output//all_tests.txt ./nvmf-tcp-uring-vg-autotest_77403/output//build-pkg-manifest.txt ./nvmf-tcp-uring-vg-autotest_77403/output//build-repo-manifest.txt ./nvmf-tcp-uring-vg-autotest_77403/output//cov_base.info.xz ./nvmf-tcp-uring-vg-autotest_77403/output//cov_test.info.xz ./nvmf-tcp-uring-vg-autotest_77403/output//cov_total.info.xz ./nvmf-tcp-uring-vg-autotest_77403/output//dmesg.init.log ./nvmf-tcp-uring-vg-autotest_77403/output//dmesg.log ./nvmf-tcp-uring-vg-autotest_77403/output//nvmf_fuzz_tgt_output.txt ./nvmf-tcp-uring-vg-autotest_77403/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-uring-vg-autotest_77403/output//power.tar.xz ./nvmf-tcp-uring-vg-autotest_77403/output//test_completions.txt ./nvmf-tcp-uring-vg-autotest_77403/output//timing.svg ./nvmf-tcp-uring-vg-autotest_77403/output//timing.txt ./nvmf-tcp-uring-vg-autotest_77403/output//..
01:08:43.171  + rmdir ./nvmf-tcp-uring-vg-autotest_77403/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-vg-autotest_77469/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-vg-autotest_77469/output//ut_coverage.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./nvmf-tcp-vg-autotest_77469/output//llvm.tar.xz ]]
01:08:43.171  + mv ./nvmf-tcp-vg-autotest_77469/output//all_tests.txt ./nvmf-tcp-vg-autotest_77469/output//build-pkg-manifest.txt ./nvmf-tcp-vg-autotest_77469/output//build-repo-manifest.txt ./nvmf-tcp-vg-autotest_77469/output//cov_base.info.xz ./nvmf-tcp-vg-autotest_77469/output//cov_test.info.xz ./nvmf-tcp-vg-autotest_77469/output//cov_total.info.xz ./nvmf-tcp-vg-autotest_77469/output//dmesg.init.log ./nvmf-tcp-vg-autotest_77469/output//dmesg.log ./nvmf-tcp-vg-autotest_77469/output//nvmf_fuzz_tgt_output.txt ./nvmf-tcp-vg-autotest_77469/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-vg-autotest_77469/output//power.tar.xz ./nvmf-tcp-vg-autotest_77469/output//test_completions.txt ./nvmf-tcp-vg-autotest_77469/output//timing.svg ./nvmf-tcp-vg-autotest_77469/output//timing.txt ./nvmf-tcp-vg-autotest_77469/output//..
01:08:43.171  + rmdir ./nvmf-tcp-vg-autotest_77469/output/
01:08:43.171  + for dir in "${out_dirs[@]}"
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./rocky9-vg-autotest_16088/output//doc.tar.xz ]]
01:08:43.171  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.171  + [[ -f ./rocky9-vg-autotest_16088/output//ut_coverage.tar.xz ]]
01:08:43.171  + tar -C ./rocky9-vg-autotest_16088/output/ -xf ./rocky9-vg-autotest_16088/output//ut_coverage.tar.xz
01:08:43.429  + rm ./rocky9-vg-autotest_16088/output//ut_coverage.tar.xz
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./rocky9-vg-autotest_16088/output//llvm.tar.xz ]]
01:08:43.429  + mv ./rocky9-vg-autotest_16088/output//all_tests.txt ./rocky9-vg-autotest_16088/output//build-pkg-manifest.txt ./rocky9-vg-autotest_16088/output//build-repo-manifest.txt ./rocky9-vg-autotest_16088/output//cov_base.info.xz ./rocky9-vg-autotest_16088/output//cov_test.info.xz ./rocky9-vg-autotest_16088/output//cov_total.info.xz ./rocky9-vg-autotest_16088/output//dmesg.init.log ./rocky9-vg-autotest_16088/output//dmesg.log ./rocky9-vg-autotest_16088/output//power.tar.xz ./rocky9-vg-autotest_16088/output//test_completions.txt ./rocky9-vg-autotest_16088/output//timing.svg ./rocky9-vg-autotest_16088/output//timing.txt ./rocky9-vg-autotest_16088/output//ut_coverage ./rocky9-vg-autotest_16088/output//..
01:08:43.429  + rmdir ./rocky9-vg-autotest_16088/output/
01:08:43.429  + for dir in "${out_dirs[@]}"
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./scanbuild-vg-autotest_63377/output//doc.tar.xz ]]
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./scanbuild-vg-autotest_63377/output//ut_coverage.tar.xz ]]
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./scanbuild-vg-autotest_63377/output//llvm.tar.xz ]]
01:08:43.429  + mv ./scanbuild-vg-autotest_63377/output//all_c_files.txt ./scanbuild-vg-autotest_63377/output//build_output.txt ./scanbuild-vg-autotest_63377/output//build-pkg-manifest.txt ./scanbuild-vg-autotest_63377/output//build-repo-manifest.txt ./scanbuild-vg-autotest_63377/output//built_c_files.txt ./scanbuild-vg-autotest_63377/output//dmesg.init.log ./scanbuild-vg-autotest_63377/output//dmesg.log ./scanbuild-vg-autotest_63377/output//power.tar.xz ./scanbuild-vg-autotest_63377/output//test_completions.txt ./scanbuild-vg-autotest_63377/output//timing.svg ./scanbuild-vg-autotest_63377/output//timing.txt ./scanbuild-vg-autotest_63377/output//unbuilt_c_files.txt ./scanbuild-vg-autotest_63377/output//..
01:08:43.429  + rmdir ./scanbuild-vg-autotest_63377/output/
01:08:43.429  + for dir in "${out_dirs[@]}"
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./short-fuzz-phy-autotest_26195/output//doc.tar.xz ]]
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./short-fuzz-phy-autotest_26195/output//ut_coverage.tar.xz ]]
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./short-fuzz-phy-autotest_26195/output//llvm.tar.xz ]]
01:08:43.429  + tar -C ./short-fuzz-phy-autotest_26195/output/ -xf ./short-fuzz-phy-autotest_26195/output//llvm.tar.xz
01:08:43.429  + rm ./short-fuzz-phy-autotest_26195/output//llvm.tar.xz
01:08:43.429  + mv ./short-fuzz-phy-autotest_26195/output//all_tests.txt ./short-fuzz-phy-autotest_26195/output//build-pkg-manifest.txt ./short-fuzz-phy-autotest_26195/output//build-repo-manifest.txt ./short-fuzz-phy-autotest_26195/output//dmesg.init.log ./short-fuzz-phy-autotest_26195/output//dmesg.log ./short-fuzz-phy-autotest_26195/output//llvm ./short-fuzz-phy-autotest_26195/output//power.tar.xz ./short-fuzz-phy-autotest_26195/output//test_completions.txt ./short-fuzz-phy-autotest_26195/output//timing.svg ./short-fuzz-phy-autotest_26195/output//timing.txt ./short-fuzz-phy-autotest_26195/output//..
01:08:43.429  + rmdir ./short-fuzz-phy-autotest_26195/output/
01:08:43.429  + for dir in "${out_dirs[@]}"
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./ubuntu20-vg-autotest_73797/output//doc.tar.xz ]]
01:08:43.429  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.429  + [[ -f ./ubuntu20-vg-autotest_73797/output//ut_coverage.tar.xz ]]
01:08:43.429  + tar -C ./ubuntu20-vg-autotest_73797/output/ -xf ./ubuntu20-vg-autotest_73797/output//ut_coverage.tar.xz
01:08:43.687  + rm ./ubuntu20-vg-autotest_73797/output//ut_coverage.tar.xz
01:08:43.687  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.687  + [[ -f ./ubuntu20-vg-autotest_73797/output//llvm.tar.xz ]]
01:08:43.687  + mv ./ubuntu20-vg-autotest_73797/output//all_tests.txt ./ubuntu20-vg-autotest_73797/output//build-pkg-manifest.txt ./ubuntu20-vg-autotest_73797/output//build-repo-manifest.txt ./ubuntu20-vg-autotest_73797/output//cov_base.info.xz ./ubuntu20-vg-autotest_73797/output//dmesg.init.log ./ubuntu20-vg-autotest_73797/output//dmesg.log ./ubuntu20-vg-autotest_73797/output//power.tar.xz ./ubuntu20-vg-autotest_73797/output//proc_list.txt ./ubuntu20-vg-autotest_73797/output//test_completions.txt ./ubuntu20-vg-autotest_73797/output//timing.txt ./ubuntu20-vg-autotest_73797/output//udev.log ./ubuntu20-vg-autotest_73797/output//ut_coverage ./ubuntu20-vg-autotest_73797/output//..
01:08:43.687  + rmdir ./ubuntu20-vg-autotest_73797/output/
01:08:43.687  + for dir in "${out_dirs[@]}"
01:08:43.687  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.687  + [[ -f ./ubuntu22-vg-autotest_37811/output//doc.tar.xz ]]
01:08:43.687  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.687  + [[ -f ./ubuntu22-vg-autotest_37811/output//ut_coverage.tar.xz ]]
01:08:43.687  + tar -C ./ubuntu22-vg-autotest_37811/output/ -xf ./ubuntu22-vg-autotest_37811/output//ut_coverage.tar.xz
01:08:43.945  + rm ./ubuntu22-vg-autotest_37811/output//ut_coverage.tar.xz
01:08:43.945  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.945  + [[ -f ./ubuntu22-vg-autotest_37811/output//llvm.tar.xz ]]
01:08:43.945  + mv ./ubuntu22-vg-autotest_37811/output//all_tests.txt ./ubuntu22-vg-autotest_37811/output//build-pkg-manifest.txt ./ubuntu22-vg-autotest_37811/output//build-repo-manifest.txt ./ubuntu22-vg-autotest_37811/output//cov_base.info.xz ./ubuntu22-vg-autotest_37811/output//cov_test.info.xz ./ubuntu22-vg-autotest_37811/output//cov_total.info.xz ./ubuntu22-vg-autotest_37811/output//dmesg.init.log ./ubuntu22-vg-autotest_37811/output//dmesg.log ./ubuntu22-vg-autotest_37811/output//power.tar.xz ./ubuntu22-vg-autotest_37811/output//test_completions.txt ./ubuntu22-vg-autotest_37811/output//timing.svg ./ubuntu22-vg-autotest_37811/output//timing.txt ./ubuntu22-vg-autotest_37811/output//ut_coverage ./ubuntu22-vg-autotest_37811/output//..
01:08:43.945  + rmdir ./ubuntu22-vg-autotest_37811/output/
01:08:43.945  + for dir in "${out_dirs[@]}"
01:08:43.945  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.945  + [[ -f ./unittest-vg-autotest_63375/output//doc.tar.xz ]]
01:08:43.945  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:43.945  + [[ -f ./unittest-vg-autotest_63375/output//ut_coverage.tar.xz ]]
01:08:43.945  + tar -C ./unittest-vg-autotest_63375/output/ -xf ./unittest-vg-autotest_63375/output//ut_coverage.tar.xz
01:08:44.202  + rm ./unittest-vg-autotest_63375/output//ut_coverage.tar.xz
01:08:44.202  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.202  + [[ -f ./unittest-vg-autotest_63375/output//llvm.tar.xz ]]
01:08:44.202  + mv ./unittest-vg-autotest_63375/output//all_tests.txt ./unittest-vg-autotest_63375/output//build-pkg-manifest.txt ./unittest-vg-autotest_63375/output//build-repo-manifest.txt ./unittest-vg-autotest_63375/output//cov_base.info.xz ./unittest-vg-autotest_63375/output//cov_test.info.xz ./unittest-vg-autotest_63375/output//cov_total.info.xz ./unittest-vg-autotest_63375/output//dmesg.init.log ./unittest-vg-autotest_63375/output//dmesg.log ./unittest-vg-autotest_63375/output//power.tar.xz ./unittest-vg-autotest_63375/output//test_completions.txt ./unittest-vg-autotest_63375/output//timing.svg ./unittest-vg-autotest_63375/output//timing.txt ./unittest-vg-autotest_63375/output//ut_coverage ./unittest-vg-autotest_63375/output//..
01:08:44.202  + rmdir ./unittest-vg-autotest_63375/output/
01:08:44.202  + for dir in "${out_dirs[@]}"
01:08:44.202  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.202  + [[ -f ./valgrind-vg-autotest_49192/output//doc.tar.xz ]]
01:08:44.203  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.203  + [[ -f ./valgrind-vg-autotest_49192/output//ut_coverage.tar.xz ]]
01:08:44.203  + tar -C ./valgrind-vg-autotest_49192/output/ -xf ./valgrind-vg-autotest_49192/output//ut_coverage.tar.xz
01:08:44.460  + rm ./valgrind-vg-autotest_49192/output//ut_coverage.tar.xz
01:08:44.460  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.460  + [[ -f ./valgrind-vg-autotest_49192/output//llvm.tar.xz ]]
01:08:44.460  + mv ./valgrind-vg-autotest_49192/output//all_tests.txt ./valgrind-vg-autotest_49192/output//build-pkg-manifest.txt ./valgrind-vg-autotest_49192/output//build-repo-manifest.txt ./valgrind-vg-autotest_49192/output//cov_base.info.xz ./valgrind-vg-autotest_49192/output//cov_test.info.xz ./valgrind-vg-autotest_49192/output//cov_total.info.xz ./valgrind-vg-autotest_49192/output//dmesg.init.log ./valgrind-vg-autotest_49192/output//dmesg.log ./valgrind-vg-autotest_49192/output//power.tar.xz ./valgrind-vg-autotest_49192/output//test_completions.txt ./valgrind-vg-autotest_49192/output//timing.svg ./valgrind-vg-autotest_49192/output//timing.txt ./valgrind-vg-autotest_49192/output//ut_coverage ./valgrind-vg-autotest_49192/output//..
01:08:44.460  + rmdir ./valgrind-vg-autotest_49192/output/
01:08:44.460  + for dir in "${out_dirs[@]}"
01:08:44.460  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.460  + [[ -f ./vfio-user-phy-autotest_26206/output//doc.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vfio-user-phy-autotest_26206/output//ut_coverage.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vfio-user-phy-autotest_26206/output//llvm.tar.xz ]]
01:08:44.461  + mv ./vfio-user-phy-autotest_26206/output//all_tests.txt ./vfio-user-phy-autotest_26206/output//build-pkg-manifest.txt ./vfio-user-phy-autotest_26206/output//build-repo-manifest.txt ./vfio-user-phy-autotest_26206/output//cov_base.info.xz ./vfio-user-phy-autotest_26206/output//cov_test.info.xz ./vfio-user-phy-autotest_26206/output//cov_total.info.xz ./vfio-user-phy-autotest_26206/output//dmesg.init.log ./vfio-user-phy-autotest_26206/output//dmesg.log ./vfio-user-phy-autotest_26206/output//power.tar.xz ./vfio-user-phy-autotest_26206/output//test_completions.txt ./vfio-user-phy-autotest_26206/output//timing.svg ./vfio-user-phy-autotest_26206/output//timing.txt ./vfio-user-phy-autotest_26206/output//..
01:08:44.461  + rmdir ./vfio-user-phy-autotest_26206/output/
01:08:44.461  + for dir in "${out_dirs[@]}"
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vhost-initiator-vg-autotest_77498/output//doc.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vhost-initiator-vg-autotest_77498/output//ut_coverage.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vhost-initiator-vg-autotest_77498/output//llvm.tar.xz ]]
01:08:44.461  + mv ./vhost-initiator-vg-autotest_77498/output//all_tests.txt ./vhost-initiator-vg-autotest_77498/output//build-pkg-manifest.txt ./vhost-initiator-vg-autotest_77498/output//build-repo-manifest.txt ./vhost-initiator-vg-autotest_77498/output//cov_base.info.xz ./vhost-initiator-vg-autotest_77498/output//cov_test.info.xz ./vhost-initiator-vg-autotest_77498/output//cov_total.info.xz ./vhost-initiator-vg-autotest_77498/output//dmesg.init.log ./vhost-initiator-vg-autotest_77498/output//dmesg.log ./vhost-initiator-vg-autotest_77498/output//power.tar.xz ./vhost-initiator-vg-autotest_77498/output//test_completions.txt ./vhost-initiator-vg-autotest_77498/output//timing.svg ./vhost-initiator-vg-autotest_77498/output//timing.txt ./vhost-initiator-vg-autotest_77498/output//vhost_fuzz_output1.txt.xz ./vhost-initiator-vg-autotest_77498/output//vhost_fuzz_output2.txt.xz ./vhost-initiator-vg-autotest_77498/output//vhost_fuzz_tgt_output.txt.xz ./vhost-initiator-vg-autotest_77498/output//..
01:08:44.461  + rmdir ./vhost-initiator-vg-autotest_77498/output/
01:08:44.461  + for dir in "${out_dirs[@]}"
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vhost-phy-autotest_26223/output//doc.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vhost-phy-autotest_26223/output//ut_coverage.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vhost-phy-autotest_26223/output//llvm.tar.xz ]]
01:08:44.461  + mv ./vhost-phy-autotest_26223/output//all_tests.txt ./vhost-phy-autotest_26223/output//build-pkg-manifest.txt ./vhost-phy-autotest_26223/output//build-repo-manifest.txt ./vhost-phy-autotest_26223/output//cov_base.info.xz ./vhost-phy-autotest_26223/output//cov_test.info.xz ./vhost-phy-autotest_26223/output//cov_total.info.xz ./vhost-phy-autotest_26223/output//dmesg.init.log ./vhost-phy-autotest_26223/output//dmesg.log ./vhost-phy-autotest_26223/output//nvmf_trace.0_shm.tar.gz ./vhost-phy-autotest_26223/output//power.tar.xz ./vhost-phy-autotest_26223/output//test_completions.txt ./vhost-phy-autotest_26223/output//timing.svg ./vhost-phy-autotest_26223/output//timing.txt ./vhost-phy-autotest_26223/output//..
01:08:44.461  + rmdir ./vhost-phy-autotest_26223/output/
01:08:44.461  + for dir in "${out_dirs[@]}"
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vmd-phy-autotest_2620/output//doc.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vmd-phy-autotest_2620/output//ut_coverage.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./vmd-phy-autotest_2620/output//llvm.tar.xz ]]
01:08:44.461  + mv ./vmd-phy-autotest_2620/output//all_tests.txt ./vmd-phy-autotest_2620/output//build-pkg-manifest.txt ./vmd-phy-autotest_2620/output//build-repo-manifest.txt ./vmd-phy-autotest_2620/output//cov_base.info.xz ./vmd-phy-autotest_2620/output//cov_test.info.xz ./vmd-phy-autotest_2620/output//cov_total.info.xz ./vmd-phy-autotest_2620/output//dmesg.init.log ./vmd-phy-autotest_2620/output//dmesg.log ./vmd-phy-autotest_2620/output//power.tar.xz ./vmd-phy-autotest_2620/output//test_completions.txt ./vmd-phy-autotest_2620/output//timing.svg ./vmd-phy-autotest_2620/output//timing.txt ./vmd-phy-autotest_2620/output//..
01:08:44.461  + rmdir ./vmd-phy-autotest_2620/output/
01:08:44.461  + for dir in "${out_dirs[@]}"
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./zns-vg-autotest_76584/output//doc.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./zns-vg-autotest_76584/output//ut_coverage.tar.xz ]]
01:08:44.461  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
01:08:44.461  + [[ -f ./zns-vg-autotest_76584/output//llvm.tar.xz ]]
01:08:44.461  + mv ./zns-vg-autotest_76584/output//all_tests.txt ./zns-vg-autotest_76584/output//build-pkg-manifest.txt ./zns-vg-autotest_76584/output//build-repo-manifest.txt ./zns-vg-autotest_76584/output//cov_base.info.xz ./zns-vg-autotest_76584/output//cov_test.info.xz ./zns-vg-autotest_76584/output//cov_total.info.xz ./zns-vg-autotest_76584/output//dmesg.init.log ./zns-vg-autotest_76584/output//dmesg.log ./zns-vg-autotest_76584/output//power.tar.xz ./zns-vg-autotest_76584/output//test_completions.txt ./zns-vg-autotest_76584/output//timing.svg ./zns-vg-autotest_76584/output//timing.txt ./zns-vg-autotest_76584/output//..
01:08:44.461  + rmdir ./zns-vg-autotest_76584/output/
01:08:44.461  + unpack_cov_files
01:08:44.461  + local info_files
01:08:44.461  + info_files=(*/cov_*.info.xz)
01:08:44.461  + xargs -P0 -r -n1 xz -d
01:08:44.461  + printf '%s\n' BlobFS-phy-autotest_26204/cov_base.info.xz BlobFS-phy-autotest_26204/cov_test.info.xz BlobFS-phy-autotest_26204/cov_total.info.xz centos7-vg-autotest_73657/cov_base.info.xz centos7-vg-autotest_73657/cov_test.info.xz centos7-vg-autotest_73657/cov_total.info.xz crypto-phy-autotest_26198/cov_base.info.xz crypto-phy-autotest_26198/cov_test.info.xz crypto-phy-autotest_26198/cov_total.info.xz dsa-phy-autotest_4898/cov_base.info.xz dsa-phy-autotest_4898/cov_test.info.xz dsa-phy-autotest_4898/cov_total.info.xz iscsi-uring-vg-autotest_77383/cov_base.info.xz iscsi-uring-vg-autotest_77383/cov_test.info.xz iscsi-uring-vg-autotest_77383/cov_total.info.xz iscsi-vg-autotest_77516/cov_base.info.xz iscsi-vg-autotest_77516/cov_test.info.xz iscsi-vg-autotest_77516/cov_total.info.xz lvol-vg-autotest_77427/cov_base.info.xz lvol-vg-autotest_77427/cov_test.info.xz lvol-vg-autotest_77427/cov_total.info.xz nvme-cmb-pmr-vg-autotest_78601/cov_base.info.xz nvme-cmb-pmr-vg-autotest_78601/cov_test.info.xz nvme-cmb-pmr-vg-autotest_78601/cov_total.info.xz nvme-phy-autotest_72370/cov_base.info.xz nvme-vg-autotest_77478/cov_base.info.xz nvmf-phy-autotest_72442/cov_base.info.xz nvmf-tcp-phy-autotest_72329/cov_base.info.xz nvmf-tcp-phy-autotest_72329/cov_test.info.xz nvmf-tcp-phy-autotest_72329/cov_total.info.xz nvmf-tcp-uring-vg-autotest_77403/cov_base.info.xz nvmf-tcp-uring-vg-autotest_77403/cov_test.info.xz nvmf-tcp-uring-vg-autotest_77403/cov_total.info.xz nvmf-tcp-vg-autotest_77469/cov_base.info.xz nvmf-tcp-vg-autotest_77469/cov_test.info.xz nvmf-tcp-vg-autotest_77469/cov_total.info.xz rocky9-vg-autotest_16088/cov_base.info.xz rocky9-vg-autotest_16088/cov_test.info.xz rocky9-vg-autotest_16088/cov_total.info.xz ubuntu20-vg-autotest_73797/cov_base.info.xz ubuntu22-vg-autotest_37811/cov_base.info.xz ubuntu22-vg-autotest_37811/cov_test.info.xz ubuntu22-vg-autotest_37811/cov_total.info.xz unittest-vg-autotest_63375/cov_base.info.xz unittest-vg-autotest_63375/cov_test.info.xz unittest-vg-autotest_63375/cov_total.info.xz valgrind-vg-autotest_49192/cov_base.info.xz valgrind-vg-autotest_49192/cov_test.info.xz valgrind-vg-autotest_49192/cov_total.info.xz vfio-user-phy-autotest_26206/cov_base.info.xz vfio-user-phy-autotest_26206/cov_test.info.xz vfio-user-phy-autotest_26206/cov_total.info.xz vhost-initiator-vg-autotest_77498/cov_base.info.xz vhost-initiator-vg-autotest_77498/cov_test.info.xz vhost-initiator-vg-autotest_77498/cov_total.info.xz vhost-phy-autotest_26223/cov_base.info.xz vhost-phy-autotest_26223/cov_test.info.xz vhost-phy-autotest_26223/cov_total.info.xz vmd-phy-autotest_2620/cov_base.info.xz vmd-phy-autotest_2620/cov_test.info.xz vmd-phy-autotest_2620/cov_total.info.xz zns-vg-autotest_76584/cov_base.info.xz zns-vg-autotest_76584/cov_test.info.xz zns-vg-autotest_76584/cov_total.info.xz
01:08:45.027  + fix_downstream_job_paths
01:08:45.027  + sed -i -e 's#^SF:/.\+/spdk/#SF:/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/#g' BlobFS-phy-autotest_26204/cov_total.info centos7-vg-autotest_73657/cov_total.info crypto-phy-autotest_26198/cov_total.info dsa-phy-autotest_4898/cov_total.info iscsi-uring-vg-autotest_77383/cov_total.info iscsi-vg-autotest_77516/cov_total.info lvol-vg-autotest_77427/cov_total.info nvme-cmb-pmr-vg-autotest_78601/cov_total.info nvmf-tcp-phy-autotest_72329/cov_total.info nvmf-tcp-uring-vg-autotest_77403/cov_total.info nvmf-tcp-vg-autotest_77469/cov_total.info rocky9-vg-autotest_16088/cov_total.info ubuntu22-vg-autotest_37811/cov_total.info unittest-vg-autotest_63375/cov_total.info valgrind-vg-autotest_49192/cov_total.info vfio-user-phy-autotest_26206/cov_total.info vhost-initiator-vg-autotest_77498/cov_total.info vhost-phy-autotest_26223/cov_total.info vmd-phy-autotest_2620/cov_total.info zns-vg-autotest_76584/cov_total.info
01:08:45.962  + ignore_ocf_headers
01:08:45.962  + sed -i 's#--show-details#--show-details --ignore-errors source#g' spdk/autorun_post.py
01:08:45.962  + create_errtrace_files_for_all_builds
01:08:45.962  + local build_log_list
01:08:45.962  + build_log_list=($(find "$WORKSPACE" -type f -name 'build.log'))
01:08:45.962  ++ find /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11 -type f -name build.log
01:08:45.962  + for build in "${build_log_list[@]}"
01:08:45.962  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/build.log
01:08:45.962  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.962  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/build.log
01:08:45.962  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/build.log
01:08:45.962  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498
01:08:45.962  + job_name=vhost-initiator-vg-autotest_77498
01:08:45.962  + start_pattern='Backtrace start'
01:08:45.962  + end_pattern='Backtrace end'
01:08:45.962  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/build.log ]]
01:08:45.962  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/build.log
01:08:45.962  + backtraces=
01:08:45.962  + [[ -n '' ]]
01:08:45.962  + for build in "${build_log_list[@]}"
01:08:45.962  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/build.log
01:08:45.962  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.962  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/build.log
01:08:45.962  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/build.log
01:08:45.962  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206
01:08:45.962  + job_name=vfio-user-phy-autotest_26206
01:08:45.962  + start_pattern='Backtrace start'
01:08:45.962  + end_pattern='Backtrace end'
01:08:45.962  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/build.log ]]
01:08:45.962  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/build.log
01:08:45.962  + backtraces=
01:08:45.962  + [[ -n '' ]]
01:08:45.962  + for build in "${build_log_list[@]}"
01:08:45.962  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/build.log
01:08:45.962  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.962  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/build.log
01:08:45.962  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/build.log
01:08:45.962  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626
01:08:45.962  + job_name=clang-vg-autotest_57626
01:08:45.962  + start_pattern='Backtrace start'
01:08:45.962  + end_pattern='Backtrace end'
01:08:45.962  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/build.log ]]
01:08:45.962  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/build.log
01:08:45.962  + backtraces=
01:08:45.962  + [[ -n '' ]]
01:08:45.962  + for build in "${build_log_list[@]}"
01:08:45.962  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/build.log
01:08:45.962  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.962  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/build.log
01:08:45.962  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/build.log
01:08:45.962  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620
01:08:45.962  + job_name=vmd-phy-autotest_2620
01:08:45.962  + start_pattern='Backtrace start'
01:08:45.962  + end_pattern='Backtrace end'
01:08:45.962  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/build.log ]]
01:08:45.962  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/build.log
01:08:45.962  + backtraces=
01:08:45.962  + [[ -n '' ]]
01:08:45.962  + for build in "${build_log_list[@]}"
01:08:45.962  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/build.log
01:08:45.962  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.962  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/build.log
01:08:45.963  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/build.log
01:08:45.963  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204
01:08:45.963  + job_name=BlobFS-phy-autotest_26204
01:08:45.963  + start_pattern='Backtrace start'
01:08:45.963  + end_pattern='Backtrace end'
01:08:45.963  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/build.log ]]
01:08:45.963  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/build.log
01:08:45.963  + backtraces=
01:08:45.963  + [[ -n '' ]]
01:08:45.963  + for build in "${build_log_list[@]}"
01:08:45.963  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/build.log
01:08:45.963  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.963  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/build.log
01:08:45.963  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/build.log
01:08:45.963  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811
01:08:45.963  + job_name=ubuntu22-vg-autotest_37811
01:08:45.963  + start_pattern='Backtrace start'
01:08:45.963  + end_pattern='Backtrace end'
01:08:45.963  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/build.log ]]
01:08:45.963  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/build.log
01:08:45.963  + backtraces=
01:08:45.963  + [[ -n '' ]]
01:08:45.963  + for build in "${build_log_list[@]}"
01:08:45.963  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/build.log
01:08:45.963  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.963  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/build.log
01:08:45.963  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/build.log
01:08:45.963  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198
01:08:45.963  + job_name=crypto-phy-autotest_26198
01:08:45.963  + start_pattern='Backtrace start'
01:08:45.963  + end_pattern='Backtrace end'
01:08:45.963  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/build.log ]]
01:08:45.963  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/build.log
01:08:45.963  + backtraces=
01:08:45.963  + [[ -n '' ]]
01:08:45.963  + for build in "${build_log_list[@]}"
01:08:45.963  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/build.log
01:08:45.963  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.963  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/build.log
01:08:45.963  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/build.log
01:08:45.963  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418
01:08:45.963  + job_name=autobuild-vg-autotest_63418
01:08:45.963  + start_pattern='Backtrace start'
01:08:45.963  + end_pattern='Backtrace end'
01:08:45.963  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/build.log ]]
01:08:45.963  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/build.log
01:08:45.963  + backtraces=
01:08:45.963  + [[ -n '' ]]
01:08:45.963  + for build in "${build_log_list[@]}"
01:08:45.963  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/build.log
01:08:45.963  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.963  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/build.log
01:08:45.963  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/build.log
01:08:45.963  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370
01:08:45.963  + job_name=nvme-phy-autotest_72370
01:08:45.963  + start_pattern='Backtrace start'
01:08:45.963  + end_pattern='Backtrace end'
01:08:45.963  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/build.log ]]
01:08:45.963  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/build.log
01:08:45.963  + backtraces='00:14:37.161    10:08:21 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ==========
01:08:45.963  00:14:37.161  
01:08:45.963  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161     710 	
01:08:45.963  00:14:37.161     711 		# We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to
01:08:45.963  00:14:37.161     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:45.963  00:14:37.161     713 		# but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do:
01:08:45.963  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.963  00:14:37.161  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.963  00:14:37.161     716 	
01:08:45.963  00:14:37.161     717 		echo "$time"
01:08:45.963  00:14:37.161     718 	)
01:08:45.963  00:14:37.161     719 	
01:08:45.963  00:14:37.161     720 	function timing_enter() {
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161     11  	# Pci bus hotplug
01:08:45.963  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.963  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.963  00:14:37.161     14  		local helper_time=0
01:08:45.963  00:14:37.161     15  	
01:08:45.963  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.963  00:14:37.161     17  		printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \
01:08:45.963  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.963  00:14:37.161     19  	}
01:08:45.963  00:14:37.161     20  	
01:08:45.963  00:14:37.161     21  	remove_attach_helper() {
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.963  00:14:37.161     106 		done
01:08:45.963  00:14:37.161     107 	
01:08:45.963  00:14:37.161     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.963  00:14:37.161     109 	
01:08:45.963  00:14:37.161  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.963  00:14:37.161     111 		# Verify reregistering hotplug poller
01:08:45.963  00:14:37.161     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.963  00:14:37.161     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.963  00:14:37.161     114 	
01:08:45.963  00:14:37.161     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161     133 	
01:08:45.963  00:14:37.161     134 	# Run pci bus hotplug test
01:08:45.963  00:14:37.161     135 	run_hotplug
01:08:45.963  00:14:37.161     136 	
01:08:45.963  00:14:37.161     137 	# Run SPDK target based hotplug
01:08:45.963  00:14:37.161  => 138 	tgt_run_hotplug
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161  
01:08:45.963  00:14:37.161  ========== Backtrace end =========='\''
01:08:45.963  00:14:37.161  ========== Backtrace start: ==========
01:08:45.963  00:14:37.161  
01:08:45.963  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161     710 	
01:08:45.963  00:14:37.161     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:45.963  00:14:37.161     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:45.963  00:14:37.161     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:45.963  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.963  00:14:37.161  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.963  00:14:37.161     716 	
01:08:45.963  00:14:37.161     717 		echo "$time"
01:08:45.963  00:14:37.161     718 	)
01:08:45.963  00:14:37.161     719 	
01:08:45.963  00:14:37.161     720 	function timing_enter() {
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161     11  	# Pci bus hotplug
01:08:45.963  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.963  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.963  00:14:37.161     14  		local helper_time=0
01:08:45.963  00:14:37.161     15  	
01:08:45.963  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.963  00:14:37.161     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:45.963  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.963  00:14:37.161     19  	}
01:08:45.963  00:14:37.161     20  	
01:08:45.963  00:14:37.161     21  	remove_attach_helper() {
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.963  00:14:37.161     106 		done
01:08:45.963  00:14:37.161     107 	
01:08:45.963  00:14:37.161     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.963  00:14:37.161     109 	
01:08:45.963  00:14:37.161  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.963  00:14:37.161     111 		# Verify reregistering hotplug poller
01:08:45.963  00:14:37.161     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.963  00:14:37.161     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.963  00:14:37.161     114 	
01:08:45.963  00:14:37.161     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.963  00:14:37.161       ...
01:08:45.963  00:14:37.161     133 	
01:08:45.963  00:14:37.161     134 	# Run pci bus hotplug test
01:08:45.963  00:14:37.161     135 	run_hotplug
01:08:45.963  00:14:37.161     136 	
01:08:45.963  00:14:37.161     137 	# Run SPDK target based hotplug
01:08:45.963  00:14:37.161  => 138 	tgt_run_hotplug
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161  
01:08:45.964  00:14:37.161  ========== Backtrace end ==========
01:08:45.964  00:14:37.161  ========== Backtrace start: ==========
01:08:45.964  00:14:37.161  
01:08:45.964  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:45.964  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.964  00:14:37.161     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.964  00:14:37.161     716 	
01:08:45.964  00:14:37.161     717 		echo "$time"
01:08:45.964  00:14:37.161  => 718 	)
01:08:45.964  00:14:37.161     719 	
01:08:45.964  00:14:37.161     720 	function timing_enter() {
01:08:45.964  00:14:37.161     721 		xtrace_disable
01:08:45.964  00:14:37.161     722 		timing "enter" "$1"
01:08:45.964  00:14:37.161     723 		xtrace_restore
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161     11  	# Pci bus hotplug
01:08:45.964  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.964  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.964  00:14:37.161     14  		local helper_time=0
01:08:45.964  00:14:37.161     15  	
01:08:45.964  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.964  00:14:37.161     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:45.964  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.964  00:14:37.161     19  	}
01:08:45.964  00:14:37.161     20  	
01:08:45.964  00:14:37.161     21  	remove_attach_helper() {
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.964  00:14:37.162     106 		done
01:08:45.964  00:14:37.162     107 	
01:08:45.964  00:14:37.162     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.964  00:14:37.162     109 	
01:08:45.964  00:14:37.162  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.964  00:14:37.162     111 		# Verify reregistering hotplug poller
01:08:45.964  00:14:37.162     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.964  00:14:37.162     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.964  00:14:37.162     114 	
01:08:45.964  00:14:37.162     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162     133 	
01:08:45.964  00:14:37.162     134 	# Run pci bus hotplug test
01:08:45.964  00:14:37.162     135 	run_hotplug
01:08:45.964  00:14:37.162     136 	
01:08:45.964  00:14:37.162     137 	# Run SPDK target based hotplug
01:08:45.964  00:14:37.162  => 138 	tgt_run_hotplug
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162  
01:08:45.964  00:14:37.162  ========== Backtrace end ==========
01:08:45.964  00:14:37.162  ========== Backtrace start: ==========
01:08:45.964  00:14:37.162  
01:08:45.964  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162     11  	# Pci bus hotplug
01:08:45.964  00:14:37.162     12  	# Helper function to remove/attach cotrollers
01:08:45.964  00:14:37.162     13  	debug_remove_attach_helper() {
01:08:45.964  00:14:37.162     14  		local helper_time=0
01:08:45.964  00:14:37.162     15  	
01:08:45.964  00:14:37.162  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.964  00:14:37.162     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:45.964  00:14:37.162     18  			"$helper_time" "$nvme_count" >&2
01:08:45.964  00:14:37.162     19  	}
01:08:45.964  00:14:37.162     20  	
01:08:45.964  00:14:37.162     21  	remove_attach_helper() {
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.964  00:14:37.162     106 		done
01:08:45.964  00:14:37.162     107 	
01:08:45.964  00:14:37.162     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.964  00:14:37.162     109 	
01:08:45.964  00:14:37.162  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.964  00:14:37.162     111 		# Verify reregistering hotplug poller
01:08:45.964  00:14:37.162     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.964  00:14:37.162     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.964  00:14:37.162     114 	
01:08:45.964  00:14:37.162     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162     133 	
01:08:45.964  00:14:37.162     134 	# Run pci bus hotplug test
01:08:45.964  00:14:37.162     135 	run_hotplug
01:08:45.964  00:14:37.162     136 	
01:08:45.964  00:14:37.162     137 	# Run SPDK target based hotplug
01:08:45.964  00:14:37.162  => 138 	tgt_run_hotplug
01:08:45.964  00:14:37.162       ...
01:08:45.964  00:14:37.162  
01:08:45.964  00:14:37.162  ========== Backtrace end ==========
01:08:45.964  00:14:37.421  ========== Backtrace start: ==========
01:08:45.964  00:14:37.421  
01:08:45.964  00:14:37.421  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh"])
01:08:45.964  00:14:37.421       ...
01:08:45.964  00:14:37.421     1117		timing_enter $test_name
01:08:45.964  00:14:37.421     1118		echo "************************************"
01:08:45.964  00:14:37.421     1119		echo "START TEST $test_name"
01:08:45.964  00:14:37.421     1120		echo "************************************"
01:08:45.964  00:14:37.421     1121		xtrace_restore
01:08:45.964  00:14:37.421     1122		time "$@"
01:08:45.964  00:14:37.421     1123		xtrace_disable
01:08:45.964  00:14:37.421     1124		echo "************************************"
01:08:45.964  00:14:37.421     1125		echo "END TEST $test_name"
01:08:45.964  00:14:37.421     1126		echo "************************************"
01:08:45.964  00:14:37.421     1127		timing_exit $test_name
01:08:45.964  00:14:37.421       ...
01:08:45.964  00:14:37.421  in /var/jenkins/workspace/nvme-phy-autotest/spdk/autotest.sh:240 -> main(["/var/jenkins/workspace/nvme-phy-autotest/autorun-spdk.conf"])
01:08:45.964  00:14:37.421       ...
01:08:45.964  00:14:37.421     235 	
01:08:45.964  00:14:37.421     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:45.964  00:14:37.421     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:45.964  00:14:37.421     238 	
01:08:45.964  00:14:37.421     239 			if [ $(uname -s) = Linux ]; then
01:08:45.964  00:14:37.421  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:45.964  00:14:37.421     241 			fi
01:08:45.964  00:14:37.421     242 	
01:08:45.964  00:14:37.421     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:45.964  00:14:37.421     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:45.964  00:14:37.421     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:45.964  00:14:37.421       ...
01:08:45.964  00:14:37.421  
01:08:45.964  00:14:37.421  ========== Backtrace end =========='
01:08:45.964  + [[ -n 00:14:37.161    10:08:21 sw_hotplug -- common/autotest_common.sh@715 -- # time='========== Backtrace start: ==========
01:08:45.964  00:14:37.161  
01:08:45.964  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161     710 	
01:08:45.964  00:14:37.161     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:45.964  00:14:37.161     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:45.964  00:14:37.161     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:45.964  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.964  00:14:37.161  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.964  00:14:37.161     716 	
01:08:45.964  00:14:37.161     717 		echo "$time"
01:08:45.964  00:14:37.161     718 	)
01:08:45.964  00:14:37.161     719 	
01:08:45.964  00:14:37.161     720 	function timing_enter() {
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161     11  	# Pci bus hotplug
01:08:45.964  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.964  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.964  00:14:37.161     14  		local helper_time=0
01:08:45.964  00:14:37.161     15  	
01:08:45.964  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.964  00:14:37.161     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:45.964  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.964  00:14:37.161     19  	}
01:08:45.964  00:14:37.161     20  	
01:08:45.964  00:14:37.161     21  	remove_attach_helper() {
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.964  00:14:37.161       ...
01:08:45.964  00:14:37.161     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.964  00:14:37.161     106 		done
01:08:45.964  00:14:37.161     107 	
01:08:45.964  00:14:37.161     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.964  00:14:37.161     109 	
01:08:45.964  00:14:37.161  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.964  00:14:37.161     111 		# Verify reregistering hotplug poller
01:08:45.964  00:14:37.161     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.964  00:14:37.161     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.964  00:14:37.161     114 	
01:08:45.964  00:14:37.161     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.964  00:14:37.161       ...
01:08:45.965  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161     133 	
01:08:45.965  00:14:37.161     134 	# Run pci bus hotplug test
01:08:45.965  00:14:37.161     135 	run_hotplug
01:08:45.965  00:14:37.161     136 	
01:08:45.965  00:14:37.161     137 	# Run SPDK target based hotplug
01:08:45.965  00:14:37.161  => 138 	tgt_run_hotplug
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161  
01:08:45.965  00:14:37.161  ========== Backtrace end =========='
01:08:45.965  00:14:37.161  ========== Backtrace start: ==========
01:08:45.965  00:14:37.161  
01:08:45.965  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161     710 	
01:08:45.965  00:14:37.161     711 		# We redirect cmd's std{out,err} to a separate fd dup'ed to stdin's device (or /dev/null) to
01:08:45.965  00:14:37.161     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:45.965  00:14:37.161     713 		# but $() will return just the time's data, hence making it possible to just do:
01:08:45.965  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.965  00:14:37.161  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.965  00:14:37.161     716 	
01:08:45.965  00:14:37.161     717 		echo "$time"
01:08:45.965  00:14:37.161     718 	)
01:08:45.965  00:14:37.161     719 	
01:08:45.965  00:14:37.161     720 	function timing_enter() {
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161     11  	# Pci bus hotplug
01:08:45.965  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.965  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.965  00:14:37.161     14  		local helper_time=0
01:08:45.965  00:14:37.161     15  	
01:08:45.965  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.965  00:14:37.161     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:45.965  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.965  00:14:37.161     19  	}
01:08:45.965  00:14:37.161     20  	
01:08:45.965  00:14:37.161     21  	remove_attach_helper() {
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.965  00:14:37.161     106 		done
01:08:45.965  00:14:37.161     107 	
01:08:45.965  00:14:37.161     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.965  00:14:37.161     109 	
01:08:45.965  00:14:37.161  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.965  00:14:37.161     111 		# Verify reregistering hotplug poller
01:08:45.965  00:14:37.161     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.965  00:14:37.161     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.965  00:14:37.161     114 	
01:08:45.965  00:14:37.161     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161     133 	
01:08:45.965  00:14:37.161     134 	# Run pci bus hotplug test
01:08:45.965  00:14:37.161     135 	run_hotplug
01:08:45.965  00:14:37.161     136 	
01:08:45.965  00:14:37.161     137 	# Run SPDK target based hotplug
01:08:45.965  00:14:37.161  => 138 	tgt_run_hotplug
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161  
01:08:45.965  00:14:37.161  ========== Backtrace end ==========
01:08:45.965  00:14:37.161  ========== Backtrace start: ==========
01:08:45.965  00:14:37.161  
01:08:45.965  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161     713 		# but $() will return just the time's data, hence making it possible to just do:
01:08:45.965  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.965  00:14:37.161     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.965  00:14:37.161     716 	
01:08:45.965  00:14:37.161     717 		echo "$time"
01:08:45.965  00:14:37.161  => 718 	)
01:08:45.965  00:14:37.161     719 	
01:08:45.965  00:14:37.161     720 	function timing_enter() {
01:08:45.965  00:14:37.161     721 		xtrace_disable
01:08:45.965  00:14:37.161     722 		timing "enter" "$1"
01:08:45.965  00:14:37.161     723 		xtrace_restore
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161     11  	# Pci bus hotplug
01:08:45.965  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.965  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.965  00:14:37.161     14  		local helper_time=0
01:08:45.965  00:14:37.161     15  	
01:08:45.965  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.965  00:14:37.161     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:45.965  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.965  00:14:37.161     19  	}
01:08:45.965  00:14:37.161     20  	
01:08:45.965  00:14:37.161     21  	remove_attach_helper() {
01:08:45.965  00:14:37.161       ...
01:08:45.965  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.965  00:14:37.162     106 		done
01:08:45.965  00:14:37.162     107 	
01:08:45.965  00:14:37.162     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.965  00:14:37.162     109 	
01:08:45.965  00:14:37.162  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.965  00:14:37.162     111 		# Verify reregistering hotplug poller
01:08:45.965  00:14:37.162     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.965  00:14:37.162     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.965  00:14:37.162     114 	
01:08:45.965  00:14:37.162     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162     133 	
01:08:45.965  00:14:37.162     134 	# Run pci bus hotplug test
01:08:45.965  00:14:37.162     135 	run_hotplug
01:08:45.965  00:14:37.162     136 	
01:08:45.965  00:14:37.162     137 	# Run SPDK target based hotplug
01:08:45.965  00:14:37.162  => 138 	tgt_run_hotplug
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162  
01:08:45.965  00:14:37.162  ========== Backtrace end ==========
01:08:45.965  00:14:37.162  ========== Backtrace start: ==========
01:08:45.965  00:14:37.162  
01:08:45.965  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162     11  	# Pci bus hotplug
01:08:45.965  00:14:37.162     12  	# Helper function to remove/attach cotrollers
01:08:45.965  00:14:37.162     13  	debug_remove_attach_helper() {
01:08:45.965  00:14:37.162     14  		local helper_time=0
01:08:45.965  00:14:37.162     15  	
01:08:45.965  00:14:37.162  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.965  00:14:37.162     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:45.965  00:14:37.162     18  			"$helper_time" "$nvme_count" >&2
01:08:45.965  00:14:37.162     19  	}
01:08:45.965  00:14:37.162     20  	
01:08:45.965  00:14:37.162     21  	remove_attach_helper() {
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.965  00:14:37.162     106 		done
01:08:45.965  00:14:37.162     107 	
01:08:45.965  00:14:37.162     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.965  00:14:37.162     109 	
01:08:45.965  00:14:37.162  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.965  00:14:37.162     111 		# Verify reregistering hotplug poller
01:08:45.965  00:14:37.162     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.965  00:14:37.162     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.965  00:14:37.162     114 	
01:08:45.965  00:14:37.162     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162     133 	
01:08:45.965  00:14:37.162     134 	# Run pci bus hotplug test
01:08:45.965  00:14:37.162     135 	run_hotplug
01:08:45.965  00:14:37.162     136 	
01:08:45.965  00:14:37.162     137 	# Run SPDK target based hotplug
01:08:45.965  00:14:37.162  => 138 	tgt_run_hotplug
01:08:45.965  00:14:37.162       ...
01:08:45.965  00:14:37.162  
01:08:45.965  00:14:37.162  ========== Backtrace end ==========
01:08:45.965  00:14:37.421  ========== Backtrace start: ==========
01:08:45.965  00:14:37.421  
01:08:45.965  00:14:37.421  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh"])
01:08:45.965  00:14:37.421       ...
01:08:45.965  00:14:37.421     1117		timing_enter $test_name
01:08:45.965  00:14:37.421     1118		echo "************************************"
01:08:45.965  00:14:37.421     1119		echo "START TEST $test_name"
01:08:45.965  00:14:37.421     1120		echo "************************************"
01:08:45.965  00:14:37.421     1121		xtrace_restore
01:08:45.965  00:14:37.421     1122		time "$@"
01:08:45.965  00:14:37.421     1123		xtrace_disable
01:08:45.965  00:14:37.421     1124		echo "************************************"
01:08:45.965  00:14:37.421     1125		echo "END TEST $test_name"
01:08:45.965  00:14:37.421     1126		echo "************************************"
01:08:45.965  00:14:37.421     1127		timing_exit $test_name
01:08:45.965  00:14:37.421       ...
01:08:45.965  00:14:37.421  in /var/jenkins/workspace/nvme-phy-autotest/spdk/autotest.sh:240 -> main(["/var/jenkins/workspace/nvme-phy-autotest/autorun-spdk.conf"])
01:08:45.965  00:14:37.421       ...
01:08:45.966  00:14:37.421     235 	
01:08:45.966  00:14:37.421     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:45.966  00:14:37.421     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:45.966  00:14:37.421     238 	
01:08:45.966  00:14:37.421     239 			if [ $(uname -s) = Linux ]; then
01:08:45.966  00:14:37.421  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:45.966  00:14:37.421     241 			fi
01:08:45.966  00:14:37.421     242 	
01:08:45.966  00:14:37.421     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:45.966  00:14:37.421     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:45.966  00:14:37.421     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:45.966  00:14:37.421       ...
01:08:45.966  00:14:37.421  
01:08:45.966  00:14:37.421  ========== Backtrace end ========== ]]
01:08:45.966  + echo '00:14:37.161    10:08:21 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ==========
01:08:45.966  00:14:37.161  
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     710 	
01:08:45.966  00:14:37.161     711 		# We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to
01:08:45.966  00:14:37.161     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:45.966  00:14:37.161     713 		# but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do:
01:08:45.966  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.966  00:14:37.161  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.966  00:14:37.161     716 	
01:08:45.966  00:14:37.161     717 		echo "$time"
01:08:45.966  00:14:37.161     718 	)
01:08:45.966  00:14:37.161     719 	
01:08:45.966  00:14:37.161     720 	function timing_enter() {
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     11  	# Pci bus hotplug
01:08:45.966  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.966  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.966  00:14:37.161     14  		local helper_time=0
01:08:45.966  00:14:37.161     15  	
01:08:45.966  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.966  00:14:37.161     17  		printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \
01:08:45.966  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.966  00:14:37.161     19  	}
01:08:45.966  00:14:37.161     20  	
01:08:45.966  00:14:37.161     21  	remove_attach_helper() {
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.966  00:14:37.161     106 		done
01:08:45.966  00:14:37.161     107 	
01:08:45.966  00:14:37.161     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.966  00:14:37.161     109 	
01:08:45.966  00:14:37.161  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.966  00:14:37.161     111 		# Verify reregistering hotplug poller
01:08:45.966  00:14:37.161     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.966  00:14:37.161     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.966  00:14:37.161     114 	
01:08:45.966  00:14:37.161     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     133 	
01:08:45.966  00:14:37.161     134 	# Run pci bus hotplug test
01:08:45.966  00:14:37.161     135 	run_hotplug
01:08:45.966  00:14:37.161     136 	
01:08:45.966  00:14:37.161     137 	# Run SPDK target based hotplug
01:08:45.966  00:14:37.161  => 138 	tgt_run_hotplug
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  
01:08:45.966  00:14:37.161  ========== Backtrace end =========='\''
01:08:45.966  00:14:37.161  ========== Backtrace start: ==========
01:08:45.966  00:14:37.161  
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     710 	
01:08:45.966  00:14:37.161     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:45.966  00:14:37.161     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:45.966  00:14:37.161     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:45.966  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.966  00:14:37.161  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.966  00:14:37.161     716 	
01:08:45.966  00:14:37.161     717 		echo "$time"
01:08:45.966  00:14:37.161     718 	)
01:08:45.966  00:14:37.161     719 	
01:08:45.966  00:14:37.161     720 	function timing_enter() {
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     11  	# Pci bus hotplug
01:08:45.966  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.966  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.966  00:14:37.161     14  		local helper_time=0
01:08:45.966  00:14:37.161     15  	
01:08:45.966  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.966  00:14:37.161     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:45.966  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.966  00:14:37.161     19  	}
01:08:45.966  00:14:37.161     20  	
01:08:45.966  00:14:37.161     21  	remove_attach_helper() {
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.966  00:14:37.161     106 		done
01:08:45.966  00:14:37.161     107 	
01:08:45.966  00:14:37.161     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.966  00:14:37.161     109 	
01:08:45.966  00:14:37.161  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.966  00:14:37.161     111 		# Verify reregistering hotplug poller
01:08:45.966  00:14:37.161     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.966  00:14:37.161     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.966  00:14:37.161     114 	
01:08:45.966  00:14:37.161     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     133 	
01:08:45.966  00:14:37.161     134 	# Run pci bus hotplug test
01:08:45.966  00:14:37.161     135 	run_hotplug
01:08:45.966  00:14:37.161     136 	
01:08:45.966  00:14:37.161     137 	# Run SPDK target based hotplug
01:08:45.966  00:14:37.161  => 138 	tgt_run_hotplug
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  
01:08:45.966  00:14:37.161  ========== Backtrace end ==========
01:08:45.966  00:14:37.161  ========== Backtrace start: ==========
01:08:45.966  00:14:37.161  
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:45.966  00:14:37.161     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:45.966  00:14:37.161     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:45.966  00:14:37.161     716 	
01:08:45.966  00:14:37.161     717 		echo "$time"
01:08:45.966  00:14:37.161  => 718 	)
01:08:45.966  00:14:37.161     719 	
01:08:45.966  00:14:37.161     720 	function timing_enter() {
01:08:45.966  00:14:37.161     721 		xtrace_disable
01:08:45.966  00:14:37.161     722 		timing "enter" "$1"
01:08:45.966  00:14:37.161     723 		xtrace_restore
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161     11  	# Pci bus hotplug
01:08:45.966  00:14:37.161     12  	# Helper function to remove/attach cotrollers
01:08:45.966  00:14:37.161     13  	debug_remove_attach_helper() {
01:08:45.966  00:14:37.161     14  		local helper_time=0
01:08:45.966  00:14:37.161     15  	
01:08:45.966  00:14:37.161  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.966  00:14:37.161     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:45.966  00:14:37.161     18  			"$helper_time" "$nvme_count" >&2
01:08:45.966  00:14:37.161     19  	}
01:08:45.966  00:14:37.161     20  	
01:08:45.966  00:14:37.161     21  	remove_attach_helper() {
01:08:45.966  00:14:37.161       ...
01:08:45.966  00:14:37.161  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.966  00:14:37.162       ...
01:08:45.966  00:14:37.162     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.966  00:14:37.162     106 		done
01:08:45.966  00:14:37.162     107 	
01:08:45.966  00:14:37.162     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.966  00:14:37.162     109 	
01:08:45.966  00:14:37.162  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.966  00:14:37.162     111 		# Verify reregistering hotplug poller
01:08:45.966  00:14:37.162     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.966  00:14:37.162     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.966  00:14:37.162     114 	
01:08:45.966  00:14:37.162     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.966  00:14:37.162       ...
01:08:45.966  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.966  00:14:37.162       ...
01:08:45.966  00:14:37.162     133 	
01:08:45.966  00:14:37.162     134 	# Run pci bus hotplug test
01:08:45.966  00:14:37.162     135 	run_hotplug
01:08:45.966  00:14:37.162     136 	
01:08:45.966  00:14:37.162     137 	# Run SPDK target based hotplug
01:08:45.966  00:14:37.162  => 138 	tgt_run_hotplug
01:08:45.966  00:14:37.162       ...
01:08:45.966  00:14:37.162  
01:08:45.966  00:14:37.162  ========== Backtrace end ==========
01:08:45.966  00:14:37.162  ========== Backtrace start: ==========
01:08:45.966  00:14:37.162  
01:08:45.967  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:45.967  00:14:37.162       ...
01:08:45.967  00:14:37.162     11  	# Pci bus hotplug
01:08:45.967  00:14:37.162     12  	# Helper function to remove/attach cotrollers
01:08:45.967  00:14:37.162     13  	debug_remove_attach_helper() {
01:08:45.967  00:14:37.162     14  		local helper_time=0
01:08:45.967  00:14:37.162     15  	
01:08:45.967  00:14:37.162  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:45.967  00:14:37.162     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:45.967  00:14:37.162     18  			"$helper_time" "$nvme_count" >&2
01:08:45.967  00:14:37.162     19  	}
01:08:45.967  00:14:37.162     20  	
01:08:45.967  00:14:37.162     21  	remove_attach_helper() {
01:08:45.967  00:14:37.162       ...
01:08:45.967  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:45.967  00:14:37.162       ...
01:08:45.967  00:14:37.162     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:45.967  00:14:37.162     106 		done
01:08:45.967  00:14:37.162     107 	
01:08:45.967  00:14:37.162     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.967  00:14:37.162     109 	
01:08:45.967  00:14:37.162  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.967  00:14:37.162     111 		# Verify reregistering hotplug poller
01:08:45.967  00:14:37.162     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:45.967  00:14:37.162     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:45.967  00:14:37.162     114 	
01:08:45.967  00:14:37.162     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:45.967  00:14:37.162       ...
01:08:45.967  00:14:37.162  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:45.967  00:14:37.162       ...
01:08:45.967  00:14:37.162     133 	
01:08:45.967  00:14:37.162     134 	# Run pci bus hotplug test
01:08:45.967  00:14:37.162     135 	run_hotplug
01:08:45.967  00:14:37.162     136 	
01:08:45.967  00:14:37.162     137 	# Run SPDK target based hotplug
01:08:45.967  00:14:37.162  => 138 	tgt_run_hotplug
01:08:45.967  00:14:37.162       ...
01:08:45.967  00:14:37.162  
01:08:45.967  00:14:37.162  ========== Backtrace end ==========
01:08:45.967  00:14:37.421  ========== Backtrace start: ==========
01:08:45.967  00:14:37.421  
01:08:45.967  00:14:37.421  in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh"])
01:08:45.967  00:14:37.421       ...
01:08:45.967  00:14:37.421     1117		timing_enter $test_name
01:08:45.967  00:14:37.421     1118		echo "************************************"
01:08:45.967  00:14:37.421     1119		echo "START TEST $test_name"
01:08:45.967  00:14:37.421     1120		echo "************************************"
01:08:45.967  00:14:37.421     1121		xtrace_restore
01:08:45.967  00:14:37.421     1122		time "$@"
01:08:45.967  00:14:37.421     1123		xtrace_disable
01:08:45.967  00:14:37.421     1124		echo "************************************"
01:08:45.967  00:14:37.421     1125		echo "END TEST $test_name"
01:08:45.967  00:14:37.421     1126		echo "************************************"
01:08:45.967  00:14:37.421     1127		timing_exit $test_name
01:08:45.967  00:14:37.421       ...
01:08:45.967  00:14:37.421  in /var/jenkins/workspace/nvme-phy-autotest/spdk/autotest.sh:240 -> main(["/var/jenkins/workspace/nvme-phy-autotest/autorun-spdk.conf"])
01:08:45.967  00:14:37.421       ...
01:08:45.967  00:14:37.421     235 	
01:08:45.967  00:14:37.421     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:45.967  00:14:37.421     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:45.967  00:14:37.421     238 	
01:08:45.967  00:14:37.421     239 			if [ $(uname -s) = Linux ]; then
01:08:45.967  00:14:37.421  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:45.967  00:14:37.421     241 			fi
01:08:45.967  00:14:37.421     242 	
01:08:45.967  00:14:37.421     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:45.967  00:14:37.421     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:45.967  00:14:37.421     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:45.967  00:14:37.421       ...
01:08:45.967  00:14:37.421  
01:08:45.967  00:14:37.421  ========== Backtrace end =========='
01:08:45.967  + for build in "${build_log_list[@]}"
01:08:45.967  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/build.log
01:08:45.967  + local build_log job_name start_pattern end_pattern backtraces
01:08:45.967  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/build.log
01:08:45.967  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/build.log
01:08:45.967  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516
01:08:45.967  + job_name=iscsi-vg-autotest_77516
01:08:45.967  + start_pattern='Backtrace start'
01:08:45.967  + end_pattern='Backtrace end'
01:08:45.967  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/build.log ]]
01:08:45.967  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088
01:08:46.230  + job_name=rocky9-vg-autotest_16088
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375
01:08:46.230  + job_name=unittest-vg-autotest_63375
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898
01:08:46.230  + job_name=dsa-phy-autotest_4898
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192
01:08:46.230  + job_name=valgrind-vg-autotest_49192
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329
01:08:46.230  + job_name=nvmf-tcp-phy-autotest_72329
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442
01:08:46.230  + job_name=nvmf-phy-autotest_72442
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383
01:08:46.230  + job_name=iscsi-uring-vg-autotest_77383
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657
01:08:46.230  + job_name=centos7-vg-autotest_73657
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.230  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/build.log
01:08:46.230  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/build.log
01:08:46.230  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377
01:08:46.230  + job_name=scanbuild-vg-autotest_63377
01:08:46.230  + start_pattern='Backtrace start'
01:08:46.230  + end_pattern='Backtrace end'
01:08:46.230  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/build.log ]]
01:08:46.230  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/build.log
01:08:46.230  + backtraces=
01:08:46.230  + [[ -n '' ]]
01:08:46.230  + for build in "${build_log_list[@]}"
01:08:46.230  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/build.log
01:08:46.230  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.231  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/build.log
01:08:46.231  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/build.log
01:08:46.231  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584
01:08:46.231  + job_name=zns-vg-autotest_76584
01:08:46.231  + start_pattern='Backtrace start'
01:08:46.231  + end_pattern='Backtrace end'
01:08:46.231  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/build.log ]]
01:08:46.231  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/build.log
01:08:46.231  + backtraces=
01:08:46.231  + [[ -n '' ]]
01:08:46.231  + for build in "${build_log_list[@]}"
01:08:46.231  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/build.log
01:08:46.231  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.231  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/build.log
01:08:46.231  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/build.log
01:08:46.231  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427
01:08:46.231  + job_name=lvol-vg-autotest_77427
01:08:46.231  + start_pattern='Backtrace start'
01:08:46.231  + end_pattern='Backtrace end'
01:08:46.231  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/build.log ]]
01:08:46.231  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/build.log
01:08:46.231  + backtraces=
01:08:46.231  + [[ -n '' ]]
01:08:46.231  + for build in "${build_log_list[@]}"
01:08:46.231  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/build.log
01:08:46.231  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.231  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/build.log
01:08:46.231  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/build.log
01:08:46.231  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469
01:08:46.231  + job_name=nvmf-tcp-vg-autotest_77469
01:08:46.231  + start_pattern='Backtrace start'
01:08:46.231  + end_pattern='Backtrace end'
01:08:46.231  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/build.log ]]
01:08:46.231  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/build.log
01:08:46.231  + backtraces=
01:08:46.231  + [[ -n '' ]]
01:08:46.231  + for build in "${build_log_list[@]}"
01:08:46.231  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/build.log
01:08:46.231  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.231  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/build.log
01:08:46.231  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/build.log
01:08:46.231  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195
01:08:46.231  + job_name=short-fuzz-phy-autotest_26195
01:08:46.231  + start_pattern='Backtrace start'
01:08:46.231  + end_pattern='Backtrace end'
01:08:46.231  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/build.log ]]
01:08:46.231  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/build.log
01:08:46.231  + backtraces=
01:08:46.231  + [[ -n '' ]]
01:08:46.231  + for build in "${build_log_list[@]}"
01:08:46.231  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/build.log
01:08:46.231  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.231  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/build.log
01:08:46.231  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/build.log
01:08:46.231  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601
01:08:46.231  + job_name=nvme-cmb-pmr-vg-autotest_78601
01:08:46.231  + start_pattern='Backtrace start'
01:08:46.231  + end_pattern='Backtrace end'
01:08:46.231  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/build.log ]]
01:08:46.231  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/build.log
01:08:46.231  + backtraces=
01:08:46.231  + [[ -n '' ]]
01:08:46.231  + for build in "${build_log_list[@]}"
01:08:46.231  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/build.log
01:08:46.231  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.231  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/build.log
01:08:46.231  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/build.log
01:08:46.231  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478
01:08:46.231  + job_name=nvme-vg-autotest_77478
01:08:46.231  + start_pattern='Backtrace start'
01:08:46.231  + end_pattern='Backtrace end'
01:08:46.231  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/build.log ]]
01:08:46.231  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/build.log
01:08:46.231  + backtraces='00:29:09.156    08:22:54 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ==========
01:08:46.231  00:29:09.156  
01:08:46.231  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156     710 	
01:08:46.231  00:29:09.156     711 		# We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to
01:08:46.231  00:29:09.156     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.231  00:29:09.156     713 		# but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do:
01:08:46.231  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.231  00:29:09.156  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.231  00:29:09.156     716 	
01:08:46.231  00:29:09.156     717 		echo "$time"
01:08:46.231  00:29:09.156     718 	)
01:08:46.231  00:29:09.156     719 	
01:08:46.231  00:29:09.156     720 	function timing_enter() {
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156     11  	# Pci bus hotplug
01:08:46.231  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.231  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.231  00:29:09.156     14  		local helper_time=0
01:08:46.231  00:29:09.156     15  	
01:08:46.231  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.231  00:29:09.156     17  		printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \
01:08:46.231  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.231  00:29:09.156     19  	}
01:08:46.231  00:29:09.156     20  	
01:08:46.231  00:29:09.156     21  	remove_attach_helper() {
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.231  00:29:09.156     106 		done
01:08:46.231  00:29:09.156     107 	
01:08:46.231  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.231  00:29:09.156     109 	
01:08:46.231  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.231  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.231  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.231  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.231  00:29:09.156     114 	
01:08:46.231  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156     133 	
01:08:46.231  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.231  00:29:09.156     135 	run_hotplug
01:08:46.231  00:29:09.156     136 	
01:08:46.231  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.231  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156  
01:08:46.231  00:29:09.156  ========== Backtrace end =========='\''
01:08:46.231  00:29:09.156  ========== Backtrace start: ==========
01:08:46.231  00:29:09.156  
01:08:46.231  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156     710 	
01:08:46.231  00:29:09.156     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:46.231  00:29:09.156     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.231  00:29:09.156     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.231  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.231  00:29:09.156  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.231  00:29:09.156     716 	
01:08:46.231  00:29:09.156     717 		echo "$time"
01:08:46.231  00:29:09.156     718 	)
01:08:46.231  00:29:09.156     719 	
01:08:46.231  00:29:09.156     720 	function timing_enter() {
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156     11  	# Pci bus hotplug
01:08:46.231  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.231  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.231  00:29:09.156     14  		local helper_time=0
01:08:46.231  00:29:09.156     15  	
01:08:46.231  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.231  00:29:09.156     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.231  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.231  00:29:09.156     19  	}
01:08:46.231  00:29:09.156     20  	
01:08:46.231  00:29:09.156     21  	remove_attach_helper() {
01:08:46.231  00:29:09.156       ...
01:08:46.231  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.231  00:29:09.156       ...
01:08:46.232  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.232  00:29:09.156     106 		done
01:08:46.232  00:29:09.156     107 	
01:08:46.232  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.232  00:29:09.156     109 	
01:08:46.232  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.232  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.232  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.232  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.232  00:29:09.156     114 	
01:08:46.232  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156     133 	
01:08:46.232  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.232  00:29:09.156     135 	run_hotplug
01:08:46.232  00:29:09.156     136 	
01:08:46.232  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.232  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156  
01:08:46.232  00:29:09.156  ========== Backtrace end ==========
01:08:46.232  00:29:09.156  ========== Backtrace start: ==========
01:08:46.232  00:29:09.156  
01:08:46.232  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.232  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.232  00:29:09.156     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.232  00:29:09.156     716 	
01:08:46.232  00:29:09.156     717 		echo "$time"
01:08:46.232  00:29:09.156  => 718 	)
01:08:46.232  00:29:09.156     719 	
01:08:46.232  00:29:09.156     720 	function timing_enter() {
01:08:46.232  00:29:09.156     721 		xtrace_disable
01:08:46.232  00:29:09.156     722 		timing "enter" "$1"
01:08:46.232  00:29:09.156     723 		xtrace_restore
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156     11  	# Pci bus hotplug
01:08:46.232  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.232  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.232  00:29:09.156     14  		local helper_time=0
01:08:46.232  00:29:09.156     15  	
01:08:46.232  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.232  00:29:09.156     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.232  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.232  00:29:09.156     19  	}
01:08:46.232  00:29:09.156     20  	
01:08:46.232  00:29:09.156     21  	remove_attach_helper() {
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.232  00:29:09.156     106 		done
01:08:46.232  00:29:09.156     107 	
01:08:46.232  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.232  00:29:09.156     109 	
01:08:46.232  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.232  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.232  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.232  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.232  00:29:09.156     114 	
01:08:46.232  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156     133 	
01:08:46.232  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.232  00:29:09.156     135 	run_hotplug
01:08:46.232  00:29:09.156     136 	
01:08:46.232  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.232  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156  
01:08:46.232  00:29:09.156  ========== Backtrace end ==========
01:08:46.232  00:29:09.156  ========== Backtrace start: ==========
01:08:46.232  00:29:09.156  
01:08:46.232  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.232  00:29:09.157       ...
01:08:46.232  00:29:09.157     11  	# Pci bus hotplug
01:08:46.232  00:29:09.157     12  	# Helper function to remove/attach cotrollers
01:08:46.232  00:29:09.157     13  	debug_remove_attach_helper() {
01:08:46.232  00:29:09.157     14  		local helper_time=0
01:08:46.232  00:29:09.157     15  	
01:08:46.232  00:29:09.157  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.232  00:29:09.157     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.232  00:29:09.157     18  			"$helper_time" "$nvme_count" >&2
01:08:46.232  00:29:09.157     19  	}
01:08:46.232  00:29:09.157     20  	
01:08:46.232  00:29:09.157     21  	remove_attach_helper() {
01:08:46.232  00:29:09.157       ...
01:08:46.232  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.232  00:29:09.157       ...
01:08:46.232  00:29:09.157     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.232  00:29:09.157     106 		done
01:08:46.232  00:29:09.157     107 	
01:08:46.232  00:29:09.157     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.232  00:29:09.157     109 	
01:08:46.232  00:29:09.157  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.232  00:29:09.157     111 		# Verify reregistering hotplug poller
01:08:46.232  00:29:09.157     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.232  00:29:09.157     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.232  00:29:09.157     114 	
01:08:46.232  00:29:09.157     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.232  00:29:09.157       ...
01:08:46.232  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.232  00:29:09.157       ...
01:08:46.232  00:29:09.157     133 	
01:08:46.232  00:29:09.157     134 	# Run pci bus hotplug test
01:08:46.232  00:29:09.157     135 	run_hotplug
01:08:46.232  00:29:09.157     136 	
01:08:46.232  00:29:09.157     137 	# Run SPDK target based hotplug
01:08:46.232  00:29:09.157  => 138 	tgt_run_hotplug
01:08:46.232  00:29:09.157       ...
01:08:46.232  00:29:09.157  
01:08:46.232  00:29:09.157  ========== Backtrace end ==========
01:08:46.232  00:29:09.981  ========== Backtrace start: ==========
01:08:46.232  00:29:09.981  
01:08:46.232  00:29:09.981  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"])
01:08:46.232  00:29:09.981       ...
01:08:46.232  00:29:09.981     1117		timing_enter $test_name
01:08:46.232  00:29:09.981     1118		echo "************************************"
01:08:46.232  00:29:09.981     1119		echo "START TEST $test_name"
01:08:46.232  00:29:09.981     1120		echo "************************************"
01:08:46.232  00:29:09.981     1121		xtrace_restore
01:08:46.232  00:29:09.981     1122		time "$@"
01:08:46.232  00:29:09.981     1123		xtrace_disable
01:08:46.232  00:29:09.981     1124		echo "************************************"
01:08:46.232  00:29:09.981     1125		echo "END TEST $test_name"
01:08:46.232  00:29:09.981     1126		echo "************************************"
01:08:46.232  00:29:09.981     1127		timing_exit $test_name
01:08:46.232  00:29:09.981       ...
01:08:46.232  00:29:09.981  in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
01:08:46.232  00:29:09.981       ...
01:08:46.232  00:29:09.981     235 	
01:08:46.232  00:29:09.981     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:46.232  00:29:09.981     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:46.232  00:29:09.981     238 	
01:08:46.232  00:29:09.981     239 			if [ $(uname -s) = Linux ]; then
01:08:46.232  00:29:09.981  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:46.232  00:29:09.981     241 			fi
01:08:46.232  00:29:09.981     242 	
01:08:46.232  00:29:09.981     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:46.232  00:29:09.981     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:46.232  00:29:09.981     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:46.232  00:29:09.981       ...
01:08:46.232  00:29:09.981  
01:08:46.232  00:29:09.981  ========== Backtrace end =========='
01:08:46.232  + [[ -n 00:29:09.156    08:22:54 sw_hotplug -- common/autotest_common.sh@715 -- # time='========== Backtrace start: ==========
01:08:46.232  00:29:09.156  
01:08:46.232  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156     710 	
01:08:46.232  00:29:09.156     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:46.232  00:29:09.156     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.232  00:29:09.156     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.232  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.232  00:29:09.156  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.232  00:29:09.156     716 	
01:08:46.232  00:29:09.156     717 		echo "$time"
01:08:46.232  00:29:09.156     718 	)
01:08:46.232  00:29:09.156     719 	
01:08:46.232  00:29:09.156     720 	function timing_enter() {
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156     11  	# Pci bus hotplug
01:08:46.232  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.232  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.232  00:29:09.156     14  		local helper_time=0
01:08:46.232  00:29:09.156     15  	
01:08:46.232  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.232  00:29:09.156     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.232  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.232  00:29:09.156     19  	}
01:08:46.232  00:29:09.156     20  	
01:08:46.232  00:29:09.156     21  	remove_attach_helper() {
01:08:46.232  00:29:09.156       ...
01:08:46.232  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.232  00:29:09.156       ...
01:08:46.233  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.233  00:29:09.156     106 		done
01:08:46.233  00:29:09.156     107 	
01:08:46.233  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.233  00:29:09.156     109 	
01:08:46.233  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.233  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.233  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.233  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.233  00:29:09.156     114 	
01:08:46.233  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     133 	
01:08:46.233  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.233  00:29:09.156     135 	run_hotplug
01:08:46.233  00:29:09.156     136 	
01:08:46.233  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.233  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  
01:08:46.233  00:29:09.156  ========== Backtrace end =========='
01:08:46.233  00:29:09.156  ========== Backtrace start: ==========
01:08:46.233  00:29:09.156  
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     710 	
01:08:46.233  00:29:09.156     711 		# We redirect cmd's std{out,err} to a separate fd dup'ed to stdin's device (or /dev/null) to
01:08:46.233  00:29:09.156     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.233  00:29:09.156     713 		# but $() will return just the time's data, hence making it possible to just do:
01:08:46.233  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.233  00:29:09.156  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.233  00:29:09.156     716 	
01:08:46.233  00:29:09.156     717 		echo "$time"
01:08:46.233  00:29:09.156     718 	)
01:08:46.233  00:29:09.156     719 	
01:08:46.233  00:29:09.156     720 	function timing_enter() {
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     11  	# Pci bus hotplug
01:08:46.233  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.233  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.233  00:29:09.156     14  		local helper_time=0
01:08:46.233  00:29:09.156     15  	
01:08:46.233  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.233  00:29:09.156     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:46.233  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.233  00:29:09.156     19  	}
01:08:46.233  00:29:09.156     20  	
01:08:46.233  00:29:09.156     21  	remove_attach_helper() {
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.233  00:29:09.156     106 		done
01:08:46.233  00:29:09.156     107 	
01:08:46.233  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.233  00:29:09.156     109 	
01:08:46.233  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.233  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.233  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.233  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.233  00:29:09.156     114 	
01:08:46.233  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     133 	
01:08:46.233  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.233  00:29:09.156     135 	run_hotplug
01:08:46.233  00:29:09.156     136 	
01:08:46.233  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.233  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  
01:08:46.233  00:29:09.156  ========== Backtrace end ==========
01:08:46.233  00:29:09.156  ========== Backtrace start: ==========
01:08:46.233  00:29:09.156  
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     713 		# but $() will return just the time's data, hence making it possible to just do:
01:08:46.233  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.233  00:29:09.156     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.233  00:29:09.156     716 	
01:08:46.233  00:29:09.156     717 		echo "$time"
01:08:46.233  00:29:09.156  => 718 	)
01:08:46.233  00:29:09.156     719 	
01:08:46.233  00:29:09.156     720 	function timing_enter() {
01:08:46.233  00:29:09.156     721 		xtrace_disable
01:08:46.233  00:29:09.156     722 		timing "enter" "$1"
01:08:46.233  00:29:09.156     723 		xtrace_restore
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     11  	# Pci bus hotplug
01:08:46.233  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.233  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.233  00:29:09.156     14  		local helper_time=0
01:08:46.233  00:29:09.156     15  	
01:08:46.233  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.233  00:29:09.156     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:46.233  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.233  00:29:09.156     19  	}
01:08:46.233  00:29:09.156     20  	
01:08:46.233  00:29:09.156     21  	remove_attach_helper() {
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.233  00:29:09.156     106 		done
01:08:46.233  00:29:09.156     107 	
01:08:46.233  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.233  00:29:09.156     109 	
01:08:46.233  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.233  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.233  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.233  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.233  00:29:09.156     114 	
01:08:46.233  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156     133 	
01:08:46.233  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.233  00:29:09.156     135 	run_hotplug
01:08:46.233  00:29:09.156     136 	
01:08:46.233  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.233  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.233  00:29:09.156       ...
01:08:46.233  00:29:09.156  
01:08:46.233  00:29:09.156  ========== Backtrace end ==========
01:08:46.233  00:29:09.156  ========== Backtrace start: ==========
01:08:46.233  00:29:09.156  
01:08:46.233  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.233  00:29:09.157       ...
01:08:46.233  00:29:09.157     11  	# Pci bus hotplug
01:08:46.233  00:29:09.157     12  	# Helper function to remove/attach cotrollers
01:08:46.233  00:29:09.157     13  	debug_remove_attach_helper() {
01:08:46.233  00:29:09.157     14  		local helper_time=0
01:08:46.233  00:29:09.157     15  	
01:08:46.233  00:29:09.157  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.233  00:29:09.157     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:46.233  00:29:09.157     18  			"$helper_time" "$nvme_count" >&2
01:08:46.233  00:29:09.157     19  	}
01:08:46.233  00:29:09.157     20  	
01:08:46.233  00:29:09.157     21  	remove_attach_helper() {
01:08:46.233  00:29:09.157       ...
01:08:46.233  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.233  00:29:09.157       ...
01:08:46.233  00:29:09.157     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.233  00:29:09.157     106 		done
01:08:46.233  00:29:09.157     107 	
01:08:46.233  00:29:09.157     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.233  00:29:09.157     109 	
01:08:46.233  00:29:09.157  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.233  00:29:09.157     111 		# Verify reregistering hotplug poller
01:08:46.233  00:29:09.157     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.233  00:29:09.157     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.233  00:29:09.157     114 	
01:08:46.233  00:29:09.157     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.233  00:29:09.157       ...
01:08:46.233  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.233  00:29:09.157       ...
01:08:46.233  00:29:09.157     133 	
01:08:46.233  00:29:09.157     134 	# Run pci bus hotplug test
01:08:46.233  00:29:09.157     135 	run_hotplug
01:08:46.233  00:29:09.157     136 	
01:08:46.233  00:29:09.157     137 	# Run SPDK target based hotplug
01:08:46.233  00:29:09.157  => 138 	tgt_run_hotplug
01:08:46.233  00:29:09.157       ...
01:08:46.233  00:29:09.157  
01:08:46.233  00:29:09.157  ========== Backtrace end ==========
01:08:46.233  00:29:09.981  ========== Backtrace start: ==========
01:08:46.233  00:29:09.981  
01:08:46.233  00:29:09.981  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"])
01:08:46.233  00:29:09.981       ...
01:08:46.233  00:29:09.981     1117		timing_enter $test_name
01:08:46.233  00:29:09.981     1118		echo "************************************"
01:08:46.233  00:29:09.981     1119		echo "START TEST $test_name"
01:08:46.233  00:29:09.981     1120		echo "************************************"
01:08:46.233  00:29:09.981     1121		xtrace_restore
01:08:46.233  00:29:09.981     1122		time "$@"
01:08:46.234  00:29:09.981     1123		xtrace_disable
01:08:46.234  00:29:09.981     1124		echo "************************************"
01:08:46.234  00:29:09.981     1125		echo "END TEST $test_name"
01:08:46.234  00:29:09.981     1126		echo "************************************"
01:08:46.234  00:29:09.981     1127		timing_exit $test_name
01:08:46.234  00:29:09.981       ...
01:08:46.234  00:29:09.981  in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
01:08:46.234  00:29:09.981       ...
01:08:46.234  00:29:09.981     235 	
01:08:46.234  00:29:09.981     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:46.234  00:29:09.981     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:46.234  00:29:09.981     238 	
01:08:46.234  00:29:09.981     239 			if [ $(uname -s) = Linux ]; then
01:08:46.234  00:29:09.981  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:46.234  00:29:09.981     241 			fi
01:08:46.234  00:29:09.981     242 	
01:08:46.234  00:29:09.981     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:46.234  00:29:09.981     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:46.234  00:29:09.981     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:46.234  00:29:09.981       ...
01:08:46.234  00:29:09.981  
01:08:46.234  00:29:09.981  ========== Backtrace end ========== ]]
01:08:46.234  + echo '00:29:09.156    08:22:54 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ==========
01:08:46.234  00:29:09.156  
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     710 	
01:08:46.234  00:29:09.156     711 		# We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to
01:08:46.234  00:29:09.156     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.234  00:29:09.156     713 		# but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do:
01:08:46.234  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.234  00:29:09.156  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.234  00:29:09.156     716 	
01:08:46.234  00:29:09.156     717 		echo "$time"
01:08:46.234  00:29:09.156     718 	)
01:08:46.234  00:29:09.156     719 	
01:08:46.234  00:29:09.156     720 	function timing_enter() {
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     11  	# Pci bus hotplug
01:08:46.234  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.234  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.234  00:29:09.156     14  		local helper_time=0
01:08:46.234  00:29:09.156     15  	
01:08:46.234  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.234  00:29:09.156     17  		printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \
01:08:46.234  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.234  00:29:09.156     19  	}
01:08:46.234  00:29:09.156     20  	
01:08:46.234  00:29:09.156     21  	remove_attach_helper() {
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.234  00:29:09.156     106 		done
01:08:46.234  00:29:09.156     107 	
01:08:46.234  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.234  00:29:09.156     109 	
01:08:46.234  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.234  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.234  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.234  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.234  00:29:09.156     114 	
01:08:46.234  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     133 	
01:08:46.234  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.234  00:29:09.156     135 	run_hotplug
01:08:46.234  00:29:09.156     136 	
01:08:46.234  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.234  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156  
01:08:46.234  00:29:09.156  ========== Backtrace end =========='\''
01:08:46.234  00:29:09.156  ========== Backtrace start: ==========
01:08:46.234  00:29:09.156  
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     710 	
01:08:46.234  00:29:09.156     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:46.234  00:29:09.156     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.234  00:29:09.156     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.234  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.234  00:29:09.156  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.234  00:29:09.156     716 	
01:08:46.234  00:29:09.156     717 		echo "$time"
01:08:46.234  00:29:09.156     718 	)
01:08:46.234  00:29:09.156     719 	
01:08:46.234  00:29:09.156     720 	function timing_enter() {
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     11  	# Pci bus hotplug
01:08:46.234  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.234  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.234  00:29:09.156     14  		local helper_time=0
01:08:46.234  00:29:09.156     15  	
01:08:46.234  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.234  00:29:09.156     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.234  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.234  00:29:09.156     19  	}
01:08:46.234  00:29:09.156     20  	
01:08:46.234  00:29:09.156     21  	remove_attach_helper() {
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.234  00:29:09.156     106 		done
01:08:46.234  00:29:09.156     107 	
01:08:46.234  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.234  00:29:09.156     109 	
01:08:46.234  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.234  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.234  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.234  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.234  00:29:09.156     114 	
01:08:46.234  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     133 	
01:08:46.234  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.234  00:29:09.156     135 	run_hotplug
01:08:46.234  00:29:09.156     136 	
01:08:46.234  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.234  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156  
01:08:46.234  00:29:09.156  ========== Backtrace end ==========
01:08:46.234  00:29:09.156  ========== Backtrace start: ==========
01:08:46.234  00:29:09.156  
01:08:46.234  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.234  00:29:09.156       ...
01:08:46.234  00:29:09.156     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.234  00:29:09.156     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.234  00:29:09.156     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.234  00:29:09.156     716 	
01:08:46.234  00:29:09.156     717 		echo "$time"
01:08:46.234  00:29:09.156  => 718 	)
01:08:46.234  00:29:09.156     719 	
01:08:46.234  00:29:09.156     720 	function timing_enter() {
01:08:46.234  00:29:09.156     721 		xtrace_disable
01:08:46.234  00:29:09.156     722 		timing "enter" "$1"
01:08:46.235  00:29:09.156     723 		xtrace_restore
01:08:46.235  00:29:09.156       ...
01:08:46.235  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.235  00:29:09.156       ...
01:08:46.235  00:29:09.156     11  	# Pci bus hotplug
01:08:46.235  00:29:09.156     12  	# Helper function to remove/attach cotrollers
01:08:46.235  00:29:09.156     13  	debug_remove_attach_helper() {
01:08:46.235  00:29:09.156     14  		local helper_time=0
01:08:46.235  00:29:09.156     15  	
01:08:46.235  00:29:09.156  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.235  00:29:09.156     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.235  00:29:09.156     18  			"$helper_time" "$nvme_count" >&2
01:08:46.235  00:29:09.156     19  	}
01:08:46.235  00:29:09.156     20  	
01:08:46.235  00:29:09.156     21  	remove_attach_helper() {
01:08:46.235  00:29:09.156       ...
01:08:46.235  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.235  00:29:09.156       ...
01:08:46.235  00:29:09.156     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.235  00:29:09.156     106 		done
01:08:46.235  00:29:09.156     107 	
01:08:46.235  00:29:09.156     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.235  00:29:09.156     109 	
01:08:46.235  00:29:09.156  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.235  00:29:09.156     111 		# Verify reregistering hotplug poller
01:08:46.235  00:29:09.156     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.235  00:29:09.156     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.235  00:29:09.156     114 	
01:08:46.235  00:29:09.156     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.235  00:29:09.156       ...
01:08:46.235  00:29:09.156  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.235  00:29:09.156       ...
01:08:46.235  00:29:09.156     133 	
01:08:46.235  00:29:09.156     134 	# Run pci bus hotplug test
01:08:46.235  00:29:09.156     135 	run_hotplug
01:08:46.235  00:29:09.156     136 	
01:08:46.235  00:29:09.156     137 	# Run SPDK target based hotplug
01:08:46.235  00:29:09.156  => 138 	tgt_run_hotplug
01:08:46.235  00:29:09.156       ...
01:08:46.235  00:29:09.156  
01:08:46.235  00:29:09.156  ========== Backtrace end ==========
01:08:46.235  00:29:09.156  ========== Backtrace start: ==========
01:08:46.235  00:29:09.156  
01:08:46.235  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.235  00:29:09.157       ...
01:08:46.235  00:29:09.157     11  	# Pci bus hotplug
01:08:46.235  00:29:09.157     12  	# Helper function to remove/attach cotrollers
01:08:46.235  00:29:09.157     13  	debug_remove_attach_helper() {
01:08:46.235  00:29:09.157     14  		local helper_time=0
01:08:46.235  00:29:09.157     15  	
01:08:46.235  00:29:09.157  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.235  00:29:09.157     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.235  00:29:09.157     18  			"$helper_time" "$nvme_count" >&2
01:08:46.235  00:29:09.157     19  	}
01:08:46.235  00:29:09.157     20  	
01:08:46.235  00:29:09.157     21  	remove_attach_helper() {
01:08:46.235  00:29:09.157       ...
01:08:46.235  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.235  00:29:09.157       ...
01:08:46.235  00:29:09.157     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.235  00:29:09.157     106 		done
01:08:46.235  00:29:09.157     107 	
01:08:46.235  00:29:09.157     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.235  00:29:09.157     109 	
01:08:46.235  00:29:09.157  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.235  00:29:09.157     111 		# Verify reregistering hotplug poller
01:08:46.235  00:29:09.157     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.235  00:29:09.157     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.235  00:29:09.157     114 	
01:08:46.235  00:29:09.157     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.235  00:29:09.157       ...
01:08:46.235  00:29:09.157  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.235  00:29:09.157       ...
01:08:46.235  00:29:09.157     133 	
01:08:46.235  00:29:09.157     134 	# Run pci bus hotplug test
01:08:46.235  00:29:09.157     135 	run_hotplug
01:08:46.235  00:29:09.157     136 	
01:08:46.235  00:29:09.157     137 	# Run SPDK target based hotplug
01:08:46.235  00:29:09.157  => 138 	tgt_run_hotplug
01:08:46.235  00:29:09.157       ...
01:08:46.235  00:29:09.157  
01:08:46.235  00:29:09.157  ========== Backtrace end ==========
01:08:46.235  00:29:09.981  ========== Backtrace start: ==========
01:08:46.235  00:29:09.981  
01:08:46.235  00:29:09.981  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"])
01:08:46.235  00:29:09.981       ...
01:08:46.235  00:29:09.981     1117		timing_enter $test_name
01:08:46.235  00:29:09.981     1118		echo "************************************"
01:08:46.235  00:29:09.981     1119		echo "START TEST $test_name"
01:08:46.235  00:29:09.981     1120		echo "************************************"
01:08:46.235  00:29:09.981     1121		xtrace_restore
01:08:46.235  00:29:09.981     1122		time "$@"
01:08:46.235  00:29:09.981     1123		xtrace_disable
01:08:46.235  00:29:09.981     1124		echo "************************************"
01:08:46.235  00:29:09.981     1125		echo "END TEST $test_name"
01:08:46.235  00:29:09.981     1126		echo "************************************"
01:08:46.235  00:29:09.981     1127		timing_exit $test_name
01:08:46.235  00:29:09.981       ...
01:08:46.235  00:29:09.981  in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
01:08:46.235  00:29:09.981       ...
01:08:46.235  00:29:09.981     235 	
01:08:46.235  00:29:09.981     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:46.235  00:29:09.981     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:46.235  00:29:09.981     238 	
01:08:46.235  00:29:09.981     239 			if [ $(uname -s) = Linux ]; then
01:08:46.235  00:29:09.981  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:46.235  00:29:09.981     241 			fi
01:08:46.235  00:29:09.981     242 	
01:08:46.235  00:29:09.981     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:46.235  00:29:09.981     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:46.235  00:29:09.981     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:46.235  00:29:09.981       ...
01:08:46.235  00:29:09.981  
01:08:46.235  00:29:09.981  ========== Backtrace end =========='
01:08:46.235  + for build in "${build_log_list[@]}"
01:08:46.235  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/build.log
01:08:46.235  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.235  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/build.log
01:08:46.235  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/build.log
01:08:46.235  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223
01:08:46.235  + job_name=vhost-phy-autotest_26223
01:08:46.235  + start_pattern='Backtrace start'
01:08:46.235  + end_pattern='Backtrace end'
01:08:46.235  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/build.log ]]
01:08:46.235  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/build.log
01:08:46.235  + backtraces=
01:08:46.235  + [[ -n '' ]]
01:08:46.235  + for build in "${build_log_list[@]}"
01:08:46.235  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/build.log
01:08:46.235  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.235  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/build.log
01:08:46.235  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/build.log
01:08:46.235  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797
01:08:46.235  + job_name=ubuntu20-vg-autotest_73797
01:08:46.235  + start_pattern='Backtrace start'
01:08:46.235  + end_pattern='Backtrace end'
01:08:46.235  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/build.log ]]
01:08:46.235  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/build.log
01:08:46.235  + backtraces='00:48:27.119    08:42:12 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ==========
01:08:46.235  00:48:27.119  
01:08:46.235  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.235  00:48:27.119       ...
01:08:46.235  00:48:27.119     710 	
01:08:46.235  00:48:27.119     711 		# We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to
01:08:46.235  00:48:27.119     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.235  00:48:27.119     713 		# but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do:
01:08:46.235  00:48:27.119     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.235  00:48:27.119  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.235  00:48:27.119     716 	
01:08:46.235  00:48:27.119     717 		echo "$time"
01:08:46.235  00:48:27.119     718 	)
01:08:46.235  00:48:27.119     719 	
01:08:46.235  00:48:27.119     720 	function timing_enter() {
01:08:46.235  00:48:27.119       ...
01:08:46.235  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.235  00:48:27.119       ...
01:08:46.235  00:48:27.119     11  	# Pci bus hotplug
01:08:46.235  00:48:27.119     12  	# Helper function to remove/attach cotrollers
01:08:46.235  00:48:27.119     13  	debug_remove_attach_helper() {
01:08:46.235  00:48:27.119     14  		local helper_time=0
01:08:46.235  00:48:27.119     15  	
01:08:46.235  00:48:27.119  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.235  00:48:27.119     17  		printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \
01:08:46.235  00:48:27.119     18  			"$helper_time" "$nvme_count" >&2
01:08:46.235  00:48:27.119     19  	}
01:08:46.236  00:48:27.119     20  	
01:08:46.236  00:48:27.119     21  	remove_attach_helper() {
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.236  00:48:27.119     106 		done
01:08:46.236  00:48:27.119     107 	
01:08:46.236  00:48:27.119     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.236  00:48:27.119     109 	
01:08:46.236  00:48:27.119  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.236  00:48:27.119     111 		# Verify reregistering hotplug poller
01:08:46.236  00:48:27.119     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.236  00:48:27.119     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.236  00:48:27.119     114 	
01:08:46.236  00:48:27.119     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119     133 	
01:08:46.236  00:48:27.119     134 	# Run pci bus hotplug test
01:08:46.236  00:48:27.119     135 	run_hotplug
01:08:46.236  00:48:27.119     136 	
01:08:46.236  00:48:27.119     137 	# Run SPDK target based hotplug
01:08:46.236  00:48:27.119  => 138 	tgt_run_hotplug
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119  
01:08:46.236  00:48:27.119  ========== Backtrace end ==========
01:08:46.236  00:48:27.119  ========== Backtrace start: ==========
01:08:46.236  00:48:27.119  
01:08:46.236  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119     710 	
01:08:46.236  00:48:27.119     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:46.236  00:48:27.119     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.236  00:48:27.119     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.236  00:48:27.119     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.236  00:48:27.119  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.236  00:48:27.119     716 	
01:08:46.236  00:48:27.119     717 		echo "$time"
01:08:46.236  00:48:27.119     718 	)
01:08:46.236  00:48:27.119     719 	
01:08:46.236  00:48:27.119     720 	function timing_enter() {
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119     11  	# Pci bus hotplug
01:08:46.236  00:48:27.119     12  	# Helper function to remove/attach cotrollers
01:08:46.236  00:48:27.119     13  	debug_remove_attach_helper() {
01:08:46.236  00:48:27.119     14  		local helper_time=0
01:08:46.236  00:48:27.119     15  	
01:08:46.236  00:48:27.119  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.236  00:48:27.119     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.236  00:48:27.119     18  			"$helper_time" "$nvme_count" >&2
01:08:46.236  00:48:27.119     19  	}
01:08:46.236  00:48:27.119     20  	
01:08:46.236  00:48:27.119     21  	remove_attach_helper() {
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.236  00:48:27.119       ...
01:08:46.236  00:48:27.119     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.236  00:48:27.119     106 		done
01:08:46.236  00:48:27.119     107 	
01:08:46.236  00:48:27.119     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.236  00:48:27.119     109 	
01:08:46.236  00:48:27.119  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.236  00:48:27.119     111 		# Verify reregistering hotplug poller
01:08:46.236  00:48:27.119     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.236  00:48:27.119     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.236  00:48:27.119     114 	
01:08:46.236  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120     133 	
01:08:46.236  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.236  00:48:27.120     135 	run_hotplug
01:08:46.236  00:48:27.120     136 	
01:08:46.236  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.236  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  
01:08:46.236  00:48:27.120  ========== Backtrace end ==========
01:08:46.236  00:48:27.120  ========== Backtrace start: ==========
01:08:46.236  00:48:27.120  
01:08:46.236  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.236  00:48:27.120     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.236  00:48:27.120     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.236  00:48:27.120     716 	
01:08:46.236  00:48:27.120     717 		echo "$time"
01:08:46.236  00:48:27.120  => 718 	)
01:08:46.236  00:48:27.120     719 	
01:08:46.236  00:48:27.120     720 	function timing_enter() {
01:08:46.236  00:48:27.120     721 		xtrace_disable
01:08:46.236  00:48:27.120     722 		timing "enter" "$1"
01:08:46.236  00:48:27.120     723 		xtrace_restore
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120     11  	# Pci bus hotplug
01:08:46.236  00:48:27.120     12  	# Helper function to remove/attach cotrollers
01:08:46.236  00:48:27.120     13  	debug_remove_attach_helper() {
01:08:46.236  00:48:27.120     14  		local helper_time=0
01:08:46.236  00:48:27.120     15  	
01:08:46.236  00:48:27.120  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.236  00:48:27.120     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.236  00:48:27.120     18  			"$helper_time" "$nvme_count" >&2
01:08:46.236  00:48:27.120     19  	}
01:08:46.236  00:48:27.120     20  	
01:08:46.236  00:48:27.120     21  	remove_attach_helper() {
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.236  00:48:27.120     106 		done
01:08:46.236  00:48:27.120     107 	
01:08:46.236  00:48:27.120     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.236  00:48:27.120     109 	
01:08:46.236  00:48:27.120  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.236  00:48:27.120     111 		# Verify reregistering hotplug poller
01:08:46.236  00:48:27.120     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.236  00:48:27.120     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.236  00:48:27.120     114 	
01:08:46.236  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120     133 	
01:08:46.236  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.236  00:48:27.120     135 	run_hotplug
01:08:46.236  00:48:27.120     136 	
01:08:46.236  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.236  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  
01:08:46.236  00:48:27.120  ========== Backtrace end ==========
01:08:46.236  00:48:27.120  ========== Backtrace start: ==========
01:08:46.236  00:48:27.120  
01:08:46.236  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120     11  	# Pci bus hotplug
01:08:46.236  00:48:27.120     12  	# Helper function to remove/attach cotrollers
01:08:46.236  00:48:27.120     13  	debug_remove_attach_helper() {
01:08:46.236  00:48:27.120     14  		local helper_time=0
01:08:46.236  00:48:27.120     15  	
01:08:46.236  00:48:27.120  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.236  00:48:27.120     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.236  00:48:27.120     18  			"$helper_time" "$nvme_count" >&2
01:08:46.236  00:48:27.120     19  	}
01:08:46.236  00:48:27.120     20  	
01:08:46.236  00:48:27.120     21  	remove_attach_helper() {
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.236  00:48:27.120     106 		done
01:08:46.236  00:48:27.120     107 	
01:08:46.236  00:48:27.120     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.236  00:48:27.120     109 	
01:08:46.236  00:48:27.120  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.236  00:48:27.120     111 		# Verify reregistering hotplug poller
01:08:46.236  00:48:27.120     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.236  00:48:27.120     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.236  00:48:27.120     114 	
01:08:46.236  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120     133 	
01:08:46.236  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.236  00:48:27.120     135 	run_hotplug
01:08:46.236  00:48:27.120     136 	
01:08:46.236  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.236  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.236  00:48:27.120       ...
01:08:46.236  00:48:27.120  
01:08:46.236  00:48:27.120  ========== Backtrace end ==========
01:08:46.236  00:48:27.640  ========== Backtrace start: ==========
01:08:46.236  00:48:27.640  
01:08:46.236  00:48:27.640  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"])
01:08:46.236  00:48:27.640       ...
01:08:46.236  00:48:27.640     1117		timing_enter $test_name
01:08:46.236  00:48:27.640     1118		echo "************************************"
01:08:46.236  00:48:27.640     1119		echo "START TEST $test_name"
01:08:46.237  00:48:27.640     1120		echo "************************************"
01:08:46.237  00:48:27.640     1121		xtrace_restore
01:08:46.237  00:48:27.640     1122		time "$@"
01:08:46.237  00:48:27.640     1123		xtrace_disable
01:08:46.237  00:48:27.640     1124		echo "************************************"
01:08:46.237  00:48:27.640     1125		echo "END TEST $test_name"
01:08:46.237  00:48:27.640     1126		echo "************************************"
01:08:46.237  00:48:27.640     1127		timing_exit $test_name
01:08:46.237  00:48:27.640       ...
01:08:46.237  00:48:27.641  in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
01:08:46.237  00:48:27.641       ...
01:08:46.237  00:48:27.641     235 	
01:08:46.237  00:48:27.641     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:46.237  00:48:27.641     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:46.237  00:48:27.641     238 	
01:08:46.237  00:48:27.641     239 			if [ $(uname -s) = Linux ]; then
01:08:46.237  00:48:27.641  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:46.237  00:48:27.641     241 			fi
01:08:46.237  00:48:27.641     242 	
01:08:46.237  00:48:27.641     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:46.237  00:48:27.641     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:46.237  00:48:27.641     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:46.237  00:48:27.641       ...
01:08:46.237  00:48:27.641  
01:08:46.237  00:48:27.641  ========== Backtrace end =========='
01:08:46.237  + [[ -n 00:48:27.119    08:42:12 sw_hotplug -- common/autotest_common.sh@715 -- # time='========== Backtrace start: ==========
01:08:46.237  00:48:27.119  
01:08:46.237  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119     710 	
01:08:46.237  00:48:27.119     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:46.237  00:48:27.119     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.237  00:48:27.119     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.237  00:48:27.119     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.237  00:48:27.119  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.237  00:48:27.119     716 	
01:08:46.237  00:48:27.119     717 		echo "$time"
01:08:46.237  00:48:27.119     718 	)
01:08:46.237  00:48:27.119     719 	
01:08:46.237  00:48:27.119     720 	function timing_enter() {
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119     11  	# Pci bus hotplug
01:08:46.237  00:48:27.119     12  	# Helper function to remove/attach cotrollers
01:08:46.237  00:48:27.119     13  	debug_remove_attach_helper() {
01:08:46.237  00:48:27.119     14  		local helper_time=0
01:08:46.237  00:48:27.119     15  	
01:08:46.237  00:48:27.119  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.237  00:48:27.119     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.237  00:48:27.119     18  			"$helper_time" "$nvme_count" >&2
01:08:46.237  00:48:27.119     19  	}
01:08:46.237  00:48:27.119     20  	
01:08:46.237  00:48:27.119     21  	remove_attach_helper() {
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.237  00:48:27.119     106 		done
01:08:46.237  00:48:27.119     107 	
01:08:46.237  00:48:27.119     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.237  00:48:27.119     109 	
01:08:46.237  00:48:27.119  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.237  00:48:27.119     111 		# Verify reregistering hotplug poller
01:08:46.237  00:48:27.119     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.237  00:48:27.119     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.237  00:48:27.119     114 	
01:08:46.237  00:48:27.119     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119     133 	
01:08:46.237  00:48:27.119     134 	# Run pci bus hotplug test
01:08:46.237  00:48:27.119     135 	run_hotplug
01:08:46.237  00:48:27.119     136 	
01:08:46.237  00:48:27.119     137 	# Run SPDK target based hotplug
01:08:46.237  00:48:27.119  => 138 	tgt_run_hotplug
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119  
01:08:46.237  00:48:27.119  ========== Backtrace end ==========
01:08:46.237  00:48:27.119  ========== Backtrace start: ==========
01:08:46.237  00:48:27.119  
01:08:46.237  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119     710 	
01:08:46.237  00:48:27.119     711 		# We redirect cmd's std{out,err} to a separate fd dup'ed to stdin's device (or /dev/null) to
01:08:46.237  00:48:27.119     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.237  00:48:27.119     713 		# but $() will return just the time's data, hence making it possible to just do:
01:08:46.237  00:48:27.119     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.237  00:48:27.119  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.237  00:48:27.119     716 	
01:08:46.237  00:48:27.119     717 		echo "$time"
01:08:46.237  00:48:27.119     718 	)
01:08:46.237  00:48:27.119     719 	
01:08:46.237  00:48:27.119     720 	function timing_enter() {
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119     11  	# Pci bus hotplug
01:08:46.237  00:48:27.119     12  	# Helper function to remove/attach cotrollers
01:08:46.237  00:48:27.119     13  	debug_remove_attach_helper() {
01:08:46.237  00:48:27.119     14  		local helper_time=0
01:08:46.237  00:48:27.119     15  	
01:08:46.237  00:48:27.119  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.237  00:48:27.119     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:46.237  00:48:27.119     18  			"$helper_time" "$nvme_count" >&2
01:08:46.237  00:48:27.119     19  	}
01:08:46.237  00:48:27.119     20  	
01:08:46.237  00:48:27.119     21  	remove_attach_helper() {
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.237  00:48:27.119       ...
01:08:46.237  00:48:27.119     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.237  00:48:27.119     106 		done
01:08:46.237  00:48:27.119     107 	
01:08:46.237  00:48:27.119     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.237  00:48:27.119     109 	
01:08:46.237  00:48:27.119  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.237  00:48:27.119     111 		# Verify reregistering hotplug poller
01:08:46.237  00:48:27.119     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.237  00:48:27.119     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.237  00:48:27.119     114 	
01:08:46.237  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.237  00:48:27.120       ...
01:08:46.237  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.237  00:48:27.120       ...
01:08:46.237  00:48:27.120     133 	
01:08:46.237  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.237  00:48:27.120     135 	run_hotplug
01:08:46.237  00:48:27.120     136 	
01:08:46.237  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.237  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.237  00:48:27.120       ...
01:08:46.237  00:48:27.120  
01:08:46.237  00:48:27.120  ========== Backtrace end ==========
01:08:46.237  00:48:27.120  ========== Backtrace start: ==========
01:08:46.237  00:48:27.120  
01:08:46.237  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.237  00:48:27.120       ...
01:08:46.237  00:48:27.120     713 		# but $() will return just the time's data, hence making it possible to just do:
01:08:46.237  00:48:27.120     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.237  00:48:27.120     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.237  00:48:27.120     716 	
01:08:46.237  00:48:27.120     717 		echo "$time"
01:08:46.237  00:48:27.120  => 718 	)
01:08:46.237  00:48:27.120     719 	
01:08:46.237  00:48:27.120     720 	function timing_enter() {
01:08:46.237  00:48:27.120     721 		xtrace_disable
01:08:46.237  00:48:27.120     722 		timing "enter" "$1"
01:08:46.237  00:48:27.120     723 		xtrace_restore
01:08:46.237  00:48:27.120       ...
01:08:46.237  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.237  00:48:27.120       ...
01:08:46.237  00:48:27.120     11  	# Pci bus hotplug
01:08:46.237  00:48:27.120     12  	# Helper function to remove/attach cotrollers
01:08:46.237  00:48:27.120     13  	debug_remove_attach_helper() {
01:08:46.237  00:48:27.120     14  		local helper_time=0
01:08:46.237  00:48:27.120     15  	
01:08:46.237  00:48:27.120  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.237  00:48:27.120     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:46.237  00:48:27.120     18  			"$helper_time" "$nvme_count" >&2
01:08:46.237  00:48:27.120     19  	}
01:08:46.237  00:48:27.120     20  	
01:08:46.237  00:48:27.120     21  	remove_attach_helper() {
01:08:46.237  00:48:27.120       ...
01:08:46.237  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.237  00:48:27.120       ...
01:08:46.237  00:48:27.120     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.237  00:48:27.120     106 		done
01:08:46.237  00:48:27.120     107 	
01:08:46.237  00:48:27.120     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.237  00:48:27.120     109 	
01:08:46.237  00:48:27.120  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.237  00:48:27.120     111 		# Verify reregistering hotplug poller
01:08:46.237  00:48:27.120     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.237  00:48:27.120     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.237  00:48:27.120     114 	
01:08:46.237  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120     133 	
01:08:46.238  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.238  00:48:27.120     135 	run_hotplug
01:08:46.238  00:48:27.120     136 	
01:08:46.238  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.238  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120  
01:08:46.238  00:48:27.120  ========== Backtrace end ==========
01:08:46.238  00:48:27.120  ========== Backtrace start: ==========
01:08:46.238  00:48:27.120  
01:08:46.238  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120     11  	# Pci bus hotplug
01:08:46.238  00:48:27.120     12  	# Helper function to remove/attach cotrollers
01:08:46.238  00:48:27.120     13  	debug_remove_attach_helper() {
01:08:46.238  00:48:27.120     14  		local helper_time=0
01:08:46.238  00:48:27.120     15  	
01:08:46.238  00:48:27.120  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.238  00:48:27.120     17  		printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \
01:08:46.238  00:48:27.120     18  			"$helper_time" "$nvme_count" >&2
01:08:46.238  00:48:27.120     19  	}
01:08:46.238  00:48:27.120     20  	
01:08:46.238  00:48:27.120     21  	remove_attach_helper() {
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.238  00:48:27.120     106 		done
01:08:46.238  00:48:27.120     107 	
01:08:46.238  00:48:27.120     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.238  00:48:27.120     109 	
01:08:46.238  00:48:27.120  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.238  00:48:27.120     111 		# Verify reregistering hotplug poller
01:08:46.238  00:48:27.120     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.238  00:48:27.120     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.238  00:48:27.120     114 	
01:08:46.238  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120     133 	
01:08:46.238  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.238  00:48:27.120     135 	run_hotplug
01:08:46.238  00:48:27.120     136 	
01:08:46.238  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.238  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.238  00:48:27.120       ...
01:08:46.238  00:48:27.120  
01:08:46.238  00:48:27.120  ========== Backtrace end ==========
01:08:46.238  00:48:27.640  ========== Backtrace start: ==========
01:08:46.238  00:48:27.640  
01:08:46.238  00:48:27.640  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"])
01:08:46.238  00:48:27.640       ...
01:08:46.238  00:48:27.640     1117		timing_enter $test_name
01:08:46.238  00:48:27.640     1118		echo "************************************"
01:08:46.238  00:48:27.640     1119		echo "START TEST $test_name"
01:08:46.238  00:48:27.640     1120		echo "************************************"
01:08:46.238  00:48:27.640     1121		xtrace_restore
01:08:46.238  00:48:27.640     1122		time "$@"
01:08:46.238  00:48:27.640     1123		xtrace_disable
01:08:46.238  00:48:27.640     1124		echo "************************************"
01:08:46.238  00:48:27.640     1125		echo "END TEST $test_name"
01:08:46.238  00:48:27.640     1126		echo "************************************"
01:08:46.238  00:48:27.640     1127		timing_exit $test_name
01:08:46.238  00:48:27.640       ...
01:08:46.238  00:48:27.641  in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
01:08:46.238  00:48:27.641       ...
01:08:46.238  00:48:27.641     235 	
01:08:46.238  00:48:27.641     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:46.238  00:48:27.641     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:46.238  00:48:27.641     238 	
01:08:46.238  00:48:27.641     239 			if [ $(uname -s) = Linux ]; then
01:08:46.238  00:48:27.641  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:46.238  00:48:27.641     241 			fi
01:08:46.238  00:48:27.641     242 	
01:08:46.238  00:48:27.641     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:46.238  00:48:27.641     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:46.238  00:48:27.641     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:46.238  00:48:27.641       ...
01:08:46.238  00:48:27.641  
01:08:46.238  00:48:27.641  ========== Backtrace end ========== ]]
01:08:46.238  + echo '00:48:27.119    08:42:12 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ==========
01:08:46.238  00:48:27.119  
01:08:46.238  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119     710 	
01:08:46.238  00:48:27.119     711 		# We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to
01:08:46.238  00:48:27.119     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.238  00:48:27.119     713 		# but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do:
01:08:46.238  00:48:27.119     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.238  00:48:27.119  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.238  00:48:27.119     716 	
01:08:46.238  00:48:27.119     717 		echo "$time"
01:08:46.238  00:48:27.119     718 	)
01:08:46.238  00:48:27.119     719 	
01:08:46.238  00:48:27.119     720 	function timing_enter() {
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119     11  	# Pci bus hotplug
01:08:46.238  00:48:27.119     12  	# Helper function to remove/attach cotrollers
01:08:46.238  00:48:27.119     13  	debug_remove_attach_helper() {
01:08:46.238  00:48:27.119     14  		local helper_time=0
01:08:46.238  00:48:27.119     15  	
01:08:46.238  00:48:27.119  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.238  00:48:27.119     17  		printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \
01:08:46.238  00:48:27.119     18  			"$helper_time" "$nvme_count" >&2
01:08:46.238  00:48:27.119     19  	}
01:08:46.238  00:48:27.119     20  	
01:08:46.238  00:48:27.119     21  	remove_attach_helper() {
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.238  00:48:27.119     106 		done
01:08:46.238  00:48:27.119     107 	
01:08:46.238  00:48:27.119     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.238  00:48:27.119     109 	
01:08:46.238  00:48:27.119  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.238  00:48:27.119     111 		# Verify reregistering hotplug poller
01:08:46.238  00:48:27.119     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.238  00:48:27.119     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.238  00:48:27.119     114 	
01:08:46.238  00:48:27.119     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119     133 	
01:08:46.238  00:48:27.119     134 	# Run pci bus hotplug test
01:08:46.238  00:48:27.119     135 	run_hotplug
01:08:46.238  00:48:27.119     136 	
01:08:46.238  00:48:27.119     137 	# Run SPDK target based hotplug
01:08:46.238  00:48:27.119  => 138 	tgt_run_hotplug
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119  
01:08:46.238  00:48:27.119  ========== Backtrace end ==========
01:08:46.238  00:48:27.119  ========== Backtrace start: ==========
01:08:46.238  00:48:27.119  
01:08:46.238  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119     710 	
01:08:46.238  00:48:27.119     711 		# We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to
01:08:46.238  00:48:27.119     712 		# catch only output from the time builtin - output from the actual cmd would be still visible,
01:08:46.238  00:48:27.119     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.238  00:48:27.119     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.238  00:48:27.119  => 715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.238  00:48:27.119     716 	
01:08:46.238  00:48:27.119     717 		echo "$time"
01:08:46.238  00:48:27.119     718 	)
01:08:46.238  00:48:27.119     719 	
01:08:46.238  00:48:27.119     720 	function timing_enter() {
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119     11  	# Pci bus hotplug
01:08:46.238  00:48:27.119     12  	# Helper function to remove/attach cotrollers
01:08:46.238  00:48:27.119     13  	debug_remove_attach_helper() {
01:08:46.238  00:48:27.119     14  		local helper_time=0
01:08:46.238  00:48:27.119     15  	
01:08:46.238  00:48:27.119  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.238  00:48:27.119     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.238  00:48:27.119     18  			"$helper_time" "$nvme_count" >&2
01:08:46.238  00:48:27.119     19  	}
01:08:46.238  00:48:27.119     20  	
01:08:46.238  00:48:27.119     21  	remove_attach_helper() {
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.238  00:48:27.119       ...
01:08:46.238  00:48:27.119     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.238  00:48:27.119     106 		done
01:08:46.238  00:48:27.119     107 	
01:08:46.238  00:48:27.119     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.238  00:48:27.119     109 	
01:08:46.238  00:48:27.119  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.238  00:48:27.119     111 		# Verify reregistering hotplug poller
01:08:46.238  00:48:27.119     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.238  00:48:27.119     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.238  00:48:27.119     114 	
01:08:46.238  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120     133 	
01:08:46.239  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.239  00:48:27.120     135 	run_hotplug
01:08:46.239  00:48:27.120     136 	
01:08:46.239  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.239  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  
01:08:46.239  00:48:27.120  ========== Backtrace end ==========
01:08:46.239  00:48:27.120  ========== Backtrace start: ==========
01:08:46.239  00:48:27.120  
01:08:46.239  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"])
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120     713 		# but $() will return just the time'\''s data, hence making it possible to just do:
01:08:46.239  00:48:27.120     714 		#  time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd)
01:08:46.239  00:48:27.120     715 		time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1)
01:08:46.239  00:48:27.120     716 	
01:08:46.239  00:48:27.120     717 		echo "$time"
01:08:46.239  00:48:27.120  => 718 	)
01:08:46.239  00:48:27.120     719 	
01:08:46.239  00:48:27.120     720 	function timing_enter() {
01:08:46.239  00:48:27.120     721 		xtrace_disable
01:08:46.239  00:48:27.120     722 		timing "enter" "$1"
01:08:46.239  00:48:27.120     723 		xtrace_restore
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120     11  	# Pci bus hotplug
01:08:46.239  00:48:27.120     12  	# Helper function to remove/attach cotrollers
01:08:46.239  00:48:27.120     13  	debug_remove_attach_helper() {
01:08:46.239  00:48:27.120     14  		local helper_time=0
01:08:46.239  00:48:27.120     15  	
01:08:46.239  00:48:27.120  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.239  00:48:27.120     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.239  00:48:27.120     18  			"$helper_time" "$nvme_count" >&2
01:08:46.239  00:48:27.120     19  	}
01:08:46.239  00:48:27.120     20  	
01:08:46.239  00:48:27.120     21  	remove_attach_helper() {
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.239  00:48:27.120     106 		done
01:08:46.239  00:48:27.120     107 	
01:08:46.239  00:48:27.120     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.239  00:48:27.120     109 	
01:08:46.239  00:48:27.120  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.239  00:48:27.120     111 		# Verify reregistering hotplug poller
01:08:46.239  00:48:27.120     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.239  00:48:27.120     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.239  00:48:27.120     114 	
01:08:46.239  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120     133 	
01:08:46.239  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.239  00:48:27.120     135 	run_hotplug
01:08:46.239  00:48:27.120     136 	
01:08:46.239  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.239  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  
01:08:46.239  00:48:27.120  ========== Backtrace end ==========
01:08:46.239  00:48:27.120  ========== Backtrace start: ==========
01:08:46.239  00:48:27.120  
01:08:46.239  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"])
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120     11  	# Pci bus hotplug
01:08:46.239  00:48:27.120     12  	# Helper function to remove/attach cotrollers
01:08:46.239  00:48:27.120     13  	debug_remove_attach_helper() {
01:08:46.239  00:48:27.120     14  		local helper_time=0
01:08:46.239  00:48:27.120     15  	
01:08:46.239  00:48:27.120  => 16  		helper_time=$(timing_cmd remove_attach_helper "$@")
01:08:46.239  00:48:27.120     17  		printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \
01:08:46.239  00:48:27.120     18  			"$helper_time" "$nvme_count" >&2
01:08:46.239  00:48:27.120     19  	}
01:08:46.239  00:48:27.120     20  	
01:08:46.239  00:48:27.120     21  	remove_attach_helper() {
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([])
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120     105 			waitforbdev "Nvme0${dev}n1" "$hotplug_wait"
01:08:46.239  00:48:27.120     106 		done
01:08:46.239  00:48:27.120     107 	
01:08:46.239  00:48:27.120     108 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.239  00:48:27.120     109 	
01:08:46.239  00:48:27.120  => 110 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.239  00:48:27.120     111 		# Verify reregistering hotplug poller
01:08:46.239  00:48:27.120     112 		rpc_cmd bdev_nvme_set_hotplug -d
01:08:46.239  00:48:27.120     113 		rpc_cmd bdev_nvme_set_hotplug -e
01:08:46.239  00:48:27.120     114 	
01:08:46.239  00:48:27.120     115 		debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([])
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120     133 	
01:08:46.239  00:48:27.120     134 	# Run pci bus hotplug test
01:08:46.239  00:48:27.120     135 	run_hotplug
01:08:46.239  00:48:27.120     136 	
01:08:46.239  00:48:27.120     137 	# Run SPDK target based hotplug
01:08:46.239  00:48:27.120  => 138 	tgt_run_hotplug
01:08:46.239  00:48:27.120       ...
01:08:46.239  00:48:27.120  
01:08:46.239  00:48:27.120  ========== Backtrace end ==========
01:08:46.239  00:48:27.640  ========== Backtrace start: ==========
01:08:46.239  00:48:27.640  
01:08:46.239  00:48:27.640  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"])
01:08:46.239  00:48:27.640       ...
01:08:46.239  00:48:27.640     1117		timing_enter $test_name
01:08:46.239  00:48:27.640     1118		echo "************************************"
01:08:46.239  00:48:27.640     1119		echo "START TEST $test_name"
01:08:46.239  00:48:27.640     1120		echo "************************************"
01:08:46.239  00:48:27.640     1121		xtrace_restore
01:08:46.239  00:48:27.640     1122		time "$@"
01:08:46.239  00:48:27.640     1123		xtrace_disable
01:08:46.239  00:48:27.640     1124		echo "************************************"
01:08:46.239  00:48:27.640     1125		echo "END TEST $test_name"
01:08:46.239  00:48:27.640     1126		echo "************************************"
01:08:46.239  00:48:27.640     1127		timing_exit $test_name
01:08:46.239  00:48:27.640       ...
01:08:46.239  00:48:27.641  in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
01:08:46.239  00:48:27.641       ...
01:08:46.239  00:48:27.641     235 	
01:08:46.239  00:48:27.641     236 			run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh
01:08:46.239  00:48:27.641     237 			run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh
01:08:46.239  00:48:27.641     238 	
01:08:46.239  00:48:27.641     239 			if [ $(uname -s) = Linux ]; then
01:08:46.239  00:48:27.641  => 240 				run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh
01:08:46.239  00:48:27.641     241 			fi
01:08:46.239  00:48:27.641     242 	
01:08:46.239  00:48:27.641     243 			if [[ $SPDK_TEST_XNVME -eq 1 ]]; then
01:08:46.239  00:48:27.641     244 				run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh
01:08:46.239  00:48:27.641     245 				run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme"
01:08:46.239  00:48:27.641       ...
01:08:46.239  00:48:27.641  
01:08:46.239  00:48:27.641  ========== Backtrace end =========='
01:08:46.239  + for build in "${build_log_list[@]}"
01:08:46.239  + create_errtrace_file /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/build.log
01:08:46.239  + local build_log job_name start_pattern end_pattern backtraces
01:08:46.239  + build_log=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/build.log
01:08:46.239  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/build.log
01:08:46.239  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403
01:08:46.239  + job_name=nvmf-tcp-uring-vg-autotest_77403
01:08:46.239  + start_pattern='Backtrace start'
01:08:46.239  + end_pattern='Backtrace end'
01:08:46.239  + [[ -f /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/build.log ]]
01:08:46.240  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/build.log
01:08:46.240  + backtraces=
01:08:46.240  + [[ -n '' ]]
01:08:46.240  + run_flamegraph
01:08:46.240  + create_timing_file
01:08:46.240  + local array file
01:08:46.240  + array=($(find $WORKSPACE -type f -name 'timing.txt'))
01:08:46.240  ++ find /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11 -type f -name timing.txt
01:08:46.498  + timing_dir=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/post_process
01:08:46.498  + mkdir -p /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/post_process
01:08:46.498  + timing_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/post_process/total_timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498
01:08:46.498  + job_name=vhost-initiator-vg-autotest_77498
01:08:46.498  + sed 's/^/vhost-initiator-vg-autotest_77498;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206
01:08:46.498  + job_name=vfio-user-phy-autotest_26206
01:08:46.498  + sed 's/^/vfio-user-phy-autotest_26206;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626
01:08:46.498  + job_name=clang-vg-autotest_57626
01:08:46.498  + sed 's/^/clang-vg-autotest_57626;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/clang-vg-autotest_57626/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620
01:08:46.498  + job_name=vmd-phy-autotest_2620
01:08:46.498  + sed 's/^/vmd-phy-autotest_2620;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204
01:08:46.498  + job_name=BlobFS-phy-autotest_26204
01:08:46.498  + sed 's/^/BlobFS-phy-autotest_26204;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811
01:08:46.498  + job_name=ubuntu22-vg-autotest_37811
01:08:46.498  + sed 's/^/ubuntu22-vg-autotest_37811;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198
01:08:46.498  + job_name=crypto-phy-autotest_26198
01:08:46.498  + sed 's/^/crypto-phy-autotest_26198;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418
01:08:46.498  + job_name=autobuild-vg-autotest_63418
01:08:46.498  + sed 's/^/autobuild-vg-autotest_63418;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/autobuild-vg-autotest_63418/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370
01:08:46.498  + job_name=nvme-phy-autotest_72370
01:08:46.498  + sed 's/^/nvme-phy-autotest_72370;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-phy-autotest_72370/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516
01:08:46.498  + job_name=iscsi-vg-autotest_77516
01:08:46.498  + sed 's/^/iscsi-vg-autotest_77516;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088
01:08:46.498  + job_name=rocky9-vg-autotest_16088
01:08:46.498  + sed 's/^/rocky9-vg-autotest_16088;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375
01:08:46.498  + job_name=unittest-vg-autotest_63375
01:08:46.498  + sed 's/^/unittest-vg-autotest_63375;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898
01:08:46.498  + job_name=dsa-phy-autotest_4898
01:08:46.498  + sed 's/^/dsa-phy-autotest_4898;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192
01:08:46.498  + job_name=valgrind-vg-autotest_49192
01:08:46.498  + sed 's/^/valgrind-vg-autotest_49192;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329
01:08:46.498  + job_name=nvmf-tcp-phy-autotest_72329
01:08:46.498  + sed 's/^/nvmf-tcp-phy-autotest_72329;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442
01:08:46.498  + job_name=nvmf-phy-autotest_72442
01:08:46.498  + sed 's/^/nvmf-phy-autotest_72442;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-phy-autotest_72442/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383
01:08:46.498  + job_name=iscsi-uring-vg-autotest_77383
01:08:46.498  + sed 's/^/iscsi-uring-vg-autotest_77383;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657
01:08:46.498  + job_name=centos7-vg-autotest_73657
01:08:46.498  + sed 's/^/centos7-vg-autotest_73657;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377
01:08:46.498  + job_name=scanbuild-vg-autotest_63377
01:08:46.498  + sed 's/^/scanbuild-vg-autotest_63377;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/scanbuild-vg-autotest_63377/timing.txt
01:08:46.498  + for file in "${array[@]}"
01:08:46.498  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/timing.txt
01:08:46.498  + local job_file job_name
01:08:46.498  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/timing.txt
01:08:46.498  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/timing.txt
01:08:46.498  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584
01:08:46.498  + job_name=zns-vg-autotest_76584
01:08:46.499  + sed 's/^/zns-vg-autotest_76584;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/timing.txt
01:08:46.499  + for file in "${array[@]}"
01:08:46.499  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/timing.txt
01:08:46.499  + local job_file job_name
01:08:46.499  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/timing.txt
01:08:46.499  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/timing.txt
01:08:46.499  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427
01:08:46.499  + job_name=lvol-vg-autotest_77427
01:08:46.499  + sed 's/^/lvol-vg-autotest_77427;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/timing.txt
01:08:46.499  + for file in "${array[@]}"
01:08:46.499  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/timing.txt
01:08:46.499  + local job_file job_name
01:08:46.499  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/timing.txt
01:08:46.499  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/timing.txt
01:08:46.499  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469
01:08:46.499  + job_name=nvmf-tcp-vg-autotest_77469
01:08:46.499  + sed 's/^/nvmf-tcp-vg-autotest_77469;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/timing.txt
01:08:46.499  + for file in "${array[@]}"
01:08:46.499  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/timing.txt
01:08:46.499  + local job_file job_name
01:08:46.499  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/timing.txt
01:08:46.499  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/timing.txt
01:08:46.499  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195
01:08:46.499  + job_name=short-fuzz-phy-autotest_26195
01:08:46.499  + sed 's/^/short-fuzz-phy-autotest_26195;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/short-fuzz-phy-autotest_26195/timing.txt
01:08:46.499  + for file in "${array[@]}"
01:08:46.499  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/timing.txt
01:08:46.499  + local job_file job_name
01:08:46.499  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/timing.txt
01:08:46.499  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/timing.txt
01:08:46.499  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601
01:08:46.499  + job_name=nvme-cmb-pmr-vg-autotest_78601
01:08:46.499  + sed 's/^/nvme-cmb-pmr-vg-autotest_78601;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/timing.txt
01:08:46.499  + for file in "${array[@]}"
01:08:46.499  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/timing.txt
01:08:46.499  + local job_file job_name
01:08:46.499  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/timing.txt
01:08:46.499  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/timing.txt
01:08:46.499  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478
01:08:46.499  + job_name=nvme-vg-autotest_77478
01:08:46.499  + sed 's/^/nvme-vg-autotest_77478;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-vg-autotest_77478/timing.txt
01:08:46.499  + for file in "${array[@]}"
01:08:46.499  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/timing.txt
01:08:46.499  + local job_file job_name
01:08:46.499  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/timing.txt
01:08:46.499  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/timing.txt
01:08:46.499  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223
01:08:46.499  + job_name=vhost-phy-autotest_26223
01:08:46.499  + sed 's/^/vhost-phy-autotest_26223;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/timing.txt
01:08:46.499  + for file in "${array[@]}"
01:08:46.499  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/timing.txt
01:08:46.499  + local job_file job_name
01:08:46.499  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/timing.txt
01:08:46.499  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/timing.txt
01:08:46.499  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797
01:08:46.499  + job_name=ubuntu20-vg-autotest_73797
01:08:46.499  + sed 's/^/ubuntu20-vg-autotest_73797;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu20-vg-autotest_73797/timing.txt
01:08:46.499  + for file in "${array[@]}"
01:08:46.499  + add_sub_job_timing /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/timing.txt
01:08:46.499  + local job_file job_name
01:08:46.499  + job_file=/var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/timing.txt
01:08:46.499  +++ dirname /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/timing.txt
01:08:46.499  ++ basename /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403
01:08:46.499  + job_name=nvmf-tcp-uring-vg-autotest_77403
01:08:46.499  + sed 's/^/nvmf-tcp-uring-vg-autotest_77403;autorun;/' /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/timing.txt
01:08:46.499  + /usr/local/FlameGraph/flamegraph.pl --title 'Build Timing' --nametype Step: --countname seconds /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/post_process/total_timing.txt
01:08:46.499  + run_autorun_post
01:08:46.499  + s_opt=
01:08:46.499  + [[ '' == true ]]
01:08:46.499  + spdk/autorun_post.py -d ./ -r ./spdk
01:08:56.462  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/module/blobfs/bdev/blobfs_bdev.c:254
01:09:11.346  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/module/blobfs/bdev/blobfs_bdev.c:254
01:09:17.894  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/module/blobfs/bdev/blobfs_bdev.c:254
01:09:27.855  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/lib/util/crc16.c:628
01:09:27.855  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/lib/util/crc16.c:634
01:09:27.855  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/lib/util/crc64.c:165
01:09:27.855  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/lib/util/xor.c:120
01:09:27.855  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/lib/util/crc32c.c:21
01:09:40.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/module/bdev/nvme/bdev_mdns_client.c:613
01:09:40.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/module/bdev/nvme/bdev_mdns_client.c:561
01:09:40.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/module/bdev/nvme/bdev_mdns_client.c:456
01:09:40.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/module/bdev/nvme/bdev_mdns_client.c:582
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/mmio.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/endian.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/nvme.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/histogram_data.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/trace.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/nvmf_transport.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/bdev_module.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/util.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/nvme_spec.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/thread.h!
01:10:06.559  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/base64.h!
01:10:07.124  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/histogram_data.h/histogram_ut.c!
01:10:09.020  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/lib/env_ocf/include/ocf/ocf_io.h!
01:10:09.020  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/lib/env_ocf/include/ocf/ocf_mngt.h!
01:10:09.020  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/spdk/lib/env_ocf/include/ocf/ocf_core.h!
01:10:23.878  -----Begin Post Process Script------
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-initiator-vg-autotest_77498/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vfio-user-phy-autotest_26206/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vmd-phy-autotest_2620/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/BlobFS-phy-autotest_26204/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/ubuntu22-vg-autotest_37811/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/crypto-phy-autotest_26198/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-vg-autotest_77516/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/rocky9-vg-autotest_16088/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/unittest-vg-autotest_63375/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/dsa-phy-autotest_4898/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/valgrind-vg-autotest_49192/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-phy-autotest_72329/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/iscsi-uring-vg-autotest_77383/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/centos7-vg-autotest_73657/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/zns-vg-autotest_76584/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/lvol-vg-autotest_77427/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-vg-autotest_77469/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvme-cmb-pmr-vg-autotest_78601/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/vhost-phy-autotest_26223/cov_total.info
01:10:23.878  /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/nvmf-tcp-uring-vg-autotest_77403/cov_total.info
01:10:23.878  
01:10:23.878  
01:10:23.878  -----Tests Executed in Build------
01:10:23.878  accel
01:10:23.878  accel_assign_opcode
01:10:23.878  accel_cdev_comp
01:10:23.878  accel_cdev_decmop_full
01:10:23.878  accel_cdev_decomp
01:10:23.878  accel_cdev_decomp_full_mcore
01:10:23.878  accel_cdev_decomp_full_mthread
01:10:23.878  accel_cdev_decomp_mcore
01:10:23.878  accel_cdev_decomp_mthread
01:10:23.878  accel_comp
01:10:23.878  accel_compare
01:10:23.878  accel_compress_verify
01:10:23.878  accel_copy
01:10:23.878  accel_copy_crc32c
01:10:23.878  accel_copy_crc32c_C2
01:10:23.878  accel_crc32c
01:10:23.878  accel_crc32c_C2
01:10:23.878  accel_decmop_full
01:10:23.878  accel_decomp
01:10:23.878  accel_decomp_full_mcore
01:10:23.878  accel_decomp_full_mthread
01:10:23.878  accel_decomp_mcore
01:10:23.878  accel_decomp_mthread
01:10:23.878  accel_dif_functional_tests
01:10:23.878  accel_dif_generate
01:10:23.878  accel_dif_generate_copy
01:10:23.878  accel_dif_verify
01:10:23.878  accel_dualcast
01:10:23.878  accel_fill
01:10:23.878  accel_help
01:10:23.878  accel_missing_filename
01:10:23.878  accel_negative_buffers
01:10:23.878  accel_rpc
01:10:23.878  accel_scan_dsa_modules
01:10:23.878  accel_scan_iaa_modules
01:10:23.878  accel_wrong_workload
01:10:23.878  accel_xor
01:10:23.878  acl
01:10:23.878  alias_rpc
01:10:23.878  allowed
01:10:23.878  app_cmdline
01:10:23.878  app_repeat
01:10:23.878  asan
01:10:23.878  autobuild
01:10:23.878  autobuild_build_doc
01:10:23.878  autobuild_check_dpdk_pci_api
01:10:23.878  autobuild_check_format
01:10:23.878  autobuild_check_so_deps
01:10:23.878  autobuild_external_code
01:10:23.878  autobuild_generated_files_check
01:10:23.878  autobuild_generated_files_check_post_clean
01:10:23.878  autobuild_header_dependency_check
01:10:23.878  autobuild_llvm_precompile
01:10:23.878  autobuild_make_install
01:10:23.878  autobuild_make_uninstall
01:10:23.878  autobuild_ocf_precompile
01:10:23.878  bdev_bounds
01:10:23.878  bdev_crypto_enomem
01:10:23.878  bdev_error
01:10:23.878  bdev_fio
01:10:23.878  bdev_fio_rw_verify
01:10:23.878  bdev_fio_trim
01:10:23.878  bdev_gpt_uuid
01:10:23.878  bdev_hello_world
01:10:23.878  bdev_json_nonarray
01:10:23.878  bdev_json_nonenclosed
01:10:23.878  bdev_nbd
01:10:23.878  bdev_nvme_reset_stuck_adm_cmd
01:10:23.878  bdev_qd_sampling
01:10:23.878  bdev_qos
01:10:23.878  bdev_qos_bw
01:10:23.878  bdev_qos_iops
01:10:23.878  bdev_qos_ro_bw
01:10:23.878  bdev_raid
01:10:23.878  bdev_stat
01:10:23.878  bdev_verify
01:10:23.878  bdev_verify_big_io
01:10:23.878  bdev_write_zeroes
01:10:23.878  bdevperf_config
01:10:23.878  blob_io_wait
01:10:23.878  blobfs
01:10:23.878  blobfs_mkfs
01:10:23.878  blobstore
01:10:23.878  blobstore_grow
01:10:23.878  blockdev_crypto_aesni
01:10:23.878  blockdev_crypto_qat
01:10:23.878  blockdev_crypto_sw
01:10:23.878  blockdev_general
01:10:23.878  blockdev_nvme
01:10:23.878  blockdev_nvme_gpt
01:10:23.878  blockdev_raid5f
01:10:23.878  blockdev_rbd
01:10:23.878  build_native_dpdk
01:10:23.878  build_rpm_from_gen_spec
01:10:23.878  build_shared_native_dpdk_rpm
01:10:23.878  build_shared_rpm
01:10:23.878  build_shared_rpm_with_rpmed_dpdk
01:10:23.878  chaining
01:10:23.878  check_header_filenames
01:10:23.878  cmb_copy
01:10:23.878  compress_compdev
01:10:23.878  compress_isal
01:10:23.878  confirm_abi_deps
01:10:23.878  confirm_makefile_deps
01:10:23.878  cpu_locks
01:10:23.878  custom_alloc
01:10:23.878  dd_bs_lt_native_bs
01:10:23.878  dd_copy_to_out_bdev
01:10:23.878  dd_double_input
01:10:23.878  dd_double_output
01:10:23.878  dd_flag_append
01:10:23.878  dd_flag_append_forced_aio
01:10:23.878  dd_flag_directory
01:10:23.878  dd_flag_directory_forced_aio
01:10:23.878  dd_flag_noatime
01:10:23.878  dd_flag_noatime_forced_aio
01:10:23.878  dd_flag_nofollow
01:10:23.879  dd_flag_nofollow_forced_aio
01:10:23.879  dd_flags_misc
01:10:23.879  dd_flags_misc_forced_aio
01:10:23.879  dd_inflate_file
01:10:23.879  dd_invalid_arguments
01:10:23.879  dd_invalid_count
01:10:23.879  dd_invalid_iflag
01:10:23.879  dd_invalid_json
01:10:23.879  dd_invalid_oflag
01:10:23.879  dd_malloc_copy
01:10:23.879  dd_no_input
01:10:23.879  dd_no_output
01:10:23.879  dd_offset_magic
01:10:23.879  dd_rw
01:10:23.879  dd_rw_offset
01:10:23.879  dd_smaller_blocksize
01:10:23.879  dd_sparse_bdev_to_file
01:10:23.879  dd_sparse_file_to_bdev
01:10:23.879  dd_sparse_file_to_file
01:10:23.879  dd_unknown_flag
01:10:23.879  dd_uring_copy
01:10:23.879  dd_wrong_blocksize
01:10:23.879  default_locks
01:10:23.879  default_locks_via_rpc
01:10:23.879  default_setup
01:10:23.879  denied
01:10:23.879  devices
01:10:23.879  dm_mount
01:10:23.879  dma
01:10:23.879  dpdk_mem_utility
01:10:23.879  driver
01:10:23.879  env
01:10:23.879  env_dpdk_post_init
01:10:23.879  env_mem_callbacks
01:10:23.879  env_memory
01:10:23.879  env_pci
01:10:23.879  env_vtophys
01:10:23.879  even_2G_alloc
01:10:23.879  event
01:10:23.879  event_perf
01:10:23.879  event_reactor
01:10:23.879  event_reactor_perf
01:10:23.879  event_scheduler
01:10:23.879  exit_on_failed_rpc_init
01:10:23.879  external_make_accel_driver_shared
01:10:23.879  external_make_accel_driver_static
01:10:23.879  external_make_accel_module_shared
01:10:23.879  external_make_accel_module_static
01:10:23.879  external_make_hello_bdev_shared_combo
01:10:23.879  external_make_hello_bdev_shared_iso
01:10:23.879  external_make_hello_bdev_static
01:10:23.879  external_make_hello_no_bdev_shared_combo
01:10:23.879  external_make_hello_no_bdev_shared_iso
01:10:23.879  external_make_hello_no_bdev_static
01:10:23.879  external_make_nvme_shared
01:10:23.879  external_make_nvme_static
01:10:23.879  external_run_accel_driver_shared
01:10:23.879  external_run_accel_driver_static
01:10:23.879  external_run_accel_module_shared
01:10:23.879  external_run_accel_module_static
01:10:23.879  external_run_hello_bdev_shared_combo
01:10:23.879  external_run_hello_bdev_shared_iso
01:10:23.879  external_run_hello_bdev_static
01:10:23.879  external_run_hello_no_bdev_shared_combo
01:10:23.879  external_run_hello_no_bdev_shared_iso
01:10:23.879  external_run_hello_no_bdev_static
01:10:23.879  external_run_nvme_shared
01:10:23.879  external_run_nvme_static
01:10:23.879  filesystem_btrfs
01:10:23.879  filesystem_ext4
01:10:23.879  filesystem_in_capsule_btrfs
01:10:23.879  filesystem_in_capsule_ext4
01:10:23.879  filesystem_in_capsule_xfs
01:10:23.879  filesystem_xfs
01:10:23.879  fio_dif_1_default
01:10:23.879  fio_dif_1_multi_subsystems
01:10:23.879  fio_dif_digest
01:10:23.879  fio_dif_rand_params
01:10:23.879  go_rpc
01:10:23.879  guess_driver
01:10:23.879  hello_blob
01:10:23.879  hugepages
01:10:23.879  is_zoned
01:10:23.879  iscsi_tgt
01:10:23.879  iscsi_tgt_bdev_io_wait
01:10:23.879  iscsi_tgt_calsoft
01:10:23.879  iscsi_tgt_digest
01:10:23.879  iscsi_tgt_digests
01:10:23.879  iscsi_tgt_ext4test
01:10:23.879  iscsi_tgt_filesystem
01:10:23.879  iscsi_tgt_filesystem_btrfs
01:10:23.879  iscsi_tgt_filesystem_ext4
01:10:23.879  iscsi_tgt_filesystem_xfs
01:10:23.879  iscsi_tgt_fio
01:10:23.879  iscsi_tgt_fuzz
01:10:23.879  iscsi_tgt_initiator
01:10:23.879  iscsi_tgt_ip_migration
01:10:23.879  iscsi_tgt_iscsi_lvol
01:10:23.879  iscsi_tgt_login_redirection
01:10:23.879  iscsi_tgt_multiconnection
01:10:23.879  iscsi_tgt_qos
01:10:23.879  iscsi_tgt_rbd
01:10:23.879  iscsi_tgt_reset
01:10:23.879  iscsi_tgt_resize
01:10:23.879  iscsi_tgt_rpc_config
01:10:23.879  iscsi_tgt_sock
01:10:23.879  iscsi_tgt_trace_record
01:10:23.879  json_config
01:10:23.879  json_config_extra_key
01:10:23.879  kernel_target_abort
01:10:23.879  keyring_file
01:10:23.879  llvm_fuzz
01:10:23.879  locking_app_on_locked_coremask
01:10:23.879  locking_app_on_unlocked_coremask
01:10:23.879  locking_overlapped_coremask
01:10:23.879  locking_overlapped_coremask_via_rpc
01:10:23.879  lvol
01:10:23.879  lvol_basic
01:10:23.879  lvol_esnap
01:10:23.879  lvol_external_copy
01:10:23.879  lvol_external_snapshot
01:10:23.879  lvol_hotremove
01:10:23.879  lvol_provisioning
01:10:23.879  lvol_rename
01:10:23.879  lvol_resize
01:10:23.879  lvol_snapshot_clone
01:10:23.879  lvol_tasting
01:10:23.879  lvs_grow_clean
01:10:23.879  lvs_grow_dirty
01:10:23.879  make
01:10:23.879  no_shrink_alloc
01:10:23.879  non_locking_app_on_locked_coremask
01:10:23.879  nvme
01:10:23.879  nvme_arbitration
01:10:23.879  nvme_cli_cuse
01:10:23.879  nvme_cli_plugin
01:10:23.879  nvme_cmb
01:10:23.879  nvme_cuse
01:10:23.879  nvme_cuse_app
01:10:23.879  nvme_cuse_rpc
01:10:23.879  nvme_doorbell_aers
01:10:23.879  nvme_e2edp
01:10:23.879  nvme_err_injection
01:10:23.879  nvme_fdp
01:10:23.879  nvme_fio
01:10:23.879  nvme_flexible_data_placement
01:10:23.879  nvme_hello_world
01:10:23.879  nvme_identify
01:10:23.879  nvme_mount
01:10:23.879  nvme_multi_aen
01:10:23.879  nvme_multi_secondary
01:10:23.879  nvme_ns_manage_cuse
01:10:23.879  nvme_overhead
01:10:23.879  nvme_perf
01:10:23.879  nvme_pmr
01:10:23.879  nvme_pmr_persistence
01:10:23.879  nvme_reserve
01:10:23.879  nvme_reset
01:10:23.879  nvme_rpc
01:10:23.879  nvme_rpc_timeouts
01:10:23.879  nvme_scc
01:10:23.879  nvme_sgl
01:10:23.879  nvme_simple_copy
01:10:23.879  nvme_single_aen
01:10:23.879  nvme_smartctl_cuse
01:10:23.879  nvme_startup
01:10:23.879  nvme_zns
01:10:23.879  nvmf_abort
01:10:23.879  nvmf_abort_qd_sizes
01:10:23.879  nvmf_aer
01:10:23.879  nvmf_async_init
01:10:23.879  nvmf_auth_host
01:10:23.879  nvmf_auth_target
01:10:23.879  nvmf_bdev_io_wait
01:10:23.879  nvmf_bdevio
01:10:23.879  nvmf_bdevio_no_huge
01:10:23.879  nvmf_bdevperf
01:10:23.879  nvmf_connect_disconnect
01:10:23.879  nvmf_connect_stress
01:10:23.879  nvmf_delete_subsystem
01:10:23.879  nvmf_device_removal
01:10:23.879  nvmf_device_removal_pci_remove
01:10:23.879  nvmf_device_removal_pci_remove_no_srq
01:10:23.879  nvmf_dif
01:10:23.879  nvmf_digest
01:10:23.879  nvmf_digest_clean
01:10:23.879  nvmf_digest_error
01:10:23.879  nvmf_discovery_remove_ifc
01:10:23.879  nvmf_example
01:10:23.879  nvmf_failover
01:10:23.879  nvmf_filesystem
01:10:23.879  nvmf_filesystem_in_capsule
01:10:23.879  nvmf_filesystem_no_in_capsule
01:10:23.879  nvmf_fio_host
01:10:23.879  nvmf_fio_target
01:10:23.879  nvmf_fips
01:10:23.879  nvmf_fused_ordering
01:10:23.879  nvmf_fuzz
01:10:23.879  nvmf_host_discovery
01:10:23.879  nvmf_host_management
01:10:23.879  nvmf_host_multipath
01:10:23.879  nvmf_host_multipath_status
01:10:23.879  nvmf_identify
01:10:23.879  nvmf_identify_kernel_target
01:10:23.879  nvmf_identify_passthru
01:10:23.879  nvmf_initiator_timeout
01:10:23.879  nvmf_invalid
01:10:23.879  nvmf_lvol
01:10:23.879  nvmf_lvs_grow
01:10:23.879  nvmf_mdns_discovery
01:10:23.879  nvmf_multiconnection
01:10:23.879  nvmf_multicontroller
01:10:23.879  nvmf_multitarget
01:10:23.879  nvmf_nmic
01:10:23.879  nvmf_ns_hotplug_stress
01:10:23.879  nvmf_ns_masking
01:10:23.879  nvmf_nvme_cli
01:10:23.879  nvmf_perf
01:10:23.879  nvmf_perf_adq
01:10:23.879  nvmf_queue_depth
01:10:23.879  nvmf_referrals
01:10:23.879  nvmf_rpc
01:10:23.879  nvmf_shutdown
01:10:23.879  nvmf_shutdown_tc1
01:10:23.879  nvmf_shutdown_tc2
01:10:23.879  nvmf_shutdown_tc3
01:10:23.879  nvmf_srq_overwhelm
01:10:23.879  nvmf_target_disconnect
01:10:23.879  nvmf_target_disconnect_tc1
01:10:23.879  nvmf_target_disconnect_tc2
01:10:23.879  nvmf_target_discovery
01:10:23.879  nvmf_target_multipath
01:10:23.879  nvmf_tcp
01:10:23.879  nvmf_timeout
01:10:23.879  nvmf_tls
01:10:23.879  nvmf_vfio_user
01:10:23.879  nvmf_vfio_user_fuzz
01:10:23.879  nvmf_vfio_user_nvme_compliance
01:10:23.879  nvmf_vhost
01:10:23.879  nvmf_zcopy
01:10:23.879  odd_alloc
01:10:23.879  packaging
01:10:23.879  per_node_1G_alloc
01:10:23.879  raid0_resize_test
01:10:23.879  raid5f_rebuild_test
01:10:23.879  raid5f_rebuild_test_sb
01:10:23.879  raid5f_state_function_test
01:10:23.879  raid5f_state_function_test_sb
01:10:23.879  raid5f_superblock_test
01:10:23.879  raid_function_test_concat
01:10:23.879  raid_function_test_raid0
01:10:23.879  raid_rebuild_test
01:10:23.879  raid_rebuild_test_io
01:10:23.879  raid_rebuild_test_sb
01:10:23.879  raid_rebuild_test_sb_4k
01:10:23.879  raid_rebuild_test_sb_io
01:10:23.879  raid_rebuild_test_sb_md_interleaved
01:10:23.879  raid_rebuild_test_sb_md_separate
01:10:23.879  raid_state_function_test
01:10:23.880  raid_state_function_test_sb
01:10:23.880  raid_state_function_test_sb_4k
01:10:23.880  raid_state_function_test_sb_md_interleaved
01:10:23.880  raid_state_function_test_sb_md_separate
01:10:23.880  raid_superblock_test
01:10:23.880  raid_superblock_test_4k
01:10:23.880  raid_superblock_test_md_interleaved
01:10:23.880  raid_superblock_test_md_separate
01:10:23.880  reactor_set_interrupt
01:10:23.880  reap_unregistered_poller
01:10:23.880  rocksdb
01:10:23.880  rocksdb_insert
01:10:23.880  rocksdb_overwrite
01:10:23.880  rocksdb_randread
01:10:23.880  rocksdb_readwrite
01:10:23.880  rocksdb_writesync
01:10:23.880  rpc
01:10:23.880  rpc_client
01:10:23.880  rpc_daemon_integrity
01:10:23.880  rpc_integrity
01:10:23.880  rpc_plugins
01:10:23.880  rpc_trace_cmd_test
01:10:23.880  rpm_packaging
01:10:23.880  scanbuild_make
01:10:23.880  scheduler_create_thread
01:10:23.880  setup.sh
01:10:23.880  skip_rpc
01:10:23.880  skip_rpc_with_delay
01:10:23.880  skip_rpc_with_json
01:10:23.880  sma
01:10:23.880  sma_crypto
01:10:23.880  sma_discovery
01:10:23.880  sma_nvmf_tcp
01:10:23.880  sma_plugins
01:10:23.880  sma_qos
01:10:23.880  sma_vfiouser_qemu
01:10:23.880  sma_vhost
01:10:23.880  spdk_dd
01:10:23.880  spdk_dd_basic_rw
01:10:23.880  spdk_dd_bdev_to_bdev
01:10:23.880  spdk_dd_malloc
01:10:23.880  spdk_dd_negative
01:10:23.880  spdk_dd_posix
01:10:23.880  spdk_dd_sparse
01:10:23.880  spdk_dd_uring
01:10:23.880  spdk_target_abort
01:10:23.880  spdkcli_iscsi
01:10:23.880  spdkcli_nvmf_tcp
01:10:23.880  spdkcli_raid
01:10:23.880  spdkcli_rbd
01:10:23.880  spdkcli_tcp
01:10:23.880  spdkcli_vhost
01:10:23.880  spdkcli_virtio
01:10:23.880  sw_hotplug
01:10:23.880  test_bdev_lvol_delete_lvstore_with_clones
01:10:23.880  test_bdev_lvol_delete_ordering
01:10:23.880  test_clone_decouple_parent
01:10:23.880  test_clone_inflate
01:10:23.880  test_clone_snapshot_relations
01:10:23.880  test_construct_lvol
01:10:23.880  test_construct_lvol_alias_conflict
01:10:23.880  test_construct_lvol_fio_clear_method_none
01:10:23.880  test_construct_lvol_fio_clear_method_unmap
01:10:23.880  test_construct_lvol_full_lvs
01:10:23.880  test_construct_lvol_inexistent_lvs
01:10:23.880  test_construct_lvols_conflict_alias
01:10:23.880  test_construct_lvs
01:10:23.880  test_construct_lvs_clear_methods
01:10:23.880  test_construct_lvs_conflict_alias
01:10:23.880  test_construct_lvs_different_cluster_size
01:10:23.880  test_construct_lvs_nonexistent_bdev
01:10:23.880  test_construct_multi_lvols
01:10:23.880  test_construct_nested_lvol
01:10:23.880  test_construct_two_lvs_on_the_same_bdev
01:10:23.880  test_create_snapshot_of_snapshot
01:10:23.880  test_create_snapshot_with_io
01:10:23.880  test_delete_lvol_store_persistent_positive
01:10:23.880  test_delete_snapshot_with_clone
01:10:23.880  test_delete_snapshot_with_snapshot
01:10:23.880  test_destroy_after_bdev_lvol_resize_positive
01:10:23.880  test_esnap_clones
01:10:23.880  test_esnap_late_arrival
01:10:23.880  test_esnap_reload
01:10:23.880  test_esnap_remove_degraded
01:10:23.880  test_hotremove_lvol_store
01:10:23.880  test_hotremove_lvol_store_base
01:10:23.880  test_hotremove_lvol_store_base_with_lvols
01:10:23.880  test_hotremove_lvol_store_multiple_lvols
01:10:23.880  test_lvol_bdev_readonly
01:10:23.880  test_lvol_list
01:10:23.880  test_lvol_rename_negative
01:10:23.880  test_rename_lvs_negative
01:10:23.880  test_rename_positive
01:10:23.880  test_resize_lvol
01:10:23.880  test_resize_lvol_negative
01:10:23.880  test_resize_lvol_with_io_traffic
01:10:23.880  test_shallow_copy_compare
01:10:23.880  test_sigterm
01:10:23.880  test_snapshot_compare_with_lvol_bdev
01:10:23.880  test_tasting
01:10:23.880  test_thin_lvol_check_integrity
01:10:23.880  test_thin_lvol_check_space
01:10:23.880  test_thin_lvol_check_zeroes
01:10:23.880  test_thin_lvol_resize
01:10:23.880  test_thin_overprovisioning
01:10:23.880  test_unregister_lvol_bdev
01:10:23.880  thread
01:10:23.880  thread_poller_perf
01:10:23.880  thread_spdk_lock
01:10:23.880  ubsan
01:10:23.880  unittest
01:10:23.880  unittest_accel
01:10:23.880  unittest_bdev
01:10:23.880  unittest_bdev_compress
01:10:23.880  unittest_bdev_crypto
01:10:23.880  unittest_bdev_raid5f
01:10:23.880  unittest_blob_blobfs
01:10:23.880  unittest_build
01:10:23.880  unittest_dma
01:10:23.880  unittest_dpdk_compressdev
01:10:23.880  unittest_event
01:10:23.880  unittest_ftl
01:10:23.880  unittest_idxd_user
01:10:23.880  unittest_include
01:10:23.880  unittest_init
01:10:23.880  unittest_ioat
01:10:23.880  unittest_iobuf
01:10:23.880  unittest_iscsi
01:10:23.880  unittest_json
01:10:23.880  unittest_keyring
01:10:23.880  unittest_lib_reduce
01:10:23.880  unittest_log
01:10:23.880  unittest_lvol
01:10:23.880  unittest_notify
01:10:23.880  unittest_nvme
01:10:23.880  unittest_nvme_cuse
01:10:23.880  unittest_nvme_rdma
01:10:23.880  unittest_nvmf
01:10:23.880  unittest_nvmf_rdma
01:10:23.880  unittest_nvmf_transport
01:10:23.880  unittest_nvmf_vfio_user
01:10:23.880  unittest_pci_event
01:10:23.880  unittest_rdma
01:10:23.880  unittest_rpc
01:10:23.880  unittest_scsi
01:10:23.880  unittest_sock
01:10:23.880  unittest_thread
01:10:23.880  unittest_util
01:10:23.880  unittest_vhost
01:10:23.880  valgrind
01:10:23.880  version
01:10:23.880  vfio_fuzz
01:10:23.880  vfio_user_nvme_fio
01:10:23.880  vfio_user_nvme_restart_vm
01:10:23.880  vfio_user_qemu
01:10:23.880  vfio_user_virtio_bdevperf
01:10:23.880  vfio_user_virtio_blk_restart_vm
01:10:23.880  vfio_user_virtio_scsi_restart_vm
01:10:23.880  vhost
01:10:23.880  vhost_blk_cores_2ctrl
01:10:23.880  vhost_blk_fs_integrity
01:10:23.880  vhost_blk_integrity
01:10:23.880  vhost_blk_lvol_integrity
01:10:23.880  vhost_blk_packed_ring_integrity
01:10:23.880  vhost_blockdev
01:10:23.880  vhost_boot
01:10:23.880  vhost_fuzz
01:10:23.880  vhost_migration
01:10:23.880  vhost_migration_tc1
01:10:23.880  vhost_migration_tc2
01:10:23.880  vhost_negative
01:10:23.880  vhost_readonly
01:10:23.880  vhost_scsi_cores_2ctrl
01:10:23.880  vhost_scsi_fs_integrity
01:10:23.880  vhost_scsi_integrity
01:10:23.880  vhost_scsi_lvol_integrity
01:10:23.880  vhost_shared
01:10:23.880  vmd
01:10:23.880  vmd_bdev_svc
01:10:23.880  vmd_fio
01:10:23.880  vmd_hello_world
01:10:23.880  vmd_identify
01:10:23.880  vmd_perf
01:10:23.880  zoned_fio
01:10:23.880  
01:10:23.880  
01:10:23.880  -----Tests Missing From Build------
01:10:23.880  balanced
01:10:23.880  blockdev_xnvme
01:10:23.880  busy
01:10:23.880  core_load
01:10:23.880  dpdk_governor
01:10:23.880  ftl
01:10:23.880  ftl_bdevperf
01:10:23.880  ftl_dirty_shutdown
01:10:23.880  ftl_fio_basic
01:10:23.881  ftl_restore
01:10:23.881  ftl_restore_fast
01:10:23.881  ftl_trim
01:10:23.881  ftl_upgrade_shutdown
01:10:23.881  idle
01:10:23.881  interrupt_mode
01:10:23.881  ioat
01:10:23.881  ioat_perf
01:10:23.881  ioat_verify
01:10:23.881  iscsi_tgt_data_digest
01:10:23.881  keyring_linux
01:10:23.881  nvme_boot_partition
01:10:23.881  nvme_bp
01:10:23.881  nvme_opal
01:10:23.881  nvme_opal_bdevio
01:10:23.881  nvme_opal_bdevperf
01:10:23.881  nvme_opal_spdk_tgt
01:10:23.881  nvme_xnvme
01:10:23.881  nvmf_digest_dsa_initiator
01:10:23.881  nvmf_digest_dsa_target
01:10:23.881  nvmf_fc
01:10:23.881  nvmf_rdma
01:10:23.881  nvmf_target_disconnect_tc3
01:10:23.881  ocf
01:10:23.881  ocf_bdevperf_iotypes
01:10:23.881  ocf_configuration_change
01:10:23.881  ocf_create_destruct
01:10:23.881  ocf_fio_modes
01:10:23.881  ocf_flush
01:10:23.881  ocf_multicore
01:10:23.881  ocf_remove
01:10:23.881  ocf_stats
01:10:23.881  scheduler
01:10:23.881  spdkcli_nvmf_fc
01:10:23.881  spdkcli_nvmf_rdma
01:10:23.881  test_create_multi_ublk
01:10:23.881  test_create_ublk
01:10:23.881  test_save_ublk_config
01:10:23.881  ublk
01:10:23.881  ublk_recovery
01:10:23.881  unittest_nvmf_fc
01:10:23.881  unittest_nvmf_fc_ls
01:10:23.881  vhost_blk_hot_remove
01:10:23.881  vhost_hotplug
01:10:23.881  vhost_scsi_hot_remove
01:10:23.881  xnvme_bdevperf
01:10:23.881  xnvme_to_malloc_dd_copy
01:10:23.881  
01:10:23.881  
01:10:23.881  -----Tests Missing ASAN------
01:10:23.881  accel_cdev_comp
01:10:23.881  accel_cdev_decmop_full
01:10:23.881  accel_cdev_decomp
01:10:23.881  accel_cdev_decomp_full_mcore
01:10:23.881  accel_cdev_decomp_full_mthread
01:10:23.881  accel_cdev_decomp_mcore
01:10:23.881  accel_cdev_decomp_mthread
01:10:23.881  autobuild_llvm_precompile
01:10:23.881  balanced
01:10:23.881  bdev_crypto_enomem
01:10:23.881  blockdev_crypto_aesni
01:10:23.881  blockdev_crypto_qat
01:10:23.881  blockdev_crypto_sw
01:10:23.881  blockdev_rbd
01:10:23.881  blockdev_xnvme
01:10:23.881  busy
01:10:23.881  chaining
01:10:23.881  cmb_copy
01:10:23.881  compress_compdev
01:10:23.881  compress_isal
01:10:23.881  core_load
01:10:23.881  dd_uring_copy
01:10:23.881  dma
01:10:23.881  dpdk_governor
01:10:23.881  filesystem_btrfs
01:10:23.881  filesystem_ext4
01:10:23.881  filesystem_in_capsule_btrfs
01:10:23.881  filesystem_in_capsule_ext4
01:10:23.881  filesystem_in_capsule_xfs
01:10:23.881  filesystem_xfs
01:10:23.881  fio_dif_1_default
01:10:23.881  fio_dif_1_multi_subsystems
01:10:23.881  fio_dif_digest
01:10:23.881  fio_dif_rand_params
01:10:23.881  ftl
01:10:23.881  ftl_bdevperf
01:10:23.881  ftl_dirty_shutdown
01:10:23.881  ftl_fio_basic
01:10:23.881  ftl_restore
01:10:23.881  ftl_restore_fast
01:10:23.881  ftl_trim
01:10:23.881  ftl_upgrade_shutdown
01:10:23.881  go_rpc
01:10:23.881  idle
01:10:23.881  interrupt_mode
01:10:23.881  ioat
01:10:23.881  ioat_perf
01:10:23.881  ioat_verify
01:10:23.881  is_zoned
01:10:23.881  iscsi_tgt
01:10:23.881  iscsi_tgt_bdev_io_wait
01:10:23.881  iscsi_tgt_calsoft
01:10:23.881  iscsi_tgt_data_digest
01:10:23.881  iscsi_tgt_digest
01:10:23.881  iscsi_tgt_digests
01:10:23.881  iscsi_tgt_ext4test
01:10:23.881  iscsi_tgt_filesystem
01:10:23.881  iscsi_tgt_filesystem_btrfs
01:10:23.881  iscsi_tgt_filesystem_ext4
01:10:23.881  iscsi_tgt_filesystem_xfs
01:10:23.881  iscsi_tgt_fio
01:10:23.881  iscsi_tgt_fuzz
01:10:23.881  iscsi_tgt_initiator
01:10:23.881  iscsi_tgt_ip_migration
01:10:23.881  iscsi_tgt_iscsi_lvol
01:10:23.881  iscsi_tgt_login_redirection
01:10:23.881  iscsi_tgt_multiconnection
01:10:23.881  iscsi_tgt_qos
01:10:23.881  iscsi_tgt_rbd
01:10:23.881  iscsi_tgt_reset
01:10:23.881  iscsi_tgt_resize
01:10:23.881  iscsi_tgt_rpc_config
01:10:23.881  iscsi_tgt_sock
01:10:23.881  iscsi_tgt_trace_record
01:10:23.881  kernel_target_abort
01:10:23.881  keyring_file
01:10:23.881  keyring_linux
01:10:23.881  llvm_fuzz
01:10:23.881  lvs_grow_clean
01:10:23.881  lvs_grow_dirty
01:10:23.881  nvme_boot_partition
01:10:23.881  nvme_bp
01:10:23.881  nvme_cli_cuse
01:10:23.881  nvme_cli_plugin
01:10:23.881  nvme_cmb
01:10:23.881  nvme_cuse
01:10:23.881  nvme_cuse_app
01:10:23.881  nvme_cuse_rpc
01:10:23.881  nvme_ns_manage_cuse
01:10:23.881  nvme_opal
01:10:23.881  nvme_opal_bdevio
01:10:23.881  nvme_opal_bdevperf
01:10:23.881  nvme_opal_spdk_tgt
01:10:23.881  nvme_pmr
01:10:23.881  nvme_pmr_persistence
01:10:23.881  nvme_smartctl_cuse
01:10:23.881  nvme_xnvme
01:10:23.881  nvme_zns
01:10:23.881  nvmf_abort
01:10:23.881  nvmf_abort_qd_sizes
01:10:23.881  nvmf_aer
01:10:23.881  nvmf_async_init
01:10:23.881  nvmf_auth_host
01:10:23.881  nvmf_auth_target
01:10:23.881  nvmf_bdev_io_wait
01:10:23.881  nvmf_bdevio
01:10:23.881  nvmf_bdevio_no_huge
01:10:23.881  nvmf_bdevperf
01:10:23.881  nvmf_connect_disconnect
01:10:23.881  nvmf_connect_stress
01:10:23.881  nvmf_delete_subsystem
01:10:23.881  nvmf_device_removal
01:10:23.881  nvmf_device_removal_pci_remove
01:10:23.881  nvmf_device_removal_pci_remove_no_srq
01:10:23.881  nvmf_dif
01:10:23.881  nvmf_digest
01:10:23.881  nvmf_digest_clean
01:10:23.881  nvmf_digest_dsa_initiator
01:10:23.881  nvmf_digest_dsa_target
01:10:23.881  nvmf_digest_error
01:10:23.881  nvmf_discovery_remove_ifc
01:10:23.881  nvmf_example
01:10:23.881  nvmf_failover
01:10:23.881  nvmf_fc
01:10:23.881  nvmf_filesystem
01:10:23.881  nvmf_filesystem_in_capsule
01:10:23.881  nvmf_filesystem_no_in_capsule
01:10:23.881  nvmf_fio_host
01:10:23.881  nvmf_fio_target
01:10:23.881  nvmf_fips
01:10:23.881  nvmf_fused_ordering
01:10:23.881  nvmf_fuzz
01:10:23.881  nvmf_host_discovery
01:10:23.881  nvmf_host_management
01:10:23.881  nvmf_host_multipath
01:10:23.881  nvmf_host_multipath_status
01:10:23.881  nvmf_identify
01:10:23.881  nvmf_identify_kernel_target
01:10:23.881  nvmf_identify_passthru
01:10:23.881  nvmf_initiator_timeout
01:10:23.881  nvmf_invalid
01:10:23.881  nvmf_lvol
01:10:23.881  nvmf_lvs_grow
01:10:23.881  nvmf_mdns_discovery
01:10:23.881  nvmf_multiconnection
01:10:23.881  nvmf_multicontroller
01:10:23.881  nvmf_multitarget
01:10:23.881  nvmf_nmic
01:10:23.881  nvmf_ns_hotplug_stress
01:10:23.881  nvmf_ns_masking
01:10:23.881  nvmf_nvme_cli
01:10:23.881  nvmf_perf
01:10:23.881  nvmf_perf_adq
01:10:23.881  nvmf_queue_depth
01:10:23.881  nvmf_rdma
01:10:24.447  nvmf_referrals
01:10:24.447  nvmf_rpc
01:10:24.447  nvmf_shutdown
01:10:24.447  nvmf_shutdown_tc1
01:10:24.447  nvmf_shutdown_tc2
01:10:24.447  nvmf_shutdown_tc3
01:10:24.447  nvmf_srq_overwhelm
01:10:24.447  nvmf_target_disconnect
01:10:24.447  nvmf_target_disconnect_tc1
01:10:24.447  nvmf_target_disconnect_tc2
01:10:24.447  nvmf_target_disconnect_tc3
01:10:24.447  nvmf_target_discovery
01:10:24.447  nvmf_target_multipath
01:10:24.447  nvmf_tcp
01:10:24.447  nvmf_timeout
01:10:24.447  nvmf_tls
01:10:24.447  nvmf_vfio_user
01:10:24.447  nvmf_vfio_user_fuzz
01:10:24.447  nvmf_vfio_user_nvme_compliance
01:10:24.447  nvmf_zcopy
01:10:24.447  ocf
01:10:24.447  ocf_bdevperf_iotypes
01:10:24.447  ocf_configuration_change
01:10:24.447  ocf_create_destruct
01:10:24.447  ocf_fio_modes
01:10:24.447  ocf_flush
01:10:24.447  ocf_multicore
01:10:24.447  ocf_remove
01:10:24.447  ocf_stats
01:10:24.447  scheduler
01:10:24.447  spdk_dd_uring
01:10:24.447  spdk_target_abort
01:10:24.447  spdkcli_iscsi
01:10:24.447  spdkcli_nvmf_fc
01:10:24.447  spdkcli_nvmf_rdma
01:10:24.447  spdkcli_nvmf_tcp
01:10:24.447  spdkcli_raid
01:10:24.447  spdkcli_rbd
01:10:24.447  spdkcli_virtio
01:10:24.447  test_create_multi_ublk
01:10:24.447  test_create_ublk
01:10:24.447  test_save_ublk_config
01:10:24.447  ublk
01:10:24.447  ublk_recovery
01:10:24.447  unittest_nvmf_fc
01:10:24.447  unittest_nvmf_fc_ls
01:10:24.447  valgrind
01:10:24.447  vfio_fuzz
01:10:24.447  vhost_blk_hot_remove
01:10:24.447  vhost_blockdev
01:10:24.447  vhost_fuzz
01:10:24.447  vhost_hotplug
01:10:24.447  vhost_scsi_hot_remove
01:10:24.447  vhost_shared
01:10:24.447  xnvme_bdevperf
01:10:24.447  xnvme_to_malloc_dd_copy
01:10:24.447  zoned_fio
01:10:24.447  
01:10:24.447  
01:10:24.447  -----Tests Missing UBSAN------
01:10:24.447  balanced
01:10:24.447  blockdev_xnvme
01:10:24.447  busy
01:10:24.447  cmb_copy
01:10:24.447  core_load
01:10:24.447  dpdk_governor
01:10:24.447  ftl
01:10:24.447  ftl_bdevperf
01:10:24.447  ftl_dirty_shutdown
01:10:24.447  ftl_fio_basic
01:10:24.447  ftl_restore
01:10:24.447  ftl_restore_fast
01:10:24.447  ftl_trim
01:10:24.447  ftl_upgrade_shutdown
01:10:24.447  idle
01:10:24.447  interrupt_mode
01:10:24.447  ioat
01:10:24.447  ioat_perf
01:10:24.447  ioat_verify
01:10:24.447  is_zoned
01:10:24.447  iscsi_tgt_data_digest
01:10:24.447  keyring_linux
01:10:24.447  nvme_boot_partition
01:10:24.447  nvme_bp
01:10:24.447  nvme_cmb
01:10:24.447  nvme_opal
01:10:24.447  nvme_opal_bdevio
01:10:24.447  nvme_opal_bdevperf
01:10:24.447  nvme_opal_spdk_tgt
01:10:24.447  nvme_pmr
01:10:24.447  nvme_pmr_persistence
01:10:24.447  nvme_xnvme
01:10:24.447  nvme_zns
01:10:24.447  nvmf_digest_dsa_initiator
01:10:24.447  nvmf_digest_dsa_target
01:10:24.447  nvmf_fc
01:10:24.447  nvmf_rdma
01:10:24.447  nvmf_target_disconnect_tc3
01:10:24.447  ocf
01:10:24.447  ocf_bdevperf_iotypes
01:10:24.447  ocf_configuration_change
01:10:24.447  ocf_create_destruct
01:10:24.447  ocf_fio_modes
01:10:24.447  ocf_flush
01:10:24.447  ocf_multicore
01:10:24.447  ocf_remove
01:10:24.447  ocf_stats
01:10:24.447  scheduler
01:10:24.447  spdkcli_nvmf_fc
01:10:24.447  spdkcli_nvmf_rdma
01:10:24.447  test_create_multi_ublk
01:10:24.447  test_create_ublk
01:10:24.447  test_save_ublk_config
01:10:24.447  ublk
01:10:24.447  ublk_recovery
01:10:24.447  unittest_nvmf_fc
01:10:24.447  unittest_nvmf_fc_ls
01:10:24.447  vhost_blk_hot_remove
01:10:24.447  vhost_hotplug
01:10:24.447  vhost_scsi_hot_remove
01:10:24.447  xnvme_bdevperf
01:10:24.447  xnvme_to_malloc_dd_copy
01:10:24.447  zoned_fio
01:10:24.447  Not all tests were run. Failing the build.
01:10:24.447  ['blockdev_xnvme', 'dpdk_governor', 'ftl', 'ftl_bdevperf', 'ftl_dirty_shutdown', 'ftl_fio_basic', 'ftl_restore', 'ftl_trim', 'ftl_upgrade_shutdown', 'idle', 'interrupt_mode', 'ioat', 'ioat_perf', 'ioat_verify', 'nvme_xnvme', 'nvmf_rdma', 'nvmf_target_disconnect_tc3', 'ocf', 'ocf_bdevperf_iotypes', 'ocf_configuration_change', 'ocf_create_destruct', 'ocf_fio_modes', 'ocf_flush', 'ocf_multicore', 'ocf_remove', 'ocf_stats', 'scheduler', 'spdkcli_nvmf_rdma', 'test_create_multi_ublk', 'test_create_ublk', 'test_save_ublk_config', 'ublk', 'ublk_recovery', 'xnvme_bdevperf', 'xnvme_to_malloc_dd_copy']
01:10:24.447  + compress_coverage_and_docs
01:10:24.447  + echo 'Start compress coverage and docs'
01:10:24.447  Start compress coverage and docs
01:10:24.447  + tar -C coverage -czf coverage_autotest-spdk-master-vs-dpdk-v23.11_425.tar.gz ./ --remove-files
01:10:25.379  + tar -C ut_coverage -czf ut_coverage_autotest-spdk-master-vs-dpdk-v23.11_425.tar.gz ./ --remove-files
01:10:26.310  + tar -C doc -czf doc_autotest-spdk-master-vs-dpdk-v23.11_425.tar.gz ./ --remove-files
01:10:26.569  + [[ -d llvm ]]
01:10:26.569  + true
01:10:26.569  + echo 'End compress coverage and docs'
01:10:26.569  End compress coverage and docs
01:10:26.569  + remove_partial_coverage_files
01:10:26.569  + find . -name 'cov_*.info' -delete
01:10:26.569  + echo 'End stage post_gen_coverage.sh'
01:10:26.569  End stage post_gen_coverage.sh
01:10:26.587  [Pipeline] }
01:10:26.590  ERROR: script returned exit code 1
01:10:26.620  [Pipeline] // catchError
01:10:26.630  [Pipeline] catchError
01:10:26.632  [Pipeline] {
01:10:26.654  [Pipeline] dir
01:10:26.654  Running in /var/jenkins/workspace/autotest-spdk-master-vs-dpdk-v23.11/post_process
01:10:26.656  [Pipeline] {