00:00:00.001  Started by user sys_sgci
00:00:00.008  Checking out git https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool into /var/jenkins_home/workspace/autotest-per-patch_script/33b20b30f0a51e6b52980845e0f6aa336787973ad45e341fbbf98d1b65b265d4 to read jbp/jenkins/jjb-config/jobs/autotest-upstream/autotest.groovy
00:00:00.009  The recommended git tool is: git
00:00:00.009  using credential 00000000-0000-0000-0000-000000000002
00:00:00.011   > git rev-parse --resolve-git-dir /var/jenkins_home/workspace/autotest-per-patch_script/33b20b30f0a51e6b52980845e0f6aa336787973ad45e341fbbf98d1b65b265d4/jbp/.git # timeout=10
00:00:00.021  Fetching changes from the remote Git repository
00:00:00.022   > git config remote.origin.url https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool # timeout=10
00:00:00.032  Using shallow fetch with depth 1
00:00:00.032  Fetching upstream changes from https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool
00:00:00.032   > git --version # timeout=10
00:00:00.041   > git --version # 'git version 2.39.2'
00:00:00.041  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:00.049  Setting http proxy: proxy-dmz.intel.com:911
00:00:00.049   > git fetch --tags --force --progress --depth=1 -- https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool refs/heads/master # timeout=5
00:00:02.222   > git rev-parse origin/FETCH_HEAD^{commit} # timeout=10
00:00:02.232   > git rev-parse FETCH_HEAD^{commit} # timeout=10
00:00:02.243  Checking out Revision 53a1a621557260e3fbfd1fd32ee65ff11a804d5b (FETCH_HEAD)
00:00:02.244   > git config core.sparsecheckout # timeout=10
00:00:02.253   > git read-tree -mu HEAD # timeout=10
00:00:02.267   > git checkout -f 53a1a621557260e3fbfd1fd32ee65ff11a804d5b # timeout=5
00:00:02.284  Commit message: "packer: Merge irdmafedora into main fedora image"
00:00:02.284   > git rev-list --no-walk 53a1a621557260e3fbfd1fd32ee65ff11a804d5b # timeout=10
00:00:02.422  [Pipeline] Start of Pipeline
00:00:02.436  [Pipeline] library
00:00:02.438  Loading library shm_lib@master
00:00:02.438  Library shm_lib@master is cached. Copying from home.
00:00:02.464  [Pipeline] node
00:00:02.470  Running on ME1 in /var/jenkins/workspace/autotest-per-patch
00:00:02.475  [Pipeline] {
00:00:02.490  [Pipeline] cleanWs
00:00:02.498  [WS-CLEANUP] Deleting project workspace...
00:00:02.498  [WS-CLEANUP] Deferred wipeout is used...
00:00:02.505  [WS-CLEANUP] done
00:00:02.510  [Pipeline] stage
00:00:02.514  [Pipeline] { (Prologue)
00:00:02.623  [Pipeline] withCredentials
00:00:02.634   > git --version # timeout=10
00:00:02.646   > git --version # 'git version 2.39.2'
00:00:02.666  Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD or $GIT_ASKPASS
00:00:02.668  [Pipeline] {
00:00:02.675  [Pipeline] retry
00:00:02.676  [Pipeline] {
00:00:02.834  [Pipeline] sh
00:00:03.113  + git ls-remote https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool refs/heads/master
00:00:05.037  [Pipeline] }
00:00:05.057  [Pipeline] // retry
00:00:05.062  [Pipeline] }
00:00:05.079  [Pipeline] // withCredentials
00:00:05.089  [Pipeline] httpRequest
00:00:05.494  [Pipeline] echo
00:00:05.495  Sorcerer 10.211.164.101 is alive
00:00:05.504  [Pipeline] retry
00:00:05.506  [Pipeline] {
00:00:05.520  [Pipeline] httpRequest
00:00:05.525  HttpMethod: GET
00:00:05.525  URL: http://10.211.164.101/packages/jbp_53a1a621557260e3fbfd1fd32ee65ff11a804d5b.tar.gz
00:00:05.526  Sending request to url: http://10.211.164.101/packages/jbp_53a1a621557260e3fbfd1fd32ee65ff11a804d5b.tar.gz
00:00:05.528  Response Code: HTTP/1.1 200 OK
00:00:05.528  Success: Status code 200 is in the accepted range: 200,404
00:00:05.529  Saving response body to /var/jenkins/workspace/autotest-per-patch/jbp_53a1a621557260e3fbfd1fd32ee65ff11a804d5b.tar.gz
00:00:05.674  [Pipeline] }
00:00:05.693  [Pipeline] // retry
00:00:05.700  [Pipeline] sh
00:00:05.985  + tar --no-same-owner -xf jbp_53a1a621557260e3fbfd1fd32ee65ff11a804d5b.tar.gz
00:00:06.004  [Pipeline] httpRequest
00:00:06.403  [Pipeline] echo
00:00:06.405  Sorcerer 10.211.164.101 is alive
00:00:06.413  [Pipeline] retry
00:00:06.415  [Pipeline] {
00:00:06.429  [Pipeline] httpRequest
00:00:06.434  HttpMethod: GET
00:00:06.435  URL: http://10.211.164.101/packages/spdk_7b38c9ede93025f415b7652489344a4dd937aed1.tar.gz
00:00:06.435  Sending request to url: http://10.211.164.101/packages/spdk_7b38c9ede93025f415b7652489344a4dd937aed1.tar.gz
00:00:06.437  Response Code: HTTP/1.1 404 Not Found
00:00:06.438  Success: Status code 404 is in the accepted range: 200,404
00:00:06.438  Saving response body to /var/jenkins/workspace/autotest-per-patch/spdk_7b38c9ede93025f415b7652489344a4dd937aed1.tar.gz
00:00:06.443  [Pipeline] }
00:00:06.469  [Pipeline] // retry
00:00:06.475  [Pipeline] sh
00:00:06.760  + rm -f spdk_7b38c9ede93025f415b7652489344a4dd937aed1.tar.gz
00:00:06.776  [Pipeline] retry
00:00:06.778  [Pipeline] {
00:00:06.799  [Pipeline] checkout
00:00:06.806  The recommended git tool is: NONE
00:00:06.819  using credential 00000000-0000-0000-0000-000000000002
00:00:06.821  Wiping out workspace first.
00:00:06.830  Cloning the remote Git repository
00:00:06.834  Honoring refspec on initial clone
00:00:06.839  Cloning repository https://review.spdk.io/gerrit/a/spdk/spdk
00:00:06.839   > git init /var/jenkins/workspace/autotest-per-patch/spdk # timeout=10
00:00:06.844  Using reference repository: /var/ci_repos/spdk_multi
00:00:06.844  Fetching upstream changes from https://review.spdk.io/gerrit/a/spdk/spdk
00:00:06.844   > git --version # timeout=10
00:00:06.845   > git --version # 'git version 2.46.0'
00:00:06.845  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:06.847  Setting http proxy: proxy-dmz.intel.com:911
00:00:06.847   > git fetch --tags --force --progress -- https://review.spdk.io/gerrit/a/spdk/spdk refs/changes/29/25029/4 +refs/heads/master:refs/remotes/origin/master # timeout=10
00:00:12.494  Avoid second fetch
00:00:12.503  Checking out Revision 7b38c9ede93025f415b7652489344a4dd937aed1 (FETCH_HEAD)
00:00:12.657  Commit message: "bdev/nvme: changed default config to multipath"
00:00:12.662  First time build. Skipping changelog.
00:00:12.485   > git config remote.origin.url https://review.spdk.io/gerrit/a/spdk/spdk # timeout=10
00:00:12.487   > git config --add remote.origin.fetch refs/changes/29/25029/4 # timeout=10
00:00:12.489   > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
00:00:12.496   > git rev-parse origin/FETCH_HEAD^{commit} # timeout=10
00:00:12.501   > git rev-parse FETCH_HEAD^{commit} # timeout=10
00:00:12.505   > git config core.sparsecheckout # timeout=10
00:00:12.507   > git checkout -f 7b38c9ede93025f415b7652489344a4dd937aed1 # timeout=10
00:00:12.660   > git rev-list --no-walk af73f170275f7ce480dcf36789b2e97532b0381d # timeout=10
00:00:12.667   > git remote # timeout=10
00:00:12.669   > git submodule init # timeout=10
00:00:12.693   > git submodule sync # timeout=10
00:00:12.717   > git config --get remote.origin.url # timeout=10
00:00:12.721   > git submodule init # timeout=10
00:00:12.743   > git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10
00:00:12.744   > git config --get submodule.dpdk.url # timeout=10
00:00:12.746   > git remote # timeout=10
00:00:12.747   > git config --get remote.origin.url # timeout=10
00:00:12.749   > git config -f .gitmodules --get submodule.dpdk.path # timeout=10
00:00:12.750   > git config --get submodule.intel-ipsec-mb.url # timeout=10
00:00:12.751   > git remote # timeout=10
00:00:12.753   > git config --get remote.origin.url # timeout=10
00:00:12.754   > git config -f .gitmodules --get submodule.intel-ipsec-mb.path # timeout=10
00:00:12.756   > git config --get submodule.isa-l.url # timeout=10
00:00:12.757   > git remote # timeout=10
00:00:12.758   > git config --get remote.origin.url # timeout=10
00:00:12.760   > git config -f .gitmodules --get submodule.isa-l.path # timeout=10
00:00:12.761   > git config --get submodule.ocf.url # timeout=10
00:00:12.762   > git remote # timeout=10
00:00:12.764   > git config --get remote.origin.url # timeout=10
00:00:12.765   > git config -f .gitmodules --get submodule.ocf.path # timeout=10
00:00:12.767   > git config --get submodule.libvfio-user.url # timeout=10
00:00:12.768   > git remote # timeout=10
00:00:12.769   > git config --get remote.origin.url # timeout=10
00:00:12.771   > git config -f .gitmodules --get submodule.libvfio-user.path # timeout=10
00:00:12.772   > git config --get submodule.xnvme.url # timeout=10
00:00:12.774   > git remote # timeout=10
00:00:12.775   > git config --get remote.origin.url # timeout=10
00:00:12.777   > git config -f .gitmodules --get submodule.xnvme.path # timeout=10
00:00:12.778   > git config --get submodule.isa-l-crypto.url # timeout=10
00:00:12.779   > git remote # timeout=10
00:00:12.781   > git config --get remote.origin.url # timeout=10
00:00:12.782   > git config -f .gitmodules --get submodule.isa-l-crypto.path # timeout=10
00:00:12.784  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:12.784  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:12.784  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:12.784  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:12.786  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:12.786  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:12.786  Setting http proxy: proxy-dmz.intel.com:911
00:00:12.786   > git submodule update --init --recursive --reference /var/ci_repos/spdk_multi dpdk # timeout=10
00:00:12.786  using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials
00:00:12.786  Setting http proxy: proxy-dmz.intel.com:911
00:00:12.786   > git submodule update --init --recursive --reference /var/ci_repos/spdk_multi ocf # timeout=10
00:00:12.787  Setting http proxy: proxy-dmz.intel.com:911
00:00:12.787   > git submodule update --init --recursive --reference /var/ci_repos/spdk_multi intel-ipsec-mb # timeout=10
00:00:12.787  Setting http proxy: proxy-dmz.intel.com:911
00:00:12.787   > git submodule update --init --recursive --reference /var/ci_repos/spdk_multi isa-l # timeout=10
00:00:12.788  Setting http proxy: proxy-dmz.intel.com:911
00:00:12.788   > git submodule update --init --recursive --reference /var/ci_repos/spdk_multi xnvme # timeout=10
00:00:12.789  Setting http proxy: proxy-dmz.intel.com:911
00:00:12.789   > git submodule update --init --recursive --reference /var/ci_repos/spdk_multi libvfio-user # timeout=10
00:00:12.789  Setting http proxy: proxy-dmz.intel.com:911
00:00:12.789   > git submodule update --init --recursive --reference /var/ci_repos/spdk_multi isa-l-crypto # timeout=10
00:00:39.053  [Pipeline] dir
00:00:39.054  Running in /var/jenkins/workspace/autotest-per-patch/spdk
00:00:39.056  [Pipeline] {
00:00:39.073  [Pipeline] sh
00:00:39.355  ++ nproc
00:00:39.355  + threads=4
00:00:39.355  + git repack -a -d --threads=4
00:00:43.538  + git submodule foreach git repack -a -d --threads=4
00:00:43.538  Entering 'dpdk'
00:00:46.820  Entering 'intel-ipsec-mb'
00:00:46.820  Entering 'isa-l'
00:00:46.820  Entering 'isa-l-crypto'
00:00:46.820  Entering 'libvfio-user'
00:00:47.079  Entering 'ocf'
00:00:47.337  Entering 'xnvme'
00:00:47.337  + find .git -type f -name alternates -print -delete
00:00:47.337  .git/objects/info/alternates
00:00:47.337  .git/modules/xnvme/objects/info/alternates
00:00:47.337  .git/modules/dpdk/objects/info/alternates
00:00:47.337  .git/modules/ocf/objects/info/alternates
00:00:47.337  .git/modules/isa-l-crypto/objects/info/alternates
00:00:47.337  .git/modules/libvfio-user/objects/info/alternates
00:00:47.337  .git/modules/isa-l/objects/info/alternates
00:00:47.337  .git/modules/intel-ipsec-mb/objects/info/alternates
00:00:47.345  [Pipeline] }
00:00:47.360  [Pipeline] // dir
00:00:47.367  [Pipeline] }
00:00:47.383  [Pipeline] // retry
00:00:47.390  [Pipeline] sh
00:00:47.674  + hash pigz
00:00:47.674  + tar -cf spdk_7b38c9ede93025f415b7652489344a4dd937aed1.tar.gz -I pigz spdk
00:00:49.589  [Pipeline] retry
00:00:49.590  [Pipeline] {
00:00:49.605  [Pipeline] httpRequest
00:00:49.612  HttpMethod: PUT
00:00:49.613  URL: http://10.211.164.101/cgi-bin/sorcerer.py?group=packages&filename=spdk_7b38c9ede93025f415b7652489344a4dd937aed1.tar.gz
00:00:49.613  Sending request to url: http://10.211.164.101/cgi-bin/sorcerer.py?group=packages&filename=spdk_7b38c9ede93025f415b7652489344a4dd937aed1.tar.gz
00:00:52.134  Response Code: HTTP/1.1 200 OK
00:00:52.140  Success: Status code 200 is in the accepted range: 200
00:00:52.143  [Pipeline] }
00:00:52.159  [Pipeline] // retry
00:00:52.164  [Pipeline] echo
00:00:52.165  
00:00:52.165  Locking
00:00:52.165  Waited 0s for lock
00:00:52.165  Everything Fine. Saved: /storage/packages/spdk_7b38c9ede93025f415b7652489344a4dd937aed1.tar.gz
00:00:52.165  
00:00:52.169  [Pipeline] sh
00:00:52.449  + git -C spdk log --oneline -n5
00:00:52.449  7b38c9ede bdev/nvme: changed default config to multipath
00:00:52.449  fefe29c8c bdev/nvme: ctrl config consistency check
00:00:52.449  09cc66129 test/unit: add mixed busy/idle mock poller function in reactor_ut
00:00:52.449  a67b3561a dpdk: update submodule to include alarm_cancel fix
00:00:52.449  43f6d3385 nvmf: remove use of STAILQ for last_wqe events
00:00:52.464  [Pipeline] setCustomBuildProperty
00:00:52.473  [Pipeline] setCustomBuildProperty
00:00:52.482  [Pipeline] catchError
00:00:52.483  [Pipeline] {
00:00:52.499  [Pipeline] sh
00:00:52.781  + git -C spdk describe --tags --abbrev=0 origin/master
00:00:52.795  [Pipeline] sh
00:00:53.075  + git -C spdk describe --tags --abbrev=0 --exclude=LTS HEAD
00:00:53.089  [Pipeline] echo
00:00:53.091  Branch: master
00:00:53.093  [Pipeline] fileExists
00:00:53.107  [Pipeline] readJSON
00:00:53.121  [Pipeline] }
00:00:53.137  [Pipeline] // catchError
00:00:53.147  [Pipeline] sh
00:00:53.432  + /var/jenkins/workspace/autotest-per-patch/jbp/jenkins/jjb-config/jobs/scripts/get-pkgdep-jobs.sh /var/jenkins/workspace/autotest-per-patch/spdk
00:00:53.448  [Pipeline] }
00:00:53.469  [Pipeline] // stage
00:00:53.486  [Pipeline] catchError
00:00:53.488  [Pipeline] {
00:00:53.504  [Pipeline] stage
00:00:53.506  [Pipeline] { (Pre tests)
00:00:53.535  [Pipeline] parallel
00:00:53.547  [Pipeline] { (Branch: check-format-docker-autotest)
00:00:53.548  [Pipeline] { (Branch: check-so-deps-docker-autotest)
00:00:53.550  [Pipeline] { (Branch: doc-docker-autotest)
00:00:53.551  [Pipeline] { (Branch: build-files-docker-autotest)
00:00:53.568  [Pipeline] retry
00:00:53.569  [Pipeline] {
00:00:53.572  [Pipeline] retry
00:00:53.572  [Pipeline] {
00:00:53.575  [Pipeline] retry
00:00:53.576  [Pipeline] {
00:00:53.579  [Pipeline] retry
00:00:53.580  [Pipeline] {
00:00:53.594  [Pipeline] build
00:00:53.661  Scheduling project: check-format-docker-autotest
00:00:53.672  [Pipeline] build
00:00:53.676  Scheduling project: check-so-deps-docker-autotest
00:00:53.687  [Pipeline] build
00:00:53.692  Scheduling project: doc-docker-autotest
00:00:53.702  [Pipeline] build
00:00:53.706  Scheduling project: build-files-docker-autotest
00:00:59.626  Starting building: build-files-docker-autotest #32615
00:00:59.628  Starting building: check-format-docker-autotest #32641
00:00:59.631  Starting building: check-so-deps-docker-autotest #32862
00:00:59.633  Starting building: doc-docker-autotest #32944
00:01:33.153  Build doc-docker-autotest #32944 completed: SUCCESS
00:01:33.155  [Pipeline] }
00:01:33.182  [Pipeline] // retry
00:01:33.187  [Pipeline] }
00:01:46.106  Build check-format-docker-autotest #32641 completed: SUCCESS
00:01:46.108  [Pipeline] }
00:01:46.142  [Pipeline] // retry
00:01:46.146  [Pipeline] }
00:03:21.483  Build build-files-docker-autotest #32615 completed: SUCCESS
00:03:21.486  [Pipeline] }
00:03:21.514  [Pipeline] // retry
00:03:21.519  [Pipeline] }
00:03:57.567  Build check-so-deps-docker-autotest #32862 completed: SUCCESS
00:03:57.570  [Pipeline] }
00:03:57.597  [Pipeline] // retry
00:03:57.602  [Pipeline] }
00:03:57.646  [Pipeline] // parallel
00:03:57.651  [Pipeline] }
00:03:57.673  [Pipeline] // stage
00:03:57.678  [Pipeline] }
00:03:57.700  [Pipeline] // catchError
00:03:57.709  [Pipeline] catchError
00:03:57.710  [Pipeline] {
00:03:57.728  [Pipeline] stage
00:03:57.729  [Pipeline] { (Tests)
00:03:57.868  [Pipeline] parallel
00:03:57.959  [Pipeline] { (Branch: unittest-clang-docker-autotest)
00:03:57.961  [Pipeline] { (Branch: unittest-main-docker-autotest)
00:03:57.962  [Pipeline] { (Branch: autobuild-external-vg-autotest)
00:03:57.964  [Pipeline] { (Branch: scan-build-docker-autotest)
00:03:57.965  [Pipeline] { (Branch: unittest-valgrind-docker-autotest)
00:03:57.967  [Pipeline] { (Branch: lvol-vg-autotest)
00:03:57.969  [Pipeline] { (Branch: vhost-initiator-vg-autotest)
00:03:57.970  [Pipeline] { (Branch: nvmf-tcp-vg-autotest)
00:03:57.972  [Pipeline] { (Branch: nvmf-tcp-uring-vg-autotest)
00:03:57.974  [Pipeline] { (Branch: nvme-vg-autotest)
00:03:57.976  [Pipeline] { (Branch: rocky9-vg-autotest)
00:03:57.977  [Pipeline] { (Branch: ubuntu22-vg-autotest)
00:03:57.979  [Pipeline] { (Branch: ubuntu24-vg-autotest)
00:03:57.981  [Pipeline] { (Branch: freebsd-vg-autotest)
00:03:57.982  [Pipeline] { (Branch: iscsi-vg-autotest)
00:03:57.984  [Pipeline] { (Branch: iscsi-uring-vg-autotest)
00:03:57.985  [Pipeline] { (Branch: zns-vg-autotest)
00:03:57.987  [Pipeline] { (Branch: nvme-cmb-pmr-vg-autotest)
00:03:57.988  [Pipeline] { (Branch: raid-vg-autotest)
00:03:57.990  [Pipeline] { (Branch: BlobFS-phy-autotest)
00:03:57.991  [Pipeline] { (Branch: crypto-phy-autotest)
00:03:57.993  [Pipeline] { (Branch: vhost-phy-autotest)
00:03:57.994  [Pipeline] { (Branch: nvmf-phy-autotest)
00:03:57.996  [Pipeline] { (Branch: nvmf-tcp-phy-autotest)
00:03:57.998  [Pipeline] { (Branch: nvme-phy-autotest)
00:03:57.999  [Pipeline] { (Branch: vfio-user-phy-autotest)
00:03:58.001  [Pipeline] { (Branch: short-fuzz-phy-autotest)
00:03:58.002  [Pipeline] { (Branch: packaging-docker-autotest)
00:03:58.004  [Pipeline] { (Branch: release-build-main-docker-autotest)
00:03:58.005  [Pipeline] { (Branch: release-build-clang-docker-autotest)
00:03:58.039  [Pipeline] retry
00:03:58.042  [Pipeline] {
00:03:58.048  [Pipeline] retry
00:03:58.051  [Pipeline] {
00:03:58.057  [Pipeline] retry
00:03:58.059  [Pipeline] {
00:03:58.065  [Pipeline] retry
00:03:58.068  [Pipeline] {
00:03:58.074  [Pipeline] retry
00:03:58.076  [Pipeline] {
00:03:58.083  [Pipeline] retry
00:03:58.085  [Pipeline] {
00:03:58.091  [Pipeline] retry
00:03:58.094  [Pipeline] {
00:03:58.100  [Pipeline] retry
00:03:58.102  [Pipeline] {
00:03:58.109  [Pipeline] retry
00:03:58.111  [Pipeline] {
00:03:58.118  [Pipeline] retry
00:03:58.120  [Pipeline] {
00:03:58.127  [Pipeline] retry
00:03:58.129  [Pipeline] {
00:03:58.135  [Pipeline] retry
00:03:58.138  [Pipeline] {
00:03:58.144  [Pipeline] retry
00:03:58.147  [Pipeline] {
00:03:58.153  [Pipeline] retry
00:03:58.156  [Pipeline] {
00:03:58.162  [Pipeline] retry
00:03:58.164  [Pipeline] {
00:03:58.170  [Pipeline] retry
00:03:58.173  [Pipeline] {
00:03:58.179  [Pipeline] retry
00:03:58.182  [Pipeline] {
00:03:58.187  [Pipeline] retry
00:03:58.190  [Pipeline] {
00:03:58.196  [Pipeline] retry
00:03:58.198  [Pipeline] {
00:03:58.204  [Pipeline] retry
00:03:58.206  [Pipeline] {
00:03:58.212  [Pipeline] retry
00:03:58.214  [Pipeline] {
00:03:58.219  [Pipeline] retry
00:03:58.221  [Pipeline] {
00:03:58.226  [Pipeline] retry
00:03:58.228  [Pipeline] {
00:03:58.233  [Pipeline] retry
00:03:58.235  [Pipeline] {
00:03:58.241  [Pipeline] retry
00:03:58.242  [Pipeline] {
00:03:58.248  [Pipeline] retry
00:03:58.249  [Pipeline] {
00:03:58.255  [Pipeline] retry
00:03:58.257  [Pipeline] {
00:03:58.262  [Pipeline] retry
00:03:58.264  [Pipeline] {
00:03:58.269  [Pipeline] retry
00:03:58.271  [Pipeline] {
00:03:58.276  [Pipeline] retry
00:03:58.278  [Pipeline] {
00:03:58.303  [Pipeline] build
00:03:58.305  Scheduling project: unittest-clang-docker-autotest
00:03:58.312  [Pipeline] build
00:03:58.315  Scheduling project: unittest-main-docker-autotest
00:03:58.321  [Pipeline] build
00:03:58.324  Scheduling project: autobuild-external-vg-autotest
00:03:58.331  [Pipeline] build
00:03:58.333  Scheduling project: scan-build-docker-autotest
00:03:58.340  [Pipeline] build
00:03:58.342  Scheduling project: unittest-valgrind-docker-autotest
00:03:58.349  [Pipeline] build
00:03:58.351  Scheduling project: lvol-vg-autotest
00:03:58.358  [Pipeline] build
00:03:58.360  Scheduling project: vhost-initiator-vg-autotest
00:03:58.367  [Pipeline] build
00:03:58.369  Scheduling project: nvmf-tcp-vg-autotest
00:03:58.376  [Pipeline] build
00:03:58.379  Scheduling project: nvmf-tcp-uring-vg-autotest
00:03:58.386  [Pipeline] build
00:03:58.388  Scheduling project: nvme-vg-autotest
00:03:58.395  [Pipeline] build
00:03:58.398  Scheduling project: rocky9-vg-autotest
00:03:58.404  [Pipeline] build
00:03:58.407  Scheduling project: ubuntu22-vg-autotest
00:03:58.413  [Pipeline] build
00:03:58.416  Scheduling project: ubuntu24-vg-autotest
00:03:58.422  [Pipeline] build
00:03:58.425  Scheduling project: freebsd-vg-autotest
00:03:58.431  [Pipeline] build
00:03:58.433  Scheduling project: iscsi-vg-autotest
00:03:58.440  [Pipeline] build
00:03:58.442  Scheduling project: iscsi-uring-vg-autotest
00:03:58.448  [Pipeline] build
00:03:58.450  Scheduling project: zns-vg-autotest
00:03:58.457  [Pipeline] build
00:03:58.460  Scheduling project: nvme-cmb-pmr-vg-autotest
00:03:58.467  [Pipeline] build
00:03:58.469  Scheduling project: raid-vg-autotest
00:03:58.476  [Pipeline] build
00:03:58.478  Scheduling project: BlobFS-phy-autotest
00:03:58.484  [Pipeline] build
00:03:58.487  Scheduling project: crypto-phy-autotest
00:03:58.493  [Pipeline] build
00:03:58.495  Scheduling project: vhost-phy-autotest
00:03:58.501  [Pipeline] build
00:03:58.504  Scheduling project: nvmf-phy-autotest
00:03:58.510  [Pipeline] build
00:03:58.512  Scheduling project: nvmf-tcp-phy-autotest
00:03:58.518  [Pipeline] build
00:03:58.520  Scheduling project: nvme-phy-autotest
00:03:58.526  [Pipeline] build
00:03:58.528  Scheduling project: vfio-user-phy-autotest
00:03:58.534  [Pipeline] build
00:03:58.537  Scheduling project: short-fuzz-phy-autotest
00:03:58.542  [Pipeline] build
00:03:58.545  Scheduling project: packaging-docker-autotest
00:03:58.551  [Pipeline] build
00:03:58.553  Scheduling project: release-build-main-docker-autotest
00:03:58.559  [Pipeline] build
00:03:58.561  Scheduling project: release-build-clang-docker-autotest
00:04:04.776  Starting building: release-build-clang-docker-autotest #2655
00:04:04.780  Starting building: release-build-main-docker-autotest #2655
00:04:04.782  Starting building: short-fuzz-phy-autotest #37805
00:04:04.785  Starting building: packaging-docker-autotest #29449
00:04:04.788  Starting building: rocky9-vg-autotest #27396
00:04:04.790  Starting building: ubuntu24-vg-autotest #6644
00:04:04.793  Starting building: unittest-clang-docker-autotest #29449
00:04:04.795  Starting building: autobuild-external-vg-autotest #30093
00:04:04.798  Starting building: vfio-user-phy-autotest #37810
00:04:04.801  Starting building: iscsi-uring-vg-autotest #90037
00:04:04.804  Starting building: lvol-vg-autotest #90085
00:04:04.807  Starting building: nvme-cmb-pmr-vg-autotest #91264
00:04:04.809  Starting building: nvme-vg-autotest #90136
00:04:04.813  Starting building: raid-vg-autotest #4805
00:04:04.816  Starting building: BlobFS-phy-autotest #37820
00:04:04.820  Starting building: crypto-phy-autotest #37809
00:04:04.823  Starting building: vhost-phy-autotest #37874
00:04:04.826  Starting building: unittest-main-docker-autotest #29451
00:04:04.829  Starting building: scan-build-docker-autotest #29456
00:04:04.834  Starting building: unittest-valgrind-docker-autotest #29454
00:04:04.838  Starting building: nvmf-tcp-uring-vg-autotest #90079
00:04:04.846  Starting building: nvmf-tcp-phy-autotest #83934
00:04:04.850  Starting building: ubuntu22-vg-autotest #49083
00:04:04.853  Starting building: zns-vg-autotest #89238
00:04:04.858  Starting building: nvmf-phy-autotest #84052
00:04:04.865  Starting building: iscsi-vg-autotest #90173
00:04:04.868  Starting building: freebsd-vg-autotest #68225
00:04:04.869  Starting building: nvme-phy-autotest #83983
00:04:04.872  Starting building: nvmf-tcp-vg-autotest #90118
00:04:04.873  Starting building: vhost-initiator-vg-autotest #90154
00:08:19.957  Build release-build-main-docker-autotest #2655 completed: SUCCESS
00:08:19.960  [Pipeline] }
00:08:19.998  [Pipeline] // retry
00:08:20.004  [Pipeline] }
00:08:35.827  Build unittest-clang-docker-autotest #29449 completed: SUCCESS
00:08:35.830  [Pipeline] }
00:08:35.870  [Pipeline] // retry
00:08:35.875  [Pipeline] }
00:08:44.462  Build release-build-clang-docker-autotest #2655 completed: SUCCESS
00:08:44.465  [Pipeline] }
00:08:44.503  [Pipeline] // retry
00:08:44.509  [Pipeline] }
00:09:32.524  Build scan-build-docker-autotest #29456 completed: SUCCESS
00:09:32.527  [Pipeline] }
00:09:32.564  [Pipeline] // retry
00:09:32.570  [Pipeline] }
00:10:13.176  Build packaging-docker-autotest #29449 completed: SUCCESS
00:10:13.179  [Pipeline] }
00:10:13.216  [Pipeline] // retry
00:10:13.222  [Pipeline] }
00:10:31.313  Build unittest-main-docker-autotest #29451 completed: SUCCESS
00:10:31.316  [Pipeline] }
00:10:31.354  [Pipeline] // retry
00:10:31.360  [Pipeline] }
00:12:12.265  Build vhost-initiator-vg-autotest #90154 completed: SUCCESS
00:12:12.267  [Pipeline] }
00:12:12.304  [Pipeline] // retry
00:12:12.310  [Pipeline] }
00:12:35.431  Build autobuild-external-vg-autotest #30093 completed: SUCCESS
00:12:35.434  [Pipeline] }
00:12:35.470  [Pipeline] // retry
00:12:35.479  [Pipeline] }
00:13:14.556  Build short-fuzz-phy-autotest #37805 completed: SUCCESS
00:13:14.560  [Pipeline] }
00:13:14.610  [Pipeline] // retry
00:13:14.616  [Pipeline] }
00:13:34.224  Build zns-vg-autotest #89238 completed: SUCCESS
00:13:34.225  [Pipeline] }
00:13:34.261  [Pipeline] // retry
00:13:34.266  [Pipeline] }
00:14:53.796  Build unittest-valgrind-docker-autotest #29454 completed: SUCCESS
00:14:53.800  [Pipeline] }
00:14:53.841  [Pipeline] // retry
00:14:53.847  [Pipeline] }
00:15:50.605  Build nvme-cmb-pmr-vg-autotest #91264 completed: SUCCESS
00:15:50.607  [Pipeline] }
00:15:50.644  [Pipeline] // retry
00:15:50.651  [Pipeline] }
00:15:52.475  Build freebsd-vg-autotest #68225 completed: SUCCESS
00:15:52.478  [Pipeline] }
00:15:52.511  [Pipeline] // retry
00:15:52.517  [Pipeline] }
00:17:00.787  Build BlobFS-phy-autotest #37820 completed: SUCCESS
00:17:00.790  [Pipeline] }
00:17:00.820  [Pipeline] // retry
00:17:00.824  [Pipeline] }
00:18:04.630  Build lvol-vg-autotest #90085 completed: SUCCESS
00:18:04.632  [Pipeline] }
00:18:04.675  [Pipeline] // retry
00:18:04.682  [Pipeline] }
00:18:55.107  Build vhost-phy-autotest #37874 completed: SUCCESS
00:18:55.110  [Pipeline] }
00:18:55.147  [Pipeline] // retry
00:18:55.155  [Pipeline] }
00:18:59.229  Build iscsi-uring-vg-autotest #90037 completed: SUCCESS
00:18:59.231  [Pipeline] }
00:18:59.265  [Pipeline] // retry
00:18:59.269  [Pipeline] }
00:20:18.178  Build crypto-phy-autotest #37809 completed: SUCCESS
00:20:18.181  [Pipeline] }
00:20:18.221  [Pipeline] // retry
00:20:18.227  [Pipeline] }
00:21:01.263  Build nvmf-tcp-uring-vg-autotest #90079 completed: FAILURE
00:21:02.728  [Pipeline] echo
00:21:02.729  No retry patterns found.
00:21:02.730  [Pipeline] }
00:21:02.763  [Pipeline] // retry
00:21:02.772  [Pipeline] error
00:21:02.780  [Pipeline] }
00:21:02.787  Failed in branch nvmf-tcp-uring-vg-autotest
00:21:12.931  Build vfio-user-phy-autotest #37810 completed: SUCCESS
00:21:12.934  [Pipeline] }
00:21:12.974  [Pipeline] // retry
00:21:12.981  [Pipeline] }
00:21:45.390  Build iscsi-vg-autotest #90173 completed: SUCCESS
00:21:45.392  [Pipeline] }
00:21:45.426  [Pipeline] // retry
00:21:45.430  [Pipeline] }
00:23:20.740  Build nvmf-tcp-vg-autotest #90118 completed: FAILURE
00:23:22.409  [Pipeline] echo
00:23:22.410  No retry patterns found.
00:23:22.411  [Pipeline] }
00:23:22.430  [Pipeline] // retry
00:23:22.438  [Pipeline] error
00:23:22.441  [Pipeline] }
00:23:22.444  Failed in branch nvmf-tcp-vg-autotest
00:24:38.223  Build rocky9-vg-autotest #27396 completed: SUCCESS
00:24:38.226  [Pipeline] }
00:24:38.271  [Pipeline] // retry
00:24:38.280  [Pipeline] }
00:25:58.080  Build ubuntu22-vg-autotest #49083 completed: SUCCESS
00:25:58.082  [Pipeline] }
00:25:58.118  [Pipeline] // retry
00:25:58.124  [Pipeline] }
00:26:02.746  Build nvmf-phy-autotest #84052 completed: FAILURE
00:26:03.783  [Pipeline] echo
00:26:03.784  No retry patterns found.
00:26:03.785  [Pipeline] }
00:26:03.809  [Pipeline] // retry
00:26:03.817  [Pipeline] error
00:26:03.822  [Pipeline] }
00:26:03.827  Failed in branch nvmf-phy-autotest
00:26:07.582  Build raid-vg-autotest #4805 completed: SUCCESS
00:26:07.584  [Pipeline] }
00:26:07.627  [Pipeline] // retry
00:26:07.633  [Pipeline] }
00:27:26.755  Build nvme-phy-autotest #83983 completed: SUCCESS
00:27:26.757  [Pipeline] }
00:27:26.788  [Pipeline] // retry
00:27:26.794  [Pipeline] }
00:28:55.046  Build ubuntu24-vg-autotest #6644 completed: SUCCESS
00:28:55.050  [Pipeline] }
00:28:55.090  [Pipeline] // retry
00:28:55.096  [Pipeline] }
00:29:02.821  Build nvme-vg-autotest #90136 completed: SUCCESS
00:29:02.823  [Pipeline] }
00:29:02.861  [Pipeline] // retry
00:29:02.868  [Pipeline] }
00:29:27.560  Build nvmf-tcp-phy-autotest #83934 completed: FAILURE
00:29:29.288  [Pipeline] echo
00:29:29.289  No retry patterns found.
00:29:29.290  [Pipeline] }
00:29:29.320  [Pipeline] // retry
00:29:29.330  [Pipeline] error
00:29:29.337  [Pipeline] }
00:29:29.342  Failed in branch nvmf-tcp-phy-autotest
00:29:29.592  [Pipeline] // parallel
00:29:29.601  [Pipeline] }
00:29:29.643  [Pipeline] // stage
00:29:29.656  [Pipeline] }
00:29:29.667  ERROR: Build nvmf-tcp-uring-vg-autotest #90079 failed
00:29:29.667  Setting overall build result to FAILURE
00:29:29.693  [Pipeline] // catchError
00:29:29.702  [Pipeline] stage
00:29:29.704  [Pipeline] { (Autorun Post and Coverage)
00:29:29.732  [Pipeline] setCustomBuildProperty
00:29:29.758  [Pipeline] dir
00:29:29.759  Running in /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944
00:29:29.760  [Pipeline] {
00:29:29.779  [Pipeline] copyArtifacts
00:29:30.055  Copied 6 artifacts from "doc-docker-autotest" build number 32944
00:29:30.059  [Pipeline] writeFile
00:29:30.081  [Pipeline] }
00:29:30.103  [Pipeline] // dir
00:29:30.119  [Pipeline] dir
00:29:30.120  Running in /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641
00:29:30.121  [Pipeline] {
00:29:30.139  [Pipeline] copyArtifacts
00:29:30.198  Copied 5 artifacts from "check-format-docker-autotest" build number 32641
00:29:30.202  [Pipeline] writeFile
00:29:30.217  [Pipeline] }
00:29:30.235  [Pipeline] // dir
00:29:30.317  [Pipeline] dir
00:29:30.317  Running in /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615
00:29:30.318  [Pipeline] {
00:29:30.335  [Pipeline] copyArtifacts
00:29:30.395  Copied 5 artifacts from "build-files-docker-autotest" build number 32615
00:29:30.398  [Pipeline] writeFile
00:29:30.451  [Pipeline] }
00:29:30.471  [Pipeline] // dir
00:29:30.518  [Pipeline] dir
00:29:30.518  Running in /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862
00:29:30.519  [Pipeline] {
00:29:30.539  [Pipeline] copyArtifacts
00:29:30.596  Copied 5 artifacts from "check-so-deps-docker-autotest" build number 32862
00:29:30.600  [Pipeline] writeFile
00:29:30.630  [Pipeline] }
00:29:30.654  [Pipeline] // dir
00:29:30.710  [Pipeline] dir
00:29:30.711  Running in /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655
00:29:30.712  [Pipeline] {
00:29:30.732  [Pipeline] copyArtifacts
00:29:30.801  Copied 5 artifacts from "release-build-main-docker-autotest" build number 2655
00:29:30.806  [Pipeline] writeFile
00:29:30.838  [Pipeline] }
00:29:30.865  [Pipeline] // dir
00:29:30.997  [Pipeline] dir
00:29:30.998  Running in /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449
00:29:30.999  [Pipeline] {
00:29:31.022  [Pipeline] copyArtifacts
00:29:31.414  Copied 6 artifacts from "unittest-clang-docker-autotest" build number 29449
00:29:31.419  [Pipeline] writeFile
00:29:31.470  [Pipeline] }
00:29:31.507  [Pipeline] // dir
00:29:31.570  [Pipeline] dir
00:29:31.571  Running in /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655
00:29:31.572  [Pipeline] {
00:29:31.598  [Pipeline] copyArtifacts
00:29:31.677  Copied 5 artifacts from "release-build-clang-docker-autotest" build number 2655
00:29:31.682  [Pipeline] writeFile
00:29:31.709  [Pipeline] }
00:29:31.739  [Pipeline] // dir
00:29:31.758  [Pipeline] dir
00:29:31.758  Running in /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456
00:29:31.760  [Pipeline] {
00:29:31.778  [Pipeline] copyArtifacts
00:29:31.890  Copied 9 artifacts from "scan-build-docker-autotest" build number 29456
00:29:31.895  [Pipeline] writeFile
00:29:31.917  [Pipeline] }
00:29:31.947  [Pipeline] // dir
00:29:32.304  [Pipeline] dir
00:29:32.305  Running in /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449
00:29:32.309  [Pipeline] {
00:29:32.367  [Pipeline] copyArtifacts
00:29:32.444  Copied 5 artifacts from "packaging-docker-autotest" build number 29449
00:29:32.448  [Pipeline] writeFile
00:29:32.579  [Pipeline] }
00:29:32.643  [Pipeline] // dir
00:29:32.782  [Pipeline] dir
00:29:32.783  Running in /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451
00:29:32.784  [Pipeline] {
00:29:32.807  [Pipeline] copyArtifacts
00:29:33.096  Copied 6 artifacts from "unittest-main-docker-autotest" build number 29451
00:29:33.101  [Pipeline] writeFile
00:29:33.151  [Pipeline] }
00:29:33.194  [Pipeline] // dir
00:29:33.514  [Pipeline] dir
00:29:33.514  Running in /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154
00:29:33.516  [Pipeline] {
00:29:33.636  [Pipeline] copyArtifacts
00:29:33.914  Copied 16 artifacts from "vhost-initiator-vg-autotest" build number 90154
00:29:33.918  [Pipeline] writeFile
00:29:34.043  [Pipeline] }
00:29:34.113  [Pipeline] // dir
00:29:34.230  [Pipeline] dir
00:29:34.230  Running in /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093
00:29:34.232  [Pipeline] {
00:29:34.254  [Pipeline] copyArtifacts
00:29:34.372  Copied 9 artifacts from "autobuild-external-vg-autotest" build number 30093
00:29:34.376  [Pipeline] writeFile
00:29:34.447  [Pipeline] }
00:29:34.487  [Pipeline] // dir
00:29:34.814  [Pipeline] dir
00:29:34.815  Running in /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805
00:29:34.816  [Pipeline] {
00:29:34.852  [Pipeline] copyArtifacts
00:29:35.132  Copied 14 artifacts from "short-fuzz-phy-autotest" build number 37805
00:29:35.136  [Pipeline] writeFile
00:29:35.290  [Pipeline] }
00:29:35.372  [Pipeline] // dir
00:29:35.580  [Pipeline] dir
00:29:35.581  Running in /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238
00:29:35.582  [Pipeline] {
00:29:35.605  [Pipeline] copyArtifacts
00:29:35.832  Copied 13 artifacts from "zns-vg-autotest" build number 89238
00:29:35.837  [Pipeline] writeFile
00:29:35.921  [Pipeline] }
00:29:35.965  [Pipeline] // dir
00:29:36.329  [Pipeline] dir
00:29:36.330  Running in /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454
00:29:36.331  [Pipeline] {
00:29:36.360  [Pipeline] copyArtifacts
00:29:36.655  Copied 6 artifacts from "unittest-valgrind-docker-autotest" build number 29454
00:29:36.660  [Pipeline] writeFile
00:29:36.796  [Pipeline] }
00:29:36.866  [Pipeline] // dir
00:29:37.113  [Pipeline] dir
00:29:37.114  Running in /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264
00:29:37.115  [Pipeline] {
00:29:37.143  [Pipeline] copyArtifacts
00:29:37.373  Copied 13 artifacts from "nvme-cmb-pmr-vg-autotest" build number 91264
00:29:37.378  [Pipeline] writeFile
00:29:37.477  [Pipeline] }
00:29:37.537  [Pipeline] // dir
00:29:37.841  [Pipeline] dir
00:29:37.842  Running in /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225
00:29:37.843  [Pipeline] {
00:29:37.872  [Pipeline] copyArtifacts
00:29:38.283  Copied 12 artifacts from "freebsd-vg-autotest" build number 68225
00:29:38.289  [Pipeline] writeFile
00:29:38.402  [Pipeline] }
00:29:38.472  [Pipeline] // dir
00:29:38.745  [Pipeline] dir
00:29:38.745  Running in /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820
00:29:38.747  [Pipeline] {
00:29:38.773  [Pipeline] copyArtifacts
00:29:39.238  Copied 29 artifacts from "BlobFS-phy-autotest" build number 37820
00:29:39.244  [Pipeline] writeFile
00:29:39.360  [Pipeline] }
00:29:39.430  [Pipeline] // dir
00:29:40.102  [Pipeline] dir
00:29:40.102  Running in /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085
00:29:40.103  [Pipeline] {
00:29:40.151  [Pipeline] copyArtifacts
00:29:40.405  Copied 13 artifacts from "lvol-vg-autotest" build number 90085
00:29:40.409  [Pipeline] writeFile
00:29:40.617  [Pipeline] }
00:29:40.707  [Pipeline] // dir
00:29:40.952  [Pipeline] dir
00:29:40.952  Running in /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874
00:29:40.953  [Pipeline] {
00:29:40.982  [Pipeline] copyArtifacts
00:29:41.235  Copied 13 artifacts from "vhost-phy-autotest" build number 37874
00:29:41.240  [Pipeline] writeFile
00:29:41.340  [Pipeline] }
00:29:41.394  [Pipeline] // dir
00:29:41.776  [Pipeline] dir
00:29:41.777  Running in /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037
00:29:41.778  [Pipeline] {
00:29:41.811  [Pipeline] copyArtifacts
00:29:44.218  Copied 17 artifacts from "iscsi-uring-vg-autotest" build number 90037
00:29:44.223  [Pipeline] writeFile
00:29:44.341  [Pipeline] }
00:29:44.404  [Pipeline] // dir
00:29:45.043  [Pipeline] dir
00:29:45.043  Running in /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809
00:29:45.044  [Pipeline] {
00:29:45.092  [Pipeline] copyArtifacts
00:29:45.369  Copied 13 artifacts from "crypto-phy-autotest" build number 37809
00:29:45.375  [Pipeline] writeFile
00:29:45.616  [Pipeline] }
00:29:45.718  [Pipeline] // dir
00:29:47.074  [Pipeline] dir
00:29:47.075  Running in /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079
00:29:47.076  [Pipeline] {
00:29:47.151  [Pipeline] copyArtifacts
00:29:47.616  Copied 13 artifacts from "nvmf-tcp-uring-vg-autotest" build number 90079
00:29:47.622  [Pipeline] writeFile
00:29:48.111  [Pipeline] }
00:29:48.314  [Pipeline] // dir
00:29:48.672  [Pipeline] dir
00:29:48.672  Running in /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810
00:29:48.673  [Pipeline] {
00:29:48.710  [Pipeline] copyArtifacts
00:29:48.946  Copied 13 artifacts from "vfio-user-phy-autotest" build number 37810
00:29:48.950  [Pipeline] writeFile
00:29:49.097  [Pipeline] }
00:29:49.183  [Pipeline] // dir
00:29:49.503  [Pipeline] dir
00:29:49.504  Running in /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173
00:29:49.505  [Pipeline] {
00:29:49.538  [Pipeline] copyArtifacts
00:29:52.905  Copied 17 artifacts from "iscsi-vg-autotest" build number 90173
00:29:52.910  [Pipeline] writeFile
00:29:53.037  [Pipeline] }
00:29:53.091  [Pipeline] // dir
00:29:54.190  [Pipeline] dir
00:29:54.190  Running in /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118
00:29:54.191  [Pipeline] {
00:29:54.259  [Pipeline] copyArtifacts
00:29:54.864  Copied 13 artifacts from "nvmf-tcp-vg-autotest" build number 90118
00:29:54.869  [Pipeline] writeFile
00:29:55.317  [Pipeline] }
00:29:55.500  [Pipeline] // dir
00:29:55.833  [Pipeline] dir
00:29:55.833  Running in /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396
00:29:55.834  [Pipeline] {
00:29:55.866  [Pipeline] copyArtifacts
00:29:56.271  Copied 13 artifacts from "rocky9-vg-autotest" build number 27396
00:29:56.275  [Pipeline] writeFile
00:29:56.404  [Pipeline] }
00:29:56.494  [Pipeline] // dir
00:29:56.949  [Pipeline] dir
00:29:56.949  Running in /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083
00:29:56.950  [Pipeline] {
00:29:56.983  [Pipeline] copyArtifacts
00:29:57.438  Copied 13 artifacts from "ubuntu22-vg-autotest" build number 49083
00:29:57.442  [Pipeline] writeFile
00:29:57.604  [Pipeline] }
00:29:57.683  [Pipeline] // dir
00:29:58.352  [Pipeline] dir
00:29:58.352  Running in /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052
00:29:58.353  [Pipeline] {
00:29:58.402  [Pipeline] copyArtifacts
00:29:58.858  Copied 13 artifacts from "nvmf-phy-autotest" build number 84052
00:29:58.862  [Pipeline] writeFile
00:29:59.093  [Pipeline] }
00:29:59.215  [Pipeline] // dir
00:30:00.334  [Pipeline] dir
00:30:00.334  Running in /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805
00:30:00.335  [Pipeline] {
00:30:00.390  [Pipeline] copyArtifacts
00:30:00.591  Copied 13 artifacts from "raid-vg-autotest" build number 4805
00:30:00.595  [Pipeline] writeFile
00:30:00.948  [Pipeline] }
00:30:01.124  [Pipeline] // dir
00:30:02.051  [Pipeline] dir
00:30:02.051  Running in /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983
00:30:02.052  [Pipeline] {
00:30:02.098  [Pipeline] copyArtifacts
00:30:02.359  Copied 13 artifacts from "nvme-phy-autotest" build number 83983
00:30:02.364  [Pipeline] writeFile
00:30:02.666  [Pipeline] }
00:30:02.798  [Pipeline] // dir
00:30:03.305  [Pipeline] dir
00:30:03.305  Running in /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644
00:30:03.306  [Pipeline] {
00:30:03.343  [Pipeline] copyArtifacts
00:30:03.790  Copied 13 artifacts from "ubuntu24-vg-autotest" build number 6644
00:30:03.795  [Pipeline] writeFile
00:30:03.956  [Pipeline] }
00:30:04.047  [Pipeline] // dir
00:30:04.669  [Pipeline] dir
00:30:04.670  Running in /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136
00:30:04.671  [Pipeline] {
00:30:04.715  [Pipeline] copyArtifacts
00:30:04.935  Copied 13 artifacts from "nvme-vg-autotest" build number 90136
00:30:04.940  [Pipeline] writeFile
00:30:05.151  [Pipeline] }
00:30:05.269  [Pipeline] // dir
00:30:06.396  [Pipeline] dir
00:30:06.397  Running in /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934
00:30:06.398  [Pipeline] {
00:30:06.475  [Pipeline] copyArtifacts
00:30:07.062  Copied 13 artifacts from "nvmf-tcp-phy-autotest" build number 83934
00:30:07.067  [Pipeline] writeFile
00:30:07.566  [Pipeline] }
00:30:07.774  [Pipeline] // dir
00:30:07.784  [Pipeline] catchError
00:30:07.787  [Pipeline] {
00:30:07.803  [Pipeline] sh
00:30:08.077  + jbp/jenkins/jjb-config/jobs/scripts/post_gen_coverage.sh
00:30:08.077  + shopt -s globstar nullglob
00:30:08.077  + echo 'Start stage post_gen_coverage.sh'
00:30:08.077  Start stage post_gen_coverage.sh
00:30:08.077  + cd /var/jenkins/workspace/autotest-per-patch
00:30:08.077  + rm -rf /var/jenkins/workspace/autotest-per-patch/spdk/doc
00:30:08.077  + trap 'compress_coverage_and_docs; remove_partial_coverage_files && echo '\''End stage post_gen_coverage.sh'\''' EXIT
00:30:08.077  + move_artifacts
00:30:08.077  + local out_dirs
00:30:08.077  + out_dirs=(./**/output/)
00:30:08.077  + for dir in "${out_dirs[@]}"
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./autobuild-external-vg-autotest_30093/output//doc.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./autobuild-external-vg-autotest_30093/output//ut_coverage.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./autobuild-external-vg-autotest_30093/output//llvm.tar.xz ]]
00:30:08.077  + mv ./autobuild-external-vg-autotest_30093/output//build-kernel-manifest.txt ./autobuild-external-vg-autotest_30093/output//build-pkg-manifest.txt ./autobuild-external-vg-autotest_30093/output//build-repo-manifest.txt ./autobuild-external-vg-autotest_30093/output//dmesg.init.log ./autobuild-external-vg-autotest_30093/output//dmesg.log ./autobuild-external-vg-autotest_30093/output//power.tar.xz ./autobuild-external-vg-autotest_30093/output//test_completions.txt ./autobuild-external-vg-autotest_30093/output//timing.svg ./autobuild-external-vg-autotest_30093/output//timing.txt ./autobuild-external-vg-autotest_30093/output//..
00:30:08.077  + rmdir ./autobuild-external-vg-autotest_30093/output/
00:30:08.077  + for dir in "${out_dirs[@]}"
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./BlobFS-phy-autotest_37820/output//doc.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./BlobFS-phy-autotest_37820/output//ut_coverage.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./BlobFS-phy-autotest_37820/output//llvm.tar.xz ]]
00:30:08.077  + mv ./BlobFS-phy-autotest_37820/output//all_tests.txt ./BlobFS-phy-autotest_37820/output//build-kernel-manifest.txt ./BlobFS-phy-autotest_37820/output//build-pkg-manifest.txt ./BlobFS-phy-autotest_37820/output//build-repo-manifest.txt ./BlobFS-phy-autotest_37820/output//cov_base.info.xz ./BlobFS-phy-autotest_37820/output//cov_test.info.xz ./BlobFS-phy-autotest_37820/output//cov_total.info.xz ./BlobFS-phy-autotest_37820/output//dmesg.init.log ./BlobFS-phy-autotest_37820/output//dmesg.log ./BlobFS-phy-autotest_37820/output//power.tar.xz ./BlobFS-phy-autotest_37820/output//rocksdb ./BlobFS-phy-autotest_37820/output//test_completions.txt ./BlobFS-phy-autotest_37820/output//timing.svg ./BlobFS-phy-autotest_37820/output//timing.txt ./BlobFS-phy-autotest_37820/output//..
00:30:08.077  + rmdir ./BlobFS-phy-autotest_37820/output/
00:30:08.077  + for dir in "${out_dirs[@]}"
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./build-files-docker-autotest_32615/output//doc.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./build-files-docker-autotest_32615/output//ut_coverage.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./build-files-docker-autotest_32615/output//llvm.tar.xz ]]
00:30:08.077  + mv ./build-files-docker-autotest_32615/output//build-repo-manifest.txt ./build-files-docker-autotest_32615/output//power.tar.xz ./build-files-docker-autotest_32615/output//test_completions.txt ./build-files-docker-autotest_32615/output//timing.svg ./build-files-docker-autotest_32615/output//timing.txt ./build-files-docker-autotest_32615/output//..
00:30:08.077  + rmdir ./build-files-docker-autotest_32615/output/
00:30:08.077  + for dir in "${out_dirs[@]}"
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./check-format-docker-autotest_32641/output//doc.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./check-format-docker-autotest_32641/output//ut_coverage.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./check-format-docker-autotest_32641/output//llvm.tar.xz ]]
00:30:08.077  + mv ./check-format-docker-autotest_32641/output//build-repo-manifest.txt ./check-format-docker-autotest_32641/output//power.tar.xz ./check-format-docker-autotest_32641/output//test_completions.txt ./check-format-docker-autotest_32641/output//timing.svg ./check-format-docker-autotest_32641/output//timing.txt ./check-format-docker-autotest_32641/output//..
00:30:08.077  + rmdir ./check-format-docker-autotest_32641/output/
00:30:08.077  + for dir in "${out_dirs[@]}"
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./check-so-deps-docker-autotest_32862/output//doc.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./check-so-deps-docker-autotest_32862/output//ut_coverage.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./check-so-deps-docker-autotest_32862/output//llvm.tar.xz ]]
00:30:08.077  + mv ./check-so-deps-docker-autotest_32862/output//build-repo-manifest.txt ./check-so-deps-docker-autotest_32862/output//power.tar.xz ./check-so-deps-docker-autotest_32862/output//test_completions.txt ./check-so-deps-docker-autotest_32862/output//timing.svg ./check-so-deps-docker-autotest_32862/output//timing.txt ./check-so-deps-docker-autotest_32862/output//..
00:30:08.077  + rmdir ./check-so-deps-docker-autotest_32862/output/
00:30:08.077  + for dir in "${out_dirs[@]}"
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./crypto-phy-autotest_37809/output//doc.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./crypto-phy-autotest_37809/output//ut_coverage.tar.xz ]]
00:30:08.077  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.077  + [[ -f ./crypto-phy-autotest_37809/output//llvm.tar.xz ]]
00:30:08.077  + mv ./crypto-phy-autotest_37809/output//all_tests.txt ./crypto-phy-autotest_37809/output//build-kernel-manifest.txt ./crypto-phy-autotest_37809/output//build-pkg-manifest.txt ./crypto-phy-autotest_37809/output//build-repo-manifest.txt ./crypto-phy-autotest_37809/output//cov_base.info.xz ./crypto-phy-autotest_37809/output//cov_test.info.xz ./crypto-phy-autotest_37809/output//cov_total.info.xz ./crypto-phy-autotest_37809/output//dmesg.init.log ./crypto-phy-autotest_37809/output//dmesg.log ./crypto-phy-autotest_37809/output//power.tar.xz ./crypto-phy-autotest_37809/output//test_completions.txt ./crypto-phy-autotest_37809/output//timing.svg ./crypto-phy-autotest_37809/output//timing.txt ./crypto-phy-autotest_37809/output//..
00:30:08.077  + rmdir ./crypto-phy-autotest_37809/output/
00:30:08.077  + for dir in "${out_dirs[@]}"
00:30:08.078  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.078  + [[ -f ./doc-docker-autotest_32944/output//doc.tar.xz ]]
00:30:08.078  + tar -C ./doc-docker-autotest_32944/output/ -xf ./doc-docker-autotest_32944/output//doc.tar.xz
00:30:08.335  + rm ./doc-docker-autotest_32944/output//doc.tar.xz
00:30:08.335  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.335  + [[ -f ./doc-docker-autotest_32944/output//ut_coverage.tar.xz ]]
00:30:08.335  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.335  + [[ -f ./doc-docker-autotest_32944/output//llvm.tar.xz ]]
00:30:08.336  + mv ./doc-docker-autotest_32944/output//build-repo-manifest.txt ./doc-docker-autotest_32944/output//doc ./doc-docker-autotest_32944/output//power.tar.xz ./doc-docker-autotest_32944/output//test_completions.txt ./doc-docker-autotest_32944/output//timing.svg ./doc-docker-autotest_32944/output//timing.txt ./doc-docker-autotest_32944/output//..
00:30:08.336  + rmdir ./doc-docker-autotest_32944/output/
00:30:08.336  + for dir in "${out_dirs[@]}"
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./freebsd-vg-autotest_68225/output//doc.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./freebsd-vg-autotest_68225/output//ut_coverage.tar.xz ]]
00:30:08.336  + tar -C ./freebsd-vg-autotest_68225/output/ -xf ./freebsd-vg-autotest_68225/output//ut_coverage.tar.xz
00:30:08.336  + rm ./freebsd-vg-autotest_68225/output//ut_coverage.tar.xz
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./freebsd-vg-autotest_68225/output//llvm.tar.xz ]]
00:30:08.336  + mv ./freebsd-vg-autotest_68225/output//all_tests.txt ./freebsd-vg-autotest_68225/output//build-pkg-manifest.txt ./freebsd-vg-autotest_68225/output//build-repo-manifest.txt ./freebsd-vg-autotest_68225/output//cov_base.info.xz ./freebsd-vg-autotest_68225/output//cov_test.info.xz ./freebsd-vg-autotest_68225/output//cov_total.info.xz ./freebsd-vg-autotest_68225/output//messages ./freebsd-vg-autotest_68225/output//power.tar.xz ./freebsd-vg-autotest_68225/output//test_completions.txt ./freebsd-vg-autotest_68225/output//timing.svg ./freebsd-vg-autotest_68225/output//timing.txt ./freebsd-vg-autotest_68225/output//ut_coverage ./freebsd-vg-autotest_68225/output//..
00:30:08.336  + rmdir ./freebsd-vg-autotest_68225/output/
00:30:08.336  + for dir in "${out_dirs[@]}"
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./iscsi-uring-vg-autotest_90037/output//doc.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./iscsi-uring-vg-autotest_90037/output//ut_coverage.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./iscsi-uring-vg-autotest_90037/output//llvm.tar.xz ]]
00:30:08.336  + mv ./iscsi-uring-vg-autotest_90037/output//all_tests.txt ./iscsi-uring-vg-autotest_90037/output//build-kernel-manifest.txt ./iscsi-uring-vg-autotest_90037/output//build-pkg-manifest.txt ./iscsi-uring-vg-autotest_90037/output//build-repo-manifest.txt ./iscsi-uring-vg-autotest_90037/output//calsoft.json ./iscsi-uring-vg-autotest_90037/output//calsoft.tar.xz ./iscsi-uring-vg-autotest_90037/output//cov_base.info.xz ./iscsi-uring-vg-autotest_90037/output//cov_test.info.xz ./iscsi-uring-vg-autotest_90037/output//cov_total.info.xz ./iscsi-uring-vg-autotest_90037/output//dmesg.init.log ./iscsi-uring-vg-autotest_90037/output//dmesg.log ./iscsi-uring-vg-autotest_90037/output//iscsi_autofuzz_logs.txt ./iscsi-uring-vg-autotest_90037/output//iscsi_autofuzz_tgt_output.txt ./iscsi-uring-vg-autotest_90037/output//power.tar.xz ./iscsi-uring-vg-autotest_90037/output//test_completions.txt ./iscsi-uring-vg-autotest_90037/output//timing.svg ./iscsi-uring-vg-autotest_90037/output//timing.txt ./iscsi-uring-vg-autotest_90037/output//..
00:30:08.336  + rmdir ./iscsi-uring-vg-autotest_90037/output/
00:30:08.336  + for dir in "${out_dirs[@]}"
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./iscsi-vg-autotest_90173/output//doc.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./iscsi-vg-autotest_90173/output//ut_coverage.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./iscsi-vg-autotest_90173/output//llvm.tar.xz ]]
00:30:08.336  + mv ./iscsi-vg-autotest_90173/output//all_tests.txt ./iscsi-vg-autotest_90173/output//build-kernel-manifest.txt ./iscsi-vg-autotest_90173/output//build-pkg-manifest.txt ./iscsi-vg-autotest_90173/output//build-repo-manifest.txt ./iscsi-vg-autotest_90173/output//calsoft.json ./iscsi-vg-autotest_90173/output//calsoft.tar.xz ./iscsi-vg-autotest_90173/output//cov_base.info.xz ./iscsi-vg-autotest_90173/output//cov_test.info.xz ./iscsi-vg-autotest_90173/output//cov_total.info.xz ./iscsi-vg-autotest_90173/output//dmesg.init.log ./iscsi-vg-autotest_90173/output//dmesg.log ./iscsi-vg-autotest_90173/output//iscsi_autofuzz_logs.txt ./iscsi-vg-autotest_90173/output//iscsi_autofuzz_tgt_output.txt ./iscsi-vg-autotest_90173/output//power.tar.xz ./iscsi-vg-autotest_90173/output//test_completions.txt ./iscsi-vg-autotest_90173/output//timing.svg ./iscsi-vg-autotest_90173/output//timing.txt ./iscsi-vg-autotest_90173/output//..
00:30:08.336  + rmdir ./iscsi-vg-autotest_90173/output/
00:30:08.336  + for dir in "${out_dirs[@]}"
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./lvol-vg-autotest_90085/output//doc.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./lvol-vg-autotest_90085/output//ut_coverage.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./lvol-vg-autotest_90085/output//llvm.tar.xz ]]
00:30:08.336  + mv ./lvol-vg-autotest_90085/output//all_tests.txt ./lvol-vg-autotest_90085/output//build-kernel-manifest.txt ./lvol-vg-autotest_90085/output//build-pkg-manifest.txt ./lvol-vg-autotest_90085/output//build-repo-manifest.txt ./lvol-vg-autotest_90085/output//cov_base.info.xz ./lvol-vg-autotest_90085/output//cov_test.info.xz ./lvol-vg-autotest_90085/output//cov_total.info.xz ./lvol-vg-autotest_90085/output//dmesg.init.log ./lvol-vg-autotest_90085/output//dmesg.log ./lvol-vg-autotest_90085/output//power.tar.xz ./lvol-vg-autotest_90085/output//test_completions.txt ./lvol-vg-autotest_90085/output//timing.svg ./lvol-vg-autotest_90085/output//timing.txt ./lvol-vg-autotest_90085/output//..
00:30:08.336  + rmdir ./lvol-vg-autotest_90085/output/
00:30:08.336  + for dir in "${out_dirs[@]}"
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./nvme-cmb-pmr-vg-autotest_91264/output//doc.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./nvme-cmb-pmr-vg-autotest_91264/output//ut_coverage.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./nvme-cmb-pmr-vg-autotest_91264/output//llvm.tar.xz ]]
00:30:08.336  + mv ./nvme-cmb-pmr-vg-autotest_91264/output//all_tests.txt ./nvme-cmb-pmr-vg-autotest_91264/output//build-kernel-manifest.txt ./nvme-cmb-pmr-vg-autotest_91264/output//build-pkg-manifest.txt ./nvme-cmb-pmr-vg-autotest_91264/output//build-repo-manifest.txt ./nvme-cmb-pmr-vg-autotest_91264/output//cov_base.info.xz ./nvme-cmb-pmr-vg-autotest_91264/output//cov_test.info.xz ./nvme-cmb-pmr-vg-autotest_91264/output//cov_total.info.xz ./nvme-cmb-pmr-vg-autotest_91264/output//dmesg.init.log ./nvme-cmb-pmr-vg-autotest_91264/output//dmesg.log ./nvme-cmb-pmr-vg-autotest_91264/output//power.tar.xz ./nvme-cmb-pmr-vg-autotest_91264/output//test_completions.txt ./nvme-cmb-pmr-vg-autotest_91264/output//timing.svg ./nvme-cmb-pmr-vg-autotest_91264/output//timing.txt ./nvme-cmb-pmr-vg-autotest_91264/output//..
00:30:08.336  + rmdir ./nvme-cmb-pmr-vg-autotest_91264/output/
00:30:08.336  + for dir in "${out_dirs[@]}"
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./nvme-phy-autotest_83983/output//doc.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./nvme-phy-autotest_83983/output//ut_coverage.tar.xz ]]
00:30:08.336  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.336  + [[ -f ./nvme-phy-autotest_83983/output//llvm.tar.xz ]]
00:30:08.336  + mv ./nvme-phy-autotest_83983/output//all_tests.txt ./nvme-phy-autotest_83983/output//build-kernel-manifest.txt ./nvme-phy-autotest_83983/output//build-pkg-manifest.txt ./nvme-phy-autotest_83983/output//build-repo-manifest.txt ./nvme-phy-autotest_83983/output//cov_base.info.xz ./nvme-phy-autotest_83983/output//cov_test.info.xz ./nvme-phy-autotest_83983/output//cov_total.info.xz ./nvme-phy-autotest_83983/output//dmesg.init.log ./nvme-phy-autotest_83983/output//dmesg.log ./nvme-phy-autotest_83983/output//power.tar.xz ./nvme-phy-autotest_83983/output//test_completions.txt ./nvme-phy-autotest_83983/output//timing.svg ./nvme-phy-autotest_83983/output//timing.txt ./nvme-phy-autotest_83983/output//..
00:30:08.336  + rmdir ./nvme-phy-autotest_83983/output/
00:30:08.595  + for dir in "${out_dirs[@]}"
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvme-vg-autotest_90136/output//doc.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvme-vg-autotest_90136/output//ut_coverage.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvme-vg-autotest_90136/output//llvm.tar.xz ]]
00:30:08.595  + mv ./nvme-vg-autotest_90136/output//all_tests.txt ./nvme-vg-autotest_90136/output//build-kernel-manifest.txt ./nvme-vg-autotest_90136/output//build-pkg-manifest.txt ./nvme-vg-autotest_90136/output//build-repo-manifest.txt ./nvme-vg-autotest_90136/output//cov_base.info.xz ./nvme-vg-autotest_90136/output//cov_test.info.xz ./nvme-vg-autotest_90136/output//cov_total.info.xz ./nvme-vg-autotest_90136/output//dmesg.init.log ./nvme-vg-autotest_90136/output//dmesg.log ./nvme-vg-autotest_90136/output//power.tar.xz ./nvme-vg-autotest_90136/output//test_completions.txt ./nvme-vg-autotest_90136/output//timing.svg ./nvme-vg-autotest_90136/output//timing.txt ./nvme-vg-autotest_90136/output//..
00:30:08.595  + rmdir ./nvme-vg-autotest_90136/output/
00:30:08.595  + for dir in "${out_dirs[@]}"
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-phy-autotest_84052/output//doc.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-phy-autotest_84052/output//ut_coverage.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-phy-autotest_84052/output//llvm.tar.xz ]]
00:30:08.595  + mv ./nvmf-phy-autotest_84052/output//all_tests.txt ./nvmf-phy-autotest_84052/output//build-kernel-manifest.txt ./nvmf-phy-autotest_84052/output//build-pkg-manifest.txt ./nvmf-phy-autotest_84052/output//build-repo-manifest.txt ./nvmf-phy-autotest_84052/output//cov_base.info.xz ./nvmf-phy-autotest_84052/output//dmesg.init.log ./nvmf-phy-autotest_84052/output//dmesg.log ./nvmf-phy-autotest_84052/output//nvmf_trace.0_shm.tar.gz ./nvmf-phy-autotest_84052/output//power.tar.xz ./nvmf-phy-autotest_84052/output//proc_list.txt ./nvmf-phy-autotest_84052/output//test_completions.txt ./nvmf-phy-autotest_84052/output//timing.txt ./nvmf-phy-autotest_84052/output//udev.log ./nvmf-phy-autotest_84052/output//..
00:30:08.595  + rmdir ./nvmf-phy-autotest_84052/output/
00:30:08.595  + for dir in "${out_dirs[@]}"
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-phy-autotest_83934/output//doc.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-phy-autotest_83934/output//ut_coverage.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-phy-autotest_83934/output//llvm.tar.xz ]]
00:30:08.595  + mv ./nvmf-tcp-phy-autotest_83934/output//all_tests.txt ./nvmf-tcp-phy-autotest_83934/output//build-kernel-manifest.txt ./nvmf-tcp-phy-autotest_83934/output//build-pkg-manifest.txt ./nvmf-tcp-phy-autotest_83934/output//build-repo-manifest.txt ./nvmf-tcp-phy-autotest_83934/output//cov_base.info.xz ./nvmf-tcp-phy-autotest_83934/output//dmesg.init.log ./nvmf-tcp-phy-autotest_83934/output//dmesg.log ./nvmf-tcp-phy-autotest_83934/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-phy-autotest_83934/output//power.tar.xz ./nvmf-tcp-phy-autotest_83934/output//proc_list.txt ./nvmf-tcp-phy-autotest_83934/output//test_completions.txt ./nvmf-tcp-phy-autotest_83934/output//timing.txt ./nvmf-tcp-phy-autotest_83934/output//udev.log ./nvmf-tcp-phy-autotest_83934/output//..
00:30:08.595  + rmdir ./nvmf-tcp-phy-autotest_83934/output/
00:30:08.595  + for dir in "${out_dirs[@]}"
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-uring-vg-autotest_90079/output//doc.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-uring-vg-autotest_90079/output//ut_coverage.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-uring-vg-autotest_90079/output//llvm.tar.xz ]]
00:30:08.595  + mv ./nvmf-tcp-uring-vg-autotest_90079/output//all_tests.txt ./nvmf-tcp-uring-vg-autotest_90079/output//build-kernel-manifest.txt ./nvmf-tcp-uring-vg-autotest_90079/output//build-pkg-manifest.txt ./nvmf-tcp-uring-vg-autotest_90079/output//build-repo-manifest.txt ./nvmf-tcp-uring-vg-autotest_90079/output//cov_base.info.xz ./nvmf-tcp-uring-vg-autotest_90079/output//dmesg.init.log ./nvmf-tcp-uring-vg-autotest_90079/output//dmesg.log ./nvmf-tcp-uring-vg-autotest_90079/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-uring-vg-autotest_90079/output//power.tar.xz ./nvmf-tcp-uring-vg-autotest_90079/output//proc_list.txt ./nvmf-tcp-uring-vg-autotest_90079/output//test_completions.txt ./nvmf-tcp-uring-vg-autotest_90079/output//timing.txt ./nvmf-tcp-uring-vg-autotest_90079/output//udev.log ./nvmf-tcp-uring-vg-autotest_90079/output//..
00:30:08.595  + rmdir ./nvmf-tcp-uring-vg-autotest_90079/output/
00:30:08.595  + for dir in "${out_dirs[@]}"
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-vg-autotest_90118/output//doc.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-vg-autotest_90118/output//ut_coverage.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./nvmf-tcp-vg-autotest_90118/output//llvm.tar.xz ]]
00:30:08.595  + mv ./nvmf-tcp-vg-autotest_90118/output//all_tests.txt ./nvmf-tcp-vg-autotest_90118/output//build-kernel-manifest.txt ./nvmf-tcp-vg-autotest_90118/output//build-pkg-manifest.txt ./nvmf-tcp-vg-autotest_90118/output//build-repo-manifest.txt ./nvmf-tcp-vg-autotest_90118/output//cov_base.info.xz ./nvmf-tcp-vg-autotest_90118/output//dmesg.init.log ./nvmf-tcp-vg-autotest_90118/output//dmesg.log ./nvmf-tcp-vg-autotest_90118/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-vg-autotest_90118/output//power.tar.xz ./nvmf-tcp-vg-autotest_90118/output//proc_list.txt ./nvmf-tcp-vg-autotest_90118/output//test_completions.txt ./nvmf-tcp-vg-autotest_90118/output//timing.txt ./nvmf-tcp-vg-autotest_90118/output//udev.log ./nvmf-tcp-vg-autotest_90118/output//..
00:30:08.595  + rmdir ./nvmf-tcp-vg-autotest_90118/output/
00:30:08.595  + for dir in "${out_dirs[@]}"
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./packaging-docker-autotest_29449/output//doc.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./packaging-docker-autotest_29449/output//ut_coverage.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./packaging-docker-autotest_29449/output//llvm.tar.xz ]]
00:30:08.595  + mv ./packaging-docker-autotest_29449/output//build-repo-manifest.txt ./packaging-docker-autotest_29449/output//power.tar.xz ./packaging-docker-autotest_29449/output//test_completions.txt ./packaging-docker-autotest_29449/output//timing.svg ./packaging-docker-autotest_29449/output//timing.txt ./packaging-docker-autotest_29449/output//..
00:30:08.595  + rmdir ./packaging-docker-autotest_29449/output/
00:30:08.595  + for dir in "${out_dirs[@]}"
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./raid-vg-autotest_4805/output//doc.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./raid-vg-autotest_4805/output//ut_coverage.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./raid-vg-autotest_4805/output//llvm.tar.xz ]]
00:30:08.595  + mv ./raid-vg-autotest_4805/output//all_tests.txt ./raid-vg-autotest_4805/output//build-kernel-manifest.txt ./raid-vg-autotest_4805/output//build-pkg-manifest.txt ./raid-vg-autotest_4805/output//build-repo-manifest.txt ./raid-vg-autotest_4805/output//cov_base.info.xz ./raid-vg-autotest_4805/output//cov_test.info.xz ./raid-vg-autotest_4805/output//cov_total.info.xz ./raid-vg-autotest_4805/output//dmesg.init.log ./raid-vg-autotest_4805/output//dmesg.log ./raid-vg-autotest_4805/output//power.tar.xz ./raid-vg-autotest_4805/output//test_completions.txt ./raid-vg-autotest_4805/output//timing.svg ./raid-vg-autotest_4805/output//timing.txt ./raid-vg-autotest_4805/output//..
00:30:08.595  + rmdir ./raid-vg-autotest_4805/output/
00:30:08.595  + for dir in "${out_dirs[@]}"
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./release-build-clang-docker-autotest_2655/output//doc.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./release-build-clang-docker-autotest_2655/output//ut_coverage.tar.xz ]]
00:30:08.595  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.595  + [[ -f ./release-build-clang-docker-autotest_2655/output//llvm.tar.xz ]]
00:30:08.596  + mv ./release-build-clang-docker-autotest_2655/output//build-repo-manifest.txt ./release-build-clang-docker-autotest_2655/output//power.tar.xz ./release-build-clang-docker-autotest_2655/output//test_completions.txt ./release-build-clang-docker-autotest_2655/output//timing.svg ./release-build-clang-docker-autotest_2655/output//timing.txt ./release-build-clang-docker-autotest_2655/output//..
00:30:08.596  + rmdir ./release-build-clang-docker-autotest_2655/output/
00:30:08.596  + for dir in "${out_dirs[@]}"
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./release-build-main-docker-autotest_2655/output//doc.tar.xz ]]
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./release-build-main-docker-autotest_2655/output//ut_coverage.tar.xz ]]
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./release-build-main-docker-autotest_2655/output//llvm.tar.xz ]]
00:30:08.596  + mv ./release-build-main-docker-autotest_2655/output//build-repo-manifest.txt ./release-build-main-docker-autotest_2655/output//power.tar.xz ./release-build-main-docker-autotest_2655/output//test_completions.txt ./release-build-main-docker-autotest_2655/output//timing.svg ./release-build-main-docker-autotest_2655/output//timing.txt ./release-build-main-docker-autotest_2655/output//..
00:30:08.596  + rmdir ./release-build-main-docker-autotest_2655/output/
00:30:08.596  + for dir in "${out_dirs[@]}"
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./rocky9-vg-autotest_27396/output//doc.tar.xz ]]
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./rocky9-vg-autotest_27396/output//ut_coverage.tar.xz ]]
00:30:08.596  + tar -C ./rocky9-vg-autotest_27396/output/ -xf ./rocky9-vg-autotest_27396/output//ut_coverage.tar.xz
00:30:08.596  + rm ./rocky9-vg-autotest_27396/output//ut_coverage.tar.xz
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./rocky9-vg-autotest_27396/output//llvm.tar.xz ]]
00:30:08.596  + mv ./rocky9-vg-autotest_27396/output//all_tests.txt ./rocky9-vg-autotest_27396/output//build-pkg-manifest.txt ./rocky9-vg-autotest_27396/output//build-repo-manifest.txt ./rocky9-vg-autotest_27396/output//cov_base.info.xz ./rocky9-vg-autotest_27396/output//cov_test.info.xz ./rocky9-vg-autotest_27396/output//cov_total.info.xz ./rocky9-vg-autotest_27396/output//dmesg.init.log ./rocky9-vg-autotest_27396/output//dmesg.log ./rocky9-vg-autotest_27396/output//power.tar.xz ./rocky9-vg-autotest_27396/output//test_completions.txt ./rocky9-vg-autotest_27396/output//timing.svg ./rocky9-vg-autotest_27396/output//timing.txt ./rocky9-vg-autotest_27396/output//ut_coverage ./rocky9-vg-autotest_27396/output//..
00:30:08.596  + rmdir ./rocky9-vg-autotest_27396/output/
00:30:08.596  + for dir in "${out_dirs[@]}"
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./scan-build-docker-autotest_29456/output//doc.tar.xz ]]
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./scan-build-docker-autotest_29456/output//ut_coverage.tar.xz ]]
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./scan-build-docker-autotest_29456/output//llvm.tar.xz ]]
00:30:08.596  + mv ./scan-build-docker-autotest_29456/output//all_c_files.txt ./scan-build-docker-autotest_29456/output//build_output.txt ./scan-build-docker-autotest_29456/output//build-repo-manifest.txt ./scan-build-docker-autotest_29456/output//built_c_files.txt ./scan-build-docker-autotest_29456/output//power.tar.xz ./scan-build-docker-autotest_29456/output//test_completions.txt ./scan-build-docker-autotest_29456/output//timing.svg ./scan-build-docker-autotest_29456/output//timing.txt ./scan-build-docker-autotest_29456/output//unbuilt_c_files.txt ./scan-build-docker-autotest_29456/output//..
00:30:08.596  + rmdir ./scan-build-docker-autotest_29456/output/
00:30:08.596  + for dir in "${out_dirs[@]}"
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./short-fuzz-phy-autotest_37805/output//doc.tar.xz ]]
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./short-fuzz-phy-autotest_37805/output//ut_coverage.tar.xz ]]
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./short-fuzz-phy-autotest_37805/output//llvm.tar.xz ]]
00:30:08.596  + tar -C ./short-fuzz-phy-autotest_37805/output/ -xf ./short-fuzz-phy-autotest_37805/output//llvm.tar.xz
00:30:08.596  + rm ./short-fuzz-phy-autotest_37805/output//llvm.tar.xz
00:30:08.596  + mv ./short-fuzz-phy-autotest_37805/output//all_tests.txt ./short-fuzz-phy-autotest_37805/output//build-kernel-manifest.txt ./short-fuzz-phy-autotest_37805/output//build-pkg-manifest.txt ./short-fuzz-phy-autotest_37805/output//build-repo-manifest.txt ./short-fuzz-phy-autotest_37805/output//cov_base.info.xz ./short-fuzz-phy-autotest_37805/output//cov_test.info.xz ./short-fuzz-phy-autotest_37805/output//cov_total.info.xz ./short-fuzz-phy-autotest_37805/output//dmesg.init.log ./short-fuzz-phy-autotest_37805/output//dmesg.log ./short-fuzz-phy-autotest_37805/output//llvm ./short-fuzz-phy-autotest_37805/output//power.tar.xz ./short-fuzz-phy-autotest_37805/output//test_completions.txt ./short-fuzz-phy-autotest_37805/output//timing.svg ./short-fuzz-phy-autotest_37805/output//timing.txt ./short-fuzz-phy-autotest_37805/output//..
00:30:08.596  + rmdir ./short-fuzz-phy-autotest_37805/output/
00:30:08.596  + for dir in "${out_dirs[@]}"
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./ubuntu22-vg-autotest_49083/output//doc.tar.xz ]]
00:30:08.596  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.596  + [[ -f ./ubuntu22-vg-autotest_49083/output//ut_coverage.tar.xz ]]
00:30:08.596  + tar -C ./ubuntu22-vg-autotest_49083/output/ -xf ./ubuntu22-vg-autotest_49083/output//ut_coverage.tar.xz
00:30:08.853  + rm ./ubuntu22-vg-autotest_49083/output//ut_coverage.tar.xz
00:30:08.853  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.853  + [[ -f ./ubuntu22-vg-autotest_49083/output//llvm.tar.xz ]]
00:30:08.853  + mv ./ubuntu22-vg-autotest_49083/output//all_tests.txt ./ubuntu22-vg-autotest_49083/output//build-pkg-manifest.txt ./ubuntu22-vg-autotest_49083/output//build-repo-manifest.txt ./ubuntu22-vg-autotest_49083/output//cov_base.info.xz ./ubuntu22-vg-autotest_49083/output//cov_test.info.xz ./ubuntu22-vg-autotest_49083/output//cov_total.info.xz ./ubuntu22-vg-autotest_49083/output//dmesg.init.log ./ubuntu22-vg-autotest_49083/output//dmesg.log ./ubuntu22-vg-autotest_49083/output//power.tar.xz ./ubuntu22-vg-autotest_49083/output//test_completions.txt ./ubuntu22-vg-autotest_49083/output//timing.svg ./ubuntu22-vg-autotest_49083/output//timing.txt ./ubuntu22-vg-autotest_49083/output//ut_coverage ./ubuntu22-vg-autotest_49083/output//..
00:30:08.853  + rmdir ./ubuntu22-vg-autotest_49083/output/
00:30:08.853  + for dir in "${out_dirs[@]}"
00:30:08.853  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.853  + [[ -f ./ubuntu24-vg-autotest_6644/output//doc.tar.xz ]]
00:30:08.853  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:08.853  + [[ -f ./ubuntu24-vg-autotest_6644/output//ut_coverage.tar.xz ]]
00:30:08.853  + tar -C ./ubuntu24-vg-autotest_6644/output/ -xf ./ubuntu24-vg-autotest_6644/output//ut_coverage.tar.xz
00:30:09.111  + rm ./ubuntu24-vg-autotest_6644/output//ut_coverage.tar.xz
00:30:09.111  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.111  + [[ -f ./ubuntu24-vg-autotest_6644/output//llvm.tar.xz ]]
00:30:09.111  + mv ./ubuntu24-vg-autotest_6644/output//all_tests.txt ./ubuntu24-vg-autotest_6644/output//build-pkg-manifest.txt ./ubuntu24-vg-autotest_6644/output//build-repo-manifest.txt ./ubuntu24-vg-autotest_6644/output//cov_base.info.xz ./ubuntu24-vg-autotest_6644/output//cov_test.info.xz ./ubuntu24-vg-autotest_6644/output//cov_total.info.xz ./ubuntu24-vg-autotest_6644/output//dmesg.init.log ./ubuntu24-vg-autotest_6644/output//dmesg.log ./ubuntu24-vg-autotest_6644/output//power.tar.xz ./ubuntu24-vg-autotest_6644/output//test_completions.txt ./ubuntu24-vg-autotest_6644/output//timing.svg ./ubuntu24-vg-autotest_6644/output//timing.txt ./ubuntu24-vg-autotest_6644/output//ut_coverage ./ubuntu24-vg-autotest_6644/output//..
00:30:09.111  + rmdir ./ubuntu24-vg-autotest_6644/output/
00:30:09.111  + for dir in "${out_dirs[@]}"
00:30:09.111  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.111  + [[ -f ./unittest-clang-docker-autotest_29449/output//doc.tar.xz ]]
00:30:09.111  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.111  + [[ -f ./unittest-clang-docker-autotest_29449/output//ut_coverage.tar.xz ]]
00:30:09.111  + tar -C ./unittest-clang-docker-autotest_29449/output/ -xf ./unittest-clang-docker-autotest_29449/output//ut_coverage.tar.xz
00:30:09.675  + rm ./unittest-clang-docker-autotest_29449/output//ut_coverage.tar.xz
00:30:09.675  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.675  + [[ -f ./unittest-clang-docker-autotest_29449/output//llvm.tar.xz ]]
00:30:09.675  + mv ./unittest-clang-docker-autotest_29449/output//build-repo-manifest.txt ./unittest-clang-docker-autotest_29449/output//power.tar.xz ./unittest-clang-docker-autotest_29449/output//test_completions.txt ./unittest-clang-docker-autotest_29449/output//timing.svg ./unittest-clang-docker-autotest_29449/output//timing.txt ./unittest-clang-docker-autotest_29449/output//ut_coverage ./unittest-clang-docker-autotest_29449/output//..
00:30:09.675  + rmdir ./unittest-clang-docker-autotest_29449/output/
00:30:09.675  + for dir in "${out_dirs[@]}"
00:30:09.675  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.675  + [[ -f ./unittest-main-docker-autotest_29451/output//doc.tar.xz ]]
00:30:09.675  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.675  + [[ -f ./unittest-main-docker-autotest_29451/output//ut_coverage.tar.xz ]]
00:30:09.675  + tar -C ./unittest-main-docker-autotest_29451/output/ -xf ./unittest-main-docker-autotest_29451/output//ut_coverage.tar.xz
00:30:09.675  + rm ./unittest-main-docker-autotest_29451/output//ut_coverage.tar.xz
00:30:09.675  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.675  + [[ -f ./unittest-main-docker-autotest_29451/output//llvm.tar.xz ]]
00:30:09.675  + mv ./unittest-main-docker-autotest_29451/output//build-repo-manifest.txt ./unittest-main-docker-autotest_29451/output//power.tar.xz ./unittest-main-docker-autotest_29451/output//test_completions.txt ./unittest-main-docker-autotest_29451/output//timing.svg ./unittest-main-docker-autotest_29451/output//timing.txt ./unittest-main-docker-autotest_29451/output//ut_coverage ./unittest-main-docker-autotest_29451/output//..
00:30:09.675  + rmdir ./unittest-main-docker-autotest_29451/output/
00:30:09.675  + for dir in "${out_dirs[@]}"
00:30:09.675  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.675  + [[ -f ./unittest-valgrind-docker-autotest_29454/output//doc.tar.xz ]]
00:30:09.675  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.675  + [[ -f ./unittest-valgrind-docker-autotest_29454/output//ut_coverage.tar.xz ]]
00:30:09.675  + tar -C ./unittest-valgrind-docker-autotest_29454/output/ -xf ./unittest-valgrind-docker-autotest_29454/output//ut_coverage.tar.xz
00:30:09.952  + rm ./unittest-valgrind-docker-autotest_29454/output//ut_coverage.tar.xz
00:30:09.952  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.952  + [[ -f ./unittest-valgrind-docker-autotest_29454/output//llvm.tar.xz ]]
00:30:09.952  + mv ./unittest-valgrind-docker-autotest_29454/output//build-repo-manifest.txt ./unittest-valgrind-docker-autotest_29454/output//power.tar.xz ./unittest-valgrind-docker-autotest_29454/output//test_completions.txt ./unittest-valgrind-docker-autotest_29454/output//timing.svg ./unittest-valgrind-docker-autotest_29454/output//timing.txt ./unittest-valgrind-docker-autotest_29454/output//ut_coverage ./unittest-valgrind-docker-autotest_29454/output//..
00:30:09.952  + rmdir ./unittest-valgrind-docker-autotest_29454/output/
00:30:09.952  + for dir in "${out_dirs[@]}"
00:30:09.952  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vfio-user-phy-autotest_37810/output//doc.tar.xz ]]
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vfio-user-phy-autotest_37810/output//ut_coverage.tar.xz ]]
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vfio-user-phy-autotest_37810/output//llvm.tar.xz ]]
00:30:09.953  + mv ./vfio-user-phy-autotest_37810/output//all_tests.txt ./vfio-user-phy-autotest_37810/output//build-kernel-manifest.txt ./vfio-user-phy-autotest_37810/output//build-pkg-manifest.txt ./vfio-user-phy-autotest_37810/output//build-repo-manifest.txt ./vfio-user-phy-autotest_37810/output//cov_base.info.xz ./vfio-user-phy-autotest_37810/output//cov_test.info.xz ./vfio-user-phy-autotest_37810/output//cov_total.info.xz ./vfio-user-phy-autotest_37810/output//dmesg.init.log ./vfio-user-phy-autotest_37810/output//dmesg.log ./vfio-user-phy-autotest_37810/output//power.tar.xz ./vfio-user-phy-autotest_37810/output//test_completions.txt ./vfio-user-phy-autotest_37810/output//timing.svg ./vfio-user-phy-autotest_37810/output//timing.txt ./vfio-user-phy-autotest_37810/output//..
00:30:09.953  + rmdir ./vfio-user-phy-autotest_37810/output/
00:30:09.953  + for dir in "${out_dirs[@]}"
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vhost-initiator-vg-autotest_90154/output//doc.tar.xz ]]
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vhost-initiator-vg-autotest_90154/output//ut_coverage.tar.xz ]]
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vhost-initiator-vg-autotest_90154/output//llvm.tar.xz ]]
00:30:09.953  + mv ./vhost-initiator-vg-autotest_90154/output//all_tests.txt ./vhost-initiator-vg-autotest_90154/output//build-kernel-manifest.txt ./vhost-initiator-vg-autotest_90154/output//build-pkg-manifest.txt ./vhost-initiator-vg-autotest_90154/output//build-repo-manifest.txt ./vhost-initiator-vg-autotest_90154/output//cov_base.info.xz ./vhost-initiator-vg-autotest_90154/output//cov_test.info.xz ./vhost-initiator-vg-autotest_90154/output//cov_total.info.xz ./vhost-initiator-vg-autotest_90154/output//dmesg.init.log ./vhost-initiator-vg-autotest_90154/output//dmesg.log ./vhost-initiator-vg-autotest_90154/output//power.tar.xz ./vhost-initiator-vg-autotest_90154/output//test_completions.txt ./vhost-initiator-vg-autotest_90154/output//timing.svg ./vhost-initiator-vg-autotest_90154/output//timing.txt ./vhost-initiator-vg-autotest_90154/output//vhost_fuzz_output1.txt.xz ./vhost-initiator-vg-autotest_90154/output//vhost_fuzz_output2.txt.xz ./vhost-initiator-vg-autotest_90154/output//vhost_fuzz_tgt_output.txt.xz ./vhost-initiator-vg-autotest_90154/output//..
00:30:09.953  + rmdir ./vhost-initiator-vg-autotest_90154/output/
00:30:09.953  + for dir in "${out_dirs[@]}"
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vhost-phy-autotest_37874/output//doc.tar.xz ]]
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vhost-phy-autotest_37874/output//ut_coverage.tar.xz ]]
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./vhost-phy-autotest_37874/output//llvm.tar.xz ]]
00:30:09.953  + mv ./vhost-phy-autotest_37874/output//all_tests.txt ./vhost-phy-autotest_37874/output//build-kernel-manifest.txt ./vhost-phy-autotest_37874/output//build-pkg-manifest.txt ./vhost-phy-autotest_37874/output//build-repo-manifest.txt ./vhost-phy-autotest_37874/output//cov_base.info.xz ./vhost-phy-autotest_37874/output//cov_test.info.xz ./vhost-phy-autotest_37874/output//cov_total.info.xz ./vhost-phy-autotest_37874/output//dmesg.init.log ./vhost-phy-autotest_37874/output//dmesg.log ./vhost-phy-autotest_37874/output//power.tar.xz ./vhost-phy-autotest_37874/output//test_completions.txt ./vhost-phy-autotest_37874/output//timing.svg ./vhost-phy-autotest_37874/output//timing.txt ./vhost-phy-autotest_37874/output//..
00:30:09.953  + rmdir ./vhost-phy-autotest_37874/output/
00:30:09.953  + for dir in "${out_dirs[@]}"
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./zns-vg-autotest_89238/output//doc.tar.xz ]]
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./zns-vg-autotest_89238/output//ut_coverage.tar.xz ]]
00:30:09.953  + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz
00:30:09.953  + [[ -f ./zns-vg-autotest_89238/output//llvm.tar.xz ]]
00:30:09.953  + mv ./zns-vg-autotest_89238/output//all_tests.txt ./zns-vg-autotest_89238/output//build-kernel-manifest.txt ./zns-vg-autotest_89238/output//build-pkg-manifest.txt ./zns-vg-autotest_89238/output//build-repo-manifest.txt ./zns-vg-autotest_89238/output//cov_base.info.xz ./zns-vg-autotest_89238/output//cov_test.info.xz ./zns-vg-autotest_89238/output//cov_total.info.xz ./zns-vg-autotest_89238/output//dmesg.init.log ./zns-vg-autotest_89238/output//dmesg.log ./zns-vg-autotest_89238/output//power.tar.xz ./zns-vg-autotest_89238/output//test_completions.txt ./zns-vg-autotest_89238/output//timing.svg ./zns-vg-autotest_89238/output//timing.txt ./zns-vg-autotest_89238/output//..
00:30:09.953  + rmdir ./zns-vg-autotest_89238/output/
00:30:09.953  + unpack_cov_files
00:30:09.953  + local info_files
00:30:09.953  + info_files=(*/cov_*.info.xz)
00:30:09.953  + xargs -P0 -r -n1 xz -d
00:30:09.953  + printf '%s\n' BlobFS-phy-autotest_37820/cov_base.info.xz BlobFS-phy-autotest_37820/cov_test.info.xz BlobFS-phy-autotest_37820/cov_total.info.xz crypto-phy-autotest_37809/cov_base.info.xz crypto-phy-autotest_37809/cov_test.info.xz crypto-phy-autotest_37809/cov_total.info.xz freebsd-vg-autotest_68225/cov_base.info.xz freebsd-vg-autotest_68225/cov_test.info.xz freebsd-vg-autotest_68225/cov_total.info.xz iscsi-uring-vg-autotest_90037/cov_base.info.xz iscsi-uring-vg-autotest_90037/cov_test.info.xz iscsi-uring-vg-autotest_90037/cov_total.info.xz iscsi-vg-autotest_90173/cov_base.info.xz iscsi-vg-autotest_90173/cov_test.info.xz iscsi-vg-autotest_90173/cov_total.info.xz lvol-vg-autotest_90085/cov_base.info.xz lvol-vg-autotest_90085/cov_test.info.xz lvol-vg-autotest_90085/cov_total.info.xz nvme-cmb-pmr-vg-autotest_91264/cov_base.info.xz nvme-cmb-pmr-vg-autotest_91264/cov_test.info.xz nvme-cmb-pmr-vg-autotest_91264/cov_total.info.xz nvme-phy-autotest_83983/cov_base.info.xz nvme-phy-autotest_83983/cov_test.info.xz nvme-phy-autotest_83983/cov_total.info.xz nvme-vg-autotest_90136/cov_base.info.xz nvme-vg-autotest_90136/cov_test.info.xz nvme-vg-autotest_90136/cov_total.info.xz nvmf-phy-autotest_84052/cov_base.info.xz nvmf-tcp-phy-autotest_83934/cov_base.info.xz nvmf-tcp-uring-vg-autotest_90079/cov_base.info.xz nvmf-tcp-vg-autotest_90118/cov_base.info.xz raid-vg-autotest_4805/cov_base.info.xz raid-vg-autotest_4805/cov_test.info.xz raid-vg-autotest_4805/cov_total.info.xz rocky9-vg-autotest_27396/cov_base.info.xz rocky9-vg-autotest_27396/cov_test.info.xz rocky9-vg-autotest_27396/cov_total.info.xz short-fuzz-phy-autotest_37805/cov_base.info.xz short-fuzz-phy-autotest_37805/cov_test.info.xz short-fuzz-phy-autotest_37805/cov_total.info.xz ubuntu22-vg-autotest_49083/cov_base.info.xz ubuntu22-vg-autotest_49083/cov_test.info.xz ubuntu22-vg-autotest_49083/cov_total.info.xz ubuntu24-vg-autotest_6644/cov_base.info.xz ubuntu24-vg-autotest_6644/cov_test.info.xz ubuntu24-vg-autotest_6644/cov_total.info.xz vfio-user-phy-autotest_37810/cov_base.info.xz vfio-user-phy-autotest_37810/cov_test.info.xz vfio-user-phy-autotest_37810/cov_total.info.xz vhost-initiator-vg-autotest_90154/cov_base.info.xz vhost-initiator-vg-autotest_90154/cov_test.info.xz vhost-initiator-vg-autotest_90154/cov_total.info.xz vhost-phy-autotest_37874/cov_base.info.xz vhost-phy-autotest_37874/cov_test.info.xz vhost-phy-autotest_37874/cov_total.info.xz zns-vg-autotest_89238/cov_base.info.xz zns-vg-autotest_89238/cov_test.info.xz zns-vg-autotest_89238/cov_total.info.xz
00:30:10.543  + fix_downstream_job_paths
00:30:10.543  + sed -i -e 's#^SF:/.\+/spdk/#SF:/var/jenkins/workspace/autotest-per-patch/spdk/#g' BlobFS-phy-autotest_37820/cov_total.info crypto-phy-autotest_37809/cov_total.info freebsd-vg-autotest_68225/cov_total.info iscsi-uring-vg-autotest_90037/cov_total.info iscsi-vg-autotest_90173/cov_total.info lvol-vg-autotest_90085/cov_total.info nvme-cmb-pmr-vg-autotest_91264/cov_total.info nvme-phy-autotest_83983/cov_total.info nvme-vg-autotest_90136/cov_total.info raid-vg-autotest_4805/cov_total.info rocky9-vg-autotest_27396/cov_total.info short-fuzz-phy-autotest_37805/cov_total.info ubuntu22-vg-autotest_49083/cov_total.info ubuntu24-vg-autotest_6644/cov_total.info vfio-user-phy-autotest_37810/cov_total.info vhost-initiator-vg-autotest_90154/cov_total.info vhost-phy-autotest_37874/cov_total.info zns-vg-autotest_89238/cov_total.info
00:30:11.474  + ignore_ocf_headers
00:30:11.474  + sed -i 's#--show-details#--show-details --ignore-errors source#g' spdk/autorun_post.py
00:30:11.474  + create_errtrace_files_for_all_builds
00:30:11.474  + local build_log_list
00:30:11.474  + build_log_list=($(find "$WORKSPACE" -type f -name 'build.log'))
00:30:11.474  ++ find /var/jenkins/workspace/autotest-per-patch -type f -name build.log
00:30:11.736  + for build in "${build_log_list[@]}"
00:30:11.736  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/build.log
00:30:11.736  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.736  + build_log=/var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/build.log
00:30:11.736  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/build.log
00:30:11.736  ++ basename /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079
00:30:11.736  + job_name=nvmf-tcp-uring-vg-autotest_90079
00:30:11.736  + start_pattern='Backtrace start'
00:30:11.736  + end_pattern='Backtrace end'
00:30:11.736  + [[ -f /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/build.log ]]
00:30:11.736  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/build.log
00:30:11.736  + backtraces='00:16:17.287  ========== Backtrace start: ==========
00:30:11.736  00:16:17.287  
00:30:11.736  00:16:17.287  in /home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.736  00:16:17.287       ...
00:30:11.736  00:16:17.287     63  	cat $testdir/try.txt
00:30:11.736  00:16:17.287     64  	# if this test fails it means we didn'\''t fail over to the second
00:30:11.736  00:16:17.287     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.736  00:16:17.287     66  	
00:30:11.736  00:16:17.287     67  	if ((count != 3)); then
00:30:11.736  00:16:17.287  => 68  		false
00:30:11.736  00:16:17.287     69  	fi
00:30:11.736  00:16:17.287     70  	
00:30:11.736  00:16:17.287     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.736  00:16:17.287     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.737  00:16:17.287     73  	bdevperf_pid=$!
00:30:11.737  00:16:17.287       ...
00:30:11.737  00:16:17.287  
00:30:11.737  00:16:17.287  ========== Backtrace end ==========
00:30:11.737  00:16:17.465  ========== Backtrace start: ==========
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     1120		timing_enter $test_name
00:30:11.737  00:16:17.465     1121		echo "************************************"
00:30:11.737  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.737  00:16:17.465     1123		echo "************************************"
00:30:11.737  00:16:17.465     1124		xtrace_restore
00:30:11.737  00:16:17.465     1125		time "$@"
00:30:11.737  00:16:17.465     1126		xtrace_disable
00:30:11.737  00:16:17.465     1127		echo "************************************"
00:30:11.737  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.737  00:16:17.465     1129		echo "************************************"
00:30:11.737  00:16:17.465     1130		timing_exit $test_name
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     20  	fi
00:30:11.737  00:16:17.465     21  	
00:30:11.737  00:16:17.465     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  ========== Backtrace end ==========
00:30:11.737  00:16:17.465  ========== Backtrace start: ==========
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     1120		timing_enter $test_name
00:30:11.737  00:16:17.465     1121		echo "************************************"
00:30:11.737  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.737  00:16:17.465     1123		echo "************************************"
00:30:11.737  00:16:17.465     1124		xtrace_restore
00:30:11.737  00:16:17.465     1125		time "$@"
00:30:11.737  00:16:17.465     1126		xtrace_disable
00:30:11.737  00:16:17.465     1127		echo "************************************"
00:30:11.737  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.737  00:16:17.465     1129		echo "************************************"
00:30:11.737  00:16:17.465     1130		timing_exit $test_name
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     11  		exit 0
00:30:11.737  00:16:17.465     12  	fi
00:30:11.737  00:16:17.465     13  	
00:30:11.737  00:16:17.465     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.737  00:16:17.465     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.737  00:16:17.465  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.737  00:16:17.465     17  	
00:30:11.737  00:16:17.465     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.737  00:16:17.465     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.737  00:16:17.465     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.737  00:16:17.465     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  ========== Backtrace end ==========
00:30:11.737  00:16:17.465  ========== Backtrace start: ==========
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     1120		timing_enter $test_name
00:30:11.737  00:16:17.465     1121		echo "************************************"
00:30:11.737  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.737  00:16:17.465     1123		echo "************************************"
00:30:11.737  00:16:17.465     1124		xtrace_restore
00:30:11.737  00:16:17.465     1125		time "$@"
00:30:11.737  00:16:17.465     1126		xtrace_disable
00:30:11.737  00:16:17.465     1127		echo "************************************"
00:30:11.737  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.737  00:16:17.465     1129		echo "************************************"
00:30:11.737  00:16:17.465     1130		timing_exit $test_name
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/autotest.sh:280 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.737  00:16:17.465     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.737  00:16:17.465     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.737  00:16:17.465     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.737  00:16:17.465     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.737  00:16:17.465  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.737  00:16:17.465     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.737  00:16:17.465     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.737  00:16:17.465     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.737  00:16:17.465     284 				fi
00:30:11.737  00:16:17.465     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  ========== Backtrace end =========='
00:30:11.737  + [[ -n 00:16:17.287  ========== Backtrace start: ==========
00:30:11.737  00:16:17.287  
00:30:11.737  00:16:17.287  in /home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.737  00:16:17.287       ...
00:30:11.737  00:16:17.287     63  	cat $testdir/try.txt
00:30:11.737  00:16:17.287     64  	# if this test fails it means we didn't fail over to the second
00:30:11.737  00:16:17.287     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.737  00:16:17.287     66  	
00:30:11.737  00:16:17.287     67  	if ((count != 3)); then
00:30:11.737  00:16:17.287  => 68  		false
00:30:11.737  00:16:17.287     69  	fi
00:30:11.737  00:16:17.287     70  	
00:30:11.737  00:16:17.287     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.737  00:16:17.287     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.737  00:16:17.287     73  	bdevperf_pid=$!
00:30:11.737  00:16:17.287       ...
00:30:11.737  00:16:17.287  
00:30:11.737  00:16:17.287  ========== Backtrace end ==========
00:30:11.737  00:16:17.465  ========== Backtrace start: ==========
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     1120		timing_enter $test_name
00:30:11.737  00:16:17.465     1121		echo "************************************"
00:30:11.737  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.737  00:16:17.465     1123		echo "************************************"
00:30:11.737  00:16:17.465     1124		xtrace_restore
00:30:11.737  00:16:17.465     1125		time "$@"
00:30:11.737  00:16:17.465     1126		xtrace_disable
00:30:11.737  00:16:17.465     1127		echo "************************************"
00:30:11.737  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.737  00:16:17.465     1129		echo "************************************"
00:30:11.737  00:16:17.465     1130		timing_exit $test_name
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     20  	fi
00:30:11.737  00:16:17.465     21  	
00:30:11.737  00:16:17.465     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  ========== Backtrace end ==========
00:30:11.737  00:16:17.465  ========== Backtrace start: ==========
00:30:11.737  00:16:17.465  
00:30:11.737  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.737  00:16:17.465       ...
00:30:11.737  00:16:17.465     1120		timing_enter $test_name
00:30:11.737  00:16:17.465     1121		echo "************************************"
00:30:11.737  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.738  00:16:17.465     1123		echo "************************************"
00:30:11.738  00:16:17.465     1124		xtrace_restore
00:30:11.738  00:16:17.465     1125		time "$@"
00:30:11.738  00:16:17.465     1126		xtrace_disable
00:30:11.738  00:16:17.465     1127		echo "************************************"
00:30:11.738  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.738  00:16:17.465     1129		echo "************************************"
00:30:11.738  00:16:17.465     1130		timing_exit $test_name
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465     11  		exit 0
00:30:11.738  00:16:17.465     12  	fi
00:30:11.738  00:16:17.465     13  	
00:30:11.738  00:16:17.465     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465     17  	
00:30:11.738  00:16:17.465     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.738  00:16:17.465     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.738  00:16:17.465     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.738  00:16:17.465     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465  
00:30:11.738  00:16:17.465  ========== Backtrace end ==========
00:30:11.738  00:16:17.465  ========== Backtrace start: ==========
00:30:11.738  00:16:17.465  
00:30:11.738  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465     1120		timing_enter $test_name
00:30:11.738  00:16:17.465     1121		echo "************************************"
00:30:11.738  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.738  00:16:17.465     1123		echo "************************************"
00:30:11.738  00:16:17.465     1124		xtrace_restore
00:30:11.738  00:16:17.465     1125		time "$@"
00:30:11.738  00:16:17.465     1126		xtrace_disable
00:30:11.738  00:16:17.465     1127		echo "************************************"
00:30:11.738  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.738  00:16:17.465     1129		echo "************************************"
00:30:11.738  00:16:17.465     1130		timing_exit $test_name
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465  in /home/vagrant/spdk_repo/spdk/autotest.sh:280 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.738  00:16:17.465     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.738  00:16:17.465     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.738  00:16:17.465  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.738  00:16:17.465     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465     284 				fi
00:30:11.738  00:16:17.465     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465  
00:30:11.738  00:16:17.465  ========== Backtrace end ========== ]]
00:30:11.738  + echo '00:16:17.287  ========== Backtrace start: ==========
00:30:11.738  00:16:17.287  
00:30:11.738  00:16:17.287  in /home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.738  00:16:17.287       ...
00:30:11.738  00:16:17.287     63  	cat $testdir/try.txt
00:30:11.738  00:16:17.287     64  	# if this test fails it means we didn'\''t fail over to the second
00:30:11.738  00:16:17.287     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.738  00:16:17.287     66  	
00:30:11.738  00:16:17.287     67  	if ((count != 3)); then
00:30:11.738  00:16:17.287  => 68  		false
00:30:11.738  00:16:17.287     69  	fi
00:30:11.738  00:16:17.287     70  	
00:30:11.738  00:16:17.287     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.738  00:16:17.287     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.738  00:16:17.287     73  	bdevperf_pid=$!
00:30:11.738  00:16:17.287       ...
00:30:11.738  00:16:17.287  
00:30:11.738  00:16:17.287  ========== Backtrace end ==========
00:30:11.738  00:16:17.465  ========== Backtrace start: ==========
00:30:11.738  00:16:17.465  
00:30:11.738  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465     1120		timing_enter $test_name
00:30:11.738  00:16:17.465     1121		echo "************************************"
00:30:11.738  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.738  00:16:17.465     1123		echo "************************************"
00:30:11.738  00:16:17.465     1124		xtrace_restore
00:30:11.738  00:16:17.465     1125		time "$@"
00:30:11.738  00:16:17.465     1126		xtrace_disable
00:30:11.738  00:16:17.465     1127		echo "************************************"
00:30:11.738  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.738  00:16:17.465     1129		echo "************************************"
00:30:11.738  00:16:17.465     1130		timing_exit $test_name
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465     20  	fi
00:30:11.738  00:16:17.465     21  	
00:30:11.738  00:16:17.465     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465  
00:30:11.738  00:16:17.465  ========== Backtrace end ==========
00:30:11.738  00:16:17.465  ========== Backtrace start: ==========
00:30:11.738  00:16:17.465  
00:30:11.738  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465     1120		timing_enter $test_name
00:30:11.738  00:16:17.465     1121		echo "************************************"
00:30:11.738  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.738  00:16:17.465     1123		echo "************************************"
00:30:11.738  00:16:17.465     1124		xtrace_restore
00:30:11.738  00:16:17.465     1125		time "$@"
00:30:11.738  00:16:17.465     1126		xtrace_disable
00:30:11.738  00:16:17.465     1127		echo "************************************"
00:30:11.738  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.738  00:16:17.465     1129		echo "************************************"
00:30:11.738  00:16:17.465     1130		timing_exit $test_name
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.738  00:16:17.465       ...
00:30:11.738  00:16:17.465     11  		exit 0
00:30:11.738  00:16:17.465     12  	fi
00:30:11.738  00:16:17.465     13  	
00:30:11.738  00:16:17.465     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.738  00:16:17.465     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.739  00:16:17.465  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.739  00:16:17.465     17  	
00:30:11.739  00:16:17.465     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.739  00:16:17.465     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.739  00:16:17.465     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.739  00:16:17.465     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.739  00:16:17.465       ...
00:30:11.739  00:16:17.465  
00:30:11.739  00:16:17.465  ========== Backtrace end ==========
00:30:11.739  00:16:17.465  ========== Backtrace start: ==========
00:30:11.739  00:16:17.465  
00:30:11.739  00:16:17.465  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.739  00:16:17.465       ...
00:30:11.739  00:16:17.465     1120		timing_enter $test_name
00:30:11.739  00:16:17.465     1121		echo "************************************"
00:30:11.739  00:16:17.465     1122		echo "START TEST $test_name"
00:30:11.739  00:16:17.465     1123		echo "************************************"
00:30:11.739  00:16:17.465     1124		xtrace_restore
00:30:11.739  00:16:17.465     1125		time "$@"
00:30:11.739  00:16:17.465     1126		xtrace_disable
00:30:11.739  00:16:17.465     1127		echo "************************************"
00:30:11.739  00:16:17.465     1128		echo "END TEST $test_name"
00:30:11.739  00:16:17.465     1129		echo "************************************"
00:30:11.739  00:16:17.465     1130		timing_exit $test_name
00:30:11.739  00:16:17.465       ...
00:30:11.739  00:16:17.465  in /home/vagrant/spdk_repo/spdk/autotest.sh:280 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
00:30:11.739  00:16:17.465       ...
00:30:11.739  00:16:17.465     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.739  00:16:17.465     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.739  00:16:17.465     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.739  00:16:17.465     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.739  00:16:17.465     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.739  00:16:17.465  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.739  00:16:17.465     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.739  00:16:17.465     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.739  00:16:17.465     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.739  00:16:17.465     284 				fi
00:30:11.739  00:16:17.465     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.739  00:16:17.465       ...
00:30:11.739  00:16:17.465  
00:30:11.739  00:16:17.465  ========== Backtrace end =========='
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644
00:30:11.739  + job_name=ubuntu24-vg-autotest_6644
00:30:11.739  + start_pattern='Backtrace start'
00:30:11.739  + end_pattern='Backtrace end'
00:30:11.739  + [[ -f /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/build.log ]]
00:30:11.739  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/build.log
00:30:11.739  + backtraces=
00:30:11.739  + [[ -n '' ]]
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944
00:30:11.739  + job_name=doc-docker-autotest_32944
00:30:11.739  + start_pattern='Backtrace start'
00:30:11.739  + end_pattern='Backtrace end'
00:30:11.739  + [[ -f /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/build.log ]]
00:30:11.739  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/build.log
00:30:11.739  + backtraces=
00:30:11.739  + [[ -n '' ]]
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810
00:30:11.739  + job_name=vfio-user-phy-autotest_37810
00:30:11.739  + start_pattern='Backtrace start'
00:30:11.739  + end_pattern='Backtrace end'
00:30:11.739  + [[ -f /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/build.log ]]
00:30:11.739  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/build.log
00:30:11.739  + backtraces=
00:30:11.739  + [[ -n '' ]]
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820
00:30:11.739  + job_name=BlobFS-phy-autotest_37820
00:30:11.739  + start_pattern='Backtrace start'
00:30:11.739  + end_pattern='Backtrace end'
00:30:11.739  + [[ -f /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/build.log ]]
00:30:11.739  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/build.log
00:30:11.739  + backtraces=
00:30:11.739  + [[ -n '' ]]
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809
00:30:11.739  + job_name=crypto-phy-autotest_37809
00:30:11.739  + start_pattern='Backtrace start'
00:30:11.739  + end_pattern='Backtrace end'
00:30:11.739  + [[ -f /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/build.log ]]
00:30:11.739  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/build.log
00:30:11.739  + backtraces=
00:30:11.739  + [[ -n '' ]]
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085
00:30:11.739  + job_name=lvol-vg-autotest_90085
00:30:11.739  + start_pattern='Backtrace start'
00:30:11.739  + end_pattern='Backtrace end'
00:30:11.739  + [[ -f /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/build.log ]]
00:30:11.739  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/build.log
00:30:11.739  + backtraces=
00:30:11.739  + [[ -n '' ]]
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615
00:30:11.739  + job_name=build-files-docker-autotest_32615
00:30:11.739  + start_pattern='Backtrace start'
00:30:11.739  + end_pattern='Backtrace end'
00:30:11.739  + [[ -f /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/build.log ]]
00:30:11.739  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/build.log
00:30:11.739  + backtraces=
00:30:11.739  + [[ -n '' ]]
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154
00:30:11.739  + job_name=vhost-initiator-vg-autotest_90154
00:30:11.739  + start_pattern='Backtrace start'
00:30:11.739  + end_pattern='Backtrace end'
00:30:11.739  + [[ -f /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/build.log ]]
00:30:11.739  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/build.log
00:30:11.739  + backtraces=
00:30:11.739  + [[ -n '' ]]
00:30:11.739  + for build in "${build_log_list[@]}"
00:30:11.739  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/build.log
00:30:11.739  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.739  + build_log=/var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/build.log
00:30:11.739  +++ dirname /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/build.log
00:30:11.739  ++ basename /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173
00:30:11.740  + job_name=iscsi-vg-autotest_90173
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/build.log
00:30:11.740  + backtraces=
00:30:11.740  + [[ -n '' ]]
00:30:11.740  + for build in "${build_log_list[@]}"
00:30:11.740  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/build.log
00:30:11.740  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.740  + build_log=/var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/build.log
00:30:11.740  +++ dirname /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/build.log
00:30:11.740  ++ basename /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037
00:30:11.740  + job_name=iscsi-uring-vg-autotest_90037
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/build.log
00:30:11.740  + backtraces=
00:30:11.740  + [[ -n '' ]]
00:30:11.740  + for build in "${build_log_list[@]}"
00:30:11.740  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/build.log
00:30:11.740  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.740  + build_log=/var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/build.log
00:30:11.740  +++ dirname /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/build.log
00:30:11.740  ++ basename /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449
00:30:11.740  + job_name=packaging-docker-autotest_29449
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/build.log
00:30:11.740  + backtraces=
00:30:11.740  + [[ -n '' ]]
00:30:11.740  + for build in "${build_log_list[@]}"
00:30:11.740  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/build.log
00:30:11.740  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.740  + build_log=/var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/build.log
00:30:11.740  +++ dirname /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/build.log
00:30:11.740  ++ basename /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093
00:30:11.740  + job_name=autobuild-external-vg-autotest_30093
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/build.log
00:30:11.740  + backtraces=
00:30:11.740  + [[ -n '' ]]
00:30:11.740  + for build in "${build_log_list[@]}"
00:30:11.740  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/build.log
00:30:11.740  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.740  + build_log=/var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/build.log
00:30:11.740  +++ dirname /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/build.log
00:30:11.740  ++ basename /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805
00:30:11.740  + job_name=short-fuzz-phy-autotest_37805
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/build.log
00:30:11.740  + backtraces=
00:30:11.740  + [[ -n '' ]]
00:30:11.740  + for build in "${build_log_list[@]}"
00:30:11.740  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/build.log
00:30:11.740  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.740  + build_log=/var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/build.log
00:30:11.740  +++ dirname /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/build.log
00:30:11.740  ++ basename /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083
00:30:11.740  + job_name=ubuntu22-vg-autotest_49083
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/build.log
00:30:11.740  + backtraces=
00:30:11.740  + [[ -n '' ]]
00:30:11.740  + for build in "${build_log_list[@]}"
00:30:11.740  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/build.log
00:30:11.740  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.740  + build_log=/var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/build.log
00:30:11.740  +++ dirname /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/build.log
00:30:11.740  ++ basename /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641
00:30:11.740  + job_name=check-format-docker-autotest_32641
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/build.log
00:30:11.740  + backtraces=
00:30:11.740  + [[ -n '' ]]
00:30:11.740  + for build in "${build_log_list[@]}"
00:30:11.740  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/build.log
00:30:11.740  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.740  + build_log=/var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/build.log
00:30:11.740  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/build.log
00:30:11.740  ++ basename /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264
00:30:11.740  + job_name=nvme-cmb-pmr-vg-autotest_91264
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/build.log
00:30:11.740  + backtraces=
00:30:11.740  + [[ -n '' ]]
00:30:11.740  + for build in "${build_log_list[@]}"
00:30:11.740  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/build.log
00:30:11.740  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.740  + build_log=/var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/build.log
00:30:11.740  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/build.log
00:30:11.740  ++ basename /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934
00:30:11.740  + job_name=nvmf-tcp-phy-autotest_83934
00:30:11.740  + start_pattern='Backtrace start'
00:30:11.740  + end_pattern='Backtrace end'
00:30:11.740  + [[ -f /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/build.log ]]
00:30:11.740  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/build.log
00:30:11.740  + backtraces='00:22:35.825  ========== Backtrace start: ==========
00:30:11.740  00:22:35.825  
00:30:11.740  00:22:35.825  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.740  00:22:35.825       ...
00:30:11.740  00:22:35.825     63  	cat $testdir/try.txt
00:30:11.740  00:22:35.825     64  	# if this test fails it means we didn'\''t fail over to the second
00:30:11.740  00:22:35.825     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.740  00:22:35.825     66  	
00:30:11.740  00:22:35.825     67  	if ((count != 3)); then
00:30:11.740  00:22:35.825  => 68  		false
00:30:11.740  00:22:35.825     69  	fi
00:30:11.740  00:22:35.825     70  	
00:30:11.740  00:22:35.825     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.740  00:22:35.825     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.740  00:22:35.825     73  	bdevperf_pid=$!
00:30:11.740  00:22:35.825       ...
00:30:11.740  00:22:35.825  
00:30:11.740  00:22:35.825  ========== Backtrace end ==========
00:30:11.740  00:22:37.918  ========== Backtrace start: ==========
00:30:11.740  00:22:37.918  
00:30:11.740  00:22:37.918  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.740  00:22:37.918       ...
00:30:11.740  00:22:37.918     1120		timing_enter $test_name
00:30:11.740  00:22:37.918     1121		echo "************************************"
00:30:11.740  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.740  00:22:37.919     1123		echo "************************************"
00:30:11.740  00:22:37.919     1124		xtrace_restore
00:30:11.740  00:22:37.919     1125		time "$@"
00:30:11.740  00:22:37.919     1126		xtrace_disable
00:30:11.740  00:22:37.919     1127		echo "************************************"
00:30:11.740  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.740  00:22:37.919     1129		echo "************************************"
00:30:11.740  00:22:37.919     1130		timing_exit $test_name
00:30:11.740  00:22:37.919       ...
00:30:11.740  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.740  00:22:37.919       ...
00:30:11.740  00:22:37.919     20  	fi
00:30:11.740  00:22:37.919     21  	
00:30:11.741  00:22:37.919     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919  
00:30:11.741  00:22:37.919  ========== Backtrace end ==========
00:30:11.741  00:22:37.919  ========== Backtrace start: ==========
00:30:11.741  00:22:37.919  
00:30:11.741  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919     1120		timing_enter $test_name
00:30:11.741  00:22:37.919     1121		echo "************************************"
00:30:11.741  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.741  00:22:37.919     1123		echo "************************************"
00:30:11.741  00:22:37.919     1124		xtrace_restore
00:30:11.741  00:22:37.919     1125		time "$@"
00:30:11.741  00:22:37.919     1126		xtrace_disable
00:30:11.741  00:22:37.919     1127		echo "************************************"
00:30:11.741  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.741  00:22:37.919     1129		echo "************************************"
00:30:11.741  00:22:37.919     1130		timing_exit $test_name
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919     11  		exit 0
00:30:11.741  00:22:37.919     12  	fi
00:30:11.741  00:22:37.919     13  	
00:30:11.741  00:22:37.919     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.741  00:22:37.919     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.741  00:22:37.919  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.741  00:22:37.919     17  	
00:30:11.741  00:22:37.919     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.741  00:22:37.919     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.741  00:22:37.919     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.741  00:22:37.919     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919  
00:30:11.741  00:22:37.919  ========== Backtrace end ==========
00:30:11.741  00:22:37.919  ========== Backtrace start: ==========
00:30:11.741  00:22:37.919  
00:30:11.741  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919     1120		timing_enter $test_name
00:30:11.741  00:22:37.919     1121		echo "************************************"
00:30:11.741  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.741  00:22:37.919     1123		echo "************************************"
00:30:11.741  00:22:37.919     1124		xtrace_restore
00:30:11.741  00:22:37.919     1125		time "$@"
00:30:11.741  00:22:37.919     1126		xtrace_disable
00:30:11.741  00:22:37.919     1127		echo "************************************"
00:30:11.741  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.741  00:22:37.919     1129		echo "************************************"
00:30:11.741  00:22:37.919     1130		timing_exit $test_name
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/autotest.sh:280 -> main(["/var/jenkins/workspace/nvmf-tcp-phy-autotest/autorun-spdk.conf"])
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.741  00:22:37.919     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.741  00:22:37.919     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.741  00:22:37.919     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.741  00:22:37.919     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.741  00:22:37.919  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.741  00:22:37.919     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.741  00:22:37.919     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.741  00:22:37.919     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.741  00:22:37.919     284 				fi
00:30:11.741  00:22:37.919     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919  
00:30:11.741  00:22:37.919  ========== Backtrace end =========='
00:30:11.741  + [[ -n 00:22:35.825  ========== Backtrace start: ==========
00:30:11.741  00:22:35.825  
00:30:11.741  00:22:35.825  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.741  00:22:35.825       ...
00:30:11.741  00:22:35.825     63  	cat $testdir/try.txt
00:30:11.741  00:22:35.825     64  	# if this test fails it means we didn't fail over to the second
00:30:11.741  00:22:35.825     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.741  00:22:35.825     66  	
00:30:11.741  00:22:35.825     67  	if ((count != 3)); then
00:30:11.741  00:22:35.825  => 68  		false
00:30:11.741  00:22:35.825     69  	fi
00:30:11.741  00:22:35.825     70  	
00:30:11.741  00:22:35.825     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.741  00:22:35.825     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.741  00:22:35.825     73  	bdevperf_pid=$!
00:30:11.741  00:22:35.825       ...
00:30:11.741  00:22:35.825  
00:30:11.741  00:22:35.825  ========== Backtrace end ==========
00:30:11.741  00:22:37.918  ========== Backtrace start: ==========
00:30:11.741  00:22:37.918  
00:30:11.741  00:22:37.918  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.741  00:22:37.918       ...
00:30:11.741  00:22:37.918     1120		timing_enter $test_name
00:30:11.741  00:22:37.918     1121		echo "************************************"
00:30:11.741  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.741  00:22:37.919     1123		echo "************************************"
00:30:11.741  00:22:37.919     1124		xtrace_restore
00:30:11.741  00:22:37.919     1125		time "$@"
00:30:11.741  00:22:37.919     1126		xtrace_disable
00:30:11.741  00:22:37.919     1127		echo "************************************"
00:30:11.741  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.741  00:22:37.919     1129		echo "************************************"
00:30:11.741  00:22:37.919     1130		timing_exit $test_name
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919     20  	fi
00:30:11.741  00:22:37.919     21  	
00:30:11.741  00:22:37.919     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.741  00:22:37.919       ...
00:30:11.741  00:22:37.919  
00:30:11.741  00:22:37.919  ========== Backtrace end ==========
00:30:11.741  00:22:37.919  ========== Backtrace start: ==========
00:30:11.741  00:22:37.919  
00:30:11.742  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919     1120		timing_enter $test_name
00:30:11.742  00:22:37.919     1121		echo "************************************"
00:30:11.742  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.742  00:22:37.919     1123		echo "************************************"
00:30:11.742  00:22:37.919     1124		xtrace_restore
00:30:11.742  00:22:37.919     1125		time "$@"
00:30:11.742  00:22:37.919     1126		xtrace_disable
00:30:11.742  00:22:37.919     1127		echo "************************************"
00:30:11.742  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.742  00:22:37.919     1129		echo "************************************"
00:30:11.742  00:22:37.919     1130		timing_exit $test_name
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919     11  		exit 0
00:30:11.742  00:22:37.919     12  	fi
00:30:11.742  00:22:37.919     13  	
00:30:11.742  00:22:37.919     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     17  	
00:30:11.742  00:22:37.919     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.742  00:22:37.919     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.742  00:22:37.919     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.742  00:22:37.919     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919  
00:30:11.742  00:22:37.919  ========== Backtrace end ==========
00:30:11.742  00:22:37.919  ========== Backtrace start: ==========
00:30:11.742  00:22:37.919  
00:30:11.742  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919     1120		timing_enter $test_name
00:30:11.742  00:22:37.919     1121		echo "************************************"
00:30:11.742  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.742  00:22:37.919     1123		echo "************************************"
00:30:11.742  00:22:37.919     1124		xtrace_restore
00:30:11.742  00:22:37.919     1125		time "$@"
00:30:11.742  00:22:37.919     1126		xtrace_disable
00:30:11.742  00:22:37.919     1127		echo "************************************"
00:30:11.742  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.742  00:22:37.919     1129		echo "************************************"
00:30:11.742  00:22:37.919     1130		timing_exit $test_name
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/autotest.sh:280 -> main(["/var/jenkins/workspace/nvmf-tcp-phy-autotest/autorun-spdk.conf"])
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.742  00:22:37.919     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.742  00:22:37.919     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.742  00:22:37.919  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.742  00:22:37.919     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     284 				fi
00:30:11.742  00:22:37.919     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919  
00:30:11.742  00:22:37.919  ========== Backtrace end ========== ]]
00:30:11.742  + echo '00:22:35.825  ========== Backtrace start: ==========
00:30:11.742  00:22:35.825  
00:30:11.742  00:22:35.825  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.742  00:22:35.825       ...
00:30:11.742  00:22:35.825     63  	cat $testdir/try.txt
00:30:11.742  00:22:35.825     64  	# if this test fails it means we didn'\''t fail over to the second
00:30:11.742  00:22:35.825     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.742  00:22:35.825     66  	
00:30:11.742  00:22:35.825     67  	if ((count != 3)); then
00:30:11.742  00:22:35.825  => 68  		false
00:30:11.742  00:22:35.825     69  	fi
00:30:11.742  00:22:35.825     70  	
00:30:11.742  00:22:35.825     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.742  00:22:35.825     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.742  00:22:35.825     73  	bdevperf_pid=$!
00:30:11.742  00:22:35.825       ...
00:30:11.742  00:22:35.825  
00:30:11.742  00:22:35.825  ========== Backtrace end ==========
00:30:11.742  00:22:37.918  ========== Backtrace start: ==========
00:30:11.742  00:22:37.918  
00:30:11.742  00:22:37.918  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.742  00:22:37.918       ...
00:30:11.742  00:22:37.918     1120		timing_enter $test_name
00:30:11.742  00:22:37.918     1121		echo "************************************"
00:30:11.742  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.742  00:22:37.919     1123		echo "************************************"
00:30:11.742  00:22:37.919     1124		xtrace_restore
00:30:11.742  00:22:37.919     1125		time "$@"
00:30:11.742  00:22:37.919     1126		xtrace_disable
00:30:11.742  00:22:37.919     1127		echo "************************************"
00:30:11.742  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.742  00:22:37.919     1129		echo "************************************"
00:30:11.742  00:22:37.919     1130		timing_exit $test_name
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919     20  	fi
00:30:11.742  00:22:37.919     21  	
00:30:11.742  00:22:37.919     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919  
00:30:11.742  00:22:37.919  ========== Backtrace end ==========
00:30:11.742  00:22:37.919  ========== Backtrace start: ==========
00:30:11.742  00:22:37.919  
00:30:11.742  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919     1120		timing_enter $test_name
00:30:11.742  00:22:37.919     1121		echo "************************************"
00:30:11.742  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.742  00:22:37.919     1123		echo "************************************"
00:30:11.742  00:22:37.919     1124		xtrace_restore
00:30:11.742  00:22:37.919     1125		time "$@"
00:30:11.742  00:22:37.919     1126		xtrace_disable
00:30:11.742  00:22:37.919     1127		echo "************************************"
00:30:11.742  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.742  00:22:37.919     1129		echo "************************************"
00:30:11.742  00:22:37.919     1130		timing_exit $test_name
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919     11  		exit 0
00:30:11.742  00:22:37.919     12  	fi
00:30:11.742  00:22:37.919     13  	
00:30:11.742  00:22:37.919     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.742  00:22:37.919     17  	
00:30:11.742  00:22:37.919     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.742  00:22:37.919     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.742  00:22:37.919     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.742  00:22:37.919     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919  
00:30:11.742  00:22:37.919  ========== Backtrace end ==========
00:30:11.742  00:22:37.919  ========== Backtrace start: ==========
00:30:11.742  00:22:37.919  
00:30:11.742  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.742  00:22:37.919       ...
00:30:11.742  00:22:37.919     1120		timing_enter $test_name
00:30:11.742  00:22:37.919     1121		echo "************************************"
00:30:11.743  00:22:37.919     1122		echo "START TEST $test_name"
00:30:11.743  00:22:37.919     1123		echo "************************************"
00:30:11.743  00:22:37.919     1124		xtrace_restore
00:30:11.743  00:22:37.919     1125		time "$@"
00:30:11.743  00:22:37.919     1126		xtrace_disable
00:30:11.743  00:22:37.919     1127		echo "************************************"
00:30:11.743  00:22:37.919     1128		echo "END TEST $test_name"
00:30:11.743  00:22:37.919     1129		echo "************************************"
00:30:11.743  00:22:37.919     1130		timing_exit $test_name
00:30:11.743  00:22:37.919       ...
00:30:11.743  00:22:37.919  in /var/jenkins/workspace/nvmf-tcp-phy-autotest/spdk/autotest.sh:280 -> main(["/var/jenkins/workspace/nvmf-tcp-phy-autotest/autorun-spdk.conf"])
00:30:11.743  00:22:37.919       ...
00:30:11.743  00:22:37.919     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.743  00:22:37.919     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.743  00:22:37.919     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:22:37.919     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:22:37.919     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.743  00:22:37.919  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:22:37.919     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.743  00:22:37.919     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:22:37.919     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:22:37.919     284 				fi
00:30:11.743  00:22:37.919     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.743  00:22:37.919       ...
00:30:11.743  00:22:37.919  
00:30:11.743  00:22:37.919  ========== Backtrace end =========='
00:30:11.743  + for build in "${build_log_list[@]}"
00:30:11.743  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/build.log
00:30:11.743  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.743  + build_log=/var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/build.log
00:30:11.743  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/build.log
00:30:11.743  ++ basename /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118
00:30:11.743  + job_name=nvmf-tcp-vg-autotest_90118
00:30:11.743  + start_pattern='Backtrace start'
00:30:11.743  + end_pattern='Backtrace end'
00:30:11.743  + [[ -f /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/build.log ]]
00:30:11.743  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/build.log
00:30:11.743  + backtraces='00:18:35.031  ========== Backtrace start: ==========
00:30:11.743  00:18:35.031  
00:30:11.743  00:18:35.031  in /home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.743  00:18:35.031       ...
00:30:11.743  00:18:35.031     63  	cat $testdir/try.txt
00:30:11.743  00:18:35.031     64  	# if this test fails it means we didn'\''t fail over to the second
00:30:11.743  00:18:35.031     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.743  00:18:35.031     66  	
00:30:11.743  00:18:35.031     67  	if ((count != 3)); then
00:30:11.743  00:18:35.031  => 68  		false
00:30:11.743  00:18:35.031     69  	fi
00:30:11.743  00:18:35.031     70  	
00:30:11.743  00:18:35.031     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.743  00:18:35.031     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.743  00:18:35.031     73  	bdevperf_pid=$!
00:30:11.743  00:18:35.031       ...
00:30:11.743  00:18:35.031  
00:30:11.743  00:18:35.031  ========== Backtrace end ==========
00:30:11.743  00:18:35.788  ========== Backtrace start: ==========
00:30:11.743  00:18:35.788  
00:30:11.743  00:18:35.788  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.743  00:18:35.788       ...
00:30:11.743  00:18:35.788     1120		timing_enter $test_name
00:30:11.743  00:18:35.788     1121		echo "************************************"
00:30:11.743  00:18:35.788     1122		echo "START TEST $test_name"
00:30:11.743  00:18:35.788     1123		echo "************************************"
00:30:11.743  00:18:35.788     1124		xtrace_restore
00:30:11.743  00:18:35.788     1125		time "$@"
00:30:11.743  00:18:35.788     1126		xtrace_disable
00:30:11.743  00:18:35.788     1127		echo "************************************"
00:30:11.743  00:18:35.788     1128		echo "END TEST $test_name"
00:30:11.743  00:18:35.788     1129		echo "************************************"
00:30:11.743  00:18:35.788     1130		timing_exit $test_name
00:30:11.743  00:18:35.788       ...
00:30:11.743  00:18:35.788  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.743  00:18:35.788       ...
00:30:11.743  00:18:35.788     20  	fi
00:30:11.743  00:18:35.788     21  	
00:30:11.743  00:18:35.788     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.788     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.788     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.788  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.788     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.789     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.789     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.789     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.789     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789  
00:30:11.743  00:18:35.789  ========== Backtrace end ==========
00:30:11.743  00:18:35.789  ========== Backtrace start: ==========
00:30:11.743  00:18:35.789  
00:30:11.743  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789     1120		timing_enter $test_name
00:30:11.743  00:18:35.789     1121		echo "************************************"
00:30:11.743  00:18:35.789     1122		echo "START TEST $test_name"
00:30:11.743  00:18:35.789     1123		echo "************************************"
00:30:11.743  00:18:35.789     1124		xtrace_restore
00:30:11.743  00:18:35.789     1125		time "$@"
00:30:11.743  00:18:35.789     1126		xtrace_disable
00:30:11.743  00:18:35.789     1127		echo "************************************"
00:30:11.743  00:18:35.789     1128		echo "END TEST $test_name"
00:30:11.743  00:18:35.789     1129		echo "************************************"
00:30:11.743  00:18:35.789     1130		timing_exit $test_name
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789     11  		exit 0
00:30:11.743  00:18:35.789     12  	fi
00:30:11.743  00:18:35.789     13  	
00:30:11.743  00:18:35.789     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:18:35.789     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:18:35.789  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:18:35.789     17  	
00:30:11.743  00:18:35.789     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.743  00:18:35.789     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.743  00:18:35.789     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.743  00:18:35.789     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789  
00:30:11.743  00:18:35.789  ========== Backtrace end ==========
00:30:11.743  00:18:35.789  ========== Backtrace start: ==========
00:30:11.743  00:18:35.789  
00:30:11.743  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789     1120		timing_enter $test_name
00:30:11.743  00:18:35.789     1121		echo "************************************"
00:30:11.743  00:18:35.789     1122		echo "START TEST $test_name"
00:30:11.743  00:18:35.789     1123		echo "************************************"
00:30:11.743  00:18:35.789     1124		xtrace_restore
00:30:11.743  00:18:35.789     1125		time "$@"
00:30:11.743  00:18:35.789     1126		xtrace_disable
00:30:11.743  00:18:35.789     1127		echo "************************************"
00:30:11.743  00:18:35.789     1128		echo "END TEST $test_name"
00:30:11.743  00:18:35.789     1129		echo "************************************"
00:30:11.743  00:18:35.789     1130		timing_exit $test_name
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789  in /home/vagrant/spdk_repo/spdk/autotest.sh:280 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.743  00:18:35.789     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.743  00:18:35.789     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:18:35.789     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:18:35.789     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.743  00:18:35.789  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:18:35.789     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.743  00:18:35.789     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:18:35.789     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.743  00:18:35.789     284 				fi
00:30:11.743  00:18:35.789     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.743  00:18:35.789       ...
00:30:11.743  00:18:35.789  
00:30:11.743  00:18:35.789  ========== Backtrace end =========='
00:30:11.743  + [[ -n 00:18:35.031  ========== Backtrace start: ==========
00:30:11.743  00:18:35.031  
00:30:11.743  00:18:35.031  in /home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.743  00:18:35.031       ...
00:30:11.744  00:18:35.031     63  	cat $testdir/try.txt
00:30:11.744  00:18:35.031     64  	# if this test fails it means we didn't fail over to the second
00:30:11.744  00:18:35.031     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.744  00:18:35.031     66  	
00:30:11.744  00:18:35.031     67  	if ((count != 3)); then
00:30:11.744  00:18:35.031  => 68  		false
00:30:11.744  00:18:35.031     69  	fi
00:30:11.744  00:18:35.031     70  	
00:30:11.744  00:18:35.031     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.744  00:18:35.031     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.744  00:18:35.031     73  	bdevperf_pid=$!
00:30:11.744  00:18:35.031       ...
00:30:11.744  00:18:35.031  
00:30:11.744  00:18:35.031  ========== Backtrace end ==========
00:30:11.744  00:18:35.788  ========== Backtrace start: ==========
00:30:11.744  00:18:35.788  
00:30:11.744  00:18:35.788  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.744  00:18:35.788       ...
00:30:11.744  00:18:35.788     1120		timing_enter $test_name
00:30:11.744  00:18:35.788     1121		echo "************************************"
00:30:11.744  00:18:35.788     1122		echo "START TEST $test_name"
00:30:11.744  00:18:35.788     1123		echo "************************************"
00:30:11.744  00:18:35.788     1124		xtrace_restore
00:30:11.744  00:18:35.788     1125		time "$@"
00:30:11.744  00:18:35.788     1126		xtrace_disable
00:30:11.744  00:18:35.788     1127		echo "************************************"
00:30:11.744  00:18:35.788     1128		echo "END TEST $test_name"
00:30:11.744  00:18:35.788     1129		echo "************************************"
00:30:11.744  00:18:35.788     1130		timing_exit $test_name
00:30:11.744  00:18:35.788       ...
00:30:11.744  00:18:35.788  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.744  00:18:35.788       ...
00:30:11.744  00:18:35.788     20  	fi
00:30:11.744  00:18:35.788     21  	
00:30:11.744  00:18:35.788     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.788     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.788     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.788  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.788     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789  
00:30:11.744  00:18:35.789  ========== Backtrace end ==========
00:30:11.744  00:18:35.789  ========== Backtrace start: ==========
00:30:11.744  00:18:35.789  
00:30:11.744  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789     1120		timing_enter $test_name
00:30:11.744  00:18:35.789     1121		echo "************************************"
00:30:11.744  00:18:35.789     1122		echo "START TEST $test_name"
00:30:11.744  00:18:35.789     1123		echo "************************************"
00:30:11.744  00:18:35.789     1124		xtrace_restore
00:30:11.744  00:18:35.789     1125		time "$@"
00:30:11.744  00:18:35.789     1126		xtrace_disable
00:30:11.744  00:18:35.789     1127		echo "************************************"
00:30:11.744  00:18:35.789     1128		echo "END TEST $test_name"
00:30:11.744  00:18:35.789     1129		echo "************************************"
00:30:11.744  00:18:35.789     1130		timing_exit $test_name
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789     11  		exit 0
00:30:11.744  00:18:35.789     12  	fi
00:30:11.744  00:18:35.789     13  	
00:30:11.744  00:18:35.789     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.744  00:18:35.789     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.744  00:18:35.789  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.744  00:18:35.789     17  	
00:30:11.744  00:18:35.789     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.744  00:18:35.789     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.744  00:18:35.789     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.744  00:18:35.789     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789  
00:30:11.744  00:18:35.789  ========== Backtrace end ==========
00:30:11.744  00:18:35.789  ========== Backtrace start: ==========
00:30:11.744  00:18:35.789  
00:30:11.744  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789     1120		timing_enter $test_name
00:30:11.744  00:18:35.789     1121		echo "************************************"
00:30:11.744  00:18:35.789     1122		echo "START TEST $test_name"
00:30:11.744  00:18:35.789     1123		echo "************************************"
00:30:11.744  00:18:35.789     1124		xtrace_restore
00:30:11.744  00:18:35.789     1125		time "$@"
00:30:11.744  00:18:35.789     1126		xtrace_disable
00:30:11.744  00:18:35.789     1127		echo "************************************"
00:30:11.744  00:18:35.789     1128		echo "END TEST $test_name"
00:30:11.744  00:18:35.789     1129		echo "************************************"
00:30:11.744  00:18:35.789     1130		timing_exit $test_name
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789  in /home/vagrant/spdk_repo/spdk/autotest.sh:280 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.744  00:18:35.789     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.744  00:18:35.789     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.744  00:18:35.789     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.744  00:18:35.789     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.744  00:18:35.789  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.744  00:18:35.789     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.744  00:18:35.789     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.744  00:18:35.789     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.744  00:18:35.789     284 				fi
00:30:11.744  00:18:35.789     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.744  00:18:35.789       ...
00:30:11.744  00:18:35.789  
00:30:11.744  00:18:35.789  ========== Backtrace end ========== ]]
00:30:11.744  + echo '00:18:35.031  ========== Backtrace start: ==========
00:30:11.744  00:18:35.031  
00:30:11.744  00:18:35.031  in /home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=tcp"])
00:30:11.744  00:18:35.031       ...
00:30:11.744  00:18:35.031     63  	cat $testdir/try.txt
00:30:11.744  00:18:35.031     64  	# if this test fails it means we didn'\''t fail over to the second
00:30:11.744  00:18:35.031     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.744  00:18:35.031     66  	
00:30:11.744  00:18:35.031     67  	if ((count != 3)); then
00:30:11.744  00:18:35.031  => 68  		false
00:30:11.744  00:18:35.031     69  	fi
00:30:11.744  00:18:35.031     70  	
00:30:11.744  00:18:35.031     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.744  00:18:35.031     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.744  00:18:35.031     73  	bdevperf_pid=$!
00:30:11.744  00:18:35.031       ...
00:30:11.744  00:18:35.031  
00:30:11.744  00:18:35.031  ========== Backtrace end ==========
00:30:11.744  00:18:35.788  ========== Backtrace start: ==========
00:30:11.744  00:18:35.788  
00:30:11.744  00:18:35.788  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/home/vagrant/spdk_repo/spdk/test/nvmf/host/failover.sh"],["--transport=tcp"])
00:30:11.744  00:18:35.788       ...
00:30:11.744  00:18:35.788     1120		timing_enter $test_name
00:30:11.744  00:18:35.788     1121		echo "************************************"
00:30:11.744  00:18:35.788     1122		echo "START TEST $test_name"
00:30:11.744  00:18:35.788     1123		echo "************************************"
00:30:11.744  00:18:35.788     1124		xtrace_restore
00:30:11.744  00:18:35.788     1125		time "$@"
00:30:11.744  00:18:35.788     1126		xtrace_disable
00:30:11.744  00:18:35.788     1127		echo "************************************"
00:30:11.744  00:18:35.788     1128		echo "END TEST $test_name"
00:30:11.744  00:18:35.788     1129		echo "************************************"
00:30:11.744  00:18:35.788     1130		timing_exit $test_name
00:30:11.744  00:18:35.788       ...
00:30:11.744  00:18:35.788  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=tcp"])
00:30:11.744  00:18:35.788       ...
00:30:11.744  00:18:35.788     20  	fi
00:30:11.744  00:18:35.788     21  	
00:30:11.744  00:18:35.788     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.788     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.788     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.788  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.788     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.744  00:18:35.789       ...
00:30:11.745  00:18:35.789  
00:30:11.745  00:18:35.789  ========== Backtrace end ==========
00:30:11.745  00:18:35.789  ========== Backtrace start: ==========
00:30:11.745  00:18:35.789  
00:30:11.745  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf_host.sh"],["--transport=tcp"])
00:30:11.745  00:18:35.789       ...
00:30:11.745  00:18:35.789     1120		timing_enter $test_name
00:30:11.745  00:18:35.789     1121		echo "************************************"
00:30:11.745  00:18:35.789     1122		echo "START TEST $test_name"
00:30:11.745  00:18:35.789     1123		echo "************************************"
00:30:11.745  00:18:35.789     1124		xtrace_restore
00:30:11.745  00:18:35.789     1125		time "$@"
00:30:11.745  00:18:35.789     1126		xtrace_disable
00:30:11.745  00:18:35.789     1127		echo "************************************"
00:30:11.745  00:18:35.789     1128		echo "END TEST $test_name"
00:30:11.745  00:18:35.789     1129		echo "************************************"
00:30:11.745  00:18:35.789     1130		timing_exit $test_name
00:30:11.745  00:18:35.789       ...
00:30:11.745  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=tcp"])
00:30:11.745  00:18:35.789       ...
00:30:11.745  00:18:35.789     11  		exit 0
00:30:11.745  00:18:35.789     12  	fi
00:30:11.745  00:18:35.789     13  	
00:30:11.745  00:18:35.789     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.745  00:18:35.789     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.745  00:18:35.789  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.745  00:18:35.789     17  	
00:30:11.745  00:18:35.789     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.745  00:18:35.789     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.745  00:18:35.789     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.745  00:18:35.789     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.745  00:18:35.789       ...
00:30:11.745  00:18:35.789  
00:30:11.745  00:18:35.789  ========== Backtrace end ==========
00:30:11.745  00:18:35.789  ========== Backtrace start: ==========
00:30:11.745  00:18:35.789  
00:30:11.745  00:18:35.789  in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_tcp"],["/home/vagrant/spdk_repo/spdk/test/nvmf/nvmf.sh"],["--transport=tcp"])
00:30:11.745  00:18:35.789       ...
00:30:11.745  00:18:35.789     1120		timing_enter $test_name
00:30:11.745  00:18:35.789     1121		echo "************************************"
00:30:11.745  00:18:35.789     1122		echo "START TEST $test_name"
00:30:11.745  00:18:35.789     1123		echo "************************************"
00:30:11.745  00:18:35.789     1124		xtrace_restore
00:30:11.745  00:18:35.789     1125		time "$@"
00:30:11.745  00:18:35.789     1126		xtrace_disable
00:30:11.745  00:18:35.789     1127		echo "************************************"
00:30:11.745  00:18:35.789     1128		echo "END TEST $test_name"
00:30:11.745  00:18:35.789     1129		echo "************************************"
00:30:11.745  00:18:35.789     1130		timing_exit $test_name
00:30:11.745  00:18:35.789       ...
00:30:11.745  00:18:35.789  in /home/vagrant/spdk_repo/spdk/autotest.sh:280 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"])
00:30:11.745  00:18:35.789       ...
00:30:11.745  00:18:35.789     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.745  00:18:35.789     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.745  00:18:35.789     277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.745  00:18:35.789     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.745  00:18:35.789     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.745  00:18:35.789  => 280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.745  00:18:35.789     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.745  00:18:35.789     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.745  00:18:35.789     283 					run_test "nvmf_identify_passthru" $rootdir/test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.745  00:18:35.789     284 				fi
00:30:11.745  00:18:35.789     285 				run_test "nvmf_dif" $rootdir/test/nvmf/target/dif.sh
00:30:11.745  00:18:35.789       ...
00:30:11.745  00:18:35.789  
00:30:11.745  00:18:35.789  ========== Backtrace end =========='
00:30:11.745  + for build in "${build_log_list[@]}"
00:30:11.745  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/build.log
00:30:11.745  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.745  + build_log=/var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/build.log
00:30:11.745  +++ dirname /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/build.log
00:30:11.745  ++ basename /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238
00:30:11.745  + job_name=zns-vg-autotest_89238
00:30:11.745  + start_pattern='Backtrace start'
00:30:11.745  + end_pattern='Backtrace end'
00:30:11.745  + [[ -f /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/build.log ]]
00:30:11.745  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/build.log
00:30:11.745  + backtraces=
00:30:11.745  + [[ -n '' ]]
00:30:11.745  + for build in "${build_log_list[@]}"
00:30:11.745  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/build.log
00:30:11.745  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.745  + build_log=/var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/build.log
00:30:11.745  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/build.log
00:30:11.745  ++ basename /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136
00:30:11.745  + job_name=nvme-vg-autotest_90136
00:30:11.745  + start_pattern='Backtrace start'
00:30:11.745  + end_pattern='Backtrace end'
00:30:11.745  + [[ -f /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/build.log ]]
00:30:11.745  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/build.log
00:30:11.745  + backtraces=
00:30:11.745  + [[ -n '' ]]
00:30:11.745  + for build in "${build_log_list[@]}"
00:30:11.745  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/build.log
00:30:11.745  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.745  + build_log=/var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/build.log
00:30:11.745  +++ dirname /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/build.log
00:30:11.745  ++ basename /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874
00:30:11.745  + job_name=vhost-phy-autotest_37874
00:30:11.745  + start_pattern='Backtrace start'
00:30:11.745  + end_pattern='Backtrace end'
00:30:11.745  + [[ -f /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/build.log ]]
00:30:11.745  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/build.log
00:30:11.745  + backtraces=
00:30:11.745  + [[ -n '' ]]
00:30:11.745  + for build in "${build_log_list[@]}"
00:30:11.745  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/build.log
00:30:11.745  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.745  + build_log=/var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/build.log
00:30:11.745  +++ dirname /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/build.log
00:30:11.745  ++ basename /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449
00:30:11.745  + job_name=unittest-clang-docker-autotest_29449
00:30:11.745  + start_pattern='Backtrace start'
00:30:11.745  + end_pattern='Backtrace end'
00:30:11.745  + [[ -f /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/build.log ]]
00:30:11.745  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/build.log
00:30:11.745  + backtraces=
00:30:11.745  + [[ -n '' ]]
00:30:11.745  + for build in "${build_log_list[@]}"
00:30:11.745  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/build.log
00:30:11.745  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.745  + build_log=/var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/build.log
00:30:11.745  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/build.log
00:30:11.745  ++ basename /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052
00:30:11.745  + job_name=nvmf-phy-autotest_84052
00:30:11.745  + start_pattern='Backtrace start'
00:30:11.745  + end_pattern='Backtrace end'
00:30:11.745  + [[ -f /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/build.log ]]
00:30:11.745  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/build.log
00:30:11.745  + backtraces='00:18:35.066  ========== Backtrace start: ==========
00:30:11.745  00:18:35.066  
00:30:11.745  00:18:35.066  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=rdma"])
00:30:11.745  00:18:35.066       ...
00:30:11.745  00:18:35.066     63  	cat $testdir/try.txt
00:30:11.745  00:18:35.066     64  	# if this test fails it means we didn'\''t fail over to the second
00:30:11.745  00:18:35.066     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.745  00:18:35.066     66  	
00:30:11.745  00:18:35.066     67  	if ((count != 3)); then
00:30:11.745  00:18:35.066  => 68  		false
00:30:11.745  00:18:35.066     69  	fi
00:30:11.745  00:18:35.066     70  	
00:30:11.745  00:18:35.066     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.745  00:18:35.066     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.745  00:18:35.066     73  	bdevperf_pid=$!
00:30:11.745  00:18:35.066       ...
00:30:11.745  00:18:35.066  
00:30:11.745  00:18:35.066  ========== Backtrace end ==========
00:30:11.745  00:18:35.598  ========== Backtrace start: ==========
00:30:11.745  00:18:35.598  
00:30:11.745  00:18:35.598  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/host/failover.sh"],["--transport=rdma"])
00:30:11.745  00:18:35.598       ...
00:30:11.745  00:18:35.598     1120		timing_enter $test_name
00:30:11.745  00:18:35.598     1121		echo "************************************"
00:30:11.745  00:18:35.598     1122		echo "START TEST $test_name"
00:30:11.745  00:18:35.598     1123		echo "************************************"
00:30:11.745  00:18:35.598     1124		xtrace_restore
00:30:11.745  00:18:35.598     1125		time "$@"
00:30:11.745  00:18:35.598     1126		xtrace_disable
00:30:11.745  00:18:35.598     1127		echo "************************************"
00:30:11.746  00:18:35.598     1128		echo "END TEST $test_name"
00:30:11.746  00:18:35.598     1129		echo "************************************"
00:30:11.746  00:18:35.599     1130		timing_exit $test_name
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=rdma"])
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599     20  	fi
00:30:11.746  00:18:35.599     21  	
00:30:11.746  00:18:35.599     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599  
00:30:11.746  00:18:35.599  ========== Backtrace end ==========
00:30:11.746  00:18:35.599  ========== Backtrace start: ==========
00:30:11.746  00:18:35.599  
00:30:11.746  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf_host.sh"],["--transport=rdma"])
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599     1120		timing_enter $test_name
00:30:11.746  00:18:35.599     1121		echo "************************************"
00:30:11.746  00:18:35.599     1122		echo "START TEST $test_name"
00:30:11.746  00:18:35.599     1123		echo "************************************"
00:30:11.746  00:18:35.599     1124		xtrace_restore
00:30:11.746  00:18:35.599     1125		time "$@"
00:30:11.746  00:18:35.599     1126		xtrace_disable
00:30:11.746  00:18:35.599     1127		echo "************************************"
00:30:11.746  00:18:35.599     1128		echo "END TEST $test_name"
00:30:11.746  00:18:35.599     1129		echo "************************************"
00:30:11.746  00:18:35.599     1130		timing_exit $test_name
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=rdma"])
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599     11  		exit 0
00:30:11.746  00:18:35.599     12  	fi
00:30:11.746  00:18:35.599     13  	
00:30:11.746  00:18:35.599     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.746  00:18:35.599     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.746  00:18:35.599  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.746  00:18:35.599     17  	
00:30:11.746  00:18:35.599     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.746  00:18:35.599     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.746  00:18:35.599     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.746  00:18:35.599     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599  
00:30:11.746  00:18:35.599  ========== Backtrace end ==========
00:30:11.746  00:18:35.599  ========== Backtrace start: ==========
00:30:11.746  00:18:35.599  
00:30:11.746  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_rdma"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf.sh"],["--transport=rdma"])
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599     1120		timing_enter $test_name
00:30:11.746  00:18:35.599     1121		echo "************************************"
00:30:11.746  00:18:35.599     1122		echo "START TEST $test_name"
00:30:11.746  00:18:35.599     1123		echo "************************************"
00:30:11.746  00:18:35.599     1124		xtrace_restore
00:30:11.746  00:18:35.599     1125		time "$@"
00:30:11.746  00:18:35.599     1126		xtrace_disable
00:30:11.746  00:18:35.599     1127		echo "************************************"
00:30:11.746  00:18:35.599     1128		echo "END TEST $test_name"
00:30:11.746  00:18:35.599     1129		echo "************************************"
00:30:11.746  00:18:35.599     1130		timing_exit $test_name
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/autotest.sh:277 -> main(["/var/jenkins/workspace/nvmf-phy-autotest/autorun-spdk.conf"])
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599     272 		if [ $SPDK_TEST_NVMF -eq 1 ]; then
00:30:11.746  00:18:35.599     273 			export NET_TYPE
00:30:11.746  00:18:35.599     274 			# The NVMe-oF run test cases are split out like this so that the parser that compiles the
00:30:11.746  00:18:35.599     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.746  00:18:35.599     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.746  00:18:35.599  => 277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.746  00:18:35.599     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.746  00:18:35.599     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.746  00:18:35.599     280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.746  00:18:35.599     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.746  00:18:35.599     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599  
00:30:11.746  00:18:35.599  ========== Backtrace end =========='
00:30:11.746  + [[ -n 00:18:35.066  ========== Backtrace start: ==========
00:30:11.746  00:18:35.066  
00:30:11.746  00:18:35.066  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=rdma"])
00:30:11.746  00:18:35.066       ...
00:30:11.746  00:18:35.066     63  	cat $testdir/try.txt
00:30:11.746  00:18:35.066     64  	# if this test fails it means we didn't fail over to the second
00:30:11.746  00:18:35.066     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.746  00:18:35.066     66  	
00:30:11.746  00:18:35.066     67  	if ((count != 3)); then
00:30:11.746  00:18:35.066  => 68  		false
00:30:11.746  00:18:35.066     69  	fi
00:30:11.746  00:18:35.066     70  	
00:30:11.746  00:18:35.066     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.746  00:18:35.066     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.746  00:18:35.066     73  	bdevperf_pid=$!
00:30:11.746  00:18:35.066       ...
00:30:11.746  00:18:35.066  
00:30:11.746  00:18:35.066  ========== Backtrace end ==========
00:30:11.746  00:18:35.598  ========== Backtrace start: ==========
00:30:11.746  00:18:35.598  
00:30:11.746  00:18:35.598  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/host/failover.sh"],["--transport=rdma"])
00:30:11.746  00:18:35.598       ...
00:30:11.746  00:18:35.598     1120		timing_enter $test_name
00:30:11.746  00:18:35.598     1121		echo "************************************"
00:30:11.746  00:18:35.598     1122		echo "START TEST $test_name"
00:30:11.746  00:18:35.598     1123		echo "************************************"
00:30:11.746  00:18:35.598     1124		xtrace_restore
00:30:11.746  00:18:35.598     1125		time "$@"
00:30:11.746  00:18:35.598     1126		xtrace_disable
00:30:11.746  00:18:35.598     1127		echo "************************************"
00:30:11.746  00:18:35.598     1128		echo "END TEST $test_name"
00:30:11.746  00:18:35.598     1129		echo "************************************"
00:30:11.746  00:18:35.599     1130		timing_exit $test_name
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=rdma"])
00:30:11.746  00:18:35.599       ...
00:30:11.746  00:18:35.599     20  	fi
00:30:11.746  00:18:35.599     21  	
00:30:11.746  00:18:35.599     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.746  00:18:35.599     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599  
00:30:11.747  00:18:35.599  ========== Backtrace end ==========
00:30:11.747  00:18:35.599  ========== Backtrace start: ==========
00:30:11.747  00:18:35.599  
00:30:11.747  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf_host.sh"],["--transport=rdma"])
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599     1120		timing_enter $test_name
00:30:11.747  00:18:35.599     1121		echo "************************************"
00:30:11.747  00:18:35.599     1122		echo "START TEST $test_name"
00:30:11.747  00:18:35.599     1123		echo "************************************"
00:30:11.747  00:18:35.599     1124		xtrace_restore
00:30:11.747  00:18:35.599     1125		time "$@"
00:30:11.747  00:18:35.599     1126		xtrace_disable
00:30:11.747  00:18:35.599     1127		echo "************************************"
00:30:11.747  00:18:35.599     1128		echo "END TEST $test_name"
00:30:11.747  00:18:35.599     1129		echo "************************************"
00:30:11.747  00:18:35.599     1130		timing_exit $test_name
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=rdma"])
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599     11  		exit 0
00:30:11.747  00:18:35.599     12  	fi
00:30:11.747  00:18:35.599     13  	
00:30:11.747  00:18:35.599     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599     17  	
00:30:11.747  00:18:35.599     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.747  00:18:35.599     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.747  00:18:35.599     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.747  00:18:35.599     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599  
00:30:11.747  00:18:35.599  ========== Backtrace end ==========
00:30:11.747  00:18:35.599  ========== Backtrace start: ==========
00:30:11.747  00:18:35.599  
00:30:11.747  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_rdma"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf.sh"],["--transport=rdma"])
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599     1120		timing_enter $test_name
00:30:11.747  00:18:35.599     1121		echo "************************************"
00:30:11.747  00:18:35.599     1122		echo "START TEST $test_name"
00:30:11.747  00:18:35.599     1123		echo "************************************"
00:30:11.747  00:18:35.599     1124		xtrace_restore
00:30:11.747  00:18:35.599     1125		time "$@"
00:30:11.747  00:18:35.599     1126		xtrace_disable
00:30:11.747  00:18:35.599     1127		echo "************************************"
00:30:11.747  00:18:35.599     1128		echo "END TEST $test_name"
00:30:11.747  00:18:35.599     1129		echo "************************************"
00:30:11.747  00:18:35.599     1130		timing_exit $test_name
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/autotest.sh:277 -> main(["/var/jenkins/workspace/nvmf-phy-autotest/autorun-spdk.conf"])
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599     272 		if [ $SPDK_TEST_NVMF -eq 1 ]; then
00:30:11.747  00:18:35.599     273 			export NET_TYPE
00:30:11.747  00:18:35.599     274 			# The NVMe-oF run test cases are split out like this so that the parser that compiles the
00:30:11.747  00:18:35.599     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.747  00:18:35.599     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.747  00:18:35.599  => 277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.747  00:18:35.599     280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.747  00:18:35.599     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599  
00:30:11.747  00:18:35.599  ========== Backtrace end ========== ]]
00:30:11.747  + echo '00:18:35.066  ========== Backtrace start: ==========
00:30:11.747  00:18:35.066  
00:30:11.747  00:18:35.066  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/host/failover.sh:68 -> main(["--transport=rdma"])
00:30:11.747  00:18:35.066       ...
00:30:11.747  00:18:35.066     63  	cat $testdir/try.txt
00:30:11.747  00:18:35.066     64  	# if this test fails it means we didn'\''t fail over to the second
00:30:11.747  00:18:35.066     65  	count="$(grep -c "Resetting controller successful" < $testdir/try.txt)"
00:30:11.747  00:18:35.066     66  	
00:30:11.747  00:18:35.066     67  	if ((count != 3)); then
00:30:11.747  00:18:35.066  => 68  		false
00:30:11.747  00:18:35.066     69  	fi
00:30:11.747  00:18:35.066     70  	
00:30:11.747  00:18:35.066     71  	# Part 2 of the test. Start removing ports, starting with the one we are connected to, confirm that the ctrlr remains active until the final trid is removed.
00:30:11.747  00:18:35.066     72  	$rootdir/build/examples/bdevperf -z -r $bdevperf_rpc_sock -q 128 -o 4096 -w verify -t 1 -f &> $testdir/try.txt &
00:30:11.747  00:18:35.066     73  	bdevperf_pid=$!
00:30:11.747  00:18:35.066       ...
00:30:11.747  00:18:35.066  
00:30:11.747  00:18:35.066  ========== Backtrace end ==========
00:30:11.747  00:18:35.598  ========== Backtrace start: ==========
00:30:11.747  00:18:35.598  
00:30:11.747  00:18:35.598  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_failover"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/host/failover.sh"],["--transport=rdma"])
00:30:11.747  00:18:35.598       ...
00:30:11.747  00:18:35.598     1120		timing_enter $test_name
00:30:11.747  00:18:35.598     1121		echo "************************************"
00:30:11.747  00:18:35.598     1122		echo "START TEST $test_name"
00:30:11.747  00:18:35.598     1123		echo "************************************"
00:30:11.747  00:18:35.598     1124		xtrace_restore
00:30:11.747  00:18:35.598     1125		time "$@"
00:30:11.747  00:18:35.598     1126		xtrace_disable
00:30:11.747  00:18:35.598     1127		echo "************************************"
00:30:11.747  00:18:35.598     1128		echo "END TEST $test_name"
00:30:11.747  00:18:35.598     1129		echo "************************************"
00:30:11.747  00:18:35.599     1130		timing_exit $test_name
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf_host.sh:25 -> main(["--transport=rdma"])
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599     20  	fi
00:30:11.747  00:18:35.599     21  	
00:30:11.747  00:18:35.599     22  	run_test "nvmf_identify" $rootdir/test/nvmf/host/identify.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     23  	run_test "nvmf_perf" $rootdir/test/nvmf/host/perf.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     24  	run_test "nvmf_fio_host" $rootdir/test/nvmf/host/fio.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599  => 25  	run_test "nvmf_failover" $rootdir/test/nvmf/host/failover.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     26  	run_test "nvmf_host_discovery" $rootdir/test/nvmf/host/discovery.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     27  	run_test "nvmf_host_multipath_status" $rootdir/test/nvmf/host/multipath_status.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     28  	run_test "nvmf_discovery_remove_ifc" $rootdir/test/nvmf/host/discovery_remove_ifc.sh "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     29  	run_test "nvmf_identify_kernel_target" "$rootdir/test/nvmf/host/identify_kernel_nvmf.sh" "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599     30  	run_test "nvmf_auth_host" "$rootdir/test/nvmf/host/auth.sh" "${TEST_ARGS[@]}"
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599  
00:30:11.747  00:18:35.599  ========== Backtrace end ==========
00:30:11.747  00:18:35.599  ========== Backtrace start: ==========
00:30:11.747  00:18:35.599  
00:30:11.747  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_host"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf_host.sh"],["--transport=rdma"])
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599     1120		timing_enter $test_name
00:30:11.747  00:18:35.599     1121		echo "************************************"
00:30:11.747  00:18:35.599     1122		echo "START TEST $test_name"
00:30:11.747  00:18:35.599     1123		echo "************************************"
00:30:11.747  00:18:35.599     1124		xtrace_restore
00:30:11.747  00:18:35.599     1125		time "$@"
00:30:11.747  00:18:35.599     1126		xtrace_disable
00:30:11.747  00:18:35.599     1127		echo "************************************"
00:30:11.747  00:18:35.599     1128		echo "END TEST $test_name"
00:30:11.747  00:18:35.599     1129		echo "************************************"
00:30:11.747  00:18:35.599     1130		timing_exit $test_name
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf.sh:16 -> main(["--transport=rdma"])
00:30:11.747  00:18:35.599       ...
00:30:11.747  00:18:35.599     11  		exit 0
00:30:11.747  00:18:35.599     12  	fi
00:30:11.747  00:18:35.599     13  	
00:30:11.747  00:18:35.599     14  	run_test "nvmf_target_core" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599     15  	run_test "nvmf_target_extra" $rootdir/test/nvmf/nvmf_target_extra.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599  => 16  	run_test "nvmf_host" $rootdir/test/nvmf/nvmf_host.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.747  00:18:35.599     17  	
00:30:11.747  00:18:35.599     18  	# Interrupt mode for now is supported only on the target, with the TCP transport and posix or ssl socket implementations.
00:30:11.747  00:18:35.599     19  	if [[ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" && $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.747  00:18:35.599     20  		run_test "nvmf_target_core_interrupt_mode" $rootdir/test/nvmf/nvmf_target_core.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.748  00:18:35.599     21  		run_test "nvmf_interrupt" $rootdir/test/nvmf/target/interrupt.sh --transport=$SPDK_TEST_NVMF_TRANSPORT --interrupt-mode
00:30:11.748  00:18:35.599       ...
00:30:11.748  00:18:35.599  
00:30:11.748  00:18:35.599  ========== Backtrace end ==========
00:30:11.748  00:18:35.599  ========== Backtrace start: ==========
00:30:11.748  00:18:35.599  
00:30:11.748  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/test/common/autotest_common.sh:1125 -> run_test(["nvmf_rdma"],["/var/jenkins/workspace/nvmf-phy-autotest/spdk/test/nvmf/nvmf.sh"],["--transport=rdma"])
00:30:11.748  00:18:35.599       ...
00:30:11.748  00:18:35.599     1120		timing_enter $test_name
00:30:11.748  00:18:35.599     1121		echo "************************************"
00:30:11.748  00:18:35.599     1122		echo "START TEST $test_name"
00:30:11.748  00:18:35.599     1123		echo "************************************"
00:30:11.748  00:18:35.599     1124		xtrace_restore
00:30:11.748  00:18:35.599     1125		time "$@"
00:30:11.748  00:18:35.599     1126		xtrace_disable
00:30:11.748  00:18:35.599     1127		echo "************************************"
00:30:11.748  00:18:35.599     1128		echo "END TEST $test_name"
00:30:11.748  00:18:35.599     1129		echo "************************************"
00:30:11.748  00:18:35.599     1130		timing_exit $test_name
00:30:11.748  00:18:35.599       ...
00:30:11.748  00:18:35.599  in /var/jenkins/workspace/nvmf-phy-autotest/spdk/autotest.sh:277 -> main(["/var/jenkins/workspace/nvmf-phy-autotest/autorun-spdk.conf"])
00:30:11.748  00:18:35.599       ...
00:30:11.748  00:18:35.599     272 		if [ $SPDK_TEST_NVMF -eq 1 ]; then
00:30:11.748  00:18:35.599     273 			export NET_TYPE
00:30:11.748  00:18:35.599     274 			# The NVMe-oF run test cases are split out like this so that the parser that compiles the
00:30:11.748  00:18:35.599     275 			# list of all tests can properly differentiate them. Please do not merge them into one line.
00:30:11.748  00:18:35.599     276 			if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
00:30:11.748  00:18:35.599  => 277 				run_test "nvmf_rdma" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.748  00:18:35.599     278 				run_test "spdkcli_nvmf_rdma" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.748  00:18:35.599     279 			elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
00:30:11.748  00:18:35.599     280 				run_test "nvmf_tcp" $rootdir/test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.748  00:18:35.599     281 				if [[ $SPDK_TEST_URING -eq 0 ]]; then
00:30:11.748  00:18:35.599     282 					run_test "spdkcli_nvmf_tcp" $rootdir/test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
00:30:11.748  00:18:35.599       ...
00:30:11.748  00:18:35.599  
00:30:11.748  00:18:35.599  ========== Backtrace end =========='
00:30:11.748  + for build in "${build_log_list[@]}"
00:30:11.748  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/build.log
00:30:11.748  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.748  + build_log=/var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/build.log
00:30:11.748  +++ dirname /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/build.log
00:30:11.748  ++ basename /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456
00:30:11.748  + job_name=scan-build-docker-autotest_29456
00:30:11.748  + start_pattern='Backtrace start'
00:30:11.748  + end_pattern='Backtrace end'
00:30:11.748  + [[ -f /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/build.log ]]
00:30:11.748  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/build.log
00:30:11.748  + backtraces=
00:30:11.748  + [[ -n '' ]]
00:30:11.748  + for build in "${build_log_list[@]}"
00:30:11.748  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/build.log
00:30:11.748  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.748  + build_log=/var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/build.log
00:30:11.748  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/build.log
00:30:11.748  ++ basename /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983
00:30:11.748  + job_name=nvme-phy-autotest_83983
00:30:11.748  + start_pattern='Backtrace start'
00:30:11.748  + end_pattern='Backtrace end'
00:30:11.748  + [[ -f /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/build.log ]]
00:30:11.748  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/build.log
00:30:11.748  + backtraces=
00:30:11.748  + [[ -n '' ]]
00:30:11.748  + for build in "${build_log_list[@]}"
00:30:11.748  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/build.log
00:30:11.748  + local build_log job_name start_pattern end_pattern backtraces
00:30:11.748  + build_log=/var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/build.log
00:30:11.748  +++ dirname /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/build.log
00:30:11.748  ++ basename /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805
00:30:11.748  + job_name=raid-vg-autotest_4805
00:30:11.748  + start_pattern='Backtrace start'
00:30:11.748  + end_pattern='Backtrace end'
00:30:11.748  + [[ -f /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/build.log ]]
00:30:11.748  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/build.log
00:30:12.006  + backtraces=
00:30:12.006  + [[ -n '' ]]
00:30:12.006  + for build in "${build_log_list[@]}"
00:30:12.006  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/build.log
00:30:12.006  + local build_log job_name start_pattern end_pattern backtraces
00:30:12.006  + build_log=/var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/build.log
00:30:12.006  +++ dirname /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/build.log
00:30:12.006  ++ basename /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396
00:30:12.006  + job_name=rocky9-vg-autotest_27396
00:30:12.006  + start_pattern='Backtrace start'
00:30:12.006  + end_pattern='Backtrace end'
00:30:12.006  + [[ -f /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/build.log ]]
00:30:12.006  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/build.log
00:30:12.006  + backtraces=
00:30:12.006  + [[ -n '' ]]
00:30:12.006  + for build in "${build_log_list[@]}"
00:30:12.006  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/build.log
00:30:12.006  + local build_log job_name start_pattern end_pattern backtraces
00:30:12.006  + build_log=/var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/build.log
00:30:12.006  +++ dirname /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/build.log
00:30:12.006  ++ basename /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862
00:30:12.006  + job_name=check-so-deps-docker-autotest_32862
00:30:12.006  + start_pattern='Backtrace start'
00:30:12.006  + end_pattern='Backtrace end'
00:30:12.006  + [[ -f /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/build.log ]]
00:30:12.006  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/build.log
00:30:12.006  + backtraces=
00:30:12.007  + [[ -n '' ]]
00:30:12.007  + for build in "${build_log_list[@]}"
00:30:12.007  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/build.log
00:30:12.007  + local build_log job_name start_pattern end_pattern backtraces
00:30:12.007  + build_log=/var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/build.log
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/build.log
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454
00:30:12.007  + job_name=unittest-valgrind-docker-autotest_29454
00:30:12.007  + start_pattern='Backtrace start'
00:30:12.007  + end_pattern='Backtrace end'
00:30:12.007  + [[ -f /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/build.log ]]
00:30:12.007  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/build.log
00:30:12.007  + backtraces=
00:30:12.007  + [[ -n '' ]]
00:30:12.007  + for build in "${build_log_list[@]}"
00:30:12.007  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/build.log
00:30:12.007  + local build_log job_name start_pattern end_pattern backtraces
00:30:12.007  + build_log=/var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/build.log
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/build.log
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655
00:30:12.007  + job_name=release-build-main-docker-autotest_2655
00:30:12.007  + start_pattern='Backtrace start'
00:30:12.007  + end_pattern='Backtrace end'
00:30:12.007  + [[ -f /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/build.log ]]
00:30:12.007  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/build.log
00:30:12.007  + backtraces=
00:30:12.007  + [[ -n '' ]]
00:30:12.007  + for build in "${build_log_list[@]}"
00:30:12.007  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/build.log
00:30:12.007  + local build_log job_name start_pattern end_pattern backtraces
00:30:12.007  + build_log=/var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/build.log
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/build.log
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655
00:30:12.007  + job_name=release-build-clang-docker-autotest_2655
00:30:12.007  + start_pattern='Backtrace start'
00:30:12.007  + end_pattern='Backtrace end'
00:30:12.007  + [[ -f /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/build.log ]]
00:30:12.007  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/build.log
00:30:12.007  + backtraces=
00:30:12.007  + [[ -n '' ]]
00:30:12.007  + for build in "${build_log_list[@]}"
00:30:12.007  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/build.log
00:30:12.007  + local build_log job_name start_pattern end_pattern backtraces
00:30:12.007  + build_log=/var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/build.log
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/build.log
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225
00:30:12.007  + job_name=freebsd-vg-autotest_68225
00:30:12.007  + start_pattern='Backtrace start'
00:30:12.007  + end_pattern='Backtrace end'
00:30:12.007  + [[ -f /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/build.log ]]
00:30:12.007  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/build.log
00:30:12.007  + backtraces=
00:30:12.007  + [[ -n '' ]]
00:30:12.007  + for build in "${build_log_list[@]}"
00:30:12.007  + create_errtrace_file /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/build.log
00:30:12.007  + local build_log job_name start_pattern end_pattern backtraces
00:30:12.007  + build_log=/var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/build.log
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/build.log
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451
00:30:12.007  + job_name=unittest-main-docker-autotest_29451
00:30:12.007  + start_pattern='Backtrace start'
00:30:12.007  + end_pattern='Backtrace end'
00:30:12.007  + [[ -f /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/build.log ]]
00:30:12.007  ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/build.log
00:30:12.007  + backtraces=
00:30:12.007  + [[ -n '' ]]
00:30:12.007  + run_flamegraph
00:30:12.007  + create_timing_file
00:30:12.007  + local array file
00:30:12.007  + array=($(find $WORKSPACE -type f -name 'timing.txt'))
00:30:12.007  ++ find /var/jenkins/workspace/autotest-per-patch -type f -name timing.txt
00:30:12.007  + timing_dir=/var/jenkins/workspace/autotest-per-patch/post_process
00:30:12.007  + mkdir -p /var/jenkins/workspace/autotest-per-patch/post_process
00:30:12.007  + timing_file=/var/jenkins/workspace/autotest-per-patch/post_process/total_timing.txt
00:30:12.007  + for file in "${array[@]}"
00:30:12.007  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/timing.txt
00:30:12.007  + local job_file job_name
00:30:12.007  + job_file=/var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/timing.txt
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/timing.txt
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079
00:30:12.007  + job_name=nvmf-tcp-uring-vg-autotest_90079
00:30:12.007  + sed 's/^/nvmf-tcp-uring-vg-autotest_90079;autorun;/' /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-uring-vg-autotest_90079/timing.txt
00:30:12.007  + for file in "${array[@]}"
00:30:12.007  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/timing.txt
00:30:12.007  + local job_file job_name
00:30:12.007  + job_file=/var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/timing.txt
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/timing.txt
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644
00:30:12.007  + job_name=ubuntu24-vg-autotest_6644
00:30:12.007  + sed 's/^/ubuntu24-vg-autotest_6644;autorun;/' /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/timing.txt
00:30:12.007  + for file in "${array[@]}"
00:30:12.007  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/timing.txt
00:30:12.007  + local job_file job_name
00:30:12.007  + job_file=/var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/timing.txt
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/timing.txt
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944
00:30:12.007  + job_name=doc-docker-autotest_32944
00:30:12.007  + sed 's/^/doc-docker-autotest_32944;autorun;/' /var/jenkins/workspace/autotest-per-patch/doc-docker-autotest_32944/timing.txt
00:30:12.007  + for file in "${array[@]}"
00:30:12.007  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/timing.txt
00:30:12.007  + local job_file job_name
00:30:12.007  + job_file=/var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/timing.txt
00:30:12.007  +++ dirname /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/timing.txt
00:30:12.007  ++ basename /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810
00:30:12.007  + job_name=vfio-user-phy-autotest_37810
00:30:12.007  + sed 's/^/vfio-user-phy-autotest_37810;autorun;/' /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/timing.txt
00:30:12.007  + for file in "${array[@]}"
00:30:12.007  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/timing.txt
00:30:12.007  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820
00:30:12.008  + job_name=BlobFS-phy-autotest_37820
00:30:12.008  + sed 's/^/BlobFS-phy-autotest_37820;autorun;/' /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809
00:30:12.008  + job_name=crypto-phy-autotest_37809
00:30:12.008  + sed 's/^/crypto-phy-autotest_37809;autorun;/' /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085
00:30:12.008  + job_name=lvol-vg-autotest_90085
00:30:12.008  + sed 's/^/lvol-vg-autotest_90085;autorun;/' /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615
00:30:12.008  + job_name=build-files-docker-autotest_32615
00:30:12.008  + sed 's/^/build-files-docker-autotest_32615;autorun;/' /var/jenkins/workspace/autotest-per-patch/build-files-docker-autotest_32615/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154
00:30:12.008  + job_name=vhost-initiator-vg-autotest_90154
00:30:12.008  + sed 's/^/vhost-initiator-vg-autotest_90154;autorun;/' /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173
00:30:12.008  + job_name=iscsi-vg-autotest_90173
00:30:12.008  + sed 's/^/iscsi-vg-autotest_90173;autorun;/' /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037
00:30:12.008  + job_name=iscsi-uring-vg-autotest_90037
00:30:12.008  + sed 's/^/iscsi-uring-vg-autotest_90037;autorun;/' /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449
00:30:12.008  + job_name=packaging-docker-autotest_29449
00:30:12.008  + sed 's/^/packaging-docker-autotest_29449;autorun;/' /var/jenkins/workspace/autotest-per-patch/packaging-docker-autotest_29449/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093
00:30:12.008  + job_name=autobuild-external-vg-autotest_30093
00:30:12.008  + sed 's/^/autobuild-external-vg-autotest_30093;autorun;/' /var/jenkins/workspace/autotest-per-patch/autobuild-external-vg-autotest_30093/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805
00:30:12.008  + job_name=short-fuzz-phy-autotest_37805
00:30:12.008  + sed 's/^/short-fuzz-phy-autotest_37805;autorun;/' /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083
00:30:12.008  + job_name=ubuntu22-vg-autotest_49083
00:30:12.008  + sed 's/^/ubuntu22-vg-autotest_49083;autorun;/' /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641
00:30:12.008  + job_name=check-format-docker-autotest_32641
00:30:12.008  + sed 's/^/check-format-docker-autotest_32641;autorun;/' /var/jenkins/workspace/autotest-per-patch/check-format-docker-autotest_32641/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264
00:30:12.008  + job_name=nvme-cmb-pmr-vg-autotest_91264
00:30:12.008  + sed 's/^/nvme-cmb-pmr-vg-autotest_91264;autorun;/' /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/timing.txt
00:30:12.008  + for file in "${array[@]}"
00:30:12.008  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/timing.txt
00:30:12.008  + local job_file job_name
00:30:12.008  + job_file=/var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/timing.txt
00:30:12.008  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/timing.txt
00:30:12.008  ++ basename /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934
00:30:12.009  + job_name=nvmf-tcp-phy-autotest_83934
00:30:12.009  + sed 's/^/nvmf-tcp-phy-autotest_83934;autorun;/' /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-phy-autotest_83934/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118
00:30:12.009  + job_name=nvmf-tcp-vg-autotest_90118
00:30:12.009  + sed 's/^/nvmf-tcp-vg-autotest_90118;autorun;/' /var/jenkins/workspace/autotest-per-patch/nvmf-tcp-vg-autotest_90118/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238
00:30:12.009  + job_name=zns-vg-autotest_89238
00:30:12.009  + sed 's/^/zns-vg-autotest_89238;autorun;/' /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136
00:30:12.009  + job_name=nvme-vg-autotest_90136
00:30:12.009  + sed 's/^/nvme-vg-autotest_90136;autorun;/' /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874
00:30:12.009  + job_name=vhost-phy-autotest_37874
00:30:12.009  + sed 's/^/vhost-phy-autotest_37874;autorun;/' /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449
00:30:12.009  + job_name=unittest-clang-docker-autotest_29449
00:30:12.009  + sed 's/^/unittest-clang-docker-autotest_29449;autorun;/' /var/jenkins/workspace/autotest-per-patch/unittest-clang-docker-autotest_29449/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052
00:30:12.009  + job_name=nvmf-phy-autotest_84052
00:30:12.009  + sed 's/^/nvmf-phy-autotest_84052;autorun;/' /var/jenkins/workspace/autotest-per-patch/nvmf-phy-autotest_84052/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456
00:30:12.009  + job_name=scan-build-docker-autotest_29456
00:30:12.009  + sed 's/^/scan-build-docker-autotest_29456;autorun;/' /var/jenkins/workspace/autotest-per-patch/scan-build-docker-autotest_29456/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983
00:30:12.009  + job_name=nvme-phy-autotest_83983
00:30:12.009  + sed 's/^/nvme-phy-autotest_83983;autorun;/' /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805
00:30:12.009  + job_name=raid-vg-autotest_4805
00:30:12.009  + sed 's/^/raid-vg-autotest_4805;autorun;/' /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396
00:30:12.009  + job_name=rocky9-vg-autotest_27396
00:30:12.009  + sed 's/^/rocky9-vg-autotest_27396;autorun;/' /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862
00:30:12.009  + job_name=check-so-deps-docker-autotest_32862
00:30:12.009  + sed 's/^/check-so-deps-docker-autotest_32862;autorun;/' /var/jenkins/workspace/autotest-per-patch/check-so-deps-docker-autotest_32862/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454
00:30:12.009  + job_name=unittest-valgrind-docker-autotest_29454
00:30:12.009  + sed 's/^/unittest-valgrind-docker-autotest_29454;autorun;/' /var/jenkins/workspace/autotest-per-patch/unittest-valgrind-docker-autotest_29454/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655
00:30:12.009  + job_name=release-build-main-docker-autotest_2655
00:30:12.009  + sed 's/^/release-build-main-docker-autotest_2655;autorun;/' /var/jenkins/workspace/autotest-per-patch/release-build-main-docker-autotest_2655/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655
00:30:12.009  + job_name=release-build-clang-docker-autotest_2655
00:30:12.009  + sed 's/^/release-build-clang-docker-autotest_2655;autorun;/' /var/jenkins/workspace/autotest-per-patch/release-build-clang-docker-autotest_2655/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/timing.txt
00:30:12.009  + local job_file job_name
00:30:12.009  + job_file=/var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/timing.txt
00:30:12.009  +++ dirname /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/timing.txt
00:30:12.009  ++ basename /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225
00:30:12.009  + job_name=freebsd-vg-autotest_68225
00:30:12.009  + sed 's/^/freebsd-vg-autotest_68225;autorun;/' /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/timing.txt
00:30:12.009  + for file in "${array[@]}"
00:30:12.009  + add_sub_job_timing /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/timing.txt
00:30:12.010  + local job_file job_name
00:30:12.010  + job_file=/var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/timing.txt
00:30:12.010  +++ dirname /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/timing.txt
00:30:12.010  ++ basename /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451
00:30:12.010  + job_name=unittest-main-docker-autotest_29451
00:30:12.010  + sed 's/^/unittest-main-docker-autotest_29451;autorun;/' /var/jenkins/workspace/autotest-per-patch/unittest-main-docker-autotest_29451/timing.txt
00:30:12.010  + /usr/local/FlameGraph/flamegraph.pl --title 'Build Timing' --nametype Step: --countname seconds /var/jenkins/workspace/autotest-per-patch/post_process/total_timing.txt
00:30:12.267  + run_autorun_post
00:30:12.267  + s_opt=
00:30:12.267  + [[ '' == true ]]
00:30:12.267  + spdk/autorun_post.py -d ./ -r ./spdk
00:30:20.372  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/module/blobfs/bdev/blobfs_bdev.c:254
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/env_dpdk/pci.c:1085
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/env_dpdk/pci.c:1078
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/module/bdev/aio/bdev_aio.c:326
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/module/bdev/aio/bdev_aio.c:164
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/module/bdev/aio/bdev_aio.c:684
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/module/bdev/aio/bdev_aio.c:344
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/module/bdev/aio/bdev_aio.c:696
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/thread/thread.c:1600
00:31:28.038  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/thread/thread.c:2786
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/thread/thread.c:1584
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/thread/thread.c:1589
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/thread/thread.c:1578
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/thread/thread.c:1594
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/uuid.c:121
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/uuid.c:115
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/uuid.c:127
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/uuid.c:74
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/uuid.c:133
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/uuid.c:139
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/uuid.c:93
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/env_dpdk/pci_event.c:226
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/env_dpdk/pci_event.c:233
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:542
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:547
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:504
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:510
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:536
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:529
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:524
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:517
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:553
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/util/fd_group.c:559
00:31:28.039  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/lib/event/reactor.c:1511
00:31:28.601  lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-per-patch/spdk/test/unit/lib/env_dpdk/pci_event.c/pci_event_ut.c:111
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/bdev_module.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/nvme.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/fsdev_module.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/base64.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/trace.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/util.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/nvme_spec.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/thread.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/endian.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/mmio.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/histogram_data.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/nvmf_transport.h!
00:31:50.510  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/histogram_data.h/histogram_ut.c!
00:31:53.034  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/lib/env_ocf/include/ocf/ocf_core.h!
00:31:53.034  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/lib/env_ocf/include/ocf/ocf_mngt.h!
00:31:53.034  genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-per-patch/spdk/lib/env_ocf/include/ocf/ocf_io.h!
00:32:14.971  -----Begin Post Process Script------
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/ubuntu24-vg-autotest_6644/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/vfio-user-phy-autotest_37810/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/BlobFS-phy-autotest_37820/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/crypto-phy-autotest_37809/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/lvol-vg-autotest_90085/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/vhost-initiator-vg-autotest_90154/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/iscsi-vg-autotest_90173/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/iscsi-uring-vg-autotest_90037/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/short-fuzz-phy-autotest_37805/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/ubuntu22-vg-autotest_49083/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/nvme-cmb-pmr-vg-autotest_91264/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/zns-vg-autotest_89238/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/nvme-vg-autotest_90136/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/vhost-phy-autotest_37874/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/nvme-phy-autotest_83983/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/raid-vg-autotest_4805/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/rocky9-vg-autotest_27396/cov_total.info
00:32:14.972  /var/jenkins/workspace/autotest-per-patch/freebsd-vg-autotest_68225/cov_total.info
00:32:14.972  
00:32:14.972  
00:32:14.972  -----Tests Executed in Build------
00:32:14.972  accel
00:32:14.972  accel_assign_opcode
00:32:14.972  accel_cdev_comp
00:32:14.972  accel_cdev_decomp
00:32:14.972  accel_cdev_decomp_full
00:32:14.972  accel_cdev_decomp_full_mcore
00:32:14.972  accel_cdev_decomp_full_mthread
00:32:14.972  accel_cdev_decomp_mcore
00:32:14.972  accel_cdev_decomp_mthread
00:32:14.972  accel_comp
00:32:14.972  accel_compare
00:32:14.972  accel_compress_verify
00:32:14.972  accel_copy
00:32:14.972  accel_copy_crc32c
00:32:14.972  accel_copy_crc32c_C2
00:32:14.972  accel_crc32c
00:32:14.972  accel_crc32c_C2
00:32:14.972  accel_decomp
00:32:14.972  accel_decomp_full
00:32:14.972  accel_decomp_full_mcore
00:32:14.972  accel_decomp_full_mthread
00:32:14.972  accel_decomp_mcore
00:32:14.972  accel_decomp_mthread
00:32:14.972  accel_dif_functional_tests
00:32:14.972  accel_dif_generate
00:32:14.972  accel_dif_generate_copy
00:32:14.972  accel_dif_verify
00:32:14.972  accel_dix_generate
00:32:14.972  accel_dix_verify
00:32:14.972  accel_dualcast
00:32:14.972  accel_fill
00:32:14.972  accel_help
00:32:14.972  accel_missing_filename
00:32:14.972  accel_negative_buffers
00:32:14.972  accel_rpc
00:32:14.972  accel_wrong_workload
00:32:14.972  accel_xor
00:32:14.972  acl
00:32:14.972  alias_rpc
00:32:14.972  allowed
00:32:14.972  app_cmdline
00:32:14.972  app_repeat
00:32:14.972  asan
00:32:14.972  autobuild
00:32:14.972  autobuild_build_doc
00:32:14.972  autobuild_check_dpdk_pci_api
00:32:14.972  autobuild_check_format
00:32:14.972  autobuild_check_so_deps
00:32:14.972  autobuild_external_code
00:32:14.972  autobuild_generated_files_check
00:32:14.972  autobuild_generated_files_check_post_clean
00:32:14.972  autobuild_header_dependency_check
00:32:14.972  autobuild_llvm_precompile
00:32:14.972  autobuild_make_install
00:32:14.972  autobuild_make_uninstall
00:32:14.972  autobuild_ocf_precompile
00:32:14.972  bdev_bounds
00:32:14.972  bdev_crypto_enomem
00:32:14.972  bdev_error
00:32:14.972  bdev_fio
00:32:14.972  bdev_fio_rw_verify
00:32:14.972  bdev_fio_trim
00:32:14.972  bdev_gpt_uuid
00:32:14.972  bdev_hello_world
00:32:14.972  bdev_json_nonarray
00:32:14.972  bdev_json_nonenclosed
00:32:14.972  bdev_nbd
00:32:14.972  bdev_nvme_reset_stuck_adm_cmd
00:32:14.972  bdev_qd_sampling
00:32:14.972  bdev_qos
00:32:14.972  bdev_qos_bw
00:32:14.972  bdev_qos_iops
00:32:14.972  bdev_qos_ro_bw
00:32:14.972  bdev_raid
00:32:14.972  bdev_stat
00:32:14.972  bdev_verify
00:32:14.972  bdev_verify_big_io
00:32:14.972  bdev_write_zeroes
00:32:14.972  bdevperf_config
00:32:14.972  blob_io_wait
00:32:14.972  blobfs
00:32:14.972  blobfs_mkfs
00:32:14.972  blobstore
00:32:14.972  blobstore_grow
00:32:14.972  blockdev_crypto_aesni
00:32:14.972  blockdev_crypto_qat
00:32:14.972  blockdev_crypto_sw
00:32:14.972  blockdev_general
00:32:14.972  blockdev_nvme
00:32:14.972  blockdev_nvme_gpt
00:32:14.972  blockdev_raid5f
00:32:14.972  blockdev_rbd
00:32:14.972  blockdev_xnvme
00:32:14.972  build_release
00:32:14.972  build_rpm_from_gen_spec
00:32:14.972  build_shared_rpm
00:32:14.972  chaining
00:32:14.972  chap_during_discovery
00:32:14.972  chap_mutual_auth
00:32:14.972  check_header_filenames
00:32:14.972  cmb_copy
00:32:14.972  compress_compdev
00:32:14.972  compress_isal
00:32:14.972  confirm_abi_deps
00:32:14.972  confirm_makefile_deps
00:32:14.972  core_isolating
00:32:14.972  cpu_locks
00:32:14.972  custom_alloc
00:32:14.972  dd_bs_lt_native_bs
00:32:14.972  dd_bs_not_multiple
00:32:14.972  dd_copy_to_out_bdev
00:32:14.972  dd_double_input
00:32:14.972  dd_double_output
00:32:14.972  dd_flag_append
00:32:14.972  dd_flag_append_forced_aio
00:32:14.972  dd_flag_directory
00:32:14.972  dd_flag_directory_forced_aio
00:32:14.972  dd_flag_noatime
00:32:14.972  dd_flag_noatime_forced_aio
00:32:14.972  dd_flag_nofollow
00:32:14.972  dd_flag_nofollow_forced_aio
00:32:14.972  dd_flags_misc
00:32:14.972  dd_flags_misc_forced_aio
00:32:14.972  dd_inflate_file
00:32:14.972  dd_invalid_arguments
00:32:14.972  dd_invalid_count
00:32:14.972  dd_invalid_iflag
00:32:14.972  dd_invalid_input_count
00:32:14.972  dd_invalid_json
00:32:14.972  dd_invalid_oflag
00:32:14.972  dd_invalid_output_count
00:32:14.972  dd_invalid_seek
00:32:14.972  dd_invalid_skip
00:32:14.972  dd_malloc_copy
00:32:14.972  dd_no_input
00:32:14.972  dd_no_output
00:32:14.972  dd_offset_magic
00:32:14.972  dd_rw
00:32:14.972  dd_rw_offset
00:32:14.972  dd_smaller_blocksize
00:32:14.972  dd_sparse_bdev_to_file
00:32:14.972  dd_sparse_file_to_bdev
00:32:14.972  dd_sparse_file_to_file
00:32:14.972  dd_unknown_flag
00:32:14.972  dd_uring_copy
00:32:14.972  dd_wrong_blocksize
00:32:14.972  default_locks
00:32:14.972  default_locks_via_rpc
00:32:14.972  denied
00:32:14.972  devices
00:32:14.972  dm_mount
00:32:14.972  dma
00:32:14.972  dpdk_governor
00:32:14.972  dpdk_mem_utility
00:32:14.972  driver
00:32:14.972  env
00:32:14.972  env_dpdk_post_init
00:32:14.972  env_mem_callbacks
00:32:14.972  env_memory
00:32:14.972  env_pci
00:32:14.972  env_vtophys
00:32:14.972  even_2G_alloc
00:32:14.972  event
00:32:14.972  event_perf
00:32:14.972  event_reactor
00:32:14.972  event_reactor_perf
00:32:14.972  event_scheduler
00:32:14.972  exit_on_failed_rpc_init
00:32:14.972  external_make_accel_driver_shared
00:32:14.972  external_make_accel_driver_static
00:32:14.972  external_make_accel_module_shared
00:32:14.972  external_make_accel_module_static
00:32:14.972  external_make_hello_bdev_shared_combo
00:32:14.972  external_make_hello_bdev_shared_iso
00:32:14.972  external_make_hello_bdev_static
00:32:14.972  external_make_hello_no_bdev_shared_combo
00:32:14.972  external_make_hello_no_bdev_shared_iso
00:32:14.972  external_make_hello_no_bdev_static
00:32:14.972  external_make_nvme_shared
00:32:14.972  external_make_nvme_static
00:32:14.972  external_run_accel_driver_shared
00:32:14.972  external_run_accel_driver_static
00:32:14.972  external_run_accel_module_shared
00:32:14.972  external_run_accel_module_static
00:32:14.972  external_run_hello_bdev_shared_combo
00:32:14.972  external_run_hello_bdev_shared_iso
00:32:14.972  external_run_hello_bdev_static
00:32:14.972  external_run_hello_no_bdev_shared_combo
00:32:14.972  external_run_hello_no_bdev_shared_iso
00:32:14.972  external_run_hello_no_bdev_static
00:32:14.972  external_run_nvme_shared
00:32:14.972  external_run_nvme_static
00:32:14.972  filesystem_btrfs
00:32:14.972  filesystem_ext4
00:32:14.972  filesystem_in_capsule_btrfs
00:32:14.972  filesystem_in_capsule_ext4
00:32:14.972  filesystem_in_capsule_xfs
00:32:14.972  filesystem_xfs
00:32:14.972  framework_get_governor
00:32:14.972  ftl
00:32:14.972  ftl_bdevperf
00:32:14.972  ftl_dirty_shutdown
00:32:14.972  ftl_fio_basic
00:32:14.972  ftl_restore
00:32:14.972  ftl_trim
00:32:14.972  ftl_upgrade_shutdown
00:32:14.972  go_rpc
00:32:14.972  guess_driver
00:32:14.972  hello_blob
00:32:14.972  hugepages
00:32:14.972  idle
00:32:14.972  interrupt_mode
00:32:14.972  ioat
00:32:14.972  ioat_perf
00:32:14.972  ioat_verify
00:32:14.972  is_zoned
00:32:14.972  iscsi_tgt
00:32:14.972  iscsi_tgt_bdev_io_wait
00:32:14.972  iscsi_tgt_calsoft
00:32:14.972  iscsi_tgt_digest
00:32:14.972  iscsi_tgt_digests
00:32:14.972  iscsi_tgt_filesystem
00:32:14.972  iscsi_tgt_filesystem_btrfs
00:32:14.972  iscsi_tgt_filesystem_ext4
00:32:14.972  iscsi_tgt_filesystem_xfs
00:32:14.972  iscsi_tgt_fio
00:32:14.972  iscsi_tgt_fuzz
00:32:14.972  iscsi_tgt_initiator
00:32:14.973  iscsi_tgt_ip_migration
00:32:14.973  iscsi_tgt_iscsi_lvol
00:32:14.973  iscsi_tgt_login_redirection
00:32:14.973  iscsi_tgt_multiconnection
00:32:14.973  iscsi_tgt_qos
00:32:14.973  iscsi_tgt_rbd
00:32:14.973  iscsi_tgt_reset
00:32:14.973  iscsi_tgt_resize
00:32:14.973  iscsi_tgt_rpc_config
00:32:14.973  iscsi_tgt_sock
00:32:14.973  iscsi_tgt_trace_record
00:32:14.973  json_config
00:32:14.973  json_config_extra_key
00:32:14.973  llvm_fuzz
00:32:14.973  locking_app_on_locked_coremask
00:32:14.973  locking_app_on_unlocked_coremask
00:32:14.973  locking_overlapped_coremask
00:32:14.973  locking_overlapped_coremask_via_rpc
00:32:14.973  lvol
00:32:14.973  lvol_basic
00:32:14.973  lvol_esnap
00:32:14.973  lvol_external_copy
00:32:14.973  lvol_external_snapshot
00:32:14.973  lvol_hotremove
00:32:14.973  lvol_provisioning
00:32:14.973  lvol_rename
00:32:14.973  lvol_resize
00:32:14.973  lvol_snapshot_clone
00:32:14.973  lvol_tasting
00:32:14.973  lvs_grow_clean
00:32:14.973  lvs_grow_dirty
00:32:14.973  make
00:32:14.973  no_shrink_alloc
00:32:14.973  non_locking_app_on_locked_coremask
00:32:14.973  nvme
00:32:14.973  nvme_arbitration
00:32:14.973  nvme_cli_cuse
00:32:14.973  nvme_cli_plugin
00:32:14.973  nvme_cmb
00:32:14.973  nvme_cuse
00:32:14.973  nvme_cuse_app
00:32:14.973  nvme_cuse_rpc
00:32:14.973  nvme_doorbell_aers
00:32:14.973  nvme_e2edp
00:32:14.973  nvme_err_injection
00:32:14.973  nvme_fdp
00:32:14.973  nvme_fio
00:32:14.973  nvme_flexible_data_placement
00:32:14.973  nvme_hello_world
00:32:14.973  nvme_identify
00:32:14.973  nvme_mount
00:32:14.973  nvme_multi_aen
00:32:14.973  nvme_multi_secondary
00:32:14.973  nvme_ns_manage_cuse
00:32:14.973  nvme_overhead
00:32:14.973  nvme_perf
00:32:14.973  nvme_pmr
00:32:14.973  nvme_pmr_persistence
00:32:14.973  nvme_reserve
00:32:14.973  nvme_reset
00:32:14.973  nvme_rpc
00:32:14.973  nvme_rpc_timeouts
00:32:14.973  nvme_scc
00:32:14.973  nvme_sgl
00:32:14.973  nvme_simple_copy
00:32:14.973  nvme_single_aen
00:32:14.973  nvme_smartctl_cuse
00:32:14.973  nvme_startup
00:32:14.973  nvme_xnvme
00:32:14.973  nvme_zns
00:32:14.973  nvmf_abort
00:32:14.973  nvmf_aer
00:32:14.973  nvmf_async_init
00:32:14.973  nvmf_auth_target
00:32:14.973  nvmf_bdev_io_wait
00:32:14.973  nvmf_bdevio
00:32:14.973  nvmf_bdevio_no_huge
00:32:14.973  nvmf_connect_disconnect
00:32:14.973  nvmf_connect_stress
00:32:14.973  nvmf_control_msg_list
00:32:14.973  nvmf_delete_subsystem
00:32:14.973  nvmf_example
00:32:14.973  nvmf_filesystem
00:32:14.973  nvmf_filesystem_in_capsule
00:32:14.973  nvmf_filesystem_no_in_capsule
00:32:14.973  nvmf_fio_host
00:32:14.973  nvmf_fio_target
00:32:14.973  nvmf_fips
00:32:14.973  nvmf_fused_ordering
00:32:14.973  nvmf_host_management
00:32:14.973  nvmf_identify
00:32:14.973  nvmf_invalid
00:32:14.973  nvmf_llvm_fuzz
00:32:14.973  nvmf_lvol
00:32:14.973  nvmf_lvs_grow
00:32:14.973  nvmf_multicontroller
00:32:14.973  nvmf_multitarget
00:32:14.973  nvmf_nmic
00:32:14.973  nvmf_ns_hotplug_stress
00:32:14.973  nvmf_ns_masking
00:32:14.973  nvmf_nvme_cli
00:32:14.973  nvmf_perf
00:32:14.973  nvmf_perf_adq
00:32:14.973  nvmf_queue_depth
00:32:14.973  nvmf_referrals
00:32:14.973  nvmf_rpc
00:32:14.973  nvmf_shutdown
00:32:14.973  nvmf_shutdown_tc1
00:32:14.973  nvmf_shutdown_tc2
00:32:14.973  nvmf_shutdown_tc3
00:32:14.973  nvmf_shutdown_tc4
00:32:14.973  nvmf_srq_overwhelm
00:32:14.973  nvmf_target_core
00:32:14.973  nvmf_target_discovery
00:32:14.973  nvmf_target_extra
00:32:14.973  nvmf_target_multipath
00:32:14.973  nvmf_tls
00:32:14.973  nvmf_vfio_user
00:32:14.973  nvmf_vfio_user_fuzz
00:32:14.973  nvmf_vfio_user_nvme_compliance
00:32:14.973  nvmf_vhost
00:32:14.973  nvmf_wait_for_buf
00:32:14.973  nvmf_zcopy
00:32:14.973  ocf
00:32:14.973  ocf_bdevperf_iotypes
00:32:14.973  ocf_configuration_change
00:32:14.973  ocf_create_destruct
00:32:14.973  ocf_fio_modes
00:32:14.973  ocf_flush
00:32:14.973  ocf_multicore
00:32:14.973  ocf_remove
00:32:14.973  ocf_stats
00:32:14.973  odd_alloc
00:32:14.973  packaging
00:32:14.973  raid0_resize_superblock_test
00:32:14.973  raid0_resize_test
00:32:14.973  raid1_resize_data_offset_test
00:32:14.973  raid1_resize_superblock_test
00:32:14.973  raid1_resize_test
00:32:14.973  raid5f_rebuild_test
00:32:14.973  raid5f_rebuild_test_sb
00:32:14.973  raid5f_state_function_test
00:32:14.973  raid5f_state_function_test_sb
00:32:14.973  raid5f_superblock_test
00:32:14.973  raid_function_test_concat
00:32:14.973  raid_function_test_raid0
00:32:14.973  raid_read_error_test
00:32:14.973  raid_rebuild_test
00:32:14.973  raid_rebuild_test_io
00:32:14.973  raid_rebuild_test_sb
00:32:14.973  raid_rebuild_test_sb_4k
00:32:14.973  raid_rebuild_test_sb_io
00:32:14.973  raid_rebuild_test_sb_md_interleaved
00:32:14.973  raid_rebuild_test_sb_md_separate
00:32:14.973  raid_state_function_test
00:32:14.973  raid_state_function_test_sb
00:32:14.973  raid_state_function_test_sb_4k
00:32:14.973  raid_state_function_test_sb_md_interleaved
00:32:14.973  raid_state_function_test_sb_md_separate
00:32:14.973  raid_superblock_test
00:32:14.973  raid_superblock_test_4k
00:32:14.973  raid_superblock_test_md_interleaved
00:32:14.973  raid_superblock_test_md_separate
00:32:14.973  raid_write_error_test
00:32:14.973  reactor_set_interrupt
00:32:14.973  reap_unregistered_poller
00:32:14.973  rocksdb
00:32:14.973  rocksdb_insert
00:32:14.973  rocksdb_overwrite
00:32:14.973  rocksdb_randread
00:32:14.973  rocksdb_readwrite
00:32:14.973  rocksdb_writesync
00:32:14.973  rpc
00:32:14.973  rpc_client
00:32:14.973  rpc_daemon_integrity
00:32:14.973  rpc_integrity
00:32:14.973  rpc_plugins
00:32:14.973  rpc_trace_cmd_test
00:32:14.973  rpm_packaging
00:32:14.973  scanbuild_make
00:32:14.973  scheduler
00:32:14.973  scheduler_create_thread
00:32:14.973  scheduler_opts
00:32:14.973  scheduler_rpc
00:32:14.973  setup.sh
00:32:14.973  single_node_setup
00:32:14.973  skip_rpc
00:32:14.973  skip_rpc_with_delay
00:32:14.973  skip_rpc_with_json
00:32:14.973  sma
00:32:14.973  sma_crypto
00:32:14.973  sma_discovery
00:32:14.973  sma_nvmf_tcp
00:32:14.973  sma_plugins
00:32:14.973  sma_qos
00:32:14.973  sma_vfiouser_qemu
00:32:14.973  sma_vhost
00:32:14.973  spdk_dd
00:32:14.973  spdk_dd_basic_rw
00:32:14.973  spdk_dd_bdev_to_bdev
00:32:14.973  spdk_dd_malloc
00:32:14.973  spdk_dd_negative
00:32:14.973  spdk_dd_posix
00:32:14.973  spdk_dd_sparse
00:32:14.973  spdk_dd_uring
00:32:14.973  spdkcli_iscsi
00:32:14.973  spdkcli_raid
00:32:14.973  spdkcli_rbd
00:32:14.973  spdkcli_tcp
00:32:14.973  spdkcli_vhost
00:32:14.973  spdkcli_virtio
00:32:14.973  static_as_default
00:32:14.973  sw_hotplug
00:32:14.973  test_bdev_lvol_delete_lvstore_with_clones
00:32:14.973  test_bdev_lvol_delete_ordering
00:32:14.973  test_clone_decouple_parent
00:32:14.973  test_clone_inflate
00:32:14.973  test_clone_snapshot_relations
00:32:14.973  test_construct_lvol
00:32:14.973  test_construct_lvol_alias_conflict
00:32:14.973  test_construct_lvol_fio_clear_method_none
00:32:14.973  test_construct_lvol_fio_clear_method_unmap
00:32:14.973  test_construct_lvol_full_lvs
00:32:14.973  test_construct_lvol_inexistent_lvs
00:32:14.973  test_construct_lvols_conflict_alias
00:32:14.973  test_construct_lvs
00:32:14.973  test_construct_lvs_clear_methods
00:32:14.973  test_construct_lvs_conflict_alias
00:32:14.973  test_construct_lvs_different_cluster_size
00:32:14.973  test_construct_lvs_nonexistent_bdev
00:32:14.973  test_construct_multi_lvols
00:32:14.973  test_construct_nested_lvol
00:32:14.973  test_construct_two_lvs_on_the_same_bdev
00:32:14.973  test_create_multi_ublk
00:32:14.973  test_create_snapshot_of_snapshot
00:32:14.973  test_create_snapshot_with_io
00:32:14.973  test_create_ublk
00:32:14.973  test_delete_lvol_store_persistent_positive
00:32:14.973  test_delete_snapshot_with_clone
00:32:14.973  test_delete_snapshot_with_snapshot
00:32:14.973  test_destroy_after_bdev_lvol_resize_positive
00:32:14.973  test_esnap_clones
00:32:14.973  test_esnap_late_arrival
00:32:14.973  test_esnap_reload
00:32:14.973  test_esnap_remove_degraded
00:32:14.973  test_hotremove_lvol_store
00:32:14.973  test_hotremove_lvol_store_base
00:32:14.973  test_hotremove_lvol_store_base_with_lvols
00:32:14.973  test_hotremove_lvol_store_multiple_lvols
00:32:14.973  test_lvol_bdev_readonly
00:32:14.973  test_lvol_list
00:32:14.973  test_lvol_rename_negative
00:32:14.973  test_lvol_set_parent_bdev_from_esnap
00:32:14.973  test_lvol_set_parent_bdev_from_none
00:32:14.973  test_lvol_set_parent_bdev_from_snapshot
00:32:14.973  test_lvol_set_parent_failed
00:32:14.973  test_lvol_set_parent_from_esnap
00:32:14.973  test_lvol_set_parent_from_none
00:32:14.973  test_lvol_set_parent_from_snapshot
00:32:14.973  test_rename_lvs_negative
00:32:14.973  test_rename_positive
00:32:14.973  test_resize_lvol
00:32:14.973  test_resize_lvol_negative
00:32:14.973  test_resize_lvol_with_io_traffic
00:32:14.973  test_save_ublk_config
00:32:14.973  test_shallow_copy_compare
00:32:14.973  test_sigterm
00:32:14.973  test_snapshot_compare_with_lvol_bdev
00:32:14.973  test_tasting
00:32:14.973  test_thin_lvol_check_integrity
00:32:14.973  test_thin_lvol_check_space
00:32:14.973  test_thin_lvol_check_zeroes
00:32:14.973  test_thin_lvol_resize
00:32:14.973  test_thin_overprovisioning
00:32:14.973  test_unregister_lvol_bdev
00:32:14.973  thread
00:32:14.973  thread_poller_perf
00:32:14.974  thread_spdk_lock
00:32:14.974  ublk
00:32:14.974  ublk_recovery
00:32:14.974  ubsan
00:32:14.974  unittest
00:32:14.974  unittest_accel
00:32:14.974  unittest_bdev
00:32:14.974  unittest_bdev_compress
00:32:14.974  unittest_bdev_crypto
00:32:14.974  unittest_bdev_raid5f
00:32:14.974  unittest_blob_blobfs
00:32:14.974  unittest_build
00:32:14.974  unittest_dma
00:32:14.974  unittest_dpdk_compressdev
00:32:14.974  unittest_event
00:32:14.974  unittest_fsdev
00:32:14.974  unittest_ftl
00:32:14.974  unittest_idxd_user
00:32:14.974  unittest_include
00:32:14.974  unittest_init
00:32:14.974  unittest_ioat
00:32:14.974  unittest_iobuf
00:32:14.974  unittest_iscsi
00:32:14.974  unittest_json
00:32:14.974  unittest_keyring
00:32:14.974  unittest_lib_reduce
00:32:14.974  unittest_log
00:32:14.974  unittest_lvol
00:32:14.974  unittest_notify
00:32:14.974  unittest_nvme
00:32:14.974  unittest_nvme_cuse
00:32:14.974  unittest_nvme_rdma
00:32:14.974  unittest_nvmf
00:32:14.974  unittest_nvmf_rdma
00:32:14.974  unittest_nvmf_transport
00:32:14.974  unittest_nvmf_vfio_user
00:32:14.974  unittest_pci_event
00:32:14.974  unittest_rdma
00:32:14.974  unittest_rpc
00:32:14.974  unittest_scsi
00:32:14.974  unittest_sock
00:32:14.974  unittest_thread
00:32:14.974  unittest_util
00:32:14.974  unittest_vhost
00:32:14.974  valgrind
00:32:14.974  version
00:32:14.974  vfio_llvm_fuzz
00:32:14.974  vfio_user_nvme_fio
00:32:14.974  vfio_user_nvme_restart_vm
00:32:14.974  vfio_user_qemu
00:32:14.974  vfio_user_virtio_bdevperf
00:32:14.974  vfio_user_virtio_blk_restart_vm
00:32:14.974  vfio_user_virtio_fs_fio
00:32:14.974  vfio_user_virtio_scsi_restart_vm
00:32:14.974  vhost
00:32:14.974  vhost_blk_lvol_integrity
00:32:14.974  vhost_blk_packed_ring_integrity
00:32:14.974  vhost_blockdev
00:32:14.974  vhost_boot
00:32:14.974  vhost_fuzz
00:32:14.974  vhost_negative
00:32:14.974  vhost_scsi_lvol_integrity
00:32:14.974  vhost_shared
00:32:14.974  xnvme_bdevperf
00:32:14.974  xnvme_to_malloc_dd_copy
00:32:14.974  zoned_fio
00:32:14.974  
00:32:14.974  
00:32:14.974  -----Tests Missing From Build------
00:32:14.974  accel_mlx5_copy
00:32:14.974  accel_mlx5_crc32c
00:32:14.974  accel_mlx5_crypto
00:32:14.974  accel_scan_dsa_modules
00:32:14.974  accel_scan_iaa_modules
00:32:14.974  balanced
00:32:14.974  build_native_dpdk
00:32:14.974  build_shared_native_dpdk_rpm
00:32:14.974  build_shared_rpm_with_rpmed_dpdk
00:32:14.974  busy
00:32:14.974  core_load
00:32:14.974  fio_dif_1_default
00:32:14.974  fio_dif_1_multi_subsystems
00:32:14.974  fio_dif_digest
00:32:14.974  fio_dif_rand_params
00:32:14.974  ftl_restore_fast
00:32:14.974  iscsi_tgt_data_digest
00:32:14.974  iscsi_tgt_ext4test
00:32:14.974  kernel_target_abort
00:32:14.974  keyring_file
00:32:14.974  keyring_linux
00:32:14.974  nvme_boot_partition
00:32:14.974  nvme_bp
00:32:14.974  nvme_opal
00:32:14.974  nvme_opal_bdevio
00:32:14.974  nvme_opal_bdevperf
00:32:14.974  nvme_opal_spdk_tgt
00:32:14.974  nvmf_abort_qd_sizes
00:32:14.974  nvmf_auth_host
00:32:14.974  nvmf_bdevperf
00:32:14.974  nvmf_device_removal_pci_remove
00:32:14.974  nvmf_device_removal_pci_remove_no_srq
00:32:14.974  nvmf_dif
00:32:14.974  nvmf_digest
00:32:14.974  nvmf_digest_clean
00:32:14.974  nvmf_digest_dsa_initiator
00:32:14.974  nvmf_digest_dsa_target
00:32:14.974  nvmf_digest_error
00:32:14.974  nvmf_discovery_remove_ifc
00:32:14.974  nvmf_failover
00:32:14.974  nvmf_fc
00:32:14.974  nvmf_fuzz
00:32:14.974  nvmf_host
00:32:14.974  nvmf_host_discovery
00:32:14.974  nvmf_host_multipath
00:32:14.974  nvmf_host_multipath_status
00:32:14.974  nvmf_identify_kernel_target
00:32:14.974  nvmf_identify_passthru
00:32:14.974  nvmf_initiator_timeout
00:32:14.974  nvmf_interrupt
00:32:14.974  nvmf_mdns_discovery
00:32:14.974  nvmf_multiconnection
00:32:14.974  nvmf_rdma
00:32:14.974  nvmf_target_core_interrupt_mode
00:32:14.974  nvmf_target_disconnect
00:32:14.974  nvmf_target_disconnect_tc1
00:32:14.974  nvmf_target_disconnect_tc2
00:32:14.974  nvmf_target_disconnect_tc3
00:32:14.974  nvmf_tcp
00:32:14.974  nvmf_timeout
00:32:14.974  spdk_target_abort
00:32:14.974  spdkcli_nvmf_fc
00:32:14.974  spdkcli_nvmf_rdma
00:32:14.974  spdkcli_nvmf_tcp
00:32:14.974  unittest_nvmf_fc
00:32:14.974  unittest_nvmf_fc_ls
00:32:14.974  vhost_blk_cores_2ctrl
00:32:14.974  vhost_blk_fs_integrity
00:32:14.974  vhost_blk_hot_remove
00:32:14.974  vhost_blk_integrity
00:32:14.974  vhost_hotplug
00:32:14.974  vhost_migration
00:32:14.974  vhost_migration_tc1
00:32:14.974  vhost_migration_tc2
00:32:14.974  vhost_readonly
00:32:14.974  vhost_scsi_cores_2ctrl
00:32:14.974  vhost_scsi_fs_integrity
00:32:14.974  vhost_scsi_hot_remove
00:32:14.974  vhost_scsi_integrity
00:32:14.974  vmd
00:32:14.974  vmd_bdev_svc
00:32:14.974  vmd_fio
00:32:14.974  vmd_hello_world
00:32:14.974  vmd_identify
00:32:14.974  vmd_perf
00:32:14.974  
00:32:14.974  
00:32:14.974  -----Tests Missing ASAN------
00:32:14.974  accel_cdev_comp
00:32:14.974  accel_cdev_decomp
00:32:14.974  accel_cdev_decomp_full
00:32:14.974  accel_cdev_decomp_full_mcore
00:32:14.974  accel_cdev_decomp_full_mthread
00:32:14.974  accel_cdev_decomp_mcore
00:32:14.974  accel_cdev_decomp_mthread
00:32:14.974  accel_mlx5_copy
00:32:14.974  accel_mlx5_crc32c
00:32:14.974  accel_mlx5_crypto
00:32:14.974  accel_scan_dsa_modules
00:32:14.974  accel_scan_iaa_modules
00:32:14.974  acl
00:32:14.974  allowed
00:32:14.974  autobuild
00:32:14.974  autobuild_build_doc
00:32:14.974  autobuild_check_dpdk_pci_api
00:32:14.974  autobuild_check_format
00:32:14.974  autobuild_check_so_deps
00:32:14.974  autobuild_external_code
00:32:14.974  autobuild_generated_files_check
00:32:14.974  autobuild_generated_files_check_post_clean
00:32:14.974  autobuild_header_dependency_check
00:32:14.974  autobuild_llvm_precompile
00:32:14.974  autobuild_make_install
00:32:14.974  autobuild_make_uninstall
00:32:14.974  autobuild_ocf_precompile
00:32:14.974  balanced
00:32:14.974  bdev_crypto_enomem
00:32:14.974  blockdev_crypto_aesni
00:32:14.974  blockdev_crypto_qat
00:32:14.974  blockdev_crypto_sw
00:32:14.974  blockdev_rbd
00:32:14.974  build_native_dpdk
00:32:14.974  build_rpm_from_gen_spec
00:32:14.974  build_shared_native_dpdk_rpm
00:32:14.974  build_shared_rpm
00:32:14.974  build_shared_rpm_with_rpmed_dpdk
00:32:14.974  busy
00:32:14.974  chaining
00:32:14.974  chap_during_discovery
00:32:14.974  chap_mutual_auth
00:32:14.974  check_header_filenames
00:32:14.974  cmb_copy
00:32:14.974  compress_compdev
00:32:14.974  compress_isal
00:32:14.974  confirm_abi_deps
00:32:14.974  confirm_makefile_deps
00:32:14.974  core_isolating
00:32:14.974  core_load
00:32:14.974  custom_alloc
00:32:14.974  dd_uring_copy
00:32:14.974  denied
00:32:14.974  devices
00:32:14.974  dm_mount
00:32:14.974  dma
00:32:14.974  dpdk_governor
00:32:14.974  driver
00:32:14.974  even_2G_alloc
00:32:14.974  external_make_accel_driver_shared
00:32:14.974  external_make_accel_driver_static
00:32:14.974  external_make_accel_module_shared
00:32:14.974  external_make_accel_module_static
00:32:14.974  external_make_hello_bdev_shared_combo
00:32:14.974  external_make_hello_bdev_shared_iso
00:32:14.974  external_make_hello_bdev_static
00:32:14.974  external_make_hello_no_bdev_shared_combo
00:32:14.974  external_make_hello_no_bdev_shared_iso
00:32:14.974  external_make_hello_no_bdev_static
00:32:14.974  external_make_nvme_shared
00:32:14.974  external_make_nvme_static
00:32:14.974  external_run_accel_driver_shared
00:32:14.974  external_run_accel_driver_static
00:32:14.974  external_run_accel_module_shared
00:32:14.974  external_run_accel_module_static
00:32:14.974  external_run_hello_bdev_shared_combo
00:32:14.974  external_run_hello_bdev_shared_iso
00:32:14.974  external_run_hello_bdev_static
00:32:14.974  external_run_hello_no_bdev_shared_combo
00:32:14.974  external_run_hello_no_bdev_shared_iso
00:32:14.974  external_run_hello_no_bdev_static
00:32:14.974  external_run_nvme_shared
00:32:14.974  external_run_nvme_static
00:32:14.974  filesystem_btrfs
00:32:14.974  filesystem_ext4
00:32:14.974  filesystem_in_capsule_btrfs
00:32:14.974  filesystem_in_capsule_ext4
00:32:14.974  filesystem_in_capsule_xfs
00:32:14.974  filesystem_xfs
00:32:14.974  fio_dif_1_default
00:32:14.974  fio_dif_1_multi_subsystems
00:32:14.974  fio_dif_digest
00:32:14.974  fio_dif_rand_params
00:32:14.974  framework_get_governor
00:32:14.974  ftl_restore_fast
00:32:14.974  go_rpc
00:32:14.974  guess_driver
00:32:14.974  hugepages
00:32:14.974  idle
00:32:14.974  interrupt_mode
00:32:14.974  ioat
00:32:14.974  ioat_perf
00:32:14.974  ioat_verify
00:32:14.974  is_zoned
00:32:14.974  iscsi_tgt
00:32:14.974  iscsi_tgt_bdev_io_wait
00:32:14.974  iscsi_tgt_calsoft
00:32:14.974  iscsi_tgt_data_digest
00:32:14.974  iscsi_tgt_digest
00:32:14.974  iscsi_tgt_digests
00:32:14.974  iscsi_tgt_ext4test
00:32:14.974  iscsi_tgt_filesystem
00:32:14.974  iscsi_tgt_filesystem_btrfs
00:32:14.974  iscsi_tgt_filesystem_ext4
00:32:14.974  iscsi_tgt_filesystem_xfs
00:32:14.974  iscsi_tgt_fio
00:32:14.974  iscsi_tgt_fuzz
00:32:14.974  iscsi_tgt_initiator
00:32:14.974  iscsi_tgt_ip_migration
00:32:14.974  iscsi_tgt_iscsi_lvol
00:32:14.974  iscsi_tgt_login_redirection
00:32:14.974  iscsi_tgt_multiconnection
00:32:14.974  iscsi_tgt_qos
00:32:14.975  iscsi_tgt_rbd
00:32:14.975  iscsi_tgt_reset
00:32:14.975  iscsi_tgt_resize
00:32:14.975  iscsi_tgt_rpc_config
00:32:14.975  iscsi_tgt_sock
00:32:14.975  iscsi_tgt_trace_record
00:32:14.975  kernel_target_abort
00:32:14.975  keyring_file
00:32:14.975  keyring_linux
00:32:14.975  llvm_fuzz
00:32:14.975  lvs_grow_clean
00:32:14.975  lvs_grow_dirty
00:32:14.975  no_shrink_alloc
00:32:14.975  nvme_boot_partition
00:32:14.975  nvme_bp
00:32:14.975  nvme_cli_cuse
00:32:14.975  nvme_cli_plugin
00:32:14.975  nvme_cmb
00:32:14.975  nvme_cuse
00:32:14.975  nvme_cuse_app
00:32:14.975  nvme_cuse_rpc
00:32:14.975  nvme_mount
00:32:14.975  nvme_ns_manage_cuse
00:32:14.975  nvme_opal
00:32:14.975  nvme_opal_bdevio
00:32:14.975  nvme_opal_bdevperf
00:32:14.975  nvme_opal_spdk_tgt
00:32:14.975  nvme_pmr
00:32:14.975  nvme_pmr_persistence
00:32:14.975  nvme_smartctl_cuse
00:32:14.975  nvme_zns
00:32:14.975  nvmf_abort
00:32:14.975  nvmf_abort_qd_sizes
00:32:14.975  nvmf_aer
00:32:14.975  nvmf_async_init
00:32:14.975  nvmf_auth_host
00:32:14.975  nvmf_auth_target
00:32:14.975  nvmf_bdev_io_wait
00:32:14.975  nvmf_bdevio
00:32:14.975  nvmf_bdevio_no_huge
00:32:14.975  nvmf_bdevperf
00:32:14.975  nvmf_connect_disconnect
00:32:14.975  nvmf_connect_stress
00:32:14.975  nvmf_control_msg_list
00:32:14.975  nvmf_delete_subsystem
00:32:14.975  nvmf_device_removal_pci_remove
00:32:14.975  nvmf_device_removal_pci_remove_no_srq
00:32:14.975  nvmf_dif
00:32:14.975  nvmf_digest
00:32:14.975  nvmf_digest_clean
00:32:14.975  nvmf_digest_dsa_initiator
00:32:14.975  nvmf_digest_dsa_target
00:32:14.975  nvmf_digest_error
00:32:14.975  nvmf_discovery_remove_ifc
00:32:14.975  nvmf_example
00:32:14.975  nvmf_failover
00:32:14.975  nvmf_fc
00:32:14.975  nvmf_filesystem
00:32:14.975  nvmf_filesystem_in_capsule
00:32:14.975  nvmf_filesystem_no_in_capsule
00:32:14.975  nvmf_fio_host
00:32:14.975  nvmf_fio_target
00:32:14.975  nvmf_fips
00:32:14.975  nvmf_fused_ordering
00:32:14.975  nvmf_fuzz
00:32:14.975  nvmf_host
00:32:14.975  nvmf_host_discovery
00:32:14.975  nvmf_host_management
00:32:14.975  nvmf_host_multipath
00:32:14.975  nvmf_host_multipath_status
00:32:14.975  nvmf_identify
00:32:14.975  nvmf_identify_kernel_target
00:32:14.975  nvmf_identify_passthru
00:32:14.975  nvmf_initiator_timeout
00:32:14.975  nvmf_interrupt
00:32:14.975  nvmf_invalid
00:32:14.975  nvmf_llvm_fuzz
00:32:14.975  nvmf_lvol
00:32:14.975  nvmf_lvs_grow
00:32:14.975  nvmf_mdns_discovery
00:32:14.975  nvmf_multiconnection
00:32:14.975  nvmf_multicontroller
00:32:14.975  nvmf_multitarget
00:32:14.975  nvmf_nmic
00:32:14.975  nvmf_ns_hotplug_stress
00:32:14.975  nvmf_ns_masking
00:32:14.975  nvmf_nvme_cli
00:32:14.975  nvmf_perf
00:32:14.975  nvmf_perf_adq
00:32:14.975  nvmf_queue_depth
00:32:14.975  nvmf_rdma
00:32:14.975  nvmf_referrals
00:32:14.975  nvmf_rpc
00:32:14.975  nvmf_shutdown
00:32:14.975  nvmf_shutdown_tc1
00:32:14.975  nvmf_shutdown_tc2
00:32:14.975  nvmf_shutdown_tc3
00:32:14.975  nvmf_shutdown_tc4
00:32:14.975  nvmf_srq_overwhelm
00:32:14.975  nvmf_target_core
00:32:14.975  nvmf_target_core_interrupt_mode
00:32:14.975  nvmf_target_disconnect
00:32:14.975  nvmf_target_disconnect_tc1
00:32:14.975  nvmf_target_disconnect_tc2
00:32:14.975  nvmf_target_disconnect_tc3
00:32:14.975  nvmf_target_discovery
00:32:14.975  nvmf_target_extra
00:32:14.975  nvmf_target_multipath
00:32:14.975  nvmf_tcp
00:32:14.975  nvmf_timeout
00:32:14.975  nvmf_tls
00:32:14.975  nvmf_vfio_user
00:32:14.975  nvmf_vfio_user_fuzz
00:32:14.975  nvmf_vfio_user_nvme_compliance
00:32:14.975  nvmf_wait_for_buf
00:32:14.975  nvmf_zcopy
00:32:14.975  ocf
00:32:14.975  ocf_bdevperf_iotypes
00:32:14.975  ocf_configuration_change
00:32:14.975  ocf_create_destruct
00:32:14.975  ocf_fio_modes
00:32:14.975  ocf_flush
00:32:14.975  ocf_multicore
00:32:14.975  ocf_remove
00:32:14.975  ocf_stats
00:32:14.975  odd_alloc
00:32:14.975  packaging
00:32:14.975  rpm_packaging
00:32:14.975  scanbuild_make
00:32:14.975  scheduler
00:32:14.975  scheduler_opts
00:32:14.975  scheduler_rpc
00:32:14.975  setup.sh
00:32:14.975  single_node_setup
00:32:14.975  spdk_dd_uring
00:32:14.975  spdk_target_abort
00:32:14.975  spdkcli_iscsi
00:32:14.975  spdkcli_nvmf_fc
00:32:14.975  spdkcli_nvmf_rdma
00:32:14.975  spdkcli_nvmf_tcp
00:32:14.975  spdkcli_rbd
00:32:14.975  spdkcli_virtio
00:32:14.975  static_as_default
00:32:14.975  unittest_bdev_compress
00:32:14.975  unittest_bdev_crypto
00:32:14.975  unittest_bdev_raid5f
00:32:14.975  unittest_dpdk_compressdev
00:32:14.975  unittest_lib_reduce
00:32:14.975  unittest_nvmf_fc
00:32:14.975  unittest_nvmf_fc_ls
00:32:14.975  unittest_nvmf_vfio_user
00:32:14.975  valgrind
00:32:14.975  vfio_llvm_fuzz
00:32:14.975  vhost_blk_cores_2ctrl
00:32:14.975  vhost_blk_fs_integrity
00:32:14.975  vhost_blk_hot_remove
00:32:14.975  vhost_blk_integrity
00:32:14.975  vhost_blockdev
00:32:14.975  vhost_fuzz
00:32:14.975  vhost_hotplug
00:32:14.975  vhost_migration
00:32:14.975  vhost_migration_tc1
00:32:14.975  vhost_migration_tc2
00:32:14.975  vhost_readonly
00:32:14.975  vhost_scsi_cores_2ctrl
00:32:14.975  vhost_scsi_fs_integrity
00:32:14.975  vhost_scsi_hot_remove
00:32:14.975  vhost_scsi_integrity
00:32:14.975  vhost_shared
00:32:14.975  vmd
00:32:14.975  vmd_bdev_svc
00:32:14.975  vmd_fio
00:32:14.975  vmd_hello_world
00:32:14.975  vmd_identify
00:32:14.975  vmd_perf
00:32:14.975  zoned_fio
00:32:14.975  
00:32:14.975  
00:32:14.975  -----Tests Missing UBSAN------
00:32:14.975  accel_mlx5_copy
00:32:14.975  accel_mlx5_crc32c
00:32:14.975  accel_mlx5_crypto
00:32:14.975  accel_scan_dsa_modules
00:32:14.975  accel_scan_iaa_modules
00:32:14.975  autobuild
00:32:14.975  autobuild_build_doc
00:32:14.975  autobuild_check_dpdk_pci_api
00:32:14.975  autobuild_check_format
00:32:14.975  autobuild_check_so_deps
00:32:14.975  autobuild_external_code
00:32:14.975  autobuild_generated_files_check
00:32:14.975  autobuild_generated_files_check_post_clean
00:32:14.975  autobuild_header_dependency_check
00:32:14.975  autobuild_make_install
00:32:14.975  autobuild_make_uninstall
00:32:14.975  balanced
00:32:14.975  build_native_dpdk
00:32:14.975  build_release
00:32:14.975  build_rpm_from_gen_spec
00:32:14.975  build_shared_native_dpdk_rpm
00:32:14.975  build_shared_rpm
00:32:14.975  build_shared_rpm_with_rpmed_dpdk
00:32:14.975  busy
00:32:14.975  check_header_filenames
00:32:14.975  cmb_copy
00:32:14.975  confirm_abi_deps
00:32:14.975  confirm_makefile_deps
00:32:14.975  core_load
00:32:14.975  external_make_accel_driver_shared
00:32:14.975  external_make_accel_driver_static
00:32:14.975  external_make_accel_module_shared
00:32:14.975  external_make_accel_module_static
00:32:14.975  external_make_hello_bdev_shared_combo
00:32:14.975  external_make_hello_bdev_shared_iso
00:32:14.975  external_make_hello_bdev_static
00:32:14.975  external_make_hello_no_bdev_shared_combo
00:32:14.975  external_make_hello_no_bdev_shared_iso
00:32:14.975  external_make_hello_no_bdev_static
00:32:14.975  external_make_nvme_shared
00:32:14.975  external_make_nvme_static
00:32:14.975  external_run_accel_driver_shared
00:32:14.975  external_run_accel_driver_static
00:32:14.975  external_run_accel_module_shared
00:32:14.975  external_run_accel_module_static
00:32:14.975  external_run_hello_bdev_shared_combo
00:32:14.975  external_run_hello_bdev_shared_iso
00:32:14.975  external_run_hello_bdev_static
00:32:14.975  external_run_hello_no_bdev_shared_combo
00:32:14.975  external_run_hello_no_bdev_shared_iso
00:32:14.975  external_run_hello_no_bdev_static
00:32:14.975  external_run_nvme_shared
00:32:14.975  external_run_nvme_static
00:32:14.975  fio_dif_1_default
00:32:14.975  fio_dif_1_multi_subsystems
00:32:14.975  fio_dif_digest
00:32:14.975  fio_dif_rand_params
00:32:14.975  ftl_restore_fast
00:32:14.975  is_zoned
00:32:14.975  iscsi_tgt_data_digest
00:32:14.975  iscsi_tgt_ext4test
00:32:14.975  kernel_target_abort
00:32:14.975  keyring_file
00:32:14.975  keyring_linux
00:32:14.975  nvme_boot_partition
00:32:14.975  nvme_bp
00:32:14.975  nvme_cmb
00:32:14.975  nvme_opal
00:32:14.975  nvme_opal_bdevio
00:32:14.975  nvme_opal_bdevperf
00:32:14.975  nvme_opal_spdk_tgt
00:32:14.975  nvme_pmr
00:32:14.975  nvme_pmr_persistence
00:32:14.975  nvme_zns
00:32:14.975  nvmf_abort_qd_sizes
00:32:14.975  nvmf_auth_host
00:32:14.975  nvmf_bdevperf
00:32:14.975  nvmf_device_removal_pci_remove
00:32:14.975  nvmf_device_removal_pci_remove_no_srq
00:32:14.975  nvmf_dif
00:32:14.975  nvmf_digest
00:32:14.975  nvmf_digest_clean
00:32:14.975  nvmf_digest_dsa_initiator
00:32:14.975  nvmf_digest_dsa_target
00:32:14.975  nvmf_digest_error
00:32:14.975  nvmf_discovery_remove_ifc
00:32:14.975  nvmf_failover
00:32:14.975  nvmf_fc
00:32:14.975  nvmf_fuzz
00:32:14.975  nvmf_host
00:32:14.975  nvmf_host_discovery
00:32:14.975  nvmf_host_multipath
00:32:14.975  nvmf_host_multipath_status
00:32:14.975  nvmf_identify_kernel_target
00:32:14.975  nvmf_identify_passthru
00:32:14.975  nvmf_initiator_timeout
00:32:14.975  nvmf_interrupt
00:32:14.975  nvmf_mdns_discovery
00:32:14.975  nvmf_multiconnection
00:32:14.975  nvmf_rdma
00:32:14.975  nvmf_target_core_interrupt_mode
00:32:14.975  nvmf_target_disconnect
00:32:14.976  nvmf_target_disconnect_tc1
00:32:14.976  nvmf_target_disconnect_tc2
00:32:14.976  nvmf_target_disconnect_tc3
00:32:14.976  nvmf_tcp
00:32:14.976  nvmf_timeout
00:32:14.976  packaging
00:32:14.976  rpm_packaging
00:32:14.976  scanbuild_make
00:32:14.976  spdk_target_abort
00:32:14.976  spdkcli_nvmf_fc
00:32:14.976  spdkcli_nvmf_rdma
00:32:14.976  spdkcli_nvmf_tcp
00:32:14.976  unittest_bdev_compress
00:32:14.976  unittest_bdev_crypto
00:32:14.976  unittest_bdev_raid5f
00:32:14.976  unittest_dpdk_compressdev
00:32:14.976  unittest_lib_reduce
00:32:14.976  unittest_nvmf_fc
00:32:14.976  unittest_nvmf_fc_ls
00:32:14.976  unittest_nvmf_vfio_user
00:32:14.976  valgrind
00:32:14.976  vhost_blk_cores_2ctrl
00:32:14.976  vhost_blk_fs_integrity
00:32:14.976  vhost_blk_hot_remove
00:32:14.976  vhost_blk_integrity
00:32:14.976  vhost_hotplug
00:32:14.976  vhost_migration
00:32:14.976  vhost_migration_tc1
00:32:14.976  vhost_migration_tc2
00:32:14.976  vhost_readonly
00:32:14.976  vhost_scsi_cores_2ctrl
00:32:14.976  vhost_scsi_fs_integrity
00:32:14.976  vhost_scsi_hot_remove
00:32:14.976  vhost_scsi_integrity
00:32:14.976  vmd
00:32:14.976  vmd_bdev_svc
00:32:14.976  vmd_fio
00:32:14.976  vmd_hello_world
00:32:14.976  vmd_identify
00:32:14.976  vmd_perf
00:32:14.976  zoned_fio
00:32:14.976  Not all tests were run. Failing the build.
00:32:14.976  ['fio_dif_1_default', 'fio_dif_1_multi_subsystems', 'fio_dif_digest', 'fio_dif_rand_params', 'kernel_target_abort', 'keyring_file', 'nvmf_abort_qd_sizes', 'nvmf_auth_host', 'nvmf_bdevperf', 'nvmf_dif', 'nvmf_digest', 'nvmf_digest_clean', 'nvmf_digest_error', 'nvmf_discovery_remove_ifc', 'nvmf_failover', 'nvmf_host', 'nvmf_host_discovery', 'nvmf_host_multipath', 'nvmf_host_multipath_status', 'nvmf_identify_kernel_target', 'nvmf_identify_passthru', 'nvmf_interrupt', 'nvmf_mdns_discovery', 'nvmf_rdma', 'nvmf_target_core_interrupt_mode', 'nvmf_target_disconnect', 'nvmf_target_disconnect_tc1', 'nvmf_target_disconnect_tc2', 'nvmf_target_disconnect_tc3', 'nvmf_tcp', 'nvmf_timeout', 'spdk_target_abort', 'spdkcli_nvmf_rdma', 'spdkcli_nvmf_tcp']
00:32:14.976  + compress_coverage_and_docs
00:32:14.976  + echo 'Start compress coverage and docs'
00:32:14.976  Start compress coverage and docs
00:32:14.976  + tar -C coverage -czf coverage_autotest-per-patch_130557.tar.gz ./ --remove-files
00:32:14.976  + tar -C ut_coverage -czf ut_coverage_autotest-per-patch_130557.tar.gz ./ --remove-files
00:32:14.976  + tar -C doc -czf doc_autotest-per-patch_130557.tar.gz ./ --remove-files
00:32:14.976  + [[ -d llvm ]]
00:32:14.976  + true
00:32:14.976  + echo 'End compress coverage and docs'
00:32:14.976  End compress coverage and docs
00:32:14.976  + remove_partial_coverage_files
00:32:14.976  + find . -name 'cov_*.info' -delete
00:32:14.976  + echo 'End stage post_gen_coverage.sh'
00:32:14.976  End stage post_gen_coverage.sh
00:32:14.990  [Pipeline] }
00:32:14.994  ERROR: script returned exit code 1
00:32:15.021  [Pipeline] // catchError
00:32:15.032  [Pipeline] catchError
00:32:15.034  [Pipeline] {
00:32:15.053  [Pipeline] dir
00:32:15.054  Running in /var/jenkins/workspace/autotest-per-patch/post_process
00:32:15.055  [Pipeline] {