00:00:00.001 Started by upstream project "nightly-trigger" build number 3087 00:00:00.001 originally caused by: 00:00:00.001 Started by timer 00:00:00.010 Checking out git https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool into /var/jenkins_home/workspace/autotest-nightly_script/33b20b30f0a51e6b52980845e0f6aa336787973ad45e341fbbf98d1b65b265d4 to read jbp/jenkins/jjb-config/jobs/autotest-upstream/autotest.groovy 00:00:00.011 The recommended git tool is: git 00:00:00.011 using credential 00000000-0000-0000-0000-000000000002 00:00:00.013 > git rev-parse --resolve-git-dir /var/jenkins_home/workspace/autotest-nightly_script/33b20b30f0a51e6b52980845e0f6aa336787973ad45e341fbbf98d1b65b265d4/jbp/.git # timeout=10 00:00:00.026 Fetching changes from the remote Git repository 00:00:00.027 > git config remote.origin.url https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool # timeout=10 00:00:00.040 Using shallow fetch with depth 1 00:00:00.040 Fetching upstream changes from https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool 00:00:00.040 > git --version # timeout=10 00:00:00.051 > git --version # 'git version 2.39.2' 00:00:00.051 using GIT_ASKPASS to set credentials SPDKCI HTTPS Credentials 00:00:00.052 Setting http proxy: proxy-dmz.intel.com:911 00:00:00.052 > git fetch --tags --force --progress --depth=1 -- https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool refs/heads/master # timeout=5 00:00:02.486 > git rev-parse origin/FETCH_HEAD^{commit} # timeout=10 00:00:02.497 > git rev-parse FETCH_HEAD^{commit} # timeout=10 00:00:02.509 Checking out Revision 10da8f6d99838e411e4e94523ded0bfebf3e7100 (FETCH_HEAD) 00:00:02.509 > git config core.sparsecheckout # timeout=10 00:00:02.519 > git read-tree -mu HEAD # timeout=10 00:00:02.537 > git checkout -f 10da8f6d99838e411e4e94523ded0bfebf3e7100 # timeout=5 00:00:02.559 Commit message: "scripts/create_git_mirror: Update path to xnvme submodule" 00:00:02.559 > git rev-list --no-walk 10da8f6d99838e411e4e94523ded0bfebf3e7100 # timeout=10 00:00:03.080 [Pipeline] Start of Pipeline 00:00:03.095 [Pipeline] library 00:00:03.097 Loading library shm_lib@master 00:00:03.097 Library shm_lib@master is cached. Copying from home. 00:00:03.117 [Pipeline] node 00:00:03.127 Running on ME2 in /var/jenkins/workspace/autotest-nightly 00:00:03.132 [Pipeline] { 00:00:03.148 [Pipeline] cleanWs 00:00:03.158 [WS-CLEANUP] Deleting project workspace... 00:00:03.158 [WS-CLEANUP] Deferred wipeout is used... 00:00:03.163 [WS-CLEANUP] done 00:00:03.169 [Pipeline] stage 00:00:03.174 [Pipeline] { (Prologue) 00:00:03.272 [Pipeline] withCredentials 00:00:03.282 > git --version # timeout=10 00:00:03.295 > git --version # 'git version 2.39.2' 00:00:03.313 Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD or $GIT_ASKPASS 00:00:03.316 [Pipeline] { 00:00:03.324 [Pipeline] retry 00:00:03.326 [Pipeline] { 00:00:03.491 [Pipeline] sh 00:00:03.775 + git ls-remote https://review.spdk.io/gerrit/a/build_pool/jenkins_build_pool refs/heads/master 00:00:05.692 [Pipeline] } 00:00:05.722 [Pipeline] // retry 00:00:05.727 [Pipeline] } 00:00:05.748 [Pipeline] // withCredentials 00:00:05.757 [Pipeline] nodesByLabel 00:00:05.759 Found a total of 1 nodes with the 'sorcerer' label 00:00:05.770 [Pipeline] httpRequest 00:00:05.775 HttpMethod: GET 00:00:05.776 URL: http://10.211.164.101/packages/jbp_10da8f6d99838e411e4e94523ded0bfebf3e7100.tar.gz 00:00:05.778 Sending request to url: http://10.211.164.101/packages/jbp_10da8f6d99838e411e4e94523ded0bfebf3e7100.tar.gz 00:00:05.780 Response Code: HTTP/1.1 200 OK 00:00:05.781 Success: Status code 200 is in the accepted range: 200,404 00:00:05.781 Saving response body to /var/jenkins/workspace/autotest-nightly/jbp_10da8f6d99838e411e4e94523ded0bfebf3e7100.tar.gz 00:00:05.920 [Pipeline] sh 00:00:06.205 + tar --no-same-owner -xf jbp_10da8f6d99838e411e4e94523ded0bfebf3e7100.tar.gz 00:00:06.229 [Pipeline] withCredentials 00:00:06.240 > git --version # timeout=10 00:00:06.253 > git --version # 'git version 2.39.2' 00:00:06.273 Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD or $GIT_ASKPASS 00:00:06.275 [Pipeline] { 00:00:06.286 [Pipeline] retry 00:00:06.289 [Pipeline] { 00:00:06.313 [Pipeline] sh 00:00:06.599 + git ls-remote https://review.spdk.io/gerrit/a/spdk/spdk refs/heads/master 00:00:09.917 [Pipeline] } 00:00:09.930 [Pipeline] // retry 00:00:09.934 [Pipeline] } 00:00:09.949 [Pipeline] // withCredentials 00:00:09.957 [Pipeline] httpRequest 00:00:09.961 HttpMethod: GET 00:00:09.961 URL: http://10.211.164.101/packages/spdk_4506c0c368f63ba9b9b013ecff216cef6ee8d0a4.tar.gz 00:00:09.962 Sending request to url: http://10.211.164.101/packages/spdk_4506c0c368f63ba9b9b013ecff216cef6ee8d0a4.tar.gz 00:00:09.964 Response Code: HTTP/1.1 200 OK 00:00:09.964 Success: Status code 200 is in the accepted range: 200,404 00:00:09.964 Saving response body to /var/jenkins/workspace/autotest-nightly/spdk_4506c0c368f63ba9b9b013ecff216cef6ee8d0a4.tar.gz 00:00:12.088 [Pipeline] sh 00:00:12.371 + tar --no-same-owner -xf spdk_4506c0c368f63ba9b9b013ecff216cef6ee8d0a4.tar.gz 00:00:14.280 [Pipeline] sh 00:00:14.557 + git -C spdk log --oneline -n5 00:00:14.557 4506c0c36 test/common: Enable inherit_errexit 00:00:14.557 b24df7cfa test: Drop superfluous calls to print_backtrace() 00:00:14.557 7b52e4c17 test/scheduler: Meassure utime of $spdk_pid threads as a fallback 00:00:14.557 1dc065205 test/scheduler: Calculate median of the cpu load samples 00:00:14.557 b22f1b34d test/scheduler: Enhance lookup of the $old_cgroup in move_proc() 00:00:14.574 [Pipeline] sh 00:00:14.855 + git -C spdk show -s --format=%s 4506c0c368f63ba9b9b013ecff216cef6ee8d0a4 00:00:14.871 [Pipeline] setCustomBuildProperty 00:00:14.879 [Pipeline] setCustomBuildProperty 00:00:14.888 [Pipeline] catchError 00:00:14.889 [Pipeline] { 00:00:14.908 [Pipeline] readJSON 00:00:14.923 [Pipeline] } 00:00:14.950 [Pipeline] // catchError 00:00:14.956 [Pipeline] } 00:00:14.984 [Pipeline] // stage 00:00:15.003 [Pipeline] catchError 00:00:15.005 [Pipeline] { 00:00:15.027 [Pipeline] stage 00:00:15.029 [Pipeline] { (Tests) 00:00:15.228 [Pipeline] parallel 00:00:15.329 [Pipeline] { (Branch: check-format-docker-autotest) 00:00:15.331 [Pipeline] { (Branch: check-so-deps-docker-autotest) 00:00:15.332 [Pipeline] { (Branch: doc-docker-autotest) 00:00:15.333 [Pipeline] { (Branch: build-files-docker-autotest) 00:00:15.334 [Pipeline] { (Branch: unittest-clang-docker-autotest) 00:00:15.335 [Pipeline] { (Branch: unittest-main-docker-autotest) 00:00:15.336 [Pipeline] { (Branch: unittest-valgrind-docker-autotest) 00:00:15.336 [Pipeline] { (Branch: scan-build-docker-autotest) 00:00:15.338 [Pipeline] { (Branch: packaging-docker-autotest) 00:00:15.339 [Pipeline] { (Branch: autobuild-external-vg-autotest) 00:00:15.340 [Pipeline] { (Branch: lvol-vg-autotest) 00:00:15.340 [Pipeline] { (Branch: vhost-initiator-vg-autotest) 00:00:15.341 [Pipeline] { (Branch: nvmf-tcp-vg-autotest) 00:00:15.342 [Pipeline] { (Branch: nvmf-tcp-uring-vg-autotest) 00:00:15.343 [Pipeline] { (Branch: nvme-vg-autotest) 00:00:15.344 [Pipeline] { (Branch: centos7-vg-autotest) 00:00:15.345 [Pipeline] { (Branch: rocky9-vg-autotest) 00:00:15.346 [Pipeline] { (Branch: ubuntu20-vg-autotest) 00:00:15.347 [Pipeline] { (Branch: ubuntu22-vg-autotest) 00:00:15.348 [Pipeline] { (Branch: freebsd-vg-autotest) 00:00:15.348 [Pipeline] { (Branch: iscsi-vg-autotest) 00:00:15.349 [Pipeline] { (Branch: iscsi-uring-vg-autotest) 00:00:15.350 [Pipeline] { (Branch: zns-vg-autotest) 00:00:15.351 [Pipeline] { (Branch: nvme-cmb-pmr-vg-autotest) 00:00:15.352 [Pipeline] { (Branch: BlobFS-phy-autotest) 00:00:15.353 [Pipeline] { (Branch: crypto-phy-autotest) 00:00:15.354 [Pipeline] { (Branch: vhost-phy-autotest) 00:00:15.355 [Pipeline] { (Branch: nvmf-phy-autotest) 00:00:15.356 [Pipeline] { (Branch: nvmf-cvl-phy-autotest) 00:00:15.357 [Pipeline] { (Branch: nvmf-tcp-phy-autotest) 00:00:15.358 [Pipeline] { (Branch: nvme-phy-autotest) 00:00:15.359 [Pipeline] { (Branch: vfio-user-phy-autotest) 00:00:15.360 [Pipeline] { (Branch: short-fuzz-phy-autotest) 00:00:15.360 [Pipeline] { (Branch: dsa-phy-autotest) 00:00:15.361 [Pipeline] { (Branch: vmd-phy-autotest) 00:00:15.395 [Pipeline] retry 00:00:15.397 [Pipeline] { 00:00:15.401 [Pipeline] retry 00:00:15.403 [Pipeline] { 00:00:15.407 [Pipeline] retry 00:00:15.409 [Pipeline] { 00:00:15.413 [Pipeline] retry 00:00:15.414 [Pipeline] { 00:00:15.419 [Pipeline] retry 00:00:15.420 [Pipeline] { 00:00:15.425 [Pipeline] retry 00:00:15.426 [Pipeline] { 00:00:15.430 [Pipeline] retry 00:00:15.432 [Pipeline] { 00:00:15.436 [Pipeline] retry 00:00:15.438 [Pipeline] { 00:00:15.442 [Pipeline] retry 00:00:15.444 [Pipeline] { 00:00:15.448 [Pipeline] retry 00:00:15.450 [Pipeline] { 00:00:15.454 [Pipeline] retry 00:00:15.455 [Pipeline] { 00:00:15.458 [Pipeline] retry 00:00:15.459 [Pipeline] { 00:00:15.462 [Pipeline] retry 00:00:15.463 [Pipeline] { 00:00:15.466 [Pipeline] retry 00:00:15.467 [Pipeline] { 00:00:15.470 [Pipeline] retry 00:00:15.471 [Pipeline] { 00:00:15.474 [Pipeline] retry 00:00:15.475 [Pipeline] { 00:00:15.478 [Pipeline] retry 00:00:15.480 [Pipeline] { 00:00:15.482 [Pipeline] retry 00:00:15.483 [Pipeline] { 00:00:15.487 [Pipeline] retry 00:00:15.488 [Pipeline] { 00:00:15.491 [Pipeline] retry 00:00:15.492 [Pipeline] { 00:00:15.495 [Pipeline] retry 00:00:15.496 [Pipeline] { 00:00:15.499 [Pipeline] retry 00:00:15.501 [Pipeline] { 00:00:15.504 [Pipeline] retry 00:00:15.506 [Pipeline] { 00:00:15.509 [Pipeline] retry 00:00:15.511 [Pipeline] { 00:00:15.514 [Pipeline] retry 00:00:15.515 [Pipeline] { 00:00:15.518 [Pipeline] retry 00:00:15.519 [Pipeline] { 00:00:15.521 [Pipeline] retry 00:00:15.523 [Pipeline] { 00:00:15.525 [Pipeline] retry 00:00:15.526 [Pipeline] { 00:00:15.529 [Pipeline] retry 00:00:15.531 [Pipeline] { 00:00:15.534 [Pipeline] retry 00:00:15.535 [Pipeline] { 00:00:15.538 [Pipeline] retry 00:00:15.540 [Pipeline] { 00:00:15.542 [Pipeline] retry 00:00:15.544 [Pipeline] { 00:00:15.547 [Pipeline] retry 00:00:15.548 [Pipeline] { 00:00:15.551 [Pipeline] retry 00:00:15.552 [Pipeline] { 00:00:15.555 [Pipeline] retry 00:00:15.556 [Pipeline] { 00:00:15.578 [Pipeline] build 00:00:15.580 Scheduling project: check-format-docker-autotest 00:00:15.585 [Pipeline] build 00:00:15.587 Scheduling project: check-so-deps-docker-autotest 00:00:15.592 [Pipeline] build 00:00:15.594 Scheduling project: doc-docker-autotest 00:00:15.599 [Pipeline] build 00:00:15.601 Scheduling project: build-files-docker-autotest 00:00:15.606 [Pipeline] build 00:00:15.607 Scheduling project: unittest-clang-docker-autotest 00:00:15.612 [Pipeline] build 00:00:15.614 Scheduling project: unittest-main-docker-autotest 00:00:15.619 [Pipeline] build 00:00:15.621 Scheduling project: unittest-valgrind-docker-autotest 00:00:15.626 [Pipeline] build 00:00:15.628 Scheduling project: scan-build-docker-autotest 00:00:15.632 [Pipeline] build 00:00:15.634 Scheduling project: packaging-docker-autotest 00:00:15.640 [Pipeline] build 00:00:15.641 Scheduling project: autobuild-external-vg-autotest 00:00:15.646 [Pipeline] build 00:00:15.648 Scheduling project: lvol-vg-autotest 00:00:15.652 [Pipeline] build 00:00:15.654 Scheduling project: vhost-initiator-vg-autotest 00:00:15.659 [Pipeline] build 00:00:15.661 Scheduling project: nvmf-tcp-vg-autotest 00:00:15.667 [Pipeline] build 00:00:15.668 Scheduling project: nvmf-tcp-uring-vg-autotest 00:00:15.673 [Pipeline] build 00:00:15.675 Scheduling project: nvme-vg-autotest 00:00:15.680 [Pipeline] build 00:00:15.682 Scheduling project: centos7-vg-autotest 00:00:15.687 [Pipeline] build 00:00:15.689 Scheduling project: rocky9-vg-autotest 00:00:15.694 [Pipeline] build 00:00:15.696 Scheduling project: ubuntu20-vg-autotest 00:00:15.701 [Pipeline] build 00:00:15.703 Scheduling project: ubuntu22-vg-autotest 00:00:15.708 [Pipeline] build 00:00:15.710 Scheduling project: freebsd-vg-autotest 00:00:15.714 [Pipeline] build 00:00:15.716 Scheduling project: iscsi-vg-autotest 00:00:15.725 [Pipeline] build 00:00:15.727 Scheduling project: iscsi-uring-vg-autotest 00:00:15.733 [Pipeline] build 00:00:15.735 Scheduling project: zns-vg-autotest 00:00:15.740 [Pipeline] build 00:00:15.742 Scheduling project: nvme-cmb-pmr-vg-autotest 00:00:15.747 [Pipeline] build 00:00:15.748 Scheduling project: BlobFS-phy-autotest 00:00:15.753 [Pipeline] build 00:00:15.755 Scheduling project: crypto-phy-autotest 00:00:15.760 [Pipeline] build 00:00:15.762 Scheduling project: vhost-phy-autotest 00:00:15.767 [Pipeline] build 00:00:15.769 Scheduling project: nvmf-phy-autotest 00:00:15.774 [Pipeline] build 00:00:15.776 Scheduling project: nvmf-cvl-phy-autotest 00:00:15.781 [Pipeline] build 00:00:15.783 Scheduling project: nvmf-tcp-phy-autotest 00:00:15.788 [Pipeline] build 00:00:15.790 Scheduling project: nvme-phy-autotest 00:00:15.795 [Pipeline] build 00:00:15.797 Scheduling project: vfio-user-phy-autotest 00:00:15.802 [Pipeline] build 00:00:15.804 Scheduling project: short-fuzz-phy-autotest 00:00:15.809 [Pipeline] build 00:00:15.811 Scheduling project: dsa-phy-autotest 00:00:15.816 [Pipeline] build 00:00:15.817 Scheduling project: vmd-phy-autotest 00:00:25.102 Starting building: vmd-phy-autotest #2617 00:00:25.106 Starting building: dsa-phy-autotest #4877 00:00:25.110 Starting building: vfio-user-phy-autotest #26185 00:00:25.113 Starting building: doc-docker-autotest #22305 00:00:25.116 Starting building: scan-build-docker-autotest #19845 00:00:25.118 Starting building: vhost-phy-autotest #26202 00:00:25.122 Starting building: nvmf-cvl-phy-autotest #3015 00:00:25.124 Starting building: check-format-docker-autotest #22125 00:00:25.128 Starting building: autobuild-external-vg-autotest #19805 00:00:25.132 Starting building: check-so-deps-docker-autotest #22138 00:00:25.134 Starting building: rocky9-vg-autotest #16067 00:00:25.137 Starting building: build-files-docker-autotest #22114 00:00:25.141 Starting building: short-fuzz-phy-autotest #26174 00:00:25.143 Starting building: unittest-main-docker-autotest #19843 00:00:25.149 Starting building: BlobFS-phy-autotest #26183 00:00:25.152 Starting building: crypto-phy-autotest #26177 00:00:25.156 Starting building: vhost-initiator-vg-autotest #77477 00:00:25.159 Starting building: centos7-vg-autotest #73636 00:00:25.162 Starting building: ubuntu20-vg-autotest #73776 00:00:25.165 Starting building: unittest-valgrind-docker-autotest #19846 00:00:25.168 Starting building: packaging-docker-autotest #19844 00:00:25.171 Starting building: nvme-phy-autotest #72349 00:00:25.173 Starting building: unittest-clang-docker-autotest #19842 00:00:25.176 Starting building: nvme-vg-autotest #77457 00:00:25.179 Starting building: ubuntu22-vg-autotest #37790 00:00:25.182 Starting building: nvmf-tcp-uring-vg-autotest #77382 00:00:25.185 Starting building: nvmf-phy-autotest #72421 00:00:25.188 Starting building: zns-vg-autotest #76563 00:00:25.191 Starting building: lvol-vg-autotest #77406 00:00:25.194 Starting building: nvmf-tcp-vg-autotest #77448 00:00:25.197 Starting building: iscsi-uring-vg-autotest #77362 00:00:25.199 Starting building: iscsi-vg-autotest #77495 00:00:25.202 Starting building: nvmf-tcp-phy-autotest #72308 00:00:25.205 Starting building: freebsd-vg-autotest #59105 00:00:25.207 Starting building: nvme-cmb-pmr-vg-autotest #78580 00:01:32.569 Build check-format-docker-autotest #22125 completed: SUCCESS 00:01:32.573 [Pipeline] } 00:01:32.626 [Pipeline] // retry 00:01:32.633 [Pipeline] } 00:01:38.040 Build doc-docker-autotest #22305 completed: SUCCESS 00:01:38.044 [Pipeline] } 00:01:38.095 [Pipeline] // retry 00:01:38.101 [Pipeline] } 00:02:48.265 Build build-files-docker-autotest #22114 completed: SUCCESS 00:02:48.269 [Pipeline] } 00:02:48.305 [Pipeline] // retry 00:02:48.313 [Pipeline] } 00:03:43.028 Build check-so-deps-docker-autotest #22138 completed: SUCCESS 00:03:43.031 [Pipeline] } 00:03:43.082 [Pipeline] // retry 00:03:43.089 [Pipeline] } 00:03:55.001 Build unittest-clang-docker-autotest #19842 completed: SUCCESS 00:03:55.003 [Pipeline] } 00:03:55.046 [Pipeline] // retry 00:03:55.052 [Pipeline] } 00:05:39.848 Build scan-build-docker-autotest #19845 completed: SUCCESS 00:05:39.851 [Pipeline] } 00:05:39.892 [Pipeline] // retry 00:05:39.899 [Pipeline] } 00:05:45.624 Build packaging-docker-autotest #19844 completed: SUCCESS 00:05:45.627 [Pipeline] } 00:05:45.675 [Pipeline] // retry 00:05:45.682 [Pipeline] } 00:06:02.095 Build unittest-main-docker-autotest #19843 completed: SUCCESS 00:06:02.098 [Pipeline] } 00:06:02.151 [Pipeline] // retry 00:06:02.158 [Pipeline] } 00:08:33.226 Build short-fuzz-phy-autotest #26174 completed: SUCCESS 00:08:33.229 [Pipeline] } 00:08:33.285 [Pipeline] // retry 00:08:33.295 [Pipeline] } 00:09:43.252 Build vmd-phy-autotest #2617 completed: SUCCESS 00:09:43.255 [Pipeline] } 00:09:43.301 [Pipeline] // retry 00:09:43.308 [Pipeline] } 00:10:15.236 Build unittest-valgrind-docker-autotest #19846 completed: SUCCESS 00:10:15.238 [Pipeline] } 00:10:15.288 [Pipeline] // retry 00:10:15.295 [Pipeline] } 00:15:21.166 Build autobuild-external-vg-autotest #19805 completed: SUCCESS 00:15:21.168 [Pipeline] } 00:15:21.222 [Pipeline] // retry 00:15:21.229 [Pipeline] } 00:17:34.492 Build nvme-phy-autotest #72349 completed: FAILURE 00:17:35.400 [Pipeline] echo 00:17:35.401 No retry patterns found. 00:17:35.402 [Pipeline] } 00:17:35.432 [Pipeline] // retry 00:17:35.442 [Pipeline] error 00:17:35.447 [Pipeline] } 00:17:35.452 Failed in branch nvme-phy-autotest 00:18:51.453 Build vhost-initiator-vg-autotest #77477 completed: SUCCESS 00:18:51.455 [Pipeline] } 00:18:51.511 [Pipeline] // retry 00:18:51.516 [Pipeline] } 00:19:38.860 Build vfio-user-phy-autotest #26185 completed: SUCCESS 00:19:38.861 [Pipeline] } 00:19:38.886 [Pipeline] // retry 00:19:38.890 [Pipeline] } 00:21:04.182 Build zns-vg-autotest #76563 completed: SUCCESS 00:21:04.185 [Pipeline] } 00:21:04.240 [Pipeline] // retry 00:21:04.247 [Pipeline] } 00:22:39.013 Build nvme-cmb-pmr-vg-autotest #78580 completed: SUCCESS 00:22:39.015 [Pipeline] } 00:22:39.063 [Pipeline] // retry 00:22:39.069 [Pipeline] } 00:23:14.786 Build nvme-vg-autotest #77457 completed: FAILURE 00:23:15.510 [Pipeline] echo 00:23:15.511 No retry patterns found. 00:23:15.512 [Pipeline] } 00:23:15.555 [Pipeline] // retry 00:23:15.564 [Pipeline] error 00:23:15.570 [Pipeline] } 00:23:15.575 Failed in branch nvme-vg-autotest 00:26:16.413 Build BlobFS-phy-autotest #26183 completed: SUCCESS 00:26:16.415 [Pipeline] } 00:26:16.465 [Pipeline] // retry 00:26:16.471 [Pipeline] } 00:28:14.339 Build lvol-vg-autotest #77406 completed: SUCCESS 00:28:14.342 [Pipeline] } 00:28:14.400 [Pipeline] // retry 00:28:14.407 [Pipeline] } 00:31:35.758 Build vhost-phy-autotest #26202 completed: SUCCESS 00:31:35.761 [Pipeline] } 00:31:35.811 [Pipeline] // retry 00:31:35.818 [Pipeline] } 00:31:55.439 Build crypto-phy-autotest #26177 completed: SUCCESS 00:31:55.442 [Pipeline] } 00:31:55.493 [Pipeline] // retry 00:31:55.500 [Pipeline] } 00:33:37.414 Build freebsd-vg-autotest #59105 completed: SUCCESS 00:33:37.417 [Pipeline] } 00:33:37.482 [Pipeline] // retry 00:33:37.489 [Pipeline] } 00:35:13.029 Build nvmf-tcp-phy-autotest #72308 completed: SUCCESS 00:35:13.031 [Pipeline] } 00:35:13.080 [Pipeline] // retry 00:35:13.087 [Pipeline] } 00:35:55.273 Build nvmf-tcp-uring-vg-autotest #77382 completed: SUCCESS 00:35:55.275 [Pipeline] } 00:35:55.307 [Pipeline] // retry 00:35:55.313 [Pipeline] } 00:37:19.998 Build nvmf-cvl-phy-autotest #3015 completed: SUCCESS 00:37:20.001 [Pipeline] } 00:37:20.045 [Pipeline] // retry 00:37:20.052 [Pipeline] } 00:38:15.959 Build iscsi-uring-vg-autotest #77362 completed: SUCCESS 00:38:15.962 [Pipeline] } 00:38:16.016 [Pipeline] // retry 00:38:16.024 [Pipeline] } 00:41:12.127 Build nvmf-phy-autotest #72421 completed: ABORTED 00:41:13.373 [Pipeline] echo 00:41:13.374 No retry patterns found. 00:41:13.375 [Pipeline] } 00:41:13.407 [Pipeline] // retry 00:41:13.414 [Pipeline] error 00:41:13.420 [Pipeline] } 00:41:13.424 Failed in branch nvmf-phy-autotest 00:41:58.259 Build dsa-phy-autotest #4877 completed: SUCCESS 00:41:58.262 [Pipeline] } 00:41:58.306 [Pipeline] // retry 00:41:58.313 [Pipeline] } 00:42:20.944 Build nvmf-tcp-vg-autotest #77448 completed: SUCCESS 00:42:20.947 [Pipeline] } 00:42:21.004 [Pipeline] // retry 00:42:21.011 [Pipeline] } 00:42:39.052 Build iscsi-vg-autotest #77495 completed: SUCCESS 00:42:39.054 [Pipeline] } 00:42:39.100 [Pipeline] // retry 00:42:39.107 [Pipeline] } 00:50:58.282 Build centos7-vg-autotest #73636 completed: SUCCESS 00:50:58.283 [Pipeline] } 00:50:58.313 [Pipeline] // retry 00:50:58.319 [Pipeline] } 00:56:04.508 Build ubuntu20-vg-autotest #73776 completed: FAILURE 00:56:06.178 [Pipeline] echo 00:56:06.179 No retry patterns found. 00:56:06.180 [Pipeline] } 00:56:06.211 [Pipeline] // retry 00:56:06.217 [Pipeline] error 00:56:06.222 [Pipeline] } 00:56:06.224 Failed in branch ubuntu20-vg-autotest 01:02:27.079 Build rocky9-vg-autotest #16067 completed: SUCCESS 01:02:27.082 [Pipeline] } 01:02:27.111 [Pipeline] // retry 01:02:27.117 [Pipeline] } 01:09:08.082 Build ubuntu22-vg-autotest #37790 completed: SUCCESS 01:09:08.085 [Pipeline] } 01:09:08.114 [Pipeline] // retry 01:09:08.120 [Pipeline] } 01:09:08.376 [Pipeline] // parallel 01:09:08.385 [Pipeline] } 01:09:08.437 [Pipeline] // stage 01:09:08.444 [Pipeline] } 01:09:08.451 ERROR: Build nvme-phy-autotest #72349 failed 01:09:08.467 [Pipeline] // catchError 01:09:08.473 [Pipeline] stage 01:09:08.474 [Pipeline] { (Autorun Post and Coverage) 01:09:08.490 [Pipeline] setCustomBuildProperty 01:09:08.504 [Pipeline] dir 01:09:08.504 Running in /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125 01:09:08.505 [Pipeline] { 01:09:08.521 [Pipeline] copyArtifacts 01:09:08.581 Copied 5 artifacts from "check-format-docker-autotest" build number 22125 01:09:08.584 [Pipeline] writeFile 01:09:08.606 [Pipeline] } 01:09:08.637 [Pipeline] // dir 01:09:08.654 [Pipeline] dir 01:09:08.654 Running in /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305 01:09:08.656 [Pipeline] { 01:09:08.679 [Pipeline] copyArtifacts 01:09:08.953 Copied 6 artifacts from "doc-docker-autotest" build number 22305 01:09:08.956 [Pipeline] writeFile 01:09:08.972 [Pipeline] } 01:09:08.992 [Pipeline] // dir 01:09:09.058 [Pipeline] dir 01:09:09.059 Running in /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114 01:09:09.060 [Pipeline] { 01:09:09.081 [Pipeline] copyArtifacts 01:09:09.156 Copied 5 artifacts from "build-files-docker-autotest" build number 22114 01:09:09.159 [Pipeline] writeFile 01:09:09.205 [Pipeline] } 01:09:09.238 [Pipeline] // dir 01:09:09.277 [Pipeline] dir 01:09:09.277 Running in /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138 01:09:09.278 [Pipeline] { 01:09:09.298 [Pipeline] copyArtifacts 01:09:09.368 Copied 5 artifacts from "check-so-deps-docker-autotest" build number 22138 01:09:09.371 [Pipeline] writeFile 01:09:09.397 [Pipeline] } 01:09:09.426 [Pipeline] // dir 01:09:09.548 [Pipeline] dir 01:09:09.548 Running in /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842 01:09:09.549 [Pipeline] { 01:09:09.575 [Pipeline] copyArtifacts 01:09:09.645 Copied 5 artifacts from "unittest-clang-docker-autotest" build number 19842 01:09:09.647 [Pipeline] writeFile 01:09:09.715 [Pipeline] } 01:09:09.755 [Pipeline] // dir 01:09:09.771 [Pipeline] dir 01:09:09.771 Running in /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845 01:09:09.772 [Pipeline] { 01:09:09.790 [Pipeline] copyArtifacts 01:09:09.926 Copied 9 artifacts from "scan-build-docker-autotest" build number 19845 01:09:09.929 [Pipeline] writeFile 01:09:09.950 [Pipeline] } 01:09:09.980 [Pipeline] // dir 01:09:10.294 [Pipeline] dir 01:09:10.295 Running in /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844 01:09:10.296 [Pipeline] { 01:09:10.335 [Pipeline] copyArtifacts 01:09:10.405 Copied 5 artifacts from "packaging-docker-autotest" build number 19844 01:09:10.408 [Pipeline] writeFile 01:09:10.553 [Pipeline] } 01:09:10.621 [Pipeline] // dir 01:09:10.739 [Pipeline] dir 01:09:10.739 Running in /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843 01:09:10.740 [Pipeline] { 01:09:10.763 [Pipeline] copyArtifacts 01:09:11.013 Copied 6 artifacts from "unittest-main-docker-autotest" build number 19843 01:09:11.017 [Pipeline] writeFile 01:09:11.120 [Pipeline] } 01:09:11.162 [Pipeline] // dir 01:09:11.446 [Pipeline] dir 01:09:11.446 Running in /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174 01:09:11.447 [Pipeline] { 01:09:11.485 [Pipeline] copyArtifacts 01:09:11.588 Copied 10 artifacts from "short-fuzz-phy-autotest" build number 26174 01:09:11.591 [Pipeline] writeFile 01:09:11.775 [Pipeline] } 01:09:11.878 [Pipeline] // dir 01:09:12.123 [Pipeline] dir 01:09:12.123 Running in /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617 01:09:12.124 [Pipeline] { 01:09:12.158 [Pipeline] copyArtifacts 01:09:12.366 Copied 12 artifacts from "vmd-phy-autotest" build number 2617 01:09:12.370 [Pipeline] writeFile 01:09:12.503 [Pipeline] } 01:09:12.568 [Pipeline] // dir 01:09:12.877 [Pipeline] dir 01:09:12.877 Running in /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846 01:09:12.878 [Pipeline] { 01:09:12.914 [Pipeline] copyArtifacts 01:09:13.194 Copied 6 artifacts from "unittest-valgrind-docker-autotest" build number 19846 01:09:13.197 [Pipeline] writeFile 01:09:13.335 [Pipeline] } 01:09:13.428 [Pipeline] // dir 01:09:13.519 [Pipeline] dir 01:09:13.519 Running in /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805 01:09:13.520 [Pipeline] { 01:09:13.543 [Pipeline] copyArtifacts 01:09:13.642 Copied 8 artifacts from "autobuild-external-vg-autotest" build number 19805 01:09:13.645 [Pipeline] writeFile 01:09:13.713 [Pipeline] } 01:09:13.755 [Pipeline] // dir 01:09:14.267 [Pipeline] dir 01:09:14.267 Running in /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349 01:09:14.268 [Pipeline] { 01:09:14.312 [Pipeline] copyArtifacts 01:09:14.691 Copied 11 artifacts from "nvme-phy-autotest" build number 72349 01:09:14.694 [Pipeline] writeFile 01:09:14.905 [Pipeline] } 01:09:15.024 [Pipeline] // dir 01:09:15.297 [Pipeline] dir 01:09:15.297 Running in /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477 01:09:15.298 [Pipeline] { 01:09:15.334 [Pipeline] copyArtifacts 01:09:15.625 Copied 16 artifacts from "vhost-initiator-vg-autotest" build number 77477 01:09:15.629 [Pipeline] writeFile 01:09:15.766 [Pipeline] } 01:09:15.841 [Pipeline] // dir 01:09:16.272 [Pipeline] dir 01:09:16.273 Running in /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185 01:09:16.274 [Pipeline] { 01:09:16.320 [Pipeline] copyArtifacts 01:09:16.550 Copied 12 artifacts from "vfio-user-phy-autotest" build number 26185 01:09:16.553 [Pipeline] writeFile 01:09:16.792 [Pipeline] } 01:09:16.907 [Pipeline] // dir 01:09:17.192 [Pipeline] dir 01:09:17.192 Running in /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563 01:09:17.193 [Pipeline] { 01:09:17.230 [Pipeline] copyArtifacts 01:09:17.445 Copied 12 artifacts from "zns-vg-autotest" build number 76563 01:09:17.450 [Pipeline] writeFile 01:09:17.596 [Pipeline] } 01:09:17.671 [Pipeline] // dir 01:09:17.987 [Pipeline] dir 01:09:17.987 Running in /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580 01:09:17.989 [Pipeline] { 01:09:18.029 [Pipeline] copyArtifacts 01:09:18.250 Copied 12 artifacts from "nvme-cmb-pmr-vg-autotest" build number 78580 01:09:18.254 [Pipeline] writeFile 01:09:18.428 [Pipeline] } 01:09:18.517 [Pipeline] // dir 01:09:18.926 [Pipeline] dir 01:09:18.926 Running in /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457 01:09:18.927 [Pipeline] { 01:09:18.969 [Pipeline] copyArtifacts 01:09:19.234 Copied 11 artifacts from "nvme-vg-autotest" build number 77457 01:09:19.239 [Pipeline] writeFile 01:09:19.422 [Pipeline] } 01:09:19.543 [Pipeline] // dir 01:09:19.832 [Pipeline] dir 01:09:19.832 Running in /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183 01:09:19.833 [Pipeline] { 01:09:19.864 [Pipeline] copyArtifacts 01:09:20.287 Copied 28 artifacts from "BlobFS-phy-autotest" build number 26183 01:09:20.292 [Pipeline] writeFile 01:09:20.422 [Pipeline] } 01:09:20.506 [Pipeline] // dir 01:09:21.034 [Pipeline] dir 01:09:21.034 Running in /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406 01:09:21.035 [Pipeline] { 01:09:21.075 [Pipeline] copyArtifacts 01:09:21.296 Copied 13 artifacts from "lvol-vg-autotest" build number 77406 01:09:21.301 [Pipeline] writeFile 01:09:21.548 [Pipeline] } 01:09:21.693 [Pipeline] // dir 01:09:22.278 [Pipeline] dir 01:09:22.278 Running in /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202 01:09:22.279 [Pipeline] { 01:09:22.334 [Pipeline] copyArtifacts 01:09:22.587 Copied 13 artifacts from "vhost-phy-autotest" build number 26202 01:09:22.591 [Pipeline] writeFile 01:09:22.843 [Pipeline] } 01:09:22.958 [Pipeline] // dir 01:09:24.141 [Pipeline] dir 01:09:24.141 Running in /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177 01:09:24.142 [Pipeline] { 01:09:24.225 [Pipeline] copyArtifacts 01:09:24.469 Copied 12 artifacts from "crypto-phy-autotest" build number 26177 01:09:24.473 [Pipeline] writeFile 01:09:24.969 [Pipeline] } 01:09:25.188 [Pipeline] // dir 01:09:25.715 [Pipeline] dir 01:09:25.716 Running in /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105 01:09:25.716 [Pipeline] { 01:09:25.764 [Pipeline] copyArtifacts 01:09:25.872 Copied 7 artifacts from "freebsd-vg-autotest" build number 59105 01:09:25.875 [Pipeline] writeFile 01:09:26.159 [Pipeline] } 01:09:26.283 [Pipeline] // dir 01:09:27.436 [Pipeline] dir 01:09:27.436 Running in /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308 01:09:27.460 [Pipeline] { 01:09:27.552 [Pipeline] copyArtifacts 01:09:27.941 Copied 14 artifacts from "nvmf-tcp-phy-autotest" build number 72308 01:09:27.944 [Pipeline] writeFile 01:09:28.544 [Pipeline] } 01:09:28.782 [Pipeline] // dir 01:09:29.482 [Pipeline] dir 01:09:29.482 Running in /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382 01:09:29.483 [Pipeline] { 01:09:29.543 [Pipeline] copyArtifacts 01:09:29.760 Copied 14 artifacts from "nvmf-tcp-uring-vg-autotest" build number 77382 01:09:29.763 [Pipeline] writeFile 01:09:30.124 [Pipeline] } 01:09:30.308 [Pipeline] // dir 01:09:31.095 [Pipeline] dir 01:09:31.095 Running in /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015 01:09:31.096 [Pipeline] { 01:09:31.173 [Pipeline] copyArtifacts 01:09:31.569 Copied 14 artifacts from "nvmf-cvl-phy-autotest" build number 3015 01:09:31.572 [Pipeline] writeFile 01:09:31.939 [Pipeline] } 01:09:32.116 [Pipeline] // dir 01:09:33.847 [Pipeline] dir 01:09:33.848 Running in /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362 01:09:33.849 [Pipeline] { 01:09:34.020 [Pipeline] copyArtifacts 01:09:36.822 Copied 17 artifacts from "iscsi-uring-vg-autotest" build number 77362 01:09:36.825 [Pipeline] writeFile 01:09:37.812 [Pipeline] } 01:09:38.245 [Pipeline] // dir 01:09:39.022 [Pipeline] dir 01:09:39.023 Running in /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421 01:09:39.023 [Pipeline] { 01:09:39.087 [Pipeline] copyArtifacts 01:09:39.505 Copied 14 artifacts from "nvmf-phy-autotest" build number 72421 01:09:39.507 [Pipeline] writeFile 01:09:39.862 [Pipeline] } 01:09:40.036 [Pipeline] // dir 01:09:40.816 [Pipeline] dir 01:09:40.817 Running in /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877 01:09:40.817 [Pipeline] { 01:09:40.879 [Pipeline] copyArtifacts 01:09:41.280 Copied 14 artifacts from "dsa-phy-autotest" build number 4877 01:09:41.283 [Pipeline] writeFile 01:09:41.676 [Pipeline] } 01:09:41.846 [Pipeline] // dir 01:09:42.637 [Pipeline] dir 01:09:42.637 Running in /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448 01:09:42.638 [Pipeline] { 01:09:42.696 [Pipeline] copyArtifacts 01:09:42.951 Copied 14 artifacts from "nvmf-tcp-vg-autotest" build number 77448 01:09:42.955 [Pipeline] writeFile 01:09:43.369 [Pipeline] } 01:09:43.544 [Pipeline] // dir 01:09:45.395 [Pipeline] dir 01:09:45.396 Running in /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495 01:09:45.397 [Pipeline] { 01:09:45.573 [Pipeline] copyArtifacts 01:09:49.226 Copied 16 artifacts from "iscsi-vg-autotest" build number 77495 01:09:49.230 [Pipeline] writeFile 01:09:50.504 [Pipeline] } 01:09:51.009 [Pipeline] // dir 01:09:51.863 [Pipeline] dir 01:09:51.863 Running in /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636 01:09:51.864 [Pipeline] { 01:09:51.919 [Pipeline] copyArtifacts 01:09:52.313 Copied 13 artifacts from "centos7-vg-autotest" build number 73636 01:09:52.321 [Pipeline] writeFile 01:09:52.663 [Pipeline] } 01:09:52.792 [Pipeline] // dir 01:09:53.752 [Pipeline] dir 01:09:53.753 Running in /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776 01:09:53.754 [Pipeline] { 01:09:53.827 [Pipeline] copyArtifacts 01:09:54.492 Copied 12 artifacts from "ubuntu20-vg-autotest" build number 73776 01:09:54.495 [Pipeline] writeFile 01:09:54.955 [Pipeline] } 01:09:55.148 [Pipeline] // dir 01:09:56.135 [Pipeline] dir 01:09:56.135 Running in /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067 01:09:56.136 [Pipeline] { 01:09:56.220 [Pipeline] copyArtifacts 01:09:56.578 Copied 13 artifacts from "rocky9-vg-autotest" build number 16067 01:09:56.581 [Pipeline] writeFile 01:09:57.035 [Pipeline] } 01:09:57.307 [Pipeline] // dir 01:09:58.537 [Pipeline] dir 01:09:58.537 Running in /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790 01:09:58.538 [Pipeline] { 01:09:58.612 [Pipeline] copyArtifacts 01:09:58.987 Copied 13 artifacts from "ubuntu22-vg-autotest" build number 37790 01:09:58.990 [Pipeline] writeFile 01:09:59.446 [Pipeline] } 01:09:59.587 [Pipeline] // dir 01:09:59.595 [Pipeline] catchError 01:09:59.596 [Pipeline] { 01:09:59.611 [Pipeline] sh 01:09:59.888 + jbp/jenkins/jjb-config/jobs/scripts/post_gen_coverage.sh 01:09:59.888 + shopt -s globstar nullglob 01:09:59.888 + echo 'Start stage post_gen_coverage.sh' 01:09:59.888 Start stage post_gen_coverage.sh 01:09:59.888 + cd /var/jenkins/workspace/autotest-nightly 01:09:59.888 + rm -rf /var/jenkins/workspace/autotest-nightly/spdk/doc 01:09:59.888 + trap 'compress_coverage_and_docs; remove_partial_coverage_files && echo '\''End stage post_gen_coverage.sh'\''' EXIT 01:09:59.888 + move_artifacts 01:09:59.888 + local out_dirs 01:09:59.888 + out_dirs=(./**/output/) 01:09:59.888 + for dir in "${out_dirs[@]}" 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./autobuild-external-vg-autotest_19805/output//doc.tar.xz ]] 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./autobuild-external-vg-autotest_19805/output//ut_coverage.tar.xz ]] 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./autobuild-external-vg-autotest_19805/output//llvm.tar.xz ]] 01:09:59.888 + mv ./autobuild-external-vg-autotest_19805/output//build-pkg-manifest.txt ./autobuild-external-vg-autotest_19805/output//build-repo-manifest.txt ./autobuild-external-vg-autotest_19805/output//dmesg.init.log ./autobuild-external-vg-autotest_19805/output//dmesg.log ./autobuild-external-vg-autotest_19805/output//power.tar.xz ./autobuild-external-vg-autotest_19805/output//test_completions.txt ./autobuild-external-vg-autotest_19805/output//timing.svg ./autobuild-external-vg-autotest_19805/output//timing.txt ./autobuild-external-vg-autotest_19805/output//.. 01:09:59.888 + rmdir ./autobuild-external-vg-autotest_19805/output/ 01:09:59.888 + for dir in "${out_dirs[@]}" 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./BlobFS-phy-autotest_26183/output//doc.tar.xz ]] 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./BlobFS-phy-autotest_26183/output//ut_coverage.tar.xz ]] 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./BlobFS-phy-autotest_26183/output//llvm.tar.xz ]] 01:09:59.888 + mv ./BlobFS-phy-autotest_26183/output//all_tests.txt ./BlobFS-phy-autotest_26183/output//build-pkg-manifest.txt ./BlobFS-phy-autotest_26183/output//build-repo-manifest.txt ./BlobFS-phy-autotest_26183/output//cov_base.info.xz ./BlobFS-phy-autotest_26183/output//cov_test.info.xz ./BlobFS-phy-autotest_26183/output//cov_total.info.xz ./BlobFS-phy-autotest_26183/output//dmesg.init.log ./BlobFS-phy-autotest_26183/output//dmesg.log ./BlobFS-phy-autotest_26183/output//power.tar.xz ./BlobFS-phy-autotest_26183/output//rocksdb ./BlobFS-phy-autotest_26183/output//test_completions.txt ./BlobFS-phy-autotest_26183/output//timing.svg ./BlobFS-phy-autotest_26183/output//timing.txt ./BlobFS-phy-autotest_26183/output//.. 01:09:59.888 + rmdir ./BlobFS-phy-autotest_26183/output/ 01:09:59.888 + for dir in "${out_dirs[@]}" 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./build-files-docker-autotest_22114/output//doc.tar.xz ]] 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./build-files-docker-autotest_22114/output//ut_coverage.tar.xz ]] 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./build-files-docker-autotest_22114/output//llvm.tar.xz ]] 01:09:59.888 + mv ./build-files-docker-autotest_22114/output//build-pkg-manifest.txt ./build-files-docker-autotest_22114/output//build-repo-manifest.txt ./build-files-docker-autotest_22114/output//power.tar.xz ./build-files-docker-autotest_22114/output//test_completions.txt ./build-files-docker-autotest_22114/output//timing.txt ./build-files-docker-autotest_22114/output//.. 01:09:59.888 + rmdir ./build-files-docker-autotest_22114/output/ 01:09:59.888 + for dir in "${out_dirs[@]}" 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./centos7-vg-autotest_73636/output//doc.tar.xz ]] 01:09:59.888 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:09:59.888 + [[ -f ./centos7-vg-autotest_73636/output//ut_coverage.tar.xz ]] 01:09:59.888 + tar -C ./centos7-vg-autotest_73636/output/ -xf ./centos7-vg-autotest_73636/output//ut_coverage.tar.xz 01:10:00.146 + rm ./centos7-vg-autotest_73636/output//ut_coverage.tar.xz 01:10:00.146 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.146 + [[ -f ./centos7-vg-autotest_73636/output//llvm.tar.xz ]] 01:10:00.147 + mv ./centos7-vg-autotest_73636/output//all_tests.txt ./centos7-vg-autotest_73636/output//build-pkg-manifest.txt ./centos7-vg-autotest_73636/output//build-repo-manifest.txt ./centos7-vg-autotest_73636/output//cov_base.info.xz ./centos7-vg-autotest_73636/output//cov_test.info.xz ./centos7-vg-autotest_73636/output//cov_total.info.xz ./centos7-vg-autotest_73636/output//dmesg.init.log ./centos7-vg-autotest_73636/output//dmesg.log ./centos7-vg-autotest_73636/output//power.tar.xz ./centos7-vg-autotest_73636/output//test_completions.txt ./centos7-vg-autotest_73636/output//timing.svg ./centos7-vg-autotest_73636/output//timing.txt ./centos7-vg-autotest_73636/output//ut_coverage ./centos7-vg-autotest_73636/output//.. 01:10:00.147 + rmdir ./centos7-vg-autotest_73636/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./check-format-docker-autotest_22125/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./check-format-docker-autotest_22125/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./check-format-docker-autotest_22125/output//llvm.tar.xz ]] 01:10:00.147 + mv ./check-format-docker-autotest_22125/output//build-pkg-manifest.txt ./check-format-docker-autotest_22125/output//build-repo-manifest.txt ./check-format-docker-autotest_22125/output//power.tar.xz ./check-format-docker-autotest_22125/output//test_completions.txt ./check-format-docker-autotest_22125/output//timing.txt ./check-format-docker-autotest_22125/output//.. 01:10:00.147 + rmdir ./check-format-docker-autotest_22125/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./check-so-deps-docker-autotest_22138/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./check-so-deps-docker-autotest_22138/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./check-so-deps-docker-autotest_22138/output//llvm.tar.xz ]] 01:10:00.147 + mv ./check-so-deps-docker-autotest_22138/output//build-pkg-manifest.txt ./check-so-deps-docker-autotest_22138/output//build-repo-manifest.txt ./check-so-deps-docker-autotest_22138/output//power.tar.xz ./check-so-deps-docker-autotest_22138/output//test_completions.txt ./check-so-deps-docker-autotest_22138/output//timing.txt ./check-so-deps-docker-autotest_22138/output//.. 01:10:00.147 + rmdir ./check-so-deps-docker-autotest_22138/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./crypto-phy-autotest_26177/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./crypto-phy-autotest_26177/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./crypto-phy-autotest_26177/output//llvm.tar.xz ]] 01:10:00.147 + mv ./crypto-phy-autotest_26177/output//all_tests.txt ./crypto-phy-autotest_26177/output//build-pkg-manifest.txt ./crypto-phy-autotest_26177/output//build-repo-manifest.txt ./crypto-phy-autotest_26177/output//cov_base.info.xz ./crypto-phy-autotest_26177/output//cov_test.info.xz ./crypto-phy-autotest_26177/output//cov_total.info.xz ./crypto-phy-autotest_26177/output//dmesg.init.log ./crypto-phy-autotest_26177/output//dmesg.log ./crypto-phy-autotest_26177/output//power.tar.xz ./crypto-phy-autotest_26177/output//test_completions.txt ./crypto-phy-autotest_26177/output//timing.svg ./crypto-phy-autotest_26177/output//timing.txt ./crypto-phy-autotest_26177/output//.. 01:10:00.147 + rmdir ./crypto-phy-autotest_26177/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./doc-docker-autotest_22305/output//doc.tar.xz ]] 01:10:00.147 + tar -C ./doc-docker-autotest_22305/output/ -xf ./doc-docker-autotest_22305/output//doc.tar.xz 01:10:00.147 + rm ./doc-docker-autotest_22305/output//doc.tar.xz 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./doc-docker-autotest_22305/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./doc-docker-autotest_22305/output//llvm.tar.xz ]] 01:10:00.147 + mv ./doc-docker-autotest_22305/output//build-pkg-manifest.txt ./doc-docker-autotest_22305/output//build-repo-manifest.txt ./doc-docker-autotest_22305/output//doc ./doc-docker-autotest_22305/output//power.tar.xz ./doc-docker-autotest_22305/output//test_completions.txt ./doc-docker-autotest_22305/output//timing.txt ./doc-docker-autotest_22305/output//.. 01:10:00.147 + rmdir ./doc-docker-autotest_22305/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./dsa-phy-autotest_4877/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./dsa-phy-autotest_4877/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./dsa-phy-autotest_4877/output//llvm.tar.xz ]] 01:10:00.147 + mv ./dsa-phy-autotest_4877/output//all_tests.txt ./dsa-phy-autotest_4877/output//build-pkg-manifest.txt ./dsa-phy-autotest_4877/output//build-repo-manifest.txt ./dsa-phy-autotest_4877/output//cov_base.info.xz ./dsa-phy-autotest_4877/output//cov_test.info.xz ./dsa-phy-autotest_4877/output//cov_total.info.xz ./dsa-phy-autotest_4877/output//dmesg.init.log ./dsa-phy-autotest_4877/output//dmesg.log ./dsa-phy-autotest_4877/output//nvmf_fuzz_tgt_output.txt ./dsa-phy-autotest_4877/output//nvmf_trace.0_shm.tar.gz ./dsa-phy-autotest_4877/output//power.tar.xz ./dsa-phy-autotest_4877/output//test_completions.txt ./dsa-phy-autotest_4877/output//timing.svg ./dsa-phy-autotest_4877/output//timing.txt ./dsa-phy-autotest_4877/output//.. 01:10:00.147 + rmdir ./dsa-phy-autotest_4877/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./freebsd-vg-autotest_59105/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./freebsd-vg-autotest_59105/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./freebsd-vg-autotest_59105/output//llvm.tar.xz ]] 01:10:00.147 + mv ./freebsd-vg-autotest_59105/output//all_tests.txt ./freebsd-vg-autotest_59105/output//build-pkg-manifest.txt ./freebsd-vg-autotest_59105/output//build-repo-manifest.txt ./freebsd-vg-autotest_59105/output//messages ./freebsd-vg-autotest_59105/output//power.tar.xz ./freebsd-vg-autotest_59105/output//test_completions.txt ./freebsd-vg-autotest_59105/output//timing.txt ./freebsd-vg-autotest_59105/output//.. 01:10:00.147 + rmdir ./freebsd-vg-autotest_59105/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./iscsi-uring-vg-autotest_77362/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./iscsi-uring-vg-autotest_77362/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./iscsi-uring-vg-autotest_77362/output//llvm.tar.xz ]] 01:10:00.147 + mv ./iscsi-uring-vg-autotest_77362/output//all_tests.txt ./iscsi-uring-vg-autotest_77362/output//build-kernel-manifest.txt ./iscsi-uring-vg-autotest_77362/output//build-pkg-manifest.txt ./iscsi-uring-vg-autotest_77362/output//build-repo-manifest.txt ./iscsi-uring-vg-autotest_77362/output//calsoft.json ./iscsi-uring-vg-autotest_77362/output//calsoft.tar.xz ./iscsi-uring-vg-autotest_77362/output//cov_base.info.xz ./iscsi-uring-vg-autotest_77362/output//cov_test.info.xz ./iscsi-uring-vg-autotest_77362/output//cov_total.info.xz ./iscsi-uring-vg-autotest_77362/output//dmesg.init.log ./iscsi-uring-vg-autotest_77362/output//dmesg.log ./iscsi-uring-vg-autotest_77362/output//iscsi_autofuzz_logs.txt ./iscsi-uring-vg-autotest_77362/output//iscsi_autofuzz_tgt_output.txt ./iscsi-uring-vg-autotest_77362/output//power.tar.xz ./iscsi-uring-vg-autotest_77362/output//test_completions.txt ./iscsi-uring-vg-autotest_77362/output//timing.svg ./iscsi-uring-vg-autotest_77362/output//timing.txt ./iscsi-uring-vg-autotest_77362/output//.. 01:10:00.147 + rmdir ./iscsi-uring-vg-autotest_77362/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./iscsi-vg-autotest_77495/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./iscsi-vg-autotest_77495/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./iscsi-vg-autotest_77495/output//llvm.tar.xz ]] 01:10:00.147 + mv ./iscsi-vg-autotest_77495/output//all_tests.txt ./iscsi-vg-autotest_77495/output//build-pkg-manifest.txt ./iscsi-vg-autotest_77495/output//build-repo-manifest.txt ./iscsi-vg-autotest_77495/output//calsoft.json ./iscsi-vg-autotest_77495/output//calsoft.tar.xz ./iscsi-vg-autotest_77495/output//cov_base.info.xz ./iscsi-vg-autotest_77495/output//cov_test.info.xz ./iscsi-vg-autotest_77495/output//cov_total.info.xz ./iscsi-vg-autotest_77495/output//dmesg.init.log ./iscsi-vg-autotest_77495/output//dmesg.log ./iscsi-vg-autotest_77495/output//iscsi_autofuzz_logs.txt ./iscsi-vg-autotest_77495/output//iscsi_autofuzz_tgt_output.txt ./iscsi-vg-autotest_77495/output//power.tar.xz ./iscsi-vg-autotest_77495/output//test_completions.txt ./iscsi-vg-autotest_77495/output//timing.svg ./iscsi-vg-autotest_77495/output//timing.txt ./iscsi-vg-autotest_77495/output//.. 01:10:00.147 + rmdir ./iscsi-vg-autotest_77495/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./lvol-vg-autotest_77406/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./lvol-vg-autotest_77406/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./lvol-vg-autotest_77406/output//llvm.tar.xz ]] 01:10:00.147 + mv ./lvol-vg-autotest_77406/output//all_tests.txt ./lvol-vg-autotest_77406/output//build-kernel-manifest.txt ./lvol-vg-autotest_77406/output//build-pkg-manifest.txt ./lvol-vg-autotest_77406/output//build-repo-manifest.txt ./lvol-vg-autotest_77406/output//cov_base.info.xz ./lvol-vg-autotest_77406/output//cov_test.info.xz ./lvol-vg-autotest_77406/output//cov_total.info.xz ./lvol-vg-autotest_77406/output//dmesg.init.log ./lvol-vg-autotest_77406/output//dmesg.log ./lvol-vg-autotest_77406/output//power.tar.xz ./lvol-vg-autotest_77406/output//test_completions.txt ./lvol-vg-autotest_77406/output//timing.svg ./lvol-vg-autotest_77406/output//timing.txt ./lvol-vg-autotest_77406/output//.. 01:10:00.147 + rmdir ./lvol-vg-autotest_77406/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./nvme-cmb-pmr-vg-autotest_78580/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./nvme-cmb-pmr-vg-autotest_78580/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./nvme-cmb-pmr-vg-autotest_78580/output//llvm.tar.xz ]] 01:10:00.147 + mv ./nvme-cmb-pmr-vg-autotest_78580/output//all_tests.txt ./nvme-cmb-pmr-vg-autotest_78580/output//build-pkg-manifest.txt ./nvme-cmb-pmr-vg-autotest_78580/output//build-repo-manifest.txt ./nvme-cmb-pmr-vg-autotest_78580/output//cov_base.info.xz ./nvme-cmb-pmr-vg-autotest_78580/output//cov_test.info.xz ./nvme-cmb-pmr-vg-autotest_78580/output//cov_total.info.xz ./nvme-cmb-pmr-vg-autotest_78580/output//dmesg.init.log ./nvme-cmb-pmr-vg-autotest_78580/output//dmesg.log ./nvme-cmb-pmr-vg-autotest_78580/output//power.tar.xz ./nvme-cmb-pmr-vg-autotest_78580/output//test_completions.txt ./nvme-cmb-pmr-vg-autotest_78580/output//timing.svg ./nvme-cmb-pmr-vg-autotest_78580/output//timing.txt ./nvme-cmb-pmr-vg-autotest_78580/output//.. 01:10:00.147 + rmdir ./nvme-cmb-pmr-vg-autotest_78580/output/ 01:10:00.147 + for dir in "${out_dirs[@]}" 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./nvme-phy-autotest_72349/output//doc.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./nvme-phy-autotest_72349/output//ut_coverage.tar.xz ]] 01:10:00.147 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.147 + [[ -f ./nvme-phy-autotest_72349/output//llvm.tar.xz ]] 01:10:00.147 + mv ./nvme-phy-autotest_72349/output//all_tests.txt ./nvme-phy-autotest_72349/output//build-pkg-manifest.txt ./nvme-phy-autotest_72349/output//build-repo-manifest.txt ./nvme-phy-autotest_72349/output//cov_base.info.xz ./nvme-phy-autotest_72349/output//dmesg.init.log ./nvme-phy-autotest_72349/output//dmesg.log ./nvme-phy-autotest_72349/output//power.tar.xz ./nvme-phy-autotest_72349/output//proc_list.txt ./nvme-phy-autotest_72349/output//test_completions.txt ./nvme-phy-autotest_72349/output//timing.txt ./nvme-phy-autotest_72349/output//udev.log ./nvme-phy-autotest_72349/output//.. 01:10:00.406 + rmdir ./nvme-phy-autotest_72349/output/ 01:10:00.406 + for dir in "${out_dirs[@]}" 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvme-vg-autotest_77457/output//doc.tar.xz ]] 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvme-vg-autotest_77457/output//ut_coverage.tar.xz ]] 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvme-vg-autotest_77457/output//llvm.tar.xz ]] 01:10:00.406 + mv ./nvme-vg-autotest_77457/output//all_tests.txt ./nvme-vg-autotest_77457/output//build-pkg-manifest.txt ./nvme-vg-autotest_77457/output//build-repo-manifest.txt ./nvme-vg-autotest_77457/output//cov_base.info.xz ./nvme-vg-autotest_77457/output//dmesg.init.log ./nvme-vg-autotest_77457/output//dmesg.log ./nvme-vg-autotest_77457/output//power.tar.xz ./nvme-vg-autotest_77457/output//proc_list.txt ./nvme-vg-autotest_77457/output//test_completions.txt ./nvme-vg-autotest_77457/output//timing.txt ./nvme-vg-autotest_77457/output//udev.log ./nvme-vg-autotest_77457/output//.. 01:10:00.406 + rmdir ./nvme-vg-autotest_77457/output/ 01:10:00.406 + for dir in "${out_dirs[@]}" 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvmf-cvl-phy-autotest_3015/output//doc.tar.xz ]] 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvmf-cvl-phy-autotest_3015/output//ut_coverage.tar.xz ]] 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvmf-cvl-phy-autotest_3015/output//llvm.tar.xz ]] 01:10:00.406 + mv ./nvmf-cvl-phy-autotest_3015/output//all_tests.txt ./nvmf-cvl-phy-autotest_3015/output//build-pkg-manifest.txt ./nvmf-cvl-phy-autotest_3015/output//build-repo-manifest.txt ./nvmf-cvl-phy-autotest_3015/output//cov_base.info.xz ./nvmf-cvl-phy-autotest_3015/output//cov_test.info.xz ./nvmf-cvl-phy-autotest_3015/output//cov_total.info.xz ./nvmf-cvl-phy-autotest_3015/output//dmesg.init.log ./nvmf-cvl-phy-autotest_3015/output//dmesg.log ./nvmf-cvl-phy-autotest_3015/output//nvmf_fuzz_tgt_output.txt ./nvmf-cvl-phy-autotest_3015/output//nvmf_trace.0_shm.tar.gz ./nvmf-cvl-phy-autotest_3015/output//power.tar.xz ./nvmf-cvl-phy-autotest_3015/output//test_completions.txt ./nvmf-cvl-phy-autotest_3015/output//timing.svg ./nvmf-cvl-phy-autotest_3015/output//timing.txt ./nvmf-cvl-phy-autotest_3015/output//.. 01:10:00.406 + rmdir ./nvmf-cvl-phy-autotest_3015/output/ 01:10:00.406 + for dir in "${out_dirs[@]}" 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvmf-phy-autotest_72421/output//doc.tar.xz ]] 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvmf-phy-autotest_72421/output//ut_coverage.tar.xz ]] 01:10:00.406 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.406 + [[ -f ./nvmf-phy-autotest_72421/output//llvm.tar.xz ]] 01:10:00.407 + mv ./nvmf-phy-autotest_72421/output//all_tests.txt ./nvmf-phy-autotest_72421/output//build-pkg-manifest.txt ./nvmf-phy-autotest_72421/output//build-repo-manifest.txt ./nvmf-phy-autotest_72421/output//cov_base.info.xz ./nvmf-phy-autotest_72421/output//cov_test.info.xz ./nvmf-phy-autotest_72421/output//cov_total.info.xz ./nvmf-phy-autotest_72421/output//dmesg.init.log ./nvmf-phy-autotest_72421/output//dmesg.log ./nvmf-phy-autotest_72421/output//nvmf_fuzz_tgt_output.txt ./nvmf-phy-autotest_72421/output//nvmf_trace.0_shm.tar.gz ./nvmf-phy-autotest_72421/output//power.tar.xz ./nvmf-phy-autotest_72421/output//test_completions.txt ./nvmf-phy-autotest_72421/output//timing.svg ./nvmf-phy-autotest_72421/output//timing.txt ./nvmf-phy-autotest_72421/output//.. 01:10:00.407 + rmdir ./nvmf-phy-autotest_72421/output/ 01:10:00.407 + for dir in "${out_dirs[@]}" 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-phy-autotest_72308/output//doc.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-phy-autotest_72308/output//ut_coverage.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-phy-autotest_72308/output//llvm.tar.xz ]] 01:10:00.407 + mv ./nvmf-tcp-phy-autotest_72308/output//all_tests.txt ./nvmf-tcp-phy-autotest_72308/output//build-pkg-manifest.txt ./nvmf-tcp-phy-autotest_72308/output//build-repo-manifest.txt ./nvmf-tcp-phy-autotest_72308/output//cov_base.info.xz ./nvmf-tcp-phy-autotest_72308/output//cov_test.info.xz ./nvmf-tcp-phy-autotest_72308/output//cov_total.info.xz ./nvmf-tcp-phy-autotest_72308/output//dmesg.init.log ./nvmf-tcp-phy-autotest_72308/output//dmesg.log ./nvmf-tcp-phy-autotest_72308/output//nvmf_fuzz_tgt_output.txt ./nvmf-tcp-phy-autotest_72308/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-phy-autotest_72308/output//power.tar.xz ./nvmf-tcp-phy-autotest_72308/output//test_completions.txt ./nvmf-tcp-phy-autotest_72308/output//timing.svg ./nvmf-tcp-phy-autotest_72308/output//timing.txt ./nvmf-tcp-phy-autotest_72308/output//.. 01:10:00.407 + rmdir ./nvmf-tcp-phy-autotest_72308/output/ 01:10:00.407 + for dir in "${out_dirs[@]}" 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-uring-vg-autotest_77382/output//doc.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-uring-vg-autotest_77382/output//ut_coverage.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-uring-vg-autotest_77382/output//llvm.tar.xz ]] 01:10:00.407 + mv ./nvmf-tcp-uring-vg-autotest_77382/output//all_tests.txt ./nvmf-tcp-uring-vg-autotest_77382/output//build-pkg-manifest.txt ./nvmf-tcp-uring-vg-autotest_77382/output//build-repo-manifest.txt ./nvmf-tcp-uring-vg-autotest_77382/output//cov_base.info.xz ./nvmf-tcp-uring-vg-autotest_77382/output//cov_test.info.xz ./nvmf-tcp-uring-vg-autotest_77382/output//cov_total.info.xz ./nvmf-tcp-uring-vg-autotest_77382/output//dmesg.init.log ./nvmf-tcp-uring-vg-autotest_77382/output//dmesg.log ./nvmf-tcp-uring-vg-autotest_77382/output//nvmf_fuzz_tgt_output.txt ./nvmf-tcp-uring-vg-autotest_77382/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-uring-vg-autotest_77382/output//power.tar.xz ./nvmf-tcp-uring-vg-autotest_77382/output//test_completions.txt ./nvmf-tcp-uring-vg-autotest_77382/output//timing.svg ./nvmf-tcp-uring-vg-autotest_77382/output//timing.txt ./nvmf-tcp-uring-vg-autotest_77382/output//.. 01:10:00.407 + rmdir ./nvmf-tcp-uring-vg-autotest_77382/output/ 01:10:00.407 + for dir in "${out_dirs[@]}" 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-vg-autotest_77448/output//doc.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-vg-autotest_77448/output//ut_coverage.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./nvmf-tcp-vg-autotest_77448/output//llvm.tar.xz ]] 01:10:00.407 + mv ./nvmf-tcp-vg-autotest_77448/output//all_tests.txt ./nvmf-tcp-vg-autotest_77448/output//build-pkg-manifest.txt ./nvmf-tcp-vg-autotest_77448/output//build-repo-manifest.txt ./nvmf-tcp-vg-autotest_77448/output//cov_base.info.xz ./nvmf-tcp-vg-autotest_77448/output//cov_test.info.xz ./nvmf-tcp-vg-autotest_77448/output//cov_total.info.xz ./nvmf-tcp-vg-autotest_77448/output//dmesg.init.log ./nvmf-tcp-vg-autotest_77448/output//dmesg.log ./nvmf-tcp-vg-autotest_77448/output//nvmf_fuzz_tgt_output.txt ./nvmf-tcp-vg-autotest_77448/output//nvmf_trace.0_shm.tar.gz ./nvmf-tcp-vg-autotest_77448/output//power.tar.xz ./nvmf-tcp-vg-autotest_77448/output//test_completions.txt ./nvmf-tcp-vg-autotest_77448/output//timing.svg ./nvmf-tcp-vg-autotest_77448/output//timing.txt ./nvmf-tcp-vg-autotest_77448/output//.. 01:10:00.407 + rmdir ./nvmf-tcp-vg-autotest_77448/output/ 01:10:00.407 + for dir in "${out_dirs[@]}" 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./packaging-docker-autotest_19844/output//doc.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./packaging-docker-autotest_19844/output//ut_coverage.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./packaging-docker-autotest_19844/output//llvm.tar.xz ]] 01:10:00.407 + mv ./packaging-docker-autotest_19844/output//build-pkg-manifest.txt ./packaging-docker-autotest_19844/output//build-repo-manifest.txt ./packaging-docker-autotest_19844/output//power.tar.xz ./packaging-docker-autotest_19844/output//test_completions.txt ./packaging-docker-autotest_19844/output//timing.txt ./packaging-docker-autotest_19844/output//.. 01:10:00.407 + rmdir ./packaging-docker-autotest_19844/output/ 01:10:00.407 + for dir in "${out_dirs[@]}" 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./rocky9-vg-autotest_16067/output//doc.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./rocky9-vg-autotest_16067/output//ut_coverage.tar.xz ]] 01:10:00.407 + tar -C ./rocky9-vg-autotest_16067/output/ -xf ./rocky9-vg-autotest_16067/output//ut_coverage.tar.xz 01:10:00.407 + rm ./rocky9-vg-autotest_16067/output//ut_coverage.tar.xz 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./rocky9-vg-autotest_16067/output//llvm.tar.xz ]] 01:10:00.407 + mv ./rocky9-vg-autotest_16067/output//all_tests.txt ./rocky9-vg-autotest_16067/output//build-pkg-manifest.txt ./rocky9-vg-autotest_16067/output//build-repo-manifest.txt ./rocky9-vg-autotest_16067/output//cov_base.info.xz ./rocky9-vg-autotest_16067/output//cov_test.info.xz ./rocky9-vg-autotest_16067/output//cov_total.info.xz ./rocky9-vg-autotest_16067/output//dmesg.init.log ./rocky9-vg-autotest_16067/output//dmesg.log ./rocky9-vg-autotest_16067/output//power.tar.xz ./rocky9-vg-autotest_16067/output//test_completions.txt ./rocky9-vg-autotest_16067/output//timing.svg ./rocky9-vg-autotest_16067/output//timing.txt ./rocky9-vg-autotest_16067/output//ut_coverage ./rocky9-vg-autotest_16067/output//.. 01:10:00.407 + rmdir ./rocky9-vg-autotest_16067/output/ 01:10:00.407 + for dir in "${out_dirs[@]}" 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./scan-build-docker-autotest_19845/output//doc.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./scan-build-docker-autotest_19845/output//ut_coverage.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./scan-build-docker-autotest_19845/output//llvm.tar.xz ]] 01:10:00.407 + mv ./scan-build-docker-autotest_19845/output//all_c_files.txt ./scan-build-docker-autotest_19845/output//build_output.txt ./scan-build-docker-autotest_19845/output//build-pkg-manifest.txt ./scan-build-docker-autotest_19845/output//build-repo-manifest.txt ./scan-build-docker-autotest_19845/output//built_c_files.txt ./scan-build-docker-autotest_19845/output//power.tar.xz ./scan-build-docker-autotest_19845/output//test_completions.txt ./scan-build-docker-autotest_19845/output//timing.txt ./scan-build-docker-autotest_19845/output//unbuilt_c_files.txt ./scan-build-docker-autotest_19845/output//.. 01:10:00.407 + rmdir ./scan-build-docker-autotest_19845/output/ 01:10:00.407 + for dir in "${out_dirs[@]}" 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./short-fuzz-phy-autotest_26174/output//doc.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./short-fuzz-phy-autotest_26174/output//ut_coverage.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./short-fuzz-phy-autotest_26174/output//llvm.tar.xz ]] 01:10:00.407 + tar -C ./short-fuzz-phy-autotest_26174/output/ -xf ./short-fuzz-phy-autotest_26174/output//llvm.tar.xz 01:10:00.407 + rm ./short-fuzz-phy-autotest_26174/output//llvm.tar.xz 01:10:00.407 + mv ./short-fuzz-phy-autotest_26174/output//all_tests.txt ./short-fuzz-phy-autotest_26174/output//build-pkg-manifest.txt ./short-fuzz-phy-autotest_26174/output//build-repo-manifest.txt ./short-fuzz-phy-autotest_26174/output//dmesg.init.log ./short-fuzz-phy-autotest_26174/output//dmesg.log ./short-fuzz-phy-autotest_26174/output//llvm ./short-fuzz-phy-autotest_26174/output//power.tar.xz ./short-fuzz-phy-autotest_26174/output//test_completions.txt ./short-fuzz-phy-autotest_26174/output//timing.svg ./short-fuzz-phy-autotest_26174/output//timing.txt ./short-fuzz-phy-autotest_26174/output//.. 01:10:00.407 + rmdir ./short-fuzz-phy-autotest_26174/output/ 01:10:00.407 + for dir in "${out_dirs[@]}" 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./ubuntu20-vg-autotest_73776/output//doc.tar.xz ]] 01:10:00.407 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.407 + [[ -f ./ubuntu20-vg-autotest_73776/output//ut_coverage.tar.xz ]] 01:10:00.407 + tar -C ./ubuntu20-vg-autotest_73776/output/ -xf ./ubuntu20-vg-autotest_73776/output//ut_coverage.tar.xz 01:10:00.665 + rm ./ubuntu20-vg-autotest_73776/output//ut_coverage.tar.xz 01:10:00.665 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.665 + [[ -f ./ubuntu20-vg-autotest_73776/output//llvm.tar.xz ]] 01:10:00.665 + mv ./ubuntu20-vg-autotest_73776/output//all_tests.txt ./ubuntu20-vg-autotest_73776/output//build-pkg-manifest.txt ./ubuntu20-vg-autotest_73776/output//build-repo-manifest.txt ./ubuntu20-vg-autotest_73776/output//cov_base.info.xz ./ubuntu20-vg-autotest_73776/output//dmesg.init.log ./ubuntu20-vg-autotest_73776/output//dmesg.log ./ubuntu20-vg-autotest_73776/output//power.tar.xz ./ubuntu20-vg-autotest_73776/output//proc_list.txt ./ubuntu20-vg-autotest_73776/output//test_completions.txt ./ubuntu20-vg-autotest_73776/output//timing.txt ./ubuntu20-vg-autotest_73776/output//udev.log ./ubuntu20-vg-autotest_73776/output//ut_coverage ./ubuntu20-vg-autotest_73776/output//.. 01:10:00.665 + rmdir ./ubuntu20-vg-autotest_73776/output/ 01:10:00.665 + for dir in "${out_dirs[@]}" 01:10:00.665 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.665 + [[ -f ./ubuntu22-vg-autotest_37790/output//doc.tar.xz ]] 01:10:00.665 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.665 + [[ -f ./ubuntu22-vg-autotest_37790/output//ut_coverage.tar.xz ]] 01:10:00.665 + tar -C ./ubuntu22-vg-autotest_37790/output/ -xf ./ubuntu22-vg-autotest_37790/output//ut_coverage.tar.xz 01:10:00.923 + rm ./ubuntu22-vg-autotest_37790/output//ut_coverage.tar.xz 01:10:00.923 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.923 + [[ -f ./ubuntu22-vg-autotest_37790/output//llvm.tar.xz ]] 01:10:00.923 + mv ./ubuntu22-vg-autotest_37790/output//all_tests.txt ./ubuntu22-vg-autotest_37790/output//build-pkg-manifest.txt ./ubuntu22-vg-autotest_37790/output//build-repo-manifest.txt ./ubuntu22-vg-autotest_37790/output//cov_base.info.xz ./ubuntu22-vg-autotest_37790/output//cov_test.info.xz ./ubuntu22-vg-autotest_37790/output//cov_total.info.xz ./ubuntu22-vg-autotest_37790/output//dmesg.init.log ./ubuntu22-vg-autotest_37790/output//dmesg.log ./ubuntu22-vg-autotest_37790/output//power.tar.xz ./ubuntu22-vg-autotest_37790/output//test_completions.txt ./ubuntu22-vg-autotest_37790/output//timing.svg ./ubuntu22-vg-autotest_37790/output//timing.txt ./ubuntu22-vg-autotest_37790/output//ut_coverage ./ubuntu22-vg-autotest_37790/output//.. 01:10:00.923 + rmdir ./ubuntu22-vg-autotest_37790/output/ 01:10:00.923 + for dir in "${out_dirs[@]}" 01:10:00.923 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.923 + [[ -f ./unittest-clang-docker-autotest_19842/output//doc.tar.xz ]] 01:10:00.923 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.923 + [[ -f ./unittest-clang-docker-autotest_19842/output//ut_coverage.tar.xz ]] 01:10:00.923 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.923 + [[ -f ./unittest-clang-docker-autotest_19842/output//llvm.tar.xz ]] 01:10:00.923 + mv ./unittest-clang-docker-autotest_19842/output//build-pkg-manifest.txt ./unittest-clang-docker-autotest_19842/output//build-repo-manifest.txt ./unittest-clang-docker-autotest_19842/output//power.tar.xz ./unittest-clang-docker-autotest_19842/output//test_completions.txt ./unittest-clang-docker-autotest_19842/output//timing.txt ./unittest-clang-docker-autotest_19842/output//.. 01:10:00.923 + rmdir ./unittest-clang-docker-autotest_19842/output/ 01:10:00.923 + for dir in "${out_dirs[@]}" 01:10:00.923 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.923 + [[ -f ./unittest-main-docker-autotest_19843/output//doc.tar.xz ]] 01:10:00.923 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:00.923 + [[ -f ./unittest-main-docker-autotest_19843/output//ut_coverage.tar.xz ]] 01:10:00.923 + tar -C ./unittest-main-docker-autotest_19843/output/ -xf ./unittest-main-docker-autotest_19843/output//ut_coverage.tar.xz 01:10:01.181 + rm ./unittest-main-docker-autotest_19843/output//ut_coverage.tar.xz 01:10:01.182 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.182 + [[ -f ./unittest-main-docker-autotest_19843/output//llvm.tar.xz ]] 01:10:01.182 + mv ./unittest-main-docker-autotest_19843/output//build-pkg-manifest.txt ./unittest-main-docker-autotest_19843/output//build-repo-manifest.txt ./unittest-main-docker-autotest_19843/output//power.tar.xz ./unittest-main-docker-autotest_19843/output//test_completions.txt ./unittest-main-docker-autotest_19843/output//timing.txt ./unittest-main-docker-autotest_19843/output//ut_coverage ./unittest-main-docker-autotest_19843/output//.. 01:10:01.182 + rmdir ./unittest-main-docker-autotest_19843/output/ 01:10:01.182 + for dir in "${out_dirs[@]}" 01:10:01.182 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.182 + [[ -f ./unittest-valgrind-docker-autotest_19846/output//doc.tar.xz ]] 01:10:01.182 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.182 + [[ -f ./unittest-valgrind-docker-autotest_19846/output//ut_coverage.tar.xz ]] 01:10:01.182 + tar -C ./unittest-valgrind-docker-autotest_19846/output/ -xf ./unittest-valgrind-docker-autotest_19846/output//ut_coverage.tar.xz 01:10:01.440 + rm ./unittest-valgrind-docker-autotest_19846/output//ut_coverage.tar.xz 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./unittest-valgrind-docker-autotest_19846/output//llvm.tar.xz ]] 01:10:01.440 + mv ./unittest-valgrind-docker-autotest_19846/output//build-pkg-manifest.txt ./unittest-valgrind-docker-autotest_19846/output//build-repo-manifest.txt ./unittest-valgrind-docker-autotest_19846/output//power.tar.xz ./unittest-valgrind-docker-autotest_19846/output//test_completions.txt ./unittest-valgrind-docker-autotest_19846/output//timing.txt ./unittest-valgrind-docker-autotest_19846/output//ut_coverage ./unittest-valgrind-docker-autotest_19846/output//.. 01:10:01.440 + rmdir ./unittest-valgrind-docker-autotest_19846/output/ 01:10:01.440 + for dir in "${out_dirs[@]}" 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vfio-user-phy-autotest_26185/output//doc.tar.xz ]] 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vfio-user-phy-autotest_26185/output//ut_coverage.tar.xz ]] 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vfio-user-phy-autotest_26185/output//llvm.tar.xz ]] 01:10:01.440 + mv ./vfio-user-phy-autotest_26185/output//all_tests.txt ./vfio-user-phy-autotest_26185/output//build-pkg-manifest.txt ./vfio-user-phy-autotest_26185/output//build-repo-manifest.txt ./vfio-user-phy-autotest_26185/output//cov_base.info.xz ./vfio-user-phy-autotest_26185/output//cov_test.info.xz ./vfio-user-phy-autotest_26185/output//cov_total.info.xz ./vfio-user-phy-autotest_26185/output//dmesg.init.log ./vfio-user-phy-autotest_26185/output//dmesg.log ./vfio-user-phy-autotest_26185/output//power.tar.xz ./vfio-user-phy-autotest_26185/output//test_completions.txt ./vfio-user-phy-autotest_26185/output//timing.svg ./vfio-user-phy-autotest_26185/output//timing.txt ./vfio-user-phy-autotest_26185/output//.. 01:10:01.440 + rmdir ./vfio-user-phy-autotest_26185/output/ 01:10:01.440 + for dir in "${out_dirs[@]}" 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vhost-initiator-vg-autotest_77477/output//doc.tar.xz ]] 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vhost-initiator-vg-autotest_77477/output//ut_coverage.tar.xz ]] 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vhost-initiator-vg-autotest_77477/output//llvm.tar.xz ]] 01:10:01.440 + mv ./vhost-initiator-vg-autotest_77477/output//all_tests.txt ./vhost-initiator-vg-autotest_77477/output//build-kernel-manifest.txt ./vhost-initiator-vg-autotest_77477/output//build-pkg-manifest.txt ./vhost-initiator-vg-autotest_77477/output//build-repo-manifest.txt ./vhost-initiator-vg-autotest_77477/output//cov_base.info.xz ./vhost-initiator-vg-autotest_77477/output//cov_test.info.xz ./vhost-initiator-vg-autotest_77477/output//cov_total.info.xz ./vhost-initiator-vg-autotest_77477/output//dmesg.init.log ./vhost-initiator-vg-autotest_77477/output//dmesg.log ./vhost-initiator-vg-autotest_77477/output//power.tar.xz ./vhost-initiator-vg-autotest_77477/output//test_completions.txt ./vhost-initiator-vg-autotest_77477/output//timing.svg ./vhost-initiator-vg-autotest_77477/output//timing.txt ./vhost-initiator-vg-autotest_77477/output//vhost_fuzz_output1.txt.xz ./vhost-initiator-vg-autotest_77477/output//vhost_fuzz_output2.txt.xz ./vhost-initiator-vg-autotest_77477/output//vhost_fuzz_tgt_output.txt.xz ./vhost-initiator-vg-autotest_77477/output//.. 01:10:01.440 + rmdir ./vhost-initiator-vg-autotest_77477/output/ 01:10:01.440 + for dir in "${out_dirs[@]}" 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vhost-phy-autotest_26202/output//doc.tar.xz ]] 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vhost-phy-autotest_26202/output//ut_coverage.tar.xz ]] 01:10:01.440 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.440 + [[ -f ./vhost-phy-autotest_26202/output//llvm.tar.xz ]] 01:10:01.441 + mv ./vhost-phy-autotest_26202/output//all_tests.txt ./vhost-phy-autotest_26202/output//build-pkg-manifest.txt ./vhost-phy-autotest_26202/output//build-repo-manifest.txt ./vhost-phy-autotest_26202/output//cov_base.info.xz ./vhost-phy-autotest_26202/output//cov_test.info.xz ./vhost-phy-autotest_26202/output//cov_total.info.xz ./vhost-phy-autotest_26202/output//dmesg.init.log ./vhost-phy-autotest_26202/output//dmesg.log ./vhost-phy-autotest_26202/output//nvmf_trace.0_shm.tar.gz ./vhost-phy-autotest_26202/output//power.tar.xz ./vhost-phy-autotest_26202/output//test_completions.txt ./vhost-phy-autotest_26202/output//timing.svg ./vhost-phy-autotest_26202/output//timing.txt ./vhost-phy-autotest_26202/output//.. 01:10:01.441 + rmdir ./vhost-phy-autotest_26202/output/ 01:10:01.441 + for dir in "${out_dirs[@]}" 01:10:01.441 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.441 + [[ -f ./vmd-phy-autotest_2617/output//doc.tar.xz ]] 01:10:01.441 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.441 + [[ -f ./vmd-phy-autotest_2617/output//ut_coverage.tar.xz ]] 01:10:01.441 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.441 + [[ -f ./vmd-phy-autotest_2617/output//llvm.tar.xz ]] 01:10:01.441 + mv ./vmd-phy-autotest_2617/output//all_tests.txt ./vmd-phy-autotest_2617/output//build-pkg-manifest.txt ./vmd-phy-autotest_2617/output//build-repo-manifest.txt ./vmd-phy-autotest_2617/output//cov_base.info.xz ./vmd-phy-autotest_2617/output//cov_test.info.xz ./vmd-phy-autotest_2617/output//cov_total.info.xz ./vmd-phy-autotest_2617/output//dmesg.init.log ./vmd-phy-autotest_2617/output//dmesg.log ./vmd-phy-autotest_2617/output//power.tar.xz ./vmd-phy-autotest_2617/output//test_completions.txt ./vmd-phy-autotest_2617/output//timing.svg ./vmd-phy-autotest_2617/output//timing.txt ./vmd-phy-autotest_2617/output//.. 01:10:01.441 + rmdir ./vmd-phy-autotest_2617/output/ 01:10:01.441 + for dir in "${out_dirs[@]}" 01:10:01.441 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.441 + [[ -f ./zns-vg-autotest_76563/output//doc.tar.xz ]] 01:10:01.441 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.441 + [[ -f ./zns-vg-autotest_76563/output//ut_coverage.tar.xz ]] 01:10:01.441 + for archive in "${dir}"/{doc,ut_coverage,llvm}.tar.xz 01:10:01.441 + [[ -f ./zns-vg-autotest_76563/output//llvm.tar.xz ]] 01:10:01.441 + mv ./zns-vg-autotest_76563/output//all_tests.txt ./zns-vg-autotest_76563/output//build-pkg-manifest.txt ./zns-vg-autotest_76563/output//build-repo-manifest.txt ./zns-vg-autotest_76563/output//cov_base.info.xz ./zns-vg-autotest_76563/output//cov_test.info.xz ./zns-vg-autotest_76563/output//cov_total.info.xz ./zns-vg-autotest_76563/output//dmesg.init.log ./zns-vg-autotest_76563/output//dmesg.log ./zns-vg-autotest_76563/output//power.tar.xz ./zns-vg-autotest_76563/output//test_completions.txt ./zns-vg-autotest_76563/output//timing.svg ./zns-vg-autotest_76563/output//timing.txt ./zns-vg-autotest_76563/output//.. 01:10:01.441 + rmdir ./zns-vg-autotest_76563/output/ 01:10:01.441 + unpack_cov_files 01:10:01.441 + local info_files 01:10:01.441 + info_files=(*/cov_*.info.xz) 01:10:01.441 + xargs -P0 -r -n1 xz -d 01:10:01.441 + printf '%s\n' BlobFS-phy-autotest_26183/cov_base.info.xz BlobFS-phy-autotest_26183/cov_test.info.xz BlobFS-phy-autotest_26183/cov_total.info.xz centos7-vg-autotest_73636/cov_base.info.xz centos7-vg-autotest_73636/cov_test.info.xz centos7-vg-autotest_73636/cov_total.info.xz crypto-phy-autotest_26177/cov_base.info.xz crypto-phy-autotest_26177/cov_test.info.xz crypto-phy-autotest_26177/cov_total.info.xz dsa-phy-autotest_4877/cov_base.info.xz dsa-phy-autotest_4877/cov_test.info.xz dsa-phy-autotest_4877/cov_total.info.xz iscsi-uring-vg-autotest_77362/cov_base.info.xz iscsi-uring-vg-autotest_77362/cov_test.info.xz iscsi-uring-vg-autotest_77362/cov_total.info.xz iscsi-vg-autotest_77495/cov_base.info.xz iscsi-vg-autotest_77495/cov_test.info.xz iscsi-vg-autotest_77495/cov_total.info.xz lvol-vg-autotest_77406/cov_base.info.xz lvol-vg-autotest_77406/cov_test.info.xz lvol-vg-autotest_77406/cov_total.info.xz nvme-cmb-pmr-vg-autotest_78580/cov_base.info.xz nvme-cmb-pmr-vg-autotest_78580/cov_test.info.xz nvme-cmb-pmr-vg-autotest_78580/cov_total.info.xz nvme-phy-autotest_72349/cov_base.info.xz nvme-vg-autotest_77457/cov_base.info.xz nvmf-cvl-phy-autotest_3015/cov_base.info.xz nvmf-cvl-phy-autotest_3015/cov_test.info.xz nvmf-cvl-phy-autotest_3015/cov_total.info.xz nvmf-phy-autotest_72421/cov_base.info.xz nvmf-phy-autotest_72421/cov_test.info.xz nvmf-phy-autotest_72421/cov_total.info.xz nvmf-tcp-phy-autotest_72308/cov_base.info.xz nvmf-tcp-phy-autotest_72308/cov_test.info.xz nvmf-tcp-phy-autotest_72308/cov_total.info.xz nvmf-tcp-uring-vg-autotest_77382/cov_base.info.xz nvmf-tcp-uring-vg-autotest_77382/cov_test.info.xz nvmf-tcp-uring-vg-autotest_77382/cov_total.info.xz nvmf-tcp-vg-autotest_77448/cov_base.info.xz nvmf-tcp-vg-autotest_77448/cov_test.info.xz nvmf-tcp-vg-autotest_77448/cov_total.info.xz rocky9-vg-autotest_16067/cov_base.info.xz rocky9-vg-autotest_16067/cov_test.info.xz rocky9-vg-autotest_16067/cov_total.info.xz ubuntu20-vg-autotest_73776/cov_base.info.xz ubuntu22-vg-autotest_37790/cov_base.info.xz ubuntu22-vg-autotest_37790/cov_test.info.xz ubuntu22-vg-autotest_37790/cov_total.info.xz vfio-user-phy-autotest_26185/cov_base.info.xz vfio-user-phy-autotest_26185/cov_test.info.xz vfio-user-phy-autotest_26185/cov_total.info.xz vhost-initiator-vg-autotest_77477/cov_base.info.xz vhost-initiator-vg-autotest_77477/cov_test.info.xz vhost-initiator-vg-autotest_77477/cov_total.info.xz vhost-phy-autotest_26202/cov_base.info.xz vhost-phy-autotest_26202/cov_test.info.xz vhost-phy-autotest_26202/cov_total.info.xz vmd-phy-autotest_2617/cov_base.info.xz vmd-phy-autotest_2617/cov_test.info.xz vmd-phy-autotest_2617/cov_total.info.xz zns-vg-autotest_76563/cov_base.info.xz zns-vg-autotest_76563/cov_test.info.xz zns-vg-autotest_76563/cov_total.info.xz 01:10:02.004 + fix_downstream_job_paths 01:10:02.005 + sed -i -e 's#^SF:/.\+/spdk/#SF:/var/jenkins/workspace/autotest-nightly/spdk/#g' BlobFS-phy-autotest_26183/cov_total.info centos7-vg-autotest_73636/cov_total.info crypto-phy-autotest_26177/cov_total.info dsa-phy-autotest_4877/cov_total.info iscsi-uring-vg-autotest_77362/cov_total.info iscsi-vg-autotest_77495/cov_total.info lvol-vg-autotest_77406/cov_total.info nvme-cmb-pmr-vg-autotest_78580/cov_total.info nvmf-cvl-phy-autotest_3015/cov_total.info nvmf-phy-autotest_72421/cov_total.info nvmf-tcp-phy-autotest_72308/cov_total.info nvmf-tcp-uring-vg-autotest_77382/cov_total.info nvmf-tcp-vg-autotest_77448/cov_total.info rocky9-vg-autotest_16067/cov_total.info ubuntu22-vg-autotest_37790/cov_total.info vfio-user-phy-autotest_26185/cov_total.info vhost-initiator-vg-autotest_77477/cov_total.info vhost-phy-autotest_26202/cov_total.info vmd-phy-autotest_2617/cov_total.info zns-vg-autotest_76563/cov_total.info 01:10:02.939 + ignore_ocf_headers 01:10:02.939 + sed -i 's#--show-details#--show-details --ignore-errors source#g' spdk/autorun_post.py 01:10:02.939 + create_errtrace_files_for_all_builds 01:10:02.939 + local build_log_list 01:10:02.939 + build_log_list=($(find "$WORKSPACE" -type f -name 'build.log')) 01:10:02.939 ++ find /var/jenkins/workspace/autotest-nightly -type f -name build.log 01:10:02.939 + for build in "${build_log_list[@]}" 01:10:02.939 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/build.log 01:10:02.939 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.939 + build_log=/var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/build.log 01:10:02.939 +++ dirname /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/build.log 01:10:02.939 ++ basename /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805 01:10:02.939 + job_name=autobuild-external-vg-autotest_19805 01:10:02.939 + start_pattern='Backtrace start' 01:10:02.939 + end_pattern='Backtrace end' 01:10:02.939 + [[ -f /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/build.log ]] 01:10:02.939 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/build.log 01:10:02.939 + backtraces= 01:10:02.939 + [[ -n '' ]] 01:10:02.939 + for build in "${build_log_list[@]}" 01:10:02.939 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/build.log 01:10:02.939 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.939 + build_log=/var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/build.log 01:10:02.939 +++ dirname /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/build.log 01:10:02.939 ++ basename /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305 01:10:02.939 + job_name=doc-docker-autotest_22305 01:10:02.939 + start_pattern='Backtrace start' 01:10:02.939 + end_pattern='Backtrace end' 01:10:02.939 + [[ -f /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/build.log ]] 01:10:02.939 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/build.log 01:10:02.939 + backtraces= 01:10:02.939 + [[ -n '' ]] 01:10:02.939 + for build in "${build_log_list[@]}" 01:10:02.939 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/build.log 01:10:02.939 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.939 + build_log=/var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/build.log 01:10:02.939 +++ dirname /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/build.log 01:10:02.939 ++ basename /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790 01:10:02.939 + job_name=ubuntu22-vg-autotest_37790 01:10:02.939 + start_pattern='Backtrace start' 01:10:02.939 + end_pattern='Backtrace end' 01:10:02.939 + [[ -f /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/build.log ]] 01:10:02.939 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/build.log 01:10:02.939 + backtraces= 01:10:02.939 + [[ -n '' ]] 01:10:02.939 + for build in "${build_log_list[@]}" 01:10:02.939 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/build.log 01:10:02.939 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.939 + build_log=/var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/build.log 01:10:02.939 +++ dirname /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/build.log 01:10:02.939 ++ basename /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636 01:10:02.939 + job_name=centos7-vg-autotest_73636 01:10:02.939 + start_pattern='Backtrace start' 01:10:02.939 + end_pattern='Backtrace end' 01:10:02.939 + [[ -f /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/build.log ]] 01:10:02.939 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/build.log 01:10:02.939 + backtraces= 01:10:02.939 + [[ -n '' ]] 01:10:02.939 + for build in "${build_log_list[@]}" 01:10:02.939 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/build.log 01:10:02.939 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.939 + build_log=/var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/build.log 01:10:02.939 +++ dirname /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/build.log 01:10:02.939 ++ basename /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067 01:10:02.939 + job_name=rocky9-vg-autotest_16067 01:10:02.939 + start_pattern='Backtrace start' 01:10:02.939 + end_pattern='Backtrace end' 01:10:02.939 + [[ -f /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/build.log ]] 01:10:02.939 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/build.log 01:10:02.939 + backtraces= 01:10:02.939 + [[ -n '' ]] 01:10:02.939 + for build in "${build_log_list[@]}" 01:10:02.939 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/build.log 01:10:02.939 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.939 + build_log=/var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/build.log 01:10:02.939 +++ dirname /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/build.log 01:10:02.939 ++ basename /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844 01:10:02.939 + job_name=packaging-docker-autotest_19844 01:10:02.939 + start_pattern='Backtrace start' 01:10:02.939 + end_pattern='Backtrace end' 01:10:02.939 + [[ -f /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/build.log ]] 01:10:02.939 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/build.log 01:10:02.939 + backtraces= 01:10:02.939 + [[ -n '' ]] 01:10:02.939 + for build in "${build_log_list[@]}" 01:10:02.939 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/build.log 01:10:02.939 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.939 + build_log=/var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/build.log 01:10:02.939 +++ dirname /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/build.log 01:10:02.939 ++ basename /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457 01:10:02.939 + job_name=nvme-vg-autotest_77457 01:10:02.939 + start_pattern='Backtrace start' 01:10:02.939 + end_pattern='Backtrace end' 01:10:02.939 + [[ -f /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/build.log ]] 01:10:02.939 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/build.log 01:10:02.939 + backtraces='00:22:05.924 03:46:54 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ========== 01:10:02.939 00:22:05.924 01:10:02.939 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.939 00:22:05.924 ... 01:10:02.939 00:22:05.924 710 01:10:02.939 00:22:05.924 711 # We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to 01:10:02.939 00:22:05.924 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.939 00:22:05.924 713 # but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do: 01:10:02.939 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.939 00:22:05.924 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.939 00:22:05.924 716 01:10:02.939 00:22:05.924 717 echo "$time" 01:10:02.939 00:22:05.924 718 ) 01:10:02.939 00:22:05.924 719 01:10:02.939 00:22:05.924 720 function timing_enter() { 01:10:02.939 00:22:05.924 ... 01:10:02.939 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.939 00:22:05.924 ... 01:10:02.939 00:22:05.924 11 # Pci bus hotplug 01:10:02.939 00:22:05.924 12 # Helper function to remove/attach cotrollers 01:10:02.939 00:22:05.924 13 debug_remove_attach_helper() { 01:10:02.939 00:22:05.924 14 local helper_time=0 01:10:02.939 00:22:05.924 15 01:10:02.939 00:22:05.924 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.939 00:22:05.924 17 printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \ 01:10:02.939 00:22:05.924 18 "$helper_time" "$nvme_count" >&2 01:10:02.939 00:22:05.924 19 } 01:10:02.939 00:22:05.924 20 01:10:02.939 00:22:05.924 21 remove_attach_helper() { 01:10:02.939 00:22:05.924 ... 01:10:02.939 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.939 00:22:05.924 ... 01:10:02.939 00:22:05.924 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.939 00:22:05.924 106 done 01:10:02.939 00:22:05.924 107 01:10:02.939 00:22:05.924 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.939 00:22:05.924 109 01:10:02.939 00:22:05.924 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.939 00:22:05.924 111 # Verify reregistering hotplug poller 01:10:02.940 00:22:05.924 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.940 00:22:05.924 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.940 00:22:05.924 114 01:10:02.940 00:22:05.924 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 133 01:10:02.940 00:22:05.924 134 # Run pci bus hotplug test 01:10:02.940 00:22:05.924 135 run_hotplug 01:10:02.940 00:22:05.924 136 01:10:02.940 00:22:05.924 137 # Run SPDK target based hotplug 01:10:02.940 00:22:05.924 => 138 tgt_run_hotplug 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 01:10:02.940 00:22:05.924 ========== Backtrace end =========='\'' 01:10:02.940 00:22:05.924 ========== Backtrace start: ========== 01:10:02.940 00:22:05.924 01:10:02.940 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 710 01:10:02.940 00:22:05.924 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:02.940 00:22:05.924 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.940 00:22:05.924 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.940 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.940 00:22:05.924 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.940 00:22:05.924 716 01:10:02.940 00:22:05.924 717 echo "$time" 01:10:02.940 00:22:05.924 718 ) 01:10:02.940 00:22:05.924 719 01:10:02.940 00:22:05.924 720 function timing_enter() { 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 11 # Pci bus hotplug 01:10:02.940 00:22:05.924 12 # Helper function to remove/attach cotrollers 01:10:02.940 00:22:05.924 13 debug_remove_attach_helper() { 01:10:02.940 00:22:05.924 14 local helper_time=0 01:10:02.940 00:22:05.924 15 01:10:02.940 00:22:05.924 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.940 00:22:05.924 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.940 00:22:05.924 18 "$helper_time" "$nvme_count" >&2 01:10:02.940 00:22:05.924 19 } 01:10:02.940 00:22:05.924 20 01:10:02.940 00:22:05.924 21 remove_attach_helper() { 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.940 00:22:05.924 106 done 01:10:02.940 00:22:05.924 107 01:10:02.940 00:22:05.924 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.940 00:22:05.924 109 01:10:02.940 00:22:05.924 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.940 00:22:05.924 111 # Verify reregistering hotplug poller 01:10:02.940 00:22:05.924 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.940 00:22:05.924 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.940 00:22:05.924 114 01:10:02.940 00:22:05.924 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 133 01:10:02.940 00:22:05.924 134 # Run pci bus hotplug test 01:10:02.940 00:22:05.924 135 run_hotplug 01:10:02.940 00:22:05.924 136 01:10:02.940 00:22:05.924 137 # Run SPDK target based hotplug 01:10:02.940 00:22:05.924 => 138 tgt_run_hotplug 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 01:10:02.940 00:22:05.924 ========== Backtrace end ========== 01:10:02.940 00:22:05.924 ========== Backtrace start: ========== 01:10:02.940 00:22:05.924 01:10:02.940 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.940 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.940 00:22:05.924 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.940 00:22:05.924 716 01:10:02.940 00:22:05.924 717 echo "$time" 01:10:02.940 00:22:05.924 => 718 ) 01:10:02.940 00:22:05.924 719 01:10:02.940 00:22:05.924 720 function timing_enter() { 01:10:02.940 00:22:05.924 721 xtrace_disable 01:10:02.940 00:22:05.924 722 timing "enter" "$1" 01:10:02.940 00:22:05.924 723 xtrace_restore 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 11 # Pci bus hotplug 01:10:02.940 00:22:05.925 12 # Helper function to remove/attach cotrollers 01:10:02.940 00:22:05.925 13 debug_remove_attach_helper() { 01:10:02.940 00:22:05.925 14 local helper_time=0 01:10:02.940 00:22:05.925 15 01:10:02.940 00:22:05.925 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.940 00:22:05.925 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.940 00:22:05.925 18 "$helper_time" "$nvme_count" >&2 01:10:02.940 00:22:05.925 19 } 01:10:02.940 00:22:05.925 20 01:10:02.940 00:22:05.925 21 remove_attach_helper() { 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.940 00:22:05.925 106 done 01:10:02.940 00:22:05.925 107 01:10:02.940 00:22:05.925 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.940 00:22:05.925 109 01:10:02.940 00:22:05.925 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.940 00:22:05.925 111 # Verify reregistering hotplug poller 01:10:02.940 00:22:05.925 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.940 00:22:05.925 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.940 00:22:05.925 114 01:10:02.940 00:22:05.925 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 133 01:10:02.940 00:22:05.925 134 # Run pci bus hotplug test 01:10:02.940 00:22:05.925 135 run_hotplug 01:10:02.940 00:22:05.925 136 01:10:02.940 00:22:05.925 137 # Run SPDK target based hotplug 01:10:02.940 00:22:05.925 => 138 tgt_run_hotplug 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 01:10:02.940 00:22:05.925 ========== Backtrace end ========== 01:10:02.940 00:22:05.925 ========== Backtrace start: ========== 01:10:02.940 00:22:05.925 01:10:02.940 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 11 # Pci bus hotplug 01:10:02.940 00:22:05.925 12 # Helper function to remove/attach cotrollers 01:10:02.940 00:22:05.925 13 debug_remove_attach_helper() { 01:10:02.940 00:22:05.925 14 local helper_time=0 01:10:02.940 00:22:05.925 15 01:10:02.940 00:22:05.925 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.940 00:22:05.925 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.940 00:22:05.925 18 "$helper_time" "$nvme_count" >&2 01:10:02.940 00:22:05.925 19 } 01:10:02.940 00:22:05.925 20 01:10:02.940 00:22:05.925 21 remove_attach_helper() { 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.940 00:22:05.925 106 done 01:10:02.940 00:22:05.925 107 01:10:02.940 00:22:05.925 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.940 00:22:05.925 109 01:10:02.940 00:22:05.925 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.940 00:22:05.925 111 # Verify reregistering hotplug poller 01:10:02.940 00:22:05.925 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.940 00:22:05.925 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.940 00:22:05.925 114 01:10:02.940 00:22:05.925 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 133 01:10:02.940 00:22:05.925 134 # Run pci bus hotplug test 01:10:02.940 00:22:05.925 135 run_hotplug 01:10:02.940 00:22:05.925 136 01:10:02.940 00:22:05.925 137 # Run SPDK target based hotplug 01:10:02.940 00:22:05.925 => 138 tgt_run_hotplug 01:10:02.940 00:22:05.925 ... 01:10:02.940 00:22:05.925 01:10:02.940 00:22:05.925 ========== Backtrace end ========== 01:10:02.940 00:22:07.303 ========== Backtrace start: ========== 01:10:02.940 00:22:07.303 01:10:02.940 00:22:07.303 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"]) 01:10:02.940 00:22:07.303 ... 01:10:02.940 00:22:07.303 1117 timing_enter $test_name 01:10:02.940 00:22:07.303 1118 echo "************************************" 01:10:02.940 00:22:07.303 1119 echo "START TEST $test_name" 01:10:02.940 00:22:07.303 1120 echo "************************************" 01:10:02.940 00:22:07.303 1121 xtrace_restore 01:10:02.940 00:22:07.303 1122 time "$@" 01:10:02.940 00:22:07.303 1123 xtrace_disable 01:10:02.940 00:22:07.303 1124 echo "************************************" 01:10:02.940 00:22:07.303 1125 echo "END TEST $test_name" 01:10:02.940 00:22:07.303 1126 echo "************************************" 01:10:02.940 00:22:07.303 1127 timing_exit $test_name 01:10:02.940 00:22:07.303 ... 01:10:02.940 00:22:07.303 in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"]) 01:10:02.940 00:22:07.303 ... 01:10:02.940 00:22:07.303 235 01:10:02.940 00:22:07.303 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:02.940 00:22:07.303 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:02.940 00:22:07.303 238 01:10:02.940 00:22:07.303 239 if [ $(uname -s) = Linux ]; then 01:10:02.940 00:22:07.303 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:02.940 00:22:07.303 241 fi 01:10:02.940 00:22:07.303 242 01:10:02.940 00:22:07.303 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:02.940 00:22:07.303 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:02.940 00:22:07.303 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:02.940 00:22:07.303 ... 01:10:02.940 00:22:07.303 01:10:02.940 00:22:07.303 ========== Backtrace end ==========' 01:10:02.940 + [[ -n 00:22:05.924 03:46:54 sw_hotplug -- common/autotest_common.sh@715 -- # time='========== Backtrace start: ========== 01:10:02.940 00:22:05.924 01:10:02.940 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 710 01:10:02.940 00:22:05.924 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:02.940 00:22:05.924 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.940 00:22:05.924 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.940 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.940 00:22:05.924 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.940 00:22:05.924 716 01:10:02.940 00:22:05.924 717 echo "$time" 01:10:02.940 00:22:05.924 718 ) 01:10:02.940 00:22:05.924 719 01:10:02.940 00:22:05.924 720 function timing_enter() { 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.940 00:22:05.924 ... 01:10:02.940 00:22:05.924 11 # Pci bus hotplug 01:10:02.940 00:22:05.924 12 # Helper function to remove/attach cotrollers 01:10:02.940 00:22:05.924 13 debug_remove_attach_helper() { 01:10:02.941 00:22:05.924 14 local helper_time=0 01:10:02.941 00:22:05.924 15 01:10:02.941 00:22:05.924 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.941 00:22:05.924 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.941 00:22:05.924 18 "$helper_time" "$nvme_count" >&2 01:10:02.941 00:22:05.924 19 } 01:10:02.941 00:22:05.924 20 01:10:02.941 00:22:05.924 21 remove_attach_helper() { 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.941 00:22:05.924 106 done 01:10:02.941 00:22:05.924 107 01:10:02.941 00:22:05.924 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.941 00:22:05.924 109 01:10:02.941 00:22:05.924 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.941 00:22:05.924 111 # Verify reregistering hotplug poller 01:10:02.941 00:22:05.924 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.941 00:22:05.924 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.941 00:22:05.924 114 01:10:02.941 00:22:05.924 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 133 01:10:02.941 00:22:05.924 134 # Run pci bus hotplug test 01:10:02.941 00:22:05.924 135 run_hotplug 01:10:02.941 00:22:05.924 136 01:10:02.941 00:22:05.924 137 # Run SPDK target based hotplug 01:10:02.941 00:22:05.924 => 138 tgt_run_hotplug 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 01:10:02.941 00:22:05.924 ========== Backtrace end ==========' 01:10:02.941 00:22:05.924 ========== Backtrace start: ========== 01:10:02.941 00:22:05.924 01:10:02.941 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 710 01:10:02.941 00:22:05.924 711 # We redirect cmd's std{out,err} to a separate fd dup'ed to stdin's device (or /dev/null) to 01:10:02.941 00:22:05.924 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.941 00:22:05.924 713 # but $() will return just the time's data, hence making it possible to just do: 01:10:02.941 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.941 00:22:05.924 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.941 00:22:05.924 716 01:10:02.941 00:22:05.924 717 echo "$time" 01:10:02.941 00:22:05.924 718 ) 01:10:02.941 00:22:05.924 719 01:10:02.941 00:22:05.924 720 function timing_enter() { 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 11 # Pci bus hotplug 01:10:02.941 00:22:05.924 12 # Helper function to remove/attach cotrollers 01:10:02.941 00:22:05.924 13 debug_remove_attach_helper() { 01:10:02.941 00:22:05.924 14 local helper_time=0 01:10:02.941 00:22:05.924 15 01:10:02.941 00:22:05.924 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.941 00:22:05.924 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:02.941 00:22:05.924 18 "$helper_time" "$nvme_count" >&2 01:10:02.941 00:22:05.924 19 } 01:10:02.941 00:22:05.924 20 01:10:02.941 00:22:05.924 21 remove_attach_helper() { 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.941 00:22:05.924 106 done 01:10:02.941 00:22:05.924 107 01:10:02.941 00:22:05.924 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.941 00:22:05.924 109 01:10:02.941 00:22:05.924 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.941 00:22:05.924 111 # Verify reregistering hotplug poller 01:10:02.941 00:22:05.924 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.941 00:22:05.924 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.941 00:22:05.924 114 01:10:02.941 00:22:05.924 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 133 01:10:02.941 00:22:05.924 134 # Run pci bus hotplug test 01:10:02.941 00:22:05.924 135 run_hotplug 01:10:02.941 00:22:05.924 136 01:10:02.941 00:22:05.924 137 # Run SPDK target based hotplug 01:10:02.941 00:22:05.924 => 138 tgt_run_hotplug 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 01:10:02.941 00:22:05.924 ========== Backtrace end ========== 01:10:02.941 00:22:05.924 ========== Backtrace start: ========== 01:10:02.941 00:22:05.924 01:10:02.941 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.924 713 # but $() will return just the time's data, hence making it possible to just do: 01:10:02.941 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.941 00:22:05.924 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.941 00:22:05.924 716 01:10:02.941 00:22:05.924 717 echo "$time" 01:10:02.941 00:22:05.924 => 718 ) 01:10:02.941 00:22:05.924 719 01:10:02.941 00:22:05.924 720 function timing_enter() { 01:10:02.941 00:22:05.924 721 xtrace_disable 01:10:02.941 00:22:05.924 722 timing "enter" "$1" 01:10:02.941 00:22:05.924 723 xtrace_restore 01:10:02.941 00:22:05.924 ... 01:10:02.941 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 11 # Pci bus hotplug 01:10:02.941 00:22:05.925 12 # Helper function to remove/attach cotrollers 01:10:02.941 00:22:05.925 13 debug_remove_attach_helper() { 01:10:02.941 00:22:05.925 14 local helper_time=0 01:10:02.941 00:22:05.925 15 01:10:02.941 00:22:05.925 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.941 00:22:05.925 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:02.941 00:22:05.925 18 "$helper_time" "$nvme_count" >&2 01:10:02.941 00:22:05.925 19 } 01:10:02.941 00:22:05.925 20 01:10:02.941 00:22:05.925 21 remove_attach_helper() { 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.941 00:22:05.925 106 done 01:10:02.941 00:22:05.925 107 01:10:02.941 00:22:05.925 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.941 00:22:05.925 109 01:10:02.941 00:22:05.925 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.941 00:22:05.925 111 # Verify reregistering hotplug poller 01:10:02.941 00:22:05.925 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.941 00:22:05.925 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.941 00:22:05.925 114 01:10:02.941 00:22:05.925 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 133 01:10:02.941 00:22:05.925 134 # Run pci bus hotplug test 01:10:02.941 00:22:05.925 135 run_hotplug 01:10:02.941 00:22:05.925 136 01:10:02.941 00:22:05.925 137 # Run SPDK target based hotplug 01:10:02.941 00:22:05.925 => 138 tgt_run_hotplug 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 01:10:02.941 00:22:05.925 ========== Backtrace end ========== 01:10:02.941 00:22:05.925 ========== Backtrace start: ========== 01:10:02.941 00:22:05.925 01:10:02.941 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 11 # Pci bus hotplug 01:10:02.941 00:22:05.925 12 # Helper function to remove/attach cotrollers 01:10:02.941 00:22:05.925 13 debug_remove_attach_helper() { 01:10:02.941 00:22:05.925 14 local helper_time=0 01:10:02.941 00:22:05.925 15 01:10:02.941 00:22:05.925 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.941 00:22:05.925 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:02.941 00:22:05.925 18 "$helper_time" "$nvme_count" >&2 01:10:02.941 00:22:05.925 19 } 01:10:02.941 00:22:05.925 20 01:10:02.941 00:22:05.925 21 remove_attach_helper() { 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.941 00:22:05.925 106 done 01:10:02.941 00:22:05.925 107 01:10:02.941 00:22:05.925 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.941 00:22:05.925 109 01:10:02.941 00:22:05.925 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.941 00:22:05.925 111 # Verify reregistering hotplug poller 01:10:02.941 00:22:05.925 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.941 00:22:05.925 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.941 00:22:05.925 114 01:10:02.941 00:22:05.925 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 133 01:10:02.941 00:22:05.925 134 # Run pci bus hotplug test 01:10:02.941 00:22:05.925 135 run_hotplug 01:10:02.941 00:22:05.925 136 01:10:02.941 00:22:05.925 137 # Run SPDK target based hotplug 01:10:02.941 00:22:05.925 => 138 tgt_run_hotplug 01:10:02.941 00:22:05.925 ... 01:10:02.941 00:22:05.925 01:10:02.941 00:22:05.925 ========== Backtrace end ========== 01:10:02.941 00:22:07.303 ========== Backtrace start: ========== 01:10:02.941 00:22:07.303 01:10:02.942 00:22:07.303 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"]) 01:10:02.942 00:22:07.303 ... 01:10:02.942 00:22:07.303 1117 timing_enter $test_name 01:10:02.942 00:22:07.303 1118 echo "************************************" 01:10:02.942 00:22:07.303 1119 echo "START TEST $test_name" 01:10:02.942 00:22:07.303 1120 echo "************************************" 01:10:02.942 00:22:07.303 1121 xtrace_restore 01:10:02.942 00:22:07.303 1122 time "$@" 01:10:02.942 00:22:07.303 1123 xtrace_disable 01:10:02.942 00:22:07.303 1124 echo "************************************" 01:10:02.942 00:22:07.303 1125 echo "END TEST $test_name" 01:10:02.942 00:22:07.303 1126 echo "************************************" 01:10:02.942 00:22:07.303 1127 timing_exit $test_name 01:10:02.942 00:22:07.303 ... 01:10:02.942 00:22:07.303 in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"]) 01:10:02.942 00:22:07.303 ... 01:10:02.942 00:22:07.303 235 01:10:02.942 00:22:07.303 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:02.942 00:22:07.303 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:02.942 00:22:07.303 238 01:10:02.942 00:22:07.303 239 if [ $(uname -s) = Linux ]; then 01:10:02.942 00:22:07.303 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:02.942 00:22:07.303 241 fi 01:10:02.942 00:22:07.303 242 01:10:02.942 00:22:07.303 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:02.942 00:22:07.303 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:02.942 00:22:07.303 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:02.942 00:22:07.303 ... 01:10:02.942 00:22:07.303 01:10:02.942 00:22:07.303 ========== Backtrace end ========== ]] 01:10:02.942 + echo '00:22:05.924 03:46:54 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ========== 01:10:02.942 00:22:05.924 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 710 01:10:02.942 00:22:05.924 711 # We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to 01:10:02.942 00:22:05.924 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.942 00:22:05.924 713 # but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do: 01:10:02.942 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.942 00:22:05.924 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.942 00:22:05.924 716 01:10:02.942 00:22:05.924 717 echo "$time" 01:10:02.942 00:22:05.924 718 ) 01:10:02.942 00:22:05.924 719 01:10:02.942 00:22:05.924 720 function timing_enter() { 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 11 # Pci bus hotplug 01:10:02.942 00:22:05.924 12 # Helper function to remove/attach cotrollers 01:10:02.942 00:22:05.924 13 debug_remove_attach_helper() { 01:10:02.942 00:22:05.924 14 local helper_time=0 01:10:02.942 00:22:05.924 15 01:10:02.942 00:22:05.924 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.942 00:22:05.924 17 printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \ 01:10:02.942 00:22:05.924 18 "$helper_time" "$nvme_count" >&2 01:10:02.942 00:22:05.924 19 } 01:10:02.942 00:22:05.924 20 01:10:02.942 00:22:05.924 21 remove_attach_helper() { 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.942 00:22:05.924 106 done 01:10:02.942 00:22:05.924 107 01:10:02.942 00:22:05.924 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.942 00:22:05.924 109 01:10:02.942 00:22:05.924 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.942 00:22:05.924 111 # Verify reregistering hotplug poller 01:10:02.942 00:22:05.924 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.942 00:22:05.924 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.942 00:22:05.924 114 01:10:02.942 00:22:05.924 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 133 01:10:02.942 00:22:05.924 134 # Run pci bus hotplug test 01:10:02.942 00:22:05.924 135 run_hotplug 01:10:02.942 00:22:05.924 136 01:10:02.942 00:22:05.924 137 # Run SPDK target based hotplug 01:10:02.942 00:22:05.924 => 138 tgt_run_hotplug 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 01:10:02.942 00:22:05.924 ========== Backtrace end =========='\'' 01:10:02.942 00:22:05.924 ========== Backtrace start: ========== 01:10:02.942 00:22:05.924 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 710 01:10:02.942 00:22:05.924 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:02.942 00:22:05.924 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.942 00:22:05.924 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.942 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.942 00:22:05.924 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.942 00:22:05.924 716 01:10:02.942 00:22:05.924 717 echo "$time" 01:10:02.942 00:22:05.924 718 ) 01:10:02.942 00:22:05.924 719 01:10:02.942 00:22:05.924 720 function timing_enter() { 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 11 # Pci bus hotplug 01:10:02.942 00:22:05.924 12 # Helper function to remove/attach cotrollers 01:10:02.942 00:22:05.924 13 debug_remove_attach_helper() { 01:10:02.942 00:22:05.924 14 local helper_time=0 01:10:02.942 00:22:05.924 15 01:10:02.942 00:22:05.924 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.942 00:22:05.924 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.942 00:22:05.924 18 "$helper_time" "$nvme_count" >&2 01:10:02.942 00:22:05.924 19 } 01:10:02.942 00:22:05.924 20 01:10:02.942 00:22:05.924 21 remove_attach_helper() { 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.942 00:22:05.924 106 done 01:10:02.942 00:22:05.924 107 01:10:02.942 00:22:05.924 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.942 00:22:05.924 109 01:10:02.942 00:22:05.924 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.942 00:22:05.924 111 # Verify reregistering hotplug poller 01:10:02.942 00:22:05.924 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.942 00:22:05.924 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.942 00:22:05.924 114 01:10:02.942 00:22:05.924 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 133 01:10:02.942 00:22:05.924 134 # Run pci bus hotplug test 01:10:02.942 00:22:05.924 135 run_hotplug 01:10:02.942 00:22:05.924 136 01:10:02.942 00:22:05.924 137 # Run SPDK target based hotplug 01:10:02.942 00:22:05.924 => 138 tgt_run_hotplug 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 01:10:02.942 00:22:05.924 ========== Backtrace end ========== 01:10:02.942 00:22:05.924 ========== Backtrace start: ========== 01:10:02.942 00:22:05.924 01:10:02.942 00:22:05.924 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.924 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.942 00:22:05.924 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.942 00:22:05.924 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.942 00:22:05.924 716 01:10:02.942 00:22:05.924 717 echo "$time" 01:10:02.942 00:22:05.924 => 718 ) 01:10:02.942 00:22:05.924 719 01:10:02.942 00:22:05.924 720 function timing_enter() { 01:10:02.942 00:22:05.924 721 xtrace_disable 01:10:02.942 00:22:05.924 722 timing "enter" "$1" 01:10:02.942 00:22:05.924 723 xtrace_restore 01:10:02.942 00:22:05.924 ... 01:10:02.942 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.942 00:22:05.925 ... 01:10:02.942 00:22:05.925 11 # Pci bus hotplug 01:10:02.942 00:22:05.925 12 # Helper function to remove/attach cotrollers 01:10:02.942 00:22:05.925 13 debug_remove_attach_helper() { 01:10:02.942 00:22:05.925 14 local helper_time=0 01:10:02.942 00:22:05.925 15 01:10:02.942 00:22:05.925 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.942 00:22:05.925 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.942 00:22:05.925 18 "$helper_time" "$nvme_count" >&2 01:10:02.942 00:22:05.925 19 } 01:10:02.942 00:22:05.925 20 01:10:02.942 00:22:05.925 21 remove_attach_helper() { 01:10:02.942 00:22:05.925 ... 01:10:02.942 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.942 00:22:05.925 ... 01:10:02.942 00:22:05.925 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.942 00:22:05.925 106 done 01:10:02.942 00:22:05.925 107 01:10:02.942 00:22:05.925 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.942 00:22:05.925 109 01:10:02.942 00:22:05.925 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.942 00:22:05.925 111 # Verify reregistering hotplug poller 01:10:02.942 00:22:05.925 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.942 00:22:05.925 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.942 00:22:05.925 114 01:10:02.942 00:22:05.925 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.942 00:22:05.925 ... 01:10:02.942 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.942 00:22:05.925 ... 01:10:02.943 00:22:05.925 133 01:10:02.943 00:22:05.925 134 # Run pci bus hotplug test 01:10:02.943 00:22:05.925 135 run_hotplug 01:10:02.943 00:22:05.925 136 01:10:02.943 00:22:05.925 137 # Run SPDK target based hotplug 01:10:02.943 00:22:05.925 => 138 tgt_run_hotplug 01:10:02.943 00:22:05.925 ... 01:10:02.943 00:22:05.925 01:10:02.943 00:22:05.925 ========== Backtrace end ========== 01:10:02.943 00:22:05.925 ========== Backtrace start: ========== 01:10:02.943 00:22:05.925 01:10:02.943 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.943 00:22:05.925 ... 01:10:02.943 00:22:05.925 11 # Pci bus hotplug 01:10:02.943 00:22:05.925 12 # Helper function to remove/attach cotrollers 01:10:02.943 00:22:05.925 13 debug_remove_attach_helper() { 01:10:02.943 00:22:05.925 14 local helper_time=0 01:10:02.943 00:22:05.925 15 01:10:02.943 00:22:05.925 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.943 00:22:05.925 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.943 00:22:05.925 18 "$helper_time" "$nvme_count" >&2 01:10:02.943 00:22:05.925 19 } 01:10:02.943 00:22:05.925 20 01:10:02.943 00:22:05.925 21 remove_attach_helper() { 01:10:02.943 00:22:05.925 ... 01:10:02.943 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.943 00:22:05.925 ... 01:10:02.943 00:22:05.925 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.943 00:22:05.925 106 done 01:10:02.943 00:22:05.925 107 01:10:02.943 00:22:05.925 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.943 00:22:05.925 109 01:10:02.943 00:22:05.925 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.943 00:22:05.925 111 # Verify reregistering hotplug poller 01:10:02.943 00:22:05.925 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.943 00:22:05.925 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.943 00:22:05.925 114 01:10:02.943 00:22:05.925 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.943 00:22:05.925 ... 01:10:02.943 00:22:05.925 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.943 00:22:05.925 ... 01:10:02.943 00:22:05.925 133 01:10:02.943 00:22:05.925 134 # Run pci bus hotplug test 01:10:02.943 00:22:05.925 135 run_hotplug 01:10:02.943 00:22:05.925 136 01:10:02.943 00:22:05.925 137 # Run SPDK target based hotplug 01:10:02.943 00:22:05.925 => 138 tgt_run_hotplug 01:10:02.943 00:22:05.925 ... 01:10:02.943 00:22:05.925 01:10:02.943 00:22:05.925 ========== Backtrace end ========== 01:10:02.943 00:22:07.303 ========== Backtrace start: ========== 01:10:02.943 00:22:07.303 01:10:02.943 00:22:07.303 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"]) 01:10:02.943 00:22:07.303 ... 01:10:02.943 00:22:07.303 1117 timing_enter $test_name 01:10:02.943 00:22:07.303 1118 echo "************************************" 01:10:02.943 00:22:07.303 1119 echo "START TEST $test_name" 01:10:02.943 00:22:07.303 1120 echo "************************************" 01:10:02.943 00:22:07.303 1121 xtrace_restore 01:10:02.943 00:22:07.303 1122 time "$@" 01:10:02.943 00:22:07.303 1123 xtrace_disable 01:10:02.943 00:22:07.303 1124 echo "************************************" 01:10:02.943 00:22:07.303 1125 echo "END TEST $test_name" 01:10:02.943 00:22:07.303 1126 echo "************************************" 01:10:02.943 00:22:07.303 1127 timing_exit $test_name 01:10:02.943 00:22:07.303 ... 01:10:02.943 00:22:07.303 in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"]) 01:10:02.943 00:22:07.303 ... 01:10:02.943 00:22:07.303 235 01:10:02.943 00:22:07.303 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:02.943 00:22:07.303 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:02.943 00:22:07.303 238 01:10:02.943 00:22:07.303 239 if [ $(uname -s) = Linux ]; then 01:10:02.943 00:22:07.303 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:02.943 00:22:07.303 241 fi 01:10:02.943 00:22:07.303 242 01:10:02.943 00:22:07.303 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:02.943 00:22:07.303 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:02.943 00:22:07.303 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:02.943 00:22:07.303 ... 01:10:02.943 00:22:07.303 01:10:02.943 00:22:07.303 ========== Backtrace end ==========' 01:10:02.943 + for build in "${build_log_list[@]}" 01:10:02.943 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/build.log 01:10:02.943 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.943 + build_log=/var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/build.log 01:10:02.943 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/build.log 01:10:02.943 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015 01:10:02.943 + job_name=nvmf-cvl-phy-autotest_3015 01:10:02.943 + start_pattern='Backtrace start' 01:10:02.943 + end_pattern='Backtrace end' 01:10:02.943 + [[ -f /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/build.log ]] 01:10:02.943 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/build.log 01:10:02.943 + backtraces= 01:10:02.943 + [[ -n '' ]] 01:10:02.943 + for build in "${build_log_list[@]}" 01:10:02.943 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/build.log 01:10:02.943 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.943 + build_log=/var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/build.log 01:10:02.943 +++ dirname /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/build.log 01:10:02.943 ++ basename /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776 01:10:02.943 + job_name=ubuntu20-vg-autotest_73776 01:10:02.943 + start_pattern='Backtrace start' 01:10:02.943 + end_pattern='Backtrace end' 01:10:02.943 + [[ -f /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/build.log ]] 01:10:02.943 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/build.log 01:10:02.943 + backtraces='00:54:55.554 04:19:44 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ========== 01:10:02.943 00:54:55.554 01:10:02.943 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.943 00:54:55.554 ... 01:10:02.943 00:54:55.554 710 01:10:02.943 00:54:55.554 711 # We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to 01:10:02.943 00:54:55.554 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.943 00:54:55.554 713 # but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do: 01:10:02.943 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.943 00:54:55.554 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.943 00:54:55.554 716 01:10:02.943 00:54:55.554 717 echo "$time" 01:10:02.943 00:54:55.554 718 ) 01:10:02.943 00:54:55.554 719 01:10:02.943 00:54:55.554 720 function timing_enter() { 01:10:02.943 00:54:55.554 ... 01:10:02.943 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.943 00:54:55.554 ... 01:10:02.943 00:54:55.554 11 # Pci bus hotplug 01:10:02.943 00:54:55.554 12 # Helper function to remove/attach cotrollers 01:10:02.943 00:54:55.554 13 debug_remove_attach_helper() { 01:10:02.943 00:54:55.554 14 local helper_time=0 01:10:02.943 00:54:55.554 15 01:10:02.943 00:54:55.554 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.943 00:54:55.554 17 printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \ 01:10:02.943 00:54:55.554 18 "$helper_time" "$nvme_count" >&2 01:10:02.943 00:54:55.554 19 } 01:10:02.943 00:54:55.554 20 01:10:02.943 00:54:55.554 21 remove_attach_helper() { 01:10:02.943 00:54:55.554 ... 01:10:02.943 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.943 00:54:55.554 ... 01:10:02.944 00:54:55.554 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.944 00:54:55.554 106 done 01:10:02.944 00:54:55.554 107 01:10:02.944 00:54:55.554 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.944 00:54:55.554 109 01:10:02.944 00:54:55.554 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.944 00:54:55.554 111 # Verify reregistering hotplug poller 01:10:02.944 00:54:55.554 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.944 00:54:55.554 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.944 00:54:55.554 114 01:10:02.944 00:54:55.554 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 133 01:10:02.944 00:54:55.554 134 # Run pci bus hotplug test 01:10:02.944 00:54:55.554 135 run_hotplug 01:10:02.944 00:54:55.554 136 01:10:02.944 00:54:55.554 137 # Run SPDK target based hotplug 01:10:02.944 00:54:55.554 => 138 tgt_run_hotplug 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 01:10:02.944 00:54:55.554 ========== Backtrace end ========== 01:10:02.944 00:54:55.554 ========== Backtrace start: ========== 01:10:02.944 00:54:55.554 01:10:02.944 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 710 01:10:02.944 00:54:55.554 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:02.944 00:54:55.554 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.944 00:54:55.554 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.944 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.944 00:54:55.554 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.944 00:54:55.554 716 01:10:02.944 00:54:55.554 717 echo "$time" 01:10:02.944 00:54:55.554 718 ) 01:10:02.944 00:54:55.554 719 01:10:02.944 00:54:55.554 720 function timing_enter() { 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 11 # Pci bus hotplug 01:10:02.944 00:54:55.554 12 # Helper function to remove/attach cotrollers 01:10:02.944 00:54:55.554 13 debug_remove_attach_helper() { 01:10:02.944 00:54:55.554 14 local helper_time=0 01:10:02.944 00:54:55.554 15 01:10:02.944 00:54:55.554 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.944 00:54:55.554 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.944 00:54:55.554 18 "$helper_time" "$nvme_count" >&2 01:10:02.944 00:54:55.554 19 } 01:10:02.944 00:54:55.554 20 01:10:02.944 00:54:55.554 21 remove_attach_helper() { 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.944 00:54:55.554 106 done 01:10:02.944 00:54:55.554 107 01:10:02.944 00:54:55.554 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.944 00:54:55.554 109 01:10:02.944 00:54:55.554 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.944 00:54:55.554 111 # Verify reregistering hotplug poller 01:10:02.944 00:54:55.554 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.944 00:54:55.554 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.944 00:54:55.554 114 01:10:02.944 00:54:55.554 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 133 01:10:02.944 00:54:55.554 134 # Run pci bus hotplug test 01:10:02.944 00:54:55.554 135 run_hotplug 01:10:02.944 00:54:55.554 136 01:10:02.944 00:54:55.554 137 # Run SPDK target based hotplug 01:10:02.944 00:54:55.554 => 138 tgt_run_hotplug 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 01:10:02.944 00:54:55.554 ========== Backtrace end ========== 01:10:02.944 00:54:55.554 ========== Backtrace start: ========== 01:10:02.944 00:54:55.554 01:10:02.944 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.944 00:54:55.554 ... 01:10:02.944 00:54:55.554 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.944 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.944 00:54:55.554 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.944 00:54:55.555 716 01:10:02.944 00:54:55.555 717 echo "$time" 01:10:02.944 00:54:55.555 => 718 ) 01:10:02.944 00:54:55.555 719 01:10:02.944 00:54:55.555 720 function timing_enter() { 01:10:02.944 00:54:55.555 721 xtrace_disable 01:10:02.944 00:54:55.555 722 timing "enter" "$1" 01:10:02.944 00:54:55.555 723 xtrace_restore 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 11 # Pci bus hotplug 01:10:02.944 00:54:55.555 12 # Helper function to remove/attach cotrollers 01:10:02.944 00:54:55.555 13 debug_remove_attach_helper() { 01:10:02.944 00:54:55.555 14 local helper_time=0 01:10:02.944 00:54:55.555 15 01:10:02.944 00:54:55.555 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.944 00:54:55.555 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.944 00:54:55.555 18 "$helper_time" "$nvme_count" >&2 01:10:02.944 00:54:55.555 19 } 01:10:02.944 00:54:55.555 20 01:10:02.944 00:54:55.555 21 remove_attach_helper() { 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.944 00:54:55.555 106 done 01:10:02.944 00:54:55.555 107 01:10:02.944 00:54:55.555 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.944 00:54:55.555 109 01:10:02.944 00:54:55.555 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.944 00:54:55.555 111 # Verify reregistering hotplug poller 01:10:02.944 00:54:55.555 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.944 00:54:55.555 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.944 00:54:55.555 114 01:10:02.944 00:54:55.555 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 133 01:10:02.944 00:54:55.555 134 # Run pci bus hotplug test 01:10:02.944 00:54:55.555 135 run_hotplug 01:10:02.944 00:54:55.555 136 01:10:02.944 00:54:55.555 137 # Run SPDK target based hotplug 01:10:02.944 00:54:55.555 => 138 tgt_run_hotplug 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 01:10:02.944 00:54:55.555 ========== Backtrace end ========== 01:10:02.944 00:54:55.555 ========== Backtrace start: ========== 01:10:02.944 00:54:55.555 01:10:02.944 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 11 # Pci bus hotplug 01:10:02.944 00:54:55.555 12 # Helper function to remove/attach cotrollers 01:10:02.944 00:54:55.555 13 debug_remove_attach_helper() { 01:10:02.944 00:54:55.555 14 local helper_time=0 01:10:02.944 00:54:55.555 15 01:10:02.944 00:54:55.555 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.944 00:54:55.555 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.944 00:54:55.555 18 "$helper_time" "$nvme_count" >&2 01:10:02.944 00:54:55.555 19 } 01:10:02.944 00:54:55.555 20 01:10:02.944 00:54:55.555 21 remove_attach_helper() { 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.944 00:54:55.555 106 done 01:10:02.944 00:54:55.555 107 01:10:02.944 00:54:55.555 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.944 00:54:55.555 109 01:10:02.944 00:54:55.555 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.944 00:54:55.555 111 # Verify reregistering hotplug poller 01:10:02.944 00:54:55.555 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.944 00:54:55.555 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.944 00:54:55.555 114 01:10:02.944 00:54:55.555 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 133 01:10:02.944 00:54:55.555 134 # Run pci bus hotplug test 01:10:02.944 00:54:55.555 135 run_hotplug 01:10:02.944 00:54:55.555 136 01:10:02.944 00:54:55.555 137 # Run SPDK target based hotplug 01:10:02.944 00:54:55.555 => 138 tgt_run_hotplug 01:10:02.944 00:54:55.555 ... 01:10:02.944 00:54:55.555 01:10:02.944 00:54:55.555 ========== Backtrace end ========== 01:10:02.944 00:54:57.195 ========== Backtrace start: ========== 01:10:02.944 00:54:57.195 01:10:02.944 00:54:57.195 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"]) 01:10:02.944 00:54:57.195 ... 01:10:02.944 00:54:57.195 1117 timing_enter $test_name 01:10:02.944 00:54:57.195 1118 echo "************************************" 01:10:02.944 00:54:57.195 1119 echo "START TEST $test_name" 01:10:02.944 00:54:57.195 1120 echo "************************************" 01:10:02.944 00:54:57.195 1121 xtrace_restore 01:10:02.944 00:54:57.195 1122 time "$@" 01:10:02.944 00:54:57.195 1123 xtrace_disable 01:10:02.944 00:54:57.195 1124 echo "************************************" 01:10:02.944 00:54:57.195 1125 echo "END TEST $test_name" 01:10:02.944 00:54:57.195 1126 echo "************************************" 01:10:02.944 00:54:57.195 1127 timing_exit $test_name 01:10:02.944 00:54:57.195 ... 01:10:02.944 00:54:57.195 in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"]) 01:10:02.945 00:54:57.195 ... 01:10:02.945 00:54:57.195 235 01:10:02.945 00:54:57.195 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:02.945 00:54:57.195 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:02.945 00:54:57.195 238 01:10:02.945 00:54:57.195 239 if [ $(uname -s) = Linux ]; then 01:10:02.945 00:54:57.195 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:02.945 00:54:57.195 241 fi 01:10:02.945 00:54:57.195 242 01:10:02.945 00:54:57.195 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:02.945 00:54:57.195 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:02.945 00:54:57.195 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:02.945 00:54:57.195 ... 01:10:02.945 00:54:57.195 01:10:02.945 00:54:57.195 ========== Backtrace end ==========' 01:10:02.945 + [[ -n 00:54:55.554 04:19:44 sw_hotplug -- common/autotest_common.sh@715 -- # time='========== Backtrace start: ========== 01:10:02.945 00:54:55.554 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 710 01:10:02.945 00:54:55.554 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:02.945 00:54:55.554 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.945 00:54:55.554 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.945 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.945 00:54:55.554 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.945 00:54:55.554 716 01:10:02.945 00:54:55.554 717 echo "$time" 01:10:02.945 00:54:55.554 718 ) 01:10:02.945 00:54:55.554 719 01:10:02.945 00:54:55.554 720 function timing_enter() { 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 11 # Pci bus hotplug 01:10:02.945 00:54:55.554 12 # Helper function to remove/attach cotrollers 01:10:02.945 00:54:55.554 13 debug_remove_attach_helper() { 01:10:02.945 00:54:55.554 14 local helper_time=0 01:10:02.945 00:54:55.554 15 01:10:02.945 00:54:55.554 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.945 00:54:55.554 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.945 00:54:55.554 18 "$helper_time" "$nvme_count" >&2 01:10:02.945 00:54:55.554 19 } 01:10:02.945 00:54:55.554 20 01:10:02.945 00:54:55.554 21 remove_attach_helper() { 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.945 00:54:55.554 106 done 01:10:02.945 00:54:55.554 107 01:10:02.945 00:54:55.554 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.945 00:54:55.554 109 01:10:02.945 00:54:55.554 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.945 00:54:55.554 111 # Verify reregistering hotplug poller 01:10:02.945 00:54:55.554 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.945 00:54:55.554 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.945 00:54:55.554 114 01:10:02.945 00:54:55.554 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 133 01:10:02.945 00:54:55.554 134 # Run pci bus hotplug test 01:10:02.945 00:54:55.554 135 run_hotplug 01:10:02.945 00:54:55.554 136 01:10:02.945 00:54:55.554 137 # Run SPDK target based hotplug 01:10:02.945 00:54:55.554 => 138 tgt_run_hotplug 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 01:10:02.945 00:54:55.554 ========== Backtrace end ========== 01:10:02.945 00:54:55.554 ========== Backtrace start: ========== 01:10:02.945 00:54:55.554 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 710 01:10:02.945 00:54:55.554 711 # We redirect cmd's std{out,err} to a separate fd dup'ed to stdin's device (or /dev/null) to 01:10:02.945 00:54:55.554 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.945 00:54:55.554 713 # but $() will return just the time's data, hence making it possible to just do: 01:10:02.945 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.945 00:54:55.554 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.945 00:54:55.554 716 01:10:02.945 00:54:55.554 717 echo "$time" 01:10:02.945 00:54:55.554 718 ) 01:10:02.945 00:54:55.554 719 01:10:02.945 00:54:55.554 720 function timing_enter() { 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 11 # Pci bus hotplug 01:10:02.945 00:54:55.554 12 # Helper function to remove/attach cotrollers 01:10:02.945 00:54:55.554 13 debug_remove_attach_helper() { 01:10:02.945 00:54:55.554 14 local helper_time=0 01:10:02.945 00:54:55.554 15 01:10:02.945 00:54:55.554 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.945 00:54:55.554 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:02.945 00:54:55.554 18 "$helper_time" "$nvme_count" >&2 01:10:02.945 00:54:55.554 19 } 01:10:02.945 00:54:55.554 20 01:10:02.945 00:54:55.554 21 remove_attach_helper() { 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.945 00:54:55.554 106 done 01:10:02.945 00:54:55.554 107 01:10:02.945 00:54:55.554 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.945 00:54:55.554 109 01:10:02.945 00:54:55.554 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.945 00:54:55.554 111 # Verify reregistering hotplug poller 01:10:02.945 00:54:55.554 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.945 00:54:55.554 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.945 00:54:55.554 114 01:10:02.945 00:54:55.554 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 133 01:10:02.945 00:54:55.554 134 # Run pci bus hotplug test 01:10:02.945 00:54:55.554 135 run_hotplug 01:10:02.945 00:54:55.554 136 01:10:02.945 00:54:55.554 137 # Run SPDK target based hotplug 01:10:02.945 00:54:55.554 => 138 tgt_run_hotplug 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 01:10:02.945 00:54:55.554 ========== Backtrace end ========== 01:10:02.945 00:54:55.554 ========== Backtrace start: ========== 01:10:02.945 00:54:55.554 01:10:02.945 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.945 00:54:55.554 ... 01:10:02.945 00:54:55.554 713 # but $() will return just the time's data, hence making it possible to just do: 01:10:02.945 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.945 00:54:55.554 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.945 00:54:55.555 716 01:10:02.945 00:54:55.555 717 echo "$time" 01:10:02.945 00:54:55.555 => 718 ) 01:10:02.945 00:54:55.555 719 01:10:02.945 00:54:55.555 720 function timing_enter() { 01:10:02.945 00:54:55.555 721 xtrace_disable 01:10:02.945 00:54:55.555 722 timing "enter" "$1" 01:10:02.945 00:54:55.555 723 xtrace_restore 01:10:02.945 00:54:55.555 ... 01:10:02.945 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.945 00:54:55.555 ... 01:10:02.945 00:54:55.555 11 # Pci bus hotplug 01:10:02.945 00:54:55.555 12 # Helper function to remove/attach cotrollers 01:10:02.945 00:54:55.555 13 debug_remove_attach_helper() { 01:10:02.945 00:54:55.555 14 local helper_time=0 01:10:02.945 00:54:55.555 15 01:10:02.945 00:54:55.555 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.945 00:54:55.555 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:02.945 00:54:55.555 18 "$helper_time" "$nvme_count" >&2 01:10:02.945 00:54:55.555 19 } 01:10:02.945 00:54:55.555 20 01:10:02.945 00:54:55.555 21 remove_attach_helper() { 01:10:02.945 00:54:55.555 ... 01:10:02.945 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.945 00:54:55.555 ... 01:10:02.945 00:54:55.555 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.945 00:54:55.555 106 done 01:10:02.945 00:54:55.555 107 01:10:02.945 00:54:55.555 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.945 00:54:55.555 109 01:10:02.945 00:54:55.555 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.945 00:54:55.555 111 # Verify reregistering hotplug poller 01:10:02.945 00:54:55.555 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.945 00:54:55.555 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.945 00:54:55.555 114 01:10:02.945 00:54:55.555 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.945 00:54:55.555 ... 01:10:02.945 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.945 00:54:55.555 ... 01:10:02.945 00:54:55.555 133 01:10:02.945 00:54:55.555 134 # Run pci bus hotplug test 01:10:02.945 00:54:55.555 135 run_hotplug 01:10:02.945 00:54:55.555 136 01:10:02.945 00:54:55.555 137 # Run SPDK target based hotplug 01:10:02.945 00:54:55.555 => 138 tgt_run_hotplug 01:10:02.945 00:54:55.555 ... 01:10:02.945 00:54:55.555 01:10:02.945 00:54:55.555 ========== Backtrace end ========== 01:10:02.945 00:54:55.555 ========== Backtrace start: ========== 01:10:02.945 00:54:55.555 01:10:02.945 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.945 00:54:55.555 ... 01:10:02.945 00:54:55.555 11 # Pci bus hotplug 01:10:02.945 00:54:55.555 12 # Helper function to remove/attach cotrollers 01:10:02.945 00:54:55.555 13 debug_remove_attach_helper() { 01:10:02.945 00:54:55.555 14 local helper_time=0 01:10:02.945 00:54:55.555 15 01:10:02.946 00:54:55.555 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.946 00:54:55.555 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:02.946 00:54:55.555 18 "$helper_time" "$nvme_count" >&2 01:10:02.946 00:54:55.555 19 } 01:10:02.946 00:54:55.555 20 01:10:02.946 00:54:55.555 21 remove_attach_helper() { 01:10:02.946 00:54:55.555 ... 01:10:02.946 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.946 00:54:55.555 ... 01:10:02.946 00:54:55.555 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.946 00:54:55.555 106 done 01:10:02.946 00:54:55.555 107 01:10:02.946 00:54:55.555 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.946 00:54:55.555 109 01:10:02.946 00:54:55.555 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.946 00:54:55.555 111 # Verify reregistering hotplug poller 01:10:02.946 00:54:55.555 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.946 00:54:55.555 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.946 00:54:55.555 114 01:10:02.946 00:54:55.555 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.946 00:54:55.555 ... 01:10:02.946 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.946 00:54:55.555 ... 01:10:02.946 00:54:55.555 133 01:10:02.946 00:54:55.555 134 # Run pci bus hotplug test 01:10:02.946 00:54:55.555 135 run_hotplug 01:10:02.946 00:54:55.555 136 01:10:02.946 00:54:55.555 137 # Run SPDK target based hotplug 01:10:02.946 00:54:55.555 => 138 tgt_run_hotplug 01:10:02.946 00:54:55.555 ... 01:10:02.946 00:54:55.555 01:10:02.946 00:54:55.555 ========== Backtrace end ========== 01:10:02.946 00:54:57.195 ========== Backtrace start: ========== 01:10:02.946 00:54:57.195 01:10:02.946 00:54:57.195 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"]) 01:10:02.946 00:54:57.195 ... 01:10:02.946 00:54:57.195 1117 timing_enter $test_name 01:10:02.946 00:54:57.195 1118 echo "************************************" 01:10:02.946 00:54:57.195 1119 echo "START TEST $test_name" 01:10:02.946 00:54:57.195 1120 echo "************************************" 01:10:02.946 00:54:57.195 1121 xtrace_restore 01:10:02.946 00:54:57.195 1122 time "$@" 01:10:02.946 00:54:57.195 1123 xtrace_disable 01:10:02.946 00:54:57.195 1124 echo "************************************" 01:10:02.946 00:54:57.195 1125 echo "END TEST $test_name" 01:10:02.946 00:54:57.195 1126 echo "************************************" 01:10:02.946 00:54:57.195 1127 timing_exit $test_name 01:10:02.946 00:54:57.195 ... 01:10:02.946 00:54:57.195 in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"]) 01:10:02.946 00:54:57.195 ... 01:10:02.946 00:54:57.195 235 01:10:02.946 00:54:57.195 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:02.946 00:54:57.195 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:02.946 00:54:57.195 238 01:10:02.946 00:54:57.195 239 if [ $(uname -s) = Linux ]; then 01:10:02.946 00:54:57.195 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:02.946 00:54:57.195 241 fi 01:10:02.946 00:54:57.195 242 01:10:02.946 00:54:57.195 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:02.946 00:54:57.195 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:02.946 00:54:57.195 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:02.946 00:54:57.195 ... 01:10:02.946 00:54:57.195 01:10:02.946 00:54:57.195 ========== Backtrace end ========== ]] 01:10:02.946 + echo '00:54:55.554 04:19:44 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ========== 01:10:02.946 00:54:55.554 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 710 01:10:02.946 00:54:55.554 711 # We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to 01:10:02.946 00:54:55.554 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.946 00:54:55.554 713 # but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do: 01:10:02.946 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.946 00:54:55.554 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.946 00:54:55.554 716 01:10:02.946 00:54:55.554 717 echo "$time" 01:10:02.946 00:54:55.554 718 ) 01:10:02.946 00:54:55.554 719 01:10:02.946 00:54:55.554 720 function timing_enter() { 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 11 # Pci bus hotplug 01:10:02.946 00:54:55.554 12 # Helper function to remove/attach cotrollers 01:10:02.946 00:54:55.554 13 debug_remove_attach_helper() { 01:10:02.946 00:54:55.554 14 local helper_time=0 01:10:02.946 00:54:55.554 15 01:10:02.946 00:54:55.554 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.946 00:54:55.554 17 printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \ 01:10:02.946 00:54:55.554 18 "$helper_time" "$nvme_count" >&2 01:10:02.946 00:54:55.554 19 } 01:10:02.946 00:54:55.554 20 01:10:02.946 00:54:55.554 21 remove_attach_helper() { 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.946 00:54:55.554 106 done 01:10:02.946 00:54:55.554 107 01:10:02.946 00:54:55.554 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.946 00:54:55.554 109 01:10:02.946 00:54:55.554 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.946 00:54:55.554 111 # Verify reregistering hotplug poller 01:10:02.946 00:54:55.554 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.946 00:54:55.554 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.946 00:54:55.554 114 01:10:02.946 00:54:55.554 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 133 01:10:02.946 00:54:55.554 134 # Run pci bus hotplug test 01:10:02.946 00:54:55.554 135 run_hotplug 01:10:02.946 00:54:55.554 136 01:10:02.946 00:54:55.554 137 # Run SPDK target based hotplug 01:10:02.946 00:54:55.554 => 138 tgt_run_hotplug 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 01:10:02.946 00:54:55.554 ========== Backtrace end ========== 01:10:02.946 00:54:55.554 ========== Backtrace start: ========== 01:10:02.946 00:54:55.554 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 710 01:10:02.946 00:54:55.554 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:02.946 00:54:55.554 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:02.946 00:54:55.554 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.946 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.946 00:54:55.554 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.946 00:54:55.554 716 01:10:02.946 00:54:55.554 717 echo "$time" 01:10:02.946 00:54:55.554 718 ) 01:10:02.946 00:54:55.554 719 01:10:02.946 00:54:55.554 720 function timing_enter() { 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 11 # Pci bus hotplug 01:10:02.946 00:54:55.554 12 # Helper function to remove/attach cotrollers 01:10:02.946 00:54:55.554 13 debug_remove_attach_helper() { 01:10:02.946 00:54:55.554 14 local helper_time=0 01:10:02.946 00:54:55.554 15 01:10:02.946 00:54:55.554 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.946 00:54:55.554 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.946 00:54:55.554 18 "$helper_time" "$nvme_count" >&2 01:10:02.946 00:54:55.554 19 } 01:10:02.946 00:54:55.554 20 01:10:02.946 00:54:55.554 21 remove_attach_helper() { 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.946 00:54:55.554 106 done 01:10:02.946 00:54:55.554 107 01:10:02.946 00:54:55.554 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.946 00:54:55.554 109 01:10:02.946 00:54:55.554 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.946 00:54:55.554 111 # Verify reregistering hotplug poller 01:10:02.946 00:54:55.554 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.946 00:54:55.554 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.946 00:54:55.554 114 01:10:02.946 00:54:55.554 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 133 01:10:02.946 00:54:55.554 134 # Run pci bus hotplug test 01:10:02.946 00:54:55.554 135 run_hotplug 01:10:02.946 00:54:55.554 136 01:10:02.946 00:54:55.554 137 # Run SPDK target based hotplug 01:10:02.946 00:54:55.554 => 138 tgt_run_hotplug 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 01:10:02.946 00:54:55.554 ========== Backtrace end ========== 01:10:02.946 00:54:55.554 ========== Backtrace start: ========== 01:10:02.946 00:54:55.554 01:10:02.946 00:54:55.554 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:02.946 00:54:55.554 ... 01:10:02.946 00:54:55.554 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:02.946 00:54:55.554 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:02.946 00:54:55.554 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:02.946 00:54:55.555 716 01:10:02.946 00:54:55.555 717 echo "$time" 01:10:02.946 00:54:55.555 => 718 ) 01:10:02.946 00:54:55.555 719 01:10:02.946 00:54:55.555 720 function timing_enter() { 01:10:02.947 00:54:55.555 721 xtrace_disable 01:10:02.947 00:54:55.555 722 timing "enter" "$1" 01:10:02.947 00:54:55.555 723 xtrace_restore 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 11 # Pci bus hotplug 01:10:02.947 00:54:55.555 12 # Helper function to remove/attach cotrollers 01:10:02.947 00:54:55.555 13 debug_remove_attach_helper() { 01:10:02.947 00:54:55.555 14 local helper_time=0 01:10:02.947 00:54:55.555 15 01:10:02.947 00:54:55.555 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.947 00:54:55.555 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.947 00:54:55.555 18 "$helper_time" "$nvme_count" >&2 01:10:02.947 00:54:55.555 19 } 01:10:02.947 00:54:55.555 20 01:10:02.947 00:54:55.555 21 remove_attach_helper() { 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.947 00:54:55.555 106 done 01:10:02.947 00:54:55.555 107 01:10:02.947 00:54:55.555 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.947 00:54:55.555 109 01:10:02.947 00:54:55.555 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.947 00:54:55.555 111 # Verify reregistering hotplug poller 01:10:02.947 00:54:55.555 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.947 00:54:55.555 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.947 00:54:55.555 114 01:10:02.947 00:54:55.555 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 133 01:10:02.947 00:54:55.555 134 # Run pci bus hotplug test 01:10:02.947 00:54:55.555 135 run_hotplug 01:10:02.947 00:54:55.555 136 01:10:02.947 00:54:55.555 137 # Run SPDK target based hotplug 01:10:02.947 00:54:55.555 => 138 tgt_run_hotplug 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 01:10:02.947 00:54:55.555 ========== Backtrace end ========== 01:10:02.947 00:54:55.555 ========== Backtrace start: ========== 01:10:02.947 00:54:55.555 01:10:02.947 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 11 # Pci bus hotplug 01:10:02.947 00:54:55.555 12 # Helper function to remove/attach cotrollers 01:10:02.947 00:54:55.555 13 debug_remove_attach_helper() { 01:10:02.947 00:54:55.555 14 local helper_time=0 01:10:02.947 00:54:55.555 15 01:10:02.947 00:54:55.555 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:02.947 00:54:55.555 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:02.947 00:54:55.555 18 "$helper_time" "$nvme_count" >&2 01:10:02.947 00:54:55.555 19 } 01:10:02.947 00:54:55.555 20 01:10:02.947 00:54:55.555 21 remove_attach_helper() { 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:02.947 00:54:55.555 106 done 01:10:02.947 00:54:55.555 107 01:10:02.947 00:54:55.555 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.947 00:54:55.555 109 01:10:02.947 00:54:55.555 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.947 00:54:55.555 111 # Verify reregistering hotplug poller 01:10:02.947 00:54:55.555 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:02.947 00:54:55.555 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:02.947 00:54:55.555 114 01:10:02.947 00:54:55.555 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 in /home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 133 01:10:02.947 00:54:55.555 134 # Run pci bus hotplug test 01:10:02.947 00:54:55.555 135 run_hotplug 01:10:02.947 00:54:55.555 136 01:10:02.947 00:54:55.555 137 # Run SPDK target based hotplug 01:10:02.947 00:54:55.555 => 138 tgt_run_hotplug 01:10:02.947 00:54:55.555 ... 01:10:02.947 00:54:55.555 01:10:02.947 00:54:55.555 ========== Backtrace end ========== 01:10:02.947 00:54:57.195 ========== Backtrace start: ========== 01:10:02.947 00:54:57.195 01:10:02.947 00:54:57.195 in /home/vagrant/spdk_repo/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/home/vagrant/spdk_repo/spdk/test/nvme/sw_hotplug.sh"]) 01:10:02.947 00:54:57.195 ... 01:10:02.947 00:54:57.195 1117 timing_enter $test_name 01:10:02.947 00:54:57.195 1118 echo "************************************" 01:10:02.947 00:54:57.195 1119 echo "START TEST $test_name" 01:10:02.947 00:54:57.195 1120 echo "************************************" 01:10:02.947 00:54:57.195 1121 xtrace_restore 01:10:02.947 00:54:57.195 1122 time "$@" 01:10:02.947 00:54:57.195 1123 xtrace_disable 01:10:02.947 00:54:57.195 1124 echo "************************************" 01:10:02.947 00:54:57.195 1125 echo "END TEST $test_name" 01:10:02.947 00:54:57.195 1126 echo "************************************" 01:10:02.947 00:54:57.195 1127 timing_exit $test_name 01:10:02.947 00:54:57.195 ... 01:10:02.947 00:54:57.195 in /home/vagrant/spdk_repo/spdk/autotest.sh:240 -> main(["/home/vagrant/spdk_repo/autorun-spdk.conf"]) 01:10:02.947 00:54:57.195 ... 01:10:02.947 00:54:57.195 235 01:10:02.947 00:54:57.195 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:02.947 00:54:57.195 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:02.947 00:54:57.195 238 01:10:02.947 00:54:57.195 239 if [ $(uname -s) = Linux ]; then 01:10:02.947 00:54:57.195 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:02.947 00:54:57.195 241 fi 01:10:02.947 00:54:57.195 242 01:10:02.947 00:54:57.195 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:02.947 00:54:57.195 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:02.947 00:54:57.195 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:02.947 00:54:57.195 ... 01:10:02.947 00:54:57.195 01:10:02.947 00:54:57.195 ========== Backtrace end ==========' 01:10:02.947 + for build in "${build_log_list[@]}" 01:10:02.947 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/build.log 01:10:02.947 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.947 + build_log=/var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/build.log 01:10:02.947 +++ dirname /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/build.log 01:10:02.947 ++ basename /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183 01:10:02.947 + job_name=BlobFS-phy-autotest_26183 01:10:02.947 + start_pattern='Backtrace start' 01:10:02.947 + end_pattern='Backtrace end' 01:10:02.947 + [[ -f /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/build.log ]] 01:10:02.947 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/build.log 01:10:02.947 + backtraces= 01:10:02.947 + [[ -n '' ]] 01:10:02.947 + for build in "${build_log_list[@]}" 01:10:02.947 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/build.log 01:10:02.947 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.947 + build_log=/var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/build.log 01:10:02.947 +++ dirname /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/build.log 01:10:02.947 ++ basename /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563 01:10:02.947 + job_name=zns-vg-autotest_76563 01:10:02.947 + start_pattern='Backtrace start' 01:10:02.947 + end_pattern='Backtrace end' 01:10:02.947 + [[ -f /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/build.log ]] 01:10:02.947 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/build.log 01:10:02.947 + backtraces= 01:10:02.947 + [[ -n '' ]] 01:10:02.947 + for build in "${build_log_list[@]}" 01:10:02.947 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/build.log 01:10:02.947 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.947 + build_log=/var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/build.log 01:10:02.947 +++ dirname /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/build.log 01:10:02.947 ++ basename /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202 01:10:02.947 + job_name=vhost-phy-autotest_26202 01:10:02.947 + start_pattern='Backtrace start' 01:10:02.947 + end_pattern='Backtrace end' 01:10:02.947 + [[ -f /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/build.log ]] 01:10:02.947 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/build.log 01:10:02.947 + backtraces= 01:10:02.947 + [[ -n '' ]] 01:10:02.947 + for build in "${build_log_list[@]}" 01:10:02.947 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/build.log 01:10:02.947 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.947 + build_log=/var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/build.log 01:10:02.947 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/build.log 01:10:02.947 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382 01:10:02.947 + job_name=nvmf-tcp-uring-vg-autotest_77382 01:10:02.947 + start_pattern='Backtrace start' 01:10:02.947 + end_pattern='Backtrace end' 01:10:02.947 + [[ -f /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/build.log ]] 01:10:02.947 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/build.log 01:10:02.947 + backtraces= 01:10:02.947 + [[ -n '' ]] 01:10:02.947 + for build in "${build_log_list[@]}" 01:10:02.947 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/build.log 01:10:02.947 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.947 + build_log=/var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/build.log 01:10:02.947 +++ dirname /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/build.log 01:10:02.947 ++ basename /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174 01:10:02.947 + job_name=short-fuzz-phy-autotest_26174 01:10:02.947 + start_pattern='Backtrace start' 01:10:02.947 + end_pattern='Backtrace end' 01:10:02.947 + [[ -f /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/build.log ]] 01:10:02.947 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/build.log 01:10:02.947 + backtraces= 01:10:02.947 + [[ -n '' ]] 01:10:02.947 + for build in "${build_log_list[@]}" 01:10:02.948 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/build.log 01:10:02.948 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.948 + build_log=/var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/build.log 01:10:02.948 +++ dirname /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/build.log 01:10:02.948 ++ basename /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125 01:10:02.948 + job_name=check-format-docker-autotest_22125 01:10:02.948 + start_pattern='Backtrace start' 01:10:02.948 + end_pattern='Backtrace end' 01:10:02.948 + [[ -f /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/build.log ]] 01:10:02.948 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/build.log 01:10:02.948 + backtraces= 01:10:02.948 + [[ -n '' ]] 01:10:02.948 + for build in "${build_log_list[@]}" 01:10:02.948 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/build.log 01:10:02.948 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.948 + build_log=/var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/build.log 01:10:02.948 +++ dirname /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/build.log 01:10:02.948 ++ basename /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495 01:10:02.948 + job_name=iscsi-vg-autotest_77495 01:10:02.948 + start_pattern='Backtrace start' 01:10:02.948 + end_pattern='Backtrace end' 01:10:02.948 + [[ -f /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/build.log ]] 01:10:02.948 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/build.log 01:10:02.948 + backtraces= 01:10:02.948 + [[ -n '' ]] 01:10:02.948 + for build in "${build_log_list[@]}" 01:10:02.948 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/build.log 01:10:02.948 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.948 + build_log=/var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/build.log 01:10:02.948 +++ dirname /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/build.log 01:10:02.948 ++ basename /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842 01:10:02.948 + job_name=unittest-clang-docker-autotest_19842 01:10:02.948 + start_pattern='Backtrace start' 01:10:02.948 + end_pattern='Backtrace end' 01:10:02.948 + [[ -f /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/build.log ]] 01:10:02.948 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/build.log 01:10:02.948 + backtraces= 01:10:02.948 + [[ -n '' ]] 01:10:02.948 + for build in "${build_log_list[@]}" 01:10:02.948 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/build.log 01:10:02.948 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.948 + build_log=/var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/build.log 01:10:02.948 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/build.log 01:10:02.948 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421 01:10:02.948 + job_name=nvmf-phy-autotest_72421 01:10:02.948 + start_pattern='Backtrace start' 01:10:02.948 + end_pattern='Backtrace end' 01:10:02.948 + [[ -f /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/build.log ]] 01:10:02.948 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/build.log 01:10:02.948 + backtraces= 01:10:02.948 + [[ -n '' ]] 01:10:02.948 + for build in "${build_log_list[@]}" 01:10:02.948 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/build.log 01:10:02.948 + local build_log job_name start_pattern end_pattern backtraces 01:10:02.948 + build_log=/var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/build.log 01:10:02.948 +++ dirname /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/build.log 01:10:02.948 ++ basename /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349 01:10:02.948 + job_name=nvme-phy-autotest_72349 01:10:02.948 + start_pattern='Backtrace start' 01:10:02.948 + end_pattern='Backtrace end' 01:10:02.948 + [[ -f /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/build.log ]] 01:10:02.948 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/build.log 01:10:03.209 + backtraces='00:14:31.430 05:39:20 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ========== 01:10:03.209 00:14:31.430 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 710 01:10:03.209 00:14:31.430 711 # We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to 01:10:03.209 00:14:31.430 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:03.209 00:14:31.430 713 # but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do: 01:10:03.209 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.209 00:14:31.430 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.209 00:14:31.430 716 01:10:03.209 00:14:31.430 717 echo "$time" 01:10:03.209 00:14:31.430 718 ) 01:10:03.209 00:14:31.430 719 01:10:03.209 00:14:31.430 720 function timing_enter() { 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 11 # Pci bus hotplug 01:10:03.209 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.209 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.209 00:14:31.430 14 local helper_time=0 01:10:03.209 00:14:31.430 15 01:10:03.209 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.209 00:14:31.430 17 printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \ 01:10:03.209 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.209 00:14:31.430 19 } 01:10:03.209 00:14:31.430 20 01:10:03.209 00:14:31.430 21 remove_attach_helper() { 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.209 00:14:31.430 106 done 01:10:03.209 00:14:31.430 107 01:10:03.209 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.209 00:14:31.430 109 01:10:03.209 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.209 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.209 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.209 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.209 00:14:31.430 114 01:10:03.209 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 133 01:10:03.209 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.209 00:14:31.430 135 run_hotplug 01:10:03.209 00:14:31.430 136 01:10:03.209 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.209 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 01:10:03.209 00:14:31.430 ========== Backtrace end =========='\'' 01:10:03.209 00:14:31.430 ========== Backtrace start: ========== 01:10:03.209 00:14:31.430 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 710 01:10:03.209 00:14:31.430 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:03.209 00:14:31.430 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:03.209 00:14:31.430 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:03.209 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.209 00:14:31.430 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.209 00:14:31.430 716 01:10:03.209 00:14:31.430 717 echo "$time" 01:10:03.209 00:14:31.430 718 ) 01:10:03.209 00:14:31.430 719 01:10:03.209 00:14:31.430 720 function timing_enter() { 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 11 # Pci bus hotplug 01:10:03.209 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.209 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.209 00:14:31.430 14 local helper_time=0 01:10:03.209 00:14:31.430 15 01:10:03.209 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.209 00:14:31.430 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:03.209 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.209 00:14:31.430 19 } 01:10:03.209 00:14:31.430 20 01:10:03.209 00:14:31.430 21 remove_attach_helper() { 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.209 00:14:31.430 106 done 01:10:03.209 00:14:31.430 107 01:10:03.209 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.209 00:14:31.430 109 01:10:03.209 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.209 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.209 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.209 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.209 00:14:31.430 114 01:10:03.209 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 133 01:10:03.209 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.209 00:14:31.430 135 run_hotplug 01:10:03.209 00:14:31.430 136 01:10:03.209 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.209 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 01:10:03.209 00:14:31.430 ========== Backtrace end ========== 01:10:03.209 00:14:31.430 ========== Backtrace start: ========== 01:10:03.209 00:14:31.430 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:03.209 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.209 00:14:31.430 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.209 00:14:31.430 716 01:10:03.209 00:14:31.430 717 echo "$time" 01:10:03.209 00:14:31.430 => 718 ) 01:10:03.209 00:14:31.430 719 01:10:03.209 00:14:31.430 720 function timing_enter() { 01:10:03.209 00:14:31.430 721 xtrace_disable 01:10:03.209 00:14:31.430 722 timing "enter" "$1" 01:10:03.209 00:14:31.430 723 xtrace_restore 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 11 # Pci bus hotplug 01:10:03.209 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.209 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.209 00:14:31.430 14 local helper_time=0 01:10:03.209 00:14:31.430 15 01:10:03.209 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.209 00:14:31.430 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:03.209 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.209 00:14:31.430 19 } 01:10:03.209 00:14:31.430 20 01:10:03.209 00:14:31.430 21 remove_attach_helper() { 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.209 00:14:31.430 106 done 01:10:03.209 00:14:31.430 107 01:10:03.209 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.209 00:14:31.430 109 01:10:03.209 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.209 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.209 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.209 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.209 00:14:31.430 114 01:10:03.209 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.209 00:14:31.430 ... 01:10:03.209 00:14:31.430 133 01:10:03.209 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.210 00:14:31.430 135 run_hotplug 01:10:03.210 00:14:31.430 136 01:10:03.210 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.210 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 01:10:03.210 00:14:31.430 ========== Backtrace end ========== 01:10:03.210 00:14:31.430 ========== Backtrace start: ========== 01:10:03.210 00:14:31.430 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 11 # Pci bus hotplug 01:10:03.210 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.210 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.210 00:14:31.430 14 local helper_time=0 01:10:03.210 00:14:31.430 15 01:10:03.210 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.210 00:14:31.430 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:03.210 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.210 00:14:31.430 19 } 01:10:03.210 00:14:31.430 20 01:10:03.210 00:14:31.430 21 remove_attach_helper() { 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.210 00:14:31.430 106 done 01:10:03.210 00:14:31.430 107 01:10:03.210 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.210 00:14:31.430 109 01:10:03.210 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.210 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.210 00:14:31.431 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.210 00:14:31.431 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.210 00:14:31.431 114 01:10:03.210 00:14:31.431 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.210 00:14:31.431 ... 01:10:03.210 00:14:31.431 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.210 00:14:31.431 ... 01:10:03.210 00:14:31.431 133 01:10:03.210 00:14:31.431 134 # Run pci bus hotplug test 01:10:03.210 00:14:31.431 135 run_hotplug 01:10:03.210 00:14:31.431 136 01:10:03.210 00:14:31.431 137 # Run SPDK target based hotplug 01:10:03.210 00:14:31.431 => 138 tgt_run_hotplug 01:10:03.210 00:14:31.431 ... 01:10:03.210 00:14:31.431 01:10:03.210 00:14:31.431 ========== Backtrace end ========== 01:10:03.210 00:14:31.689 ========== Backtrace start: ========== 01:10:03.210 00:14:31.689 01:10:03.210 00:14:31.689 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh"]) 01:10:03.210 00:14:31.689 ... 01:10:03.210 00:14:31.689 1117 timing_enter $test_name 01:10:03.210 00:14:31.689 1118 echo "************************************" 01:10:03.210 00:14:31.689 1119 echo "START TEST $test_name" 01:10:03.210 00:14:31.689 1120 echo "************************************" 01:10:03.210 00:14:31.689 1121 xtrace_restore 01:10:03.210 00:14:31.689 1122 time "$@" 01:10:03.210 00:14:31.689 1123 xtrace_disable 01:10:03.210 00:14:31.689 1124 echo "************************************" 01:10:03.210 00:14:31.689 1125 echo "END TEST $test_name" 01:10:03.210 00:14:31.689 1126 echo "************************************" 01:10:03.210 00:14:31.689 1127 timing_exit $test_name 01:10:03.210 00:14:31.689 ... 01:10:03.210 00:14:31.690 in /var/jenkins/workspace/nvme-phy-autotest/spdk/autotest.sh:240 -> main(["/var/jenkins/workspace/nvme-phy-autotest/autorun-spdk.conf"]) 01:10:03.210 00:14:31.690 ... 01:10:03.210 00:14:31.690 235 01:10:03.210 00:14:31.690 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:03.210 00:14:31.690 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:03.210 00:14:31.690 238 01:10:03.210 00:14:31.690 239 if [ $(uname -s) = Linux ]; then 01:10:03.210 00:14:31.690 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:03.210 00:14:31.690 241 fi 01:10:03.210 00:14:31.690 242 01:10:03.210 00:14:31.690 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:03.210 00:14:31.690 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:03.210 00:14:31.690 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:03.210 00:14:31.690 ... 01:10:03.210 00:14:31.690 01:10:03.210 00:14:31.690 ========== Backtrace end ==========' 01:10:03.210 + [[ -n 00:14:31.430 05:39:20 sw_hotplug -- common/autotest_common.sh@715 -- # time='========== Backtrace start: ========== 01:10:03.210 00:14:31.430 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 710 01:10:03.210 00:14:31.430 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:03.210 00:14:31.430 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:03.210 00:14:31.430 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:03.210 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.210 00:14:31.430 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.210 00:14:31.430 716 01:10:03.210 00:14:31.430 717 echo "$time" 01:10:03.210 00:14:31.430 718 ) 01:10:03.210 00:14:31.430 719 01:10:03.210 00:14:31.430 720 function timing_enter() { 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 11 # Pci bus hotplug 01:10:03.210 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.210 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.210 00:14:31.430 14 local helper_time=0 01:10:03.210 00:14:31.430 15 01:10:03.210 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.210 00:14:31.430 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:03.210 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.210 00:14:31.430 19 } 01:10:03.210 00:14:31.430 20 01:10:03.210 00:14:31.430 21 remove_attach_helper() { 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.210 00:14:31.430 106 done 01:10:03.210 00:14:31.430 107 01:10:03.210 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.210 00:14:31.430 109 01:10:03.210 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.210 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.210 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.210 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.210 00:14:31.430 114 01:10:03.210 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 133 01:10:03.210 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.210 00:14:31.430 135 run_hotplug 01:10:03.210 00:14:31.430 136 01:10:03.210 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.210 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 01:10:03.210 00:14:31.430 ========== Backtrace end ==========' 01:10:03.210 00:14:31.430 ========== Backtrace start: ========== 01:10:03.210 00:14:31.430 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 710 01:10:03.210 00:14:31.430 711 # We redirect cmd's std{out,err} to a separate fd dup'ed to stdin's device (or /dev/null) to 01:10:03.210 00:14:31.430 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:03.210 00:14:31.430 713 # but $() will return just the time's data, hence making it possible to just do: 01:10:03.210 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.210 00:14:31.430 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.210 00:14:31.430 716 01:10:03.210 00:14:31.430 717 echo "$time" 01:10:03.210 00:14:31.430 718 ) 01:10:03.210 00:14:31.430 719 01:10:03.210 00:14:31.430 720 function timing_enter() { 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 11 # Pci bus hotplug 01:10:03.210 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.210 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.210 00:14:31.430 14 local helper_time=0 01:10:03.210 00:14:31.430 15 01:10:03.210 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.210 00:14:31.430 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:03.210 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.210 00:14:31.430 19 } 01:10:03.210 00:14:31.430 20 01:10:03.210 00:14:31.430 21 remove_attach_helper() { 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.210 00:14:31.430 106 done 01:10:03.210 00:14:31.430 107 01:10:03.210 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.210 00:14:31.430 109 01:10:03.210 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.210 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.210 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.210 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.210 00:14:31.430 114 01:10:03.210 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.210 00:14:31.430 ... 01:10:03.210 00:14:31.430 133 01:10:03.210 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.211 00:14:31.430 135 run_hotplug 01:10:03.211 00:14:31.430 136 01:10:03.211 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.211 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 01:10:03.211 00:14:31.430 ========== Backtrace end ========== 01:10:03.211 00:14:31.430 ========== Backtrace start: ========== 01:10:03.211 00:14:31.430 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 713 # but $() will return just the time's data, hence making it possible to just do: 01:10:03.211 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.211 00:14:31.430 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.211 00:14:31.430 716 01:10:03.211 00:14:31.430 717 echo "$time" 01:10:03.211 00:14:31.430 => 718 ) 01:10:03.211 00:14:31.430 719 01:10:03.211 00:14:31.430 720 function timing_enter() { 01:10:03.211 00:14:31.430 721 xtrace_disable 01:10:03.211 00:14:31.430 722 timing "enter" "$1" 01:10:03.211 00:14:31.430 723 xtrace_restore 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 11 # Pci bus hotplug 01:10:03.211 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.211 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.211 00:14:31.430 14 local helper_time=0 01:10:03.211 00:14:31.430 15 01:10:03.211 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.211 00:14:31.430 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:03.211 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.211 00:14:31.430 19 } 01:10:03.211 00:14:31.430 20 01:10:03.211 00:14:31.430 21 remove_attach_helper() { 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.211 00:14:31.430 106 done 01:10:03.211 00:14:31.430 107 01:10:03.211 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.211 00:14:31.430 109 01:10:03.211 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.211 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.211 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.211 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.211 00:14:31.430 114 01:10:03.211 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 133 01:10:03.211 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.211 00:14:31.430 135 run_hotplug 01:10:03.211 00:14:31.430 136 01:10:03.211 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.211 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 01:10:03.211 00:14:31.430 ========== Backtrace end ========== 01:10:03.211 00:14:31.430 ========== Backtrace start: ========== 01:10:03.211 00:14:31.430 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 11 # Pci bus hotplug 01:10:03.211 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.211 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.211 00:14:31.430 14 local helper_time=0 01:10:03.211 00:14:31.430 15 01:10:03.211 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.211 00:14:31.430 17 printf 'remove_attach_helper took %ss to complete (handling %u nvme drive(s))' \ 01:10:03.211 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.211 00:14:31.430 19 } 01:10:03.211 00:14:31.430 20 01:10:03.211 00:14:31.430 21 remove_attach_helper() { 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.211 00:14:31.430 106 done 01:10:03.211 00:14:31.430 107 01:10:03.211 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.211 00:14:31.430 109 01:10:03.211 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.211 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.211 00:14:31.431 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.211 00:14:31.431 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.211 00:14:31.431 114 01:10:03.211 00:14:31.431 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.211 00:14:31.431 ... 01:10:03.211 00:14:31.431 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.211 00:14:31.431 ... 01:10:03.211 00:14:31.431 133 01:10:03.211 00:14:31.431 134 # Run pci bus hotplug test 01:10:03.211 00:14:31.431 135 run_hotplug 01:10:03.211 00:14:31.431 136 01:10:03.211 00:14:31.431 137 # Run SPDK target based hotplug 01:10:03.211 00:14:31.431 => 138 tgt_run_hotplug 01:10:03.211 00:14:31.431 ... 01:10:03.211 00:14:31.431 01:10:03.211 00:14:31.431 ========== Backtrace end ========== 01:10:03.211 00:14:31.689 ========== Backtrace start: ========== 01:10:03.211 00:14:31.689 01:10:03.211 00:14:31.689 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh"]) 01:10:03.211 00:14:31.689 ... 01:10:03.211 00:14:31.689 1117 timing_enter $test_name 01:10:03.211 00:14:31.689 1118 echo "************************************" 01:10:03.211 00:14:31.689 1119 echo "START TEST $test_name" 01:10:03.211 00:14:31.689 1120 echo "************************************" 01:10:03.211 00:14:31.689 1121 xtrace_restore 01:10:03.211 00:14:31.689 1122 time "$@" 01:10:03.211 00:14:31.689 1123 xtrace_disable 01:10:03.211 00:14:31.689 1124 echo "************************************" 01:10:03.211 00:14:31.689 1125 echo "END TEST $test_name" 01:10:03.211 00:14:31.689 1126 echo "************************************" 01:10:03.211 00:14:31.689 1127 timing_exit $test_name 01:10:03.211 00:14:31.689 ... 01:10:03.211 00:14:31.690 in /var/jenkins/workspace/nvme-phy-autotest/spdk/autotest.sh:240 -> main(["/var/jenkins/workspace/nvme-phy-autotest/autorun-spdk.conf"]) 01:10:03.211 00:14:31.690 ... 01:10:03.211 00:14:31.690 235 01:10:03.211 00:14:31.690 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:03.211 00:14:31.690 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:03.211 00:14:31.690 238 01:10:03.211 00:14:31.690 239 if [ $(uname -s) = Linux ]; then 01:10:03.211 00:14:31.690 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:03.211 00:14:31.690 241 fi 01:10:03.211 00:14:31.690 242 01:10:03.211 00:14:31.690 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:03.211 00:14:31.690 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:03.211 00:14:31.690 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:03.211 00:14:31.690 ... 01:10:03.211 00:14:31.690 01:10:03.211 00:14:31.690 ========== Backtrace end ========== ]] 01:10:03.211 + echo '00:14:31.430 05:39:20 sw_hotplug -- common/autotest_common.sh@715 -- # time='\''========== Backtrace start: ========== 01:10:03.211 00:14:31.430 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 710 01:10:03.211 00:14:31.430 711 # We redirect cmd'\''\'\'''\''s std{out,err} to a separate fd dup'\''\'\'''\''ed to stdin'\''\'\'''\''s device (or /dev/null) to 01:10:03.211 00:14:31.430 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:03.211 00:14:31.430 713 # but $() will return just the time'\''\'\'''\''s data, hence making it possible to just do: 01:10:03.211 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.211 00:14:31.430 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.211 00:14:31.430 716 01:10:03.211 00:14:31.430 717 echo "$time" 01:10:03.211 00:14:31.430 718 ) 01:10:03.211 00:14:31.430 719 01:10:03.211 00:14:31.430 720 function timing_enter() { 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 11 # Pci bus hotplug 01:10:03.211 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.211 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.211 00:14:31.430 14 local helper_time=0 01:10:03.211 00:14:31.430 15 01:10:03.211 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.211 00:14:31.430 17 printf '\''\'\'''\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\''\'\'''\'' \ 01:10:03.211 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.211 00:14:31.430 19 } 01:10:03.211 00:14:31.430 20 01:10:03.211 00:14:31.430 21 remove_attach_helper() { 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.211 00:14:31.430 106 done 01:10:03.211 00:14:31.430 107 01:10:03.211 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.211 00:14:31.430 109 01:10:03.211 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.211 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.211 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.211 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.211 00:14:31.430 114 01:10:03.211 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.211 00:14:31.430 ... 01:10:03.211 00:14:31.430 133 01:10:03.212 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.212 00:14:31.430 135 run_hotplug 01:10:03.212 00:14:31.430 136 01:10:03.212 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.212 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 01:10:03.212 00:14:31.430 ========== Backtrace end =========='\'' 01:10:03.212 00:14:31.430 ========== Backtrace start: ========== 01:10:03.212 00:14:31.430 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:715 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 710 01:10:03.212 00:14:31.430 711 # We redirect cmd'\''s std{out,err} to a separate fd dup'\''ed to stdin'\''s device (or /dev/null) to 01:10:03.212 00:14:31.430 712 # catch only output from the time builtin - output from the actual cmd would be still visible, 01:10:03.212 00:14:31.430 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:03.212 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.212 00:14:31.430 => 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.212 00:14:31.430 716 01:10:03.212 00:14:31.430 717 echo "$time" 01:10:03.212 00:14:31.430 718 ) 01:10:03.212 00:14:31.430 719 01:10:03.212 00:14:31.430 720 function timing_enter() { 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 11 # Pci bus hotplug 01:10:03.212 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.212 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.212 00:14:31.430 14 local helper_time=0 01:10:03.212 00:14:31.430 15 01:10:03.212 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.212 00:14:31.430 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:03.212 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.212 00:14:31.430 19 } 01:10:03.212 00:14:31.430 20 01:10:03.212 00:14:31.430 21 remove_attach_helper() { 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.212 00:14:31.430 106 done 01:10:03.212 00:14:31.430 107 01:10:03.212 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.212 00:14:31.430 109 01:10:03.212 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.212 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.212 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.212 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.212 00:14:31.430 114 01:10:03.212 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 133 01:10:03.212 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.212 00:14:31.430 135 run_hotplug 01:10:03.212 00:14:31.430 136 01:10:03.212 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.212 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 01:10:03.212 00:14:31.430 ========== Backtrace end ========== 01:10:03.212 00:14:31.430 ========== Backtrace start: ========== 01:10:03.212 00:14:31.430 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:718 -> timing_cmd(["remove_attach_helper"],["3"],["6"],["true"]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 713 # but $() will return just the time'\''s data, hence making it possible to just do: 01:10:03.212 00:14:31.430 714 # time_of_super_verbose_cmd=$(timing_cmd super_verbose_cmd) 01:10:03.212 00:14:31.430 715 time=$({ time "$@" >&"$cmd_out" 2>&1; } 2>&1) 01:10:03.212 00:14:31.430 716 01:10:03.212 00:14:31.430 717 echo "$time" 01:10:03.212 00:14:31.430 => 718 ) 01:10:03.212 00:14:31.430 719 01:10:03.212 00:14:31.430 720 function timing_enter() { 01:10:03.212 00:14:31.430 721 xtrace_disable 01:10:03.212 00:14:31.430 722 timing "enter" "$1" 01:10:03.212 00:14:31.430 723 xtrace_restore 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 11 # Pci bus hotplug 01:10:03.212 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.212 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.212 00:14:31.430 14 local helper_time=0 01:10:03.212 00:14:31.430 15 01:10:03.212 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.212 00:14:31.430 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:03.212 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.212 00:14:31.430 19 } 01:10:03.212 00:14:31.430 20 01:10:03.212 00:14:31.430 21 remove_attach_helper() { 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.212 00:14:31.430 106 done 01:10:03.212 00:14:31.430 107 01:10:03.212 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.212 00:14:31.430 109 01:10:03.212 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.212 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.212 00:14:31.430 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.212 00:14:31.430 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.212 00:14:31.430 114 01:10:03.212 00:14:31.430 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 133 01:10:03.212 00:14:31.430 134 # Run pci bus hotplug test 01:10:03.212 00:14:31.430 135 run_hotplug 01:10:03.212 00:14:31.430 136 01:10:03.212 00:14:31.430 137 # Run SPDK target based hotplug 01:10:03.212 00:14:31.430 => 138 tgt_run_hotplug 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 01:10:03.212 00:14:31.430 ========== Backtrace end ========== 01:10:03.212 00:14:31.430 ========== Backtrace start: ========== 01:10:03.212 00:14:31.430 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:16 -> debug_remove_attach_helper(["3"],["6"],["true"]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 11 # Pci bus hotplug 01:10:03.212 00:14:31.430 12 # Helper function to remove/attach cotrollers 01:10:03.212 00:14:31.430 13 debug_remove_attach_helper() { 01:10:03.212 00:14:31.430 14 local helper_time=0 01:10:03.212 00:14:31.430 15 01:10:03.212 00:14:31.430 => 16 helper_time=$(timing_cmd remove_attach_helper "$@") 01:10:03.212 00:14:31.430 17 printf '\''remove_attach_helper took %ss to complete (handling %u nvme drive(s))'\'' \ 01:10:03.212 00:14:31.430 18 "$helper_time" "$nvme_count" >&2 01:10:03.212 00:14:31.430 19 } 01:10:03.212 00:14:31.430 20 01:10:03.212 00:14:31.430 21 remove_attach_helper() { 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:110 -> tgt_run_hotplug([]) 01:10:03.212 00:14:31.430 ... 01:10:03.212 00:14:31.430 105 waitforbdev "Nvme0${dev}n1" "$hotplug_wait" 01:10:03.212 00:14:31.430 106 done 01:10:03.212 00:14:31.430 107 01:10:03.212 00:14:31.430 108 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.212 00:14:31.430 109 01:10:03.212 00:14:31.430 => 110 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.212 00:14:31.430 111 # Verify reregistering hotplug poller 01:10:03.212 00:14:31.431 112 rpc_cmd bdev_nvme_set_hotplug -d 01:10:03.212 00:14:31.431 113 rpc_cmd bdev_nvme_set_hotplug -e 01:10:03.212 00:14:31.431 114 01:10:03.212 00:14:31.431 115 debug_remove_attach_helper "$hotplug_events" "$hotplug_wait" true 01:10:03.212 00:14:31.431 ... 01:10:03.212 00:14:31.431 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh:138 -> main([]) 01:10:03.212 00:14:31.431 ... 01:10:03.212 00:14:31.431 133 01:10:03.212 00:14:31.431 134 # Run pci bus hotplug test 01:10:03.212 00:14:31.431 135 run_hotplug 01:10:03.212 00:14:31.431 136 01:10:03.212 00:14:31.431 137 # Run SPDK target based hotplug 01:10:03.212 00:14:31.431 => 138 tgt_run_hotplug 01:10:03.212 00:14:31.431 ... 01:10:03.212 00:14:31.431 01:10:03.212 00:14:31.431 ========== Backtrace end ========== 01:10:03.212 00:14:31.689 ========== Backtrace start: ========== 01:10:03.212 00:14:31.689 01:10:03.212 00:14:31.689 in /var/jenkins/workspace/nvme-phy-autotest/spdk/test/common/autotest_common.sh:1122 -> run_test(["sw_hotplug"],["/var/jenkins/workspace/nvme-phy-autotest/spdk/test/nvme/sw_hotplug.sh"]) 01:10:03.212 00:14:31.689 ... 01:10:03.212 00:14:31.689 1117 timing_enter $test_name 01:10:03.212 00:14:31.689 1118 echo "************************************" 01:10:03.212 00:14:31.689 1119 echo "START TEST $test_name" 01:10:03.212 00:14:31.689 1120 echo "************************************" 01:10:03.212 00:14:31.689 1121 xtrace_restore 01:10:03.212 00:14:31.689 1122 time "$@" 01:10:03.212 00:14:31.689 1123 xtrace_disable 01:10:03.212 00:14:31.689 1124 echo "************************************" 01:10:03.212 00:14:31.689 1125 echo "END TEST $test_name" 01:10:03.212 00:14:31.689 1126 echo "************************************" 01:10:03.212 00:14:31.689 1127 timing_exit $test_name 01:10:03.212 00:14:31.689 ... 01:10:03.212 00:14:31.690 in /var/jenkins/workspace/nvme-phy-autotest/spdk/autotest.sh:240 -> main(["/var/jenkins/workspace/nvme-phy-autotest/autorun-spdk.conf"]) 01:10:03.212 00:14:31.690 ... 01:10:03.212 00:14:31.690 235 01:10:03.212 00:14:31.690 236 run_test "nvme_rpc" $rootdir/test/nvme/nvme_rpc.sh 01:10:03.212 00:14:31.690 237 run_test "nvme_rpc_timeouts" $rootdir/test/nvme/nvme_rpc_timeouts.sh 01:10:03.212 00:14:31.690 238 01:10:03.212 00:14:31.690 239 if [ $(uname -s) = Linux ]; then 01:10:03.212 00:14:31.690 => 240 run_test "sw_hotplug" $rootdir/test/nvme/sw_hotplug.sh 01:10:03.212 00:14:31.690 241 fi 01:10:03.212 00:14:31.690 242 01:10:03.212 00:14:31.690 243 if [[ $SPDK_TEST_XNVME -eq 1 ]]; then 01:10:03.212 00:14:31.690 244 run_test "nvme_xnvme" $rootdir/test/nvme/xnvme/xnvme.sh 01:10:03.213 00:14:31.690 245 run_test "blockdev_xnvme" $rootdir/test/bdev/blockdev.sh "xnvme" 01:10:03.213 00:14:31.690 ... 01:10:03.213 00:14:31.690 01:10:03.213 00:14:31.690 ========== Backtrace end ==========' 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843 01:10:03.213 + job_name=unittest-main-docker-autotest_19843 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406 01:10:03.213 + job_name=lvol-vg-autotest_77406 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877 01:10:03.213 + job_name=dsa-phy-autotest_4877 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617 01:10:03.213 + job_name=vmd-phy-autotest_2617 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845 01:10:03.213 + job_name=scan-build-docker-autotest_19845 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138 01:10:03.213 + job_name=check-so-deps-docker-autotest_22138 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580 01:10:03.213 + job_name=nvme-cmb-pmr-vg-autotest_78580 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114 01:10:03.213 + job_name=build-files-docker-autotest_22114 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846 01:10:03.213 + job_name=unittest-valgrind-docker-autotest_19846 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/build.log 01:10:03.213 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.213 + build_log=/var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/build.log 01:10:03.213 +++ dirname /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/build.log 01:10:03.213 ++ basename /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177 01:10:03.213 + job_name=crypto-phy-autotest_26177 01:10:03.213 + start_pattern='Backtrace start' 01:10:03.213 + end_pattern='Backtrace end' 01:10:03.213 + [[ -f /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/build.log ]] 01:10:03.213 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/build.log 01:10:03.213 + backtraces= 01:10:03.213 + [[ -n '' ]] 01:10:03.213 + for build in "${build_log_list[@]}" 01:10:03.213 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/build.log 01:10:03.214 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.214 + build_log=/var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/build.log 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/build.log 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308 01:10:03.214 + job_name=nvmf-tcp-phy-autotest_72308 01:10:03.214 + start_pattern='Backtrace start' 01:10:03.214 + end_pattern='Backtrace end' 01:10:03.214 + [[ -f /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/build.log ]] 01:10:03.214 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/build.log 01:10:03.214 + backtraces= 01:10:03.214 + [[ -n '' ]] 01:10:03.214 + for build in "${build_log_list[@]}" 01:10:03.214 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/build.log 01:10:03.214 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.214 + build_log=/var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/build.log 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/build.log 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185 01:10:03.214 + job_name=vfio-user-phy-autotest_26185 01:10:03.214 + start_pattern='Backtrace start' 01:10:03.214 + end_pattern='Backtrace end' 01:10:03.214 + [[ -f /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/build.log ]] 01:10:03.214 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/build.log 01:10:03.214 + backtraces= 01:10:03.214 + [[ -n '' ]] 01:10:03.214 + for build in "${build_log_list[@]}" 01:10:03.214 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/build.log 01:10:03.214 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.214 + build_log=/var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/build.log 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/build.log 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362 01:10:03.214 + job_name=iscsi-uring-vg-autotest_77362 01:10:03.214 + start_pattern='Backtrace start' 01:10:03.214 + end_pattern='Backtrace end' 01:10:03.214 + [[ -f /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/build.log ]] 01:10:03.214 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/build.log 01:10:03.214 + backtraces= 01:10:03.214 + [[ -n '' ]] 01:10:03.214 + for build in "${build_log_list[@]}" 01:10:03.214 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/build.log 01:10:03.214 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.214 + build_log=/var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/build.log 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/build.log 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477 01:10:03.214 + job_name=vhost-initiator-vg-autotest_77477 01:10:03.214 + start_pattern='Backtrace start' 01:10:03.214 + end_pattern='Backtrace end' 01:10:03.214 + [[ -f /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/build.log ]] 01:10:03.214 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/build.log 01:10:03.214 + backtraces= 01:10:03.214 + [[ -n '' ]] 01:10:03.214 + for build in "${build_log_list[@]}" 01:10:03.214 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/build.log 01:10:03.214 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.214 + build_log=/var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/build.log 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/build.log 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448 01:10:03.214 + job_name=nvmf-tcp-vg-autotest_77448 01:10:03.214 + start_pattern='Backtrace start' 01:10:03.214 + end_pattern='Backtrace end' 01:10:03.214 + [[ -f /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/build.log ]] 01:10:03.214 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/build.log 01:10:03.214 + backtraces= 01:10:03.214 + [[ -n '' ]] 01:10:03.214 + for build in "${build_log_list[@]}" 01:10:03.214 + create_errtrace_file /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/build.log 01:10:03.214 + local build_log job_name start_pattern end_pattern backtraces 01:10:03.214 + build_log=/var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/build.log 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/build.log 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105 01:10:03.214 + job_name=freebsd-vg-autotest_59105 01:10:03.214 + start_pattern='Backtrace start' 01:10:03.214 + end_pattern='Backtrace end' 01:10:03.214 + [[ -f /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/build.log ]] 01:10:03.214 ++ sed -n '/Backtrace start/,/Backtrace end/p' /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/build.log 01:10:03.214 + backtraces= 01:10:03.214 + [[ -n '' ]] 01:10:03.214 + run_flamegraph 01:10:03.214 + create_timing_file 01:10:03.214 + local array file 01:10:03.214 + array=($(find $WORKSPACE -type f -name 'timing.txt')) 01:10:03.214 ++ find /var/jenkins/workspace/autotest-nightly -type f -name timing.txt 01:10:03.214 + timing_dir=/var/jenkins/workspace/autotest-nightly/post_process 01:10:03.214 + mkdir -p /var/jenkins/workspace/autotest-nightly/post_process 01:10:03.214 + timing_file=/var/jenkins/workspace/autotest-nightly/post_process/total_timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805 01:10:03.214 + job_name=autobuild-external-vg-autotest_19805 01:10:03.214 + sed 's/^/autobuild-external-vg-autotest_19805;autorun;/' /var/jenkins/workspace/autotest-nightly/autobuild-external-vg-autotest_19805/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305 01:10:03.214 + job_name=doc-docker-autotest_22305 01:10:03.214 + sed 's/^/doc-docker-autotest_22305;autorun;/' /var/jenkins/workspace/autotest-nightly/doc-docker-autotest_22305/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790 01:10:03.214 + job_name=ubuntu22-vg-autotest_37790 01:10:03.214 + sed 's/^/ubuntu22-vg-autotest_37790;autorun;/' /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636 01:10:03.214 + job_name=centos7-vg-autotest_73636 01:10:03.214 + sed 's/^/centos7-vg-autotest_73636;autorun;/' /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067 01:10:03.214 + job_name=rocky9-vg-autotest_16067 01:10:03.214 + sed 's/^/rocky9-vg-autotest_16067;autorun;/' /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844 01:10:03.214 + job_name=packaging-docker-autotest_19844 01:10:03.214 + sed 's/^/packaging-docker-autotest_19844;autorun;/' /var/jenkins/workspace/autotest-nightly/packaging-docker-autotest_19844/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457 01:10:03.214 + job_name=nvme-vg-autotest_77457 01:10:03.214 + sed 's/^/nvme-vg-autotest_77457;autorun;/' /var/jenkins/workspace/autotest-nightly/nvme-vg-autotest_77457/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015 01:10:03.214 + job_name=nvmf-cvl-phy-autotest_3015 01:10:03.214 + sed 's/^/nvmf-cvl-phy-autotest_3015;autorun;/' /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/timing.txt 01:10:03.214 + local job_file job_name 01:10:03.214 + job_file=/var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/timing.txt 01:10:03.214 +++ dirname /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/timing.txt 01:10:03.214 ++ basename /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776 01:10:03.214 + job_name=ubuntu20-vg-autotest_73776 01:10:03.214 + sed 's/^/ubuntu20-vg-autotest_73776;autorun;/' /var/jenkins/workspace/autotest-nightly/ubuntu20-vg-autotest_73776/timing.txt 01:10:03.214 + for file in "${array[@]}" 01:10:03.214 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183 01:10:03.215 + job_name=BlobFS-phy-autotest_26183 01:10:03.215 + sed 's/^/BlobFS-phy-autotest_26183;autorun;/' /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563 01:10:03.215 + job_name=zns-vg-autotest_76563 01:10:03.215 + sed 's/^/zns-vg-autotest_76563;autorun;/' /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202 01:10:03.215 + job_name=vhost-phy-autotest_26202 01:10:03.215 + sed 's/^/vhost-phy-autotest_26202;autorun;/' /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382 01:10:03.215 + job_name=nvmf-tcp-uring-vg-autotest_77382 01:10:03.215 + sed 's/^/nvmf-tcp-uring-vg-autotest_77382;autorun;/' /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174 01:10:03.215 + job_name=short-fuzz-phy-autotest_26174 01:10:03.215 + sed 's/^/short-fuzz-phy-autotest_26174;autorun;/' /var/jenkins/workspace/autotest-nightly/short-fuzz-phy-autotest_26174/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125 01:10:03.215 + job_name=check-format-docker-autotest_22125 01:10:03.215 + sed 's/^/check-format-docker-autotest_22125;autorun;/' /var/jenkins/workspace/autotest-nightly/check-format-docker-autotest_22125/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495 01:10:03.215 + job_name=iscsi-vg-autotest_77495 01:10:03.215 + sed 's/^/iscsi-vg-autotest_77495;autorun;/' /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842 01:10:03.215 + job_name=unittest-clang-docker-autotest_19842 01:10:03.215 + sed 's/^/unittest-clang-docker-autotest_19842;autorun;/' /var/jenkins/workspace/autotest-nightly/unittest-clang-docker-autotest_19842/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421 01:10:03.215 + job_name=nvmf-phy-autotest_72421 01:10:03.215 + sed 's/^/nvmf-phy-autotest_72421;autorun;/' /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349 01:10:03.215 + job_name=nvme-phy-autotest_72349 01:10:03.215 + sed 's/^/nvme-phy-autotest_72349;autorun;/' /var/jenkins/workspace/autotest-nightly/nvme-phy-autotest_72349/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/timing.txt 01:10:03.215 +++ dirname /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/timing.txt 01:10:03.215 ++ basename /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843 01:10:03.215 + job_name=unittest-main-docker-autotest_19843 01:10:03.215 + sed 's/^/unittest-main-docker-autotest_19843;autorun;/' /var/jenkins/workspace/autotest-nightly/unittest-main-docker-autotest_19843/timing.txt 01:10:03.215 + for file in "${array[@]}" 01:10:03.215 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/timing.txt 01:10:03.215 + local job_file job_name 01:10:03.215 + job_file=/var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/timing.txt 01:10:03.473 +++ dirname /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/timing.txt 01:10:03.473 ++ basename /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406 01:10:03.473 + job_name=lvol-vg-autotest_77406 01:10:03.473 + sed 's/^/lvol-vg-autotest_77406;autorun;/' /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/timing.txt 01:10:03.473 + for file in "${array[@]}" 01:10:03.473 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/timing.txt 01:10:03.473 + local job_file job_name 01:10:03.473 + job_file=/var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/timing.txt 01:10:03.473 +++ dirname /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/timing.txt 01:10:03.473 ++ basename /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877 01:10:03.473 + job_name=dsa-phy-autotest_4877 01:10:03.473 + sed 's/^/dsa-phy-autotest_4877;autorun;/' /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/timing.txt 01:10:03.473 + for file in "${array[@]}" 01:10:03.473 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/timing.txt 01:10:03.473 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617 01:10:03.474 + job_name=vmd-phy-autotest_2617 01:10:03.474 + sed 's/^/vmd-phy-autotest_2617;autorun;/' /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845 01:10:03.474 + job_name=scan-build-docker-autotest_19845 01:10:03.474 + sed 's/^/scan-build-docker-autotest_19845;autorun;/' /var/jenkins/workspace/autotest-nightly/scan-build-docker-autotest_19845/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138 01:10:03.474 + job_name=check-so-deps-docker-autotest_22138 01:10:03.474 + sed 's/^/check-so-deps-docker-autotest_22138;autorun;/' /var/jenkins/workspace/autotest-nightly/check-so-deps-docker-autotest_22138/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580 01:10:03.474 + job_name=nvme-cmb-pmr-vg-autotest_78580 01:10:03.474 + sed 's/^/nvme-cmb-pmr-vg-autotest_78580;autorun;/' /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114 01:10:03.474 + job_name=build-files-docker-autotest_22114 01:10:03.474 + sed 's/^/build-files-docker-autotest_22114;autorun;/' /var/jenkins/workspace/autotest-nightly/build-files-docker-autotest_22114/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846 01:10:03.474 + job_name=unittest-valgrind-docker-autotest_19846 01:10:03.474 + sed 's/^/unittest-valgrind-docker-autotest_19846;autorun;/' /var/jenkins/workspace/autotest-nightly/unittest-valgrind-docker-autotest_19846/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177 01:10:03.474 + job_name=crypto-phy-autotest_26177 01:10:03.474 + sed 's/^/crypto-phy-autotest_26177;autorun;/' /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308 01:10:03.474 + job_name=nvmf-tcp-phy-autotest_72308 01:10:03.474 + sed 's/^/nvmf-tcp-phy-autotest_72308;autorun;/' /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185 01:10:03.474 + job_name=vfio-user-phy-autotest_26185 01:10:03.474 + sed 's/^/vfio-user-phy-autotest_26185;autorun;/' /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362 01:10:03.474 + job_name=iscsi-uring-vg-autotest_77362 01:10:03.474 + sed 's/^/iscsi-uring-vg-autotest_77362;autorun;/' /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477 01:10:03.474 + job_name=vhost-initiator-vg-autotest_77477 01:10:03.474 + sed 's/^/vhost-initiator-vg-autotest_77477;autorun;/' /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448 01:10:03.474 + job_name=nvmf-tcp-vg-autotest_77448 01:10:03.474 + sed 's/^/nvmf-tcp-vg-autotest_77448;autorun;/' /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/timing.txt 01:10:03.474 + for file in "${array[@]}" 01:10:03.474 + add_sub_job_timing /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/timing.txt 01:10:03.474 + local job_file job_name 01:10:03.474 + job_file=/var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/timing.txt 01:10:03.474 +++ dirname /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/timing.txt 01:10:03.474 ++ basename /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105 01:10:03.474 + job_name=freebsd-vg-autotest_59105 01:10:03.474 + sed 's/^/freebsd-vg-autotest_59105;autorun;/' /var/jenkins/workspace/autotest-nightly/freebsd-vg-autotest_59105/timing.txt 01:10:03.474 + /usr/local/FlameGraph/flamegraph.pl --title 'Build Timing' --nametype Step: --countname seconds /var/jenkins/workspace/autotest-nightly/post_process/total_timing.txt 01:10:03.474 + run_autorun_post 01:10:03.474 + s_opt= 01:10:03.474 + [[ '' == true ]] 01:10:03.474 + spdk/autorun_post.py -d ./ -r ./spdk 01:10:07.654 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/lib/util/crc32c.c:21 01:10:07.654 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/lib/util/crc16.c:628 01:10:07.654 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/lib/util/crc16.c:634 01:10:07.654 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/lib/util/xor.c:120 01:10:07.911 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/lib/util/crc64.c:165 01:10:16.015 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/module/blobfs/bdev/blobfs_bdev.c:254 01:11:12.225 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/module/bdev/nvme/bdev_mdns_client.c:613 01:11:12.225 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/module/bdev/nvme/bdev_mdns_client.c:561 01:11:12.225 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/module/bdev/nvme/bdev_mdns_client.c:582 01:11:12.225 lcov: WARNING: function data mismatch at /var/jenkins/workspace/autotest-nightly/spdk/module/bdev/nvme/bdev_mdns_client.c:456 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/nvme.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/trace.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/base64.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/thread.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/bdev_module.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/histogram_data.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/nvmf_transport.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/nvme_spec.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/endian.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/mmio.h! 01:11:22.188 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/util.h! 01:11:23.121 genhtml: WARNING: cannot read /var/jenkins/workspace/autotest-nightly/spdk/histogram_data.h/histogram_ut.c! 01:11:41.190 -----Begin Post Process Script------ 01:11:41.190 /var/jenkins/workspace/autotest-nightly/ubuntu22-vg-autotest_37790/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/centos7-vg-autotest_73636/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/rocky9-vg-autotest_16067/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/nvmf-cvl-phy-autotest_3015/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/BlobFS-phy-autotest_26183/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/zns-vg-autotest_76563/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/vhost-phy-autotest_26202/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/nvmf-tcp-uring-vg-autotest_77382/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/iscsi-vg-autotest_77495/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/nvmf-phy-autotest_72421/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/lvol-vg-autotest_77406/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/dsa-phy-autotest_4877/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/vmd-phy-autotest_2617/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/nvme-cmb-pmr-vg-autotest_78580/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/crypto-phy-autotest_26177/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/nvmf-tcp-phy-autotest_72308/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/vfio-user-phy-autotest_26185/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/iscsi-uring-vg-autotest_77362/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/vhost-initiator-vg-autotest_77477/cov_total.info 01:11:41.190 /var/jenkins/workspace/autotest-nightly/nvmf-tcp-vg-autotest_77448/cov_total.info 01:11:41.190 01:11:41.190 01:11:41.190 -----Tests Executed in Build------ 01:11:41.190 accel 01:11:41.190 accel_assign_opcode 01:11:41.190 accel_cdev_comp 01:11:41.190 accel_cdev_decmop_full 01:11:41.190 accel_cdev_decomp 01:11:41.190 accel_cdev_decomp_full_mcore 01:11:41.190 accel_cdev_decomp_full_mthread 01:11:41.190 accel_cdev_decomp_mcore 01:11:41.190 accel_cdev_decomp_mthread 01:11:41.190 accel_comp 01:11:41.190 accel_compare 01:11:41.190 accel_compress_verify 01:11:41.190 accel_copy 01:11:41.190 accel_copy_crc32c 01:11:41.190 accel_copy_crc32c_C2 01:11:41.190 accel_crc32c 01:11:41.190 accel_crc32c_C2 01:11:41.190 accel_decmop_full 01:11:41.190 accel_decomp 01:11:41.190 accel_decomp_full_mcore 01:11:41.190 accel_decomp_full_mthread 01:11:41.190 accel_decomp_mcore 01:11:41.190 accel_decomp_mthread 01:11:41.190 accel_dif_functional_tests 01:11:41.190 accel_dif_generate 01:11:41.190 accel_dif_generate_copy 01:11:41.190 accel_dif_verify 01:11:41.190 accel_dualcast 01:11:41.190 accel_fill 01:11:41.190 accel_help 01:11:41.190 accel_missing_filename 01:11:41.190 accel_negative_buffers 01:11:41.190 accel_rpc 01:11:41.190 accel_scan_dsa_modules 01:11:41.190 accel_scan_iaa_modules 01:11:41.190 accel_wrong_workload 01:11:41.190 accel_xor 01:11:41.190 acl 01:11:41.190 alias_rpc 01:11:41.190 allowed 01:11:41.190 app_cmdline 01:11:41.190 app_repeat 01:11:41.190 asan 01:11:41.190 autobuild 01:11:41.190 autobuild_build_doc 01:11:41.190 autobuild_check_dpdk_pci_api 01:11:41.190 autobuild_check_format 01:11:41.190 autobuild_check_so_deps 01:11:41.190 autobuild_external_code 01:11:41.190 autobuild_generated_files_check 01:11:41.190 autobuild_generated_files_check_post_clean 01:11:41.190 autobuild_header_dependency_check 01:11:41.190 autobuild_llvm_precompile 01:11:41.190 autobuild_make_install 01:11:41.190 autobuild_make_uninstall 01:11:41.190 autobuild_ocf_precompile 01:11:41.190 bdev_bounds 01:11:41.190 bdev_crypto_enomem 01:11:41.190 bdev_error 01:11:41.190 bdev_fio 01:11:41.190 bdev_fio_rw_verify 01:11:41.190 bdev_fio_trim 01:11:41.190 bdev_gpt_uuid 01:11:41.190 bdev_hello_world 01:11:41.190 bdev_json_nonarray 01:11:41.190 bdev_json_nonenclosed 01:11:41.190 bdev_nbd 01:11:41.190 bdev_nvme_reset_stuck_adm_cmd 01:11:41.190 bdev_qd_sampling 01:11:41.190 bdev_qos 01:11:41.190 bdev_qos_bw 01:11:41.190 bdev_qos_iops 01:11:41.190 bdev_qos_ro_bw 01:11:41.190 bdev_raid 01:11:41.190 bdev_stat 01:11:41.190 bdev_verify 01:11:41.190 bdev_verify_big_io 01:11:41.190 bdev_write_zeroes 01:11:41.190 bdevperf_config 01:11:41.190 blob_io_wait 01:11:41.190 blobfs 01:11:41.190 blobfs_mkfs 01:11:41.190 blobstore 01:11:41.190 blobstore_grow 01:11:41.190 blockdev_crypto_aesni 01:11:41.190 blockdev_crypto_qat 01:11:41.190 blockdev_crypto_sw 01:11:41.190 blockdev_general 01:11:41.190 blockdev_nvme 01:11:41.190 blockdev_nvme_gpt 01:11:41.190 blockdev_raid5f 01:11:41.190 blockdev_rbd 01:11:41.190 build_rpm_from_gen_spec 01:11:41.190 build_shared_rpm 01:11:41.190 build_shared_rpm_with_rpmed_dpdk 01:11:41.190 chaining 01:11:41.190 check_header_filenames 01:11:41.190 cmb_copy 01:11:41.190 compress_compdev 01:11:41.190 compress_isal 01:11:41.190 confirm_abi_deps 01:11:41.190 confirm_makefile_deps 01:11:41.190 cpu_locks 01:11:41.190 custom_alloc 01:11:41.190 dd_bs_lt_native_bs 01:11:41.190 dd_copy_to_out_bdev 01:11:41.190 dd_double_input 01:11:41.190 dd_double_output 01:11:41.190 dd_flag_append 01:11:41.190 dd_flag_append_forced_aio 01:11:41.190 dd_flag_directory 01:11:41.190 dd_flag_directory_forced_aio 01:11:41.190 dd_flag_noatime 01:11:41.190 dd_flag_noatime_forced_aio 01:11:41.190 dd_flag_nofollow 01:11:41.190 dd_flag_nofollow_forced_aio 01:11:41.190 dd_flags_misc 01:11:41.190 dd_flags_misc_forced_aio 01:11:41.190 dd_inflate_file 01:11:41.190 dd_invalid_arguments 01:11:41.190 dd_invalid_count 01:11:41.191 dd_invalid_iflag 01:11:41.191 dd_invalid_json 01:11:41.191 dd_invalid_oflag 01:11:41.191 dd_malloc_copy 01:11:41.191 dd_no_input 01:11:41.191 dd_no_output 01:11:41.191 dd_offset_magic 01:11:41.191 dd_rw 01:11:41.191 dd_rw_offset 01:11:41.191 dd_smaller_blocksize 01:11:41.191 dd_sparse_bdev_to_file 01:11:41.191 dd_sparse_file_to_bdev 01:11:41.191 dd_sparse_file_to_file 01:11:41.191 dd_unknown_flag 01:11:41.191 dd_uring_copy 01:11:41.191 dd_wrong_blocksize 01:11:41.191 default_locks 01:11:41.191 default_locks_via_rpc 01:11:41.191 default_setup 01:11:41.191 denied 01:11:41.191 devices 01:11:41.191 dm_mount 01:11:41.191 dma 01:11:41.191 dpdk_mem_utility 01:11:41.191 driver 01:11:41.191 env 01:11:41.191 env_dpdk_post_init 01:11:41.191 env_mem_callbacks 01:11:41.191 env_memory 01:11:41.191 env_pci 01:11:41.191 env_vtophys 01:11:41.191 even_2G_alloc 01:11:41.191 event 01:11:41.191 event_perf 01:11:41.191 event_reactor 01:11:41.191 event_reactor_perf 01:11:41.191 event_scheduler 01:11:41.191 exit_on_failed_rpc_init 01:11:41.191 external_make_accel_driver_shared 01:11:41.191 external_make_accel_driver_static 01:11:41.191 external_make_accel_module_shared 01:11:41.191 external_make_accel_module_static 01:11:41.191 external_make_hello_bdev_shared_combo 01:11:41.191 external_make_hello_bdev_shared_iso 01:11:41.191 external_make_hello_bdev_static 01:11:41.191 external_make_hello_no_bdev_shared_combo 01:11:41.191 external_make_hello_no_bdev_shared_iso 01:11:41.191 external_make_hello_no_bdev_static 01:11:41.191 external_make_nvme_shared 01:11:41.191 external_make_nvme_static 01:11:41.191 external_run_accel_driver_shared 01:11:41.191 external_run_accel_driver_static 01:11:41.191 external_run_accel_module_shared 01:11:41.191 external_run_accel_module_static 01:11:41.191 external_run_hello_bdev_shared_combo 01:11:41.191 external_run_hello_bdev_shared_iso 01:11:41.191 external_run_hello_bdev_static 01:11:41.191 external_run_hello_no_bdev_shared_combo 01:11:41.191 external_run_hello_no_bdev_shared_iso 01:11:41.191 external_run_hello_no_bdev_static 01:11:41.191 external_run_nvme_shared 01:11:41.191 external_run_nvme_static 01:11:41.191 filesystem_btrfs 01:11:41.191 filesystem_ext4 01:11:41.191 filesystem_in_capsule_btrfs 01:11:41.191 filesystem_in_capsule_ext4 01:11:41.191 filesystem_in_capsule_xfs 01:11:41.191 filesystem_xfs 01:11:41.191 fio_dif_1_default 01:11:41.191 fio_dif_1_multi_subsystems 01:11:41.191 fio_dif_digest 01:11:41.191 fio_dif_rand_params 01:11:41.191 go_rpc 01:11:41.191 guess_driver 01:11:41.191 hello_blob 01:11:41.191 hugepages 01:11:41.191 is_zoned 01:11:41.191 iscsi_tgt 01:11:41.191 iscsi_tgt_bdev_io_wait 01:11:41.191 iscsi_tgt_calsoft 01:11:41.191 iscsi_tgt_digest 01:11:41.191 iscsi_tgt_digests 01:11:41.191 iscsi_tgt_ext4test 01:11:41.191 iscsi_tgt_filesystem 01:11:41.191 iscsi_tgt_filesystem_btrfs 01:11:41.191 iscsi_tgt_filesystem_ext4 01:11:41.191 iscsi_tgt_filesystem_xfs 01:11:41.191 iscsi_tgt_fio 01:11:41.191 iscsi_tgt_fuzz 01:11:41.191 iscsi_tgt_initiator 01:11:41.191 iscsi_tgt_ip_migration 01:11:41.191 iscsi_tgt_iscsi_lvol 01:11:41.191 iscsi_tgt_login_redirection 01:11:41.191 iscsi_tgt_multiconnection 01:11:41.191 iscsi_tgt_qos 01:11:41.191 iscsi_tgt_rbd 01:11:41.191 iscsi_tgt_reset 01:11:41.191 iscsi_tgt_resize 01:11:41.191 iscsi_tgt_rpc_config 01:11:41.191 iscsi_tgt_sock 01:11:41.191 iscsi_tgt_trace_record 01:11:41.191 json_config 01:11:41.191 json_config_extra_key 01:11:41.191 kernel_target_abort 01:11:41.191 keyring_file 01:11:41.191 llvm_fuzz 01:11:41.191 locking_app_on_locked_coremask 01:11:41.191 locking_app_on_unlocked_coremask 01:11:41.191 locking_overlapped_coremask 01:11:41.191 locking_overlapped_coremask_via_rpc 01:11:41.191 lvol 01:11:41.191 lvol_basic 01:11:41.191 lvol_esnap 01:11:41.191 lvol_external_copy 01:11:41.191 lvol_external_snapshot 01:11:41.191 lvol_hotremove 01:11:41.191 lvol_provisioning 01:11:41.191 lvol_rename 01:11:41.191 lvol_resize 01:11:41.191 lvol_snapshot_clone 01:11:41.191 lvol_tasting 01:11:41.191 lvs_grow_clean 01:11:41.191 lvs_grow_dirty 01:11:41.191 make 01:11:41.191 no_shrink_alloc 01:11:41.191 non_locking_app_on_locked_coremask 01:11:41.191 nvme 01:11:41.191 nvme_arbitration 01:11:41.191 nvme_cli_cuse 01:11:41.191 nvme_cli_plugin 01:11:41.191 nvme_cmb 01:11:41.191 nvme_cuse 01:11:41.191 nvme_cuse_app 01:11:41.191 nvme_cuse_rpc 01:11:41.191 nvme_doorbell_aers 01:11:41.191 nvme_e2edp 01:11:41.191 nvme_err_injection 01:11:41.191 nvme_fdp 01:11:41.191 nvme_fio 01:11:41.191 nvme_flexible_data_placement 01:11:41.191 nvme_hello_world 01:11:41.191 nvme_identify 01:11:41.191 nvme_mount 01:11:41.191 nvme_multi_aen 01:11:41.191 nvme_multi_secondary 01:11:41.191 nvme_ns_manage_cuse 01:11:41.191 nvme_overhead 01:11:41.191 nvme_perf 01:11:41.191 nvme_pmr 01:11:41.191 nvme_pmr_persistence 01:11:41.191 nvme_reserve 01:11:41.191 nvme_reset 01:11:41.191 nvme_rpc 01:11:41.191 nvme_rpc_timeouts 01:11:41.191 nvme_scc 01:11:41.191 nvme_sgl 01:11:41.191 nvme_simple_copy 01:11:41.191 nvme_single_aen 01:11:41.191 nvme_smartctl_cuse 01:11:41.191 nvme_startup 01:11:41.191 nvme_zns 01:11:41.191 nvmf_abort 01:11:41.191 nvmf_abort_qd_sizes 01:11:41.191 nvmf_aer 01:11:41.191 nvmf_async_init 01:11:41.191 nvmf_auth_host 01:11:41.191 nvmf_auth_target 01:11:41.191 nvmf_bdev_io_wait 01:11:41.191 nvmf_bdevio 01:11:41.191 nvmf_bdevio_no_huge 01:11:41.191 nvmf_bdevperf 01:11:41.191 nvmf_connect_disconnect 01:11:41.191 nvmf_connect_stress 01:11:41.191 nvmf_delete_subsystem 01:11:41.191 nvmf_device_removal 01:11:41.191 nvmf_device_removal_pci_remove 01:11:41.191 nvmf_device_removal_pci_remove_no_srq 01:11:41.191 nvmf_dif 01:11:41.191 nvmf_digest 01:11:41.191 nvmf_digest_clean 01:11:41.191 nvmf_digest_dsa_initiator 01:11:41.191 nvmf_digest_dsa_target 01:11:41.191 nvmf_digest_error 01:11:41.191 nvmf_discovery_remove_ifc 01:11:41.191 nvmf_example 01:11:41.191 nvmf_failover 01:11:41.191 nvmf_filesystem 01:11:41.191 nvmf_filesystem_in_capsule 01:11:41.191 nvmf_filesystem_no_in_capsule 01:11:41.191 nvmf_fio_host 01:11:41.191 nvmf_fio_target 01:11:41.191 nvmf_fips 01:11:41.191 nvmf_fused_ordering 01:11:41.191 nvmf_fuzz 01:11:41.191 nvmf_host_discovery 01:11:41.191 nvmf_host_management 01:11:41.191 nvmf_host_multipath 01:11:41.191 nvmf_host_multipath_status 01:11:41.191 nvmf_identify 01:11:41.191 nvmf_identify_kernel_target 01:11:41.191 nvmf_identify_passthru 01:11:41.191 nvmf_initiator_timeout 01:11:41.191 nvmf_invalid 01:11:41.191 nvmf_lvol 01:11:41.191 nvmf_lvs_grow 01:11:41.191 nvmf_mdns_discovery 01:11:41.191 nvmf_multiconnection 01:11:41.191 nvmf_multicontroller 01:11:41.191 nvmf_multitarget 01:11:41.191 nvmf_nmic 01:11:41.191 nvmf_ns_hotplug_stress 01:11:41.191 nvmf_ns_masking 01:11:41.191 nvmf_nvme_cli 01:11:41.191 nvmf_perf 01:11:41.191 nvmf_perf_adq 01:11:41.191 nvmf_queue_depth 01:11:41.191 nvmf_rdma 01:11:41.191 nvmf_referrals 01:11:41.191 nvmf_rpc 01:11:41.191 nvmf_shutdown 01:11:41.191 nvmf_shutdown_tc1 01:11:41.191 nvmf_shutdown_tc2 01:11:41.191 nvmf_shutdown_tc3 01:11:41.191 nvmf_srq_overwhelm 01:11:41.191 nvmf_target_disconnect 01:11:41.191 nvmf_target_disconnect_tc1 01:11:41.191 nvmf_target_disconnect_tc2 01:11:41.191 nvmf_target_disconnect_tc3 01:11:41.191 nvmf_target_discovery 01:11:41.191 nvmf_target_multipath 01:11:41.191 nvmf_tcp 01:11:41.191 nvmf_timeout 01:11:41.191 nvmf_tls 01:11:41.191 nvmf_vfio_user 01:11:41.191 nvmf_vfio_user_fuzz 01:11:41.191 nvmf_vfio_user_nvme_compliance 01:11:41.191 nvmf_vhost 01:11:41.191 nvmf_zcopy 01:11:41.191 odd_alloc 01:11:41.191 packaging 01:11:41.191 per_node_1G_alloc 01:11:41.191 raid0_resize_test 01:11:41.191 raid5f_rebuild_test 01:11:41.191 raid5f_rebuild_test_sb 01:11:41.191 raid5f_state_function_test 01:11:41.191 raid5f_state_function_test_sb 01:11:41.191 raid5f_superblock_test 01:11:41.191 raid_function_test_concat 01:11:41.191 raid_function_test_raid0 01:11:41.191 raid_rebuild_test 01:11:41.191 raid_rebuild_test_io 01:11:41.191 raid_rebuild_test_sb 01:11:41.191 raid_rebuild_test_sb_4k 01:11:41.191 raid_rebuild_test_sb_io 01:11:41.191 raid_rebuild_test_sb_md_interleaved 01:11:41.191 raid_rebuild_test_sb_md_separate 01:11:41.191 raid_state_function_test 01:11:41.191 raid_state_function_test_sb 01:11:41.191 raid_state_function_test_sb_4k 01:11:41.191 raid_state_function_test_sb_md_interleaved 01:11:41.191 raid_state_function_test_sb_md_separate 01:11:41.191 raid_superblock_test 01:11:41.191 raid_superblock_test_4k 01:11:41.191 raid_superblock_test_md_interleaved 01:11:41.191 raid_superblock_test_md_separate 01:11:41.191 reactor_set_interrupt 01:11:41.191 reap_unregistered_poller 01:11:41.191 rocksdb 01:11:41.191 rocksdb_insert 01:11:41.191 rocksdb_overwrite 01:11:41.191 rocksdb_randread 01:11:41.191 rocksdb_readwrite 01:11:41.191 rocksdb_writesync 01:11:41.191 rpc 01:11:41.191 rpc_client 01:11:41.191 rpc_daemon_integrity 01:11:41.191 rpc_integrity 01:11:41.191 rpc_plugins 01:11:41.191 rpc_trace_cmd_test 01:11:41.191 rpm_packaging 01:11:41.191 scanbuild_make 01:11:41.191 scheduler_create_thread 01:11:41.191 setup.sh 01:11:41.192 skip_rpc 01:11:41.192 skip_rpc_with_delay 01:11:41.192 skip_rpc_with_json 01:11:41.192 sma 01:11:41.192 sma_crypto 01:11:41.192 sma_discovery 01:11:41.192 sma_nvmf_tcp 01:11:41.192 sma_plugins 01:11:41.192 sma_qos 01:11:41.192 sma_vfiouser_qemu 01:11:41.192 sma_vhost 01:11:41.192 spdk_dd 01:11:41.192 spdk_dd_basic_rw 01:11:41.192 spdk_dd_bdev_to_bdev 01:11:41.192 spdk_dd_malloc 01:11:41.192 spdk_dd_negative 01:11:41.192 spdk_dd_posix 01:11:41.192 spdk_dd_sparse 01:11:41.192 spdk_dd_uring 01:11:41.192 spdk_target_abort 01:11:41.192 spdkcli_iscsi 01:11:41.192 spdkcli_nvmf_rdma 01:11:41.192 spdkcli_nvmf_tcp 01:11:41.192 spdkcli_raid 01:11:41.192 spdkcli_rbd 01:11:41.192 spdkcli_tcp 01:11:41.192 spdkcli_vhost 01:11:41.192 spdkcli_virtio 01:11:41.192 sw_hotplug 01:11:41.192 test_bdev_lvol_delete_lvstore_with_clones 01:11:41.192 test_bdev_lvol_delete_ordering 01:11:41.192 test_clone_decouple_parent 01:11:41.192 test_clone_inflate 01:11:41.192 test_clone_snapshot_relations 01:11:41.192 test_construct_lvol 01:11:41.192 test_construct_lvol_alias_conflict 01:11:41.192 test_construct_lvol_fio_clear_method_none 01:11:41.192 test_construct_lvol_fio_clear_method_unmap 01:11:41.192 test_construct_lvol_full_lvs 01:11:41.192 test_construct_lvol_inexistent_lvs 01:11:41.192 test_construct_lvols_conflict_alias 01:11:41.192 test_construct_lvs 01:11:41.192 test_construct_lvs_clear_methods 01:11:41.192 test_construct_lvs_conflict_alias 01:11:41.192 test_construct_lvs_different_cluster_size 01:11:41.192 test_construct_lvs_nonexistent_bdev 01:11:41.192 test_construct_multi_lvols 01:11:41.192 test_construct_nested_lvol 01:11:41.192 test_construct_two_lvs_on_the_same_bdev 01:11:41.192 test_create_snapshot_of_snapshot 01:11:41.192 test_create_snapshot_with_io 01:11:41.192 test_delete_lvol_store_persistent_positive 01:11:41.192 test_delete_snapshot_with_clone 01:11:41.192 test_delete_snapshot_with_snapshot 01:11:41.192 test_destroy_after_bdev_lvol_resize_positive 01:11:41.192 test_esnap_clones 01:11:41.192 test_esnap_late_arrival 01:11:41.192 test_esnap_reload 01:11:41.192 test_esnap_remove_degraded 01:11:41.192 test_hotremove_lvol_store 01:11:41.192 test_hotremove_lvol_store_base 01:11:41.192 test_hotremove_lvol_store_base_with_lvols 01:11:41.192 test_hotremove_lvol_store_multiple_lvols 01:11:41.192 test_lvol_bdev_readonly 01:11:41.192 test_lvol_list 01:11:41.192 test_lvol_rename_negative 01:11:41.192 test_rename_lvs_negative 01:11:41.192 test_rename_positive 01:11:41.192 test_resize_lvol 01:11:41.192 test_resize_lvol_negative 01:11:41.192 test_resize_lvol_with_io_traffic 01:11:41.192 test_shallow_copy_compare 01:11:41.192 test_sigterm 01:11:41.192 test_snapshot_compare_with_lvol_bdev 01:11:41.192 test_tasting 01:11:41.192 test_thin_lvol_check_integrity 01:11:41.192 test_thin_lvol_check_space 01:11:41.192 test_thin_lvol_check_zeroes 01:11:41.192 test_thin_lvol_resize 01:11:41.192 test_thin_overprovisioning 01:11:41.192 test_unregister_lvol_bdev 01:11:41.192 thread 01:11:41.192 thread_poller_perf 01:11:41.192 thread_spdk_lock 01:11:41.192 ubsan 01:11:41.192 unittest 01:11:41.192 unittest_accel 01:11:41.192 unittest_bdev 01:11:41.192 unittest_bdev_compress 01:11:41.192 unittest_bdev_crypto 01:11:41.192 unittest_bdev_raid5f 01:11:41.192 unittest_blob_blobfs 01:11:41.192 unittest_build 01:11:41.192 unittest_dma 01:11:41.192 unittest_dpdk_compressdev 01:11:41.192 unittest_event 01:11:41.192 unittest_ftl 01:11:41.192 unittest_idxd_user 01:11:41.192 unittest_include 01:11:41.192 unittest_init 01:11:41.192 unittest_ioat 01:11:41.192 unittest_iobuf 01:11:41.192 unittest_iscsi 01:11:41.192 unittest_json 01:11:41.192 unittest_keyring 01:11:41.192 unittest_lib_reduce 01:11:41.192 unittest_log 01:11:41.192 unittest_lvol 01:11:41.192 unittest_notify 01:11:41.192 unittest_nvme 01:11:41.192 unittest_nvme_cuse 01:11:41.192 unittest_nvme_rdma 01:11:41.192 unittest_nvmf 01:11:41.192 unittest_nvmf_rdma 01:11:41.192 unittest_nvmf_transport 01:11:41.192 unittest_nvmf_vfio_user 01:11:41.192 unittest_pci_event 01:11:41.192 unittest_rdma 01:11:41.192 unittest_rpc 01:11:41.192 unittest_scsi 01:11:41.192 unittest_sock 01:11:41.192 unittest_thread 01:11:41.192 unittest_util 01:11:41.192 unittest_vhost 01:11:41.192 valgrind 01:11:41.192 version 01:11:41.192 vfio_fuzz 01:11:41.192 vfio_user_nvme_fio 01:11:41.192 vfio_user_nvme_restart_vm 01:11:41.192 vfio_user_qemu 01:11:41.192 vfio_user_virtio_bdevperf 01:11:41.192 vfio_user_virtio_blk_restart_vm 01:11:41.192 vfio_user_virtio_scsi_restart_vm 01:11:41.192 vhost 01:11:41.192 vhost_blk_cores_2ctrl 01:11:41.192 vhost_blk_fs_integrity 01:11:41.192 vhost_blk_integrity 01:11:41.192 vhost_blk_lvol_integrity 01:11:41.192 vhost_blk_packed_ring_integrity 01:11:41.192 vhost_blockdev 01:11:41.192 vhost_boot 01:11:41.192 vhost_fuzz 01:11:41.192 vhost_migration 01:11:41.192 vhost_migration_tc1 01:11:41.192 vhost_migration_tc2 01:11:41.192 vhost_negative 01:11:41.192 vhost_readonly 01:11:41.192 vhost_scsi_cores_2ctrl 01:11:41.192 vhost_scsi_fs_integrity 01:11:41.192 vhost_scsi_integrity 01:11:41.192 vhost_scsi_lvol_integrity 01:11:41.192 vhost_shared 01:11:41.192 vmd 01:11:41.192 vmd_bdev_svc 01:11:41.192 vmd_fio 01:11:41.192 vmd_hello_world 01:11:41.192 vmd_identify 01:11:41.192 vmd_perf 01:11:41.192 zoned_fio 01:11:41.192 01:11:41.192 01:11:41.192 -----Tests Missing From Build------ 01:11:41.192 balanced 01:11:41.192 blockdev_xnvme 01:11:41.192 build_native_dpdk 01:11:41.192 build_shared_native_dpdk_rpm 01:11:41.192 busy 01:11:41.192 core_load 01:11:41.192 dpdk_governor 01:11:41.192 ftl 01:11:41.192 ftl_bdevperf 01:11:41.192 ftl_dirty_shutdown 01:11:41.192 ftl_fio_basic 01:11:41.192 ftl_restore 01:11:41.192 ftl_restore_fast 01:11:41.192 ftl_trim 01:11:41.192 ftl_upgrade_shutdown 01:11:41.192 idle 01:11:41.192 interrupt_mode 01:11:41.192 ioat 01:11:41.192 ioat_perf 01:11:41.192 ioat_verify 01:11:41.192 iscsi_tgt_data_digest 01:11:41.192 keyring_linux 01:11:41.192 nvme_boot_partition 01:11:41.192 nvme_bp 01:11:41.192 nvme_opal 01:11:41.192 nvme_opal_bdevio 01:11:41.192 nvme_opal_bdevperf 01:11:41.192 nvme_opal_spdk_tgt 01:11:41.192 nvme_xnvme 01:11:41.192 nvmf_fc 01:11:41.192 ocf 01:11:41.192 ocf_bdevperf_iotypes 01:11:41.192 ocf_configuration_change 01:11:41.192 ocf_create_destruct 01:11:41.192 ocf_fio_modes 01:11:41.192 ocf_flush 01:11:41.192 ocf_multicore 01:11:41.192 ocf_remove 01:11:41.192 ocf_stats 01:11:41.192 scheduler 01:11:41.192 spdkcli_nvmf_fc 01:11:41.192 test_create_multi_ublk 01:11:41.192 test_create_ublk 01:11:41.192 test_save_ublk_config 01:11:41.192 ublk 01:11:41.192 ublk_recovery 01:11:41.192 unittest_nvmf_fc 01:11:41.192 unittest_nvmf_fc_ls 01:11:41.192 vhost_blk_hot_remove 01:11:41.192 vhost_hotplug 01:11:41.192 vhost_scsi_hot_remove 01:11:41.192 xnvme_bdevperf 01:11:41.192 xnvme_to_malloc_dd_copy 01:11:41.192 01:11:41.192 01:11:41.192 -----Tests Missing ASAN------ 01:11:41.192 accel_cdev_comp 01:11:41.192 accel_cdev_decmop_full 01:11:41.192 accel_cdev_decomp 01:11:41.192 accel_cdev_decomp_full_mcore 01:11:41.192 accel_cdev_decomp_full_mthread 01:11:41.192 accel_cdev_decomp_mcore 01:11:41.192 accel_cdev_decomp_mthread 01:11:41.192 autobuild 01:11:41.192 autobuild_build_doc 01:11:41.192 autobuild_check_dpdk_pci_api 01:11:41.192 autobuild_check_format 01:11:41.192 autobuild_check_so_deps 01:11:41.192 autobuild_external_code 01:11:41.192 autobuild_generated_files_check 01:11:41.192 autobuild_generated_files_check_post_clean 01:11:41.192 autobuild_header_dependency_check 01:11:41.192 autobuild_llvm_precompile 01:11:41.192 autobuild_make_install 01:11:41.192 autobuild_make_uninstall 01:11:41.192 autobuild_ocf_precompile 01:11:41.192 balanced 01:11:41.192 bdev_crypto_enomem 01:11:41.192 blockdev_crypto_aesni 01:11:41.192 blockdev_crypto_qat 01:11:41.192 blockdev_crypto_sw 01:11:41.192 blockdev_rbd 01:11:41.192 blockdev_xnvme 01:11:41.192 build_native_dpdk 01:11:41.192 build_shared_native_dpdk_rpm 01:11:41.192 busy 01:11:41.192 chaining 01:11:41.192 check_header_filenames 01:11:41.192 cmb_copy 01:11:41.192 compress_compdev 01:11:41.192 compress_isal 01:11:41.192 confirm_abi_deps 01:11:41.192 confirm_makefile_deps 01:11:41.192 core_load 01:11:41.192 dd_uring_copy 01:11:41.192 dpdk_governor 01:11:41.192 external_make_accel_driver_shared 01:11:41.192 external_make_accel_driver_static 01:11:41.192 external_make_accel_module_shared 01:11:41.192 external_make_accel_module_static 01:11:41.192 external_make_hello_bdev_shared_combo 01:11:41.192 external_make_hello_bdev_shared_iso 01:11:41.192 external_make_hello_bdev_static 01:11:41.192 external_make_hello_no_bdev_shared_combo 01:11:41.192 external_make_hello_no_bdev_shared_iso 01:11:41.192 external_make_hello_no_bdev_static 01:11:41.192 external_make_nvme_shared 01:11:41.192 external_make_nvme_static 01:11:41.192 external_run_accel_driver_shared 01:11:41.192 external_run_accel_driver_static 01:11:41.192 external_run_accel_module_shared 01:11:41.192 external_run_accel_module_static 01:11:41.192 external_run_hello_bdev_shared_combo 01:11:41.192 external_run_hello_bdev_shared_iso 01:11:41.192 external_run_hello_bdev_static 01:11:41.192 external_run_hello_no_bdev_shared_combo 01:11:41.192 external_run_hello_no_bdev_shared_iso 01:11:41.192 external_run_hello_no_bdev_static 01:11:41.192 external_run_nvme_shared 01:11:41.192 external_run_nvme_static 01:11:41.192 ftl 01:11:41.192 ftl_bdevperf 01:11:41.192 ftl_dirty_shutdown 01:11:41.193 ftl_fio_basic 01:11:41.193 ftl_restore 01:11:41.193 ftl_restore_fast 01:11:41.193 ftl_trim 01:11:41.193 ftl_upgrade_shutdown 01:11:41.193 go_rpc 01:11:41.193 idle 01:11:41.193 interrupt_mode 01:11:41.193 ioat 01:11:41.193 ioat_perf 01:11:41.193 ioat_verify 01:11:41.193 is_zoned 01:11:41.193 iscsi_tgt 01:11:41.193 iscsi_tgt_bdev_io_wait 01:11:41.193 iscsi_tgt_calsoft 01:11:41.193 iscsi_tgt_data_digest 01:11:41.193 iscsi_tgt_digest 01:11:41.193 iscsi_tgt_digests 01:11:41.193 iscsi_tgt_ext4test 01:11:41.193 iscsi_tgt_filesystem 01:11:41.193 iscsi_tgt_filesystem_btrfs 01:11:41.193 iscsi_tgt_filesystem_ext4 01:11:41.193 iscsi_tgt_filesystem_xfs 01:11:41.193 iscsi_tgt_fio 01:11:41.193 iscsi_tgt_fuzz 01:11:41.193 iscsi_tgt_initiator 01:11:41.193 iscsi_tgt_ip_migration 01:11:41.193 iscsi_tgt_iscsi_lvol 01:11:41.193 iscsi_tgt_login_redirection 01:11:41.193 iscsi_tgt_multiconnection 01:11:41.193 iscsi_tgt_qos 01:11:41.193 iscsi_tgt_rbd 01:11:41.193 iscsi_tgt_reset 01:11:41.193 iscsi_tgt_resize 01:11:41.193 iscsi_tgt_rpc_config 01:11:41.193 iscsi_tgt_sock 01:11:41.193 iscsi_tgt_trace_record 01:11:41.193 keyring_linux 01:11:41.193 llvm_fuzz 01:11:41.193 nvme_boot_partition 01:11:41.193 nvme_bp 01:11:41.193 nvme_cli_cuse 01:11:41.193 nvme_cli_plugin 01:11:41.193 nvme_cmb 01:11:41.193 nvme_cuse 01:11:41.193 nvme_cuse_app 01:11:41.193 nvme_cuse_rpc 01:11:41.193 nvme_ns_manage_cuse 01:11:41.193 nvme_opal 01:11:41.193 nvme_opal_bdevio 01:11:41.193 nvme_opal_bdevperf 01:11:41.193 nvme_opal_spdk_tgt 01:11:41.193 nvme_pmr 01:11:41.193 nvme_pmr_persistence 01:11:41.193 nvme_smartctl_cuse 01:11:41.193 nvme_xnvme 01:11:41.193 nvme_zns 01:11:41.193 nvmf_bdevperf 01:11:41.193 nvmf_device_removal 01:11:41.193 nvmf_device_removal_pci_remove 01:11:41.193 nvmf_device_removal_pci_remove_no_srq 01:11:41.193 nvmf_digest_clean 01:11:41.193 nvmf_fc 01:11:41.193 nvmf_host_multipath 01:11:41.193 nvmf_mdns_discovery 01:11:41.193 nvmf_nvme_cli 01:11:41.193 nvmf_perf_adq 01:11:41.193 nvmf_rdma 01:11:41.193 nvmf_shutdown 01:11:41.193 nvmf_shutdown_tc1 01:11:41.193 nvmf_shutdown_tc2 01:11:41.193 nvmf_shutdown_tc3 01:11:41.193 nvmf_srq_overwhelm 01:11:41.193 nvmf_target_disconnect 01:11:41.193 nvmf_target_disconnect_tc1 01:11:41.193 nvmf_target_disconnect_tc2 01:11:41.193 nvmf_target_disconnect_tc3 01:11:41.193 nvmf_timeout 01:11:41.193 nvmf_vfio_user 01:11:41.193 nvmf_vfio_user_fuzz 01:11:41.451 nvmf_vfio_user_nvme_compliance 01:11:41.451 ocf 01:11:41.451 ocf_bdevperf_iotypes 01:11:41.451 ocf_configuration_change 01:11:41.451 ocf_create_destruct 01:11:41.451 ocf_fio_modes 01:11:41.451 ocf_flush 01:11:41.451 ocf_multicore 01:11:41.451 ocf_remove 01:11:41.451 ocf_stats 01:11:41.451 scanbuild_make 01:11:41.451 scheduler 01:11:41.451 spdk_dd_uring 01:11:41.451 spdkcli_iscsi 01:11:41.451 spdkcli_nvmf_fc 01:11:41.451 spdkcli_nvmf_rdma 01:11:41.451 spdkcli_raid 01:11:41.451 spdkcli_rbd 01:11:41.451 spdkcli_virtio 01:11:41.451 test_create_multi_ublk 01:11:41.451 test_create_ublk 01:11:41.451 test_save_ublk_config 01:11:41.451 ublk 01:11:41.451 ublk_recovery 01:11:41.451 unittest_bdev_compress 01:11:41.451 unittest_bdev_crypto 01:11:41.451 unittest_dpdk_compressdev 01:11:41.451 unittest_lib_reduce 01:11:41.451 unittest_nvmf_fc 01:11:41.451 unittest_nvmf_fc_ls 01:11:41.451 unittest_nvmf_vfio_user 01:11:41.451 valgrind 01:11:41.451 vfio_fuzz 01:11:41.451 vhost_blk_hot_remove 01:11:41.451 vhost_blockdev 01:11:41.451 vhost_fuzz 01:11:41.451 vhost_hotplug 01:11:41.451 vhost_scsi_hot_remove 01:11:41.451 vhost_shared 01:11:41.451 xnvme_bdevperf 01:11:41.451 xnvme_to_malloc_dd_copy 01:11:41.451 zoned_fio 01:11:41.451 01:11:41.451 01:11:41.451 -----Tests Missing UBSAN------ 01:11:41.451 autobuild 01:11:41.451 autobuild_build_doc 01:11:41.451 autobuild_check_dpdk_pci_api 01:11:41.451 autobuild_check_format 01:11:41.451 autobuild_check_so_deps 01:11:41.451 autobuild_external_code 01:11:41.451 autobuild_generated_files_check 01:11:41.451 autobuild_generated_files_check_post_clean 01:11:41.451 autobuild_header_dependency_check 01:11:41.451 autobuild_make_install 01:11:41.451 autobuild_make_uninstall 01:11:41.451 balanced 01:11:41.451 blockdev_xnvme 01:11:41.451 build_native_dpdk 01:11:41.451 build_rpm_from_gen_spec 01:11:41.451 build_shared_native_dpdk_rpm 01:11:41.451 build_shared_rpm 01:11:41.451 build_shared_rpm_with_rpmed_dpdk 01:11:41.451 busy 01:11:41.451 check_header_filenames 01:11:41.451 cmb_copy 01:11:41.451 confirm_abi_deps 01:11:41.451 confirm_makefile_deps 01:11:41.451 core_load 01:11:41.451 dpdk_governor 01:11:41.451 external_make_accel_driver_shared 01:11:41.451 external_make_accel_driver_static 01:11:41.451 external_make_accel_module_shared 01:11:41.451 external_make_accel_module_static 01:11:41.451 external_make_hello_bdev_shared_combo 01:11:41.451 external_make_hello_bdev_shared_iso 01:11:41.451 external_make_hello_bdev_static 01:11:41.451 external_make_hello_no_bdev_shared_combo 01:11:41.451 external_make_hello_no_bdev_shared_iso 01:11:41.451 external_make_hello_no_bdev_static 01:11:41.451 external_make_nvme_shared 01:11:41.451 external_make_nvme_static 01:11:41.451 external_run_accel_driver_shared 01:11:41.451 external_run_accel_driver_static 01:11:41.451 external_run_accel_module_shared 01:11:41.451 external_run_accel_module_static 01:11:41.451 external_run_hello_bdev_shared_combo 01:11:41.451 external_run_hello_bdev_shared_iso 01:11:41.451 external_run_hello_bdev_static 01:11:41.451 external_run_hello_no_bdev_shared_combo 01:11:41.451 external_run_hello_no_bdev_shared_iso 01:11:41.451 external_run_hello_no_bdev_static 01:11:41.451 external_run_nvme_shared 01:11:41.451 external_run_nvme_static 01:11:41.451 ftl 01:11:41.451 ftl_bdevperf 01:11:41.451 ftl_dirty_shutdown 01:11:41.451 ftl_fio_basic 01:11:41.451 ftl_restore 01:11:41.451 ftl_restore_fast 01:11:41.451 ftl_trim 01:11:41.451 ftl_upgrade_shutdown 01:11:41.451 idle 01:11:41.451 interrupt_mode 01:11:41.451 ioat 01:11:41.451 ioat_perf 01:11:41.451 ioat_verify 01:11:41.451 is_zoned 01:11:41.451 iscsi_tgt_data_digest 01:11:41.451 keyring_linux 01:11:41.451 nvme_boot_partition 01:11:41.451 nvme_bp 01:11:41.451 nvme_cmb 01:11:41.451 nvme_opal 01:11:41.451 nvme_opal_bdevio 01:11:41.451 nvme_opal_bdevperf 01:11:41.451 nvme_opal_spdk_tgt 01:11:41.451 nvme_pmr 01:11:41.451 nvme_pmr_persistence 01:11:41.451 nvme_xnvme 01:11:41.451 nvme_zns 01:11:41.451 nvmf_fc 01:11:41.451 ocf 01:11:41.451 ocf_bdevperf_iotypes 01:11:41.451 ocf_configuration_change 01:11:41.451 ocf_create_destruct 01:11:41.452 ocf_fio_modes 01:11:41.452 ocf_flush 01:11:41.452 ocf_multicore 01:11:41.452 ocf_remove 01:11:41.452 ocf_stats 01:11:41.452 packaging 01:11:41.452 rpm_packaging 01:11:41.452 scanbuild_make 01:11:41.452 scheduler 01:11:41.452 spdkcli_nvmf_fc 01:11:41.452 test_create_multi_ublk 01:11:41.452 test_create_ublk 01:11:41.452 test_save_ublk_config 01:11:41.452 ublk 01:11:41.452 ublk_recovery 01:11:41.452 unittest_bdev_compress 01:11:41.452 unittest_bdev_crypto 01:11:41.452 unittest_dpdk_compressdev 01:11:41.452 unittest_lib_reduce 01:11:41.452 unittest_nvmf_fc 01:11:41.452 unittest_nvmf_fc_ls 01:11:41.452 unittest_nvmf_vfio_user 01:11:41.452 valgrind 01:11:41.452 vhost_blk_hot_remove 01:11:41.452 vhost_hotplug 01:11:41.452 vhost_scsi_hot_remove 01:11:41.452 xnvme_bdevperf 01:11:41.452 xnvme_to_malloc_dd_copy 01:11:41.452 zoned_fio 01:11:41.452 Not all tests were run. Failing the build. 01:11:41.452 ['blockdev_xnvme', 'dpdk_governor', 'ftl', 'ftl_bdevperf', 'ftl_dirty_shutdown', 'ftl_fio_basic', 'ftl_restore', 'ftl_trim', 'ftl_upgrade_shutdown', 'idle', 'interrupt_mode', 'ioat', 'ioat_perf', 'ioat_verify', 'nvme_xnvme', 'ocf', 'ocf_bdevperf_iotypes', 'ocf_configuration_change', 'ocf_create_destruct', 'ocf_fio_modes', 'ocf_flush', 'ocf_multicore', 'ocf_remove', 'ocf_stats', 'scheduler', 'test_create_multi_ublk', 'test_create_ublk', 'test_save_ublk_config', 'ublk', 'ublk_recovery', 'xnvme_bdevperf', 'xnvme_to_malloc_dd_copy'] 01:11:41.452 + compress_coverage_and_docs 01:11:41.452 + echo 'Start compress coverage and docs' 01:11:41.452 Start compress coverage and docs 01:11:41.452 + tar -C coverage -czf coverage_autotest-nightly_3706.tar.gz ./ --remove-files 01:11:42.386 + tar -C ut_coverage -czf ut_coverage_autotest-nightly_3706.tar.gz ./ --remove-files 01:11:43.319 + tar -C doc -czf doc_autotest-nightly_3706.tar.gz ./ --remove-files 01:11:43.579 + [[ -d llvm ]] 01:11:43.579 + true 01:11:43.579 + echo 'End compress coverage and docs' 01:11:43.579 End compress coverage and docs 01:11:43.579 + remove_partial_coverage_files 01:11:43.579 + find . -name 'cov_*.info' -delete 01:11:43.579 + echo 'End stage post_gen_coverage.sh' 01:11:43.579 End stage post_gen_coverage.sh 01:11:43.597 [Pipeline] } 01:11:43.601 ERROR: script returned exit code 1 01:11:43.637 [Pipeline] // catchError 01:11:43.649 [Pipeline] catchError 01:11:43.651 [Pipeline] { 01:11:43.673 [Pipeline] dir 01:11:43.674 Running in /var/jenkins/workspace/autotest-nightly/post_process 01:11:43.676 [Pipeline] {