[00:58:43] *** Quits: ziyeyang_ (~ziyeyang@192.55.46.36) (Read error: Connection reset by peer) [01:03:27] Project autotest-nightly build #415: STILL FAILING in 46 min. See https://ci.spdk.io/spdk-jenkins for results. [01:16:59] Project autotest-nightly-failing build #286: STILL FAILING in 50 min. See https://ci.spdk.io/spdk-jenkins for results. [01:56:09] *** Joins: travis-ci (~travis-ci@ec2-3-95-156-67.compute-1.amazonaws.com) [01:56:10] (spdk/master) bdev: remove delete_bdev RPC (Jim Harris) [01:56:10] Diff URL: https://github.com/spdk/spdk/compare/5f3c92c2fdbf...db524432dc35 [01:56:10] *** Parts: travis-ci (~travis-ci@ec2-3-95-156-67.compute-1.amazonaws.com) () [06:09:19] *** Joins: travis-ci (~travis-ci@ec2-3-82-58-228.compute-1.amazonaws.com) [06:09:20] (spdk/master) bdev/aio: cleanup bdev_aio.h (Jim Harris) [06:09:20] Diff URL: https://github.com/spdk/spdk/compare/db524432dc35...b5f13b368b83 [06:09:20] *** Parts: travis-ci (~travis-ci@ec2-3-82-58-228.compute-1.amazonaws.com) () [07:24:13] *** Joins: vmysak (~vmysak@134.134.139.76) [08:15:12] jimharris, any idea off the top of your head why the memset of the vol params->vol_size would take a crazy amount of time on Ubuntu 18 but maybe 10 secs on Fedora 29? Its takes so long on my Ubuntu machine the rpc create times out. Different systems but same HW, fresh install of Ubuntu [08:15:39] (sometimes times out, sometimes finishes fast enough) [08:22:20] ...read something about maybe VTD could be the culprit, I think its off on my Fed system and was on w/Ubuntu, will give it a shot... [08:34:47] peluse: maybe params->vol_size is unitialized and happens to have different values on different compilers? [08:35:04] huge on ubuntu, small on fedora [08:35:06] no, they're identical on both systems [08:35:17] all params [08:35:26] okay [08:36:01] turning off VTD in VIOS made it work like my Fedora system, down from like 45 secs to < 10 [08:36:29] maybe we need to think about using IOAT to clear this block of mem? [08:41:13] how big is the vol_size? [08:41:46] this is all bare metal i assume? [09:01:32] yes, all bare metal. i don't know the size of the top of my head, I'll look next time I'm in a good spot. Debugging an issue with delete right now [09:10:11] jimharris, "(gdb) p backing_dev_size [09:10:11] $8 = 400088457216 [09:10:11] (gdb) p params->vol_size [09:10:11] $9 = 400086351872" [09:10:39] *** Joins: travis-ci (~travis-ci@ec2-52-201-167-86.compute-1.amazonaws.com) [09:10:40] (spdk/master) nvme: clarify nvme_ctrlr_update_namespaces assignment (Jim Harris) [09:10:40] Diff URL: https://github.com/spdk/spdk/compare/b5f13b368b83...4680db9e0925 [09:10:40] *** Parts: travis-ci (~travis-ci@ec2-52-201-167-86.compute-1.amazonaws.com) () [09:13:53] where is this memset? or did you mean writing zeroes to the SSD? [09:15:45] but we don't even write zeroes to the SSD - we only fill out the pm file metadata (which should be much smaller than the vol size) [09:15:58] no [09:16:03] one sec [09:16:53] memset(vol->pm_logical_map, 0xFF, vol->pm_file.size - sizeof(*vol->backing_super)); in spdk_reduce_vol_init [09:18:27] also... pmem_msync can take way too long but not on first creation. Only when I create/delete/create again. Can talk about it later just wanted to mention that one too [09:19:26] ok - pm_file.size makes more sense [09:19:40] what's the size of pm_file.size in this case? [09:20:45] this is really expected to run a lot more slowly if the pm file isn't really on persistent memory [09:24:16] yeah, we need to do something for testing though, RPC timeout is too small. I'll look at pm file size when I'm there again. In the meantime, the memset thing is real... [09:25:05] we just need to do testing on a bdev smaller than a full NVMe SSD [09:27:51] that's too easy :) [09:28:50] wonder if should print a warning or something when we don't have pmem for dev purposes so others don't run into stuff like this without a clue as to why things are taking so long [09:29:14] and for regular use even with pm, the memset thing? [09:45:43] anyone having trouble with signing in to GerritHub today? [09:46:18] sure - an SPDK_ERRLOG or SPDK_WARNLOG in lib/reduce would be fine I think [09:47:41] we may need to think about how we should handle these kinds of operations - i could see other types of RPCs that also take a long time to complete [09:47:58] maybe we modify rpc.py to extend the timeout for certain RPCs? [10:06:01] yeah, that'd be good to have the capability to override the def TO for an RPC [10:06:06] I can add that [10:29:46] more java.io exceptions [10:29:47] https://10.102.17.104:8080/job/Other/job/blockdev_autotest/19691/console [13:00:24] *** Quits: vmysak (~vmysak@134.134.139.76) (Ping timeout: 250 seconds) [13:43:52] *** Quits: gila (~gila@5ED4D979.cm-7-5d.dynamic.ziggo.nl) () [15:25:49] *** Joins: sethhowe (~sethhowe@134.134.139.75) [15:52:28] *** Joins: travis-ci (~travis-ci@ec2-52-200-244-196.compute-1.amazonaws.com) [15:52:29] (spdk/master) iscsi: Factor out check data segment length into a helper function (Shuhei Matsumoto) [15:52:29] Diff URL: https://github.com/spdk/spdk/compare/4680db9e0925...abccec1fc812 [15:52:29] *** Parts: travis-ci (~travis-ci@ec2-52-200-244-196.compute-1.amazonaws.com) () [15:55:32] *** Joins: travis-ci (~travis-ci@ec2-3-80-36-176.compute-1.amazonaws.com) [15:55:33] (spdk/master) bdev/passthru: updaate rpc param description (paul luse) [15:55:33] Diff URL: https://github.com/spdk/spdk/compare/abccec1fc812...fa5b07ffef3e [15:55:33] *** Parts: travis-ci (~travis-ci@ec2-3-80-36-176.compute-1.amazonaws.com) () [15:58:57] *** Joins: travis-ci (~travis-ci@ec2-34-229-117-129.compute-1.amazonaws.com) [15:58:58] (spdk/master) test/lvol: Implement test case 653 (Pawel Kaminski) [15:58:58] Diff URL: https://github.com/spdk/spdk/compare/fa5b07ffef3e...32c888e57576 [15:58:58] *** Parts: travis-ci (~travis-ci@ec2-34-229-117-129.compute-1.amazonaws.com) () [23:26:53] Project autotest-nightly-failing build #287: STILL FAILING in 26 min. See https://ci.spdk.io/spdk-jenkins for results. [23:40:11] Project autotest-nightly build #416: STILL FAILING in 40 min. See https://ci.spdk.io/spdk-jenkins for results.