[00:04:23] hi nate12112_spdk [00:04:36] could you please share your results on mailing list? [00:23:12] *** Quits: mszwed (~mszwed@192.55.54.39) (Ping timeout: 250 seconds) [00:40:03] *** Joins: mszwed (mszwed@nat/intel/x-fuirbdupaszkzkfk) [00:45:17] *** Joins: travis-ci (~travis-ci@ec2-54-81-97-97.compute-1.amazonaws.com) [00:45:18] (spdk/master) bdev/ocf: Add missing error handling in bottom adapter (Robert Bałdyga) [00:45:18] Diff URL: https://github.com/spdk/spdk/compare/a1c5442d166f...ec50de0957fe [00:45:18] *** Parts: travis-ci (~travis-ci@ec2-54-81-97-97.compute-1.amazonaws.com) () [00:56:10] Project autotest-nightly build #407: STILL FAILING in 33 min. See https://ci.spdk.io/spdk-jenkins for results. [01:01:14] Project autotest-nightly-failing build #278: STILL FAILING in 32 min. See https://ci.spdk.io/spdk-jenkins for results. [02:11:41] *** Joins: travis-ci (~travis-ci@ec2-54-162-117-7.compute-1.amazonaws.com) [02:11:42] (spdk/master) bdev/nvme: set hotplug poller with default period value (Changpeng Liu) [02:11:42] Diff URL: https://github.com/spdk/spdk/compare/ec50de0957fe...f8dfbc5e9f77 [02:11:42] *** Parts: travis-ci (~travis-ci@ec2-54-162-117-7.compute-1.amazonaws.com) () [05:31:01] *** Quits: guerby (~guerby@april/board/guerby) (Remote host closed the connection) [05:35:38] *** Joins: guerby (~guerby@april/board/guerby) [07:39:18] *** Joins: tomzawadzki (uid327004@gateway/web/irccloud.com/x-wvsqtzxecsecenpi) [08:01:20] *** Joins: gila (~gila@5ED4D979.cm-7-5d.dynamic.ziggo.nl) [08:08:52] jimharris, hey I figured out why my 18.04 and then 18.10 can't build w/crypto but now I don't see exactly how they're working every else :) [08:09:38] the -I we have in the dpdkbuil/Makefile points to one dir behind the isa-l dir such that the includes in isa-l.h can be founf using isa-l/include/.h [08:10:07] however on ubuntu make isn't picking up isa-l.h as included by DPDK because its in the isa-l root dir [08:10:13] so, this works for me: DPDK_CFLAGS += -I$(ISAL_DIR)/.. [08:10:13] +DPDK_CFLAGS += -I$(ISAL_DIR) [08:10:55] so that both the isa-l dir and one behind it are in the search path. ANyone have any idea how dpdk is finding isa-lh without the 2nd -I? [08:24:00] there's no isa-l header files in your system include directories? [08:25:42] there's no supposed to be right, we're not installing it? There might be on my other systesm from back when we were installing [08:27:03] yeah, there is on my other dev systems but those have old installations [08:37:23] darsto: this vhost 2MB multiple check was just added in 19.01; besides this failure that liang reported, is there any reason qemu 2.11 shouldn't work, at least for vhost-scsi? [08:38:48] jimharris: no, i'm not aware of any. My comment only based on that 2MB alignment error [08:38:52] *** Joins: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) [08:39:13] ok - did you see the comment i just added on github? i'm wondering if we should be checking mmap_size instead of size for that 2mb multiple check [08:39:21] we could ask liang to try that [08:40:25] in the code i can see mmap_size = RTE_ALIGN_CEIL(mmap_size, get_blk_size(region_fd)); [08:40:47] so that will be always a 2mb multiple for hugepages [08:41:04] hmmm - yeah, it sort of defeats the purpose of the check [08:41:43] the check was for non-hugepage based clients, wasn't it? [08:42:19] oh - wait, no I think that would work - if user tried to connect with non-hugepage memory, get_blk_size should return 4KB not 2MB (or 1GB) [08:42:21] anyway, what bad can happen if we start writing to VGA BIOS memory? [08:42:57] nothing really - i mean on more recent QEMU, it doesn't split the memory regions at all [08:43:18] the VGA BIOS hole is still there, but it's not getting communicated via SET_MEM_TABLE [08:43:44] so that memory still belongs to the guest VM - if something in the guest VM stupidly decides to DMA something to VGA memory, then we'll do it [08:44:42] *** Joins: felipef (~felipef@62.254.189.133) [08:44:43] oh okay [08:45:17] btw [08:45:42] do you recall why we register guest memory with "start = FLOOR_2MB(region->mmap_addr);"? [08:46:14] shouldn't that address be aligned as well? [08:47:22] I mean it should be 2MB aligned from the very beginning [08:49:47] i need to run - i'll be back in an hour [09:59:39] klateck: for the nightly test failing the connect/disconnect test - is it using soft roce or a real RNIC? [10:40:21] bwalker the failing one is using softroce [10:40:32] real hardware test passes [10:49:24] guys, I am very confused why I do not see significant difference when switching from nvme-cli to spdk [10:49:53] I used an enterprise drive so the drive itself is NOT the limitation [10:51:04] klateck: this is a soft roce bug. I just notified linux-rdma yesterday [10:51:18] sethhowe might have a solid workaround coming though [10:51:43] nate12112_spdk: can you try doing benchmarks of 4k I/O at queue depth 1? [10:52:40] you mean 4K random reads or 4K Random Writes with Queue Depth 1x1 [10:52:41] ? [10:53:12] try random reads [10:54:09] k...brb [10:54:35] *** Quits: tomzawadzki (uid327004@gateway/web/irccloud.com/x-wvsqtzxecsecenpi) (Quit: Connection closed for inactivity) [10:56:42] bwalker: I don't see that failure on the latest nightly tests, Do you know what day it failed so I can look at the logs? I assume this is the problem with completion queue elements not getting decremented on disconnect. [10:57:37] I haven't seen any logs either, just this patch: https://review.gerrithub.io/c/spdk/spdk/+/445660 [10:57:47] but it has the error message we were getting when we ran out of memory [10:58:06] because rxe wasn't completing the dummy recv [10:59:55] Yeah. That sounds right. I need to get back to determining the proper solution there. I just haven't set aside time to do the tests I need to. I'll prioritize that. [11:07:51] *** Joins: travis-ci (~travis-ci@ec2-54-81-97-97.compute-1.amazonaws.com) [11:07:52] (spdk/master) nvme: add memory barrier in completion path for arm64 (heyang) [11:07:52] Diff URL: https://github.com/spdk/spdk/compare/80ff32cfcc07...7cd3a6f5e061 [11:07:52] *** Parts: travis-ci (~travis-ci@ec2-54-81-97-97.compute-1.amazonaws.com) () [11:09:22] *** Joins: travis-ci (~travis-ci@ec2-54-83-86-95.compute-1.amazonaws.com) [11:09:23] (spdk/master) scripts: vm_setup.sh fix OCF github repo path (Vitaliy Mysak) [11:09:23] Diff URL: https://github.com/spdk/spdk/compare/f8dfbc5e9f77...80ff32cfcc07 [11:09:23] *** Parts: travis-ci (~travis-ci@ec2-54-83-86-95.compute-1.amazonaws.com) () [11:12:06] *** Joins: travis-ci (~travis-ci@ec2-54-173-246-246.compute-1.amazonaws.com) [11:12:07] (spdk/master) nvmf/tcp: fix error message printing in spdk_nvmf_tcp_qpair_set_recv_state (Ziye Yang) [11:12:07] Diff URL: https://github.com/spdk/spdk/compare/7cd3a6f5e061...2da86de69f2a [11:12:07] *** Parts: travis-ci (~travis-ci@ec2-54-173-246-246.compute-1.amazonaws.com) () [11:12:17] @bwalker --> NVME-Cli read: IOPS=399k, BW=1560MiB/s (1636MB/s)(8192MiB/5252msec) [11:12:27] *** Joins: travis-ci (~travis-ci@ec2-54-226-199-254.compute-1.amazonaws.com) [11:12:28] (spdk/master) doc: add documentation for gdb_macros (shahar salzman) [11:12:28] Diff URL: https://github.com/spdk/spdk/compare/2da86de69f2a...d047e25c3f9b [11:12:28] *** Parts: travis-ci (~travis-ci@ec2-54-226-199-254.compute-1.amazonaws.com) () [11:12:41] @bwalker ---> SPDK read: IOPS=410k, BW=1601MiB/s (1679MB/s)(8192MiB/5116msec) [11:13:30] @bwalker --> NVME-Cli read: IOPS=399k, BW=1560MiB/s (1636MB/s)(8192MiB/5252msec) [11:14:25] @bwalker ---> SPDK iops : min=405887, max=411429, avg=409930.60, stdev=2293.25, samples=5 [11:14:56] bwalker ----> NVME-Cli iops : min=398377, max=399928, avg=399328.80, stdev=691.13, samples=5 [11:15:36] wait....that is at Queue Depth 32...shoot...let me change my script [11:23:42] *** Quits: felipef (~felipef@62.254.189.133) (Remote host closed the connection) [11:25:14] @bwalker --> NVMe-Cli read: IOPS=101k, BW=393MiB/s (412MB/s)(8192MiB/20838msec) iops : min=96748, max=101974, avg=100638.75, stdev=996.29, samples=20 [11:25:47] @bwalker ----> read: IOPS=148k, BW=579MiB/s (607MB/s)(8192MiB/14154msec) iops : min=147969, max=148339, avg=148165.21, stdev=118.73, samples=14 [11:45:03] what physical device was this again? [11:45:41] because 101k to 148k seems like a pretty good improvement at QD 1 [11:45:49] 50% almost [12:08:18] at 32 QD you see the performance gap disappear [12:08:44] it was Intel P3700 Enterprise series [12:09:26] here are some examples: [12:10:27] SPDK read: IOPS=410k, BW=1601MiB/s (1679MB/s)(8192MiB/5116msec) iops : min=405887, max=411429, avg=409930.60, stdev=2293.25, samples=5 [12:10:58] NVMe-Cli read: IOPS=399k, BW=1560MiB/s (1636MB/s)(8192MiB/5252msec) iops : min=398377, max=399928, avg=399328.80, stdev=691.13, samples=5 [12:22:21] @bwalker....any explanation for the 32 Queue Depth [12:22:23] ? [12:32:36] bwalker, sethhowe: can we merge https://review.gerrithub.io/c/spdk/spdk/+/445660 for now, while you debug the soft-roce /dev/nvme-fabrics error message? [12:51:39] peluse: do you have dpdk compressdev building with that 445767 patch + removing isa-l from system directories? [12:53:12] why does SPDK perform approx 50% better on fio test with Queue Depth 1 when compared with NVMe-Cli versus with Queue Depth 32 (At Queue Depth 32 they are virtually the same performance) [12:56:11] peluse: i don't think it works [12:57:39] checking [12:59:31] no it does not build *without* the patch (after I removed ISAL from my dev system) and yes it does build with the patch [12:59:38] why don't you think it works? [13:00:54] nate12112_spdk: I suspect that's because your device is saturating [13:01:14] jimharris, have a call... I'll reply on GH but in short DPDK can't find isa-l.h unless (a) its in the system dir or (b) we add -I to the right dir [13:02:59] jimharris: merged. We know what the bug is (rxe is broken) but we need to work around it for iwarp too actually. sethhowe is going to come up with a fix [13:04:23] nate12112_spdk: what's the capacity of the drive? [13:04:46] they generally can only do ~450k iops [13:05:05] which you may not be able to reach at 32 queue depth, but it's close [13:05:31] I usually test SPDK on a system with 12-24 SSDs - you'll see a giant difference in that scenario [13:06:00] for example, libaio can saturate about 1 of those SSDs per core [13:06:05] spdk can saturate 7 of them [13:07:56] *** Joins: travis-ci (~travis-ci@ec2-52-91-242-50.compute-1.amazonaws.com) [13:07:57] (spdk/master) test/nvmf: move connect_disconnect to NIGHTLY_FAILING (Karol Latecki) [13:07:57] Diff URL: https://github.com/spdk/spdk/compare/95fc5d35816b...0d5cbbed16a4 [13:07:57] *** Parts: travis-ci (~travis-ci@ec2-52-91-242-50.compute-1.amazonaws.com) () [13:17:12] peluse: so on your system you have it building with this patch? [13:17:57] because isa-l.h includes isa-l/crc.h - but that's not how the isa-l submodule is laid out [13:18:21] i'm assuming you must have a magic trick here so please fill me in :) [13:30:44] bwalker: darsto has a question for you on https://review.gerrithub.io/c/spdk/spdk/+/440762 [13:32:23] answered - basically I had that as part of the series earlier on, but then decided things merged more smoothly with that coming much later on [13:32:34] and I just haven't gotten to the "much later on" patches in order to resubmit that one [13:32:53] so it will get updated shortly [13:38:06] *** Joins: travis-ci (~travis-ci@ec2-54-83-86-95.compute-1.amazonaws.com) [13:38:07] (spdk/master) nvme: mv submit_tick assignments to generic qpair code (lorneli) [13:38:07] Diff URL: https://github.com/spdk/spdk/compare/0d5cbbed16a4...815f82b17bda [13:38:07] *** Parts: travis-ci (~travis-ci@ec2-54-83-86-95.compute-1.amazonaws.com) () [14:17:05] @bwalker it is a 2TB drive [14:17:51] @bwalker doing an NVMe format does not help with QD 32? [14:22:37] jimharris, sorry if this is a stupid question but the symlink solution works, I put one symlink per .h file in the isa-l dir since the DPDK includes look like this: #include Is there a better way, I'd have to put them in the submodule and then update it if not of course [14:45:06] doing an nvme format, if that means unmapping the whole drive, will make random reads fast regardless of queue depth [14:47:28] peluse: i think you'll need to keep both the existing $(ISAL_DIR)/.. and add your new $(ISAL_DIR) [14:47:44] and then just symlink isa-l/include to isa-l/isa-l [14:48:12] oh no - sorry, you should only need your new directory - you can remove the existing $(ISAL_DIR)/.. [14:48:27] if you point to $(ISAL_DIR), it contains isa-l.h which DPDK includes [14:48:36] isa-l.h includes isa-l/crc.h [14:48:49] so the symlink would enable that [14:48:52] does that make sense? [14:52:13] haha, its kind of a tangled mess :) I'll give it a whirl... [14:52:39] you shouldn't need to symlink each file - you should be able to just symlink the include directory [14:53:58] yup, understood [14:54:11] peluse: This discussion about the isa-l stuff has caught my attention. I've got to resolve how we will build the SPDK pkg going forward without use of a git-submodule, so anything special in this arena is of high interest to me. [14:57:23] I don't have the full context of this apparent need for creating symlinks related to the isa-l. [14:58:19] Reviewing the irclog, I'm guessing perhaps this started as a sidebar discussion between you and @jimharris? [15:00:21] Can anyone take a look at recent blockdev_autotest fails in Jenkins? [15:00:51] It seems to be failing also when run on current master [15:01:26] lhodev: isa-l header files get installed in system directories in a different hierarchy than they exist in the spdk submodule [15:01:35] or rather the isa-l submodule in spdk [15:02:06] so we're actually aligning spdk's use of the isa-l submodule to match what the installed packages look like [15:02:45] So "isa-l" is available as an installable package? [15:04:29] yes [15:04:57] but we have a submodule and install in the spdk root under isa-l, not in the system dirs [15:06:34] I did a search for "isa-l" on one of my Oracle Linux boxes, but failed to get a hit. I may not have all the needed yum repo channels enabled, though. Do you know which Linux distro's are carrying isa-l ? [15:07:56] lhodev, see https://github.com/01org/isa-l/wiki/Ports--Repos [15:08:27] klateck: not sure what the problem is - i pulled master on my system and ran this test and it passed [15:08:33] but my systems doesn't have IOMMU enabled [15:08:36] my system [15:09:06] looks like this command fails during cleanup: [15:09:07] sudo test/env/env_dpdk_post_init/env_dpdk_post_init -c 0x1 --base-virtaddr=0x200000000000 [15:09:41] peluse: Thanks for the link. I see at that URL no mention of RedHat :-( [15:10:45] yeah, I jsut asked Greg (owner sits next to me in the lab) and he said they're going to try again soon... [15:10:49] to get in there I mean [15:12:40] jimharris, OK so with 2 changes (1) dpdkbuild/Make file pointing to just $(ISA_LDIR) and (b) new symlink in the isa-l subdir "isa-l -> include" I can build. This is what you meant right? [15:13:54] exactly [15:14:13] sweet, OK I'll udpate the patch and submit a new one to the ISAL fork to add the symlink [15:16:10] jimharris, do you want to create a branch in the isal fork first? I don't think I have the clearance for that [15:16:49] * peluse BRB [15:17:47] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [15:30:30] i don't think we need a fork - just create the symlink in isalbuild/Makefile [15:38:00] oh, didn't know you could do that. will try [16:19:08] *** Quits: gila (~gila@5ED4D979.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [16:57:18] jimharris, OK just updated the patch. I was unable to make the symlink in the isalbuild Makefile because I couldn't get the top level Makefile ordering to hit isa-l first so DPDK would get built before isa-l and fail [16:57:57] so I added it in the top level Makefile but don't know Makefile syntax from a shinola but I did try several methods. Take a look and let me know if there's a cleaner solution [16:59:46] BTW I tried conditionally adding the new target to $(DIRS-y) but it kept thinking it was a dir so that's why I redefined the all target the way I did [17:30:10] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [17:34:41] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 255 seconds) [17:42:41] *** Joins: travis-ci (~travis-ci@ec2-54-224-249-80.compute-1.amazonaws.com) [17:42:42] (spdk/master) nvmf: Expose bdev's PI setting to NVMe-oF Initiator (Shuhei Matsumoto) [17:42:42] Diff URL: https://github.com/spdk/spdk/compare/815f82b17bda...df99e281587c [17:42:42] *** Parts: travis-ci (~travis-ci@ec2-54-224-249-80.compute-1.amazonaws.com) () [20:03:17] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds) [23:29:31] Project autotest-nightly-failing build #279: STILL FAILING in 29 min. See https://ci.spdk.io/spdk-jenkins for results. [23:48:59] Project autotest-nightly build #408: STILL FAILING in 48 min. See https://ci.spdk.io/spdk-jenkins for results. [23:50:02] there is a slight delay before https://ci.spdk.io/spdk-jenkins gets updated [23:50:27] currently it still points at the previous nightly as a latest one - #407 [23:52:15] and nightly #408 gives 404 on the logs, of course [23:52:18] I'm done