[00:01:01] *** Joins: tomzawadzki (~tomzawadz@134.134.139.76) [00:01:25] *** Joins: tzawadzki (~tomzawadz@134.134.139.76) [00:01:25] *** Quits: tomzawadzki (~tomzawadz@134.134.139.76) (Remote host closed the connection) [01:10:02] *** Joins: tkulasek (~tkulasek@192.55.54.45) [01:37:20] *** Quits: alessio (~alessio@140.105.207.227) (Quit: alessio) [01:45:57] *** Joins: alessio (~alessio@140.105.207.227) [03:29:37] *** Quits: alessio (~alessio@140.105.207.227) (Quit: alessio) [03:46:27] *** Quits: dlw (~Thunderbi@114.255.44.143) (Quit: dlw) [04:11:14] *** Joins: tzawadzki_ (tomzawadzk@nat/intel/x-kizwnmxjyhrwhxia) [04:14:05] *** Quits: tzawadzki (~tomzawadz@134.134.139.76) (Ping timeout: 240 seconds) [04:55:33] *** Joins: alessio (~alessio@140.105.207.227) [05:15:39] *** Quits: alessio (~alessio@140.105.207.227) (Quit: alessio) [05:17:13] *** Joins: alessio (~alessio@140.105.207.227) [05:57:20] *** Joins: dlw (~Thunderbi@114.246.78.47) [05:58:05] *** Quits: dlw (~Thunderbi@114.246.78.47) (Client Quit) [06:44:52] *** Joins: jkkariu (jkkariu@nat/intel/x-twnjjhkdwgdohdox) [07:07:30] *** Joins: dlw (~Thunderbi@114.246.78.47) [07:07:31] *** Quits: dlw (~Thunderbi@114.246.78.47) (Client Quit) [07:14:03] hmmm, failure on last night's build. https://ci.spdk.io/spdk/status/ I'm in a meeting but will look shortly, has anyone else already done any triage? [07:46:51] 3 test systems failed with different errors on each [07:47:32] on centos, I think I've run into this every once in a while in my dev env. What would cause this in what should be a more stable/predictable setup? [07:47:34] EAL: probe driver: 8086:5845 spdk_nvme [07:47:34] Cannot create lock on device /tmp/spdk_pci_lock_0000:00:04.0, probably process 10128 has claimed it [07:47:34] nvme_pcie.c: 787:nvme_pcie_ctrlr_construct: *ERROR*: could not claim device 0000:00:04.0 [07:47:34] nvme.c: 347:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe controller for SSD: 0000:00:04.0 [07:47:35] EAL: Requested device 0000:00:04.0 cannot be used [07:47:39] . [07:48:35] on fedora 4 I've not seen this before, seems a little out of place, maybe the result of some earlier not so easy to spot failure in the log and/or test setup? [07:48:36] EAL: probe driver: 8086:5845 spdk_nvme [07:48:37] Cannot create lock on device /tmp/spdk_pci_lock_0000:00:04.0, probably process 10128 has claimed it [07:48:38] nvme_pcie.c: 787:nvme_pcie_ctrlr_construct: *ERROR*: could not claim device 0000:00:04.0 [07:48:39] nvme.c: 347:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe controller for SSD: 0000:00:04.0 [07:48:41] EAL: Requested device 0000:00:04.0 cannot be used [07:48:44] , [07:49:29] and on ubuntu 16.04 did we just caught with ASLR in a multi-proc setup? [07:49:30] EAL: Auto-detected process type: SECONDARY [07:49:30] EAL: Multi-process socket /var/run/.spdk0_unix_26875_65d5ada5f54 [07:49:30] EAL: Probing VFIO support... [07:49:30] EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel. [07:49:31] EAL: This may cause issues with mapping memory into secondary processes [07:49:32] EAL: Could not mmap 1906311168 bytes in /dev/zero at [0x7f48e6c00000], got [0x7f48808ff000] - please use '--base-virtaddr' option [07:49:35] EAL: It is recommended to disable ASLR in the kernel and retry running both primary and secondary processes [07:49:39] EAL: FATAL: Cannot init memory [07:49:41] EAL: Cannot init memory [07:49:43] Failed to initialize DPDK [07:49:45] . [08:16:34] *** Quits: tzawadzki_ (tomzawadzk@nat/intel/x-kizwnmxjyhrwhxia) (Ping timeout: 264 seconds) [08:23:23] *** Quits: sage___ (~quassel@2607:f298:5:101d:f816:3eff:fe21:1966) (Read error: Connection reset by peer) [08:23:45] *** Joins: sage_ (~quassel@2607:f298:5:101d:f816:3eff:fe21:1966) [08:25:53] *** Quits: alessio (~alessio@140.105.207.227) (Remote host closed the connection) [08:28:52] on the centos system - it's a multi-proc failure - the problem is a bit further up from the bottom of the log: [08:28:54] EAL: WARNING! Base virtual address hint (0x1000000000 != 0x7f2bce773000) not respected! [08:28:55] EAL: This may cause issues with mapping memory into secondary processes [08:30:12] ah, cool. missed that one. that's 2/3 of them I think [08:30:13] odd - we get 4 of these errors - indicating that the rest of the hugepage mappings respected the requested base virtual address [08:30:39] the fedora-04 one I have been debugging in my free time [08:30:52] it's not a new failure [08:31:59] *** Joins: alessio (~alessio@140.105.207.227) [08:43:57] *** Joins: travis-ci (~travis-ci@ec2-54-197-66-174.compute-1.amazonaws.com) [08:43:58] (spdk/master) test/iscsi: use veth interfaces instead of loopback (Tomasz Zawadzki) [08:43:58] Diff URL: https://github.com/spdk/spdk/compare/2393edd538d0...7da3afd1f782 [08:43:58] *** Parts: travis-ci (~travis-ci@ec2-54-197-66-174.compute-1.amazonaws.com) () [09:17:06] jimharris: I think the DPDK mapping ending up somewhere else than we put it is related to ASAN being re-enabled recently [09:17:33] I suspect ASLR causes the allocations that ASAN makes to sometimes end up on top of our --base-virtaddr region [09:28:30] should we disable ASLR in the test scripts that do multi-process, if ASAN is enabled? [09:28:50] that seems reasonable to me [09:29:00] or just as part of vm_setup in general [09:29:59] *** Quits: tkulasek (~tkulasek@192.55.54.45) (Ping timeout: 268 seconds) [10:01:42] *** Joins: travis-ci (~travis-ci@ec2-54-198-123-118.compute-1.amazonaws.com) [10:01:43] (spdk/master) blobfs: Remove all uses of strncpy (Ben Walker) [10:01:43] Diff URL: https://github.com/spdk/spdk/compare/7da3afd1f782...38d75b56f4d4 [10:01:43] *** Parts: travis-ci (~travis-ci@ec2-54-198-123-118.compute-1.amazonaws.com) () [10:08:59] I think we need to work out the semantics of what "snapshot" and "clone" really mean - or someone needs to explain them to me [10:09:13] I think I see what the code is doing, but I'm not sure it is intuitive [10:09:31] so on snapshot, if I started with volume A [10:09:55] it makes a new volume, volume A' that is read only, and then updates volume A to be thin provisioned backed by A' [10:10:23] and on clone, if I started with volume A [10:10:37] it makes a new volume, volume B, backed by volume A? [10:10:52] and volume A in the clone case must have already been read only? [10:13:35] yeah http://www.spdk.io/doc/logical_volumes.html could use an update with pictures, labels and scenarios for thin provisioning, snapshot and close... [10:14:10] I'll add it to the next Euro comm meeting as an agenda topic so we don't forget [10:14:46] well, Asia is next so we'll likely talk about it twice briefly :) [10:14:56] the mechanics of snapshot seem clear and intuitive to me [10:15:01] clone is the one where I'm not so sure [10:15:45] and we may be better served by exposing "spdk_blob_snapshot" objects in the public API [10:15:51] they can just be blobs that are read-only internally [10:20:24] jimharris: I'm catching up on some of the messages above, and I'm confused on the "...should we disable ASLR in the test scripts that do multi-process, if ASAN is enabled?" My understanding is/was that ASAN is entirely unrelated to ASLR-enablement. Fundamentally, from past readings I thought that ASLR had to be disabled, period, for any multi-process SPDK apps owing to the way some shared addresses are hash'd. [10:24:52] drv: Since you thought the above "seems reasonable", then I'm left to conclude I'm misunderstood something (or something changed) with respect to ASLR and multi-process SPDK apps? [10:25:53] lhodev: I haven't verified my assumptions yet, but I think what is happening is that when ASAN is enabled, it allocates a (large) region of virtual address space to use for tracking which memory is valid/freed [10:26:12] and when ASLR is enabled, this can sometimes overlap the region where DPDK wants to map its shared memory region (--base-virtaddr option) [10:26:34] and ASAN gets to start up very early on, so this happens before DPDK gets a chance to map it [10:27:43] if we turn off ASLR for the test machines, this should cause the layout of allocations to be consistent across runs, so that ASAN will always get the same region of memory that should never interfere with the DPDK shared memory region [10:29:05] For the sake of a base-line understanding, can we (if possible) set aside any mention of ASAN? That is, did I misunderstand that there's a requirement that ASLR must be disabled for multi-process SPDK apps? Or, did something change that allows multi-process SPDK apps to run properly with ASLR enabled? [10:30:19] multi-process works most of the time with ASLR (assuming ASAN is turned off) because we specify the DPDK --base-virtaddr option pointing to a region that should normally not be used by ASLR-randomized allocations [10:32:54] in theory, because we're specifying that --base-virtaddr argument to DPDK, multiprocess and ASLR should work together [10:33:05] drv: Thanks Daniel. I'm going to attempt to locate the doc and/or code that led me to this (mis)understanding so that I grok the ramifications of the address layouts. [10:33:06] prior to that fix, they did not [10:33:41] it's still not guaranteed that ASLR won't put stuff into the region we ask DPDK to use for shared memory [10:34:06] we're using implementation details about how Linux currently lays out its address space [10:34:22] bwalker: Re: your "prior to that fix" -- you mean the addition of a new option, '--base-virtaddr', or was there some fix to the implementation of exactly how that operated? [10:34:33] the addition of our use of --base-virtaddr [10:34:40] when we start up DPDK [10:34:56] bwalker: Ok, thanks Ben! [10:35:17] and in particular, the value we're passing there is key. drv is right that the value is chosen knowing implementation details about Linux [10:36:21] but even all of that put together, we were still seeing failures periodically where --base-virtaddr couldn't be respected. That's the problem we're discussing today and we think the reason there is something at that address is that ASAN put it there. [10:39:09] bwalker, drv: Does that (i.e. chosen knowing implementation details about Linux) put us at risk making use of that knowledge? That is, it could change from underneath us in the future? Or, is there mechanism by which we can derive address layout info (parsing ELF sections)? [10:42:13] well, one sure-fire thing that should always be safe is "don't use multiprocess" :) [10:42:33] I don't think there's any way around this issue for multi-process in general, since the kernel is free to lay things out differently in different processes [10:43:06] we're definitely vulnerable, but the way it breaks is that the primary process just puts the data in a different place [10:43:10] and the secondary fails to start up [10:43:28] so multiprocess gets broken, but everything else continues to function [10:47:39] I need to go grab some lunch before I gnaw off my arm. Should we put this put as a topic of (additional) discussion for tomorrow's meeting? [10:48:02] when we start the primary process, we could dump pmap to the log [10:48:38] just for debugging purposes - should help prove/disprove whether there was a conflict allocation to that address [10:49:07] *** Joins: travis-ci (~travis-ci@ec2-54-80-116-49.compute-1.amazonaws.com) [10:49:08] (spdk/master) lvol: add snapshots and clones (Tomasz Zawadzki) [10:49:09] Diff URL: https://github.com/spdk/spdk/compare/38d75b56f4d4...97934c5291e1 [10:49:09] *** Parts: travis-ci (~travis-ci@ec2-54-80-116-49.compute-1.amazonaws.com) () [10:50:35] lhodev: there is a thread on the mailing list to discuss some major issues with multi-process that was just started. I'm about to weight in. I'd love to hear what you think [10:51:02] *** Parts: alessio (~alessio@140.105.207.227) ("Good Bye") [10:52:19] the other option for making multi-process work reliably would be to rewrite anything that touches shared memory to use a base + offset addressing model [10:52:27] that would allow the shared memory region to be mapped in different places in each process [10:53:01] e.g. no direct pointers into the shared memory region would be allowed [10:53:06] except by default DPDK maps each huge page separately [10:53:39] yeah, that is a problem, plus it would require rewriting pretty much everything in DPDK, which is not really viable [12:17:40] *** Joins: param (~param@157.49.215.100) [12:20:27] ben : spdk_nvmf_transport_poll_group_create is called in spdk_nvmf_tgt_listen.. but spdk_nvmf_poll_group_destroy is called during spdk_nvmf_tgt_destroy_poll_group. so should spdk_nvmf_poll_group_destroy shoould be called in function which will perform the opposite of spdk_nvmf_tgt_listen? [12:46:47] *** Joins: travis-ci (~travis-ci@ec2-54-80-116-49.compute-1.amazonaws.com) [12:46:48] (spdk/master) nvme: use AER configuation structure when starting controller (Changpeng Liu) [12:46:48] Diff URL: https://github.com/spdk/spdk/compare/af9c9a415ac7...2d192cf8fb5d [12:46:48] *** Parts: travis-ci (~travis-ci@ec2-54-80-116-49.compute-1.amazonaws.com) () [12:48:53] *** Joins: travis-ci (~travis-ci@ec2-54-80-116-49.compute-1.amazonaws.com) [12:48:54] (spdk/master) nvmf: fix non-deallocate Dataset Management status (Daniel Verkamp) [12:48:55] Diff URL: https://github.com/spdk/spdk/compare/2a3d7bf373ac...565932d27a33 [12:48:55] *** Parts: travis-ci (~travis-ci@ec2-54-80-116-49.compute-1.amazonaws.com) () [12:49:32] *** Joins: travis-ci (~travis-ci@ec2-54-198-123-118.compute-1.amazonaws.com) [12:49:32] (spdk/master) bdev: update the name of Passthru section (GangCao) [12:49:32] Diff URL: https://github.com/spdk/spdk/compare/565932d27a33...9bad29c18c0a [12:49:32] *** Parts: travis-ci (~travis-ci@ec2-54-198-123-118.compute-1.amazonaws.com) () [14:07:04] *** Quits: param (~param@157.49.215.100) (Remote host closed the connection) [16:07:47] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [16:27:40] *** Joins: travis-ci (~travis-ci@ec2-54-198-123-118.compute-1.amazonaws.com) [16:27:41] (spdk/master) bdev: Enable lvol resize RPC (Slawomir Mrozowicz) [16:27:41] Diff URL: https://github.com/spdk/spdk/compare/e433001a4773...7552707ef16a [16:27:41] *** Parts: travis-ci (~travis-ci@ec2-54-198-123-118.compute-1.amazonaws.com) () [16:28:53] param: If you tear down the last listener for a transport, you should probably uninitialize the whole transport. But you can leave that for a separate patch. [16:45:20] *** Quits: guerby (~guerby@april/board/guerby) (Remote host closed the connection) [16:45:27] *** Joins: guerby (~guerby@ip165.tetaneutral.net) [16:45:27] *** Quits: guerby (~guerby@ip165.tetaneutral.net) (Changing host) [16:45:27] *** Joins: guerby (~guerby@april/board/guerby) [17:55:51] *** Joins: dlw (~Thunderbi@114.255.44.143) [18:10:45] *** Joins: mphardy (~mphardy@pool-72-83-7-2.washdc.fios.verizon.net) [18:13:19] *** Parts: mphardy (~mphardy@pool-72-83-7-2.washdc.fios.verizon.net) () [18:13:22] *** Joins: mphardy (~mphardy@pool-72-83-7-2.washdc.fios.verizon.net) [18:54:02] FYI another multi-proc ASLR failure: https://ci.spdk.io/spdk/builds/review/fbb625370fe2d10bca851f748df592a156d16503.1523404157/ubuntu17.10/build.log [18:54:11] but it passed on Jenkins :) [18:54:20] can someone please restart CI on https://review.gerrithub.io/#/c/407222/ [19:20:19] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 260 seconds) [20:01:45] *** Joins: ziyeyang_ (~ziyeyang@134.134.139.74) [20:06:00] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [20:33:49] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 260 seconds) [23:57:46] *** Joins: tomzawadzki (tomzawadzk@nat/intel/x-ksqrzwqvfagovfuu)