[00:10:14] *** Joins: travis-ci (~travis-ci@ec2-54-90-147-132.compute-1.amazonaws.com) [00:10:15] (spdk/master) tcp: Initialize mutex only if everything else succeeded (Maciej Szwed) [00:10:15] Diff URL: https://github.com/spdk/spdk/compare/c1ce8db0df7f...be0eb272d806 [00:10:15] *** Parts: travis-ci (~travis-ci@ec2-54-90-147-132.compute-1.amazonaws.com) () [02:19:23] *** Joins: drv_ (daniel@oak.drv.nu) [02:19:24] *** ChanServ sets mode: +o drv_ [02:25:12] *** Quits: drv (daniel@oak.drv.nu) (*.net *.split) [02:43:13] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Remote host closed the connection) [03:26:40] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [03:34:06] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 244 seconds) [06:49:31] klateck, saw your Jenkinx IRC question, right on! If I don't see a reply in 6 hours or so I'll repeat it for you in case their main folks are in a TZ where they aren't likely to see/reply to questions at this particular time of day [06:59:36] Thanks peluse! [06:59:49] I think I am going also to try some stackoverflow or someplace else too [07:21:42] good plan [08:05:31] darsto: can you take a look at https://review.gerrithub.io/#/c/spdk/spdk/+/434833/ [08:05:49] it seems the code to try to wait for nbd to become ready isn't really working [08:06:52] jimharris: i was thinking - can we just move the modprobe to the very beginning of our tests? [08:07:19] yeah - i'd be fine with that too [08:07:32] let me make that change now [08:09:12] that's not super urgent though [08:09:26] i'm more often seeing the nvmf failure [08:09:55] i was just looking through my makefile cleanup series and saw this issue 5 times already but not the nvmf failure [09:36:23] jimharris: Just and idea: could we convert all makefiles to use just "LIBS+=-lspdk_XXX" ? [09:37:27] the in common makefile we could just add proper "-L/path/to/spdk/libs" or not doing this if building against system installed libraries [09:39:47] anyway just an idea, no need to use it. [09:40:25] pwodkowx: yes - we could do that - you've probably seen the long makefile cleanup series, i'm planning to further simplify this to add some helper variables which list the core set of libraries that all apps link in [09:40:51] and then we could do your simplification after that [09:56:27] *** Joins: travis-ci (~travis-ci@ec2-54-144-55-204.compute-1.amazonaws.com) [09:56:28] (spdk/master) doc: update a payload example about rpc_client (WangHaiLiang) [09:56:28] Diff URL: https://github.com/spdk/spdk/compare/be0eb272d806...2f5feaf5b14c [09:56:28] *** Parts: travis-ci (~travis-ci@ec2-54-144-55-204.compute-1.amazonaws.com) () [11:09:50] *** Joins: felipef (~felipef@92.40.249.191.threembb.co.uk) [11:16:03] *** Quits: felipef (~felipef@92.40.249.191.threembb.co.uk) (Remote host closed the connection) [11:21:13] *** Joins: travis-ci (~travis-ci@ec2-54-159-157-78.compute-1.amazonaws.com) [11:21:14] (spdk/master) test: modprobe nbd at the start of autotest (Jim Harris) [11:21:14] Diff URL: https://github.com/spdk/spdk/compare/2f5feaf5b14c...50656e7a078e [11:21:14] *** Parts: travis-ci (~travis-ci@ec2-54-159-157-78.compute-1.amazonaws.com) () [11:23:22] *** Joins: travis-ci (~travis-ci@ec2-54-144-55-204.compute-1.amazonaws.com) [11:23:23] (spdk/master) bdev/qos: assert io channel when acquiring new reference (Tomasz Zawadzki) [11:23:23] Diff URL: https://github.com/spdk/spdk/compare/50656e7a078e...6177d220fb62 [11:23:23] *** Parts: travis-ci (~travis-ci@ec2-54-144-55-204.compute-1.amazonaws.com) () [11:31:16] *** Joins: travis-ci (~travis-ci@ec2-54-147-22-164.compute-1.amazonaws.com) [11:31:17] (spdk/master) bdev/crypto: code simplification (Paul Luse) [11:31:17] Diff URL: https://github.com/spdk/spdk/compare/6177d220fb62...fa4def5ceb98 [11:31:17] *** Parts: travis-ci (~travis-ci@ec2-54-147-22-164.compute-1.amazonaws.com) () [11:37:52] *** Joins: travis-ci (~travis-ci@ec2-54-144-55-204.compute-1.amazonaws.com) [11:37:53] (spdk/master) lib/trace: records num-trace-entries for lcores (Liu Xiaodong) [11:37:53] Diff URL: https://github.com/spdk/spdk/compare/fa4def5ceb98...9d2f39ab0bca [11:37:53] *** Parts: travis-ci (~travis-ci@ec2-54-144-55-204.compute-1.amazonaws.com) () [12:08:58] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Ping timeout: 246 seconds) [12:43:00] *** Joins: travis-ci (~travis-ci@ec2-54-146-247-127.compute-1.amazonaws.com) [12:43:01] (spdk/master) trace: verify tpoint_id does not execeed SPDK_TRACE_MAX_TPOINT_ID (Tomasz Zawadzki) [12:43:01] Diff URL: https://github.com/spdk/spdk/compare/9d2f39ab0bca...9e2f1bff8146 [12:43:01] *** Parts: travis-ci (~travis-ci@ec2-54-146-247-127.compute-1.amazonaws.com) () [13:39:53] jimharris: could you take a look at https://review.gerrithub.io/c/spdk/spdk/+/434317 ? [13:40:05] sure [13:40:29] oh yeah - i was just reviewing this [13:40:38] on the nvmf mr issue... [13:40:56] yeah, "io_device Nvme not unregistered" [13:41:31] is there a specific test case that demonstrates something like 16MB getting added, and then later a portion of that getting freed? [13:41:38] via the DPDK memory allocation callbacks [13:42:09] i feel like these callbacks should always free at the same granularity it was alloced [13:42:15] i don't think that's possible right now [13:42:47] but didn't we purposely change spdk to split the allocs on huge page boundaries? [13:42:51] ah wait, it is [13:44:29] that patch looks fine but made a request to improve one of the comments [13:45:00] i'm not sure how hard it would be for dpdk to make sure it frees at the same granularity [13:45:19] but i feel like malloc/free work on this principle - i can't malloc 2MB and then just free a portion of it [13:46:03] something like that: ./spdk_tgt -s 8MB; rte_malloc(16MB) -> additional 16MB gets allocated, the allocation possibly occupies the pre-reserved 8MB; rte_malloc(4MB) -> reuses the leftover 8MB from the previous allocation, rte_free(4MB) -> frees 4MB of a 16MB region [13:48:17] if i specify -s 8MB, it doesn't fail if I try to allocate more than 8MB? [13:48:32] of course not [13:48:49] that's how it used to work - before dynamic memory allocation [13:48:54] it's just an extra memory that won't be ever released to the system [13:49:03] *** Joins: lhodev (~lhodev@inet-hqmc02-o.oracle.com) [13:49:41] can we still use the old behavior, if user specifies a -s value? [13:50:18] that's an ambiguous question [13:50:26] why? to fix nvmf? [13:50:53] no - to keep the old behavior - if I specify -s 1024, then I only want this process to use 1024MB of memory [13:51:06] if I specify -s 0, then it means do dynamic memory allocation [13:51:29] we don't have a way now to restrict an application to X amount of memory [13:52:19] we do [13:52:21] let me find a link [13:53:18] https://review.gerrithub.io/c/spdk/spdk/+/432040/3/doc/system_configuration.md#38 [13:53:42] and that is creating a separate hugetlbfs mount for you application [13:53:58] *your application [13:54:00] that's not the old behavior though :) [13:54:39] ./spdk_tgt -s 1G --huge-dir /hugetlbfs/with/1GB [13:55:43] --legacy-mem additionally gives us physically contiguous memory, but we don't need that [13:56:37] if i'm understanding you correctly, the only way we can run into this case where dpdk sends us an RTE_MEM_EVENT_FREE event that doesn't match the original RTE_MEM_EVENT_ALLOC event is the case where we mix up pre-allocated and dynamically allocated memory [13:57:31] do you still have test code lying around that demonstrates this behavior by chance? [13:57:59] that's the only one I can think of right now [13:58:27] this still depends on the system configuration [14:01:41] ok - but you were able to demonstrate this problem with an empirical test? [14:01:42] nope, I don't have it in my reflog, must have stashed it at some point [14:02:46] so what would I be demonstrating exactly? [14:03:13] a test case that shows an RTE_MEM_EVENT_FREE that doesn't match the size of the associated RTE_MEM_EVENT_ALLOC [14:03:34] ah, you don't believe me? :) [14:03:39] what if we turn on --legacy-mem if the user specified a -s? [14:03:58] i believe you - i just want a test case that we can point to when we try to fix this in dpdk :) [14:04:26] *** Joins: felipef (~felipef@109.144.208.251) [14:05:02] is it important to support the case where the user specifies some amount of memory to pre-allocate, which may not be enough for the application such that it will need to allocate more? [14:05:23] k, I'll craft something [14:05:57] what if we made these two changes? [14:06:27] turning --legacy-mem on just makes you lose the dynamic allocation features [14:06:40] why would you want that? [14:06:49] 1) use --legacy-mem if the user specified a -s (maybe make this the default behavior, and add a command line option to still enable dynamic memory allocation if user specifies -s value) [14:07:03] preallocating memory may be useful when you're doing a lot of rte_malloc/rte_free [14:07:09] i mean you only turn on --legacy-mem if user specifies -s > 0 [14:08:45] since the preallocated mem is never returned to the system, I can think of it as a cache [14:09:33] you know you'll be freeing and reallocating a lot memory -> preallocate some [14:10:12] right - but i'd argue that if you're doing a lot of rte_malloc/free, that you should make sure you preallocate all that you need and don't rely on dynamic memory allocation [14:11:03] sure we can turn on --legacy-mem if the user specifies -s > 0, but why would we do that? [14:11:15] that's "legacy" mem for a reason [14:11:36] it's slow, takes forever to allocate [14:11:54] right - which is why for most cases people will just use -s 0 [14:12:44] if i'm understanding correctly, turning on --legacy-mem in this case will allow us to remove splitting an allocated region into separate MRs for each huge page [14:13:50] yup, there'll be one or more MRs - one for each physically contiguous memory chunk [14:14:11] and with `./spdk_tgt -s 1G --huge-dir /hugetlbfs/with/1GB` there'll be always exactly 1 MR [14:14:22] if you have 1GB hugepages [14:14:28] with 2MB too [14:14:57] the preallocated memory is never released to the system, so we register it as a one big MR [14:15:03] there's no risk of freeing just a part of it [14:16:04] hmmm - i see, memory_iter_cb is what gets invoked on preallocated memory? [14:16:18] looking [14:17:23] it still means that if we are instructing users to just use -s 0, we'll get one MR per hugepage [14:17:24] yup, that gets invoked on each VA-contiguous area [14:17:40] I think I documented that somewhere [14:17:43] nvmf guide maybe? [14:18:24] here https://spdk.io/doc/nvmf.html#nvmf_rdma_limitations [14:20:20] i still think dpdk should give us FREE notifications that match the ALLOC notifications - then all of these problems go away [14:21:46] but we still need proper MR splitting in nvmf [14:24:43] But only on the initiator side right? Since on the target side all of the buffers are allocated out of an spdk_mempool. [14:24:58] we need multiple SGE support in NVMF - but not sure we need MR splitting except to workaround this issue [14:26:15] we shouldn't need MR splitting on the initiator side either [14:37:57] fionatrahe, you around? [15:03:32] jimharris: https://review.gerrithub.io/c/spdk/spdk/+/434889 [15:03:48] that's without preallocated memory though [15:03:55] I found it in my git stash [15:04:34] sweet - i'll check it out [15:05:36] allocating a hotplugs 10MB of memory (8MB [15:05:46] 8MB + rte_malloc header) [15:06:10] then freeing it removes just 6MB of memory [15:06:38] the remaining 4 is kept allocated until `y` is freed [15:19:32] jimharris: since dpdk 18.08 there is an additional cmdline option --socket-limit [15:20:18] limits the amount of hugepages you can allocate [15:22:29] we could temporarily sneak it into dpdk args if user specifies -s > 0 [16:01:49] *** Quits: lhodev (~lhodev@inet-hqmc02-o.oracle.com) (Remote host closed the connection) [16:02:27] *** Joins: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) [16:42:52] For tomorrows community call do not use the WebEx info on the webpage, we're going to trial the new service again and officially switch once we've tried it in both geos. See email I just to dist list for more info [16:43:11] Here is the conf details for tomorrow "Tue Nov 27 Euro Call: [16:43:11] Access code: 652734# [16:43:11] Online meeting ID: paul_e_luse [16:43:11] Join the online meeting: https://join.freeconferencecall.com/paul_e_luse [16:43:11] " [16:43:23] (same code as last time for anyone keeping track) [16:48:46] darsto: what if we had a way to tell dpdk to *not* free memory that was dynamically allocated? [16:49:23] i'll have a patch for you shortly [16:57:17] *** Quits: felipef (~felipef@109.144.208.251) (Remote host closed the connection) [17:28:08] fionatrahe, jimharris - able to do my 2 operation test (bdevio doing 1 write then 1 read) and confirm compression and decompression through the DPDK API are functional. Thanks Fiona!! [17:28:30] this was just a super basic sanity check before moving onto reduce integration, but really cool to see it actually work [17:29:39] and, BTW, fionatrahe that 4K block of 0xa3 doesn't compress to 8 bytes (knew that seemed too good), it actually compresses to 28 :) [18:32:04] *** drv_ is now known as drv [18:39:21] *** Joins: felipef (~felipef@109.144.208.131) [19:22:17] *** Joins: ziyeyang_ (~ziyeyang@192.55.46.36) [20:23:09] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Quit: Textual IRC Client: www.textualapp.com) [20:26:16] *** Joins: lhodev (~lhodev@inet-hqmc04-o.oracle.com) [20:29:22] How far back (or how long) are changes viewable on gerrit after they're merged ? [20:33:47] I attempted to search for a specific commit (dated from Jan 2017) on gerrit, but my search failed. [20:36:39] I then did a search by the author and it appeared the first commit I can find for that individual on gerrit currently shows a date of Feb 2017 [20:38:41] *** Quits: lhodev (~lhodev@inet-hqmc04-o.oracle.com) (Remote host closed the connection) [20:40:57] *** Joins: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) [20:58:52] hi lhodev: I don't think we were using gerrithub yet in January 2017 [22:11:02] @lhodev, previously we used patch work and some internal tools...