[01:59:20] *** Joins: travis-ci (~travis-ci@ec2-34-238-137-18.compute-1.amazonaws.com) [01:59:21] (spdk/master) lvol: fix Null-checking after dereferenced (Jie Wang) [01:59:21] Diff URL: https://github.com/spdk/spdk/compare/1c54ba1c174b...e24da0913cee [01:59:21] *** Parts: travis-ci (~travis-ci@ec2-34-238-137-18.compute-1.amazonaws.com) () [02:02:14] *** Joins: travis-ci (~travis-ci@ec2-184-73-70-169.compute-1.amazonaws.com) [02:02:15] (spdk/master) lib: Fix spelling error in vhost_nvme.c (Arshad Hussain) [02:02:15] Diff URL: https://github.com/spdk/spdk/compare/e24da0913cee...4cec12f20885 [02:02:15] *** Parts: travis-ci (~travis-ci@ec2-184-73-70-169.compute-1.amazonaws.com) () [03:52:49] *** Joins: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) [04:06:06] Hello everyone. I am trying to play with my null bdev comparing with nullbdev in spdk. Have some question. If i finish io (spdk_bdev_io_complete) immediately in submit_request function but not in poller do I see any performance degradation? [05:15:26] alekseymmm: if you complete an I/O immediately, the internal bdev code will still defer that completion by sending an spdk event to the current thread [05:16:22] polling all I/Os from a single poller is way more efficient than using a separate spdk_event for each single I/O [05:23:59] To sum up. It is faster to have poller per io channel and complete in poll function than immediately in make request. Am I correct? [05:42:20] correct [05:42:32] sending an spdk event has a constant overhead [05:43:04] you don't have that overhead when completing all I/Os from a poller [05:59:45] thank you [06:16:22] *** Quits: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) (Quit: Page closed) [06:46:06] *** Joins: ziyeyang_ (~ziyeyang@192.102.204.38) [07:07:20] *** Joins: travis-ci (~travis-ci@ec2-34-238-137-18.compute-1.amazonaws.com) [07:07:21] (spdk/master) test/env_dpdk_post_init: use only a single core (Darek Stojaczyk) [07:07:21] Diff URL: https://github.com/spdk/spdk/compare/4cec12f20885...0f068a6b6568 [07:07:21] *** Parts: travis-ci (~travis-ci@ec2-34-238-137-18.compute-1.amazonaws.com) () [07:09:23] *** Quits: ziyeyang_ (~ziyeyang@192.102.204.38) (Quit: Leaving) [07:10:34] *** Joins: travis-ci (~travis-ci@ec2-35-171-28-152.compute-1.amazonaws.com) [07:10:35] (spdk/master) doc/lvol: add json rpc description for set_read_only_lvol_bdev() RPC (Tomasz Zawadzki) [07:10:35] Diff URL: https://github.com/spdk/spdk/compare/0f068a6b6568...27290f500dd3 [07:10:35] *** Parts: travis-ci (~travis-ci@ec2-35-171-28-152.compute-1.amazonaws.com) () [07:10:44] *** Joins: travis-ci (~travis-ci@ec2-54-159-202-0.compute-1.amazonaws.com) [07:10:45] (spdk/master) bdevperf: add check functions for input parsing (Chunyang Hui) [07:10:45] Diff URL: https://github.com/spdk/spdk/compare/27290f500dd3...35cdee26ad0d [07:10:45] *** Parts: travis-ci (~travis-ci@ec2-54-159-202-0.compute-1.amazonaws.com) () [07:41:51] *** Joins: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) [07:42:56] hello, I have not been working with spdk for awhile. And recently returned to work. Noticed alot of changes. One of them is traces. Could I turn the ,on and off? to see the impact on performance [08:49:26] klateck, you there? [08:50:08] Hey Paul, Im here [08:50:20] whats up? [08:51:09] cool, quick question on your interpretation of the current Jenkins load. Go take a quick look, seems like we have a bunch waiting on an exec and many test nodes idle, I'm just not sure if all of those test nodes are actively involved with auto-test-per-patch though. Wondering is we should bump execs again? [08:52:33] Not all of them are involved in a-p-p. And some of these involved which are idle have already finished their portion of the tests [08:52:55] I wouldn't increase the execs though - take a look at build history [08:53:03] K, one sec [08:53:44] what specifically are you pointing out? [08:54:02] Well, today it's not most fortunate to back up my point of view, but in cases where there are many consecutive passes then single test run may take up to 50 mins [08:54:22] because none of the builds running on the execuors would exit early [08:54:35] so they have to wait for nodes to become available [08:54:38] what do you mean by exit early? why? [08:54:51] Unit early - for example on unit tests phase [08:54:52] none, sorry thought I read "some" [08:54:54] Exit* [08:56:10] and it appears that each autotest job schedules the subjobs in the same order each time right? I wonder if there was a way to randomize or let Jenkins skip a subjob if a resources is busy or something [08:57:12] but I do see your point on # of execs now, you're saying the bottleneck is in the # of available test nodes; ie the current executing jobs are all competing for the same nodes right? [08:57:14] First question - you mean the spawned subjob builds? No idea. I think the order is random. [08:57:30] random would be good, yeah that's what I mean [08:59:47] As for skipping - Jenkins runs what is available at the moment, so any subjubs will run on available nodes. [08:59:59] But yeah, we have a bottleneck o physical systems I think [09:00:02] on* [09:01:42] Sorry, got to go. Hear you later! [09:02:03] alekseymmm, sorry, jumped in and scrolled a bunch of text before anyone got to your question. I thought someone just recently updated some docs or comments but the person I'm thinking about is on sabbatical now. If nobody jumps in with an answer I'll help dig into with you (I haven't used our tracing) [09:28:09] *** Joins: travis-ci (~travis-ci@ec2-54-196-189-124.compute-1.amazonaws.com) [09:28:10] (spdk/master) nvme_perf: Remove unnecessary zeroing the buffer by memset (Shuhei Matsumoto) [09:28:10] Diff URL: https://github.com/spdk/spdk/compare/35cdee26ad0d...307631e10a75 [09:28:10] *** Parts: travis-ci (~travis-ci@ec2-54-196-189-124.compute-1.amazonaws.com) () [10:11:18] *** Joins: travis-ci (~travis-ci@ec2-54-224-205-35.compute-1.amazonaws.com) [10:11:19] (spdk/master) histograms: add unit tests for public bdev api (Piotr Pelplinski) [10:11:20] Diff URL: https://github.com/spdk/spdk/compare/307631e10a75...42dba6047b05 [10:11:20] *** Parts: travis-ci (~travis-ci@ec2-54-224-205-35.compute-1.amazonaws.com) () [10:33:12] *** Joins: travis-ci (~travis-ci@ec2-54-224-205-35.compute-1.amazonaws.com) [10:33:13] (spdk/master) UT: fix the sock_ut failure because of the port conflict (tone.zhang) [10:33:14] Diff URL: https://github.com/spdk/spdk/compare/42dba6047b05...88ddf0f22e6e [10:33:14] *** Parts: travis-ci (~travis-ci@ec2-54-224-205-35.compute-1.amazonaws.com) () [10:46:54] *** Quits: tomzawadzki (uid327004@gateway/web/irccloud.com/x-vbdxmkcnnwdnayyu) (Quit: Connection closed for inactivity) [12:25:39] *** Joins: travis-ci (~travis-ci@ec2-54-144-117-120.compute-1.amazonaws.com) [12:25:40] (spdk/master) json_config: verify jsonrpc client request allocation (Tomasz Zawadzki) [12:25:40] Diff URL: https://github.com/spdk/spdk/compare/88ddf0f22e6e...a96748a78212 [12:25:40] *** Parts: travis-ci (~travis-ci@ec2-54-144-117-120.compute-1.amazonaws.com) () [12:26:09] *** Joins: travis-ci (~travis-ci@ec2-54-80-118-170.compute-1.amazonaws.com) [12:26:10] (spdk/master) json_config: dont dereference when no next config entry (Tomasz Zawadzki) [12:26:10] Diff URL: https://github.com/spdk/spdk/compare/a96748a78212...01455bb15bd6 [12:26:10] *** Parts: travis-ci (~travis-ci@ec2-54-80-118-170.compute-1.amazonaws.com) () [12:50:06] *** Joins: travis-ci (~travis-ci@ec2-34-238-137-18.compute-1.amazonaws.com) [12:50:07] (spdk/master) lvol: make unique_id an array (Pawel Wodkowski) [12:50:07] Diff URL: https://github.com/spdk/spdk/compare/01455bb15bd6...1da6e2f5c50d [12:50:07] *** Parts: travis-ci (~travis-ci@ec2-34-238-137-18.compute-1.amazonaws.com) () [14:08:30] *** Joins: alekseymmm_ (050811aa@gateway/web/freenode/ip.5.8.17.170) [14:09:52] Hello. small question. I work on my bdev in spdk and allocate some structures on datapath, like request that wraps bdev_io, what is the best way to allocate it spdk_mempool_get or spdk_dma_zmalloc? [14:10:55] spdk_mempool_get is faster - but in each bdev_io is a scratch space that you can request [14:11:05] so you could put your data structure in there for free [14:11:49] ok, IO [14:12:02] you control the size by implementing the "get_ctx" size function pointer in your bdev module. [14:12:06] I think i need some allocations though for different reasons [14:12:11] sorry get_ctx_size() [14:12:28] you'll want to use a mempool for performance [14:13:25] were there any recent changes in mempools? since i am testing my old code and the performance is really different compare to couple month ago... [14:19:13] I don't think so. I know there can be some major differences based on whether you are calling the mempool functions from a thread that DPDK spawned vs. one that you spawned [14:19:22] but I am not up to speed on the exact status of that [14:20:14] it seems very like to what I see. [14:20:23] what about fio threads? I use fio for perf tests [14:21:01] yeah the fio threads are not DPDK threads [14:21:22] jimharris might remember better than me whether DPDK made some changes to fix this scenario [14:21:40] we definitely made some changes in the bdev layer but I think that was quite awhile ago [14:21:51] And couple months ago there were no such difference? since I used fio at that time [14:22:01] are you seeing it as much faster now? [14:22:14] faster in the pst [14:22:25] oh - well nothing should have happened to make it slower [14:22:34] hm [14:22:44] may be I am following the wrong path... [14:22:48] mempool are only "fast" if you can use the per-thread caches - otherwise it grabs a global lock on each alloc/free to touch the global pool [14:23:25] DPDK doesn't have any 'fixes' for this - you can hack fio to set a dpdk lcore_id in each of the fio threads, but that's really ugly [14:23:58] for the bdev layer, we ended up going our own per-thread caching - and just use the DPDK mempool without per-thread caches for the threads to get buffers above their cache limit [14:23:58] How do they (mempools) know whether they are used per thread or not ? [14:24:18] there's a thread-local lcore_id on each thread [14:24:34] if the lcore_id isn't set, DPDK reverts to using the global pool including the locks [14:26:40] do I have to use mempools without caching? [14:26:58] if you're calling the mempool functions from a thread that DPDK didn't spawn, yes [14:27:16] since previously It was with SPDK_MEMPOOL_DEFAULT_CACHE_SIZE as argument on mempool create [14:27:36] you can allocate caches for the mempool, DPDK just isn't going to use them from a non-DPDK thread [14:27:55] is it recent changes ? my last tests were around october and I didn't see any problems with mempool like right now [14:28:06] no this is the way it has always been [14:30:13] sad... Since the performance of my bdev from October is much better then with the latest spdk and I have no idea where to investigate. So many changes [14:30:44] *than [14:32:49] are you running on top of a null bdev? [14:34:08] my bdev is like pasthru (with some changes and allocations and calculations), but yes I test it on top null [14:36:05] can you just do a run in like perftop and see what's different between the two? [14:36:12] On my past test when I use fio for random read test and increse numjobs the total IOPS significantly increased with the number of jobs, With the latest code it is opposite: more numjobs the less IOPS [14:36:28] sounds like some kind of lock contention [14:36:34] ok, in a moment [14:40:57] with the current vversion of spdk perf top says [14:41:10] 93.55% fio_plugin [.] common_ring_mc_dequeue [14:41:28] on top of what i see in perf top [14:41:37] on the old code there is no such thing [14:42:26] I have seen some changes in fio plagin though (in email from github) but has no chance to llok at them [14:43:38] to be more precise. On the old code I see [14:43:40] 2.93% fio_plugin [.] common_ring_mc_dequeue [14:44:50] were there any changes in ring dequeue mechanics in dpdp? I believe that release of dpdk itself changed a lot [14:49:47] Probably my results are somehow related to the changes of how fio use dpdk threads [15:03:41] *** Quits: alekseymmm_ (050811aa@gateway/web/freenode/ip.5.8.17.170) (Ping timeout: 256 seconds) [15:06:32] *** Joins: alekseymmm_ (050811aa@gateway/web/freenode/ip.5.8.17.170) [15:06:41] Any thoughts? [15:29:56] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [15:40:19] I failed to understand the changes in fio_plugin that leads to my observed results. Hope to ask @bwalker latter, it is time for me to sleep. but I believe all my problems are due to patches from 1,2 Nov 2018 about fio_plugin [15:42:12] I'll have to take a look - did you narrow it down to between the 1st and 2nd of Nov. for sure? [15:42:25] the DPDK mempools are implemented on top of rings btw [15:46:16] not sure. I am looking at all the cahnges in fio_plugin since that time [15:46:38] do those perf tops clarify something? [15:46:46] well the problem is a ring dequeue [15:46:50] that's pretty clear [15:47:04] that ring dequeue could be part of a buffer pool, because that's how they're implemented internally [15:47:11] so would be nice to know if that was the case [15:47:34] and beyond that, we have to figure out what changed [15:49:08] I think most of the changes in plugin was in december. I think I have to test different commits [15:54:36] "ring dequeue could be part of a buffer pool" what do you mean by that ? [15:54:50] internally, DPDK implements the buffer pool as a ring [15:55:03] so that common_ring_mc_dequeue call could be inside of a buffer pool [15:55:31] i.e. it could be part of an spdk_mempool_get [15:59:12] ok, thanks, need to dig a bit there. may be test some smaller examples. [16:24:51] *** Quits: alekseymmm_ (050811aa@gateway/web/freenode/ip.5.8.17.170) (Quit: Page closed) [16:26:56] *** Joins: travis-ci (~travis-ci@ec2-54-144-117-120.compute-1.amazonaws.com) [16:26:57] (spdk/master) QoS/Bdev: add RPC support for Read/Write separate bandwidth limits (GangCao) [16:26:57] Diff URL: https://github.com/spdk/spdk/compare/1da6e2f5c50d...05b43152b2d2 [16:26:57] *** Parts: travis-ci (~travis-ci@ec2-54-144-117-120.compute-1.amazonaws.com) () [16:27:08] *** Joins: travis-ci (~travis-ci@ec2-35-171-28-152.compute-1.amazonaws.com) [16:27:09] (spdk/master) ftl: Added unit tests for FTL library (Wojciech Malikowski) [16:27:09] Diff URL: https://github.com/spdk/spdk/compare/05b43152b2d2...70b86ec995c2 [16:27:09] *** Parts: travis-ci (~travis-ci@ec2-35-171-28-152.compute-1.amazonaws.com) () [16:29:26] *** Joins: travis-ci (~travis-ci@ec2-34-238-137-18.compute-1.amazonaws.com) [16:29:27] (spdk/master) vhost: move around some struct definitions (Darek Stojaczyk) [16:29:27] Diff URL: https://github.com/spdk/spdk/compare/70b86ec995c2...d171896a40fb [16:29:27] *** Parts: travis-ci (~travis-ci@ec2-34-238-137-18.compute-1.amazonaws.com) () [18:48:04] *** Joins: ziyeyang_ (ziyeyang@nat/intel/x-xlybwrsiangtpzrf) [18:48:32] Hi Jim and Paul, could you review the patch again [18:50:15] *** Joins: travis-ci (~travis-ci@ec2-54-159-202-0.compute-1.amazonaws.com) [18:50:16] (spdk/master) test/nvmf: fix the error message in nvmf_lvol test (Ziye Yang) [18:50:17] Diff URL: https://github.com/spdk/spdk/compare/d171896a40fb...0652c0a6c4a4 [18:50:17] *** Parts: travis-ci (~travis-ci@ec2-54-159-202-0.compute-1.amazonaws.com) () [18:54:24] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Ping timeout: 246 seconds) [20:08:33] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds) [20:20:07] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [20:33:29] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds) [20:45:01] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97)