[00:41:10] *** Joins: ziyeyang__ (~ziyeyang@192.55.54.42) [00:41:10] *** Quits: ziyeyang_ (~ziyeyang@134.134.139.74) (Remote host closed the connection) [00:56:04] *** Quits: ziyeyang__ (~ziyeyang@192.55.54.42) (Quit: Leaving) [00:59:04] *** Joins: tkulasek (~tkulasek@192.55.54.42) [03:53:09] *** Quits: tomzawadzki (tomzawadzk@nat/intel/x-ksqrzwqvfagovfuu) (Ping timeout: 264 seconds) [03:54:24] *** Quits: dlw (~Thunderbi@114.255.44.143) (Quit: dlw) [05:55:27] *** Joins: dlw (~Thunderbi@222.129.236.83) [06:44:21] *** Quits: dlw (~Thunderbi@222.129.236.83) (Ping timeout: 256 seconds) [07:27:51] *** Joins: travis-ci (~travis-ci@ec2-54-226-151-214.compute-1.amazonaws.com) [07:27:52] (spdk/master) iscsi: Remove all uses of strncpy (Ben Walker) [07:27:52] Diff URL: https://github.com/spdk/spdk/compare/7552707ef16a...c0d6eb9d1140 [07:27:52] *** Parts: travis-ci (~travis-ci@ec2-54-226-151-214.compute-1.amazonaws.com) () [09:06:29] *** Quits: tkulasek (~tkulasek@192.55.54.42) (Ping timeout: 260 seconds) [09:13:43] *** Joins: travis-ci (~travis-ci@ec2-54-226-151-214.compute-1.amazonaws.com) [09:13:44] (spdk/master) include/net.h: remove non public functions to private header file (Yanbo Zhou) [09:13:44] Diff URL: https://github.com/spdk/spdk/compare/c0d6eb9d1140...c1497365964f [09:13:44] *** Parts: travis-ci (~travis-ci@ec2-54-226-151-214.compute-1.amazonaws.com) () [09:23:26] someone is going to need to sit down and spend a bunch of time with this at some point: https://github.com/anatolyburakov/dpdk [09:23:39] it's all the dynamic memory changes going into DPDK [09:33:12] i'd be up for it [09:34:40] last time i checked these patches didn't really work for anything but some unit tests [09:34:51] that was a ~month ago though [09:34:55] I think they're really gearing up now [09:35:04] people are fixing bugs in drivers and such to accomodate [09:37:27] I was getting random segfaults in rte_malloc_* at the time, I guess I'll try again now [10:18:42] *** Quits: sethhowe (~sethhowe@134.134.139.76) (Remote host closed the connection) [10:20:26] *** Joins: sethhowe (~sethhowe@134.134.139.76) [11:43:39] *** Joins: param (~param@157.49.107.243) [11:50:14] ben : jim : drv : I have pushed the code https://review.gerrithub.io/405489 transport_listen & transport_stop_listen during RPC [13:01:48] darsto: that DPDK dynamic memory patch series was just merged [13:01:59] so it will definitely be in 18.05 [13:02:34] we'll see how far we can go with that - hopefully fully dynamic allocation of hugepages [13:02:54] and eventually, if an IOMMU is present and enabled, it may allow us to not even use hugepages at all [13:03:12] or at least, transparent hugepages if we still want better performance [13:06:10] *** Joins: travis-ci (~travis-ci@ec2-54-166-6-76.compute-1.amazonaws.com) [13:06:11] (spdk/master) blobfs: sync length if append occurs after cache eviction (Jim Harris) [13:06:11] Diff URL: https://github.com/spdk/spdk/compare/c1497365964f...1831b086d000 [13:06:11] *** Parts: travis-ci (~travis-ci@ec2-54-166-6-76.compute-1.amazonaws.com) () [13:10:45] *** Joins: param_ (~param@157.49.107.243) [13:14:32] *** Quits: param (~param@157.49.107.243) (Ping timeout: 276 seconds) [13:14:40] *** param_ is now known as param [13:19:09] *** Quits: param (~param@157.49.107.243) (Read error: Connection reset by peer) [14:35:26] bwalker: could you please expand on that? My understanding is that hugepages provided two (2) things: 1) the obvious performance benefit by reducing page faults and, 2) that hugepages are always pinned thus serving the need for physical memory for DMA transfers. How does employment of an IOMMU obviate those? [14:36:29] when the IOMMU is enabled, you can program I/O virtual addresses (iova) into the DMA engine hardware on the device [14:36:53] and those messages are intercepted on their way from the PCI device to main memory by the IOMMU, which translates them to physical memory addresses (pa) [14:37:29] the IOMMU is "in" on the memory management occurring in the MMU, so it's aware of page table updates and movement [14:37:56] it actually stores a mapping of iova -> va -> pa [14:38:10] so that if the mapping of va -> pa is modified, the iova mapping is still going to the right spot [14:38:23] so memory no longer has to be pinned to a single physical address [14:39:13] I wrote up a bunch of stuff on this here: http://www.spdk.io/doc/memory.html [14:39:16] IOMMU section at the end [14:41:02] bwalker, so I grok the concept of the I/O virtual address abstraction, but I'm unclear about the related physical one. I would imagine that you'd want to ensure the translations result in those addresses backed by a physical page; i.e. one would not want to incur a fault requiring a need to grab some physical memory when the DMA is initiated. I'll go look at the link you provided to see if that helps me understand this better. [14:42:20] it doesn't protect against memory getting paged out [14:42:30] we mostly assume that swap is disabled [14:42:37] but we could also call mlock to stop swapping [14:42:52] the more devious problem is that pages can get moved behind the scenes [14:43:11] so that a va ends up being remapped to a different pa at some point [14:43:41] this can happen for a lot of reasons - NUMA rebalancing is a big one, as is ECC error detection and recovery [14:44:11] so if we have a va to pa mapping in our process address space, that can be invalidated and we aren't notified [14:44:20] so we end up programming an incorrect pa into the device to start a DMA transfer [14:44:35] Right, got that part. [14:44:47] so hugepages, at least so far, never get moved [14:44:58] there just isn't code to do that in linux, except as explicit request [14:45:04] so we were safe [14:45:22] but with an IOMMU, we just program it to say I want my iova to be X and it points at va Y [14:45:31] and if va Y gets remapped later to a different pa, it just works out [14:45:46] the IOMMU knows about the mappings - it's part of the MMU [14:46:35] But, if one does not employ an mlock, is there a possibility that there is no physical page available at that instant in which case an exception is taken to provide one during the first DMA xfer? And, if so, it can be done quickly enough such that the endpoint's DMA engine does not timeout/fail? [14:47:21] if there is no physical page, I'm not sure if the IOMMU can properly generate a page fault [14:47:34] I doubt it, but I can ask the hardware guys [14:48:08] my best guess would be that the DMA transfer fails, which then fails our NVMe command [14:48:24] I sure would appreciate you passing on such details! Very eager to better understand exactly how that operates. [14:49:30] I'll see if I can find someone that knows the answer [14:50:40] Mucho gracias ! [15:52:23] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [18:23:11] *** Joins: dlw (~Thunderbi@114.255.44.143) [20:42:14] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 260 seconds) [20:56:14] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [20:59:26] Hi All, I'll join soon. [21:00:32] Network is not stable now, so I'll need a couple of minutes, sorry for inconvenience. [21:02:52] ok shuhei - no problem [21:14:49] *** Joins: travis-ci (~travis-ci@ec2-54-226-151-214.compute-1.amazonaws.com) [21:14:50] (spdk/master) jsonrpc: convert send queue to singly linked tail queue (Pawel Wodkowski) [21:14:50] Diff URL: https://github.com/spdk/spdk/compare/ffb4d54a79b1...b066126b0b9a [21:14:50] *** Parts: travis-ci (~travis-ci@ec2-54-226-151-214.compute-1.amazonaws.com) () [21:23:17] *** Joins: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) [21:23:17] *** ChanServ sets mode: +o bwalker_ [21:23:25] does anyone have admin on this webex? [21:53:59] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 260 seconds) [21:55:59] *** Quits: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) (Quit: Leaving)