[00:43:42] *** Joins: darsto_ (~darsto@89-68-126-89.dynamic.chello.pl) [00:44:40] *** Quits: darsto (~darsto@89-68-126-89.dynamic.chello.pl) (Ping timeout: 244 seconds) [00:44:40] *** darsto_ is now known as darsto [01:14:55] *** Joins: alekseymmm_ (050811aa@gateway/web/freenode/ip.5.8.17.170) [01:18:27] *** Quits: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) (Ping timeout: 252 seconds) [01:55:15] *** Quits: alekseymmm_ (050811aa@gateway/web/freenode/ip.5.8.17.170) (Quit: Page closed) [01:55:45] *** Joins: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) [02:52:44] Hello everyone. Do you have FC support in SPDK? [04:11:51] *** Quits: dlw (~Thunderbi@114.255.44.143) (Ping timeout: 268 seconds) [05:51:11] *** Joins: lyan (~lyan@2605:a000:160e:2dd:4a4d:7eff:fef2:eea3) [05:51:40] *** lyan is now known as Guest27464 [07:45:02] *** Joins: lyan (~lyan@2605:a000:160e:2dd:4a4d:7eff:fef2:eea3) [07:45:25] *** lyan is now known as Guest4506 [07:46:10] *** Quits: Guest27464 (~lyan@2605:a000:160e:2dd:4a4d:7eff:fef2:eea3) (Read error: Connection reset by peer) [08:02:44] peluse: gerrithub needs to be synced with github [08:04:27] github spdk-18.05 branch is updated now: https://github.com/spdk/dpdk/commits/spdk-18.05 [08:14:58] alekseymmm: darsto abandoned that patch since it was solely for debugging purposes and not suitable for upstream - but there's no reason we couldn't add an fio config option to the bdev fio plugin to specify debug log flag [08:17:11] Good morning guys. RDMA-related question. What happens when a RDMA application dies without deregistering memory regions (i.e. no calls to ibv_dereg_mr) ? I've google'd this and come up empty-handed. [08:17:26] and there is no FC support today, although there is work being started in this area (see include/spdk/nvmf_fc_spec.h and lib/nvmf/nvmf_fc.h) [08:18:15] i'm honestly not sure what to do with fio_plugin [08:18:27] we have a similar problem with other example applications - bdevio/bdevperf [08:19:25] they don't parse some options that we will need sooner or later [08:20:19] we can't just use the standrard app param parser in them, because there'd be conflicts [08:21:03] but also we're running out of available characters for command line parameters, so it's inevitable to implement full-name params [08:21:19] so one cheap way is to support those full-name params in bdevio/bdevperf [08:21:39] but it still doesn't cover fio_plugin, which is just a plugin and not a separate app [08:37:46] lhodev: That's a great question. I don't know for sure, but I think really bad things. The docs indicate that an application should not free memory associated with an MR until they have called ibv_dereg_mr. If your application exits without deregistering that memory, It effectively frees it for use by another application without notifying the NIC. I definitely see potential for memory corruption there. But would need to do more research. [08:41:38] sethhowe: Right. Generally when user processes die, most of their resources are reaped by the OS, but I've yet to locate precisely what happens in this failure to dereg case as it, as you know, impacts resources that an adapter manages. I'm wonder if this is a form of memory leak. I've tried to find diagnostic commands that I might be able to issue to the adapter to dump or list memory regions, request to purge them, etc., but [08:41:38] to no avail. [08:45:14] sethhowe: related to this topic, I'm working with a team now where they're finding *every* time, currently, that they launch nvmf_tgt, they see ibv_mr_reg() failures (-ENOSPC). Initially, though, this was not occurring. I'm wondering if after terminating nvmf_tgt by various signals, that perhaps a lack of dereg's has led to this. Pure speculation on my part. [08:50:33] lhodev: Thank you for this information. I am actually doing a lot more digging into NVMe-oF and DMA code right now. Especially the shutdown path. All of this information is really useful. [08:55:10] sethhowe: I've also spent a little time looking through the signal handling by the reactor framework and nvmf_tgt. I've yet to find that upon a signal, all memory regions will be dereg'd as part of the shutdown/cleanup. I hasten to add that it *might* be there and I just haven't spotted it yet. It's some complex code together with the ability of an application to supply a cleanup handler too. That aside, though, if a process [08:55:10] dies abruptly due to a bug, then it still leaves the question: what happens with a lack of explicit dereg calls? [08:56:52] *** Quits: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) (Quit: Page closed) [09:01:36] *** Joins: peter_turschm (~peter_tur@2604:4080:1133:0:c8e:9eda:e39f:5c77) [10:11:06] *** Quits: peter_turschm (~peter_tur@2604:4080:1133:0:c8e:9eda:e39f:5c77) (Remote host closed the connection) [10:11:19] *** Joins: peter_turschm (~peter_tur@2604:4080:1133:0:c8e:9eda:e39f:5c77) [10:11:52] *** Quits: peter_turschm (~peter_tur@2604:4080:1133:0:c8e:9eda:e39f:5c77) (Remote host closed the connection) [10:48:58] *** Joins: peter_turschm (~peter_tur@2604:4080:1133:0:c8e:9eda:e39f:5c77) [10:53:30] *** Quits: peter_turschm (~peter_tur@2604:4080:1133:0:c8e:9eda:e39f:5c77) (Ping timeout: 260 seconds) [11:06:11] *** Joins: peter_turschm (~peter_tur@66.193.132.66) [11:23:24] *** Joins: darsto_ (~darsto@89-68-126-89.dynamic.chello.pl) [11:24:56] *** Quits: darsto (~darsto@89-68-126-89.dynamic.chello.pl) (Ping timeout: 265 seconds) [11:24:56] *** darsto_ is now known as darsto [11:42:16] jimharris: https://review.gerrithub.io/c/spdk/spdk/+/420758 [13:29:34] setthowe: I've got a little more info on the cleanup of RDMA-related objects upon process exit if you're interested. [13:29:58] sethhowe: I've got a little more info on the cleanup of RDMA-related objects upon process exit if you're interested. [13:31:55] I'm interested [13:38:10] lhodev: yeah, absolutely. [13:41:29] sethhowe: under the covers within the rdma userspace code, a file descriptor is opened to /dev/infiniband/uverbs* and the memory registrations are associated with that descriptor. As such, when the process terminates -- even dies hard -- then a close() is done on the descriptor which results in some kernel code getting invoked -- depending on your kernel version -- resulting in the automatic cleanup of those objects. [13:43:07] sethhowe: by "depending on your kernel version", I was referring to the point that the exact name of the cleanup function has changed over time so that will depend on your kernel. [13:45:15] lhodev: Oh that's good to know. Just to clarify what you said then, that kernel code informs your device that those memory regions are no longer valid taking care of any potential memory leaks / corruption? [13:46:43] sethhowe: Correct. That was a big relief for me to learn as I felt that was leaving an egregiously bad opportunity to leak critical resources. [13:52:33] Definitely. Thank you for sharing that. [14:39:23] *** Quits: Guest4506 (~lyan@2605:a000:160e:2dd:4a4d:7eff:fef2:eea3) (Quit: Leaving) [15:10:37] *** Joins: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) [15:12:23] Hello everyone. Have a question. In _spdk_bdev_io_complete you have assert(spdk_get_thread() == spdk_io_channel_get_thread(bdev_io->internal.ch->channel)); . But spdk_get_thread acquire pthread_mutex_lock. Does it affect performance, acquiring mutex lock for every end io ? [15:15:12] asserts don't compile in release builds [15:15:53] ok got it [15:15:58] thaxnks [15:16:00] np [15:44:27] *** Quits: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) (Quit: Page closed) [17:21:41] *** Quits: peter_turschm (~peter_tur@66.193.132.66) (Remote host closed the connection) [18:44:16] *** Joins: dlw (~Thunderbi@114.255.44.143) [21:05:58] *** Quits: kjakimia (kjakimia@nat/intel/x-wnivetlcilimajxj) (Quit: ZNC - http://znc.in) [21:05:58] *** Quits: ziyeyang (ziyeyang@nat/intel/x-bsjsvdwajzwyhdfp) (Quit: ZNC - http://znc.in) [21:06:18] *** Quits: jkkariu (jkkariu@nat/intel/x-frharzkpigcqiliq) (Quit: ZNC 1.7.0 - https://znc.in) [21:06:55] *** Quits: peluse (~peluse@134.134.139.72) (Quit: ZNC - http://znc.in) [21:07:42] *** Joins: ziyeyang (ziyeyang@nat/intel/x-zmcjhfjdtzsafgbo) [21:10:14] *** Joins: peluse (~peluse@134.134.139.72) [21:11:14] *** Joins: kjakimia (kjakimia@nat/intel/x-ptuhgynfrsyhaxgn) [21:12:50] *** Joins: jkkariu (jkkariu@nat/intel/x-wukkbesdwijofanf) [23:13:21] *** Joins: lhodev_ (~lhodev@66-90-218-190.dyn.grandenetworks.net) [23:14:56] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Ping timeout: 276 seconds) [23:27:53] *** Joins: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) [23:30:38] *** Quits: lhodev_ (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Ping timeout: 276 seconds) [23:31:13] *** Joins: darsto_ (~darsto@89-68-126-89.dynamic.chello.pl) [23:32:21] *** Quits: darsto (~darsto@89-68-126-89.dynamic.chello.pl) (Ping timeout: 268 seconds) [23:32:22] *** darsto_ is now known as darsto [23:57:05] I create small fix for memory leaks: https://review.gerrithub.io/c/spdk/spdk/+/420845 [23:57:17] it would be good to have it in release, so please review :) [23:57:49] *** Joins: tomzawadzki (tomzawadzk@nat/intel/x-kqhqmzfbjafbexjz)