[01:53:49] *** Joins: tkulasek (~tkulasek@192.55.54.45) [03:45:57] *** Quits: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) (*.net *.split) [04:12:03] *** Quits: tomzawadzki (uid327004@gateway/web/irccloud.com/x-mqpzbqxnkniaxvqn) (Quit: Connection closed for inactivity) [04:43:43] *** Quits: tkulasek (~tkulasek@192.55.54.45) (Quit: Leaving) [06:59:44] *** Joins: tomzawadzki (uid327004@gateway/web/irccloud.com/x-axywpncnlptirvne) [07:30:39] i think the pic looks fine [07:30:59] that photographer did a fine job [07:31:04] lol [07:53:40] lol [07:54:01] that dude in the back looks sick though [07:54:45] jimharris, did you rebook the euro series? I thought I did but it wn't let me log in saying the meeting hasnt started [07:55:09] no [07:55:24] hmm, now its kidna working [07:56:30] OK, I'm in and sharing [08:02:34] "The meeting has ended" [08:02:35] OK [08:02:59] same for me [08:03:28] paul left and is coming back in because his mic isn't working. I think that is why it ended. [08:03:36] *** Joins: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) [08:03:36] *** ChanServ sets mode: +o bwalker_ [08:04:40] *** Quits: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) (Client Quit) [08:16:07] *** Joins: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) [08:16:07] *** ChanServ sets mode: +o bwalker_ [08:17:49] jimharris: /away in a meeting [09:01:35] *** Quits: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) (Read error: Connection reset by peer) [10:42:26] jimharris: bwalker: Please take a look at: https://review.gerrithub.io/#/c/spdk/spdk/+/431169/ Want to include this into 18.10.1 as it impacts packaging. [11:13:30] *** Joins: travis-ci (~travis-ci@ec2-54-161-94-58.compute-1.amazonaws.com) [11:13:31] (spdk/master) CHANGELOG: mention long name cmdline parameters (Darek Stojaczyk) [11:13:31] Diff URL: https://github.com/spdk/spdk/compare/3e2297140ca1...dba21f69d930 [11:13:31] *** Parts: travis-ci (~travis-ci@ec2-54-161-94-58.compute-1.amazonaws.com) () [11:30:07] jimharris, bwalker can one of you take a quick look at these basic PT changes? This chain leads up to the compress bdev and Pawel marked them -1 with some questions I'd like confirmation on... https://review.gerrithub.io/#/c/spdk/spdk/+/428723/ [12:41:58] *** Joins: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) [12:45:51] Good evening. I am noticing strange behavior of my bdev. I submit io via spdk_bdev_read to nvme device. but end_io callback of this bdev_io happens on the thread different from the one I submitted it. Why could this happen? This only happens on real nvme devices. Everything works well on null or malloc and only with fio testing [12:46:38] Any thoughts appreciated. I have been spending too much time on this issue.. ( [12:47:42] hi alekseymmm - which application are you using to send I/O to the bdev? [12:49:25] fio [12:49:42] LD_PRELOAD fio plugin and so on [12:49:59] numjobs=1 iodepth=1 [12:51:27] Could it be related to the fact that fio use its own pthreads not spdk_threads? [12:51:45] or it is not that way? [12:52:02] you have a custom bdev module that is claiming nvme namespace bdevs? [12:52:15] yes [12:52:25] kinda similar to pastrhu [12:53:46] May be some directions to investigate? [12:55:01] I found out that the exact place of the issue is that in _spdk_bdev_io_complete spdk_get_thread returns NULL in assert checking. [12:55:55] This happens before end io callback for io that I send by spdk_bdev_read(). [12:56:34] So for me it looks like spdk_bdev_io_complete happens not in spdk thread [12:56:44] Is it possible ? [12:59:18] without knowing exactly what your custom bdev looks like, it is hard to say [12:59:33] the bdev nvme driver will only complete io from the context of its poller [13:01:00] meaning the bdev nvme driver does not send messages between different threads to transfer I/O handling [13:01:08] sad but true. [13:01:21] But may be fio handle its threads in some crazy way? [13:02:03] Are they exactly the same as spdk_threads like in "normal" apps ? [13:02:11] they should be, yes [13:02:21] ok( [13:02:29] you're running the fio plugin with thread=1 right? [13:02:39] otherwise it spawns processes and that would not work [13:02:40] sure [13:02:42] what's the backtrace when you hit this condition in _spdk_bdev_io_complete? [13:02:50] otherwise it doesn't work [13:05:12] see https://pastebin.com/pcwv0KHJ [13:05:29] and then thread.c: 215:spdk_get_thread: *ERROR*: No thread allocated fio: bdev.c:2953: _spdk_bdev_io_complete: Assertion `spdk_get_thread() == spdk_io_channel_get_thread(bdev_io->internal.ch->channel)' failed. [13:05:50] bt doesnt help me [13:07:33] one sec - ben and i are chatting on this here in the lab - there might be an fio plugin error here [13:07:45] may be it is related to the fact that in bdev_nvme_poll (arg=0x7839c0) and in spdk_fio_poll_thread (fio_thread=0x77c310) [13:07:47] ? [13:08:28] thanks to all gods! if it is fio bug not mine [13:13:18] spdk_fio_init_env and spdk_init_thread_poll have a problem [13:13:47] the code starting at "rc = spdk_fio_init_thread" to the pthread_create call needs to be moved inside of spdk_init_thread_poll [13:14:15] and then spdk_init_thread_poll needs to set a condition variable to signal spdk_fio_init_env that it can continue and return after pthread_create() [13:14:47] similar work needs to be done moving code from spdk_fio_finish_env to spdk_init_thread_poll as well [13:15:32] *** Quits: tomzawadzki (uid327004@gateway/web/irccloud.com/x-axywpncnlptirvne) (Quit: Connection closed for inactivity) [13:18:00] do you have a patch ? [13:20:32] I would really like to have these changes [13:20:44] no patch yet [13:21:05] we'd be happy to accept a patch if it's something you'd like to work on [13:21:33] either way, it would be good to have a github issue filed for this to make sure it gets assigned at the next bug scrub [13:22:11] how could I explain this in github without referencing my unfinished bdev ? [13:22:51] or I could just write what you have said about moving code around in fio_plugin [13:23:11] if you can just explain briefly what you wrote here in IRC, I will add more details to it afterwards [13:23:32] ok [13:23:39] please post the backtrace into the github issue too - that was important [13:23:47] sure [13:34:12] filled the issue on github . #489 . Please correct me or attach any useful info [13:34:28] yep - adding more details now - thanks! [13:34:55] When do you think will be mentioned bug scrub ? [13:36:47] there is a bug scrub scheduled for today - see http://spdk.io/community for details [13:36:54] about 8 hours from now [13:40:07] it will be 7 am my timezone... Really hard to wake up but really wish to participate) [13:42:09] your attendance is of course welcome, but for this bug i don't think it's necessary - i certainly understand the bug and proposed fix and can communicate that during the meeting [13:42:51] discussion of a bug can also be during a community meeting, if that time works better for you [13:43:18] (but we just had our Euro-friendly community meeting this morning - next one will be in two weeks) [13:44:56] bah - peluse just corrected me - bug scrub is tomorrow, not today [13:45:15] I am so unlucky on scheduling. Ok I am looking forward those changes in fio_plugin anyway [13:45:26] oh. tomorrow is much better) [14:04:07] @jimharris Thanks a lot for help with this issue. Hope fixes will do something [14:29:22] bwalker, can you take a quick look at this, I have a few spdk repo patches that depend on this 2 liner https://review.gerrithub.io/#/c/spdk/dpdk/+/431413/ [14:33:53] done [14:45:01] thanks!!! jimharris you're next ;) [16:35:49] *** Quits: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) (Quit: Page closed) [20:47:30] *** Joins: tomzawadzki (uid327004@gateway/web/irccloud.com/x-spskmlpsqhjfazah)