[01:52:10] *** Joins: tkulasek (tkulasek@nat/intel/x-yxqblwetnpkencyc) [06:23:20] does anyone remember what's the story behind the "usleep(500 * 1000);" in our pci device init code? [06:23:32] how was this value picked? [07:08:08] interesting.... :) [08:14:14] *** Joins: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) [09:15:06] I will try to remember the exact scenario, but basically we'd request the device to do a PCI reset [09:15:14] and the kernel would say sure and return immediately [09:15:22] but we need to wait for the reset to finish [09:15:48] it may very well be a kernel bug (the not waiting before returning) [09:29:42] *** Joins: waelhalbawi (94571707@gateway/web/freenode/ip.148.87.23.7) [09:35:24] I have one system with a 8086:0953 nvme where 500ms sleep is not enough [09:36:05] it's a fedora26 w/ 4.14.16 [09:36:25] increasing the sleep to 1s makes the problem go away [09:37:10] but then on another machine - ubuntu18 w/ 4.15.0 - i can remove the sleep entirely and everything still works [09:42:21] (ubuntu18 uses 8086:0953 as well) [10:10:15] *** Quits: waelhalbawi (94571707@gateway/web/freenode/ip.148.87.23.7) (Ping timeout: 256 seconds) [10:21:41] *** Quits: tkulasek (tkulasek@nat/intel/x-yxqblwetnpkencyc) (Quit: Leaving) [10:27:42] peluse: replied "bdev: do not finish unitialized modules" https://review.gerrithub.io/c/spdk/spdk/+/421158 [10:28:38] darsto, cool, yeah I thought it looked good but wasn't sure about the IRC comment, thanks [10:28:54] okay, cool [10:52:54] bwalker: do you recall what device type did you see this issue with? [10:53:02] was it 0953 as well? [10:53:35] it was a P3700 [10:53:53] and I think some other types of devices too - the problem isn't the device but rather that the kernel does not wait [10:54:07] oh [10:54:27] 0953 used to have 100us delay in our nvme driver as well [10:54:35] that was a different problem though [10:55:17] i'm not able to encounter any problems with i/oat though [10:55:28] it's only the 0953 nvme that's failing without that vfio usleep [10:55:46] well, i don't have any other nvmes to test with [10:56:35] from my memory it's definitely a race condition - sometimes the device comes back quickly and it works [10:56:50] the kernel could have changed since we hit this too [10:59:51] btw. how do i check which device id does p3700 have? [11:00:12] is there some public database on that? [11:02:50] I think all NVMe devices have 0x0953 [11:03:34] I think it might be set by the PCI specification committee [11:04:16] nvme quirks are assigned by pci device id though [11:04:21] at least in our nvme driver [11:04:43] oh yeah you're right, there are several of them [11:04:49] let me take a look at where those come from [11:06:14] yeah that's the class code that is generic, not the device code [11:08:08] ok, they're made up by Intel [11:08:13] unique within Intel's vendor id space [11:08:31] http://pci-ids.ucw.cz/read/PC/ [11:08:35] there is a database of all of them [11:09:26] yeah, i just found this - https://pci-ids.ucw.cz/read/PC/8086/0953 [11:10:13] it's very likely that the list is incomplete, but still, the P3700 is there [11:11:32] but that doesn't help me in any way. how bad would it be if I pushed a patch that increases the vfio sleep from 500ms to 1s? [11:11:51] it just makes it take longer to start up [11:11:56] not the end of the world [11:12:10] we should figure out what needs to change in the kernel to make it wait after a PCI reset [11:12:37] yeah - but if you're using I/OAT and there are 16 different VFs... [11:16:08] what if we try to issue another vfio reset inside spdk? [11:16:28] or an software-only reset (a vfio addition) [11:16:52] just thinking loud - i'll try that tomorrow [11:17:44] if you issue the reset, how do you wait for it to finish? [11:18:00] do we have access to the PCI config space? I think we just have the BAR [11:18:18] maybe it'll wait on some internal mutex? [11:18:37] we have the config space as well [11:19:11] it appears to be read-only though, the writes don't make it all the way down to the hardware [11:19:42] or the hardware doesn't react to them [11:23:21] i tried pushing the patch "nvme_ctrlr: retry setting CC.EN = 1" https://review.gerrithub.io/c/spdk/spdk/+/426334 [11:24:25] and the enable bit gets set and stays so, but the device does not set the ready state anyway [11:44:20] ok. thanks ben, i'll push a patch that increases the sleep time for now, but will continue fiddling with this over this week [11:45:04] for now now - have a good night [12:55:53] I went to lunch but I think if you set EN too early (which is what happens) the device is just bricked until you PCI reset it again [12:56:06] so setting CC.EN to 1 later on doesn't do anything [15:30:23] *** Joins: waelhalbawi (94571705@gateway/web/freenode/ip.148.87.23.5) [15:53:03] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [18:56:41] *** Quits: waelhalbawi (94571705@gateway/web/freenode/ip.148.87.23.5) (Ping timeout: 256 seconds) [20:50:00] *** Joins: travis-ci (~travis-ci@ec2-54-158-90-20.compute-1.amazonaws.com) [20:50:01] (spdk/master) vhost_blk: Add support for blk_request_queue_io in vhost_blk (Ni Xun) [20:50:02] Diff URL: https://github.com/spdk/spdk/compare/99850ca7d5cb...4ada23dbf432 [20:50:02] *** Parts: travis-ci (~travis-ci@ec2-54-158-90-20.compute-1.amazonaws.com) () [21:22:15] *** Joins: travis-ci (~travis-ci@ec2-54-81-62-178.compute-1.amazonaws.com) [21:22:16] (spdk/master) test/nvmf: add fio and lvol test cases with raid bdev (Chen Wang) [21:22:16] Diff URL: https://github.com/spdk/spdk/compare/4ada23dbf432...104f7d857503 [21:22:16] *** Parts: travis-ci (~travis-ci@ec2-54-81-62-178.compute-1.amazonaws.com) () [22:11:48] *** Joins: travis-ci (~travis-ci@ec2-54-163-34-26.compute-1.amazonaws.com) [22:11:49] (spdk/master) bdev: Fix spdk_bdev_part_io_type_supported() (Wael Halbawi) [22:11:49] Diff URL: https://github.com/spdk/spdk/compare/104f7d857503...e80b3d5f1328 [22:11:49] *** Parts: travis-ci (~travis-ci@ec2-54-163-34-26.compute-1.amazonaws.com) () [23:32:05] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds)