[00:44:08] *** Quits: tomzawadzki (~tzawadzk@192.55.54.39) (Ping timeout: 240 seconds) [00:45:29] *** Joins: tomzawadzki (~tzawadzk@134.134.139.76) [00:55:02] *** Quits: ziyeyang_ (~ziyeyang@134.134.139.78) (Quit: Leaving) [05:20:30] *** Quits: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [05:56:01] *** Quits: tomzawadzki (~tzawadzk@134.134.139.76) (Ping timeout: 246 seconds) [08:15:54] *** Joins: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) [08:23:12] *** Quits: peluse_ (~peluse@2600:8800:140a:ae18:41ef:3342:fc78:9c7f) (Ping timeout: 258 seconds) [08:38:22] *** Quits: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [08:46:20] *** Joins: peluse_ (~peluse@2600:8800:140a:ae18:3986:cbbb:3bb7:ddaa) [09:11:20] *** Joins: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) [09:17:22] *** Joins: jan (~textual@ip-94-113-131-57.net.upcbroadband.cz) [09:17:45] *** jan is now known as Guest72860 [09:26:44] *** Quits: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [09:42:13] *** Quits: peluse_ (~peluse@2600:8800:140a:ae18:3986:cbbb:3bb7:ddaa) (Ping timeout: 276 seconds) [09:56:20] *** Joins: Darren (c6ba0002@gateway/web/freenode/ip.198.186.0.2) [09:56:44] *** Darren is now known as Guest55689 [09:57:49] I was chatting yesterday on the 4ms delay. I finally got to the bottom. It is related to the pre-fetch patch you listed (4766849) [09:58:20] so if you rever that, it works? [09:58:24] revert [09:58:36] The delay was not the detection of the change in the wq. The delay is the write to the bar registers to ack the entry, then another bar to push new work [09:59:10] so it sees the change, but it write combines the ack MMIO [09:59:13] Yes. I also tried doing a read back after every write. That works too. [09:59:30] is there a reason your device is reporting its BAR as prefetchable in the PCI header? [09:59:44] if you don't report prefetchable there, our code won't map it with write combining [10:00:28] correct. I have 2 writes going back (to different bars) they would both be held for the 4ms. If I do a read back after each write, things are great. Not a great solution [10:00:57] what are the sizes of these writes? [10:01:07] can you just do an sfence after each write? [10:01:30] I have to check the history on this. This card family has been around for 20yrs. [10:01:34] writes are 32-bit [10:01:45] you can use spdk_wmb() macro in spdk/barrier.h [10:02:06] that's a good idea. Let me try that. [10:02:45] hopefully that workaround fixes it - but it's definitely a bug in the PCI header for that device [10:02:58] we're doing what the device is telling us to do [10:06:05] prefetchable BARs just means that there no side effect on reads and that host bridges *can* merge writes [10:07:13] it sounds like for this device, that is the case - we probably need to better document this though, that writes to prefetchable BARs will be write-combined and will require an sfence in cases like this (presuming that fixes the issue) [10:07:52] I assume these devices work because other drivers aren't quite so aggressive in enabling and utilizing all performance features [10:08:08] so even though it can be mapped with write combining, they probably don't [10:12:13] it does support the write combining required to be pre-fetchable on these bar's. It works fine after the 4ms delay happens and the write goes out. [10:13:12] The problem is that the code waits 4ms for another write that may/may not appear. The only way around this is to code with the wmb to cause the flush [10:14:08] have you confirmed that the wmb fixes the issue? [10:14:51] next thing to try. Just put the dpdk patch back in [10:16:45] *** Joins: peluse_ (~peluse@2600:8800:140a:ae18:acd8:57ba:123d:d97d) [10:17:04] bwalker: FYI blobstore example ready for review... [10:19:47] *** Joins: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) [10:40:30] *** Quits: Guest55689 (c6ba0002@gateway/web/freenode/ip.198.186.0.2) (Ping timeout: 260 seconds) [10:59:14] *** Joins: Darren (c6ba0002@gateway/web/freenode/ip.198.186.0.2) [10:59:37] *** Darren is now known as Guest44154 [10:59:59] Adding the spdk_wmb worked also to remove the 4ms delay. [11:28:43] *** Quits: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [11:42:57] *** Joins: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) [12:12:10] great! [12:22:24] *** Joins: nKumar (uid239884@gateway/web/irccloud.com/x-fvgbkhbcenzuzckj) [13:13:04] *** Quits: Guest72860 (~textual@ip-94-113-131-57.net.upcbroadband.cz) (Quit: Textual IRC Client: www.textualapp.com) [13:23:33] *** Quits: Guest44154 (c6ba0002@gateway/web/freenode/ip.198.186.0.2) (Quit: Page closed) [14:53:50] *** Quits: nKumar (uid239884@gateway/web/irccloud.com/x-fvgbkhbcenzuzckj) (Quit: Connection closed for inactivity) [14:56:49] *** Quits: peluse_ (~peluse@2600:8800:140a:ae18:acd8:57ba:123d:d97d) (Ping timeout: 276 seconds) [15:38:39] *** Joins: sethhowe_ (~sethhowe@134.134.139.82) [15:42:45] *** Quits: vermavis (~vermavis@192.55.54.44) (Excess Flood) [15:43:25] *** Quits: sethhowe (~sethhowe@192.55.54.42) (Remote host closed the connection) [15:43:31] *** Joins: vermavis (~vermavis@192.55.54.44) [15:45:47] *** Quits: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) (Ping timeout: 269 seconds) [15:47:24] so I'm missing something on the test side with the hello_world example, the git porcelain check in autobuild.sh is finding the binary still sitting there so all test systems fail. Not sure what I missed, maybe something in one of the Makefiles? [15:48:29] *** Joins: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) [15:51:06] add a .gitignore file [15:51:11] to ignore the executable [15:51:33] yeah, I looked there to see if other example code binaires where in there but I didn't see any, maybe I looked to quickly [15:51:53] put a new .gitignore file in the directory examples/blob/hello_world [15:52:08] see e.g. examples/nvme/perf/.gitignore [15:52:15] ohhhh... duh [15:52:30] that's what this git status --porcelain check is detecting :) [15:52:36] whether you forgot to add the .gitignore [15:52:40] because everyone did [15:53:10] except me, I have never forgotten a .gitignore [15:53:17] lol [15:53:40] correction to my statement - /s/everyone/jim/g [15:54:49] haha, yeah I was looking at the root .gitignore, didn't realize there were sprinkled elsewhere - pushed :) [16:26:55] *** Joins: ziyeyang_ (~ziyeyang@134.134.139.72) [16:37:17] *** Quits: gila (~gila@5ED4FE92.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [17:06:24] *** Quits: ziyeyang_ (~ziyeyang@134.134.139.72) (Remote host closed the connection) [17:08:00] *** Joins: ziyeyang_ (~ziyeyang@192.55.54.40) [17:11:28] *** Quits: ziyeyang_ (~ziyeyang@192.55.54.40) (Remote host closed the connection) [17:14:43] *** Joins: ziyeyang_ (~ziyeyang@192.55.54.40) [17:18:32] *** Quits: ziyeyang_ (~ziyeyang@192.55.54.40) (Remote host closed the connection) [17:19:59] *** Joins: ziyeyang_ (~ziyeyang@192.55.54.40) [17:40:59] *** Joins: ziyeyang__ (ziyeyang@nat/intel/x-bdswaznddhlvktjq) [17:40:59] *** Quits: ziyeyang_ (~ziyeyang@192.55.54.40) (Remote host closed the connection) [17:57:41] *** Quits: ziyeyang__ (ziyeyang@nat/intel/x-bdswaznddhlvktjq) (Remote host closed the connection) [18:00:21] *** Joins: ziyeyang_ (ziyeyang@nat/intel/x-wqozahfbxedhjafc) [18:12:05] *** Quits: ziyeyang_ (ziyeyang@nat/intel/x-wqozahfbxedhjafc) (Remote host closed the connection) [18:12:22] *** Joins: ziyeyang_ (~ziyeyang@192.55.54.40) [18:17:06] *** Quits: ziyeyang_ (~ziyeyang@192.55.54.40) (Remote host closed the connection) [18:18:16] *** Joins: ziyeyang_ (ziyeyang@nat/intel/x-vzyoslyzyvhwzdzx) [18:20:58] *** Quits: ziyeyang_ (ziyeyang@nat/intel/x-vzyoslyzyvhwzdzx) (Remote host closed the connection) [18:24:16] *** Joins: ziyeyang_ (ziyeyang@nat/intel/x-zxkljvrrziwpmdrk) [18:33:45] *** Joins: ziyeyang__ (ziyeyang@nat/intel/x-fxcuidvxukrnulzk) [18:33:45] *** Quits: ziyeyang_ (ziyeyang@nat/intel/x-zxkljvrrziwpmdrk) (Remote host closed the connection) [18:41:29] *** Joins: ziyeyang_ (~ziyeyang@192.55.54.40) [18:41:29] *** Quits: ziyeyang__ (ziyeyang@nat/intel/x-fxcuidvxukrnulzk) (Remote host closed the connection) [23:08:44] *** Joins: tomzawadzki (~tzawadzk@192.55.54.36)