[00:30:24] *** Joins: darsto_ (~darsto@89-68-128-62.dynamic.chello.pl) [00:31:27] *** Quits: darsto (~darsto@89-68-128-62.dynamic.chello.pl) (Ping timeout: 240 seconds) [00:31:28] *** darsto_ is now known as darsto [00:34:47] *** Joins: dlw1 (~Thunderbi@114.255.44.139) [00:36:05] *** Quits: dlw (~Thunderbi@114.255.44.143) (Ping timeout: 240 seconds) [00:36:07] *** dlw1 is now known as dlw [00:53:13] *** Joins: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) [00:53:50] *** Quits: dlw (~Thunderbi@114.255.44.139) (Remote host closed the connection) [00:54:20] *** Joins: dlw (~Thunderbi@114.255.44.139) [00:55:39] Hello spdk developers. I have a question. If in my bdev module it is possible to pass incoming bdev_io from one io_channel to the other , then what is the proper way to complete this bdev_io ? In _spdk_bdev_io_complete() you only chesk if (spdk_unlikely(bdev_io->internal.in_submit_request || bdev_io->internal.io_submit_ch)) to Send the completion to the thread that originally submitted the I/O. [00:55:56] But how could I set for this bio bdev_io->internal.io_submit_ch [00:56:10] since it is internal [01:00:28] alekseymm: don't think it is wise to send it to other channel [01:00:58] there is explicit assert in _spdk_bdev_io_complete() [01:00:59] assert(spdk_get_thread() == spdk_io_channel_get_thread(bdev_io->internal.ch->channel)); [01:01:16] so don't think it will work. [01:01:26] question is why you want to do that? [01:01:33] QoS do that [01:01:39] so why cannot I ? [01:01:45] it is said in comment [01:01:55] *** Joins: tkulasek (~tkulasek@192.55.54.40) [01:02:14] /* * Send the completion to the thread that originally submitted the I/O, * which may not be the current thread in the case of QoS. */ [01:02:23] in _spdk_bdev_io_complete() [01:02:28] so it seems possible [01:05:17] you can send bdevio as a attached data but can't complete it on other thread. This is why it is send back to the original thread. [01:10:44] why cannot I pass completion to original thread ? we have bdev_io->internal.io_submit_ch [01:11:20] An it seems to me tjat in _spd_bdev_io_complete you do exactly what I want [01:11:21] spdk_thread_send_msg(spdk_io_channel_get_thread(bdev_io->internal.ch->channel), _spdk_bdev_io_complete, bdev_io); [01:12:27] I mean you restore internal.ch from internal.io_submit_ch (initial channel I believe) an send completion via msg [01:12:46] ohhh, yes, you are right. [01:13:28] so my question is how to indicate for bdev_io that it should follow this logic [01:13:34] it send it to original thread [01:13:54] since the only way i see is to pass the checking if (spdk_unlikely(bdev_io->internal.in_submit_request || bdev_io->internal.io_submit_ch) [01:14:19] I don feel it is a good idea to change bdev_io->internal.in_submit_request, [01:14:38] is it ok if I set up bdev_io->internal.io_submit_ch in my module make _request function ? [01:16:47] *** Joins: Aneesh (81fdb3a1@gateway/web/freenode/ip.129.253.179.161) [01:17:23] *not make_request but submit_request module function [01:19:30] *** Joins: Rakesh (81fdb3a1@gateway/web/freenode/ip.129.253.179.161) [01:21:18] hmmm... this whole routing stuff to other thread seem to be specific for QoS. It look like reusing this will be rather a hack than using some feature SPDK provides. [01:22:14] @bwalker, @jimharris, @drv Could you please rerun the CI for the patch "https://review.gerrithub.io/c/spdk/spdk/+/409297" [01:22:55] The error coming from the build log is not related to the changes I've done. [01:22:57] can't give you the answer right now, can you send email to mailing list? someone with deeper knowledge will pick it up. [01:25:39] Rakesh: pls rebase you patch on master, I think we recently fixed this issue you hit in this patch. [01:28:28] @powdkowx Thank you for the response Pawel Wodkowski, I'll do it. [02:00:32] ok thanks you any way [02:07:12] *** Quits: Aneesh (81fdb3a1@gateway/web/freenode/ip.129.253.179.161) (Ping timeout: 252 seconds) [03:45:22] *** Quits: dlw (~Thunderbi@114.255.44.139) (Ping timeout: 264 seconds) [03:52:08] *** Quits: guerby (~guerby@april/board/guerby) (Remote host closed the connection) [03:56:03] *** Joins: guerby (~guerby@ip165.tetaneutral.net) [03:56:04] *** Quits: guerby (~guerby@ip165.tetaneutral.net) (Changing host) [03:56:04] *** Joins: guerby (~guerby@april/board/guerby) [04:23:37] *** Joins: dlw (~Thunderbi@111.197.237.248) [04:32:27] *** Quits: dlw (~Thunderbi@111.197.237.248) (Ping timeout: 240 seconds) [04:38:42] *** Quits: Rakesh (81fdb3a1@gateway/web/freenode/ip.129.253.179.161) (Quit: Page closed) [04:56:16] *** Joins: lyan (~lyan@2605:a000:160e:2dd:4a4d:7eff:fef2:eea3) [04:56:41] *** lyan is now known as Guest22237 [04:58:24] jimharris: about those asserts in blobstore - I was debbuging issue #339 so target crashed several times during VM boot. But this was release build, the I rebuild for debug and hit this issue. So it is very likely that corruption was long before it was detected. [04:59:19] * ... then I rebuild with debug enabled ... [05:17:38] *** Joins: dlw (~Thunderbi@111.197.237.248) [05:22:09] *** Quits: dlw (~Thunderbi@111.197.237.248) (Ping timeout: 260 seconds) [05:29:37] *** Joins: dlw (~Thunderbi@111.197.237.248) [05:38:21] pwodkowx: I have observed same issue debugging cluster allocation path when md thread was unexpectedly ended during updating metadata. [06:27:27] *** Quits: dlw (~Thunderbi@111.197.237.248) (Ping timeout: 240 seconds) [06:27:59] *** Joins: dlw (~Thunderbi@111.197.237.248) [06:58:16] *** Quits: dlw (~Thunderbi@111.197.237.248) (Ping timeout: 244 seconds) [07:18:08] *** Joins: philipp-sk (~Philipp@ktnron0916w-lp130-02-76-66-162-159.dsl.bell.ca) [07:44:08] *** Joins: dlw (~Thunderbi@111.197.237.248) [07:59:10] hi alekseymmm - you may want to look at the iscsi initiator driver [07:59:22] lib/bdev/iscsi [07:59:52] it uses libiscsi and has just one connection to the target, so it funnels all I/O to one thread [07:59:58] (one connection for each LUN) [08:01:26] jimharris, let me know when the ipsec spadk branch is pointing to .49 and I'll make the changes and update the submodule commit. no hurry of course [08:02:19] and FYI I'm home today. Wife stung by scorpion late last night and it pretty numb all over. Nobody else home so if she gets really bad I may take her to ER... damn scorps! [08:05:48] *** Quits: alekseymmm (050811aa@gateway/web/freenode/ip.5.8.17.170) (Ping timeout: 252 seconds) [08:40:59] *** Quits: tomzawadzki (~tomzawadz@192.55.54.44) (Ping timeout: 268 seconds) [08:48:16] scorpions suck [08:48:28] ipsec spdk branch is updated - check it out and let me know how it works [08:50:07] OK, cool. Will do [08:50:23] i updated both the gerrithub and github spdk branches [08:50:46] yeah, they do suck. one got my toe last year and I was numb from the knee down for over 24 hours! Penny's entire leg is numb and even her hands are tingling and she got it in the foot [08:57:54] jimharris, see https://review.gerrithub.io/#/c/spdk/intel-ipsec-mb/+/420500 [08:58:37] once pushed I'll try that commit in my ipsec submodule and if it works, along with your other dpdk 18.05 patch I'll push one to move the ipsec submodule to that commit [09:07:48] *** Joins: peter_turschm (~peter_tur@66.193.132.66) [09:25:39] Hi guys, need help with git submodules (intel-ipsec-mb) as builds on Jenkins seem to be failing. What I mean is - go to Jenkins main UI, autotest-per-patch, build 8262 and check the output log [09:25:54] dpdk is getting cloned [09:26:09] but ipsec fails with "reference is not a tree" [09:26:17] ok - i'll look - i just did some updates with this and must have messed something up [09:26:57] The build I mention was a current master top build, so it's probably something merged recently [09:27:41] @peluse wow, thats bad. Is she OK? [09:29:54] klateck, thanks. yeah she'll be fine I'm sure. happens all the time in AZ and even with bark scorpions (the worst kind) adults are generally good after a day or two. Its kids and dogs that are at real risk [09:30:37] damned intel-ipsec-mb [09:31:20] Good to hear that. So its not THAT bad right? I was imagining these really posionous small, white or bright yellow scorps. [09:32:42] the bark scorpion is indeed the worst and they're all over the place in AZ. I think we're all just sort of used to it so it doesn't seem so bad. I suppose it is though :) [09:32:47] *** Quits: dlw (~Thunderbi@111.197.237.248) (Ping timeout: 268 seconds) [09:32:58] its amazing what just a drop of their poison can do though... crazy [09:35:12] jimharris, might make sense to get the ipsec patch that I pushed landed and submodule updated before trying to address whatever that reference error is, it might go away in the process of getting everything "re syncd" [09:36:18] it's fixed [09:36:25] what did you have to do? [09:36:33] Yeah I am looking at the consle output now:) [09:36:43] peluse - magic [09:36:47] :) [09:37:00] always... and my magic wand is in the shop [09:37:17] i did a force push with the new 0.49 changes - git probably figured out that then nothing was referencing the old commit we had in our submodule reference - so when jenkins cloned it, it didn't pull down that reference [09:37:34] ahhh [09:37:38] so i just made a new spdk-0.48 branch to point to the 'current' submodule commit and pushed that [09:37:59] OK, so the pending patch I have should still be valid then right? [09:38:20] yes [09:38:27] rock on [09:39:31] @jimharris Thanks! that way quick [09:40:00] you literally caught the problem 2 minutes after i created it so i knew exactly where to look :) [09:40:44] heh [10:04:57] *** Quits: tkulasek (~tkulasek@192.55.54.40) (Ping timeout: 240 seconds) [10:13:16] peluse: looking at your patch - i'm just curious, why didn't you have this additional change on the v0.48 branch? [10:13:47] earlier you just had it on the static (non-shared) part of that ifeq [10:14:16] i'm fine having it on both (which is probably more correct) was just curious [11:32:55] bwalker: jimharris: I think I've resolved the segfault problem with nvme/fio. I've submitted a patch. Please keep an eye out for it once it gets through the test pool. [11:34:16] great - I was just about to ping you on that [11:39:07] bwalker: https://review.gerrithub.io/#/c/spdk/spdk/+/420575/ [12:05:40] jimharris, it was on there [12:07:15] jimharris, see https://github.com/spdk/intel-ipsec-mb/commit/60a1f479c20d5374b7efe333e3c1b6c908d66d7d [12:08:20] oh, I see what you mean now [12:09:33] did that .48 Makefile change make it to the .49 branch? It looks like it. If so we don't need the patch I jsut submitted [12:14:57] yeah, not needed. [12:23:10] jimharris, submodule update... pretty sure I did this right :) https://review.gerrithub.io/c/spdk/spdk/+/420579 [12:31:45] @lhodev, your fast. Thanks for working on the issue. I will download the patch sometime tomorrow and test it [12:33:25] jkkariu: happy to help! [12:59:07] *** Joins: alekseymmm (bcf3adf1@gateway/web/freenode/ip.188.243.173.241) [14:12:33] gila: ping [14:12:48] Hey @jimharris [14:13:09] bwalker and i are looking at your backtrace [14:13:57] spdk_bdev_readv (frame #8) shows parameter nbytes=0 which is wacky [14:14:07] frame #9 looks ok [14:14:26] i have no idea how this could be aio-specific but one thing at a time :) [14:14:33] do you still have that gdb session handy? [14:14:39] Yes sir [14:14:59] is this a debug build? [14:15:44] Well, i used --enable-debug during configure, so I assume so [14:16:32] perfect [14:16:49] can you go up to frame 16 - spdk_iscsi_op_scsi? [14:17:02] f 16 [14:17:11] whoops [14:17:18] yes [14:17:19] there. [14:17:39] p *task [14:17:54] or specifically p task->scsi.transfer_len [14:18:21] (gdb) p task->scsi.transfer_len [14:18:21] $3 = 0 [14:18:21] (gdb) [14:19:02] can you do p *pdu [14:19:07] same frame [14:19:48] (gdb) p *pdu [14:19:49] $4 = {bhs = {opcode = 1 '\001', immediate = 0 '\000', reserved = 0 '\000', flags = 129 '\201', rsv = "\000", total_ahs_len = 0 '\000', data_segment_len = "\000\000", [14:19:49] lun = 0, itt = 805306382, ttt = 0, stat_sn = 1460273152, exp_stat_sn = 16777216, max_stat_sn = 40, res3 = "\000\000\000\000\b\000\000\000\000\000\000"}, mobj = 0x0, [14:19:49] data_buf = 0x0, data = 0x0, header_digest = "\000\000\000", data_digest = "\000\000\000", data_segment_len = 0, bhs_valid_bytes = 48, ahs_valid_bytes = 0, [14:19:49] data_valid_bytes = 0, hdigest_valid_bytes = 0, ddigest_valid_bytes = 0, ref = 2, data_from_mempool = false, task = 0x0, cmd_sn = 2647, writev_offset = 0, tailq = { [14:19:49] tqe_next = 0x0, tqe_prev = 0x0}, ahs = '\000' , sense = {length = 0, data = '\000' }} [14:20:00] just a sec [14:20:27] https://pastebin.com/SLPAKEGv [14:20:28] there [14:24:31] p *(struct iscsi_bhs_scsi_req *)(&pdu->bhs) [14:24:47] i think sg_raw is doing something weird here [14:25:29] yeah, the cdb says to read 8 blocks, but the iscsi header says 0 bytes [14:25:51] can you try: [14:26:19] sg_raw -r 4096 /dev/sdc 28 00 00 00 00 00 00 00 08 00 [14:26:34] i think without the -r 4096, it doesn't populate the expected transfer size [14:27:48] Well, im not sure if this comes from sg_raw here -- because the kernel fails to attach the device (due to all the failures) and when I issue sg_raw, the kernel tries to attach again so this might be a TUR as well. [14:28:13] @jimharris here is the gdb output: [14:28:30] https://pastebin.com/rbRHd5CZ [14:28:40] I will restart SPDK and try the sg_raw cmd now [14:30:26] So right after restarting SPDK, login in/out to the LU: [14:31:32] Hello everyone. I have some question. I have been working with my bdev module and on some tests with fio I get spdk_bdev_io_complete: Assertion `shared_resource->io_outstanding > 0' failed. What could be the general reason for that ? what is shared_resource ? [14:31:36] https://pastebin.com/w2V1Z2G8 [14:33:12] @jimharris sg_raw output the same [14:33:55] I can print/dump out all the PDU's during attach if that would help [14:36:22] *** Joins: dlw (~Thunderbi@111.197.237.248) [14:37:53] @gila - hmmm, can you try a simple dd command? [14:38:00] alekseymmm: my guess is that you're completing a bdev_io on a different core than it was submitted on [14:38:18] dd if=/dev/sdX iflag=direct of=/dev/null bs=512 count=1 [14:38:36] the shared_resource thing is a big complex subject [14:38:39] i want to see if this is sg_raw related [14:39:08] but essentially, it's a method of fairly allocating bdev_io within a single thread [14:39:30] so no individual bdev or io_channel gets starved [14:39:35] @jimharris sure [14:39:47] but it yields: [14:39:52] dd: error reading '/dev/sdc': Input/output error [14:40:36] bwalker you are right . And it is my intend. As someone suggested me to looak at bdev/iscsi for similar functionality I do spdk_thread_send_msg(submit_td, _bdev_io_complete, bdev_io); hoping that this will redirect end io to initial thread [14:40:45] is it correct understanding? [14:40:59] the dd commands never get through if i'm not mistaking.. as the open() on the disk fales because its not properly attached yet [14:41:44] if submit_td is the thread the bdev_io was originally submitted on, then yes that will work [14:42:36] i store submit_td in module make_request function using spdk_io_channel_get_thread() [14:42:40] note the assertion is indicating just a problem with the total count - so the particular I/O that causes the assertion may not be the problem [14:43:17] on each thread, we track the number of times submit was called and the number of times complete was called [14:43:22] It makes it harder to investigate [14:43:24] reference count, so increment and decrement [14:43:38] the assert is saying that a completion came in, but the count is already 0 [14:43:42] so something is off somewhere [14:45:10] but you have 2 assertions there assert(bdev_ch->io_outstanding > 0); assert(shared_resource->io_outstanding > 0); [14:45:23] and only the second one yield [14:45:27] you could add much more explicit checks in your module by saving the thread into your bdev_io context on submit and asserting it matches the current thread (spdk_get_thread()) before calling spdk_bdev_io_complete [14:45:39] yeah - the shared resource is per thread, but across all channels on that thread [14:45:41] so the bdev_ch->io_outstanding seems to be fine [14:45:51] so the total count on the current channel lines up [14:46:01] but across the collection of channels, something is wrong [14:46:24] shared resource is across all the thread ? [14:46:36] across all channels on a given thread [14:47:25] gila: i'm still here - trying to repro on my system [14:47:52] ok [14:48:10] so it is like I open many bdevs in some thread, hence create many channels , and during these channels there are some leaking io ? [14:48:19] what is this 1GB AIO device? [14:48:46] something like that - a mismatch between submissions and completions of some sort [14:48:52] alekseymmm: that's weird. you cannot really decouple shared resource from a bdev_ch [14:49:23] if you use a channel from two threads at once somehow [14:49:28] * jimharris wonders when darsto sleeps [14:49:28] you could maybe corrupt the count? [14:49:51] darsto : This is exactly what surprised me , since I greped in code where they are increased and decreased and found out that they increased and decreased together [14:50:13] is it illigal to use a channel from 2 threads? [14:50:18] how many threads exist in your test application currently? [14:50:24] 2 fio threads [14:50:28] oh very much so - a channel must be bound to a thread [14:50:33] ^ [14:50:37] a channel is essentially a per-thread context [14:50:46] gila: what is this 1GB AIO device? [14:51:01] a regular VM disk [14:51:05] scsi [14:51:15] It seems U have to look at my code more carefully , thanks for info [14:51:23] * I have to [14:51:25] virtio-scsi? [14:51:38] or some kind of emulated scsi? [14:52:09] *** Quits: philipp-sk (~Philipp@ktnron0916w-lp130-02-76-66-162-159.dsl.bell.ca) (Quit: Leaving) [14:52:21] "Cannot open network namespace: No such file or directory" [14:52:25] a new TP intermittent failure? [14:52:31] anyone seen that? [14:52:48] @jimharris not virtrio [14:53:15] i'm using vmware fusion [14:53:19] whatever vmware uses =) [14:53:27] *** Quits: dlw (~Thunderbi@111.197.237.248) (Ping timeout: 268 seconds) [14:56:14] haven't been able to repro on my system - i'm backing AIO with an NVMe namespace and then using iSCSI loopback [14:56:59] is it a VM? [14:57:29] no - it's bare metal - i don't have fusion on my macbook (at least not yet!) [14:58:03] Ok well, i had the same issue on an other machine, which uses bhyve (BSD) [14:58:24] I've also tried AIO backed by NVMe with fusion, same issue [14:58:25] that's a good data point [14:58:48] when using the nvme bdev on the nvme device -- all is fine (same goes for malloc) [14:58:53] could you try restarting the spdk iscsi target with "-L scsi" option? [15:00:38] Yes [15:01:52] @jimharris this is right after restart and login [15:03:50] https://pastebin.com/FiM2Fqq7 [15:09:55] don't see anything obvious in there [15:10:16] Nope neither did I [15:10:33] The only thing that stands out and where I was focussed on is this: [15:10:34] could you try bdevperf - this will just run some local I/O to this AIO block device [15:10:46] [sdc] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00 [15:10:59] in dmesg [15:11:21] If I understand correctly what it tries to do -- is read the partition table [15:11:24] hence the read [15:11:37] yeah [15:12:17] it is odd that it is trying to read it over and over again [15:12:43] only when the device gets opened [15:13:13] could you try pulling latest master? bwalker pushed a patch earlier this week which affects the buffer alignment code in the bdev layer - i'd like to see if that has any effect on your system [15:13:22] i'm going to go back and try the commit you're testing with [15:14:07] its up2date :) [15:14:23] commit e152aa8e51798a090a7a7ecb2799f605df930761 [15:14:24] Author: Ziye Yang [15:14:24] meaning just now - or everything you've been sending today is with the latest? [15:14:50] everything i've been sending today is with latest but also a little older [15:14:51] gila: pull again now [15:15:06] ah! [15:15:24] sorry - for whatever reason the automatic GerritHub -> GitHub sync is busted. We're working on that [15:15:28] so I manually sync it each day [15:15:47] sounds like a job for lamda functions ;=) [15:16:01] it's supposed to just be a built-in feature of GerritHub [15:16:28] bwalker: https://review.gerrithub.io/#/c/spdk/dpdk/+/420448/ [15:16:35] Yeah gerritthub to me is very confusing ... [15:17:35] *** Joins: travis-ci (~travis-ci@ec2-54-221-101-66.compute-1.amazonaws.com) [15:17:36] (spdk/master) bdev: defer call of spdk_bdev_init_failed() when a module fails (Paul Luse) [15:17:36] Diff URL: https://github.com/spdk/spdk/compare/e152aa8e5179...18f534465ca5 [15:17:36] *** Parts: travis-ci (~travis-ci@ec2-54-221-101-66.compute-1.amazonaws.com) () [15:18:03] whee! @jimharris it works [15:18:15] I fixed it! [15:18:22] I don't know what bug it was, but I fixed it! [15:18:28] those are the best ones =) [15:18:56] I couldn't figure out why it was the way it was, so I rewrote it the way I thought it should be [15:18:59] thanks gila - you indirectly got us to the bottom of this blobstore issue too :) [15:19:27] well -- you are most well come =) [15:19:35] @bwalker i saw you removed this: [15:19:38] - aligned_buf = (void *)(((uintptr_t)buf + 511) & ~511UL); [15:19:39] - len = len - ((uintptr_t)aligned_buf - (uintptr_t)buf); [15:19:52] that indeed is what was causing the iov to have a len of 8725 [15:20:02] in my case atleast [15:20:23] I changed the iov to have the length of the "used" part of the buffer, instead of the total length of the buffer [15:20:42] the buffers come from a pool, so if you wanted a 4k buffer, you actually get an 8k + a little for alignment [15:20:56] I now set the iov element length to 4k [15:21:04] whereas before it was 8k [15:21:28] but the length of an I/O shouldn't be based on the iov elements - it should be pulled from the command itself [15:21:53] I only changed it because I assumed other people would make the mistake of using the iov element length as the I/O length and wanted to head that off [15:22:11] Yes that makes sense [15:24:32] @jimharris shall i close the bug (and refer to the commit from @bwalker) ? [15:25:19] please refer to the commit - i'll add another note once the blobstore thing is fixed and committed too [15:25:27] but don't close for now [15:25:52] ok [15:27:25] well thanks guys, it was fun. Going to bed its past midnight, cheers! [15:27:41] thanks for the help [15:33:46] hmmm... where have I seen someone bitten by that iov length thing recently ;) [15:48:09] *** Quits: Guest22237 (~lyan@2605:a000:160e:2dd:4a4d:7eff:fef2:eea3) (Quit: Leaving) [15:50:42] jimharris: an intermittent failure that might be related to dpdk 18.05 update - https://ci.spdk.io/spdk-jenkins/results/autotest-per-patch/builds/8304/archive/ubuntu17_autotest/build.log [15:59:54] yeah- i've seen this pop up a few times [16:31:56] bwalker , you were right about my issue with io_outstanding , correct usage of spdk_thread_send_msg helped me. Thanks a lot [16:34:39] sethhowe: fedora-04 is down in chandler test pool [16:39:29] *** Quits: alekseymmm (bcf3adf1@gateway/web/freenode/ip.188.243.173.241) (Quit: Page closed) [16:42:08] jimharris: thx [16:56:36] jimharris: I forgot to change the bios setting when I swapped out that machine a couple months ago. It's fixed now. [16:58:58] darsto: can you dig into that intermittent failure some more tomorrow? [17:00:07] yeah [17:00:47] can i get a script running and stress the TP over the night? [17:02:42] i think it would be ok to use ubuntu16.04 [17:02:49] from the chandler test pool [17:02:59] you'll need to talk to sethhowe for logistics [17:03:38] i could just push new patchsets over and over [17:03:57] sethhowe: this seems eerily similar to an issue we were debugging a while back on those ubuntu systems [17:04:25] yeah - maybe comment out all of the tests except for the ones needed to repro this failure, so they run through the test pool a lot faster [17:38:14] *** Joins: dlw (~Thunderbi@171.82.178.135) [17:51:13] *** Quits: dlw (~Thunderbi@171.82.178.135) (Ping timeout: 248 seconds) [17:53:40] *** Quits: peter_turschm (~peter_tur@66.193.132.66) (Remote host closed the connection) [20:04:12] *** Joins: dlw (~Thunderbi@114.255.44.143)