[00:44:34] *** Joins: tomzawadzki (~tomzawadz@134.134.139.76) [01:00:35] *** Quits: ziyeyang_ (ziyeyang@nat/intel/x-kgmkykwspflkzvrs) (Remote host closed the connection) [01:10:34] *** Quits: Guest61101 (~James@2601:640:8300:10f3:1c7f:f779:d77c:8d51) (Remote host closed the connection) [01:28:54] *** Joins: gila (~gila@5ED4D9C8.cm-7-5d.dynamic.ziggo.nl) [01:57:14] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 260 seconds) [02:31:25] *** Joins: baruch (~baruch@bzq-82-81-85-138.red.bezeqint.net) [06:15:22] *** Quits: gila (~gila@5ED4D9C8.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [06:21:21] *** Joins: gila (~gila@5ED4D9C8.cm-7-5d.dynamic.ziggo.nl) [06:38:26] *** Quits: baruch (~baruch@bzq-82-81-85-138.red.bezeqint.net) (Ping timeout: 265 seconds) [07:05:57] *** Quits: tomzawadzki (~tomzawadz@134.134.139.76) (Ping timeout: 240 seconds) [07:44:10] *** Quits: gila (~gila@5ED4D9C8.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [08:01:00] *** Joins: gila (~gila@5ED4D9C8.cm-7-5d.dynamic.ziggo.nl) [08:40:34] *** Joins: lhodev (~Adium@inet-hqmc08-o.oracle.com) [08:40:39] *** Parts: lhodev (~Adium@inet-hqmc08-o.oracle.com) () [08:43:51] hmmm, my reviews query is now showing everything so either I didn't have enough coffee yesterday or something changed on the backend? [08:44:46] FYI blobstore programmers guide updated to remove metadata structure details: https://review.gerrithub.io/384118 [08:49:50] *** Joins: tomzawadzki (~tomzawadz@134.134.139.83) [08:53:12] FYI on that PG in CI, haven't seen this failure before "nl: ./test/vhost/initiator/blockdev.sh: No such file or directory" [09:22:29] *** Quits: gila (~gila@5ED4D9C8.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [09:37:53] *** Quits: tomzawadzki (~tomzawadz@134.134.139.83) (Ping timeout: 248 seconds) [10:04:00] peluse: that nl error is something going wrong with the shell script backtrace functionality we have, but the actual error is before that [10:04:23] there's currently an intermittent issue where the vhost target sometimes hangs on shutdown and causes the test to time out [10:04:50] drv, thx [10:04:58] I've rescheduled the build [10:26:47] bwalker: can you take a look at these two small patches? https://review.gerrithub.io/#/c/393680/ [10:26:52] should I be able to run just the iscsi_tgt tests via ~/spdk$ sudo test/iscsi_tgt/iscsi_tgt.sh ? [10:27:16] would like to get these in - it affects the nightly build so once the second patch goes in I need to modify the nightly cron job [10:29:39] peluse: yes [10:30:16] if you are on Fedora 26 or 27, there is a bug in the distro-packaged version of iscsiadm that causes it to crash, though [10:30:46] jimharris: done [10:30:48] ugh, OK. I'll mess with it some more as its not working for me. Ubuntu 16.04 [10:31:02] sethhowe found a patch and has it scripted in test/config/vm_setup.sh (this is what our Fedora test machines are running) [10:31:13] hm, ok, let us know what is failing - we should try to make that work out of the box [10:31:31] you can try running some of the individual iscsi_tgt/*/... test scripts [10:31:47] drv, thx will do [10:47:29] *** Joins: James (~James@2601:640:8300:10f3:1c7f:f779:d77c:8d51) [10:47:53] *** James is now known as Guest80611 [11:00:54] *** Quits: Guest80611 (~James@2601:640:8300:10f3:1c7f:f779:d77c:8d51) (Remote host closed the connection) [11:10:11] *** Joins: gila (~gila@5ED4D9C8.cm-7-5d.dynamic.ziggo.nl) [11:36:57] *** Joins: lhodev (~Adium@inet-hqmc08-o.oracle.com) [11:37:10] *** Parts: lhodev (~Adium@inet-hqmc08-o.oracle.com) () [11:44:46] *** Joins: James (~James@2601:640:8300:10f3:1c7f:f779:d77c:8d51) [11:45:10] *** James is now known as Guest62373 [12:14:34] *** Quits: Guest62373 (~James@2601:640:8300:10f3:1c7f:f779:d77c:8d51) (Remote host closed the connection) [12:32:03] drv, so I had trouble with the first iscsi test (filesystem) with iscsiadm so tried on Fedora 25 and it worked fine so tried running the rest and now get to test/iscsi_tgt/reset/reset.sh but it exits on the check for 'sg_reset' but I don't immediately see what that is or where it comes from?? [12:32:45] it's part of sg3_utils [12:33:23] hmmm - we have that in vm_setup.sh but not in pkgdep.sh [12:33:24] test/config/vm_setup.sh should install everything needed for the tests, but that's only for Fedora right now [12:33:45] is there any reason not to add sg3_utils to pkgdep.sh? [12:33:48] yeah, it isn't in pkgdep because it isn't needed to build, only for the tests - not sure how we want that to work [12:34:01] it probably doesn't hurt to add it to pkgdep, but it's not strictly a build dependency [12:34:31] maybe we can have some flags for pkgdep - install deps for build only, doc, tests, etc. [12:34:32] we have astyle and pep8 in pkgdep and those are really test tools and not used for building [12:34:40] yeah, fair enough [12:34:57] we should definitely de-dup pkgdep and vm_setup.sh [12:35:04] probably vm_setup should call pkgdep [12:37:00] for the short term, I'm OK with adding sg3_utils (and anything else needed to run the tests) to pkgdep [12:44:51] cool, thanks. [12:45:07] patch pushed [12:45:10] *** Joins: James (~James@208.185.211.2) [12:45:20] for now it only adds sg3_utils [12:45:33] *** James is now known as Guest67891 [12:45:45] there is a *lot* of stuff in vm_setup.sh that I'm not sure we want in pkgdep.sh [12:46:19] yeah, vm_setup.sh should probably still be separate, but it should be able to share the installation of the build deps at least [12:46:50] gracias, yeah when I saw the filename I assumed it was one of our things, so used to seeing sg as the prefix to our old group name :) [12:46:57] haha [12:47:05] I don't know why it is sg3_utils, either [12:47:10] were there sg1 and 2? [12:48:46] The sg3_utils and sg_utils are two closely related packages of utilities that send SCSI commands to Linux devices. As well as devices on SCSI transports (e.g. Fibre channel and the SCSI parallel interface) many other devices use SCSI command sets. Examples are ATAPI cd/dvd writers and sATA disks (typically via a translation layer or bridge device). The SCSI command set is divided into what are called "primary" commands (e.g. SPC-3) and device [12:48:47] class specific sets (e.g. SBC-2 for disks and MMC-4 for cd/dvd devices). SCSI command sets and transports definitions can be found at www.t10.org . [12:48:47] The main difference between the two packages is the linux kernel series that they target: sg_utils for the lk 2.2 (with some support for the lk 2.0 series); sg3_utils for the lk 2.4 and 2.6 series. The sg3_utils package is still in active development while sg_utils is in maintenance mode. This document describes version 1.12 for sg3_utils and version 1.02 for sg_utils. [12:49:02] http://sg.danny.cz/sg/uu_index.html [12:50:09] *** Quits: Guest67891 (~James@208.185.211.2) (Ping timeout: 264 seconds) [12:50:28] guess they skipped version 2 in order to make it appear more mature - where have I seen that done before?? [12:51:05] should have started SPDK at version 3 [12:51:19] like RSTe? [12:51:27] LOL, exactly! [12:51:56] so I get farther now but still fail running fio.py, will keep poking [12:57:11] if the test ran part of the way through, it might have left stuff in a bad state - probably worth checking if you have some leftover iSCSI configuration [12:57:14] or just reboot :) [12:57:46] the test scripts have 'trap' statements that are supposed to clean up on failure, but those really don't get exercised [13:03:24] drv: i just committed 393212 and now I see you were suggesting to not commit that patch :) [13:10:53] *** Joins: James (~James@208.185.211.2) [13:11:17] *** James is now known as Guest10742 [13:16:38] jimharris: that's ok, then Shuhei's patch will just need to be rebased [13:50:49] bwalker: can you fix this typo quick before all of your doc patches go through? https://review.gerrithub.io/#/c/392982/ [13:52:55] re-uploaded - beat all but one [15:42:05] *** Joins: wmeng (6c3c6103@gateway/web/freenode/ip.108.60.97.3) [16:06:36] *** Joins: ziyeyang_ (~ziyeyang@134.134.139.83) [16:11:22] *** Joins: Anna_ (c066cc24@gateway/web/freenode/ip.192.102.204.36) [16:30:30] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [16:40:33] Hi Jim, Daniel, I would like to revert, modify, and submit again for https://review.gerrithub.io/#/c/393582/. [16:41:30] Hi Shuhei, confirmed, it is not your fault [16:41:49] Do not need to be revert [16:42:38] Hi Ziye, thank you, I'm sorry but I could not understand well yet. [16:43:07] It is my fault due to rebasing. These days, I am doing stress test on iSCSI target: e.g., 160 connections, and each with 512KB IO SIZE with 256 QD [16:47:02] Hi Ziye, thank you. I'll stay but check the patch again anyway. Hi Jim, Daniel, please stay without anything, sorry for confusing you. [16:47:54] hi Shuhei [16:48:08] if you have a fix in mind, we can just add that on top of the current patch [16:48:18] or if you would like to revert, fix, and resubmit, that is OK too [16:48:48] I'm surprised that we do not have a test for large I/O size to catch this - maybe ziyeyang is already working on adding this to autotest :) [16:51:05] Hi @drv @shuhei, Shuhei's current patch is OK. I just did some stress tests. Due to my rebasing fault, the first test failed. [16:51:15] oh, I see [16:51:20] After my rebasing, the test passed [16:52:10] For large I/O test, we only need to change the I/O size, it seems that not a problem. But for long time stress test, I think that we should do it nightly [16:52:51] Hi Ziye, Daniel, even if Ziye's test passed, removing the code to update data_transferred for read is my fault and I would like to fix that by a new patch. [16:53:04] sure, no problem [16:53:37] Thank you, reveting is a little tricky for me and I'll create a new patch. [16:54:11] just for reference, if we wanted to revert a patch, that could be done by pushing the revert as a review to GerritHub like a normal patch [16:54:18] (but we don't need to do it in this case) [16:56:05] thanks I'll remember. [16:56:46] Hi Ziye, anyway thank you so much for your testing and notification. [17:00:39] Hi， Shuhei, thank you for the iscsi_target patches, will you add integration and component test cases to cover the changes? [17:01:12] seems that io_size=512kb qd=128 fails currently: https://ci.spdk.io/spdk/builds/review/b72fcd1204222129ac4a0022b5cf043ecfbd63ae.1515109932/ [17:01:27] I didn't try to bisect this yet, so I'm not sure if it was working before the recent patches [17:02:25] I think that some interity tests, i.e., plus verify needed [17:02:46] My test this morning is only for purely iscsi read stress test [17:03:18] yes, I think some of the nightly tests that are being ported from DTS should cover these cases, or if not, we can add them separately [17:09:15] Hi Ziye, Daniel, Anna, I pushed a patch as https://review.gerrithub.io/#/c/393713/. I'll create the test code to cover this case by another patch. Sorry for inconvenience and thank you for your support. [17:12:18] oh, I did spot one thing I wasn't sure about - since my patch got merged first, the data_transferred accumulation is now in spdk_iscsi_task_cpl() in conn.c, but I think your patch intended to remove this [17:12:41] is this still needed, or should it be removed as well as adding back the one in the read path in your new patch? [17:14:49] *** Quits: Anna_ (c066cc24@gateway/web/freenode/ip.192.102.204.36) (Ping timeout: 260 seconds) [17:15:00] Thank you. Please give a couple of minutes. I'll reply soon. [17:16:35] I noticed what was mistake. [17:17:03] I'll revise my patch but I'll explain now. [17:19:18] For write, only byte_completed should accumulate, hence in spdk_iscsi_task_cpl() data_transferred accumulation should be removed after Daniel's patch. However I removed data_transferred accumulation for read path. I did not remove data_transferred accumulation for write path in spdk_iscsi_task_cpl(). [17:21:04] Hi Daniel, I should have spent more time to prepare the patch because patch sequence by multiple committer was the first for me. [17:21:59] My explanation make sense to you? [17:29:25] Hi Daniel, Ziye, Jim, please review https://review.gerrithub.io/#/c/393713/. I hope you understand well by combining the patch 393582. [17:31:43] Shuhei: yes, I think your updated patch looks right to me [17:33:44] Hi Daniel Thanks. [17:35:32] I rebased the new 512KB/128qd fio test on your patch, so we can verify that it works again [17:36:09] *** Joins: lhodev (~Adium@66-90-218-190.dyn.grandenetworks.net) [17:45:58] looks good - the FIO test passes again after your updated patch [17:51:25] Looks OK to me [17:51:34] Hope that we can merge it soon [17:52:21] *** Quits: ziyeyang_ (~ziyeyang@134.134.139.83) (Quit: Leaving) [19:16:44] *** Joins: VItta (b7521395@gateway/web/freenode/ip.183.82.19.149) [19:51:59] *** Joins: ziyeyang_ (ziyeyang@nat/intel/x-kwwlwanytfhxsudp) [20:06:25] *** Quits: Guest10742 (~James@208.185.211.2) (Remote host closed the connection) [20:31:40] hi all.. I'm facing an issue while running 'scripts/setup.sh' command in google cloud.. [20:33:16] below is error: vitta@spdk1:~/spdk$ sudo HUGEMEM=1024 scripts/setup.sh modprobe: FATAL: Module uio_pci_generic not found in directory /lib/modules/4.13.0-1002-gcp 0000:00:03.0 (1af4 1004): virtio-pci -> uio_pci_generic scripts/setup.sh: line 27: /sys/bus/pci/drivers/uio_pci_generic/new_id: No such file or directory [20:33:39] after this, no commands works until I restart the instance [20:34:08] can anyone help me in pointing to info on setting up spdk on google cloud? [20:34:47] this is my lspci output: vitta@spdk1:~/spdk$ lspci 2 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 3 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03) 4 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 5 00:03.0 Non-VGA unclassified device: Red Hat, Inc Virtio SCSI 6 00:04.0 Non-VGA unclassified device: Red Hat, Inc Virtio SCSI 7 00:05.0 Ethernet controlle [20:35:05] VItta: we have just recently (today) merged a fix for the virtio-pci device being unbound when it is used for a mounted filesystem [20:35:07] https://github.com/spdk/spdk/commit/f869082a911e7fb5d16a58b14cd09b7e3736c29c [20:35:17] do you know if you are using an up-to-date SPDK or one before that commit? [20:36:15] I cloned spdk just few mins back, and compiled and ran that command.. [20:36:33] doing reset of my instance.. I can share git log info once it comes up [20:36:46] hmm, maybe it is not correctly detecting that the virtio device is in use, then [20:37:27] can you run 'scripts/setup.sh status'? (this should just report the detected devices, not make any changes) [20:37:40] sure.. will do that [20:38:42] this is output: itta@spdk1:~/spdk$ ./scripts/setup.sh status NVMe devices BDF Numa Node Driver name Device name I/OAT DMA BDF Numa Node Driver Name virtio BDF Numa Node Driver Name 0000:00:03.0 -1 virtio-pci 0000:00:04.0 -1 virtio-pci [20:39:03] is there a way to paste large data to irc? sorry, I'm new to irc [20:40:29] you could post it to something like https://pastebin.com/ and post the link [20:40:44] got it.. will do that.. thansk @drv [20:43:07] 'setup.sh status' output at https://pastebin.com/c4nP3qXi [20:47:03] oops, I guess it does not currently print the detected devices for virtio, one sec [20:48:12] can you download this as a file and apply with 'git apply filename.txt', then run setup.sh status again: https://pastebin.com/UD0w3GYF [20:48:30] that will let us see if it's detecting that the virtio devices are in use or not [20:52:35] will do ^^^ [21:01:39] *** Joins: James (~James@73.93.143.109) [21:02:03] *** James is now known as Guest45676 [21:03:53] output at: https://pastebin.com/wvV4VdQm [21:05:37] my git log shows at this commit: [21:05:51] commit 60f1d526052795052655d67593759eb2db043ee1 [21:05:55] Author: Shuhei Matsumoto Date: Fri Jan 5 08:58:14 2018 +0900 [21:06:17] *** Quits: Guest45676 (~James@73.93.143.109) (Ping timeout: 252 seconds) [21:06:44] hmm, that looks good, it is detecting that you have block devices on those virtio devices [21:08:20] can you also post the output of 'mount'? [21:08:36] *** Quits: wmeng (6c3c6103@gateway/web/freenode/ip.108.60.97.3) (Quit: Page closed) [21:08:40] it looks like uio_pci_generic module is not available? [21:10:31] mount output at: https://pastebin.com/6C4ZuUVK [21:11:08] ok, so sdb and sdc (the virtio device names from the status output) are not mounted, does that sound correct? [21:11:24] yes.. uio_pci_generic doesn't seem to be present.. @jimharris [21:11:39] yes right @drv [21:19:35] I ran scripts/pkgdep.sh on ubuntu 16.04 google cloud instance.. do I need to get uio_pci_generic? @jimharris [21:22:02] hi VItta - uio_pci_generic is a kernel module - looks like maybe google's ubuntu 16.04 does not include it [21:23:38] got it.. thanks @jimharris [21:23:51] looks like there is some information here: https://help.ubuntu.com/lts/serverguide/DPDK.html [21:24:02] SPDK leverages DPDK so the information there applies [21:24:38] can you check if you can install a linux-image-extra-XXX package? [21:25:50] will install that.. ^^^ [21:30:37] installed it @jimharris.. output is at: https://pastebin.com/QeVfv7dk [21:31:01] but still, setup.sh is not working.. and erroring out. after this, no commands works until I reset the instance [21:31:40] thanks for that link @jimharris.. will go thru that [21:34:02] i guess you need to install coreutils package too [21:34:24] i thought normally this was installed by default on ubuntu server but maybe not for the google cloud instances [21:34:41] drv - we should add these to our pkgdep.sh script [21:35:57] coreutils is already installed installed @jimharris [21:36:13] error that you are seeing in output is becz of setup.sh command [21:36:34] 'cd', and 'ls' commands also won't work after that error from setup.sh [21:39:05] hmm, odd, is loading the uio driver on that virtio device causing the root filesystem to stop working somehow? [21:39:36] yeah - what is sda attached to? [21:41:43] in google cloud instance, I took ubuntu 16.04, and added two extra disks (one is persistent SSD of 100G, and local scratch disk of 375, apart from 10G standard persistent disk for booting) [21:42:15] do you know how the persistent disk is provided? is it an emulated SATA controller or something like that? [21:42:37] here is output of fdisk and lsblk: https://pastebin.com/mJwrRn6Q [21:43:05] if it was virtio, I'd think we would see it in the setup.sh status output [21:43:27] I also don't know about that drv [21:46:05] lspci output: https://pastebin.com/xEkVb4ew [21:50:24] I removed persistent SSD and checked 'setup.sh status' output: https://pastebin.com/fgC957wT [21:50:47] it seems they are sharing same controller [21:51:04] should we try modifying setup.sh to avoid that controller? [21:51:15] *** Joins: Robert__ (495dd02d@gateway/web/freenode/ip.73.93.208.45) [21:51:22] sorry, if I sound dumb [21:51:32] yeah, that seems like our detection of in-use virtio devices is not sufficient [21:52:26] if you try setup.sh in that new configuration, it should probably work (since it is now showing sda, which is mounted as /) [21:52:38] ok.. will try now [21:53:16] yeah.. it worked.. it skipped for 00.03.0 [21:53:23] thanks drv [21:53:30] and thanks jimharris [21:53:37] yeah, obviously that is not the real fix, since it should work with the persistent ssd inserted, but at least it's a start [21:53:48] yeah understood [21:54:09] darsto_: if you are reading this later, check out the log above - looks like the virtio device detection in setup.sh might need some tweaks [21:54:30] true.. thats for device detection [21:54:48] but, I want to use it for persistent ssds.. becz, local scratch disks are limited [21:55:09] need to check in google cloud, how to create persistent ssds on different controller..? do I sound correct? [21:55:23] yeah, I wonder if there is a way to make the google cloud infrastructure insert it as a new controller [21:56:15] right.. will explore that.. (actually, I tried a bit, but, not found a way to add new controller..) [21:56:23] thanks once again [21:58:55] *** Quits: Robert__ (495dd02d@gateway/web/freenode/ip.73.93.208.45) (Quit: Page closed) [22:02:17] *** Joins: James (~James@73.93.143.109) [22:02:40] *** James is now known as Guest35209 [22:11:02] *** Quits: Guest35209 (~James@73.93.143.109) (Remote host closed the connection) [23:18:14] *** Quits: VItta (b7521395@gateway/web/freenode/ip.183.82.19.149) (Ping timeout: 260 seconds) [23:57:29] *** Joins: James (~James@c-67-180-75-111.hsd1.ca.comcast.net) [23:57:31] *** Quits: James (~James@c-67-180-75-111.hsd1.ca.comcast.net) (Remote host closed the connection) [23:58:13] *** Joins: James (~James@c-67-180-75-111.hsd1.ca.comcast.net) [23:58:28] *** James is now known as Guest28167