LVM2.2.02.132

[Fernando de Oliveira]

​ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.132.tgz

​ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.132.tgz.asc

​ftp://sources.redhat.com/pub/lvm2/md5.sum

80af5af726949bbbb2aceb15b24b7d20 LVM2.2.02.132.tgz

​ftp://sources.redhat.com/pub/lvm2/WHATS_NEW

Version 2.02.132 - 22nd September 2015
======================================
 • Fix lvmconf to set locking_type=2 if external locking library is
   requested.
 • Remove verbose message when rescanning an unchanged device.
   (2.02.119)
 • Add origin_uuid, mirror_log_uuid, move_pv_uuid, convert_lv_uuid
   report fields.
 • Add pool_lv_uuid, metadata_lv_uuid, data_lv_uuid reporting fields.
 • Fix PV label processing failure after pvcreate in lvm shell with
   lvmetad.

Version 2.02.131 - 15th September 2015
======================================
 • Rename 'make install_full_man' to install_all_man and add all_man
 •  target.
 • Fix vgimportclone cache_dir path name (2.02.115).
 • Swapping of LV identifiers handles more complex LVs.
 • Use passed list of PVS when allocating space in lvconvert --thinpool.
 • Disallow usage of --stripe and --stripesize when creating cache pool.
 • Warn user when caching raid or thin pool data LV.
 • When layering LV, move LV flags with segments.
 • Ignore persistent cache if configuration changed. (2.02.127)
 • Fix devices/filter to be applied before disk-accessing filters.
 •  (2.02.112)
 • Make tags only when requested via 'make tags'.
 • Configure supports --disable-dependency-tracking for one-time builds.
 • Fix usage of configure.h when building in srcdir != builddir.

​ftp://sources.redhat.com/pub/lvm2/WHATS_NEW_DM

Version 1.02.109 - 22nd September 2016
======================================
 • Update man pages for dmsetup and dmstats.
 • Improve help text for dmsetup.
 • Use --noflush and --nolockfs when removing device with --force.
 • Parse new Overflow status string for snapshot target.
 • Check dir path components are valid if using dm_create_dir, error out
   if not.
 • Fix /dev/mapper handling to remove dangling entries if symlinks are
   found.
 • Make it possible to use blank value as selection for string list
   report field.

Version 1.02.108 - 15th September 2015
=====================================
 • Do not check for full thin pool when activating without messages
 •  (1.02.107).

[Fernando de Oliveira]

New point release 2.02.132

[Pierre Labastie]

Tests seem to hang at lvcreate-large-raid.sh (more than 45 mn without activity). This is with kernel-4.2.3. Trying 4.1.10.

[Fernando de Oliveira]

Unfortunately, I cannot test in 7.8. For 7.7, linux-4.1.6, version 130 and three others before, I get them skipped:

$ xzgrep lvcreate-large-raid.sh \
    lvm2-2.02.130-make-k-check-2015.09.07-11h46m06s.log.xz
###      skipped: [ndev-vanilla] shell/lvcreate-large-raid.sh
###      skipped: [ndev-cluster] shell/lvcreate-large-raid.sh

[Pierre Labastie]

Thanks, that's good information. Guess we'd have to compare our kernel configurations. Actually, with 4.1.10, I left the test running (or stalled) for 3 hours, and when I came back, I had to hard reset the VM where I do the tests. I'll try with 130. I'll let you know the result.

[Pierre Labastie]

Again, machine stalled, unresponsive. Problem is: it may also come from qemu. I've no way to try on bare metal right now. I'll do the other updates first, then build lfs-7.8 on my real machine.

[Pierre Labastie]

Tests completed on a real machine, but took 19 SBU! lvcreate-large-raid.sh has not been skipped, but failed. Looking at those tests, I am not sure they mean so much. They pretend to create and use 256TB logical volumes on loopbask devices whose size is ~3GB... I'll leave Fernando's figures for the tests, and just add that they may hang under some conditions, depending on kernel options and whether a real or a virtual machine is used.

[Pierre Labastie]

I see that my test results are much worse than those previously reported (version 2.0.2.125 by Bruce). What does it mean? I'll update the book for now, but it might be interesting to have independant testing.

[bdubbs@…]

Is there an easy way to disable the lvcreate-large-raid.sh test?

[Pierre Labastie]

Problem is not only with lvcreate-large-raid.sh: Bruce's results for 2.02.125:

448 tests: 141 passed, 302 skipped, 5 broken, 0 failed

Mine for 2.02.132:

460 tests: 121 passed, 307 skipped, 21 broken, 11 failed

[Pierre Labastie]

I've commited the update at r16510. Will leave open for now, until we reach a conclusion about those tests...

[Fernando de Oliveira]

My results with lvm2-2.02.130:

### 452 tests: 133 passed, 293 skipped, 26 broken, 0 failed

[Pierre Labastie]

I understand that all the test results except mine are for LFS-7.7, aren't they? What I fear is that the test failures do not come from LVM per se, but from some other component. My first thought is udev, then the kernel. I'll build a new VM with older udev and kernel and try again. But of course, another possibility is on my side: didn't I miss some essential configuration item in the kernel?

[bdubbs@…]

I tried the tests on the system and it really got fouled up. It was doing something in the kernel (migration, whatever that is) and even top didn't show anything significant that I could tell.

I even did a reboot and it hung trying to stop dbus. So I did a reset via switch. I then changed my lvm partitions to regular linux partitions and created a new lvm partition.

Then I ran:

    ./configure --prefix=/usr       \
                --exec-prefix=      \
                --with-confdir=/etc \
                --enable-applib     \
                --enable-cmdlib     \
                --enable-pkgconfig  \
                --enable-udev_sync && 
    make -j1                   &&

    # Must be root for checks
    rm test/shell/lvcreate-large-raid.sh  &&
    sudo make -k check_system

and got:

### 458 tests: 140 passed, 312 skipped, 6 broken, 0 failed
make[1]: Leaving directory '/tmp/lvm2/LVM2.2.02.132/test'
334.7 Elapsed Time -  LVM2.2.02.132
SBU=3.598
1820 /usr/src/lvm2/LVM2.2.02.132.tgz size (1.777 MB)
36160 kilobytes build size (35.312 MB)
md5sum : 80af5af726949bbbb2aceb15b24b7d20  /usr/src/lvm2/LVM2.2.02.132.tgz

Then I ran 'sudo make -k check_local' and got:

### 229 tests: 141 passed, 82 skipped, 6 broken, 0 failed

I did not try to run check_cluster because it didn't seem to be appropriate.

There are a couple of things we do not build: BUILD_LVMETAD and BUILD_LVMPOLLD that would generate additional tests.

I do not think we need to try to run the lvcreate-large-raid.sh test.

Running a full 'make -k check' without logging gives a better output. The tests say running: and there is a timer clock. When the test is done, the 'running:' label is changed to passed:, skipped:, or warnings: as appropriate.

All the cluster checks were skipped and I got:

### 458 tests: 141 passed, 311 skipped, 6 broken, 0 failed

There are results in the test/results/ directory, but they don't have meaning for me.

[Pierre Labastie]

So it seems that removing the offending test might prevent the others to fail? I'll test tomorrow (was busy with the GIMP stack).

[Pierre Labastie]

Tried a couple of things, but no joy...

• Remove test/shell/lvcreate-large-raid.sh. Got:

  458 tests: 121 passed, 306 skipped, 21 broken, 10 failed
  

• Created an ext4 filesystem on a regular partition (not LVM). Mounted /tmp on it. Exactly the same result.
• I noticed that my kernel options had:

  # CONFIG_MD_LINEAR is not set
  # CONFIG_MD_RAID0 is not set
  CONFIG_MD_RAID1=y
  CONFIG_MD_RAID10=y
  CONFIG_MD_RAID456=y
  CONFIG_MD_MULTIPATH=y
  

So I decided to set both MD_LINEAR and MD_RAID0. No change...

• There is no file telling what failed. I tried:

  grep -r failed: test/results
  

Got:

ndev-vanilla:shell_lock-blocking.sh.txt:[ 0:10]   @TESTDIR@/var/lock/lvm/P_orphans: flock failed: Resource temporarily unavailable
ndev-vanilla:shell_pvcreate-operation.sh.txt:[ 0:01]   @PREFIX@.mybackupfile: stat failed: No such file or directory
ndev-vanilla:shell_pvcreate-operation.sh.txt:[ 0:01]   @PREFIX@.mybackupfile: stat failed: No such file or directory
ndev-vanilla:shell_pvcreate-operation.sh.txt:[ 0:01]   @PREFIX@.mybackupfile: stat failed: No such file or directory
ndev-vanilla:shell_lvcreate-repair.sh.txt:[ 0:01] # device-mapper: create ioctl failed: Device or resource busy
ndev-vanilla:shell_lvcreate-repair.sh.txt:[ 0:01]   device-mapper: create ioctl on @PREFIX@vg-lvol0LVM-UDsVoUddAI18tR1FlR2PWRC1uykTVVHy07F6rBR1t5shmMVyH3gc5LhjQ9E0QlSY failed: Device or resource busy
ndev-vanilla:shell_lvcreate-repair.sh.txt:## DEBUG: ioctl/libdm-iface.c:1870   device-mapper: create ioctl on @PREFIX@vg-lvol0LVM-UDsVoUddAI18tR1FlR2PWRC1uykTVVHy07F6rBR1t5shmMVyH3gc5LhjQ9E0QlSY failed: Device or resource busy
ndev-vanilla:shell_mirror-vgreduce-removemissing.sh.txt:[ 0:04] 27,8949,1426140495,-;udevd[7041]: inotify_add_watch(7, /dev/dm-13, 10) failed: No such file or directory
ndev-vanilla:shell_mirror-vgreduce-removemissing.sh.txt:[ 0:06] 27,8957,1428159816,-;udevd[7041]: inotify_add_watch(7, /dev/dm-12, 10) failed: No such file or directory
ndev-vanilla:shell_activate-missing.sh.txt:[ 0:00] 27,909,235863666,-;udevd[19276]: inotify_add_watch(7, /dev/dm-12, 10) failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: /etc/machine-id: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: /etc/machine-id: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: /etc/machine-id: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01] WARNING: /etc/machine-id: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: /etc/machine-id: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: /etc/machine-id: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01] WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01] WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_system_id.sh.txt:[ 0:01]   WARNING: etc/lvm_test.conf: fopen failed: No such file or directory
ndev-vanilla:shell_lvchange-syncaction-raid.sh.txt:[ 0:03] # device-mapper: message ioctl on  failed: Device or resource busy
ndev-vanilla:shell_lvchange-syncaction-raid.sh.txt:[ 0:06] # device-mapper: message ioctl on  failed: Device or resource busy
ndev-vanilla:shell_lvchange-syncaction-raid.sh.txt:[ 0:06]   device-mapper: message ioctl on LVM-Nl5za6CMkW20k8fAMaHrcVndTH5Szt37e6WFM30fo6KTrsSwXPyRn2Uac1YgQhOX failed: Device or resource busy
ndev-vanilla:shell_lvchange-syncaction-raid.sh.txt:## DEBUG: ioctl/libdm-iface.c:1870   device-mapper: message ioctl on LVM-Nl5za6CMkW20k8fAMaHrcVndTH5Szt37e6WFM30fo6KTrsSwXPyRn2Uac1YgQhOX failed: Device or resource busy
ndev-vanilla:shell_lvchange-syncaction-raid.sh.txt:## Line: 38   # device-mapper: message ioctl on  failed: Device or resource busy
ndev-vanilla:shell_lvchange-syncaction-raid.sh.txt:## Line: 66   # device-mapper: message ioctl on  failed: Device or resource busy
ndev-vanilla:shell_name-mangling.sh.txt:[ 0:00] device-mapper: reload ioctl on @PREFIX@abc failed: Invalid argument

Note that the /etc/machine-id, which is warned as missing, is a systemd thing...

I found in /test/shell/lvchange-syncaction-raid.sh:

# FIXME
# Some delay - there is currently race in upstream kernel
# Some delay - there is currently race in upstream kernel
# test may occasionaly fail with:
# device-mapper: message ioctl on  failed: Device or resource busy
#
# Heinz's kernel seems to fix this particular issue but
# has some other problem for now

More in a moment (some real job duty now).

[Pierre Labastie]

Moved the LFS root from LVM to regular partition. Exactly the same errors again. The only thing I have not tried: deactivating all LVM partititions (vgchange -an). I that last test fails, will (try to) improve the wording for tests and close the ticket.

[Pierre Labastie]

Found a little more information:

• the test/results/list file contains a summary of the results.
• the tests can use reiserfs and which (will add to optional deps).
• thin-provisioning-tools may be used at build and run time (will add to optional deps too).
• I am not sure an LVM partition (type 8E00) is needed anymore. Everything is done in loop devices anyway.

On another machine, got 458 tests: 139 passed, 312 skipped, 6 broken, 1 failed. Also, the timing is around 10 SBU, much larger than Bruce's above...

[Pierre Labastie]

Progress report (if it is possible to talk about progress):

• I have no machine without LVM partitions. I have tried to test on a virtual machine with only regular partitions (/dev/sda1: ext2, /dev/sda2: ext4, /dev/sda3: ext4, / on /dev/sda3, /boot on /dev/sda1, /dev/sda2 not mounted). I had to remove lvcreate-large-raid.sh, but got another hanging test: snapshot-remove-dmsetup.sh. Will try without this test, too.
• The machine where tests are better has a slower CPU, but has SCSI disks, while the one for bad tests has a faster CPU (SBU is 100 instead of 160), and SATA disks. Don't know whether this is relevant.
• The "ndev" tests mean that udev is not used. To check using udev, the command is make check_system

[Pierre Labastie]

(Small) Progress again (?):

• On the virtual machine described at comment:19, I got only two failed tests, after removing shell/snapshot-remove-dmsetup.sh".
• Tried a virgin installation of lfs (SVN) on /dev/sda2 on the same VM. All tests timeout (after 180 s)!!! Reason: if udev is installed, the tests wait for udev synchronization when they manipulate the LV's. But since LVM is not installed yet, udev rules are not installed too. So udev does not do the synchronization and the tests wait forever! (Am I allowed to hate upstream?). The workaround is to install first the package (at least the udev rules, dmsetup and libdevmapper).

[Pierre Labastie]

BTW, did I write that it seems that there is no need for partition with 8E00 label?

[Fernando de Oliveira]

Yes. I suspected it, because I never had it during the tests, but my knowledge about this is just to update (running the tests) and test if it is working.

I wrote in one ticket about this. Without complaints I went on.

I am very thankful for this work you are doing. It will become easier to update, after this work.

[bdubbs@…]

There are a couple of packages where we do not run the tests until after they are installed. It sounds like that would be the easiest way to solve this problem.

I agree with Fernando. Your efforts on this ticket are much appreciated.

[Pierre Labastie]

Thanks to you both for your support. I think I've reached a conclusion for now. At several places, the tests use the target DM_DELAY (I/O delaying target), which I had not selected on some machines, but was selected on others (the ones with good tests results). The help for this switch says:

CONFIG_DM_DELAY:

A target that delays reads and/or writes and can send
them to different devices.  Useful for testing.

If unsure, say N.

Well, using something which is useful for testing when doing tests is not totally wrong, actually. What is wrong is that the tests are marked failed when that something is not there...

So you have to select almost all the targets (either as modules or inline). I select everything except the ones marked "EXPERIMENTAL". Then, removing lvcreate-large-raid.sh, and shell/snapshot-remove-dmsetup.sh, and running make check_local (the -k is not needed), I get on all machines I have tried:

228 tests: 148 passed, 47 skipped, 31 broken, 2 failed

Note that removing the Thin provisioning target decreases the number of broken tests and increases the number of skipped ones.

Note also that make check_local runs the same tests as make check, except that it does not try to run ndev-cluster tests, which are all skipped eventually.

[Pierre Labastie]

Two more things:

• adding:

  --with-thin-check=    \
  --with-thin-dump=     \
  --with-thin-repair=   \
  --with-thin-restore=  \
  --with-cache-check=   \
  --with-cache-dump=    \
  --with-cache-repair=  \
  --with-cache-restore= \
  

to configure switches removes a lot of broken tests:

228 tests: 173 passed, 47 skipped, 6 broken, 2 failed

• The snapshot-remove-dmsetup.sh test was added in September. Commits seem to tell that it is not ready yet... For example, the last commit log begins with(​https://git.fedorahosted.org/cgit/lvm2.git/commit/test/shell/snapshot-remove-dmsetup.sh?id=4c2cc782aa8c7e6bdc18339215f65e82ea1489ae):

  This test is somewhat 'stupid' and simulates mostly unsupported situation [...]
  

Note that this commit prevents the test from blocking (it is just broken).

[Pierre Labastie]

I have tried to improve the page, respective to tests, and also added some information in the introduction, at r16565. Closing now, but feel free to comment: I would agree that the test section is too verbose...

[Fernando de Oliveira]

I dont think it is verbose, after all complications that you fixed. On th contrary, I would like to see more info, e.g. from comment:25.

I mean, after all work you have done, more info of what you found would be beneficial to users and to developers.

Thanks, again.

[Pierre Labastie]

Do you mean I could write about the --with-thin_check= etc, switches?

[Fernando de Oliveira]

Yes, wouldn't be useful? I may be wrong.

[Pierre Labastie]

I am not very good at knowing what is useful or not. Those switches work around a defect in some tests, which issue a warning when they discover that thin-provisionning-tools is not installed, although they do not need those tools.

I would agree with comment:29 that using those switches would be useful for testing, but I am not sure how to mention that in the text. Will try to commit something, but please review.

[Fernando de Oliveira]

OK, thanks. Will check after you do, with pleasure.

[Pierre Labastie]

New commit at r16573, thanks for review.

[Fernando de Oliveira]

Very good, thanks!

I have one question, though.

Can we use those options since the beginning, for installing, or is it a two step process? My understanding was that if one doesn't have the thin-provisioning-tools, which seems to be the usual situation for BLFS, then just include since the beginning. Is that correct?

[Pierre Labastie]

Replying to fo:

Very good, thanks!

I have one question, though.

Can we use those options since the beginning, for installing, or is it a two step process? My understanding was that if one doesn't have the thin-provisioning-tools, which seems to be the usual situation for BLFS, then just include since the beginning. Is that correct?

Yes that is correct. Those switches are not needed for using LVM, but they do not hurt, and they improve test results. Thanks again for all your questions.