Review test failures before release

[Bruce Dubbs]

See ​http://lists.linuxfromscratch.org/pipermail/lfs-dev/2020-August/074060.html

[thomas]

Here a test run on a i3. All packages built without any optimization flags set (unset CFLAGS; unset CXXFLAGS; unset MAKEFLAGS), gcc including languages c,c++,d. Arch x86_64, host ArchLinux:

802-tcl, 803-expect, 804-dejagnu

No failures

806-glibc

FAIL: io/tst-lchmod

Summary of test results:
      1 FAIL
   4234 PASS
     29 UNSUPPORTED
     17 XFAIL
      2 XPASS

807-zlib, 809-xz, 811-file, 813-m4, 814-bc, 815-flex, 816-binutils, 817-gmp, 818-mpfr, 819-mpc, 820-attr, 822-libcap

No failures

824-gcc

			=== g++ tests ===

Running target unix
FAIL: g++.dg/asan/asan_test.C   -O2  (test for excess errors)
UNRESOLVED: g++.dg/asan/asan_test.C   -O2  compilation failed to produce executable
FAIL: g++.dg/coroutines/torture/co-ret-17-void-ret-coro.C execution test
FAIL: g++.dg/coroutines/torture/pr95519-05-gro.C execution test
FAIL: g++.dg/coroutines/torture/co-ret-17-void-ret-coro.C   -O0  execution test
FAIL: g++.dg/coroutines/torture/co-ret-17-void-ret-coro.C   -O1  execution test
FAIL: g++.dg/coroutines/torture/co-ret-17-void-ret-coro.C   -O2  execution test
FAIL: g++.dg/coroutines/torture/co-ret-17-void-ret-coro.C   -O3 -g  execution test
FAIL: g++.dg/coroutines/torture/co-ret-17-void-ret-coro.C   -Os  execution test
FAIL: g++.dg/coroutines/torture/co-ret-17-void-ret-coro.C   -O2 -flto -fno-use-linker-plugin -flto-partition=none  execution test
FAIL: g++.dg/coroutines/torture/co-ret-17-void-ret-coro.C   -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  execution test
FAIL: g++.dg/coroutines/torture/pr95519-05-gro.C   -O0  execution test
FAIL: g++.dg/coroutines/torture/pr95519-05-gro.C   -O1  execution test
FAIL: g++.dg/coroutines/torture/pr95519-05-gro.C   -O2  execution test
FAIL: g++.dg/coroutines/torture/pr95519-05-gro.C   -O3 -g  execution test
FAIL: g++.dg/coroutines/torture/pr95519-05-gro.C   -Os  execution test
FAIL: g++.dg/coroutines/torture/pr95519-05-gro.C   -O2 -flto -fno-use-linker-plugin -flto-partition=none  execution test
FAIL: g++.dg/coroutines/torture/pr95519-05-gro.C   -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  execution test

		=== gcc tests ===

Running target unix
FAIL: gcc.dg/asan/pr80166.c   -O0  (test for excess errors)
FAIL: gcc.dg/asan/pr80166.c   -O1  (test for excess errors)
FAIL: gcc.dg/asan/pr80166.c   -O2  (test for excess errors)
FAIL: gcc.dg/asan/pr80166.c   -O3 -g  (test for excess errors)
FAIL: gcc.dg/asan/pr80166.c   -Os  (test for excess errors)
FAIL: gcc.dg/asan/pr80166.c   -O2 -flto -fno-use-linker-plugin -flto-partition=none  (test for excess errors)
FAIL: gcc.dg/asan/pr80166.c   -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  (test for excess errors)

	=== libstdc++ tests ===

Running target unix
FAIL: 22_locale/time_get/get_time/char/2.cc execution test
FAIL: 22_locale/time_get/get_time/char/wrapped_env.cc execution test
FAIL: 22_locale/time_get/get_time/char/wrapped_locale.cc execution test
FAIL: 22_locale/time_get/get_time/wchar_t/2.cc execution test
FAIL: 22_locale/time_get/get_time/wchar_t/wrapped_env.cc execution test
FAIL: 22_locale/time_get/get_time/wchar_t/wrapped_locale.cc execution test

Summary:
- g++      17
- gcc       7
- libstdc++ 6

825-pkg-config, 827-sed, 829-gettext, 830-bison, 831-grep, 832-bash

No failures

833-libtool

...
Standalone Libltdl.

123: compiling softlinked libltdl                    FAILED (standalone.at:35)
124: compiling copied libltdl                        FAILED (standalone.at:50)
125: installable libltdl                             FAILED (standalone.at:67)
126: linking libltdl without autotools               FAILED (standalone.at:85)

Subproject Libltdl.

...
130: linking libltdl without autotools               FAILED (subproject.at:115)
...

834-gdbm, 835-gperf, 836-expat, 837-inetutils, 838-perl, 839-xml-parser, 840-intltool

No failures

841-autoconf

ERROR: 450 tests were run,
137 failed (4 expected failures).
53 tests were skipped.

Many failures; known as testsuite is broken

842-automake, 844-libelf, 845-libffi, 846-openssl

No failures

847-python

no testsuite ran by scripts

848-ninja

No failures

850-coreutils

...
PASS: test-getloadavg
PASS: test-getlogin
PASS: test-getndelim2
...
No failures   (also test-getlogin passed)

851-check, 852-diffutils, 853-gawk, 854-findutils, 858-gzip, 860-kbd, 861-libpipeline, 862-make, 863-patch, 864-man-db

No failures

865-tar

223: capabilities: binary store/restore              FAILED (capabs_raw01.at:28)

866-texinfo, 867-vim,

No failures

868-udev

FAIL: udev-test.pl

869-procps-ng, 870-util-linux, 871-e2fsprogs

No failures

util-linux:
...
       column: columnate: [01] fill-cols-80                  ... OK
       column: invalid multibyte                             ... OK
       column: multiple files                                ... OK
...

[Bruce Dubbs]

I agree with all the above with the exception of 868-eudev. In that package I have:

PASS: udev-test.pl
PASS: rules-test.sh
============================================================================
Testsuite summary for eudev 3.2.9
============================================================================
# TOTAL: 2
# PASS:  2
...

At one point we had in the book:

sed -r -i 's|/usr(/bin/test)|\1|' test/udev-test.pl

but I didn't have that when I ran my tests.

I think what we have now is good enough for release. Closing the ticket.