MySQL test failures and charset support

[alexander@…]

Currently, MySQL instructions disable certain tests:

for TESTFILE in information_schema innodb join mysql_client_test \
                ps_2myisam ps_3innodb ps_4heap ps_5merge \
                sp-error strict type_float type_ranges
do
  mv mysql-test/t/$TESTFILE.test mysql-test/t/$TESTFILE.test.disabled
done
unset TESTFILE

However, all tests (including the tests listed above) pass without any errors when one adds --with-extra-charsets=all to the configure line. This switch is also necessary if one wants to push UTF-8 data into MySQL tables as strings and return correctly-sorted results.

[alexander@…]

Sorry, the statement that all tests passed in fact applies to an older gcc3-based system. MySQL compiled on modern systems has these failures:

• ctype_ucs_binlog: unexplained, there's some evidence suggesting that an external library (glibc?) is at fault (a binary passing this test on the old system fails it when copied to the new system)
• mysql_client_test: gcc4 issue, can be fixed by recompiling with gcc-3.4.6, see ​http://bugs.mysql.com/bug.php?id=16287
• the rest of failures go away after recompilation --with-extra-charsets=all

Also, it may be interesting to run "make test-force" instead of "make test".

[archaic@…]

In response to each of Alex's comments:

The testsuite is supposed to run 2 passes. make test-force will only allow a given "pass" of the testsuite to finish. The second pass still will not run if there are any errors in the first pass (but then running pass2 would be rather pointless anyway).

As for glibc (or any other external library) problems, this one is either not true, or the external library isn't the real fault and was instead miscompiled by gcc-4.0.3.

--with-extra-charsets=all is still needed regardless of gcc version.

I built 2 identical udev_update systems save for one difference. The 2nd system I used gcc-3.4.6. With that, the mysql testsuite runs to completion with zero errors.

Where to go from here, or which package (gcc or mysql) is at fault is beyond my level of experience. mysql devs say it's a gcc problem. (See link in previous comment.)

A google search for '(int) i8_data == rc' yields similar bug reports from various distros (including LFS).

As for ctype_ucs_binlog (which is a charset failure), if glibc is the culprit, then evidence seems to support that gcc-4.0.3 is miscompiling glibc.

[archaic@…]

Tested 5.0.20. Things got worse. For each matching test run between 5.0.19 and 5.0.20, the latter had more failures or more severe failures. Here's the comparison:

5.0.19-pass1 of testsuite (added --with-extra charsets=all to book instructions)

ctype_ucs_binlog mysql_client_test mysqlcheck

5.0.19-pass2 of testsuite (added --with-extra charsets=all to book instructions)

ctype_ucs_binlog join mysql_client_test mysqlcheck ps_2myisam ps_3innodb ps_4heap ps_5merge sp-error strict type_float type_newdecimal type_ranges union

5.0.19-pass1 of testsuite (by the book)

information_schema innodb mysql mysql_client_test mysqlbinlog mysqlcheck mysqldump mysqlshow null_key

5.0.19-pass2 of testsuite (by the book)

information_schema innodb join mysql mysql_client_test mysqlbinlog mysqlcheck mysqldump mysqlshow null_key ps_2myisam ps_3innodb ps_4heap ps_5merge sp-error strict type_float type_newdecimal type_ranges union

5.0.20-pass1 of testsuite (added --with-extra charsets=all to book instructions)

ctype_ucs_binlog mysql_client_test mysqlcheck mysqldump mysqlshow

5.0.20-pass2 of testsuite (added --with-extra charsets=all to book instructions)

ctype_ucs_binlog join mysql_client_test mysqlcheck mysqldump mysqlshow ps_2myisam ps_3innodb ps_4heap ps_5merge sp-error strict type_float type_newdecimal type_ranges union

5.0.20-pass1 of testsuite (by the book)

ERROR: /home/archaic/mysql-5.0.20-2/mysql-test/var/run/master.pid was not created in 400 seconds; Aborting

5.0.20-pass2 of testsuite (by the book)

ERROR: /home/archaic/mysql-5.0.20-2/mysql-test/var/run/master.pid was not created in 400 seconds; Aborting

5.0.19 and 5.0.20 using svn trunk with a single difference (downgrade to gcc-3.4.6) and with --with-extra-charsets=all produces a flawless test run on both passes.

[bdubbs@…]

A couple of discoveries... I was able to get all the tests to pass using gcc 4.0.2 with:

CFLAGS="-O3 -DPIC -fPIC -DUNDEF_HAVE_INITGROUPS -fno-strict-aliasing" \
CXXFLAGS="-O3 -fno-strict-aliasing -felide-constructors -fno-exceptions -fno-rtti -fPIC -DPIC -DUNDEF_HAVE_INITGROUPS" \
    ./configure --prefix=/usr \
                --sysconfdir=/etc \
                --libexecdir=/usr/sbin \
                --localstatedir=/srv/mysql \
                --enable-thread-safe-client \
                --enable-assembler \
                --enable-local-infile \
                --with-unix-socket-path=/var/run/mysql/mysql.sock \
                --without-debug \
                --without-bench \
                --without-readline \
                --with-extra-charsets=all

I found this on the mysql site via google.

The default FLAGS are:

CFLAGS = -O3 -DDBUG_OFF
CXXFLAGS = -O3 -DDBUG_OFF -fno-implicit-templates -fno-exceptions -fno-rtti

Some other observations:

1. The tests seem to be processor independent. My activity monitors stay quite low during the tests. top was only showing loads of 0.25 to 0.35.

2. There are optional dependencies that are not listed. In particular, BerkeleyDB. Looking at my log, I see:

   checking for BerkeleyDB... no
   Not using Berkeley DB
   checking for Innodb... Using Innodb
   checking for NDB Cluster... Not using NDB Cluster
   checking for MySQL federated storage engine... no
   checking for yaSSL... no
   checking for OpenSSL... no
   

I do have BerkeleyDB and OpenSSL installed.

It looks like we need to pass --with-openssl to configure for OpenSSL and --with-berkeley-db for bdb support. They do not ssem to be automatically detected.

3. We mention Libedit which is a non-GPL replacement for the readline library. I believe we ran into a problem with this before. Do we want to remove it or mention that it is not recommended?

[alexander@…]

Today I got a report via ICQ that MySQL fails to build. It appeared to be related to the fact that the (clueless) user installed libedit. So the libedit dependency should be just dropped as it is simply impossible to build MySQL against it.

[alexander@…]

I will retest this with GCC-4.0.3 soon and either confirm or deny the CFLAGS effect. Please wait a bit...

[Randy McMurchy]

As far as OpenSSL support, this is no different than 100 other packages in the book (perhapss 200) that you look at configure --help, determine what you have installed, and what you want MySQL linked to and add whatever appropriate switches as desired. We certainly can't put openssl as a default switch, but I suppose we could list it in the command explanation if you feel it necessary. I don't think so, because I expect folks to use --help.

As far as BDB, you could spend a few hours and then discover MySQL just won't use any recent version. Perhaps you could patch the sources and try that, but is is worth it? It uses an internal copy of BDB if it can't use the system copy.

[bdubbs@…]

You are right about OpenSSL support.

I'm not sure about bdb support however. When --with-berkeley-db is not passed, none of the bdb tests are run.

[alexander@…]

All 374 tests were successful with Bruce's instructions and GCC-4.0.3

[dnicholson@…]

1. That's great that the tests are successful with those settings. Someone should ping upstream about this situation. Also, it would be nice to see if Archaic sees the same behavior.

2. It's policy that BDB isn't mentioned as a dependency anymore, remember?

[dnicholson@…]

Oh, I forgot

3. Drop libedit. It's been dropped for other packages like bc, so there should be some consistency. I think we should unceremoniously remove its mention from the book.

[bdubbs@…]

Fixed at revision 5959.

[archaic@…]

Just replying to Dan for archival purposes. At least part of Alex's testing was done on the same box where we initially discovered this problem, so if he says it is good, I'd tend to trust him.