Parallel make for "openssl" package

[yuksiy]

I suggest the following change in ​BLFS 2016-03-02 "OpenSSL-1.0.2g".
(If there are any mistakes in my English, please correct them.)

In "Installation of OpenSSL":

+ Note 1
+ This package does not support parallel test.

+ Note 2
+ This package may install broken shared libraries during the install phase when using -jN. (N>=2)

-make test
+make -j1 test

-make MANDIR=/usr/share/man MANSUFFIX=ssl install &&
+make -j1 MANDIR=/usr/share/man MANSUFFIX=ssl install &&

The reasons are as follows.

About addition of Note 1 and change to "make -j1 test"

The command-line for test was written as "make -j1 test" in ​BLFS 7.8 "OpenSSL-1.0.2d",
But "-j1" was deleted in r16708.
Because the commit message does not mention this deletion, I don't understand the reason about this deletion.

But "openssl-1.0.2g/PROBLEMS" file describes as follows.

* Parallell make leads to errors
(snip)
The solution is simple for now: don't run parallell make when testing.

About addition of Note 2 and change to "make -j1 (snip) install"

I noticed accidentally once, but parallel install of the openssl package seems to cause the problem that installed shared libraries are often broken (= 0 bytes) even if "make -jN (snip) install (N>=2)" finished with a status of zero.

The instruction to reproduce the symptom is as follows.

cd /sources

cat > openssl_build_1.sh << "EOF"
cd /sources                    &&
rm -fr   openssl-1.0.2g        &&
tar -xvf openssl-1.0.2g.tar.gz &&
cd       openssl-1.0.2g        &&
./config --prefix=/usr         \
         --openssldir=/etc/ssl \
         --libdir=lib          \
         shared                \
         zlib-dynamic          &&
make -j4                       &&
make -j4 INSTALL_PREFIX=`pwd`/destdir MANDIR=/usr/share/man MANSUFFIX=ssl install \
  && echo "NO ERROR" \
  || echo "ERROR"
ls -Al ./destdir/usr/lib/*.so.*
EOF

sh openssl_build_1.sh

When I tried the instruction mentioned above, the problem occurred three of five times.
(See file size in TRY2, TRY4, TRY5.)

# TRY1
NO ERROR
-r-xr-xr-x 1 root root 2702392 Mar  3 18:43 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root  516896 Mar  3 18:43 ./destdir/usr/lib/libssl.so.1.0.0

# TRY2
NO ERROR
-r-xr-xr-x 1 root root 2702392 Mar  3 18:46 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root       0 Mar  3 18:46 ./destdir/usr/lib/libssl.so.1.0.0

# TRY3
NO ERROR
-r-xr-xr-x 1 root root 2696848 Mar  3 18:50 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root  516896 Mar  3 18:50 ./destdir/usr/lib/libssl.so.1.0.0

# TRY4
NO ERROR
-r-xr-xr-x 1 root root 2702392 Mar  3 18:53 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root       0 Mar  3 18:53 ./destdir/usr/lib/libssl.so.1.0.0

# TRY5
NO ERROR
-r-xr-xr-x 1 root root      0 Mar  3 18:55 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root 516896 Mar  3 18:55 ./destdir/usr/lib/libssl.so.1.0.0

Then, I changed "j4" to "j1" and retried.

cat openssl_build_1.sh \
  | sed '/make -j4 INSTALL_PREFIX/ s,j4,j1,' \
  > openssl_build_2.sh

sh openssl_build_2.sh

When I tried the instruction mentioned above, no problem occurred at least within five times.

# TRY1
NO ERROR
-r-xr-xr-x 1 root root 2702392 Mar  3 19:06 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root  516896 Mar  3 19:06 ./destdir/usr/lib/libssl.so.1.0.0

# TRY2
NO ERROR
-r-xr-xr-x 1 root root 2702392 Mar  3 19:09 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root  516896 Mar  3 19:09 ./destdir/usr/lib/libssl.so.1.0.0

# TRY3
NO ERROR
-r-xr-xr-x 1 root root 2702392 Mar  3 19:11 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root  516896 Mar  3 19:11 ./destdir/usr/lib/libssl.so.1.0.0

# TRY4
NO ERROR
-r-xr-xr-x 1 root root 2702392 Mar  3 19:13 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root  516896 Mar  3 19:13 ./destdir/usr/lib/libssl.so.1.0.0

# TRY5
NO ERROR
-r-xr-xr-x 1 root root 2702392 Mar  3 19:16 ./destdir/usr/lib/libcrypto.so.1.0.0
-r-xr-xr-x 1 root root  516896 Mar  3 19:16 ./destdir/usr/lib/libssl.so.1.0.0

[ken@…]

I was under the impression that we have recently only been using -j1 for openssl, but a quick glance at the page shows that is no longer mentioned. I do not have time to dig into why -j1 was removed from the book at the moment (it might even have been an accident), but I support reinstating -j1 for this package.

A long while ago (the early versions of 1.0.2) we had a patch to support parallel builds, but it did not allow parallel tests : at that time, specifying -j1 for the tests made sense.

But then we dropped the patch (people still had problems with parallel builds) and used -j1 for make. If a package specifies -j1 for make, assume that is required for tests and install. So it made sense to remove the second -j1 from the tests. Actually, parallel install is generally untested.

I will leave this for the moment to see if anybody disagrees about parallel builds, but for the moment I think we should revert to -j1 for 7.9.

[Fernando de Oliveira]

I removed for version openssl-1.0.2e. Unfortunately, we both forgot to check for 1.0.2f and g.

However, a cannot confirm that make -jN install (N > 1) gives me any problem.

Whatever you decide I agree.

[ken@…]

On my A10 I was running a 4.5-rc kernel (just finished bisecting a problem there, and reverted one commit) and at first I thought I had a new kernel issue - using make -j5 the CPU percentage seemed to max out at 100%. But I've now rebooted to 4.4.2 and I think they must have changed the Makefile - for the first part of the build it mostly runs at 100% CPU or less, but later there are some times when it briefly touches 200%, maybe a little more, no sign of anything close to 400% (I only have 4 cores). Both runs apparently ended normally.

So, make -jN appears to be ok, therefore there is no need to change the book unless somebody can break the build on a powerful machine.

For the tests, either they work or they don't, and if they don't the answer is to not run parallel jobs. There have been one or two packages where parallel make for the tests is mentioned, for others such as this where we do not specify anything then 'make test' or 'make check' without specifying multiple jobs is what I would expect people to run.

For make -jN install : anybody doing that is either braver than I am, or likes to learn how to unpick damage.

[Fernando de Oliveira]

LOL.

Yes, I agree. Have noticed in the LXQt sensors that the overuse of CPU, (not in numbers, just graphics).

For the tests, they don't work in parallel.

For make -jN install, I think I am braver than you (funny, I thought I was never braver than anybody). But it is not a problem that you state explicitly make -j1 install (due to problems in some systems).

Again, I am more than happy with the way you write the page and close this ticket. And more than thankful that you took it to solve. :-)

[bdubbs@…]

Just a general comment. It is a rare package where the install part of the package takes a significant amount of time. I always use -j1 for installs.

[bdubbs@…]

Fixed at revision 17090.