source: introduction/important/building-notes.xml@ c6c57f7

plabs/python-mods qt5new trunk
Last change on this file since c6c57f7 was c6c57f7, checked in by Pierre Labastie <pierre.labastie@…>, 4 months ago

Use an <informaltable> for explaining CMake flags

  • Property mode set to 100644
File size: 43.5 KB
Line 
1<?xml version="1.0" encoding="ISO-8859-1"?>
2<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
3 "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
4 <!ENTITY % general-entities SYSTEM "../../general.ent">
5 %general-entities;
6]>
7
8<sect1 id="unpacking">
9 <?dbhtml filename="notes-on-building.html"?>
10
11 <sect1info>
12 <date>$Date$</date>
13 </sect1info>
14
15 <title>Notes on Building Software</title>
16
17 <para>Those people who have built an LFS system may be aware
18 of the general principles of downloading and unpacking software. Some
19 of that information is repeated here for those new to building
20 their own software.</para>
21
22 <para>Each set of installation instructions contains a URL from which you
23 can download the package. The patches; however, are stored on the LFS
24 servers and are available via HTTP. These are referenced as needed in the
25 installation instructions.</para>
26
27 <para>While you can keep the source files anywhere you like, we assume that
28 you have unpacked the package and changed into the directory created by the
29 unpacking process (the 'build' directory). We also assume you have
30 uncompressed any required patches and they are in the directory immediately
31 above the 'build' directory.</para>
32
33 <para>We can not emphasize strongly enough that you should start from a
34 <emphasis>clean source tree</emphasis> each time. This means that if
35 you have had an error during configuration or compilation, it's usually
36 best to delete the source tree and
37 re-unpack it <emphasis>before</emphasis> trying again. This obviously
38 doesn't apply if you're an advanced user used to hacking
39 <filename>Makefile</filename>s and C code, but if in doubt, start from a
40 clean tree.</para>
41
42 <sect2>
43 <title>Building Software as an Unprivileged (non-root) User</title>
44
45 <para>The golden rule of Unix System Administration is to use your
46 superpowers only when necessary. Hence, BLFS recommends that you
47 build software as an unprivileged user and only become the
48 <systemitem class='username'>root</systemitem> user when installing the
49 software. This philosophy is followed in all the packages in this book.
50 Unless otherwise specified, all instructions should be executed as an
51 unprivileged user. The book will advise you on instructions that need
52 <systemitem class='username'>root</systemitem> privileges.</para>
53
54 </sect2>
55
56 <sect2>
57 <title>Unpacking the Software</title>
58
59 <para>If a file is in <filename class='extension'>.tar</filename> format
60 and compressed, it is unpacked by running one of the following
61 commands:</para>
62
63<screen><userinput>tar -xvf filename.tar.gz
64tar -xvf filename.tgz
65tar -xvf filename.tar.Z
66tar -xvf filename.tar.bz2</userinput></screen>
67
68 <note>
69 <para>You may omit using the <option>v</option> parameter in the commands
70 shown above and below if you wish to suppress the verbose listing of all
71 the files in the archive as they are extracted. This can help speed up the
72 extraction as well as make any errors produced during the extraction
73 more obvious to you.</para>
74 </note>
75
76 <para>You can also use a slightly different method:</para>
77
78<screen><userinput>bzcat filename.tar.bz2 | tar -xv</userinput></screen>
79
80 <para>Finally, you sometimes need to be able to unpack patches which are
81 generally not in <filename class='extension'>.tar</filename> format. The
82 best way to do this is to copy the patch file to the parent of the 'build'
83 directory and then run one of the following commands depending on whether
84 the file is a <filename class='extension'>.gz</filename> or <filename
85 class='extension'>.bz2</filename> file:</para>
86
87<screen><userinput>gunzip -v patchname.gz
88bunzip2 -v patchname.bz2</userinput></screen>
89
90 </sect2>
91
92 <sect2>
93 <title>Verifying File Integrity</title>
94
95 <para>Generally, to verify that the downloaded file is complete,
96 many package maintainers also distribute md5sums of the files. To verify the
97 md5sum of the downloaded files, download both the file and the
98 corresponding md5sum file to the same directory (preferably from different
99 on-line locations), and (assuming <filename>file.md5sum</filename> is the
100 md5sum file downloaded) run the following command:</para>
101
102<screen><userinput>md5sum -c file.md5sum</userinput></screen>
103
104 <para>If there are any errors, they will be reported. Note that the BLFS
105 book includes md5sums for all the source files also. To use the BLFS
106 supplied md5sums, you can create a <filename>file.md5sum</filename> (place
107 the md5sum data and the exact name of the downloaded file on the same
108 line of a file, separated by white space) and run the command shown above.
109 Alternately, simply run the command shown below and compare the output
110 to the md5sum data shown in the BLFS book.</para>
111
112<screen><userinput>md5sum <replaceable>&lt;name_of_downloaded_file&gt;</replaceable></userinput></screen>
113
114 <para>MD5 is not cryptographically secure, so the md5sums are only
115 provided for detecting unmalicious changes to the file content. For
116 example, an error or truncation introduced during network transfer, or
117 a <quote>stealth</quote> update to the package from the upstream
118 (updating the content of a released tarball instead of making a new
119 release properly).</para>
120
121 <para>There is no <quote>100%</quote> secure way to make
122 sure the genuity of the source files. Assuming the upstream is managing
123 their website correctly (the private key is not leaked and the domain is
124 not hijacked), and the trust anchors have been set up correctly using
125 <xref linkend="make-ca"/> on the BLFS system, we can reasonably trust
126 download URLs to the upstream official website
127 <emphasis role="bold">with https protocol</emphasis>. Note that
128 BLFS book itself is published on a website with https, so you should
129 already have some confidence in https protocol or you wouldn't trust the
130 book content.</para>
131
132 <para>If the package is downloaded from an unofficial location (for
133 example a local mirror), checksums generated by cryptographically secure
134 digest algorithms (for example SHA256) can be used to verify the
135 genuity of the package. Download the checksum file from the upstream
136 <emphasis role="bold">official</emphasis> website (or somewhere
137 <emphasis role="bold">you can trust</emphasis>) and compare the
138 checksum of the package from unofficial location with it. For example,
139 SHA256 checksum can be checked with the command:</para>
140
141 <note>
142 <para>If the checksum and the package are downloaded from the same
143 untrusted location, you won't gain security enhancement by verifying
144 the package with the checksum. The attacker can fake the checksum as
145 well as compromising the package itself.</para>
146 </note>
147
148<screen><userinput>sha256sum -c <replaceable>file</replaceable>.sha256sum</userinput></screen>
149
150 <para>If <xref linkend="gnupg2"/> is installed, you can also verify the
151 genuity of the package with a GPG signature. Import the upstream GPG
152 public key with:</para>
153
154<screen><userinput>gpg --recv-key <replaceable>keyID</replaceable></userinput></screen>
155
156 <para><replaceable>keyID</replaceable> should be replaced with the key ID
157 from somewhere <emphasis role="bold">you can trust</emphasis> (for
158 example, copy it from the upstream official website using https). Now
159 you can verify the signature with:</para>
160
161<screen><userinput>gpg --recv-key <replaceable>file</replaceable>.sig <replaceable>file</replaceable></userinput></screen>
162
163 <para>The advantage of <application>GnuPG</application> signature is,
164 once you imported a public key which can be trusted, you can download
165 both the package and its signature from the same unofficial location and
166 verify them with the public key. So you won't need to connect to the
167 official upstream website to retrieve a checksum for each new release.
168 You only need to update the public key if it's expired or revoked.
169 </para>
170
171 </sect2>
172
173 <sect2>
174 <title>Creating Log Files During Installation</title>
175
176 <para>For larger packages, it is convenient to create log files instead of
177 staring at the screen hoping to catch a particular error or warning. Log
178 files are also useful for debugging and keeping records. The following
179 command allows you to create an installation log. Replace
180 <replaceable>&lt;command&gt;</replaceable> with the command you intend to execute.</para>
181
182<screen><userinput>( <replaceable>&lt;command&gt;</replaceable> 2&gt;&amp;1 | tee compile.log &amp;&amp; exit $PIPESTATUS )</userinput></screen>
183
184 <para><option>2&gt;&amp;1</option> redirects error messages to the same
185 location as standard output. The <command>tee</command> command allows
186 viewing of the output while logging the results to a file. The parentheses
187 around the command run the entire command in a subshell and finally the
188 <command>exit $PIPESTATUS</command> command ensures the result of the
189 <replaceable>&lt;command&gt;</replaceable> is returned as the result and not the
190 result of the <command>tee</command> command.</para>
191
192 </sect2>
193
194 <sect2 id="parallel-builds" xreflabel="Using Multiple Processors">
195 <title>Using Multiple Processors</title>
196
197 <para>For many modern systems with multiple processors (or cores) the
198 compilation time for a package can be reduced by performing a "parallel
199 make" by either setting an environment variable or telling the make program
200 how many processors are available. For instance, a Core2Duo can support two
201 simultaneous processes with: </para>
202
203 <screen><userinput>export MAKEFLAGS='-j2'</userinput></screen>
204
205 <para>or just building with:</para>
206
207 <screen><userinput>make -j2</userinput></screen>
208
209 <para>
210 If you have applied the optional <command>sed</command> when building
211 <application>ninja</application> in LFS, you can use:
212 </para>
213
214 <screen><userinput>export NINJAJOBS=2</userinput></screen>
215
216 <para>
217 when a package uses <command>ninja</command>, or just:
218 </para>
219
220 <screen><userinput>ninja -j2</userinput></screen>
221
222 <para>
223 but for ninja, the default number of jobs is &lt;N&gt;+2, where &lt;N&gt;
224 is the number of processors available, so that using the above commands
225 is rather for limiting the number of jobs (see below for why this could
226 be necessary).
227 </para>
228
229 <para>Generally the number of processes should not exceed the number of
230 cores supported by the CPU. To list the processors on your
231 system, issue: <userinput>grep processor /proc/cpuinfo</userinput>.
232 </para>
233
234 <para>In some cases, using multiple processes may result in a 'race'
235 condition where the success of the build depends on the order of the
236 commands run by the <command>make</command> program. For instance, if an
237 executable needs File A and File B, attempting to link the program before
238 one of the dependent components is available will result in a failure.
239 This condition usually arises because the upstream developer has not
240 properly designated all the prerequisites needed to accomplish a step in the
241 Makefile.</para>
242
243 <para>If this occurs, the best way to proceed is to drop back to a
244 single processor build. Adding '-j1' to a make command will override
245 the similar setting in the <envar>MAKEFLAGS</envar> environment
246 variable.</para>
247
248 <note><para>When running the package tests or the install portion of the
249 package build process, we do not recommend using an option greater than
250 '-j1' unless specified otherwise. The installation procedures or checks
251 have not been validated using parallel procedures and may fail with issues
252 that are difficult to debug.</para></note>
253
254 <important>
255 <para>
256 Another problem may occur with modern CPU's, which have a lot of cores.
257 Each job started consumes memory, and if the sum of the needed
258 memory for each job exceeds the available memory, you may encounter
259 either an OOM (Out of Memory) kernel interrupt or intense swapping
260 that will slow the build beyond reasonable limits.
261 </para>
262
263 <para>
264 Some compilations with <command>g++</command> may consume up to 2.5 GB
265 of memory, so to be safe, you should restrict the number of jobs
266 to (Total Memory in GB)/2.5, at least for big packages such as LLVM,
267 WebKitGtk, QtWebEngine, or libreoffice.
268 </para>
269 </important>
270 </sect2>
271
272 <sect2 id="automating-builds" xreflabel="Automated Building Procedures">
273 <title>Automated Building Procedures</title>
274
275 <para>There are times when automating the building of a package can come in
276 handy. Everyone has their own reasons for wanting to automate building,
277 and everyone goes about it in their own way. Creating
278 <filename>Makefile</filename>s, <application>Bash</application> scripts,
279 <application>Perl</application> scripts or simply a list of commands used
280 to cut and paste are just some of the methods you can use to automate
281 building BLFS packages. Detailing how and providing examples of the many
282 ways you can automate the building of packages is beyond the scope of this
283 section. This section will expose you to using file redirection and the
284 <command>yes</command> command to help provide ideas on how to automate
285 your builds.</para>
286
287 <bridgehead renderas="sect3">File Redirection to Automate Input</bridgehead>
288
289 <para>You will find times throughout your BLFS journey when you will come
290 across a package that has a command prompting you for information. This
291 information might be configuration details, a directory path, or a response
292 to a license agreement. This can present a challenge to automate the
293 building of that package. Occasionally, you will be prompted for different
294 information in a series of questions. One method to automate this type of
295 scenario requires putting the desired responses in a file and using
296 redirection so that the program uses the data in the file as the answers to
297 the questions.</para>
298
299 <para>Building the <application>CUPS</application> package is a good
300 example of how redirecting a file as input to prompts can help you automate
301 the build. If you run the test suite, you are asked to respond to a series
302 of questions regarding the type of test to run and if you have any
303 auxiliary programs the test can use. You can create a file with your
304 responses, one response per line, and use a command similar to the
305 one shown below to automate running the test suite:</para>
306
307<screen><userinput>make check &lt; ../cups-1.1.23-testsuite_parms</userinput></screen>
308
309 <para>This effectively makes the test suite use the responses in the file
310 as the input to the questions. Occasionally you may end up doing a bit of
311 trial and error determining the exact format of your input file for some
312 things, but once figured out and documented you can use this to automate
313 building the package.</para>
314
315 <bridgehead renderas="sect3">Using <command>yes</command> to Automate
316 Input</bridgehead>
317
318 <para>Sometimes you will only need to provide one response, or provide the
319 same response to many prompts. For these instances, the
320 <command>yes</command> command works really well. The
321 <command>yes</command> command can be used to provide a response (the same
322 one) to one or more instances of questions. It can be used to simulate
323 pressing just the <keycap>Enter</keycap> key, entering the
324 <keycap>Y</keycap> key or entering a string of text. Perhaps the easiest
325 way to show its use is in an example.</para>
326
327 <para>First, create a short <application>Bash</application> script by
328 entering the following commands:</para>
329
330<screen><userinput>cat &gt; blfs-yes-test1 &lt;&lt; "EOF"
331<literal>#!/bin/bash
332
333echo -n -e "\n\nPlease type something (or nothing) and press Enter ---> "
334
335read A_STRING
336
337if test "$A_STRING" = ""; then A_STRING="Just the Enter key was pressed"
338else A_STRING="You entered '$A_STRING'"
339fi
340
341echo -e "\n\n$A_STRING\n\n"</literal>
342EOF
343chmod 755 blfs-yes-test1</userinput></screen>
344
345 <para>Now run the script by issuing <command>./blfs-yes-test1</command> from
346 the command line. It will wait for a response, which can be anything (or
347 nothing) followed by the <keycap>Enter</keycap> key. After entering
348 something, the result will be echoed to the screen. Now use the
349 <command>yes</command> command to automate the entering of a
350 response:</para>
351
352<screen><userinput>yes | ./blfs-yes-test1</userinput></screen>
353
354 <para>Notice that piping <command>yes</command> by itself to the script
355 results in <keycap>y</keycap> being passed to the script. Now try it with a
356 string of text:</para>
357
358<screen><userinput>yes 'This is some text' | ./blfs-yes-test1</userinput></screen>
359
360 <para>The exact string was used as the response to the script. Finally,
361 try it using an empty (null) string:</para>
362
363<screen><userinput>yes '' | ./blfs-yes-test1</userinput></screen>
364
365 <para>Notice this results in passing just the press of the
366 <keycap>Enter</keycap> key to the script. This is useful for times when the
367 default answer to the prompt is sufficient. This syntax is used in the
368 <xref linkend="net-tools-automate-example"/> instructions to accept all the
369 defaults to the many prompts during the configuration step. You may now
370 remove the test script, if desired.</para>
371
372 <bridgehead renderas="sect3">File Redirection to Automate Output</bridgehead>
373
374 <para>In order to automate the building of some packages, especially those
375 that require you to read a license agreement one page at a time, requires
376 using a method that avoids having to press a key to display each page.
377 Redirecting the output to a file can be used in these instances to assist
378 with the automation. The previous section on this page touched on creating
379 log files of the build output. The redirection method shown there used the
380 <command>tee</command> command to redirect output to a file while also
381 displaying the output to the screen. Here, the output will only be sent to
382 a file.</para>
383
384 <para>Again, the easiest way to demonstrate the technique is to show an
385 example. First, issue the command:</para>
386
387<screen><userinput>ls -l /usr/bin | more</userinput></screen>
388
389 <para>Of course, you'll be required to view the output one page at a time
390 because the <command>more</command> filter was used. Now try the same
391 command, but this time redirect the output to a file. The special file
392 <filename>/dev/null</filename> can be used instead of the filename shown,
393 but you will have no log file to examine:</para>
394
395<screen><userinput>ls -l /usr/bin | more &gt; redirect_test.log 2&gt;&amp;1</userinput></screen>
396
397 <para>Notice that this time the command immediately returned to the shell
398 prompt without having to page through the output. You may now remove the
399 log file.</para>
400
401 <para>The last example will use the <command>yes</command> command in
402 combination with output redirection to bypass having to page through the
403 output and then provide a <keycap>y</keycap> to a prompt. This technique
404 could be used in instances when otherwise you would have to page through
405 the output of a file (such as a license agreement) and then answer the
406 question of <quote>do you accept the above?</quote>. For this example,
407 another short <application>Bash</application> script is required:</para>
408
409<screen><userinput>cat &gt; blfs-yes-test2 &lt;&lt; "EOF"
410<literal>#!/bin/bash
411
412ls -l /usr/bin | more
413
414echo -n -e "\n\nDid you enjoy reading this? (y,n) "
415
416read A_STRING
417
418if test "$A_STRING" = "y"; then A_STRING="You entered the 'y' key"
419else A_STRING="You did NOT enter the 'y' key"
420fi
421
422echo -e "\n\n$A_STRING\n\n"</literal>
423EOF
424chmod 755 blfs-yes-test2</userinput></screen>
425
426 <para>This script can be used to simulate a program that requires you to
427 read a license agreement, then respond appropriately to accept the
428 agreement before the program will install anything. First, run the script
429 without any automation techniques by issuing
430 <command>./blfs-yes-test2</command>.</para>
431
432 <para>Now issue the following command which uses two automation techniques,
433 making it suitable for use in an automated build script:</para>
434
435<screen><userinput>yes | ./blfs-yes-test2 &gt; blfs-yes-test2.log 2&gt;&amp;1</userinput></screen>
436
437 <para>If desired, issue <command>tail blfs-yes-test2.log</command> to see
438 the end of the paged output, and confirmation that <keycap>y</keycap> was
439 passed through to the script. Once satisfied that it works as it should,
440 you may remove the script and log file.</para>
441
442 <para>Finally, keep in mind that there are many ways to automate and/or
443 script the build commands. There is not a single <quote>correct</quote> way
444 to do it. Your imagination is the only limit.</para>
445
446 </sect2>
447
448 <sect2>
449 <title>Dependencies</title>
450
451 <para>For each package described, BLFS lists the known dependencies.
452 These are listed under several headings, whose meaning is as follows:</para>
453
454 <itemizedlist>
455 <listitem>
456 <para><emphasis>Required</emphasis> means that the target package
457 cannot be correctly built without the dependency having first been
458 installed.</para>
459 </listitem>
460 <listitem>
461 <para><emphasis>Recommended</emphasis> means that BLFS strongly
462 suggests this package is installed first for a clean and trouble-free
463 build, that won't have issues either during the build process, or at
464 run-time. The instructions in the book assume these packages are
465 installed. Some changes or workarounds may be required if these
466 packages are not installed.</para>
467 </listitem>
468 <listitem>
469 <para><emphasis>Optional</emphasis> means that this package might be
470 installed for added functionality. Often BLFS will describe the
471 dependency to explain the added functionality that will result.</para>
472 </listitem>
473 </itemizedlist>
474
475 </sect2>
476
477 <sect2 id="package_updates">
478 <title>Using the Most Current Package Sources</title>
479
480 <para>On occasion you may run into a situation in the book when a package
481 will not build or work properly. Though the Editors attempt to ensure
482 that every package in the book builds and works properly, sometimes a
483 package has been overlooked or was not tested with this particular version
484 of BLFS.</para>
485
486 <para>If you discover that a package will not build or work properly, you
487 should see if there is a more current version of the package. Typically
488 this means you go to the maintainer's web site and download the most current
489 tarball and attempt to build the package. If you cannot determine the
490 maintainer's web site by looking at the download URLs, use Google and query
491 the package's name. For example, in the Google search bar type:
492 'package_name download' (omit the quotes) or something similar. Sometimes
493 typing: 'package_name home page' will result in you finding the
494 maintainer's web site.</para>
495
496 </sect2>
497
498 <sect2 id="stripping">
499 <title>Stripping One More Time</title>
500
501 <para>
502 In LFS, stripping of debugging symbols and unneeded symbol table
503 entries was discussed a couple of times. When building BLFS packages,
504 there are generally no special instructions that discuss stripping
505 again. Stripping can be done while installing a package, or
506 afterwards.
507 </para>
508
509 <bridgehead renderas="sect3" id="stripping-install">Stripping while Installing a Package</bridgehead>
510
511 <para>
512 There are several ways to strip executables installed by a
513 package. They depend on the build system used (see below <link
514 linkend="buildsystems">the section about build systems</link>),
515 so only some
516 generalities can be listed here:
517 </para>
518
519 <note>
520 <para>
521 The following methods using the feature of a building system
522 (autotools, meson, or cmake) will not strip static libraries if any
523 is installed. Fortunately there are not too many static libraries
524 in BLFS, and a static library can always be stripped safely by
525 running <command>strip --strip-unneeded</command> on it manually.
526 </para>
527 </note>
528
529 <itemizedlist>
530 <listitem>
531 <para>
532 The packages using autotools usually have an
533 <parameter>install-strip</parameter> target in their generated
534 <filename>Makefile</filename> files. So installing stripped
535 executables is just a matter of using
536 <command>make install-strip</command> instead of
537 <command>make install</command>.
538 </para>
539 </listitem>
540 <listitem>
541 <para>
542 The packages using the meson build system can accept
543 <parameter>-Dstrip=true</parameter> when running
544 <command>meson</command>. If you've forgot to add this option
545 running the <command>meson</command>, you can also run
546 <command>meson install --strip</command> instead of
547 <command>ninja install</command>.
548 </para>
549 </listitem>
550 <listitem>
551 <para>
552 <command>cmake</command> generates
553 <parameter>install/strip</parameter> targets for both the
554 <parameter>Unix Makefiles</parameter> and
555 <parameter>Ninja</parameter> generators (the default is
556 <parameter>Unix Makefiles</parameter> on linux). So just run
557 <command>make install/strip</command> or
558 <command>ninja install/strip</command> instead of the
559 <command>install</command> counterparts.
560 </para>
561 </listitem>
562 <listitem>
563 <para>
564 Removing (or not generating) debug symbols can also be
565 achieved by removing the
566 <parameter>-g&lt;something&gt;</parameter> options
567 in C/C++ calls. How to do that is very specific for each
568 package. And, it does not remove unneeded symbol table entries.
569 So it will not be explained in detail here. See also below
570 the paragraphs about optimization.
571 </para>
572 </listitem>
573 </itemizedlist>
574
575 <bridgehead renderas="sect3" id="stripping-installed">Stripping Installed Executables</bridgehead>
576
577 <para>
578 The <command>strip</command> utility changes files in place, which
579 may break anything using it if it is loaded in memory. Note that
580 if a file is in use but just removed from the disk (i.e. not overwritten
581 nor modified), this is not a problem since the kernel can use
582 <quote>deleted</quote> files.
583 Look at <filename>/proc/*/maps</filename>, and it is likely that
584 you'll see some <emphasis>(deleted)</emphasis> entries. The
585 <command>mv</command> just removes the destination file from the
586 directory but does not touch its content, so that it satisfies the
587 condition for the kernel to use the old (deleted) file. The script
588 below is just an example. Feel free to add stronger error detection,
589 other directories to scan, etcetera. It should be run as the &root; user:
590 </para>
591
592<screen role="root"><userinput>{ find /usr/lib -type f \( -name \*.a -o \
593 \( -name \*.so* ! -name \*dbg \) \)
594 find /usr/{bin,sbin,libexec} -type f; } | while read file; do
595 if readelf -h $file >/dev/null 2>&amp;1; then
596 cp -a $file ${file}.tmp &amp;&amp;
597 strip --strip-unneeded ${file}.tmp &amp;&amp;
598 mv ${file}.tmp $file
599 fi
600 done</userinput></screen>
601
602 <para>
603 If you install programs in other directories such as <filename
604 class="directory">/opt</filename> or <filename
605 class="directory">/usr/local</filename>, you may want to strip the files
606 there too.
607 </para>
608
609 <para>
610 For more information on stripping, see <ulink
611 url="https://www.technovelty.org/linux/stripping-shared-libraries.html"/>.
612 </para>
613
614 </sect2>
615
616<!--
617 <sect2 id="libtool">
618 <title>Libtool files</title>
619
620 <para>
621 One of the side effects of packages that use Autotools, including
622 libtool, is that they create many files with an .la extension. These
623 files are not needed in an LFS environment. If there are conflicts with
624 pkgconfig entries, they can actually prevent successful builds. You
625 may want to consider removing these files periodically:
626 </para>
627
628<screen><userinput>find /lib /usr/lib -not -path "*Image*" -a -name \*.la -delete</userinput></screen>
629
630 <para>
631 The above command removes all .la files with the exception of those that
632 have <quote>Image</quote> or <quote>openldap</quote> as a part of the
633 path. These .la files are used by the ImageMagick and openldap programs,
634 respectively. There may be other exceptions by packages not in BLFS.
635 </para>
636
637 </sect2>
638-->
639 <sect2 id="buildsystems">
640 <title>Working with different build systems</title>
641
642 <para>
643 There are now three different build systems in common use for
644 converting C or C++ source code into compiled programs or
645 libraries and their details (particularly, finding out about available
646 options and their default values) differ. It may be easiest to understand
647 the issues caused by some choices (typically slow execution or
648 unexpected use of, or omission of, optimizatons) by starting with
649 the CFLAGS and CXXFLAGS environment variables. There are also some
650 programs which use rust.
651 </para>
652
653 <para>
654 Most LFS and BLFS builders are probably aware of the basics of CFLAGS
655 and CXXFLAGS for altering how a program is compiled. Typically, some
656 form of optimization is used by upstream developers (-O2 or -O3),
657 sometimes with the creation of debug symbols (-g), as defaults.
658 </para>
659
660 <para>
661 If there are contradictory flags (e.g. multiple different -O values),
662 the <emphasis>last</emphasis> value will be used. Sometimes this means
663 that flags specified in environment variables will be picked up before
664 values hardcoded in the Makefile, and therefore ignored. For example,
665 where a user specifies '-O2' and that is followed by '-O3' the build will
666 use '-O3'.
667 </para>
668
669 <para>
670 There are various other things which can be passed in CFLAGS or
671 CXXFLAGS, such as forcing compilation for a specific microarchitecture
672 (e.g. -march=amdfam10, -march=native) or specifying a specific standard
673 for C or C++ (-std=c++17 for example). But one thing which has now come
674 to light is that programmers might include debug assertions in their
675 code, expecting them to be disabled in releases by using -DNDEBUG.
676 Specifically, if <xref linkend="mesa"/> is built with these assertions
677 enabled, some activities such as loading levels of games can take
678 extremely long times, even on high-class video cards.
679 </para>
680
681 <bridgehead renderas="sect3" id="autotools-info">Autotools with Make</bridgehead>
682
683 <para>
684 This combination is often described as 'CMMI' (configure, make, make
685 install) and is used here to also cover the few packages which have a
686 configure script that is not generated by autotools.
687 </para>
688
689 <para>
690 Sometimes running <command>./configure --help</command> will produce
691 useful options about switches which might be used. At other times,
692 after looking at the output from configure you may need to look
693 at the details of the script to find out what it was actually searching
694 for.
695 </para>
696
697 <para>
698 Many configure scripts will pick up any CFLAGS or CXXFLAGS from the
699 environment, but CMMI packages vary about how these will be mixed with
700 any flags which would otherwise be used (<emphasis>variously</emphasis>:
701 ignored, used to replace the programmer's suggestion, used before the
702 programmer's suggestion, or used after the programmer's suggestion).
703 </para>
704
705 <para>
706 In most CMMI packages, running 'make' will list each command and run
707 it, interspersed with any warnings. But some packages try to be 'silent'
708 and only show which file they are compiling or linking instead of showing
709 the command line. If you need to inspect the command, either because of
710 an error, or just to see what options and flags are being used, adding
711 'V=1' to the make invocation may help.
712 </para>
713
714 <bridgehead renderas="sect3" id="cmake-info">CMake</bridgehead>
715
716 <para>
717 CMake works in a very different way, and it has two backends which can
718 be used on BLFS: 'make' and 'ninja'. The default backend is make, but
719 ninja can be faster on large packages with multiple processors. To
720 use ninja, specify '-G Ninja' in the cmake command. However, there are
721 some packages which create fatal errors in their ninja files but build
722 successfully using the default of Unix Makefiles.
723 </para>
724
725 <para>
726 The hardest part of using CMake is knowing what options you might wish
727 to specify. The only way to get a list of what the package knows about
728 is to run <command>cmake -LAH</command> and look at the output for that
729 default configuration.
730 </para>
731
732 <para>
733 Perhaps the most-important thing about CMake is that it has a variety
734 of CMAKE_BUILD_TYPE values, and these affect the flags. The default
735 is that this is not set and no flags are generated. Any CFLAGS or
736 CXXFLAGS in the environment will be used. If the programmer has coded
737 any debug assertions, those will be enabled unless -DNDEBUG is used.
738 The following CMAKE_BUILD_TYPE values will generate the flags shown,
739 and these will come <emphasis>after</emphasis> any flags in the
740 environment and therefore take precedence.
741 </para>
742
743 <informaltable align="center">
744 <tgroup cols="2">
745 <colspec colnum="1" align="center"/>
746 <colspec colnum="2" align="center"/>
747 <thead>
748 <row><entry>Value</entry><entry>Flags</entry></row>
749 </thead>
750 <tbody>
751 <row>
752 <entry>Debug</entry><entry><option>-g</option></entry>
753 </row>
754 <row>
755 <entry>Release</entry><entry><option>-O3 -DNDEBUG</option></entry>
756 </row>
757 <row>
758 <entry>RelWithDebInfo</entry><entry><option>-O2 -g -DNDEBUG</option></entry>
759 </row>
760 <row>
761 <entry>MinSizeRel</entry><entry><option>-Os -DNDEBUG</option></entry>
762 </row>
763 </tbody>
764 </tgroup>
765 </informaltable>
766
767 <para>
768 CMake tries to produce quiet builds. To see the details of the commands
769 which are being run, use <command>make VERBOSE=1</command> or
770 <command>ninja -v</command>.
771 </para>
772
773 <bridgehead renderas="sect3" id="meson-info">Meson</bridgehead>
774
775 <para>
776 Meson has some similarities to CMake, but many differences. To get
777 details of the defines that you may wish to change you can look at
778 <filename>meson_options.txt</filename> which is usually in the
779 top-level directory.
780 </para>
781
782 <para>
783 If you have already configured the package by running
784 <command>meson</command> and now wish to change one or more settings,
785 you can either remove the build directory, recreate it, and use the
786 altered options, or within the build directory run <command>meson
787 configure</command>, e.g. to set an option:
788 </para>
789
790<screen><userinput>meson configure -D&lt;some_option&gt;=true</userinput></screen>
791
792 <para>
793 If you do that, the file <filename>meson-private/cmd_line.txt</filename>
794 will show the <emphasis>last</emphasis> commands which were used.
795 </para>
796
797 <para>
798 Meson provides the following buildtype values, and the flags they enable
799 come <emphasis>after</emphasis> any flags supplied in the environment and
800 therefore take precedence.
801 </para>
802
803 <itemizedlist>
804 <listitem>
805 <para>plain : no added flags. This is for distributors to supply their
806 own CLFAGS, CXXFLAGS and LDFLAGS. There is no obvious reason to use
807 this in BLFS.</para>
808 </listitem>
809 <listitem>
810 <para>debug : '-g' - this is the default if nothing is specified
811 in either <filename>meson.build</filename> or the command line.
812 However it results large and slow binaries, so we should override
813 it in BLFS.</para>
814 </listitem>
815 <listitem>
816 <para>debugoptimized : '-O2 -g' : this is the default specified in
817 <filename>meson.build</filename> of some packages.</para>
818 </listitem>
819 <listitem>
820 <para>release : '-O3 -DNDEBUG' (but occasionally a package will force
821 -O2 here)</para>
822 </listitem>
823 </itemizedlist>
824
825 <para>
826 Although the 'release' buildtype is described as enabling -DNDEBUG, and all
827 CMake Release builds pass that, it has so far only been observed (in
828 verbose builds) for <xref linkend="mesa"/>. That suggests that it might
829 only be used when there are debug assertions present.
830 </para>
831
832 <para>
833 The -DNDEBUG flag can also be provided by passing
834 <command>-Db_ndebug=true</command>.
835 </para>
836
837 <para>
838 To see the details of the commands which are being run in a package using
839 meson, use 'ninja -v'.
840 </para>
841
842 <bridgehead renderas="sect3" id="rust-info">Rustc and Cargo</bridgehead>
843
844 <para>
845 Most released rustc programs are provided as crates (source tarballs)
846 which will query a server to check current versions of dependencies
847 and then download them as necessary. These packages are built using
848 <command>cargo --release</command>. In theory, you can manipulate the
849 RUSTFLAGS to change the optimize-level (default is 3, like -O3, e.g.
850 <literal>-Copt-level=3</literal>) or to force it to build for the
851 machine it is being compiled on, using
852 <literal>-Ctarget-cpu=native</literal> but in practice this seems to
853 make no significant difference.
854 </para>
855
856 <para>
857 If you find an interesting rustc program which is only provided as
858 unpackaged source, you should at least specify
859 <literal>RUSTFLAGS=-Copt-level=2</literal> otherwise it will do an
860 unoptimized compile with debug info and run <emphasis>much</emphasis>
861 slower.
862 </para>
863
864 <para>
865 The rust developers seem to assume that everyone will compile on a
866 machine dedicated to producing builds, so by default all CPUs are used.
867 This can often be worked around, either by exporting
868 CARGO_BUILD_JOBS=&lt;N&gt; or passing --jobs &lt;N&gt; to cargo. For
869 compiling rustc itself, specifying --jobs &lt;N&gt; on invocations of
870 x.py (together with the <envar>CARGO_BUILD_JOBS</envar> environment
871 variable, which looks like a "belt and braces" approach but seems to be
872 necessary) mostly works. The exception is running the tests when building
873 rustc, some of them will nevertheless use all online CPUs, at least as of
874 rustc-1.42.0.
875 </para>
876
877 </sect2>
878
879 <sect2 id="optimizations">
880 <title>Optimizing the build</title>
881
882 <para>
883 Many people will prefer to optimize compiles as they see fit, by providing
884 CFLAGS or CXXFLAGS. For an introduction to the options available with gcc
885 and g++ see <ulink
886 url="https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html"/> and <ulink
887 url="https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html"/>
888 and <command>info gcc</command>.
889
890 </para>
891
892 <para>
893 Some packages default to '-O2 -g', others to '-O3 -g', and if CFLAGS or
894 CXXFLAGS are supplied they might be added to the package's defaults,
895 replace the package's defaults, or even be ignored. There are details
896 on some desktop packages which were mostly current in April 2019 at
897 <ulink url="https://www.linuxfromscratch.org/~ken/tuning/"/> - in
898 particular, README.txt, tuning-1-packages-and-notes.txt, and
899 tuning-notes-2B.txt. The particular thing to remember is that if you
900 want to try some of the more interesting flags you may need to force
901 verbose builds to confirm what is being used.
902 </para>
903
904 <para>
905 Clearly, if you are optimizing your own program you can spend time to
906 profile it and perhaps recode some of it if it is too slow. But for
907 building a whole system that approach is impractical. In general,
908 -O3 usually produces faster programs than -O2. Specifying
909 -march=native is also beneficial, but means that you cannot move the
910 binaries to an incompatible machine - this can also apply to newer
911 machines, not just to older machines. For example programs compiled for
912 'amdfam10' run on old Phenoms, Kaveris, and Ryzens : but programs
913 compiled for a Kaveri will not run on a Ryzen because certain op-codes
914 are not present. Similarly, if you build for a Haswell not everything
915 will run on a SandyBridge.
916 </para>
917
918 <para>
919 There are also various other options which some people claim are
920 beneficial. At worst, you get to recompile and test, and then
921 discover that in your usage the options do not provide a benefit.
922 </para>
923
924 <para>
925 If building Perl or Python modules, or Qt packages which use qmake,
926 in general the CFLAGS and CXXFLAGS used are those which were used by
927 those 'parent' packages.
928 </para>
929
930 </sect2>
931
932 <sect2 id="hardening">
933 <title>Options for hardening the build</title>
934
935 <para>
936 Even on desktop systems, there are still a lot of exploitable
937 vulnerabilities. For many of these, the attack comes via javascript
938 in a browser. Often, a series of vulnerabilities are used to gain
939 access to data (or sometimes to pwn, i.e. own, the machine and
940 install rootkits). Most commercial distros will apply various
941 hardening measures.
942 </para>
943
944 <para>
945 For hardening options which are reasonably cheap, there is some
946 discussion in the 'tuning' link above (occasionally, one or more
947 of these options might be inappropriate for a package). These
948 options are -D_FORTIFY_SOURCE=2, -fstack-protector=strong, and
949 (for C++) -D_GLIBCXX_ASSERTIONS. On modern machines these should
950 only have a little impact on how fast things run, and often they
951 will not be noticeable.
952 </para>
953
954 <para>
955 In the past, there was Hardened LFS where gcc (a much older version)
956 was forced to use hardening (with options to turn some of it off on a
957 per-package basis. What is being covered here is different - first you
958 have to make sure that the package is indeed using your added flags and
959 not over-riding them.
960 </para>
961
962 <para>
963 The main distros use much more, such as RELRO (Relocation Read Only)
964 and perhaps -fstack-clash-protection. You may also encounter the
965 so-called 'userspace retpoline' (-mindirect-branch=thunk etc.) which
966 is the equivalent of the spectre mitigations applied to the linux
967 kernel in late 2018). The kernel mitigations caused a lot of complaints
968 about lost performance, if you have a production server you might wish
969 to consider testing that, along with the other available options, to
970 see if performance is still sufficient.
971 </para>
972
973 <para>
974 Whilst gcc has many hardening options, clang/LLVM's strengths lie
975 elsewhere. Some options which gcc provides are said to be less effective
976 in clang/LLVM.
977 </para>
978
979 </sect2>
980
981</sect1>
Note: See TracBrowser for help on using the repository browser.