source: introduction/important/building-notes.xml@ 7e255b8

12.0 kea lazarus lxqt qt5new trunk xry111/xf86-video-removal
Last change on this file since 7e255b8 was 7e255b8, checked in by Xi Ruoyao <xry111@…>, 6 months ago

building-notes: Fix stupid typo in stripping script

I should have copied tested script file into the XML, instead of editing
it...

  • Property mode set to 100644
File size: 47.3 KB
Line 
1<?xml version="1.0" encoding="ISO-8859-1"?>
2<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
3 "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
4 <!ENTITY % general-entities SYSTEM "../../general.ent">
5 %general-entities;
6]>
7
8<sect1 id="unpacking">
9 <?dbhtml filename="notes-on-building.html"?>
10
11
12 <title>Notes on Building Software</title>
13
14 <para>Those people who have built an LFS system may be aware
15 of the general principles of downloading and unpacking software. Some
16 of that information is repeated here for those new to building
17 their own software.</para>
18
19 <para>Each set of installation instructions contains a URL from which you
20 can download the package. The patches; however, are stored on the LFS
21 servers and are available via HTTP. These are referenced as needed in the
22 installation instructions.</para>
23
24 <para>While you can keep the source files anywhere you like, we assume that
25 you have unpacked the package and changed into the directory created by the
26 unpacking process (the 'build' directory). We also assume you have
27 uncompressed any required patches and they are in the directory immediately
28 above the 'build' directory.</para>
29
30 <para>We can not emphasize strongly enough that you should start from a
31 <emphasis>clean source tree</emphasis> each time. This means that if
32 you have had an error during configuration or compilation, it's usually
33 best to delete the source tree and
34 re-unpack it <emphasis>before</emphasis> trying again. This obviously
35 doesn't apply if you're an advanced user used to hacking
36 <filename>Makefile</filename>s and C code, but if in doubt, start from a
37 clean tree.</para>
38
39 <sect2>
40 <title>Building Software as an Unprivileged (non-root) User</title>
41
42 <para>The golden rule of Unix System Administration is to use your
43 superpowers only when necessary. Hence, BLFS recommends that you
44 build software as an unprivileged user and only become the
45 <systemitem class='username'>root</systemitem> user when installing the
46 software. This philosophy is followed in all the packages in this book.
47 Unless otherwise specified, all instructions should be executed as an
48 unprivileged user. The book will advise you on instructions that need
49 <systemitem class='username'>root</systemitem> privileges.</para>
50
51 </sect2>
52
53 <sect2>
54 <title>Unpacking the Software</title>
55
56 <para>If a file is in <filename class='extension'>.tar</filename> format
57 and compressed, it is unpacked by running one of the following
58 commands:</para>
59
60<screen><userinput>tar -xvf filename.tar.gz
61tar -xvf filename.tgz
62tar -xvf filename.tar.Z
63tar -xvf filename.tar.bz2</userinput></screen>
64
65 <note>
66 <para>You may omit using the <option>v</option> parameter in the commands
67 shown above and below if you wish to suppress the verbose listing of all
68 the files in the archive as they are extracted. This can help speed up the
69 extraction as well as make any errors produced during the extraction
70 more obvious to you.</para>
71 </note>
72
73 <para>You can also use a slightly different method:</para>
74
75<screen><userinput>bzcat filename.tar.bz2 | tar -xv</userinput></screen>
76
77 <para>Finally, you sometimes need to be able to unpack patches which are
78 generally not in <filename class='extension'>.tar</filename> format. The
79 best way to do this is to copy the patch file to the parent of the 'build'
80 directory and then run one of the following commands depending on whether
81 the file is a <filename class='extension'>.gz</filename> or <filename
82 class='extension'>.bz2</filename> file:</para>
83
84<screen><userinput>gunzip -v patchname.gz
85bunzip2 -v patchname.bz2</userinput></screen>
86
87 </sect2>
88
89 <sect2>
90 <title>Verifying File Integrity</title>
91
92 <para>Generally, to verify that the downloaded file is complete,
93 many package maintainers also distribute md5sums of the files. To verify the
94 md5sum of the downloaded files, download both the file and the
95 corresponding md5sum file to the same directory (preferably from different
96 on-line locations), and (assuming <filename>file.md5sum</filename> is the
97 md5sum file downloaded) run the following command:</para>
98
99<screen><userinput>md5sum -c file.md5sum</userinput></screen>
100
101 <para>If there are any errors, they will be reported. Note that the BLFS
102 book includes md5sums for all the source files also. To use the BLFS
103 supplied md5sums, you can create a <filename>file.md5sum</filename> (place
104 the md5sum data and the exact name of the downloaded file on the same
105 line of a file, separated by white space) and run the command shown above.
106 Alternately, simply run the command shown below and compare the output
107 to the md5sum data shown in the BLFS book.</para>
108
109<screen><userinput>md5sum <replaceable>&lt;name_of_downloaded_file&gt;</replaceable></userinput></screen>
110
111 <para>MD5 is not cryptographically secure, so the md5sums are only
112 provided for detecting unmalicious changes to the file content. For
113 example, an error or truncation introduced during network transfer, or
114 a <quote>stealth</quote> update to the package from the upstream
115 (updating the content of a released tarball instead of making a new
116 release properly).</para>
117
118 <para>There is no <quote>100%</quote> secure way to make
119 sure the genuity of the source files. Assuming the upstream is managing
120 their website correctly (the private key is not leaked and the domain is
121 not hijacked), and the trust anchors have been set up correctly using
122 <xref linkend="make-ca"/> on the BLFS system, we can reasonably trust
123 download URLs to the upstream official website
124 <emphasis role="bold">with https protocol</emphasis>. Note that
125 BLFS book itself is published on a website with https, so you should
126 already have some confidence in https protocol or you wouldn't trust the
127 book content.</para>
128
129 <para>If the package is downloaded from an unofficial location (for
130 example a local mirror), checksums generated by cryptographically secure
131 digest algorithms (for example SHA256) can be used to verify the
132 genuity of the package. Download the checksum file from the upstream
133 <emphasis role="bold">official</emphasis> website (or somewhere
134 <emphasis role="bold">you can trust</emphasis>) and compare the
135 checksum of the package from unofficial location with it. For example,
136 SHA256 checksum can be checked with the command:</para>
137
138 <note>
139 <para>If the checksum and the package are downloaded from the same
140 untrusted location, you won't gain security enhancement by verifying
141 the package with the checksum. The attacker can fake the checksum as
142 well as compromising the package itself.</para>
143 </note>
144
145<screen><userinput>sha256sum -c <replaceable>file</replaceable>.sha256sum</userinput></screen>
146
147 <para>If <xref linkend="gnupg2"/> is installed, you can also verify the
148 genuity of the package with a GPG signature. Import the upstream GPG
149 public key with:</para>
150
151<screen><userinput>gpg --recv-key <replaceable>keyID</replaceable></userinput></screen>
152
153 <para><replaceable>keyID</replaceable> should be replaced with the key ID
154 from somewhere <emphasis role="bold">you can trust</emphasis> (for
155 example, copy it from the upstream official website using https). Now
156 you can verify the signature with:</para>
157
158<screen><userinput>gpg --recv-key <replaceable>file</replaceable>.sig <replaceable>file</replaceable></userinput></screen>
159
160 <para>The advantage of <application>GnuPG</application> signature is,
161 once you imported a public key which can be trusted, you can download
162 both the package and its signature from the same unofficial location and
163 verify them with the public key. So you won't need to connect to the
164 official upstream website to retrieve a checksum for each new release.
165 You only need to update the public key if it's expired or revoked.
166 </para>
167
168 </sect2>
169
170 <sect2>
171 <title>Creating Log Files During Installation</title>
172
173 <para>For larger packages, it is convenient to create log files instead of
174 staring at the screen hoping to catch a particular error or warning. Log
175 files are also useful for debugging and keeping records. The following
176 command allows you to create an installation log. Replace
177 <replaceable>&lt;command&gt;</replaceable> with the command you intend to execute.</para>
178
179<screen><userinput>( <replaceable>&lt;command&gt;</replaceable> 2&gt;&amp;1 | tee compile.log &amp;&amp; exit $PIPESTATUS )</userinput></screen>
180
181 <para><option>2&gt;&amp;1</option> redirects error messages to the same
182 location as standard output. The <command>tee</command> command allows
183 viewing of the output while logging the results to a file. The parentheses
184 around the command run the entire command in a subshell and finally the
185 <command>exit $PIPESTATUS</command> command ensures the result of the
186 <replaceable>&lt;command&gt;</replaceable> is returned as the result and not the
187 result of the <command>tee</command> command.</para>
188
189 </sect2>
190
191 <sect2 id="parallel-builds" xreflabel="Using Multiple Processors">
192 <title>Using Multiple Processors</title>
193
194 <para>For many modern systems with multiple processors (or cores) the
195 compilation time for a package can be reduced by performing a "parallel
196 make" by either setting an environment variable or telling the make program
197 how many processors are available. For instance, a Core2Duo can support two
198 simultaneous processes with: </para>
199
200 <screen><userinput>export MAKEFLAGS='-j2'</userinput></screen>
201
202 <para>or just building with:</para>
203
204 <screen><userinput>make -j2</userinput></screen>
205
206 <para>
207 If you have applied the optional <command>sed</command> when building
208 <application>ninja</application> in LFS, you can use:
209 </para>
210
211 <screen><userinput>export NINJAJOBS=2</userinput></screen>
212
213 <para>
214 when a package uses <command>ninja</command>, or just:
215 </para>
216
217 <screen><userinput>ninja -j2</userinput></screen>
218
219 <para>
220 but for ninja, the default number of jobs is &lt;N&gt;+2, where &lt;N&gt;
221 is the number of processors available, so that using the above commands
222 is rather for limiting the number of jobs (see below for why this could
223 be necessary).
224 </para>
225
226 <para>Generally the number of processes should not exceed the number of
227 cores supported by the CPU. To list the processors on your
228 system, issue: <userinput>grep processor /proc/cpuinfo</userinput>.
229 </para>
230
231 <para>In some cases, using multiple processes may result in a 'race'
232 condition where the success of the build depends on the order of the
233 commands run by the <command>make</command> program. For instance, if an
234 executable needs File A and File B, attempting to link the program before
235 one of the dependent components is available will result in a failure.
236 This condition usually arises because the upstream developer has not
237 properly designated all the prerequisites needed to accomplish a step in the
238 Makefile.</para>
239
240 <para>If this occurs, the best way to proceed is to drop back to a
241 single processor build. Adding '-j1' to a make command will override
242 the similar setting in the <envar>MAKEFLAGS</envar> environment
243 variable.</para>
244
245 <note><para>When running the package tests or the install portion of the
246 package build process, we do not recommend using an option greater than
247 '-j1' unless specified otherwise. The installation procedures or checks
248 have not been validated using parallel procedures and may fail with issues
249 that are difficult to debug.</para></note>
250
251 <important>
252 <para>
253 Another problem may occur with modern CPU's, which have a lot of cores.
254 Each job started consumes memory, and if the sum of the needed
255 memory for each job exceeds the available memory, you may encounter
256 either an OOM (Out of Memory) kernel interrupt or intense swapping
257 that will slow the build beyond reasonable limits.
258 </para>
259
260 <para>
261 Some compilations with <command>g++</command> may consume up to 2.5 GB
262 of memory, so to be safe, you should restrict the number of jobs
263 to (Total Memory in GB)/2.5, at least for big packages such as LLVM,
264 WebKitGtk, QtWebEngine, or libreoffice.
265 </para>
266 </important>
267 </sect2>
268
269 <sect2 id="automating-builds" xreflabel="Automated Building Procedures">
270 <title>Automated Building Procedures</title>
271
272 <para>There are times when automating the building of a package can come in
273 handy. Everyone has their own reasons for wanting to automate building,
274 and everyone goes about it in their own way. Creating
275 <filename>Makefile</filename>s, <application>Bash</application> scripts,
276 <application>Perl</application> scripts or simply a list of commands used
277 to cut and paste are just some of the methods you can use to automate
278 building BLFS packages. Detailing how and providing examples of the many
279 ways you can automate the building of packages is beyond the scope of this
280 section. This section will expose you to using file redirection and the
281 <command>yes</command> command to help provide ideas on how to automate
282 your builds.</para>
283
284 <bridgehead renderas="sect3">File Redirection to Automate Input</bridgehead>
285
286 <para>You will find times throughout your BLFS journey when you will come
287 across a package that has a command prompting you for information. This
288 information might be configuration details, a directory path, or a response
289 to a license agreement. This can present a challenge to automate the
290 building of that package. Occasionally, you will be prompted for different
291 information in a series of questions. One method to automate this type of
292 scenario requires putting the desired responses in a file and using
293 redirection so that the program uses the data in the file as the answers to
294 the questions.</para>
295
296 <para>Building the <application>CUPS</application> package is a good
297 example of how redirecting a file as input to prompts can help you automate
298 the build. If you run the test suite, you are asked to respond to a series
299 of questions regarding the type of test to run and if you have any
300 auxiliary programs the test can use. You can create a file with your
301 responses, one response per line, and use a command similar to the
302 one shown below to automate running the test suite:</para>
303
304<screen><userinput>make check &lt; ../cups-1.1.23-testsuite_parms</userinput></screen>
305
306 <para>This effectively makes the test suite use the responses in the file
307 as the input to the questions. Occasionally you may end up doing a bit of
308 trial and error determining the exact format of your input file for some
309 things, but once figured out and documented you can use this to automate
310 building the package.</para>
311
312 <bridgehead renderas="sect3">Using <command>yes</command> to Automate
313 Input</bridgehead>
314
315 <para>Sometimes you will only need to provide one response, or provide the
316 same response to many prompts. For these instances, the
317 <command>yes</command> command works really well. The
318 <command>yes</command> command can be used to provide a response (the same
319 one) to one or more instances of questions. It can be used to simulate
320 pressing just the <keycap>Enter</keycap> key, entering the
321 <keycap>Y</keycap> key or entering a string of text. Perhaps the easiest
322 way to show its use is in an example.</para>
323
324 <para>First, create a short <application>Bash</application> script by
325 entering the following commands:</para>
326
327<screen><userinput>cat &gt; blfs-yes-test1 &lt;&lt; "EOF"
328<literal>#!/bin/bash
329
330echo -n -e "\n\nPlease type something (or nothing) and press Enter ---> "
331
332read A_STRING
333
334if test "$A_STRING" = ""; then A_STRING="Just the Enter key was pressed"
335else A_STRING="You entered '$A_STRING'"
336fi
337
338echo -e "\n\n$A_STRING\n\n"</literal>
339EOF
340chmod 755 blfs-yes-test1</userinput></screen>
341
342 <para>Now run the script by issuing <command>./blfs-yes-test1</command> from
343 the command line. It will wait for a response, which can be anything (or
344 nothing) followed by the <keycap>Enter</keycap> key. After entering
345 something, the result will be echoed to the screen. Now use the
346 <command>yes</command> command to automate the entering of a
347 response:</para>
348
349<screen><userinput>yes | ./blfs-yes-test1</userinput></screen>
350
351 <para>Notice that piping <command>yes</command> by itself to the script
352 results in <keycap>y</keycap> being passed to the script. Now try it with a
353 string of text:</para>
354
355<screen><userinput>yes 'This is some text' | ./blfs-yes-test1</userinput></screen>
356
357 <para>The exact string was used as the response to the script. Finally,
358 try it using an empty (null) string:</para>
359
360<screen><userinput>yes '' | ./blfs-yes-test1</userinput></screen>
361
362 <para>Notice this results in passing just the press of the
363 <keycap>Enter</keycap> key to the script. This is useful for times when the
364 default answer to the prompt is sufficient. This syntax is used in the
365 <xref linkend="net-tools-automate-example"/> instructions to accept all the
366 defaults to the many prompts during the configuration step. You may now
367 remove the test script, if desired.</para>
368
369 <bridgehead renderas="sect3">File Redirection to Automate Output</bridgehead>
370
371 <para>In order to automate the building of some packages, especially those
372 that require you to read a license agreement one page at a time, requires
373 using a method that avoids having to press a key to display each page.
374 Redirecting the output to a file can be used in these instances to assist
375 with the automation. The previous section on this page touched on creating
376 log files of the build output. The redirection method shown there used the
377 <command>tee</command> command to redirect output to a file while also
378 displaying the output to the screen. Here, the output will only be sent to
379 a file.</para>
380
381 <para>Again, the easiest way to demonstrate the technique is to show an
382 example. First, issue the command:</para>
383
384<screen><userinput>ls -l /usr/bin | more</userinput></screen>
385
386 <para>Of course, you'll be required to view the output one page at a time
387 because the <command>more</command> filter was used. Now try the same
388 command, but this time redirect the output to a file. The special file
389 <filename>/dev/null</filename> can be used instead of the filename shown,
390 but you will have no log file to examine:</para>
391
392<screen><userinput>ls -l /usr/bin | more &gt; redirect_test.log 2&gt;&amp;1</userinput></screen>
393
394 <para>Notice that this time the command immediately returned to the shell
395 prompt without having to page through the output. You may now remove the
396 log file.</para>
397
398 <para>The last example will use the <command>yes</command> command in
399 combination with output redirection to bypass having to page through the
400 output and then provide a <keycap>y</keycap> to a prompt. This technique
401 could be used in instances when otherwise you would have to page through
402 the output of a file (such as a license agreement) and then answer the
403 question of <quote>do you accept the above?</quote>. For this example,
404 another short <application>Bash</application> script is required:</para>
405
406<screen><userinput>cat &gt; blfs-yes-test2 &lt;&lt; "EOF"
407<literal>#!/bin/bash
408
409ls -l /usr/bin | more
410
411echo -n -e "\n\nDid you enjoy reading this? (y,n) "
412
413read A_STRING
414
415if test "$A_STRING" = "y"; then A_STRING="You entered the 'y' key"
416else A_STRING="You did NOT enter the 'y' key"
417fi
418
419echo -e "\n\n$A_STRING\n\n"</literal>
420EOF
421chmod 755 blfs-yes-test2</userinput></screen>
422
423 <para>This script can be used to simulate a program that requires you to
424 read a license agreement, then respond appropriately to accept the
425 agreement before the program will install anything. First, run the script
426 without any automation techniques by issuing
427 <command>./blfs-yes-test2</command>.</para>
428
429 <para>Now issue the following command which uses two automation techniques,
430 making it suitable for use in an automated build script:</para>
431
432<screen><userinput>yes | ./blfs-yes-test2 &gt; blfs-yes-test2.log 2&gt;&amp;1</userinput></screen>
433
434 <para>If desired, issue <command>tail blfs-yes-test2.log</command> to see
435 the end of the paged output, and confirmation that <keycap>y</keycap> was
436 passed through to the script. Once satisfied that it works as it should,
437 you may remove the script and log file.</para>
438
439 <para>Finally, keep in mind that there are many ways to automate and/or
440 script the build commands. There is not a single <quote>correct</quote> way
441 to do it. Your imagination is the only limit.</para>
442
443 </sect2>
444
445 <sect2>
446 <title>Dependencies</title>
447
448 <para>For each package described, BLFS lists the known dependencies.
449 These are listed under several headings, whose meaning is as follows:</para>
450
451 <itemizedlist>
452 <listitem>
453 <para><emphasis>Required</emphasis> means that the target package
454 cannot be correctly built without the dependency having first been
455 installed, except if the dependency is said to be
456 <quote>runtime</quote>, which means the target package can be built but
457 cannot function without it.</para>
458 <para>
459 Note that a target package can start to <quote>function</quote>
460 in many subtle ways: an installed configuration file can make the
461 init system, cron daemon, or bus daemon to run a program
462 automatically; another package using the target package as an
463 dependency can run a program from the target package in the
464 building system; and the configuration sections in the BLFS book
465 may also run a program from a just installed package. So if
466 you are installing the target package without a
467 <emphasis>Required (runtime)</emphasis> dependency installed,
468 You should install the dependency as soon as possible after the
469 installation of the target package.
470 </para>
471 </listitem>
472 <listitem>
473 <para><emphasis>Recommended</emphasis> means that BLFS strongly
474 suggests this package is installed first (except if said to be
475 <quote>runtime</quote>, see below) for a clean and trouble-free
476 build, that won't have issues either during the build process, or at
477 run-time. The instructions in the book assume these packages are
478 installed. Some changes or workarounds may be required if these
479 packages are not installed. If a recommended dependency is said
480 to be <quote>runtime</quote>, it means that BLFS strongly suggests
481 that this dependency is installed before using the package, for
482 getting full funtionality.</para>
483 </listitem>
484 <listitem>
485 <para><emphasis>Optional</emphasis> means that this package might be
486 installed for added functionality. Often BLFS will describe the
487 dependency to explain the added functionality that will result.
488 An optional dependency may be automatically pick up by the target
489 package if the dependency is installed, but another some optional
490 dependency may also need additional configuration options to enable
491 them when the target package is built. Such additional options are
492 often documented in the BLFS book. If an optional dependency is
493 said to be <quote>runtime</quote>, it means you may install
494 the dependency after installing the target package to support some
495 optional features of the target package if you need these
496 features.</para>
497 <para>An optional dependency may be out of BLFS. If you need such
498 an <emphasis>external</emphasis> optional dependency for some
499 features you need, read <xref linkend='beyond'/> for the general
500 hint about installing an out-of-BLFS package.</para>
501 </listitem>
502 </itemizedlist>
503
504 </sect2>
505
506 <sect2 id="package_updates">
507 <title>Using the Most Current Package Sources</title>
508
509 <para>On occasion you may run into a situation in the book when a package
510 will not build or work properly. Though the Editors attempt to ensure
511 that every package in the book builds and works properly, sometimes a
512 package has been overlooked or was not tested with this particular version
513 of BLFS.</para>
514
515 <para>If you discover that a package will not build or work properly, you
516 should see if there is a more current version of the package. Typically
517 this means you go to the maintainer's web site and download the most current
518 tarball and attempt to build the package. If you cannot determine the
519 maintainer's web site by looking at the download URLs, use Google and query
520 the package's name. For example, in the Google search bar type:
521 'package_name download' (omit the quotes) or something similar. Sometimes
522 typing: 'package_name home page' will result in you finding the
523 maintainer's web site.</para>
524
525 </sect2>
526
527 <sect2 id="stripping">
528 <title>Stripping One More Time</title>
529
530 <para>
531 In LFS, stripping of debugging symbols and unneeded symbol table
532 entries was discussed a couple of times. When building BLFS packages,
533 there are generally no special instructions that discuss stripping
534 again. Stripping can be done while installing a package, or
535 afterwards.
536 </para>
537
538 <bridgehead renderas="sect3" id="stripping-install">Stripping while Installing a Package</bridgehead>
539
540 <para>
541 There are several ways to strip executables installed by a
542 package. They depend on the build system used (see below <link
543 linkend="buildsystems">the section about build systems</link>),
544 so only some
545 generalities can be listed here:
546 </para>
547
548 <note>
549 <para>
550 The following methods using the feature of a building system
551 (autotools, meson, or cmake) will not strip static libraries if any
552 is installed. Fortunately there are not too many static libraries
553 in BLFS, and a static library can always be stripped safely by
554 running <command>strip --strip-unneeded</command> on it manually.
555 </para>
556 </note>
557
558 <itemizedlist>
559 <listitem>
560 <para>
561 The packages using autotools usually have an
562 <parameter>install-strip</parameter> target in their generated
563 <filename>Makefile</filename> files. So installing stripped
564 executables is just a matter of using
565 <command>make install-strip</command> instead of
566 <command>make install</command>.
567 </para>
568 </listitem>
569 <listitem>
570 <para>
571 The packages using the meson build system can accept
572 <parameter>-Dstrip=true</parameter> when running
573 <command>meson</command>. If you've forgot to add this option
574 running the <command>meson</command>, you can also run
575 <command>meson install --strip</command> instead of
576 <command>ninja install</command>.
577 </para>
578 </listitem>
579 <listitem>
580 <para>
581 <command>cmake</command> generates
582 <parameter>install/strip</parameter> targets for both the
583 <parameter>Unix Makefiles</parameter> and
584 <parameter>Ninja</parameter> generators (the default is
585 <parameter>Unix Makefiles</parameter> on linux). So just run
586 <command>make install/strip</command> or
587 <command>ninja install/strip</command> instead of the
588 <command>install</command> counterparts.
589 </para>
590 </listitem>
591 <listitem>
592 <para>
593 Removing (or not generating) debug symbols can also be
594 achieved by removing the
595 <parameter>-g&lt;something&gt;</parameter> options
596 in C/C++ calls. How to do that is very specific for each
597 package. And, it does not remove unneeded symbol table entries.
598 So it will not be explained in detail here. See also below
599 the paragraphs about optimization.
600 </para>
601 </listitem>
602 </itemizedlist>
603
604 <bridgehead renderas="sect3" id="stripping-installed">Stripping Installed Executables</bridgehead>
605
606 <para>
607 The <command>strip</command> utility changes files in place, which may
608 break anything using it if it is loaded in memory. Note that if a file is
609 in use but just removed from the disk (i.e. not overwritten nor
610 modified), this is not a problem since the kernel can use
611 <quote>deleted</quote> files. Look at <filename>/proc/*/maps</filename>
612 and it is likely that you'll see some <emphasis>(deleted)</emphasis>
613 entries. The <command>mv</command> just removes the destination file from
614 the directory but does not touch its content, so that it satisfies the
615 condition for the kernel to use the old (deleted) file.
616 But this approach can detach hard links into duplicated copies,
617 causing a bloat which is obviously unwanted as we are stripping to
618 reduce system size. If two files in a same file system share the
619 same inode number, they are hard links to each other and we should
620 reconstruct the link. The script below is just an example.
621 It should be run as the &root; user:
622 </para>
623
624<screen><userinput>cat &gt; /usr/sbin/strip-all.sh &lt;&lt; "EOF"
625<literal>#!/usr/bin/bash
626
627if [ $EUID -ne 0 ]; then
628 echo "Need to be root"
629 exit 1
630fi
631
632last_fs_inode=
633last_file=
634
635{ find /usr/lib -type f -name '*.so*' ! -name '*dbg'
636 find /usr/lib -type f -name '*.a'
637 find /usr/{bin,sbin,libexec} -type f
638} | xargs stat -c '%m %i %n' | sort | while read fs inode file; do
639 if ! readelf -h $file >/dev/null 2>&amp;1; then continue; fi
640 if file $file | grep --quiet --invert-match 'not stripped'; then continue; fi
641
642 if [ "$fs $inode" = "$last_fs_inode" ]; then
643 ln -f $last_file $file;
644 continue;
645 fi
646
647 cp --preserve $file ${file}.tmp
648 strip --strip-unneeded ${file}.tmp
649 mv ${file}.tmp $file
650
651 last_fs_inode="$fs $inode"
652 last_file=$file
653done</literal>
654EOF
655chmod 744 /usr/sbin/strip-all.sh</userinput></screen>
656
657 <para>
658 If you install programs in other directories such as <filename
659 class="directory">/opt</filename> or <filename
660 class="directory">/usr/local</filename>, you may want to strip the files
661 there too. Just add other directories to scan in the compound list of
662 <command>find</command> commands between the braces.
663 </para>
664
665 <para>
666 For more information on stripping, see <ulink
667 url="https://www.technovelty.org/linux/stripping-shared-libraries.html"/>.
668 </para>
669
670 </sect2>
671
672<!--
673 <sect2 id="libtool">
674 <title>Libtool files</title>
675
676 <para>
677 One of the side effects of packages that use Autotools, including
678 libtool, is that they create many files with an .la extension. These
679 files are not needed in an LFS environment. If there are conflicts with
680 pkgconfig entries, they can actually prevent successful builds. You
681 may want to consider removing these files periodically:
682 </para>
683
684<screen><userinput>find /lib /usr/lib -not -path "*Image*" -a -name \*.la -delete</userinput></screen>
685
686 <para>
687 The above command removes all .la files with the exception of those that
688 have <quote>Image</quote> or <quote>openldap</quote> as a part of the
689 path. These .la files are used by the ImageMagick and openldap programs,
690 respectively. There may be other exceptions by packages not in BLFS.
691 </para>
692
693 </sect2>
694-->
695 <sect2 id="buildsystems">
696 <title>Working with different build systems</title>
697
698 <para>
699 There are now three different build systems in common use for
700 converting C or C++ source code into compiled programs or
701 libraries and their details (particularly, finding out about available
702 options and their default values) differ. It may be easiest to understand
703 the issues caused by some choices (typically slow execution or
704 unexpected use of, or omission of, optimizatons) by starting with
705 the CFLAGS and CXXFLAGS environment variables. There are also some
706 programs which use rust.
707 </para>
708
709 <para>
710 Most LFS and BLFS builders are probably aware of the basics of CFLAGS
711 and CXXFLAGS for altering how a program is compiled. Typically, some
712 form of optimization is used by upstream developers (-O2 or -O3),
713 sometimes with the creation of debug symbols (-g), as defaults.
714 </para>
715
716 <para>
717 If there are contradictory flags (e.g. multiple different -O values),
718 the <emphasis>last</emphasis> value will be used. Sometimes this means
719 that flags specified in environment variables will be picked up before
720 values hardcoded in the Makefile, and therefore ignored. For example,
721 where a user specifies '-O2' and that is followed by '-O3' the build will
722 use '-O3'.
723 </para>
724
725 <para>
726 There are various other things which can be passed in CFLAGS or
727 CXXFLAGS, such as forcing compilation for a specific microarchitecture
728 (e.g. -march=amdfam10, -march=native) or specifying a specific standard
729 for C or C++ (-std=c++17 for example). But one thing which has now come
730 to light is that programmers might include debug assertions in their
731 code, expecting them to be disabled in releases by using -DNDEBUG.
732 Specifically, if <xref linkend="mesa"/> is built with these assertions
733 enabled, some activities such as loading levels of games can take
734 extremely long times, even on high-class video cards.
735 </para>
736
737 <bridgehead renderas="sect3" id="autotools-info">Autotools with Make</bridgehead>
738
739 <para>
740 This combination is often described as 'CMMI' (configure, make, make
741 install) and is used here to also cover the few packages which have a
742 configure script that is not generated by autotools.
743 </para>
744
745 <para>
746 Sometimes running <command>./configure --help</command> will produce
747 useful options about switches which might be used. At other times,
748 after looking at the output from configure you may need to look
749 at the details of the script to find out what it was actually searching
750 for.
751 </para>
752
753 <para>
754 Many configure scripts will pick up any CFLAGS or CXXFLAGS from the
755 environment, but CMMI packages vary about how these will be mixed with
756 any flags which would otherwise be used (<emphasis>variously</emphasis>:
757 ignored, used to replace the programmer's suggestion, used before the
758 programmer's suggestion, or used after the programmer's suggestion).
759 </para>
760
761 <para>
762 In most CMMI packages, running 'make' will list each command and run
763 it, interspersed with any warnings. But some packages try to be 'silent'
764 and only show which file they are compiling or linking instead of showing
765 the command line. If you need to inspect the command, either because of
766 an error, or just to see what options and flags are being used, adding
767 'V=1' to the make invocation may help.
768 </para>
769
770 <bridgehead renderas="sect3" id="cmake-info">CMake</bridgehead>
771
772 <para>
773 CMake works in a very different way, and it has two backends which can
774 be used on BLFS: 'make' and 'ninja'. The default backend is make, but
775 ninja can be faster on large packages with multiple processors. To
776 use ninja, specify '-G Ninja' in the cmake command. However, there are
777 some packages which create fatal errors in their ninja files but build
778 successfully using the default of Unix Makefiles.
779 </para>
780
781 <para>
782 The hardest part of using CMake is knowing what options you might wish
783 to specify. The only way to get a list of what the package knows about
784 is to run <command>cmake -LAH</command> and look at the output for that
785 default configuration.
786 </para>
787
788 <para>
789 Perhaps the most-important thing about CMake is that it has a variety
790 of CMAKE_BUILD_TYPE values, and these affect the flags. The default
791 is that this is not set and no flags are generated. Any CFLAGS or
792 CXXFLAGS in the environment will be used. If the programmer has coded
793 any debug assertions, those will be enabled unless -DNDEBUG is used.
794 The following CMAKE_BUILD_TYPE values will generate the flags shown,
795 and these will come <emphasis>after</emphasis> any flags in the
796 environment and therefore take precedence.
797 </para>
798
799 <informaltable align="center">
800 <tgroup cols="2">
801 <colspec colnum="1" align="center"/>
802 <colspec colnum="2" align="center"/>
803 <thead>
804 <row><entry>Value</entry><entry>Flags</entry></row>
805 </thead>
806 <tbody>
807 <row>
808 <entry>Debug</entry><entry><option>-g</option></entry>
809 </row>
810 <row>
811 <entry>Release</entry><entry><option>-O3 -DNDEBUG</option></entry>
812 </row>
813 <row>
814 <entry>RelWithDebInfo</entry><entry><option>-O2 -g -DNDEBUG</option></entry>
815 </row>
816 <row>
817 <entry>MinSizeRel</entry><entry><option>-Os -DNDEBUG</option></entry>
818 </row>
819 </tbody>
820 </tgroup>
821 </informaltable>
822
823 <para>
824 CMake tries to produce quiet builds. To see the details of the commands
825 which are being run, use <command>make VERBOSE=1</command> or
826 <command>ninja -v</command>.
827 </para>
828
829 <para>
830 By default, CMake treats file installation differently from the other
831 build systems: if a file already exists and is not newer than a file
832 that would overwrite it, then the file is not installed. This may be
833 a problem if a user wants to record which file belongs to a package,
834 either using <envar>LD_PRELOAD</envar>, or by listing files newer
835 than a timestamp. The default can be changed by setting the variable
836 <envar>CMAKE_INSTALL_ALWAYS</envar> to 1 in the
837 <emphasis>environment</emphasis>, for example by
838 <command>export</command>'ing it.
839 </para>
840
841 <bridgehead renderas="sect3" id="meson-info">Meson</bridgehead>
842
843 <para>
844 Meson has some similarities to CMake, but many differences. To get
845 details of the defines that you may wish to change you can look at
846 <filename>meson_options.txt</filename> which is usually in the
847 top-level directory.
848 </para>
849
850 <para>
851 If you have already configured the package by running
852 <command>meson</command> and now wish to change one or more settings,
853 you can either remove the build directory, recreate it, and use the
854 altered options, or within the build directory run <command>meson
855 configure</command>, e.g. to set an option:
856 </para>
857
858<screen><userinput>meson configure -D&lt;some_option&gt;=true</userinput></screen>
859
860 <para>
861 If you do that, the file <filename>meson-private/cmd_line.txt</filename>
862 will show the <emphasis>last</emphasis> commands which were used.
863 </para>
864
865 <para>
866 Meson provides the following buildtype values, and the flags they enable
867 come <emphasis>after</emphasis> any flags supplied in the environment and
868 therefore take precedence.
869 </para>
870
871 <itemizedlist>
872 <listitem>
873 <para>plain : no added flags. This is for distributors to supply their
874 own CLFAGS, CXXFLAGS and LDFLAGS. There is no obvious reason to use
875 this in BLFS.</para>
876 </listitem>
877 <listitem>
878 <para>debug : '-g' - this is the default if nothing is specified
879 in either <filename>meson.build</filename> or the command line.
880 However it results large and slow binaries, so we should override
881 it in BLFS.</para>
882 </listitem>
883 <listitem>
884 <para>debugoptimized : '-O2 -g' : this is the default specified in
885 <filename>meson.build</filename> of some packages.</para>
886 </listitem>
887 <listitem>
888 <para>release : '-O3 -DNDEBUG' (but occasionally a package will force
889 -O2 here)</para>
890 </listitem>
891 </itemizedlist>
892
893 <para>
894 Although the 'release' buildtype is described as enabling -DNDEBUG, and all
895 CMake Release builds pass that, it has so far only been observed (in
896 verbose builds) for <xref linkend="mesa"/>. That suggests that it might
897 only be used when there are debug assertions present.
898 </para>
899
900 <para>
901 The -DNDEBUG flag can also be provided by passing
902 <command>-Db_ndebug=true</command>.
903 </para>
904
905 <para>
906 To see the details of the commands which are being run in a package using
907 meson, use 'ninja -v'.
908 </para>
909
910 <bridgehead renderas="sect3" id="rust-info">Rustc and Cargo</bridgehead>
911
912 <para>
913 Most released rustc programs are provided as crates (source tarballs)
914 which will query a server to check current versions of dependencies
915 and then download them as necessary. These packages are built using
916 <command>cargo --release</command>. In theory, you can manipulate the
917 RUSTFLAGS to change the optimize-level (default is 3, like -O3, e.g.
918 <literal>-Copt-level=3</literal>) or to force it to build for the
919 machine it is being compiled on, using
920 <literal>-Ctarget-cpu=native</literal> but in practice this seems to
921 make no significant difference.
922 </para>
923
924 <para>
925 If you find an interesting rustc program which is only provided as
926 unpackaged source, you should at least specify
927 <literal>RUSTFLAGS=-Copt-level=2</literal> otherwise it will do an
928 unoptimized compile with debug info and run <emphasis>much</emphasis>
929 slower.
930 </para>
931
932 <para>
933 The rust developers seem to assume that everyone will compile on a
934 machine dedicated to producing builds, so by default all CPUs are used.
935 This can often be worked around, either by exporting
936 CARGO_BUILD_JOBS=&lt;N&gt; or passing --jobs &lt;N&gt; to cargo. For
937 compiling rustc itself, specifying --jobs &lt;N&gt; on invocations of
938 x.py (together with the <envar>CARGO_BUILD_JOBS</envar> environment
939 variable, which looks like a "belt and braces" approach but seems to be
940 necessary) mostly works. The exception is running the tests when building
941 rustc, some of them will nevertheless use all online CPUs, at least as of
942 rustc-1.42.0.
943 </para>
944
945 </sect2>
946
947 <sect2 id="optimizations">
948 <title>Optimizing the build</title>
949
950 <para>
951 Many people will prefer to optimize compiles as they see fit, by providing
952 CFLAGS or CXXFLAGS. For an introduction to the options available with gcc
953 and g++ see <ulink
954 url="https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html"/> and <ulink
955 url="https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html"/>
956 and <command>info gcc</command>.
957
958 </para>
959
960 <para>
961 Some packages default to '-O2 -g', others to '-O3 -g', and if CFLAGS or
962 CXXFLAGS are supplied they might be added to the package's defaults,
963 replace the package's defaults, or even be ignored. There are details
964 on some desktop packages which were mostly current in April 2019 at
965 <ulink url="https://www.linuxfromscratch.org/~ken/tuning/"/> - in
966 particular, README.txt, tuning-1-packages-and-notes.txt, and
967 tuning-notes-2B.txt. The particular thing to remember is that if you
968 want to try some of the more interesting flags you may need to force
969 verbose builds to confirm what is being used.
970 </para>
971
972 <para>
973 Clearly, if you are optimizing your own program you can spend time to
974 profile it and perhaps recode some of it if it is too slow. But for
975 building a whole system that approach is impractical. In general,
976 -O3 usually produces faster programs than -O2. Specifying
977 -march=native is also beneficial, but means that you cannot move the
978 binaries to an incompatible machine - this can also apply to newer
979 machines, not just to older machines. For example programs compiled for
980 'amdfam10' run on old Phenoms, Kaveris, and Ryzens : but programs
981 compiled for a Kaveri will not run on a Ryzen because certain op-codes
982 are not present. Similarly, if you build for a Haswell not everything
983 will run on a SandyBridge.
984 </para>
985
986 <para>
987 There are also various other options which some people claim are
988 beneficial. At worst, you get to recompile and test, and then
989 discover that in your usage the options do not provide a benefit.
990 </para>
991
992 <para>
993 If building Perl or Python modules, or Qt packages which use qmake,
994 in general the CFLAGS and CXXFLAGS used are those which were used by
995 those 'parent' packages.
996 </para>
997
998 </sect2>
999
1000 <sect2 id="hardening">
1001 <title>Options for hardening the build</title>
1002
1003 <para>
1004 Even on desktop systems, there are still a lot of exploitable
1005 vulnerabilities. For many of these, the attack comes via javascript
1006 in a browser. Often, a series of vulnerabilities are used to gain
1007 access to data (or sometimes to pwn, i.e. own, the machine and
1008 install rootkits). Most commercial distros will apply various
1009 hardening measures.
1010 </para>
1011
1012 <para>
1013 In the past, there was Hardened LFS where gcc (a much older version)
1014 was forced to use hardening (with options to turn some of it off on a
1015 per-package basis). The current LFS and BLFS books are carrying
1016 forward a part of its spirit by enabling PIE
1017 (<option>-fPIE -pie</option>) and SSP
1018 (<option>-fstack-protector-strong</option>) as the defaults
1019 for GCC and clang. What is being covered here is different - first
1020 you have to make sure that the package is indeed using your added
1021 flags and not over-riding them.
1022 </para>
1023
1024 <para>
1025 For hardening options which are reasonably cheap, there is some
1026 discussion in the 'tuning' link above (occasionally, one or more
1027 of these options might be inappropriate for a package). These
1028 options are <option>-D_FORTIFY_SOURCE=2</option> and
1029 (for C++) <option>-D_GLIBCXX_ASSERTIONS</option>. On modern
1030 machines these should only have a little impact on how fast things
1031 run, and often they will not be noticeable.
1032 </para>
1033
1034 <para>
1035 The main distros use much more, such as RELRO (Relocation Read Only)
1036 and perhaps <option>-fstack-clash-protection</option>. You may also
1037 encounter the so-called <quote>userspace retpoline</quote>
1038 (<option>-mindirect-branch=thunk</option> etc.) which
1039 is the equivalent of the spectre mitigations applied to the linux
1040 kernel in late 2018. The kernel mitigations caused a lot of complaints
1041 about lost performance, if you have a production server you might wish
1042 to consider testing that, along with the other available options, to
1043 see if performance is still sufficient.
1044 </para>
1045
1046 <para>
1047 Whilst gcc has many hardening options, clang/LLVM's strengths lie
1048 elsewhere. Some options which gcc provides are said to be less effective
1049 in clang/LLVM.
1050 </para>
1051
1052 </sect2>
1053
1054</sect1>
Note: See TracBrowser for help on using the repository browser.