source: introduction/important/building-notes.xml@ 3334a3d

12.2 lazarus trunk xry111/for-12.3
Last change on this file since 3334a3d was 3334a3d, checked in by Xi Ruoyao <xry111@…>, 2 months ago

Fix typos

Signed-off-by: Andrew Kreimer <algonell@…>

  • Property mode set to 100644
File size: 60.8 KB
Line 
1<?xml version="1.0" encoding="UTF-8"?>
2<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
3 "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
4 <!ENTITY % general-entities SYSTEM "../../general.ent">
5 %general-entities;
6]>
7
8<sect1 id="unpacking">
9 <?dbhtml filename="notes-on-building.html"?>
10
11
12 <title>Notes on Building Software</title>
13
14 <para>Those people who have built an LFS system may be aware
15 of the general principles of downloading and unpacking software. Some
16 of that information is repeated here for those new to building
17 their own software.</para>
18
19 <para>Each set of installation instructions contains a URL from which you
20 can download the package. The patches; however, are stored on the LFS
21 servers and are available via HTTP. These are referenced as needed in the
22 installation instructions.</para>
23
24 <para>While you can keep the source files anywhere you like, we assume that
25 you have unpacked the package and changed into the directory created by the
26 unpacking process (the source directory). We also assume you have
27 uncompressed any required patches and they are in the directory
28 immediately above the source directory.</para>
29
30 <para>We can not emphasize strongly enough that you should start from a
31 <emphasis>clean source tree</emphasis> each time. This means that if
32 you have had an error during configuration or compilation, it's usually
33 best to delete the source tree and
34 re-unpack it <emphasis>before</emphasis> trying again. This obviously
35 doesn't apply if you're an advanced user used to hacking
36 <filename>Makefile</filename>s and C code, but if in doubt, start from a
37 clean tree.</para>
38
39 <sect2>
40 <title>Building Software as an Unprivileged (non-root) User</title>
41
42 <para>The golden rule of Unix System Administration is to use your
43 superpowers only when necessary. Hence, BLFS recommends that you
44 build software as an unprivileged user and only become the
45 <systemitem class='username'>root</systemitem> user when installing the
46 software. This philosophy is followed in all the packages in this book.
47 Unless otherwise specified, all instructions should be executed as an
48 unprivileged user. The book will advise you on instructions that need
49 <systemitem class='username'>root</systemitem> privileges.</para>
50
51 </sect2>
52
53 <sect2>
54 <title>Unpacking the Software</title>
55
56 <para>If a file is in <filename class='extension'>.tar</filename> format
57 and compressed, it is unpacked by running one of the following
58 commands:</para>
59
60<screen><userinput>tar -xvf filename.tar.gz
61tar -xvf filename.tgz
62tar -xvf filename.tar.Z
63tar -xvf filename.tar.bz2</userinput></screen>
64
65 <note>
66 <para>You may omit using the <option>v</option> parameter in the commands
67 shown above and below if you wish to suppress the verbose listing of all
68 the files in the archive as they are extracted. This can help speed up the
69 extraction as well as make any errors produced during the extraction
70 more obvious to you.</para>
71 </note>
72
73 <para>You can also use a slightly different method:</para>
74
75<screen><userinput>bzcat filename.tar.bz2 | tar -xv</userinput></screen>
76
77 <para>
78 Finally, sometimes we have a compressed patch file in
79 <filename class='extension'>.patch.gz</filename> or
80 <filename class='extension'>.patch.bz2</filename> format.
81 The best way to apply the patch is piping the output of the
82 decompressor to the <command>patch</command> utility. For example:
83 </para>
84
85 <screen><userinput>gzip -cd ../patchname.patch.gz | patch -p1</userinput></screen>
86
87 <para>
88 Or for a patch compressed with <command>bzip2</command>:
89 </para>
90
91 <screen><userinput>bzcat ../patchname.patch.bz2 | patch -p1</userinput></screen>
92
93 </sect2>
94
95 <sect2>
96 <title>Verifying File Integrity</title>
97
98 <para>Generally, to verify that the downloaded file is complete,
99 many package maintainers also distribute md5sums of the files. To verify the
100 md5sum of the downloaded files, download both the file and the
101 corresponding md5sum file to the same directory (preferably from different
102 on-line locations), and (assuming <filename>file.md5sum</filename> is the
103 md5sum file downloaded) run the following command:</para>
104
105<screen><userinput>md5sum -c file.md5sum</userinput></screen>
106
107 <para>If there are any errors, they will be reported. Note that the BLFS
108 book includes md5sums for all the source files also. To use the BLFS
109 supplied md5sums, you can create a <filename>file.md5sum</filename> (place
110 the md5sum data and the exact name of the downloaded file on the same
111 line of a file, separated by white space) and run the command shown above.
112 Alternately, simply run the command shown below and compare the output
113 to the md5sum data shown in the BLFS book.</para>
114
115<screen><userinput>md5sum <replaceable>&lt;name_of_downloaded_file&gt;</replaceable></userinput></screen>
116
117 <para>MD5 is not cryptographically secure, so the md5sums are only
118 provided for detecting unmalicious changes to the file content. For
119 example, an error or truncation introduced during network transfer, or
120 a <quote>stealth</quote> update to the package from the upstream
121 (updating the content of a released tarball instead of making a new
122 release properly).</para>
123
124 <para>There is no <quote>100%</quote> secure way to make
125 sure the genuity of the source files. Assuming the upstream is managing
126 their website correctly (the private key is not leaked and the domain is
127 not hijacked), and the trust anchors have been set up correctly using
128 <xref linkend="make-ca"/> on the BLFS system, we can reasonably trust
129 download URLs to the upstream official website
130 <emphasis role="bold">with https protocol</emphasis>. Note that
131 BLFS book itself is published on a website with https, so you should
132 already have some confidence in https protocol or you wouldn't trust the
133 book content.</para>
134
135 <para>If the package is downloaded from an unofficial location (for
136 example a local mirror), checksums generated by cryptographically secure
137 digest algorithms (for example SHA256) can be used to verify the
138 genuity of the package. Download the checksum file from the upstream
139 <emphasis role="bold">official</emphasis> website (or somewhere
140 <emphasis role="bold">you can trust</emphasis>) and compare the
141 checksum of the package from unofficial location with it. For example,
142 SHA256 checksum can be checked with the command:</para>
143
144 <note>
145 <para>If the checksum and the package are downloaded from the same
146 untrusted location, you won't gain security enhancement by verifying
147 the package with the checksum. The attacker can fake the checksum as
148 well as compromising the package itself.</para>
149 </note>
150
151<screen><userinput>sha256sum -c <replaceable>file</replaceable>.sha256sum</userinput></screen>
152
153 <para>If <xref linkend="gnupg2"/> is installed, you can also verify the
154 genuity of the package with a GPG signature. Import the upstream GPG
155 public key with:</para>
156
157<screen><userinput>gpg --recv-key <replaceable>keyID</replaceable></userinput></screen>
158
159 <para><replaceable>keyID</replaceable> should be replaced with the key ID
160 from somewhere <emphasis role="bold">you can trust</emphasis> (for
161 example, copy it from the upstream official website using https). Now
162 you can verify the signature with:</para>
163
164<screen><userinput>gpg --recv-key <replaceable>file</replaceable>.sig <replaceable>file</replaceable></userinput></screen>
165
166 <para>The advantage of <application>GnuPG</application> signature is,
167 once you imported a public key which can be trusted, you can download
168 both the package and its signature from the same unofficial location and
169 verify them with the public key. So you won't need to connect to the
170 official upstream website to retrieve a checksum for each new release.
171 You only need to update the public key if it's expired or revoked.
172 </para>
173
174 </sect2>
175
176 <sect2>
177 <title>Creating Log Files During Installation</title>
178
179 <para>For larger packages, it is convenient to create log files instead of
180 staring at the screen hoping to catch a particular error or warning. Log
181 files are also useful for debugging and keeping records. The following
182 command allows you to create an installation log. Replace
183 <replaceable>&lt;command&gt;</replaceable> with the command you intend to execute.</para>
184
185<screen><userinput>( <replaceable>&lt;command&gt;</replaceable> 2&gt;&amp;1 | tee compile.log &amp;&amp; exit $PIPESTATUS )</userinput></screen>
186
187 <para><option>2&gt;&amp;1</option> redirects error messages to the same
188 location as standard output. The <command>tee</command> command allows
189 viewing of the output while logging the results to a file. The parentheses
190 around the command run the entire command in a subshell and finally the
191 <command>exit $PIPESTATUS</command> command ensures the result of the
192 <replaceable>&lt;command&gt;</replaceable> is returned as the result and not the
193 result of the <command>tee</command> command.</para>
194
195 </sect2>
196
197 <sect2 id="parallel-builds" xreflabel="Using Multiple Processors">
198 <title>Using Multiple Processors</title>
199
200 <para>For many modern systems with multiple processors (or cores) the
201 compilation time for a package can be reduced by performing a "parallel
202 make" by either setting an environment variable or telling the make program
203 to simultaneously execute multiple jobs.</para>
204
205 <para>For instance, an Intel Core i9-13900K CPU contains 8 performance
206 (P) cores and 16 efficiency (E) cores, and the P cores support SMT
207 (Simultaneous MultiThreading, also known as
208 <quote>Hyper-Threading</quote>) so each P core can run two threads
209 simultaneously and the Linux kernel will treat each P core as two
210 logical cores. As a result, there are 32 logical cores in total.
211 To utilize all these logical cores running <command>make</command>, we
212 can set an environment variable to tell <command>make</command> to
213 run 32 jobs simultaneously:</para>
214
215 <screen><userinput>export MAKEFLAGS='-j32'</userinput></screen>
216
217 <para>or just building with:</para>
218
219 <screen><userinput>make -j32</userinput></screen>
220
221 <para>
222 If you have applied the optional <command>sed</command> when building
223 <application>ninja</application> in LFS, you can use:
224 </para>
225
226 <screen><userinput>export NINJAJOBS=32</userinput></screen>
227
228 <para>
229 when a package uses <command>ninja</command>, or just:
230 </para>
231
232 <screen><userinput>ninja -j32</userinput></screen>
233
234 <para>
235 If you are not sure about the number of logical cores, run the
236 <command>nproc</command> command.
237 </para>
238
239 <para>
240 For <command>make</command>, the default number of jobs is 1. But
241 for <command>ninja</command>, the default number of jobs is N + 2 if
242 the number of logical cores N is greater than 2; or N + 1 if
243 N is 1 or 2. The reason to use a number of jobs slightly greater
244 than the number of logical cores is keeping all logical
245 processors busy even if some jobs are performing I/O operations.
246 </para>
247
248 <para>
249 Note that the <option>-j</option> switches only limits the parallel
250 jobs started by <command>make</command> or <command>ninja</command>,
251 but each job may still spawn its own processes or threads. For
252 example, <command>ld.gold</command> will use multiple threads for
253 linking, and some tests of packages can spawn multiple threads for
254 testing thread safety properties. There is no generic way for the
255 building system to know the number of processes or threads spawned by
256 a job. So generally we should not consider the value passed with
257 <option>-j</option> a hard limit of the number of logical cores to
258 use. Read <xref linkend='build-in-cgroup'/> if you want to set such
259 a hard limit.
260 </para>
261
262 <para>Generally the number of processes should not exceed the number of
263 cores supported by the CPU too much. To list the processors on your
264 system, issue: <userinput>grep processor /proc/cpuinfo</userinput>.
265 </para>
266
267 <para>In some cases, using multiple processes may result in a race
268 condition where the success of the build depends on the order of the
269 commands run by the <command>make</command> program. For instance, if an
270 executable needs File A and File B, attempting to link the program before
271 one of the dependent components is available will result in a failure.
272 This condition usually arises because the upstream developer has not
273 properly designated all the prerequisites needed to accomplish a step in the
274 Makefile.</para>
275
276 <para>If this occurs, the best way to proceed is to drop back to a
277 single processor build. Adding <option>-j1</option> to a make command
278 will override the similar setting in the <envar>MAKEFLAGS</envar>
279 environment variable.</para>
280
281 <important>
282 <para>
283 Another problem may occur with modern CPU's, which have a lot of cores.
284 Each job started consumes memory, and if the sum of the needed
285 memory for each job exceeds the available memory, you may encounter
286 either an OOM (Out of Memory) kernel interrupt or intense swapping
287 that will slow the build beyond reasonable limits.
288 </para>
289
290 <para>
291 Some compilations with <command>g++</command> may consume up to 2.5 GB
292 of memory, so to be safe, you should restrict the number of jobs
293 to (Total Memory in GB)/2.5, at least for big packages such as LLVM,
294 WebKitGtk, QtWebEngine, or libreoffice.
295 </para>
296 </important>
297 </sect2>
298
299 <sect2 id="build-in-cgroup">
300 <title>Use Linux Control Group to Limit the Resource Usage</title>
301
302 <para>
303 Sometimes we want to limit the resource usage when we build a
304 package. For example, when we have 8 logical cores, we may want
305 to use only 6 cores for building the package and reserve another
306 2 cores for playing a movie. The Linux kernel provides a feature
307 called control groups (cgroup) for such a need.
308 </para>
309
310 <para>
311 Enable control group in the kernel configuration, then rebuild the
312 kernel and reboot if necessary:
313 </para>
314
315 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
316 href="cgroup-kernel.xml"/>
317
318 <!-- We need cgroup2 mounted at /sys/fs/cgroup. It's done by
319 systemd itself in LFS systemd, mountvirtfs script in LFS sysv. -->
320
321 <para revision='systemd'>
322 Ensure <xref linkend='systemd'/> and <xref linkend='shadow'/> have
323 been rebuilt with <xref linkend='linux-pam'/> support (if you are
324 interacting via a SSH or graphical session, also ensure the
325 <xref linkend='openssh'/> server or the desktop manager has been
326 built with <xref linkend='linux-pam'/>). As the &root; user, create
327 a configuration file to allow resource control without &root;
328 privilege, and instruct <command>systemd</command> to reload the
329 configuration:
330 </para>
331
332 <screen revision="systemd" role="nodump"><userinput>mkdir -pv /etc/systemd/system/user@.service.d &amp;&amp;
333cat &gt; /etc/systemd/system/user@.service.d/delegate.conf &lt;&lt; EOF &amp;&amp;
334<literal>[Service]
335Delegate=memory cpuset</literal>
336systemctl daemon-reload</userinput></screen>
337
338 <para revision='systemd'>
339 Then logout and login again. Now to run <command>make -j5</command>
340 with the first 4 logical cores and 8 GB of system memory, issue:
341 </para>
342
343 <screen revision="systemd" role="nodump"><userinput>systemctl --user start dbus &amp;&amp;
344systemd-run --user --pty --pipe --wait -G -d \
345 -p MemoryHigh=8G \
346 -p AllowedCPUs=0-3 \
347 make -j5</userinput></screen>
348
349 <para revision='sysv'>
350 Ensure <xref linkend='sudo'/> is installed. To run
351 <command>make -j5</command> with the first 4 logical cores and 8 GB
352 of system memory, issue:
353 </para>
354
355 <!-- "\EOF" because we expect $$ to be expanded by the "bash -e"
356 shell, not the current shell.
357
358 TODO: can we use elogind to delegate the controllers (like
359 systemd) to avoid relying on sudo? -->
360 <screen revision="sysv" role="nodump"><userinput>bash -e &lt;&lt; \EOF
361 sudo mkdir /sys/fs/cgroup/$$
362 sudo sh -c \
363 "echo +memory +cpuset > /sys/fs/cgroup/cgroup.subtree_control"
364 sudo sh -c \
365 "echo 0-3 > /sys/fs/cgroup/$$/cpuset.cpus"
366 sudo sh -c \
367 "echo $(bc -e '8*2^30') > /sys/fs/cgroup/$$/memory.high"
368 (
369 sudo sh -c "echo $BASHPID > /sys/fs/cgroup/$$/cgroup.procs"
370 exec make -j5
371 )
372 sudo rmdir /sys/fs/cgroup/$$
373EOF</userinput></screen>
374
375 <para>
376 With
377 <phrase revision='systemd'>
378 <parameter>MemoryHigh=8G</parameter>
379 </phrase>
380 <phrase revision='sysv'>
381 <literal>8589934592</literal> (the output of
382 <userinput>bc -e '8*2^30'</userinput>, 2^30 represents
383 2<superscript>30</superscript>, i.e. a Gigabyte) in the
384 <filename>memory.high</filename> entry
385 </phrase>, a soft limit of memory usage is set.
386 If the processes in the cgroup (<command>make</command> and all the
387 descendants of it) uses more than 8 GB of system memory in total,
388 the kernel will throttle down the processes and try to reclaim the
389 system memory from them. But they can still use more than 8 GB of
390 system memory. If you want to make a hard limit instead, replace
391 <phrase revision='systemd'>
392 <parameter>MemoryHigh</parameter> with
393 <parameter>MemoryMax</parameter>.
394 </phrase>
395 <phrase revision='sysv'>
396 <filename>memory.high</filename> with
397 <filename>memory.max</filename>.
398 </phrase>
399 But doing so will cause the processes killed if 8 GB is not enough
400 for them.
401 </para>
402
403 <para>
404 <phrase revision='systemd'>
405 <parameter>AllowedCPUs=0-3</parameter>
406 </phrase>
407 <phrase revision='sysv'>
408 <literal>0-3</literal> in the <filename>cpuset.cpus</filename>
409 entry
410 </phrase> makes the kernel only run the processes in the cgroup on
411 the logical cores with numbers 0, 1, 2, or 3. You may need to
412 adjust this setting based the mapping between the logical cores and the
413 physical cores. For example, with an Intel Core i9-13900K CPU,
414 the logical cores 0, 2, 4, ..., 14 are mapped to the first threads of
415 the eight physical P cores, the logical cores 1, 3, 5, ..., 15 are
416 mapped to the second threads of the physical P cores, and the logical
417 cores 16, 17, ..., 31 are mapped to the 16 physical E cores. So if
418 we want to use four threads from four different P cores, we need to
419 specify <literal>0,2,4,6</literal> instead of <literal>0-3</literal>.
420 Note that the other CPU models may use a different mapping scheme.
421 If you are not sure about the mapping between the logical cores
422 and the physical cores, run <command>grep -E '^processor|^core'
423 /proc/cpuinfo</command> which will output logical core IDs in the
424 <computeroutput>processor</computeroutput> lines, and physical core
425 IDs in the <computeroutput>core id</computeroutput> lines.
426 </para>
427
428 <para>
429 When the <command>nproc</command> or <command>ninja</command> command
430 runs in a cgroup, it will use the number of logical cores assigned to
431 the cgroup as the <quote>system logical core count.</quote> For
432 example, in a cgroup with logical cores 0-3 assigned,
433 <command>nproc</command> will print
434 <computeroutput>4</computeroutput>, and <command>ninja</command>
435 will run 6 (4 + 2) jobs simultaneously if no <option>-j</option>
436 setting is explicitly given.
437 </para>
438
439 <para revision="systemd">
440 Read the man pages <ulink role='man'
441 url='&man;systemd-run.1'>systemd-run(1)</ulink> and
442 <ulink role='man'
443 url='&man;systemd.resource-control.5'>systemd.resource-control(5)</ulink>
444 for the detailed explanation of parameters in the command.
445 </para>
446
447 <para revision="sysv">
448 Read the <filename>Documentation/admin-guide/cgroup-v2.rst</filename>
449 file in the Linux kernel source tree for the detailed explanation of
450 <systemitem class="filesystem">cgroup2</systemitem> pseudo file
451 system entries referred in the command.
452 </para>
453
454 </sect2>
455
456 <sect2 id="automating-builds" xreflabel="Automated Building Procedures">
457 <title>Automated Building Procedures</title>
458
459 <para>There are times when automating the building of a package can come in
460 handy. Everyone has their own reasons for wanting to automate building,
461 and everyone goes about it in their own way. Creating
462 <filename>Makefile</filename>s, <application>Bash</application> scripts,
463 <application>Perl</application> scripts or simply a list of commands used
464 to cut and paste are just some of the methods you can use to automate
465 building BLFS packages. Detailing how and providing examples of the many
466 ways you can automate the building of packages is beyond the scope of this
467 section. This section will expose you to using file redirection and the
468 <command>yes</command> command to help provide ideas on how to automate
469 your builds.</para>
470
471 <bridgehead renderas="sect3">File Redirection to Automate Input</bridgehead>
472
473 <para>You will find times throughout your BLFS journey when you will come
474 across a package that has a command prompting you for information. This
475 information might be configuration details, a directory path, or a response
476 to a license agreement. This can present a challenge to automate the
477 building of that package. Occasionally, you will be prompted for different
478 information in a series of questions. One method to automate this type of
479 scenario requires putting the desired responses in a file and using
480 redirection so that the program uses the data in the file as the answers to
481 the questions.</para>
482<!-- outdated
483 <para>Building the <application>CUPS</application> package is a good
484 example of how redirecting a file as input to prompts can help you automate
485 the build. If you run the test suite, you are asked to respond to a series
486 of questions regarding the type of test to run and if you have any
487 auxiliary programs the test can use. You can create a file with your
488 responses, one response per line, and use a command similar to the
489 one shown below to automate running the test suite:</para>
490
491<screen><userinput>make check &lt; ../cups-1.1.23-testsuite_parms</userinput></screen>
492-->
493 <para>This effectively makes the test suite use the responses in the file
494 as the input to the questions. Occasionally you may end up doing a bit of
495 trial and error determining the exact format of your input file for some
496 things, but once figured out and documented you can use this to automate
497 building the package.</para>
498
499 <bridgehead renderas="sect3">Using <command>yes</command> to Automate
500 Input</bridgehead>
501
502 <para>Sometimes you will only need to provide one response, or provide the
503 same response to many prompts. For these instances, the
504 <command>yes</command> command works really well. The
505 <command>yes</command> command can be used to provide a response (the same
506 one) to one or more instances of questions. It can be used to simulate
507 pressing just the <keycap>Enter</keycap> key, entering the
508 <keycap>Y</keycap> key or entering a string of text. Perhaps the easiest
509 way to show its use is in an example.</para>
510
511 <para>First, create a short <application>Bash</application> script by
512 entering the following commands:</para>
513
514<screen><userinput>cat &gt; blfs-yes-test1 &lt;&lt; "EOF"
515<literal>#!/bin/bash
516
517echo -n -e "\n\nPlease type something (or nothing) and press Enter ---> "
518
519read A_STRING
520
521if test "$A_STRING" = ""; then A_STRING="Just the Enter key was pressed"
522else A_STRING="You entered '$A_STRING'"
523fi
524
525echo -e "\n\n$A_STRING\n\n"</literal>
526EOF
527chmod 755 blfs-yes-test1</userinput></screen>
528
529 <para>Now run the script by issuing <command>./blfs-yes-test1</command> from
530 the command line. It will wait for a response, which can be anything (or
531 nothing) followed by the <keycap>Enter</keycap> key. After entering
532 something, the result will be echoed to the screen. Now use the
533 <command>yes</command> command to automate the entering of a
534 response:</para>
535
536<screen><userinput>yes | ./blfs-yes-test1</userinput></screen>
537
538 <para>Notice that piping <command>yes</command> by itself to the script
539 results in <keycap>y</keycap> being passed to the script. Now try it with a
540 string of text:</para>
541
542<screen><userinput>yes 'This is some text' | ./blfs-yes-test1</userinput></screen>
543
544 <para>The exact string was used as the response to the script. Finally,
545 try it using an empty (null) string:</para>
546
547<screen><userinput>yes '' | ./blfs-yes-test1</userinput></screen>
548
549 <para>Notice this results in passing just the press of the
550 <keycap>Enter</keycap> key to the script. This is useful for times when the
551 default answer to the prompt is sufficient. This syntax is used in the
552 <xref linkend="net-tools-automate-example"/> instructions to accept all the
553 defaults to the many prompts during the configuration step. You may now
554 remove the test script, if desired.</para>
555
556 <bridgehead renderas="sect3">File Redirection to Automate Output</bridgehead>
557
558 <para>In order to automate the building of some packages, especially those
559 that require you to read a license agreement one page at a time, requires
560 using a method that avoids having to press a key to display each page.
561 Redirecting the output to a file can be used in these instances to assist
562 with the automation. The previous section on this page touched on creating
563 log files of the build output. The redirection method shown there used the
564 <command>tee</command> command to redirect output to a file while also
565 displaying the output to the screen. Here, the output will only be sent to
566 a file.</para>
567
568 <para>Again, the easiest way to demonstrate the technique is to show an
569 example. First, issue the command:</para>
570
571<screen><userinput>ls -l /usr/bin | less</userinput></screen>
572
573 <para>Of course, you'll be required to view the output one page at a time
574 because the <command>less</command> filter was used. Now try the same
575 command, but this time redirect the output to a file. The special file
576 <filename>/dev/null</filename> can be used instead of the filename shown,
577 but you will have no log file to examine:</para>
578
579<screen><userinput>ls -l /usr/bin | less &gt; redirect_test.log 2&gt;&amp;1</userinput></screen>
580
581 <para>Notice that this time the command immediately returned to the shell
582 prompt without having to page through the output. You may now remove the
583 log file.</para>
584
585 <para>The last example will use the <command>yes</command> command in
586 combination with output redirection to bypass having to page through the
587 output and then provide a <keycap>y</keycap> to a prompt. This technique
588 could be used in instances when otherwise you would have to page through
589 the output of a file (such as a license agreement) and then answer the
590 question of <computeroutput>do you accept the above?</computeroutput>.
591 For this example,
592 another short <application>Bash</application> script is required:</para>
593
594<screen><userinput>cat &gt; blfs-yes-test2 &lt;&lt; "EOF"
595<literal>#!/bin/bash
596
597ls -l /usr/bin | less
598
599echo -n -e "\n\nDid you enjoy reading this? (y,n) "
600
601read A_STRING
602
603if test "$A_STRING" = "y"; then A_STRING="You entered the 'y' key"
604else A_STRING="You did NOT enter the 'y' key"
605fi
606
607echo -e "\n\n$A_STRING\n\n"</literal>
608EOF
609chmod 755 blfs-yes-test2</userinput></screen>
610
611 <para>This script can be used to simulate a program that requires you to
612 read a license agreement, then respond appropriately to accept the
613 agreement before the program will install anything. First, run the script
614 without any automation techniques by issuing
615 <command>./blfs-yes-test2</command>.</para>
616
617 <para>Now issue the following command which uses two automation techniques,
618 making it suitable for use in an automated build script:</para>
619
620<screen><userinput>yes | ./blfs-yes-test2 &gt; blfs-yes-test2.log 2&gt;&amp;1</userinput></screen>
621
622 <para>If desired, issue <command>tail blfs-yes-test2.log</command> to see
623 the end of the paged output, and confirmation that <keycap>y</keycap> was
624 passed through to the script. Once satisfied that it works as it should,
625 you may remove the script and log file.</para>
626
627 <para>Finally, keep in mind that there are many ways to automate and/or
628 script the build commands. There is not a single <quote>correct</quote> way
629 to do it. Your imagination is the only limit.</para>
630
631 </sect2>
632
633 <sect2>
634 <title>Dependencies</title>
635
636 <para>For each package described, BLFS lists the known dependencies.
637 These are listed under several headings, whose meaning is as follows:</para>
638
639 <itemizedlist>
640 <listitem>
641 <para><emphasis>Required</emphasis> means that the target package
642 cannot be correctly built without the dependency having first been
643 installed, except if the dependency is said to be
644 <quote>runtime</quote> which means the target package can be built but
645 cannot function without it.</para>
646 <para>
647 Note that a target package can start to <quote>function</quote>
648 in many subtle ways: an installed configuration file can make the
649 init system, cron daemon, or bus daemon to run a program
650 automatically; another package using the target package as a
651 dependency can run a program from the target package in the
652 building system; and the configuration sections in the BLFS book
653 may also run a program from a just installed package. So if
654 you are installing the target package without a
655 <emphasis>Required (runtime)</emphasis> dependency installed,
656 You should install the dependency as soon as possible after the
657 installation of the target package.
658 </para>
659 </listitem>
660 <listitem>
661 <para><emphasis>Recommended</emphasis> means that BLFS strongly
662 suggests this package is installed first (except if said to be
663 <quote>runtime,</quote> see below) for a clean and trouble-free
664 build, that won't have issues either during the build process, or at
665 run-time. The instructions in the book assume these packages are
666 installed. Some changes or workarounds may be required if these
667 packages are not installed. If a recommended dependency is said
668 to be <quote>runtime,</quote> it means that BLFS strongly suggests
669 that this dependency is installed before using the package, for
670 getting full functionality.</para>
671 </listitem>
672 <listitem>
673 <para><emphasis>Optional</emphasis> means that this package might be
674 installed for added functionality. Often BLFS will describe the
675 dependency to explain the added functionality that will result.
676 An optional dependency may be automatically picked up by the target
677 package if the dependency is installed, but another some optional
678 dependency may also need additional configuration options to enable
679 them when the target package is built. Such additional options are
680 often documented in the BLFS book. If an optional dependency is
681 said to be <quote>runtime,</quote> it means you may install
682 the dependency after installing the target package to support some
683 optional features of the target package if you need these
684 features.</para>
685 <para>An optional dependency may be out of BLFS. If you need such
686 an <emphasis>external</emphasis> optional dependency for some
687 features you need, read <xref linkend='beyond'/> for the general
688 hint about installing an out-of-BLFS package.</para>
689 </listitem>
690 </itemizedlist>
691
692 </sect2>
693
694 <sect2 id="package_updates">
695 <title>Using the Most Current Package Sources</title>
696
697 <para>On occasion you may run into a situation in the book when a package
698 will not build or work properly. Though the Editors attempt to ensure
699 that every package in the book builds and works properly, sometimes a
700 package has been overlooked or was not tested with this particular version
701 of BLFS.</para>
702
703 <para>If you discover that a package will not build or work properly, you
704 should see if there is a more current version of the package. Typically
705 this means you go to the maintainer's web site and download the most current
706 tarball and attempt to build the package. If you cannot determine the
707 maintainer's web site by looking at the download URLs, use Google and query
708 the package's name. For example, in the Google search bar type:
709 'package_name download' (omit the quotes) or something similar. Sometimes
710 typing: 'package_name home page' will result in you finding the
711 maintainer's web site.</para>
712
713 </sect2>
714
715 <sect2 id="stripping">
716 <title>Stripping One More Time</title>
717
718 <para>
719 In LFS, stripping of debugging symbols and unneeded symbol table
720 entries was discussed a couple of times. When building BLFS packages,
721 there are generally no special instructions that discuss stripping
722 again. Stripping can be done while installing a package, or
723 afterwards.
724 </para>
725
726 <bridgehead renderas="sect3" id="stripping-install">Stripping while Installing a Package</bridgehead>
727
728 <para>
729 There are several ways to strip executables installed by a
730 package. They depend on the build system used (see below <link
731 linkend="buildsystems">the section about build systems</link>),
732 so only some
733 generalities can be listed here:
734 </para>
735
736 <note>
737 <para>
738 The following methods using the feature of a building system
739 (autotools, meson, or cmake) will not strip static libraries if any
740 is installed. Fortunately there are not too many static libraries
741 in BLFS, and a static library can always be stripped safely by
742 running <command>strip --strip-unneeded</command> on it manually.
743 </para>
744 </note>
745
746 <itemizedlist>
747 <listitem>
748 <para>
749 The packages using autotools usually have an
750 <parameter>install-strip</parameter> target in their generated
751 <filename>Makefile</filename> files. So installing stripped
752 executables is just a matter of using
753 <command>make install-strip</command> instead of
754 <command>make install</command>.
755 </para>
756 </listitem>
757 <listitem>
758 <para>
759 The packages using the meson build system can accept
760 <parameter>-D strip=true</parameter> when running
761 <command>meson</command>. If you've forgot to add this option
762 running the <command>meson</command>, you can also run
763 <command>meson install --strip</command> instead of
764 <command>ninja install</command>.
765 </para>
766 </listitem>
767 <listitem>
768 <para>
769 <command>cmake</command> generates
770 <parameter>install/strip</parameter> targets for both the
771 <parameter>Unix Makefiles</parameter> and
772 <parameter>Ninja</parameter> generators (the default is
773 <parameter>Unix Makefiles</parameter> on linux). So just run
774 <command>make install/strip</command> or
775 <command>ninja install/strip</command> instead of the
776 <command>install</command> counterparts.
777 </para>
778 </listitem>
779 <listitem>
780 <para>
781 Removing (or not generating) debug symbols can also be
782 achieved by removing the
783 <parameter>-g&lt;something&gt;</parameter> options
784 in C/C++ calls. How to do that is very specific for each
785 package. And, it does not remove unneeded symbol table entries.
786 So it will not be explained in detail here. See also below
787 the paragraphs about optimization.
788 </para>
789 </listitem>
790 </itemizedlist>
791
792 <bridgehead renderas="sect3" id="stripping-installed">Stripping Installed Executables</bridgehead>
793
794 <para>
795 The <command>strip</command> utility changes files in place, which may
796 break anything using it if it is loaded in memory. Note that if a file is
797 in use but just removed from the disk (i.e. not overwritten nor
798 modified), this is not a problem since the kernel can use
799 <quote>deleted</quote> files. Look at <filename>/proc/*/maps</filename>
800 and it is likely that you'll see some <emphasis>(deleted)</emphasis>
801 entries. The <command>mv</command> just removes the destination file from
802 the directory but does not touch its content, so that it satisfies the
803 condition for the kernel to use the old (deleted) file.
804 But this approach can detach hard links into duplicated copies,
805 causing a bloat which is obviously unwanted as we are stripping to
806 reduce system size. If two files in a same file system share the
807 same inode number, they are hard links to each other and we should
808 reconstruct the link. The script below is just an example.
809 It should be run as the &root; user:
810 </para>
811
812<screen><userinput>cat &gt; /usr/sbin/strip-all.sh &lt;&lt; "EOF"
813<literal>#!/usr/bin/bash
814
815if [ $EUID -ne 0 ]; then
816 echo "Need to be root"
817 exit 1
818fi
819
820last_fs_inode=
821last_file=
822
823{ find /usr/lib -type f -name '*.so*' ! -name '*dbg'
824 find /usr/lib -type f -name '*.a'
825 find /usr/{bin,sbin,libexec} -type f
826} | xargs stat -c '%m %i %n' | sort | while read fs inode file; do
827 if ! readelf -h $file >/dev/null 2>&amp;1; then continue; fi
828 if file $file | grep --quiet --invert-match 'not stripped'; then continue; fi
829
830 if [ "$fs $inode" = "$last_fs_inode" ]; then
831 ln -f $last_file $file;
832 continue;
833 fi
834
835 cp --preserve $file ${file}.tmp
836 strip --strip-unneeded ${file}.tmp
837 mv ${file}.tmp $file
838
839 last_fs_inode="$fs $inode"
840 last_file=$file
841done</literal>
842EOF
843chmod 744 /usr/sbin/strip-all.sh</userinput></screen>
844
845 <para>
846 If you install programs in other directories such as <filename
847 class="directory">/opt</filename> or <filename
848 class="directory">/usr/local</filename>, you may want to strip the files
849 there too. Just add other directories to scan in the compound list of
850 <command>find</command> commands between the braces.
851 </para>
852
853 <para>
854 For more information on stripping, see <ulink
855 url="https://www.technovelty.org/linux/stripping-shared-libraries.html"/>.
856 </para>
857
858 </sect2>
859
860<!--
861 <sect2 id="libtool">
862 <title>Libtool files</title>
863
864 <para>
865 One of the side effects of packages that use Autotools, including
866 libtool, is that they create many files with an .la extension. These
867 files are not needed in an LFS environment. If there are conflicts with
868 pkgconfig entries, they can actually prevent successful builds. You
869 may want to consider removing these files periodically:
870 </para>
871
872<screen><userinput>find /lib /usr/lib -not -path "*Image*" -a -name \*.la -delete</userinput></screen>
873
874 <para>
875 The above command removes all .la files with the exception of those that
876 have <quote>Image</quote> or <quote>openldap</quote> as a part of the
877 path. These .la files are used by the ImageMagick and openldap programs,
878 respectively. There may be other exceptions by packages not in BLFS.
879 </para>
880
881 </sect2>
882-->
883 <sect2 id="buildsystems">
884 <title>Working with different build systems</title>
885
886 <para>
887 There are now three different build systems in common use for
888 converting C or C++ source code into compiled programs or
889 libraries and their details (particularly, finding out about available
890 options and their default values) differ. It may be easiest to understand
891 the issues caused by some choices (typically slow execution or
892 unexpected use of, or omission of, optimizations) by starting with
893 the <envar>CFLAGS</envar>, <envar>CXXFLAGS</envar>, and
894 <envar>LDFLAGS</envar> environment variables. There are also some
895 programs which use Rust.
896 </para>
897
898 <para>
899 Most LFS and BLFS builders are probably aware of the basics of
900 <envar>CFLAGS</envar> and <envar>CXXFLAGS</envar> for altering how a
901 program is compiled. Typically, some form of optimization is used by
902 upstream developers (<option>-O2</option> or <option>-O3</option>),
903 sometimes with the creation of debug symbols (<option>-g</option>),
904 as defaults.
905 </para>
906
907 <para>
908 If there are contradictory flags (e.g. multiple different
909 <option>-O</option> values),
910 the <emphasis>last</emphasis> value will be used. Sometimes this means
911 that flags specified in environment variables will be picked up before
912 values hardcoded in the Makefile, and therefore ignored. For example,
913 where a user specifies <option>-O2</option> and that is followed by
914 <option>-O3</option> the build will use <option>-O3</option>.
915 </para>
916
917 <para>
918 There are various other things which can be passed in CFLAGS or
919 CXXFLAGS, such as allowing using the instruction set extensions
920 available with a specific microarchitecture (e.g.
921 <option>-march=amdfam10</option> or <option>-march=native</option>),
922 tune the generated code for a specific microarchitecture (e. g.
923 <option>-mtune=tigerlake</option> or <option>-mtune=native</option>,
924 if <option>-mtune=</option> is not used, the microarchitecture from
925 <option>-march=</option> setting will be used), or specifying a
926 specific standard for C or C++ (<option>-std=c++17</option> for
927 example). But one thing which has now come to light is that
928 programmers might include debug assertions in their code, expecting
929 them to be disabled in releases by using <option>-D NDEBUG</option>.
930 Specifically, if <xref linkend="mesa"/> is built with these
931 assertions enabled, some activities such as loading levels of games
932 can take extremely long times, even on high-class video cards.
933 </para>
934
935 <bridgehead renderas="sect3" id="autotools-info">Autotools with Make</bridgehead>
936
937 <para>
938 This combination is often described as <quote>CMMI</quote>
939 (configure, make, make install) and is used here to also cover
940 the few packages which have a configure script that is not
941 generated by autotools.
942 </para>
943
944 <para>
945 Sometimes running <command>./configure --help</command> will produce
946 useful options about switches which might be used. At other times,
947 after looking at the output from configure you may need to look
948 at the details of the script to find out what it was actually searching
949 for.
950 </para>
951
952 <para>
953 Many configure scripts will pick up any CFLAGS or CXXFLAGS from the
954 environment, but CMMI packages vary about how these will be mixed with
955 any flags which would otherwise be used (<emphasis>variously</emphasis>:
956 ignored, used to replace the programmer's suggestion, used before the
957 programmer's suggestion, or used after the programmer's suggestion).
958 </para>
959
960 <para>
961 In most CMMI packages, running <command>make</command> will list
962 each command and run it, interspersed with any warnings. But some
963 packages try to be <quote>silent</quote> and only show which file
964 they are compiling or linking instead of showing the command line.
965 If you need to inspect the command, either because of an error, or
966 just to see what options and flags are being used, adding
967 <option>V=1</option> to the make invocation may help.
968 </para>
969
970 <bridgehead renderas="sect3" id="cmake-info">CMake</bridgehead>
971
972 <para>
973 CMake works in a very different way, and it has two backends which
974 can be used on BLFS: <command>make</command> and
975 <command>ninja</command>. The default backend is make, but
976 ninja can be faster on large packages with multiple processors. To
977 use ninja, specify <option>-G Ninja</option> in the cmake command.
978 However, there are some packages which create fatal errors in their
979 ninja files but build successfully using the default of Unix
980 Makefiles.
981 </para>
982
983 <para>
984 The hardest part of using CMake is knowing what options you might wish
985 to specify. The only way to get a list of what the package knows about
986 is to run <command>cmake -LAH</command> and look at the output for that
987 default configuration.
988 </para>
989
990 <para>
991 Perhaps the most-important thing about CMake is that it has a variety
992 of CMAKE_BUILD_TYPE values, and these affect the flags. The default
993 is that this is not set and no flags are generated. Any
994 <envar>CFLAGS</envar> or <envar>CXXFLAGS</envar> in the environment
995 will be used. If the programmer has coded any debug assertions,
996 those will be enabled unless -D NDEBUG is used. The following
997 CMAKE_BUILD_TYPE values will generate the flags shown, and these
998 will come <emphasis>after</emphasis> any flags in the environment
999 and therefore take precedence.
1000 </para>
1001
1002 <informaltable align="center">
1003 <tgroup cols="2">
1004 <colspec colnum="1" align="center"/>
1005 <colspec colnum="2" align="center"/>
1006 <thead>
1007 <row><entry>Value</entry><entry>Flags</entry></row>
1008 </thead>
1009 <tbody>
1010 <row>
1011 <entry>Debug</entry><entry><option>-g</option></entry>
1012 </row>
1013 <row>
1014 <entry>Release</entry><entry><option>-O3 -D NDEBUG</option></entry>
1015 </row>
1016 <row>
1017 <entry>RelWithDebInfo</entry><entry><option>-O2 -g -D NDEBUG</option></entry>
1018 </row>
1019 <row>
1020 <entry>MinSizeRel</entry><entry><option>-Os -D NDEBUG</option></entry>
1021 </row>
1022 </tbody>
1023 </tgroup>
1024 </informaltable>
1025
1026 <para>
1027 CMake tries to produce quiet builds. To see the details of the commands
1028 which are being run, use <command>make VERBOSE=1</command> or
1029 <command>ninja -v</command>.
1030 </para>
1031
1032 <para>
1033 By default, CMake treats file installation differently from the other
1034 build systems: if a file already exists and is not newer than a file
1035 that would overwrite it, then the file is not installed. This may be
1036 a problem if a user wants to record which file belongs to a package,
1037 either using <envar>LD_PRELOAD</envar>, or by listing files newer
1038 than a timestamp. The default can be changed by setting the variable
1039 <envar>CMAKE_INSTALL_ALWAYS</envar> to 1 in the
1040 <emphasis>environment</emphasis>, for example by
1041 <command>export</command>'ing it.
1042 </para>
1043
1044 <bridgehead renderas="sect3" id="meson-info">Meson</bridgehead>
1045
1046 <para>
1047 Meson has some similarities to CMake, but many differences. To get
1048 details of the defines that you may wish to change you can look at
1049 <filename>meson_options.txt</filename> which is usually in the
1050 top-level directory.
1051 </para>
1052
1053 <para>
1054 If you have already configured the package by running
1055 <command>meson</command> and now wish to change one or more settings,
1056 you can either remove the build directory, recreate it, and use the
1057 altered options, or within the build directory run <command>meson
1058 configure</command>, e.g. to set an option:
1059 </para>
1060
1061<screen><userinput>meson configure -D &lt;some_option&gt;=true</userinput></screen>
1062
1063 <para>
1064 If you do that, the file <filename>meson-private/cmd_line.txt</filename>
1065 will show the <emphasis>last</emphasis> commands which were used.
1066 </para>
1067
1068 <para>
1069 Meson provides the following buildtype values, and the flags they enable
1070 come <emphasis>after</emphasis> any flags supplied in the environment and
1071 therefore take precedence.
1072 </para>
1073
1074 <itemizedlist>
1075 <listitem>
1076 <para>plain : no added flags. This is for distributors to supply their
1077 own <envar>CFLAGS</envar>, <envar>CXXFLAGS</envar> and
1078 <envar>LDFLAGS</envar>. There is no obvious reason to use
1079 this in BLFS.</para>
1080 </listitem>
1081 <listitem>
1082 <para>debug : <option>-g</option> - this is the default if
1083 nothing is specified in either <filename>meson.build</filename>
1084 or the command line. However it results large and slow binaries,
1085 so we should override it in BLFS.</para>
1086 </listitem>
1087 <listitem>
1088 <para>debugoptimized : <option>-O2 -g</option> - this is the
1089 default specified in <filename>meson.build</filename> of some
1090 packages.</para>
1091 </listitem>
1092 <listitem>
1093 <para>release : <option>-O3</option> (occasionally a package will
1094 force <option>-O2</option> here) - this is the buildtype we use
1095 for most packages with Meson build system in BLFS.</para>
1096 </listitem>
1097 </itemizedlist>
1098
1099 <!-- From https://mesonbuild.com/Builtin-options.html#core-options:
1100 b_ndebug: Default value = false, Possible values are
1101 true, false, if-release. Some packages sets it to if-release
1102 so we mistakenly believed if-release had been the default. -->
1103 <para>
1104 The <option>-D NDEBUG</option> flag is implied by the release
1105 buildtype for some packages (for example <xref linkend='mesa'/>).
1106 It can also be provided explicitly by passing
1107 <option>-D b_ndebug=true</option>.
1108 </para>
1109
1110 <para>
1111 To see the details of the commands which are being run in a package using
1112 meson, use <command>ninja -v</command>.
1113 </para>
1114
1115 <bridgehead renderas="sect3" id="rust-info">Rustc and Cargo</bridgehead>
1116
1117 <para>
1118 Most released rustc programs are provided as crates (source tarballs)
1119 which will query a server to check current versions of dependencies
1120 and then download them as necessary. These packages are built using
1121 <command>cargo --release</command>. In theory, you can manipulate the
1122 RUSTFLAGS to change the optimize-level (default for
1123 <option>--release</option> is 3, i. e.
1124 <option>-Copt-level=3</option>, like <option>-O3</option>) or to
1125 force it to build for the machine it is being compiled on, using
1126 <option>-Ctarget-cpu=native</option> but in practice this seems to
1127 make no significant difference.
1128 </para>
1129
1130 <para>
1131 If you are compiling a standalone Rust program (as an unpackaged
1132 <filename class='extension'>.rs</filename> file) by running
1133 <command>rustc</command> directly, you should specify
1134 <option>-O</option> (the abbreviation of
1135 <option>-Copt-level=2</option>) or <option>-Copt-level=3</option>
1136 otherwise it will do an unoptimized compile and run
1137 <emphasis>much</emphasis> slower. If you are compiling the program
1138 for debugging it, replace the <option>-O</option> or
1139 <option>-Copt-level=</option> options with <option>-g</option> to
1140 produce an unoptimized program with debug info.
1141 </para>
1142
1143 <para>
1144 Like <command>ninja</command>, by default <command>cargo</command>
1145 uses all logical cores. This can often be worked around,
1146 either by exporting
1147 <envar>CARGO_BUILD_JOBS=<replaceable>&lt;N&gt;</replaceable></envar>
1148 or passing
1149 <option>--jobs <replaceable>&lt;N&gt;</replaceable></option> to
1150 <command>cargo</command>.
1151 For compiling rustc itself, specifying
1152 <option>--jobs <replaceable>&lt;N&gt;</replaceable></option> for
1153 invocations of <command>x.py</command>
1154 (together with the <envar>CARGO_BUILD_JOBS</envar> environment
1155 variable, which looks like a <quote>belt and braces</quote>
1156 approach but seems to be necessary) mostly works. The exception is
1157 running the tests when building rustc, some of them will
1158 nevertheless use all online CPUs, at least as of rustc-1.42.0.
1159 </para>
1160
1161 </sect2>
1162
1163 <sect2 id="optimizations">
1164 <title>Optimizing the build</title>
1165
1166 <para>
1167 Many people will prefer to optimize compiles as they see fit, by providing
1168 <envar>CFLAGS</envar> or <envar>CXXFLAGS</envar>. For an
1169 introduction to the options available with gcc and g++ see <ulink
1170 url="https://gcc.gnu.org/onlinedocs/gcc-&gcc-version;/gcc/Optimize-Options.html"/>.
1171 The same content can be also found in <command>info gcc</command>.
1172 </para>
1173
1174 <para>
1175 Some packages default to <option>-O2 -g</option>, others to
1176 <option>-O3 -g</option>, and if <envar>CFLAGS</envar> or
1177 <envar>CXXFLAGS</envar> are supplied they might be added to the
1178 package's defaults, replace the package's defaults, or even be
1179 ignored. There are details on some desktop packages which were
1180 mostly current in April 2019 at
1181 <ulink url="https://www.linuxfromscratch.org/~ken/tuning/"/> - in
1182 particular, <filename>README.txt</filename>,
1183 <filename>tuning-1-packages-and-notes.txt</filename>, and
1184 <filename>tuning-notes-2B.txt</filename>. The particular thing to
1185 remember is that if you want to try some of the more interesting
1186 flags you may need to force verbose builds to confirm what is being
1187 used.
1188 </para>
1189
1190 <para>
1191 Clearly, if you are optimizing your own program you can spend time to
1192 profile it and perhaps recode some of it if it is too slow. But for
1193 building a whole system that approach is impractical. In general,
1194 <option>-O3</option> usually produces faster programs than
1195 <option>-O2</option>. Specifying
1196 <option>-march=native</option> is also beneficial, but means that
1197 you cannot move the binaries to an incompatible machine - this can
1198 also apply to newer machines, not just to older machines. For
1199 example programs compiled for <literal>amdfam10</literal> run on
1200 old Phenoms, Kaveris, and Ryzens : but programs compiled for a
1201 Kaveri will not run on a Ryzen because certain op-codes are not
1202 present. Similarly, if you build for a Haswell not everything will
1203 run on a SandyBridge.
1204 </para>
1205
1206 <note>
1207 <para>
1208 Be careful that the name of a <option>-march</option> setting
1209 does not always match the baseline of the microarchitecture
1210 with the same name. For example, the Skylake-based Intel Celeron
1211 processors do not support AVX at all, but
1212 <option>-march=skylake</option> assumes AVX and even AVX2.
1213 </para>
1214 </note>
1215
1216 <para>
1217 When a shared library is built by GCC, a feature named
1218 <quote>semantic interposition</quote> is enabled by default. When
1219 the shared library refers to a symbol name with external linkage
1220 and default visibility, if the symbol exists in both the shared
1221 library and the main executable, semantic interposition guarantees
1222 the symbol in the main executable is always used. This feature
1223 was invented in an attempt to make the behavior of linking a shared
1224 library and linking a static library as similar as possible. Today
1225 only a small number of packages still depend on semantic
1226 interposition, but the feature is still on by the default of GCC,
1227 causing many optimizations disabled for shared libraries because
1228 they conflict with semantic interposition. The
1229 <option>-fno-semantic-interposition</option> option can be passed
1230 to <command>gcc</command> or <command>g++</command> to disable
1231 semantic interposition and enable more optimizations for shared
1232 libraries. This option is used as the default of some packages
1233 (for example <xref linkend='python3'/>), and it's also the default
1234 of Clang.
1235 </para>
1236
1237 <para>
1238 There are also various other options which some people claim are
1239 beneficial. At worst, you get to recompile and test, and then
1240 discover that in your usage the options do not provide a benefit.
1241 </para>
1242
1243 <para>
1244 If building Perl or Python modules,
1245 in general the <envar>CFLAGS</envar> and <envar>CXXFLAGS</envar>
1246 used are those which were used by those <quote>parent</quote>
1247 packages.
1248 </para>
1249
1250 <para>
1251 For <envar>LDFLAGS</envar>, there are three options can be used
1252 for optimization. They are quite safe to use and the building
1253 system of some packages use some of these options as the default.
1254 </para>
1255
1256 <para>
1257 With <option>-Wl,-O1</option>, the linker will
1258 optimize the hash table to speed up the dynamic linking.
1259 Note that <option>-Wl,-O1</option> is completely unrelated to the
1260 compiler optimization flag <option>-O1</option>.
1261 </para>
1262
1263 <para>
1264 With <option>-Wl,--as-needed</option>, the linker will disregard
1265 unnecessary <option>-l<replaceable>foo</replaceable></option> options
1266 from the command line, i. e. the shared library <systemitem
1267 class='library'>lib<replaceable>foo</replaceable></systemitem>
1268 will only be linked if a symbol in <systemitem
1269 class='library'>lib<replaceable>foo</replaceable></systemitem> is
1270 really referred from the executable or shared library being linked.
1271 This can sometimes mitigate the <quote>excessive dependencies to
1272 shared libraries</quote> issues caused by
1273 <application>libtool</application>.
1274 </para>
1275
1276 <para>
1277 With <option>-Wl,-z,pack-relative-relocs</option>, the linker
1278 generates a more compacted form of the relative relocation entries
1279 for PIEs and shared libraries. It reduces the size of the linked
1280 PIE or shared library, and speeds up the loading of the PIE or
1281 shared library.
1282 </para>
1283
1284 <para>
1285 The <option>-Wl,</option> prefix is necessary because despite the
1286 variable is named <envar>LDFLAGS</envar>, its content is actually
1287 passed to <command>gcc</command> (or <command>g++</command>,
1288 <command>clang</command>, etc.) during the link stage, not directly
1289 passed to <command>ld</command>.
1290 </para>
1291
1292 </sect2>
1293
1294 <sect2 id="hardening">
1295 <title>Options for hardening the build</title>
1296
1297 <para>
1298 Even on desktop systems, there are still a lot of exploitable
1299 vulnerabilities. For many of these, the attack comes via javascript
1300 in a browser. Often, a series of vulnerabilities are used to gain
1301 access to data (or sometimes to pwn, i.e. own, the machine and
1302 install rootkits). Most commercial distros will apply various
1303 hardening measures.
1304 </para>
1305
1306 <para>
1307 In the past, there was Hardened LFS where gcc (a much older version)
1308 was forced to use hardening (with options to turn some of it off on a
1309 per-package basis). The current LFS and BLFS books are carrying
1310 forward a part of its spirit by enabling PIE
1311 (<option>-fPIE -pie</option>) and SSP
1312 (<option>-fstack-protector-strong</option>) as the defaults
1313 for GCC and clang. What is being covered here is different - first
1314 you have to make sure that the package is indeed using your added
1315 flags and not over-riding them.
1316 </para>
1317
1318 <para>
1319 For hardening options which are reasonably cheap, there is some
1320 discussion in the 'tuning' link above (occasionally, one or more
1321 of these options might be inappropriate for a package). These
1322 options are <option>-D _FORTIFY_SOURCE=2</option>
1323 (or <option>-D _FORTIFY_SOURCE=3</option> which is more secure but
1324 with a larger performance overhead) and
1325 (for C++) <option>-D _GLIBCXX_ASSERTIONS</option>. On modern
1326 machines these should only have a little impact on how fast things
1327 run, and often they will not be noticeable.
1328 </para>
1329
1330 <para>
1331 The main distros use much more, such as RELRO (Relocation Read Only)
1332 and perhaps <option>-fstack-clash-protection</option>. You may also
1333 encounter the so-called <quote>userspace retpoline</quote>
1334 (<option>-mindirect-branch=thunk</option> etc.) which
1335 is the equivalent of the spectre mitigations applied to the linux
1336 kernel in late 2018. The kernel mitigations caused a lot of complaints
1337 about lost performance, if you have a production server you might wish
1338 to consider testing that, along with the other available options, to
1339 see if performance is still sufficient.
1340 </para>
1341
1342 <para>
1343 Whilst gcc has many hardening options, clang/LLVM's strengths lie
1344 elsewhere. Some options which gcc provides are said to be less effective
1345 in clang/LLVM.
1346 </para>
1347
1348 </sect2>
1349
1350</sect1>
Note: See TracBrowser for help on using the repository browser.