1 | <?xml version="1.0" encoding="ISO-8859-1"?>
|
---|
2 | <!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
|
---|
3 | "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
|
---|
4 | <!ENTITY % general-entities SYSTEM "../../general.ent">
|
---|
5 | %general-entities;
|
---|
6 | ]>
|
---|
7 |
|
---|
8 | <sect1 id="unpacking">
|
---|
9 | <?dbhtml filename="notes-on-building.html"?>
|
---|
10 |
|
---|
11 |
|
---|
12 | <title>Notes on Building Software</title>
|
---|
13 |
|
---|
14 | <para>Those people who have built an LFS system may be aware
|
---|
15 | of the general principles of downloading and unpacking software. Some
|
---|
16 | of that information is repeated here for those new to building
|
---|
17 | their own software.</para>
|
---|
18 |
|
---|
19 | <para>Each set of installation instructions contains a URL from which you
|
---|
20 | can download the package. The patches; however, are stored on the LFS
|
---|
21 | servers and are available via HTTP. These are referenced as needed in the
|
---|
22 | installation instructions.</para>
|
---|
23 |
|
---|
24 | <para>While you can keep the source files anywhere you like, we assume that
|
---|
25 | you have unpacked the package and changed into the directory created by the
|
---|
26 | unpacking process (the source directory). We also assume you have
|
---|
27 | uncompressed any required patches and they are in the directory
|
---|
28 | immediately above the source directory.</para>
|
---|
29 |
|
---|
30 | <para>We can not emphasize strongly enough that you should start from a
|
---|
31 | <emphasis>clean source tree</emphasis> each time. This means that if
|
---|
32 | you have had an error during configuration or compilation, it's usually
|
---|
33 | best to delete the source tree and
|
---|
34 | re-unpack it <emphasis>before</emphasis> trying again. This obviously
|
---|
35 | doesn't apply if you're an advanced user used to hacking
|
---|
36 | <filename>Makefile</filename>s and C code, but if in doubt, start from a
|
---|
37 | clean tree.</para>
|
---|
38 |
|
---|
39 | <sect2>
|
---|
40 | <title>Building Software as an Unprivileged (non-root) User</title>
|
---|
41 |
|
---|
42 | <para>The golden rule of Unix System Administration is to use your
|
---|
43 | superpowers only when necessary. Hence, BLFS recommends that you
|
---|
44 | build software as an unprivileged user and only become the
|
---|
45 | <systemitem class='username'>root</systemitem> user when installing the
|
---|
46 | software. This philosophy is followed in all the packages in this book.
|
---|
47 | Unless otherwise specified, all instructions should be executed as an
|
---|
48 | unprivileged user. The book will advise you on instructions that need
|
---|
49 | <systemitem class='username'>root</systemitem> privileges.</para>
|
---|
50 |
|
---|
51 | </sect2>
|
---|
52 |
|
---|
53 | <sect2>
|
---|
54 | <title>Unpacking the Software</title>
|
---|
55 |
|
---|
56 | <para>If a file is in <filename class='extension'>.tar</filename> format
|
---|
57 | and compressed, it is unpacked by running one of the following
|
---|
58 | commands:</para>
|
---|
59 |
|
---|
60 | <screen><userinput>tar -xvf filename.tar.gz
|
---|
61 | tar -xvf filename.tgz
|
---|
62 | tar -xvf filename.tar.Z
|
---|
63 | tar -xvf filename.tar.bz2</userinput></screen>
|
---|
64 |
|
---|
65 | <note>
|
---|
66 | <para>You may omit using the <option>v</option> parameter in the commands
|
---|
67 | shown above and below if you wish to suppress the verbose listing of all
|
---|
68 | the files in the archive as they are extracted. This can help speed up the
|
---|
69 | extraction as well as make any errors produced during the extraction
|
---|
70 | more obvious to you.</para>
|
---|
71 | </note>
|
---|
72 |
|
---|
73 | <para>You can also use a slightly different method:</para>
|
---|
74 |
|
---|
75 | <screen><userinput>bzcat filename.tar.bz2 | tar -xv</userinput></screen>
|
---|
76 |
|
---|
77 | <para>
|
---|
78 | Finally, sometimes we have a compressed patch file in
|
---|
79 | <filename class='extension'>.patch.gz</filename> or
|
---|
80 | <filename class='extension'>.patch.bz2</filename> format.
|
---|
81 | The best way to apply the patch is piping the output of the
|
---|
82 | decompressor to the <command>patch</command> utility. For example:
|
---|
83 | </para>
|
---|
84 |
|
---|
85 | <screen><userinput>gzip -cd ../patchname.patch.gz | patch -p1</userinput></screen>
|
---|
86 |
|
---|
87 | <para>
|
---|
88 | Or for a patch compressed with <command>bzip2</command>:
|
---|
89 | </para>
|
---|
90 |
|
---|
91 | <screen><userinput>bzcat ../patchname.patch.bz2 | patch -p1</userinput></screen>
|
---|
92 |
|
---|
93 | </sect2>
|
---|
94 |
|
---|
95 | <sect2>
|
---|
96 | <title>Verifying File Integrity</title>
|
---|
97 |
|
---|
98 | <para>Generally, to verify that the downloaded file is complete,
|
---|
99 | many package maintainers also distribute md5sums of the files. To verify the
|
---|
100 | md5sum of the downloaded files, download both the file and the
|
---|
101 | corresponding md5sum file to the same directory (preferably from different
|
---|
102 | on-line locations), and (assuming <filename>file.md5sum</filename> is the
|
---|
103 | md5sum file downloaded) run the following command:</para>
|
---|
104 |
|
---|
105 | <screen><userinput>md5sum -c file.md5sum</userinput></screen>
|
---|
106 |
|
---|
107 | <para>If there are any errors, they will be reported. Note that the BLFS
|
---|
108 | book includes md5sums for all the source files also. To use the BLFS
|
---|
109 | supplied md5sums, you can create a <filename>file.md5sum</filename> (place
|
---|
110 | the md5sum data and the exact name of the downloaded file on the same
|
---|
111 | line of a file, separated by white space) and run the command shown above.
|
---|
112 | Alternately, simply run the command shown below and compare the output
|
---|
113 | to the md5sum data shown in the BLFS book.</para>
|
---|
114 |
|
---|
115 | <screen><userinput>md5sum <replaceable><name_of_downloaded_file></replaceable></userinput></screen>
|
---|
116 |
|
---|
117 | <para>MD5 is not cryptographically secure, so the md5sums are only
|
---|
118 | provided for detecting unmalicious changes to the file content. For
|
---|
119 | example, an error or truncation introduced during network transfer, or
|
---|
120 | a <quote>stealth</quote> update to the package from the upstream
|
---|
121 | (updating the content of a released tarball instead of making a new
|
---|
122 | release properly).</para>
|
---|
123 |
|
---|
124 | <para>There is no <quote>100%</quote> secure way to make
|
---|
125 | sure the genuity of the source files. Assuming the upstream is managing
|
---|
126 | their website correctly (the private key is not leaked and the domain is
|
---|
127 | not hijacked), and the trust anchors have been set up correctly using
|
---|
128 | <xref linkend="make-ca"/> on the BLFS system, we can reasonably trust
|
---|
129 | download URLs to the upstream official website
|
---|
130 | <emphasis role="bold">with https protocol</emphasis>. Note that
|
---|
131 | BLFS book itself is published on a website with https, so you should
|
---|
132 | already have some confidence in https protocol or you wouldn't trust the
|
---|
133 | book content.</para>
|
---|
134 |
|
---|
135 | <para>If the package is downloaded from an unofficial location (for
|
---|
136 | example a local mirror), checksums generated by cryptographically secure
|
---|
137 | digest algorithms (for example SHA256) can be used to verify the
|
---|
138 | genuity of the package. Download the checksum file from the upstream
|
---|
139 | <emphasis role="bold">official</emphasis> website (or somewhere
|
---|
140 | <emphasis role="bold">you can trust</emphasis>) and compare the
|
---|
141 | checksum of the package from unofficial location with it. For example,
|
---|
142 | SHA256 checksum can be checked with the command:</para>
|
---|
143 |
|
---|
144 | <note>
|
---|
145 | <para>If the checksum and the package are downloaded from the same
|
---|
146 | untrusted location, you won't gain security enhancement by verifying
|
---|
147 | the package with the checksum. The attacker can fake the checksum as
|
---|
148 | well as compromising the package itself.</para>
|
---|
149 | </note>
|
---|
150 |
|
---|
151 | <screen><userinput>sha256sum -c <replaceable>file</replaceable>.sha256sum</userinput></screen>
|
---|
152 |
|
---|
153 | <para>If <xref linkend="gnupg2"/> is installed, you can also verify the
|
---|
154 | genuity of the package with a GPG signature. Import the upstream GPG
|
---|
155 | public key with:</para>
|
---|
156 |
|
---|
157 | <screen><userinput>gpg --recv-key <replaceable>keyID</replaceable></userinput></screen>
|
---|
158 |
|
---|
159 | <para><replaceable>keyID</replaceable> should be replaced with the key ID
|
---|
160 | from somewhere <emphasis role="bold">you can trust</emphasis> (for
|
---|
161 | example, copy it from the upstream official website using https). Now
|
---|
162 | you can verify the signature with:</para>
|
---|
163 |
|
---|
164 | <screen><userinput>gpg --recv-key <replaceable>file</replaceable>.sig <replaceable>file</replaceable></userinput></screen>
|
---|
165 |
|
---|
166 | <para>The advantage of <application>GnuPG</application> signature is,
|
---|
167 | once you imported a public key which can be trusted, you can download
|
---|
168 | both the package and its signature from the same unofficial location and
|
---|
169 | verify them with the public key. So you won't need to connect to the
|
---|
170 | official upstream website to retrieve a checksum for each new release.
|
---|
171 | You only need to update the public key if it's expired or revoked.
|
---|
172 | </para>
|
---|
173 |
|
---|
174 | </sect2>
|
---|
175 |
|
---|
176 | <sect2>
|
---|
177 | <title>Creating Log Files During Installation</title>
|
---|
178 |
|
---|
179 | <para>For larger packages, it is convenient to create log files instead of
|
---|
180 | staring at the screen hoping to catch a particular error or warning. Log
|
---|
181 | files are also useful for debugging and keeping records. The following
|
---|
182 | command allows you to create an installation log. Replace
|
---|
183 | <replaceable><command></replaceable> with the command you intend to execute.</para>
|
---|
184 |
|
---|
185 | <screen><userinput>( <replaceable><command></replaceable> 2>&1 | tee compile.log && exit $PIPESTATUS )</userinput></screen>
|
---|
186 |
|
---|
187 | <para><option>2>&1</option> redirects error messages to the same
|
---|
188 | location as standard output. The <command>tee</command> command allows
|
---|
189 | viewing of the output while logging the results to a file. The parentheses
|
---|
190 | around the command run the entire command in a subshell and finally the
|
---|
191 | <command>exit $PIPESTATUS</command> command ensures the result of the
|
---|
192 | <replaceable><command></replaceable> is returned as the result and not the
|
---|
193 | result of the <command>tee</command> command.</para>
|
---|
194 |
|
---|
195 | </sect2>
|
---|
196 |
|
---|
197 | <sect2 id="parallel-builds" xreflabel="Using Multiple Processors">
|
---|
198 | <title>Using Multiple Processors</title>
|
---|
199 |
|
---|
200 | <para>For many modern systems with multiple processors (or cores) the
|
---|
201 | compilation time for a package can be reduced by performing a "parallel
|
---|
202 | make" by either setting an environment variable or telling the make program
|
---|
203 | how many processors are available. For instance, a Core2Duo can support two
|
---|
204 | simultaneous processes with: </para>
|
---|
205 |
|
---|
206 | <screen><userinput>export MAKEFLAGS='-j2'</userinput></screen>
|
---|
207 |
|
---|
208 | <para>or just building with:</para>
|
---|
209 |
|
---|
210 | <screen><userinput>make -j2</userinput></screen>
|
---|
211 |
|
---|
212 | <para>
|
---|
213 | If you have applied the optional <command>sed</command> when building
|
---|
214 | <application>ninja</application> in LFS, you can use:
|
---|
215 | </para>
|
---|
216 |
|
---|
217 | <screen><userinput>export NINJAJOBS=2</userinput></screen>
|
---|
218 |
|
---|
219 | <para>
|
---|
220 | when a package uses <command>ninja</command>, or just:
|
---|
221 | </para>
|
---|
222 |
|
---|
223 | <screen><userinput>ninja -j2</userinput></screen>
|
---|
224 |
|
---|
225 | <para>
|
---|
226 | but for ninja, the default number of jobs is N + 2, if
|
---|
227 | the number of logical processors N is greater than 2; or N + 1 if
|
---|
228 | N is 1 or 2. The reason to use a number of jobs slightly greater
|
---|
229 | than the number of logical processors is keeping all logical
|
---|
230 | processors busy even if some jobs are performing I/O operations.
|
---|
231 | </para>
|
---|
232 |
|
---|
233 | <para>Generally the number of processes should not exceed the number of
|
---|
234 | cores supported by the CPU too much. To list the processors on your
|
---|
235 | system, issue: <userinput>grep processor /proc/cpuinfo</userinput>.
|
---|
236 | </para>
|
---|
237 |
|
---|
238 | <para>In some cases, using multiple processes may result in a race
|
---|
239 | condition where the success of the build depends on the order of the
|
---|
240 | commands run by the <command>make</command> program. For instance, if an
|
---|
241 | executable needs File A and File B, attempting to link the program before
|
---|
242 | one of the dependent components is available will result in a failure.
|
---|
243 | This condition usually arises because the upstream developer has not
|
---|
244 | properly designated all the prerequisites needed to accomplish a step in the
|
---|
245 | Makefile.</para>
|
---|
246 |
|
---|
247 | <para>If this occurs, the best way to proceed is to drop back to a
|
---|
248 | single processor build. Adding <option>-j1</option> to a make command
|
---|
249 | will override the similar setting in the <envar>MAKEFLAGS</envar>
|
---|
250 | environment variable.</para>
|
---|
251 | <!-- outdated
|
---|
252 | <note><para>When running the package tests or the install portion of the
|
---|
253 | package build process, we do not recommend using an option greater than
|
---|
254 | '-j1' unless specified otherwise. The installation procedures or checks
|
---|
255 | have not been validated using parallel procedures and may fail with issues
|
---|
256 | that are difficult to debug.</para></note>
|
---|
257 | -->
|
---|
258 | <important>
|
---|
259 | <para>
|
---|
260 | Another problem may occur with modern CPU's, which have a lot of cores.
|
---|
261 | Each job started consumes memory, and if the sum of the needed
|
---|
262 | memory for each job exceeds the available memory, you may encounter
|
---|
263 | either an OOM (Out of Memory) kernel interrupt or intense swapping
|
---|
264 | that will slow the build beyond reasonable limits.
|
---|
265 | </para>
|
---|
266 |
|
---|
267 | <para>
|
---|
268 | Some compilations with <command>g++</command> may consume up to 2.5 GB
|
---|
269 | of memory, so to be safe, you should restrict the number of jobs
|
---|
270 | to (Total Memory in GB)/2.5, at least for big packages such as LLVM,
|
---|
271 | WebKitGtk, QtWebEngine, or libreoffice.
|
---|
272 | </para>
|
---|
273 | </important>
|
---|
274 | </sect2>
|
---|
275 |
|
---|
276 | <sect2 id="automating-builds" xreflabel="Automated Building Procedures">
|
---|
277 | <title>Automated Building Procedures</title>
|
---|
278 |
|
---|
279 | <para>There are times when automating the building of a package can come in
|
---|
280 | handy. Everyone has their own reasons for wanting to automate building,
|
---|
281 | and everyone goes about it in their own way. Creating
|
---|
282 | <filename>Makefile</filename>s, <application>Bash</application> scripts,
|
---|
283 | <application>Perl</application> scripts or simply a list of commands used
|
---|
284 | to cut and paste are just some of the methods you can use to automate
|
---|
285 | building BLFS packages. Detailing how and providing examples of the many
|
---|
286 | ways you can automate the building of packages is beyond the scope of this
|
---|
287 | section. This section will expose you to using file redirection and the
|
---|
288 | <command>yes</command> command to help provide ideas on how to automate
|
---|
289 | your builds.</para>
|
---|
290 |
|
---|
291 | <bridgehead renderas="sect3">File Redirection to Automate Input</bridgehead>
|
---|
292 |
|
---|
293 | <para>You will find times throughout your BLFS journey when you will come
|
---|
294 | across a package that has a command prompting you for information. This
|
---|
295 | information might be configuration details, a directory path, or a response
|
---|
296 | to a license agreement. This can present a challenge to automate the
|
---|
297 | building of that package. Occasionally, you will be prompted for different
|
---|
298 | information in a series of questions. One method to automate this type of
|
---|
299 | scenario requires putting the desired responses in a file and using
|
---|
300 | redirection so that the program uses the data in the file as the answers to
|
---|
301 | the questions.</para>
|
---|
302 | <!-- outdated
|
---|
303 | <para>Building the <application>CUPS</application> package is a good
|
---|
304 | example of how redirecting a file as input to prompts can help you automate
|
---|
305 | the build. If you run the test suite, you are asked to respond to a series
|
---|
306 | of questions regarding the type of test to run and if you have any
|
---|
307 | auxiliary programs the test can use. You can create a file with your
|
---|
308 | responses, one response per line, and use a command similar to the
|
---|
309 | one shown below to automate running the test suite:</para>
|
---|
310 |
|
---|
311 | <screen><userinput>make check < ../cups-1.1.23-testsuite_parms</userinput></screen>
|
---|
312 | -->
|
---|
313 | <para>This effectively makes the test suite use the responses in the file
|
---|
314 | as the input to the questions. Occasionally you may end up doing a bit of
|
---|
315 | trial and error determining the exact format of your input file for some
|
---|
316 | things, but once figured out and documented you can use this to automate
|
---|
317 | building the package.</para>
|
---|
318 |
|
---|
319 | <bridgehead renderas="sect3">Using <command>yes</command> to Automate
|
---|
320 | Input</bridgehead>
|
---|
321 |
|
---|
322 | <para>Sometimes you will only need to provide one response, or provide the
|
---|
323 | same response to many prompts. For these instances, the
|
---|
324 | <command>yes</command> command works really well. The
|
---|
325 | <command>yes</command> command can be used to provide a response (the same
|
---|
326 | one) to one or more instances of questions. It can be used to simulate
|
---|
327 | pressing just the <keycap>Enter</keycap> key, entering the
|
---|
328 | <keycap>Y</keycap> key or entering a string of text. Perhaps the easiest
|
---|
329 | way to show its use is in an example.</para>
|
---|
330 |
|
---|
331 | <para>First, create a short <application>Bash</application> script by
|
---|
332 | entering the following commands:</para>
|
---|
333 |
|
---|
334 | <screen><userinput>cat > blfs-yes-test1 << "EOF"
|
---|
335 | <literal>#!/bin/bash
|
---|
336 |
|
---|
337 | echo -n -e "\n\nPlease type something (or nothing) and press Enter ---> "
|
---|
338 |
|
---|
339 | read A_STRING
|
---|
340 |
|
---|
341 | if test "$A_STRING" = ""; then A_STRING="Just the Enter key was pressed"
|
---|
342 | else A_STRING="You entered '$A_STRING'"
|
---|
343 | fi
|
---|
344 |
|
---|
345 | echo -e "\n\n$A_STRING\n\n"</literal>
|
---|
346 | EOF
|
---|
347 | chmod 755 blfs-yes-test1</userinput></screen>
|
---|
348 |
|
---|
349 | <para>Now run the script by issuing <command>./blfs-yes-test1</command> from
|
---|
350 | the command line. It will wait for a response, which can be anything (or
|
---|
351 | nothing) followed by the <keycap>Enter</keycap> key. After entering
|
---|
352 | something, the result will be echoed to the screen. Now use the
|
---|
353 | <command>yes</command> command to automate the entering of a
|
---|
354 | response:</para>
|
---|
355 |
|
---|
356 | <screen><userinput>yes | ./blfs-yes-test1</userinput></screen>
|
---|
357 |
|
---|
358 | <para>Notice that piping <command>yes</command> by itself to the script
|
---|
359 | results in <keycap>y</keycap> being passed to the script. Now try it with a
|
---|
360 | string of text:</para>
|
---|
361 |
|
---|
362 | <screen><userinput>yes 'This is some text' | ./blfs-yes-test1</userinput></screen>
|
---|
363 |
|
---|
364 | <para>The exact string was used as the response to the script. Finally,
|
---|
365 | try it using an empty (null) string:</para>
|
---|
366 |
|
---|
367 | <screen><userinput>yes '' | ./blfs-yes-test1</userinput></screen>
|
---|
368 |
|
---|
369 | <para>Notice this results in passing just the press of the
|
---|
370 | <keycap>Enter</keycap> key to the script. This is useful for times when the
|
---|
371 | default answer to the prompt is sufficient. This syntax is used in the
|
---|
372 | <xref linkend="net-tools-automate-example"/> instructions to accept all the
|
---|
373 | defaults to the many prompts during the configuration step. You may now
|
---|
374 | remove the test script, if desired.</para>
|
---|
375 |
|
---|
376 | <bridgehead renderas="sect3">File Redirection to Automate Output</bridgehead>
|
---|
377 |
|
---|
378 | <para>In order to automate the building of some packages, especially those
|
---|
379 | that require you to read a license agreement one page at a time, requires
|
---|
380 | using a method that avoids having to press a key to display each page.
|
---|
381 | Redirecting the output to a file can be used in these instances to assist
|
---|
382 | with the automation. The previous section on this page touched on creating
|
---|
383 | log files of the build output. The redirection method shown there used the
|
---|
384 | <command>tee</command> command to redirect output to a file while also
|
---|
385 | displaying the output to the screen. Here, the output will only be sent to
|
---|
386 | a file.</para>
|
---|
387 |
|
---|
388 | <para>Again, the easiest way to demonstrate the technique is to show an
|
---|
389 | example. First, issue the command:</para>
|
---|
390 |
|
---|
391 | <screen><userinput>ls -l /usr/bin | less</userinput></screen>
|
---|
392 |
|
---|
393 | <para>Of course, you'll be required to view the output one page at a time
|
---|
394 | because the <command>less</command> filter was used. Now try the same
|
---|
395 | command, but this time redirect the output to a file. The special file
|
---|
396 | <filename>/dev/null</filename> can be used instead of the filename shown,
|
---|
397 | but you will have no log file to examine:</para>
|
---|
398 |
|
---|
399 | <screen><userinput>ls -l /usr/bin | less > redirect_test.log 2>&1</userinput></screen>
|
---|
400 |
|
---|
401 | <para>Notice that this time the command immediately returned to the shell
|
---|
402 | prompt without having to page through the output. You may now remove the
|
---|
403 | log file.</para>
|
---|
404 |
|
---|
405 | <para>The last example will use the <command>yes</command> command in
|
---|
406 | combination with output redirection to bypass having to page through the
|
---|
407 | output and then provide a <keycap>y</keycap> to a prompt. This technique
|
---|
408 | could be used in instances when otherwise you would have to page through
|
---|
409 | the output of a file (such as a license agreement) and then answer the
|
---|
410 | question of <quote>do you accept the above?</quote>. For this example,
|
---|
411 | another short <application>Bash</application> script is required:</para>
|
---|
412 |
|
---|
413 | <screen><userinput>cat > blfs-yes-test2 << "EOF"
|
---|
414 | <literal>#!/bin/bash
|
---|
415 |
|
---|
416 | ls -l /usr/bin | less
|
---|
417 |
|
---|
418 | echo -n -e "\n\nDid you enjoy reading this? (y,n) "
|
---|
419 |
|
---|
420 | read A_STRING
|
---|
421 |
|
---|
422 | if test "$A_STRING" = "y"; then A_STRING="You entered the 'y' key"
|
---|
423 | else A_STRING="You did NOT enter the 'y' key"
|
---|
424 | fi
|
---|
425 |
|
---|
426 | echo -e "\n\n$A_STRING\n\n"</literal>
|
---|
427 | EOF
|
---|
428 | chmod 755 blfs-yes-test2</userinput></screen>
|
---|
429 |
|
---|
430 | <para>This script can be used to simulate a program that requires you to
|
---|
431 | read a license agreement, then respond appropriately to accept the
|
---|
432 | agreement before the program will install anything. First, run the script
|
---|
433 | without any automation techniques by issuing
|
---|
434 | <command>./blfs-yes-test2</command>.</para>
|
---|
435 |
|
---|
436 | <para>Now issue the following command which uses two automation techniques,
|
---|
437 | making it suitable for use in an automated build script:</para>
|
---|
438 |
|
---|
439 | <screen><userinput>yes | ./blfs-yes-test2 > blfs-yes-test2.log 2>&1</userinput></screen>
|
---|
440 |
|
---|
441 | <para>If desired, issue <command>tail blfs-yes-test2.log</command> to see
|
---|
442 | the end of the paged output, and confirmation that <keycap>y</keycap> was
|
---|
443 | passed through to the script. Once satisfied that it works as it should,
|
---|
444 | you may remove the script and log file.</para>
|
---|
445 |
|
---|
446 | <para>Finally, keep in mind that there are many ways to automate and/or
|
---|
447 | script the build commands. There is not a single <quote>correct</quote> way
|
---|
448 | to do it. Your imagination is the only limit.</para>
|
---|
449 |
|
---|
450 | </sect2>
|
---|
451 |
|
---|
452 | <sect2>
|
---|
453 | <title>Dependencies</title>
|
---|
454 |
|
---|
455 | <para>For each package described, BLFS lists the known dependencies.
|
---|
456 | These are listed under several headings, whose meaning is as follows:</para>
|
---|
457 |
|
---|
458 | <itemizedlist>
|
---|
459 | <listitem>
|
---|
460 | <para><emphasis>Required</emphasis> means that the target package
|
---|
461 | cannot be correctly built without the dependency having first been
|
---|
462 | installed, except if the dependency is said to be
|
---|
463 | <quote>runtime</quote>, which means the target package can be built but
|
---|
464 | cannot function without it.</para>
|
---|
465 | <para>
|
---|
466 | Note that a target package can start to <quote>function</quote>
|
---|
467 | in many subtle ways: an installed configuration file can make the
|
---|
468 | init system, cron daemon, or bus daemon to run a program
|
---|
469 | automatically; another package using the target package as an
|
---|
470 | dependency can run a program from the target package in the
|
---|
471 | building system; and the configuration sections in the BLFS book
|
---|
472 | may also run a program from a just installed package. So if
|
---|
473 | you are installing the target package without a
|
---|
474 | <emphasis>Required (runtime)</emphasis> dependency installed,
|
---|
475 | You should install the dependency as soon as possible after the
|
---|
476 | installation of the target package.
|
---|
477 | </para>
|
---|
478 | </listitem>
|
---|
479 | <listitem>
|
---|
480 | <para><emphasis>Recommended</emphasis> means that BLFS strongly
|
---|
481 | suggests this package is installed first (except if said to be
|
---|
482 | <quote>runtime</quote>, see below) for a clean and trouble-free
|
---|
483 | build, that won't have issues either during the build process, or at
|
---|
484 | run-time. The instructions in the book assume these packages are
|
---|
485 | installed. Some changes or workarounds may be required if these
|
---|
486 | packages are not installed. If a recommended dependency is said
|
---|
487 | to be <quote>runtime</quote>, it means that BLFS strongly suggests
|
---|
488 | that this dependency is installed before using the package, for
|
---|
489 | getting full functionality.</para>
|
---|
490 | </listitem>
|
---|
491 | <listitem>
|
---|
492 | <para><emphasis>Optional</emphasis> means that this package might be
|
---|
493 | installed for added functionality. Often BLFS will describe the
|
---|
494 | dependency to explain the added functionality that will result.
|
---|
495 | An optional dependency may be automatically pick up by the target
|
---|
496 | package if the dependency is installed, but another some optional
|
---|
497 | dependency may also need additional configuration options to enable
|
---|
498 | them when the target package is built. Such additional options are
|
---|
499 | often documented in the BLFS book. If an optional dependency is
|
---|
500 | said to be <quote>runtime</quote>, it means you may install
|
---|
501 | the dependency after installing the target package to support some
|
---|
502 | optional features of the target package if you need these
|
---|
503 | features.</para>
|
---|
504 | <para>An optional dependency may be out of BLFS. If you need such
|
---|
505 | an <emphasis>external</emphasis> optional dependency for some
|
---|
506 | features you need, read <xref linkend='beyond'/> for the general
|
---|
507 | hint about installing an out-of-BLFS package.</para>
|
---|
508 | </listitem>
|
---|
509 | </itemizedlist>
|
---|
510 |
|
---|
511 | </sect2>
|
---|
512 |
|
---|
513 | <sect2 id="package_updates">
|
---|
514 | <title>Using the Most Current Package Sources</title>
|
---|
515 |
|
---|
516 | <para>On occasion you may run into a situation in the book when a package
|
---|
517 | will not build or work properly. Though the Editors attempt to ensure
|
---|
518 | that every package in the book builds and works properly, sometimes a
|
---|
519 | package has been overlooked or was not tested with this particular version
|
---|
520 | of BLFS.</para>
|
---|
521 |
|
---|
522 | <para>If you discover that a package will not build or work properly, you
|
---|
523 | should see if there is a more current version of the package. Typically
|
---|
524 | this means you go to the maintainer's web site and download the most current
|
---|
525 | tarball and attempt to build the package. If you cannot determine the
|
---|
526 | maintainer's web site by looking at the download URLs, use Google and query
|
---|
527 | the package's name. For example, in the Google search bar type:
|
---|
528 | 'package_name download' (omit the quotes) or something similar. Sometimes
|
---|
529 | typing: 'package_name home page' will result in you finding the
|
---|
530 | maintainer's web site.</para>
|
---|
531 |
|
---|
532 | </sect2>
|
---|
533 |
|
---|
534 | <sect2 id="stripping">
|
---|
535 | <title>Stripping One More Time</title>
|
---|
536 |
|
---|
537 | <para>
|
---|
538 | In LFS, stripping of debugging symbols and unneeded symbol table
|
---|
539 | entries was discussed a couple of times. When building BLFS packages,
|
---|
540 | there are generally no special instructions that discuss stripping
|
---|
541 | again. Stripping can be done while installing a package, or
|
---|
542 | afterwards.
|
---|
543 | </para>
|
---|
544 |
|
---|
545 | <bridgehead renderas="sect3" id="stripping-install">Stripping while Installing a Package</bridgehead>
|
---|
546 |
|
---|
547 | <para>
|
---|
548 | There are several ways to strip executables installed by a
|
---|
549 | package. They depend on the build system used (see below <link
|
---|
550 | linkend="buildsystems">the section about build systems</link>),
|
---|
551 | so only some
|
---|
552 | generalities can be listed here:
|
---|
553 | </para>
|
---|
554 |
|
---|
555 | <note>
|
---|
556 | <para>
|
---|
557 | The following methods using the feature of a building system
|
---|
558 | (autotools, meson, or cmake) will not strip static libraries if any
|
---|
559 | is installed. Fortunately there are not too many static libraries
|
---|
560 | in BLFS, and a static library can always be stripped safely by
|
---|
561 | running <command>strip --strip-unneeded</command> on it manually.
|
---|
562 | </para>
|
---|
563 | </note>
|
---|
564 |
|
---|
565 | <itemizedlist>
|
---|
566 | <listitem>
|
---|
567 | <para>
|
---|
568 | The packages using autotools usually have an
|
---|
569 | <parameter>install-strip</parameter> target in their generated
|
---|
570 | <filename>Makefile</filename> files. So installing stripped
|
---|
571 | executables is just a matter of using
|
---|
572 | <command>make install-strip</command> instead of
|
---|
573 | <command>make install</command>.
|
---|
574 | </para>
|
---|
575 | </listitem>
|
---|
576 | <listitem>
|
---|
577 | <para>
|
---|
578 | The packages using the meson build system can accept
|
---|
579 | <parameter>-Dstrip=true</parameter> when running
|
---|
580 | <command>meson</command>. If you've forgot to add this option
|
---|
581 | running the <command>meson</command>, you can also run
|
---|
582 | <command>meson install --strip</command> instead of
|
---|
583 | <command>ninja install</command>.
|
---|
584 | </para>
|
---|
585 | </listitem>
|
---|
586 | <listitem>
|
---|
587 | <para>
|
---|
588 | <command>cmake</command> generates
|
---|
589 | <parameter>install/strip</parameter> targets for both the
|
---|
590 | <parameter>Unix Makefiles</parameter> and
|
---|
591 | <parameter>Ninja</parameter> generators (the default is
|
---|
592 | <parameter>Unix Makefiles</parameter> on linux). So just run
|
---|
593 | <command>make install/strip</command> or
|
---|
594 | <command>ninja install/strip</command> instead of the
|
---|
595 | <command>install</command> counterparts.
|
---|
596 | </para>
|
---|
597 | </listitem>
|
---|
598 | <listitem>
|
---|
599 | <para>
|
---|
600 | Removing (or not generating) debug symbols can also be
|
---|
601 | achieved by removing the
|
---|
602 | <parameter>-g<something></parameter> options
|
---|
603 | in C/C++ calls. How to do that is very specific for each
|
---|
604 | package. And, it does not remove unneeded symbol table entries.
|
---|
605 | So it will not be explained in detail here. See also below
|
---|
606 | the paragraphs about optimization.
|
---|
607 | </para>
|
---|
608 | </listitem>
|
---|
609 | </itemizedlist>
|
---|
610 |
|
---|
611 | <bridgehead renderas="sect3" id="stripping-installed">Stripping Installed Executables</bridgehead>
|
---|
612 |
|
---|
613 | <para>
|
---|
614 | The <command>strip</command> utility changes files in place, which may
|
---|
615 | break anything using it if it is loaded in memory. Note that if a file is
|
---|
616 | in use but just removed from the disk (i.e. not overwritten nor
|
---|
617 | modified), this is not a problem since the kernel can use
|
---|
618 | <quote>deleted</quote> files. Look at <filename>/proc/*/maps</filename>
|
---|
619 | and it is likely that you'll see some <emphasis>(deleted)</emphasis>
|
---|
620 | entries. The <command>mv</command> just removes the destination file from
|
---|
621 | the directory but does not touch its content, so that it satisfies the
|
---|
622 | condition for the kernel to use the old (deleted) file.
|
---|
623 | But this approach can detach hard links into duplicated copies,
|
---|
624 | causing a bloat which is obviously unwanted as we are stripping to
|
---|
625 | reduce system size. If two files in a same file system share the
|
---|
626 | same inode number, they are hard links to each other and we should
|
---|
627 | reconstruct the link. The script below is just an example.
|
---|
628 | It should be run as the &root; user:
|
---|
629 | </para>
|
---|
630 |
|
---|
631 | <screen><userinput>cat > /usr/sbin/strip-all.sh << "EOF"
|
---|
632 | <literal>#!/usr/bin/bash
|
---|
633 |
|
---|
634 | if [ $EUID -ne 0 ]; then
|
---|
635 | echo "Need to be root"
|
---|
636 | exit 1
|
---|
637 | fi
|
---|
638 |
|
---|
639 | last_fs_inode=
|
---|
640 | last_file=
|
---|
641 |
|
---|
642 | { find /usr/lib -type f -name '*.so*' ! -name '*dbg'
|
---|
643 | find /usr/lib -type f -name '*.a'
|
---|
644 | find /usr/{bin,sbin,libexec} -type f
|
---|
645 | } | xargs stat -c '%m %i %n' | sort | while read fs inode file; do
|
---|
646 | if ! readelf -h $file >/dev/null 2>&1; then continue; fi
|
---|
647 | if file $file | grep --quiet --invert-match 'not stripped'; then continue; fi
|
---|
648 |
|
---|
649 | if [ "$fs $inode" = "$last_fs_inode" ]; then
|
---|
650 | ln -f $last_file $file;
|
---|
651 | continue;
|
---|
652 | fi
|
---|
653 |
|
---|
654 | cp --preserve $file ${file}.tmp
|
---|
655 | strip --strip-unneeded ${file}.tmp
|
---|
656 | mv ${file}.tmp $file
|
---|
657 |
|
---|
658 | last_fs_inode="$fs $inode"
|
---|
659 | last_file=$file
|
---|
660 | done</literal>
|
---|
661 | EOF
|
---|
662 | chmod 744 /usr/sbin/strip-all.sh</userinput></screen>
|
---|
663 |
|
---|
664 | <para>
|
---|
665 | If you install programs in other directories such as <filename
|
---|
666 | class="directory">/opt</filename> or <filename
|
---|
667 | class="directory">/usr/local</filename>, you may want to strip the files
|
---|
668 | there too. Just add other directories to scan in the compound list of
|
---|
669 | <command>find</command> commands between the braces.
|
---|
670 | </para>
|
---|
671 |
|
---|
672 | <para>
|
---|
673 | For more information on stripping, see <ulink
|
---|
674 | url="https://www.technovelty.org/linux/stripping-shared-libraries.html"/>.
|
---|
675 | </para>
|
---|
676 |
|
---|
677 | </sect2>
|
---|
678 |
|
---|
679 | <!--
|
---|
680 | <sect2 id="libtool">
|
---|
681 | <title>Libtool files</title>
|
---|
682 |
|
---|
683 | <para>
|
---|
684 | One of the side effects of packages that use Autotools, including
|
---|
685 | libtool, is that they create many files with an .la extension. These
|
---|
686 | files are not needed in an LFS environment. If there are conflicts with
|
---|
687 | pkgconfig entries, they can actually prevent successful builds. You
|
---|
688 | may want to consider removing these files periodically:
|
---|
689 | </para>
|
---|
690 |
|
---|
691 | <screen><userinput>find /lib /usr/lib -not -path "*Image*" -a -name \*.la -delete</userinput></screen>
|
---|
692 |
|
---|
693 | <para>
|
---|
694 | The above command removes all .la files with the exception of those that
|
---|
695 | have <quote>Image</quote> or <quote>openldap</quote> as a part of the
|
---|
696 | path. These .la files are used by the ImageMagick and openldap programs,
|
---|
697 | respectively. There may be other exceptions by packages not in BLFS.
|
---|
698 | </para>
|
---|
699 |
|
---|
700 | </sect2>
|
---|
701 | -->
|
---|
702 | <sect2 id="buildsystems">
|
---|
703 | <title>Working with different build systems</title>
|
---|
704 |
|
---|
705 | <para>
|
---|
706 | There are now three different build systems in common use for
|
---|
707 | converting C or C++ source code into compiled programs or
|
---|
708 | libraries and their details (particularly, finding out about available
|
---|
709 | options and their default values) differ. It may be easiest to understand
|
---|
710 | the issues caused by some choices (typically slow execution or
|
---|
711 | unexpected use of, or omission of, optimizations) by starting with
|
---|
712 | the <envar>CFLAGS</envar>, <envar>CXXFLAGS</envar>, and
|
---|
713 | <envar>LDFLAGS</envar> environment variables. There are also some
|
---|
714 | programs which use Rust.
|
---|
715 | </para>
|
---|
716 |
|
---|
717 | <para>
|
---|
718 | Most LFS and BLFS builders are probably aware of the basics of
|
---|
719 | <envar>CFLAGS</envar> and <envar>CXXFLAGS</envar> for altering how a
|
---|
720 | program is compiled. Typically, some form of optimization is used by
|
---|
721 | upstream developers (<option>-O2</option> or <option>-O3</option>),
|
---|
722 | sometimes with the creation of debug symbols (<option>-g</option>),
|
---|
723 | as defaults.
|
---|
724 | </para>
|
---|
725 |
|
---|
726 | <para>
|
---|
727 | If there are contradictory flags (e.g. multiple different
|
---|
728 | <option>-O</option> values),
|
---|
729 | the <emphasis>last</emphasis> value will be used. Sometimes this means
|
---|
730 | that flags specified in environment variables will be picked up before
|
---|
731 | values hardcoded in the Makefile, and therefore ignored. For example,
|
---|
732 | where a user specifies <option>-O2</option> and that is followed by
|
---|
733 | <option>-O3</option> the build will use <option>-O3</option>.
|
---|
734 | </para>
|
---|
735 |
|
---|
736 | <para>
|
---|
737 | There are various other things which can be passed in CFLAGS or
|
---|
738 | CXXFLAGS, such as allowing using the instruction set extensions
|
---|
739 | available with a specific microarchitecture (e.g.
|
---|
740 | <option>-march=amdfam10</option> or <option>-march=native</option>),
|
---|
741 | tune the generated code for a specific microarchitecture (e. g.
|
---|
742 | <option>-mtune=tigerlake</option> or <option>-mtune=native</option>,
|
---|
743 | if <option>-mtune=</option> is not used, the microarchitecture from
|
---|
744 | <option>-march=</option> setting will be used), or specifying a
|
---|
745 | specific standard for C or C++ (<option>-std=c++17</option> for
|
---|
746 | example). But one thing which has now come to light is that
|
---|
747 | programmers might include debug assertions in their code, expecting
|
---|
748 | them to be disabled in releases by using <option>-DNDEBUG</option>.
|
---|
749 | Specifically, if <xref linkend="mesa"/> is built with these
|
---|
750 | assertions enabled, some activities such as loading levels of games
|
---|
751 | can take extremely long times, even on high-class video cards.
|
---|
752 | </para>
|
---|
753 |
|
---|
754 | <bridgehead renderas="sect3" id="autotools-info">Autotools with Make</bridgehead>
|
---|
755 |
|
---|
756 | <para>
|
---|
757 | This combination is often described as <quote>CMMI</quote>
|
---|
758 | (configure, make, make install) and is used here to also cover
|
---|
759 | the few packages which have a configure script that is not
|
---|
760 | generated by autotools.
|
---|
761 | </para>
|
---|
762 |
|
---|
763 | <para>
|
---|
764 | Sometimes running <command>./configure --help</command> will produce
|
---|
765 | useful options about switches which might be used. At other times,
|
---|
766 | after looking at the output from configure you may need to look
|
---|
767 | at the details of the script to find out what it was actually searching
|
---|
768 | for.
|
---|
769 | </para>
|
---|
770 |
|
---|
771 | <para>
|
---|
772 | Many configure scripts will pick up any CFLAGS or CXXFLAGS from the
|
---|
773 | environment, but CMMI packages vary about how these will be mixed with
|
---|
774 | any flags which would otherwise be used (<emphasis>variously</emphasis>:
|
---|
775 | ignored, used to replace the programmer's suggestion, used before the
|
---|
776 | programmer's suggestion, or used after the programmer's suggestion).
|
---|
777 | </para>
|
---|
778 |
|
---|
779 | <para>
|
---|
780 | In most CMMI packages, running <command>make</command> will list
|
---|
781 | each command and run it, interspersed with any warnings. But some
|
---|
782 | packages try to be <quote>silent</quote> and only show which file
|
---|
783 | they are compiling or linking instead of showing the command line.
|
---|
784 | If you need to inspect the command, either because of an error, or
|
---|
785 | just to see what options and flags are being used, adding
|
---|
786 | <option>V=1</option> to the make invocation may help.
|
---|
787 | </para>
|
---|
788 |
|
---|
789 | <bridgehead renderas="sect3" id="cmake-info">CMake</bridgehead>
|
---|
790 |
|
---|
791 | <para>
|
---|
792 | CMake works in a very different way, and it has two backends which
|
---|
793 | can be used on BLFS: <command>make</command> and
|
---|
794 | <command>ninja</command>. The default backend is make, but
|
---|
795 | ninja can be faster on large packages with multiple processors. To
|
---|
796 | use ninja, specify <option>-G Ninja</option> in the cmake command.
|
---|
797 | However, there are some packages which create fatal errors in their
|
---|
798 | ninja files but build successfully using the default of Unix
|
---|
799 | Makefiles.
|
---|
800 | </para>
|
---|
801 |
|
---|
802 | <para>
|
---|
803 | The hardest part of using CMake is knowing what options you might wish
|
---|
804 | to specify. The only way to get a list of what the package knows about
|
---|
805 | is to run <command>cmake -LAH</command> and look at the output for that
|
---|
806 | default configuration.
|
---|
807 | </para>
|
---|
808 |
|
---|
809 | <para>
|
---|
810 | Perhaps the most-important thing about CMake is that it has a variety
|
---|
811 | of CMAKE_BUILD_TYPE values, and these affect the flags. The default
|
---|
812 | is that this is not set and no flags are generated. Any
|
---|
813 | <envar>CFLAGS</envar> or <envar>CXXFLAGS</envar> in the environment
|
---|
814 | will be used. If the programmer has coded any debug assertions,
|
---|
815 | those will be enabled unless -DNDEBUG is used. The following
|
---|
816 | CMAKE_BUILD_TYPE values will generate the flags shown, and these
|
---|
817 | will come <emphasis>after</emphasis> any flags in the environment
|
---|
818 | and therefore take precedence.
|
---|
819 | </para>
|
---|
820 |
|
---|
821 | <informaltable align="center">
|
---|
822 | <tgroup cols="2">
|
---|
823 | <colspec colnum="1" align="center"/>
|
---|
824 | <colspec colnum="2" align="center"/>
|
---|
825 | <thead>
|
---|
826 | <row><entry>Value</entry><entry>Flags</entry></row>
|
---|
827 | </thead>
|
---|
828 | <tbody>
|
---|
829 | <row>
|
---|
830 | <entry>Debug</entry><entry><option>-g</option></entry>
|
---|
831 | </row>
|
---|
832 | <row>
|
---|
833 | <entry>Release</entry><entry><option>-O3 -DNDEBUG</option></entry>
|
---|
834 | </row>
|
---|
835 | <row>
|
---|
836 | <entry>RelWithDebInfo</entry><entry><option>-O2 -g -DNDEBUG</option></entry>
|
---|
837 | </row>
|
---|
838 | <row>
|
---|
839 | <entry>MinSizeRel</entry><entry><option>-Os -DNDEBUG</option></entry>
|
---|
840 | </row>
|
---|
841 | </tbody>
|
---|
842 | </tgroup>
|
---|
843 | </informaltable>
|
---|
844 |
|
---|
845 | <para>
|
---|
846 | CMake tries to produce quiet builds. To see the details of the commands
|
---|
847 | which are being run, use <command>make VERBOSE=1</command> or
|
---|
848 | <command>ninja -v</command>.
|
---|
849 | </para>
|
---|
850 |
|
---|
851 | <para>
|
---|
852 | By default, CMake treats file installation differently from the other
|
---|
853 | build systems: if a file already exists and is not newer than a file
|
---|
854 | that would overwrite it, then the file is not installed. This may be
|
---|
855 | a problem if a user wants to record which file belongs to a package,
|
---|
856 | either using <envar>LD_PRELOAD</envar>, or by listing files newer
|
---|
857 | than a timestamp. The default can be changed by setting the variable
|
---|
858 | <envar>CMAKE_INSTALL_ALWAYS</envar> to 1 in the
|
---|
859 | <emphasis>environment</emphasis>, for example by
|
---|
860 | <command>export</command>'ing it.
|
---|
861 | </para>
|
---|
862 |
|
---|
863 | <bridgehead renderas="sect3" id="meson-info">Meson</bridgehead>
|
---|
864 |
|
---|
865 | <para>
|
---|
866 | Meson has some similarities to CMake, but many differences. To get
|
---|
867 | details of the defines that you may wish to change you can look at
|
---|
868 | <filename>meson_options.txt</filename> which is usually in the
|
---|
869 | top-level directory.
|
---|
870 | </para>
|
---|
871 |
|
---|
872 | <para>
|
---|
873 | If you have already configured the package by running
|
---|
874 | <command>meson</command> and now wish to change one or more settings,
|
---|
875 | you can either remove the build directory, recreate it, and use the
|
---|
876 | altered options, or within the build directory run <command>meson
|
---|
877 | configure</command>, e.g. to set an option:
|
---|
878 | </para>
|
---|
879 |
|
---|
880 | <screen><userinput>meson configure -D<some_option>=true</userinput></screen>
|
---|
881 |
|
---|
882 | <para>
|
---|
883 | If you do that, the file <filename>meson-private/cmd_line.txt</filename>
|
---|
884 | will show the <emphasis>last</emphasis> commands which were used.
|
---|
885 | </para>
|
---|
886 |
|
---|
887 | <para>
|
---|
888 | Meson provides the following buildtype values, and the flags they enable
|
---|
889 | come <emphasis>after</emphasis> any flags supplied in the environment and
|
---|
890 | therefore take precedence.
|
---|
891 | </para>
|
---|
892 |
|
---|
893 | <itemizedlist>
|
---|
894 | <listitem>
|
---|
895 | <para>plain : no added flags. This is for distributors to supply their
|
---|
896 | own <envar>CFLAGS</envar>, <envar>CXXFLAGS</envar> and
|
---|
897 | <envar>LDFLAGS</envar>. There is no obvious reason to use
|
---|
898 | this in BLFS.</para>
|
---|
899 | </listitem>
|
---|
900 | <listitem>
|
---|
901 | <para>debug : <option>-g</option> - this is the default if
|
---|
902 | nothing is specified in either <filename>meson.build</filename>
|
---|
903 | or the command line. However it results large and slow binaries,
|
---|
904 | so we should override it in BLFS.</para>
|
---|
905 | </listitem>
|
---|
906 | <listitem>
|
---|
907 | <para>debugoptimized : <option>-O2 -g</option> - this is the
|
---|
908 | default specified in <filename>meson.build</filename> of some
|
---|
909 | packages.</para>
|
---|
910 | </listitem>
|
---|
911 | <listitem>
|
---|
912 | <para>release : <option>-O3</option> (occasionally a package will
|
---|
913 | force <option>-O2</option> here) - this is the buildtype we use
|
---|
914 | for most packages with Meson build system in BLFS.</para>
|
---|
915 | </listitem>
|
---|
916 | </itemizedlist>
|
---|
917 |
|
---|
918 | <!-- From https://mesonbuild.com/Builtin-options.html#core-options:
|
---|
919 | b_ndebug: Default value = false, Possible values are
|
---|
920 | true, false, if-release. Some packages sets it to if-release
|
---|
921 | so we mistakenly believed if-release had been the default. -->
|
---|
922 | <para>
|
---|
923 | The <option>-DNDEBUG</option> flag is implied by the release
|
---|
924 | buildtype for some packages (for example <xref linkend='mesa'/>).
|
---|
925 | It can also be provided explicitly by passing
|
---|
926 | <option>-Db_ndebug=true</option>.
|
---|
927 | </para>
|
---|
928 |
|
---|
929 | <para>
|
---|
930 | To see the details of the commands which are being run in a package using
|
---|
931 | meson, use <command>ninja -v</command>.
|
---|
932 | </para>
|
---|
933 |
|
---|
934 | <bridgehead renderas="sect3" id="rust-info">Rustc and Cargo</bridgehead>
|
---|
935 |
|
---|
936 | <para>
|
---|
937 | Most released rustc programs are provided as crates (source tarballs)
|
---|
938 | which will query a server to check current versions of dependencies
|
---|
939 | and then download them as necessary. These packages are built using
|
---|
940 | <command>cargo --release</command>. In theory, you can manipulate the
|
---|
941 | RUSTFLAGS to change the optimize-level (default for
|
---|
942 | <option>--release</option> is 3, i. e.
|
---|
943 | <option>-Copt-level=3</option>, like <option>-O3</option>) or to
|
---|
944 | force it to build for the machine it is being compiled on, using
|
---|
945 | <option>-Ctarget-cpu=native</option> but in practice this seems to
|
---|
946 | make no significant difference.
|
---|
947 | </para>
|
---|
948 |
|
---|
949 | <para>
|
---|
950 | If you are compiling a standalone Rust program (as an unpackaged
|
---|
951 | <filename class='extension'>.rs</filename> file) by running
|
---|
952 | <command>rustc</command> directly, you should specify
|
---|
953 | <option>-O</option> (the abbreviation of
|
---|
954 | <option>-Copt-level=2</option>) or <option>-Copt-level=3</option>
|
---|
955 | otherwise it will do an unoptimized compile and run
|
---|
956 | <emphasis>much</emphasis> slower. If are compiling the program
|
---|
957 | for debugging it, replace the <option>-O</option> or
|
---|
958 | <option>-Copt-level=</option> options with <option>-g</option> to
|
---|
959 | produce an unoptimized program with debug info.
|
---|
960 | </para>
|
---|
961 |
|
---|
962 | <para>
|
---|
963 | Like <command>ninja</command>, by default <command>cargo</command>
|
---|
964 | uses all logical processors. This can often be worked around,
|
---|
965 | either by exporting
|
---|
966 | <envar>CARGO_BUILD_JOBS=<replaceable><N></replaceable></envar>
|
---|
967 | or passing
|
---|
968 | <option>--jobs <replaceable><N></replaceable></option> to
|
---|
969 | <command>cargo</command>.
|
---|
970 | For compiling rustc itself, specifying
|
---|
971 | <option>--jobs <replaceable><N></replaceable></option> for
|
---|
972 | invocations of <command>x.py</command>
|
---|
973 | (together with the <envar>CARGO_BUILD_JOBS</envar> environment
|
---|
974 | variable, which looks like a <quote>belt and braces</quote>
|
---|
975 | approach but seems to be necessary) mostly works. The exception is
|
---|
976 | running the tests when building rustc, some of them will
|
---|
977 | nevertheless use all online CPUs, at least as of rustc-1.42.0.
|
---|
978 | </para>
|
---|
979 |
|
---|
980 | </sect2>
|
---|
981 |
|
---|
982 | <sect2 id="optimizations">
|
---|
983 | <title>Optimizing the build</title>
|
---|
984 |
|
---|
985 | <para>
|
---|
986 | Many people will prefer to optimize compiles as they see fit, by providing
|
---|
987 | <envar>CFLAGS</envar> or <envar>CXXFLAGS</envar>. For an
|
---|
988 | introduction to the options available with gcc and g++ see <ulink
|
---|
989 | url="https://gcc.gnu.org/onlinedocs/gcc-&gcc-version;/gcc/Optimize-Options.html"/>.
|
---|
990 | The same content can be also found in <command>info gcc</command>.
|
---|
991 | </para>
|
---|
992 |
|
---|
993 | <para>
|
---|
994 | Some packages default to <option>-O2 -g</option>, others to
|
---|
995 | <option>-O3 -g</option>, and if <envar>CFLAGS</envar> or
|
---|
996 | <envar>CXXFLAGS</envar> are supplied they might be added to the
|
---|
997 | package's defaults, replace the package's defaults, or even be
|
---|
998 | ignored. There are details on some desktop packages which were
|
---|
999 | mostly current in April 2019 at
|
---|
1000 | <ulink url="https://www.linuxfromscratch.org/~ken/tuning/"/> - in
|
---|
1001 | particular, <filename>README.txt</filename>,
|
---|
1002 | <filename>tuning-1-packages-and-notes.txt</filename>, and
|
---|
1003 | <filename>tuning-notes-2B.txt</filename>. The particular thing to
|
---|
1004 | remember is that if you want to try some of the more interesting
|
---|
1005 | flags you may need to force verbose builds to confirm what is being
|
---|
1006 | used.
|
---|
1007 | </para>
|
---|
1008 |
|
---|
1009 | <para>
|
---|
1010 | Clearly, if you are optimizing your own program you can spend time to
|
---|
1011 | profile it and perhaps recode some of it if it is too slow. But for
|
---|
1012 | building a whole system that approach is impractical. In general,
|
---|
1013 | <option>-O3</option> usually produces faster programs than
|
---|
1014 | <option>-O2</option>. Specifying
|
---|
1015 | <option>-march=native</option> is also beneficial, but means that
|
---|
1016 | you cannot move the binaries to an incompatible machine - this can
|
---|
1017 | also apply to newer machines, not just to older machines. For
|
---|
1018 | example programs compiled for <literal>amdfam10</literal> run on
|
---|
1019 | old Phenoms, Kaveris, and Ryzens : but programs compiled for a
|
---|
1020 | Kaveri will not run on a Ryzen because certain op-codes are not
|
---|
1021 | present. Similarly, if you build for a Haswell not everything will
|
---|
1022 | run on a SandyBridge.
|
---|
1023 | </para>
|
---|
1024 |
|
---|
1025 | <note>
|
---|
1026 | <para>
|
---|
1027 | Be careful that the name of a <option>-march</option> setting
|
---|
1028 | does not always match the baseline of the microarchitecture
|
---|
1029 | with the same name. For example, the Skylake-based Intel Celeron
|
---|
1030 | processors do not support AVX at all, but
|
---|
1031 | <option>-march=skylake</option> assumes AVX and even AVX2.
|
---|
1032 | </para>
|
---|
1033 | </note>
|
---|
1034 |
|
---|
1035 | <para>
|
---|
1036 | When a shared library is built by GCC, a feature named
|
---|
1037 | <quote>semantic interposition</quote> is enabled by default. When
|
---|
1038 | the shared library refers to a symbol name with external linkage
|
---|
1039 | and default visibility, if the symbol exists in both the shared
|
---|
1040 | library and the main executable, semantic interposition guarantees
|
---|
1041 | the symbol in the main executable is always used. This feature
|
---|
1042 | was invented in an attempt to make the behavior of linking a shared
|
---|
1043 | library and linking a static library as similar as possible. Today
|
---|
1044 | only a small number of packages still depend on semantic
|
---|
1045 | interposition, but the feature is still on by the default of GCC,
|
---|
1046 | causing many optimizations disabled for shared libraries because
|
---|
1047 | they conflict with semantic interposition. The
|
---|
1048 | <option>-fno-semantic-interposition</option> option can be passed
|
---|
1049 | to <command>gcc</command> or <command>g++</command> to disable
|
---|
1050 | semantic interposition and enable more optimizations for shared
|
---|
1051 | libraries. This option is used as the default of some packages
|
---|
1052 | (for example <xref linkend='python3'/>), and it's also the default
|
---|
1053 | of Clang.
|
---|
1054 | </para>
|
---|
1055 |
|
---|
1056 | <para>
|
---|
1057 | There are also various other options which some people claim are
|
---|
1058 | beneficial. At worst, you get to recompile and test, and then
|
---|
1059 | discover that in your usage the options do not provide a benefit.
|
---|
1060 | </para>
|
---|
1061 |
|
---|
1062 | <para>
|
---|
1063 | If building Perl or Python modules,
|
---|
1064 | in general the <envar>CFLAGS</envar> and <envar>CXXFLAGS</envar>
|
---|
1065 | used are those which were used by those <quote>parent</quote>
|
---|
1066 | packages.
|
---|
1067 | </para>
|
---|
1068 |
|
---|
1069 | <para>
|
---|
1070 | For <envar>LDFLAGS</envar>, there are three options can be used
|
---|
1071 | for optimization. They are quite safe to use and the building
|
---|
1072 | system of some packages use some of these options as the default.
|
---|
1073 | </para>
|
---|
1074 |
|
---|
1075 | <para>
|
---|
1076 | With <option>-Wl,-O1</option>, the linker will
|
---|
1077 | optimize the hash table to speed up the dynamic linking.
|
---|
1078 | Note that <option>-Wl,-O1</option> is completely unrelated to the
|
---|
1079 | compiler optimization flag <option>-O1</option>.
|
---|
1080 | </para>
|
---|
1081 |
|
---|
1082 | <para>
|
---|
1083 | With <option>-Wl,--as-needed</option>, the linker will disregard
|
---|
1084 | unnecessary <option>-l<replaceable>foo</replaceable></option> options
|
---|
1085 | from the command line, i. e. the shared library <systemitem
|
---|
1086 | class='library'>lib<replaceable>foo</replaceable></systemitem>
|
---|
1087 | will only be linked if a symbol in <systemitem
|
---|
1088 | class='library'>lib<replaceable>foo</replaceable></systemitem> is
|
---|
1089 | really referred from the executable or shared library being linked.
|
---|
1090 | This can sometimes mitigate the <quote>excessive dependencies to
|
---|
1091 | shared libraries</quote> issues caused by
|
---|
1092 | <application>libtool</application>.
|
---|
1093 | </para>
|
---|
1094 |
|
---|
1095 | <para>
|
---|
1096 | With <option>-Wl,-z,pack-relative-relocs</option>, the linker
|
---|
1097 | generates a more compacted form of the relative relocation entries
|
---|
1098 | for PIEs and shared libraries. It reduces the size of the linked
|
---|
1099 | PIE or shared library, and speeds up the loading of the PIE or
|
---|
1100 | shared library.
|
---|
1101 | </para>
|
---|
1102 |
|
---|
1103 | <para>
|
---|
1104 | The <option>-Wl,</option> prefix is necessary because despite the
|
---|
1105 | variable is named <envar>LDFLAGS</envar>, its content is actually
|
---|
1106 | passed to <command>gcc</command> (or <command>g++</command>,
|
---|
1107 | <command>clang</command>, etc.) during the link stage, not directly
|
---|
1108 | passed to <command>ld</command>.
|
---|
1109 | </para>
|
---|
1110 |
|
---|
1111 | </sect2>
|
---|
1112 |
|
---|
1113 | <sect2 id="hardening">
|
---|
1114 | <title>Options for hardening the build</title>
|
---|
1115 |
|
---|
1116 | <para>
|
---|
1117 | Even on desktop systems, there are still a lot of exploitable
|
---|
1118 | vulnerabilities. For many of these, the attack comes via javascript
|
---|
1119 | in a browser. Often, a series of vulnerabilities are used to gain
|
---|
1120 | access to data (or sometimes to pwn, i.e. own, the machine and
|
---|
1121 | install rootkits). Most commercial distros will apply various
|
---|
1122 | hardening measures.
|
---|
1123 | </para>
|
---|
1124 |
|
---|
1125 | <para>
|
---|
1126 | In the past, there was Hardened LFS where gcc (a much older version)
|
---|
1127 | was forced to use hardening (with options to turn some of it off on a
|
---|
1128 | per-package basis). The current LFS and BLFS books are carrying
|
---|
1129 | forward a part of its spirit by enabling PIE
|
---|
1130 | (<option>-fPIE -pie</option>) and SSP
|
---|
1131 | (<option>-fstack-protector-strong</option>) as the defaults
|
---|
1132 | for GCC and clang. What is being covered here is different - first
|
---|
1133 | you have to make sure that the package is indeed using your added
|
---|
1134 | flags and not over-riding them.
|
---|
1135 | </para>
|
---|
1136 |
|
---|
1137 | <para>
|
---|
1138 | For hardening options which are reasonably cheap, there is some
|
---|
1139 | discussion in the 'tuning' link above (occasionally, one or more
|
---|
1140 | of these options might be inappropriate for a package). These
|
---|
1141 | options are <option>-D_FORTIFY_SOURCE=2</option> and
|
---|
1142 | (for C++) <option>-D_GLIBCXX_ASSERTIONS</option>. On modern
|
---|
1143 | machines these should only have a little impact on how fast things
|
---|
1144 | run, and often they will not be noticeable.
|
---|
1145 | </para>
|
---|
1146 |
|
---|
1147 | <para>
|
---|
1148 | The main distros use much more, such as RELRO (Relocation Read Only)
|
---|
1149 | and perhaps <option>-fstack-clash-protection</option>. You may also
|
---|
1150 | encounter the so-called <quote>userspace retpoline</quote>
|
---|
1151 | (<option>-mindirect-branch=thunk</option> etc.) which
|
---|
1152 | is the equivalent of the spectre mitigations applied to the linux
|
---|
1153 | kernel in late 2018. The kernel mitigations caused a lot of complaints
|
---|
1154 | about lost performance, if you have a production server you might wish
|
---|
1155 | to consider testing that, along with the other available options, to
|
---|
1156 | see if performance is still sufficient.
|
---|
1157 | </para>
|
---|
1158 |
|
---|
1159 | <para>
|
---|
1160 | Whilst gcc has many hardening options, clang/LLVM's strengths lie
|
---|
1161 | elsewhere. Some options which gcc provides are said to be less effective
|
---|
1162 | in clang/LLVM.
|
---|
1163 | </para>
|
---|
1164 |
|
---|
1165 | </sect2>
|
---|
1166 |
|
---|
1167 | </sect1>
|
---|