1. INTRODUCTION:: To automate package builds from the BLFS book instructions is a huge task. Some of the issues are: the BLFS book isn't linear; some package pages use a custom layout; there are circular dependencies; several packages can be installed on a non-default prefix; build commands can change based on what dependencies will be used, etc. That being said, the goal of the blfs-tool is to help you solve package dependencies, create build scripts and a Makefile. Not all the auto-generated build scripts and Makefile will work "as is", thus, as a general rule, you will need to review and edit the scripts while reading the book. The blfs tools allow also to update packages from the LFS book. LFS packages which may be updated appear in the menu interface. When selected, their scriptlet is generated in the same manner as for BLFS packages. (TODO: presently, when an LFS package needs a patch, you'll have to donwload it manually to your $SRC_ARCHIVE directory (usually /sources)). 2. PREREQUISITES:: In addition to a full LFS system, the following packages and their dependencies are needed by this tool: - required: libxml2, libxslt, DocBook XML DTD - recommended: wget (to download the package tarballs) and sudo (to build as a user) - optional: lynx (allows to read the generated linearized book), GPM (to cut and paste commands from the book), git (to update the book sources) Note that the optional dependencies are recommended for ease of use of the tool. You should also have the following personal skills: - Ability to write and debug shell scripts: as said in the introduction, not all the generated scripts can be used directly. They need to be edited to produce an error free build. - Ability to debug build failures, like missing dependencies or installation directories not known to the system (when you install in /opt for example). - Ability to choose the tools you need to configure and administrate your system: in the BLFS book, nothing is mandatory, nothing is useless. You are on your own in choosing what to build, but wrong decisions may lead to a non functional system... 3. INSTALL:: There are two ways to install the BLFS tools on an LFS system, described in paragraphs 3.1 and 3.2, respectively: 3.1 INSTALLATION ON A RUNNING SYSTEM Select "Use Book --> Beyond Linux From Scratch" in the jhalfs menu: The tools are installed in $HOME$BLFS_ROOT (the default for $BLFS_ROOT is /blfs_root). The BLFS book is downloaded or copied to its directory. The tracking directory (see below) is created (if it does not already exist) and initialized. Before running "make", you should ensure the tracking directory (default location /var/lib/jhalfs/BLFS) can be: - either created by the user running "make", if it does not exist - or that it is writable by the user running "make", if it exists. After the intallation, you should perform the following additional steps: - Configure sudo, adding the needed privileges for the user. - Although it is not strictly necessary, it is recommended to install the bash shell startup files (as per `3. After LFS Configuration Issues' of the BLFS book), as some instructions in BLFS rely on their being present. - At this point, the tool has no way to know which versions of LFS packages are installed, so that the menu interface will show all the LFS packages, as if they were not installed. If you have a released version of LFS, or the date of your GIT version of LFS is known, you should run the update-lfs.sh script. If you have updated some LFS packages since first installation, or have been using a custom working copy of the LFS book, the only (tedious) way is to create empty files with names - in the tracking directory, and run the tool. - If you have also installed some BLFS packages, they are not in the tracking file. The only way is to create empty files with names - in the tracking directory, and run the tool. 3.2 INSTALLATION ON A JUST BUILT LFS SYSTEM For books that support it (only LFS), there is an option to install the BLFS tools right after building the LFS system: just tick `BOOK Settings/Add blfs-tool support' in jhalfs configuration menu. The tools are installed in $BLFS_ROOT (default /blfs_root) on the LFS system, and the dependencies are built at the end of the jhalfs run, before the custom tools. After booting the new LFS system some steps are needed to finish the installation of the automated tools: - A user account must be created. You must be logged on that user account to use blfs-tool. This is not strictly necessary, since the packages can be built as root, too, but it is never a good idea to build packages as root. - Move /blfs-root to that user's home and change ownership of the directory and files to the user. - Give the user read and write privileges over the $TRACKING_DIR directory and the files that it contains. - Configure sudo and add the bash shell startup files, as described above - Note that the versions of LFS packages are automatically known to the tool in this case, and there is no need to run the update-lfs.sh script. We assume that the BLFS tools will be used on a booted LFS system. Using them to build BLFS packages in a chroot jail is also possible, but not supported. 3.3 DIRECTORY LAYOUT IN THE $BLFS_ROOT DIRECTORY blfs-xml/* GIT tree of the selected BLFS book version lfs-xml/* GIT tree of the selected LFS book version lib/constants.inc functions libraries /func_dependencies for building the dependency tree menu/* menuconfig source code xsl/gen_pkg_list.xsl XSL stylesheet to generate the package database /gen_config.xsl XSL stylesheet to generate the Config.in file for use in the menuconfig system /dependencies.xsl XSL stylesheet to generate the dependency list of a package /make_book.xsl XSL stylesheet to generate the linear book.xml /lfs_make_book.xsl XSL stylesheet to incoporate LFS pages into the linear book.xml /scripts.xsl XSL stylesheet to generate the scriptlets from book.xml /bump.xsl XSL stylesheet to update the tracking file /process-install.xsl XSL stylesheet included by scripts.xsl, for outputting cleanly install instructions /process-replaceable.xsl XSL stylesheet included by scripts.xsl, for generating correct instructions when a tag is encountered. README.BLFS this file TODO developers notes (well, not updated often) gen_pkg_book.sh resolves dependencies, generates a linear BLFS book, and finally generates build scripts gen-makefile.sh generates the target Makefile progress_bar.sh the target Makefile progress bar gen-special.sh Helper script for generating the package database Makefile Used by make to update the package database from the GIT tree, then launch the menuconfig interface, and run gen_pkg_book.sh based on configuration settings packdesc.dtd a simple DTD describing the format of the package database and the tracking file. Working files: several files are generated when first running the tool: packages.xml auto-generated package database Config.in input file for the menu driven choices configuration file generated by the menuconfig process dependencies/* files recording the dependency tree book.xml the linearized book book-html/* the linearized book rendered in html scripts/* the scriptlets 3.4 INSTALLED PACKAGES TRACKING SYSTEM: This tool includes a very simple tracking system to log which packages have been installed using the tool. It is used to skip installed packages from target selection menu and to test if an installed package has been updated in the BLFS book. Do not rely on this feature as a package management tool. The tracking system itself is an XML file: instpkg.xml. It is initialized when is first run in blfs_root. It resides in a directory, which is created when needed during the process of building custom tools or blfs dependencies, right after LFS. You can specify that directory location in the blfs-tools sub-menu of jhalfs. You may need to update permissions and/or ownership of this directory before using the blfs tool (see README in jhalfs). The default location of the tracking directory is /var/lib/jhalfs/BLFS. NB : after the initial build, that directory is only used to contain instpkg.xml, unless custom tools have been built. In the latter case, it also contains empty files whose name are "$PKG-$VERSION" for each versionned package built. The information about those packages is included into instpkg.xml the next time the tool is run. 4. USAGE:: From now on, all the work must be done from inside the installation root directory. Due to the complexity of the BLFS book, the scripts and Makefile generation is done in several steps: 4.1 UPDATING BOOK SOURCES:: If you are using the development book version and you want to update installed packages to the latest version found in that book, you need to update the XML sources and packages database. To do that, run "make update". This is not necessary if you just built LFS, and you can skip to step 4.2. On the next configuration run, packages already installed but listed with a new version in the book will be available for target selection and used to solve dependencies. 4.2 CONFIGURING AND PARSING THE BOOK:: The next step is to create a book and build scripts in dependency build order for one or several packages. Run to launch the configuration interface. The main menu contains four blocks: individual package selection, Build settings, Build layout, and Optimization. In the package selection block, menus and submenus are organized as the book's parts, chapters and sections. You can navigate those menus and select as many targets as you want. But we suggest to not select too many at a time to be able to sort issues! In the "Build settings" submenu, the dependency level and default packages used to solve alternatives are set (currently, only for the MTA). You can also select whether the build will be made as a normal user or as root, whether to use "porg style" package management, whether to remove ".la" files, and wheter statistics for the package are generated (build time, memory footprint and "DESTDIR" install). If you use package management, you have to enter the path to the packInstall.sh script too. In the "Build layout" submenu, you can select where the source tarballs reside and are downloaded, where the packages are built, and whether to keep the build tree after installation. In the "Optimization" submenu, you can select the number of parallel jobs, and set the usual CFLAGS, CXXFLAGS, and LDFLAGS. the special keyword "EMPTY" can be used for those flags to ensure they are unset. Note that there are help strings associated to those menus. Please read them for details! Those settings are saved to be reused in future configuration runs. When you are done with the menu, a few checks occur, and the dependency chain is generated. Each dependency appears with its priority (required, recommended, optional, or external), and it's level. There is a root level 1. The selected packages have level 2. The dependencies of selected packages have level 3, the dependencies of the dependencies have level 4, and so on. When circular dependencies are found, they appear with a priority of "circular". This means that two (or more) dependency chains arrive at the same package. The algorithm chooses the chain with the highest priority and reorders dependencies to remove the other chain(s). This is not always the solution an user would prefer, but we have found no way to do it better. You end up with a book.xml file which contains the linearized book, and a rendered HTML, in the directory book-html, which you can browse with "lynx book-html/index.html" (or with any other browser). Furthermore, there is a directory "scripts", which contains the generated scriptlets. There is yet another directory, "dependencies" that contains files generated while resolving dependencies. 4.3 EDITING BUILD SCRIPTS:: Now it is time to review the generated book and scripts, making any changes to the scripts necessary to fix generation bugs or to suit your needs. Scripts for additional packages (i.e., for non-BLFS packages) can be easily inserted. For example, if you want to install the external dependency "bar" before "foo" package and the "foo" script is named "064-z-foo", you just need to create a "064-y-bar" build script. Remember, the package tracking system isn't a package management tool and knows nothing about packages not in the BLFS book. 4.4 CREATING THE MAKEFILE:: When the build scripts are ready to be run, the Makefile can be created. Create an empty subdirectory (for example "mkdir work") and cd to that directory. Then run ../gen-makefile.sh. Note that the directory is completely emptied before generating the Makefile, so to prevent erasing useful data, the script ensures that the name of the current working directory starts with "work". Review the Makefile, and, if all looks sane, start the build by running "make". 5. GENERATED BUILD SCRIPTS ISSUES:: In this section, known issues with the generated build scripts are discussed. They are due to build procedures and/or BLFS layout particularities that we can't handle. In several cases, editing the build scripts is mandatory. You may also need to insert some build scripts created by you to resolve unhandled dependencies and/or to remove some script installing an unneeded package (unneeded packages may be pulled in the dependency chain, if they occur as an "or" with another package). When there are circular dependencies (only one known in BLFS 8.0 for recommended dependencies), you may need to move around scripts so that they run in the order script-A script-B script-A. This involves copying script-A to another name (using the xxx-a- fields), and possibly renaming the xxx-a- fields of each involved script. 5.1 BLFS BOOTSCRIPTS:: Normally, bootscript installation should work. On the other hand, the book does not give instruction for running them, so you might have to manually insert "/etc/init.d/rc.d/ start" at some place during the build. 5.2 PACKAGE CONFIGURATION:: For those packages that have a "Configuration" section, you should edit the build script to fit the needs of your system. Sometimes, the bash startup files are modified. The generated scripts contain a line 'source /etc/profile', which ensures that the proper environment variables are used. 5.3 PAGES WITH TWO OR MORE PACKAGES:: For example: sane, poppler, audacious, freetts, which, etc. On the pages for those packages, the BLFS book actually has instructions to download and install two or more packages. You must edit the scripts to fix this. A common pitfall is that the variable PACKAGE may be used for several tarballs. Be sure to save the PACKAGE variable to some other name (for example PKG1, PKG2, etc) after each download. The unpacking instructions may need to be repeated for each tarball in turn. 5.4 XORG7 The book has special page layouts for the Xorg7 packages. The tool breaks those pages into individual pages for each packages in the linear book. Also, the menu gives the choice to select each package individually. To build the whole Xorg7 chapter, select xinit. The (recommended) dependency chain brings in the whole set of Xorg packages. 5.5 PATCHES Please, make sure that all scripts have the commands to download/apply the required patches. Due to book layout issues, some patches may be missing (as of BLFS 8.0, all the patches seem to be downloaded). 5.6 ROOT COMMANDS If building as a normal user (the default setting), be sure that all commands that require root privileges are run using sudo. Also make sure necessary root privilege commands are visible in your PATH. The scripts ensure that /usr/sbin is appended to the user's PATH when running privileged commands. For commands necessitating root privileges, the generated scripts wrap them with the construct: sudo -E sh -e << ROOT_EOF ROOT_EOF The "-e" switch to sh ensures the command block exits with error if an error occurs. The "-E" switch to sudo ensures the whole environment is passed to the commands to be run with root privileges. It is effective only if the /etc/sudoers file contains `Defaults setenv', or SETENV in the user attributes (this is implicit if the command the user is allowed to run is `ALL'). If you think it is a security issue, you may forbid this flag in /etc/sudoers, but then, you have to un-escape `$' for variables coming from the environment in the instructions. Although this construct is rather strong, it can fail in some corner cases, so carefully review those instructions. Due to book layout issues, some sudo commands may be missing. 5.7 OTHERS There may be other issues that we are not aware of. If you find any, please report it to .