Must re-evaluate package order then document the rationale.

[greg@…]

More info in the thread starting at the above URL.

Alphabetize where possible. Be careful of test suites. Check purity iteration effects.

After 5.0 release sometime.

[gerard@…]

Downgrading to P3. Would be nice to have fixed in the next release but is not of the utmost importance if it doesn't.

[ken@…]

One small data point for this - cross-lfs had relegated iana-etc to near the

end of the build, and had some errors in the perl testsuite (particularly, a few messages about a particular protocol). By restoring iana-etc to where it is in regular lfs, the perl testsuite completes without error.

[archaic@…]

Updated trunk (r7070). Re-open if you feel --infodir is needed.

[archaic@…]

Aak! I was on the wrong bug! :/

[chris@…]

I'm currently testing an lfs build with nearly-alphabetical package order right now. Here is what I have so far:

Chapter 5

Ncurses-5.5 Bash-3.0 Bzip2-1.0.3 Coreutils-5.92 Diffutils-2.8.1 Findutils-4.2.25 Gawk-3.1.5 Gettext-0.14.5 Grep-2.5.1a Gzip-1.3.5 M4-1.4.4 Make-3.80 Patch-2.5.4 Perl-5.8.7 Sed-4.1.4 Tar-1.15.1 Texinfo-4.8 Util-linux-2.12r

Chapter 6

Ncurses-5.5 Readline-5.0 Zlib-1.2.3 Autoconf-2.59 Automake-1.9.6 Bash-3.0 Bison-2.1 Bzip2-1.0.3 Coreutils-5.92 Diffutils-2.8.1 E2fsprogs-1.38 File-4.16 Findutils-4.2.25 Flex-2.5.31 GRUB-0.97 Gawk-3.1.5 Gettext-0.14.5 Grep-2.5.1a Groff-1.19.2 Gzip-1.3.5 Hotplug-2004_09_23 IPRoute2-051007 Iana-Etc-2.00 Inetutils-1.4.2 Kbd-1.12 Less-382 Libtool-1.5.20 M4-1.4.4 Make-3.80 Man-1.6b Mktemp-1.5 Module-Init-Tools-3.1 Patch-2.5.4 Perl-5.8.7 Procps-3.2.5 Psmisc-21.6 Sed-4.1.4 Shadow-4.0.12 Sysklogd-1.4.1 Sysvinit-2.86 Tar-1.15.1 Texinfo-4.8 Udev-071 Util-linux-2.12r Vim-6.3

Also, is there any reason that dejagnu is after tcl and expect? Does it require either of them? Same with the toolchain build order in chap. 5, after the adjustment - is there a specific reason why gcc Pass 2 needs to be installed before binutils Pass 2, particularly since both Pass 1 and Chap. 6 build binutils first?

[chris@…]

I've made a little more progress...

1. Autoconf needs Perl (more than just the minimal Perl installed in Chap. 5) so

Perl needs to be among the 1st packages installed.

2. The Perl testsuite needs Iana-Etc, so Iana-Etc also needs to be moved up and

should be added to the list of Perl's dependencies.

3. Bash's testsuite has hard-wired paths for rm, chmod, and other Coreutils

programs, so Coreutils needs to be moved (back) to the beginning of the list.

I've just built a complete system with this build order, and it seems to be worknig fine so far. Here's the slightly modified build order...

Coreutils-5.92 Iana-Etc-2.00 Ncurses-5.5 Perl-5.8.7 Readline-5.0 Zlib-1.2.3 Autoconf-2.59 Automake-1.9.6 Bash-3.0 Bison-2.1 Bzip2-1.0.3 Diffutils-2.8.1 E2fsprogs-1.38 File-4.16 Findutils-4.2.25 Flex-2.5.31 GRUB-0.97 Gawk-3.1.5 Gettext-0.14.5 Grep-2.5.1a Groff-1.19.2 Gzip-1.3.5 Hotplug-2004_09_23 Inetutils-1.4.2 IPRoute2-051007 Kbd-1.12 Less-382 Libtool-1.5.20 M4-1.4.4 Make-3.80 Man-1.6b Mktemp-1.5 Module-Init-Tools-3.1 Patch-2.5.4 Procps-3.2.5 Psmisc-21.6 Sed-4.1.4 Shadow-4.0.12 Sysklogd-1.4.1 Sysvinit-2.86 Tar-1.15.1 Texinfo-4.8 Udev-071 Util-linux-2.12r Vim-6.3

[Jeremy Huntwork]

I like this a great deal. Chris did you make any changes to the order of the tool chain packages or the testsuite packages, either in chap. 5 or chap. 6? I'd like to get this into trunk and start playing with it a bit more.

[chris@…]

No, I made no changes to the toolchain. I did ask a couple questions about the order there (toolchain and testing tools) but didn't change them.

[Jeremy Huntwork]

After discussing this briefly with Matt, we'd like to test out your suggested order, Chris. We'll create a new branch from trunk at this point and re-arrange the packages as you've listed, toolchain and testsuite packages will remain as is (with the possible exception of putting dejagnu before tcl and expect, if it builds correctly.)

After the branch is created and the orders adjusted, jhalfs can be used to quickly (in a matter of a day or so) build 3 builds:

1) Trunk as it is now 2) The new branch 3) The new branch built from the new branch

We can then compare the builds and logs to see if any major flaws have arisen from the new build order.

[randy@…]

Well, I can save you a little bit of time. Dejagnu won't build before Tcl as it looks for the tclsh program.

[Jeremy Huntwork]

Thank you, Randy. So all testsuite and toolchain packages will stay as is.

[chris@…]

(In reply to comment #10)

Well, I can save you a little bit of time. Dejagnu won't build before Tcl as it looks for the tclsh program.

Dejagnu does look for tclsh, but it still builds without it. I don't know if that causes dejagnu to be missing functionality so I'm not sure if it should be done, but it does build and install without tcl.

[Matthew Burgess]

From comment 1:

Alphabetize where possible. Be careful of test suites. Check purity iteration effects.

So, looks like we're OK on parts 1 and 2 from that list. Doing part 3 (i.e. build LFS using Chris' build order, then use that to build another LFS, then compare logs and binaries for differences) should let you know whether or not any of the packages (not just dejagnu as mentioned in comment 12) suffer from any functionality changes.

Looks great so far, though. Thanks for tackling this, Chris!

[chris@…]

Another issue...in Chapter 6, m4 needs to be moved up (between ncurses and iana-etc) because autoconf creates the "autom4te" script with the location of the m4 binary hard-wired into it. This issue has already been mentioned on the mailing list and accounted for in the alphabetical branch, but I'm mentioning here as well to keep all the info about this in one location.

Also, I think it would be a good idea to list *all* dependencies where build order is important, even the ones that are already fulfilled by packages installed alphabetically. For example...

1. bzip2 and gzip must be installed before tar
2. ncurses must be installed before readline, which must be installed before bash
3. ncurses must be installed before texinfo, util-linux, and inetutils

...and so on. Listing every known dependency would make things much easier should a package be added, removed, or replaced in the future.

[Jeremy Huntwork]

(In reply to comment #14)

Another issue...in Chapter 6, m4 needs to be moved up (between ncurses and iana-etc) because autoconf creates the "autom4te" script with the location of the m4 binary hard-wired into it.

Similar adjustment has been done for sed. It now comes just after readline early in chapter 6. e2fsprogs and libtool were hard-coding the full path to the sed in /tools into their scripts.

Also, I think it would be a good idea to list *all* dependencies where build order is important, even the ones that are already fulfilled by packages installed alphabetically.

Yes, that is part of the bug that needs to be fixed. Since you've done so much of the initial research on this, I would appreciate any patches to the book's XML (or at least revised text for each package) that you could provide, assuming you've documented this information already.

[chris@…]

(In reply to comment #15)

(In reply to comment #14)

Another issue...in Chapter 6, m4 needs to be moved up (between ncurses and iana-etc) because autoconf creates the "autom4te" script with the location of the m4 binary hard-wired into it.

Similar adjustment has been done for sed. It now comes just after readline early in chapter 6. e2fsprogs and libtool were hard-coding the full path to the sed in /tools into their scripts.

Also, I think it would be a good idea to list *all* dependencies where build order is important, even the ones that are already fulfilled by packages installed alphabetically.

Yes, that is part of the bug that needs to be fixed. Since you've done so much of the initial research on this, I would appreciate any patches to the book's XML (or at least revised text for each package) that you could provide, assuming you've documented this information already.

I have no idea how to create patches, but I will gladly suggest alternate text for any packages that need it. Although I think all the build order explanations

should go in a single place in the book, maybe adding a new section just for

this (or adding to the "How things are going to be done" section). As for each individual package, the only things that need be changed are the list of what each package "depends on", and I think that could stay as just a simple list of other packages it needs - the complete explanation for what package needs what and in what order can all be put into one section, making it easy for people who just want to follow the book and build a system to skip over that one section if they don't really care about why everything is ordered the way it is. Any other thoughts on how this should be done?

[Matthew Burgess]

(In reply to comment #16)

Although I think all the build order explanations should go in a single place in the book, maybe adding a new section just for this (or adding to the "How things are going to be done" section).

I agree, a single section explaining why things are built in the order they are is preferable. Briefly looking at the book, I think ' 1.1. How to Build an LFS System' is too high-level for this info, and the focus of "5.2. Toolchain Technical Notes" is on the toolchain, rather than general build order issues.

So, perhaps a new section "1.2. Package Build Order"? This could (perhaps should?) point to 5.2 as containing further discussions regarding the order of the toolchain packages.

As for each individual package, the only things that need be changed are the list of what each package "depends on", and I think that could stay as just a simple list of other packages it needs

Agreed.

[chris@…]

I just realized GRUB isn't listed in alphabetical order (it's the first package listed that starts with a "G") but the sort I did put it first because it's in all-caps. It should be between Groff and gzip.

[Jeremy Huntwork]

(In reply to comment #18)

I just realized GRUB isn't listed in alphabetical order (it's the first package listed that starts with a "G") but the sort I did put it first because it's in all-caps. It should be between Groff and gzip.

I noticed this too, and was going to change it, but I realized that the LFS book refers to it as GRUB which would indeed put it at the beginning of the list. If this is really the only package that is an acronym, perhaps it should stay where it is. If there are others then perhaps we should move it.

[chris@…]

OK, here's what I have so far in listing build-order dependencies. Each package has a list of (what I know so far) what packages must be installed before that package, and the reason (if needed...if it's anything other than the obvious "links to libraries")...

Perl:

Iana-Etc (needed for testsuite)

Readline:

Ncurses

Autoconf:

M4 (hard-wires path to m4 binary in some scripts) Perl (needs more than the minimal Perl installed in chap. 5)

Automake:

Autoconf

Bash:

Ncurses Readline

Bison:

M4 (hard-wires path to m4 binary in bison executable)

Diffutils:

Coreutils (hard-wires path to pr program)

File:

Sed (hard-wires path to sed binary)

Findutils:

Coreutils (updatedb script hard-wires path to sort binary)

Inetutils:

Ncurses

Kbd:

Bison Flex

Less:

Ncurses

Libtool:

Sed (hard-wires path to sed binary)

Man:

Bzip2 (path to bzip2 in man.conf) Diffutils (path to CMP in man.conf) Gawk (makewhatis script hard-wires path to awk binary) Gzip (path to gunzip in man.conf) Less (path to PAGER in man.conf)

Module-Init-Tools

Zlib

Procps:

Ncurses

Psmisc:

Ncurses

Shadow:

Sed (several scripts (useradd, userdel, vipw) hard-wire path to sed binary)

Tar:

Bzip2 Gzip

Texinfo:

Ncurses

Util-linux:

Ncurses

Vim:

Ncurses

In addition to the above list of what packages must be built in what order, some packages need additions to the plain list of what they "Depend on"...

Perl - Iana-Etc Bash - Readline Man - Bzip2, Diffutils, Gzip, Less Module-Init-Tools - Zlib Tar - Bzip2, Gzip

Unfortuanately I didn't do complete testing for this because I forgot to run testsuites for most packages. Will start finding what (if any) additional dependencies would be needed for package testsuites to pass whenever I get time to try it...

[chris@…]

Adding some more...

Bash - need Coreutils (testsuite has location for rm and chmod (possibly others as well) binaries hard-wired into it)

Findutils - needs Coreutils (/bin/echo referred to in the testsuite)

Flex - needs Bison (flex will build without bison, but the testsuite looks for bison and fails without it)

GRUB - needs Ncurses - it will build without ncurses, but probably missing some functionality (though I can't find anything specific about *what* would be different)

Udev - needs Coreutils (testsuite has "/bin/echo"). This isn't all though...without coreutils installed into /usr and /bin, udev's testsuite complains that it "Can't exec 'tree'" and fails 26 tests. Somehow it seems "tree" is in the "echo" binary because symlinking /bin/echo ---> /tools/bin/echo causes mosts of the failing tests to pass...but there are still 3 more failures, also complaining about "tree". So I don't know what else is needed... Well, it seems it's something else in coreutils (I just installed coreutils into /usr and the udev testsuite worked with no failures).

Util-linux - needs zlib

[Matthew Burgess]

(In reply to comment #21)

Udev - needs Coreutils (testsuite has "/bin/echo"). This isn't all though...without coreutils installed into /usr and /bin, udev's testsuite complains that it "Can't exec 'tree'" and fails 26 tests.

When a Udev test fails it tries to run `tree' in order to provide detailed feedback on what the differences are between the expected result and the actual result. As we don't install 'tree' then this then causes the 'Can't exec 'tree'" results, but that's not the actual cause of the test failure.

I assume comment #21 was aimed at helping us improve the accuracy of our dependency lists, as for the most part the deps you mention are resolved by installing in packages in alphabetical order (bash->coreutils, grub->ncurses and util-linux->zlib being the exceptions).

[chris@…]

(In reply to comment #22)

(In reply to comment #21)

Udev - needs Coreutils (testsuite has "/bin/echo"). This isn't all though...without coreutils installed into /usr and /bin, udev's testsuite complains that it "Can't exec 'tree'" and fails 26 tests.

When a Udev test fails it tries to run `tree' in order to provide detailed feedback on what the differences are between the expected result and the actual result. As we don't install 'tree' then this then causes the 'Can't exec 'tree'" results, but that's not the actual cause of the test failure.

Ahhh, thanks for the clarification.

I assume comment #21 was aimed at helping us improve the accuracy of our dependency lists, as for the most part the deps you mention are resolved by installing in packages in alphabetical order (bash->coreutils, grub->ncurses and util-linux->zlib being the exceptions).

Yeah, the idea is to list *all* dependencies that depend on build order, even if it's "automatically" covered by putting them in alphabetical order. You may notice that comment #20 also lists several package dependencies that are covered by alphabetical order.

[chris@…]

I am building a system with this order, but moved vim near the beginning (suggestion from Gerard in the mailing lists describing the reason why vim is put near the beginning of Chapter 6), between sed and zlib. It seems to work fine, as all of its dependencies are already fulfilled. The only packages that are built before vim in the existing devel-LFS but not in the alphabetical branch are zlib, mktemp, and findutils, and none of those are listed in vim's list of dependencies.

[chris@…]

Update the list of installation dependencies for several packages in the book

[chris@…]

Update the list of installation dependencies for several packages in the book

[chris@…]

Also, perl needs procps (uses ps in the testsuite).

[chris@…]

Update the list of installation dependencies for many packages in the book

[chris@…]

I'm starting to write a new page to be added to the book describing the reasons for the build order, and containing the information about what packages depend on each other to build. Any ideas on what this should look like? A couple possibilities...

1. A table...

Column 1-This package... Column 2-Depends On... Column 3-Because...

2. nested list, like an outline...

1. Bash
   1. Readline
      1. Ncurses

Anyone have any other thoughts on this?

[chris@…]

Adding a little more dependency information...

Findutils needs DejaGNU (for the testsuite) M4 does NOT need perl (the book currently says it does)

[dbn.lists@…]

Chris, you're knocking this one out! I'll add whatever info I have from the ICA runs tomorrow including the build order currently used. I'm out of time tonight, though.

[chris@…]

Great work on all the ICA testing Dan! I just have a question...I believe the ICA testing checks whether the LFS system can reproduce itself flawlessly. Has there been any testing done to see if you get the same system starting from different hosts?

[dbn.lists@…]

Great work on all the ICA testing Dan! I just have a question...I believe the ICA testing checks whether the LFS system can reproduce itself flawlessly. Has there been any testing done to see if you get the same system starting from different hosts?

ICA could potentially highlight any errors coming from different hosts, but you'd have to try building from different hosts. Hopefully the Ch.5 in the book works well enough that by the time you get to Ch.6 it's independent of the host, but I don't know. I've never tested whether the end result from 2 different hosts is the same. It would be a time consuming test to set up, but it could be done.

The ICA testing has all been done on anduin, which is LFS SVN-20050109 on an Athlon XP 2100. At home I have a couple partitions set up, one with LFS ~20051006, and one with FC4. The problem is that it's a PIII 500, so it takes a LONG time to build anything. And it's my only box, so I can't just let it run for a couple days. (My girlfriend would kill me.)

If there's another host to use, we could do the test. Preferably a non-LFS host.

[dbn.lists@…]

ICA verified build order

[chris@…]

Found some more dependencies that need to be added...

Bash needs Bash and Texinfo IPRoute2 needs Bison and Coreutils Autoconf needs Texinfo Automake needs Texinfo Diffutils needs Texinfo Gzip needs Texinfo Man needs Groff Make needs Perl (for the testsuite) and Texinfo Mktemp needs Bash Module-Init-Tools needs Gawk (for the testsuite) Tar needs Texinfo Util-Linux needs E2fsprogs

[chris@…]

And one more...GRUB needs Texinfo

[dbn.lists@…]

I reversed the order to find new dependencies. Here's what I've got so far in addition to the last patch. These are just the show stoppers, i.e. the build crashes without them. The ICA should point out a couple more.

man-db: groff groff: bison (uses yacc)

[dbn.lists@…]

ICA was able to show some more dependencies. Here's what I have. Some are repeats of what Chris already has.

man-db: groff less grep gzip bzip2 flex

groff: bison

kbd: flex gettext

automake: autoconf

libtool: grep

util-linux: e2fsprogs gettext

tar: bison inetutils

I'm attaching order2.txt. This is an ICA clean run. The only difference between this and the previous is that I moved up bdb to be more alphabetical, and it had no effect on the results.

[dbn.lists@…]

New ICA clean alphabetical build order

[Jeremy Huntwork]

OK, the alphabetical branch has been adjusted to match your build order, Dan. Now I need to finish getting the dependencies listed. The book is rendered in my home dir again, so anything that's missing feel free to drop a line about.

[dbn.lists@…]

Jeremy, I see one missing thing in there. Since groff isn't installed yet in Ch. 6, perl makes the brilliant decision that you don't want the man pages. You can force it by changing the configure command to

./configure.gnu --prefix=/usr \
    -Dman1dir=/usr/share/man/man1 \
    -Dman3dir=/usr/share/man/man3 \
    -Dpager="/usr/bin/less -isR"

[dbn.lists@…]

A little more dependency info. All the attachments except order2.txt are missing, so I can't see whether Chris's last patch has this or not.

iproute2

• bdb - needed to build arpd
• bison and flex - needed to build /sbin/tc and /usr/lib/tc/q_netem.so

[dbn.lists@…]

Chris, on 11/06/05 you asked why gcc-pass2 comes before binutils-pass2 in Ch. 5. The reason is because of gcc-4.0.2-specs-1.patch. Since gcc is the driver of this whole operation, you want to get the new one installed that has /tools/lib/ld-linux.so.2 as the default dynamic linker and gets /usr/include out of the header search path.

gcc should probably be the first package built after both of the adjustments. I can't back that up with any technical information right now. The only reason the testsuite packages come before gcc-pass2 is for testing gcc-pass2.

[Chris Staub]

I believe the most recent patch I submitted has already been applied to the Alphabetical branch. In any case, I know there certainly wasn't anything on there about db, and nothing about iproute2 needing bison and flex.

[Chris Staub]

Just found some more dependencies...autoconf can use automake and libtool in its testsuite. Since automake requires autoconf to build at all, I am not sure what the solution would be. However, libtool doesn't appear to have any dependency on autoconf or automake so it could simply be moved up (or auto{conf,make} could be moved down).

[Chris Staub]

Adding some more details on possible solutions:

Moving libtool should be simple enough. It hard-wires paths to sed and grep so either libtool, along with sed and grep, could be moved up, or the other autotools could be moved down, right after libtool.

Autoconf/automake is a bit more complicated. Obviously, automake requires autoconf to build, so having autoconf use automake would require building autoconf twice. Is it really worth bothering to built it twice just to have a couple of autoconf tests pass? Would it be better to just add a note to the book mentioning that autoconf can use automake in its testsuite?

[Chris Staub]

Automake can also use libtool in its testsuite, as well as bison and flex.

[Chris Staub]

Automake testsuite can also use DejaGNU, Expect, Gettext (more than just the msgfmt binary), gzip, and gcc

Findutils needs Texinfo

[Chris Staub]

Just a couple more... (really!)

Gettext testsuite uses perl Grep testsuite uses gawk

[Chris Staub]

Libtool needs texinfo

[Chris Staub]

e2fsprogs can use gzip (in the test suite, and to compress its info docs)
Mktemp does NOT need zlib

[dbn.lists@…]

For some reason, I haven't been getting email updates on the bug.

Anyway, this libtool/autoconf/automake thing has me at a loss. If we move up libtool, then grep has to go with it. sed is already in the first block. So, that order would look something like this:

... coreutils grep iana-etc m4 ncurses procps sed libtool perl ...

Or, we could leave grep where it is and add /bin/grep -> /tools/bin/grep to the Essential Symlinks.

I say we ignore automake in autoconf testsuite. They're circular, so either one has to be in /tools to get full coverage, or we just pick one that goes first. No one is going to put autoconf in /tools since it already fights such a bitter battle to stay in LFS at all. So, I say we keep the order alphabetical with the unfortunate effect that autoconf loses some test suite coverage.

[dbn.lists@…]

Also, Gerard asked some time back that Vim be built as early as possible. We could probably move it right after Perl with no detrimental effects.

Is this still the case?

[Chris Staub]

I know this is something that was brought up a while ago, but I'm mentioning it again. DejaGNU does not, in fact, need either tcl or expect to build. I just installed dejagnu on my current system (into /dejagnu), which does not have either tcl or system, and on the new system I'm still building into and chrooted into, which does still have tcl and expect in /tools. I diffed /dejagnu and /mnt/lfs/dejagnu, and they are identical.

[dbn.lists@…]

As for DejaGnu. tcl seems to be used implicitly through expect in the DejaGnu testsuite. tcl is also used in make dist, but that's not really our concern. Look at /tools/bin/runtest. It definitely shows run-time dependency on expect. We might be able to remove tcl from the deps list, though, as it is implied through expect.

[Chris Staub]

I just noticed this from Dan's list of additional dependencies found - tar: bison inetutils. Never knew about that one, and can't quite find out where tar needs those packages. Any more details on this?

[Chris Staub]

Found another one...seems Bash now depends on Bison. The patch modifies a .y file so bash's Makefile uses bison/yacc.

[Jeremy Huntwork]

Yay! Closing finally! We may spot an item or two that will need to be adjusted in connection with this ticket, but by far the majority of the work is done and in the book. From this point any changes are minor updates and corrections.

Done as of r7536.