rustc will be needed for firefox-53.

[ken@…]

LOL, muggins created a ticket for changing the LLVM build, but forgot to create one for rustc.

​https://www.rust-lang.org/en-US/other-installers.html#source - at the very bottom of the page, the first link is for Stable and labelled '.tar.gz'. When I tried that a few days ago I got rustc-1.15.1.

Although configure still permits the

--disable-rustbuild

switch (fedora use that), it reports

WARNING: the makefile-based build system is deprecated in favor of rustbuild

It is recommended you avoid passing --disable-rustbuild to get your
build working as the makefiles will be deleted on 2017-02-02.

So I suppose best not to do that.

Lots of configure options, and rustbuild downloads a binary from rust and uses that to bootstrap the build. For the moment I'm using the shipped LLVM.

I particularly wish to note two things:

First, the build defaults to silence (nothing until it completes or fails), but

VERBOSE=1 make

provides output.

Second, it uses all cores, so trying to do timing with only 4 cores on a well-endowed machine is probably not going to be possible. And adding MAKEFLAGS such as

-j2 -O

either after 'make' or as

MAKEFLAGS='-j2 -O'

in front of make is both ineffective (still uses all 4 cores on my i3) AND silences the build!

AFAICS, the cargo crates which this will enable are packages of metadata to ensure a repeatable compile, so I _think_ this is still ultimately from source.

[ken@…]

What a parcel of rogues!

There was apparently a fix for DESTDIR installs in january, but I can't make sense of the multi-file diff at ​https://github.com/rust-lang/rust/pull/38654/files (that was because gentoo complained DESTDIR no longer worked).

I see that gentoo are still using

--disable-rustbuild

for 1.15.1, whilst Arch (who seem to use DESTDIR without problems) are on 1.14.0 which presumably pre-dates the change to use rustbuild.

If there was a better browser than firefox, for those of us who build from source, I would use it.

[ken@…]

The more I look at rust, the more its build systems offend me: no doubt it provides the security its adherents claim, but for people who build from source on each machine it is a heavy burden (e.g. with the shipped llvm it builds for each possible architecture (AArch64, MIPS, PowerPC, SystemZ) and the only option is to not *install* the non-host architectures. I have now found, at ​https://developer.mozilla.org/en-US/Firefox/Building_Firefox_with_Rust_code that there may be a small delay:

Disabling Rust code

Support for building without Rust code was removed in bug 1284816. For Firefox 54 it's possible to build without Rust by patching out that and subsequent changes, but starting with Firefox 55 this becomes increasingly difficult and affects user security and available features.

That implies that ff-53 (not ff-52) will be the last non-rust version. But even for ff-54, patching out that commit plus all subsequent commits sounds a large activity, and obviously some of those subsequent commits will be security related. So, perhaps 6 weeks more than initially threatened.

[ken@…]

A few comments before I set this aside for a few days:

1. 1.16 is due for release on St Patrick's day. This is currently the beta branch and includes the commit which may allow DESTDIR builds with rustbuild. I say may because I can't build that branch and the commit looks unlikely, with a description which suggests it might be a first stage.

2. At the moment, the stable branch is essentially at 1.15.1, looks as if that will be the last 1.15 release. I see that Arch were using 1.14.0 when I last looked - if we have to build this, we might need to use a not-latest version. For 1.15.1 DESTDIR works and it is also possible to prevent it installing (but not building) arches such as AArch64, MIPS, PowerPC, etc.

3. In both the master (development) and beta branches I get similar errors (using rustbuild, it cannot be disabled in those versions). At first I thought this was a missing file, but the original error is some way above that - a file which is named as if it is a script, but is actually an ELF file, fails without explanation.

4. There are references to running ./x.py in the configure output, but 1.15.1 does not have that. Using the master branch, I _thought_ I had persuaded that to use config.toml to override certain defaults. But after returning to 1.15.1 to get something that would actually build and install, I found it installed in /usr/local instead of my intended /opt/rust (I don't want a real install until I can tune it, that is a next-best place in the absence of DESTDIR) and apparently ignored all the settings I had provided. Possibly, config.toml is only useful where ./x.py can be invoked.

5. With rustbuild there is (theoretically, in the light of point 4 above) an option to specify a different number of codegen-units : this seems to NOT be the same as 'make -jN'. From what I can see, rustbuild forces its own optimizations (-O3) on release builds, and the codegen-units seem to be soemthing along the lines of "build more quickly, but with poor code, or build more slowly with good code" - for releases, the latter option is chosen. This seems to be aimed at "build once on a build server, deploy widely", rather than "build for the machine you are using".

6. And rustbuild appears to grab all the cores of hte build machine when it can (from time to time, e.g. linking or copying a solib to somewhere, only 100% CPU is used, but on my i7 it gets close to 800% usage for some of the build.

7. With old-fashioned configure builds, it is possible to restrict which architectures get installed (although they all appear to get compiled - I was surprised that system LLVM with only the host architecture managed to do this). In master this can apparently be done in config.toml.

8. On a powerful machine, building rust takes a lot longer than building LLVM. I suggest that if we have to build this, we should encourage people who upgrade firefox to use their current LLVM and rust versions whilst those are still adequate - for the moment, it looks as if current rust will not be ready for LLVM > 3.9 (at the moment there is a test for 3.7 to 3.9 as the only supported versions).

[ken@…]

With 1.16.0, --disable-rustbuild is still supported, and still produces the same warning. I'm initially using

./configure --prefix=/usr --llvm-root=/usr --enable-llvm-link-shared --disable-jemalloc --disable-rustbuild

and

python x.py build --verbose

(a mix of gentoo and Arch) : when it can, this uses all CPUs. This is on LLVM-3.9, but make DESTDIR=... seems to be rebuilding from scratch using gcc with -j1 <sigh/>

[ken@…]

Nasty, very nasty, but maybe debian can fix it.

Please recall that this is llvm-3.9. In 'configure' is the following test:

    case $LLVM_VERSION in
        (3.[7-9]*)
            msg "found ok version of LLVM: $LLVM_VERSION"
            ;;
        (*)
            err "bad LLVM version: $LLVM_VERSION, need >=3.7"
            ;;
    esac

so this version definitely will NOT build with LLVM-4.

On my phenom, reconfiguring with the same options, then using

make VERBOSE=1 -j4 -O

it completed in about 15 SBU : not sure if the makeflags were obeyed, or whether it just uses all cores when it can (long periods using only 1 core, probably during the linking).

It then managed a DESTDIR (I forgot to note the time) - 2.0GB source, 369 MB installed. The nasty part is that the three programs in the DESTDIR are linked to the libs in the DESTDIR (proved with ldd and strace). Debian has a patch for 1.15.1 that might solve this.

I then ran make check tee'd to a log : that used all cores, and ultimately failed in a jemalloc test (there are reports that it makes assumptions about jemalloc, I can't be bothered to care about the details). Again, debian has a patch which might work.

From earlier testing, ISTR that building with the shipped LLVM is a lot bigger and a lot slower.

[ken@…]

The debian patches are, at least partially, applied in 1.16. Long story short - if I install rustc and one lib (librustc_driver-*.so : initially ldd said that was not found), and run ldconfig, ldd shows that lib is now found and all the others are missing (I renamed the DESTDIR to be absolutely certain).

I ran the tests after that - the jemalloc test appears to not be run, but I now get 104 failures from all of the gdb tests (because gdb is not present). I suppose that means gdb is needed for the testsuite.

This was using rustbuild, but with configure and using *static* llvm (because rust loves static libs, and things might break in future versions of either rust or LLVM). For that, I needed a sed from fedora to ensure -lffi is picked up when system LLVM (built against libffi) is used: there have been alternative fixes, apparently they no longer work (the shipped LLVM is not built against libffi). :

sed -i -e '$a #[link(name = "ffi")] extern {}' src/librustc_llvm/lib.rs

and then

./configure --prefix=/usr --llvm-root=/usr
./x.py build
DESTDIR=/some/path ./x.py dist --install
./x.py test

The tests can be run after the build, but I was more concerned about the DESTDIR.

Without the tests, build tree was 1.6G, install 264M. The tests added 0.3G to the build tree.

This was cleaner than previously: I ran each command | tee somelog : the output on the term showed which file was being built, no need to force verbosity. It takes all the cores, of course. When I did get an error (one was local, second eventually led me to the sed) the output did eventually show the error - 'eventually' because the first time it took about 3 minutes waiting for other jobs to stop, with 1 core at 100%, and then it stopped and reported the detail.

No idea how much of it used cmake - I suspect that only really gets used if building the shipped LLVM. That will be my next step, along with trying to stop it installing for other architectures, and using a versioned docdir.

The docs are in /usr/share/doc/rust (unversioned).

[ken@…]

I've now done a build, test, DESTDIR install with gdb installed - using the shipped LLVM.

./configure --prefix=/usr --docdir=/usr/share/doc/rust-1.16.0 --enable-dist-host-only
./x.py build
./x.py test
DESTDIR=/some/path ./x.py dist --install

That --enable-dist-host-only ensures files for e.g. AArch64, ARM, MIPS, PowerPC, Sparc, SystemZ are only compiled, not installed (no option to prevent them getting compiled).

This seems to do the right thing, but is (of course) much slower building LLVM. The tests took 22m25 on this machine (about 8.5 SBU) - in the previous build without gdb the tests were only just over 10 minutes.

Unfortunately, 10 of hte gdb tests still failed.

I think this is now ready to give it a go (I suspect that I will probably need to build cargo before I can use this for firefox, and maybe there is still some problem that will stop it working, but it looks as if it might work.

Summary of build sizes and SBUs (all with 4 cores)

configure, make using shared system LLVM about 2.3G (369M installed), 15 SBU

configure, rustbuild using static system LLVM about 2.1G (264M installed), 11 SBU

configure, rustbuild using shipped LLVM about 3.5G (288M installed), 20 SBU.

So, in an ideal world static linking is best for this, and using system LLVM. In reality, we have already moved LLVM to 4.0 (and the next version will apparently be inflated to 5.0) but it seems unlikely that rust will cope with newer versions by the time we need to use it.

[ken@…]

To actually *use* rust, firefox also requires cargo which is if anything even worse than rust for building from source (if cargo is not present, need to manually download the previous version to bootstrap - not sure if older installed version would work), and it ships without rust-installer which prevents it from installing. Long notes, I don't want this to be wasted if I get knocked over by a bus.

I've now got it building firefox-52.0.1, so I assume what I've got is adequate (the phrase "good enough" doesn't really fit something which is big, slow, and a new requirement). And I'm loathe to describe this as "quick and dirty", it doesn't feel quick ;-)

With rust installed, download cargo-0.17.0 and then ​https://github.com/rust-lang/cargo/archive/cargo-0.16.0-x86_64-unknown-linux-gnu.tar.gz and get rust-installer-master.zip from github.

Untar cargo, within that directory

tar -xvf ../cargo-0.16.0-x86_64-unknown-linux-gnu.tar.gz
unzip ../rust-installer-master.zip
mv -v rust-installer-master/* src/rust-installer

No, that doesn't move all the test/ files for the installer, and they don't seem to be needed Build with

./configure --prefix=/usr \
 --sysconfdir=/etc \
 --docdir=/usr/share/doc/ \
 --cargo=./cargo-nightly-x86_64-unknown-linux-gnu/cargo/bin/cargo
make -jN
CFG_DISABLE_CROSS_TESTS=1 \
 ./target/x86_64-unknown-linux-gnu/release/cargo test
make install

The tests all pass, I didn't count how many there were and lots of individual summaries.

grep 'result.*failed' testlog | cut -d ';' -f2 | uniq

should produce one line of output, ' 0 failed'

Firefox-52.0.1 now built and DESTDIR installed, time and space similar to a build without rust, but in the log I did see at least one reference to a rust lib with its hashed version (plus, my initial attempt without cargo failed quickly), so this process seems to be adequate.

Setting this aside until whichever version of firefox really does require rustc.

[ken@…]

Changed title - I just tried to build 53.0b7, it does indeed need rust.

checking for rustc... not found
checking for cargo... not found
ERROR: Rust compiler not found.
To compile rust language sources, you must have 'rustc' in your path.
See https//www.rust-lang.org/ for more information.

You can install rust by running './mach bootstrap'
or by directly running the installer from https://rustup.rs/

*** Fix above errors and then restart with\
               "make -f client.mk build"
make[2]: *** [/scratch/ken/firefox-53.0b7/client.mk:383: configure] Error 1

And just in case, I tried that command:

Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

NotImplementedError: Bootstrap support for this Linux distro not yet available.

And it does still need the libevent sed.

[ken@…]

It is also possible to build rustc using an existing rustc, so that it does not have to download the stage0 binary tarball. Unfortunately, that involves building stage0 using the existing rustc, so on my SandyBridge i3 it takes an extra 4 SBU with 4 cores, and it seemed to only save about 100 MB of disk space. So, although the "keep it pure after the initial bootstrap" people will want to do this, it seems hard to recommend unless you have either a very fast machine or a slow/capped internet.

For me, it also downloaded some cargo files - that might have been because I had wiped out ~/.cargo - not sure.

The cargo files are actually renamed .tar.gz files, containing source, docs, etc. Tar can extract them, vim doesn't understand what they are, so using view to look at the contents does not work.

[ken@…]

The rust tests are random. I've now raised ​https://github.com/rust-lang/rust/issues/41127 - for the moment I can see no point in running whichever subset of the tests will be randomly chosen (at the test time, not at the build time) for the current run, particularly since some selections take a long time on a 4-core machine.

I still need to tar up a git clone of the rust installer for cargo (only available as a snapshot, and that gives a zip file), then double check the commands for that, but I'm nearly ready to add these packages.

[ken@…]

Done in r18591, firefox-53 is expected next week.

[bdubbs@…]

Milestone renamed

[bdubbs@…]

Milestone renamed