Make offline-build the standard method for rust-based packages

[Rainer]

Contrary to what is said in the book, an internet connection is not mandatory for building rust-based packages like cargo-c, cbindgen or librsvg. With "cargo vendor" a simple alternate method is available which makes an offline build possible.

From a user perspective an offline build has several advantages:

• It increases transparency and better control as it allows the user to take a look at the downloaded packages _before_ the build starts.
• It thus may help users who have qualms about having to blindly download and accept stuff when building software for their systems.
• By copying directory "vendor" (and perhaps other needed files), the user creates a backup which can be reused if the build fails. After all, users are admonished to start each build with a freshly unpacked source which otherwise would mean to repeat the whole downloading-act.
• It also may make the user aware that the often tiny download-size of the target-package may be misleading and may just be the lesser part of the download-piper.

Thus I think that - where possible - "cargo vendor" plus ensuing offline-build should be the preferred method for rust-based packages in BLFS.

Edit: On request, I've added an example for librsvg-2.61.4, see below. The formatting is not as intended but I don't know how to get this right here, sorry.

Rainer

--

Example for librsvg-2.61.4

# The build of librsvg-2.61.4 requires a lot of packages ("crates") which must be downloaded. With an internet connection active, issue

cargo vendor

# After that, the internet connection is no longer needed. If you like, inspect the contents of directory "vendor" which was created by "cargo vendor" and contains the "crates". For possible reuse, make a copy:

cp -r vendor ../vendor_librsvg-2.61.4

# Create directory ".cargo":

mkdir -v .cargo

# Create file .cargo/config.toml:

cat > .cargo/config.toml << "EOF"
[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "vendor"
EOF

# Again, for possible later reuse:

cp -r .cargo ../.cargo-librsvg-2.61.4

# If you later want to repeat the offline-build with a freshly unpacked source:

# ln -sv ../vendor_librsvg-2.61.4 vendor

# ln -sv ../.cargo-librsvg-2.61.4 .cargo

# First, fix the installation path of the API documentation:

sed -e "/OUTDIR/s|,| / 'librsvg-2.61.4', '--no-namespace-dir',|" \ [...]

[Xi Ruoyao]

IMO this should be a section in "Notes on Building Software" instead of a change to all the packages, similar to other things which provide more control on building and optionally applicable to many packages (like verifying GPG signature or hardening the build).

We should also explain how cargo works and why we require a "locked" build (notably, for cargo-c where we download the Cargo.lock file to ensure that) there.

[Bruce Dubbs]

I like this idea. We can also put an xref to that from the individual packages that can use it.

As far as explaining how cargo works, I'm not sure I understand the "how" or "why". Can we get someone to create a rough outline?

[pierre]

Not sure I am the best suited for doing such a task, but I have begun learning rust through tutorials, and might be able to explain the "why" and "how"...

[pierre]

Interestingly, searching for packages that have Cargo.toml files in their tarball return 39 packages in LFS/BLFS. To my surprise, even meson has a handful of such files. After further examination, it seems that they are only for tests of rust subprojects, and they don't have external dependencies.

It looks like using cargo fetch while setting CARGO_HOME is what we want. I'll test on the 38 remaining packages...

[Xi Ruoyao]

I still prefer cargo vendor instead of cargo fetch with CARGO_HOME. cargo vendor is more idiomatic.

[pierre]

Replying to Xi Ruoyao:

I still prefer cargo vendor instead of cargo fetch with CARGO_HOME. cargo vendor is more idiomatic.

I may be out of my depth here, but:

export CARGO_HOME=/location/of/local/crates/repo
#optionally fetch or copy a Cargo.lock file
cargo fetch
cargo build --frozen --release

seems simpler to me than creating a .cargo/config.toml directory+file.

[pierre]

Also, I just tried cargo vendor and it seems to store downloaded crates at two places (in directory vendor and in ~/.cargo).

[Xi Ruoyao]

~/.cargo is the machine-wide cache and vendor is the package-wide directory. I.e. if two rust packages need the same crate, when you build the second one it can be directly reused from ~/.cargo (or copied to vendor if you use cargo vendor).

With CARGO_HOME it will be downloaded all over again.

And users may need ~/.cargo/config.toml to configure a nearby mirror of crates.io, using CARGO_HOME will require to copy it into the new CARGO_HOME.

[pierre]

Replying to Xi Ruoyao:

~/.cargo is the machine-wide cache and vendor is the package-wide directory. I.e. if two rust packages need the same crate, when you build the second one it can be directly reused from ~/.cargo (or copied to vendor if you use cargo vendor).

Thanks for the clarification, but I think the next sentence is not true:

With CARGO_HOME it will be downloaded all over again.

If you use the same CARGO_HOME for all packages, then it plays exactly the same role as $HOME/.cargo (the doc says $HOME/.cargo is the default if CARGO_HOME is not set). My idea is that specifying CARGO_HOME allows the user to choose where crates are downloaded, and that it may be a good idea to have a crate store in /sources (for example CARGO_HOME=/sources/cargo)

And users may need ~/.cargo/config.toml to configure a nearby mirror of crates.io, using CARGO_HOME will require to copy it into the new CARGO_HOME.

That's another interesting point, that we may want to put into the explanations, but without direct relation to building offline.

Note that I agree all those explanations have to go into "Notes on Building Software".

[pierre]

I begin to be convinced by the cargo vendor route: the problem is that packages that use meson as a build system redefine CARGO_HOME (at least glycin and loupe), so that running cargo fetch with CARGO_HOME set to some location does not prevent downloading packages again to the redefined location...

[pierre]

There are several packages in BLFS that ship one or more files named Cargo.toml. There is also a couple of package in LFS (gettext, meson). I've gone through all of them to check how they can be built offline. In the following, standard build will mean:

export CARGO_HOME=/location/of/local/crates/repo
#optionally fetch or copy a Cargo.lock file
cargo fetch
#then build offline
cargo build --frozen --release

Vendor build will mean:

cargo vendor
mkdir -v .cargo
# Create file .cargo/config.toml:

cat > .cargo/config.toml << "EOF"
[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "vendor"
EOF
#then build offline using book instructions

Note that when standard build works, vendor build works as well.

• cargo-c: standard build works
• cbindgen: standard build works
• firefox: does its own vendor build, no need to do anything for building offline
• fontconfig: there is a rust subproject (fontations), but it is only built if using meson, and we use autotools. Even if using meson, it is an opt-in project.
• gcc: has a rust component with a few vendored sources, we do not build it anyway.
• git: has POCs of rust libraries to access git internals. We do not build them.
• glad: cargo is used for tests, but offline anyway.
• glycin: wraps cargo inside a meson build system, which defines CARGO_HOME to an internal location. We already use the vendor build in the book, so that everything can be done offline after the vendor part.
• gst-plugins-rs-gstreamer: standard build ok
• gtk-vnc: there is a Cargo.toml in a subproject, which seems unused
• harfbuzz: rust can be optionally used, but it uses unstable features (-Z option), so cannot be built in BLFS.
• libreoffice: building offline is not only a rust problem, so not considered here. I am not sure about the right method for the rust part anyway.
• librsvg: can be built offline after running cargo fetch; then use book build instructions
• loupe: the build system redefines CARGO_HOME. So use a vendor build.
• mariadb: there is a Cargo.toml in bundled wolfssl, which is used if system openssl is not found. That is, does not prevent building offline.
• maturin: standard build works, but tests cannot be run offline.
• mercurial: note that the rust part is opt in. If we want it: cd rust; cargo fetch; cd .. then build offline works (NB (not relevant for this ticket) rust tests can be made to pass)
• meson: cargo may be used in tests, but without external dependencies, so can be testsed offline.
• mupdf: there is a Cargo.toml in bundled zxing-cpp, which is opt in. If building with barcode=yes, we should use the system zxing-cpp anyway.
• node: there is a Cargo.toml in ittapi, which does not seem to be used
• numpy: there is a vendored-meson, which contains some Cargo.toml files (see above at meson). I am not sure which meson is used (vendored or system), but anyway, tests are not run.
• protobuf: there are several rust projects in the rust directory, but nothing uses them.
• qemu: there is a rust subproject, but we do not enable it
• qt6: Cargo.toml only in qtwebengine
• qtwebengine: a lot of Cargo.toml in 3rdpaty, but rust is not enabled in our build, so I think they are not used
• ruby: cargo is not used for release builds (according to doc), so no risk to download anything. Note that rust is directly called.
• rust-bindgen: standard build ok. Not related to this ticket: tests should be run after installing, since cargo test recompiles bindgen...
• rustc: use ./x.py vendor, then build can be done offline
• samba: there is a Cargo.toml file, which is just a placeholder for future rust integration
• seamonkey: same as firefox
• setuptools_rust: the build does not use rust. There are Cargo.toml files under the examples directory, so not relevant here.
• snapshot: uses vendored sources, so builds offline
• texlive: Cargo.toml is in a subdirectory of vendored harfbuzz: we don't use it (see above harfbuzz)
• thunderbird: same as firefox
• uv_build: needs cargo fetch, then book instructions can be run offline.
• webkitgtk: there is a Cargo.toml symlink pointing to a non existent file under thirdparty dir. Nothing to worry about.
• zxing-cpp: there is a rust wrapper that is not built by default.

[pierre]

Fixed at 0353eb21e3