qtwebengine after 5.15.2

[ken@…]

As expected, future qt5 releases will be limited to commercial customers at first ​https://www.qt.io/blog/commercial-lts-qt-5.15.3-released and will apparently become available to the rest of us when 6.1 is released (maybe in April). When this was announced there were reports that qtwebengine had to be made available because of its license. Now that 5.15.3 has been reported (see e.g. phoronix) I googled.

Gentoo are using a git version from 24th February, apparently with extra workarounds because this is from git ​https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-qt/qtwebengine/qtwebengine-5.15.2_p20210224.ebuild. I've eventually managed to assemble qtwebengine (5.15.3 branch) and the submodules (chromium, gn, ninja) from that date.

I cannot match gentoo's tarball - it looks as if they have cherry-picked items. In particular, they still apply patches for ICU68 but in my chromium source the changes appear to have all been made.

In the qtwebengine chromium tree the latest commits fixed CVE-2021-21149 to -21156 (all rated High by chromium, see ​https://chromereleases.googleblog.com/2021/02/stable-channel-update-for-desktop_16.html).

[ken@…]

Removed my earlier comments, will add the process I used once I've got the (expletive deleted) source to build. I think some of it needs patching for glibc-2.33, my test build on LFS-10.1 last night didn't get very far (backtrace in python) but on 10.0 it is about half-way through.

[ken@…]

Failed to build my own version, taking a look at gentoo. I was wrong about the ICU68 patches, they only patch a couple of v8 files. My source does not have those changes, nor does our ICU68-2 patch. This is from upstream chromium for ICU 68 migration, supposedly required (ICU 68-1 changed the output skeleton format).

[ken@…]

There are also earlier fixes, CVE-2020-16044 and CVE-2021-21118 to CVE-2021-21123, CVE-2021-21125 to CVE-2021-21127, CVE-2021-21132, CVE-2021-21135, CVE-2021-21137, CVE-2021-21140, CVE-2021-21141, CVE-2021-21145 to CVE-2021-21148.

[ken@…]

I've got a bit further (managed to build it and do an INSTALL_ROOT install on BLFS-10.0, which highlighted a few things and worked around possible remaining glibc-2.33 issues.

Although the qtwebengine 5.15.3 branch is visible online, using that gave me many differences from what gentoo were using. Twice very near the end of the build I had failures. For the first one I found that gentoo had reverted a patch from chromium. For the second I looked at git blame and found that the FTBFS was related to a merge of 5.15.3 into -dev (which is the ongoing development). There is also a 5.15 branch, using that changed the module version from 6.0+ to 5.15.3. Of course, in the absence of Qt-5.15.3 we need to build for 5.15.2.

This does not need the revert from chromium.

I'm currently calling this qtwebengine-5.15branch_20210303 to indicate where this came from.

From an existing qtwebengine git pull I've done:

git submodule update --init
git pull --recurse-submodules
git fetch origin
git branch -r # to show remote branches
git checkout -b 5.15 origin/5.15

git-archive ignores submodules, so tarring this up is
a multi-stage process:

git archive --prefix qtwebengine-5.15branch_20210303/ \
 -o /path/to/qtwebengine-only.tar

then go down to src/thirdparty. There are three modules:
chromium, gn, ninja. Go into each in turn and create an
archive (prefix chromium/ etc).

In somewhere with plenty of space (the chromium submodule
is somewhat large, it is also different from upstream chromium,
with backports and a qtwebengine directory in it) untar the
qtwebengine tarball, then cd into that and go down to 
src/submodules. Untar each submodule.

The completed source can now be tarred up with tar -cJf, it
takes some time and includes files such as .gitignore but not
any git history.

[ken@…]

FWIW, I found a reference to why qtwebengine is available (it's LGPL) in ​https://lists.qt-project.org/pipermail/interest/2021-March/036402.html

[ken@…]

I still can't build this on LFS-10.1, latest details of what I used, and how it continues to fail with a python error in catapult, are in the thread at ​http://lists.linuxfromscratch.org/pipermail/blfs-dev/2021-March/037997.html. My last attempt used what I think Arch are using, but the result was no different.

Oddly, all the variants I have tried built ok on LFS-10.0.

Adding Caution paragraphs to qtwebengine and falkon, and will raise a Security Advisory warning people about the unresolved vulnerabilities.

Giving the ticket back because I cannot fix it.

[Douglas R. Reno]

Hi Ken,

I was looking at Arch and I noticed that they now have 5.15.3: ​https://github.com/archlinux/svntogit-packages/blob/packages/qt5-webengine/trunk/PKGBUILD

Looking at the PKGBUILD file, they got their files by doing the following:

  mkdir -p build

  cd ${_pkgfqn}
  git submodule init
  git submodule set-url src/3rdparty "$srcdir"/qtwebengine-chromium
  git submodule set-branch --branch 87-based src/3rdparty
  git submodule update

  patch -p1 -i "$srcdir"/qt5-webengine-glibc-2.33.patch # Fix text rendering when building with glibc 2.33

[ken@…]

Replying to renodr:

Hi Ken,

I was looking at Arch and I noticed that they now have 5.15.3: ​https://github.com/archlinux/svntogit-packages/blob/packages/qt5-webengine/trunk/PKGBUILD

Looking at the PKGBUILD file, they got their files by doing the following:

  mkdir -p build

  cd ${_pkgfqn}
  git submodule init
  git submodule set-url src/3rdparty "$srcdir"/qtwebengine-chromium
  git submodule set-branch --branch 87-based src/3rdparty
  git submodule update

  patch -p1 -i "$srcdir"/qt5-webengine-glibc-2.33.patch # Fix text rendering when building with glibc 2.33

Yes, thanks. I eventually looked at that, and the 87-based chromium is essential for some of the fixes. But it did not solve my FTBFS. I'm now using that version on the old 10.0 system (after updating node.js - and that *is* now required). Will be trying to use gdb.

[ken@…]

I've now tried to get exactly what Arch are using:

git clone git://code.qt.io/qt/qtwebengine.git
git submodule init
git submodule set-branch --branch 87-based src/3rdparty
git submodule update
git fetch origin
git branch -r to list
git checkout origin/5.15.3

That set HEAD to what Arch are using: a059e740 Update Chromium

Tried to tar up the whole thing, with tar, but with all the .git files the source is over 4GB. So, used git archive on qtwebengine itself and each of the submodules (it seems to get all three), then tarred that up as qtwebengine-5.15.3.tar.xz (306MB).

But build still fails like before:

 [2892/23843] /usr/bin/python2 ../../../../src/3rdparty/chromium/third_party/catapult/tracing/bin/generate_about_tracing_contents --outdir gen/content/browser/tracing
FAILED: gen/content/browser/tracing/about_tracing.js gen/content/browser/tracing/about_tracing.html 
/usr/bin/python2 ../../../../src/3rdparty/chromium/third_party/catapult/tracing/bin/generate_about_tracing_contents --outdir gen/content/browser/tracing
Traceback (most recent call last):
  File "../../../../src/3rdparty/chromium/third_party/catapult/tracing/bin/generate_about_tracing_contents", line 14, in <module>
    sys.exit(generate_about_tracing_contents.Main(sys.argv[1:]))
  File "/scratch/working/qtwebengine-5.15.3/src/3rdparty/chromium/third_party/catapult/tracing/tracing_build/generate_about_tracing_contents.py", line 46, in Main
    minify=not args.no_min)
  File "/scratch/working/qtwebengine-5.15.3/src/3rdparty/chromium/third_party/catapult/common/py_vulcanize/py_vulcanize/generate.py", line 287, in GenerateStandaloneHTMLToFile
    module.AppendHTMLContentsToFile(output_file, controller, minify=minify)
  File "/scratch/working/qtwebengine-5.15.3/src/3rdparty/chromium/third_party/catapult/common/py_vulcanize/py_vulcanize/html_module.py", line 72, in AppendHTMLContentsToFile
    for piece in self._parser_results.YieldHTMLInPieces(ctl, minify=minify):
  File "/scratch/working/qtwebengine-5.15.3/src/3rdparty/chromium/third_party/catapult/common/py_vulcanize/py_vulcanize/parse_html_deps.py", line 198, in YieldHTMLInPieces
    yield self.GenerateHTML(controller, minify)
  File "/scratch/working/qtwebengine-5.15.3/src/3rdparty/chromium/third_party/catapult/common/py_vulcanize/py_vulcanize/parse_html_deps.py", line 242, in GenerateHTML
    html = controller.GetHTMLForStylesheetHRef(stylesheet_link['href'])
  File "/scratch/working/qtwebengine-5.15.3/src/3rdparty/chromium/third_party/catapult/common/py_vulcanize/py_vulcanize/generate.py", line 282, in GetHTMLForStylesheetHRef
    text = _MinifyCSS(text)
  File "/scratch/working/qtwebengine-5.15.3/src/3rdparty/chromium/third_party/catapult/common/py_vulcanize/py_vulcanize/generate.py", line 210, in _MinifyCSS
    stderr=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I have given up trying to debug that error, wherever I try to run the code from it completely fails to find the in-tree modules (tried strace, it looks around then starts looking in /usr/lib/python2.7) and ends with the same error - but I'm not convinced that the real build is failing in the same way.

[Douglas R. Reno]

Ken, can you make sure that you have six installed? I just made a tarball and got past there, but only after I had 'six' installed for python2

[Douglas R. Reno]

Scratch that, it failed later on. Looking into it a bit more

[Douglas R. Reno]

I found Gentoo's tarball here: ​https://dev.gentoo.org/~asturm/distfiles/qtwebengine-5.15.2_p20210224.tar.xz

Going to try with theirs real quick and see what we get.

[Douglas R. Reno]

Hi Ken,

I think I figured out a solution for that particular problem. Now we have ICU issues to deal with though

Edit src/3rdparty/chromium/third_party/catapult/tracing/tracing_build/generate_about_tracing_contents.py and set default=True to default=False in parser.add_argument('--no-min').

The actual problem is a bit more complex. It's calling on 'python' rather than 'python2' to execute a file in src/3rdparty/chromium/third_party/catapult/common/py_vulcanize/py_vulcanize/generate.py called rcssmin, and the arguments that it wants don't really work well. I got an Exception out of it, which was different than a "File not found error", by manually changing the path in the line "rccmin_args = ['python', rcssmin_path]" to "../third_party/rcssmin/rcssmin.py" instead of "rcssmin_path". That gives me an exception because it's trying to process /* comments as CSS, which doesn't work. We can bypass this problem entirely by just changing "default=False" to "default=True" as described above.

about:tracing (about_tracing.html) isn't used by QtWebEngine, so we're fine to just ignore this issue by telling it not to try minimizing the CSS on the page.

For future reference, the [Errno 2] No such file or directory was for rcssmin.py stored in src/3rdparty/chromium/third_party/catapult/common/py_vulcanice/third_party/rcssmin

When you upload your tarball, can you merge in the ICU patch, ​https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-qt/qtwebengine/files/qtwebengine-5.15.2_p20210224-disable-git.patch, and ​https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-qt/qtwebengine/files/qtwebengine-5.15.2_p20210224-fix-crash-w-app-locales.patch?

There is a newer version of the ICU patch at ​https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-qt/qtwebengine/files/qtwebengine-5.15.2_p20210224-chromium-87-v8-icu68.patch but I'm not sure how much different that it is from ours. I know that ours has some fixes for seccomp sandboxes rolled in too I think.

[Douglas R. Reno]

Also, /usr/bin/python was a symlink to python2 in 10.0, so that could've been related as well

[ken@…]

Not yet completing its build, but getting closer.

[ken@…]

Still failing. What confuses me is why neither gentoo (with their 5.15.2_p20210224) nor Arch (their original build should match the tarball I've uploaded to anduin) are seeing this. Obviously needs the (updated) glibc-2.33 patch at *runtime* (use from Arch), but I don't seem to be any closer to completing the compilation.

Will also benefit from gentoo's qtwebengine-5.15.2_p20210224-fix-crash-w-app-locales.patch at runtime if you use one of the affected locales (Arch cherry pick that in their build as of Friday).

[ken@…]

A (lack of) progress report. Using a python -> python2 symlink with gentoo's tarball, their fix-crash-w-app-locales.patch and Arch's glibc-2.33 patch (both of the latter are runtime fixes), and using a sed instead of gentoo's patch to make git not required, I'm still hitting

/scratch/ken/testbuild/LFS-10.1/qtwebengine-5.15.2_p20210224/src/core/net/qrc_url_scheme_handler.h:43:10: fatal error: QtWebEngineCore/private/qtwebenginecoreglobal_p.h: No such file or directory
   43 | #include <QtWebEngineCore/private/qtwebenginecoreglobal_p.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Note that I have not created the .git directories which were in gentoo's ebuild when I last looked, nor their 5.15.2 icu runtime fix patch (we didn't apparently need that for 5.15.2).

Arch, of course use git (if I read their PKGBUILD correctly they force people to each clone before building although the details of what exact commands their process uses escape me).

[ken@…]

Successful manual INSTALL_ROOT (i.e. DESTDIR) build on one machine, now trying a real install with updated patch and my script.

Tarball: last commit from the 5.15 branch before the version was moved on to 5.15.4: [ ​https://code.qt.io/cgit/qt/qtwebengine.git/commit/?h=5.15&id=029771bcd254b6cf873badd538e3f762d6344f6b] and chromium submodules from the (qtwebengine) 87-branch at 7c8217b36a95.

Patch: Amalgamate gentoo qtwebengine-5.15.2_p20210224-fix-crash-w-app-locales.patch (from upstream after 5.15.4 was opened), gentoo qtwebengine-5.15.2_p20210224-disable-git.patch, arch qt5-webengine-glibc-2.33.patch. The glibc-2.33 patch had to be fixed up for changes in this version.

Additional dependencies: /usr/bin/python (symlink to python2) - disgusting, but chromium is stuck on this. node-js.

Additional steps before building: Create .git and src/3rdparty/chromium.git directories (per gentoo) Sed to change the version to 5.15.2 to match Qt.

The build output was slightly odd - after the initial 178 targets for gn, there were 23844 reported targets, but then at the end it found another 716 targets in Qpdf. Also, my own CFLAGS (passed to CXXFLAGS based on previous Qt5) for things like -march=native and some hardening were mostly ignored until late in the build when some of the chromium files used them.

I hesitate to say this, but after 18 attempts or more, I think I'm getting there.

[ken@…]

On two AMD machines this is working, and falkon is working. But on my intel haswell starting falkon from a term reports 'Trace/breakpoint trap'.

[ken@…]

Note that a debug build of webengine is huge (20GB build, and in excess of 4.5GB installed (primarily libQt5WebEngineCore.so.5.15.2). I already had a "by the book" (no added CFLAGS) build of Qt-5.15.2 in a different prefix, ready for measuring, but I ended up having to move everything to a different, large, partition to get enough room.

The debug build of webengine is achieved by adding ' -config debug' before '--' in the qmake command. Also, rename the usual /opt/qt5, fix up /etc/ld.so.conf and run ldconfig. After webengine, rebuild falkon to use the debug version of webengine.

Trying to run falkon now gave me a backtrace, it failed in sandbox_linux.cc, called from zygote_host_impl_linux.cc. That caller talks about fun and games with namespaces and the PID namespace.

Now, one of my earlier attempts about a week ago involved enabling user namespaces in the kernel config on this machine (they are normally disabled). So I looked at what options I had on the three machines. One was using all the current namespace defaults (UTS, TIME, IPC, PID, NET all enabled, USER not enabled), another had enabled TIME, and this problematic box had enabled TIME and USER.

So, despite the comments in the zygote code about the PID namespace, that is not required (works on one of my machiens where that is not yet enabled).

I also manually compared the Arch chromium glibc-2.33 patch (against a version of 89, so should be new enough) and that is identical with the qtwebengine patch.

I've now built linux-5.10.24 on this machine, using the default namespace config options, and rebooted. The debug build now works.

[ken@…]

Fixed at r24382. Security advisory updated.

[ken@…]

Kernel namespace options reworked at r24390 - user namespace is ok if (at least) PID namespace is also enabled.

[Bruce Dubbs]

Milestone renamed