#17794 closed enhancement (fixed)
thunderbird-102.9.0
| Reported by: | Bruce Dubbs | Owned by: | Douglas R. Reno |
|---|---|---|---|
| Priority: | elevated | Milestone: | 12.0 |
| Component: | BOOK | Version: | git |
| Severity: | normal | Keywords: | |
| Cc: |
Description ¶
New minor version.
Change History (20)
comment:1 by , 2 years ago
| Priority: | normal → elevated |
|---|
comment:2 by , 2 years ago
| Owner: | changed from to |
|---|---|
| Status: | new → assigned |
comment:3 by , 2 years ago
comment:4 by , 2 years ago
Thunderbird builds with the patch, but some initial testing causes an instantaneous crash. Investigating...
comment:5 by , 2 years ago
I've attempted to move my profile out of the way and have had no luck there either, and upstream doesn't seem to have any bugs.
We might be the only group building with rustc-1.68 *and* llvm-16 though. I'm going to rebuild with debugging symbols because it doesn't give me much to go with:
(gdb) bt full
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
tid = <optimized out>
ret = 0
pd = <optimized out>
old_mask = {__val = {0}}
ret = <optimized out>
#1 0x00007f7e8687c0ff in __pthread_kill_internal (signo=11, threadid=<optimized out>) at pthread_kill.c:78
#2 0x00007f7e8682e462 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
ret = <optimized out>
#3 0x00007f7e80b06b4c in () at /usr/lib/thunderbird/libxul.so
#4 0x00007f7e815ca9a7 in () at /usr/lib/thunderbird/libxul.so
#5 0x00007f7e8682e500 in <signal handler called> () at /usr/lib/libc.so.6
#6 0x00007f7e81d17fb6 in () at /usr/lib/thunderbird/libxul.so
#7 0x00007f7e81863586 in () at /usr/lib/thunderbird/libxul.so
#8 0x00007f7e81862e4c in () at /usr/lib/thunderbird/libxul.so
#9 0x0000000000000000 in ()
For console output, I get:
[ImapModuleLoader] Using nsImapService.cpp [NntpModuleLoader] Using NntpService.jsm [Pop3ModuleLoader] Using Pop3Service.jsm ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. [calBackendLoader] Using Thunderbird's ical.js backend Segmentation fault (core dumped)
I need a breadcrumb/clue of some kind to go with...
I did investigate the "ATTENTION: default value of option mesa_glthread overridden by environment." warning by running 'env' and looking through the output for anything that could be related, but I haven't been able to find anything.
comment:6 by , 2 years ago
Running a build now with --disable-debug and --disable-debug-symbols commented out.
I did circle back and verify that Firefox works as well, so I do not believe that it's my patch at fault (though mine really only impacts gecko-profiler and I don't think we're getting that far just yet)
comment:7 by , 2 years ago
Trying to build it in debug mode presents an even more curious error:
27:10.54 error: Cannot represent a difference across sections 27:32.84 error: could not compile `gkrust` due to previous error
Going to just try a build with just debugging symbols turned on next to see if that has any clues. Something seems very odd to me here though, and I wonder if it's an issue in gkrust. A good test for that would be to try with rustc-1.67.1 since we know that works, and I have that in /opt still.
comment:8 by , 2 years ago
My build with --disable-debug uncommented, but with --disable-debug-symbols commented out, is still resulting in the same error:
2:08.80 3 warnings generated. 2:09.78 3 warnings generated. 2:09.82 media/ffvpx/libavcodec/libmozavcodec.so 7:20.27 error: Cannot represent a difference across sections 7:37.44 error: could not compile `gkrust` due to previous error
I'm going to try backing down to rustc-1.67.1 real quick just to remove a potential rust bug from the list of possibilities.
comment:9 by , 2 years ago
Note though that the number one result for "Cannot represent a difference across sections" suggests that it might be a bug in LLVM/Clang. Not sure I believe that yet, but I might build a copy of LLVM-15.0.7 in /opt to see if it triggers if downgrading rust doesn't make a difference.
I'm not inclined to think that this is a thunderbird-specific bug now just because no distributions that I've looked at have anything for this, and Thunderbird's source trees (and bugzilla) don't have anything for this problem either. That being said though we'll see what happens, I might need to make a bug report somewhere...
To recap - segmentation fault when executing, and when attempting to build with debug symbols or debug mode on, the equivalent of an internal compiler error.
comment:10 by , 2 years ago
The plot thickens a bit.
With rustc-1.67.1, it not only compiles a release build properly, but also runs it.
Going to go try a debug build with rustc-1.67.1 real quick because I want to see if the two errors return. In the meantime, I'm going to go start poking around at rust...
Glad it's not LLVM, at least not from the looks of it right now. Not happy that it's rust though.
follow-up: 13 comment:12 by , 2 years ago
Restored my symlink so that I'm using rustc-1.68 again. One thing I've tried now is CC=gcc CXX=g++. It does build fine (just like clang/clang++, which is the default)... but it immediately crashes upon startup still. A debug build crashes after about 20 minutes like normal, with the faulty crate being gkrust once again (confirmed by running 'ps' while waiting on C++ code to compile. gkrust takes a really long time to build/link and I hadn't seen any rust related messages in a while, and I've seen some gkrust stuff earlier as well).
Meanwhile, I just realized that my old version of rust is using LLVM-15, while the new one is using LLVM-16, so my comment about LLVM in comment:10 is moot now.
/opt/rustc-1.67.1/lib/librustc_driver-0ba5ef7854983ef2.so:
linux-vdso.so.1 (0x00007ffcfa9a1000)
libstd-11f3850e9ce3f8b7.so => /opt/rustc-1.67.1/lib/../lib/libstd-11f3850e9ce3f8b7.so (0x00007ff906b0e000)
libLLVM-15.so => /usr/lib/libLLVM-15.so (0x00007ff8ff31c000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ff8ff0f0000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007ff8ff0cf000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007ff8feff0000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007ff8fee0f000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff90b100000)
libffi.so.8 => /usr/lib/../lib/libffi.so.8 (0x00007ff8fedfd000)
libz.so.1 => /usr/lib/../lib/libz.so.1 (0x00007ff8fedde000)
libzstd.so.1 => /usr/lib/../lib/libzstd.so.1 (0x00007ff8fecf1000)
libncursesw.so.6 => /usr/lib/../lib/libncursesw.so.6 (0x00007ff8fec7b000)
libxml2.so.2 => /usr/lib/../lib/libxml2.so.2 (0x00007ff8feb14000)
liblzma.so.5 => /usr/lib/../lib/liblzma.so.5 (0x00007ff8feae3000)
An interesting test would be to recompile rustc-1.67.1 (but into a different spot in /opt, like /opt/rustc-1.67.1-llvm16) and see if the issue surfaces on that version too. That would help isolate things a bit further, and I'd know where to bring this issue to (and I can also suggest a revert of the relevant package for now if that's what needs to happen, but it doesn't help that I can't tell which package to revert just yet).
I note that rustc-1.68.1 also came out a couple days ago, though the release isn't in Github's releases page so the currency script probably hasn't picked up on it for that reason. This doesn't seem like a panic in the compiler though, rather a miscompilation of some kind, so I'm not inclined to think that it'll fix it. Still though, it's another spot to look...
Note that the rustc-1.67.1 attempt (which used LLVM-15 for code generation from the looks of what libraries rustc's libraries are linked to) did build and execute properly... and I was even able to get a debug version to work. I won't know if this changes when rustc-1.67.1 is compiled with LLVM-16 until tomorrow though. So far though I'm inclined to believe that one of LLVM or rustc is causing a miscompilation, which is returning an internal error when compiling in debug mode, and an application crash when attempting to bring up the UI in normal mode.
Back at this in the morning, it's well after midnight now. First stop in the morning: rustc-1.67.1 in /opt/rustc-1.67.1-llvm16 w/ LLVM-16. A couple hours after I start that I'll be able to tell whether the issue is with rust, or with LLVM. I'll probably still try rustc-1.68.1 anyway to see if anything changes though, and we'll go from there.
follow-up: 15 comment:13 by , 2 years ago
Replying to Douglas R. Reno:
An interesting test would be to recompile rustc-1.67.1 (but into a different spot in /opt, like /opt/rustc-1.67.1-llvm16) and see if the issue surfaces on that version too.
Unfortunately it won't work. Rustc-1.67 will just FTBFS with system LLVM-16. If it worked I would not update rustc to 1.68 for the book (though I'll do it for my own system).
comment:14 by , 2 years ago
Not sure it is related, but firefox segfaults on the machine with Alder Lake CPU and AMD GPU, and doesn't segfaults on the Haswell with integrated graphics. Will try building thunderbird on both. EDIT: looks like I hadn't rebuilt firefox with the new tools on the Haswell. So trying now...
follow-up: 16 comment:15 by , 2 years ago
Replying to Xi Ruoyao:
Replying to Douglas R. Reno:
An interesting test would be to recompile rustc-1.67.1 (but into a different spot in /opt, like /opt/rustc-1.67.1-llvm16) and see if the issue surfaces on that version too.
Unfortunately it won't work. Rustc-1.67 will just FTBFS with system LLVM-16. If it worked I would not update rustc to 1.68 for the book (though I'll do it for my own system).
Thank you for that :)
Going to try rustc-1.68.1 in that case, though I'm not sure it's going to make much of a difference.
It would be really nasty to do this, but I might look at using shipped LLVM with rustc-1.68 if rustc-1.68.1 does not work. It's 15.0.6 according to CMakeLists.txt.
comment:16 by , 2 years ago
Replying to Douglas R. Reno:
Replying to Xi Ruoyao:
Replying to Douglas R. Reno:
An interesting test would be to recompile rustc-1.67.1 (but into a different spot in /opt, like /opt/rustc-1.67.1-llvm16) and see if the issue surfaces on that version too.
Unfortunately it won't work. Rustc-1.67 will just FTBFS with system LLVM-16. If it worked I would not update rustc to 1.68 for the book (though I'll do it for my own system).
Thank you for that :)
Going to try rustc-1.68.1 in that case, though I'm not sure it's going to make much of a difference.
It would be really nasty to do this, but I might look at using shipped LLVM with rustc-1.68 if rustc-1.68.1 does not work. It's 15.0.6 according to CMakeLists.txt.
I guess you can use rustup in a dedicated user account for trying different Rustc versions. It would be easier.
comment:17 by , 2 years ago
https://bugzilla.mozilla.org/show_bug.cgi?id=1824544 for "Thunderbird built with rustc-1.68.0 and LLVM-16 segfaults on startup"
Going to revert to LLVM-15 for now and update rust to 1.68.1 to prevent a compiler panic in some situations.
comment:18 by , 2 years ago
Here's a bit of an update on things...
Xi is going to try this with rust nightly once rust nightly finishes upgrading to LLVM.
Pierre is going to try building Thunderbird's latest from Mozilla's Mercurial repository
We're definitely thinking it's a miscompilation caused by LLVM (or another LLVM-specific bug) at this point.
Pierre is also experiencing crashes on Firefox with his Alder Lake system w/ AMDGPU, when compiled with LLVM-16 and rustc-1.68.
I reverted LLVM to LLVM-15 at 218e13ce4ba083720efc136aedb40c5fa0d9eaf2
I'm now working on updating the book to rustc-1.68.1. I've made some progress on that but don't think I'll finish tonight, though we shall see. Still need to update Mercurial and librsvg and then test the other rust consumers. I'm experiencing 5 test failures that might be related to GDB, so I'm going to dig into that a bit. Seeing how much we've had to use GDB in the past few days, I'd really like for that to work (note that I had those same test failures on 1.68.0 as well).
In the meantime, I should be able to close this ticket tomorrow and will switch gears to the cURL security update right afterwards.
Thunderbird is confirmed to be impacted by the same issue as Firefox.
2:45.28 cargo:rerun-if-changed=/usr/lib/clang/16/include/stdint.h 2:45.28 --- stderr 2:45.28 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/mozilla/cxxalloc.h:32:1: warning: replacement function 'operator new' cannot be declared 'inline' [-Winline-new-delete], err: false 2:45.28 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/mozilla/cxxalloc.h:36:1: warning: replacement function 'operator new' cannot be declared 'inline' [-Winline-new-delete], err: false 2:45.28 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/mozilla/cxxalloc.h:41:1: warning: replacement function 'operator new[]' cannot be declared 'inline' [-Winline-new-delete], err: false 2:45.28 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/mozilla/cxxalloc.h:45:1: warning: replacement function 'operator new[]' cannot be declared 'inline' [-Winline-new-delete], err: false 2:45.28 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/mozilla/cxxalloc.h:50:1: warning: replacement function 'operator delete' cannot be declared 'inline' [-Winline-new-delete], err: false 2:45.28 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/mozilla/cxxalloc.h:54:1: warning: replacement function 'operator delete' cannot be declared 'inline' [-Winline-new-delete], err: false 2:45.28 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/mozilla/cxxalloc.h:59:1: warning: replacement function 'operator delete[]' cannot be declared 'inline' [-Winline-new-delete], err: false 2:45.28 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/mozilla/cxxalloc.h:63:1: warning: replacement function 'operator delete[]' cannot be declared 'inline' [-Winline-new-delete], err: false 2:45.29 /sources/thunderbird-102.9.0/thunderbird-102.9.0/obj-x86_64-pc-linux- gnu/dist/include/js/HeapAPI.h:259:38: warning: offset of on non-standard-layout type 'TenuredChunkBase' [-Winvalid-offsetof], err: false 2:45.29 thread 'main' panicked at '"Vector_(unnamed_enum_at_/sources/thunderbird- 102_9_0/thunderbird-102_9_0/obj-x86_64-pc-linux-gnu/dist/include/mozilla /Vector_h_457_3)" is not a valid Ident', /sources/thunderbird-102.9.0/thunderbird- 102.9.0/third_party/rust/proc-macro2/src/fallback.rs:701:9 2:45.29 stack backtrace: 2:45.29 0: 0x561b7cb62de9 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb7c5ac78366d82a2 2:45.29 1: 0x561b7cb7d24f - core::fmt::write::h5f2882dd7340ff67 2:45.29 2: 0x561b7cb43fd5 - std::io::Write::write_fmt::h094f825d8b33998e 2:45.29 3: 0x561b7cb62ba5 - std::sys_common::backtrace::print::he84c8d432046661e 2:45.29 4: 0x561b7cb493ee - std::panicking::default_hook:: {{closure}}::h7d5178e766500307 2:45.29 5: 0x561b7cb490a9 - std::panicking::default_hook::h2bfae49f9b7452c0 2:45.29 6: 0x561b7cb49c21 - std::panicking::rust_panic_with_hook::h85202f59b67b7f5b 2:45.29 7: 0x561b7cb63b09 - std::panicking::begin_panic_handler:: {{closure}}::hd84b2c51c3bd13a1 2:45.29 8: 0x561b7cb62f3c - std::sys_common::backtrace::__rust_end_short_backtrace::h63cae59b53e5f3e6 2:45.29 9: 0x561b7cb497c2 - rust_begin_unwind 2:45.29 10: 0x561b7c9bf693 - core::panicking::panic_fmt::h86404400f963d4c0 2:45.29 11: 0x561b7cb24cf8 - proc_macro2::fallback::Ident::_new::ha42e2e3aba658295 2:45.29 12: 0x561b7cb26ab7 - proc_macro2::Ident::new::h2d344bc997948098 2:45.29 13: 0x561b7c9e15f4 - bindgen::ir::context::BindgenContext::rust_ident_raw::h0466e8f54c340fcf 2:45.29 14: 0x561b7c9f5c58 - <bindgen::ir::enum_ty::Enum as bindgen::codegen::CodeGenerator>::codegen::hd7a799dfcb2f5820 2:45.29 15: 0x561b7ca1d8f7 - <bindgen::ir::item::Item as bindgen::codegen::CodeGenerator>::codegen::hc2ba00e5dffeff50 2:45.29 16: 0x561b7ca2a5d0 - bindgen::codegen::CodegenResult::inner::h238bf7c0a27bea75 2:45.29 17: 0x561b7ca4dc13 - <bindgen::ir::module::Module as bindgen::codegen::CodeGenerator>::codegen::h27d91220885949f7 2:45.29 18: 0x561b7ca1d8c9 - <bindgen::ir::item::Item as bindgen::codegen::CodeGenerator>::codegen::hc2ba00e5dffeff50 2:45.29 19: 0x561b7ca2a5d0 - bindgen::codegen::CodegenResult::inner::h238bf7c0a27bea75 2:45.29 20: 0x561b7ca4dc13 - <bindgen::ir::module::Module as bindgen::codegen::CodeGenerator>::codegen::h27d91220885949f7 2:45.29 21: 0x561b7ca1d8c9 - <bindgen::ir::item::Item as bindgen::codegen::CodeGenerator>::codegen::hc2ba00e5dffeff50 2:45.29 22: 0x561b7c9e4b7a - bindgen::ir::context::BindgenContext::gen::h17321d68b9de83e8 2:45.29 23: 0x561b7ca31576 - bindgen::Builder::generate::hf0836b09f6eb0943 2:45.29 24: 0x561b7c9c27e1 - build_script_build::main::hd8cb202d23e4e729 2:45.29 25: 0x561b7c9c34a3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h430eeb6b61adfef4 2:45.29 26: 0x561b7c9c1c69 - std::rt::lang_start:: {{closure}}::hd971d611e15d6b41 2:45.29 27: 0x561b7cb48233 - std::rt::lang_start_internal::h46d236810b236b7f 2:45.29 28: 0x561b7c9c2d05 - main 2:45.29 29: 0x7f1189c4768a - __libc_start_call_main 2:45.29 at /sources/glibc-2.37/csu/../sysdeps /nptl/libc_start_call_main.h:58:16 2:45.29 30: 0x7f1189c47745 - __libc_start_main_impl 2:45.29 at /sources/glibc-2.37/csu/../csu/libc- start.c:360:3 2:45.29 31: 0x561b7c9bfb51 - _start 2:45.29 at /sources/glibc-2.37/csu/../sysdeps/x86_64 /start.S:115 2:45.29 32: 0x0 - <unknown> 2:45.29 warning: build failed, waiting for other jobs to finish...Now going to attempt using the existing patch that I made for Firefox. :)