Locale selection instructions are still incorrect

[alexander@…]

Hello,

Our instructions lead to selection of e.g. de_DE.ISO-8859-15@euro locale that Xlib doesn't like. I propose to:

1) Add installation of /usr/share/i18n/SUPPORTED file to glibc page. Suggested text:

========================== With both methods of locale installation, it is a good idea to copy the list of locales that Glibc maintainers declare as supported, for documentational purposes:

install -m644 ../glibc-2.3.4/localedata/SUPPORTED /usr/share/i18n ==========================

2) On the "The Bash Shell Startup Files" page, after the words "This results in a final locale setting of “en_GB.ISO-8859-1”, add:

========================== However, there are cases when the heuristic procedure outlined above produces wrong results. E.g., unpatched Xlib doesn't accept the de_DE.ISO-8859-15@euro locale (use de_DE@euro in this case) and the message "Locale is not supported by the X library" is printed. In such cases, try dropping the character set field or modifiers, guessing, searching in Google, or looking into the /usr/share/i18n/SUPPORTED file (the first slash-separated field). Verify yourself by running the following commands:

LC_ALL=[locale name] locale country LC_ALL=[locale name] locale language LC_ALL=[locale name] locale charmap LC_ALL=[locale name] locale int_curr_symbol LC_ALL=[locale name] locale int_prefix

This should print the country and language names, the character encoding used by the locale, the local currency and the prefix to dial before the telephone number in order to get into the country. The commands above should give no errors. ==========================

[Matthew Burgess]

If this only affects xlib, why is it not a BLFS problem/xlib upstream problem?

[alexander@…]

The patch for xlib is known, but BLFS editors didn't apply it as "insufficiently tested". See ​http://www.linuxfromscratch.org/~alexander/patches/xorg-6.8.2-locale_names-1.patch

Many upstream authors just won't do the right-for-glibc thing (although I don't know any other situations where the current LFS procedure leads to incorect results) because the nl_langinfo() function is not standartized enough. So the fact is: proper locale setting is black magic.

[alexander@…]

also note that LFS-6.1.1 targets BLFS 6.1, which doesn't contain the patch.

[alexander@…]

Xlib patch is in fact an incomplete solution, so please add the proposed text to the book.

This potentially affects all programs/libraries that don't use the nl_langinfo function for getting the character set out of the locale name.

[Matthew Burgess]

Proposed patch

[alexander@…]

The proposed patch doesn't address the problem at all. The prescribed tests are there in order to test that glibc accepts the locale. They have nothing to do with Xlib.

More constructive criticism:

1) The patch does ensure that the locale is accepted by glibc, but omits the important fact that some applications do custom parsing of locale names and thus can reject or otherwise mistreat locale names that are valid for glibc.

2) The "de_DE.ISO-8859-15@euro" locale passes all the tests added by the patch, but is still rejected by Xlib.

3) Messages that come as a result of locale rejection are not in the patch.

That's what happens if the C library doesn't understand the locale:

patrakov@home:~$ LC_ALL=fo_BA.ISO-8859-1 locale charmap locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_MESSAGES to default locale: No such file or directory locale: Cannot set LC_ALL to default locale: No such file or directory ANSI_X3.4-1968

That's what happens if the C library accepts the locale, but Xlib doesn't:

patrakov@home:~$ LC_ALL=ru_RU.KoI8R locale charmap KOI8-R patrakov@home:~$ LC_ALL=ru_RU.KoI8R xterm Warning: locale not supported by Xlib, locale set to C

[alexander@…]

As for the SUPPORTED file, it is installed, but never referenced further in the book. Either don't install it, or reference it somewhere.

[Matthew Burgess]

Proposed patch v.2

[alexander@…]

The SUPPORTED file is still installed but not used.

The localedata/SUPPORTED lists the locales officially supported by the Glibc

developers.

I'd rather not stress that fact. Leave the upper part as it is. Don't install that file. Instead of "which generates and installs every locale Glibc is aware of" write "which generates and installs every locale listed in the ../glibc-2.3.5/localedata/SUPPORTED file".

The commands above may fail if the locale you have selected is not supported

by Glibc.

"may" -> "will"

maybe: "not supported by glibc" -> "not supported by glibc or not installed"

One possible fix is to remove the charmap part of the locale specification

only as long as this doesn't change the output of the "locale charmap" and other commands given above.

(using the example on this page, for instance, en_GB.ISO-8859-1 would become

en_GB)

I'd rather use the real-world de_DE.ISO-8859-15@euro -> de_DE@euro example here. Also it would be nice to explain the reason for such failures: glibc and xlib parse locale names differently.

(other packages should produce similar, though probably not identical, messages)

or even malfunction without any error messages containing the word "locale". UTF-8 specific example, not for the book: in ru_RU.utf8 locale, the "less" pager prints <XX> instead of many non-ASCII characters, where X is a hexadecimal digit. Reason: it thinks that this is not an UTF-8 locale. Fix: use ru_RU.UTF-8 (that's what our heuristic produces).

[Matthew Burgess]

Proposed patch v.3

[alexander@…]

Still slightly wrong.

"The glibc-&glibc-version;/localedata/SUPPORTED lists the locales officially supported by the Glibc developers. Install that file...". We don't install that file, don't say that we do. Also the "official support" status means absulutely nothing, there's at least one non-working locale listed there (vi_VN.TCVN). Please revert the text before the command to its original state. The text after the command is good.

"When trying such a fix, be sure to check that Glibc supports the newly chosen locale too, using the tests outlined earlier in this section."

Basically you currently just say that with both old and new locale, the "locale charmap" command should give no errors. The actual requirement is more strict: it should print exactly the same.

The idea is to choose a locale name that is known good for glibc and matches the user's expectations about character map and currency. Then, in the case Xlib fails to recognize this locale, find something that is treated 100% equivalently by glibc, but recognized by Xlib.

Also please change:

"using a real-world example that Xlib lacks support for" -> "using a real-world locale example that Xlib doesn't recognize"

As for the modifiers, "locale -a" does show some examples that contain modifiers such as "@euro". So the task is to explain the fact that the "@" sign and everything after it form a modifier.

"If you suffer any of the symptoms shown above" -> never happens due to glibc test if the reader follows the book. But let it stay as it is, because I can't find any better wording to introduce glibc error message.

"Due to improper locale chioce, other packages may also fail to function correctly, with or without outputting any error messages" -> please put this to a separate paragraph at the bottom. Otherwise, there's no good place for me to add other failure modes in the UTF-8 book.

[Matthew Burgess]

(In reply to comment #11)

Still slightly wrong.

"The glibc-&glibc-version;/localedata/SUPPORTED lists the locales officially supported by the Glibc developers. Install that file...". We don't install that file, don't say that we do.

Whoops, when you said "Leave the upper part as it is." in comment 9, that's exactly what I did!

Also the "official support" status means absulutely nothing, there's at least one non-working locale listed there (vi_VN.TCVN). Please revert the text before the command to its original state. The text after the command is good.

Done.

"When trying such a fix, be sure to check that Glibc supports the newly chosen locale too, using the tests outlined earlier in this section."

Basically you currently just say that with both old and new locale, the "locale charmap" command should give no errors. The actual requirement is more strict: it should print exactly the same.

Well, that text currently caters for both Glibc and BLFS packages failing. If Glibc fails, you obviously *don't* want the newly chosen locale to print exactly the same for the Glibc based locale tests!

The idea is to choose a locale name that is known good for glibc and matches the user's expectations about character map and currency. Then, in the case Xlib fails to recognize this locale, find something that is treated 100% equivalently by glibc, but recognized by Xlib.

Yep, understood.

Also please change:

"using a real-world example that Xlib lacks support for" -> "using a real-world locale example that Xlib doesn't recognize"

Done.

As for the modifiers, "locale -a" does show some examples that contain modifiers such as "@euro". So the task is to explain the fact that the "@" sign and everything after it form a modifier.

Hmm, couldn't see a decent place to put this.

"If you suffer any of the symptoms shown above" -> never happens due to glibc test if the reader follows the book. But let it stay as it is, because I can't find any better wording to introduce glibc error message.

"Due to improper locale chioce, other packages may also fail to function correctly, with or without outputting any error messages" -> please put this to a separate paragraph at the bottom. Otherwise, there's no good place for me to add other failure modes in the UTF-8 book.

Well, I can't see a decent way to deal with any of these other points, so I'm going to have to request you come up with a patch yourself this time, Alexander.

I'll then review it for spelling/grammar and that it flows OK.

Regards,

Matt.

[Matthew Burgess]

Proposed patch v.4

[alexander@…]

glibc.xml part is good

If Glibc fails, you obviously *don't* want the newly chosen locale to print

exactly the same for the Glibc based locale tests!

Assuming that the reader follows the book, Glibc fails only with no_NO.ISO-8859-1 locale, and removal of the charmap doesn't help here.

Proposed text for profile.xml:

This results in a final locale setting of en_GB.ISO-8859-1.

It is important that the locale found using the heuristic above is tested prior to it being added to the Bash startup files: LC_ALL=[locale name] locale country LC_ALL=[locale name] locale language LC_ALL=[locale name] locale charmap LC_ALL=[locale name] locale int_curr_symbol LC_ALL=[locale name] locale int_prefix

This should print the country and language names, the character encoding used by the locale, the local currency and the prefix to dial before the telephone number in order to get into the country. If any of the commands above fails with the message similar to the following,

locale: Cannot set LC_something to default locale: No such file or directory

this means that your locale is either not installed in Chapter 6, or not supported by the default installation of Glibc. If this happens, you should either install the wanted locale using the "localedef" command, or consider choosing a different one. Further instructions asume that there are no such error messages from Glibc.

Some packages beyond LFS may also lack support for your chosen locale. One example is the X library (part of X Window System), which outputs the following error message:

Warning: locale not supported by Xlib, locale set to C

Sometimes it is possible to fix this by removing the charmap part of the locale specification, if that doesn't change Glibc's idea of the character map associated with this locale (check yourself by running the "locale charmap" command in both locales). E.g., one would have to change "de_DE.ISO-8859-15@euro" to "de_DE@euro" in order to get this locale recognized by Xlib.

Other packages can also function improperly (with or without error messages) if the locale name doesn't meet their expectations. In any case, investigating how other Linux distributions support your locale might provide some useful information.

[Matthew Burgess]

Proposed patch v.5

[alexander@…]

Looks good. Please commit, and I will pick that up in the UTF-8 book.