source: introduction/important/locale-issues.xml@ 5aeb97df

10.0 10.1 11.0 11.1 11.2 6.2 6.2.0 6.2.0-rc1 6.2.0-rc2 6.3 6.3-rc1 6.3-rc2 6.3-rc3 7.10 7.4 7.5 7.6 7.6-blfs 7.6-systemd 7.7 7.8 7.9 8.0 8.1 8.2 8.3 8.4 9.0 9.1 basic bdubbs/svn elogind gnome kde5-13430 kde5-14269 kde5-14686 krejzi/svn lazarus nosym perl-modules qt5new systemd-11177 systemd-13485 trunk upgradedb xry111/intltool xry111/soup3 xry111/test-20220226
Last change on this file since 5aeb97df was 5aeb97df, checked in by Alexander E. Patrakov <alexander@…>, 16 years ago

Stopped mentioning obsolete locale-related problems

git-svn-id: svn://svn.linuxfromscratch.org/BLFS/trunk/BOOK@6446 af4574ff-66df-0310-9fd7-8a98e5e911e0

  • Property mode set to 100644
File size: 13.3 KB
Line 
1<?xml version="1.0" encoding="ISO-8859-1"?>
2<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
3 "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
4 <!ENTITY % general-entities SYSTEM "../../general.ent">
5 %general-entities;
6]>
7
8<sect1 id="locale-issues" xreflabel="Locale Related Issues">
9 <?dbhtml filename="locale-issues.html"?>
10
11 <sect1info>
12 <othername>$LastChangedBy$</othername>
13 <date>$Date$</date>
14 </sect1info>
15
16 <title>Locale Related Issues</title>
17
18 <para>This page contains information about locale related problems and
19 issues. In the following paragraphs you'll find a generic overview of
20 things that can come up when configuring your system for various locales.
21 Many (but not all) existing locale-related problems can be classified
22 and fall under one of the headings below. The severity ratings below use
23 the following criteria:</para>
24
25 <itemizedlist>
26 <listitem>
27 <para>Critical: The program doesn't perform its main function.
28 The fix would be very intrusive, it's better to search for a
29 replacement.</para>
30 </listitem>
31 <listitem>
32 <para>High: Part of the functionality that the program provides
33 is not usable. If that functionality is required, it's better to
34 search for a replacement.</para>
35 </listitem>
36 <listitem>
37 <para>Low: The program works in all typical use cases, but lacks
38 some functionality normally provided by its equivalents.</para>
39 </listitem>
40 </itemizedlist>
41
42 <para>If there is a known workaround for a specific package, it will
43 appear on that package's page.</para>
44
45 <sect2 id="locale-not-valid-option"
46 xreflabel="Needed Encoding Not a Valid Option">
47
48 <title>The Needed Encoding is Not a Valid Option in the Program</title>
49
50 <para>Severity: Critical</para>
51
52 <para>Some programs require the user to specify the character encoding
53 for their input or output data and present only a limited choice of
54 encodings. This is the case for the <option>-X</option> option in
55 <xref linkend="a2ps"/> and <xref linkend="enscript"/>,
56 the <option>-input-charset</option> option in unpatched
57 <xref linkend="cdrtools"/>, and the character sets offered for display
58 in the menu of <xref linkend="links"/>. If the required encoding is not
59 in the list, the program usually becomes completely unusable. For
60 non-interactive programs, it may be possible to work around this by
61 converting the document to a supported input character set before
62 submitting to the program.</para>
63
64 <para>A solution to this type of problem is to implement the necessary
65 support for the missing encoding as a patch to the original program
66 (as done for <xref linkend="cdrtools"/> in this book), or to find a
67 replacement.</para>
68
69 </sect2>
70
71 <sect2 id="locale-assumed-encoding"
72 xreflabel="Program Assumes Encoding">
73
74 <title>The Program Assumes the Locale-Based Encoding of External
75 Documents</title>
76
77 <para>Severity: High for non-text documents, low for text
78 documents</para>
79
80 <para>Some programs, <xref linkend="nano"/> or
81 <xref linkend="joe"/> for example, assume that documents are always
82 in the encoding implied by the current locale. While this assumption
83 may be valid for the user-created documents, it is not safe for
84 external ones. When this assumption fails, non-ASCII characters are
85 displayed incorrectly, and the document may become unreadable.</para>
86
87 <para>If the external document is entirely text based, it can be
88 converted to the current locale encoding using the
89 <command>iconv</command> program.</para>
90
91 <para>For documents that are not text-based, this is not possible.
92 In fact, the assumption made in the program may be completely
93 invalid for documents where the Microsoft Windows operating system
94 has set de facto standards. An example of this problem is ID3v1 tags
95 in MP3 files (see <ulink url="&blfs-wiki;/ID3v1Coding">this page</ulink>
96 for more details). For these cases, the only solution is to find a
97 replacement program that doesn't have the issue (e.g., one that
98 will allow you to specify the assumed document encoding).</para>
99
100 <para>Among BLFS packages, this problem applies to
101 <xref linkend="nano"/>, <xref linkend="joe"/>, and all media players
102 except <xref linkend="audacious"/>.</para>
103
104 <para>Another problem in this category is when someone cannot read
105 the documents you've sent them because their operating system is
106 set up to handle character encodings differently. This can happen
107 often when the other person is using Microsoft Windows, which only
108 provides one character encoding for a given country. For example,
109 this causes problems with UTF-8 encoded TeX documents created in
110 Linux. On Windows, most applications will assume that these documents
111 have been created using the default Windows 8-bit encoding. See the
112 <ulink url="&blfs-wiki;/tetex">teTeX</ulink> Wiki page for more
113 details.</para>
114
115 <para>In extreme cases, Windows encoding compatibility issues may be
116 solved only by running Windows programs under
117 <ulink url="http://www.winehq.com/">Wine</ulink>.</para>
118
119 </sect2>
120
121 <sect2 id="locale-wrong-filename-encoding"
122 xreflabel="Wrong Filename Encoding">
123
124 <title>The Program Uses or Creates Filenames in the Wrong Encoding</title>
125
126 <para>Severity: Critical</para>
127
128 <para>The POSIX standard mandates that the filename encoding is
129 the encoding implied by the current LC_CTYPE locale category. This
130 information is well-hidden on the page which specifies the behavior
131 of <application>Tar</application> and <application>Cpio</application>
132 programs. Some programs get it wrong by default (or simply don't
133 have enough information to get it right). The result is that they
134 create filenames which are not subsequently shown correctly by
135 <command>ls</command>, or they refuse to accept filenames that
136 <command>ls</command> shows properly. For the <xref linkend="glib2"/>
137 library, the problem can be corrected by setting the
138 <envar>G_FILENAME_ENCODING</envar> environment variable to the special
139 "@locale" value. <application>Glib2</application> based programs that
140 don't respect that environment variable are buggy.</para>
141
142 <para>The <xref linkend="zip"/>, <xref linkend="unzip"/>, and
143 <xref linkend="nautilus-cd-burner"/> have this problem because
144 they hard-code the expected filename encoding.
145 <application>UnZip</application> contains a hard-coded conversion
146 table between the CP850 (DOS) and ISO-8859-1 (UNIX) encodings and
147 uses this table when extracting archives created under DOS or
148 Microsoft Windows. However, this assumption only works for those
149 in the US and not for anyone using a UTF-8 locale. Non-ASCII
150 characters will be mangled in the extracted filenames.</para>
151
152 <para>On the other hand,
153 <application>Nautilus CD Burner</application> checks names of
154 files added to its window for UTF-8 validity. This is wrong for
155 users of non-UTF-8 locales. Also,
156 <application>Nautilus CD Burner</application> unconditionally
157 calls <command>mkisofs</command> with the
158 <parameter>-input-charset UTF-8</parameter> parameter, which is
159 only correct in UTF-8 locales.</para>
160
161 <para>The general rule for avoiding this class of problems is to
162 avoid installing broken programs. If this is impossible, the
163 <ulink url="http://j3e.de/linux/convmv/">convmv</ulink>
164 command-line tool can be used to fix filenames created by these
165 broken programs, or intentionally mangle the existing filenames
166 to meet the broken expectations of such programs.</para>
167
168 <para>In other cases, a similar problem is caused by importing
169 filenames from a system using a different locale with a tool that
170 is not locale-aware (e.g., <xref linkend="nfs-utils"/> or
171 <xref linkend="openssh"/>). In order to avoid mangling non-ASCII
172 characters when transferring files to a system with a different
173 locale, any of the following methods can be used:</para>
174
175 <itemizedlist>
176 <listitem>
177 <para>Transfer anyway, fix the damage with
178 <command>convmv</command>.</para>
179 </listitem>
180 <listitem>
181 <para>On the sending side, create a tar archive with the
182 <parameter>--format=posix</parameter> switch passed to
183 <command>tar</command> (this will be the default in a future
184 version of <command>tar</command>).</para>
185 </listitem>
186 <listitem>
187 <para>Mail the files as attachments. Mail clients specify the
188 encoding of attached filenames.</para>
189 </listitem>
190 <listitem>
191 <para>Write the files to a removable disk formatted with a FAT or
192 FAT32 filesystem.</para>
193 </listitem>
194 <listitem>
195 <para>Transfer the files using Samba.</para>
196 </listitem>
197 <listitem>
198 <para>Transfer the files via FTP using RFC2640-aware server
199 (this currently means only wu-ftpd, which has bad security history)
200 and client (e.g., lftp).</para>
201 </listitem>
202 </itemizedlist>
203
204 <para>The last four methods work because the filenames are automatically
205 converted from the sender's locale to UNICODE and stored or sent in this
206 form. They are then transparently converted from UNICODE to the
207 recipient's locale encoding.</para>
208
209 </sect2>
210
211 <sect2 id="locale-wrong-multibyte-characters"
212 xreflabel="Wrong Multibyte Characters">
213
214 <title>The Program Breaks Multibyte Characters or Doesn't Count
215 Character Cells Correctly</title>
216
217 <para>Severity: High or critical</para>
218
219 <para>Many programs were written in an older era where multibyte
220 locales were not common. Such programs assume that C "char" data
221 type, which is one byte, can be used to store single characters.
222 Further, they assume that any sequence of characters is a valid
223 string and that every character occupies a single character cell.
224 Such assumptions completely break in UTF-8 locales. The visible
225 manifestation is that the program truncates strings prematurely
226 (i.e., at 80 bytes instead of 80 characters). Terminal-based
227 programs don't place the cursor correctly on the screen, don't react
228 to the "Backspace" key by erasing one character, and leave junk
229 characters around when updating the screen, usually turning the
230 screen into a complete mess.</para>
231
232 <para>Fixing this kind of problems is a tedious task from a
233 programmer's point of view, like all other cases of retrofitting new
234 concepts into the old flawed design. In this case, one has to redesign
235 all data structures in order to accommodate to the fact that a complete
236 character may span a variable number of "char"s (or switch to wchar_t
237 and convert as needed). Also, for every call to the "strlen" and
238 similar functions, find out whether a number of bytes, a number of
239 characters, or the width of the string was really meant. Sometimes it
240 is faster to write a program with the same functionality from scratch.
241 </para>
242
243 <para>Among BLFS packages, this problem applies to
244 <xref linkend="ed"/>, <xref linkend="xine-ui"/> and all shells.</para>
245
246 </sect2>
247
248 <sect2 id="locale-wrong-manpage-encoding"
249 xreflabel="Incorrect Manual Page Encoding">
250
251 <title>The Package Installs Manual Pages in Incorrect or
252 Non-Displayable Encoding</title>
253
254 <para>Severity: Low</para>
255
256 <para>LFS expects that manual pages are in the language-specific (usually
257 8-bit) encoding, as specified on <ulink
258 url="&lfs-root;/chapter06/man-db.html"/>. However, some packages install
259 translated manual pages in UTF-8 encoding (e.g., Shadow, already dealt
260 with), or manual pages in languages not in the table. Not all BLFS packages
261 have been audited for conformance with the requirements put in LFS (the
262 large majority have been checked, and fixes placed in the book for packages
263 known to install non-conforming manual pages). If you find a manual page
264 installed by any of BLFS packages that is obviously in the wrong encoding,
265 please remove or convert it as needed, and report this to BLFS team as a
266 bug.</para>
267
268 <para>You can easily check your system for any non-conforming manual pages
269 by copying the following short shell script to some accessible location,
270
271<screen><literal>#!/bin/sh
272# Begin checkman.sh
273# Usage: find /usr/share/man -type f | xargs checkman.sh
274for a in "$@"
275do
276 # echo "Checking $a..."
277 # Pure-ASCII manual page (possibly except comments) is OK
278 grep -v '.\\"' "$a" | iconv -f US-ASCII -t US-ASCII >/dev/null 2>&amp;1 &amp;&amp; continue
279 # Non-UTF-8 manual page is OK
280 iconv -f UTF-8 -t UTF-8 "$a" >/dev/null 2>&amp;1 || continue
281 # If we got here, we found UTF-8 manual page, bad.
282 echo "UTF-8 manual page: $a" >&amp;2
283done
284# End checkman.sh
285</literal></screen>
286
287 and then issuing the following command (modify the command below if the
288 <command>checkman.sh</command> script is not in your <envar>PATH</envar>
289 environment variable):</para>
290
291<screen><userinput>find /usr/share/man -type f | xargs checkman.sh</userinput></screen>
292
293 <para>Note that if you have manual pages installed in any location other
294 than <filename class='directory'>/usr/share/man</filename> (e.g.,
295 <filename class='directory'>/usr/local/share/man</filename>), you must
296 modify the above command to include this additional location.</para>
297
298 </sect2>
299
300</sect1>
Note: See TracBrowser for help on using the repository browser.