This chapter covers the special issues relating to non-ASCII characters and how they are stored in strings and buffers.
Emacs has two text representations---two ways to represent text in a string or buffer. These are called unibyte and multibyte. Each string, and each buffer, uses one of these two representations. For most purposes, you can ignore the issue of representations, because Emacs converts text between them as appropriate. Occasionally in Lisp programming you will need to pay attention to the difference.
In unibyte representation,
each character occupies one byte and therefore the possible
character codes range from 0 to 255. Codes 0 through 127 are ASCII
characters; the codes from 128 through 255 are used for one
non-ASCII character set (you can choose which character set by
setting the variable nonascii-insert-offset).
In multibyte representation, a character may occupy more than one byte, and as a result, the full range of Emacs character codes can be stored. The first byte of a multibyte character is always in the range 128 through 159 (octal 0200 through 0237). These values are called leading codes. The second and subsequent bytes of a multibyte character are always in the range 160 through 255 (octal 0240 through 0377); these values are trailing codes.
In a buffer, the buffer-local value of the variable
enable-multibyte-characters specifies the
representation used. The representation for a string is determined
based on the string contents when the string is constructed.
nil, the buffer
contains multibyte text; otherwise, it contains unibyte text.
You cannot set this variable directly; instead, use the function
set-buffer-multibyte to change a buffer's
representation.
(default-value 'enable-multibyte-characters), and
setting this variable changes that default value. Setting the local
binding of enable-multibyte-characters in a specific
buffer is not allowed, but changing the default value is supported,
and it is a reasonable thing to do, because it has no effect on
existing buffers. The `--unibyte' command line option does its job by
setting the default value to nil early in startup.
t if string
contains multibyte characters.
Emacs can convert unibyte text to multibyte; it can also convert multibyte text to unibyte, though this conversion loses information. In general these conversions happen when inserting text into a buffer, or when putting text from several strings together in one string. You can also explicitly convert a string's contents to either representation.
Emacs chooses the representation for a string based on the text that it is constructed from. The general rule is to convert unibyte text to multibyte text when combining it with other multibyte text, because the multibyte representation is more general and can hold whatever characters the unibyte text has.
When inserting text into a buffer, Emacs converts the text to
the buffer's representation, as specified by
enable-multibyte-characters in that buffer. In
particular, when you insert multibyte text into a unibyte buffer,
Emacs converts the text to unibyte, even though this conversion
cannot in general preserve all the characters that might be in the
multibyte text. The other natural alternative, to convert the
buffer contents to multibyte, is not acceptable because the
buffer's representation is a choice made by the user that cannot be
overridden automatically.
Converting unibyte text to multibyte text leaves ASCII
characters unchanged, and likewise 128 through 159. It converts the
non-ASCII codes 160 through 255 by adding the value
nonascii-insert-offset to each character code. By
setting this variable, you specify which character set the unibyte
characters correspond to (see section Character Sets). For example, if
nonascii-insert-offset is 2048, which is (-
(make-char 'latin-iso8859-1) 128), then the unibyte
non-ASCII characters correspond to Latin 1. If it is 2688, which is
(- (make-char 'greek-iso8859-7) 128), then they
correspond to Greek letters.
Converting multibyte text to unibyte is simpler: it performs
logical-and of each character code with 255. If
nonascii-insert-offset has a reasonable value,
corresponding to the beginning of some character set, this
conversion is the inverse of the other: converting unibyte text to
multibyte and back to unibyte reproduces the original unibyte
text.
self-insert-command inserts a
character in the unibyte non-ASCII range, 128 through 255. However,
the function insert-char does not perform this
conversion. The right value to use to select character set cs is
(- (make-char cs) 128). If the value of
nonascii-insert-offset is zero, then conversion
actually uses the value for the Latin 1 character set, rather than
zero.
nonascii-insert-offset. You can use it
to specify independently how to translate each code in the range of
128 through 255 into a multibyte character. The value should be a
vector, or nil. If this is non-nil, it
overrides nonascii-insert-offset.
Sometimes it is useful to examine an existing buffer or string as multibyte when it was unibyte, or vice versa.
nil, the buffer
becomes multibyte. If multibyte is nil, the
buffer becomes unibyte. This function leaves the buffer contents unchanged when viewed as a sequence of bytes. As a consequence, it can change the contents viewed as characters; a sequence of two bytes which is treated as one character in multibyte representation will count as two characters in unibyte representation.
This function sets enable-multibyte-characters to
record which representation is in use. It also adjusts various data
in the buffer (including overlays, text properties and markers) so
that they cover the same text as they did before.
If string is unibyte already, then the value is string itself.
If string is multibyte already, then the value is string itself.
The unibyte and multibyte text representations use different character codes. The valid character codes for unibyte representation range from 0 to 255--the values that can fit in one byte. The valid character codes for multibyte representation range from 0 to 524287, but not all values in that range are valid. In particular, the values 128 through 255 are not legitimate in multibyte text (though they can occur in "raw bytes"; see section Explicit Encoding and Decoding). Only the ASCII codes 0 through 127 are fully legitimate in both representations.
t
if charcode is valid for either one of the two text
representations.
(char-valid-p 65)
=> t
(char-valid-p 256)
=> nil
(char-valid-p 2248)
=> t
Emacs classifies characters into various character sets, each of which has a name which is a symbol. Each character belongs to one and only one character set.
In general, there is one character set for each distinct script.
For example, latin-iso8859-1 is one character set,
greek-iso8859-7 is another, and ascii is
another. An Emacs character set can hold at most 9025 characters;
therefore, in some cases, characters that would logically be
grouped together are split into several character sets. For
example, one set of Chinese characters, generally known as Big 5,
is divided into two Emacs character sets,
chinese-big5-1 and chinese-big5-2.
t if object is a
character set name symbol, nil otherwise.
In multibyte representation, each character occupies one or more bytes. Each character set has an introduction sequence, which is normally one or two bytes long. (Exception: the ASCII character set has a zero-length introduction sequence.) The introduction sequence is the beginning of the byte sequence for any character in the character set. The rest of the character's bytes distinguish it from the other characters in the same character set. Depending on the character set, there are either one or two distinguishing bytes; the number of such bytes is called the dimension of the character set.
This is the simplest way to determine the byte length of a character set's introduction sequence:
(- (char-bytes (make-char charset)) (charset-dimension charset))
The functions in this section convert between characters and the byte values used to represent them. For most purposes, there is no need to be concerned with the sequence of bytes used to represent a character, because Emacs translates automatically when necessary.
(char-bytes 2248)
=> 2
(char-bytes 65)
=> 1
(char-bytes 192)
=> 1
The reason this function can give correct results for both multibyte and unibyte representations is that the non-ASCII character codes used in those two representations do not overlap.
(split-char 2248)
=> (latin-iso8859-1 72)
(split-char 65)
=> (ascii 65)
Unibyte non-ASCII characters are considered as part of the
ascii character set:
(split-char 192)
=> (ascii 192)
split-char. Normally, you should specify either one or
two byte-values, according to the dimension of
charset. For example,
(make-char 'latin-iso8859-1 72)
=> 2248
If you call
make-char with no byte-values, the result
is a generic character which stands for
charset. A generic character is an integer, but it is
not valid for insertion in the buffer as a character. It
can be used in char-table-range to refer to the whole
character set (see section Char-Tables). char-valid-p
returns nil for generic characters. For example:
(make-char 'latin-iso8859-1)
=> 2176
(char-valid-p 2176)
=> nil
(split-char 2176)
=> (latin-iso8859-1 0)
Sometimes it is useful to find out which character sets appear in a part of a buffer or a string. One use for this is in determining which coding systems (see section Coding Systems) are capable of representing all of the text in question.
The optional argument translation specifies a
translation table to be used in scanning the text (see section Translation of Characters). If it is
non-nil, then each character in the region is
translated through this table, and the value returned describes the
translated characters instead of the characters actually in the
buffer.
The optional argument translation specifies a
translation table; see find-charset-region, above.
A translation table specifies a mapping of characters into characters. These tables are used in encoding and decoding, and for other purposes. Some coding systems specify their own particular translation tables; there are also default translation tables which apply to all other coding systems.
(from . to);
this says to translate the character from into
to. You can also map one whole character set into another character set with the same dimension. To do this, you specify a generic character (which designates a character set) for from (see section Splitting Characters). In this case, to should also be a generic character, for another character set of the same dimension. Then the translation table translates each character of from's character set into the corresponding character of to's character set.
In decoding, the translation table's translations are applied to
the characters that result from ordinary decoding. If a coding
system has property
character-translation-table-for-decode, that specifies
the translation table to use. Otherwise, if
standard-character-translation-table-for-decode is
non-nil, decoding uses that table.
In encoding, the translation table's translations are applied to
the characters in the buffer, and the result of translation is
actually encoded. If a coding system has property
character-translation-table-for-encode, that specifies
the translation table to use. Otherwise the variable
standard-character-translation-table-for-encode
specifies the translation table.
When Emacs reads or writes a file, and when Emacs sends text to a subprocess or receives text from a subprocess, it normally performs character code conversion and end-of-line conversion as specified by a particular coding system.
Character code conversion involves conversion between the encoding used inside Emacs and some other encoding. Emacs supports many different encodings, in that it can convert to and from them. For example, it can convert text to or from encodings such as Latin 1, Latin 2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022. In some cases, Emacs supports several alternative encodings for the same characters; for example, there are three coding systems for the Cyrillic (Russian) alphabet: ISO, Alternativnyj, and KOI8.
Most coding systems specify a particular character code for conversion, but some of them leave this unspecified--to be chosen heuristically based on the data.
End of line conversion handles three different conventions used on various systems for representing end of line in files. The Unix convention is to use the linefeed character (also called newline). The DOS convention is to use the two character sequence, carriage-return linefeed, at the end of a line. The Mac convention is to use just carriage-return.
Base coding systems such as
latin-1 leave the end-of-line conversion unspecified,
to be chosen based on the data. Variant coding systems
such as latin-1-unix, latin-1-dos and
latin-1-mac specify the end-of-line conversion
explicitly as well. Most base coding systems have three
corresponding variants whose names are formed by adding
`-unix', `-dos' and
`-mac'.
The coding system raw-text is special in that it
prevents character code conversion, and causes the buffer visited
with that coding system to be a unibyte buffer. It does not specify
the end-of-line conversion, allowing that to be determined as usual
by the data, and has the usual three variants which specify the
end-of-line conversion. no-conversion is equivalent to
raw-text-unix: it specifies no conversion of either
character codes or end-of-line.
The coding system emacs-mule specifies that the
data is represented in the internal Emacs encoding. This is like
raw-text in that no code conversion happens, but
different in that the result is multibyte data.
mime-charset. That property's value is the
name used in MIME for the character coding which this coding system
can read and write. Examples:
(coding-system-get 'iso-latin-1 'mime-charset)
=> iso-8859-1
(coding-system-get 'iso-2022-cn 'mime-charset)
=> iso-2022-cn
(coding-system-get 'cyrillic-koi8 'mime-charset)
=> koi8-r
The value of the mime-charset property is also
defined as an alias for the coding system.
The principal purpose of coding systems is for use in reading
and writing files. The function insert-file-contents
uses a coding system for decoding the file data, and
write-region uses one to encode the buffer
contents.
You can specify the coding system to use either explicitly (see
section Specifying a Coding System for
One Operation), or implicitly using the defaulting mechanism
(see section Default Coding
Systems). But these methods may not completely specify what to
do. For example, they may choose a coding system such as
undefined which leaves the character code conversion
to be determined from the data. In these cases, the I/O operation
finishes the job of choosing a coding system. Very often you will
want to find out afterwards which coding system was chosen.
write-region. When those operations ask the user to
specify a different coding system,
buffer-file-coding-system is updated to the coding
system specified.
write-region. When saving the buffer asks the user to
specify a different coding system, and
save-buffer-coding-system was used, then it is updated
to the coding system that was specified.
Warning: Since receiving subprocess output sets this variable, it can change whenever Emacs waits; therefore, you should use copy the value shortly after the function call which stores the value you are interested in.
The variable selection-coding-system specifies how
to encode selections for the window system. See section Window System Selections.
Here are Lisp facilities for working with coding systems;
nil, the value includes only the base coding
systems. Otherwise, it includes variant coding systems as
well.
t if
object is a coding system name.
coding-system-error.
eol-type. eol-type should
be unix, dos, mac, or
nil. If it is nil, the returned coding
system determines the end-of-line conversion from the data.
nil, it returns
undecided, or one of its variants according to
eol-coding.
If the text contains no multibyte characters, the function
returns the list (undecided).
(undecided).
Normally this function returns a list of coding systems that
could handle decoding the text that was scanned. They are listed in
order of decreasing priority. But if highest is
non-nil, then the return value is just one coding
system, the one that is highest in priority.
If the region contains only ASCII characters, the value is
undecided or (undecided).
detect-coding-region except that it operates on the
contents of string instead of bytes in the buffer.
See section Process Information, for how to examine or set the coding systems used for I/O to a subprocess.
The optional argument preferred-coding-system
specifies a coding system to try first. If that one can handle the
text in the specified region, then it is used. If this argument is
omitted, the current buffer's value of
buffer-file-coding-system is tried first.
If the region contains some multibyte characters that the preferred coding system cannot encode, this function asks the user to choose from a list of coding systems which can encode the text, and returns the user's choice.
One other kludgy feature: if from is a string, the string is the target text, and to is ignored.
Here are two functions you can use to let the user specify a coding system, with completion. See section Completion.
This section describes variables that specify the default coding system for certain files or when running certain subprograms, and the function that I/O operations use to access them.
The idea of these variables is that you set them once and for
all to the defaults you want, and then do not change them again. To
specify a particular coding system for a particular operation in a
Lisp program, don't change these variables; instead, override them
using coding-system-for-read and
coding-system-for-write (see section Specifying a Coding System for One
Operation).
(pattern .
coding), where pattern is a regular
expression that matches certain file names. The element applies to
file names that match pattern. The CDR of the element, coding, should be either a coding system, a cons cell containing two coding systems, or a function symbol. If val is a coding system, that coding system is used for both reading the file and writing it. If val is a cons cell containing two coding systems, its CAR specifies the coding system for decoding, and its CDR specifies the coding system for encoding.
If val is a function symbol, the function must return a coding system or a cons cell containing two coding systems. This value is used as described above.
file-coding-system-alist, except that
pattern is matched against the program name used to
start the subprocess. The coding system or systems specified in
this alist are used to initialize the coding systems used for I/O
to the subprocess, but you can specify other coding systems later
using set-process-coding-system.
Warning: Coding systems such as
undecided which determine the coding system from the
data do not work entirely reliably with asynchronous subprocess
output. This is because Emacs handles asynchronous subprocess
output in batches, as it arrives. If the coding system leaves the
character code conversion unspecified, or leaves the end-of-line
conversion unspecified, Emacs must try to detect the proper
conversion from one batch at a time, and this does not always
work.
Therefore, with an asynchronous subprocess, if at all possible,
use a coding system which determines both the character code
conversion and the end of line conversion--that is, one like
latin-1-unix, rather than undecided or
latin-1.
file-coding-system-alist, with the difference that the
pattern in an element may be either a port number or a
regular expression. If it is a regular expression, it is matched
against the network service name used to open the network
stream.
The value should be a cons cell of the form
(input-coding . output-coding).
Here input-coding applies to input from the subprocess,
and output-coding applies to output to it.
(decoding-system encoding-system)
The first element, decoding-system, is the coding system to use for decoding (in case operation does decoding), and encoding-system is the coding system for encoding (in case operation does encoding).
The argument operation should be an Emacs I/O
primitive: insert-file-contents,
write-region, call-process,
call-process-region, start-process, or
open-network-stream.
The remaining arguments should be the same arguments that might
be given to that I/O primitive. Depending on which primitive, one
of those arguments is selected as the target. For example,
if operation does file I/O, whichever argument specifies
the file name is the target. For subprocess primitives, the process
name is the target. For open-network-stream, the
target is the service name or port number.
This function looks up the target in
file-coding-system-alist,
process-coding-system-alist, or
network-coding-system-alist, depending on
operation. See section Default Coding Systems.
You can specify the coding system for a specific operation by
binding the variables coding-system-for-read and/or
coding-system-for-write.
nil, it
specifies the coding system to use for reading a file, or for input
from a synchronous subprocess. It also applies to any asynchronous subprocess or network
stream, but in a different way: the value of
coding-system-for-read when you start the subprocess
or open the network stream specifies the input decoding method for
that subprocess or network stream. It remains in use for that
subprocess or network stream unless and until overridden.
The right way to use this variable is to bind it with
let for a specific I/O operation. Its global value is
normally nil, and you should not globally set it to
any other value. Here is an example of the right way to use the
variable:
;; Read the file with no character code conversion. ;; Assume CRLF represents end-of-line. (let ((coding-system-for-write 'emacs-mule-dos)) (insert-file-contents filename))
When its value is non-nil,
coding-system-for-read takes precedence over all other
methods of specifying a coding system to use for input, including
file-coding-system-alist,
process-coding-system-alist and
network-coding-system-alist.
coding-system-for-read, except that it applies to
output rather than input. It affects writing to files,
subprocesses, and net connections. When a single operation does both input and output, as do
call-process-region and start-process,
both coding-system-for-read and
coding-system-for-write affect it.
nil, no
end-of-line conversion is done, no matter which coding system is
specified. This applies to all the Emacs I/O and subprocess
primitives, and to the explicit encoding and decoding functions
(see section Explicit Encoding and
Decoding).
All the operations that transfer text in and out of Emacs have the ability to use a coding system to encode or decode the text. You can also explicitly encode and decode text using the functions in this section.
The result of encoding, and
the input to decoding, are not ordinary text. They are "raw
bytes"---bytes that represent text in the same way that an external
file would. When a buffer contains raw bytes, it is most natural to
mark that buffer as using unibyte representation, using
set-buffer-multibyte (see section Selecting a Representation), but this
is not required. If the buffer's contents are only temporarily raw,
leave the buffer multibyte, which will be correct after you decode
them.
The usual way to get raw bytes in a buffer, for explicit
decoding, is to read them from a file with
insert-file-contents-literally (see section Reading from Files) or specify a
non-nil rawfile argument when visiting a
file with find-file-noselect.
The usual way to use the raw bytes that result from explicitly
encoding text is to copy them to a file or process--for example, to
write them with write-region (see section Writing to Files), and suppress
encoding for that write-region call by binding
coding-system-for-write to
no-conversion.
Raw bytes sometimes contain overlong byte-sequences that look like a proper multibyte character plus extra bytes containing trailing codes. For most purposes, Emacs treats such a sequence in a buffer or string as a single character, and if you look at its character code, you get the value that corresponds to the multibyte character sequence--the extra bytes are disregarded. This behavior is not quite clean, but raw bytes are used only in limited places in Emacs, so as a practical matter problems can be avoided.
Emacs can decode keyboard input using a coding system, and
encode terminal output. This is useful for terminals that transmit
or display text using a particular encoding such as Latin-1. Emacs
does not set last-coding-system-used for encoding or
decoding for the terminal.
nil if no
coding system is to be used.
nil,
that means do not decode keyboard input.
nil for no
encoding.
nil,
that means do not encode terminal output.
Emacs on MS-DOS and on MS-Windows recognizes certain file names as text files or binary files. By "binary file" we mean a file of literal byte values that are not necessary meant to be characters. Emacs does no end-of-line conversion and no character code conversion for a binary file. Meanwhile, when you create a new file which is marked by its name as a "text file", Emacs uses DOS end-of-line conversion.
buffer-file-coding-system, this variable
is used to determine which coding system to use when writing the
contents of the buffer. It should be nil for text,
t for binary. If it is t, the coding
system is no-conversion. Otherwise,
undecided-dos is used. Normally this variable is set by visiting a file; it is set to
nil if the file was visited without any actual
conversion.
nil for text, t for binary, or a function
to call to compute which. If it is a function, then it is called
with a single argument (the file name) and should return
t or nil. Emacs when running on MS-DOS or MS-Windows checks this alist to
decide which coding system to use when reading a file. For a text
file, undecided-dos is used. For a binary file,
no-conversion is used.
If no element in this alist matches a given file name, then
default-buffer-file-type says how to treat the
file.
file-name-buffer-file-type-alist says nothing about
the type. If this variable is non-nil, then these files are
treated as binary: the coding system no-conversion is
used. Otherwise, nothing special is done for them--the coding
system is deduced solely from the file contents, in the usual Emacs
fashion.
Input methods provide convenient ways of entering non-ASCII characters from the keyboard. Unlike coding systems, which translate non-ASCII characters to and from encodings meant to be read by programs, input methods provide human-friendly commands. (See section `Input Methods' in The GNU Emacs Manual, for information on how users use input methods to enter text.) How to define input methods is not yet documented in this manual, but here we describe how to use them.
Each input method has a name, which is currently a string; in the future, symbols may also be usable as input method names.
nil if no input method is active in
the buffer now.
current-input-method, this variable is normally
global.
default-input-method to
input-method. If input-method is
nil, this function deactivates any input method for
the current buffer.
nil,
that is returned by default, if the user enters empty input.
However, if inhibit-null is non-nil, empty
input signals an error. The returned value is a string.
(input-method language-env activate-func title description args...)
Here input-method is the input method name, a string; language-env is another string, the name of the language environment this input method is recommended for. (That serves only for documentation purposes.)
title is a string to display in the mode line while this method is active. description is a string describing this method and what it is good for.
activate-func is a function to call to activate this method. The args, if any, are passed as arguments to activate-func. All told, the arguments to activate-func are input-method and the args.
The fundamental interface to input methods is through the
variable input-method-function. See section Reading One Event.