source: trunk/essentials/dev-lang/perl/pod/perlunicode.pod@ 3310

=head1 NAME

perlunicode - Unicode support in Perl

=head1 DESCRIPTION

=head2 Important Caveats

Unicode support is an extensive requirement. While Perl does not
implement the Unicode standard or the accompanying technical reports
from cover to cover, Perl does support many Unicode features.

=over 4

=item Input and Output Layers

Perl knows when a filehandle uses Perl's internal Unicode encodings
(UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with
the ":utf8" layer.  Other encodings can be converted to Perl's
encoding on input or from Perl's encoding on output by use of the
":encoding(...)"  layer.  See L<open>.

To indicate that Perl source itself is using a particular encoding,
see L<encoding>.

=item Regular Expressions

The regular expression compiler produces polymorphic opcodes.  That is,
the pattern adapts to the data and automatically switches to the Unicode
character scheme when presented with Unicode data--or instead uses
a traditional byte scheme when presented with byte data.

=item C<use utf8> still needed to enable UTF-8/UTF-EBCDIC in scripts

As a compatibility measure, the C<use utf8> pragma must be explicitly
included to enable recognition of UTF-8 in the Perl scripts themselves
(in string or regular expression literals, or in identifier names) on
ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based
machines.  B<These are the only times when an explicit C<use utf8>
is needed.>  See L<utf8>.

You can also use the C<encoding> pragma to change the default encoding
of the data in your script; see L<encoding>.

=item BOM-marked scripts and UTF-16 scripts autodetected

If a Perl script begins marked with the Unicode BOM (UTF-16LE, UTF16-BE,
or UTF-8), or if the script looks like non-BOM-marked UTF-16 of either
endianness, Perl will correctly read in the script as Unicode.
(BOMless UTF-8 cannot be effectively recognized or differentiated from
ISO 8859-1 or other eight-bit encodings.)

=item C<use encoding> needed to upgrade non-Latin-1 byte strings

By default, there is a fundamental asymmetry in Perl's unicode model:
implicit upgrading from byte strings to Unicode strings assumes that
they were encoded in I<ISO 8859-1 (Latin-1)>, but Unicode strings are
downgraded with UTF-8 encoding.  This happens because the first 256
codepoints in Unicode happens to agree with Latin-1.  

If you wish to interpret byte strings as UTF-8 instead, use the
C<encoding> pragma:

    use encoding 'utf8';

See L</"Byte and Character Semantics"> for more details.

=back

=head2 Byte and Character Semantics

Beginning with version 5.6, Perl uses logically-wide characters to
represent strings internally.

In future, Perl-level operations will be expected to work with
characters rather than bytes.

However, as an interim compatibility measure, Perl aims to
provide a safe migration path from byte semantics to character
semantics for programs.  For operations where Perl can unambiguously
decide that the input data are characters, Perl switches to
character semantics.  For operations where this determination cannot
be made without additional information from the user, Perl decides in
favor of compatibility and chooses to use byte semantics.

This behavior preserves compatibility with earlier versions of Perl,
which allowed byte semantics in Perl operations only if
none of the program's inputs were marked as being as source of Unicode
character data.  Such data may come from filehandles, from calls to
external programs, from information provided by the system (such as %ENV),
or from literals and constants in the source text.

The C<bytes> pragma will always, regardless of platform, force byte
semantics in a particular lexical scope.  See L<bytes>.

The C<utf8> pragma is primarily a compatibility device that enables
recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
Note that this pragma is only required while Perl defaults to byte
semantics; when character semantics become the default, this pragma
may become a no-op.  See L<utf8>.

Unless explicitly stated, Perl operators use character semantics
for Unicode data and byte semantics for non-Unicode data.
The decision to use character semantics is made transparently.  If
input data comes from a Unicode source--for example, if a character
encoding layer is added to a filehandle or a literal Unicode
string constant appears in a program--character semantics apply.
Otherwise, byte semantics are in effect.  The C<bytes> pragma should
be used to force byte semantics on Unicode data.

If strings operating under byte semantics and strings with Unicode
character data are concatenated, the new string will be created by
decoding the byte strings as I<ISO 8859-1 (Latin-1)>, even if the
old Unicode string used EBCDIC.  This translation is done without
regard to the system's native 8-bit encoding.  To change this for
systems with non-Latin-1 and non-EBCDIC native encodings, use the
C<encoding> pragma.  See L<encoding>.

Under character semantics, many operations that formerly operated on
bytes now operate on characters. A character in Perl is
logically just a number ranging from 0 to 2**31 or so. Larger
characters may encode into longer sequences of bytes internally, but
this internal detail is mostly hidden for Perl code.
See L<perluniintro> for more.

=head2 Effects of Character Semantics

Character semantics have the following effects:

=over 4

=item *

Strings--including hash keys--and regular expression patterns may
contain characters that have an ordinal value larger than 255.

If you use a Unicode editor to edit your program, Unicode characters
may occur directly within the literal strings in one of the various
Unicode encodings (UTF-8, UTF-EBCDIC, UCS-2, etc.), but will be recognized
as such and converted to Perl's internal representation only if the
appropriate L<encoding> is specified.

Unicode characters can also be added to a string by using the
C<\x{...}> notation.  The Unicode code for the desired character, in
hexadecimal, should be placed in the braces. For instance, a smiley
face is C<\x{263A}>.  This encoding scheme only works for characters
with a code of 0x100 or above.

Additionally, if you

   use charnames ':full';

you can use the C<\N{...}> notation and put the official Unicode
character name within the braces, such as C<\N{WHITE SMILING FACE}>.


=item *

If an appropriate L<encoding> is specified, identifiers within the
Perl script may contain Unicode alphanumeric characters, including
ideographs.  Perl does not currently attempt to canonicalize variable
names.

=item *

Regular expressions match characters instead of bytes.  "." matches
a character instead of a byte.  The C<\C> pattern is provided to force
a match a single byte--a C<char> in C, hence C<\C>.

=item *

Character classes in regular expressions match characters instead of
bytes and match against the character properties specified in the