source: trunk/essentials/dev-lang/python/Doc/ref/ref2.tex@ 3400

\chapter{Lexical analysis\label{lexical}}

A Python program is read by a \emph{parser}.  Input to the parser is a
stream of \emph{tokens}, generated by the \emph{lexical analyzer}.  This
chapter describes how the lexical analyzer breaks a file into tokens.
\index{lexical analysis}
\index{parser}
\index{token}

Python uses the 7-bit \ASCII{} character set for program text.
\versionadded[An encoding declaration can be used to indicate that 
string literals and comments use an encoding different from ASCII]{2.3}
For compatibility with older versions, Python only warns if it finds
8-bit characters; those warnings should be corrected by either declaring
an explicit encoding, or using escape sequences if those bytes are binary
data, instead of characters.


The run-time character set depends on the I/O devices connected to the
program but is generally a superset of \ASCII.

\strong{Future compatibility note:} It may be tempting to assume that the
character set for 8-bit characters is ISO Latin-1 (an \ASCII{}
superset that covers most western languages that use the Latin
alphabet), but it is possible that in the future Unicode text editors
will become common.  These generally use the UTF-8 encoding, which is
also an \ASCII{} superset, but with very different use for the
characters with ordinals 128-255.  While there is no consensus on this
subject yet, it is unwise to assume either Latin-1 or UTF-8, even
though the current implementation appears to favor Latin-1.  This
applies both to the source character set and the run-time character
set.


\section{Line structure\label{line-structure}}

A Python program is divided into a number of \emph{logical lines}.
\index{line structure}


\subsection{Logical lines\label{logical}}

The end of
a logical line is represented by the token NEWLINE.  Statements cannot
cross logical line boundaries except where NEWLINE is allowed by the
syntax (e.g., between statements in compound statements).
A logical line is constructed from one or more \emph{physical lines}
by following the explicit or implicit \emph{line joining} rules.
\index{logical line}
\index{physical line}
\index{line joining}
\index{NEWLINE token}


\subsection{Physical lines\label{physical}}

A physical line is a sequence of characters terminated by an end-of-line
sequence.  In source files, any of the standard platform line
termination sequences can be used - the \UNIX{} form using \ASCII{} LF
(linefeed), the Windows form using the \ASCII{} sequence CR LF (return
followed by linefeed), or the Macintosh form using the \ASCII{} CR
(return) character.  All of these forms can be used equally, regardless
of platform.

When embedding Python, source code strings should be passed to Python
APIs using the standard C conventions for newline characters (the
\code{\e n} character, representing \ASCII{} LF, is the line
terminator).


\subsection{Comments\label{comments}}

A comment starts with a hash character (\code{\#}) that is not part of
a string literal, and ends at the end of the physical line.  A comment
signifies the end of the logical line unless the implicit line joining
rules are invoked.
Comments are ignored by the syntax; they are not tokens.
\index{comment}
\index{hash character}


\subsection{Encoding declarations\label{encodings}}
\index{source character set}
\index{encodings}

If a comment in the first or second line of the Python script matches
the regular expression \regexp{coding[=:]\e s*([-\e w.]+)}, this comment is
processed as an encoding declaration; the first group of this
expression names the encoding of the source code file. The recommended
forms of this expression are

\begin{verbatim}
# -*- coding: <encoding-name> -*-
\end{verbatim}

which is recognized also by GNU Emacs, and

\begin{verbatim}
# vim:fileencoding=<encoding-name>
\end{verbatim}

which is recognized by Bram Moolenaar's VIM. In addition, if the first
bytes of the file are the UTF-8 byte-order mark
(\code{'\e xef\e xbb\e xbf'}), the declared file encoding is UTF-8
(this is supported, among others, by Microsoft's \program{notepad}).

If an encoding is declared, the encoding name must be recognized by
Python. % XXX there should be a list of supported encodings.
The encoding is used for all lexical analysis, in particular to find
the end of a string, and to interpret the contents of Unicode literals.
String literals are converted to Unicode for syntactical analysis,
then converted back to their original encoding before interpretation
starts. The encoding declaration must appear on a line of its own.

\subsection{Explicit line joining\label{explicit-joining}}

Two or more physical lines may be joined into logical lines using
backslash characters (\code{\e}), as follows: when a physical line ends
in a backslash that is not part of a string literal or comment, it is
joined with the following forming a single logical line, deleting the
backslash and the following end-of-line character.  For example:
\index{physical line}
\index{line joining}
\index{line continuation}
\index{backslash character}
%
\begin{verbatim}
if 1900 < year < 2100 and 1 <= month <= 12 \
   and 1 <= day <= 31 and 0 <= hour < 24 \
   and 0 <= minute < 60 and 0 <= second < 60:   # Looks like a valid date
        return 1
\end{verbatim}

A line ending in a backslash cannot carry a comment.  A backslash does
not continue a comment.  A backslash does not continue a token except
for string literals (i.e., tokens other than string literals cannot be
split across physical lines using a backslash).  A backslash is
illegal elsewhere on a line outside a string literal.


\subsection{Implicit line joining\label{implicit-joining}}

Expressions in parentheses, square brackets or curly braces can be
split over more than one physical line without using backslashes.
For example:

\begin{verbatim}
month_names = ['Januari', 'Februari', 'Maart',      # These are the
               'April',   'Mei',      'Juni',       # Dutch names
               'Juli',    'Augustus', 'September',  # for the months
               'Oktober', 'November', 'December']   # of the year
\end{verbatim}

Implicitly continued lines can carry comments.  The indentation of the
continuation lines is not important.  Blank continuation lines are
allowed.  There is no NEWLINE token between implicit continuation
lines.  Implicitly continued lines can also occur within triple-quoted
strings (see below); in that case they cannot carry comments.


\subsection{Blank lines \label{blank-lines}}

\index{blank line}
A logical line that contains only spaces, tabs, formfeeds and possibly
a comment, is ignored (i.e., no NEWLINE token is generated).  During
interactive input of statements, handling of a blank line may differ
depending on the implementation of the read-eval-print loop.  In the
standard implementation, an entirely blank logical line (i.e.\ one
containing not even whitespace or a comment) terminates a multi-line
statement.


\subsection{Indentation\label{indentation}}

Leading whitespace (spaces and tabs) at the beginning of a logical
line is used to compute the indentation level of the line, which in
turn is used to determine the grouping of statements.
\index{indentation}
\index{whitespace}
\index{leading whitespace}
\index{space}
\index{tab}
\index{grouping}
\index{statement grouping}

First, tabs are replaced (from left to right) by one to eight spaces
such that the total number of characters up to and including the
replacement is a multiple of
eight (this is intended to be the same rule as used by \UNIX).  The
total number of spaces preceding the first non-blank character then
determines the line's indentation.  Indentation cannot be split over
multiple physical lines using backslashes; the whitespace up to the
first backslash determines the indentation.

\strong{Cross-platform compatibility note:} because of the nature of
text editors on non-UNIX platforms, it is unwise to use a mixture of
spaces and tabs for the indentation in a single source file.  It
should also be noted that different platforms may explicitly limit the
maximum indentation level.

A formfeed character may be present at the start of the line; it will
be ignored for the indentation calculations above.  Formfeed
characters occurring elsewhere in the leading whitespace have an
undefined effect (for instance, they may reset the space count to
zero).

The indentation levels of consecutive lines are used to generate
INDENT and DEDENT tokens, using a stack, as follows.
\index{INDENT token}