source: trunk/essentials/dev-lang/perl/pod/perlpodspec.pod@ 3184

=head1 NAME

perlpodspec - Plain Old Documentation: format specification and notes

=head1 DESCRIPTION

This document is detailed notes on the Pod markup language.  Most
people will only have to read L<perlpod|perlpod> to know how to write
in Pod, but this document may answer some incidental questions to do
with parsing and rendering Pod.

In this document, "must" / "must not", "should" /
"should not", and "may" have their conventional (cf. RFC 2119)
meanings: "X must do Y" means that if X doesn't do Y, it's against
this specification, and should really be fixed.  "X should do Y"
means that it's recommended, but X may fail to do Y, if there's a
good reason.  "X may do Y" is merely a note that X can do Y at
will (although it is up to the reader to detect any connotation of
"and I think it would be I<nice> if X did Y" versus "it wouldn't
really I<bother> me if X did Y").

Notably, when I say "the parser should do Y", the
parser may fail to do Y, if the calling application explicitly
requests that the parser I<not> do Y.  I often phrase this as
"the parser should, by default, do Y."  This doesn't I<require>
the parser to provide an option for turning off whatever
feature Y is (like expanding tabs in verbatim paragraphs), although
it implicates that such an option I<may> be provided.

=head1 Pod Definitions

Pod is embedded in files, typically Perl source files -- although you
can write a file that's nothing but Pod.

A B<line> in a file consists of zero or more non-newline characters,
terminated by either a newline or the end of the file.

A B<newline sequence> is usually a platform-dependent concept, but
Pod parsers should understand it to mean any of CR (ASCII 13), LF
(ASCII 10), or a CRLF (ASCII 13 followed immediately by ASCII 10), in
addition to any other system-specific meaning.  The first CR/CRLF/LF
sequence in the file may be used as the basis for identifying the
newline sequence for parsing the rest of the file.

A B<blank line> is a line consisting entirely of zero or more spaces
(ASCII 32) or tabs (ASCII 9), and terminated by a newline or end-of-file.
A B<non-blank line> is a line containing one or more characters other
than space or tab (and terminated by a newline or end-of-file).

(I<Note:> Many older Pod parsers did not accept a line consisting of
spaces/tabs and then a newline as a blank line -- the only lines they
considered blank were lines consisting of I<no characters at all>,
terminated by a newline.)

B<Whitespace> is used in this document as a blanket term for spaces,
tabs, and newline sequences.  (By itself, this term usually refers
to literal whitespace.  That is, sequences of whitespace characters
in Pod source, as opposed to "EE<lt>32>", which is a formatting
code that I<denotes> a whitespace character.)

A B<Pod parser> is a module meant for parsing Pod (regardless of
whether this involves calling callbacks or building a parse tree or
directly formatting it).  A B<Pod formatter> (or B<Pod translator>)
is a module or program that converts Pod to some other format (HTML,
plaintext, TeX, PostScript, RTF).  A B<Pod processor> might be a
formatter or translator, or might be a program that does something
else with the Pod (like wordcounting it, scanning for index points,
etc.).

Pod content is contained in B<Pod blocks>.  A Pod block starts with a
line that matches <m/\A=[a-zA-Z]/>, and continues up to the next line
that matches C<m/\A=cut/> -- or up to the end of the file, if there is
no C<m/\A=cut/> line.

=for comment
 The current perlsyn says:
 [beginquote]
   Note that pod translators should look at only paragraphs beginning
   with a pod directive (it makes parsing easier), whereas the compiler
   actually knows to look for pod escapes even in the middle of a
   paragraph.  This means that the following secret stuff will be ignored
   by both the compiler and the translators.
      $a=3;
      =secret stuff
       warn "Neither POD nor CODE!?"
      =cut back
      print "got $a\n";
   You probably shouldn't rely upon the warn() being podded out forever.
   Not all pod translators are well-behaved in this regard, and perhaps
   the compiler will become pickier.
 [endquote]
 I think that those paragraphs should just be removed; paragraph-based
 parsing  seems to have been largely abandoned, because of the hassle
 with non-empty blank lines messing up what people meant by "paragraph".
 Even if the "it makes parsing easier" bit were especially true,
 it wouldn't be worth the confusion of having perl and pod2whatever
 actually disagree on what can constitute a Pod block.

Within a Pod block, there are B<Pod paragraphs>.  A Pod paragraph
consists of non-blank lines of text, separated by one or more blank
lines.

For purposes of Pod processing, there are four types of paragraphs in
a Pod block:

=over

=item *

A command paragraph (also called a "directive").  The first line of
this paragraph must match C<m/\A=[a-zA-Z]/>.  Command paragraphs are
typically one line, as in:

  =head1 NOTES

  =item *

But they may span several (non-blank) lines:

  =for comment
  Hm, I wonder what it would look like if
  you tried to write a BNF for Pod from this.

  =head3 Dr. Strangelove, or: How I Learned to
  Stop Worrying and Love the Bomb

I<Some> command paragraphs allow formatting codes in their content
(i.e., after the part that matches C<m/\A=[a-zA-Z]\S*\s*/>), as in:

  =head1 Did You Remember to C<use strict;>?

In other words, the Pod processing handler for "head1" will apply the
same processing to "Did You Remember to CE<lt>use strict;>?" that it
would to an ordinary paragraph -- i.e., formatting codes (like
"CE<lt>...>") are parsed and presumably formatted appropriately, and
whitespace in the form of literal spaces and/or tabs is not
significant.

=item *

A B<verbatim paragraph>.  The first line of this paragraph must be a
literal space or tab, and this paragraph must not be inside a "=begin
I<identifier>", ... "=end I<identifier>" sequence unless
"I<identifier>" begins with a colon (":").  That is, if a paragraph
starts with a literal space or tab, but I<is> inside a
"=begin I<identifier>", ... "=end I<identifier>" region, then it's
a data paragraph, unless "I<identifier>" begins with a colon.

Whitespace I<is> significant in verbatim paragraphs (although, in
processing, tabs are probably expanded).

=item *

An B<ordinary paragraph>.  A paragraph is an ordinary paragraph
if its first line matches neither C<m/\A=[a-zA-Z]/> nor
C<m/\A[ \t]/>, I<and> if it's not inside a "=begin I<identifier>",
... "=end I<identifier>" sequence unless "I<identifier>" begins with
a colon (":").

=item *

A B<data paragraph>.  This is a paragraph that I<is> inside a "=begin
I<identifier>" ... "=end I<identifier>" sequence where
"I<identifier>" does I<not> begin with a literal colon (":").  In
some sense, a data paragraph is not part of Pod at all (i.e.,
effectively it's "out-of-band"), since it's not subject to most kinds
of Pod parsing; but it is specified here, since Pod
parsers need to be able to call an event for it, or store it in some
form in a parse tree, or at least just parse I<around> it.

=back

For example: consider the following paragraphs:

  # <- that's the 0th column

  =head1 Foo

  Stuff

    $foo->bar

  =cut

Here, "=head1 Foo" and "=cut" are command paragraphs because the first
line of each matches C<m/\A=[a-zA-Z]/>.  "I<[space][space]>$foo->bar"
is a verbatim paragraph, because its first line starts with a literal
whitespace character (and there's no "=begin"..."=end" region around).

The "=begin I<identifier>" ... "=end I<identifier>" commands stop
paragraphs that they surround from being parsed as data or verbatim
paragraphs, if I<identifier> doesn't begin with a colon.  This
is discussed in detail in the section
L</About Data Paragraphs and "=beginE<sol>=end" Regions>.

=head1 Pod Commands

This section is intended to supplement and clarify the discussion in
L<perlpod/"Command Paragraph">.  These are the currently recognized
Pod commands:

=over

=item "=head1", "=head2", "=head3", "=head4"

This command indicates that the text in the remainder of the paragraph
is a heading.  That text may contain formatting codes.  Examples:

  =head1 Object Attributes

  =head3 What B<Not> to Do!

=item "=pod"

This command indicates that this paragraph begins a Pod block.  (If we
are already in the middle of a Pod block, this command has no effect at
all.)  If there is any text in this command paragraph after "=pod",
it must be ignored.  Examples:

  =pod

  This is a plain Pod paragraph.

  =pod This text is ignored.

=item "=cut"

This command indicates that this line is the end of this previously
started Pod block.  If there is any text after "=cut" on the line, it must be
ignored.  Examples:

  =cut

  =cut The documentation ends here.

  =cut
  # This is the first line of program text.
  sub foo { # This is the second.

It is an error to try to I<start> a Pod block with a "=cut" command.  In
that case, the Pod processor must halt parsing of the input file, and
must by default emit a warning.

=item "=over"

This command indicates that this is the start of a list/indent
region.  If there is any text following the "=over", it must consist
of only a nonzero positive numeral.  The semantics of this numeral is
explained in the L</"About =over...=back Regions"> section, further
below.  Formatting codes are not expanded.  Examples:

  =over 3

  =over 3.5

  =over

=item "=item"

This command indicates that an item in a list begins here.  Formatting
codes are processed.  The semantics of the (optional) text in the
remainder of this paragraph are
explained in the L</"About =over...=back Regions"> section, further
below.  Examples:

  =item

  =item *

  =item      *    

  =item 14

  =item   3.

  =item C<< $thing->stuff(I<dodad>) >>

  =item For transporting us beyond seas to be tried for pretended
  offenses

  =item He is at this time transporting large armies of foreign
  mercenaries to complete the works of death, desolation and
  tyranny, already begun with circumstances of cruelty and perfidy
  scarcely paralleled in the most barbarous ages, and totally
  unworthy the head of a civilized nation.

=item "=back"

This command indicates that this is the end of the region begun
by the most recent "=over" command.  It permits no text after the
"=back" command.

=item "=begin formatname"

This marks the following paragraphs (until the matching "=end
formatname") as being for some special kind of processing.  Unless
"formatname" begins with a colon, the contained non-command
paragraphs are data paragraphs.  But if "formatname" I<does> begin
with a colon, then non-command paragraphs are ordinary paragraphs
or data paragraphs.  This is discussed in detail in the section
L</About Data Paragraphs and "=beginE<sol>=end" Regions>.

It is advised that formatnames match the regexp
C<m/\A:?[-a-zA-Z0-9_]+\z/>.  Implementors should anticipate future
expansion in the semantics and syntax of the first parameter
to "=begin"/"=end"/"=for".

=item "=end formatname"

This marks the end of the region opened by the matching
"=begin formatname" region.  If "formatname" is not the formatname
of the most recent open "=begin formatname" region, then this
is an error, and must generate an error message.  This
is discussed in detail in the section
L</About Data Paragraphs and "=beginE<sol>=end" Regions>.

=item "=for formatname text..."

This is synonymous with:

     =begin formatname

     text...

     =end formatname

That is, it creates a region consisting of a single paragraph; that
paragraph is to be treated as a normal paragraph if "formatname"
begins with a ":"; if "formatname" I<doesn't> begin with a colon,
then "text..." will constitute a data paragraph.  There is no way
to use "=for formatname text..." to express "text..." as a verbatim
paragraph.

=item "=encoding encodingname"

This command, which should occur early in the document (at least
before any non-US-ASCII data!), declares that this document is
encoded in the encoding I<encodingname>, which must be
an encoding name that L<Encoding> recognizes.  (Encoding's list
of supported encodings, in L<Encoding::Supported>, is useful here.)
If the Pod parser cannot decode the declared encoding, it 
should emit a warning and may abort parsing the document
altogether.

A document having more than one "=encoding" line should be
considered an error.  Pod processors may silently tolerate this if
the not-first "=encoding" lines are just duplicates of the
first one (e.g., if there's a "=use utf8" line, and later on
another "=use utf8" line).  But Pod processors should complain if
there are contradictory "=encoding" lines in the same document
(e.g., if there is a "=encoding utf8" early in the document and
"=encoding big5" later).  Pod processors that recognize BOMs
may also complain if they see an "=encoding" line
that contradicts the BOM (e.g., if a document with a UTF-16LE
BOM has an "=encoding shiftjis" line).

=back

If a Pod processor sees any command other than the ones listed
above (like "=head", or "=haed1", or "=stuff", or "=cuttlefish",
or "=w123"), that processor must by default treat this as an
error.  It must not process the paragraph beginning with that
command, must by default warn of this as an error, and may
abort the parse.  A Pod parser may allow a way for particular
applications to add to the above list of known commands, and to
stipulate, for each additional command, whether formatting
codes should be processed.

Future versions of this specification may add additional
commands.



=head1 Pod Formatting Codes

(Note that in previous drafts of this document and of perlpod,
formatting codes were referred to as "interior sequences", and
this term may still be found in the documentation for Pod parsers,
and in error messages from Pod processors.)

There are two syntaxes for formatting codes:

=over

=item *

A formatting code starts with a capital letter (just US-ASCII [A-Z])
followed by a "<", any number of characters, and ending with the first
matching ">".  Examples:

    That's what I<you> think!

    What's C<dump()> for?

    X<C<chmod> and C<unlink()> Under Different Operating Systems>

=item *

A formatting code starts with a capital letter (just US-ASCII [A-Z])
followed by two or more "<"'s, one or more whitespace characters,
any number of characters, one or more whitespace characters,
and ending with the first matching sequence of two or more ">"'s, where
the number of ">"'s equals the number of "<"'s in the opening of this
formatting code.  Examples:

    That's what I<< you >> think!

    C<<< open(X, ">>thing.dat") || die $! >>>

    B<< $foo->bar(); >>

With this syntax, the whitespace character(s) after the "CE<lt><<"
and before the ">>" (or whatever letter) are I<not> renderable -- they
do not signify whitespace, are merely part of the formatting codes
themselves.  That is, these are all synonymous:

    C<thing>
    C<< thing >>
    C<<           thing     >>
    C<<<   thing >>>
    C<<<<
    thing
               >>>>

and so on.

=back

In parsing Pod, a notably tricky part is the correct parsing of
(potentially nested!) formatting codes.  Implementors should
consult the code in the C<parse_text> routine in Pod::Parser as an
example of a correct implementation.

=over

=item C<IE<lt>textE<gt>> -- italic text

See the brief discussion in L<perlpod/"Formatting Codes">.

=item C<BE<lt>textE<gt>> -- bold text

See the brief discussion in L<perlpod/"Formatting Codes">.

=item C<CE<lt>codeE<gt>> -- code text

See the brief discussion in L<perlpod/"Formatting Codes">.

=item C<FE<lt>filenameE<gt>> -- style for filenames

See the brief discussion in L<perlpod/"Formatting Codes">.

=item C<XE<lt>topic nameE<gt>> -- an index entry

See the brief discussion in L<perlpod/"Formatting Codes">.

This code is unusual in that most formatters completely discard
this code and its content.  Other formatters will render it with
invisible codes that can be used in building an index of
the current document.

=item C<ZE<lt>E<gt>> -- a null (zero-effect) formatting code

Discussed briefly in L<perlpod/"Formatting Codes">.

This code is unusual is that it should have no content.  That is,
a processor may complain if it sees C<ZE<lt>potatoesE<gt>>.  Whether
or not it complains, the I<potatoes> text should ignored.

=item C<LE<lt>nameE<gt>> -- a hyperlink

The complicated syntaxes of this code are discussed at length in
L<perlpod/"Formatting Codes">, and implementation details are
discussed below, in L</"About LE<lt>...E<gt> Codes">.  Parsing the
contents of LE<lt>content> is tricky.  Notably, the content has to be
checked for whether it looks like a URL, or whether it has to be split
on literal "|" and/or "/" (in the right order!), and so on,
I<before> EE<lt>...> codes are resolved.

=item C<EE<lt>escapeE<gt>> -- a character escape

See L<perlpod/"Formatting Codes">, and several points in
L</Notes on Implementing Pod Processors>.

=item C<SE<lt>textE<gt>> -- text contains non-breaking spaces

This formatting code is syntactically simple, but semantically
complex.  What it means is that each space in the printable
content of this code signifies a non-breaking space.

Consider:

    C<$x ? $y    :  $z>

    S<C<$x ? $y     :  $z>>

Both signify the monospace (c[ode] style) text consisting of
"$x", one space, "?", one space, ":", one space, "$z".  The
difference is that in the latter, with the S code, those spaces
are not "normal" spaces, but instead are non-breaking spaces.

=back


If a Pod processor sees any formatting code other than the ones
listed above (as in "NE<lt>...>", or "QE<lt>...>", etc.), that
processor must by default treat this as an error.
A Pod parser may allow a way for particular
applications to add to the above list of known formatting codes;
a Pod parser might even allow a way to stipulate, for each additional
command, whether it requires some form of special processing, as
LE<lt>...> does.

Future versions of this specification may add additional
formatting codes.

Historical note:  A few older Pod processors would not see a ">" as
closing a "CE<lt>" code, if the ">" was immediately preceded by
a "-".  This was so that this:

    C<$foo->bar>

would parse as equivalent to this:

    C<$foo-E<gt>bar>

instead of as equivalent to a "C" formatting code containing 
only "$foo-", and then a "bar>" outside the "C" formatting code.  This
problem has since been solved by the addition of syntaxes like this:

    C<< $foo->bar >>

Compliant parsers must not treat "->" as special.

Formatting codes absolutely cannot span paragraphs.  If a code is
opened in one paragraph, and no closing code is found by the end of
that paragraph, the Pod parser must close that formatting code,
and should complain (as in "Unterminated I code in the paragraph
starting at line 123: 'Time objects are not...'").  So these
two paragraphs:

  I<I told you not to do this!

  Don't make me say it again!>

...must I<not> be parsed as two paragraphs in italics (with the I
code starting in one paragraph and starting in another.)  Instead,