source: trunk/essentials/dev-lang/perl/pod/perlopentut.pod@ 3368

=head1 NAME

perlopentut - tutorial on opening things in Perl

=head1 DESCRIPTION

Perl has two simple, built-in ways to open files: the shell way for
convenience, and the C way for precision.  The shell way also has 2- and
3-argument forms, which have different semantics for handling the filename.
The choice is yours.

=head1 Open E<agrave> la shell

Perl's C<open> function was designed to mimic the way command-line
redirection in the shell works.  Here are some basic examples
from the shell:

    $ myprogram file1 file2 file3
    $ myprogram    <  inputfile
    $ myprogram    >  outputfile
    $ myprogram    >> outputfile
    $ myprogram    |  otherprogram 
    $ otherprogram |  myprogram

And here are some more advanced examples:

    $ otherprogram      | myprogram f1 - f2
    $ otherprogram 2>&1 | myprogram -
    $ myprogram     <&3
    $ myprogram     >&4

Programmers accustomed to constructs like those above can take comfort
in learning that Perl directly supports these familiar constructs using
virtually the same syntax as the shell.

=head2 Simple Opens

The C<open> function takes two arguments: the first is a filehandle,
and the second is a single string comprising both what to open and how
to open it.  C<open> returns true when it works, and when it fails,
returns a false value and sets the special variable C<$!> to reflect
the system error.  If the filehandle was previously opened, it will
be implicitly closed first.

For example:

    open(INFO,      "datafile") || die("can't open datafile: $!");
    open(INFO,   "<  datafile") || die("can't open datafile: $!");
    open(RESULTS,">  runstats") || die("can't open runstats: $!");
    open(LOG,    ">> logfile ") || die("can't open logfile:  $!");

If you prefer the low-punctuation version, you could write that this way:

    open INFO,   "<  datafile"  or die "can't open datafile: $!";
    open RESULTS,">  runstats"  or die "can't open runstats: $!";
    open LOG,    ">> logfile "  or die "can't open logfile:  $!";

A few things to notice.  First, the leading less-than is optional.
If omitted, Perl assumes that you want to open the file for reading.

Note also that the first example uses the C<||> logical operator, and the
second uses C<or>, which has lower precedence.  Using C<||> in the latter
examples would effectively mean

    open INFO, ( "<  datafile"  || die "can't open datafile: $!" );

which is definitely not what you want.

The other important thing to notice is that, just as in the shell,
any whitespace before or after the filename is ignored.  This is good,
because you wouldn't want these to do different things:

    open INFO,   "<datafile"   
    open INFO,   "< datafile" 
    open INFO,   "<  datafile"

Ignoring surrounding whitespace also helps for when you read a filename
in from a different file, and forget to trim it before opening:

    $filename = <INFO>;         # oops, \n still there
    open(EXTRA, "< $filename") || die "can't open $filename: $!";

This is not a bug, but a feature.  Because C<open> mimics the shell in
its style of using redirection arrows to specify how to open the file, it
also does so with respect to extra whitespace around the filename itself
as well.  For accessing files with naughty names, see 
L<"Dispelling the Dweomer">.

There is also a 3-argument version of C<open>, which lets you put the
special redirection characters into their own argument:

    open( INFO, ">", $datafile ) || die "Can't create $datafile: $!";

In this case, the filename to open is the actual string in C<$datafile>,
so you don't have to worry about C<$datafile> containing characters
that might influence the open mode, or whitespace at the beginning of
the filename that would be absorbed in the 2-argument version.  Also,
any reduction of unnecessary string interpolation is a good thing.

=head2 Indirect Filehandles

C<open>'s first argument can be a reference to a filehandle.  As of
perl 5.6.0, if the argument is uninitialized, Perl will automatically
create a filehandle and put a reference to it in the first argument,
like so:

    open( my $in, $infile )   or die "Couldn't read $infile: $!";
    while ( <$in> ) {
        # do something with $_
    }
    close $in;

Indirect filehandles make namespace management easier.  Since filehandles
are global to the current package, two subroutines trying to open
C<INFILE> will clash.  With two functions opening indirect filehandles
like C<my $infile>, there's no clash and no need to worry about future
conflicts.

Another convenient behavior is that an indirect filehandle automatically
closes when it goes out of scope or when you undefine it:

    sub firstline {
        open( my $in, shift ) && return scalar <$in>;
        # no close() required
    }

=head2 Pipe Opens

In C, when you want to open a file using the standard I/O library,
you use the C<fopen> function, but when opening a pipe, you use the
C<popen> function.  But in the shell, you just use a different redirection
character.  That's also the case for Perl.  The C<open> call 
remains the same--just its argument differs.  

If the leading character is a pipe symbol, C<open> starts up a new
command and opens a write-only filehandle leading into that command.
This lets you write into that handle and have what you write show up on
that command's standard input.  For example:

    open(PRINTER, "| lpr -Plp1")    || die "can't run lpr: $!";
    print PRINTER "stuff\n";
    close(PRINTER)                  || die "can't close lpr: $!";

If the trailing character is a pipe, you start up a new command and open a
read-only filehandle leading out of that command.  This lets whatever that
command writes to its standard output show up on your handle for reading.
For example:

    open(NET, "netstat -i -n |")    || die "can't fork netstat: $!";
    while (<NET>) { }               # do something with input
    close(NET)                      || die "can't close netstat: $!";

What happens if you try to open a pipe to or from a non-existent
command?  If possible, Perl will detect the failure and set C<$!> as
usual.  But if the command contains special shell characters, such as
C<E<gt>> or C<*>, called 'metacharacters', Perl does not execute the
command directly.  Instead, Perl runs the shell, which then tries to
run the command.  This means that it's the shell that gets the error
indication.  In such a case, the C<open> call will only indicate
failure if Perl can't even run the shell.  See L<perlfaq8/"How can I
capture STDERR from an external command?"> to see how to cope with
this.  There's also an explanation in L<perlipc>.

If you would like to open a bidirectional pipe, the IPC::Open2
library will handle this for you.  Check out 
L<perlipc/"Bidirectional Communication with Another Process">

=head2 The Minus File

Again following the lead of the standard shell utilities, Perl's
C<open> function treats a file whose name is a single minus, "-", in a
special way.  If you open minus for reading, it really means to access
the standard input.  If you open minus for writing, it really means to
access the standard output.

If minus can be used as the default input or default output, what happens
if you open a pipe into or out of minus?  What's the default command it
would run?  The same script as you're currently running!  This is actually
a stealth C<fork> hidden inside an C<open> call.  See 
L<perlipc/"Safe Pipe Opens"> for details.

=head2 Mixing Reads and Writes

It is possible to specify both read and write access.  All you do is
add a "+" symbol in front of the redirection.  But as in the shell,
using a less-than on a file never creates a new file; it only opens an
existing one.  On the other hand, using a greater-than always clobbers
(truncates to zero length) an existing file, or creates a brand-new one
if there isn't an old one.  Adding a "+" for read-write doesn't affect
whether it only works on existing files or always clobbers existing ones.

    open(WTMP, "+< /usr/adm/wtmp") 
        || die "can't open /usr/adm/wtmp: $!";

    open(SCREEN, "+> lkscreen")
        || die "can't open lkscreen: $!";

    open(LOGFILE, "+>> /var/log/applog"
        || die "can't open /var/log/applog: $!";

The first one won't create a new file, and the second one will always
clobber an old one.  The third one will create a new file if necessary
and not clobber an old one, and it will allow you to read at any point
in the file, but all writes will always go to the end.  In short,
the first case is substantially more common than the second and third
cases, which are almost always wrong.  (If you know C, the plus in
Perl's C<open> is historically derived from the one in C's fopen(3S),
which it ultimately calls.)

In fact, when it comes to updating a file, unless you're working on
a binary file as in the WTMP case above, you probably don't want to
use this approach for updating.  Instead, Perl's B<-i> flag comes to
the rescue.  The following command takes all the C, C++, or yacc source
or header files and changes all their foo's to bar's, leaving
the old version in the original filename with a ".orig" tacked
on the end:

    $ perl -i.orig -pe 's/\bfoo\b/bar/g' *.[Cchy]

This is a short cut for some renaming games that are really
the best way to update textfiles.  See the second question in 
L<perlfaq5> for more details.

=head2 Filters 

One of the most common uses for C<open> is one you never
even notice.  When you process the ARGV filehandle using
C<< <ARGV> >>, Perl actually does an implicit open 
on each file in @ARGV.  Thus a program called like this:

    $ myprogram file1 file2 file3

Can have all its files opened and processed one at a time
using a construct no more complex than:

    while (<>) {
        # do something with $_
    } 

If @ARGV is empty when the loop first begins, Perl pretends you've opened
up minus, that is, the standard input.  In fact, $ARGV, the currently
open file during C<< <ARGV> >> processing, is even set to "-"
in these circumstances.

You are welcome to pre-process your @ARGV before starting the loop to
make sure it's to your liking.  One reason to do this might be to remove
command options beginning with a minus.  While you can always roll the
simple ones by hand, the Getopts modules are good for this:

    use Getopt::Std;

    # -v, -D, -o ARG, sets $opt_v, $opt_D, $opt_o
    getopts("vDo:");            

    # -v, -D, -o ARG, sets $args{v}, $args{D}, $args{o}
    getopts("vDo:", \%args);    

Or the standard Getopt::Long module to permit named arguments:

    use Getopt::Long;
    GetOptions( "verbose"  => \$verbose,        # --verbose
                "Debug"    => \$debug,          # --Debug
                "output=s" => \$output );       
            # --output=somestring or --output somestring

Another reason for preprocessing arguments is to make an empty
argument list default to all files:

    @ARGV = glob("*") unless @ARGV;

You could even filter out all but plain, text files.  This is a bit
silent, of course, and you might prefer to mention them on the way.

    @ARGV = grep { -f && -T } @ARGV;

If you're using the B<-n> or B<-p> command-line options, you
should put changes to @ARGV in a C<BEGIN{}> block.

Remember that a normal C<open> has special properties, in that it might
call fopen(3S) or it might called popen(3S), depending on what its
argument looks like; that's why it's sometimes called "magic open".
Here's an example:

    $pwdinfo = `domainname` =~ /^(\(none\))?$/
                    ? '< /etc/passwd'
                    : 'ypcat passwd |';

    open(PWD, $pwdinfo)                 
                or die "can't open $pwdinfo: $!";

This sort of thing also comes into play in filter processing.  Because
C<< <ARGV> >> processing employs the normal, shell-style Perl C<open>,
it respects all the special things we've already seen:

    $ myprogram f1 "cmd1|" - f2 "cmd2|" f3 < tmpfile

That program will read from the file F<f1>, the process F<cmd1>, standard
input (F<tmpfile> in this case), the F<f2> file, the F<cmd2> command,
and finally the F<f3> file.

Yes, this also means that if you have files named "-" (and so on) in
your directory, they won't be processed as literal files by C<open>.
You'll need to pass them as "./-", much as you would for the I<rm> program,
or you could use C<sysopen> as described below.

One of the more interesting applications is to change files of a certain
name into pipes.  For example, to autoprocess gzipped or compressed
files by decompressing them with I<gzip>:

    @ARGV = map { /^\.(gz|Z)$/ ? "gzip -dc $_ |" : $_  } @ARGV;

Or, if you have the I<GET> program installed from LWP,
you can fetch URLs before processing them:

    @ARGV = map { m#^\w+://# ? "GET $_ |" : $_ } @ARGV;

It's not for nothing that this is called magic C<< <ARGV> >>.
Pretty nifty, eh?

=head1 Open E<agrave> la C

If you want the convenience of the shell, then Perl's C<open> is
definitely the way to go.  On the other hand, if you want finer precision
than C's simplistic fopen(3S) provides you should look to Perl's
C<sysopen>, which is a direct hook into the open(2) system call.
That does mean it's a bit more involved, but that's the price of 
precision.

C<sysopen> takes 3 (or 4) arguments.

    sysopen HANDLE, PATH, FLAGS, [MASK]

The HANDLE argument is a filehandle just as with C<open>.  The PATH is
a literal path, one that doesn't pay attention to any greater-thans or
less-thans or pipes or minuses, nor ignore whitespace.  If it's there,
it's part of the path.  The FLAGS argument contains one or more values
derived from the Fcntl module that have been or'd together using the
bitwise "|" operator.  The final argument, the MASK, is optional; if
present, it is combined with the user's current umask for the creation
mode of the file.  You should usually omit this.

Although the traditional values of read-only, write-only, and read-write
are 0, 1, and 2 respectively, this is known not to hold true on some
systems.  Instead, it's best to load in the appropriate constants first
from the Fcntl module, which supplies the following standard flags:

    O_RDONLY            Read only
    O_WRONLY            Write only
    O_RDWR              Read and write
    O_CREAT             Create the file if it doesn't exist
    O_EXCL              Fail if the file already exists
    O_APPEND            Append to the file
    O_TRUNC             Truncate the file
    O_NONBLOCK          Non-blocking access

Less common flags that are sometimes available on some operating
systems include C<O_BINARY>, C<O_TEXT>, C<O_SHLOCK>, C<O_EXLOCK>,
C<O_DEFER>, C<O_SYNC>, C<O_ASYNC>, C<O_DSYNC>, C<O_RSYNC>,
C<O_NOCTTY>, C<O_NDELAY> and C<O_LARGEFILE>.  Consult your open(2)
manpage or its local equivalent for details.  (Note: starting from
Perl release 5.6 the C<O_LARGEFILE> flag, if available, is automatically
added to the sysopen() flags because large files are the default.)

Here's how to use C<sysopen> to emulate the simple C<open> calls we had
before.  We'll omit the C<|| die $!> checks for clarity, but make sure
you always check the return values in real code.  These aren't quite
the same, since C<open> will trim leading and trailing whitespace,
but you'll get the idea.

To open a file for reading:

    open(FH, "< $path");
    sysopen(FH, $path, O_RDONLY);

To open a file for writing, creating a new file if needed or else truncating
an old file:

    open(FH, "> $path");
    sysopen(FH, $path, O_WRONLY | O_TRUNC | O_CREAT);

To open a file for appending, creating one if necessary:

    open(FH, ">> $path");
    sysopen(FH, $path, O_WRONLY | O_APPEND | O_CREAT);

To open a file for update, where the file must already exist:

    open(FH, "+< $path");
    sysopen(FH, $path, O_RDWR);

And here are things you can do with C<sysopen> that you cannot do with
a regular C<open>.  As you'll see, it's just a matter of controlling the
flags in the third argument.

To open a file for writing, creating a new file which must not previously
exist:

    sysopen(FH, $path, O_WRONLY | O_EXCL | O_CREAT);

To open a file for appending, where that file must already exist:

    sysopen(FH, $path, O_WRONLY | O_APPEND);

To open a file for update, creating a new file if necessary:

    sysopen(FH, $path, O_RDWR | O_CREAT);

To open a file for update, where that file must not already exist:

    sysopen(FH, $path, O_RDWR | O_EXCL | O_CREAT);

To open a file without blocking, creating one if necessary:

    sysopen(FH, $path, O_WRONLY | O_NONBLOCK | O_CREAT);

=head2 Permissions E<agrave> la mode

If you omit the MASK argument to C<sysopen>, Perl uses the octal value
0666.  The normal MASK to use for executables and directories should
be 0777, and for anything else, 0666.

Why so permissive?  Well, it isn't really.  The MASK will be modified
by your process's current C<umask>.  A umask is a number representing
I<disabled> permissions bits; that is, bits that will not be turned on
in the created files' permissions field.

For example, if your C<umask> were 027, then the 020 part would
disable the group from writing, and the 007 part would disable others
from reading, writing, or executing.  Under these conditions, passing
C<sysopen> 0666 would create a file with mode 0640, since C<0666 & ~027>
is 0640.

You should seldom use the MASK argument to C<sysopen()>.  That takes
away the user's freedom to choose what permission new files will have.
Denying choice is almost always a bad thing.  One exception would be for
cases where sensitive or private data is being stored, such as with mail
folders, cookie files, and internal temporary files.

=head1 Obscure Open Tricks

=head2 Re-Opening Files (dups)

Sometimes you already have a filehandle open, and want to make another
handle that's a duplicate of the first one.  In the shell, we place an
ampersand in front of a file descriptor number when doing redirections.
For example, C<< 2>&1 >> makes descriptor 2 (that's STDERR in Perl)
be redirected into descriptor 1 (which is usually Perl's STDOUT).
The same is essentially true in Perl: a filename that begins with an
ampersand is treated instead as a file descriptor if a number, or as a
filehandle if a string.

    open(SAVEOUT, ">&SAVEERR") || die "couldn't dup SAVEERR: $!";
    open(MHCONTEXT, "<&4")     || die "couldn't dup fd4: $!";

That means that if a function is expecting a filename, but you don't
want to give it a filename because you already have the file open, you
can just pass the filehandle with a leading ampersand.  It's best to
use a fully qualified handle though, just in case the function happens
to be in a different package:

    somefunction("&main::LOGFILE");

This way if somefunction() is planning on opening its argument, it can
just use the already opened handle.  This differs from passing a handle,
because with a handle, you don't open the file.  Here you have something
you can pass to open.

If you have one of those tricky, newfangled I/O objects that the C++
folks are raving about, then this doesn't work because those aren't a
proper filehandle in the native Perl sense.  You'll have to use fileno()
to pull out the proper descriptor number, assuming you can:

    use IO::Socket;
    $handle = IO::Socket::INET->new("www.perl.com:80");
    $fd = $handle->fileno;
    somefunction("&$fd");  # not an indirect function call

It can be easier (and certainly will be faster) just to use real
filehandles though:

    use IO::Socket;
    local *REMOTE = IO::Socket::INET->new("www.perl.com:80");
    die "can't connect" unless defined(fileno(REMOTE));
    somefunction("&main::REMOTE");

If the filehandle or descriptor number is preceded not just with a simple
"&" but rather with a "&=" combination, then Perl will not create a
completely new descriptor opened to the same place using the dup(2)
system call.  Instead, it will just make something of an alias to the
existing one using the fdopen(3S) library call  This is slightly more
parsimonious of systems resources, although this is less a concern
these days.  Here's an example of that:

    $fd = $ENV{"MHCONTEXTFD"};
    open(MHCONTEXT, "<&=$fd")   or die "couldn't fdopen $fd: $!";

If you're using magic C<< <ARGV> >>, you could even pass in as a
command line argument in @ARGV something like C<"<&=$MHCONTEXTFD">,
but we've never seen anyone actually do this.

=head2 Dispelling the Dweomer

Perl is more of a DWIMmer language than something like Java--where DWIM
is an acronym for "do what I mean".  But this principle sometimes leads
to more hidden magic than one knows what to do with.  In this way, Perl