source: trunk/flex/doc/flex.info-3@ 3032

This is flex.info, produced by makeinfo version 4.5 from flex.texi.

INFO-DIR-SECTION Programming
START-INFO-DIR-ENTRY
* flex: (flex).      Fast lexical analyzer generator (lex replacement).
END-INFO-DIR-ENTRY


   The flex manual is placed under the same licensing conditions as the
rest of flex:

   Copyright (C) 1990, 1997 The Regents of the University of California.
All rights reserved.

   This code is derived from software contributed to Berkeley by Vern
Paxson.

   The United States Government has rights in this work pursuant to
contract no. DE-AC03-76SF00098 between the United States Department of
Energy and the University of California.

   Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

  1.  Redistributions of source code must retain the above copyright
     notice, this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright
     notice, this list of conditions and the following disclaimer in the
     documentation and/or other materials provided with the
     distribution.
   Neither the name of the University nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.

   THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

File: flex.info,  Node: Debugging Options,  Next: Miscellaneous Options,  Prev: Options for Scanner Speed and Size,  Up: Scanner Options

Debugging Options
=================

`-b, --backup, `%option backup''
     Generate backing-up information to `lex.backup'.  This is a list of
     scanner states which require backing up and the input characters on
     which they do so.  By adding rules one can remove backing-up
     states.  If _all_ backing-up states are eliminated and `-Cf' or
     `-CF' is used, the generated scanner will run faster (see the
     `--perf-report' flag).  Only users who wish to squeeze every last
     cycle out of their scanners need worry about this option.  (*note
     Performance::).

`-d, --debug, `%option debug''
     makes the generated scanner run in "debug" mode.  Whenever a
     pattern is recognized and the global variable `yy_flex_debug' is
     non-zero (which is the default), the scanner will write to
     `stderr' a line of the form:


              -accepting rule at line 53 ("the matched text")

     The line number refers to the location of the rule in the file
     defining the scanner (i.e., the file that was fed to flex).
     Messages are also generated when the scanner backs up, accepts the
     default rule, reaches the end of its input buffer (or encounters a
     NUL; at this point, the two look the same as far as the scanner's
     concerned), or reaches an end-of-file.

`-p, --perf-report, `%option perf-report''
     generates a performance report to `stderr'.  The report consists of
     comments regarding features of the `flex' input file which will
     cause a serious loss of performance in the resulting scanner.  If
     you give the flag twice, you will also get comments regarding
     features that lead to minor performance losses.

     Note that the use of `REJECT', and variable trailing context
     (*note Limitations::) entails a substantial performance penalty;
     use of `yymore()', the `^' operator, and the `--interactive' flag
     entail minor performance penalties.

`-s, --nodefault, `%option nodefault''
     causes the _default rule_ (that unmatched scanner input is echoed
     to `stdout)' to be suppressed.  If the scanner encounters input
     that does not match any of its rules, it aborts with an error.
     This option is useful for finding holes in a scanner's rule set.

`-T, --trace, `%option trace''
     makes `flex' run in "trace" mode.  It will generate a lot of
     messages to `stderr' concerning the form of the input and the
     resultant non-deterministic and deterministic finite automata.
     This option is mostly for use in maintaining `flex'.

`-w, --nowarn, `%option nowarn''
     suppresses warning messages.

`-v, --verbose, `%option verbose''
     specifies that `flex' should write to `stderr' a summary of
     statistics regarding the scanner it generates.  Most of the
     statistics are meaningless to the casual `flex' user, but the
     first line identifies the version of `flex' (same as reported by
     `--version'), and the next line the flags used when generating the
     scanner, including those that are on by default.

`--warn, `%option warn''
     warn about certain things. In particular, if the default rule can
     be matched but no defualt rule has been given, the flex will warn
     you.  We recommend using this option always.



File: flex.info,  Node: Miscellaneous Options,  Prev: Debugging Options,  Up: Scanner Options

Miscellaneous Options
=====================

`-c'
     is a do-nothing option included for POSIX compliance.

     generates

`-h, -?, --help'
     generates a "help" summary of `flex''s options to `stdout' and
     then exits.

`-n'
     is another do-nothing option included only for POSIX compliance.

`-V, --version'
     prints the version number to `stdout' and exits.



File: flex.info,  Node: Performance,  Next: Cxx,  Prev: Scanner Options,  Up: Top

Performance Considerations
**************************

   The main design goal of `flex' is that it generate high-performance
scanners.  It has been optimized for dealing well with large sets of
rules.  Aside from the effects on scanner speed of the table compression
`-C' options outlined above, there are a number of options/actions
which degrade performance.  These are, from most expensive to least:


         REJECT
         arbitrary trailing context
     
         pattern sets that require backing up
         %option yylineno
         %array
     
         %option interactive
         %option always-interactive
     
         @samp{^} beginning-of-line operator
         yymore()

   with the first two all being quite expensive and the last two being
quite cheap.  Note also that `unput()' is implemented as a routine call
that potentially does quite a bit of work, while `yyless()' is a
quite-cheap macro. So if you are just putting back some excess text you
scanned, use `ss()'.

   `REJECT' should be avoided at all costs when performance is
important.  It is a particularly expensive option.

   There is one case when `%option yylineno' can be expensive. That is
when your patterns match long tokens that could _possibly_ contain a
newline character. There is no performance penalty for rules that can
not possibly match newlines, since flex does not need to check them for
newlines.  In general, you should avoid rules such as `[^f]+', which
match very long tokens, including newlines, and may possibly match your
entire file! A better approach is to separate `[^f]+' into two rules:


     %option yylineno
     %%
         [^f\n]+
         \n+

   The above scanner does not incur a performance penalty.

   Getting rid of backing up is messy and often may be an enormous
amount of work for a complicated scanner.  In principal, one begins by
using the `-b' flag to generate a `lex.backup' file.  For example, on
the input:


         %%
         foo        return TOK_KEYWORD;
         foobar     return TOK_KEYWORD;

   the file looks like:


         State #6 is non-accepting -
          associated rule line numbers:
                2       3
          out-transitions: [ o ]
          jam-transitions: EOF [ \001-n  p-\177 ]
     
         State #8 is non-accepting -
          associated rule line numbers:
                3
          out-transitions: [ a ]
          jam-transitions: EOF [ \001-`  b-\177 ]
     
         State #9 is non-accepting -
          associated rule line numbers:
                3
          out-transitions: [ r ]
          jam-transitions: EOF [ \001-q  s-\177 ]
     
         Compressed tables always back up.

   The first few lines tell us that there's a scanner state in which it
can make a transition on an 'o' but not on any other character, and
that in that state the currently scanned text does not match any rule.
The state occurs when trying to match the rules found at lines 2 and 3
in the input file.  If the scanner is in that state and then reads
something other than an 'o', it will have to back up to find a rule
which is matched.  With a bit of headscratching one can see that this
must be the state it's in when it has seen `fo'.  When this has
happened, if anything other than another `o' is seen, the scanner will
have to back up to simply match the `f' (by the default rule).

   The comment regarding State #8 indicates there's a problem when
`foob' has been scanned.  Indeed, on any character other than an `a',
the scanner will have to back up to accept "foo".  Similarly, the
comment for State #9 concerns when `fooba' has been scanned and an `r'
does not follow.

   The final comment reminds us that there's no point going to all the
trouble of removing backing up from the rules unless we're using `-Cf'
or `-CF', since there's no performance gain doing so with compressed
scanners.

   The way to remove the backing up is to add "error" rules:


         %%
         foo         return TOK_KEYWORD;
         foobar      return TOK_KEYWORD;
     
         fooba       |
         foob        |
         fo          {
                     /* false alarm, not really a keyword */
                     return TOK_ID;
                     }

   Eliminating backing up among a list of keywords can also be done
using a "catch-all" rule:


         %%
         foo         return TOK_KEYWORD;
         foobar      return TOK_KEYWORD;
     
         [a-z]+      return TOK_ID;

   This is usually the best solution when appropriate.

   Backing up messages tend to cascade.  With a complicated set of rules
it's not uncommon to get hundreds of messages.  If one can decipher
them, though, it often only takes a dozen or so rules to eliminate the
backing up (though it's easy to make a mistake and have an error rule
accidentally match a valid token.  A possible future `flex' feature
will be to automatically add rules to eliminate backing up).

   It's important to keep in mind that you gain the benefits of
eliminating backing up only if you eliminate _every_ instance of
backing up.  Leaving just one means you gain nothing.

   _Variable_ trailing context (where both the leading and trailing
parts do not have a fixed length) entails almost the same performance
loss as `REJECT' (i.e., substantial).  So when possible a rule like:


         %%
         mouse|rat/(cat|dog)   run();

   is better written:


         %%
         mouse/cat|dog         run();
         rat/cat|dog           run();

   or as


         %%
         mouse|rat/cat         run();
         mouse|rat/dog         run();

   Note that here the special '|' action does _not_ provide any
savings, and can even make things worse (*note Limitations::).

   Another area where the user can increase a scanner's performance (and
one that's easier to implement) arises from the fact that the longer the