tokenize.htm

<HTML>
   <HEAD>
      <TITLE> Tokenizer: a C++ tokenizer </TITLE>
   </HEAD>

   <BODY>

      <H1> Tokenizer: a C++ tokenizer  </H1>
      <pre>
$Header: /repository/documentation/tokenize.htm,v 1.2 2006/04/29 00:24:44 joerg Exp $
      </pre>


      <H2> ideas </H2>

      <P> a Tokenizer is an input iterator, which reads an input iterator Source. the
         value_type of Tokenizer is Token, the value_type of Source is Character. </P>

      <P> Character is a arbitrary type, can be but not has to be char. </P>

      <P> Token is a derived class of Token_basis< Character >. the main task of
         Token_basis is to store the original sequence of Character's. so it is possible 
         to putback a Token. </P>

      <P> it is possible to putback as many Token's as wished, there is no limit. that means,
         that inside Tokenizer there must be a container storing putback'ed Character's, because
         it is not requiered from Source, to be able to do that. the only requirenment for
         Source is to be an output iterator. </P>

      <P> if there are several levels of syntax analysis, the output iterator Tokenizer of the 
         first level will be the Source of the second level Tokenizer. </P>

      <P> a Token_context is a trait class, which defines: what is a white space, a delimiter,
         a string, a comment, an operator, an escape Character, ... </P>

      <P> there can be nested Token_context's. e.g. while analysing C++ code, inside a ""
         the Token_context is different. </P>

      <P> 

      
      <HR>
      
      <P> <A HREF="index.htm">the silicon brain home page</A> </P>
      <P> <A HREF="mailto:info@siliconbrain.com">contact 
         SiliconBrain</A></P>
      
      <ADDRESS>
         <A HREF="mailto:info@siliconbrain.com">info@siliconbrain.com</A> 
      </ADDRESS>
      
      <P> this web page was designed by joerg kunze.</P>

      <P> copyright; 2000, 2006 joerg kunze </P>

      <P> this web page is free software; you can redistribute it and/or 
         modify it under the terms of the GNU General Public License
         as published by the Free Software Foundation; either version 2
         of the license or (at your opinion) any later version.</P>

      <P> this web page is distributed in the hope that it is useful, but
         <STRONG>without any warranty</STRONG>; without even the implied warranty of
         <STRONG>merchantability</STRONG> or <STRONG>fitness for a particular
         purpose</STRONG>. see the GNU General Public License for details.</P>

      <P> you should have received a <A HREF="gpl.html">copy of the GNU 
         General Public License</A>
         along with these web page; if not, write to the
         <A HREF="http://www.fsf.org/">Free Software Foundation</A>, Inc.,
         59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.</P>

      <hr>
      <pre>
$Log: tokenize.htm,v $
Revision 1.2  2006/04/29 00:24:44  joerg
add Header and Log cvs keywords

      </pre>


   </BODY>
</HTML>