DeSR Dependency Parser

Parser::Tokenizer Class Reference

Read a plain text file, split into tokens. More...

#include <Tokenizer.h>

Inheritance diagram for Parser::Tokenizer:

Inheritance graph
Collaboration diagram for Parser::Tokenizer:

Collaboration graph

List of all members.

Public Member Functions

 Tokenizer (std::istream *is, char const *lang=0)
 Creates a new Tokenizer.
Scanner::Token const * Current ()
 Returns the current token.
bool MoveNext ()
 Advance to the next token and return true if there is one available.
void Reset ()
 Restart.


Detailed Description

Read a plain text file, split into tokens.

Provides an Enumerator interface, since it needs to look at next token to determine if there are more. An Iterator interface would be more cumbersome to implement.

Definition at line 109 of file Tokenizer.h.


Constructor & Destructor Documentation

Parser::Tokenizer::Tokenizer ( std::istream *  is,
char const *  lang = 0 
)

Creates a new Tokenizer.

Parameters:
is the stream containing the sentence to read.

Definition at line 49 of file Tokenizer.cpp.


The documentation for this class was generated from the following files:
 
Copyright © 2005-2007 G. Attardi. Generated on 13 Aug 2009 by doxygen 1.5.7.1.