DeSR Dependency Parser |
#include <Tokenizer.h>
Public Member Functions | |
Tokenizer (std::istream *is, char const *lang=0) | |
Creates a new Tokenizer . | |
Scanner::Token const * | Current () |
Returns the current token. | |
bool | MoveNext () |
Advance to the next token and return true if there is one available. | |
void | Reset () |
Restart. |
Provides an Enumerator interface, since it needs to look at next token to determine if there are more. An Iterator interface would be more cumbersome to implement.
Definition at line 109 of file Tokenizer.h.
Parser::Tokenizer::Tokenizer | ( | std::istream * | is, | |
char const * | lang = 0 | |||
) |
Creates a new Tokenizer
.
is | the stream containing the sentence to read. |
Definition at line 49 of file Tokenizer.cpp.