java.lang.Object
- gw.internal.gosu.parser.SourceCodeTokenizer

All Implemented Interfaces:: ISourceCodeTokenizer

public final class SourceCodeTokenizer
extends java.lang.Object
implements ISourceCodeTokenizer

Field Summary

Fields
Modifier and Type Field Description

private SourceCodeTokenizerInternal _internal

private Stack<ITokenizerOffsetMarker> _offsetMarkers

private int _state
- Fields inherited from interface gw.lang.parser.ISourceCodeTokenizer
  TT_COMMENT, TT_EOF, TT_EOL, TT_INTEGER, TT_KEYWORD, TT_NOTHING, TT_NUMBER, TT_OPERATOR, TT_WHITESPACE, TT_WORD

Constructor Summary

Constructors
Modifier	Constructor	Description
`private`	`SourceCodeTokenizer()`
`private`	`SourceCodeTokenizer(boolean initForCopy)`
	`SourceCodeTokenizer(SourceCodeReader reader)`
	`SourceCodeTokenizer(SourceCodeReader reader, ITokenizerInstructor instructor)`
	`SourceCodeTokenizer(java.io.Reader reader)`
	`SourceCodeTokenizer(java.lang.CharSequence sourceCode)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`commentChar(int ch)`
`IToken`	`copy()`
`IToken`	`copyInto(IToken t)`
`int`	`countMatches(java.lang.String s)`
`int`	`countMatches(java.lang.String s, int tokenType)`
`void`	`eolIsSignificant(boolean bFlag)`
`Token`	`getCurrentToken()`
`ITokenizerInstructor`	`getInstructor()`
`SourceCodeTokenizerInternal`	`getInternal()`
`int`	`getLineNumber()`
`int`	`getLineOffset()`
`ITokenizerOffsetMarker`	`getOffsetMarker()`
`Token`	`getPriorToken()`
`Token`	`getPriorToken(boolean bSkipWhitespace)`
`Token`	`getPriorToken(boolean bSkipWhitespace, boolean bSkipComments)`
`SourceCodeReader`	`getReader()`
`int`	`getRestoreState()`
`java.lang.String`	`getSource()`
`int`	`getState()`
`java.lang.String`	`getStringValue()`
`java.lang.String`	`getTokenAsString()`
`Token`	`getTokenAt(int iTokenIndex)`
`IToken`	`getTokenAtPosition(int docPosition)`
`int`	`getTokenColumn()`
`int`	`getTokenEnd()`
`Stack<Token>`	`getTokens()`
`int`	`getTokenStart()`
`int`	`getType()`
`(package private) void`	`goToPosition(int iOffset)`
`boolean`	`isAnalyzingDirective()`
`boolean`	`isAnalyzingSeparately()`
`boolean`	`isCommentsSignificant()`
`boolean`	`isEOF()`
`boolean`	`isNOTHING()`
`boolean`	`isOperator(java.lang.String strOperator)`
`boolean`	`isParseDotsAsOperators()`
`boolean`	`isPositioned()`
`boolean`	`isSupportsKeywords()`
`boolean`	`isUnterminatedComment()`
`boolean`	`isUnterminatedString()`
`boolean`	`isWhitespaceSignificant()`
`SourceCodeTokenizer`	`lightweightRestore()`
`int`	`lookaheadType(int iTokens, boolean bSkipSpaces)`
`void`	`lowerCaseMode(boolean bLowerCaseMode)`
`int`	`mark()`
`private int`	`maybeSkipWhitespaceAndComments(int state, Stack<Token> tokens, int count)`
`void`	`nextToken()`
`void`	`operatorChars(int iLow, int iHigh)`
`void`	`operators(java.lang.String[] astrOperators)`
`void`	`ordinaryChar(int ch)`
`void`	`ordinaryChars(int iLow, int iHigh)`
`void`	`parseNumbers()`
`DocCommentBlock`	`popLastComment()`
`void`	`popOffsetMarker(ITokenizerOffsetMarker offsetMarker)`
`void`	`pushOffsetMarker(ITokenizerOffsetMarker offsetMarker)`
`void`	`quoteChar(int ch)`
`void`	`reset()`
`void`	`reset(SourceCodeReader reader)`
`private void`	`reset(SourceCodeReader reader, boolean bResetReader)`
`void`	`reset(java.io.Reader reader)`
`void`	`resetButKeepTokens()`
`void`	`restoreToMark(int iMarkedOffset)`
`void`	`setCommentsSignificant(boolean bCommentsSignificant)`
`void`	`setInstructor(ITokenizerInstructor instructor)`
`void`	`setParseDotsAsOperators(boolean parseDotsAsOperators)`
`void`	`setSupportsKeywords(boolean supportsKeywords)`
`void`	`setWhitespaceSignificant(boolean bWhitespaceSignificant)`
`private int`	`skipSpaces(int iPos)`
`java.lang.String`	`toString()`
`void`	`whitespaceChars(int iLow, int iHigh)`
`void`	`wordChars(int iLow, int iHigh)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - _state
```
private int _state
```
  - _internal
```
private SourceCodeTokenizerInternal _internal
```
  - _offsetMarkers
```
private Stack<ITokenizerOffsetMarker> _offsetMarkers
```
- Constructor Detail
  - SourceCodeTokenizer
```
private SourceCodeTokenizer()
```
  - SourceCodeTokenizer
```
private SourceCodeTokenizer(boolean initForCopy)
```
  - SourceCodeTokenizer
```
public SourceCodeTokenizer(java.lang.CharSequence sourceCode)
```
  - SourceCodeTokenizer
```
public SourceCodeTokenizer(java.io.Reader reader)
```
  - SourceCodeTokenizer
```
public SourceCodeTokenizer(SourceCodeReader reader)
```
  - SourceCodeTokenizer
```
public SourceCodeTokenizer(SourceCodeReader reader,
                           ITokenizerInstructor instructor)
```
- Method Detail
  - copy
```
public IToken copy()
```
    Specified by:
    
    copy in interface ISourceCodeTokenizer
  - copyInto
```
public IToken copyInto(IToken t)
```
    Specified by:
    
    copyInto in interface ISourceCodeTokenizer
  - getInternal
```
public SourceCodeTokenizerInternal getInternal()
```
  - lightweightRestore
```
public SourceCodeTokenizer lightweightRestore()
```
    Specified by:
    
    lightweightRestore in interface ISourceCodeTokenizer
    
    Returns:
    
    a very lightweight restored copy of this tokenizer, without deep cloning anything. This method should be used with care, as it does not clone internal state and cannot actually tokenize. It should be used only for offset information.
  - isPositioned
```
public boolean isPositioned()
```
    Specified by:
    
    isPositioned in interface ISourceCodeTokenizer
  - reset
```
public void reset()
```
    Specified by:
    
    reset in interface ISourceCodeTokenizer
  - reset
```
public void reset(java.io.Reader reader)
```
    Specified by:
    
    reset in interface ISourceCodeTokenizer
  - reset
```
public void reset(SourceCodeReader reader)
```
    Specified by:
    
    reset in interface ISourceCodeTokenizer
  - reset
```
private void reset(SourceCodeReader reader,
                   boolean bResetReader)
```
  - getReader
```
public SourceCodeReader getReader()
```
    Specified by:
    
    getReader in interface ISourceCodeTokenizer
  - getSource
```
public java.lang.String getSource()
```
    Specified by:
    
    getSource in interface ISourceCodeTokenizer
  - getInstructor
```
public ITokenizerInstructor getInstructor()
```
    Specified by:
    
    getInstructor in interface ISourceCodeTokenizer
  - setInstructor
```
public void setInstructor(ITokenizerInstructor instructor)
```
    Specified by:
    
    setInstructor in interface ISourceCodeTokenizer
  - isWhitespaceSignificant
```
public boolean isWhitespaceSignificant()
```
    Specified by:
    
    isWhitespaceSignificant in interface ISourceCodeTokenizer
  - setWhitespaceSignificant
```
public void setWhitespaceSignificant(boolean bWhitespaceSignificant)
```
    Specified by:
    
    setWhitespaceSignificant in interface ISourceCodeTokenizer
  - isCommentsSignificant
```
public boolean isCommentsSignificant()
```
    Specified by:
    
    isCommentsSignificant in interface ISourceCodeTokenizer
  - setCommentsSignificant
```
public void setCommentsSignificant(boolean bCommentsSignificant)
```
    Specified by:
    
    setCommentsSignificant in interface ISourceCodeTokenizer
  - getCurrentToken
```
public final Token getCurrentToken()
```
    Specified by:
    
    getCurrentToken in interface ISourceCodeTokenizer
  - getTokenAt
```
public Token getTokenAt(int iTokenIndex)
```
    Specified by:
    
    getTokenAt in interface ISourceCodeTokenizer
  - getTokenAtPosition
```
public IToken getTokenAtPosition(int docPosition)
```
    Specified by:
    
    getTokenAtPosition in interface ISourceCodeTokenizer
  - getLineNumber
```
public int getLineNumber()
```
    Specified by:
    
    getLineNumber in interface ISourceCodeTokenizer
  - getLineOffset
```
public int getLineOffset()
```
    Specified by:
    
    getLineOffset in interface ISourceCodeTokenizer
  - getTokenColumn
```
public int getTokenColumn()
```
    Specified by:
    
    getTokenColumn in interface ISourceCodeTokenizer
  - wordChars
```
public void wordChars(int iLow,
                      int iHigh)
```
    Specified by:
    
    wordChars in interface ISourceCodeTokenizer
  - whitespaceChars
```
public void whitespaceChars(int iLow,
                            int iHigh)
```
    Specified by:
    
    whitespaceChars in interface ISourceCodeTokenizer
  - ordinaryChars
```
public void ordinaryChars(int iLow,
                          int iHigh)
```
    Specified by:
    
    ordinaryChars in interface ISourceCodeTokenizer
  - ordinaryChar
```
public void ordinaryChar(int ch)
```
    Specified by:
    
    ordinaryChar in interface ISourceCodeTokenizer
  - operators
```
public void operators(java.lang.String[] astrOperators)
```
    Specified by:
    
    operators in interface ISourceCodeTokenizer
  - operatorChars
```
public void operatorChars(int iLow,
                          int iHigh)
```
    Specified by:
    
    operatorChars in interface ISourceCodeTokenizer
  - isOperator
```
public boolean isOperator(java.lang.String strOperator)
```
    Specified by:
    
    isOperator in interface ISourceCodeTokenizer
  - commentChar
```
public void commentChar(int ch)
```
    Specified by:
    
    commentChar in interface ISourceCodeTokenizer
  - quoteChar
```
public void quoteChar(int ch)
```
    Specified by:
    
    quoteChar in interface ISourceCodeTokenizer
  - parseNumbers
```
public void parseNumbers()
```
    Specified by:
    
    parseNumbers in interface ISourceCodeTokenizer
  - eolIsSignificant
```
public void eolIsSignificant(boolean bFlag)
```
    Specified by:
    
    eolIsSignificant in interface ISourceCodeTokenizer
  - lowerCaseMode
```
public void lowerCaseMode(boolean bLowerCaseMode)
```
    Specified by:
    
    lowerCaseMode in interface ISourceCodeTokenizer
  - isUnterminatedString
```
public boolean isUnterminatedString()
```
    Specified by:
    
    isUnterminatedString in interface ISourceCodeTokenizer
  - isUnterminatedComment
```
public boolean isUnterminatedComment()
```
    Specified by:
    
    isUnterminatedComment in interface ISourceCodeTokenizer
  - setParseDotsAsOperators
```
public void setParseDotsAsOperators(boolean parseDotsAsOperators)
```
    Specified by:
    
    setParseDotsAsOperators in interface ISourceCodeTokenizer
  - isParseDotsAsOperators
```
public boolean isParseDotsAsOperators()
```
    Specified by:
    
    isParseDotsAsOperators in interface ISourceCodeTokenizer
  - popLastComment
```
public DocCommentBlock popLastComment()
```
  - getTokenStart
```
public int getTokenStart()
```
    Specified by:
    
    getTokenStart in interface ISourceCodeTokenizer
  - getTokenEnd
```
public int getTokenEnd()
```
    Specified by:
    
    getTokenEnd in interface ISourceCodeTokenizer
  - getTokenAsString
```
public java.lang.String getTokenAsString()
```
    Specified by:
    
    getTokenAsString in interface ISourceCodeTokenizer
  - mark
```
public int mark()
```
    Specified by:
    
    mark in interface ISourceCodeTokenizer
  - restoreToMark
```
public void restoreToMark(int iMarkedOffset)
```
    Specified by:
    
    restoreToMark in interface ISourceCodeTokenizer
  - getRestoreState
```
public int getRestoreState()
```
    Specified by:
    
    getRestoreState in interface ISourceCodeTokenizer
  - getTokens
```
public final Stack<Token> getTokens()
```
    Specified by:
    
    getTokens in interface ISourceCodeTokenizer
  - nextToken
```
public void nextToken()
```
    Specified by:
    
    nextToken in interface ISourceCodeTokenizer
  - maybeSkipWhitespaceAndComments
```
private int maybeSkipWhitespaceAndComments(int state,
                                           Stack<Token> tokens,
                                           int count)
```
  - getStringValue
```
public java.lang.String getStringValue()
```
    Specified by:
    
    getStringValue in interface ISourceCodeTokenizer
  - toString
```
public java.lang.String toString()
```
    Overrides:
    
    toString in class java.lang.Object
  - getOffsetMarker
```
public ITokenizerOffsetMarker getOffsetMarker()
```
  - pushOffsetMarker
```
public void pushOffsetMarker(ITokenizerOffsetMarker offsetMarker)
```
    Specified by:
    
    pushOffsetMarker in interface ISourceCodeTokenizer
  - popOffsetMarker
```
public void popOffsetMarker(ITokenizerOffsetMarker offsetMarker)
```
    Specified by:
    
    popOffsetMarker in interface ISourceCodeTokenizer
  - getState
```
public int getState()
```
    Specified by:
    
    getState in interface ISourceCodeTokenizer
  - countMatches
```
public int countMatches(java.lang.String s)
```
    Specified by:
    
    countMatches in interface ISourceCodeTokenizer
  - countMatches
```
public int countMatches(java.lang.String s,
                        int tokenType)
```
    Specified by:
    
    countMatches in interface ISourceCodeTokenizer
  - getType
```
public int getType()
```
    Specified by:
    
    getType in interface ISourceCodeTokenizer
  - isEOF
```
public boolean isEOF()
```
    Specified by:
    
    isEOF in interface ISourceCodeTokenizer
  - isNOTHING
```
public boolean isNOTHING()
```
  - goToPosition
```
void goToPosition(int iOffset)
           throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - resetButKeepTokens
```
public void resetButKeepTokens()
```
  - isAnalyzingSeparately
```
public boolean isAnalyzingSeparately()
```
    Specified by:
    
    isAnalyzingSeparately in interface ISourceCodeTokenizer
  - isAnalyzingDirective
```
public boolean isAnalyzingDirective()
```
    Specified by:
    
    isAnalyzingDirective in interface ISourceCodeTokenizer
  - isSupportsKeywords
```
public boolean isSupportsKeywords()
```
    Specified by:
    
    isSupportsKeywords in interface ISourceCodeTokenizer
  - setSupportsKeywords
```
public void setSupportsKeywords(boolean supportsKeywords)
```
    Specified by:
    
    setSupportsKeywords in interface ISourceCodeTokenizer
  - getPriorToken
```
public Token getPriorToken()
```
  - getPriorToken
```
public Token getPriorToken(boolean bSkipWhitespace)
```
  - getPriorToken
```
public Token getPriorToken(boolean bSkipWhitespace,
                           boolean bSkipComments)
```
  - lookaheadType
```
public int lookaheadType(int iTokens,
                         boolean bSkipSpaces)
```
  - skipSpaces
```
private int skipSpaces(int iPos)
```

Modifier and Type	Field	Description
`private SourceCodeTokenizerInternal`	`_internal`
`private Stack<ITokenizerOffsetMarker>`	`_offsetMarkers`
`private int`	`_state`

Class SourceCodeTokenizer

Field Summary

Fields inherited from interface gw.lang.parser.ISourceCodeTokenizer

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

_state

_internal

_offsetMarkers

Constructor Detail

SourceCodeTokenizer

SourceCodeTokenizer

SourceCodeTokenizer

SourceCodeTokenizer

SourceCodeTokenizer

SourceCodeTokenizer

Method Detail

copy

copyInto

getInternal

lightweightRestore

isPositioned

reset

reset

reset

reset

getReader

getSource

getInstructor

setInstructor

isWhitespaceSignificant

setWhitespaceSignificant

isCommentsSignificant

setCommentsSignificant

getCurrentToken

getTokenAt

getTokenAtPosition

getLineNumber

getLineOffset

getTokenColumn

wordChars

whitespaceChars

ordinaryChars

ordinaryChar

operators

operatorChars

isOperator

commentChar

quoteChar

parseNumbers

eolIsSignificant

lowerCaseMode

isUnterminatedString

isUnterminatedComment

setParseDotsAsOperators

isParseDotsAsOperators

popLastComment

getTokenStart

getTokenEnd

getTokenAsString

mark

restoreToMark

getRestoreState

getTokens

nextToken

maybeSkipWhitespaceAndComments

getStringValue

toString

getOffsetMarker

pushOffsetMarker

popOffsetMarker

getState

countMatches

countMatches

getType

isEOF

isNOTHING

goToPosition

resetButKeepTokens