java.lang.Object
- gw.internal.gosu.parser.SourceCodeTokenizerInternal

```
public final class SourceCodeTokenizerInternal
extends java.lang.Object
```
This class is adapted from java.io.SourceCodeTokenizer. It adds the notion of operator to the mix. You can define your own operators or use the default set of operators, which are taken from the Java Language Spec. It also captures state information for use by our parser e.g., current token location, line number and column.

Field Summary

Fields
Modifier and Type	Field	Description
`private boolean`	`_bCommentsSignificant`
`private boolean`	`_bEOLIsSignificantP`
`private boolean`	`_bForceLower`
`private boolean`	`_bParseDotsAsOperators`
`boolean`	`_bUnterminatedComment`
`boolean`	`_bUnterminatedString`
`private boolean`	`_bWhitespaceSignificant`
`private int[]`	`_ctype`
`private Token`	`_eof`
`private int`	`_iColumn`
`int`	`_iInvalidCharPos`
`private int`	`_iLineNum`
`private int`	`_iLineOffset`
`private ITokenizerInstructor`	`_instructor`
`private int`	`_iPos`
`private int`	`_iTokenColumn`
`private int`	`_iTokenStart`
`int`	`_iType`
`Keyword`	`_keyword`
`private DocCommentBlock`	`_lastComment`
`private java.util.Set<java.lang.String>`	`_operators`
`(package private) int`	`_peekc`
`private SourceCodeReader`	`_reader`
`java.lang.String`	`_strValue`
`private boolean`	`_supportsKeywords`
`private Stack<Token>`	`_tokens`
`private static java.util.Set<java.lang.String>`	`BITSHIFT_OPERATORS`
`protected static int`	`CT_ALPHA`
`protected static int`	`CT_BITSHIFT_OPERATOR`
`protected static int`	`CT_CHARQUOTE`
`protected static int`	`CT_COMMENT`
`protected static int`	`CT_DIGIT`
`protected static int`	`CT_OPERATOR`
`protected static int`	`CT_QUOTE`
`protected static int`	`CT_WHITESPACE`
`private static java.util.Set<java.lang.String>`	`DEFAULT_OPERATORS`

Constructor Summary

Constructors
Constructor	Description
`SourceCodeTokenizerInternal(boolean initForCopy)`
`SourceCodeTokenizerInternal(SourceCodeReader reader)`
`SourceCodeTokenizerInternal(SourceCodeReader reader, ITokenizerInstructor instructor)`
`SourceCodeTokenizerInternal(java.io.Reader reader)`
`SourceCodeTokenizerInternal(java.lang.CharSequence sourceCode)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`private int`	`_nextTokenImpl()`
`void`	`bitshiftOperatorChars(int iLow, int iHigh)`
`void`	`charQuoteChar(int ch)`
`void`	`commentChar(int ch)`
`private void`	`consumeBlockComment()`
`Token`	`copy()`
`void`	`eolIsSignificant(boolean bFlag)`
`static java.lang.String[]`	`getBitshiftOperators()`
`static java.util.List<java.lang.String>`	`getDefaultBindingOperators()`
`static java.lang.String[]`	`getDefaultOperators()`
`Token`	`getEofToken()`
`ITokenizerInstructor`	`getInstructor()`
`int`	`getLineNumber()`
`int`	`getLineOffset()`
`SourceCodeReader`	`getReader()`
`java.lang.String`	`getSource()`
`int`	`getTokenColumn()`
`int`	`getTokenEnd()`
`Stack<Token>`	`getTokens()`
`int`	`getTokenStart()`
`int`	`getType()`
`(package private) void`	`goToPosition(int iOffset)`
`protected void`	`incrementLineNumber()`
`private void`	`initEofToken()`
`(package private) Token`	`initToken(Token token)`
`boolean`	`isAnalyzingDirective()`
`boolean`	`isAnalyzingSeparately()`
`private boolean`	`isAtIgnorePos()`
`boolean`	`isCommentsSignificant()`
`boolean`	`isEOF()`
`private boolean`	`isHexDigit(int c)`
`private boolean`	`isLeftOpenIntervalOp(java.lang.String strOpTest)`
`boolean`	`isNOTHING()`
`boolean`	`isOperator(java.lang.String strOperator)`
`boolean`	`isParseDotsAsOperators()`
`private boolean`	`isReserved()`
`boolean`	`isSupportsKeywords()`
`boolean`	`isUnterminatedComment()`
`boolean`	`isUnterminatedString()`
`boolean`	`isWhitespaceSignificant()`
`void`	`lowerCaseMode(boolean bLowerCaseMode)`
`int`	`nextToken()`
`void`	`operatorChars(int iLow, int iHigh)`
`void`	`operators(java.lang.String[] astrOperators)`
`void`	`ordinaryChar(int ch)`
`void`	`ordinaryChars(int iLow, int iHigh)`
`void`	`parseNumbers()`
`DocCommentBlock`	`popLastComment()`
`private void`	`pushToken()`
`private void`	`pushToken(Token token)`
`private void`	`pushWhitespaceToken(java.lang.StringBuilder sbWhitespace)`
`void`	`quoteChar(int ch)`
`private int`	`read()`
`private int`	`readOne()`
`protected int`	`readWithInstructions(int c)`
`void`	`reset()`
`void`	`reset(SourceCodeReader reader)`
`(package private) void`	`reset(SourceCodeReader reader, boolean bResetReader)`
`void`	`reset(java.io.Reader reader)`
`(package private) void`	`rip()`
`private void`	`setBitshiftOperatorChars(java.util.Set<java.lang.String> operators)`
`void`	`setCommentsSignificant(boolean bCommentsSignificant)`
`void`	`setInstructor(ITokenizerInstructor instructor)`
`private void`	`setOperatorChars(java.util.Set<java.lang.String> operators)`
`void`	`setParseDotsAsOperators(boolean parseDotsAsOperators)`
`void`	`setSupportsKeywords(boolean supportsKeywords)`
`(package private) void`	`setTokens(Stack<Token> tokens)`
`void`	`setWhitespaceSignificant(boolean bWhitespaceSignificant)`
`private boolean`	`stopOnDot(int c)`
`java.lang.String`	`toString()`
`void`	`whitespaceChars(int iLow, int iHigh)`
`void`	`wordChars(int iLow, int iHigh)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

CT_WHITESPACE
```
protected static final int CT_WHITESPACE
```
See Also:

Constant Field Values

CT_DIGIT
```
protected static final int CT_DIGIT
```
See Also:

Constant Field Values

CT_ALPHA
```
protected static final int CT_ALPHA
```
See Also:

Constant Field Values

CT_QUOTE
```
protected static final int CT_QUOTE
```
See Also:

Constant Field Values

CT_COMMENT
```
protected static final int CT_COMMENT
```
See Also:

Constant Field Values

CT_OPERATOR
```
protected static final int CT_OPERATOR
```
See Also:

Constant Field Values

CT_BITSHIFT_OPERATOR
```
protected static final int CT_BITSHIFT_OPERATOR
```
See Also:

Constant Field Values

CT_CHARQUOTE
```
protected static final int CT_CHARQUOTE
```
See Also:

Constant Field Values

DEFAULT_OPERATORS

private static final java.util.Set<java.lang.String> DEFAULT_OPERATORS

BITSHIFT_OPERATORS

private static final java.util.Set<java.lang.String> BITSHIFT_OPERATORS

_reader
```
private SourceCodeReader _reader
```

_peekc
```
int _peekc
```

_bForceLower
```
private boolean _bForceLower
```

_bEOLIsSignificantP
```
private boolean _bEOLIsSignificantP
```

_ctype
```
private int[] _ctype
```

_operators

private java.util.Set<java.lang.String> _operators

_iPos
```
private int _iPos
```

_iLineNum
```
private int _iLineNum
```

_iColumn
```
private int _iColumn
```

_iTokenStart
```
private int _iTokenStart
```

_iTokenColumn
```
private int _iTokenColumn
```

_bWhitespaceSignificant

private boolean _bWhitespaceSignificant

_bCommentsSignificant
```
private boolean _bCommentsSignificant
```

_iType
```
public int _iType
```

_keyword
```
public Keyword _keyword
```

_strValue
```
public java.lang.String _strValue
```

_iInvalidCharPos
```
public int _iInvalidCharPos
```

_bUnterminatedString
```
public boolean _bUnterminatedString
```

_bUnterminatedComment
```
public boolean _bUnterminatedComment
```

_instructor

private ITokenizerInstructor _instructor

_lastComment
```
private DocCommentBlock _lastComment
```

_bParseDotsAsOperators
```
private boolean _bParseDotsAsOperators
```

_iLineOffset
```
private int _iLineOffset
```

_tokens
```
private Stack<Token> _tokens
```

_eof
```
private Token _eof
```

_supportsKeywords
```
private boolean _supportsKeywords
```

Constructor Detail

SourceCodeTokenizerInternal

SourceCodeTokenizerInternal(boolean initForCopy)

SourceCodeTokenizerInternal

public SourceCodeTokenizerInternal(java.lang.CharSequence sourceCode)

SourceCodeTokenizerInternal

public SourceCodeTokenizerInternal(java.io.Reader reader)

SourceCodeTokenizerInternal

public SourceCodeTokenizerInternal(SourceCodeReader reader)

SourceCodeTokenizerInternal

public SourceCodeTokenizerInternal(SourceCodeReader reader,
                                   ITokenizerInstructor instructor)

Method Detail

copy
```
public Token copy()
```

reset
```
public void reset()
```

reset

public void reset(java.io.Reader reader)

reset

public void reset(SourceCodeReader reader)

reset

void reset(SourceCodeReader reader,
           boolean bResetReader)

getReader
```
public SourceCodeReader getReader()
```

getSource
```
public java.lang.String getSource()
```

getInstructor

public ITokenizerInstructor getInstructor()

setInstructor

public void setInstructor(ITokenizerInstructor instructor)

isWhitespaceSignificant

public boolean isWhitespaceSignificant()

setWhitespaceSignificant

public void setWhitespaceSignificant(boolean bWhitespaceSignificant)

isCommentsSignificant

public boolean isCommentsSignificant()

setCommentsSignificant

public void setCommentsSignificant(boolean bCommentsSignificant)

getLineNumber
```
public int getLineNumber()
```

getLineOffset
```
public int getLineOffset()
```

incrementLineNumber
```
protected void incrementLineNumber()
```

getTokenColumn
```
public int getTokenColumn()
```

wordChars

public void wordChars(int iLow,
                      int iHigh)

whitespaceChars

public void whitespaceChars(int iLow,
                            int iHigh)

ordinaryChars

public void ordinaryChars(int iLow,
                          int iHigh)

ordinaryChar
```
public void ordinaryChar(int ch)
```

getDefaultOperators

public static java.lang.String[] getDefaultOperators()

getDefaultBindingOperators

public static java.util.List<java.lang.String> getDefaultBindingOperators()

getBitshiftOperators

public static java.lang.String[] getBitshiftOperators()

operators

public void operators(java.lang.String[] astrOperators)

setOperatorChars

private void setOperatorChars(java.util.Set<java.lang.String> operators)

operatorChars

public void operatorChars(int iLow,
                          int iHigh)

setBitshiftOperatorChars

private void setBitshiftOperatorChars(java.util.Set<java.lang.String> operators)

bitshiftOperatorChars

public void bitshiftOperatorChars(int iLow,
                                  int iHigh)

isOperator

public boolean isOperator(java.lang.String strOperator)

commentChar
```
public void commentChar(int ch)
```

quoteChar
```
public void quoteChar(int ch)
```

charQuoteChar
```
public void charQuoteChar(int ch)
```

parseNumbers
```
public void parseNumbers()
```

eolIsSignificant

public void eolIsSignificant(boolean bFlag)

lowerCaseMode

public void lowerCaseMode(boolean bLowerCaseMode)

isUnterminatedString
```
public boolean isUnterminatedString()
```

isUnterminatedComment

public boolean isUnterminatedComment()

setParseDotsAsOperators

public void setParseDotsAsOperators(boolean parseDotsAsOperators)

isParseDotsAsOperators

public boolean isParseDotsAsOperators()

popLastComment

public DocCommentBlock popLastComment()

getTokenStart
```
public int getTokenStart()
```

getTokenEnd
```
public int getTokenEnd()
```

read

private int read()
          throws java.io.IOException

Throws:: java.io.IOException

readOne

private int readOne()
             throws java.io.IOException

Throws:: java.io.IOException

readWithInstructions

protected int readWithInstructions(int c)
                            throws java.io.IOException

Throws:: java.io.IOException

pushWhitespaceToken

private void pushWhitespaceToken(java.lang.StringBuilder sbWhitespace)

pushToken
```
private void pushToken()
```

initEofToken
```
private void initEofToken()
```

initToken
```
Token initToken(Token token)
```

pushToken

private void pushToken(Token token)

isAnalyzingSeparately

public boolean isAnalyzingSeparately()

isAnalyzingDirective
```
public boolean isAnalyzingDirective()
```

getTokens
```
public final Stack<Token> getTokens()
```

rip
```
void rip()
```

nextToken

public int nextToken()
              throws java.io.IOException

Throws:: java.io.IOException

_nextTokenImpl

private int _nextTokenImpl()
                    throws java.io.IOException

Throws:: java.io.IOException

consumeBlockComment

private void consumeBlockComment()
                          throws java.io.IOException

Throws:: java.io.IOException

isReserved
```
private boolean isReserved()
```

isAtIgnorePos
```
private boolean isAtIgnorePos()
```

isHexDigit
```
private boolean isHexDigit(int c)
```

isLeftOpenIntervalOp

private boolean isLeftOpenIntervalOp(java.lang.String strOpTest)

stopOnDot
```
private boolean stopOnDot(int c)
```

toString
```
public java.lang.String toString()
```
Overrides:

toString in class java.lang.Object

getType
```
public int getType()
```

isEOF
```
public boolean isEOF()
```

isNOTHING
```
public boolean isNOTHING()
```

goToPosition

void goToPosition(int iOffset)
           throws java.io.IOException

Throws:: java.io.IOException

setTokens

void setTokens(Stack<Token> tokens)

getEofToken
```
public Token getEofToken()
```

isSupportsKeywords
```
public boolean isSupportsKeywords()
```

setSupportsKeywords

public void setSupportsKeywords(boolean supportsKeywords)

Class SourceCodeTokenizerInternal

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

CT_WHITESPACE

CT_DIGIT

CT_ALPHA

CT_QUOTE

CT_COMMENT

CT_OPERATOR

CT_BITSHIFT_OPERATOR

CT_CHARQUOTE

DEFAULT_OPERATORS

BITSHIFT_OPERATORS

_reader

_peekc

_bForceLower

_bEOLIsSignificantP

_ctype

_operators

_iPos

_iLineNum

_iColumn

_iTokenStart

_iTokenColumn

_bWhitespaceSignificant

_bCommentsSignificant

_iType

_keyword

_strValue

_iInvalidCharPos

_bUnterminatedString

_bUnterminatedComment

_instructor

_lastComment

_bParseDotsAsOperators

_iLineOffset

_tokens

_eof

_supportsKeywords

Constructor Detail

SourceCodeTokenizerInternal

SourceCodeTokenizerInternal

SourceCodeTokenizerInternal

SourceCodeTokenizerInternal

SourceCodeTokenizerInternal

Method Detail

copy

reset

reset

reset

reset

getReader

getSource

getInstructor

setInstructor

isWhitespaceSignificant

setWhitespaceSignificant

isCommentsSignificant

setCommentsSignificant

getLineNumber

getLineOffset

incrementLineNumber

getTokenColumn

wordChars

whitespaceChars

ordinaryChars

ordinaryChar

getDefaultOperators

getDefaultBindingOperators

getBitshiftOperators

operators

setOperatorChars

operatorChars

setBitshiftOperatorChars

bitshiftOperatorChars

isOperator

commentChar