A "manual" provider of tokens, returning tokens in a binary form.

Hierarchy

  • EncodedTokensProvider

Index

Methods

getInitialState

  • The initial state of a language. Will be the state passed in to tokenize the first line.

    Returns IState

tokenizeEncoded