class SimpleTemplates::Lexer

The SimpleTemplates::Lexer tokenizes the raw input into a more usable form for the SimpleTemplates::Parser.

Constants

Token

A Struct for a Lexer::Token that takes the type, the content and position of a token.

VALID_PLACEHOLDER

A Hash with the allowed Regexp for a valid placeholder name @return [Hash{Symbol => Regexp}] a hash with the allowed Regexp for the

placeholder name +:ph_name+

Public Class Methods

new(delimiter, input) click to toggle source

Initializes a new Lexer @param delimiter [SimpleTemplates::Delimiter] a delimiter object @param input [String] a raw input for the lexer

# File lib/simple_templates/lexer.rb, line 22
def initialize(delimiter, input)
  @input    = input.clone.freeze
  @matchers = delimiter.to_h.merge(text: /./m).freeze
  @matchers_with_placeholder_name = VALID_PLACEHOLDER.merge(@matchers)
end

Public Instance Methods

tokenize() click to toggle source

Tokenizes the raw input and returns a list of tokens. @return <Array>

# File lib/simple_templates/lexer.rb, line 30
def tokenize
  tokens = []
  ss = StringScanner.new(@input)

  until ss.eos?
    tok = next_token(tokens, ss)

    if tokens.any? && tok.type == :text && tokens.last.type == :text
      tokens.last.content += tok.content
    else
      tokens << tok
    end
  end

  tokens
end

Private Instance Methods

current_matchers(tokens) click to toggle source

Checks if the last token was the start of a placeholder to use include

the placeholder name +:ph_name+ in the hash of matchers

@param tokens <Array> the list of tokens @return [Hash {Symbol => Regexp}] (see next_token)

# File lib/simple_templates/lexer.rb, line 65
def current_matchers(tokens)
  if tokens.any? && tokens.last.type == :ph_start
    @matchers_with_placeholder_name
  else
    @matchers
  end
end
next_token(tokens, ss) click to toggle source

Returns a new token and moves to the next position in the StringScanner @param tokens <Array> list of tokens @param ss [StringScanner] a StringScanner for the input @return [SimpleTemplates::Lexer::Token] the next token

# File lib/simple_templates/lexer.rb, line 53
def next_token(tokens, ss)
  pos = ss.pos
  token_type, _ = current_matchers(tokens).find { |_, pattern| ss.scan(pattern) }

  Token.new(token_type, ss.matched, pos)
end