class SimpleTemplates::Lexer
The SimpleTemplates::Lexer
tokenizes the raw input into a more usable form for the SimpleTemplates::Parser
.
Constants
- Token
A
Struct
for a Lexer::Token that takes the type, the content and position of a token.- VALID_PLACEHOLDER
A Hash with the allowed
Regexp
for a valid placeholder name @return [Hash{Symbol => Regexp}] a hash with the allowed Regexp for theplaceholder name +:ph_name+
Public Class Methods
Initializes a new Lexer
@param delimiter [SimpleTemplates::Delimiter] a delimiter object @param input [String] a raw input for the lexer
# File lib/simple_templates/lexer.rb, line 22 def initialize(delimiter, input) @input = input.clone.freeze @matchers = delimiter.to_h.merge(text: /./m).freeze @matchers_with_placeholder_name = VALID_PLACEHOLDER.merge(@matchers) end
Public Instance Methods
Tokenizes the raw input and returns a list of tokens. @return <Array>
# File lib/simple_templates/lexer.rb, line 30 def tokenize tokens = [] ss = StringScanner.new(@input) until ss.eos? tok = next_token(tokens, ss) if tokens.any? && tok.type == :text && tokens.last.type == :text tokens.last.content += tok.content else tokens << tok end end tokens end
Private Instance Methods
Checks if the last token was the start of a placeholder to use include
the placeholder name +:ph_name+ in the hash of matchers
@param tokens <Array> the list of tokens @return [Hash {Symbol => Regexp}] (see next_token
)
# File lib/simple_templates/lexer.rb, line 65 def current_matchers(tokens) if tokens.any? && tokens.last.type == :ph_start @matchers_with_placeholder_name else @matchers end end
Returns a new token and moves to the next position in the StringScanner
@param tokens <Array> list of tokens @param ss [StringScanner] a StringScanner
for the input @return [SimpleTemplates::Lexer::Token] the next token
# File lib/simple_templates/lexer.rb, line 53 def next_token(tokens, ss) pos = ss.pos token_type, _ = current_matchers(tokens).find { |_, pattern| ss.scan(pattern) } Token.new(token_type, ss.matched, pos) end