module Drudge::Parsers::Tokenizer
tokenization of commandline arguments.
Public Instance Methods
tokenize(argv)
click to toggle source
tokenizes the arg-v list into an array of sexps the sexps are then suitable for the Drudge::parsers parser combinators
# File lib/drudge/parsers/tokenizer.rb, line 11 def tokenize(argv) argv.map.with_index do |arg, index| [:val, arg, loc(index, arg.length)] end end
underline_token(input, token, underline_char: '~')
click to toggle source
produces a string that underlines a specific token if no token is provided, the end of string is underlined
# File lib/drudge/parsers/tokenizer.rb, line 30 def underline_token(input, token, underline_char: '~') line = untokenize(input) if token _, _, meta = token location = meta[:loc] _, _, token_length = location white_space = index_of_sexp_in_untokenized(input, location) else white_space = line.length + 1 token_length = 1 underline_char = '^' end " " * white_space + underline_char * token_length end
untokenize(sexps)
click to toggle source
given an array of sexps (as returned by tokenize) produce a string representatio of that
# File lib/drudge/parsers/tokenizer.rb, line 19 def untokenize(sexps) sexps.map do |type, arg, *_| case type when :val arg end end.join(" ") end
Private Instance Methods
index_of_sexp_in_untokenized(input, loc)
click to toggle source
# File lib/drudge/parsers/tokenizer.rb, line 54 def index_of_sexp_in_untokenized(input, loc) l_index, l_start, l_len = loc prefix = if l_index == 0 0 else input[0..l_index - 1].map { |_, _, meta| meta[:loc] } .reduce(0) { |sum, (_, _, len)| sum + len + 1 } end prefix + l_start end
loc(index, start = 0, len)
click to toggle source
# File lib/drudge/parsers/tokenizer.rb, line 50 def loc(index, start = 0, len) {loc: [index, start, len]} end