module Drudge::Parsers::Tokenizer

tokenization of commandline arguments.

Public Instance Methods

tokenize(argv) click to toggle source

tokenizes the arg-v list into an array of sexps the sexps are then suitable for the Drudge::parsers parser combinators

# File lib/drudge/parsers/tokenizer.rb, line 11
def tokenize(argv)
  argv.map.with_index do |arg, index|
    [:val, arg, loc(index, arg.length)]
  end
end
underline_token(input, token, underline_char: '~') click to toggle source

produces a string that underlines a specific token if no token is provided, the end of string is underlined

# File lib/drudge/parsers/tokenizer.rb, line 30
def underline_token(input, token, underline_char: '~')
  line                 = untokenize(input)

  if token
    _, _, meta         = token
    location           = meta[:loc]
    _, _, token_length = location
    white_space        = index_of_sexp_in_untokenized(input, location)
  else
    white_space        = line.length + 1
    token_length       = 1
    underline_char     = '^'
  end

  " " * white_space + underline_char * token_length
end
untokenize(sexps) click to toggle source

given an array of sexps (as returned by tokenize) produce a string representatio of that

# File lib/drudge/parsers/tokenizer.rb, line 19
def untokenize(sexps)
  sexps.map do |type, arg, *_|
    case type
    when :val
      arg
    end
  end.join(" ")
end

Private Instance Methods

index_of_sexp_in_untokenized(input, loc) click to toggle source
# File lib/drudge/parsers/tokenizer.rb, line 54
def index_of_sexp_in_untokenized(input, loc)
  l_index, l_start, l_len = loc

  prefix =
  if l_index == 0 
    0
  else 
    input[0..l_index - 1].map       { |_, _, meta|       meta[:loc]    }
    .reduce(0) { |sum, (_, _, len)| sum + len + 1 } 
  end

  prefix + l_start
end
loc(index, start = 0, len) click to toggle source
# File lib/drudge/parsers/tokenizer.rb, line 50
def loc(index, start = 0, len)
  {loc: [index, start, len]}
end