class Tokenizer
Base Class for all Tokenizers Inherits from Array
Attributes
items[R]
parsed[RW]
Public Class Methods
new(str,args={})
click to toggle source
Base class version of initialize Will raise an exception if a Lexer is not passed in. Takes a string str and and hash arguments and creates a Lexical token reference of str It will also parse the lexical tokens into an array of items. :keep_escape determines weather or not to keep all escape backslashes, default false
Calls superclass method
# File libs/lexer.rb, line 426 def initialize(str,args={}) super() raise NoLexer.new if args[:lexer].nil? debug(8,:msg=>"Initial String",:var=>str.inspect) replace(str.lexer_parse(args[:lexer])) #replace self with array from lexer_parse debug(8,:msg=>"Tokens",:var=>self) debug(8,:msg=>"Tokens(Length)",:var=>length) @available_tokens=args[:lexer].available_tokens @class_options||=[] if @class_options.include?(:remove_whitespace) delete_if do |i| i.kind==:whitespace end end end
options(*options)
click to toggle source
Creates a factory to dynamicly generate a new descendant object with the options passed in, which are available via the variable @class_options *options : array of symbols
# File libs/lexer.rb, line 451 def self.options(*options) if options.nil? self else class_name=self.to_s+"_"+options.map {|i| i.to_s}.join str =<<-EOF class #{class_name} < self def initialize(str,args={}) @class_options=#{options.inspect} super(str,args) end end EOF eval(str) eval(class_name) end end
Public Instance Methods
parse(args={})
click to toggle source
# File libs/lexer.rb, line 444 def parse(args={}) raise BaseClassError.new end