class Tokenizer

Base Class for all Tokenizers Inherits from Array

Attributes

items[R]
parsed[RW]

Public Class Methods

new(str,args={}) click to toggle source

Base class version of initialize Will raise an exception if a Lexer is not passed in. Takes a string str and and hash arguments and creates a Lexical token reference of str It will also parse the lexical tokens into an array of items. :keep_escape determines weather or not to keep all escape backslashes, default false

Calls superclass method
# File libs/lexer.rb, line 426
def initialize(str,args={})
  super()
  raise NoLexer.new if args[:lexer].nil?
  debug(8,:msg=>"Initial String",:var=>str.inspect)
  replace(str.lexer_parse(args[:lexer]))  #replace self with array from lexer_parse
  debug(8,:msg=>"Tokens",:var=>self)
  debug(8,:msg=>"Tokens(Length)",:var=>length)
  @available_tokens=args[:lexer].available_tokens
  @class_options||=[]

  if @class_options.include?(:remove_whitespace)
    delete_if do |i|
      i.kind==:whitespace
    end
  end

end
options(*options) click to toggle source

Creates a factory to dynamicly generate a new descendant object with the options passed in, which are available via the variable @class_options *options : array of symbols

# File libs/lexer.rb, line 451
  def self.options(*options)
    if options.nil?
      self
    else
      class_name=self.to_s+"_"+options.map {|i| i.to_s}.join
      str =<<-EOF
        class #{class_name} < self
          def initialize(str,args={})
            @class_options=#{options.inspect}
            super(str,args)
          end
        end
      EOF
      eval(str)
      eval(class_name)
    end
  end

Public Instance Methods

parse(args={}) click to toggle source
# File libs/lexer.rb, line 444
def parse(args={})
  raise BaseClassError.new
end