class EBNF::LL1::Lexer::Token

Represents a lexer token.

@example Creating a new token

token = EBNF::LL1::Lexer::Token.new(:LANGTAG, "en")
token.type   #=> :LANGTAG
token.value  #=> "en"

@see en.wikipedia.org/wiki/Lexical_analysis#Token

Attributes

lineno[R]

The line number where the token was encountered.

@return [Integer]

options[R]

Any additional options for the token.

@return [Hash]

type[R]

The token’s symbol type.

@return [Symbol]

value[R]

The token’s value.

@return [String]

Public Class Methods

new(type, value, **options) click to toggle source

Initializes a new token instance.

@param [Symbol] type @param [String] value @param [Hash{Symbol => Object}] options @option options [Integer] :lineno (nil)

# File lib/ebnf/ll1/lexer.rb, line 382
def initialize(type, value, **options)
  @type = type.to_s.to_sym if type
  @value = value.to_s
  @options = options.dup
  @lineno  = @options.delete(:lineno)
end

Public Instance Methods

===(value) click to toggle source

Returns ‘true` if the given `value` matches either the type or value of this token.

@example Matching using the symbolic type

EBNF::LL1::Lexer::Token.new(:NIL) === :NIL     #=> true

@example Matching using the string value

EBNF::LL1::Lexer::Token.new(nil, "{") === "{"  #=> true

@param [Symbol, String] value @return [Boolean]

# File lib/ebnf/ll1/lexer.rb, line 415
def ===(value)
  case value
    when Symbol
      value == @type
    when ::String
      @value == (@options[:case_insensitive] ? value.to_s.downcase : value.to_s)
    else value == @value
  end
end
[](key) click to toggle source

Returns the attribute named by ‘key`.

@param [Symbol] key @return [Object]

# File lib/ebnf/ll1/lexer.rb, line 394
def [](key)
  key = key.to_s.to_sym unless key.is_a?(Integer) || key.is_a?(Symbol)
  case key
    when 0, :type    then @type
    when 1, :value   then @value
    else nil
  end
end
inspect() click to toggle source

Returns a developer-friendly representation of this token.

@return [String]

# File lib/ebnf/ll1/lexer.rb, line 457
def inspect
  "#{@value.inspect}#{'(' + @type.to_s + ')' if @type}"
end
representation() click to toggle source

Returns type, if not nil, otherwise value

# File lib/ebnf/ll1/lexer.rb, line 441
def representation
  @type ? @type : @value
end
to_a() click to toggle source

Returns an array representation of this token.

@return [Array]

# File lib/ebnf/ll1/lexer.rb, line 449
def to_a
  [@type, @value]
end
to_hash() click to toggle source

Returns a hash table representation of this token.

@return [Hash]

# File lib/ebnf/ll1/lexer.rb, line 429
def to_hash
  {type: @type, value: @value}
end
to_s() click to toggle source

Readable version of token

# File lib/ebnf/ll1/lexer.rb, line 435
def to_s
  @type ? @type.inspect : @value
end