class Parser::Lexer
line 3 “lib/parser/lexer.rl”
BEFORE YOU START ===¶ ↑
Read the Ruby Hacking Guide chapter 11, available in English at whitequark.org/blog/2013/04/01/ruby-hacking-guide-ch-11-finite-state-lexer/
Remember two things about Ragel scanners:
1) Longest match wins. 2) If two matches have the same length, the first in source code wins.
General rules of making Ragel and Bison happy:
* `p` (position) and `@te` contain the index of the character they're pointing to ("current"), plus one. `@ts` contains the index of the corresponding character. The code for extracting matched token is: @source_buffer.slice(@ts...@te) * If your input is `foooooooobar` and the rule is: 'f' 'o'+ the result will be: foooooooobar ^ ts=0 ^ p=te=9 * A Ragel lexer action should not emit more than one token, unless you know what you are doing. * All Ragel commands (fnext, fgoto, ...) end with a semicolon. * If an action emits the token and transitions to another state, use these Ragel commands: emit($whatever) fnext $next_state; fbreak; If you perform `fgoto` in an action which does not emit a token nor rewinds the stream pointer, the parser's side-effectful, context-sensitive lookahead actions will break in a hard to detect and debug way. * If an action does not emit a token: fgoto $next_state; * If an action features lookbehind, i.e. matches characters with the intent of passing them to another action: p = @ts - 1 fgoto $next_state; or, if the lookbehind consists of a single character: fhold; fgoto $next_state; * Ragel merges actions. So, if you have `e_lparen = '(' %act` and `c_lparen = '('` and a lexer action `e_lparen | c_lparen`, the result _will_ invoke the action `act`. e_something stands for "something with **e**mbedded action". * EOF is explicit and is matched by `c_eof`. If you want to introspect the state of the lexer, add this rule to the state: c_eof => do_eof; * If you proceed past EOF, the lexer will complain: NoMethodError: undefined method `ord' for nil:NilClass
line 3 “lib/parser/lexer.rl”
BEFORE YOU START ===¶ ↑
Read the Ruby Hacking Guide chapter 11, available in English at whitequark.org/blog/2013/04/01/ruby-hacking-guide-ch-11-finite-state-lexer/
Remember two things about Ragel scanners:
1) Longest match wins. 2) If two matches have the same length, the first in source code wins.
General rules of making Ragel and Bison happy:
* `p` (position) and `@te` contain the index of the character they're pointing to ("current"), plus one. `@ts` contains the index of the corresponding character. The code for extracting matched token is: @source_buffer.slice(@ts...@te) * If your input is `foooooooobar` and the rule is: 'f' 'o'+ the result will be: foooooooobar ^ ts=0 ^ p=te=9 * A Ragel lexer action should not emit more than one token, unless you know what you are doing. * All Ragel commands (fnext, fgoto, ...) end with a semicolon. * If an action emits the token and transitions to another state, use these Ragel commands: emit($whatever) fnext $next_state; fbreak; If you perform `fgoto` in an action which does not emit a token nor rewinds the stream pointer, the parser's side-effectful, context-sensitive lookahead actions will break in a hard to detect and debug way. * If an action does not emit a token: fgoto $next_state; * If an action features lookbehind, i.e. matches characters with the intent of passing them to another action: p = @ts - 1 fgoto $next_state; or, if the lookbehind consists of a single character: fhold; fgoto $next_state; * Ragel merges actions. So, if you have `e_lparen = '(' %act` and `c_lparen = '('` and a lexer action `e_lparen | c_lparen`, the result _will_ invoke the action `act`. e_something stands for "something with **e**mbedded action". * EOF is explicit and is matched by `c_eof`. If you want to introspect the state of the lexer, add this rule to the state: c_eof => do_eof; * If you proceed past EOF, the lexer will complain: NoMethodError: undefined method `ord' for nil:NilClass
Constants
- ESCAPE_WHITESPACE
- KEYWORDS
- KEYWORDS_BEGIN
- LEX_STATES
- PUNCTUATION
Mapping of strings to parser tokens.
- PUNCTUATION_BEGIN
Attributes
_lex_actions[RW]
_lex_eof_trans[RW]
_lex_from_state_actions[RW]
_lex_index_offsets[RW]
_lex_indicies[RW]
_lex_key_spans[RW]
_lex_to_state_actions[RW]
_lex_trans_actions[RW]
_lex_trans_keys[RW]
_lex_trans_targs[RW]
lex_en_expr_arg[RW]
lex_en_expr_beg[RW]
lex_en_expr_cmdarg[RW]
lex_en_expr_dot[RW]
lex_en_expr_end[RW]
lex_en_expr_endarg[RW]
lex_en_expr_endfn[RW]
lex_en_expr_fname[RW]
lex_en_expr_labelarg[RW]
lex_en_expr_mid[RW]
lex_en_expr_value[RW]
lex_en_expr_variable[RW]
lex_en_inside_string[RW]
lex_en_leading_dot[RW]
lex_en_line_begin[RW]
lex_en_line_comment[RW]
lex_error[RW]
lex_start[RW]
cmdarg[RW]
cmdarg_stack[R]
command_start[RW]
comments[RW]
cond[RW]
cond_stack[R]
context[RW]
diagnostics[RW]
force_utf32[RW]
lambda_stack[R]
paren_nest[R]
source_buffer[R]
line 82 “lib/parser/lexer.rl” %
static_env[RW]
tokens[RW]
version[R]
Public Class Methods
new(version)
click to toggle source
# File lib/parser/lexer-F0.rb, line 8382 def initialize(version) @version = version @static_env = nil @context = nil @tokens = nil @comments = nil @_lex_actions = if self.class.respond_to?(:_lex_actions, true) self.class.send :_lex_actions else [] end @emit_integer = lambda { |chars, p| emit(:tINTEGER, chars); p } @emit_rational = lambda { |chars, p| emit(:tRATIONAL, Rational(chars)); p } @emit_imaginary = lambda { |chars, p| emit(:tIMAGINARY, Complex(0, chars)); p } @emit_imaginary_rational = lambda { |chars, p| emit(:tIMAGINARY, Complex(0, Rational(chars))); p } @emit_integer_re = lambda { |chars, p| emit(:tINTEGER, chars, @ts, @te - 2); p - 2 } @emit_integer_if = lambda { |chars, p| emit(:tINTEGER, chars, @ts, @te - 2); p - 2 } @emit_integer_rescue = lambda { |chars, p| emit(:tINTEGER, chars, @ts, @te - 6); p - 6 } @emit_float = lambda { |chars, p| emit(:tFLOAT, Float(chars)); p } @emit_imaginary_float = lambda { |chars, p| emit(:tIMAGINARY, Complex(0, Float(chars))); p } @emit_float_if = lambda { |chars, p| emit(:tFLOAT, Float(chars), @ts, @te - 2); p - 2 } @emit_float_rescue = lambda { |chars, p| emit(:tFLOAT, Float(chars), @ts, @te - 6); p - 6 } reset end
Public Instance Methods
advance()
click to toggle source
Return next token: [type, value].
# File lib/parser/lexer-F0.rb, line 8542 def advance unless @token_queue.empty? return @token_queue.shift end # Ugly, but dependent on Ragel output. Consider refactoring it somehow. klass = self.class _lex_trans_keys = klass.send :_lex_trans_keys _lex_key_spans = klass.send :_lex_key_spans _lex_index_offsets = klass.send :_lex_index_offsets _lex_indicies = klass.send :_lex_indicies _lex_trans_targs = klass.send :_lex_trans_targs _lex_trans_actions = klass.send :_lex_trans_actions _lex_to_state_actions = klass.send :_lex_to_state_actions _lex_from_state_actions = klass.send :_lex_from_state_actions _lex_eof_trans = klass.send :_lex_eof_trans _lex_actions = @_lex_actions pe = @source_pts.size + 2 p, eof = @p, pe cmd_state = @command_start @command_start = false # line 8568 "lib/parser/lexer-F0.rb" begin # ragel flat testEof = false _slen, _trans, _keys, _inds, _acts, _nacts = nil _goto_level = 0 _resume = 10 _eof_trans = 15 _again = 20 _test_eof = 30 _out = 40 while true _trigger_goto = false if _goto_level <= 0 if p == pe _goto_level = _test_eof next end if @cs == 0 _goto_level = _out next end end if _goto_level <= _resume _acts = _lex_from_state_actions[ @cs] _nacts = _lex_actions[_acts] _acts += 1 while _nacts > 0 _nacts -= 1 _acts += 1 case _lex_actions[_acts - 1] when 52 then # line 1 "NONE" begin @ts = p end # line 8603 "lib/parser/lexer-F0.rb" end # from state action switch end if _trigger_goto next end _keys = @cs << 1 _inds = _lex_index_offsets[ @cs] _slen = _lex_key_spans[ @cs] _wide = ( (@source_pts[p] || 0)) _trans = if ( _slen > 0 && _lex_trans_keys[_keys] <= _wide && _wide <= _lex_trans_keys[_keys + 1] ) then _lex_indicies[ _inds + _wide - _lex_trans_keys[_keys] ] else _lex_indicies[ _inds + _slen ] end end if _goto_level <= _eof_trans @cs = _lex_trans_targs[_trans] if _lex_trans_actions[_trans] != 0 _acts = _lex_trans_actions[_trans] _nacts = _lex_actions[_acts] _acts += 1 while _nacts > 0 _nacts -= 1 _acts += 1 case _lex_actions[_acts - 1] when 0 then # line 539 "lib/parser/lexer.rl" begin # Record position of a newline for precise location reporting on tNL # tokens. # # This action is embedded directly into c_nl, as it is idempotent and # there are no cases when we need to skip it. @newline_s = p end when 1 then # line 673 "lib/parser/lexer.rl" begin @num_xfrm = @emit_integer end when 2 then # line 674 "lib/parser/lexer.rl" begin @num_xfrm = @emit_rational end when 3 then # line 675 "lib/parser/lexer.rl" begin @num_xfrm = @emit_imaginary end when 4 then # line 676 "lib/parser/lexer.rl" begin @num_xfrm = @emit_imaginary_rational end when 5 then # line 677 "lib/parser/lexer.rl" begin @num_xfrm = @emit_integer_re end when 6 then # line 678 "lib/parser/lexer.rl" begin @num_xfrm = @emit_integer_if end when 7 then # line 679 "lib/parser/lexer.rl" begin @num_xfrm = @emit_integer_rescue end when 8 then # line 682 "lib/parser/lexer.rl" begin @num_xfrm = @emit_float end when 9 then # line 683 "lib/parser/lexer.rl" begin @num_xfrm = @emit_imaginary_float end when 10 then # line 684 "lib/parser/lexer.rl" begin @num_xfrm = @emit_float_if end when 11 then # line 688 "lib/parser/lexer.rl" begin @num_xfrm = @emit_rational end when 12 then # line 689 "lib/parser/lexer.rl" begin @num_xfrm = @emit_imaginary_rational end when 13 then # line 690 "lib/parser/lexer.rl" begin @num_xfrm = @emit_float_rescue end when 14 then # line 696 "lib/parser/lexer.rl" begin e_lbrace end when 15 then # line 700 "lib/parser/lexer.rl" begin if @strings.close_interp_on_current_literal(p) p = p - 1; @cs = 128; begin p += 1 _trigger_goto = true _goto_level = _out break end end @paren_nest -= 1 end when 16 then # line 723 "lib/parser/lexer.rl" begin p = on_newline(p) end when 17 then # line 733 "lib/parser/lexer.rl" begin @sharp_s = p - 1 end when 18 then # line 736 "lib/parser/lexer.rl" begin emit_comment_from_range(p, pe) end when 19 then # line 777 "lib/parser/lexer.rl" begin tm = p end when 20 then # line 778 "lib/parser/lexer.rl" begin tm = p - 2 end when 21 then # line 783 "lib/parser/lexer.rl" begin tm = p end when 22 then # line 784 "lib/parser/lexer.rl" begin tm = p - 2 end when 23 then # line 785 "lib/parser/lexer.rl" begin tm = p - 2 end when 24 then # line 786 "lib/parser/lexer.rl" begin tm = p - 2 end when 25 then # line 787 "lib/parser/lexer.rl" begin tm = p - 3 end when 26 then # line 792 "lib/parser/lexer.rl" begin tm = p - 2 end when 27 then # line 797 "lib/parser/lexer.rl" begin tm = p - 2 end when 28 then # line 803 "lib/parser/lexer.rl" begin @cond.push(false); @cmdarg.push(false) @paren_nest += 1 end when 29 then # line 809 "lib/parser/lexer.rl" begin @paren_nest -= 1 end when 30 then # line 816 "lib/parser/lexer.rl" begin @cond.push(false); @cmdarg.push(false) @paren_nest += 1 if version?(18) @command_start = true end end when 31 then # line 826 "lib/parser/lexer.rl" begin @paren_nest -= 1 end when 32 then # line 1061 "lib/parser/lexer.rl" begin tm = p end when 33 then # line 1074 "lib/parser/lexer.rl" begin tm = p end when 34 then # line 1102 "lib/parser/lexer.rl" begin tm = p end when 35 then # line 1299 "lib/parser/lexer.rl" begin heredoc_e = p end when 36 then # line 1300 "lib/parser/lexer.rl" begin new_herebody_s = p end when 37 then # line 1400 "lib/parser/lexer.rl" begin tm = p - 1; diag_msg = :ivar_name end when 38 then # line 1401 "lib/parser/lexer.rl" begin tm = p - 2; diag_msg = :cvar_name end when 39 then # line 1488 "lib/parser/lexer.rl" begin tm = p end when 40 then # line 1595 "lib/parser/lexer.rl" begin ident_tok = tok; ident_ts = @ts; ident_te = @te; end when 41 then # line 1781 "lib/parser/lexer.rl" begin @num_base = 16; @num_digits_s = p end when 42 then # line 1782 "lib/parser/lexer.rl" begin @num_base = 10; @num_digits_s = p end when 43 then # line 1783 "lib/parser/lexer.rl" begin @num_base = 8; @num_digits_s = p end when 44 then # line 1784 "lib/parser/lexer.rl" begin @num_base = 2; @num_digits_s = p end when 45 then # line 1785 "lib/parser/lexer.rl" begin @num_base = 10; @num_digits_s = @ts end when 46 then # line 1786 "lib/parser/lexer.rl" begin @num_base = 8; @num_digits_s = @ts end when 47 then # line 1787 "lib/parser/lexer.rl" begin @num_suffix_s = p end when 48 then # line 1830 "lib/parser/lexer.rl" begin @num_suffix_s = p end when 49 then # line 1831 "lib/parser/lexer.rl" begin @num_suffix_s = p end when 50 then # line 2034 "lib/parser/lexer.rl" begin tm = p end when 53 then # line 1 "NONE" begin @te = p+1 end when 54 then # line 848 "lib/parser/lexer.rl" begin @te = p+1 begin emit_global_var @cs = (stack_pop); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 55 then # line 848 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_global_var @cs = (stack_pop); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 56 then # line 855 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_class_var @cs = (stack_pop); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 57 then # line 862 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_instance_var @cs = (stack_pop); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 58 then # line 880 "lib/parser/lexer.rl" begin @act = 4; end when 59 then # line 884 "lib/parser/lexer.rl" begin @act = 5; end when 60 then # line 888 "lib/parser/lexer.rl" begin @act = 6; end when 61 then # line 880 "lib/parser/lexer.rl" begin @te = p+1 begin emit_table(KEYWORDS_BEGIN); @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 62 then # line 888 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tIDENTIFIER) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 63 then # line 892 "lib/parser/lexer.rl" begin @te = p+1 begin p = @ts - 1 @cs = 516; begin @stack[ @top] = @cs @top+= 1 @cs = 129 _trigger_goto = true _goto_level = _again break end end end when 64 then # line 901 "lib/parser/lexer.rl" begin @te = p+1 begin emit_table(PUNCTUATION) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 65 then # line 905 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 66 then # line 911 "lib/parser/lexer.rl" begin @te = p+1 begin if version?(23) type, delimiter = tok[0..-2], tok[-1].chr @strings.push_literal(type, delimiter, @ts) begin @cs = 128 _trigger_goto = true _goto_level = _again break end else p = @ts - 1 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end end when 67 then # line 925 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 68 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 69 then # line 880 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(KEYWORDS_BEGIN); @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 70 then # line 884 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tCONSTANT) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 71 then # line 888 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tIDENTIFIER) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 72 then # line 892 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = @ts - 1 @cs = 516; begin @stack[ @top] = @cs @top+= 1 @cs = 129 _trigger_goto = true _goto_level = _again break end end end when 73 then # line 901 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(PUNCTUATION) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 74 then # line 908 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 75 then # line 922 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 76 then # line 925 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 77 then # line 901 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin emit_table(PUNCTUATION) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 78 then # line 925 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 79 then # line 1 "NONE" begin case @act when 4 then begin begin p = (( @te))-1; end emit_table(KEYWORDS_BEGIN); @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 5 then begin begin p = (( @te))-1; end emit(:tCONSTANT) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 6 then begin begin p = (( @te))-1; end emit(:tIDENTIFIER) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 80 then # line 937 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tLABEL, tok(@ts, @te - 2), @ts, @te - 1) p = p - 1; @cs = 501; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 81 then # line 941 "lib/parser/lexer.rl" begin @te = p+1 begin if @version >= 31 && @context.in_argdef emit(:tBDOT3, '...'.freeze) # emit(:tNL, "\n".freeze, @te - 1, @te) @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end else p -= 3; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end end when 82 then # line 955 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 83 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 84 then # line 952 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 85 then # line 955 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 86 then # line 955 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 87 then # line 981 "lib/parser/lexer.rl" begin @te = p+1 begin emit_table(PUNCTUATION) @cs = 276; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 88 then # line 987 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 89 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 90 then # line 966 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tCONSTANT) @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 91 then # line 970 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tIDENTIFIER) @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 92 then # line 974 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tFID, tok(@ts, tm), @ts, tm) @cs = (arg_or_cmdarg(cmd_state)); p = tm - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 93 then # line 981 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(PUNCTUATION) @cs = 276; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 94 then # line 984 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 95 then # line 987 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 96 then # line 1065 "lib/parser/lexer.rl" begin @act = 33; end when 97 then # line 1075 "lib/parser/lexer.rl" begin @act = 34; end when 98 then # line 1114 "lib/parser/lexer.rl" begin @act = 39; end when 99 then # line 1119 "lib/parser/lexer.rl" begin @act = 40; end when 100 then # line 1047 "lib/parser/lexer.rl" begin @te = p+1 begin # Unlike expr_beg as invoked in the next rule, do not warn p = @ts - 1 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 101 then # line 1065 "lib/parser/lexer.rl" begin @te = p+1 begin check_ambiguous_slash(tm) p = tm - 1 begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 102 then # line 1086 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 103 then # line 1094 "lib/parser/lexer.rl" begin @te = p+1 begin p = @ts - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 104 then # line 1103 "lib/parser/lexer.rl" begin @te = p+1 begin p = tm - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 105 then # line 1114 "lib/parser/lexer.rl" begin @te = p+1 begin p = @ts - 1 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 106 then # line 1128 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 107 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 108 then # line 1003 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if version?(18) emit(:tLPAREN2, '('.freeze, @te - 1, @te) @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end else emit(:tLPAREN_ARG, '('.freeze, @te - 1, @te) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 109 then # line 1016 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tLPAREN2, '('.freeze) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 110 then # line 1022 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tLBRACK, '['.freeze, @te - 1, @te) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 111 then # line 1028 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if @lambda_stack.last == @paren_nest @lambda_stack.pop emit(:tLAMBEG, '{'.freeze, @te - 1, @te) else emit(:tLCURLY, '{'.freeze, @te - 1, @te) end @command_start = true @paren_nest += 1 @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 112 then # line 1056 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 113 then # line 1075 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin diagnostic :warning, :ambiguous_prefix, { :prefix => tok(tm, @te) }, range(tm, @te) p = tm - 1 begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 114 then # line 1091 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 115 then # line 1114 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = @ts - 1 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 116 then # line 1119 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 117 then # line 1122 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 118 then # line 1125 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 119 then # line 1128 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 120 then # line 1119 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end end when 121 then # line 1128 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 122 then # line 1 "NONE" begin case @act when 33 then begin begin p = (( @te))-1; end check_ambiguous_slash(tm) p = tm - 1 begin @cs = 345 _trigger_goto = true _goto_level = _again break end end when 34 then begin begin p = (( @te))-1; end diagnostic :warning, :ambiguous_prefix, { :prefix => tok(tm, @te) }, range(tm, @te) p = tm - 1 begin @cs = 345 _trigger_goto = true _goto_level = _again break end end when 39 then begin begin p = (( @te))-1; end p = @ts - 1 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end else begin begin p = (( @te))-1; end end end end when 123 then # line 1151 "lib/parser/lexer.rl" begin @act = 46; end when 124 then # line 1164 "lib/parser/lexer.rl" begin @act = 47; end when 125 then # line 1164 "lib/parser/lexer.rl" begin @te = p+1 begin p = @ts - 1 begin @cs = 276 _trigger_goto = true _goto_level = _again break end end end when 126 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 127 then # line 1141 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tLPAREN_ARG, '('.freeze, @te - 1, @te) if version?(18) @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end else @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 128 then # line 1164 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = @ts - 1 begin @cs = 276 _trigger_goto = true _goto_level = _again break end end end when 129 then # line 1164 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin p = @ts - 1 begin @cs = 276 _trigger_goto = true _goto_level = _again break end end end when 130 then # line 1 "NONE" begin case @act when 46 then begin begin p = (( @te))-1; end if @cond.active? emit(:kDO_COND, 'do'.freeze, @te - 2, @te) else emit(:kDO, 'do'.freeze, @te - 2, @te) end @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 47 then begin begin p = (( @te))-1; end p = @ts - 1 begin @cs = 276 _trigger_goto = true _goto_level = _again break end end end end when 131 then # line 1200 "lib/parser/lexer.rl" begin @te = p+1 begin emit_do(true) @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 132 then # line 1206 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 133 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 134 then # line 1187 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if @lambda_stack.last == @paren_nest @lambda_stack.pop emit(:tLAMBEG, '{'.freeze) else emit(:tLBRACE_ARG, '{'.freeze) end @paren_nest += 1 @command_start = true @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 135 then # line 1203 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 136 then # line 1206 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 137 then # line 1218 "lib/parser/lexer.rl" begin @act = 54; end when 138 then # line 1222 "lib/parser/lexer.rl" begin @act = 55; end when 139 then # line 1230 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 140 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 141 then # line 1222 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = @ts - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 142 then # line 1224 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 143 then # line 1227 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 144 then # line 1230 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 145 then # line 1 "NONE" begin case @act when 54 then begin begin p = (( @te))-1; end emit_table(KEYWORDS) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 55 then begin begin p = (( @te))-1; end p = @ts - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end end when 146 then # line 1245 "lib/parser/lexer.rl" begin @act = 60; end when 147 then # line 1348 "lib/parser/lexer.rl" begin @act = 67; end when 148 then # line 1442 "lib/parser/lexer.rl" begin @act = 76; end when 149 then # line 1483 "lib/parser/lexer.rl" begin @act = 80; end when 150 then # line 1489 "lib/parser/lexer.rl" begin @act = 81; end when 151 then # line 1495 "lib/parser/lexer.rl" begin @act = 82; end when 152 then # line 1586 "lib/parser/lexer.rl" begin @act = 86; end when 153 then # line 831 "lib/parser/lexer.rl" begin @act = 87; end when 154 then # line 1632 "lib/parser/lexer.rl" begin @act = 91; end when 155 then # line 1245 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tUNARY_NUM, tok(@ts, @ts + 1), @ts, @ts + 1) p = p - 1; @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 156 then # line 1262 "lib/parser/lexer.rl" begin @te = p+1 begin type = delimiter = tok[0].chr @strings.push_literal(type, delimiter, @ts) p = p - 1; begin @cs = 128 _trigger_goto = true _goto_level = _again break end end end when 157 then # line 1272 "lib/parser/lexer.rl" begin @te = p+1 begin type, delimiter = @source_buffer.slice(@ts, 1).chr, tok[-1].chr @strings.push_literal(type, delimiter, @ts) begin @cs = 128 _trigger_goto = true _goto_level = _again break end end end when 158 then # line 1280 "lib/parser/lexer.rl" begin @te = p+1 begin type, delimiter = tok[0..-2], tok[-1].chr @strings.push_literal(type, delimiter, @ts) begin @cs = 128 _trigger_goto = true _goto_level = _again break end end end when 159 then # line 1357 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; p = p - 1; emit(:tSYMBEG, tok(@ts, @ts + 1), @ts, @ts + 1) begin @cs = 134 _trigger_goto = true _goto_level = _again break end end end when 160 then # line 1365 "lib/parser/lexer.rl" begin @te = p+1 begin type, delimiter = tok, tok[-1].chr @strings.push_literal(type, delimiter, @ts); begin @cs = 128 _trigger_goto = true _goto_level = _again break end end end when 161 then # line 1375 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tSYMBOL, tok(@ts + 1, @ts + 2)) @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 162 then # line 1389 "lib/parser/lexer.rl" begin @te = p+1 begin gvar_name = tok(@ts + 1) if @version >= 33 && gvar_name.start_with?('$0') && gvar_name.length > 2 diagnostic :error, :gvar_name, { :name => gvar_name }, range(@ts + 1, @te) end emit(:tSYMBOL, gvar_name, @ts) @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 163 then # line 1416 "lib/parser/lexer.rl" begin @te = p+1 begin p, next_state = @strings.read_character_constant(@ts) p = p - 1; # Ragel will do `p += 1` to consume input, prevent it # If strings lexer founds a character constant (?a) emit it, # otherwise read ternary operator if @token_queue.empty? begin @cs = (next_state) _trigger_goto = true _goto_level = _again break end else @cs = (next_state); begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 164 then # line 1431 "lib/parser/lexer.rl" begin @te = p+1 begin diagnostic :fatal, :incomplete_escape, nil, range(@ts, @ts + 1) end end when 165 then # line 1483 "lib/parser/lexer.rl" begin @te = p+1 begin emit_table(PUNCTUATION_BEGIN) begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 166 then # line 1504 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; if version?(18) ident = tok(@ts, @te - 2) emit((@source_buffer.slice(@ts, 1) =~ /[A-Z]/) ? :tCONSTANT : :tIDENTIFIER, ident, @ts, @te - 2) p = p - 1; # continue as a symbol if !@static_env.nil? && @static_env.declared?(ident) @cs = 516; else @cs = (arg_or_cmdarg(cmd_state)); end else emit(:tLABEL, tok(@ts, @te - 2), @ts, @te - 1) @cs = 501; end begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 167 then # line 1542 "lib/parser/lexer.rl" begin @te = p+1 begin # Here we scan and conditionally emit "\n": # + if it's there # + and emitted we do nothing # + and not emitted we return `p` to "\n" to process it on the next scan # + if it's not there we do nothing followed_by_nl = @te - 1 == @newline_s nl_emitted = false dots_te = followed_by_nl ? @te - 1 : @te if @version >= 30 if @lambda_stack.any? && @lambda_stack.last + 1 == @paren_nest # To reject `->(...)` like `->...` emit(:tDOT3, '...'.freeze, @ts, dots_te) else emit(:tBDOT3, '...'.freeze, @ts, dots_te) if @version >= 31 && followed_by_nl && @context.in_argdef emit(:tNL, @te - 1, @te) nl_emitted = true end end elsif @version >= 27 emit(:tBDOT3, '...'.freeze, @ts, dots_te) else emit(:tDOT3, '...'.freeze, @ts, dots_te) end if followed_by_nl && !nl_emitted # return "\n" to process it on the next scan p = p - 1; end @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 168 then # line 1597 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tIDENTIFIER, ident_tok, ident_ts, ident_te) p = ident_te - 1 if !@static_env.nil? && @static_env.declared?(ident_tok) && @version < 25 @cs = 247; else @cs = 307; end begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 169 then # line 1616 "lib/parser/lexer.rl" begin @te = p+1 begin p = @ts - 1 @cs_before_block_comment = @cs begin @cs = 710 _trigger_goto = true _goto_level = _again break end end end when 170 then # line 1632 "lib/parser/lexer.rl" begin @te = p+1 begin p = @ts - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 171 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 172 then # line 1245 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tUNARY_NUM, tok(@ts, @ts + 1), @ts, @ts + 1) p = p - 1; @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 173 then # line 1252 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tSTAR, '*'.freeze) begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 174 then # line 1287 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin diagnostic :fatal, :string_eof, nil, range(@ts, @ts + 1) end end when 175 then # line 1301 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin tok(@ts, heredoc_e) =~ /^<<(-?)(~?)(["'`]?)(.*)\3$/m indent = !$1.empty? || !$2.empty? dedent_body = !$2.empty? type = $3.empty? ? '<<"'.freeze : ('<<'.freeze + $3) delimiter = $4 if @version >= 27 if delimiter.count("\n") > 0 || delimiter.count("\r") > 0 diagnostic :error, :unterminated_heredoc_id, nil, range(@ts, @ts + 1) end elsif @version >= 24 if delimiter.count("\n") > 0 if delimiter.end_with?("\n") diagnostic :warning, :heredoc_id_ends_with_nl, nil, range(@ts, @ts + 1) delimiter = delimiter.rstrip else diagnostic :fatal, :heredoc_id_has_newline, nil, range(@ts, @ts + 1) end end end if dedent_body && version?(18, 19, 20, 21, 22) emit(:tLSHFT, '<<'.freeze, @ts, @ts + 2) p = @ts + 1 @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end else @strings.push_literal(type, delimiter, @ts, heredoc_e, indent, dedent_body); @strings.herebody_s ||= new_herebody_s p = @strings.herebody_s - 1 @cs = 128; end end end when 176 then # line 1348 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin diagnostic :error, :unterminated_heredoc_id, nil, range(@ts, @ts + 1) end end when 177 then # line 1381 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tSYMBOL, tok(@ts + 1, tm), @ts, tm) p = tm - 1 @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 178 then # line 1389 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin gvar_name = tok(@ts + 1) if @version >= 33 && gvar_name.start_with?('$0') && gvar_name.length > 2 diagnostic :error, :gvar_name, { :name => gvar_name }, range(@ts + 1, @te) end emit(:tSYMBOL, gvar_name, @ts) @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 179 then # line 1403 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_colon_with_digits(p, tm, diag_msg) @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 180 then # line 1431 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin diagnostic :fatal, :incomplete_escape, nil, range(@ts, @ts + 1) end end when 181 then # line 1459 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if @lambda_stack.last == @paren_nest @lambda_stack.pop @command_start = true emit(:tLAMBEG, '{'.freeze) else emit(:tLBRACE, '{'.freeze) end @paren_nest += 1 begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 182 then # line 1473 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tLBRACK, '['.freeze) begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 183 then # line 1478 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tLPAREN, '('.freeze) begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 184 then # line 1483 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(PUNCTUATION_BEGIN) begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 185 then # line 1489 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:kRESCUE, 'rescue'.freeze, @ts, tm) p = tm - 1 @cs = 321; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 186 then # line 1531 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if @version >= 27 emit(:tBDOT2) else emit(:tDOT2) end @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 187 then # line 1542 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin # Here we scan and conditionally emit "\n": # + if it's there # + and emitted we do nothing # + and not emitted we return `p` to "\n" to process it on the next scan # + if it's not there we do nothing followed_by_nl = @te - 1 == @newline_s nl_emitted = false dots_te = followed_by_nl ? @te - 1 : @te if @version >= 30 if @lambda_stack.any? && @lambda_stack.last + 1 == @paren_nest # To reject `->(...)` like `->...` emit(:tDOT3, '...'.freeze, @ts, dots_te) else emit(:tBDOT3, '...'.freeze, @ts, dots_te) if @version >= 31 && followed_by_nl && @context.in_argdef emit(:tNL, @te - 1, @te) nl_emitted = true end end elsif @version >= 27 emit(:tBDOT3, '...'.freeze, @ts, dots_te) else emit(:tDOT3, '...'.freeze, @ts, dots_te) end if followed_by_nl && !nl_emitted # return "\n" to process it on the next scan p = p - 1; end @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 188 then # line 1586 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = @ts - 1 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 189 then # line 831 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tIDENTIFIER) if !@static_env.nil? && @static_env.declared?(tok) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end elsif @version >= 32 && tok =~ /\A_[1-9]\z/ @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end else @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 190 then # line 1613 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 191 then # line 1616 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = @ts - 1 @cs_before_block_comment = @cs begin @cs = 710 _trigger_goto = true _goto_level = _again break end end end when 192 then # line 1632 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = @ts - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 193 then # line 1287 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin diagnostic :fatal, :string_eof, nil, range(@ts, @ts + 1) end end when 194 then # line 1348 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin diagnostic :error, :unterminated_heredoc_id, nil, range(@ts, @ts + 1) end end when 195 then # line 831 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin emit(:tIDENTIFIER) if !@static_env.nil? && @static_env.declared?(tok) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end elsif @version >= 32 && tok =~ /\A_[1-9]\z/ @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end else @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 196 then # line 1613 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end end when 197 then # line 1632 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin p = @ts - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 198 then # line 1 "NONE" begin case @act when 60 then begin begin p = (( @te))-1; end emit(:tUNARY_NUM, tok(@ts, @ts + 1), @ts, @ts + 1) p = p - 1; @cs = 516; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 67 then begin begin p = (( @te))-1; end diagnostic :error, :unterminated_heredoc_id, nil, range(@ts, @ts + 1) end when 76 then begin begin p = (( @te))-1; end if @version >= 27 emit(:tPIPE, tok(@ts, @ts + 1), @ts, @ts + 1) p = p - 1; @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end else p -= 2 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 80 then begin begin p = (( @te))-1; end emit_table(PUNCTUATION_BEGIN) begin p += 1 _trigger_goto = true _goto_level = _out break end end when 81 then begin begin p = (( @te))-1; end emit(:kRESCUE, 'rescue'.freeze, @ts, tm) p = tm - 1 @cs = 321; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 82 then begin begin p = (( @te))-1; end emit_table(KEYWORDS_BEGIN) @command_start = true @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 86 then begin begin p = (( @te))-1; end p = @ts - 1 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end when 87 then begin begin p = (( @te))-1; end emit(:tIDENTIFIER) if !@static_env.nil? && @static_env.declared?(tok) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end elsif @version >= 32 && tok =~ /\A_[1-9]\z/ @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end else @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 91 then begin begin p = (( @te))-1; end p = @ts - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end end when 199 then # line 1652 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 200 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 201 then # line 1640 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 202 then # line 1643 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if @context.in_kwarg p = p - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end else begin @cs = 710 _trigger_goto = true _goto_level = _again break end end end end when 203 then # line 1652 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 204 then # line 1662 "lib/parser/lexer.rl" begin @te = p+1 begin p = @ts - 1 begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 205 then # line 1667 "lib/parser/lexer.rl" begin @te = p+1 begin @strings.push_literal(tok, tok, @ts) begin @cs = 128 _trigger_goto = true _goto_level = _again break end end end when 206 then # line 1678 "lib/parser/lexer.rl" begin @te = p+1 begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 207 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 208 then # line 1672 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 209 then # line 1675 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin begin @cs = 710 _trigger_goto = true _goto_level = _again break end end end when 210 then # line 1678 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 211 then # line 1678 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin p = p - 1; begin @cs = 345 _trigger_goto = true _goto_level = _again break end end end when 212 then # line 1697 "lib/parser/lexer.rl" begin @act = 104; end when 213 then # line 1726 "lib/parser/lexer.rl" begin @act = 105; end when 214 then # line 1730 "lib/parser/lexer.rl" begin @act = 106; end when 215 then # line 1735 "lib/parser/lexer.rl" begin @act = 107; end when 216 then # line 1740 "lib/parser/lexer.rl" begin @act = 108; end when 217 then # line 1745 "lib/parser/lexer.rl" begin @act = 109; end when 218 then # line 1749 "lib/parser/lexer.rl" begin @act = 110; end when 219 then # line 1760 "lib/parser/lexer.rl" begin @act = 111; end when 220 then # line 1774 "lib/parser/lexer.rl" begin @act = 112; end when 221 then # line 1788 "lib/parser/lexer.rl" begin @act = 113; end when 222 then # line 1806 "lib/parser/lexer.rl" begin @act = 115; end when 223 then # line 1818 "lib/parser/lexer.rl" begin @act = 116; end when 224 then # line 1833 "lib/parser/lexer.rl" begin @act = 117; end when 225 then # line 1862 "lib/parser/lexer.rl" begin @act = 119; end when 226 then # line 831 "lib/parser/lexer.rl" begin @act = 123; end when 227 then # line 1884 "lib/parser/lexer.rl" begin @act = 124; end when 228 then # line 1910 "lib/parser/lexer.rl" begin @act = 126; end when 229 then # line 1916 "lib/parser/lexer.rl" begin @act = 127; end when 230 then # line 1689 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tLAMBDA, '->'.freeze, @ts, @ts + 2) @lambda_stack.push @paren_nest @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 231 then # line 1730 "lib/parser/lexer.rl" begin @te = p+1 begin emit_singleton_class @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 232 then # line 1851 "lib/parser/lexer.rl" begin @te = p+1 begin type, delimiter = tok, tok[-1].chr @strings.push_literal(type, delimiter, @ts, nil, false, false, true); begin @cs = 128 _trigger_goto = true _goto_level = _again break end end end when 233 then # line 1870 "lib/parser/lexer.rl" begin @te = p+1 begin p = @ts - 1; begin @stack[ @top] = @cs @top+= 1 @cs = 129 _trigger_goto = true _goto_level = _again break end end end when 234 then # line 1877 "lib/parser/lexer.rl" begin @te = p+1 begin emit_table(PUNCTUATION) @cs = 255; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 235 then # line 1901 "lib/parser/lexer.rl" begin @te = p+1 begin emit_table(PUNCTUATION) @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 236 then # line 1910 "lib/parser/lexer.rl" begin @te = p+1 begin emit_table(PUNCTUATION); @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 237 then # line 1937 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tOP_ASGN, tok(@ts, @te - 1)) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 238 then # line 1941 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tEH, '?'.freeze) @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 239 then # line 1949 "lib/parser/lexer.rl" begin @te = p+1 begin if @paren_nest == 0 diagnostic :warning, :triple_dot_at_eol, nil, range(@ts, @te - 1) end emit(:tDOT3, '...'.freeze, @ts, @te - 1) p = p - 1; @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 240 then # line 1960 "lib/parser/lexer.rl" begin @te = p+1 begin emit_table(PUNCTUATION) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 241 then # line 1973 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tSEMI, ';'.freeze) @command_start = true @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 242 then # line 1977 "lib/parser/lexer.rl" begin @te = p+1 begin diagnostic :error, :bare_backslash, nil, range(@ts, @ts + 1) p = p - 1; end end when 243 then # line 1983 "lib/parser/lexer.rl" begin @te = p+1 begin diagnostic :fatal, :unexpected, { :character => tok.inspect[1..-2] } end end when 244 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 245 then # line 1697 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if @lambda_stack.last == @paren_nest @lambda_stack.pop if tok == '{'.freeze emit(:tLAMBEG, '{'.freeze) else # 'do' emit(:kDO_LAMBDA, 'do'.freeze) end else if tok == '{'.freeze emit(:tLCURLY, '{'.freeze) else # 'do' emit_do end end if tok == '{'.freeze @paren_nest += 1 end @command_start = true @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 246 then # line 1726 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(KEYWORDS) @cs = 134; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 247 then # line 1730 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_singleton_class @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 248 then # line 1740 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(KEYWORDS) @command_start = true @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 249 then # line 1788 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin digits = numeric_literal_int if version?(18, 19, 20) emit(:tINTEGER, digits.to_i(@num_base), @ts, @num_suffix_s) p = @num_suffix_s - 1 else p = @num_xfrm.call(digits.to_i(@num_base), p) end begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 250 then # line 1801 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin diagnostic :error, :no_dot_digit_literal end end when 251 then # line 1833 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin digits = tok(@ts, @num_suffix_s) if version?(18, 19, 20) emit(:tFLOAT, Float(digits), @ts, @num_suffix_s) p = @num_suffix_s - 1 else p = @num_xfrm.call(digits, p) end begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 252 then # line 1862 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tCONSTANT) @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 253 then # line 1866 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tCONSTANT, tok(@ts, tm), @ts, tm) p = tm - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 254 then # line 1870 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = @ts - 1; begin @stack[ @top] = @cs @top+= 1 @cs = 129 _trigger_goto = true _goto_level = _again break end end end when 255 then # line 1877 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(PUNCTUATION) @cs = 255; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 256 then # line 831 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tIDENTIFIER) if !@static_env.nil? && @static_env.declared?(tok) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end elsif @version >= 32 && tok =~ /\A_[1-9]\z/ @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end else @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 257 then # line 1884 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if tm == @te # Suffix was consumed, e.g. foo! emit(:tFID) else # Suffix was not consumed, e.g. foo!= emit(:tIDENTIFIER, tok(@ts, tm), @ts, tm) p = tm - 1 end @cs = 276; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 258 then # line 1901 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(PUNCTUATION) @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 259 then # line 1910 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(PUNCTUATION); @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 260 then # line 1916 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(PUNCTUATION) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 261 then # line 1920 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_rbrace_rparen_rbrack if tok == '}'.freeze || tok == ']'.freeze if @version >= 25 @cs = 516; else @cs = 313; end else # ) # fnext expr_endfn; ? end begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 262 then # line 1945 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tLBRACK2, '['.freeze) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 263 then # line 1960 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_table(PUNCTUATION) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 264 then # line 1967 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 265 then # line 1970 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin begin @cs = 696 _trigger_goto = true _goto_level = _again break end end end when 266 then # line 1983 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin diagnostic :fatal, :unexpected, { :character => tok.inspect[1..-2] } end end when 267 then # line 1788 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin digits = numeric_literal_int if version?(18, 19, 20) emit(:tINTEGER, digits.to_i(@num_base), @ts, @num_suffix_s) p = @num_suffix_s - 1 else p = @num_xfrm.call(digits.to_i(@num_base), p) end begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 268 then # line 1801 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin diagnostic :error, :no_dot_digit_literal end end when 269 then # line 1833 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin digits = tok(@ts, @num_suffix_s) if version?(18, 19, 20) emit(:tFLOAT, Float(digits), @ts, @num_suffix_s) p = @num_suffix_s - 1 else p = @num_xfrm.call(digits, p) end begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 270 then # line 1983 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin diagnostic :fatal, :unexpected, { :character => tok.inspect[1..-2] } end end when 271 then # line 1 "NONE" begin case @act when 104 then begin begin p = (( @te))-1; end if @lambda_stack.last == @paren_nest @lambda_stack.pop if tok == '{'.freeze emit(:tLAMBEG, '{'.freeze) else # 'do' emit(:kDO_LAMBDA, 'do'.freeze) end else if tok == '{'.freeze emit(:tLCURLY, '{'.freeze) else # 'do' emit_do end end if tok == '{'.freeze @paren_nest += 1 end @command_start = true @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 105 then begin begin p = (( @te))-1; end emit_table(KEYWORDS) @cs = 134; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 106 then begin begin p = (( @te))-1; end emit_singleton_class @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 107 then begin begin p = (( @te))-1; end emit_table(KEYWORDS) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 108 then begin begin p = (( @te))-1; end emit_table(KEYWORDS) @command_start = true @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 109 then begin begin p = (( @te))-1; end emit_table(KEYWORDS) @cs = 321; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 110 then begin begin p = (( @te))-1; end emit_table(KEYWORDS) if version?(18) && tok == 'not'.freeze @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end else @cs = 276; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 111 then begin begin p = (( @te))-1; end if version?(18) emit(:tIDENTIFIER) unless !@static_env.nil? && @static_env.declared?(tok) @cs = (arg_or_cmdarg(cmd_state)); end else emit(:k__ENCODING__, '__ENCODING__'.freeze) end begin p += 1 _trigger_goto = true _goto_level = _out break end end when 112 then begin begin p = (( @te))-1; end emit_table(KEYWORDS) begin p += 1 _trigger_goto = true _goto_level = _out break end end when 113 then begin begin p = (( @te))-1; end digits = numeric_literal_int if version?(18, 19, 20) emit(:tINTEGER, digits.to_i(@num_base), @ts, @num_suffix_s) p = @num_suffix_s - 1 else p = @num_xfrm.call(digits.to_i(@num_base), p) end begin p += 1 _trigger_goto = true _goto_level = _out break end end when 115 then begin begin p = (( @te))-1; end if version?(18, 19, 20) diagnostic :error, :trailing_in_number, { :character => tok(@te - 1, @te) }, range(@te - 1, @te) else emit(:tINTEGER, tok(@ts, @te - 1).to_i, @ts, @te - 1) p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 116 then begin begin p = (( @te))-1; end if version?(18, 19, 20) diagnostic :error, :trailing_in_number, { :character => tok(@te - 1, @te) }, range(@te - 1, @te) else emit(:tFLOAT, tok(@ts, @te - 1).to_f, @ts, @te - 1) p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 117 then begin begin p = (( @te))-1; end digits = tok(@ts, @num_suffix_s) if version?(18, 19, 20) emit(:tFLOAT, Float(digits), @ts, @num_suffix_s) p = @num_suffix_s - 1 else p = @num_xfrm.call(digits, p) end begin p += 1 _trigger_goto = true _goto_level = _out break end end when 119 then begin begin p = (( @te))-1; end emit(:tCONSTANT) @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end when 123 then begin begin p = (( @te))-1; end emit(:tIDENTIFIER) if !@static_env.nil? && @static_env.declared?(tok) @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end elsif @version >= 32 && tok =~ /\A_[1-9]\z/ @cs = 247; begin p += 1 _trigger_goto = true _goto_level = _out break end else @cs = (arg_or_cmdarg(cmd_state)); begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 124 then begin begin p = (( @te))-1; end if tm == @te # Suffix was consumed, e.g. foo! emit(:tFID) else # Suffix was not consumed, e.g. foo!= emit(:tIDENTIFIER, tok(@ts, tm), @ts, tm) p = tm - 1 end @cs = 276; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 126 then begin begin p = (( @te))-1; end emit_table(PUNCTUATION); @cs = 508; begin p += 1 _trigger_goto = true _goto_level = _out break end end when 127 then begin begin p = (( @te))-1; end emit_table(PUNCTUATION) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 272 then # line 1999 "lib/parser/lexer.rl" begin @act = 140; end when 273 then # line 2038 "lib/parser/lexer.rl" begin @act = 144; end when 274 then # line 2024 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tNL, nil, @newline_s, @newline_s + 1) if @version < 27 p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end else emit(:tBDOT3) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 275 then # line 2035 "lib/parser/lexer.rl" begin @te = p+1 begin p = tm - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 276 then # line 2038 "lib/parser/lexer.rl" begin @te = p+1 begin emit(:tNL, nil, @newline_s, @newline_s + 1) p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 277 then # line 1999 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin if @version < 27 # Ruby before 2.7 doesn't support comments before leading dot. # If a line after "a" starts with a comment then "a" is a self-contained statement. # So in that case we emit a special tNL token and start reading the # next line as a separate statement. # # Note: block comments before leading dot are not supported on any version of Ruby. emit(:tNL, nil, @newline_s, @newline_s + 1) p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 278 then # line 2013 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tNL, nil, @newline_s, @newline_s + 1) if @version < 27 p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end else emit(:tBDOT2) @cs = 345; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 279 then # line 2035 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin p = tm - 1; begin @cs = 516 _trigger_goto = true _goto_level = _again break end end end when 280 then # line 2038 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit(:tNL, nil, @newline_s, @newline_s + 1) p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 281 then # line 1999 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin if @version < 27 # Ruby before 2.7 doesn't support comments before leading dot. # If a line after "a" starts with a comment then "a" is a self-contained statement. # So in that case we emit a special tNL token and start reading the # next line as a separate statement. # # Note: block comments before leading dot are not supported on any version of Ruby. emit(:tNL, nil, @newline_s, @newline_s + 1) p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 282 then # line 2038 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin emit(:tNL, nil, @newline_s, @newline_s + 1) p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 283 then # line 1 "NONE" begin case @act when 140 then begin begin p = (( @te))-1; end if @version < 27 # Ruby before 2.7 doesn't support comments before leading dot. # If a line after "a" starts with a comment then "a" is a self-contained statement. # So in that case we emit a special tNL token and start reading the # next line as a separate statement. # # Note: block comments before leading dot are not supported on any version of Ruby. emit(:tNL, nil, @newline_s, @newline_s + 1) p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 144 then begin begin p = (( @te))-1; end emit(:tNL, nil, @newline_s, @newline_s + 1) p = p - 1; @cs = 710; begin p += 1 _trigger_goto = true _goto_level = _out break end end end end when 284 then # line 2048 "lib/parser/lexer.rl" begin @te = p+1 begin emit_comment(@eq_begin_s, @te) begin @cs = (@cs_before_block_comment) _trigger_goto = true _goto_level = _again break end end end when 285 then # line 2053 "lib/parser/lexer.rl" begin @te = p+1 end when 286 then # line 2048 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin emit_comment(@eq_begin_s, @te) begin @cs = (@cs_before_block_comment) _trigger_goto = true _goto_level = _again break end end end when 287 then # line 2056 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin diagnostic :fatal, :embedded_document, nil, range(@eq_begin_s, @eq_begin_s + '=begin'.length) end end when 288 then # line 2066 "lib/parser/lexer.rl" begin @te = p+1 begin @eq_begin_s = @ts begin @cs = 704 _trigger_goto = true _goto_level = _again break end end end when 289 then # line 2070 "lib/parser/lexer.rl" begin @te = p+1 begin p = pe - 3 end end when 290 then # line 2073 "lib/parser/lexer.rl" begin @te = p+1 begin cmd_state = true; p = p - 1; begin @cs = 508 _trigger_goto = true _goto_level = _again break end end end when 291 then # line 566 "lib/parser/lexer.rl" begin @te = p+1 begin # Sit at EOF indefinitely. #advance would return $eof each time. # This allows to feed the lexer more data if needed; this is only used # in tests. # # Note that this action is not embedded into e_eof like e_nl and e_bs # below. This is due to the fact that scanner state at EOF is observed # by tests, and encapsulating it in a rule would break the introspection. p = p - 1; begin p += 1 _trigger_goto = true _goto_level = _out break end end end when 292 then # line 2063 "lib/parser/lexer.rl" begin @te = p p = p - 1; end when 293 then # line 2066 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin @eq_begin_s = @ts begin @cs = 704 _trigger_goto = true _goto_level = _again break end end end when 294 then # line 2073 "lib/parser/lexer.rl" begin @te = p p = p - 1; begin cmd_state = true; p = p - 1; begin @cs = 508 _trigger_goto = true _goto_level = _again break end end end when 295 then # line 2073 "lib/parser/lexer.rl" begin begin p = (( @te))-1; end begin cmd_state = true; p = p - 1; begin @cs = 508 _trigger_goto = true _goto_level = _again break end end end when 296 then # line 2080 "lib/parser/lexer.rl" begin @te = p+1 begin p, next_state = @strings.advance(p) p = p - 1; # Ragel will do `p += 1` to consume input, prevent it @cs = (next_state); begin p += 1 _trigger_goto = true _goto_level = _out break end end end # line 12621 "lib/parser/lexer-F0.rb" end # action switch end end if _trigger_goto next end end if _goto_level <= _again _acts = _lex_to_state_actions[ @cs] _nacts = _lex_actions[_acts] _acts += 1 while _nacts > 0 _nacts -= 1 _acts += 1 case _lex_actions[_acts - 1] when 51 then # line 1 "NONE" begin @ts = nil; end # line 12641 "lib/parser/lexer-F0.rb" end # to state action switch end if _trigger_goto next end if @cs == 0 _goto_level = _out next end p += 1 if p != pe _goto_level = _resume next end end if _goto_level <= _test_eof if p == eof if _lex_eof_trans[ @cs] > 0 _trans = _lex_eof_trans[ @cs] - 1; _goto_level = _eof_trans next; end end end if _goto_level <= _out break end end end # line 281 "lib/parser/lexer.rl" # % # Ragel creates a local variable called `testEof` but it doesn't use # it in any assignment. This dead code is here to swallow the warning. # It has no runtime cost because Ruby doesn't produce any instructions from it. if false testEof end @p = p if @token_queue.any? @token_queue.shift elsif @cs == klass.lex_error [ false, [ '$error'.freeze, range(p - 1, p) ] ] else eof = @source_pts.size [ false, [ '$eof'.freeze, range(eof, eof) ] ] end end
dedent_level()
click to toggle source
# File lib/parser/lexer-F0.rb, line 8537 def dedent_level @strings.dedent_level end
encoding()
click to toggle source
# File lib/parser/lexer-F0.rb, line 8490 def encoding @source_buffer.source.encoding end
pop_cmdarg()
click to toggle source
# File lib/parser/lexer-F0.rb, line 8524 def pop_cmdarg @cmdarg = @cmdarg_stack.pop end
pop_cond()
click to toggle source
# File lib/parser/lexer-F0.rb, line 8533 def pop_cond @cond = @cond_stack.pop end
push_cmdarg()
click to toggle source
# File lib/parser/lexer-F0.rb, line 8519 def push_cmdarg @cmdarg_stack.push(@cmdarg) @cmdarg = StackState.new("cmdarg.#{@cmdarg_stack.count}") end
push_cond()
click to toggle source
# File lib/parser/lexer-F0.rb, line 8528 def push_cond @cond_stack.push(@cond) @cond = StackState.new("cond.#{@cond_stack.count}") end
reset(reset_state=true)
click to toggle source
# File lib/parser/lexer-F0.rb, line 8413 def reset(reset_state=true) # Ragel state: if reset_state # Unit tests set state prior to resetting lexer. @cs = self.class.lex_en_line_begin @cond = StackState.new('cond') @cmdarg = StackState.new('cmdarg') @cond_stack = [] @cmdarg_stack = [] end @force_utf32 = false # Set to true by some tests @source_pts = nil # @source as a codepoint array @p = 0 # stream position (saved manually in #advance) @ts = nil # token start @te = nil # token end @act = 0 # next action @stack = [] # state stack @top = 0 # state stack top pointer # Lexer state: @token_queue = [] @eq_begin_s = nil # location of last encountered =begin @sharp_s = nil # location of last encountered # @newline_s = nil # location of last encountered newline @num_base = nil # last numeric base @num_digits_s = nil # starting position of numeric digits @num_suffix_s = nil # starting position of numeric suffix @num_xfrm = nil # numeric suffix-induced transformation # Ruby 1.9 ->() lambdas emit a distinct token if do/{ is # encountered after a matching closing parenthesis. @paren_nest = 0 @lambda_stack = [] # If the lexer is in `command state' (aka expr_value) # at the entry to #advance, it will transition to expr_cmdarg # instead of expr_arg at certain points. @command_start = true # State before =begin / =end block comment @cs_before_block_comment = self.class.lex_en_line_begin @strings = Parser::LexerStrings.new(self, @version) end
source_buffer=(source_buffer)
click to toggle source
# File lib/parser/lexer-F0.rb, line 8466 def source_buffer=(source_buffer) @source_buffer = source_buffer if @source_buffer source = @source_buffer.source if source.encoding == Encoding::UTF_8 @source_pts = source.unpack('U*') else @source_pts = source.unpack('C*') end if @source_pts[0] == 0xfeff # Skip byte order mark. @p = 1 end else @source_pts = nil end @strings.source_buffer = @source_buffer @strings.source_pts = @source_pts end
state()
click to toggle source
# File lib/parser/lexer-F0.rb, line 8511 def state LEX_STATES.invert.fetch(@cs, @cs) end
state=(state)
click to toggle source
# File lib/parser/lexer-F0.rb, line 8515 def state=(state) @cs = LEX_STATES.fetch(state) end
Protected Instance Methods
arg_or_cmdarg(cmd_state)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12738 def arg_or_cmdarg(cmd_state) if cmd_state self.class.lex_en_expr_cmdarg else self.class.lex_en_expr_arg end end
check_ambiguous_slash(tm)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12800 def check_ambiguous_slash(tm) if tok(tm, tm + 1) == '/'.freeze # Ambiguous regexp literal. if @version < 30 diagnostic :warning, :ambiguous_literal, nil, range(tm, tm + 1) else diagnostic :warning, :ambiguous_regexp, nil, range(tm, tm + 1) end end end
diagnostic(type, reason, arguments=nil, location=range, highlights=[])
click to toggle source
# File lib/parser/lexer-F0.rb, line 12762 def diagnostic(type, reason, arguments=nil, location=range, highlights=[]) @diagnostics.process( Parser::Diagnostic.new(type, reason, arguments, location, highlights)) end
e_lbrace()
click to toggle source
# File lib/parser/lexer-F0.rb, line 12768 def e_lbrace @cond.push(false); @cmdarg.push(false) current_literal = @strings.literal if current_literal current_literal.start_interp_brace end end
emit(type, value = tok, s = @ts, e = @te)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12712 def emit(type, value = tok, s = @ts, e = @te) token = [ type, [ value, range(s, e) ] ] @token_queue.push(token) @tokens.push(token) if @tokens token end
emit_class_var(ts = @ts, te = @te)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12821 def emit_class_var(ts = @ts, te = @te) if tok(ts, te) =~ /^@@[0-9]/ diagnostic :error, :cvar_name, { :name => tok(ts, te) } end emit(:tCVAR, tok(ts, te), ts, te) end
emit_colon_with_digits(p, tm, diag_msg)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12849 def emit_colon_with_digits(p, tm, diag_msg) if @version >= 27 diagnostic :error, diag_msg, { name: tok(tm, @te) }, range(tm, @te) else emit(:tCOLON, tok(@ts, @ts + 1), @ts, @ts + 1) p = @ts end p end
emit_comment(s = @ts, e = @te)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12746 def emit_comment(s = @ts, e = @te) if @comments @comments.push(Parser::Source::Comment.new(range(s, e))) end if @tokens @tokens.push([ :tCOMMENT, [ tok(s, e), range(s, e) ] ]) end nil end
emit_comment_from_range(p, pe)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12758 def emit_comment_from_range(p, pe) emit_comment(@sharp_s, p == pe ? p - 2 : p) end
emit_do(do_block=false)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12728 def emit_do(do_block=false) if @cond.active? emit(:kDO_COND, 'do'.freeze) elsif @cmdarg.active? || do_block emit(:kDO_BLOCK, 'do'.freeze) else emit(:kDO, 'do'.freeze) end end
emit_global_var(ts = @ts, te = @te)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12811 def emit_global_var(ts = @ts, te = @te) if tok(ts, te) =~ /^\$([1-9][0-9]*)$/ emit(:tNTH_REF, tok(ts + 1, te).to_i, ts, te) elsif tok =~ /^\$([&`'+])$/ emit(:tBACK_REF, tok(ts, te), ts, te) else emit(:tGVAR, tok(ts, te), ts, te) end end
emit_instance_var(ts = @ts, te = @te)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12829 def emit_instance_var(ts = @ts, te = @te) if tok(ts, te) =~ /^@[0-9]/ diagnostic :error, :ivar_name, { :name => tok(ts, te) } end emit(:tIVAR, tok(ts, te), ts, te) end
emit_rbrace_rparen_rbrack()
click to toggle source
# File lib/parser/lexer-F0.rb, line 12837 def emit_rbrace_rparen_rbrack emit_table(PUNCTUATION) if @version < 24 @cond.lexpop @cmdarg.lexpop else @cond.pop @cmdarg.pop end end
emit_singleton_class()
click to toggle source
# File lib/parser/lexer-F0.rb, line 12859 def emit_singleton_class emit(:kCLASS, 'class'.freeze, @ts, @ts + 5) emit(:tLSHFT, '<<'.freeze, @te - 2, @te) end
emit_table(table, s = @ts, e = @te)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12722 def emit_table(table, s = @ts, e = @te) value = tok(s, e) emit(table[value], value, s, e) end
numeric_literal_int()
click to toggle source
# File lib/parser/lexer-F0.rb, line 12777 def numeric_literal_int digits = tok(@num_digits_s, @num_suffix_s) if digits.end_with? '_'.freeze diagnostic :error, :trailing_in_number, { :character => '_'.freeze }, range(@te - 1, @te) elsif digits.empty? && @num_base == 8 && version?(18) # 1.8 did not raise an error on 0o. digits = '0'.freeze elsif digits.empty? diagnostic :error, :empty_numeric elsif @num_base == 8 && (invalid_idx = digits.index(/[89]/)) invalid_s = @num_digits_s + invalid_idx diagnostic :error, :invalid_octal, nil, range(invalid_s, invalid_s + 1) end digits end
on_newline(p)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12796 def on_newline(p) @strings.on_newline(p) end
range(s = @ts, e = @te)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12708 def range(s = @ts, e = @te) Parser::Source::Range.new(@source_buffer, s, e) end
stack_pop()
click to toggle source
# File lib/parser/lexer-F0.rb, line 12699 def stack_pop @top -= 1 @stack[@top] end
tok(s = @ts, e = @te)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12704 def tok(s = @ts, e = @te) @source_buffer.slice(s, e - s) end
version?(*versions)
click to toggle source
# File lib/parser/lexer-F0.rb, line 12695 def version?(*versions) versions.include?(@version) end