class Puppet::Pops::Parser::Lexer2
Constants
- KEYWORDS
Keywords are all singleton tokens with pre calculated lengths. Booleans are pre-calculated (rather than evaluating the strings “false” “true” repeatedly.
- KEYWORD_NAMES
Reverse lookup of keyword name to string
- PATTERN_BARE_WORD
- PATTERN_CLASSREF
The NAME and CLASSREF in 4x are strict. Each segment must start with a letter a-z and may not contain dashes (w includes letters, digits and _).
- PATTERN_COMMENT
The single line comment includes the line ending.
- PATTERN_DOLLAR_VAR
- PATTERN_MLCOMMENT
- PATTERN_NAME
- PATTERN_NON_WS
- PATTERN_NUMBER
- PATTERN_REGEX
- PATTERN_REGEX_A
- PATTERN_REGEX_END
- PATTERN_REGEX_ESC
- PATTERN_REGEX_Z
- PATTERN_WS
- STRING_BSLASH_SLASH
PERFORMANCE NOTE: Comparison against a frozen string is faster (than unfrozen).
- TOKEN_APPENDS
- TOKEN_AT
- TOKEN_ATAT
- TOKEN_COLON
- TOKEN_COMMA
- TOKEN_DELETES
- TOKEN_DIV
- TOKEN_DOT
- TOKEN_DQMID
- TOKEN_DQPOS
- TOKEN_DQPRE
- TOKEN_EPPEND
- TOKEN_EPPEND_TRIM
- TOKEN_EPPSTART
EPP_START is currently a marker token, may later get syntax
- TOKEN_EQUALS
- TOKEN_FARROW
- TOKEN_GREATEREQUAL
- TOKEN_GREATERTHAN
- TOKEN_HEREDOC
HEREDOC has syntax as an argument.
- TOKEN_IN_EDGE
- TOKEN_IN_EDGE_SUB
- TOKEN_ISEQUAL
- TOKEN_LBRACE
- TOKEN_LBRACK
ALl tokens have three slots, the token name (a Symbol), the token text (String), and a token text length. All operator and punctuation tokens reuse singleton arrays Tokens that require unique values create a unique array per token.
PEFORMANCE NOTES: This construct reduces the amount of object that needs to be created for operators and punctuation. The length is pre-calculated for all singleton tokens. The length is used both to signal the length of the token, and to advance the scanner position (without having to advance it with a scan(regexp)).
- TOKEN_LCOLLECT
- TOKEN_LESSEQUAL
- TOKEN_LESSTHAN
- TOKEN_LISTSTART
- TOKEN_LLCOLLECT
- TOKEN_LPAREN
- TOKEN_LSHIFT
- TOKEN_MATCH
- TOKEN_MINUS
- TOKEN_MODULO
- TOKEN_NOMATCH
- TOKEN_NOT
- TOKEN_NOTEQUAL
- TOKEN_NUMBER
- TOKEN_OTHER
This is used for unrecognized tokens, will always be a single character. This particular instance is not used, but is kept here for documentation purposes.
- TOKEN_OUT_EDGE
- TOKEN_OUT_EDGE_SUB
- TOKEN_PARROW
- TOKEN_PIPE
- TOKEN_PLUS
- TOKEN_QMARK
- TOKEN_RBRACE
- TOKEN_RBRACK
- TOKEN_RCOLLECT
- TOKEN_REGEXP
- TOKEN_RPAREN
- TOKEN_RRCOLLECT
- TOKEN_RSHIFT
- TOKEN_SELBRACE
- TOKEN_SEMIC
- TOKEN_STRING
Tokens that are always unique to what has been lexed
- TOKEN_TILDE
- TOKEN_TIMES
- TOKEN_VARIABLE
- TOKEN_VARIABLE_EMPTY
- TOKEN_WORD
- TOKEN_WSLPAREN
Attributes
Public Class Methods
# File lib/puppet/pops/parser/lexer2.rb 182 def initialize() 183 @selector = { 184 '.' => lambda { emit(TOKEN_DOT, @scanner.pos) }, 185 ',' => lambda { emit(TOKEN_COMMA, @scanner.pos) }, 186 '[' => lambda do 187 before = @scanner.pos 188 # Must check the preceding character to see if it is whitespace. 189 # The fastest thing to do is to simply byteslice to get the string ending at the offset before 190 # and then check what the last character is. (This is the same as what an locator.char_offset needs 191 # to compute, but with less overhead of trying to find out the global offset from a local offset in the 192 # case when this is sublocated in a heredoc). 193 if before == 0 || @scanner.string.byteslice(0, before)[-1] =~ /[[:blank:]\r\n]+/ 194 emit(TOKEN_LISTSTART, before) 195 else 196 emit(TOKEN_LBRACK, before) 197 end 198 end, 199 ']' => lambda { emit(TOKEN_RBRACK, @scanner.pos) }, 200 '(' => lambda do 201 before = @scanner.pos 202 # If first on a line, or only whitespace between start of line and '(' 203 # then the token is special to avoid being taken as start of a call. 204 line_start = @lexing_context[:line_lexical_start] 205 if before == line_start || @scanner.string.byteslice(line_start, before - line_start) =~ /\A[[:blank:]\r]+\Z/ 206 emit(TOKEN_WSLPAREN, before) 207 else 208 emit(TOKEN_LPAREN, before) 209 end 210 end, 211 ')' => lambda { emit(TOKEN_RPAREN, @scanner.pos) }, 212 ';' => lambda { emit(TOKEN_SEMIC, @scanner.pos) }, 213 '?' => lambda { emit(TOKEN_QMARK, @scanner.pos) }, 214 '*' => lambda { emit(TOKEN_TIMES, @scanner.pos) }, 215 '%' => lambda do 216 scn = @scanner 217 before = scn.pos 218 la = scn.peek(2) 219 if la[1] == '>' && @lexing_context[:epp_mode] 220 scn.pos += 2 221 if @lexing_context[:epp_mode] == :expr 222 enqueue_completed(TOKEN_EPPEND, before) 223 end 224 @lexing_context[:epp_mode] = :text 225 interpolate_epp 226 else 227 emit(TOKEN_MODULO, before) 228 end 229 end, 230 '{' => lambda do 231 # The lexer needs to help the parser since the technology used cannot deal with 232 # lookahead of same token with different precedence. This is solved by making left brace 233 # after ? into a separate token. 234 # 235 @lexing_context[:brace_count] += 1 236 emit(if @lexing_context[:after] == :QMARK 237 TOKEN_SELBRACE 238 else 239 TOKEN_LBRACE 240 end, @scanner.pos) 241 end, 242 '}' => lambda do 243 @lexing_context[:brace_count] -= 1 244 emit(TOKEN_RBRACE, @scanner.pos) 245 end, 246 247 248 # TOKENS @, @@, @( 249 '@' => lambda do 250 scn = @scanner 251 la = scn.peek(2) 252 if la[1] == '@' 253 emit(TOKEN_ATAT, scn.pos) # TODO; Check if this is good for the grammar 254 elsif la[1] == '(' 255 heredoc 256 else 257 emit(TOKEN_AT, scn.pos) 258 end 259 end, 260 261 # TOKENS |, |>, |>> 262 '|' => lambda do 263 scn = @scanner 264 la = scn.peek(3) 265 emit(la[1] == '>' ? (la[2] == '>' ? TOKEN_RRCOLLECT : TOKEN_RCOLLECT) : TOKEN_PIPE, scn.pos) 266 end, 267 268 # TOKENS =, =>, ==, =~ 269 '=' => lambda do 270 scn = @scanner 271 la = scn.peek(2) 272 emit(case la[1] 273 when '=' 274 TOKEN_ISEQUAL 275 when '>' 276 TOKEN_FARROW 277 when '~' 278 TOKEN_MATCH 279 else 280 TOKEN_EQUALS 281 end, scn.pos) 282 end, 283 284 # TOKENS '+', '+=', and '+>' 285 '+' => lambda do 286 scn = @scanner 287 la = scn.peek(2) 288 emit(case la[1] 289 when '=' 290 TOKEN_APPENDS 291 when '>' 292 TOKEN_PARROW 293 else 294 TOKEN_PLUS 295 end, scn.pos) 296 end, 297 298 # TOKENS '-', '->', and epp '-%>' (end of interpolation with trim) 299 '-' => lambda do 300 scn = @scanner 301 la = scn.peek(3) 302 before = scn.pos 303 if @lexing_context[:epp_mode] && la[1] == '%' && la[2] == '>' 304 scn.pos += 3 305 if @lexing_context[:epp_mode] == :expr 306 enqueue_completed(TOKEN_EPPEND_TRIM, before) 307 end 308 interpolate_epp(:with_trim) 309 else 310 emit(case la[1] 311 when '>' 312 TOKEN_IN_EDGE 313 when '=' 314 TOKEN_DELETES 315 else 316 TOKEN_MINUS 317 end, before) 318 end 319 end, 320 321 # TOKENS !, !=, !~ 322 '!' => lambda do 323 scn = @scanner 324 la = scn.peek(2) 325 emit(case la[1] 326 when '=' 327 TOKEN_NOTEQUAL 328 when '~' 329 TOKEN_NOMATCH 330 else 331 TOKEN_NOT 332 end, scn.pos) 333 end, 334 335 # TOKENS ~>, ~ 336 '~' => lambda do 337 scn = @scanner 338 la = scn.peek(2) 339 emit(la[1] == '>' ? TOKEN_IN_EDGE_SUB : TOKEN_TILDE, scn.pos) 340 end, 341 342 '#' => lambda { @scanner.skip(PATTERN_COMMENT); nil }, 343 344 # TOKENS '/', '/*' and '/ regexp /' 345 '/' => lambda do 346 scn = @scanner 347 la = scn.peek(2) 348 if la[1] == '*' 349 lex_error(Issues::UNCLOSED_MLCOMMENT) if scn.skip(PATTERN_MLCOMMENT).nil? 350 nil 351 else 352 before = scn.pos 353 # regexp position is a regexp, else a div 354 value = scn.scan(PATTERN_REGEX) if regexp_acceptable? 355 if value 356 # Ensure an escaped / was not matched 357 while escaped_end(value) 358 more = scn.scan_until(PATTERN_REGEX_END) 359 return emit(TOKEN_DIV, before) unless more 360 value << more 361 end 362 regex = value.sub(PATTERN_REGEX_A, '').sub(PATTERN_REGEX_Z, '').gsub(PATTERN_REGEX_ESC, '/') 363 emit_completed([:REGEX, Regexp.new(regex), scn.pos-before], before) 364 else 365 emit(TOKEN_DIV, before) 366 end 367 end 368 end, 369 370 # TOKENS <, <=, <|, <<|, <<, <-, <~ 371 '<' => lambda do 372 scn = @scanner 373 la = scn.peek(3) 374 emit(case la[1] 375 when '<' 376 if la[2] == '|' 377 TOKEN_LLCOLLECT 378 else 379 TOKEN_LSHIFT 380 end 381 when '=' 382 TOKEN_LESSEQUAL 383 when '|' 384 TOKEN_LCOLLECT 385 when '-' 386 TOKEN_OUT_EDGE 387 when '~' 388 TOKEN_OUT_EDGE_SUB 389 else 390 TOKEN_LESSTHAN 391 end, scn.pos) 392 end, 393 394 # TOKENS >, >=, >> 395 '>' => lambda do 396 scn = @scanner 397 la = scn.peek(2) 398 emit(case la[1] 399 when '>' 400 TOKEN_RSHIFT 401 when '=' 402 TOKEN_GREATEREQUAL 403 else 404 TOKEN_GREATERTHAN 405 end, scn.pos) 406 end, 407 408 # TOKENS :, ::CLASSREF, ::NAME 409 ':' => lambda do 410 scn = @scanner 411 la = scn.peek(3) 412 before = scn.pos 413 if la[1] == ':' 414 # PERFORMANCE NOTE: This could potentially be speeded up by using a case/when listing all 415 # upper case letters. Alternatively, the 'A', and 'Z' comparisons may be faster if they are 416 # frozen. 417 # 418 la2 = la[2] 419 if la2 >= 'A' && la2 <= 'Z' 420 # CLASSREF or error 421 value = scn.scan(PATTERN_CLASSREF) 422 if value && scn.peek(2) != '::' 423 after = scn.pos 424 emit_completed([:CLASSREF, value.freeze, after-before], before) 425 else 426 # move to faulty position ('::<uc-letter>' was ok) 427 scn.pos = scn.pos + 3 428 lex_error(Issues::ILLEGAL_FULLY_QUALIFIED_CLASS_REFERENCE) 429 end 430 else 431 value = scn.scan(PATTERN_BARE_WORD) 432 if value 433 if value =~ PATTERN_NAME 434 emit_completed([:NAME, value.freeze, scn.pos - before], before) 435 else 436 emit_completed([:WORD, value.freeze, scn.pos - before], before) 437 end 438 else 439 # move to faulty position ('::' was ok) 440 scn.pos = scn.pos + 2 441 lex_error(Issues::ILLEGAL_FULLY_QUALIFIED_NAME) 442 end 443 end 444 else 445 emit(TOKEN_COLON, before) 446 end 447 end, 448 449 '$' => lambda do 450 scn = @scanner 451 before = scn.pos 452 value = scn.scan(PATTERN_DOLLAR_VAR) 453 if value 454 emit_completed([:VARIABLE, value[1..-1].freeze, scn.pos - before], before) 455 else 456 # consume the $ and let higher layer complain about the error instead of getting a syntax error 457 emit(TOKEN_VARIABLE_EMPTY, before) 458 end 459 end, 460 461 '"' => lambda do 462 # Recursive string interpolation, 'interpolate' either returns a STRING token, or 463 # a DQPRE with the rest of the string's tokens placed in the @token_queue 464 interpolate_dq 465 end, 466 467 "'" => lambda do 468 scn = @scanner 469 before = scn.pos 470 emit_completed([:STRING, slurp_sqstring.freeze, scn.pos - before], before) 471 end, 472 473 "\n" => lambda do 474 # If heredoc_cont is in effect there are heredoc text lines to skip over 475 # otherwise just skip the newline. 476 # 477 ctx = @lexing_context 478 if ctx[:newline_jump] 479 @scanner.pos = ctx[:newline_jump] 480 ctx[:newline_jump] = nil 481 else 482 @scanner.pos += 1 483 end 484 ctx[:line_lexical_start] = @scanner.pos 485 nil 486 end, 487 '' => lambda { nil } # when the peek(1) returns empty 488 } 489 490 [ ' ', "\t", "\r" ].each { |c| @selector[c] = lambda { @scanner.skip(PATTERN_WS); nil } } 491 492 [ '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'].each do |c| 493 @selector[c] = lambda do 494 scn = @scanner 495 before = scn.pos 496 value = scn.scan(PATTERN_NUMBER) 497 if value 498 length = scn.pos - before 499 assert_numeric(value, before) 500 emit_completed([:NUMBER, value.freeze, length], before) 501 else 502 invalid_number = scn.scan_until(PATTERN_NON_WS) 503 if before > 1 504 after = scn.pos 505 scn.pos = before - 1 506 if scn.peek(1) == '.' 507 # preceded by a dot. Is this a bad decimal number then? 508 scn.pos = before - 2 509 while scn.peek(1) =~ /^\d$/ 510 invalid_number = nil 511 before = scn.pos 512 break if before == 0 513 scn.pos = scn.pos - 1 514 end 515 end 516 scn.pos = before 517 invalid_number = scn.peek(after - before) unless invalid_number 518 end 519 assert_numeric(invalid_number, before) 520 scn.pos = before + 1 521 lex_error(Issues::ILLEGAL_NUMBER, {:value => invalid_number}) 522 end 523 end 524 end 525 ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 526 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '_'].each do |c| 527 @selector[c] = lambda do 528 scn = @scanner 529 before = scn.pos 530 value = scn.scan(PATTERN_BARE_WORD) 531 if value && value =~ PATTERN_NAME 532 emit_completed(KEYWORDS[value] || @taskm_keywords[value] || [:NAME, value.freeze, scn.pos - before], before) 533 elsif value 534 emit_completed([:WORD, value.freeze, scn.pos - before], before) 535 else 536 # move to faulty position ([a-z_] was ok) 537 scn.pos = scn.pos + 1 538 fully_qualified = scn.match?(/::/) 539 if fully_qualified 540 lex_error(Issues::ILLEGAL_FULLY_QUALIFIED_NAME) 541 else 542 lex_error(Issues::ILLEGAL_NAME_OR_BARE_WORD) 543 end 544 end 545 end 546 end 547 548 ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 549 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'].each do |c| 550 @selector[c] = lambda do 551 scn = @scanner 552 before = scn.pos 553 value = @scanner.scan(PATTERN_CLASSREF) 554 if value && @scanner.peek(2) != '::' 555 emit_completed([:CLASSREF, value.freeze, scn.pos - before], before) 556 else 557 # move to faulty position ([A-Z] was ok) 558 scn.pos = scn.pos + 1 559 lex_error(Issues::ILLEGAL_CLASS_REFERENCE) 560 end 561 end 562 end 563 564 @selector.default = lambda do 565 # In case of unicode spaces of various kinds that are captured by a regexp, but not by the 566 # simpler case expression above (not worth handling those special cases with better performance). 567 scn = @scanner 568 if scn.skip(PATTERN_WS) 569 nil 570 else 571 # "unrecognized char" 572 emit([:OTHER, scn.peek(0), 1], scn.pos) 573 end 574 end 575 @selector.each { |k,v| k.freeze } 576 @selector.freeze 577 end
Public Instance Methods
Clears the lexer state (it is not required to call this as it will be garbage collected and the next lex call (lex_string
, lex_file
) will reset the internal state.
# File lib/puppet/pops/parser/lexer2.rb 597 def clear() 598 # not really needed, but if someone wants to ensure garbage is collected as early as possible 599 @scanner = nil 600 @locator = nil 601 @lexing_context = nil 602 end
Emits (produces) a token [:tokensymbol, TokenValue] and moves the scanner's position past the token
# File lib/puppet/pops/parser/lexer2.rb 724 def emit(token, byte_offset) 725 @scanner.pos = byte_offset + token[2] 726 [token[0], TokenValue.new(token, byte_offset, @locator)] 727 end
Emits the completed token on the form [:tokensymbol, TokenValue. This method does not alter the scanner's position.
# File lib/puppet/pops/parser/lexer2.rb 732 def emit_completed(token, byte_offset) 733 [token[0], TokenValue.new(token, byte_offset, @locator)] 734 end
Allows subprocessors for heredoc etc to enqueue tokens that are tokenized by a different lexer instance
# File lib/puppet/pops/parser/lexer2.rb 743 def enqueue(emitted_token) 744 @token_queue << emitted_token 745 end
Enqueues a completed token at the given offset
# File lib/puppet/pops/parser/lexer2.rb 737 def enqueue_completed(token, byte_offset) 738 @token_queue << emit_completed(token, byte_offset) 739 end
Determine if last char of value is escaped by a backslash
# File lib/puppet/pops/parser/lexer2.rb 580 def escaped_end(value) 581 escaped = false 582 if value.end_with?(STRING_BSLASH_SLASH) 583 value[1...-1].each_codepoint do |cp| 584 if cp == 0x5c # backslash 585 escaped = !escaped 586 else 587 escaped = false 588 end 589 end 590 end 591 escaped 592 end
TODO: This method should not be used, callers should get the locator since it is most likely required to compute line, position etc given offsets.
# File lib/puppet/pops/parser/lexer2.rb 645 def file 646 @locator ? @locator.file : nil 647 end
Convenience method, and for compatibility with older lexer. Use the lex_file
instead. (Bad form to use overloading of assignment operator for something that is not really an assignment).
# File lib/puppet/pops/parser/lexer2.rb 638 def file=(file) 639 lex_file(file) 640 end
Scans all of the content and returns it in an array Note that the terminating [false, false] token is included in the result.
# File lib/puppet/pops/parser/lexer2.rb 674 def fullscan 675 result = [] 676 scan {|token| result.push(token) } 677 result 678 end
# File lib/puppet/pops/parser/lexer2.rb 659 def initvars 660 @token_queue = [] 661 # NOTE: additional keys are used; :escapes, :uq_slurp_pattern, :newline_jump, :epp_* 662 @lexing_context = { 663 :brace_count => 0, 664 :after => nil, 665 :line_lexical_start => 0 666 } 667 # Use of --tasks introduces the new keyword 'plan' 668 @taskm_keywords = Puppet[:tasks] ? { 'plan' => [:PLAN, 'plan', 4], 'apply' => [:APPLY, 'apply', 5] }.freeze : EMPTY_HASH 669 end
Initializes lexing of the content of the given file. An empty string is used if the file does not exist.
# File lib/puppet/pops/parser/lexer2.rb 651 def lex_file(file) 652 initvars 653 contents = Puppet::FileSystem.exist?(file) ? Puppet::FileSystem.read(file, :mode => 'rb', :encoding => 'utf-8') : '' 654 assert_not_bom(contents) 655 @scanner = StringScanner.new(contents.freeze) 656 @locator = Locator.locator(contents, file) 657 end
# File lib/puppet/pops/parser/lexer2.rb 613 def lex_string(string, path=nil) 614 initvars 615 assert_not_bom(string) 616 @scanner = StringScanner.new(string) 617 @locator = Locator.locator(string, path) 618 end
This lexes one token at the current position of the scanner. PERFORMANCE NOTE: Any change to this logic should be performance measured.
# File lib/puppet/pops/parser/lexer2.rb 718 def lex_token 719 @selector[@scanner.peek(1)].call 720 end
Lexes an unquoted string. @param string [String] the string to lex @param locator [Locator] the locator to use (a default is used if nil is given) @param escapes [Array<String>] array of character strings representing the escape sequences to transform @param interpolate [Boolean] whether interpolation of expressions should be made or not.
# File lib/puppet/pops/parser/lexer2.rb 626 def lex_unquoted_string(string, locator, escapes, interpolate) 627 initvars 628 assert_not_bom(string) 629 @scanner = StringScanner.new(string) 630 @locator = locator || Locator.locator(string, '') 631 @lexing_context[:escapes] = escapes || UQ_ESCAPES 632 @lexing_context[:uq_slurp_pattern] = interpolate ? (escapes.include?('$') ? SLURP_UQ_PATTERN : SLURP_UQNE_PATTERN) : SLURP_ALL_PATTERN 633 end
Answers after which tokens it is acceptable to lex a regular expression. PERFORMANCE NOTE: It may be beneficial to turn this into a hash with default value of true for missing entries. A case expression with literal values will however create a hash internally. Since a reference is always needed to the hash, this access is almost as costly as a method call.
# File lib/puppet/pops/parser/lexer2.rb 753 def regexp_acceptable? 754 case @lexing_context[:after] 755 756 # Ends of (potential) R-value generating expressions 757 when :RPAREN, :RBRACK, :RRCOLLECT, :RCOLLECT 758 false 759 760 # End of (potential) R-value - but must be allowed because of case expressions 761 # Called out here to not be mistaken for a bug. 762 when :RBRACE 763 true 764 765 # Operands (that can be followed by DIV (even if illegal in grammar) 766 when :NAME, :CLASSREF, :NUMBER, :STRING, :BOOLEAN, :DQPRE, :DQMID, :DQPOST, :HEREDOC, :REGEX, :VARIABLE, :WORD 767 false 768 769 else 770 true 771 end 772 end
A block must be passed to scan. It will be called with two arguments, a symbol for the token, and an instance of LexerSupport::TokenValue
PERFORMANCE NOTE: The TokenValue is designed to reduce the amount of garbage / temporary data and to only convert the lexer's internal tokens on demand. It is slightly more costly to create an instance of a class defined in Ruby than an Array or Hash, but the gain is much bigger since transformation logic is avoided for many of its members (most are never used (e.g. line/pos information which is only of value in general for error messages, and for some expressions (which the lexer does not know about).
# File lib/puppet/pops/parser/lexer2.rb 688 def scan 689 # PERFORMANCE note: it is faster to access local variables than instance variables. 690 # This makes a small but notable difference since instance member access is avoided for 691 # every token in the lexed content. 692 # 693 scn = @scanner 694 lex_error_without_pos(Issues::NO_INPUT_TO_LEXER) unless scn 695 696 ctx = @lexing_context 697 queue = @token_queue 698 selector = @selector 699 700 scn.skip(PATTERN_WS) 701 702 # This is the lexer's main loop 703 until queue.empty? && scn.eos? do 704 token = queue.shift || selector[scn.peek(1)].call 705 if token 706 ctx[:after] = token[0] 707 yield token 708 end 709 end 710 711 # Signals end of input 712 yield [false, false] 713 end
Convenience method, and for compatibility with older lexer. Use the lex_string
instead which allows passing the path to use without first having to call file= (which reads the file if it exists). (Bad form to use overloading of assignment operator for something that is not really an assignment. Also, overloading of = does not allow passing more than one argument).
# File lib/puppet/pops/parser/lexer2.rb 609 def string=(string) 610 lex_string(string, nil) 611 end