class Puppet::Pops::Parser::Lexer2

Constants

KEYWORDS

Keywords are all singleton tokens with pre calculated lengths. Booleans are pre-calculated (rather than evaluating the strings “false” “true” repeatedly.

KEYWORD_NAMES

Reverse lookup of keyword name to string

PATTERN_BARE_WORD
PATTERN_CLASSREF

The NAME and CLASSREF in 4x are strict. Each segment must start with a letter a-z and may not contain dashes (w includes letters, digits and _).

PATTERN_COMMENT

The single line comment includes the line ending.

PATTERN_DOLLAR_VAR
PATTERN_MLCOMMENT
PATTERN_NAME
PATTERN_NON_WS
PATTERN_NUMBER
PATTERN_REGEX
PATTERN_REGEX_A
PATTERN_REGEX_END
PATTERN_REGEX_ESC
PATTERN_REGEX_Z
PATTERN_WS
STRING_BSLASH_SLASH

PERFORMANCE NOTE: Comparison against a frozen string is faster (than unfrozen).

TOKEN_APPENDS
TOKEN_AT
TOKEN_ATAT
TOKEN_COLON
TOKEN_COMMA
TOKEN_DELETES
TOKEN_DIV
TOKEN_DOT
TOKEN_DQMID
TOKEN_DQPOS
TOKEN_DQPRE
TOKEN_EPPEND
TOKEN_EPPEND_TRIM
TOKEN_EPPSTART

EPP_START is currently a marker token, may later get syntax

TOKEN_EQUALS
TOKEN_FARROW
TOKEN_GREATEREQUAL
TOKEN_GREATERTHAN
TOKEN_HEREDOC

HEREDOC has syntax as an argument.

TOKEN_IN_EDGE
TOKEN_IN_EDGE_SUB
TOKEN_ISEQUAL
TOKEN_LBRACE
TOKEN_LBRACK

ALl tokens have three slots, the token name (a Symbol), the token text (String), and a token text length. All operator and punctuation tokens reuse singleton arrays Tokens that require unique values create a unique array per token.

PEFORMANCE NOTES: This construct reduces the amount of object that needs to be created for operators and punctuation. The length is pre-calculated for all singleton tokens. The length is used both to signal the length of the token, and to advance the scanner position (without having to advance it with a scan(regexp)).

TOKEN_LCOLLECT
TOKEN_LESSEQUAL
TOKEN_LESSTHAN
TOKEN_LISTSTART
TOKEN_LLCOLLECT
TOKEN_LPAREN
TOKEN_LSHIFT
TOKEN_MATCH
TOKEN_MINUS
TOKEN_MODULO
TOKEN_NOMATCH
TOKEN_NOT
TOKEN_NOTEQUAL
TOKEN_NUMBER
TOKEN_OTHER

This is used for unrecognized tokens, will always be a single character. This particular instance is not used, but is kept here for documentation purposes.

TOKEN_OUT_EDGE
TOKEN_OUT_EDGE_SUB
TOKEN_PARROW
TOKEN_PIPE
TOKEN_PLUS
TOKEN_QMARK
TOKEN_RBRACE
TOKEN_RBRACK
TOKEN_RCOLLECT
TOKEN_REGEXP
TOKEN_RPAREN
TOKEN_RRCOLLECT
TOKEN_RSHIFT
TOKEN_SELBRACE
TOKEN_SEMIC
TOKEN_STRING

Tokens that are always unique to what has been lexed

TOKEN_TILDE
TOKEN_TIMES
TOKEN_VARIABLE
TOKEN_VARIABLE_EMPTY
TOKEN_WORD
TOKEN_WSLPAREN

Attributes

locator[R]

Public Class Methods

new() click to toggle source
    # File lib/puppet/pops/parser/lexer2.rb
182 def initialize()
183   @selector = {
184     '.' =>  lambda { emit(TOKEN_DOT, @scanner.pos) },
185     ',' => lambda {  emit(TOKEN_COMMA, @scanner.pos) },
186     '[' => lambda do
187       before = @scanner.pos
188       # Must check the preceding character to see if it is whitespace.
189       # The fastest thing to do is to simply byteslice to get the string ending at the offset before
190       # and then check what the last character is. (This is the same as  what an locator.char_offset needs
191       # to compute, but with less overhead of trying to find out the global offset from a local offset in the
192       # case when this is sublocated in a heredoc).
193       if before == 0 || @scanner.string.byteslice(0, before)[-1] =~ /[[:blank:]\r\n]+/
194         emit(TOKEN_LISTSTART, before)
195       else
196         emit(TOKEN_LBRACK, before)
197       end
198     end,
199     ']' => lambda { emit(TOKEN_RBRACK, @scanner.pos) },
200     '(' => lambda do
201       before = @scanner.pos
202       # If first on a line, or only whitespace between start of line and '('
203       # then the token is special to avoid being taken as start of a call.
204       line_start = @lexing_context[:line_lexical_start]
205       if before == line_start || @scanner.string.byteslice(line_start, before - line_start) =~ /\A[[:blank:]\r]+\Z/
206         emit(TOKEN_WSLPAREN, before)
207       else
208         emit(TOKEN_LPAREN, before)
209       end
210     end,
211     ')' => lambda { emit(TOKEN_RPAREN, @scanner.pos) },
212     ';' => lambda { emit(TOKEN_SEMIC, @scanner.pos) },
213     '?' => lambda { emit(TOKEN_QMARK, @scanner.pos) },
214     '*' => lambda { emit(TOKEN_TIMES, @scanner.pos) },
215     '%' => lambda do
216       scn = @scanner
217       before = scn.pos
218       la = scn.peek(2)
219       if la[1] == '>' && @lexing_context[:epp_mode]
220         scn.pos += 2
221         if @lexing_context[:epp_mode] == :expr
222           enqueue_completed(TOKEN_EPPEND, before)
223         end
224         @lexing_context[:epp_mode] = :text
225         interpolate_epp
226       else
227         emit(TOKEN_MODULO, before)
228       end
229     end,
230     '{' => lambda do
231       # The lexer needs to help the parser since the technology used cannot deal with
232       # lookahead of same token with different precedence. This is solved by making left brace
233       # after ? into a separate token.
234       #
235       @lexing_context[:brace_count] += 1
236       emit(if @lexing_context[:after] == :QMARK
237              TOKEN_SELBRACE
238            else
239              TOKEN_LBRACE
240            end, @scanner.pos)
241     end,
242     '}' => lambda do
243       @lexing_context[:brace_count] -= 1
244       emit(TOKEN_RBRACE, @scanner.pos)
245     end,
246 
247 
248     # TOKENS @, @@, @(
249     '@' => lambda do
250       scn = @scanner
251       la = scn.peek(2)
252       if la[1] == '@'
253         emit(TOKEN_ATAT, scn.pos) # TODO; Check if this is good for the grammar
254       elsif la[1] == '('
255         heredoc
256       else
257         emit(TOKEN_AT, scn.pos)
258       end
259     end,
260 
261     # TOKENS |, |>, |>>
262     '|' => lambda do
263       scn = @scanner
264       la = scn.peek(3)
265       emit(la[1] == '>' ? (la[2] == '>' ? TOKEN_RRCOLLECT : TOKEN_RCOLLECT) : TOKEN_PIPE, scn.pos)
266     end,
267 
268     # TOKENS =, =>, ==, =~
269     '=' => lambda do
270       scn = @scanner
271       la = scn.peek(2)
272       emit(case la[1]
273            when '='
274              TOKEN_ISEQUAL
275            when '>'
276              TOKEN_FARROW
277            when '~'
278              TOKEN_MATCH
279            else
280              TOKEN_EQUALS
281            end, scn.pos)
282     end,
283 
284     # TOKENS '+', '+=', and '+>'
285     '+' => lambda do
286       scn = @scanner
287       la = scn.peek(2)
288       emit(case la[1]
289            when '='
290              TOKEN_APPENDS
291            when '>'
292              TOKEN_PARROW
293            else
294              TOKEN_PLUS
295            end, scn.pos)
296     end,
297 
298     # TOKENS '-', '->', and epp '-%>' (end of interpolation with trim)
299     '-' => lambda do
300       scn = @scanner
301       la = scn.peek(3)
302       before = scn.pos
303       if @lexing_context[:epp_mode] && la[1] == '%' && la[2] == '>'
304         scn.pos += 3
305         if @lexing_context[:epp_mode] == :expr
306           enqueue_completed(TOKEN_EPPEND_TRIM, before)
307         end
308         interpolate_epp(:with_trim)
309       else
310         emit(case la[1]
311              when '>'
312                TOKEN_IN_EDGE
313              when '='
314                TOKEN_DELETES
315              else
316                TOKEN_MINUS
317              end, before)
318       end
319     end,
320 
321     # TOKENS !, !=, !~
322     '!' => lambda do
323       scn = @scanner
324       la = scn.peek(2)
325       emit(case la[1]
326            when '='
327              TOKEN_NOTEQUAL
328            when '~'
329              TOKEN_NOMATCH
330            else
331              TOKEN_NOT
332            end, scn.pos)
333     end,
334 
335     # TOKENS ~>, ~
336     '~' => lambda do
337       scn = @scanner
338       la = scn.peek(2)
339       emit(la[1] == '>' ? TOKEN_IN_EDGE_SUB : TOKEN_TILDE, scn.pos)
340     end,
341 
342     '#' => lambda { @scanner.skip(PATTERN_COMMENT); nil },
343 
344     # TOKENS '/', '/*' and '/ regexp /'
345     '/' => lambda do
346       scn = @scanner
347       la = scn.peek(2)
348       if la[1] == '*'
349         lex_error(Issues::UNCLOSED_MLCOMMENT) if scn.skip(PATTERN_MLCOMMENT).nil?
350         nil
351       else
352         before = scn.pos
353         # regexp position is a regexp, else a div
354         value = scn.scan(PATTERN_REGEX) if regexp_acceptable?
355         if value
356           # Ensure an escaped / was not matched
357           while escaped_end(value)
358             more = scn.scan_until(PATTERN_REGEX_END)
359             return emit(TOKEN_DIV, before) unless more
360             value << more
361           end
362           regex = value.sub(PATTERN_REGEX_A, '').sub(PATTERN_REGEX_Z, '').gsub(PATTERN_REGEX_ESC, '/')
363           emit_completed([:REGEX, Regexp.new(regex), scn.pos-before], before)
364         else
365           emit(TOKEN_DIV, before)
366         end
367       end
368     end,
369 
370     # TOKENS <, <=, <|, <<|, <<, <-, <~
371     '<' => lambda do
372       scn = @scanner
373       la = scn.peek(3)
374       emit(case la[1]
375            when '<'
376              if la[2] == '|'
377                TOKEN_LLCOLLECT
378              else
379                TOKEN_LSHIFT
380              end
381            when '='
382              TOKEN_LESSEQUAL
383            when '|'
384              TOKEN_LCOLLECT
385            when '-'
386              TOKEN_OUT_EDGE
387            when '~'
388              TOKEN_OUT_EDGE_SUB
389            else
390              TOKEN_LESSTHAN
391            end, scn.pos)
392     end,
393 
394     # TOKENS >, >=, >>
395     '>' => lambda do
396       scn = @scanner
397       la = scn.peek(2)
398       emit(case la[1]
399            when '>'
400              TOKEN_RSHIFT
401            when '='
402              TOKEN_GREATEREQUAL
403            else
404              TOKEN_GREATERTHAN
405            end, scn.pos)
406     end,
407 
408     # TOKENS :, ::CLASSREF, ::NAME
409     ':' => lambda do
410       scn = @scanner
411       la = scn.peek(3)
412       before = scn.pos
413       if la[1] == ':'
414         # PERFORMANCE NOTE: This could potentially be speeded up by using a case/when listing all
415         # upper case letters. Alternatively, the 'A', and 'Z' comparisons may be faster if they are
416         # frozen.
417         #
418         la2 = la[2]
419         if la2 >= 'A' && la2 <= 'Z'
420           # CLASSREF or error
421           value = scn.scan(PATTERN_CLASSREF)
422           if value && scn.peek(2) != '::'
423             after = scn.pos
424             emit_completed([:CLASSREF, value.freeze, after-before], before)
425           else
426             # move to faulty position ('::<uc-letter>' was ok)
427             scn.pos = scn.pos + 3
428             lex_error(Issues::ILLEGAL_FULLY_QUALIFIED_CLASS_REFERENCE)
429           end
430         else
431           value = scn.scan(PATTERN_BARE_WORD)
432           if value
433             if value =~ PATTERN_NAME
434               emit_completed([:NAME, value.freeze, scn.pos - before], before)
435             else
436               emit_completed([:WORD, value.freeze, scn.pos - before], before)
437             end
438           else
439             # move to faulty position ('::' was ok)
440             scn.pos = scn.pos + 2
441             lex_error(Issues::ILLEGAL_FULLY_QUALIFIED_NAME)
442           end
443         end
444       else
445         emit(TOKEN_COLON, before)
446       end
447     end,
448 
449     '$' => lambda do
450       scn = @scanner
451       before = scn.pos
452       value = scn.scan(PATTERN_DOLLAR_VAR)
453       if value
454         emit_completed([:VARIABLE, value[1..-1].freeze, scn.pos - before], before)
455       else
456         # consume the $ and let higher layer complain about the error instead of getting a syntax error
457         emit(TOKEN_VARIABLE_EMPTY, before)
458       end
459     end,
460 
461     '"' => lambda do
462       # Recursive string interpolation, 'interpolate' either returns a STRING token, or
463       # a DQPRE with the rest of the string's tokens placed in the @token_queue
464       interpolate_dq
465     end,
466 
467     "'" => lambda do
468       scn = @scanner
469       before = scn.pos
470       emit_completed([:STRING, slurp_sqstring.freeze, scn.pos - before], before)
471     end,
472 
473     "\n" => lambda do
474       # If heredoc_cont is in effect there are heredoc text lines to skip over
475       # otherwise just skip the newline.
476       #
477       ctx = @lexing_context
478       if ctx[:newline_jump]
479         @scanner.pos = ctx[:newline_jump]
480         ctx[:newline_jump] = nil
481       else
482         @scanner.pos += 1
483       end
484       ctx[:line_lexical_start] = @scanner.pos
485       nil
486     end,
487     '' => lambda { nil } # when the peek(1) returns empty
488   }
489 
490   [ ' ', "\t", "\r" ].each { |c| @selector[c] = lambda { @scanner.skip(PATTERN_WS); nil } }
491 
492   [ '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'].each do |c|
493     @selector[c] = lambda do
494       scn = @scanner
495       before = scn.pos
496       value = scn.scan(PATTERN_NUMBER)
497       if value
498         length = scn.pos - before
499         assert_numeric(value, before)
500         emit_completed([:NUMBER, value.freeze, length], before)
501       else
502         invalid_number = scn.scan_until(PATTERN_NON_WS)
503         if before > 1
504           after = scn.pos
505           scn.pos = before - 1
506           if scn.peek(1) == '.'
507             # preceded by a dot. Is this a bad decimal number then?
508             scn.pos = before - 2
509             while scn.peek(1) =~ /^\d$/
510               invalid_number = nil
511               before = scn.pos
512               break if before == 0
513               scn.pos = scn.pos - 1
514             end
515           end
516           scn.pos = before
517           invalid_number = scn.peek(after - before) unless invalid_number
518         end
519         assert_numeric(invalid_number, before)
520         scn.pos = before + 1
521         lex_error(Issues::ILLEGAL_NUMBER, {:value => invalid_number})
522       end
523     end
524   end
525   ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
526     'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '_'].each do |c|
527     @selector[c] = lambda do
528       scn = @scanner
529       before = scn.pos
530       value = scn.scan(PATTERN_BARE_WORD)
531       if value && value =~ PATTERN_NAME
532         emit_completed(KEYWORDS[value] || @taskm_keywords[value] || [:NAME, value.freeze, scn.pos - before], before)
533       elsif value
534         emit_completed([:WORD, value.freeze, scn.pos - before], before)
535       else
536         # move to faulty position ([a-z_] was ok)
537         scn.pos = scn.pos + 1
538         fully_qualified = scn.match?(/::/)
539         if fully_qualified
540           lex_error(Issues::ILLEGAL_FULLY_QUALIFIED_NAME)
541         else
542           lex_error(Issues::ILLEGAL_NAME_OR_BARE_WORD)
543         end
544       end
545     end
546   end
547 
548   ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
549     'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'].each do |c|
550     @selector[c] = lambda do
551       scn = @scanner
552       before = scn.pos
553       value = @scanner.scan(PATTERN_CLASSREF)
554       if value && @scanner.peek(2) != '::'
555         emit_completed([:CLASSREF, value.freeze, scn.pos - before], before)
556       else
557         # move to faulty position ([A-Z] was ok)
558         scn.pos = scn.pos + 1
559         lex_error(Issues::ILLEGAL_CLASS_REFERENCE)
560       end
561     end
562   end
563 
564   @selector.default = lambda do
565     # In case of unicode spaces of various kinds that are captured by a regexp, but not by the
566     # simpler case expression above (not worth handling those special cases with better performance).
567     scn = @scanner
568     if scn.skip(PATTERN_WS)
569       nil
570     else
571       # "unrecognized char"
572       emit([:OTHER, scn.peek(0), 1], scn.pos)
573     end
574   end
575   @selector.each { |k,v| k.freeze }
576   @selector.freeze
577 end

Public Instance Methods

clear() click to toggle source

Clears the lexer state (it is not required to call this as it will be garbage collected and the next lex call (lex_string, lex_file) will reset the internal state.

    # File lib/puppet/pops/parser/lexer2.rb
597 def clear()
598   # not really needed, but if someone wants to ensure garbage is collected as early as possible
599   @scanner = nil
600   @locator = nil
601   @lexing_context = nil
602 end
emit(token, byte_offset) click to toggle source

Emits (produces) a token [:tokensymbol, TokenValue] and moves the scanner's position past the token

    # File lib/puppet/pops/parser/lexer2.rb
724 def emit(token, byte_offset)
725   @scanner.pos = byte_offset + token[2]
726   [token[0], TokenValue.new(token, byte_offset, @locator)]
727 end
emit_completed(token, byte_offset) click to toggle source

Emits the completed token on the form [:tokensymbol, TokenValue. This method does not alter the scanner's position.

    # File lib/puppet/pops/parser/lexer2.rb
732 def emit_completed(token, byte_offset)
733   [token[0], TokenValue.new(token, byte_offset, @locator)]
734 end
enqueue(emitted_token) click to toggle source

Allows subprocessors for heredoc etc to enqueue tokens that are tokenized by a different lexer instance

    # File lib/puppet/pops/parser/lexer2.rb
743 def enqueue(emitted_token)
744   @token_queue << emitted_token
745 end
enqueue_completed(token, byte_offset) click to toggle source

Enqueues a completed token at the given offset

    # File lib/puppet/pops/parser/lexer2.rb
737 def enqueue_completed(token, byte_offset)
738   @token_queue << emit_completed(token, byte_offset)
739 end
escaped_end(value) click to toggle source

Determine if last char of value is escaped by a backslash

    # File lib/puppet/pops/parser/lexer2.rb
580 def escaped_end(value)
581   escaped = false
582   if value.end_with?(STRING_BSLASH_SLASH)
583     value[1...-1].each_codepoint do |cp|
584       if cp == 0x5c # backslash
585         escaped = !escaped
586       else
587         escaped = false
588       end
589     end
590   end
591   escaped
592 end
file() click to toggle source

TODO: This method should not be used, callers should get the locator since it is most likely required to compute line, position etc given offsets.

    # File lib/puppet/pops/parser/lexer2.rb
645 def file
646   @locator ? @locator.file : nil
647 end
file=(file) click to toggle source

Convenience method, and for compatibility with older lexer. Use the lex_file instead. (Bad form to use overloading of assignment operator for something that is not really an assignment).

    # File lib/puppet/pops/parser/lexer2.rb
638 def file=(file)
639   lex_file(file)
640 end
fullscan() click to toggle source

Scans all of the content and returns it in an array Note that the terminating [false, false] token is included in the result.

    # File lib/puppet/pops/parser/lexer2.rb
674 def fullscan
675   result = []
676   scan {|token| result.push(token) }
677   result
678 end
initvars() click to toggle source
    # File lib/puppet/pops/parser/lexer2.rb
659 def initvars
660   @token_queue = []
661   # NOTE: additional keys are used; :escapes, :uq_slurp_pattern, :newline_jump, :epp_*
662   @lexing_context = {
663     :brace_count => 0,
664     :after => nil,
665     :line_lexical_start => 0
666   }
667   # Use of --tasks introduces the new keyword 'plan'
668   @taskm_keywords = Puppet[:tasks] ? { 'plan' => [:PLAN, 'plan',  4], 'apply' => [:APPLY, 'apply', 5] }.freeze : EMPTY_HASH
669 end
lex_file(file) click to toggle source

Initializes lexing of the content of the given file. An empty string is used if the file does not exist.

    # File lib/puppet/pops/parser/lexer2.rb
651 def lex_file(file)
652   initvars
653   contents = Puppet::FileSystem.exist?(file) ? Puppet::FileSystem.read(file, :mode => 'rb', :encoding => 'utf-8') : ''
654   assert_not_bom(contents)
655   @scanner = StringScanner.new(contents.freeze)
656   @locator = Locator.locator(contents, file)
657 end
lex_string(string, path=nil) click to toggle source
    # File lib/puppet/pops/parser/lexer2.rb
613 def lex_string(string, path=nil)
614   initvars
615   assert_not_bom(string)
616   @scanner = StringScanner.new(string)
617   @locator = Locator.locator(string, path)
618 end
lex_token() click to toggle source

This lexes one token at the current position of the scanner. PERFORMANCE NOTE: Any change to this logic should be performance measured.

    # File lib/puppet/pops/parser/lexer2.rb
718 def lex_token
719   @selector[@scanner.peek(1)].call
720 end
lex_unquoted_string(string, locator, escapes, interpolate) click to toggle source

Lexes an unquoted string. @param string [String] the string to lex @param locator [Locator] the locator to use (a default is used if nil is given) @param escapes [Array<String>] array of character strings representing the escape sequences to transform @param interpolate [Boolean] whether interpolation of expressions should be made or not.

    # File lib/puppet/pops/parser/lexer2.rb
626 def lex_unquoted_string(string, locator, escapes, interpolate)
627   initvars
628   assert_not_bom(string)
629   @scanner = StringScanner.new(string)
630   @locator = locator || Locator.locator(string, '')
631   @lexing_context[:escapes] = escapes || UQ_ESCAPES
632   @lexing_context[:uq_slurp_pattern] = interpolate ? (escapes.include?('$') ? SLURP_UQ_PATTERN : SLURP_UQNE_PATTERN) : SLURP_ALL_PATTERN
633 end
regexp_acceptable?() click to toggle source

Answers after which tokens it is acceptable to lex a regular expression. PERFORMANCE NOTE: It may be beneficial to turn this into a hash with default value of true for missing entries. A case expression with literal values will however create a hash internally. Since a reference is always needed to the hash, this access is almost as costly as a method call.

    # File lib/puppet/pops/parser/lexer2.rb
753 def regexp_acceptable?
754   case @lexing_context[:after]
755 
756   # Ends of (potential) R-value generating expressions
757   when :RPAREN, :RBRACK, :RRCOLLECT, :RCOLLECT
758     false
759 
760   # End of (potential) R-value - but must be allowed because of case expressions
761   # Called out here to not be mistaken for a bug.
762   when :RBRACE
763     true
764 
765   # Operands (that can be followed by DIV (even if illegal in grammar)
766   when :NAME, :CLASSREF, :NUMBER, :STRING, :BOOLEAN, :DQPRE, :DQMID, :DQPOST, :HEREDOC, :REGEX, :VARIABLE, :WORD
767     false
768 
769   else
770     true
771   end
772 end
scan() { |token| ... } click to toggle source

A block must be passed to scan. It will be called with two arguments, a symbol for the token, and an instance of LexerSupport::TokenValue PERFORMANCE NOTE: The TokenValue is designed to reduce the amount of garbage / temporary data and to only convert the lexer's internal tokens on demand. It is slightly more costly to create an instance of a class defined in Ruby than an Array or Hash, but the gain is much bigger since transformation logic is avoided for many of its members (most are never used (e.g. line/pos information which is only of value in general for error messages, and for some expressions (which the lexer does not know about).

    # File lib/puppet/pops/parser/lexer2.rb
688 def scan
689   # PERFORMANCE note: it is faster to access local variables than instance variables.
690   # This makes a small but notable difference since instance member access is avoided for
691   # every token in the lexed content.
692   #
693   scn   = @scanner
694   lex_error_without_pos(Issues::NO_INPUT_TO_LEXER) unless scn
695 
696   ctx   = @lexing_context
697   queue = @token_queue
698   selector = @selector
699 
700   scn.skip(PATTERN_WS)
701 
702   # This is the lexer's main loop
703   until queue.empty? && scn.eos? do
704     token = queue.shift || selector[scn.peek(1)].call
705     if token
706       ctx[:after] = token[0]
707       yield token
708     end
709   end
710 
711   # Signals end of input
712   yield [false, false]
713 end
string=(string) click to toggle source

Convenience method, and for compatibility with older lexer. Use the lex_string instead which allows passing the path to use without first having to call file= (which reads the file if it exists). (Bad form to use overloading of assignment operator for something that is not really an assignment. Also, overloading of = does not allow passing more than one argument).

    # File lib/puppet/pops/parser/lexer2.rb
609 def string=(string)
610   lex_string(string, nil)
611 end