class Grammar

Represents a Textmate Grammar

Attributes

name[RW]
repository[RW]
scope_name[RW]

Public Class Methods

fromTmLanguage(path) click to toggle source

import an existing grammar from a file

@note the imported grammar is write only access to imported keys will raise an error

@param [String] path path to a json or plist grammar

@return [Grammar] The imported grammar

# File lib/ruby_grammar_builder/grammar.rb, line 43
def self.fromTmLanguage(path)
    begin
        import_grammar = JSON.parse File.read(path)
    rescue JSON::ParserError
        require 'plist'
        import_grammar = Plist.parse_xml File.read(path)
    end

    grammar = ImportGrammar.new(
        name: import_grammar["name"],
        scope_name: import_grammar["scopeName"],
        version: import_grammar["version"] || "",
        description: import_grammar["description"] || nil,
        information_for_contributors: import_grammar["information_for_contributors"] || nil,
        fileTypes: import_grammar["fileTypes"] || nil,
    )
    # import "patterns" into @repository[:$initial_context]
    grammar.repository[:$initial_context] = import_grammar["patterns"]
    # import the rest of the repository
    import_grammar["repository"].each do |key, value|
        # repository keys are kept as a hash
        grammar.repository[key.to_sym] = value
    end
    grammar
end
import(path_or_export) click to toggle source

Import a grammar partial

@note the import is “dynamic”, changes made to the grammar partial after the import

wil be reflected in the parent grammar

@param [String, ExportableGrammar] path_or_export the grammar partial or the file

in which the grammar partial is declared

@return [ExportableGrammar]

# File lib/ruby_grammar_builder/grammar.rb, line 80
def self.import(path_or_export)
    export = path_or_export
    unless path_or_export.is_a? ExportableGrammar
        # allow for relative paths
        if not Pathname.new(path_or_export).absolute?
            relative_path = File.dirname(caller_locations[0].path)
            if not Pathname.new(relative_path).absolute?
                relative_path = File.join(Dir.pwd,relative_path)
            end
            path_or_export = File.join(relative_path, path_or_export)
        end
        require path_or_export
        resolved = File.expand_path resolve_require(path_or_export)

        export = @@export_grammars.dig(resolved, :grammar)
        unless export.is_a? ExportableGrammar
            raise "#{path_or_export} does not create a Exportable Grammar"
        end
    end

    return export.export
end
new(keys) click to toggle source

Create a new Grammar

@param [Hash] keys The grammar keys @option keys [String] :name The name of the grammar @option keys [String] :scope_name The scope_name of teh grammar, must start with

+source.+ or +text.+

@option keys [String, :auto] :version (:auto) the version of the grammar, :auto uses

the current git commit as the version

@option keys [Array] :patterns ([]) ignored, will be replaced with the initial context @option keys [Hash] :repository ({}) ignored, will be replaced by the generated rules @option keys all remaining options will be copied to the grammar without change

# File lib/ruby_grammar_builder/grammar.rb, line 116
def initialize(keys)
    required_keys = [:name, :scope_name]
    unless required_keys & keys.keys == required_keys
        puts "Missing one or more of the required grammar keys"
        puts "Missing: #{required_keys - (required_keys & keys.keys)}"
        puts "The required grammar keys are: #{required_keys}"
        raise "See above error"
    end

    @name = keys[:name]
    @scope_name = keys[:scope_name]
    @repository = {}

    keys.delete :name
    keys.delete :scope_name

    # auto versioning, when save_to is called grab the latest git commit or "" if not
    # a git repo
    keys[:version] ||= :auto
    @keys = keys.compact
    return if @scope_name == "export" || @scope_name.start_with?("source.", "text.")

    puts "Warning: grammar scope name should start with `source.' or `text.'"
    puts "Examples: source.cpp text.html text.html.markdown source.js.regexp"
end
new_exportable_grammar() click to toggle source

Create a new Exportable Grammar (Grammar Partial)

@return [ExportableGrammar] the new exportable Grammar

# File lib/ruby_grammar_builder/grammar.rb, line 30
def self.new_exportable_grammar
    ExportableGrammar.new
end
plugins() click to toggle source

Gets all registered plugins

@api private

@return [Array<GrammarPlugin>] A list of all plugins

# File lib/ruby_grammar_builder/grammar_plugin.rb, line 148
def self.plugins
    @@linters + @@transforms.values.flatten.map { |v| v[:transform] }
end
register_linter(linter) click to toggle source

Register a linter plugin

@param [GrammarLinter] linter the linter plugin

@return [void] nothing

# File lib/ruby_grammar_builder/grammar_plugin.rb, line 108
def self.register_linter(linter)
    @@linters << linter
end
register_transform(transform, priority = 150) click to toggle source

Register a transformation plugin

@param [GrammarTransform] transform the transformation plugin @param [Numeric] priority an optional priority

@note The priority controls when a transformation runs in relation to

other events in addition to ordering transformations
priorities < 100 have their pre transform run before pre linters
priorities >= 100 have their pre transform run after pre linters
priorities >= 200 do not have their pre_transform function ran
priorities < 300 have their post transorm run before post linters
priorities >= 300 have their post transorm run before post linters

@return [void] nothing

# File lib/ruby_grammar_builder/grammar_plugin.rb, line 128
def self.register_transform(transform, priority = 150)
    key = if priority < 100 then :before_pre_linter
          elsif priority < 200 then :after_pre_linter
          elsif priority < 300 then :before_post_linter
          else :after_pre_linter
          end

    @@transforms[key] << {
        priority: priority,
        transform: transform,
    }
end
remove_plugin(plugin) click to toggle source

Removes a plugin whose classname matches plugin

@param [#to_s] plugin The plugin name to remove

@return [void]

# File lib/ruby_grammar_builder/grammar_plugin.rb, line 159
def self.remove_plugin(plugin)
    @@linters.delete_if { |linter| linter.class.to_s == plugin.to_s }
    @@transforms[:before_pre_linter].delete_if { |t| t[:transform].class.to_s == plugin.to_s }
    @@transforms[:after_pre_linter].delete_if { |t| t[:transform].class.to_s == plugin.to_s }
    @@transforms[:before_post_linter].delete_if { |t| t[:transform].class.to_s == plugin.to_s }
    @@transforms[:after_post_linter].delete_if { |t| t[:transform].class.to_s == plugin.to_s }
end

Public Instance Methods

[](key) click to toggle source

Access a pattern in the grammar

@param [Symbol] key The key the pattern is stored in

@return [PatternBase, Symbol, Array<PatternBase, Symbol>] The stored pattern

# File lib/ruby_grammar_builder/grammar.rb, line 149
def [](key)
    if key.is_a?(Regexp)
        tokenMatching(key) # see tokens.rb
    else
        @repository.fetch(key, PlaceholderPattern.new(key))
    end
end
[]=(key, value) click to toggle source

Store a pattern

A pattern must be stored in the grammar for it to appear in the final grammar

The special key :$initial_context is the pattern that will be matched at the beginning of the document or whenever the root of the grammar is to be matched

@param [Symbol] key The key to store the pattern in @param [PatternBase, Symbol, Array<PatternBase, Symbol>] value the pattern to store

@return [PatternBase, Symbol, Array<PatternBase, Symbol>] the stored pattern

# File lib/ruby_grammar_builder/grammar.rb, line 170
def []=(key, value)
    unless key.is_a? Symbol
        raise "Use symbols not strings" unless key.is_a? Symbol
    end

    if key.to_s.start_with?("$") && !([:$initial_context, :$base, :$self].include? key)
        puts "#{key} is not a valid repository name"
        puts "repository names starting with $ are reserved"
        raise "See above error"
    end

    if key.to_s == "repository"
        puts "#{key} is not a valid repository name"
        puts "the name 'repository' is a reserved name"
        raise "See above error"
    end

    # add it to the repository
    @repository[key] = fixup_value(value)
    @repository[key]
end
auto_version() click to toggle source

Returns the version information

@api private

@return [String] The version string to use

# File lib/ruby_grammar_builder/grammar.rb, line 512
def auto_version
    return @keys[:version] unless @keys[:version] == :auto

    `git rev-parse HEAD`.strip
rescue StandardError
    ""
end
generate(options) click to toggle source

Convert the grammar into a hash suitable for exporting to a file

@param [Symbol] inherit_or_embedded Is this grammar being inherited

from, or will be embedded, this controls if :$initial_context is mapped to
+"$base"+ or +"$self"+

@return [Hash] the generated grammar

# File lib/ruby_grammar_builder/grammar.rb, line 296
def generate(options)
    default = {
        inherit_or_embedded: :embedded,
        should_lint: true,
    }
    options = default.merge(options)
    
    repo = @repository.__deep_clone__
    repo = run_pre_transform_stage(repo, :before_pre_linter)

    if options[:should_lint]
        @@linters.each do |linter|
            repo.each do |_, potential_pattern|
                [potential_pattern].flatten.each do |each_potential_pattern|
                    raise "linting failed, see above error" unless linter.pre_lint(
                        each_potential_pattern,
                        filter_options(
                            linter,
                            each_potential_pattern,
                            grammar: self,
                            repository: repo,
                        ),
                    )
                end
            end
        end
    end

    repo = run_pre_transform_stage(repo, :after_pre_linter)

    convert_initial_context = lambda do |potential_pattern|
        if potential_pattern == :$initial_context
            return (options[:inherit_or_embedded] == :embedded) ? :$self : :$base
        end

        if potential_pattern.is_a? Array
            return potential_pattern.map do |nested_potential_pattern|
                convert_initial_context.call(nested_potential_pattern)
            end
        end

        if potential_pattern.is_a? PatternBase
            return potential_pattern.transform_includes do |each_nested_potential_pattern|
                # transform includes will call this block again if each_* is a patternBase
                if each_nested_potential_pattern.is_a? PatternBase
                    next each_nested_potential_pattern
                end

                convert_initial_context.call(each_nested_potential_pattern)
            end
        end

        return potential_pattern
    end
    repo = repo.transform_values do |each_potential_pattern|
        convert_initial_context.call(each_potential_pattern)
    end

    output = {
        name: @name,
        scopeName: @scope_name,
    }

    to_tag = lambda do |potential_pattern|
        case potential_pattern
        when Array
            return {
                "patterns" => potential_pattern.map do |nested_potential_pattern|
                    to_tag.call(nested_potential_pattern)
                end,
            }
        when Symbol then return {"include" => "#" + potential_pattern.to_s}
        when Hash then return potential_pattern
        when String then return {"include" => potential_pattern}
        when PatternBase then return potential_pattern.to_tag
        else raise "Unexpected value: #{potential_pattern.class}"
        end
    end

    output[:repository] = repo.transform_values do |each_potential_pattern|
        to_tag.call(each_potential_pattern)
    end
    # sort repos by key name
    output[:repository] = Hash[output[:repository].sort_by { |key, _| key.to_s }]

    output[:patterns] = output[:repository][:$initial_context]
    output[:patterns] ||= []
    output[:patterns] = output[:patterns]["patterns"] if output[:patterns].is_a? Hash
    output[:repository].delete(:$initial_context)

    output[:version] = auto_version
    output.merge!(@keys) { |_key, old, _new| old }

    output = run_post_transform_stage(output, :before_pre_linter)
    output = run_post_transform_stage(output, :after_pre_linter)
    output = run_post_transform_stage(output, :before_post_linter)

    @@linters.each do |linter|
        raise "linting failed, see above error" unless linter.post_lint(output)
    end

    output = run_post_transform_stage(output, :after_post_linter)

    Hash[
        output.sort_by do |key, _|
            order = {
                information_for_contributors: 0,
                version: 1,
                name: 2,
                scopeName: 3,
                fileTypes: 4,
                unknown_keys: 5,
                patterns: 6,
                repository: 7,
                uuid: 8,
            }
            next order[key.to_sym] if order.has_key? key.to_sym

            order[:unknown_keys]
        end
    ]
end
import(path_or_export) click to toggle source

Import a grammar partial into this grammar

@note the import is “dynamic”, changes made to the grammar partial after the import

wil be reflected in the parent grammar

@param [String, ExportableGrammar] path_or_export the grammar partial or the file

in which the grammar partial is declared

@return [void] nothing

# File lib/ruby_grammar_builder/grammar.rb, line 203
def import(path_or_export)
    
    unless path_or_export.is_a? ExportableGrammar
        relative_path = File.dirname(caller_locations[0].path)
        if not Pathname.new(relative_path).absolute?
            relative_path = File.join(Dir.pwd,relative_path)
        end
        # allow for relative paths
        if not Pathname.new(path_or_export).absolute?
            path_or_export = File.join(relative_path, path_or_export)
        end
    end

    export = Grammar.import(path_or_export)
    export.parent_grammar = self

    # import the repository
    @repository = @repository.merge export.repository do |_key, old_val, new_val|
        [old_val, new_val].flatten.uniq
    end
end
parseTokenSyntax(argument) click to toggle source

convert a regex value into a proc filter used to select patterns

@param [Regexp] argument A value that uses the tokenParsing syntax (explained below)

@note The syntax for tokenParsing is simple, there are:

- `adjectives` ex: isAClass
- the `not` operator ex: !isAClass
- the `or` operator ex: isAClass || isAPrimitive
- the `and` operator ex: isAClass && isAPrimitive
- paraentheses ex: (!isAClass) && isAPrimitive
_ 
anything matching /[a-zA-Z0-9_]+/ is considered an "adjective"
whitespace, including newlines, are removed/ignored
all other characters are invalid
_
using only an adjective, ex: /isAClass/ means to only include
Patterns that have that adjective in their adjective list

@return [proc] a function that accepts a Pattern as input, and returns

a boolean of whether or not that pattern should 
be included
# File lib/ruby_grammar_builder/tokens.rb, line 64
    def parseTokenSyntax(argument)
        # validate input type
        if !argument.is_a?(Regexp)
            raise <<~HEREDOC
                
                
                Trying to call parseTokenSyntax() but the argument isn't Regexp its #{argument.class}
                value: #{argument}
            HEREDOC
        end
        # just remove the //'s from the string
        regex_content = argument.inspect[1...-1]
        
        # remove all invalid characters, make sure length didn't change
        invalid_characters_removed = regex_content.gsub(/[^a-zA-Z0-9_&|\(\)! \n]/, "")
        if invalid_characters_removed.length != regex_content.length
            raise <<~HEREDOC
                
                
                It appears the tokenSyntax #{argument.inspect} contains some invalid characters
                with invalid characters: #{regex_content.inspect}
                without invalid characters: #{invalid_characters_removed.inspect}
            HEREDOC
        end
        
        # find broken syntax
        if regex_content =~ /[a-zA-Z0-9_]+\s+[a-zA-Z0-9_]+/
            raise <<~HEREDOC
                
                Inside a tokenSyntax: #{argument.inspect}
                this part of the syntax is invalid: #{$&.inspect}
                (theres a space between two adjectives)
                My guess is that it was half-edited
                or an accidental space was added
            HEREDOC
        end
        
        # convert all adjectives into inclusion checks
        regex_content.gsub!(/\s+/," ")
        regex_content.gsub!(/[a-zA-Z0-9_]+/, 'pattern.arguments[:adjectives].include?(:\0)')
        # convert it into a proc
        return ->(pattern) do
            puts "regex_content is: #{regex_content} "
            eval(regex_content) if pattern.is_a?(PatternBase) && pattern.arguments[:adjectives].is_a?(Array)
        end
    end
run_post_transform_stage(output, stage) click to toggle source

Runs a set of post transformations

@param [Hash] output The generated grammar @param [Symbol] stage the stage to run

@return [Hash] The modified grammar

# File lib/ruby_grammar_builder/grammar.rb, line 278
def run_post_transform_stage(output, stage)
    @@transforms[stage]
        .sort { |a, b| a[:priority] <=> b[:priority] }
        .map { |a| a[:transform] }
        .each { |transform| output = transform.post_transform(output) }

    output
end
run_pre_transform_stage(repository, stage) click to toggle source

Runs a set of pre transformations

@api private

@param [Hash] repository The repository @param [:before_pre_linter,:after_pre_linter] stage the stage to run

@return [Hash] the modified repository

# File lib/ruby_grammar_builder/grammar.rb, line 235
def run_pre_transform_stage(repository, stage)
    @@transforms[stage]
        .sort { |a, b| a[:priority] <=> b[:priority] }
        .map { |a| a[:transform] }
        .each do |transform|
            repository = repository.transform_values do |potential_pattern|
                if potential_pattern.is_a? Array
                    potential_pattern.map do |each|
                        transform.pre_transform(
                            each,
                            filter_options(
                                transform,
                                each,
                                grammar: self,
                                repository: repository,
                            ),
                        )
                    end
                else
                    transform.pre_transform(
                        potential_pattern,
                        filter_options(
                            transform,
                            potential_pattern,
                            grammar: self,
                            repository: repository,
                        ),
                    )
                end
            end
        end

    repository
end
save_to(options) click to toggle source

Save the grammar to a path

@param [Hash] options options to save_to @option options :inherit_or_embedded (:embedded) see generate @option options :generate_tags [Boolean] (true) generate a list of all +:tag_as+s @option options :directory [String] the location to generate the files @option options :tag_dir [String] (File.join(options,“language_tags”)) the

directory to generate language tags in

@option options :syntax_dir [String] (File.join(options,“syntaxes”)) the

directory to generate the syntax file in

@option options :syntax_format [:json,:vscode,:plist,:textmate,:tm_language,:xml]

(:json) The format to generate the syntax file in

@option options :syntax_name [String] (“#{@name}.tmLanguage”) the name of the syntax

file to generate without the extension

@option options :tag_name [String] (“#{@name}-scopes.txt”) the name of the tag list

file to generate without the extension

@note all keys except :directory is optional @note :directory is optional if both :tag_dir and :syntax_dir are specified @note currently :vscode is an alias for :json @note currently :textmate, :tm_language, and :xml are aliases for :plist @note later the aliased :syntax_type choices may enable compatibility features

@return [void] nothing

# File lib/ruby_grammar_builder/grammar.rb, line 445
def save_to(options)
    options[:directory] ||= "."
    
    # make the path absolute
    absolute_path_from_caller = File.dirname(caller_locations[0].path)
    if not Pathname.new(absolute_path_from_caller).absolute?
        absolute_path_from_caller = File.join(Dir.pwd,absolute_path_from_caller)
    end
    if not Pathname.new(options[:directory]).absolute?
        options[:directory] = File.join(absolute_path_from_caller, options[:directory])
    end
    
    default = {
        inherit_or_embedded: :embedded,
        generate_tags: true,
        syntax_format: :json,
        syntax_name: "#{@scope_name.split('.').drop(1).join('.')}",
        syntax_dir: options[:directory],
        tag_name: "#{@scope_name.split('.').drop(1).join('.')}_scopes.txt",
        tag_dir: options[:directory],
        should_lint: true,
    }
    options = default.merge(options)
    
    output = generate(options)

    if [:json, :vscode].include? options[:syntax_format]
        file_name = File.join(
            options[:syntax_dir],
            "#{options[:syntax_name]}.tmLanguage.json",
        )
        out_file = File.open(file_name, "w")
        out_file.write(JSON.pretty_generate(output))
        out_file.close
    elsif [:plist, :textmate, :tm_language, :xml].include? options[:syntax_format]
        require 'plist'
        file_name = File.join(
            options[:syntax_dir],
            options[:syntax_name],
        )
        out_file = File.open(file_name, "w")
        out_file.write(Plist::Emit.dump(output))
        out_file.close
    else
        puts "unexpected syntax format #{options[:syntax_format]}"
        puts "expected one of [:json, :vscode, :plist, :textmate, :tm_language, :xml]"
        raise "see above error"
    end

    return unless options[:generate_tags]

    file_name = File.join(
        options[:tag_dir],
        options[:tag_name],
    )
    new_file = File.open(file_name, "w")
    new_file.write(get_tags(output).to_a.sort.join("\n"))
    new_file.close
end
tokenMatching(token_pattern) click to toggle source

convert a regex value into a proc filter used to select patterns

@param [Regexp] argument A value that uses the tokenParsing syntax (explained below)

@note The syntax for tokenParsing is simple, there are:

- `adjectives` ex: isAClass
- the `not` operator ex: !isAClass
- the `or` operator ex: isAClass || isAPrimitive
- the `and` operator ex: isAClass && isAPrimitive
- paraentheses ex: (!isAClass) && isAPrimitive
_ 
anything matching /[a-zA-Z0-9_]+/ is considered an "adjective"
whitespace, including newlines, are removed/ignored
all other characters are invalid
_
using only an adjective, ex: /isAClass/ means to only include
Patterns that have that adjective in their adjective list

@return [TokenPattern]

# File lib/ruby_grammar_builder/tokens.rb, line 30
def tokenMatching(token_pattern)
    # create the normal pattern that will act as a placeholder until the very end
    token_pattern = TokenPattern.new({
        match: /(?#tokens)/,
        pattern_filter: parseTokenSyntax(token_pattern),
    })
    # tell it what it needs to select-later
    return token_pattern
end