class Grammar
Represents a Textmate Grammar
Attributes
Public Class Methods
import an existing grammar from a file
@note the imported grammar is write only access to imported keys will raise an error
@param [String] path path to a json or plist grammar
@return [Grammar] The imported grammar
# File lib/textmate_grammar/grammar.rb, line 43 def self.fromTmLanguage(path) begin import_grammar = JSON.parse File.read(path) rescue JSON::ParserError require 'plist' import_grammar = Plist.parse_xml File.read(path) end grammar = ImportGrammar.new( name: import_grammar["name"], scope_name: import_grammar["scopeName"], version: import_grammar["version"] || "", description: import_grammar["description"] || nil, information_for_contributors: import_grammar["information_for_contributors"] || nil, fileTypes: import_grammar["fileTypes"] || nil, ) # import "patterns" into @repository[:$initial_context] grammar.repository[:$initial_context] = import_grammar["patterns"] # import the rest of the repository import_grammar["repository"].each do |key, value| # repository keys are kept as a hash grammar.repository[key.to_sym] = value end grammar end
Import a grammar partial
@note the import is “dynamic”, changes made to the grammar partial after the import
wil be reflected in the parent grammar
@param [String, ExportableGrammar] path_or_export the grammar partial or the file
in which the grammar partial is declared
@return [ExportableGrammar]
# File lib/textmate_grammar/grammar.rb, line 80 def self.import(path_or_export) export = path_or_export unless path_or_export.is_a? ExportableGrammar # allow for relative paths if not Pathname.new(path_or_export).absolute? relative_path = File.dirname(caller_locations[0].path) if not Pathname.new(relative_path).absolute? relative_path = File.join(Dir.pwd,relative_path) end path_or_export = File.join(relative_path, path_or_export) end require path_or_export resolved = File.expand_path resolve_require(path_or_export) export = @@export_grammars.dig(resolved, :grammar) unless export.is_a? ExportableGrammar raise "#{path_or_export} does not create a Exportable Grammar" end end return export.export end
Create a new Grammar
@param [Hash] keys The grammar keys @option keys [String] :name The name of the grammar @option keys [String] :scope_name The scope_name
of teh grammar, must start with
+source.+ or +text.+
@option keys [String, :auto] :version (:auto) the version of the grammar, :auto uses
the current git commit as the version
@option keys [Array] :patterns ([]) ignored, will be replaced with the initial context @option keys [Hash] :repository ({}) ignored, will be replaced by the generated rules @option keys all remaining options will be copied to the grammar without change
# File lib/textmate_grammar/grammar.rb, line 116 def initialize(keys) required_keys = [:name, :scope_name] unless required_keys & keys.keys == required_keys puts "Missing one or more of the required grammar keys" puts "Missing: #{required_keys - (required_keys & keys.keys)}" puts "The required grammar keys are: #{required_keys}" raise "See above error" end @name = keys[:name] @scope_name = keys[:scope_name] @repository = {} keys.delete :name keys.delete :scope_name # auto versioning, when save_to is called grab the latest git commit or "" if not # a git repo keys[:version] ||= :auto @keys = keys.compact return if @scope_name == "export" || @scope_name.start_with?("source.", "text.") puts "Warning: grammar scope name should start with `source.' or `text.'" puts "Examples: source.cpp text.html text.html.markdown source.js.regexp" end
Gets all registered plugins
@api private
@return [Array<GrammarPlugin>] A list of all plugins
# File lib/textmate_grammar/grammar_plugin.rb, line 148 def self.plugins @@linters + @@transforms.values.flatten.map { |v| v[:transform] } end
Register a linter plugin
@param [GrammarLinter] linter the linter plugin
@return [void] nothing
# File lib/textmate_grammar/grammar_plugin.rb, line 108 def self.register_linter(linter) @@linters << linter end
Register a transformation plugin
@param [GrammarTransform] transform the transformation plugin @param [Numeric] priority an optional priority
@note The priority controls when a transformation runs in relation to
other events in addition to ordering transformations priorities < 100 have their pre transform run before pre linters priorities >= 100 have their pre transform run after pre linters priorities >= 200 do not have their pre_transform function ran priorities < 300 have their post transorm run before post linters priorities >= 300 have their post transorm run before post linters
@return [void] nothing
# File lib/textmate_grammar/grammar_plugin.rb, line 128 def self.register_transform(transform, priority = 150) key = if priority < 100 then :before_pre_linter elsif priority < 200 then :after_pre_linter elsif priority < 300 then :before_post_linter else :after_pre_linter end @@transforms[key] << { priority: priority, transform: transform, } end
Removes a plugin whose classname matches plugin
@param [#to_s] plugin The plugin name to remove
@return [void]
# File lib/textmate_grammar/grammar_plugin.rb, line 159 def self.remove_plugin(plugin) @@linters.delete_if { |linter| linter.class.to_s == plugin.to_s } @@transforms[:before_pre_linter].delete_if { |t| t[:transform].class.to_s == plugin.to_s } @@transforms[:after_pre_linter].delete_if { |t| t[:transform].class.to_s == plugin.to_s } @@transforms[:before_post_linter].delete_if { |t| t[:transform].class.to_s == plugin.to_s } @@transforms[:after_post_linter].delete_if { |t| t[:transform].class.to_s == plugin.to_s } end
Public Instance Methods
Access a pattern in the grammar
@param [Symbol] key The key the pattern is stored in
@return [PatternBase, Symbol, Array<PatternBase, Symbol>] The stored pattern
# File lib/textmate_grammar/grammar.rb, line 149 def [](key) if key.is_a?(Regexp) tokenMatching(key) # see tokens.rb else @repository.fetch(key, PlaceholderPattern.new(key)) end end
Store a pattern
A pattern must be stored in the grammar for it to appear in the final grammar
The special key :$initial_context is the pattern that will be matched at the beginning of the document or whenever the root of the grammar is to be matched
@param [Symbol] key The key to store the pattern in @param [PatternBase, Symbol, Array<PatternBase, Symbol>] value the pattern to store
@return [PatternBase, Symbol, Array<PatternBase, Symbol>] the stored pattern
# File lib/textmate_grammar/grammar.rb, line 170 def []=(key, value) unless key.is_a? Symbol raise "Use symbols not strings" unless key.is_a? Symbol end if key.to_s.start_with?("$") && !([:$initial_context, :$base, :$self].include? key) puts "#{key} is not a valid repository name" puts "repository names starting with $ are reserved" raise "See above error" end if key.to_s == "repository" puts "#{key} is not a valid repository name" puts "the name 'repository' is a reserved name" raise "See above error" end # add it to the repository @repository[key] = fixup_value(value) @repository[key] end
Returns the version information
@api private
@return [String] The version string to use
# File lib/textmate_grammar/grammar.rb, line 512 def auto_version return @keys[:version] unless @keys[:version] == :auto `git rev-parse HEAD`.strip rescue StandardError "" end
Convert the grammar into a hash suitable for exporting to a file
@param [Symbol] inherit_or_embedded Is this grammar being inherited
from, or will be embedded, this controls if :$initial_context is mapped to +"$base"+ or +"$self"+
@return [Hash] the generated grammar
# File lib/textmate_grammar/grammar.rb, line 296 def generate(options) default = { inherit_or_embedded: :embedded, should_lint: true, } options = default.merge(options) repo = @repository.__deep_clone__ repo = run_pre_transform_stage(repo, :before_pre_linter) if options[:should_lint] @@linters.each do |linter| repo.each do |_, potential_pattern| [potential_pattern].flatten.each do |each_potential_pattern| raise "linting failed, see above error" unless linter.pre_lint( each_potential_pattern, filter_options( linter, each_potential_pattern, grammar: self, repository: repo, ), ) end end end end repo = run_pre_transform_stage(repo, :after_pre_linter) convert_initial_context = lambda do |potential_pattern| if potential_pattern == :$initial_context return (options[:inherit_or_embedded] == :embedded) ? :$self : :$base end if potential_pattern.is_a? Array return potential_pattern.map do |nested_potential_pattern| convert_initial_context.call(nested_potential_pattern) end end if potential_pattern.is_a? PatternBase return potential_pattern.transform_includes do |each_nested_potential_pattern| # transform includes will call this block again if each_* is a patternBase if each_nested_potential_pattern.is_a? PatternBase next each_nested_potential_pattern end convert_initial_context.call(each_nested_potential_pattern) end end return potential_pattern end repo = repo.transform_values do |each_potential_pattern| convert_initial_context.call(each_potential_pattern) end output = { name: @name, scopeName: @scope_name, } to_tag = lambda do |potential_pattern| case potential_pattern when Array return { "patterns" => potential_pattern.map do |nested_potential_pattern| to_tag.call(nested_potential_pattern) end, } when Symbol then return {"include" => "#" + potential_pattern.to_s} when Hash then return potential_pattern when String then return {"include" => potential_pattern} when PatternBase then return potential_pattern.to_tag else raise "Unexpected value: #{potential_pattern.class}" end end output[:repository] = repo.transform_values do |each_potential_pattern| to_tag.call(each_potential_pattern) end # sort repos by key name output[:repository] = Hash[output[:repository].sort_by { |key, _| key.to_s }] output[:patterns] = output[:repository][:$initial_context] output[:patterns] ||= [] output[:patterns] = output[:patterns]["patterns"] if output[:patterns].is_a? Hash output[:repository].delete(:$initial_context) output[:version] = auto_version output.merge!(@keys) { |_key, old, _new| old } output = run_post_transform_stage(output, :before_pre_linter) output = run_post_transform_stage(output, :after_pre_linter) output = run_post_transform_stage(output, :before_post_linter) @@linters.each do |linter| raise "linting failed, see above error" unless linter.post_lint(output) end output = run_post_transform_stage(output, :after_post_linter) Hash[ output.sort_by do |key, _| order = { information_for_contributors: 0, version: 1, name: 2, scopeName: 3, fileTypes: 4, unknown_keys: 5, patterns: 6, repository: 7, uuid: 8, } next order[key.to_sym] if order.has_key? key.to_sym order[:unknown_keys] end ] end
Import a grammar partial into this grammar
@note the import is “dynamic”, changes made to the grammar partial after the import
wil be reflected in the parent grammar
@param [String, ExportableGrammar] path_or_export the grammar partial or the file
in which the grammar partial is declared
@return [void] nothing
# File lib/textmate_grammar/grammar.rb, line 203 def import(path_or_export) unless path_or_export.is_a? ExportableGrammar relative_path = File.dirname(caller_locations[0].path) if not Pathname.new(relative_path).absolute? relative_path = File.join(Dir.pwd,relative_path) end # allow for relative paths if not Pathname.new(path_or_export).absolute? path_or_export = File.join(relative_path, path_or_export) end end export = Grammar.import(path_or_export) export.parent_grammar = self # import the repository @repository = @repository.merge export.repository do |_key, old_val, new_val| [old_val, new_val].flatten.uniq end end
convert a regex value into a proc filter used to select patterns
@param [Regexp] argument A value that uses the tokenParsing syntax (explained below)
@note The syntax for tokenParsing is simple, there are:
- `adjectives` ex: isAClass - the `not` operator ex: !isAClass - the `or` operator ex: isAClass || isAPrimitive - the `and` operator ex: isAClass && isAPrimitive - paraentheses ex: (!isAClass) && isAPrimitive _ anything matching /[a-zA-Z0-9_]+/ is considered an "adjective" whitespace, including newlines, are removed/ignored all other characters are invalid _ using only an adjective, ex: /isAClass/ means to only include Patterns that have that adjective in their adjective list
@return [proc] a function that accepts a Pattern
as input, and returns
a boolean of whether or not that pattern should be included
# File lib/textmate_grammar/tokens.rb, line 64 def parseTokenSyntax(argument) # validate input type if !argument.is_a?(Regexp) raise <<~HEREDOC Trying to call parseTokenSyntax() but the argument isn't Regexp its #{argument.class} value: #{argument} HEREDOC end # just remove the //'s from the string regex_content = argument.inspect[1...-1] # remove all invalid characters, make sure length didn't change invalid_characters_removed = regex_content.gsub(/[^a-zA-Z0-9_&|\(\)! \n]/, "") if invalid_characters_removed.length != regex_content.length raise <<~HEREDOC It appears the tokenSyntax #{argument.inspect} contains some invalid characters with invalid characters: #{regex_content.inspect} without invalid characters: #{invalid_characters_removed.inspect} HEREDOC end # find broken syntax if regex_content =~ /[a-zA-Z0-9_]+\s+[a-zA-Z0-9_]+/ raise <<~HEREDOC Inside a tokenSyntax: #{argument.inspect} this part of the syntax is invalid: #{$&.inspect} (theres a space between two adjectives) My guess is that it was half-edited or an accidental space was added HEREDOC end # convert all adjectives into inclusion checks regex_content.gsub!(/\s+/," ") regex_content.gsub!(/[a-zA-Z0-9_]+/, 'pattern.arguments[:adjectives].include?(:\0)') # convert it into a proc return ->(pattern) do puts "regex_content is: #{regex_content} " eval(regex_content) if pattern.is_a?(PatternBase) && pattern.arguments[:adjectives].is_a?(Array) end end
Runs a set of post transformations
@param [Hash] output The generated grammar @param [Symbol] stage the stage to run
@return [Hash] The modified grammar
# File lib/textmate_grammar/grammar.rb, line 278 def run_post_transform_stage(output, stage) @@transforms[stage] .sort { |a, b| a[:priority] <=> b[:priority] } .map { |a| a[:transform] } .each { |transform| output = transform.post_transform(output) } output end
Runs a set of pre transformations
@api private
@param [Hash] repository The repository @param [:before_pre_linter,:after_pre_linter] stage the stage to run
@return [Hash] the modified repository
# File lib/textmate_grammar/grammar.rb, line 235 def run_pre_transform_stage(repository, stage) @@transforms[stage] .sort { |a, b| a[:priority] <=> b[:priority] } .map { |a| a[:transform] } .each do |transform| repository = repository.transform_values do |potential_pattern| if potential_pattern.is_a? Array potential_pattern.map do |each| transform.pre_transform( each, filter_options( transform, each, grammar: self, repository: repository, ), ) end else transform.pre_transform( potential_pattern, filter_options( transform, potential_pattern, grammar: self, repository: repository, ), ) end end end repository end
Save the grammar to a path
@param [Hash] options options to save_to
@option options :inherit_or_embedded (:embedded) see generate
@option options :generate_tags [Boolean] (true) generate a list of all +:tag_as+s @option options :directory [String] the location to generate the files @option options :tag_dir [String] (File.join(options,“language_tags”)) the
directory to generate language tags in
@option options :syntax_dir [String] (File.join(options,“syntaxes”)) the
directory to generate the syntax file in
@option options :syntax_format [:json,:vscode,:plist,:textmate,:tm_language,:xml]
(:json) The format to generate the syntax file in
@option options :syntax_name [String] (“#{@name}.tmLanguage”) the name of the syntax
file to generate without the extension
@option options :tag_name [String] (“#{@name}-scopes.txt”) the name of the tag list
file to generate without the extension
@note all keys except :directory is optional @note :directory is optional if both :tag_dir and :syntax_dir are specified @note currently :vscode is an alias for :json @note currently :textmate, :tm_language, and :xml are aliases for :plist @note later the aliased :syntax_type choices may enable compatibility features
@return [void] nothing
# File lib/textmate_grammar/grammar.rb, line 445 def save_to(options) options[:directory] ||= "." # make the path absolute absolute_path_from_caller = File.dirname(caller_locations[0].path) if not Pathname.new(absolute_path_from_caller).absolute? absolute_path_from_caller = File.join(Dir.pwd,absolute_path_from_caller) end if not Pathname.new(options[:directory]).absolute? options[:directory] = File.join(absolute_path_from_caller, options[:directory]) end default = { inherit_or_embedded: :embedded, generate_tags: true, syntax_format: :json, syntax_name: "#{@scope_name.split('.').drop(1).join('.')}", syntax_dir: options[:directory], tag_name: "#{@scope_name.split('.').drop(1).join('.')}_scopes.txt", tag_dir: options[:directory], should_lint: true, } options = default.merge(options) output = generate(options) if [:json, :vscode].include? options[:syntax_format] file_name = File.join( options[:syntax_dir], "#{options[:syntax_name]}.tmLanguage.json", ) out_file = File.open(file_name, "w") out_file.write(JSON.pretty_generate(output)) out_file.close elsif [:plist, :textmate, :tm_language, :xml].include? options[:syntax_format] require 'plist' file_name = File.join( options[:syntax_dir], options[:syntax_name], ) out_file = File.open(file_name, "w") out_file.write(Plist::Emit.dump(output)) out_file.close else puts "unexpected syntax format #{options[:syntax_format]}" puts "expected one of [:json, :vscode, :plist, :textmate, :tm_language, :xml]" raise "see above error" end return unless options[:generate_tags] file_name = File.join( options[:tag_dir], options[:tag_name], ) new_file = File.open(file_name, "w") new_file.write(get_tags(output).to_a.sort.join("\n")) new_file.close end
convert a regex value into a proc filter used to select patterns
@param [Regexp] argument A value that uses the tokenParsing syntax (explained below)
@note The syntax for tokenParsing is simple, there are:
- `adjectives` ex: isAClass - the `not` operator ex: !isAClass - the `or` operator ex: isAClass || isAPrimitive - the `and` operator ex: isAClass && isAPrimitive - paraentheses ex: (!isAClass) && isAPrimitive _ anything matching /[a-zA-Z0-9_]+/ is considered an "adjective" whitespace, including newlines, are removed/ignored all other characters are invalid _ using only an adjective, ex: /isAClass/ means to only include Patterns that have that adjective in their adjective list
@return [TokenPattern]
# File lib/textmate_grammar/tokens.rb, line 30 def tokenMatching(token_pattern) # create the normal pattern that will act as a placeholder until the very end token_pattern = TokenPattern.new({ match: /(?#tokens)/, pattern_filter: parseTokenSyntax(token_pattern), }) # tell it what it needs to select-later return token_pattern end