class RequestLogAnalyzer::Controller

The RequestLogAnalyzer::Controller class creates a LogParser instance for the requested file format and connect it with sources and aggregators.

Sources are streams or files from which the requests will be parsed. Aggregators will handle every passed request to yield a meaningfull results.

Note that the order of sources can be imported if you have log files than succeed eachother. Requests that span over succeeding files will be parsed correctly if the sources are registered in the correct order. This can be helpful to parse requests from several logrotated log files.

Attributes

aggregators[R]
filters[R]
options[R]
output[R]
source[R]

Public Class Methods

build(options) click to toggle source

Build a new controller. Returns a new RequestLogAnalyzer::Controller object.

Options

  • :after Drop all requests after this date (Date, DateTime, Time, or a String in “YYYY-MM-DD hh:mm:ss” format)

  • :aggregator Array of aggregators (Strings or Symbols for the builtin aggregators or a RequestLogAnalyzer::Aggregator class - Defaults to [:summarizer]).

  • :boring Do not show color on STDOUT (Defaults to false).

  • :before Drop all requests before this date (Date, DateTime, Time or a String in “YYYY-MM-DD hh:mm:ss” format)

  • :database Database file to insert encountered requests to.

  • :debug Enables echo aggregator which will echo each request analyzed.

  • :file Filestring, File or StringIO.

  • :format :rails, {:apache => ‘FORMATSTRING’}, :merb, :amazon_s3, :mysql or RequestLogAnalyzer::FileFormat class. (Defaults to :rails).

  • :mail Email the results to this email address.

  • :mailhost Email the results to this mail server.

  • :mailfrom Set the Email sender address.

  • :mailfrom_alias Set the Email sender name.

  • :mailsubject Email subject.

  • :no_progress Do not display the progress bar (increases parsing speed).

  • :output ‘FixedWidth’, ‘HTML’ or RequestLogAnalyzer::Output class. Defaults to ‘FixedWidth’.

  • :reject Reject specific {:field => :value} combination (expects a single hash).

  • :report_width Width of reports in characters for FixedWidth reports. (Defaults to 80)

  • :reset_database Reset the database before starting.

  • :select Select specific {:field => :value} combination (expects a single hash).

  • :source_files Source files to analyze. Provide either File, array of files or STDIN.

  • :yaml Output to YAML file.

  • :silent Minimal output automatically implies :no_progress

  • :source The class to instantiate to grab the requestes, must be a RequestLogAnalyzer::Source::Base descendant. (Defaults to RequestLogAnalyzer::Source::LogParser)

Example

RequestLogAnalyzer::Controller.build(

:output       => :HTML,
:mail         => 'root@localhost',
:after        => Time.now - 24*60*60,
:source_files => '/var/log/passenger.log'

).run!

Todo

  • Check if defaults work (Aggregator defaults seem wrong).

  • Refactor :database => options, :dump => options away from contoller intialization.

    # File lib/request_log_analyzer/controller.rb
133 def self.build(options)
134   # Defaults
135   options[:output]        ||= 'FixedWidth'
136   options[:format]        ||= :rails
137   options[:aggregator]    ||= [:summarizer]
138   options[:report_width]  ||= 80
139   options[:report_amount] ||= 20
140   options[:report_sort]   ||= 'sum,mean'
141   options[:boring]        ||= false
142   options[:silent]        ||= false
143   options[:source]        ||= RequestLogAnalyzer::Source::LogParser
144 
145   options[:no_progress] = true if options[:silent]
146 
147   # Deprecation warnings
148   if options[:dump]
149     warn '[DEPRECATION] `:dump` is deprecated.  Please use `:yaml` instead.'
150     options[:yaml]          = options[:dump]
151   end
152 
153   # Set the output class
154   output_args   = {}
155   output_object = nil
156   if options[:output].is_a?(Class)
157     output_class = options[:output]
158   else
159     output_class = RequestLogAnalyzer::Output.const_get(options[:output])
160   end
161 
162   output_sort   = options[:report_sort].split(',').map { |s| s.to_sym }
163   output_amount = options[:report_amount] == 'all' ? :all : options[:report_amount].to_i
164 
165   if options[:file]
166     output_object = %w(        File StringIO        ).include?(options[:file].class.name) ? options[:file] : File.new(options[:file], 'w+')
167     output_args   = { width: 80, color: false, characters: :ascii, sort: output_sort, amount: output_amount }
168   elsif options[:mail]
169     output_object = RequestLogAnalyzer::Mailer.new(options[:mail], options[:mailhost], subject: options[:mailsubject], from: options[:mailfrom], from_alias: options[:mailfrom_name])
170     output_args   = { width: 80, color: false, characters: :ascii, sort: output_sort, amount: output_amount  }
171   else
172     output_object = STDOUT
173     output_args   = { width: options[:report_width].to_i, color: !options[:boring],
174                     characters: (options[:boring] ? :ascii : :utf), sort: output_sort, amount: output_amount }
175   end
176 
177   output_instance = output_class.new(output_object, output_args)
178 
179   # Create the controller with the correct file format
180   if options[:format].is_a?(Hash)
181     file_format = RequestLogAnalyzer::FileFormat.load(options[:format].keys[0], options[:format].values[0])
182   else
183     file_format = RequestLogAnalyzer::FileFormat.load(options[:format])
184   end
185 
186   # Kickstart the controller
187   controller =
188     Controller.new(options[:source].new(file_format,
189                                         source_files: options[:source_files],
190                                         parse_strategy: options[:parse_strategy]),
191                    output: output_instance,
192                    database: options[:database],                # FUGLY!
193                    yaml: options[:yaml],
194                    reset_database: options[:reset_database],
195                    no_progress: options[:no_progress],
196                    silent: options[:silent]
197                    )
198 
199   # register filters
200   if options[:after] || options[:before]
201     filter_options = {}
202     [:after, :before].each do |filter|
203       case options[filter]
204       when Date, DateTime, Time
205         filter_options[filter] = options[filter]
206       when String
207         filter_options[filter] = DateTime.parse(options[filter])
208       end
209     end
210     controller.add_filter(:timespan, filter_options)
211   end
212 
213   if options[:reject]
214     options[:reject].each do |(field, value)|
215       controller.add_filter(:field, mode: :reject, field: field, value: value)
216     end
217   end
218 
219   if options[:select]
220     options[:select].each do |(field, value)|
221       controller.add_filter(:field, mode: :select, field: field, value: value)
222     end
223   end
224 
225   # register aggregators
226   options[:aggregator].each { |agg| controller.add_aggregator(agg) }
227   controller.add_aggregator(:summarizer)          if options[:aggregator].empty?
228   controller.add_aggregator(:echo)                if options[:debug]
229   controller.add_aggregator(:database_inserter)   if options[:database] && !options[:aggregator].include?('database')
230 
231   file_format.setup_environment(controller)
232   controller
233 end
build_from_arguments(arguments) click to toggle source

Builds a RequestLogAnalyzer::Controller given parsed command line arguments <tt>arguments<tt> A CommandLine::Arguments hash containing parsed commandline parameters.

   # File lib/request_log_analyzer/controller.rb
20 def self.build_from_arguments(arguments)
21   options = {}
22 
23   # Copy fields
24   options[:database]       = arguments[:database]
25   options[:reset_database] = arguments[:reset_database]
26   options[:debug]          = arguments[:debug]
27   options[:yaml]           = arguments[:yaml] || arguments[:dump]
28   options[:mail]           = arguments[:mail]
29   options[:no_progress]    = arguments[:no_progress]
30   options[:format]         = arguments[:format]
31   options[:output]         = arguments[:output]
32   options[:file]           = arguments[:file]
33   options[:after]          = arguments[:after]
34   options[:before]         = arguments[:before]
35   options[:reject]         = arguments[:reject]
36   options[:select]         = arguments[:select]
37   options[:boring]         = arguments[:boring]
38   options[:aggregator]     = arguments[:aggregator]
39   options[:report_width]   = arguments[:report_width]
40   options[:report_sort]    = arguments[:report_sort]
41   options[:report_amount]  = arguments[:report_amount]
42   options[:mailhost]       = arguments[:mailhost]
43   options[:mailfrom]       = arguments[:mailfrom]
44   options[:mailfrom_name]  = arguments[:mailfrom_name]
45   options[:mailsubject]    = arguments[:mailsubject]
46   options[:silent]         = arguments[:silent]
47   options[:parse_strategy] = arguments[:parse_strategy]
48 
49   # Apache format workaround
50   if arguments[:rails_format]
51     options[:format] = { rails: arguments[:rails_format] }
52   elsif arguments[:apache_format]
53     options[:format] = { apache: arguments[:apache_format] }
54   end
55 
56   # Handle output format casing
57   if options[:output].class == String
58     options[:output] = 'HTML'       if options[:output] =~ /^html$/i
59     options[:output] = 'FixedWidth' if options[:output] =~ /^fixed_?width$/i
60   end
61 
62   # Register sources
63   if arguments.parameters.length == 1
64     file = arguments.parameters[0]
65     if file == '-' || file == 'STDIN'
66       options.store(:source_files, $stdin)
67     elsif File.exist?(file)
68       options.store(:source_files, file)
69     else
70       puts "File not found: #{file}"
71       exit(0)
72     end
73   else
74     options.store(:source_files, arguments.parameters)
75   end
76 
77   # Guess file format
78   if !options[:format] && options[:source_files]
79     options[:format] = :rails3 # Default
80 
81     if options[:source_files] != $stdin
82       if options[:source_files].class == String
83         options[:format] = RequestLogAnalyzer::FileFormat.autodetect(options[:source_files])
84 
85       elsif options[:source_files].class == Array && options[:source_files].first != $stdin
86         options[:format] = RequestLogAnalyzer::FileFormat.autodetect(options[:source_files].first)
87       end
88     end
89   end
90 
91   build(options)
92 end
new(source, options = {}) click to toggle source

Builds a new Controller for the given log file format. format Logfile format. Defaults to :rails Options are passd on to the LogParser.

  • :database Database the controller should use.

  • :yaml Yaml Dump the contrller should use.

  • :output All report outputs get << through this output.

  • :no_progress No progress bar

  • :silent Minimal output, only error

    # File lib/request_log_analyzer/controller.rb
243 def initialize(source, options = {})
244   @source      = source
245   @options     = options
246   @aggregators = []
247   @filters     = []
248   @output      = options[:output]
249   @interrupted = false
250 
251   # Register the request format for this session after checking its validity
252   fail 'Invalid file format!' unless @source.file_format.valid?
253 
254   # Install event handlers for wrnings, progress updates and source changes
255   @source.warning        = lambda { |type, message, lineno|  @aggregators.each { |agg| agg.warning(type, message, lineno) } }
256   @source.progress       = lambda { |message, value| handle_progress(message, value) } unless options[:no_progress]
257   @source.source_changes = lambda { |change, filename| handle_source_change(change, filename) }
258 end

Public Instance Methods

>>(agg)
Alias for: add_aggregator
add_aggregator(agg) click to toggle source

Adds an aggregator to the controller. The aggregator will be called for every request that is parsed from the provided sources (see add_source)

    # File lib/request_log_analyzer/controller.rb
288 def add_aggregator(agg)
289   agg = RequestLogAnalyzer::Aggregator.const_get(RequestLogAnalyzer.to_camelcase(agg)) if agg.is_a?(String) || agg.is_a?(Symbol)
290   @aggregators << agg.new(@source, @options)
291 end
Also aliased as: >>
add_filter(filter, filter_options = {}) click to toggle source

Adds a request filter to the controller.

    # File lib/request_log_analyzer/controller.rb
296 def add_filter(filter, filter_options = {})
297   filter = RequestLogAnalyzer::Filter.const_get(RequestLogAnalyzer.to_camelcase(filter)) if filter.is_a?(Symbol)
298   @filters << filter.new(source.file_format, @options.merge(filter_options))
299 end
aggregate_request(request) click to toggle source

Push a request to all the aggregators (@aggregators). request The request to push to the aggregators.

    # File lib/request_log_analyzer/controller.rb
314 def aggregate_request(request)
315   return false unless request
316   @aggregators.each { |agg| agg.aggregate(request) }
317   true
318 end
filter_request(request) click to toggle source

Push a request through the entire filterchain (@filters). request The request to filter. Returns the filtered request or nil.

    # File lib/request_log_analyzer/controller.rb
304 def filter_request(request)
305   @filters.each do |filter|
306     request = filter.filter(request)
307     return nil if request.nil?
308   end
309   request
310 end
handle_progress(message, value = nil) click to toggle source

Progress function. Expects :started with file, :progress with current line and :finished or :interrupted when done. message Current state (:started, :finished, :interupted or :progress). value File or current line.

    # File lib/request_log_analyzer/controller.rb
264 def handle_progress(message, value = nil)
265   case message
266   when :started
267     @progress_bar = CommandLine::ProgressBar.new(File.basename(value), File.size(value), STDERR)
268   when :finished
269     @progress_bar.finish
270     @progress_bar = nil
271   when :interrupted
272     if @progress_bar
273       @progress_bar.halt
274       @progress_bar = nil
275     end
276   when :progress
277     @progress_bar.set(value)
278   end
279 end
handle_source_change(change, filename) click to toggle source

Source change handler

    # File lib/request_log_analyzer/controller.rb
282 def handle_source_change(change, filename)
283   @aggregators.each { |agg| agg.source_change(change, File.expand_path(filename, Dir.pwd)) }
284 end
install_signal_handlers() click to toggle source
    # File lib/request_log_analyzer/controller.rb
361 def install_signal_handlers
362   Signal.trap('INT') do
363     handle_progress(:interrupted)
364     puts 'Caught interrupt! Stopping parsing...'
365     @interrupted = true
366   end
367 end
run!() click to toggle source

Runs RequestLogAnalyzer

  1. Call prepare on every aggregator

  2. Generate requests from source object

  3. Filter out unwanted requests

  4. Call aggregate for remaning requests on every aggregator

  5. Call finalize on every aggregator

  6. Call report on every aggregator

  7. Finalize Source

    # File lib/request_log_analyzer/controller.rb
328 def run!
329   # @aggregators.each{|agg| p agg}
330 
331   @aggregators.each { |agg| agg.prepare }
332   install_signal_handlers
333 
334   @source.each_request do |request|
335     break if @interrupted
336     aggregate_request(filter_request(request))
337   end
338 
339   @aggregators.each { |agg| agg.finalize }
340 
341   @output.header
342   @aggregators.each { |agg| agg.report(@output) }
343   @output.footer
344 
345   @source.finalize
346 
347   if @output.io.is_a?(File)
348     unless @options[:silent]
349       puts
350       puts 'Report written to: ' + File.expand_path(@output.io.path)
351       puts 'Need an expert to analyze your application?'
352       puts 'Mail to contact@railsdoctors.com or visit us at http://railsdoctors.com'
353       puts 'Thanks for using request-log-analyzer!'
354     end
355     @output.io.close
356   elsif @output.io.is_a?(RequestLogAnalyzer::Mailer)
357     @output.io.mail
358   end
359 end