class MachineLearningWorkbench::NeuralNetwork::Base

Neural Network base class

Attributes

act_fn[R]

@!attribute [r] layers

List of matrices, each being the weights
connecting a layer's inputs (rows) to a layer's neurons (columns),
hence its shape is `[ninputs, nneurs]`
@return [Array<NArray>] list of weight matrices, each uniquely describing a layer
TODO: return a NArray after the usage of `#map` is figured out

@!attribute [r] state

It's a list of one-dimensional matrices, each an input to a layer, plus the output layer's output. The first element is the input to the first layer of the network, which is composed of the network's input, possibly the first layer's activation on the last input (recursion), and a bias (fixed `1`). The second to but-last entries follow the same structure, but with the previous layer's output in place of the network's input. The last entry is the activation of the output layer, without additions since it's not used as an input by anyone.
TODO: return a NArray after the usage of `#map` is figured out
@return [Array<NArray>] current state of the network.

@!attribute [r] act_fn

activation function, common to all neurons (for now)
@return [#call] activation function

@!attribute [r] struct

list of number of (inputs or) neurons in each layer
@return [Array<Integer>] structure of the network
act_fn_name[R]

@!attribute [r] layers

List of matrices, each being the weights
connecting a layer's inputs (rows) to a layer's neurons (columns),
hence its shape is `[ninputs, nneurs]`
@return [Array<NArray>] list of weight matrices, each uniquely describing a layer
TODO: return a NArray after the usage of `#map` is figured out

@!attribute [r] state

It's a list of one-dimensional matrices, each an input to a layer, plus the output layer's output. The first element is the input to the first layer of the network, which is composed of the network's input, possibly the first layer's activation on the last input (recursion), and a bias (fixed `1`). The second to but-last entries follow the same structure, but with the previous layer's output in place of the network's input. The last entry is the activation of the output layer, without additions since it's not used as an input by anyone.
TODO: return a NArray after the usage of `#map` is figured out
@return [Array<NArray>] current state of the network.

@!attribute [r] act_fn

activation function, common to all neurons (for now)
@return [#call] activation function

@!attribute [r] struct

list of number of (inputs or) neurons in each layer
@return [Array<Integer>] structure of the network
layers[R]

@!attribute [r] layers

List of matrices, each being the weights
connecting a layer's inputs (rows) to a layer's neurons (columns),
hence its shape is `[ninputs, nneurs]`
@return [Array<NArray>] list of weight matrices, each uniquely describing a layer
TODO: return a NArray after the usage of `#map` is figured out

@!attribute [r] state

It's a list of one-dimensional matrices, each an input to a layer, plus the output layer's output. The first element is the input to the first layer of the network, which is composed of the network's input, possibly the first layer's activation on the last input (recursion), and a bias (fixed `1`). The second to but-last entries follow the same structure, but with the previous layer's output in place of the network's input. The last entry is the activation of the output layer, without additions since it's not used as an input by anyone.
TODO: return a NArray after the usage of `#map` is figured out
@return [Array<NArray>] current state of the network.

@!attribute [r] act_fn

activation function, common to all neurons (for now)
@return [#call] activation function

@!attribute [r] struct

list of number of (inputs or) neurons in each layer
@return [Array<Integer>] structure of the network
state[R]

@!attribute [r] layers

List of matrices, each being the weights
connecting a layer's inputs (rows) to a layer's neurons (columns),
hence its shape is `[ninputs, nneurs]`
@return [Array<NArray>] list of weight matrices, each uniquely describing a layer
TODO: return a NArray after the usage of `#map` is figured out

@!attribute [r] state

It's a list of one-dimensional matrices, each an input to a layer, plus the output layer's output. The first element is the input to the first layer of the network, which is composed of the network's input, possibly the first layer's activation on the last input (recursion), and a bias (fixed `1`). The second to but-last entries follow the same structure, but with the previous layer's output in place of the network's input. The last entry is the activation of the output layer, without additions since it's not used as an input by anyone.
TODO: return a NArray after the usage of `#map` is figured out
@return [Array<NArray>] current state of the network.

@!attribute [r] act_fn

activation function, common to all neurons (for now)
@return [#call] activation function

@!attribute [r] struct

list of number of (inputs or) neurons in each layer
@return [Array<Integer>] structure of the network
struct[R]

@!attribute [r] layers

List of matrices, each being the weights
connecting a layer's inputs (rows) to a layer's neurons (columns),
hence its shape is `[ninputs, nneurs]`
@return [Array<NArray>] list of weight matrices, each uniquely describing a layer
TODO: return a NArray after the usage of `#map` is figured out

@!attribute [r] state

It's a list of one-dimensional matrices, each an input to a layer, plus the output layer's output. The first element is the input to the first layer of the network, which is composed of the network's input, possibly the first layer's activation on the last input (recursion), and a bias (fixed `1`). The second to but-last entries follow the same structure, but with the previous layer's output in place of the network's input. The last entry is the activation of the output layer, without additions since it's not used as an input by anyone.
TODO: return a NArray after the usage of `#map` is figured out
@return [Array<NArray>] current state of the network.

@!attribute [r] act_fn

activation function, common to all neurons (for now)
@return [#call] activation function

@!attribute [r] struct

list of number of (inputs or) neurons in each layer
@return [Array<Integer>] structure of the network

Public Class Methods

new(struct, act_fn: nil, **act_fn_args) click to toggle source

@param struct [Array<Integer>] list of layer sizes @param act_fn [Symbol] choice of activation function for the neurons

# File lib/machine_learning_workbench/neural_network/base.rb, line 30
def initialize struct, act_fn: nil, **act_fn_args
  @struct = struct
  @act_fn_name = act_fn || :sigmoid
  @act_fn = send act_fn_name, **act_fn_args
  # @state holds both inputs, possibly recurrency, and bias
  # it is a complete input for the next layer, hence size from layer sizes
  @state = layer_row_sizes.collect do |size|
    NArray.zeros [1, size]
  end
  # to this, append a matrix to hold the final network output
  @state.push NArray.zeros [1, nneurs(-1)]
  reset_state
end

Public Instance Methods

activate(input) click to toggle source

Activate the network on a given input @param input [Array<Float>] the given input @return [Array] the activation of the output layer

# File lib/machine_learning_workbench/neural_network/base.rb, line 147
def activate input
  raise ArgumentError unless input.size == struct.first
  # load input in first state
  state[0][0...struct.first] = input
  # activate layers in sequence
  nlayers.times.each do |i|
    act = activate_layer i
    state[i+1][0...act.size] = act
  end
  return out
end
deep_reset() click to toggle source

Resets memoization: needed to play with structure modification

# File lib/machine_learning_workbench/neural_network/base.rb, line 63
def deep_reset
  # reset memoization
  [:@layer_row_sizes, :@layer_col_sizes, :@nlayers, :@layer_shapes,
   :@nweights_per_layer, :@nweights].each do |sym|
     instance_variable_set sym, nil
  end
  reset_state
end
init_random() click to toggle source

Initialize the network with random weights

# File lib/machine_learning_workbench/neural_network/base.rb, line 54
def init_random
  # Reusing `#load_weights` instead helps catching bugs
  deep_reset
  load_weights NArray.new(nweights).rand(-1,1)
end
layer_col_sizes() click to toggle source

Number of neurons per layer. Although this implementation includes inputs in the layer counts, this methods correctly ignores the input as not having neurons. @return [Array] list of neurons per each (proper) layer (i.e. no inputs)

# File lib/machine_learning_workbench/neural_network/base.rb, line 101
def layer_col_sizes
  @layer_col_sizes ||= struct.drop(1)
end
layer_shapes() click to toggle source

Shapes for the weight matrices, each corresponding to a layer @return [Array<Array[Integer, Integer]>] Weight matrix shapes

# File lib/machine_learning_workbench/neural_network/base.rb, line 109
def layer_shapes
  @layer_shapes ||= layer_row_sizes.zip layer_col_sizes
end
lecun_hyperbolic() click to toggle source

LeCun hyperbolic activation @see yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf Section 4.4

# File lib/machine_learning_workbench/neural_network/base.rb, line 177
def lecun_hyperbolic
  -> (vec) { 1.7159 * NMath.tanh(2.0*vec/3.0) + 1e-3*vec }
end
load_weights(weights) click to toggle source

Loads a plain list of weights into the weight matrices (one per layer). Preserves order. Reuses allocated memory if available. @input weights [Array<Float>] weights to load @return [true] always true. If something's wrong it simply fails, and if

all goes well there's nothing to return but a confirmation to the caller.
# File lib/machine_learning_workbench/neural_network/base.rb, line 127
def load_weights weights
  raise ArgumentError unless weights.size == nweights
  weights = weights.to_na unless weights.kind_of? NArray
  from = 0
  @layers = layer_shapes.collect do |shape|
    to = from + shape.reduce(:*)
    lay_w = weights[from...to].reshape *shape
    from = to
    lay_w
  end
  reset_state
  return true
end
logistic(steepness: 1)
Alias for: sigmoid
nlayers() click to toggle source

Count the layers. This is a computation helper, and for this implementation the inputs are considered as if a layer like the others. @return [Integer] number of layers

# File lib/machine_learning_workbench/neural_network/base.rb, line 87
def nlayers
  @nlayers ||= layer_shapes.size
end
nneurs(nlay=nil) click to toggle source

Count the neurons in a particular layer or in the whole network. @param nlay [Integer, nil] the layer of interest, 1-indexed.

`0` will return the number of inputs.
`nil` will compute the total neurons in the network.

@return [Integer] the number of neurons in a given layer, or in all network, or the number of inputs

# File lib/machine_learning_workbench/neural_network/base.rb, line 118
def nneurs nlay=nil
  nlay.nil? ? struct.reduce(:+) : struct[nlay]
end
nweights() click to toggle source

Total weights in the network @return [Integer] total number of weights

# File lib/machine_learning_workbench/neural_network/base.rb, line 74
def nweights
  @nweights ||= nweights_per_layer.reduce(:+)
end
nweights_per_layer() click to toggle source

List of per-layer number of weights @return [Array<Integer>] list of weights per each layer

# File lib/machine_learning_workbench/neural_network/base.rb, line 80
def nweights_per_layer
  @nweights_per_layer ||= layer_shapes.collect { |shape| shape.reduce(:*) }
end
out() click to toggle source

Extract and convert the output layer's activation @return [NArray] the activation of the output layer

# File lib/machine_learning_workbench/neural_network/base.rb, line 161
def out
  state.last.flatten
end
relu() click to toggle source

Rectified Linear Unit (ReLU)

# File lib/machine_learning_workbench/neural_network/base.rb, line 182
def relu
  -> (vec) { (vec>0).all? && vec || vec.class.zeros(vec.shape) }
end
reset_state() click to toggle source

Reset the network to the initial state

# File lib/machine_learning_workbench/neural_network/base.rb, line 45
def reset_state
  state.each do |s|
    s.fill 0           # reset state to zero
    s[-1] = 1        # add bias
  end
  state[-1][-1] = 0  # last layer has no bias
end
sigmoid(steepness: 1) click to toggle source

Traditional sigmoid (logistic) with variable steepness

# File lib/machine_learning_workbench/neural_network/base.rb, line 168
def sigmoid steepness: 1
  # steepness:  0<s<1 is flatter, 1<s is flatter
  # flatter makes activation less sensitive, better with large number of inputs
  -> (vec) { 1.0 / (NMath.exp(-steepness * vec) + 1.0) }
end
Also aliased as: logistic
weights() click to toggle source

Returns the weight matrix @return [Array<NArray>] list of NArray matrices of weights (one per layer).

# File lib/machine_learning_workbench/neural_network/base.rb, line 93
def weights
  layers
end