Neural Network

A neural network is a parametric model. It is used with weights that are fit by loss minimization.

Introduction

A visualization of a neural network is:

There is a vector of inputs (x) with d_x elements, which forms the first layer, and a vector of outputs (y) with d_y elements, forming the last layer. In between there are hidden layers.

Layers are composed of nodes called neurons. Each member of the input and output vectors are nodes; the number of nodes in a hidden layer is a parameter.

In every layer (except the last), each node is connected by an edge to every node in the subsequent layer. Each edge has an unknown weight. The vector of weights (w) has d_w elements; the number is calculated as...

• The dimensions of every layer are indexed from 0 to h.

• Let d₀ = d_x and d_h = d_y.

• d_w = d_hd_h-1 + d_h-1d_h-2 + ... + d₁d₀.

Description

A neural network is a continuous mapping as

The neuromanifold (i.e., function space that can be parameterized) of ϕ is notated as ℳ_ϕ and is defined by

Properties

ϕ is piecewise smooth so ℳ_ϕ is a manifold with singularities.

so dim ℳ_ϕ < d_w.

Linear neural network

A linear neural network does not feature any activation functions. It can be characterized by the following:

• Let r = min{d₀, d₁, ..., d_h}.

• Let M be the matrix product of all weights W_hW_h-1...W₂W₁.

• ϕ(w,x) = Mx.

• 

Properties

If r = min{d₀,d_h} then .

This can be interpretted as the neuromanifold not being bounded in the vector space; it has a filling architecture. On the other hand, if r < min{d₀,d_h}, then the neuromanifold does not form a vector space; it has a non-filling architecture and is a determinantal variety.

CategoryRicottone