= Neural Network =

A '''neural network''' is a parametric model. It is used with weights that are fit by loss minimization.

<<TableOfContents>>

----



== Introduction ==

A visualization of a neural network is:

{{attachment:visual.svg}}

There is a vector of inputs (''x'') with ''d,,x,,'' elements, which forms the first layer, and a vector of outputs (''y'') with ''d,,y,,'' elements, forming the last layer. In between there are hidden layers.

Layers are composed of nodes called neurons. Each member of the input and output vectors are nodes; the number of nodes in a hidden layer is a parameter.

In every layer (except the last), each node is connected by an edge to ''every'' node in the subsequent layer. Each edge has an unknown weight. The vector of weights (''w'') has ''d,,w,,'' elements; the number is calculated as...
 * The dimensions of every layer are indexed from 0 to ''h''.
 * Let ''d,,0,, = d,,x,,'' and ''d,,h,, = d,,y,,''.
 * ''d,,w,, = d,,h,,d,,h-1,, + d,,h-1,,d,,h-2,, + ... + d,,1,,d,,0,,''.

----



== Description ==

A neural network is a continuous mapping as {{attachment:mapping.svg}}

The neuromanifold (i.e., function space that can be parameterized) of ''ϕ'' is notated as ''ℳ,,ϕ,,'' and is defined by {{attachment:setbuilder.svg}}



=== Properties ===

''ϕ'' is piecewise smooth so ''ℳ,,ϕ,,'' is a manifold with singularities.

{{attachment:subset.svg}} so ''dim ℳ,,ϕ,, < d,,w,,''.

----



== Linear neural network ==

A linear neural network does not feature any activation functions. It can be characterized by the following:
 * Let ''r = min{d,,0,,, d,,1,,, ..., d,,h,,}''.
 * Let ''M'' be the matrix product of all weights ''W,,h,,W,,h-1,,...W,,2,,W,,1,,''.
 * ''ϕ(w,x) = Mx''.
 * {{attachment:setbuilder2.svg}}



=== Properties ===

If ''r = min{d,,0,,,d,,h,,}'' then {{attachment:setbuilder3.svg}}.

This can be interpretted as the neuromanifold not being bounded in the vector space; it has a '''filling architecture'''. On the other hand, if ''r < min{d,,0,,,d,,h,,}'', then the neuromanifold does not form a vector space; it has a '''non-filling architecture''' and is a '''determinantal variety'''.



----
CategoryRicottone