Example Back Propagation Algorithm Multi-Layer Perceptron

[tsh73]

follow-up to this thread
justbasiccom.proboards.com/thread/1025/help-learning-program

The base source is www.youtube.com/watch?v=n2L1J5JYgUk
Solved Example Back Propagation Algorithm Multi-Layer Perceptron Network by Dr. Mahesh Huddar.

Taken that video, implemented all formulas
It spits same numbers as in video (sans scientific formatting)
========== Pass 1 ==========
forward --------------------
a(4)=-0.7     O(4)=0.33181223
a(5)=0.1      O(5)=0.52497919
a(6)=-0.10453951            O(6)=0.4738889
Output is 0.4738889         Error=0.5261111
backward --------------------
delta(6)=0.13116908
delta(4)=-0.8724562e-2
delta(5)=-0.65420851e-2
dw(1,4)=-0.78521058e-2      w(1,4)=0.19214789
dw(1,5)=-0.58878766e-2      w(1,5)=-0.30588788
dw(2,4)=0     w(2,4)=0.4
dw(2,5)=0     w(2,5)=0.1
dw(3,4)=-0.78521058e-2      w(3,4)=-0.50785211
dw(3,5)=-0.58878766e-2      w(3,5)=0.19411212
dw(4,6)=0.39171154e-1       w(4,6)=-0.26082885
dw(5,6)=0.61974932e-1       w(5,6)=-0.13802507
b(4)=-0.40785211
b(5)=0.19411212
b(6)=0.21805217
========== Pass 2 ==========
forward --------------------
a(4)=-0.72355632            O(4)=0.32661034
a(5)=0.8233637e-1           O(5)=0.52057247
a(6)=0.61010722e-1          O(6)=0.51524795
Output is 0.51524795        Error=0.48475205

If you set number of passes big, output approaches target value (1)
Now, will it be of any help?
'back propagation example, by YouTube video
'link https://www.youtube.com/watch?v=n2L1J5JYgUk
'v1: forward propagation
'v2: + backward
'v3: + second pass

nX=3    'number of inputs
nHid=2    '2 hidded neurons, 1 layer, numbered 4,5
nOut=1    '1 output neuron, numbered 6
dim x(nX)    'inputs
dim w(nX+nHid,nX+nHid+nOut)    'weights
dim u(nX+nHid,nX+nHid+nOut)    '1 if used, to skip 0 then printing
dim dw(nX+nHid,nX+nHid+nOut)    'delta of weights
dim b(nX+nHid+nOut)    'bias
dim a(nX+nHid+nOut)    'temp (pre-sigmoid?) output
dim O(nX+nHid+nOut)    'neuron output
dim delta(nX+nHid+nOut)

[readData]
print "Initializing test data"
restore [teachingData]
for i = 1 to nX
   read x:x(i)=x
   O(i)=x
   print x(i);" ";
next
read Target
print Target
print
print "-----------"

[readNeurons]
print "Initializing neurons"
for i = 1 to nHid+nOut
   read idN, numInp
   print idN, numInp
   for j = 1 to numInp
       read idInp, w
       w(idInp,idN)=w
       u(idInp,idN)=1
       print , idInp, w(idInp,idN)
   next
   read b: b(idN)=b
   print ,,b(idN)
next
print "-----------"

numPasses=200    'indeed it does converge to 1
numPasses=2
for pass= 1 to numPasses
print "========== Pass ";pass;" =========="
[forvPass]
print "forward --------------------"
[hidden]    '4,5
[output]    '6
for j = 4 to 6
   a(j)=b(j)
   for i = 1 to 5    'all possible inputs, w is 0 then not needed
       a(j)=a(j)+w(i,j)*O(i)    'x1..x3 copied to O1..O3
   next
   O(j)=sigm(a(j))
   print "a(";j;")=";a(j), "O(";j;")=";O(j)
next

theErr=Target-O(6)
print "Output is ";O(6),"Error=";theErr
if numPasses=pass then exit for 'do not back pass on last one

[backPass]
print "backward --------------------"
[outputUnit]
i=6
delta(6)=O(i)*(1-O(i))*(Target-O(i))
print "delta(";i;")=";delta(i)
j=6
for i = 4 to 5
   delta(i)=O(i)*(1-O(i))*w(i,j)*delta(j)
   print "delta(";i;")=";delta(i)
next
nu=0.9
for j=1 to 5
   for i = 1 to 6
       if u(j,i) then
           dw(j,i)=nu*delta(i)*O(j)
           print "dw(";j;",";i;")=";dw(j,i),
           w(j,i)=w(j,i)+dw(j,i)
           print "w(";j;",";i;")=";w(j,i)
       end if
   next
next
'bias
for i = 4 to 6
   b(i)=b(i)+nu*delta(i)
   print "b(";i;")=";b(i)
next
next    'pass

'-------------------------------------
input "press Enter to exit";dummy$

[teachingData]
'x1 x2 x3 output
data 1, 0, 1, 1

[neuronData]
'idNeuron,
'    numInputs*(idInput, Winput),
'        bias
data 4,3
data    1, 0.2
data     2, 0.4
data    3, -0.5
data         -0.4

data 5,3
data    1, -0.3
data     2, 0.1
data    3, 0.2
data         0.2

data 6,2
data    4, -0.3
data     5, -0.2
data         0.1

function sigm(a)
   sigm=1/(1+exp(0-a))
end function


[tsh73]

Ok I tried to make it learn more test data
[teachingData]
'x1 x2 x3 output
data 1, 0, 1, 1
data 1, 1, 1, 0.5
data 1, 0, 0, 0
data 1, 1, 1, 0.5
data 0, 0, 1, 1
First was in example, others I just made up

Now on each iteration (say 200) (actually until bored)
. I read one line of teaching data, 5 of them
.. do forward pass - get network output and compare it to this line goal
.. do backward pass and correct weights so error decreased

Here is error on the first pass
========== Pass 1 ==========
data 1 0 1 Goal 1           Output is 0.4738889         Error=0.5261111
data 1 1 1 Goal 0.5         Output is 0.50831596        Error=-0.83159555e-2
data 1 0 0 Goal 0           Output is 0.50844091        Error=-0.50844091
data 1 1 1 Goal 0.5         Output is 0.46597935        Error=0.34020653e-1
data 0 0 1 Goal 1           Output is 0.4769881         Error=0.5230119

And here is on the last pass. Error definitely got smaller, and will get smaller still given enough time.
========== Pass 200 ==========
data 1 0 1 Goal 1           Output is 0.89766235        Error=0.10233765
data 1 1 1 Goal 0.5         Output is 0.51026331        Error=-0.10263312e-1
data 1 0 0 Goal 0           Output is 0.93918224e-1     Error=-0.93918224e-1
data 1 1 1 Goal 0.5         Output is 0.49888           Error=0.11199989e-2
data 0 0 1 Goal 1           Output is 0.96040252        Error=0.39597479e-1

On last pass I do print weights and biases - only w() and b() matters
dw(1,4)=0     w(1,4)=1.648503
dw(1,5)=0     w(1,5)=-1.0734708
dw(2,4)=0     w(2,4)=1.83196876
dw(2,5)=0     w(2,5)=-0.63260881
dw(3,4)=-0.2315867e-3       w(3,4)=-3.33942343
dw(3,5)=0.33434864e-3       w(3,5)=2.13509225
dw(4,6)=0.52333842e-4       w(4,6)=-4.60286829
dw(5,6)=0.12178961e-2       w(5,6)=2.70932598
b(4)=0.12421581
b(5)=0.47635041e-1
b(6)=0.93410395

These digits is actually "learned" network.

Whole program
'back propagation example, by YouTube video
'link https://www.youtube.com/watch?v=n2L1J5JYgUk
'v1: forward propagation
'v2: + backward
'v3: + second pass
'v4: +  try to teach to different tasks

'printBack=1
'printForth=1

nData=5
nX=3    'number of inputs
nHid=2    '2 hidded neurons, 1 layer, numbered 4,5
nOut=1    '1 output neuron, numbered 6
dim x(nX)    'inputs
dim w(nX+nHid,nX+nHid+nOut)    'weights
dim u(nX+nHid,nX+nHid+nOut)    '1 if used, to skip 0 then printing
dim dw(nX+nHid,nX+nHid+nOut)    'delta of weights
dim b(nX+nHid+nOut)    'bias
dim a(nX+nHid+nOut)    'temp (pre-sigmoid?) output
dim O(nX+nHid+nOut)    'neuron output
dim delta(nX+nHid+nOut)

[readData]
' print "Initializing test data"
' restore [teachingData]

' print "data ";
' for i = 1 to nX
'    read x:x(i)=x
'    O(i)=x
'    print x(i);" ";
' next
' read Target
' print "Goal ";Target,
' print
' print "-----------"

[readNeurons]
print "Initializing neurons"
restore [neuronData]
for i = 1 to nHid+nOut
   read idN, numInp
   print idN, numInp
   for j = 1 to numInp
       read idInp, w
       w(idInp,idN)=w
       u(idInp,idN)=1
       print , idInp, w(idInp,idN)
   next
   read b: b(idN)=b
   print ,,b(idN)
next
print "-----------"

numPasses=200    'indeed it does converge to 1
'numPasses=2
for pass= 1 to numPasses
print "========== Pass ";pass;" =========="
restore [teachingData]
for n= 1 to nData
print "data ";
for i = 1 to nX
   read x:x(i)=x
   O(i)=x
   print x(i);" ";
next
read Target
print "Goal ";Target,

[forvPass]
'print "forward --------------------"
[hidden]    '4,5
[output]    '6
for j = 4 to 6
   a(j)=b(j)
   for i = 1 to 5    'all possible inputs, w is 0 then not needed
       a(j)=a(j)+w(i,j)*O(i)    'x1..x3 copied to O1..O3
   next
   O(j)=sigm(a(j))
if printForth then print "a(";j;")=";a(j), "O(";j;")=";O(j)
next

theErr=Target-O(6)
print "Output is ";O(6),"Error=";theErr
'if numPasses=pass then exit for 'do not back pass on last one
'does not work for several test cases
if numPasses=pass and n= nData then printBack  =1 'show weights after teaching

[backPass]
'print "backward --------------------"
[outputUnit]
i=6
delta(6)=O(i)*(1-O(i))*(Target-O(i))
if printBack then print "delta(";i;")=";delta(i)
j=6
for i = 4 to 5
   delta(i)=O(i)*(1-O(i))*w(i,j)*delta(j)
   if printBack then print "delta(";i;")=";delta(i)
next
nu=0.9
for j=1 to 5
   for i = 1 to 6
       if u(j,i) then
           dw(j,i)=nu*delta(i)*O(j)
           w(j,i)=w(j,i)+dw(j,i)
   if printBack then
           print "dw(";j;",";i;")=";dw(j,i),
            print "w(";j;",";i;")=";w(j,i)
end if
       end if
   next
next
'bias
for i = 4 to 6
   b(i)=b(i)+nu*delta(i)
   if printBack then   print "b(";i;")=";b(i)
next
next 'test data
next    'pass

'-------------------------------------
input "press Enter to exit";dummy$

[teachingData]
'x1 x2 x3 output
data 1, 0, 1, 1
data 1, 1, 1, 0.5
data 1, 0, 0, 0
data 1, 1, 1, 0.5
data 0, 0, 1, 1

[neuronData]
'idNeuron,
'    numInputs*(idInput, Winput),
'        bias
data 4,3
data    1, 0.2
data     2, 0.4
data    3, -0.5
data         -0.4

data 5,3
data    1, -0.3
data     2, 0.1
data    3, 0.2
data         0.2

data 6,2
data    4, -0.3
data     5, -0.2
data         0.1

function sigm(a)
   sigm=1/(1+exp(0-a))
end function