Creating neural networks for embedded systems
Screen Shot Download Page Report
MLP Edit is a development tool for creating Multi Layered Perceptron Neural Networks for embedded systems. It allows you to set up, train and test an MLP network and then export it as an ADA95-package.
This file is divided into four sections:
The main window
When the application is executed, the main window opens (maximised to full window size), displaying the main menu & toolbar, the vertical and horizontal scrollbars, and an outline of the current network ( three fully connected layers with two neurons each ).
The scroll bars can be used to move the outline of the network, such that the most extreme scrollbar position will place the corresponding edge of the network near the centre of the window. This is useful for large networks.
Note: The outline of the current network will not be displayed if there are more than 512 neurons in one of the layers or if the total number of connections between the neurons exceeds 32768.
The main menu
The main-menu consists of File, Network, Data, Application, and Help, which contain the following menu items:
| File: | Load Network, Save Network, Export Network as …ADA Source. |
| Network: | Network Configuration, Randomise Weights, Train Network, Run Network. |
| Data: | Text Editor. |
| Application: | Set Defaults. |
| Help: | MLP Edit, About. |
Please note:
Only one dialogue-box / window can be open at a time.
The main toolbar
The tool-bar consists of three groups of speed-buttons. Moving the mouse-pointer over any one of these will display a tool-tip naming the corresponding menu item; going from left to right these are: Load Network, Save Network, Export Network as ADA Source, Network Configuration, Randomise Weights, Train Network, Run Network and Text Editor.
Default settings
Certain defaults can be set which are then held in the application's ini-file; see Set Defaults for details.
1. Load Network
Loads an existing Neural Network which was previously created using MLP Edit. A dialogue box appears, listing the networks saved in the default MLP-network folder. Select the network which is to be loaded.
2. Save Network
Saves the current Neural Network. A dialogue box appears listing the networks saved in the default MLP-network folder. Select or type in a filename (including the ".mlp" extension). If the file already exists, you will be alerted.
3. Export Network as ADA Source
Exports the current Neural Network as ADA95 source code. A dialogue box opens with one text-entry-box and three radio-buttons, plus an Ok push-button and a Cancel push-button. Type the desired name of the ADA95 package into the text-entry-box and then select the type of exception handling the package is to perform:
Standard allows all exceptions to propagate to the calling procedure.
Simplified intercepts all exceptions and raises an error exception (defined in the package).
Detailed also intercepts all exceptions and raises an error exception; however, in addition the name of the intercepted exception is included and can be retrieved using the standard Ada.Exceptions.Exception_Information( ) function.
When you press the Ok push-button, a further dialogue box opens ( listing the ADA95 packages in the default ADA95 folder ) and you select the folder into which the package source code is to be written. The application then generates the filenames of the source code by adding ".ads" and ".adb" to the package name. If the files already exists, you will be alerted to this and asked if they are to be over-written, otherwise the full path and filenames are presented to you for validation, and in both cases clicking Yes will then cause the package source code to be generated.
Warning: The package name is not checked for ADA-compliance.
Usage of the generated packages:
Each such package contains four public functions:
procedure Wr_Input(Neuron:in Input_Neuron_Number;Value:in Neuron_Value);
function Rd_Input(Neuron:in Input_Neuron_Number) return Neuron_Value;
procedure Classify;
function Rd_Output(Neuron:in Output_Neuron_Number) return Neuron_value;
Wr_Input writes the input-value Value into the input-neuron Neuron, where Neuron is an integer between one and number-of-input-neurons.
Rd_Input returns the current value of the input-neuron Neuron, where Neuron is an integer between one and number-of-input-neurons.
Classify calculates the values of the output-neurons from the values of the input-neurons.
Rd_Output returns the current value of the output-neuron Neuron, where Neuron is an integer between one and number-of-output-neurons.
4. Network Configuration
Changes the number of layers and the number of neurons in each layer. A window opens with a slider, a text-entry-box, and two push-buttons ( Insert Layer and Remove Layer ). The total number of connections between neurons is also displayed, as this is the biggest factor affecting the speed and physical size of the network.
The slider is used to select a layer, and the number of neurons in that layer is displayed in the text-entry-box. You may then change the number of neurons in that layer by typing in the required number of neurons and pressing the enter-key. A new layer may be inserted behind the selected layer, or the selected layer may be removed from the network, by pressing the respective push-buttons.
Bounds:
The network must contain a minimum of 3 layers and a maximum of 32 layers.
Each layer must contain at least 2 neurons.
The input layer may contain a maximum of 16384 neurons (128x128).
The hidden layers may contain a maximum of 128 neurons each.
The output layer may contain a maximum of 512 neurons.
5. Randomise Weights
Gives each weight in each neuron a new random value in the range -1.0 through +1.0, such that the network is ready for training.
6. Train Network
Trains the current network using training sets loaded from text files. A window opens which has three subdivisions (Training instructions, Error Graph and Training Status) and a Start/Stop push-button. Initially the Start/Stop push-button is not enabled, as no training file has been loaded.
Training instructions: There are two types of training targets. "Error < N" or "N iterations". With "Error < N", the network is trained until the error of the network is less than the given target, while "N iterations" trains the network for a specified number of iterations. Training automatically ceases when the target has been reached. The learning rate may also be specified. The training file is loaded by typing in the file name and pressing return or by pressing the Browse push-button ( upon which a dialogue-box appears listing the training files stored in the default data folder ) and selecting a file. Once this has taken place, the Start/Stop push-button becomes enabled. Creating training files is described in "8. Text Editor".
Bounds:
The training targets must be greater than zero.
The learning rate must be greater than zero and less than or equal to one.
Error Graph: The history of the changes in the error-value of the network will be displayed on a graph if the Display check-box is ticked. The horizontal resolution of the graph may then also be specified using the slider. The number of displayed iterations is shown beneath the slider.
Training Status: Once training has commenced, the current error value and the number of training iterations (i.e. the number of times each training pattern has been presented to the network) are displayed. In addition, you may tick the Delta check-box, in which case the difference between the current error value and the previous error value is displayed in ten to the power of … format. This can useful in determining whether the network is converging or not.
7. Run Network
Tests a trained network. There are two text-grids: Inputs and Outputs. Each column in the Inputs text-grid represents an input neuron, while each row represents an input pattern. In the same way, each column in the Outputs text-grid represents an output neuron, while each row represents the classification of the corresponding input pattern; i.e. row n in the Inputs grid contains the values of the input neurons, while row n in the Outputs grid contains the corresponding values of the output neurons.
Above the text-grids there are four speed-buttons:
Load Input File, loads input data from a text file. A dialogue box appears, listing the input data files saved in the default data folder. Select the input data file which is to be loaded. Creating input data files is described in "8. Text Editor".
Add Noise to Input Data adds a random number to each input value, which is proportional to the average value of the current input values.
Classify Input Data calculates and displays the output for each input pattern.
Save Input File writes the current input data (which may have had noise added to it) into a text file. A dialogue box appears, listing the input data files saved in the default data folder. Select or type in a filename (including the ".inp" extension). If the file already exists, you will be alerted. The data is written to the file in such a way that the text-file contents can then be cut-and-pasted into a spread-sheet.
8. Text Editor
Create, Save and Load training files and input data files. This is a simple built-in text editor which allows you to create or modify training and input data files without leaving MLP Edit.
Use CTRL-X, CTRL-C and CTRL-V to cut, copy and paste, respectively.
Three speed-buttons are provided:
New deletes the current text. If the current text has been modified, you are alerted and asked if the text is to be saved, and clicking Yes will cause the save dialogue-box to appear.
Open loads text from a file. As above, if the current text has been modified, you are alerted and asked if the text is to be saved. In any case, an open dialogue-box then appears, listing the files saved in the default data folder. Select the file which is to be loaded.
Save writes text to a file. A dialogue-box appears, listing the files saved in the default data folder. Select or type in a filename (including the appropriate extension). If the file already exists, you will be alerted.
File formats:
The file format descriptions below are very simple, and examples are included with this application. A training file has a ".trn" file-extension, while an input data file has a ".inp" file-extension. Both are simple text files containing a sequence of items written in plain text and separated by space characters.
Note: Each input value may be any real number, but each output value needs to be either zero or one.
Training file format: The first item of text is a number specifying how many training patterns there are in this file. This is followed by an open square bracket, then the number of inputs per pattern, then the number of outputs per pattern, and then a close square bracket. Next comes the first training pattern: Each input value is stated in sequence, after which all the corresponding desired output values are stated in sequence. This forms one training pattern. Each training file may contain any number of training patterns.
Input data file format: The first item of text is a number specifying how many input patterns there are in this file. This is followed by an open square bracket, then the number of inputs per pattern, and then a close square bracket. Next comes the first input pattern: Each input value is stated in sequence. This forms one input pattern. Each input data file may contain any number of input data patterns.
9. Set Defaults
For convenience, certain defaults may be specified, which are held in the application's ini-file and therefore only need to be set once. A dialogue box opens with three pages:
Folders: There are three default directories: Network is where the MLP-Networks are stored, Ada95 is where the Ada95 packages are stored, and Data is where the training files and data input files are stored.
Note that directory names may not contain spaces.
Ada95: One of the three Ada95 exception handling modes can be set as the default.
Error Graph: The dimensions of the error graph can be changed. The Graph Width determines how many iterations can be displayed when the graph is set to its highest detail during training, while the Graph Height determines the highest error value that can be displayed (note: if the Graph Height is 100 then the highest error value that can be displayed is 1.0 ) .
At the bottom of the dialogue box there are three push-buttons: Save As Default sets the current values as the permanent default values, while Use only changes them until the application is closed. Cancel resets the default values to what they were before the dialogue box was opened.
This tutorial is intended to guide you through the process of creating a neural network for a specific purpose and exporting it as an ADA95 package.
In this tutorial, a network for classifying a dice-face will be produced. A dice-face consists of a 3x3 matrix of points which are either black or white. Therefore, the network will require nine input neurons, one for each point. Each roll of the dice has six possible outcomes, therefore the network will require six output neurons.
It is time to produce the training file. Click on the text-editor speed-button. As there are six basic dice-face patterns, our training file will require six training patterns. The first entry in the training file is therefore a '6'. This is followed by an open square bracket. The number of inputs and outputs has already been determined to be nine and six, respectively, so the next two entries are a '9' and a '6'. Finally, a close square bracket needs to be entered. Having typed in these entries (each separated by a space character), it is wise to press enter twice to keep the file readable. It is now time to enter the training values: Each input pattern is entered, followed by the desired corresponding output pattern. Here then is the full training file:
6 [ 9 6 ]
0 0 0
0 1 0
0 0 0
1 0 0 0 0 0
1 0 0
0 0 0
0 0 1
0 1 0 0 0 0
0 0 1
0 1 0
1 0 0
0 0 1 0 0 0
1 0 1
0 0 0
1 0 1
0 0 0 1 0 0
1 0 1
0 1 0
1 0 1
0 0 0 0 1 0
1 0 1
1 0 1
1 0 1
0 0 0 0 0 1
Copy this into the Text Editor and save it as "Dice.trn", then close the Text Editor window.
The network must now be configured. Click on the Network Configuration speed-button. Change the number of neurons in the input layer to 9, the number of neurons in the hidden layer to 4 and the number of neurons in the output layer to 6. Be sure to press the enter key each time. Notice that the number-of-connections is now sixty.
Close the Network Configuration window and click on the Train Network speed-button. Click on the Browse push-button, and select "Dice.trn" from the file-dialogue box. Press the Start/Stop push-button, and watch the error value go down to 0.01 . When this figure has been reached, the network automatically stops training.
The network must now be tested. Close the window and click on the Text Editor speed-button again. Copy the following input data and save it as "Dice.inp".
6 [ 9 ]
0 0 0
0 1 0
0 0 0
1 0 0
0 0 0
0 0 1
0 0 1
0 1 0
1 0 0
1 0 1
0 0 0
1 0 1
1 0 1
0 1 0
1 0 1
1 0 1
1 0 1
1 0 1
Then close the Text Editor window and click on the Run Network speed-button.
Load "Dice.inp" from disk by clicking on the Load Input File speed-button and then click on the Classify Input Data speed-button. Notice how for each input pattern there is one output neuron which is near one while all the other output neurons in that pattern are near zero.
It is time to save the network for future use. Close the Run Network window and click the Save Network speed-button. Save the network as "Dice.mlp".
Next click on the Export network as ADA Source speed-button. Under Package Name enter "Mlp_Dice", and click on the Ok push-button. In the file dialogue box simply click "Save" and in the following message-box click Yes.
Congratulations, one fully functional ADA95-package consisting of a trained neural network has just been generated !
MLP Edit uses the following network-architecture:
A fully connected network with multiple layers of perceptron-neurons. The output from each neuron in one layer is connected to the inputs of each neuron in the next layer. Each layer also includes one bias-neuron, who's output value is always one; however, this bias neuron is hidden from the user in MLP Edit. The output of each neuron is calculated by multiplying each input to the neuron with the corresponding weight of the neuron and passing the sum of these weighted inputs through the sigmoid function. Therefore the inputs to the network may be any real number, while the outputs from the network will always be between zero and one. Training of the network consists of calculating the difference between the desired output and the actual output and reducing this difference by adjusting the weights of each neuron through back-propagation using a gradient descent method.
For more information on this type of neural-network, see Chapter 3 of "Introduction to Neural Networks", by Phil Picton (1994, MacMillan Press Ltd, ISBN 0-333-61832-7).