, ,
|| _ ||
_-_, =||= ,._-_ < \, =||=
||_. || || /-|| ||
~ || || || (( || ||
,-_- \\, \\, \/\\ \\,
A neural network with no math? - (and machine code optimizations)
Uses C
Neural networks are expensive to compute, like REALLY expensive. To get the result of a typical network it takes thousands of multiplications across a ton of data. So my idea was to make one that used only simple one clock instructions like bitwise operators. And if you exclude addition it becomes trivial to translate to simple logic gates.
My first idea to do this was to have it operate similarly to a normal network, simply operating on 1 bit instead of full numbers. Unfortunatly, I couldn't come up with a decent way of combining multiple operations on one node (like when every node of the previous layer is multiplied by a constant then added). The problem is that with numbers you can add it together and get a continuous output but with bits if you added it they would just flip flop and it might as well be random, and if you tried counting how many are 1s and how many are 0s instead it would be inefficient and slow, defeating the purpose.
The implementation that I finally came to was, in essence, an ML turing machine. If you've never heard of that being done before, that's because it's a terrible idea, but I'll get to that later. The way it works is that arc4buffer generates me a nice completely random list of instructions in a binary format specifically designed such that random slop is perfectly valid. It then compiles that into machine code and executes it with a virtual memory of 64kB.
Machine code?!
In a traditional network, every operation is the same, just multiply over and over again with different numbers. But for a turing machine the part that gets trained IS the operation. A naïve approach would be a string of if statements or a switch, and, while that gets the job done, doing it that way would be horridly slow.
To solve this problem, I made a very simple bytecode interpreter to turn the model's representation into executable code. Because of the amount of processing power used for compiling, it is only useful to do this when running the same model several times in a row. This is perfectly fine though since the only way I found to train it was evolution--considering I don't know any way to use the derivative of a bitwise operation. maybe I just need to learn discrete calculus...
The basic gist of how the assembler works is as follows:
1: Each opcode is broken into 2 nibbles which dictate what it does with the input data.
/* * op large bits: * xxx0 a value * xxx1 a pointer * xx0x b value * xx1x b pointer * * x0xx pass a * x1xx inv a * 0xxx pass b * 1xxx inv b */
The first 2 bits of the signifigant nibble dictate how the a and b values are used, the first one for A and the second for B. If the bit is 0 then it uses the value provided; if 1, it treats it as a pointer. The second 2 determine if the value is inverted or not.
/* * op small bits: * 0000 buf * 0001 and * 0010 or * 0011 xor * * 0100 shl * 0101 shr * 0110 nop * 0111 nop * * 1xxx not */
The small bits are used to determine the operation applied, according to the above comment.
2: It sets the edx register as A and the ecx register as B to either values or the values at the pointers. It also inverts if necessary.
3: The operations are then applied and the output is stored to ram.
The method used to write the machine code is rather crude but it works, here is an example:
void notd(struct machinecode *mc) { mc->d[mc->i++] = 0xf7; mc->d[mc->i++] = 0xd2; }
This function inverts the edx register.
Why does it suck?
Turns out, when you give a model direct access to 64kB of ram, it doesn't know what to do with it. In a traditional feedforward network, every node is connected to every other. But for a turing machine with 64kB of ram every operation has a 1 in 64k chance of being from spot A, another 1 in 64k chance of taking from spot B, and yet another 1 in 64k chance of outputing to spot C. This results in an approximately 1 in 2.8147498e+14 chance of it actually even doing anything with relevant and putting it in a reasonable place. As a result, while it does "train", it does so incredably slowly with almost all random mutations not even touching the input nor the output.
In addition to that, the binary produced by the compiler can be around 320kB, and with the tape at 64kB, each run is massively bottlenecked by ram. And if that wasn't bad enough, the constant random access by the randomly generated instructions of the model mean the entire run is a massive cache miss, annihilating any chance it could've had at having any semblance of speed. Therefore, dispite my best optimization efforts, defeating the entire point to begin with. :<