A version of a neural network which is inspired by a popular model of the brain but doesn't perform significantly well on normal machine learning tasks.
A neural network model which doesn't learn via backpropagation like typical neural networks do. The model is (typically) hierarchical and each layer consists of state neurons and then predictions of these state neurons which are produced by weights applied to the higher level state neurons. The bottom layer states are the real world sensory inputs and the idea is that the higher levels represent more abstract and compact notions of the current situation. E.g. the top layer is that you are watching a car drive past and then this drives lower level predictions that your eyes are likely to see the car object move further in the same direction by a consistent amount. The model's objective is to minimise prediction error across all layers. The benefit of these models is that they are relatively biologically plausible models of neural networks in the brain. The update rules are local - they don't depend on values many layers away (which would require some additional memory to compute). It can learn from unlabelled input data (like the brain does). Many psychological effects are explained nicely with a predictive processing model and modern neuroscience seems to support a bayesian brain which aims to reduce prediction error based on prior knowledge. This research was inspired from reading Surfing Uncertainty by Andy Clark which is a neuroscience/psychology book explaining Andy Clark's view of the predictive processing framework and how it explains psychological phenomena. Additionally, Clark's model has 'uncertainty' for each neuron's errors e.g. vision inputs may be less reliable in the dark than auditory signals. Clark postulates that this precision weighting can be used for high level effects like gating of neural circuits and can explain conditions such as schizophrenia as incorrect precision weighting of high level beliefs versus sensory input. As good as some of the literature makes them sounds, predictive coding networks are much slower to train and run inference on than traditional feedforward neural networks. They are more unstable in their training and scaling the architecture doesn't work as well as it does for traditional deep learning models. For these reasons (and others), they aren't used for any practical purposes but there is ongoing research about them. My research aims to first get a decent implementation of traditional predictive coding networks and then add the features that I like the sound of such as precision weighting and then see if I can produce any results which are noticeably better than traditional neural networks.
I've tried the typical formulation of predictive coding networks on classification where they didn't perform particularly well. There are variations in what's considered the normal predictive coding, most notably in the way the weights are updated. The literature agrees that the states incrementally update but with the weights, I've seen four different ways of updating the weights. I've compared these and incremental predictive coding seems to perform the best. I wasn't particularly keen on the formulation of classification models - forcing the high level state to be mapped to a one hot encoding seems a bit restrictive and differs from the predictive processing model. So, I tried the network on auto-associative retrieval of images. I've then been experimenting with adding layers which predict states in their own layer (lateral weights) and then also precision weighting. The lateral weights are beneficial. However, precision weighting is not currently working as desired. The model aims to reduce precision weighted error and so its solution to this is to lower the precisions. Adding a regulatory term helps keep a stable mean value of precision weights but reduces the standard deviation of precision weights which suggests that they aren't performing any useful functions.
The short term goal is to fix my formulation of precision weighting. The objective function doesn't cause the model to utilise the precision weights in an 'intelligent' way. I'll try to recreate more concrete use cases that Andy Clark describes. Currently, I reset the precision weights for each image. From what I understand, precision weightings values should change at a slower timescale than neurons but faster than weights. An intermediate timescale and being reset on each image may be incompatible. Potentially, testing on something like video or physics predictions may yield better results - something more biologically relevant than remembering images. If I find something noteworthy, I'll write a report/paper style document but it's quite enjoyable just experimenting.