Relearning your data and other possible means of increasing data for learning

In deep learning, it is often noted that large amount of data is needed to train the neural network to learn and recognize the pattern.

But is this necessary?   Is it possible to accelerate the process instead?

Often, after you have gained new insight, revisiting old books and then you will suddenly gained new perspective and insights.   So how about revisiting old data, after acquiring some amount of experience and knowledge?     How to force the system to acquire and absorb the new knowledge gained from one set of data into recognizing the same set of patterns in another set of data?

There are several heuristics possible to implement this "insight recognition enforced":

1.  Subsetting the data into different sets, and preferably, all the elements within each set should be easily related to one another, and not something totally different and cannot be associated with one another.    And after learning all the pattern for each set, next is to feed it with pattern from another set - by doing this you are forcing the learning of a new set of pattern using the old knowledge as the base.   In this way, the same set of data can be used many different times, each time it is used immediately after another different set of pattern.   So if there are 10 sets of data, swapping in different orders, there should be 9x10 different set of two, 8x9x10 set of threes etc.   This heuristic will hugely blow up the amount of data available to pattern learning.   Another perspective:   any piece of data, can have some unknown effects on the brains.   But we can be quite sure if the braine has been highly trained with more different set of data, it will be able to recognize and learn new set of data faster/easier.

2.   Instead of selecting subsets, or ordering in different orders of subset, another way is to throw out useless data, or to reformat the current data in a more structure way, using the newly gained neural network knowledge, so that the training will be even faster.   Keywords:  cleanup, removal of data, restructuring of existing data.

3.   Sometimes ordering of the above sets matters - just like ordering of lessons matters sometimes in learning logically or temporally.   Keywords: reordering of data according to degree of complexity, or dependencies of information from one to another, or hierarchically - different cores and subfunctions etc.

4.   Random/systematic introduction of data - it can be structured, or formed from existing data etc.   Keywords:   injection of new data.

Another heuristic we often used in knowledge acquisition is top-down breakdown of knowledge component hierarchically.

For example, at the course granularity, we know that CPU execute instruction and output its values.   Next we add the concept of memory, and we know it retrieve its instructions from memory, execute them and load the output back to the memory.   Going to the lower hierarchy, we can differentiate many different types of CPU instructions, and different types of memory (ring 0 or ring 3), and so on.

Using the same mechanism described above, going lower and lower, we can generate more and more data - all data will have different perspective of the same set of courser refinement data.

For example, using different LLVM/gcc optimization it is possible to generate many different alternative set of representations for the original program, and thus multiplying the original data by a lot more, and expanding the depth of representations.