Repeated Sequences in Tree Genetic Programming

W. B. Langdon
Computer Science

Paper presented at EuroGP-2005 Pages 190-202

Introduction

Langdon + Banzhaf in Memorial University Repeated Sequences in Biology
Linear and Tree Genetic Programming
Test problems
Repeated sequences, fragments and subtrees
So what?
- Where does this lead next?
- Other emergent phenomena?
Conclusions
More information and Data set

Repeats in DNA

50% of DNA is part of repetitive sequence. Many different types of repeated DNA sequence. Classified by repeat sequence length, number of repeats, location in DNA molecule etc. etc.
- Some may have biological meaning, e.g. as a clock counting cell divisions and enforcing limit, cell life limited, so cancer prevented.
- Repeated sequences in both expressed (protein coding) and non-expressed DNA.
DNA whose sequence is not maintained by selection will develop periodicities as a result of random crossover [G.P. Smith, 1976].

Demonstration problems

Want to run GP for many generations. Hard problems, not immediately solved.
Want range of different problems
- Time series modelling. One variable, short integers (byte) arithmetic
- Bioinformatics. Binary classification, floating point, 20 inputs.

Mackey-Glass Chaotic Time Series

Hard (impossible) since chaotic time series.
IEEE benchmark, 1201 data points.
Fast signal processing (integer arithmetic)
7 time lags: 1, 2, 4, …, 128 steps ago.

Mackey-Glass

Predicting Protein Location

Given only number of each amino acid (i.e. cheap info, Swissprot) in a protein, predict what it is. Very hard.
Easier: predict where the protein will be found
- Simplified [A. Reinhardt and T. Hubbard,1998] which covers animals and microbes, to just animals and two classes: In the cell nucleus or not.

Animal Nuclear Proteins

Non-linear 2D projection from 20 Dimensional Space Nuclear and Non-Nuclear Animal Proteins

Animal Nuclear Proteins

Non-linear 2D projection from 20 Dimensional Space

Genetic Programming Approaches

Linear GPengine (Nordin)
Linear Machine Code Discipulus
Tree GP
TinyGP

Performance

(all approaches solve problems)

Predicting M-G chaotic Time Series
	RMS error	Mean
Linear GP Nordin 2XO	1.60-5.37	3.79
Tree GP 2X0 500gens	1.08-3.78	2.41
Nuclear Protein prediction (holdout set)
Discipulus	78-82%	80%
Tree GP 2X0 50gens	80-83%	82%

Evolution of Nuclear Protein Prediction Accuracy

Protein Prediction. Tree GP, Two point XO

Evolution of Protein program Size

Size of Repeated Tree Fragments

Evolution of Exact Repeats in First Protein Prediction 2XO run

Repeated Tree Fragments

Red 128 Blue 85 Green 79 Grey 11-63

Repeated Subtrees

Red 39 Blue 19 Green 15 Black 11 Grey 7

Largest Repeats M-G and Protein

Largest repeated fragment in Mackey-Glass and Protein Location best of run programs

Fraction of Program made of Repeated Subtrees

First Protein prediction 2XO Best program's repeated subtrees

Semantically repeated subtrees

First Protein Nuclear 2XO run best program Correlated subtrees

"Fitness" of Subtrees

Function and terminal set have do not have side effects. Therefore each subtree can be evaluated as if it where the whole program.
Use full fitness measure.
Calculate correlation with training data.
Plot anti-correlation and anti-fitness as if correlated or as if fit.

Entropy first Protein prediction run

Entropy in best program at end of first Protein prediction run

In most cases variation across the training data increases monotonically from the leafs (bottom) to the root (top).

Subtree Fitness in first Protein prediction run

Subtree Fitness in best program at end of first Protein prediction run

Important nodes first Protein prediction run

Important nodes in best program at end of first Protein prediction run

Black changes >10 training cases

Repeated Subtrees

Red 39 Blue 19 Green 15 Black 11 Grey 7

Discussion

In trees, can get diffuse introns whereby whole program depends only on fraction of tree. Not classic introns, since most functions do depend on both arguments.
Crossover evolves trees similar fractal shape properties as random trees BUT
Repeats not random.
Many subtrees have high fitness and pass information towards root, BUT
Much of program can be discarded with little impact on fitness
Genetic programming on simple problems assembles complete solutions by gradually, randomly, reusing existing partial solutions to get small improvements, rendering existing parts less important.

Conclusions

GP is putting together fit subtrees
On different problems and different GPs (2 linear and 2 tree) where size is not constrained, repeated sequences/subtrees/fragments emerge from crossover
Repeats cover large fraction of fit programs.
This is an example of emergence.

More information

More information on GP

References:

Langdon, W. B. Genetic Programming and Data Structures: Genetic Programming + Data Structures = Automatic Programming!, vol. 1 of Genetic Programming. \newblock Kluwer, Boston, 1998.
Langdon, W. B. Size fair and homologous tree genetic programming crossovers. Genetic Programming and Evolvable Machines 1, 1/2 (Apr. 2000), 95-119.
Langdon, W. B. and S. J. Barrett. Genetic programming in data mining for drug discovery. In Ashish Ghosh and Lakhmi C. Jain, editors, Evolutionary Computing in Data Mining, volume 163 of Studies in Fuzziness and Soft Computing, chapter 10, pages 211-235. Springer, 2004.
Repeated Sequences in Linear Genetic Programming Genomes, W. B. Langdon and W. Banzhaf, Complex Systems in press. (GECCO'2004 late breaking paper PDF gzipped postscript. Movie. Poster).
Repeated Patterns in Tree Genetic Programming, W. B. Langdon and W. Banzhaf, EuroGP-2005 LNCS 3447, p190-202 doi:10.1007/b107383
A. Reinhardt and T. Hubbard. Using neural networks for prediction of the subcellular location of proteins. Nucleic Acids Research, 26(9):2230-2236, 1 May 1998.
G. P. Smith Evolution of repeated DNA sequences by unequal crossover. Science 191, 4227:528-535, 13 Feb 1976.

GP Parameters for Mackey-Glass

time series prediction

Function set: MUL ADD DIV SUB operating on unsigned bytes
Terminal set: Registers are initialised with historical values of time series. D128 128 time steps ago, D64 64, D32 32, D16 16, D8 8, D4 4, D2 2 and finally D1 with the previous value. Time points before the start of the series are set to zero. Constants 0..127.
Fitness: RMS error
Selection: non elitist, tournament size 7. Pop Size 500
Initial pop: ramped half-and-half (2:6) (50% of terminals are constants)
Parameters: 50% mutation (point 22.5%, constants 22.5%, shrink 2.5% subtree 2.5%). Max tree size 1000. Either 50% subtree crossover or 50% size fair crossover (90% on internal nodes), FXO fragments 30 [Langdon,2000]
Termination: 50 generations

Nuclear v Non-nuclear Protein Prediction

Function set: MUL ADD DIV SUB operating on floats
Terminal set: Number (integer) of each of the 20 amino acids in the protein. (Counts for code B were split evenly between aspartic acid D and asparagine N. Those for Z, between glutamic acid E and glutamine Q.) 100 unique constants randomly chosen from tangent distribution (50% between -10.0 and 10.0) . (By chance none are integers.)
Fitness: True 0.5 Positive rate + 0.5 True Negative rate [langdon,2004]
Selection: non elitist, tournament size 7. Pop Size 5000.
Initial pop: ramped half-and-half (2:6) (50% of terminals are constants)
Parameters: 50% mutation (point 22.5%, constants 22.5%, shrink 2.5% subtree 2.5%). Max tree size 1000. Either 50% subtree crossover or 50% size fair crossover (90% on internal nodes), FXO fragments 30 [Langdon,2000]
Termination: 50 generations
Data set