Terminal_Set() constructor issue

[omidr1370]

Hello. Thanks for this Tensorial approach. I have one question.
It seems "Terminal_Set()" constructor is not working. I want to define four 2x2 tensors in my terminal. I'll be grateful if you let me know the correct syntax to define the terminal.

def` calc_fit(**kwargs):
# read parameters
population = kwargs.get('population')
generation = kwargs.get('generation')
tensors = kwargs.get('tensors')
f_path = kwargs.get('f_path')
_stf = kwargs.get('stf')
target = kwargs.get('target')

fn = f_path + "gen_" + str(generation).zfill(5)
fitness = []
times = []
best_ind = 0

# set objective function according to min/max
fit = 0
condition = lambda: (fit < max_fit)  # minimizing
max_fit = float('inf')

for i in range(len(tensors)):

    start_ind = time.time()
    fit = tf_rmse(tensors[i], target).numpy()

    if condition():
        max_fit = fit
        best_ind = i

    times.append((time.time() - start_ind) * 1000.0)
    fitness.append(fit)
    population[i]['fitness'] = fit

#if generation == gens:
#    save_image(tensors[best_ind], best_ind, fn, 2)

return` population, population[best_ind]`

Different types of function sets

normal_set = {'add', 'sub'}

if __name__ == "__main__":

# GP params
dev = '/gpu:0'  # device to run, write '/cpu_0' to tun on cpu
gens = 250 
pop_size = 2500  
tour_size = 3
mut_rate = 0.1
cross_rate = 0.9
max_tree_dep = 10
max_init_depth = 10
elite_size = 1 # 0 to turn off
runs = 1 # Number of average runs

# problems
optimal_solution = tf.convert_to_tensor(F)   #F is a (19828, 2, 2) tensor

terminal

indexing_variables = 3
domain_range = [19828, 2, 2]
my_terminal_set = Terminal_set(indexing_variables, domain_range)
my_terminal_set.add_to_set("T_1", tf.convert_to_tensor(T_1, shape=(19828, 2, 2)))
my_terminal_set.add_to_set("T_2", tf.convert_to_tensor(T_2, shape=(19828, 2, 2)))
my_terminal_set.add_to_set("T_3", tf.convert_to_tensor(T_3, shape=(19828, 2, 2)))
my_terminal_set.add_to_set("T_4", tf.convert_to_tensor(T_4, shape=(19828, 2, 2)))

problems = [optimal_solution] # Add to run more problems

# Domains dimensions
test_cases = [[19828, 2, 2]]

for p in problems:
    

    for res in test_cases:
        
        for r in range(runs):
            
            #seeds = random.randint(0, 0x7fffffff)
            seeds = 39485793482 # reproducibility
            
            # create engine
            engine = Engine(terminal_set=my_terminal_set,
                            fitness_func=calc_fit,
                            population_size=pop_size,
                            tournament_size=tour_size,
                            mutation_rate=mut_rate,
                            crossover_rate=cross_rate,
                            max_tree_depth=max_tree_dep,
                            target_dims=res,
                            target=optimal_solution,
                            #elitism=elite_size,
                            method='ramped half-and-half',
                            max_init_depth=max_init_depth,
                            objective='minimizing',
                            device=dev,
                            stop_criteria='generation',
                            stop_value=gens,
                            effective_dims=2,
                            min_domain=-0.5,
                            max_domain=0.5,
                            operators=normal_set,
                            seed=seeds,
                            save_to_file=10,
                            save_graphics=False,
                            show_graphics=False,
                            write_log=False,
                            write_gen_stats=False,
                            read_init_pop_from_file=None)

            # run evolutionary process
            engine.run()

error:

NameError: name 'Terminal_set' is not defined

[AwardOfSky]

Hello @omidr1370, thanks for your interest. The correct method is called Terminal_Set not Terminal_set.
I understand why you made that error as there was a typo in the readme. It's fixed now.
Thanks for the feedback!

[omidr1370]

Thanks. It seems it solved my problem. However, I still have some difficulties defining the terminal. I have the target "F" which is a tensor with shape (19828, 2, 2). I want to define four tensors(T_1,T_2,T_3,T_4) with the same shape like the target(19828, 2, 2). I receive an error. I really appreciate your help if you let me know the correct set-up for such a problem. Below is my code:

def calc_fit(**kwargs):
# read parameters
population = kwargs.get('population')
generation = kwargs.get('generation')
tensors = kwargs.get('tensors')
f_path = kwargs.get('f_path')
_stf = kwargs.get('stf')
target = kwargs.get('target')

fn = f_path + "gen_" + str(generation).zfill(5)
fitness = []
times = []
best_ind = 0

# set objective function according to min/max
fit = 0
condition = lambda: (fit < max_fit)  # minimizing
max_fit = float('inf')

for i in range(len(tensors)):

    start_ind = time.time()
    fit = tf_rmse(tensors[i], target).numpy()

    if condition():
        max_fit = fit
        best_ind = i

    times.append((time.time() - start_ind) * 1000.0)
    fitness.append(fit)
    population[i]['fitness'] = fit

#if generation == gens:
#    save_image(tensors[best_ind], best_ind, fn, 2)

return population, population[best_ind]

normal_set = {'add', 'sub'}

if name == "main":

# GP params
dev = '/gpu:0'  # device to run, write '/cpu_0' to tun on cpu
gens = 250 
pop_size = 2500  
tour_size = 3
mut_rate = 0.1
cross_rate = 0.9
max_tree_dep = 10
max_init_depth = 10
elite_size = 1 # 0 to turn off
runs = 1 # Number of average runs

# problems
optimal_solution = tf.convert_to_tensor(F) #F is a (19828, 2, 2) tensor
optimal_solution = tf.cast(optimal_solution, tf.float32)

indexing_variables = 3
domain_range = [19828, 2, 2]
my_terminal_set = Terminal_Set(indexing_variables, domain_range)
my_terminal_set.add_to_set("T_1", T_1)  # T_1 is a (19828, 2, 2)  tensor
my_terminal_set.add_to_set("T_2", T_2)  # T_2 is a (19828, 2, 2)  tensor
my_terminal_set.add_to_set("T_3", T_3)  # T_3 is a (19828, 2, 2)  tensor
my_terminal_set.add_to_set("T_4", T_4)  # T_4 is a (19828, 2, 2)  tensor

problems = [optimal_solution] # Add to run more problems







# Domains dimensions
test_cases = [[19828, 2, 2]]

for p in problems:


    for res in test_cases:
    
        for r in range(runs):
        
            #seeds = random.randint(0, 0x7fffffff)
            seeds = 39485793482 # reproducibility
        
            # create engine
            engine = Engine(uniform_scalar_prob=0,
                            terminal_set=my_terminal_set,
                            fitness_func=calc_fit,
                            population_size=pop_size,
                            tournament_size=tour_size,
                            mutation_rate=mut_rate,
                            crossover_rate=cross_rate,
                            max_tree_depth=max_tree_dep,
                            target_dims=res,
                            target=optimal_solution,
                            #elitism=elite_size,
                            method='ramped half-and-half',
                            max_init_depth=max_init_depth,
                            objective='minimizing',
                            device=dev,
                            stop_criteria='generation',
                            stop_value=gens,
                            effective_dims=3,
                            min_domain=-0.5,
                            max_domain=0.5,
                            operators=normal_set,
                            seed=seeds,
                            save_to_file=10,
                            save_graphics=False,
                            show_graphics=False,
                            write_log=False,
                            write_gen_stats=False,
                            read_init_pop_from_file=None)

            # run evolutionary process
            engine.run()

Error:

KeyError: 'T'

[AwardOfSky]

Hello @omidr1370. I tried to emulate your setup and had no problems but I don't know how you are defining the tensors you provide. If I had to guess you may be missing a conversion to tensor where you define the terminals:
Try this:

my_terminal_set.add_to_set("T_1", tf.convert_to_tensor(T_1, tf.float32))
my_terminal_set.add_to_set("T_2", tf.convert_to_tensor(T_2, tf.float32))
my_terminal_set.add_to_set("T_3", tf.convert_to_tensor(T_3, tf.float32))
my_terminal_set.add_to_set("T_4", tf.convert_to_tensor(T_4, tf.float32))

If this fails to solve your issue, I would appreciate if you posted the full traceback of the error to see where the program crashes.

[omidr1370]

Hello. Thank you so much for your help. I already converted tensors before using them in the terminal. My problem still remains and I receive an error. It runs only for the first iteration. I use "Jupyter Notebook" and import everything. I put the error at the end of this text.
So, I will explain everything with one more question for better clarification. Perhaps, it will help to find a solution easier. First, I have five tensors, "F" which is the target, and "T_1","T_2","T_3","T_4" which I want to use in my terminal. All these tensors have the same shape of (19828, 2, 2). My goal is to get an equation in this form:
F= C1T_1+C2T_2+C3T_3+C4T_4.
Here C1,.....,C4 are constants. As I don't want any constants to appear alone(C1T_1+C2T_2+C3T_3+C4T_4+C5), I put "uniform_scalar_prob=0". Each of these tensors has 19828 data points which consist of 2x2 tensors. For example, T_1[0]=array([[ 0.10702974, 1.9503768 ],[ 1.9503768 , -0.10702974]]). Moreover, I don't want two tensors multiplied by each other. So, I use only addition(add) and substraction(sub) for my "operators".

I'm not sure about these options for my problem. Did I put these correctly?
indexing_variables = ? 3
domain_range = ? [19828, 2, 2]

Here is the error:

Device /cpu:0 successfully tested, using this device.
Engine seed: 39485793482

[ gen, fit avg, fit std, fit best, dep avg, dep std, dep best, nod avg, nod std, nod best]

[ 0, 1.334728, 0.399250, 0.001294, 5.874, 2.650684, 2, 278.814, 498.216824, 7]

KeyError Traceback (most recent call last)
in
125
126 # run evolutionary process
--> 127 engine.run()

C:\TensorGP-master\tensorgp\engine.py in run(self)
1290
1291 # calculate fitness of the new population
-> 1292 new_population, _new_best = self.fitness_func_wrap(population=new_population,
1293 f_path=self.experiment.current_directory)
1294 # update best and population

C:\TensorGP-master\tensorgp\engine.py in fitness_func_wrap(self, population, f_path)
999 if self.debug > 4: print("\nEvaluating generation: " + str(self.current_generation))
1000 with tf.device(self.device):
-> 1001 tensors, time_taken = self.calculate_tensors(population)
1002 if self.debug > 4: print("Calculated " + str(len(tensors)) + " tensors in (s): " + str(time_taken))
1003

C:\TensorGP-master\tensorgp\engine.py in calculate_tensors(self, population)
979 #_start = time.time()
980
--> 981 test_tens = p['tree'].get_tensor(self)
982 tens = self.final_transform_domain(test_tens)
983 tensors.append(tens)

C:\TensorGP-master\tensorgp\engine.py in get_tensor(self, engref)
294 tens_list = []
295 for c in self.children:
--> 296 tens_list.append(c.get_tensor(engref))
297
298 return self.node_tensor(tens_list, engref)

C:\TensorGP-master\tensorgp\engine.py in get_tensor(self, engref)
294 tens_list = []
295 for c in self.children:
--> 296 tens_list.append(c.get_tensor(engref))
297
298 return self.node_tensor(tens_list, engref)

C:\TensorGP-master\tensorgp\engine.py in get_tensor(self, engref)
290
291 else:
--> 292 return engref.terminal.set[self.value]
293 else:
294 tens_list = []

KeyError: 'T'

[AwardOfSky]

Hey again @omidr1370.
There was an error in the method for replacing modes in point mutation when the terminals had more than 1 character. It's fixed now.
Once again thanks for your patience and feedback!
Let me know if you have any more problems.

PS: the only file that I updated was engine.py in the tensorgp folder.

[omidr1370]

Thank you so much. It works.
Now, I want to define my own operators for my problem. I receive an error in "Function_Set". I put my code and the error below:

code:

def calc_fit(**kwargs):
# read parameters
population = kwargs.get('population')
generation = kwargs.get('generation')
tensors = kwargs.get('tensors')
f_path = kwargs.get('f_path')
_stf = kwargs.get('stf')
target = kwargs.get('target')

fn = f_path + "gen_" + str(generation).zfill(5)
fitness = []
times = []
best_ind = 0

# set objective function according to min/max
fit = 0
condition = lambda: (fit < max_fit)  # minimizing
max_fit = float('inf')

for i in range(len(tensors)):

    start_ind = time.time()
    fit = tf_rmse(tensors[i], target).numpy()

    if condition():
        max_fit = fit
        best_ind = i

    times.append((time.time() - start_ind) * 1000.0)
    fitness.append(fit)
    population[i]['fitness'] = fit

#if generation == gens:
#    save_image(tensors[best_ind], best_ind, fn, 2)

return population, population[best_ind]

def node_add(child1, child2, dims=[]): #only two tensors will be added to each other
if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
return tf.add(child1, child2)
else:
return tf.constant(0,dtype=tf.float32)

def node_sub(child1, child2, dims=[]): #only two tensors will be substracted
if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
return tf.subtract(child1, child2)
else:
return tf.constant(0,dtype=tf.float32)

def node_mult(child1, child2, dims=[]): #only a tensor and a scalar will be multiplied
if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
return tf.constant(0,dtype=tf.float32)
elif len(tf.shape(child1))==0 and len(tf.shape(child2))==0:
return tf.constant(0,dtype=tf.float32)
else:
return tf.convert_to_tensor(child1*child2, dtype=tf.float32)

if name == "main":

# GP params
dev = '/cpu:0'  # device to run, write '/cpu_0' to tun on cpu
gens = 50 
pop_size = 1000  
tour_size = 3
mut_rate = 0.1
cross_rate = 0.9
max_tree_dep = 10
max_init_depth = 10
elite_size = 1 # 0 to turn off
runs = 1 # Number of average runs

# problems
optimal_solution = tf.convert_to_tensor(F) #F is a (19828, 2, 2) tensor
optimal_solution = tf.cast(optimal_solution, tf.float32)

indexing_variables = 3
domain_range = [797,2,2]
my_terminal_set = Terminal_Set(indexing_variables, domain_range)
my_terminal_set.add_to_set("T_1", tf.convert_to_tensor(T_1, tf.float32))
my_terminal_set.add_to_set("T_2", tf.convert_to_tensor(T_2, tf.float32))
my_terminal_set.add_to_set("T_3", tf.convert_to_tensor(T_3, tf.float32))
my_terminal_set.add_to_set("T_4", tf.convert_to_tensor(T_4, tf.float32))
my_terminal_set.remove_from_set("x")
my_terminal_set.remove_from_set("y")
my_terminal_set.remove_from_set("z")


# Define subset of internally implemented operators
primitives = {'add', 'sub', 'mult', 'div'}
my_function_set = Function_Set(primitives)

# Add rmse_node, which has 2 arguments with name "rmse"
my_function_set.add_to_set("node_add", 2, node_add)
my_function_set.add_to_set("node_sub", 2, node_sub)
my_function_set.add_to_set("node_mult", 2, node_mult)


# You can also remove operators from the set
my_function_set-remove_from_set("add")
my_function_set-remove_from_set("sub")
my_function_set-remove_from_set("mult")
my_function_set-remove_from_set("div")






problems = [optimal_solution] # Add to run more problems







# Domains dimensions
test_cases = [[797,2,2]]


for p in problems:


    for res in test_cases:
    
        for r in range(runs):
        
            #seeds = random.randint(0, 0x7fffffff)
            seeds = 39485793482 # reproducibility
        
            # create engine
            engine = Engine(uniform_scalar_prob=0,
                            scalar_prob=0.55,
                            terminal_prob=0.2,
                            terminal_set=my_terminal_set,
                            fitness_func=calc_fit,
                            population_size=pop_size,
                            tournament_size=tour_size,
                            mutation_rate=mut_rate,
                            crossover_rate=cross_rate,
                            max_tree_depth=max_tree_dep,
                            target_dims=res,
                            target=optimal_solution,
                            #elitism=elite_size,
                            method='ramped half-and-half',
                            max_init_depth=max_init_depth,
                            objective='minimizing',
                            device=dev,
                            stop_criteria='generation',
                            stop_value=gens,
                            effective_dims=3,
                            min_domain=-0.25,
                            max_domain=0.25,
                            operators=my_function_set,
                            seed=seeds,
                            save_to_file=10,
                            save_graphics=False,
                            show_graphics=True,
                            write_log=False,
                            write_gen_stats=False,
                            read_init_pop_from_file=None)

            # run evolutionary process
            engine.run()

Error:

---

TypeError Traceback (most recent call last)
in
106 # Define subset of internally implemented operators
107 primitives = {'add', 'sub', 'mult', 'div'}
--> 108 my_function_set = Function_Set(primitives)
109
110 # Add rmse_node, which has 2 arguments with name "rmse"

TypeError: init() missing 2 required positional arguments: 'fset' and 'edim'

[AwardOfSky]

Hello @omidr1370. First and foremost sorry for the delayed response, I have been quite busy.

In the function set you should define the number of variables you want to index, as you did for the engine constructor.
However, by defining the function set the parameter that you want to define is function_set not operators. Again thanks for pointing this out to me because it was also wrong in the readme.

So what you do is

my_function_set = Function_Set(primitives, indexing_variables)

and then:

engine = Engine(function_set=my_function_set,
		... # other parameters,
		)
engine.run()

To leave out any doubts, below I will post an example of a working code based on yours:

    # GP params
    dev = '/cpu:0'  # device to run, write '/cpu_0' to tun on cpu
    gens = 50
    pop_size = 1000
    tour_size = 3
    mut_rate = 0.1
    cross_rate = 0.9
    max_tree_dep = 10
    max_init_depth = 10
    elite_size = 1  # 0 to turn off
    runs = 1  # Number of average runs

    # problems
    optimal_solution = tf.convert_to_tensor(F)  # F is a (19828, 2, 2) tensor
    optimal_solution = tf.cast(optimal_solution, tf.float32)

    indexing_variables = 3
    domain_range = [797, 2, 2]
    my_terminal_set = Terminal_Set(indexing_variables, domain_range)
    my_terminal_set.add_to_set("T_1", tf.convert_to_tensor(T_1, tf.float32))
    my_terminal_set.add_to_set("T_2", tf.convert_to_tensor(T_2, tf.float32))
    my_terminal_set.add_to_set("T_3", tf.convert_to_tensor(T_3, tf.float32))
    my_terminal_set.add_to_set("T_4", tf.convert_to_tensor(T_4, tf.float32))
    my_terminal_set.remove_from_set("x")
    my_terminal_set.remove_from_set("y")
    my_terminal_set.remove_from_set("z")

    # Define subset of internally implemented operators
    primitives = {'add', 'sub', 'mult', 'div'}
    my_function_set = Function_Set(primitives, indexing_variables)

    # Add rmse_node, which has 2 arguments with name "rmse"
    my_function_set.add_to_set("node_add", 2, node_add)
    my_function_set.add_to_set("node_sub", 2, node_sub)
    my_function_set.add_to_set("node_mult", 2, node_mult)

    # You can also remove operators from the set
    my_function_set.remove_from_set("add")
    my_function_set.remove_from_set("sub")
    my_function_set.remove_from_set("mult")
    my_function_set.remove_from_set("div")


    problems = [optimal_solution]  # Add to run more problems

    # Domains dimensions
    test_cases = [[797, 2, 2]]

    for p in problems:

        for res in test_cases:

            for r in range(runs):
                # seeds = random.randint(0, 0x7fffffff)
                seeds = 39485793482  # reproducibility

                # create engine
                engine = Engine(uniform_scalar_prob=0,
                                scalar_prob=0.55,
                                terminal_prob=0.2,
                                terminal_set=my_terminal_set,
                                fitness_func=calc_fit,
                                population_size=pop_size,
                                tournament_size=tour_size,
                                mutation_rate=mut_rate,
                                crossover_rate=cross_rate,
                                max_tree_depth=max_tree_dep,
                                target_dims=res,
                                target=optimal_solution,
                                # elitism=elite_size,
                                method='ramped half-and-half',
                                max_init_depth=max_init_depth,
                                objective='minimizing',
                                device=dev,
                                stop_criteria='generation',
                                stop_value=gens,
                                effective_dims=3,
                                min_domain=-0.25,
                                max_domain=0.25,
                                function_set=my_function_set,
                                seed=seeds,
                                save_to_file=10,
                                save_graphics=False,
                                show_graphics=True,
                                write_log=False,
                                write_gen_stats=False,
                                read_init_pop_from_file=None)

                # run evolutionary process
                engine.run()

    print("Done!")

PS: For future reference, the operators parameter is there for when you want to use a subset of the operators defined by the engine but you don't want to go through the hassle of defining the whole function set (this does work for custom ops like you have here). So, for instance:

ops = {'add, 'sub'}
engine = Engine(operators =ops,
		... # other parameters,
		)
engine.run()

would start an evolutionary run with only the standard sub and add operators defined internally by the engine.

[omidr1370]

Thank you so much for your time and clarification. I still have problem and receive an error due to "edim" inside "Function_Set" class. Do you use the same "engine.py" code? I used the same as the one you suggested but I receive the following error. Again, I am really grateful for the time you put to answer my questions.

---

TypeError Traceback (most recent call last)
in
105 # Define subset of internally implemented operators
106 primitives = {'add', 'sub', 'mult', 'div'}
--> 107 my_function_set = Function_Set(primitives, indexing_variables)
108
109 # Add rmse_node, which has 2 arguments with name "rmse"

TypeError: init() missing 1 required positional argument: 'edim'

[AwardOfSky]

Yup, there was an outdated version of the Function Set. Should be solved now.

[omidr1370]

Thanks. Now, it's running but I can't understand why I can't see custom functions in the result. For example, these are my custom functions.

def node_add(child1, child2, dims=[]):           #only two tensors will be added to each other
    if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
        return tf.add(child1, child2)
    else:
        return tf.constant(0,dtype=tf.float32)


def node_sub(child1, child2, dims=[]):          #only two tensors will be substracted
    if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
        return tf.subtract(child1, child2)
    else:
        return tf.constant(0,dtype=tf.float32)


def node_mult(child1, child2, dims=[]):        #only a tensor and a scalar will be multiplied
    if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
        return tf.constant(0,dtype=tf.float32)
    elif len(tf.shape(child1))==0 and len(tf.shape(child2))==0:
        return tf.constant(0,dtype=tf.float32)
    else:
        return tf.convert_to_tensor(child1*child2, dtype=tf.float32)

And this is my setup and the result of such setup. As you see, two tensors multiplied to each other which I don't want.

Setup

    # GP params
    dev = '/cpu:0'  # device to run, write '/cpu_0' to tun on cpu
    gens = 25
    pop_size = 1000
    tour_size = 3
    mut_rate = 0.1
    cross_rate = 0.9
    max_tree_dep = 10
    max_init_depth = 10
    elite_size = 1  # 0 to turn off
    runs = 1  # Number of average runs

    # problems
    optimal_solution = tf.convert_to_tensor(F)  # F is a (19828, 2, 2) tensor
    optimal_solution = tf.cast(optimal_solution, tf.float32)

    indexing_variables = 3
    domain_range = [7329, 2, 2]
    my_terminal_set = Terminal_Set(indexing_variables, domain_range)
    my_terminal_set.add_to_set("T_1", tf.convert_to_tensor(T_1, tf.float32))
    my_terminal_set.add_to_set("T_2", tf.convert_to_tensor(T_2, tf.float32))
    my_terminal_set.add_to_set("T_3", tf.convert_to_tensor(T_3, tf.float32))
    my_terminal_set.add_to_set("T_4", tf.convert_to_tensor(T_4, tf.float32))
    my_terminal_set.remove_from_set("x")
    my_terminal_set.remove_from_set("y")
    #my_terminal_set.remove_from_set("z")

    # Define subset of internally implemented operators
    primitives = {'add', 'sub', 'mult', 'div'}
    my_function_set = Function_Set(primitives, indexing_variables)

    # Add rmse_node, which has 2 arguments with name "rmse"
    my_function_set.add_to_set("node_add", 2, node_add)
    my_function_set.add_to_set("node_sub", 2, node_sub)
    my_function_set.add_to_set("node_mult", 2, node_mult)

    # You can also remove operators from the set
    my_function_set.remove_from_set("add")
    my_function_set.remove_from_set("sub")
    my_function_set.remove_from_set("mult")
    my_function_set.remove_from_set("div")


    problems = [optimal_solution]  # Add to run more problems

    # Domains dimensions
    test_cases = [[7329, 2, 2]]

    for p in problems:

        for res in test_cases:

            for r in range(runs):
                # seeds = random.randint(0, 0x7fffffff)
                seeds = 39485793482  # reproducibility

                # create engine
                engine = Engine(uniform_scalar_prob=0.8,
                                scalar_prob=0,
                                terminal_prob=0.2,
                                terminal_set=my_terminal_set,
                                fitness_func=calc_fit,
                                population_size=pop_size,
                                tournament_size=tour_size,
                                mutation_rate=mut_rate,
                                crossover_rate=cross_rate,
                                max_tree_depth=max_tree_dep,
                                target_dims=res,
                                target=optimal_solution,
                                # elitism=elite_size,
                                method='ramped half-and-half',
                                max_init_depth=max_init_depth,
                                objective='minimizing',
                                device=dev,
                                stop_criteria='generation',
                                stop_value=gens,
                                effective_dims=3,
                                min_domain=-0.25,
                                max_domain=0.25,
                                function_set=my_function_set,
                                seed=seeds,
                                save_to_file=10,
                                save_graphics=False,
                                show_graphics=True,
                                write_log=False,
                                write_gen_stats=False,
                                read_init_pop_from_file=None)

                # run evolutionary process
                engine.run()

    print("Done!")

Result

Best individual:
node_add(node_sub(T_4, T_1), node_sub(node_sub(T_4, T_3), node_add(node_add(node_mult(node_sub(T_1, T_4), node_mult(T_4, T_2)), node_sub(node_add(T_3, T_2), node_add(T_1, T_4))), T_3))) 

My desired form of the equation is:

C₀T₁+C₁T₂+C₂T₃+C₃T₄

which C₀,C₁,C₂ and C₃ are constants.

[omidr1370]

To ask my question clearly, I see my functions in the result but I can't see them working properly. For example, two tensors are multiplied by each other(T_4 and T_2) which I limited to avoid this multiplication in my custom function. Do you know the reason?
Thank you in advance for your time.

[AwardOfSky]

The functions that you define in the terminal set are the ones that are used by the engine.
Maybe I did not understand your question, but the fact that you see node_mult(T4, T2) in the resulting individuals does not mean that they are being multiplied, simply that your custom node_mult operator is being called.

I did a test with this code:

    T_1 = np.ones([797, 2, 2], np.float)
    T_2 = np.ones([797, 2, 2], np.float)
    tf_node_mult = node_mult(T_1, T_2)
    print(tf_node_mult.numpy())

And the result is 0.0 as expected according to your implementation of node_mult, because both T_1 and T_2 are rank 3 tensors.

Btw, your custom operators should always return tensors of the same shape so there are no conflicts in subsequent tensors operations
(EDIT: some TensorFlow operations between tensors of different shapes are undefined, but if you can get a working system like this, maybe it pays off with the lesser bottleneck).
Nevertheless, I would encourage you to change your node_mult to have the arguments as your input using the dims parameter (that's the use for this parameter) and only then optimize. For example, you should first do something like:

def node_mult(child1, child2, dims=[]):
    if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
        return tf.constant(0,shape=dims,dtype=tf.float32)
    elif len(tf.shape(child1))==0 and len(tf.shape(child2))==0:
        return tf.constant(0,shape=dims,dtype=tf.float32)
    else:
        return tf.convert_to_tensor(child1*child2, dtype=tf.float32) #oh, and there is no need to explicitly convert to tensor here

[omidr1370]

Thank you so much for your suggestion and explanation.
There is a problem here. When I allow scalars to appear, changing the engine parameters, the prediction seems most of the time to be zero.

So, I don't want:

1. Two tensors or two scalars multiplied by each other.
2. Two scalars added or subtracted by each other.(The only exception is zero)
3. One tensor and one scalar added or subtracted by each other.

In the above situations, my code will return zero.

For example, this is the result of one of the runs:

node_sub(node_add(node_sub(scalar(-0.024643402271970538), T_3), node_mult(node_sub(node_mult(scalar(-0.008758578372655368), scalar(-0.05651795832957618)), node_add(T_2, T_1)), node_mult(scalar(-0.09503844302366904), scalar(-0.10830596509618018)))), scalar(0.2486832856702118)) 

If you just take a look at the last term scalar(0.2486832856702118), you see it will be subtracted finally by something which will definitely be zero. The reason is that only two tensors can be added or subtracted.

To summarize again my goal of using this approach, I want to receive an equation in the following form:

C0T1+C1T2+C2T3+C3T4

C0,C1,C2,C3 are constants/scalars.
Something similar to linear regression. Is this possible with this approach?

I will put my code again here for better discussion.
my custom functions

def node_add(child1, child2, dims=[]):           #only two tensors will be added to each other
    if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
        return tf.add(child1, child2)
    elif np.all(child1==0) or np.all(child2==0):
        return tf.add(child1, child2)
    else:
        return tf.constant(0,shape=dims,dtype=tf.float32)


def node_sub(child1, child2, dims=[]):          #only two tensors will be substracted
    if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
        return tf.math.subtract(child1, child2)
    elif np.all(child1==0) or np.all(child2==0):
        return tf.math.subtract(child1, child2)
    else:
        return tf.constant(0,shape=dims,dtype=tf.float32)


def node_mult(child1, child2, dims=[]):
    if len(tf.shape(child1))==3 and len(tf.shape(child2))==3:
        return tf.constant(0,shape=dims,dtype=tf.float32)
    elif len(tf.shape(child1))==0 and len(tf.shape(child2))==0:
        return tf.constant(0,shape=dims,dtype=tf.float32)
    else:
        return tf.convert_to_tensor(child1*child2, dtype=tf.float32) 

And here is the engine

                engine = Engine(uniform_scalar_prob=0,
                                scalar_prob=0.35,
                                terminal_prob=0.2,
                                terminal_set=my_terminal_set,
                                fitness_func=calc_fit,
                                population_size=pop_size,
                                tournament_size=tour_size,
                                mutation_rate=mut_rate,
                                crossover_rate=cross_rate,
                                max_tree_depth=max_tree_dep,
                                target_dims=res,
                                target=optimal_solution,
                                # elitism=elite_size,
                                method='ramped half-and-half',
                                max_init_depth=max_init_depth,
                                objective='minimizing',
                                device=dev,
                                stop_criteria='generation',
                                stop_value=gens,
                                effective_dims=3,
                                min_domain=-0.25,
                                max_domain=0.25,
                                #operators=normal_set,
                                function_set=my_function_set,
                                seed=seeds,
                                save_to_file=10,
                                save_graphics=False,
                                show_graphics=False,
                                write_log=False,
                                write_gen_stats=False,
                                read_init_pop_from_file=None)

                # run evolutionary process
                engine.run()

    print("Done!")

[AwardOfSky]

Hello @omidr1370 . If I understood correctly what you are trying to achieve is linear regression, which can be seen as a subgroup of symbolic regression.

However, the aim of TensorGP (and GP as a whole) is to evolve populations of expressions freely without restrictions (without domain-specific knowledge about the problem).
If you wish all individuals of the population to follow the same formula, you could do it by editing the engine.py file.
However, I would first encourage you to perform a standard evolutionary run first.
If a linear formula is the most efficient way to approximate your target, then, in theory, with enough generations (and probably some parameter tweaking), evolution will lead individuals to that formula.

Then again, there are other tools specifically designed for that kind of approximation.
Sorry if this sounds discouraging, maybe I did not understand your problem fully.