A question about test_optim_adam_z_memory_check_1

[hkr04]

With my implementation and an old version from others, I got this assertion error:

But if I print one of the m, v, grad，it will pass (for both my implementation and another one):

Note that I only transform the m and v to ndl.Tensor for once and use the .data in calculation, and I can pass the previous test_optim_sgd_z_memory_check_1.

Could anyone explain this? I'm quite confused why this would influence the count of tensors. Thanks for help :)

[chenjisuan]

same question，my result is 2180, i use .data everywhere, still cannot make it 1132.

[chenjisuan]

same question，my result is 2180, i use .data everywhere, still cannot make it 1132.

i debug my code, and find it that it is BatchNorm1d rather than optimizer.Adam uses too many tensor.

debug code herer:

import gc
import pytest
import time

@pytest.fixture(autouse=True)
def before_each(request):
    print_global_tensor_count("[before]" + request.node.name)
    yield

@pytest.fixture(autouse=True)
def after_each(request):
    yield
    print_global_tensor_count("[after]" + request.node.name)
    # add gc do clear all the tensor, maybe there is cycle reference in my code, cause Tensor.__del is called after a while.
    gc.collect()
    print_global_tensor_count("[after]" + request.node.name)  

def print_global_tensor_count(prefix):
    print("%s: %d\n" % (prefix, global_tensor_count()))

here is what i found during debuging.

• without gc, tensor count do increase by each test case. with gc, it is zero after gc is called.
• next step, which test case increase tensor count the most.
• i found two case, case test_optim_adam_batchnorm_1 increases much more than test_optim_adam_weight_decay_1, and the main diff is test_optim_adam_batchnorm_1 use an additional BatchNorm1d.
• i check my BatchNorm1d code, i find it that i misuse tensor with running mean and var.

# old 
self.running_mean = (1-self.momentum) * self.running_mean + self.momentum * mean_raw
self.running_var = (1-self.momentum) * self.running_var + self.momentum * var_raw

# new
self.running_mean.data = (1-self.momentum) * self.running_mean + self.momentum * mean_raw
self.running_var.data = (1-self.momentum) * self.running_var + self.momentum * var_raw

we dont need to calculate the gradient of running mean and var, it unnessary to keep compute relationship(even a ndarray is enough).
and final tensor count decrease from 2180 to 941.

my advise:

• check Adam first, reduce tensor op that may cause increasing compute graph nodes.
• check which test case increase much more than others, analyze it.
• check modules of a test case
• check ops implement
• two thing will increase unnessary tensor count.
  • unnessary tensor op, try use .data or detach() if you dont want to keep a computing relationship.
  • optimize you code, a function can be implemented by many ways, they may use different numbers of ops. (i use sum、reshape、 broadcast to get mean and var of a tensor, it is easy to think of, but i guess there is better implementation)