root@c1fed017e0b8:/workspace/physicsnemo/examples/additive_manufacturing/sintering_physics# python train.py
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1777139445.563010 771 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1777139445.945102 771 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1777139447.430253 771 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/usr/local/lib/python3.12/dist-packages/physicsnemo/distributed/manager.py:415: UserWarning: Could not initialize using ENV, SLURM or OPENMPI methods. Assuming this is a single process job
warn(
check cfg loading: {'general': {'mode': 'train', 'eval_split': 'inference', 'device': 'cuda:0', 'message_passing_devices': "['cuda:0']", 'fp16': False}, 'train_options': {'batch_size': 1, 'num_steps': 20000000.0, 'eval_steps': 1, 'prefetch_buffer_size': 100, 'input_seq_len': 5, 'pred_len': 1, 'loss': 'me', 'loss_decay_factor': 0.6, 'l_plane': 30, 'l_me': 3}, 'data_options': {'data_path': './data/test_validation', 'noise_std': 1e-09, 'ckpt_path_vfgn': 'models/test24', 'output_path': 'rollouts/test2404', 'NUM_PARTICLE_TYPES': 3, 'KINEMATIC_PARTICLE_ID': 0, 'METAL_PARTICLE_ID': 2, 'ANCHOR_PLANE_PARTICLE_ID': 1}, 'test_options': {'rollout_refine': False, 'rollout_path': 'rollouts/test24/rollout_test_0.json', 'metadata_path': './data/2024-3-absnorm', 'step_stride': 3, 'block_on_show': True, 'ds_type': 'str = "standard"', 'test_build_name': 'test0', 'plot_tolerance_range': True, 'plot_3d': False}}
Error executing job with overrides: []
Traceback (most recent call last):
File "/workspace/physicsnemo/examples/additive_manufacturing/sintering_physics/train.py", line 721, in main
Train(rank_zero_logger, dist, cfg)
File "/workspace/physicsnemo/examples/additive_manufacturing/sintering_physics/train.py", line 76, in Train
dataset = GraphDataset(
^^^^^^^^^^^^^
File "/workspace/physicsnemo/examples/additive_manufacturing/sintering_physics/graph_dataset.py", line 239, in __init__
self.dataset = get_input_fn(
^^^^^^^^^^^^^
File "/workspace/physicsnemo/examples/additive_manufacturing/sintering_physics/graph_dataset.py", line 93, in input_fn
metadata = _read_metadata(data_path)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/physicsnemo/examples/additive_manufacturing/sintering_physics/utils.py", line 62, in _read_metadata
with open(os.path.join(data_path, "metadata.json"), "rt") as fp:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './data/test_validation/metadata.json'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Version
25.08
On which installation method(s) does this occur?
Docker
Describe the issue
Minimum reproducible example
Relevant log output
Environment details