[2025-09-25T21:26:04.805Z] + docker exec 0e5430b26af20dda4900ad2e28194dc0ee5e8f929492e59c3a1bcc6e2de8a3a6 bazel --bazelrc=tensorflow/tools/tf_sig_build_dockerfiles/devel.usertools/rocm.bazelrc test --local_ram_resources=60000 --local_cpu_resources=32 --jobs=64 --verbose_failures --disk_cache=/tf/cache --config=sigbuild_local_cache --config=rocm --config=nonpip_multi_gpu --repo_env=USE_PYWRAP_RULES=True --action_env=TF_PYTHON_VERSION=3.10 --test_env=TF_TESTS_PER_GPU=1 --test_env=TF_GPU_COUNT=2 --local_test_jobs=2 --repo_env=TF_ROCM_AMDGPU_TARGETS=gfx90a
[2025-09-25T21:26:04.805Z] 2025/09/25 21:26:04 Downloading https://releases.bazel.build/6.5.0/release/bazel-6.5.0-linux-x86_64...
[2025-09-25T21:26:05.720Z] Extracting Bazel installation...
[2025-09-25T21:26:06.678Z] Starting local Bazel server and connecting to it...
[2025-09-25T21:26:08.792Z] INFO: Invocation ID: aa539ad6-dc41-44c2-b16c-0fee78d59db8
[2025-09-25T21:26:08.792Z] INFO: Reading 'startup' options from /tf/tensorflow/.bazelrc: --windows_enable_symlinks
[2025-09-25T21:26:08.792Z] INFO: Options provided by the client:
[2025-09-25T21:26:08.792Z]   Inherited 'common' options: --isatty=0 --terminal_columns=80
[2025-09-25T21:26:08.792Z] INFO: Reading rc options for 'test' from /tf/tensorflow/.bazelrc:
[2025-09-25T21:26:08.792Z]   Inherited 'common' options: --experimental_repo_remote_exec
[2025-09-25T21:26:08.792Z] INFO: Reading rc options for 'test' from /tf/tensorflow/.bazelrc:
[2025-09-25T21:26:08.793Z]   Inherited 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
[2025-09-25T21:26:08.793Z] INFO: Reading rc options for 'test' from /tf/tensorflow/tensorflow/tools/tf_sig_build_dockerfiles/devel.usertools/gpu.bazelrc:
[2025-09-25T21:26:08.793Z]   Inherited 'build' options: --action_env=CACHEBUSTER=565341047
[2025-09-25T21:26:08.793Z] INFO: Reading rc options for 'test' from /tf/tensorflow/.bazelrc:
[2025-09-25T21:26:08.793Z]   'test' options: --test_env=GTEST_INSTALL_FAILURE_SIGNAL_HANDLER=1
[2025-09-25T21:26:08.793Z] INFO: Reading rc options for 'test' from /tf/tensorflow/tensorflow/tools/tf_sig_build_dockerfiles/devel.usertools/gpu.bazelrc:
[2025-09-25T21:26:08.793Z]   'test' options: --test_output=errors --test_timeout=920,2400,7200,9600 --local_test_jobs=4 --run_under=//tensorflow/tools/ci_build/gpu_build:parallel_gpu_execute
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:short_logs in file /tf/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:v2 in file /tf/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:sigbuild_local_cache in file /tf/tensorflow/tensorflow/tools/tf_sig_build_dockerfiles/devel.usertools/gpu.bazelrc: --disk_cache=/tf/cache
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:rocm in file /tf/tensorflow/.bazelrc: --config=rocm_base --config=release_cpu_linux_base --action_env=CLANG_COMPILER_PATH=/usr/lib/llvm-18/bin/clang --action_env=TF_ROCM_CLANG=1 --linkopt=-fuse-ld=lld --host_linkopt=-fuse-ld=lld --linkopt=-Wl,--undefined-version --copt=-Wno-gnu-offsetof-extensions --copt=-Wno-unused-result
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:rocm_base in file /tf/tensorflow/.bazelrc: --crosstool_top=@local_config_rocm//crosstool:toolchain --define=using_rocm_hipcc=true --define=tensorflow_mkldnn_contraction_kernel=0 --define=xnn_enable_avxvnniint8=false --define=xnn_enable_avx512fp16=false --repo_env TF_NEED_ROCM=1 --config=no_tfrt
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:no_tfrt in file /tf/tensorflow/.bazelrc: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/ir,tensorflow/compiler/mlir/tfrt/ir/mlrt,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ifrt,tensorflow/compiler/mlir/tfrt/tests/mlrt,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/compiler/mlir/tfrt/transforms/mlrt,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/runtime_fallback/test,tensorflow/core/runtime_fallback/test/gpu,tensorflow/core/runtime_fallback/test/saved_model,tensorflow/core/runtime_fallback/test/testdata,tensorflow/core/tfrt/stubs,tensorflow/core/tfrt/tfrt_session,tensorflow/core/tfrt/mlrt,tensorflow/core/tfrt/mlrt/attribute,tensorflow/core/tfrt/mlrt/kernel,tensorflow/core/tfrt/mlrt/bytecode,tensorflow/core/tfrt/mlrt/interpreter,tensorflow/compiler/mlir/tfrt/translate/mlrt,tensorflow/compiler/mlir/tfrt/translate/mlrt/testdata,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils,tensorflow/core/tfrt/utils/debug,tensorflow/core/tfrt/saved_model/python,tensorflow/core/tfrt/graph_executor/python,tensorflow/core/tfrt/saved_model/utils
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:release_cpu_linux_base in file /tf/tensorflow/.bazelrc: --repo_env=CC=/usr/lib/llvm-18/bin/clang --repo_env=BAZEL_COMPILER=/usr/lib/llvm-18/bin/clang --action_env=CLANG_COMPILER_PATH=/usr/lib/llvm-18/bin/clang --linkopt=-fuse-ld=lld
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition test:rocm in file /tf/tensorflow/tensorflow/tools/tf_sig_build_dockerfiles/devel.usertools/gpu.bazelrc: --test_env=HSA_TOOLS_LIB=libroctracer64.so --test_sharding_strategy=disabled --action_env=TF_ENABLE_ONEDNN_OPTS=0 --action_env=OPENBLAS_CORETYPE=Haswell
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition test:nonpip_multi_gpu in file /tf/tensorflow/tensorflow/tools/tf_sig_build_dockerfiles/devel.usertools/gpu.bazelrc: --config=nonpip_filters_multi_gpu -- //tensorflow/core/nccl:nccl_manager_test_2gpu //tensorflow/python/distribute/integration_test:mwms_peer_failure_test_2gpu //tensorflow/python/distribute:checkpoint_utils_test_2gpu //tensorflow/python/distribute:checkpointing_test_2gpu //tensorflow/python/distribute:collective_all_reduce_strategy_test_xla_2gpu //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu //tensorflow/python/distribute:custom_training_loop_input_test_2gpu //tensorflow/python/distribute:distribute_utils_test_2gpu //tensorflow/python/distribute:input_lib_test_2gpu //tensorflow/python/distribute:input_lib_type_spec_test_2gpu //tensorflow/python/distribute:metrics_v1_test_2gpu //tensorflow/python/distribute:mirrored_variable_test_2gpu //tensorflow/python/distribute:parameter_server_strategy_test_2gpu //tensorflow/python/distribute:ps_values_test_2gpu //tensorflow/python/distribute:random_generator_test_2gpu //tensorflow/python/distribute:test_util_test_2gpu //tensorflow/python/distribute:tf_function_test_2gpu //tensorflow/python/distribute:vars_test_2gpu //tensorflow/python/distribute:warm_starting_util_test_2gpu //tensorflow/python/training:saver_test_2gpu
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition test:nonpip_filters_multi_gpu in file /tf/tensorflow/tensorflow/tools/tf_sig_build_dockerfiles/devel.usertools/gpu.bazelrc: --test_tag_filters=-no_gpu,-cuda-only --build_tag_filters=-no_gpu,-cuda-only --test_lang_filters=py --flaky_test_attempts=2 --test_size_filters=small,medium,large --test_env=TF_PER_DEVICE_MEMORY_LIMIT_MB=2048
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:linux in file /tf/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --linkopt=-Wl,--undefined-version --host_linkopt=-Wl,--undefined-version --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
[2025-09-25T21:26:08.793Z] INFO: Found applicable config definition build:dynamic_kernels in file /tf/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
[2025-09-25T21:26:08.793Z] Loading: 
[2025-09-25T21:26:09.750Z] Loading: 
[2025-09-25T21:26:10.228Z] DEBUG: /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/external/local_tsl/third_party/py/python_repo.bzl:83:14: !!!Using pywrap rules instead of directly creating .so objects!!!
[2025-09-25T21:26:10.228Z] DEBUG: /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/external/local_tsl/third_party/py/python_repo.bzl:88:10: 
[2025-09-25T21:26:10.228Z] =============================
[2025-09-25T21:26:10.228Z] Hermetic Python configuration:
[2025-09-25T21:26:10.228Z] Version: "3.10"
[2025-09-25T21:26:10.228Z] Kind: ""
[2025-09-25T21:26:10.228Z] Interpreter: "default" (provided by rules_python)
[2025-09-25T21:26:10.228Z] Requirements_lock label: "@python_version_repo//:requirements_lock_3_10.txt"
[2025-09-25T21:26:10.228Z] =====================================
[2025-09-25T21:26:10.704Z] Loading: 
[2025-09-25T21:26:12.184Z] Loading: 
[2025-09-25T21:26:13.098Z] Loading: 
[2025-09-25T21:26:14.589Z] Loading: 
[2025-09-25T21:26:15.506Z] Loading: 
[2025-09-25T21:26:37.259Z] Loading: 
[2025-09-25T21:26:37.734Z] Loading: 
[2025-09-25T21:26:38.692Z] Loading: 
[2025-09-25T21:26:39.824Z] Loading: 
[2025-09-25T21:26:40.303Z] Loading: 
[2025-09-25T21:26:40.303Z] Loading: 0 packages loaded
[2025-09-25T21:26:41.349Z] Analyzing: 17 targets (4 packages loaded, 0 targets configured)
[2025-09-25T21:26:42.308Z] Analyzing: 17 targets (46 packages loaded, 10 targets configured)
[2025-09-25T21:26:43.891Z] Analyzing: 17 targets (46 packages loaded, 10 targets configured)
[2025-09-25T21:26:45.415Z] Analyzing: 17 targets (46 packages loaded, 10 targets configured)
[2025-09-25T21:26:47.574Z] Analyzing: 17 targets (46 packages loaded, 10 targets configured)
[2025-09-25T21:26:49.859Z] Analyzing: 17 targets (83 packages loaded, 306 targets configured)
[2025-09-25T21:26:50.420Z] Analyzing: 17 targets (336 packages loaded, 12584 targets configured)
[2025-09-25T21:26:51.630Z] Analyzing: 17 targets (535 packages loaded, 25301 targets configured)
[2025-09-25T21:26:52.838Z] Analyzing: 17 targets (570 packages loaded, 32331 targets configured)
[2025-09-25T21:26:53.399Z] Analyzing: 17 targets (731 packages loaded, 49765 targets configured)
[2025-09-25T21:26:54.605Z] Analyzing: 17 targets (754 packages loaded, 59835 targets configured)
[2025-09-25T21:26:55.769Z] Analyzing: 17 targets (757 packages loaded, 60958 targets configured)
[2025-09-25T21:26:57.017Z] Analyzing: 17 targets (763 packages loaded, 63980 targets configured)
[2025-09-25T21:26:58.184Z] ERROR: /tf/tensorflow/tensorflow/python/BUILD:1509:15: in _pywrap_common_split_library rule //tensorflow/python:_libtensorflow_framework.so.2_split: 
[2025-09-25T21:26:58.184Z] Traceback (most recent call last):
[2025-09-25T21:26:58.184Z] 	File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/external/local_tsl/third_party/py/rules_pywrap/pywrap.impl.bzl", line 343, column 44, in _pywrap_common_split_library_impl
[2025-09-25T21:26:58.184Z] 		return _construct_split_library_cc_info(ctx, split_linker_inputs, [], [])
[2025-09-25T21:26:58.184Z] 	File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/external/local_tsl/third_party/py/rules_pywrap/pywrap.impl.bzl", line 371, column 59, in _construct_split_library_cc_info
[2025-09-25T21:26:58.184Z] 		dependency_libraries = _construct_dependency_libraries(
[2025-09-25T21:26:58.184Z] 	File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/external/local_tsl/third_party/py/rules_pywrap/pywrap.impl.bzl", line 404, column 60, in _construct_dependency_libraries
[2025-09-25T21:26:58.184Z] 		lib_copy = cc_common.create_library_to_link(
[2025-09-25T21:26:58.184Z] Error in create_library_to_link: Must pass at least one artifact
[2025-09-25T21:26:58.184Z] ERROR: /tf/tensorflow/tensorflow/python/BUILD:1509:15: Analysis of target '//tensorflow/python:_libtensorflow_framework.so.2_split' failed
[2025-09-25T21:26:58.184Z] ERROR: Analysis of target '//tensorflow/python/distribute:checkpointing_test_2gpu' failed; build aborted: 
[2025-09-25T21:26:58.184Z] INFO: Elapsed time: 52.367s
[2025-09-25T21:26:58.184Z] INFO: 0 processes.
[2025-09-25T21:26:58.184Z] FAILED: Build did NOT complete successfully (763 packages loaded, 64062 targets configured)
[2025-09-25T21:26:58.184Z] ERROR: Couldn't start the build. Unable to run tests