Second GPU corrupts output?

[Voraciousreader5]

So for whatever reason the second worker is seemingly having issues communicating on a second pass running my image through a high res fix node. It ends with one of my 2 images coming out corrupted and I believe it has to due with the issue the worker is having as I've ruled out just about everything else.

I'm relatively new to using ComfyUI so I am unsure if this is simply a limitation due to my workflows setup not being compatible or something else but I do see the second worker struggling in the logs so I figured I would come here to share this issue and maybe get some help solving it or raise awareness to a bug if it is actually one.

Heres the logs from my second worker + my workflow

Worker 1 log.txt

12.1-Batch Generation.json

[robertvoy]

what setup do you have? Meaning what GPUs are you using with master and worker?

[Voraciousreader5]

what setup do you have? Meaning what GPUs are you using with master and worker?

2 NVIDIA RTX 6000 Ada Generation, forgive me if thats not what you were asking for I'm fairly new to all this.

[robertvoy]

Can you try a simple txt2img distributed workflow from this repo?

[Voraciousreader5]

Can you try a simple txt2img distributed workflow from this repo?

Yes I've tried your sample workflow and it does work the same actually goes for my workflow, but I guess maybe it is a specific issue to my workflow that when we finish generating two images and then it runs it through the high quality fix the second image comes out corrupted. I guess you could say its just a me issue but it does seem like for whatever reason the worker stops communicating correctly and possibly thats a bug or maybe its just not built in such a way that it can handle what I am doing in my workflow

[robertvoy]

Try connecting the collector like this

[Voraciousreader5]

Try connecting the collector like this

Yeah looks like i still get the same result

I did a lot of trial and error and monitoring of when it fails and seemingly this is the part in the worker log where it does

loaded partially; 0.00 MB usable, 0.00 MB loaded, 242.00 MB offloaded, 22.78 MB buffer reserved, lowvram patches: 0
[Distributed] Worker - Failed to send chunk to master: 404, message='Not Found', url='http://127.0.0.1:7801/Error/404'
!!! Exception during processing !!! 404, message='Not Found', url='http://127.0.0.1:7801/Error/404'
Traceback (most recent call last):
File "/home/Ubuntu/apps/ComfyUI/execution.py", line 510, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
File "/home/Ubuntu/apps/ComfyUI/execution.py", line 324, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
File "/home/Ubuntu/apps/ComfyUI/execution.py", line 298, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/Ubuntu/apps/ComfyUI/execution.py", line 286, in process_inputs
result = f(**inputs)
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/distributed.py", line 1878, in run
result = run_async_in_server_loop(
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/utils/async_helpers.py", line 49, in run_async_in_server_loop
raise error
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/utils/async_helpers.py", line 34, in wrapper
result = await coro
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/distributed.py", line 1931, in execute
await self.send_batch_to_master(images, multi_job_id, master_url, worker_id)
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/distributed.py", line 1920, in send_batch_to_master
response.raise_for_status()
File "/home/Ubuntu/apps/StableSwarmUI/dlbackend/ComfyUI/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 638, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://127.0.0.1:7801/Error/404'

[robertvoy]

Try increasing this value to 1200

[Voraciousreader5]

Try increasing this value to 1200

Seemingly didnt help unfortunately

Still getting the:

Worker - Failed to send chunk to master: 404, message='Not Found', url='http://127.0.0.1:7801/Error/404'
!!! Exception during processing !!! 404, message='Not Found', url='http://127.0.0.1:7801/Error/404'
Traceback (most recent call last):
File "/home/Ubuntu/apps/ComfyUI/execution.py", line 510, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
File "/home/Ubuntu/apps/ComfyUI/execution.py", line 324, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
File "/home/Ubuntu/apps/ComfyUI/execution.py", line 298, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/Ubuntu/apps/ComfyUI/execution.py", line 286, in process_inputs
result = f(**inputs)
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/distributed.py", line 1878, in run
result = run_async_in_server_loop(
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/utils/async_helpers.py", line 49, in run_async_in_server_loop
raise error
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/utils/async_helpers.py", line 34, in wrapper
result = await coro
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/distributed.py", line 1931, in execute
await self.send_batch_to_master(images, multi_job_id, master_url, worker_id)
File "/home/Ubuntu/apps/ComfyUI/custom_nodes/ComfyUI-Distributed/distributed.py", line 1920, in send_batch_to_master
response.raise_for_status()
File "/home/Ubuntu/apps/StableSwarmUI/dlbackend/ComfyUI/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 638, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://127.0.0.1:7801/Error/404'