break with "Resume Training" ,cannot finish

[anthony-zg]

Operating System

• macOS
• Linux
• Windows

Deployment Method

• Docker
• non-Docker

CUDA Usage

• Yes
• No

Training Process Details (if applicable)

Model：QWen2.5-1.5B-Instruct
Machine Base config: RAM 192GB，24GB VRAM(4090)

Second-Me version

179c52e (HEAD -> master, tag: v1.0.1, origin/master, origin/HEAD)

Describe the bug

always break training,show "resume training" button.but click that button, show same logs

2025-05-25 17:31:30 [WARNING] chroma_utils.py:60 - Unknown embedding model: text-embedding-v3, defaulting to 1536 dimensions
2025-05-25 17:31:30 [INFO] embedding_service.py:30 - Detected embedding dimension: 1536 for model: text-embedding-v3
2025-05-25 17:31:30 [WARNING] chroma_utils.py:60 - Unknown embedding model: text-embedding-v3, defaulting to 1536 dimensions
2025-05-25 17:31:30 [INFO] embedding_service.py:30 - Detected embedding dimension: 1536 for model: text-embedding-v3
2025-05-25 17:31:30 [WARNING] chroma_utils.py:60 - Unknown embedding model: text-embedding-v3, defaulting to 1536 dimensions
2025-05-25 17:31:30 [INFO] embedding_service.py:30 - Detected embedding dimension: 1536 for model: text-embedding-v3
2025-05-25 17:31:30 [INFO] train.py:200 - Initializing training with memory optimizations
2025-05-25 17:31:30 [INFO] train.py:204 - Applying memory optimizations to training configuration
2025-05-25 17:31:30 [INFO] train.py:234 - Loading model with automatic memory management from /app/resources/L2/base_models/Qwen2.5-1.5B-Instruct
2025-05-25 17:32:00 [INFO] train.py:286 - Enabling gradient checkpointing for memory efficiency

Current Behavior

see

Expected Behavior

see

Reproduction Steps

see

Possible Workaround

No response

Additional Information

No response

Link to related Github discussion or issue

No response

[anthony-zg]

retries some times, it's break at final step.and when i click "resume training" , just 10 seconds, it's break -- change to "resume training"

2025-05-26 16:14:29 [WARNING] chroma_utils.py:60 - Unknown embedding model: text-embedding-v3, defaulting to 1536 dimensions
2025-05-26 16:14:29 [INFO] embedding_service.py:30 - Detected embedding dimension: 1536 for model: text-embedding-v3
2025-05-26 16:14:29 [WARNING] chroma_utils.py:60 - Unknown embedding model: text-embedding-v3, defaulting to 1536 dimensions
2025-05-26 16:14:29 [INFO] embedding_service.py:30 - Detected embedding dimension: 1536 for model: text-embedding-v3
2025-05-26 16:14:30 [WARNING] chroma_utils.py:60 - Unknown embedding model: text-embedding-v3, defaulting to 1536 dimensions
2025-05-26 16:14:30 [INFO] embedding_service.py:30 - Detected embedding dimension: 1536 for model: text-embedding-v3
2025-05-26 16:14:30 [INFO] train.py:200 - Initializing training with memory optimizations
2025-05-26 16:14:30 [INFO] train.py:204 - Applying memory optimizations to training configuration
2025-05-26 16:14:30 [INFO] train.py:234 - Loading model with automatic memory management from /app/resources/L2/base_models/Qwen2.5-1.5B-Instruct
2025-05-26 16:15:01 [INFO] train.py:286 - Enabling gradient checkpointing for memory efficiency
CUDA enabled by user configuration.
CUDA parameter received: True
CUDA_VISIBLE_DEVICES set to 0
CUDA_LAUNCH_BLOCKING set to 0 for better performance
Using training parameters:
Learning rate: 0.0001
Number of epochs: 3
Concurrency threads: 2
Data synthesis mode: medium
Use CUDA: True
Is chain of thought: False
Set thread environment variables to 2
Enabling BF16 half precision for non-Apple platform with CUDA
Environment configuration:
CUDA_VISIBLE_DEVICES: 0
PYTORCH_CUDA_ALLOC_CONF: max_split_size_mb:128
Using half precision: True
2025-05-27 00:34:24 [INFO] train.py:185 - Begin training...
2025-05-27 00:34:24 [INFO] train.py:185 - Begin training...
2025-05-27 00:34:25 [INFO] memory_manager.py:61 - Memory after cleanup: RAM: 7.05GB / 94.18GB, VRAM: 0.00GB / 23.99GB
2025-05-27 00:34:25 [INFO] train.py:195 - BEGIN release ollama models early
2025-05-27 00:34:25 [INFO] train.py:195 - BEGIN release ollama models early