|
1 | | -# Release Notes - v2.3.3 |
| 1 | +# Release Notes - v3.0.0 |
2 | 2 |
|
3 | | -> **Release Date:** December 31, 2025 |
| 3 | +> **Release Date:** January 2026 |
| 4 | +> **The HuggingFace-Native Release** |
4 | 5 |
|
5 | | -## 🎉 Community Features |
| 6 | +## Highlights |
6 | 7 |
|
7 | | -This release incorporates contributions and suggestions from the community: |
| 8 | +Version 3.0.0 is a major release that brings **full HuggingFace CLI compatibility**. Your downloads are now stored in the standard HuggingFace cache structure, making them instantly accessible to Transformers, Diffusers, and any other HuggingFace-based tools. |
8 | 9 |
|
9 | | -### ✨ New Features |
| 10 | +--- |
| 11 | + |
| 12 | +## New Features |
| 13 | + |
| 14 | +### HuggingFace Cache Structure (Default) |
10 | 15 |
|
11 | | -#### 🚫 Exclude Patterns (`--exclude`, `-E`) |
12 | | -Exclude specific files from downloads. Thanks to **@jeroenkroese** ([#41](https://github.com/bodaay/HuggingFaceModelDownloader/pull/41))! |
| 16 | +Downloads now go directly to `~/.cache/huggingface/hub/` - the same location used by `huggingface_hub`, Transformers, and other HuggingFace tools. |
13 | 17 |
|
14 | 18 | ```bash |
15 | | -# Exclude markdown and ONNX files |
16 | | -hfdownloader download TheBloke/Mistral-7B-GGUF -E .md,onnx |
| 19 | +# Download a model - it's immediately available to Transformers |
| 20 | +hfdownloader download microsoft/DialoGPT-medium |
17 | 21 |
|
18 | | -# Exclude full precision models |
19 | | -hfdownloader download owner/repo -E fp16,fp32 |
| 22 | +# Use it directly in Python - no copying needed! |
| 23 | +from transformers import AutoModel |
| 24 | +model = AutoModel.from_pretrained("microsoft/DialoGPT-medium") |
20 | 25 | ``` |
21 | 26 |
|
22 | | -#### 🌍 Custom Endpoint (`--endpoint`) |
23 | | -Use HuggingFace mirrors or enterprise endpoints. Thanks to **@windtail** ([#38](https://github.com/bodaay/HuggingFaceModelDownloader/pull/38))! |
| 27 | +### Dual-Layer Storage |
24 | 28 |
|
25 | | -```bash |
26 | | -# Use China mirror |
27 | | -hfdownloader download owner/repo --endpoint https://hf-mirror.com |
| 29 | +Get the best of both worlds: |
| 30 | +- **HuggingFace cache**: Content-addressable blobs for deduplication |
| 31 | +- **Human-readable symlinks**: Easy file browsing at `~/.cache/huggingface/hub/models--{owner}--{repo}/snapshots/{revision}/` |
28 | 32 |
|
29 | | -# Use enterprise endpoint |
30 | | -hfdownloader serve --endpoint https://your-enterprise.com/hf |
31 | | -``` |
| 33 | +### Multi-Revision Support |
32 | 34 |
|
33 | | -#### 🐳 Docker Support |
34 | | -Run hfdownloader in containers. Thanks to **@cdeving** ([#50](https://github.com/bodaay/HuggingFaceModelDownloader/pull/50))! |
| 35 | +Download specific branches, tags, or commits: |
35 | 36 |
|
36 | 37 | ```bash |
37 | | -# Build Docker image |
38 | | -docker build -t hfdownloader . |
39 | | - |
40 | | -# Run CLI in container |
41 | | -docker run --rm -v ./models:/data hfdownloader download TheBloke/Mistral-7B-GGUF -o /data |
| 38 | +# Download a specific branch |
| 39 | +hfdownloader download TheBloke/Mistral-7B-GGUF --revision main |
42 | 40 |
|
43 | | -# Run web server in container |
44 | | -docker run --rm -p 8080:8080 -v ./models:/data hfdownloader serve --models-dir /data/Models |
| 41 | +# Download a specific tag |
| 42 | +hfdownloader download owner/repo --revision v1.0.0 |
45 | 43 | ``` |
46 | 44 |
|
47 | | ---- |
| 45 | +### Model Analysis |
48 | 46 |
|
49 | | -## 📋 Changes |
| 47 | +Understand models before downloading with the new `analyze` command: |
50 | 48 |
|
51 | | -- Added `--exclude` / `-E` flag to CLI download command |
52 | | -- Added `--endpoint` flag to both download and serve commands |
53 | | -- Added `Excludes` field to Job struct |
54 | | -- Added `Endpoint` field to Settings struct |
55 | | -- Created `Dockerfile` with multi-stage build |
56 | | -- Updated API to support `excludes` in download requests |
57 | | -- Updated settings endpoint to include custom endpoint config |
58 | | -- **Web UI:** Added "Exclude" input field to Model and Dataset download forms |
59 | | -- **Web UI:** Updated version display to v2.3.1 |
| 49 | +```bash |
| 50 | +hfdownloader analyze microsoft/DialoGPT-medium |
| 51 | +``` |
60 | 52 |
|
61 | | ---- |
| 53 | +Shows: |
| 54 | +- Model architecture and framework |
| 55 | +- File types and sizes |
| 56 | +- Quantization formats available |
| 57 | +- Recommended filters for your use case |
62 | 58 |
|
63 | | -## 🙏 Acknowledgments |
| 59 | +### Enhanced Web UI |
64 | 60 |
|
65 | | -Special thanks to the community members whose PRs inspired these features: |
| 61 | +- **Revision picker** - Select branches/tags from a dropdown |
| 62 | +- **Model analysis** - Analyze before downloading |
| 63 | +- **Cache browser** - Explore your local HuggingFace cache |
| 64 | +- **Authentication** - Secure your web server with `--auth-user` and `--auth-pass` |
66 | 65 |
|
67 | | -| Contributor | PR | Feature | |
68 | | -|-------------|-----|---------| |
69 | | -| **[@jeroenkroese](https://github.com/jeroenkroese)** | [#41](https://github.com/bodaay/HuggingFaceModelDownloader/pull/41) | Exclude file patterns | |
70 | | -| **[@windtail](https://github.com/windtail)** | [#38](https://github.com/bodaay/HuggingFaceModelDownloader/pull/38) | Custom HuggingFace endpoint | |
71 | | -| **[@cdeving](https://github.com/cdeving)** | [#50](https://github.com/bodaay/HuggingFaceModelDownloader/pull/50) | Docker support | |
| 66 | +### Web Authentication |
72 | 67 |
|
73 | | -Also thanks to the community for bug reports and PRs that helped identify issues in v2.3.0: |
74 | | -- URL escaping fix (related to #60) |
75 | | -- TUI speed improvements (related to #59) |
76 | | -- API 400 fixes (related to #58) |
| 68 | +Secure your web server when exposing to networks: |
77 | 69 |
|
78 | | ---- |
79 | | - |
80 | | -# Release Notes - v2.3.0 |
| 70 | +```bash |
| 71 | +hfdownloader serve --auth-user admin --auth-pass secret |
| 72 | +``` |
81 | 73 |
|
82 | | -> **Release Date:** December 31, 2025 |
| 74 | +### Legacy Mode |
83 | 75 |
|
84 | | -## 🎉 Highlights |
| 76 | +Still need the old flat directory structure? Use `--legacy`: |
85 | 77 |
|
86 | | -This is a **major release** introducing a brand new **Web UI**, complete project restructuring, and numerous bug fixes. The project has been reorganized into a clean, modular architecture following Go best practices. |
| 78 | +```bash |
| 79 | +hfdownloader download TheBloke/Mistral-7B-GGUF --legacy -o ./models |
| 80 | +``` |
87 | 81 |
|
88 | 82 | --- |
89 | 83 |
|
90 | | -## ✨ New Features |
| 84 | +## Docker |
91 | 85 |
|
92 | | -### 🌐 Web Interface |
93 | | -- **Beautiful Terminal-Noir themed Web UI** for managing downloads |
94 | | -- Real-time progress updates via WebSocket |
95 | | -- Separate pages for downloading **Models** and **Datasets** |
96 | | -- Per-file progress bars with live status updates |
97 | | -- Settings management (connections, retries, verification mode) |
98 | | -- Job deduplication - prevents duplicate downloads of the same repo |
| 86 | +Docker images are now automatically published to GitHub Container Registry: |
99 | 87 |
|
100 | | -### 🚀 One-Liner Web Mode |
101 | | -Start the web UI instantly with: |
102 | 88 | ```bash |
103 | | -bash <(curl -sSL https://g.bodaay.io/hfd) -w |
104 | | -``` |
105 | | -Automatically opens your browser to `http://localhost:8080` |
106 | | - |
107 | | -### 🔧 New CLI Commands |
108 | | -- `hfdownloader serve` - Start the web server |
109 | | -- `hfdownloader version` - Show version information |
110 | | -- `hfdownloader config` - Manage configuration |
| 89 | +# Pull the image |
| 90 | +docker pull ghcr.io/bodaay/huggingfacemodeldownloader:3.0.0 |
111 | 91 |
|
112 | | -### 📦 Reusable Go Package |
113 | | -The downloader is now available as an importable package: |
114 | | -```go |
115 | | -import "github.com/bodaay/HuggingFaceModelDownloader/pkg/hfdownloader" |
| 92 | +# Run with HuggingFace cache mount |
| 93 | +docker run --rm -p 8080:8080 \ |
| 94 | + -v ~/.cache/huggingface:/home/hfdownloader/.cache/huggingface \ |
| 95 | + ghcr.io/bodaay/huggingfacemodeldownloader:3.0.0 serve |
116 | 96 | ``` |
117 | 97 |
|
118 | 98 | --- |
119 | 99 |
|
120 | | -## 🐛 Bug Fixes |
121 | | - |
122 | | -### Fixed: "error: tree API failed: 400 Bad Request" |
123 | | -- Repository paths with slashes (e.g., `Qwen/Qwen3-0.6B`) were being incorrectly URL-escaped |
124 | | -- Now correctly handles repo IDs without double-escaping the slash |
| 100 | +## Quick Start |
125 | 101 |
|
126 | | -### Fixed: TUI Speed/ETA Display Jumping Around |
127 | | -- Implemented **Exponential Moving Average (EMA)** for smooth speed calculations |
128 | | -- Added minimum time interval (50ms) before recalculating speed |
129 | | -- Both per-file and overall speeds are now stable and readable |
| 102 | +```bash |
| 103 | +# Analyze a model (no download) |
| 104 | +bash <(curl -sSL https://g.bodaay.io/hfd) analyze TheBloke/Mistral-7B-GGUF |
130 | 105 |
|
131 | | -### Fixed: TUI Total File Size Fluctuating |
132 | | -- File totals no longer get overwritten with incorrect values during progress updates |
133 | | -- Now only updates total if a valid value is provided |
| 106 | +# Download Q4_K_M quantization only |
| 107 | +bash <(curl -sSL https://g.bodaay.io/hfd) download TheBloke/Mistral-7B-GGUF:q4_k_m |
134 | 108 |
|
135 | | -### Fixed: Downloads Appearing Stuck |
136 | | -- Removed blocking HEAD requests during repository scanning |
137 | | -- Large repos (90+ files) now start downloading within seconds instead of minutes |
138 | | -- Assumed LFS files support range requests (they always do on HuggingFace) |
| 109 | +# Start web UI with authentication |
| 110 | +bash <(curl -sSL https://g.bodaay.io/hfd) serve --auth-user admin --auth-pass secret |
139 | 111 |
|
140 | | -### Fixed: Web UI Progress Not Updating |
141 | | -- Added `progressReader` wrapper for real-time progress during single-file downloads |
142 | | -- Progress events now use correct `Downloaded` field (cumulative bytes) |
143 | | -- UI throttled to 10fps to prevent DOM thrashing |
| 112 | +# Install permanently |
| 113 | +bash <(curl -sSL https://g.bodaay.io/hfd) -i |
| 114 | +``` |
144 | 115 |
|
145 | 116 | --- |
146 | 117 |
|
147 | | -## 🏗️ Architecture Changes |
| 118 | +## Migration from V2 |
148 | 119 |
|
149 | | -### Project Structure |
150 | | -The codebase has been completely reorganized: |
| 120 | +V3 uses a different storage structure by default. Your V2 downloads remain intact. |
151 | 121 |
|
| 122 | +**Option 1**: Keep using legacy mode for existing workflows |
| 123 | +```bash |
| 124 | +hfdownloader download owner/repo --legacy -o ./models |
152 | 125 | ``` |
153 | | -├── cmd/hfdownloader/ # CLI entry point |
154 | | -├── internal/ |
155 | | -│ ├── cli/ # CLI commands (Cobra) |
156 | | -│ ├── server/ # Web server & API |
157 | | -│ ├── tui/ # Terminal UI |
158 | | -│ └── assets/ # Embedded web assets |
159 | | -├── pkg/hfdownloader/ # Reusable download library |
160 | | -└── scripts/ # Installation scripts |
161 | | -``` |
162 | | - |
163 | | -### Security Improvements |
164 | | -- **Output path is server-controlled** - Cannot be changed via API or Web UI |
165 | | -- Separate directories for models (`./Models/`) and datasets (`./Datasets/`) |
166 | | -- Token is never logged or exposed in API responses |
167 | | - |
168 | | -### Testing |
169 | | -- Comprehensive unit tests for `JobManager`, API handlers, and WebSocket |
170 | | -- Integration tests for end-to-end download flows |
171 | | -- Test coverage for job deduplication and cancellation |
172 | 126 |
|
173 | | ---- |
174 | | - |
175 | | -## 📊 Performance Improvements |
176 | | - |
177 | | -| Improvement | Before | After | |
178 | | -|-------------|--------|-------| |
179 | | -| Large repo scan (90+ files) | 5+ minutes | ~2 seconds | |
180 | | -| Progress update frequency | 1 second | 200ms | |
181 | | -| Speed display stability | Jumpy/erratic | Smooth (EMA) | |
182 | | -| Web UI responsiveness | Laggy | Throttled 10fps | |
| 127 | +**Option 2**: Re-download to HuggingFace cache (recommended) |
| 128 | +```bash |
| 129 | +# New downloads go to ~/.cache/huggingface/hub/ by default |
| 130 | +hfdownloader download owner/repo |
| 131 | +``` |
183 | 132 |
|
184 | 133 | --- |
185 | 134 |
|
186 | | -## 🔄 Breaking Changes |
| 135 | +## Breaking Changes |
187 | 136 |
|
188 | | -- Main package moved from `hfdownloader/` to `pkg/hfdownloader/` |
189 | | -- CLI now uses Cobra commands instead of flags-only |
190 | | -- `main.go` replaced with `cmd/hfdownloader/main.go` |
191 | | -- Old `makefile` replaced with `build.sh` |
| 137 | +- **Default storage location changed**: Downloads now go to `~/.cache/huggingface/hub/` instead of `./Models/` |
| 138 | +- **Directory structure changed**: Uses HuggingFace blob/symlink structure instead of flat files |
| 139 | +- **`-o` flag behavior changed**: In V3 mode, sets `HF_HOME`; in legacy mode, sets direct output path |
192 | 140 |
|
193 | 141 | --- |
194 | 142 |
|
195 | | -## 📥 Installation |
| 143 | +## Full Changelog |
196 | 144 |
|
197 | | -### One-Liner (Recommended) |
198 | | -```bash |
199 | | -# Install to /usr/local/bin |
200 | | -bash <(curl -sSL https://g.bodaay.io/hfd) -i |
| 145 | +### New Features |
| 146 | +- HuggingFace cache structure as default storage |
| 147 | +- `analyze` command for model inspection |
| 148 | +- `--revision` flag for multi-branch downloads |
| 149 | +- Web UI revision picker |
| 150 | +- Web UI cache browser |
| 151 | +- Web UI authentication (`--auth-user`, `--auth-pass`) |
| 152 | +- GitHub Actions for automated releases |
| 153 | +- GitHub Container Registry for Docker images |
201 | 154 |
|
202 | | -# Start Web UI |
203 | | -bash <(curl -sSL https://g.bodaay.io/hfd) -w |
| 155 | +### Improvements |
| 156 | +- Dual-layer storage (blobs + symlinks) |
| 157 | +- Better progress display |
| 158 | +- Improved error messages |
204 | 159 |
|
205 | | -# Download a model directly |
206 | | -bash <(curl -sSL https://g.bodaay.io/hfd) download TheBloke/Mistral-7B-GGUF |
207 | | -``` |
208 | | - |
209 | | -### From Source |
210 | | -```bash |
211 | | -git clone https://github.com/bodaay/HuggingFaceModelDownloader |
212 | | -cd HuggingFaceModelDownloader |
213 | | -go build -o hfdownloader ./cmd/hfdownloader |
214 | | -``` |
| 160 | +### Legacy Support |
| 161 | +- `--legacy` flag for V2-style flat directory structure |
| 162 | +- Existing V2 workflows continue to work |
215 | 163 |
|
216 | 164 | --- |
217 | 165 |
|
218 | | -## 📋 Full Changelog |
219 | | - |
220 | | -**New Files:** |
221 | | -- `cmd/hfdownloader/main.go` - New CLI entry point |
222 | | -- `internal/server/*` - Complete web server implementation |
223 | | -- `internal/assets/*` - Embedded web UI (HTML/CSS/JS) |
224 | | -- `pkg/hfdownloader/*` - Modular download library |
225 | | -- `build.sh` - Cross-platform build script |
226 | | - |
227 | | -**Modified:** |
228 | | -- `scripts/gist_gethfd.sh` - Added `-w` flag for web mode |
229 | | -- `README.md` - Updated documentation with web UI info |
230 | | -- `go.mod` - Added new dependencies (Cobra, Gorilla WebSocket) |
231 | | - |
232 | | -**Removed:** |
233 | | -- `hfdownloader/` - Moved to `pkg/hfdownloader/` |
234 | | -- `main.go` - Replaced by `cmd/hfdownloader/main.go` |
235 | | -- `makefile` - Replaced by `build.sh` |
| 166 | +--- |
236 | 167 |
|
| 168 | +**Full Changelog**: https://github.com/bodaay/HuggingFaceModelDownloader/compare/v2.3.3...v3.0.0 |
0 commit comments