|
| 1 | +# Binary Compression Quick Reference |
| 2 | + |
| 3 | +## TL;DR |
| 4 | + |
| 5 | +```bash |
| 6 | +# Compression is ENABLED BY DEFAULT (it's called "smol" for a reason!) |
| 7 | +node scripts/build.mjs |
| 8 | + |
| 9 | +# Disable compression if needed |
| 10 | +COMPRESS_BINARY=0 node scripts/build.mjs |
| 11 | + |
| 12 | +# Output location |
| 13 | +ls -lh build/out/Compressed/ |
| 14 | +# node (~10-12 MB compressed binary) |
| 15 | +# socket_macho_decompress (~86 KB decompression tool) |
| 16 | + |
| 17 | +# Test it |
| 18 | +cd build/out/Compressed |
| 19 | +./socket_macho_decompress ./node --version |
| 20 | +``` |
| 21 | + |
| 22 | +## Why Use Compression? |
| 23 | + |
| 24 | +**Size reduction:** 23-27 MB → 10-12 MB (**70% smaller**) |
| 25 | +**Better than UPX:** 75-79% compression vs UPX's 50-60% |
| 26 | +**macOS compatible:** Works with code signing (UPX breaks it) |
| 27 | +**No AV flags:** Uses native platform APIs, zero false positives |
| 28 | + |
| 29 | +## Platform Support |
| 30 | + |
| 31 | +| Platform | Algorithm | Size Reduction | Decompressor | |
| 32 | +|----------|-----------|----------------|--------------| |
| 33 | +| **macOS** | LZFSE | ~30% | socket_macho_decompress (86 KB) | |
| 34 | +| **macOS** | LZMA | ~34% | socket_macho_decompress (86 KB) | |
| 35 | +| **Linux** | LZMA | ~75-77% | socket_elf_decompress (~90 KB) | |
| 36 | +| **Windows** | LZMS | ~73% | socket_pe_decompress.exe (~95 KB) | |
| 37 | + |
| 38 | +## Quick Start |
| 39 | + |
| 40 | +### 1. Build Compression Tools (First Time Only) |
| 41 | + |
| 42 | +```bash |
| 43 | +cd packages/node-smol-builder/additions/tools |
| 44 | +make all |
| 45 | + |
| 46 | +# Verify |
| 47 | +ls -lh socket_*compress* |
| 48 | +``` |
| 49 | + |
| 50 | +### 2. Build Node.js with Compression |
| 51 | + |
| 52 | +```bash |
| 53 | +cd packages/node-smol-builder |
| 54 | +COMPRESS_BINARY=1 node scripts/build.mjs |
| 55 | +``` |
| 56 | + |
| 57 | +### 3. Test Compressed Binary |
| 58 | + |
| 59 | +```bash |
| 60 | +cd build/out/Compressed |
| 61 | + |
| 62 | +# Test directly |
| 63 | +./socket_macho_decompress ./node --version |
| 64 | +# Output: v24.10.0 |
| 65 | + |
| 66 | +# Test with script |
| 67 | +./socket_macho_decompress ./node -e "console.log('Hello')" |
| 68 | +# Output: Hello |
| 69 | +``` |
| 70 | + |
| 71 | +## Build Output Structure |
| 72 | + |
| 73 | +``` |
| 74 | +build/out/ |
| 75 | +├── Release/ # Unstripped binary (44 MB) |
| 76 | +├── Stripped/ # Stripped binary (23-27 MB) |
| 77 | +├── Signed/ # Stripped + signed (23-27 MB, macOS only) |
| 78 | +├── Final/ # Final uncompressed (23-27 MB) |
| 79 | +├── Compressed/ # ✨ Compressed output (COMPRESS_BINARY=1) |
| 80 | +│ ├── node # Compressed binary (10-12 MB) |
| 81 | +│ └── socket_*_decompress # Decompression tool (~90 KB) |
| 82 | +├── Sea/ # For SEA builds |
| 83 | +└── Distribution/ # Distribution copy |
| 84 | +``` |
| 85 | + |
| 86 | +## Distribution |
| 87 | + |
| 88 | +### Option 1: Distribute Compressed (Recommended) |
| 89 | + |
| 90 | +```bash |
| 91 | +cd build/out/Compressed |
| 92 | +tar -czf socket-node-macos-arm64.tar.gz node socket_macho_decompress |
| 93 | + |
| 94 | +# Users extract and run: |
| 95 | +tar -xzf socket-node-macos-arm64.tar.gz |
| 96 | +./socket_macho_decompress ./node --version |
| 97 | +``` |
| 98 | + |
| 99 | +**Pros:** |
| 100 | +- 70% smaller download |
| 101 | +- Better than UPX compression |
| 102 | +- Works with macOS code signing |
| 103 | + |
| 104 | +**Cons:** |
| 105 | +- Requires bundling decompression tool |
| 106 | +- ~100-500ms startup overhead (first run) |
| 107 | + |
| 108 | +### Option 2: Distribute Uncompressed |
| 109 | + |
| 110 | +```bash |
| 111 | +cd build/out/Final |
| 112 | +tar -czf socket-node-macos-arm64.tar.gz node |
| 113 | + |
| 114 | +# Users extract and run: |
| 115 | +tar -xzf socket-node-macos-arm64.tar.gz |
| 116 | +./node --version |
| 117 | +``` |
| 118 | + |
| 119 | +**Pros:** |
| 120 | +- No decompression overhead |
| 121 | +- Simpler distribution |
| 122 | + |
| 123 | +**Cons:** |
| 124 | +- 2-3x larger download |
| 125 | +- Still need to ship 23-27 MB binary |
| 126 | + |
| 127 | +## Configuration |
| 128 | + |
| 129 | +### Environment Variables |
| 130 | + |
| 131 | +```bash |
| 132 | +# Disable compression (opt-out) |
| 133 | +COMPRESS_BINARY=0 |
| 134 | + |
| 135 | +# Values: "0", "false" to disable (case-sensitive) |
| 136 | +# Default: compression ENABLED (smol = small!) |
| 137 | +``` |
| 138 | + |
| 139 | +### Compression Algorithms |
| 140 | + |
| 141 | +**Automatically selected based on platform:** |
| 142 | +- **macOS:** LZFSE (default) or LZMA |
| 143 | +- **Linux:** LZMA |
| 144 | +- **Windows:** LZMS |
| 145 | + |
| 146 | +To override (advanced): |
| 147 | +```bash |
| 148 | +# Edit build.mjs line 1466 |
| 149 | +const compressionQuality = 'lzma' # Options: lzfse, lzma, lz4, zlib (macOS) |
| 150 | +``` |
| 151 | + |
| 152 | +## Performance |
| 153 | + |
| 154 | +### Decompression Overhead |
| 155 | + |
| 156 | +**First run (cold cache):** |
| 157 | +- macOS LZFSE: ~100-200ms |
| 158 | +- macOS LZMA: ~300-500ms |
| 159 | +- Linux LZMA: ~200-400ms |
| 160 | +- Windows LZMS: ~250-450ms |
| 161 | + |
| 162 | +**Subsequent runs (warm cache):** |
| 163 | +- macOS: ~10-20ms |
| 164 | +- Linux: ~15-30ms |
| 165 | +- Windows: ~20-40ms |
| 166 | + |
| 167 | +### Runtime Performance |
| 168 | + |
| 169 | +**Zero impact** - Same performance as uncompressed binary after decompression. |
| 170 | + |
| 171 | +## Code Signing (macOS) |
| 172 | + |
| 173 | +Compression preserves code signatures: |
| 174 | + |
| 175 | +```bash |
| 176 | +# Check outer signature (compressed wrapper) |
| 177 | +codesign -dv build/out/Compressed/node |
| 178 | +# Shows: adhoc signature on compressed binary |
| 179 | + |
| 180 | +# Inner signature (original binary) preserved inside compression |
| 181 | +# Verified by decompressor at runtime |
| 182 | +``` |
| 183 | + |
| 184 | +**Signing flow:** |
| 185 | +1. Build → Strip → **Sign original** → Compress → **Re-sign compressed** |
| 186 | +2. Both signatures are valid and preserved |
| 187 | +3. Gatekeeper checks outer signature |
| 188 | +4. Decompressor verifies inner signature |
| 189 | + |
| 190 | +## Troubleshooting |
| 191 | + |
| 192 | +### "Decompression tool not found" |
| 193 | + |
| 194 | +```bash |
| 195 | +# Build tools first |
| 196 | +cd packages/node-smol-builder/additions/tools |
| 197 | +make all |
| 198 | + |
| 199 | +# macOS prerequisites (built-in): |
| 200 | +# - Xcode Command Line Tools |
| 201 | +# - Apple Compression framework |
| 202 | + |
| 203 | +# Linux prerequisites: |
| 204 | +sudo apt-get install liblzma-dev # Debian/Ubuntu |
| 205 | +sudo dnf install xz-devel # Fedora/RHEL |
| 206 | + |
| 207 | +# Windows prerequisites: |
| 208 | +choco install mingw |
| 209 | +``` |
| 210 | + |
| 211 | +### "Binary larger than expected" |
| 212 | + |
| 213 | +```bash |
| 214 | +# Check if compression was applied |
| 215 | +ls -lh build/out/Compressed/node |
| 216 | + |
| 217 | +# Expected: |
| 218 | +# macOS: ~11-12 MB |
| 219 | +# Linux: ~10-11 MB |
| 220 | +# Windows: ~12-13 MB |
| 221 | + |
| 222 | +# If larger, verify COMPRESS_BINARY=1 was set |
| 223 | +echo $COMPRESS_BINARY |
| 224 | +``` |
| 225 | + |
| 226 | +### "Command failed: compress-binary.mjs" |
| 227 | + |
| 228 | +```bash |
| 229 | +# Check compression tools are built |
| 230 | +ls -lh additions/tools/socket_*_compress* |
| 231 | + |
| 232 | +# If missing, rebuild: |
| 233 | +cd additions/tools |
| 234 | +make clean |
| 235 | +make all |
| 236 | +``` |
| 237 | + |
| 238 | +### "codesign: code object is not signed at all" |
| 239 | + |
| 240 | +This is expected for non-macOS or non-ARM64 builds. Code signing only applies to macOS ARM64. |
| 241 | + |
| 242 | +## Integration with Socket CLI |
| 243 | + |
| 244 | +### For pkg Builds |
| 245 | + |
| 246 | +```bash |
| 247 | +# 1. Build compressed Node |
| 248 | +COMPRESS_BINARY=1 node packages/node-smol-builder/scripts/build.mjs |
| 249 | + |
| 250 | +# 2. Option A: Use compressed binary with pkg |
| 251 | +# (Copy to pkg cache - pkg will use compressed version internally) |
| 252 | +cp build/out/Compressed/node ~/.pkg-cache/v3.5/built-v24.10.0-darwin-arm64 |
| 253 | + |
| 254 | +# 3. Build Socket CLI |
| 255 | +pnpm exec pkg . |
| 256 | + |
| 257 | +# Result: Socket CLI uses compressed Node.js (~70% smaller) |
| 258 | +``` |
| 259 | + |
| 260 | +### For Direct Distribution |
| 261 | + |
| 262 | +```bash |
| 263 | +# Distribute decompressor alongside Socket CLI |
| 264 | +socket-cli-macos-arm64/ |
| 265 | +├── socket # Socket CLI executable |
| 266 | +├── socket_macho_decompress # Decompressor |
| 267 | +└── README.md |
| 268 | + |
| 269 | +# Users run via wrapper |
| 270 | +./socket_macho_decompress ./socket --version |
| 271 | +``` |
| 272 | + |
| 273 | +## Documentation |
| 274 | + |
| 275 | +### Comprehensive Guides |
| 276 | + |
| 277 | +- **[docs/binary-compression-distribution.md](./docs/binary-compression-distribution.md)** - Complete architecture and distribution strategy |
| 278 | +- **[QUICKSTART-COMPRESSION.md](./QUICKSTART-COMPRESSION.md)** - Original compression quick start |
| 279 | +- **[TEST-RESULTS.md](./TEST-RESULTS.md)** - Compression benchmarks and comparisons |
| 280 | + |
| 281 | +### Quick Links |
| 282 | + |
| 283 | +- **Build script:** `scripts/build.mjs` (compression at line 1449-1560) |
| 284 | +- **Compression script:** `scripts/compress-binary.mjs` |
| 285 | +- **Decompression script:** `scripts/decompress-binary.mjs` |
| 286 | +- **Tools source:** `additions/tools/socket_*_compress*.{cc,c}` |
| 287 | + |
| 288 | +## Comparison to Alternatives |
| 289 | + |
| 290 | +### vs UPX |
| 291 | + |
| 292 | +| Feature | UPX | Socket Compression | |
| 293 | +|---------|-----|-------------------| |
| 294 | +| Compression | 50-60% | **75-79%** ⭐ | |
| 295 | +| macOS Code Signing | ❌ Breaks | ✅ Works | |
| 296 | +| AV False Positives | ❌ 15-30% | ✅ 0% | |
| 297 | +| Gatekeeper | ❌ Blocked | ✅ No warnings | |
| 298 | +| Distribution | Self-extracting | External decompressor | |
| 299 | + |
| 300 | +### vs No Compression |
| 301 | + |
| 302 | +| Metric | Uncompressed | Compressed | |
| 303 | +|--------|--------------|------------| |
| 304 | +| **Download Size** | 23-27 MB | **10-12 MB** | |
| 305 | +| **Startup Time** | 0ms | 100-500ms (first run) | |
| 306 | +| **Runtime Performance** | ✅ | ✅ (identical) | |
| 307 | +| **Distribution Complexity** | Simple | +Decompressor (~90 KB) | |
| 308 | + |
| 309 | +## FAQ |
| 310 | + |
| 311 | +**Q: Do I need to compress?** |
| 312 | +A: Optional. Recommended for production releases to reduce download size. |
| 313 | + |
| 314 | +**Q: Does compression affect performance?** |
| 315 | +A: Only startup time (~100-500ms first run, ~10-40ms cached). No runtime impact. |
| 316 | + |
| 317 | +**Q: Will this work with pkg?** |
| 318 | +A: Yes! Copy compressed binary to pkg cache, pkg will use it. |
| 319 | + |
| 320 | +**Q: Is this safe for production?** |
| 321 | +A: Yes. Uses native platform APIs, fully code-signed, zero AV flags. |
| 322 | + |
| 323 | +**Q: Can I skip the decompression tool in distribution?** |
| 324 | +A: No. Users need the decompressor to run the compressed binary. Bundle it (~90 KB overhead). |
| 325 | + |
| 326 | +**Q: Why not self-extracting?** |
| 327 | +A: Self-extracting archives write to disk (~1-2 MB overhead, slower startup). Our approach decompresses to memory (faster, no disk I/O). |
| 328 | + |
| 329 | +## Examples |
| 330 | + |
| 331 | +### Test Script |
| 332 | + |
| 333 | +```bash |
| 334 | +#!/bin/bash |
| 335 | +# Test compressed Node.js binary |
| 336 | + |
| 337 | +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
| 338 | +DECOMPRESS="$SCRIPT_DIR/socket_macho_decompress" |
| 339 | +NODE_BIN="$SCRIPT_DIR/node" |
| 340 | + |
| 341 | +echo "Testing compressed Node.js binary..." |
| 342 | +"$DECOMPRESS" "$NODE_BIN" --version |
| 343 | +"$DECOMPRESS" "$NODE_BIN" -e "console.log('✓ Compression working')" |
| 344 | +``` |
| 345 | + |
| 346 | +### Wrapper for Users |
| 347 | + |
| 348 | +```bash |
| 349 | +#!/bin/bash |
| 350 | +# Socket CLI launcher with decompression |
| 351 | + |
| 352 | +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
| 353 | +exec "$SCRIPT_DIR/socket_macho_decompress" "$SCRIPT_DIR/socket" "$@" |
| 354 | +``` |
| 355 | + |
| 356 | +### Size Comparison Script |
| 357 | + |
| 358 | +```bash |
| 359 | +#!/bin/bash |
| 360 | +# Compare sizes |
| 361 | + |
| 362 | +echo "Size comparison:" |
| 363 | +echo " Uncompressed: $(du -h build/out/Final/node | cut -f1)" |
| 364 | +echo " Compressed: $(du -h build/out/Compressed/node | cut -f1)" |
| 365 | +echo " Decompressor: $(du -h build/out/Compressed/socket_macho_decompress | cut -f1)" |
| 366 | +echo " Total: $(du -ch build/out/Compressed/* | tail -1 | cut -f1)" |
| 367 | +``` |
| 368 | + |
| 369 | +## Support |
| 370 | + |
| 371 | +**Issues?** Check: |
| 372 | +1. Compression tools built: `ls additions/tools/socket_*_compress*` |
| 373 | +2. Environment variable set: `echo $COMPRESS_BINARY` |
| 374 | +3. Platform supported: macOS, Linux, or Windows |
| 375 | +4. Logs in build output for errors |
| 376 | + |
| 377 | +**Questions?** See full documentation in `docs/binary-compression-distribution.md` |
0 commit comments