Skip to content

Fix Dockerfile for successful builds, improve /chat response quality, and add dev workflow tooling #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sriharip123 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Fix Dockerfile for successful builds, improve /chat response quality, and add dev workflow tooling #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
81eb61b
fix: Dockerfile build issues and add docker-compose + Postman collection
sriharip123 Mar 17, 2026
d407312
fix: use /v1/chat/completions with LLaMA 3 chat template for clean re…
sriharip123 Mar 17, 2026
33e3d31
feat: add volume mounts and uvicorn --reload for hot-reloading code c…
sriharip123 Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,23 +10,29 @@ RUN git clone --recursive https://github.com/microsoft/BitNet.git /tmp/BitNet &&
rm -rf /tmp/BitNet

# Install dependencies
RUN apt-get update && apt-get install -y \
RUN apt-get update --fix-missing && apt-get install -y --no-install-recommends \
wget \
lsb-release \
software-properties-common \
gnupg \
cmake \
clang \
&& bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" \
ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

# Patch const-correctness error in BitNet source (clang is stricter than gcc)
RUN sed -i 's/int8_t \* y_col = y + col \* by;/const int8_t * y_col = y + col * by;/' /code/src/ggml-bitnet-mad.cpp

# Install Python dependencies
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt && \
pip install "fastapi[standard]" "uvicorn[standard]" httpx fastapi-mcp psutil

# (Optional) Run your setup_env.py if needed
RUN python /code/setup_env.py -md /code/models/BitNet-b1.58-2B-4T -q i2_s
# Download pre-built GGUF model (skips the broken HF-to-GGUF conversion)
# Use the exact model name "BitNet-b1.58-2B-4T" so setup_env.py recognizes it
RUN huggingface-cli download microsoft/bitnet-b1.58-2B-4T-gguf --local-dir /code/models/BitNet-b1.58-2B-4T

# Run setup (compile + codegen, model already has the gguf so conversion is skipped)
RUN python /code/setup_env.py -md /code/models/BitNet-b1.58-2B-4T -q i2_s 2>&1 \
|| (cat /code/logs/*.log 2>/dev/null; exit 1)

EXPOSE 8080

Expand Down
Loading