Skip to content

cli: add option to connect to server via http(s) #21674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
pwilkin wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

cli: add option to connect to server via http(s) #21674

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
14d349e
cli: add option to connect to server via http(s)
pwilkin Apr 9, 2026
42ae57c
fix Mac failing build
pwilkin Apr 9, 2026
0f5cb2d
Rewrite to http-only.
pwilkin Apr 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions common/arg.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -591,9 +591,9 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
common_params_handle_model(params.vocoder.model, params.hf_token, params.offline);
}

// model is required (except for server)
// model is required (except for server, or when using --endpoint in CLI)
// TODO @ngxson : maybe show a list of available models in CLI in this case
if (params.model.path.empty() && ctx_arg.ex != LLAMA_EXAMPLE_SERVER && !skip_model_download && !params.usage && !params.completion) {
if (params.model.path.empty() && ctx_arg.ex != LLAMA_EXAMPLE_SERVER && !skip_model_download && !params.usage && !params.completion && params.endpoint.empty()) {
throw std::invalid_argument("error: --model is required\n");
}

Expand Down Expand Up @@ -1380,6 +1380,13 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.show_timings = value;
}
).set_examples({LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_SHOW_TIMINGS"));
add_opt(common_arg(
{"--endpoint"}, "URL",
Copy link
Copy Markdown
Contributor

@ngxson ngxson Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this should be a bit more specific, smth like --server-base

and in the future when server becomes a daemon, maybe with a fixed port like 8800, we can move --server-base to default mode and use the mentioned port

string_format("connect to a running llama-server at URL instead of loading a model locally (e.g. http://localhost:8080)"),
[](common_params & params, const std::string & value) {
params.endpoint = value;
}
).set_examples({LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_ENDPOINT"));
add_opt(common_arg(
{"-f", "--file"}, "FNAME",
"a file containing the prompt (default: none)",
Expand Down
4 changes: 4 additions & 0 deletions common/common.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,10 @@ struct common_params {

bool single_turn = false; // single turn chat conversation

// remote server endpoint for CLI (e.g. "http://localhost:8080")
// when set, CLI connects to a running server instead of loading a model
std::string endpoint = ""; // NOLINT

ggml_type cache_type_k = GGML_TYPE_F16; // KV cache data type for the K
ggml_type cache_type_v = GGML_TYPE_F16; // KV cache data type for the V

Expand Down
6 changes: 4 additions & 2 deletions tools/cli/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
set(TARGET llama-cli)
add_executable(${TARGET} cli.cpp)
target_link_libraries(${TARGET} PRIVATE server-context PUBLIC common ${CMAKE_THREAD_LIBS_INIT})
add_executable(${TARGET} cli.cpp cli-backend.cpp)
target_link_libraries(${TARGET} PRIVATE cpp-httplib PUBLIC common ${CMAKE_THREAD_LIBS_INIT})
target_compile_features(${TARGET} PRIVATE cxx_std_17)

include_directories(../server)
include_directories(../mtmd)
include_directories(../../vendor)

if(LLAMA_TOOLS_INSTALL)
install(TARGETS ${TARGET} RUNTIME)
Expand Down
Loading
Loading