Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Commit 7e1dd01

Browse files
author
Gabrielle Ong
authored
Merge branch 'dev' into j/add-docs-cors
2 parents ed39e16 + 24bebed commit 7e1dd01

File tree

102 files changed

+4296
-990
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

102 files changed

+4296
-990
lines changed

‎.github/workflows/beta-build.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ jobs:
8989
new_version: ${{ needs.get-update-version.outputs.new_version }}
9090
runs-on: ubuntu-latest
9191
cmake-extra-flags: "-DCORTEX_VARIANT=prod"
92-
tags: "menloltd/cortex:${{ needs.get-update-version.outputs.new_version }}"
92+
tags: "menloltd/cortex:beta-${{ needs.get-update-version.outputs.new_version }}"
9393

9494
update_release:
9595
needs: [get-update-version, create-draft-release, build-macos, build-windows-x64, build-linux-x64]
@@ -140,4 +140,5 @@ jobs:
140140
- Network Installer: https://github.com/janhq/cortex.cpp/releases/download/v${{ env.VERSION }}/cortex-${{ env.VERSION }}-linux-amd64-network-installer.deb
141141
- Local Installer: https://github.com/janhq/cortex.cpp/releases/download/v${{ env.VERSION }}/cortex-${{ env.VERSION }}-linux-amd64-local-installer.deb
142142
- Binary: https://github.com/janhq/cortex.cpp/releases/download/v${{ env.VERSION }}/cortex-${{ env.VERSION }}-linux-amd64.tar.gz
143+
- Docker: menloltd/cortex:beta-${{ env.VERSION }}
143144
- Github Release: https://github.com/janhq/cortex.cpp/releases/tag/v${{ env.VERSION }}

‎.github/workflows/cortex-cpp-quality-gate.yml‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,6 @@ jobs:
9999
100100
- name: Run setup config
101101
run: |
102-
rm ~/.cortexrc
103102
cd engine
104103
echo "huggingFaceToken: ${{ secrets.HUGGINGFACE_TOKEN_READ }}" > ~/.cortexrc
105104
echo "gitHubToken: ${{ secrets.PAT_SERVICE_ACCOUNT }}" >> ~/.cortexrc
@@ -115,7 +114,6 @@ jobs:
115114

116115
- name: Run setup config
117116
run: |
118-
rm ~/.cortexrc
119117
cd engine
120118
echo "huggingFaceToken: ${{ secrets.HUGGINGFACE_TOKEN_READ }}" > ~/.cortexrc
121119
echo "gitHubToken: ${{ secrets.PAT_SERVICE_ACCOUNT }}" >> ~/.cortexrc
@@ -189,3 +187,41 @@ jobs:
189187
AWS_ACCESS_KEY_ID: "${{ secrets.MINIO_ACCESS_KEY_ID }}"
190188
AWS_SECRET_ACCESS_KEY: "${{ secrets.MINIO_SECRET_ACCESS_KEY }}"
191189
AWS_DEFAULT_REGION: "${{ secrets.MINIO_REGION }}"
190+
191+
build-docker-and-test:
192+
runs-on: ubuntu-latest
193+
steps:
194+
- name: Getting the repo
195+
uses: actions/checkout@v3
196+
with:
197+
submodules: 'recursive'
198+
199+
- name: Set up QEMU
200+
uses: docker/setup-qemu-action@v3
201+
202+
- name: Set up Docker Buildx
203+
uses: docker/setup-buildx-action@v3
204+
205+
- name: Run Docker
206+
run: |
207+
docker build -t menloltd/cortex:test -f docker/Dockerfile .
208+
docker run -it -d -p 3928:39281 --name cortex menloltd/cortex:test
209+
210+
- name: use python
211+
uses: actions/setup-python@v5
212+
with:
213+
python-version: "3.10"
214+
215+
- name: Run e2e tests
216+
run: |
217+
cd engine
218+
python -m pip install --upgrade pip
219+
python -m pip install -r e2e-test/requirements.txt
220+
pytest e2e-test/test_api_docker.py
221+
222+
- name: Run Docker
223+
continue-on-error: true
224+
if: always()
225+
run: |
226+
docker stop cortex
227+
docker rm cortex

‎.github/workflows/template-noti-discord.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,5 +42,5 @@ jobs:
4242
- Network Installer: https://delta.jan.ai/cortex/v${{ env.VERSION }}/linux-amd64/cortex-${{ env.VERSION }}-linux-amd64-network-installer.deb
4343
- Local Installer: https://delta.jan.ai/cortex/v${{ env.VERSION }}/linux-amd64/cortex-${{ env.VERSION }}-linux-amd64-local-installer.deb
4444
- Binary: https://delta.jan.ai/cortex/v${{ env.VERSION }}/linux-amd64/cortex-nightly.tar.gz
45-
- Docker: menloltd/cortex:${{ env.VERSION }}
45+
- Docker: menloltd/cortex:nightly-${{ env.VERSION }}
4646
- Github action run: https://github.com/janhq/cortex.cpp/actions/runs/${{ env.RUNNER_ID }}

‎docker/entrypoint.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,13 @@
22

33
# Install cortex.llamacpp engine
44

5+
echo "apiServerHost: 0.0.0.0" > /root/.cortexrc
6+
echo "enableCors: true" >> /root/.cortexrc
7+
58
cortex engines install llama-cpp -s /opt/cortex.llamacpp
6-
cortex -v
79

810
# Start the cortex server
911

10-
sed -i 's/apiServerHost: 127.0.0.1/apiServerHost: 0.0.0.0/' /root/.cortexrc
11-
1212
cortex start
1313

1414
# Keep the container running by tailing the log files

‎docs/docs/capabilities/embeddings.md‎

Lines changed: 99 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,103 @@
11
---
22
title: Embeddings
33
---
4-
54
:::info
6-
🚧 Cortex is currently under development, and this page is a stub for future development.
7-
:::
5+
🚧 Cortex is currently under development, and this page is a stub for future development.
6+
:::
7+
8+
cortex.cpp now support embeddings endpoint with fully OpenAI compatible.
9+
10+
For embeddings API usage please refer to [API references](/api-reference#tag/chat/POST/v1/embeddings). This tutorial show you how to use embeddings in cortex with openai python SDK.
11+
12+
## Embedding with openai compatible
13+
14+
### 1. Start server and run model
15+
16+
```
17+
cortex run llama3.1:8b-gguf-q4-km
18+
```
19+
20+
### 2. Create script `embeddings.py` with this content
21+
22+
```
23+
from datetime import datetime
24+
from openai import OpenAI
25+
from pydantic import BaseModel
26+
ENDPOINT = "http://localhost:39281/v1"
27+
MODEL = "llama3.1:8bb-gguf-q4-km"
28+
client = OpenAI(
29+
base_url=ENDPOINT,
30+
api_key="not-needed"
31+
)
32+
```
33+
34+
### 3. Create embeddings
35+
36+
```
37+
response = client.embeddings.create(input = "embedding", model=MODEL, encoding_format="base64")
38+
print(response)
39+
```
40+
41+
The reponse will be like this
42+
43+
```
44+
CreateEmbeddingResponse(
45+
data=[
46+
Embedding(
47+
embedding='hjuAPOD8TryuPU8...',
48+
index=0,
49+
object='embedding'
50+
)
51+
],
52+
model='meta-llama3.1-8b-instruct',
53+
object='list',
54+
usage=Usage(
55+
prompt_tokens=2,
56+
total_tokens=2
57+
)
58+
)
59+
```
60+
61+
62+
The output embeddings is encoded as base64 string. Default the model will output the embeddings in float mode.
63+
64+
```
65+
response = client.embeddings.create(input = "embedding", model=MODEL)
66+
print(response)
67+
```
68+
69+
Result will be
70+
71+
```
72+
CreateEmbeddingResponse(
73+
data=[
74+
Embedding(
75+
embedding=[0.1, 0.3, 0.4 ....],
76+
index=0,
77+
object='embedding'
78+
)
79+
],
80+
model='meta-llama3.1-8b-instruct',
81+
object='list',
82+
usage=Usage(
83+
prompt_tokens=2,
84+
total_tokens=2
85+
)
86+
)
87+
```
88+
89+
Cortex also supports all input types as [OpenAI](https://platform.openai.com/docs/api-reference/embeddings/create#embeddings-create-input).
90+
91+
```sh
92+
# input as string
93+
response = client.embeddings.create(input = "embedding", model=MODEL)
94+
95+
# input as array of string
96+
response = client.embeddings.create(input = ["embedding"], model=MODEL)
97+
98+
# input as array of tokens
99+
response = client.embeddings.create(input = [12,44,123], model=MODEL)
100+
101+
# input as array of arrays contain tokens
102+
response = client.embeddings.create(input = [[912,312,54],[12,433,1241]], model=MODEL)
103+
```

docs/docs/capabilities/hardware/index.md renamed to docs/docs/capabilities/hardware/index.mdx

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,13 @@
11
---
22
title: Hardware Awareness
3-
draft: True
3+
description: The Hardware Awareness section overview
44
---
55

6+
:::warning
7+
🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.
8+
:::
9+
10+
611
# Hardware Awareness
712

813
Cortex is designed to be hardware aware, meaning it can detect your hardware configuration and automatically set parameters to optimize compatibility and performance, and avoid hardware-related errors.

‎docs/docs/capabilities/models/sources/index.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -272,7 +272,7 @@ curl --request POST \
272272
Clients can abort a downloading task using the task ID. Below is a sample `curl` command to abort a download task:
273273

274274
```sh
275-
curl --location --request DELETE 'http://127.0.0.1:3928/models/pull' \
275+
curl --location --request DELETE 'http://127.0.0.1:39281/v1/models/pull' \
276276
--header 'Content-Type: application/json' \
277277
--data '{
278278
"taskId": "tinyllama:1b-gguf-q2-k"

‎docs/docs/cli/hardware/index.mdx‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
---
2+
title: Cortex Hardware
3+
---
4+
5+
import Tabs from "@theme/Tabs";
6+
import TabItem from "@theme/TabItem";
7+
8+
# `cortex hardware`
9+
10+
This command allows you manage and monitor hardware resources.
11+
12+
13+
**Usage**:
14+
:::info
15+
You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`.
16+
:::
17+
<Tabs>
18+
<TabItem value="MacOs/Linux" label="MacOs/Linux">
19+
```sh
20+
cortex hardware [options] [subcommand]
21+
```
22+
</TabItem>
23+
<TabItem value="Windows" label="Windows">
24+
```sh
25+
cortex.exe hardware [options]
26+
27+
```
28+
</TabItem>
29+
</Tabs>
30+
31+
**Options**:
32+
33+
| Option | Description | Required | Default value | Example |
34+
|-------------------|-------------------------------------------------------|----------|---------------|-----------------|
35+
| `-h`, `--help` | Display help information for the command. | No | - | `-h` |
36+
37+
---
38+
# Subcommands:
39+
40+
## `cortex hardware list`
41+
:::info
42+
This CLI command calls the following API endpoint:
43+
- [List Model](/api-reference#tag/hardware/get/v1/hardware)
44+
:::
45+
This command lists all the hardware resources.
46+
47+
**Usage**:
48+
<Tabs>
49+
<TabItem value="MacOs/Linux" label="MacOs/Linux">
50+
```sh
51+
cortex hardware list [options]
52+
```
53+
</TabItem>
54+
<TabItem value="Windows" label="Windows">
55+
```sh
56+
cortex.exe hardware list [options]
57+
```
58+
</TabItem>
59+
</Tabs>
60+
61+
For example, it returns the following:
62+
```bash
63+
OS Information:
64+
+---+---------------------------+--------------------+
65+
| # | Version | Name |
66+
+---+---------------------------+--------------------+
67+
| 1 | 24.04.1 LTS (Noble Numbat)| Ubuntu 24.04.1 LTS |
68+
+---+---------------------------+--------------------+
69+
```
70+
71+
**Options**:
72+
73+
| Option | Description | Required | Default value | Example |
74+
|---------------------------|----------------------------------------------------|----------|---------------|----------------------|
75+
| `-h`, `--help` | Display help for command. | No | - | `-h` |
76+
|`--cpu` | Display CPU information | No | - | `--cpu` |
77+
|`--os` | Display OS information | No | - | `--os` |
78+
|`--ram` | Display RAM information | No | - | `--ram` |
79+
|`--storage` | Display Storage information | No | - | `--storage` |
80+
|`--gpu` | Display GPU information | No | - | `--gpu` |
81+
|`--power` | Display Power information | No | - | `--power` |
82+
|`--monitors` | Display Monitors information | No | - | `--monitors` |
83+
84+
## `cortex hardware activate`
85+
86+
::info
87+
This CLI command calls the following API endpoint:
88+
- [List Model](/api-reference#tag/hardware/post/v1/hardware/activate)
89+
:::
90+
This command activates the Cortex's hardware, currently support only GPUs.
91+
92+
**Usage**:
93+
<Tabs>
94+
<TabItem value="MacOs/Linux" label="MacOs/Linux">
95+
```sh
96+
cortex hardware activate [options]
97+
```
98+
</TabItem>
99+
<TabItem value="Windows" label="Windows">
100+
```sh
101+
cortex.exe hardware activate [options]
102+
```
103+
</TabItem>
104+
</Tabs>
105+
106+
For example, it returns the following:
107+
```bash
108+
Activated GPUs: 0
109+
```
110+
111+
**Options**:
112+
113+
| Option | Description | Required | Default value | Example |
114+
|---------------------------|----------------------------------------------------|----------|---------------|----------------------|
115+
| `-h`, `--help` | Display help for command. | No | - | `-h` |
116+
|`--gpus` | List of GPUs to activate | Yes | - | `[0, 1]` |

‎docs/docs/cli/models/index.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,7 @@ This command uses a `model_id` from the model that you have downloaded or availa
157157
| Option | Description | Required | Default value | Example |
158158
|---------------------------|---------------------------------------------------------------------------|----------|----------------------------------------------|------------------------|
159159
| `model_id` | The identifier of the model you want to start. | Yes | `Prompt to select from the available models` | `mistral` |
160+
| `--gpus` | List of GPUs to use. | No | - | `[0,1]` |
160161
| `-h`, `--help` | Display help information for the command. | No | - | `-h` |
161162

162163
## `cortex models stop`

‎docs/docs/cli/models/start.md‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -12,34 +12,28 @@ description: Cortex models subcommands.
1212
This command starts a model defined by a `model_id`.
1313

1414

15-
1615
## Usage
1716

1817
```bash
1918
# Start a model
2019
cortex models start [model_id]
2120

22-
# Start a model with a preset
23-
cortex models start [model_id] [options]
24-
2521
# Start with a specified engine
2622
cortex models start [model_id]:[engine] [options]
2723
```
2824

2925

3026
:::info
3127
- This command uses a `model_id` from the model that you have downloaded or available in your file system.
32-
- Model preset is applied only at the start of the model and does not change during the chat session.
3328
:::
3429

3530
## Options
3631

37-
| Option | Description | Required | Default value | Example |
38-
|---------------------------|---------------------------------------------------------------------------|----------|----------------------------------------------|------------------------|
39-
| `model_id` | The identifier of the model you want to start. | No | `Prompt to select from the available models` | `mistral` |
40-
| `-a`, `--attach` | Attach to an interactive chat session. | No | `false` | `-a` |
41-
| `-p`, `--preset <preset>` | Apply a chat preset to the chat session. | No | `false` | `-p friendly` |
42-
| `-h`, `--help` | Display help information for the command. | No | - | `-h` |
32+
| Option | Description | Required | Default value | Example |
33+
|---------------------------|----------------------------------------------------------|----------|----------------------------------------------|-------------------|
34+
| `model_id` | The identifier of the model you want to start. | No | `Prompt to select from the available models` | `mistral` |
35+
| `--gpus` | List of GPUs to use. | No | - | `[0,1]` |
36+
| `-h`, `--help` | Display help information for the command. | No | - | `-h` |
4337

4438

4539

0 commit comments

Comments
 (0)