You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jul 4, 2025. It is now read-only.
Before installation, ensure that you have installed the following:
69
74
70
75
-**Node.js**: Required for running the installation.
71
76
-**NPM**: Needed to manage packages.
72
77
-**CPU Instruction Sets**: Available for download from the [Cortex GitHub Releases](https://github.com/janhq/cortex/releases) page.
73
78
74
-
<aside>
75
-
💡 The **CPU instruction sets** are not required for the initial installation of Cortex. This dependency will be automatically installed during the Cortex initialization if they are not already on your system.
76
79
77
-
</aside>
80
+
>💡 The **CPU instruction sets** are not required for the initial installation of Cortex. This dependency will be automatically installed during the Cortex initialization if they are not already on your system.
78
81
79
-
#### **Hardware**
82
+
83
+
### **Hardware**
80
84
81
85
Ensure that your system meets the following requirements to run Cortex:
82
86
@@ -96,85 +100,31 @@ Ensure that your system meets the following requirements to run Cortex:
96
100
97
101
-**Disk**: At least 10GB for app and model download.
98
102
99
-
### Cortex Installation
100
-
101
-
To install Cortex, follow the steps below:
102
-
103
-
#### Step 1: Install Cortex
104
-
105
-
Run the following command to install Cortex globally on your machine:
106
-
107
-
```bash
108
-
# Install using NPM globally
103
+
## Quickstart
104
+
1. Install the NPM package:
105
+
```bash
109
106
npm i -g @janhq/cortex
110
107
```
111
108
112
-
#### Step 2: Verify the Installation
113
-
114
-
After installation, you can verify that Cortex is installed correctly by getting help information.
115
-
116
-
```bash
117
-
# Get the help information
118
-
cortex -h
119
-
```
120
-
121
-
#### Step 3: Initialize Cortex
122
-
123
-
Once verified, you need to initialize the Cortex engine.
4. Select CPU instructions (will be deprecated soon).
148
-
149
-
```bash
150
-
? Select CPU instructions (Use arrow keys)
151
-
> AVX2
152
-
AVX
153
-
AVX-512
123
+
5. Start chatting with the model
124
+
```bash
125
+
cortex chat tell me a joke
154
126
```
155
127
156
-
5. Cortex will download the required CPU instruction sets if you choose `CPU` mode. If you choose `GPU` mode, Cortex will download the necessary dependencies to use your GPU.
157
-
6. Once downloaded, Cortex is ready to use!
158
-
159
-
#### Step 4: Pull a model
160
-
161
-
From HuggingFace
162
-
163
-
```bash
164
-
cortex pull janhq/phi-3-medium-128k-instruct-GGUF
165
-
```
166
-
167
-
From Jan Hub (TBD)
168
-
169
-
```bash
170
-
cortex pull llama3
171
-
```
172
-
173
-
#### Step 5: Chat
174
-
175
-
```bash
176
-
cortex run janhq/phi-3-medium-128k-instruct-GGUF
177
-
```
178
128
179
129
## Run as an API server
180
130
To run Cortex as an API server:
@@ -211,44 +161,24 @@ The following CLI commands are currently available:
211
161
> ⚠️ **Cortex is currently in Development**: More commands will be added soon!
212
162
213
163
```bash
214
-
cortex --help
215
-
Usage: cortex <command>
216
-
217
-
Commands:
218
-
chat Send a query to the chat service.
219
-
Example: cortex chat "tell me a joke" --stream
220
164
165
+
serve Providing API endpoint for Cortex backend
166
+
chat Send a chat request to a model
167
+
init|setup Init settings and download cortex's dependencies
168
+
ps Show running models and their status
169
+
kill Kill running cortex processes
170
+
pull|download Download a model. Working with HuggingFace model id.
171
+
run [options] EXPERIMENTAL: Shortcut to start a model and chat
172
+
models Subcommands for managing models
221
173
models list List all available models.
222
-
Example: cortex models list
223
-
224
174
models pull Download a specified model.
225
-
Example: cortex models pull llama3:8b
226
-
227
175
models remove Delete a specified model.
228
-
Example: cortex models remove llama3:8b
229
-
230
176
models get Retrieve the configuration of a specified model.
231
-
Example: cortex models config llama3:8b
232
-
233
177
models start Start a specified model.
234
-
Example: cortex models start llama3:8b
235
-
236
178
models stop Stop a specified model.
237
-
Example: cortex models stop llama3:8b
238
-
239
179
models update Update the configuration of a specified model.
240
-
Example: cortex models update llama3:8b --ngl 32
241
-
242
180
engines Execute a specified command related to engines.
> ⚠️ **cortex-cpp is currently in Development**: Expect breaking changes and bugs!
12
12
13
-
## Features
14
-
- Fast Inference: Built on top of the cutting-edge inference library llama.cpp, modified to be production ready.
15
-
- Lightweight: Only 3MB, ideal for resource-sensitive environments.
16
-
- Easily Embeddable: Simple integration into existing applications, offering flexibility.
17
-
- Quick Setup: Approximately 10-second initialization for swift deployment.
18
-
- Enhanced Web Framework: Incorporates drogon cpp to boost web service efficiency.
19
-
20
13
## About cortex-cpp
21
14
22
-
cortex-cpp is a high-efficiency C++ inference engine for edge computing, powering [Jan](https://jan.ai/). It is lightweight and embeddable, ideal for product integration.
15
+
Cortex.cpp is a stateless, C++ server that is 100% compatible with OpenAI API (stateless endpoints). It includes a Drogon server, with request queues, model orchestration logic, and hardware telemetry, and more, for prod environments.
23
16
24
17
The binary of cortex-cpp after zipped is only ~3mb in size with none to minimal dependencies (if you use a GPU need CUDA for example) make it desirable for any edge/server deployment 👍.
25
18
26
-
> Read more about Nitro at https://nitro.jan.ai/
19
+
> Read more about Cortex-cpp at https://jan.ai/cortex
20
+
21
+
### Features
22
+
- Fast Inference: Built on top of the cutting-edge inference library llama.cpp, modified to be production ready.
23
+
- Lightweight: Only 3MB, ideal for resource-sensitive environments.
24
+
- Easily Embeddable: Simple integration into existing applications, offering flexibility.
25
+
- Quick Setup: Approximately 10-second initialization for swift deployment.
26
+
- Enhanced Web Framework: Incorporates drogon cpp to boost web service efficiency.
27
27
28
28
### Repo Structure
29
29
30
30
```
31
31
.
32
-
├── controllers
33
-
├── docs
34
-
├── llama.cpp -> Upstream llama C++
35
-
├── cortex-cpp-deps -> Dependencies of the cortex-cpp project as a sub-project
36
-
└── utils
32
+
├── common # Common libraries or shared resources
33
+
├── controllers # Controller scripts or modules for managing interactions
34
+
├── cortex-common # Shared components across different cortex modules
35
+
├── cortex-cpp-deps # Dependencies specific to the cortex-cpp module
36
+
├── engines # Different processing or computational engines
37
+
├── examples # Example scripts or applications demonstrating usage
38
+
├── test # Test scripts and testing frameworks
39
+
└── utils # Utility scripts and helper functions
40
+
37
41
```
38
42
39
43
## Quickstart
@@ -43,13 +47,13 @@ The binary of cortex-cpp after zipped is only ~3mb in size with none to minimal
Download the latest version of Nitro at https://nitro.jan.ai/ or visit the **[GitHub Releases](https://github.com/janhq/cortex/releases)** to download any previous release.
195
+
Download the latest version of Cortex-cpp at https://jan.ai/ or visit the **[GitHub Releases](https://github.com/janhq/cortex/releases)** to download any previous release.
192
196
193
197
## Nightly Build
194
198
195
-
Nightly build is a process where the software is built automatically every night. This helps in detecting and fixing bugs early in the development cycle. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml)
199
+
Nightly build is a process where the software is built automatically every night. This helps detect and fix bugs early in the development cycle. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml)
196
200
197
-
You can join our Discord server [here](https://discord.gg/FTk2MvZwJH) and go to channel [github-nitro](https://discordapp.com/channels/1107178041848909847/1151022176019939328) to monitor the build process.
201
+
You can join our Discord server [here](https://discord.gg/FTk2MvZwJH) and go to channel [cortex](https://discordapp.com/channels/1107178041848909847/1151022176019939328) to monitor the build process.
198
202
199
203
The nightly build is triggered at 2:00 AM UTC every day.
200
204
201
-
The nightly build can be downloaded from the url notified in the Discord channel. Please access the url from the browser and download the build artifacts from there.
205
+
The nightly build can be downloaded from the URL notified in the Discord channel. Please access the URL from the browser and download the build artifacts from there.
202
206
203
207
## Manual Build
204
-
205
-
Manual build is a process where the software is built manually by the developers. This is usually done when a new feature is implemented or a bug is fixed. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml)
208
+
Manual build is a process in which the developers build the software manually. This is usually done when a new feature is implemented or a bug is fixed. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml)
206
209
207
210
It is similar to the nightly build process, except that it is triggered manually by the developers.
208
211
@@ -214,4 +217,4 @@ It is similar to the nightly build process, except that it is triggered manually
214
217
215
218
## Star History
216
219
217
-
[](https://star-history.com/#janhq/nitro&Date)
220
+
[](https://star-history.com/#janhq/cortex-cpp&Date)
0 commit comments