You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jul 4, 2025. It is now read-only.
> ⚠️ **Cortex is currently in Development**: Expect breaking changes and bugs!
6
9
7
-
### **Dependencies**
10
+
## About
11
+
Cortex is an openAI-compatible local AI server that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and a Typescript client library. It can be used as a standalone server, or imported as a library.
12
+
13
+
Cortex currently supports two inference engines:
14
+
15
+
- Llama.cpp
16
+
- TensorRT-LLM
17
+
18
+
> Read more about Cortex at https://jan.ai/cortex
19
+
20
+
### Repo Structure
21
+
```
22
+
# Entity Definitions
23
+
domain/ # This is the core directory where the domains are defined.
24
+
abstracts/ # Abstract base classes for common attributes and methods.
25
+
models/ # Domain interface definitions, e.g. model, assistant.
Before installation, ensure that you have installed the following:
10
69
@@ -17,7 +76,7 @@ Before installation, ensure that you have installed the following:
17
76
18
77
</aside>
19
78
20
-
### **Hardware**
79
+
####**Hardware**
21
80
22
81
Ensure that your system meets the following requirements to run Cortex:
23
82
@@ -37,11 +96,11 @@ Ensure that your system meets the following requirements to run Cortex:
37
96
38
97
-**Disk**: At least 10GB for app and model download.
39
98
40
-
## Cortex Installation
99
+
###Cortex Installation
41
100
42
101
To install Cortex, follow the steps below:
43
102
44
-
### Step 1: Install Cortex
103
+
####Step 1: Install Cortex
45
104
46
105
Run the following command to install Cortex globally on your machine:
47
106
@@ -50,7 +109,7 @@ Run the following command to install Cortex globally on your machine:
50
109
npm i -g @janhq/cortex
51
110
```
52
111
53
-
### Step 2: Verify the Installation
112
+
####Step 2: Verify the Installation
54
113
55
114
After installation, you can verify that Cortex is installed correctly by getting help information.
56
115
@@ -59,7 +118,7 @@ After installation, you can verify that Cortex is installed correctly by getting
59
118
cortex -h
60
119
```
61
120
62
-
### Step 3: Initialize Cortex
121
+
####Step 3: Initialize Cortex
63
122
64
123
Once verified, you need to initialize the Cortex engine.
65
124
@@ -69,23 +128,23 @@ Once verified, you need to initialize the Cortex engine.
69
128
cortex init
70
129
```
71
130
72
-
1. Select between `CPU` and `GPU` modes.
131
+
2. Select between `CPU` and `GPU` modes.
73
132
74
133
```bash
75
134
? Select run mode (Use arrow keys)
76
135
> CPU
77
136
GPU
78
137
```
79
138
80
-
2. Select between GPU types.
139
+
3. Select between GPU types.
81
140
82
141
```bash
83
142
? Select GPU types (Use arrow keys)
84
143
> Nvidia
85
144
Others (Vulkan)
86
145
```
87
146
88
-
3. Select CPU instructions (will be deprecated soon).
147
+
4. Select CPU instructions (will be deprecated soon).
89
148
90
149
```bash
91
150
? Select CPU instructions (Use arrow keys)
@@ -94,10 +153,10 @@ cortex init
94
153
AVX-512
95
154
```
96
155
97
-
1. Cortex will download the required CPU instruction sets if you choose `CPU` mode. If you choose `GPU` mode, Cortex will download the necessary dependencies to use your GPU.
98
-
2. Once downloaded, Cortex is ready to use!
156
+
5. Cortex will download the required CPU instruction sets if you choose `CPU` mode. If you choose `GPU` mode, Cortex will download the necessary dependencies to use your GPU.
157
+
6. Once downloaded, Cortex is ready to use!
99
158
100
-
### Step 4: Pull a model
159
+
####Step 4: Pull a model
101
160
102
161
From HuggingFace
103
162
@@ -111,14 +170,14 @@ From Jan Hub (TBD)
111
170
cortex pull llama3
112
171
```
113
172
114
-
### Step 5: Chat
173
+
####Step 5: Chat
115
174
116
175
```bash
117
176
cortex run janhq/phi-3-medium-128k-instruct-GGUF
118
177
```
119
178
120
179
## Run as an API server
121
-
180
+
To run Cortex as an API server:
122
181
```bash
123
182
cortex serve
124
183
```
@@ -135,18 +194,62 @@ To install Cortex from the source, follow the steps below:
0 commit comments