V2 update plan

[PicoCreator]

Latest version of https://github.com/saharNooby/rwkv.cpp has new quantization format (breaking change?) and GPU offload (!!!)
Since this might be potentially breaking changes, its gonna be a v2 update.

• update to the newer version, which has breaking change on the model (might be backwards compat)
• (to confirm if not backwards compat) create a new set of ".bin" files for the new version
• make changes to the API, to add support for GPU offload (its a param now in the new version, on how many layers you want to offload to the GPU)
• for input inference, update to the new batch mode API (10x faster)
• (stretch goal) change to async API
• support for world model / world tokenizer (we can detect this using the token count)

[PicoCreator]

<= 67.2 ms on 7B with partial GPU offload (3060 Ti 8G) is a huge win win

[cahya-wirawan]

Hi,
I converted a pth file to bin format, unfortunately it crashed with following error message:

% rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin 
--------------------------------------
Starting RWKV chat mode
--------------------------------------
Loading model from ./rwkv-7b-369-Q5_1.bin ...
Unsupported file version 101
/Users/eugene/Desktop/RWKV/rwkv.cpp/rwkv.cpp:211: version == RWKV_FILE_VERSION
zsh: segmentation fault  rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin

Is it the compatibility issue mentioned here?
Thanks (Looking forward to the world version :-) )

[PicoCreator]

Yup, the new rwkv.cpp - is now merged into v2 (publishing now)

[PicoCreator]

Merged in : 74655de

This resolves all issues, EXCEPT support for world model tokenizer (needs a new JS tokenizer)

[cgisky1980]

Yup, the new rwkv.cpp - is now merged into v2 (publishing now)

update the docs please

[cgisky1980]

Merged in : 74655de

This resolves all issues, EXCEPT support for world model tokenizer (needs a new JS tokenizer)

waiting for it ~~

[cahya-wirawan]

Hi, I reinstalled the package, but when I run it, I get only following result:

% rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin
--------------------------------------
Starting RWKV chat mode
--------------------------------------
Loading model with {"path":"./rwkv-7b-369-Q5_1.bin","threads":4,"gpuOffload":0} ...
The following is a conversation between the User and the Bot ...
--------------------------------------
? User:  Hi how are you
Bot: <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

Did I convert the model wrong?

[cgisky1980]

Hi, I reinstalled the package, but when I run it, I get only following result:

% rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin
--------------------------------------
Starting RWKV chat mode
--------------------------------------
Loading model with {"path":"./rwkv-7b-369-Q5_1.bin","threads":4,"gpuOffload":0} ...
The following is a conversation between the User and the Bot ...
--------------------------------------
? User:  Hi how are you
Bot: <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

Did I convert the model wrong?

world models not support yet

[cahya-wirawan]

Hi, I reinstalled the package, but when I run it, I get only following result:

% rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin
--------------------------------------
Starting RWKV chat mode
--------------------------------------
Loading model with {"path":"./rwkv-7b-369-Q5_1.bin","threads":4,"gpuOffload":0} ...
The following is a conversation between the User and the Bot ...
--------------------------------------
? User:  Hi how are you
Bot: <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

Did I convert the model wrong?

world models not support yet

It is just a normal finetuned raven model