Example Code - Web Conversation Guide

How to deploy and chat with offline LLMs on a DFRobot RK3576 board: install RK182x drivers on Debian 12, verify the RK1828 NPU via rknn-smi info, then download pre-converted RKNN models and the offline-model app, place model files with consistent prefixes under /userdata/models, and copy rkllm3_server.sh, run.sh, and switch_to.sh into each model folder. It’s like setting up a local AI kiosk: you prepare the hardware, load models such as Qwen2.5-3B or MiniCPM-V-4, and use switch_to.sh to select the active model. Finally, run start.sh to install or start Docker, then open :5000 in a browser to access the web chat UI, monitor logs with docker logs, and manage the container using standard Docker commands.

Test Environment

Linux Version: Debian 12
RK182x SDK Version: V1.0.4
rknn-smi Version: 1.3.0
Hardware: DFRobot RK3576 + RK1828

Environment Setup

Install Drivers

Download the driver installation package (select RELEASE_V1.0.4 or newer).
Download link: https://console.box.lenovo.com/l/PHk0PF
Transfer the file to the development board, extract and install it.

tar -zxf rknn3_rk182x_m2_installer_arm64.tgz
sudo ./install.sh

Verify Test Environment

Check RK182x status

# If you are not logged in as root, switch to root user
linaro@linaro-alip:/$ [ $(whoami) != "root" ] && sudo su -
root@linaro-alip:/# rknn-smi info
+------------------------+---------------+---------------+----------------------+
| rknn-smi Version: 1.3.0 |
+========================+===============+===============+======================+
| Device Status | Health | Power(mW) | Npu(%) |
| Chip Name | Bus-Id | Temp(C) | Memory-Usage(MB) |
+========================+===============+===============+======================+
| 0 Online | OK | 81 | 0 |
| 0 RK1828 | 0000:01:00.0 | 44 | 32 / 5120 |
+========================+===============+===============+======================+

Download Model and Application Files

Download model files

Pre-converted RKNN model cloud disk link: RKNN3_SDK (https://console.box.lenovo.com/l/H1fig1, access code: rknn), path: RKNN3_SDK/rknn3_models/v1.0.4
Download application files: offline-model
Place model files under /userdata/models/

Note: Keep consistent prefixes for model files.
Put rkllm3_server.sh from the offline-model folder into /userdata/models/; Put run.sh and switch_to.sh from the offline-model folder into each model subdirectory.

Reference Folder Structure

/userdata/models/
├── rkllm3_server.sh
├── Qwen2.5-3B/
│   ├── Qwen2.5-3B-Instruct.rknn
│   ├── Qwen2.5-3B-Instruct.tokenizer.gguf
│   ├── Qwen2.5-3B-Instruct.embed.bin
│   ├── Qwen2.5-3B-Instruct.weight
│   ├── run.sh
│   └── switch_to.sh
└── MiniCPM-V-4/
    ├── MiniCPM-V-4-llm.rknn
    ├── MiniCPM-V-4-llm.tokenizer.gguf
    ├── MiniCPM-V-4.embed.bin
    ├── MiniCPM-V-4.weight            ← The above four files belong to the LLM part and must share the same file prefix
    ├── MiniCPM-V-4-vision.rknn       ← Multimodal models may contain an extra RKNN file
    ├── run.sh
    └── switch_to.sh

Launch Application

Extract the offline-model image package and place it under /home/linaro/
Execute commands

sudo bash /home/linaro/offline-model/start.sh --install-docker   # First launch: automatically install Docker
sudo bash /home/linaro/offline-model/start.sh                    # Subsequent launches

3. Open a browser and visit <Development-Board-IP>:5000 to access the web chat interface

You can select models and languages in the top-right corner.

Common Commands

docker logs -f offline-model                         # View real-time logs
docker exec -it offline-model bash                   # Enter the container
docker restart offline-model                         # Restart container
docker stop offline-model && docker rm -f offline-model  # Stop and remove container
sudo bash /home/linaro/offline-model/start.sh        # Start / restart service

Script List

Script locations:

Script	Location	Quantity
`rkllm3_server.sh`	`/userdata/models/`	1 copy
`run.sh`	`/userdata/models/<each model>/`	1 per model
`switch_to.sh`	`/userdata/models/<each model>/`	1 per model
`run_rkllm3_server.sh`	`/userdata/models/`	1 symlink (auto-created)

run_rkllm3_server.sh is a symbolic link automatically created when switch_to.sh runs:

# Content (symbolic link, no physical file):
run_rkllm3_server.sh -> /userdata/models/MiniCPM-V-4/run.sh

No error will be triggered if the symlink is missing on first deployment. start.sh will automatically fall back to the first model directory containing run.sh.
You can also run switch_to.sh inside each model folder to switch to that model as the currently loaded model.