How to use the Local AI Finetuner

1. Overview

The fine-tuning system consists of two components that work together over a local network:

Server: finetune_server.py — A FastAPI server that runs on any machine with a CUDA GPU (Linux recommended), or on any machine without a GPU using CPU mode. It accepts training data, fine-tunes a model using LoRA via the Hugging Face ecosystem, converts the result to GGUF format, and serves the file for download.

Client (Windows machine): Local AI Finetuner — A Windows desktop application. Double-click the icon on your desktop to launch it. It connects to the server over LAN, uploads training data, monitors progress in real time, downloads the finished GGUF, and registers the model with Ollama.

The server and client communicate over HTTP on port 8765. Both machines must be on the same local network, or the server port must be forwarded if connecting remotely.

---

2. Server Requirements

2.1 Hardware

GPU mode (recommended)

• NVIDIA GPU with CUDA support (minimum 8 GB VRAM recommended for 7B models)
• RAM: 32 GB recommended (merge step runs on CPU)
• Disk: 60 GB free minimum per fine-tuning job (model cache + merged model + GGUF)
• OS: Ubuntu 20.04 or later (Windows is supported but Linux is preferred)

CPU mode

• Any modern multi-core CPU. The more cores available, the better — the server allocates all cores automatically, reserving a small number for the OS and client.
• RAM: 32 GB minimum for 7B models (the full model is loaded in float32)
• Disk: 60 GB free minimum per fine-tuning job
• OS: Windows or Linux

Note: CPU mode is significantly slower than GPU mode. A 7B model with 200 training examples may take one to several hours depending on hardware. Larger datasets will take proportionally longer.

2.2 System Software

• Python 3.10 or later (Python 3.12 or 3.13 are supported; see note on PyTorch wheels below)
• CUDA 11.8 or 12.x if using GPU mode (match your GPU driver — check with nvidia-smi)
• git (for cloning llama.cpp)
• A Python virtual environment (created automatically by the setup scripts)

Windows only: VS Build Tools and CMake are not required. The setup scripts use convert_hf_to_gguf.py, which is a pure Python script and does not need any C++ compiler toolchain.

2.3 Python Packages

The quickest way to install all dependencies is to run the supplied setup script (see section 3.1). The scripts install a pinned, tested dependency stack. If you need to install manually:

python3 -m venv ~/finetune_venv
source ~/finetune_venv/bin/activate   # Linux
# or: finetune_venv\Scripts\activate  # Windows

# GPU mode — install PyTorch with CUDA support
# Python 3.10–3.12: use cu121
pip install "torch>=2.3.0" --index-url https://download.pytorch.org/whl/cu121
# Python 3.13+: use cu124
pip install "torch>=2.3.0" --index-url https://download.pytorch.org/whl/cu124

# CPU mode — install standard PyTorch (no CUDA required)
pip install "torch>=2.3.0"

# Pinned ML stack (same for both modes)
pip install transformers==4.46.2 trl==0.12.2 peft==0.13.2 accelerate==0.34.2

# Supporting packages
pip install datasets fastapi uvicorn click python-multipart sentencepiece protobuf rich

# numpy pinned below 2.0 for llama.cpp compatibility
pip install "numpy>=1.24,<2.0"

Package	Pinned version	Purpose
torch	>=2.3.0 (cu121 or cu124)	Deep learning framework
transformers	4.46.2	Model loading, tokenisation, training
trl	0.12.2	SFTTrainer for supervised fine-tuning
peft	0.13.2	LoRA fine-tuning
accelerate	0.34.2	Distributed training and mixed precision
bitsandbytes	latest	4-bit quantisation (QLoRA, GPU mode only)
datasets	latest	Dataset loading and processing
fastapi	latest	HTTP API server framework
uvicorn + click	latest	ASGI server for FastAPI
numpy	>=1.24,<2.0	Required for llama.cpp converter compatibility

2.4 llama.cpp (GGUF Conversion)

Required to convert the fine-tuned model to GGUF format for use with Ollama. Only the Python conversion script is used — no C++ compilation is required.

cd ~
git clone --depth=1 https://github.com/ggerganov/llama.cpp
pip install gguf sentencepiece

The server automatically searches for convert_hf_to_gguf.py first in the same directory as finetune_server.py, then in ~/llama.cpp/. If llama.cpp is not found, the server will zip the merged Hugging Face model as a fallback — but this cannot be registered directly with Ollama.

---

3. Server Setup & Launch

3.1 Recommended: Automated Setup

Windows

Two setup scripts are supplied: server_setup_gpu.bat (for machines with an NVIDIA GPU) and server_setup_cpu.bat (for CPU-only machines). Each script:

• Checks Python is installed and the version is supported
• Detects GPU presence (GPU script only) and warns if VRAM is below 6 GB
• Creates a virtual environment at finetune_venv in the script directory
• Installs the correct PyTorch wheel (automatically selecting cu121 or cu124 based on your Python version)
• Installs all pinned ML dependencies
• Clones llama.cpp and installs the GGUF conversion requirements
• Verifies all imports before finishing

Place finetune_server.py, server_setup_gpu.bat, server_setup_cpu.bat, and launch_finetune_server.bat in the same folder, then double-click the appropriate setup script. No administrator rights or compiler tools are required.

Linux

The supplied setup_finetune_server.sh performs the same steps as the Windows scripts:

chmod +x setup_finetune_server.sh
./setup_finetune_server.sh

It detects GPU availability, installs build-essential and cmake via apt if needed, creates a virtual environment at ~/finetune_venv, installs the pinned dependency stack, clones llama.cpp, and starts the server on completion.

3.2 Launching the Server (after initial setup)

Windows — recommended method

Double-click launch_finetune_server.bat. This launcher:

• Detects whether the virtual environment exists
• If it does not exist (first run), automatically calls server_setup_gpu.bat or server_setup_cpu.bat based on whether an NVIDIA GPU is present
• Activates the virtual environment and starts finetune_server.py

Place a shortcut to launch_finetune_server.bat on the desktop if convenient. The script must remain in the same folder as finetune_server.py — do not copy the .bat file itself to the desktop, only a shortcut.

Windows — manual launch

finetune_venv\Scripts\activate
python finetune_server.py

Linux — manual launch

source ~/finetune_venv/bin/activate
python finetune_server.py

On a machine with a GPU, a successful start looks like:

============================================================
Local AI Translator — Fine-Tuning Server
GPU: NVIDIA GeForce GTX 1080 Ti  (11.7 GB)  backend=cuda
CPU threads: 10 / 12  (2 reserved for OS/client)
Work dir: /home/{username}/finetune_jobs
Listening on http://0.0.0.0:8765
============================================================

On a machine without a GPU:

============================================================
Local AI Translator — Fine-Tuning Server
GPU: None  (N/A)  backend=cpu  (CPU mode available for GPU-less machines)
CPU threads: 10 / 12  (2 reserved for OS/client)
Work dir: /home/{username}/finetune_jobs
Listening on http://0.0.0.0:8765
============================================================

If the GPU shows as None on a machine that does have a GPU, PyTorch is not finding CUDA. Verify with:

python -c "import torch; print(torch.cuda.is_available())"

If this returns False, reinstall PyTorch with the correct CUDA wheel (see section 2.3).

3.3 Optional Arguments

python finetune_server.py --host 0.0.0.0 --port 8765

The default host 0.0.0.0 makes the server reachable from other machines on the LAN. Change to 127.0.0.1 to restrict to local access only.

3.4 Verifying the Server

From any browser or terminal, check the health endpoint:

http://192.168.1.x:8765/health

A GPU machine returns:

{
  "status": "ok",
  "gpu": {
    "available": true,
    "name": "NVIDIA GeForce GTX 1080 Ti",
    "vram_gb": 11.7,
    "backend": "cuda",
    "cpu_fallback_available": true
  }
}

A CPU-only machine returns:

{
  "status": "ok",
  "gpu": {
    "available": false,
    "name": "None (CPU mode available)",
    "vram_gb": 0,
    "backend": "cpu",
    "cpu_fallback_available": true
  }
}

3.5 Disk Space Management

Each completed fine-tuning job leaves files on disk. For a 7B model:

• Hugging Face model cache (~/.cache/huggingface): ~15 GB — kept between jobs
• Merged model (temporary, per job): ~14 GB — deleted after GGUF conversion
• GGUF output file: ~7 GB (q8_0 format)

After downloading the GGUF to your Windows machine, delete the job directory to recover disk space:

rm -rf ~/finetune_jobs/<job_id>   # Linux
rmdir /s /q finetune_jobs\<job_id>  # Windows

Minimum recommended free disk space before starting a job: 25 GB. The merge step writes a full fp16 copy of the model to disk before GGUF conversion. In CPU mode the model is held in float32, which increases RAM requirements but does not affect disk usage during the merge.

---

4. Client Requirements (Windows Machine)

4.1 Software

• Ollama installed and running (https://ollama.com)

4.2 Launching the Client

Double-click the Local AI Finetuner icon on your desktop. No installation or command line is required — the application is a self-contained Windows executable.

---

5. Fine-Tuning Workflow

5.1 Prepare Training Data

The Finetuner accepts bilingual translation data in any of the following formats:

• TMX (.tmx) — exported from SDL Trados, memoQ, OmegaT, Wordfast, or any TMX-compatible CAT tool
• XLIFF 1.2 and XLIFF 2.0 (.xlf, .xliff)
• TSV (.tsv) — two-column tab-separated file with language headings
• Bilingual Word document (.docx) — a single two-column table with language headings
• Excel spreadsheet (.xlsx) — bilingual columns across one or more sheets, with language headings

5.2 Selecting GPU or CPU Mode

When the client connects to the server, it reads the health endpoint automatically. If the server reports no GPU, CPU mode is enabled in the client and a warning is displayed. You can also enable CPU mode manually by ticking the CPU mode checkbox in the GPU Server section before clicking Check Server.

When CPU mode is active, 4-bit quantisation is disabled automatically — it is not supported without a GPU. A confirmation dialog will appear when you click Start Fine-Tuning in CPU mode, reminding you that training may take several hours or days.

5.3 Running a Job

1. Start the server on the GPU or CPU machine (double-click launch_finetune_server.bat on Windows, or run the Python file directly on Linux).
2. Double-click the Local AI Finetuner icon on your Windows desktop.
3. Enter the server IP and port (default: 192.168.x.x:8765) and click Check Server.
4. Select your training file and configure parameters (model, epochs, batch size, LoRA settings).
5. If using a CPU-only server, ensure CPU mode is ticked.
6. Click Start Fine-Tuning. Progress and logs stream in real time.
7. On completion, the GGUF is downloaded automatically — choose a save location when prompted.
8. Click Register with Ollama to make the model available in Local AI Translator.

5.4 Training Parameters

Parameter	Default	Purpose
Model	e.g. Qwen/Qwen2.5-7B-Instruct	Hugging Face model ID
Epochs	2	Number of passes through the training data
Batch size	4	Samples per training step
Learning rate	2e-4	Step size for gradient updates
LoRA r	16	LoRA rank — higher = more capacity, more VRAM
LoRA alpha	32	LoRA scaling factor (typically 2× rank)
Max seq length	512	Maximum token length per training example
Quantize	True	Load model in 4-bit (QLoRA) — GPU mode only
CPU mode	False	Train without a GPU — expect hours to days

---

6. Troubleshooting

GPU not detected at startup Run: python -c "import torch; print(torch.cuda.is_available())" If False, reinstall torch with the correct CUDA wheel. For Python 3.10–3.12: use the cu121 index. For Python 3.13+: use the cu124 index (see section 2.3). If you do not have a GPU, enable CPU mode in the client and proceed normally.

Training is very slow in CPU mode This is expected. A 7B model on CPU may take one to several hours for a small dataset, and longer for larger ones. The server allocates CPU threads automatically. Do not close the server or the client while a CPU job is running — there is no way to resume a job that has been interrupted mid-training. Use the Resume Job ID field to reconnect to a running job if the client window is closed.

Out of memory during training (GPU mode)

• Reduce batch size to 1 or 2
• Reduce max seq length to 256
• Ensure Quantize is ticked to use 4-bit QLoRA

Out of memory during training (CPU mode)

• Reduce batch size to 1
• Reduce max seq length to 256
• Close other applications to free RAM — a 7B model in float32 requires approximately 28 GB of RAM

Out of disk space during merge The merge step requires ~14 GB of free space for a 7B model. Delete old job directories:

rm -rf ~/finetune_jobs/*          # Linux
rmdir /s /q finetune_jobs         # Windows (recreate the folder afterwards)

To move the model cache to an external drive and symlink it (Linux):

mv ~/.cache/huggingface /mnt/external/huggingface_cache
ln -s /mnt/external/huggingface_cache ~/.cache/huggingface

GGUF conversion failed

• Ensure llama.cpp is cloned (to the same folder as finetune_server.py, or to ~/llama.cpp on Linux)
• Install its Python requirements: pip install gguf sentencepiece "numpy>=1.24,<2.0"
• Test manually: ls ~/llama.cpp/convert_hf_to_gguf.py (Linux) or check the script folder on Windows

Poll timeout errors on the client These appear as Read timed out warnings in the client log during the merge and conversion phase. This is normal — the server is busy with post-training work and cannot respond to polls. The job is still running. Warnings will stop once the job completes.

Job not found after server restart If the server is restarted mid-job, the in-memory job store is cleared and the client will show a Job not found warning and stop polling. The training data and any completed checkpoints remain on disk in ~/finetune_jobs/<job_id> but the job cannot be resumed automatically. Start a new job with the same training file.

Import errors on server startup If you see errors about missing or incompatible modules, re-run the setup script for your platform. If running manually:

pip install --upgrade --force-reinstall transformers==4.46.2 trl==0.12.2 peft==0.13.2 accelerate==0.34.2 --no-cache-dir

Use python -m pip rather than pip directly, to ensure packages are installed into the correct virtual environment.

click module not found (uvicorn startup error) This indicates click was not pulled in as a transitive dependency of uvicorn. Install it explicitly:

pip install click

---

7. Server API Reference

All endpoints are available at http://<server-ip>:8765.

Endpoint	Purpose
GET /health	Check server, GPU status, and CPU fallback availability
GET /jobs	List all jobs and their status
GET /jobs/{id}	Get full status and last 100 log lines for a job
GET /jobs/{id}/stream	Server-Sent Events stream of live logs
POST /finetune	Start a new fine-tuning job (multipart form)
POST /jobs/{id}/cancel	Cancel a running job
GET /jobs/{id}/download	Download the completed GGUF file

---