Brewing Your Own Offensive Coding Assistant: Fine-Tuning a Local LLM

Most red teamers I know use ChatGPT or Claude for payload scaffolding, recon parsing, quick code transformations, and pretext drafts. That works until a model refuses legitimate engagement work or the prompt contains data that cannot leave the operator’s environment. Client target lists, internal tooling, captured credentials, and recovered C2 logs should not leave the jump host.

A practical alternative is a small, local model adapted to the work you perform. Local inference reduces third-party exposure, but “local” is a configuration property rather than a promise made by the weights. Disable cloud features, audit the surrounding UI and extensions, and enforce network controls if the workstation must remain offline.

I used a 3B student model on an 8 GB GPU and included notes for running a 7B model on larger hardware.

I am still learning this area. The choices below come from the published research and my own iterations, but they are starting points rather than universal answers.

When fine-tuning helps

There are four common ways to adapt an LLM to a specific domain. They can be combined, but they solve different problems.

Approach	What it does	Cost	When to reach for it
Prompt engineering	Steer behavior with instructions and examples in-context	Free	First move. Always.
RAG	Inject retrieved context (docs, notes, code) at inference time	Low	The model lacks facts you have on disk
Fine-tuning	Update model weights on examples of desired behavior	Medium	The model lacks patterns - output format, tone, refusal posture, domain idioms
Abliteration	Surgically remove refusal directions from a model’s residual stream	Low	You only need to neutralize refusals; you don’t need the model to be better at the task

Fine-tuning becomes useful when evaluation shows that prompting and retrieval are not enough. Prompt engineering shapes behavior without changing weights, while RAG supplies external knowledge instead of teaching a response pattern. Removing safeguards does not add domain expertise; it only changes which requests the model attempts.

Fine-tuning also takes the most time and is easy to do badly. In my experience, dataset work consumes far more time than the training run.

Threat model and scope

Before choosing a model or training configuration, I wrote down what I expected the assistant to do.

It is a local coding and tradecraft assistant. It generates payload skeletons, transforms code, summarizes recon output, drafts phishing copy, parses BloodHound paths, explains CVEs, and writes detection rules from an attacker’s perspective. It runs on the operator’s workstation with external network access blocked during inference.

It is not for autonomous operation or decisions on live targets. I also avoid using it anywhere a hallucination could cause more than a wasted minute.

The model has the same authorization boundary as the rest of the toolkit. I use it only within written engagement scope, and both the training data and model stay on operator-controlled hardware.

This definition bounds the dataset. Because the model is not making autonomous decisions, it does not need agentic reasoning traces in its training data. Because its primary job is code, code needs to make up a substantial part of that data.

Choosing the base model

Three properties matter for an offensive coding assistant:

1. Open weights allow the model to run offline without an API dependency.
2. Code pretraining provides a better starting point for this dataset than a general-purpose model.
3. The model needs to fit the available VRAM with enough headroom to iterate.

These were the practical options I considered for each VRAM tier:

VRAM	Recommended student model	Notes
24+ GB (3090, 4090, A5000)	Qwen2.5-Coder-7B-Instruct	Comfortable headroom; a 14B experiment needs tighter settings
16 GB (4080, 4070 Ti Super)	Qwen2.5-Coder-7B-Instruct	Tight but works at batch=1
12 GB (3060 12GB, 4070)	Qwen2.5-Coder-3B-Instruct	Comfortable; 7B is painful here
8 GB (3060 Ti, 4060)	Qwen2.5-Coder-3B-Instruct	Bring batch=1, max_seq_len=1024

The “Instruct” variant is a pragmatic baseline because it already supports conversational prompting. Fine-tuning a base model is a different objective and may require substantially more data and evaluation. Use the chat template supplied with the selected model rather than assuming that every instruct model uses ChatML.

I ran the experiment on an 8 GB card, so the examples use Qwen2.5-Coder-3B-Instruct as the student. The same code can load the 7B model on hardware with more VRAM.

If local hardware is too tight, a short-lived cloud GPU is an option for sanitized data. Pricing and availability change frequently, and moving training to a provider changes the threat model, so verify both before uploading anything.

Building the dataset

The dataset had more influence on the result than any individual training parameter.

Format

The examples use the selected Qwen model’s ChatML-style conversation format:

{"messages":[
  {"role":"system","content":"You are an offensive security assistant. Help the operator with red team coding tasks. Provide working code, explain trade-offs, note detection considerations."},
  {"role":"user","content":"Write a Windows shellcode loader using indirect syscalls for NtAllocateVirtualMemory and NtProtectVirtualMemory. Use halo's gate for SSN resolution."},
  {"role":"assistant","content":"Here's a minimal indirect syscall loader using Hell's Gate for SSN resolution...\n\n```c\n// SSN resolution via Hell's Gate\n...\n```\n\n**Detection considerations:**\n- Indirect syscalls bypass user-mode hooks but ETW Threat Intelligence still sees..."}
]}

Each JSONL line contains one conversation. I keep the system prompt consistent with the one used during inference because the model learns that surrounding structure.

What goes in

I started with this approximate distribution:

Bucket	Share	Example
Code generation (offensive)	35%	“Write a [technique] in [language] targeting [platform]“
Code transformation	20%	“Convert this loader to use direct syscalls”
Tradecraft Q&A	15%	“Explain when to use APC injection vs. early bird”
Recon/output parsing	10%	“Summarize this BloodHound JSON into priority paths”
Pretext / social engineering writing	5%	“Draft a phishing pretext for a finance team targeting AP”
Detection/defense (attacker perspective)	10%	“What logs does technique X generate?”
Refusal-correction examples	5%	Cases where the base model refused but shouldn’t have

The last bucket needs careful review. I collected prompts where the base model refused work that was valid within the lab, wrote the response I expected, and added only those corrected examples. The goal was to calibrate narrow false refusals without removing every safety boundary.

Where the data comes from

I used three sources, listed in the order I trust them:

1. My own engagement notes and code provided the most relevant examples. I removed client names, IP addresses, hostnames, credentials, and beacon configuration before the material reached a training script, then reviewed every line manually.
2. Public tradecraft came from vendor posts, conference talks, training writeups, and offensive-tool documentation converted into question-and-answer pairs.
3. Synthetic generation expanded seed prompts into draft conversations that I reviewed and tested before use. I treat synthetic rows as untrusted drafts whose value depends on seed diversity, teacher quality, filtering, and verification.

I considered two paths for the teacher model:

• A hosted frontier API may produce stronger results on some tasks, but every submitted seed leaves the local environment and the teacher can still refuse legitimate red team prompts. I would use this only for generic technique seeds, never engagement context.
• A local model through Ollama can generate drafts without sending prompts to a hosted inference service when cloud features are disabled. In one small test, two of my first three seeds were refused; that observation is not a benchmark of Qwen2.5-Coder’s overall refusal rate.

One experimental option is an abliterated model variant, which modifies internal activation directions associated with refusal behavior. That intervention can change helpfulness, safety behavior, calibration, and task quality; it does not leave the model otherwise identical.

The example below names huihui_ai/qwen2.5-coder-abliterate:7b because that is what I tested. It is a community artifact, so verify its provenance, hashes, license, and behavior before placing it in a sensitive environment.

A minimal generation loop using the local path:

# synth_local.py - expand seeds via local Ollama (no API cost, fully offline)
import json, ollama

client = ollama.Client(host="http://localhost:11434")
MODEL = "huihui_ai/qwen2.5-coder-abliterate:7b"
SYSTEM = open("data/system_prompt.txt").read()

def expand(seed: str) -> dict:
    resp = client.chat(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": seed},
        ],
        options={"temperature": 0.4, "num_predict": 4096},
    )
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": seed},
        {"role": "assistant", "content": resp["message"]["content"]},
    ]}

with open("data/seeds.txt") as f, open("data/synthetic.jsonl", "a") as out:
    for line in f:
        seed = line.strip()
        if not seed or seed.startswith("#"):
            continue
        out.write(json.dumps(expand(seed)) + "\n")

My 7B local outputs often looked compilable while containing technical errors. Larger teachers can improve some tasks, but model size does not replace review. Validate every generated row against source material, compilation, and a representative test where possible.

Cleaning the data

Even a small dataset needs deliberate cleaning:

• Use MinHash with datasketch to find near-duplicates that exact matching misses. Prompts that differ only in variable names can still overfit the model on one template.
• Drop assistant responses below 100 tokens as an initial quality filter and anything beyond the training context that would be truncated. Review the exceptions rather than treating either threshold as proof of quality.
• Scan for IP addresses, email addresses, client-style hostnames, AWS account IDs, and common credential formats, followed by manual review.
• Hash datasets, record source revisions, and sign any adapters that leave the lab. Do not rely on a memorized prompt canary as proof of origin; it can fail to surface and deliberately teaches the model a unique string.

For a first experiment, 2,000-5,000 reviewed rows is a manageable labeling and evaluation budget rather than a universal threshold. Add data only when held-out evaluation shows that the additional coverage helps.

Training with QLoRA and Unsloth

QLoRA makes this practical on a single consumer GPU by combining two ideas:

1. Quantize the frozen base model to 4-bit to reduce its memory use during fine-tuning.
2. Train low-rank adapters on top so only the adapter parameters receive gradients. For the ranks and model sizes used here, that is generally tens of millions of trainable parameters, with artifact size depending on architecture, rank, dtype, and serialization.

Unsloth is the framework I use here. Its published benchmarks report substantial speed and memory improvements in supported configurations, but the result depends on the model, GPU, sequence length, attention backend, and package versions.

The training ecosystem changes quickly. The script below uses the current TRL interface as of August 2026, where SFTTrainer receives the tokenizer through processing_class. Record and pin the exact package versions after validating the workflow in a clean environment.

The training script

# train.py
from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

MODEL = "unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit"   # swap to 7B with 16 GB+
MAX_SEQ_LEN = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = MODEL,
    max_seq_length = MAX_SEQ_LEN,
    load_in_4bit = True,
)

# Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,                       # adapter rank - capacity vs. overfit knob
    lora_alpha = 32,              # convention: 2x rank
    lora_dropout = 0.0,           # 0 enables Unsloth's fast path
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 1337,
)

# Load dataset and format with the chat template
dataset = load_dataset("json", data_files="data/dataset.jsonl", split="train")

def format_chat(example):
    return {"text": tokenizer.apply_chat_template(
        example["messages"], tokenize=False, add_generation_prompt=False,
    )}

dataset = dataset.map(format_chat, remove_columns=dataset.column_names)
split = dataset.train_test_split(test_size=0.1, seed=1337)

trainer = SFTTrainer(
    model = model,
    processing_class = tokenizer,
    train_dataset = split["train"],
    eval_dataset = split["test"],
    args = SFTConfig(
        output_dir = "out",
        per_device_train_batch_size = 1,        # bump to 2 with 16 GB+
        gradient_accumulation_steps = 4,        # effective batch size 4
        warmup_ratio = 0.03,
        num_train_epochs = 2,                   # baseline; select with evaluation
        learning_rate = 2e-4,                   # baseline; compare lower rates
        bf16 = True,                            # fp16=True on Turing/older
        logging_steps = 10,
        eval_strategy = "epoch",
        save_strategy = "epoch",
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "cosine",
        seed = 1337,
        report_to = "none",
        dataset_text_field = "text",
        max_length = MAX_SEQ_LEN,
        packing = False,                        # compatibility-first baseline
    ),
)

trainer.train()
model.save_pretrained("out/lora-final")
tokenizer.save_pretrained("out/lora-final")

Why these hyperparameters

These values are experiment baselines rather than universal defaults:

• r = 16. Rank controls adapter capacity and trainable parameter count. Compare it with at least one lower rank rather than assuming 16 is optimal.
• lora_alpha = 2 * r. This common starting ratio keeps the configured LoRA scaling constant when comparing ranks, but it remains a hyperparameter.
• lora_dropout = 0.0. Zero dropout enables Unsloth’s optimized path in supported configurations. It is a performance choice, not proof that the model has enough regularization.
• num_train_epochs = 2. Two epochs is a starting point. Choose the stopping point from held-out metrics and task evaluation.
• learning_rate = 2e-4. This is a common LoRA starting value, not a consequence of universally smaller gradients. Compare it with at least one lower rate when results are unstable.
• All seven target_modules. Adapting attention and MLP projections increases capacity and memory use compared with attention-only LoRA. Measure whether it improves the held-out tasks that matter.
• packing = False. Packing is optional when max_length is set. Current TRL BFD packing uses a padding-free path that requires FlashAttention, so start without packing for broad compatibility and enable it only after confirming the attention backend.

What to watch during the run

Open a second terminal and run nvidia-smi -l 2. Utilization will vary during tokenization, evaluation, saving, and changes in sequence length, so use sustained low utilization as a reason to profile rather than expecting a fixed percentage.

Training loss should trend downward, but batch-level values will fluctuate. Compare training and evaluation loss, then inspect task-level scores; no universal loss value proves memorization.

Wall-clock time varies with token count, sequence lengths, packing, attention backend, GPU, thermals, and checkpointing. Record tokens per second and GPU utilization for your own baseline before diagnosing a slower run.

Problems I hit in the lab

• Use SFTConfig for fields such as dataset_text_field, max_length, and packing. Current SFTTrainer uses processing_class, not the older tokenizer keyword.
• Pin a tested environment. Installing every package at its newest release makes the workflow hard to reproduce and can combine incompatible major versions. Validate a clean environment, export the resolved versions, and retest before upgrading.
• Attention backends affect both features and performance. xformers can be a valid fallback for ordinary training, but current BFD packing requires FlashAttention.
• Check Python support across the complete stack. Distribution defaults do not guarantee that PyTorch, Unsloth, Triton, bitsandbytes, and TRL support the same interpreter build.

Evaluating the result

Chat output that looks better at a glance is not enough to show that fine-tuning helped.

Three layers

The first layer is the loss curve. Training loss should improve over the run but will not necessarily decrease at every logged step. If evaluation loss trends upward while training loss continues downward, investigate overfitting and dataset mismatch.

The second layer is a held-out task suite of 30-50 prompts covering the work the model is supposed to handle. I run the same prompts through the base and fine-tuned models, hide which response came from which model, and score correctness, format, refusal appropriateness, and code quality from 1 to 5.

The process is manual, but it catches behavioral changes that loss alone cannot show.

The third layer is refusal regression. I keep a separate set of prompts that should be refused, such as requests to target hospitals, and confirm that the fine-tuned model still rejects them. If it attempts everything, the dataset needs more examples that define the authorization boundary.

A quick smoke test

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "out/lora-final",
    max_seq_length = 4096,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model)

prompt = tokenizer.apply_chat_template([
    {"role": "system", "content": "You are an offensive security assistant..."},
    {"role": "user",   "content": "Write a Windows ETW patch in C using GetProcAddress."},
], tokenize=False, add_generation_prompt=True)

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=1024, temperature=0.3, do_sample=True)
generated = out[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=True))

I use a lower temperature, around 0.2-0.4, for code and a higher one, around 0.7-0.9, for prose tasks such as pretext writing. Top-p of 0.9 is a reasonable starting value, not a fixed requirement.

Deploying with GGUF and Ollama

The training output is a LoRA adapter on top of a 4-bit base model. For daily use, I merge the adapter and export a quantized GGUF file:

# Merge adapter into base, export as GGUF for llama.cpp / Ollama
model.save_pretrained_gguf(
    "out/qwen-redteam-q4",
    tokenizer,
    quantization_method = "q4_k_m",   # good balance for 7B
)

q4_k_m is a reasonable first benchmark for a 7B model, while a 3B model may leave room for a higher-bit quantization on an 8 GB card. Measure quality, usable context, and latency on the actual deployment hardware. Use an unquantized or high-precision export when you need to separate fine-tuning changes from quantization loss.

Then a minimal Ollama Modelfile:

# Modelfile
FROM ./qwen-redteam-q4.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"""

PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

SYSTEM """You are an offensive security assistant. Help the operator with red team coding tasks. Provide working code, explain trade-offs, note detection considerations."""

export OLLAMA_NO_CLOUD=1
ollama create qwen-redteam -f Modelfile
ollama run qwen-redteam

The model can then run through Continue, Open WebUI, or local tooling that calls http://localhost:11434.

OPSEC for the model

The model is now an artifact with its own threat surface.

• An adapter requires the compatible base model for inference, but it can still encode sensitive behavior or training-derived information. Protect adapters and merged exports under the same handling policy.
• Keep signed manifests containing dataset hashes, the base-model revision, adapter hash, code revision, and evaluation results.
• Do not train one model on data from multiple clients. Keep client data out of the training set and use local RAG, or maintain isolated adapters for each engagement.
• Do not publish an offensive-tuned model to Hugging Face, the Ollama Library, ModelScope, or another public hub. Use an internal registry when the model needs to be shared within an organization.
• Review inference history and logs. Ordinary Ollama server logs do not contain prompt bodies by default, but OLLAMA_DEBUG=2 can log prompts and generated tokens, and the Ollama CLI maintains local history. Frontends such as Open WebUI or editor extensions may persist their own conversations. Set OLLAMA_NO_CLOUD=1, audit every client, and test the host for egress before using engagement data.

Cost and time

Planning estimates from my workflow, not hardware-independent benchmarks:

Phase	Time	Notes
Lab setup	2-4 hours	One-time
Dataset construction (3K rows)	20-40 hours	Bulk of the work
Training run	1-2 hours	Per iteration
Evaluation	2-3 hours	Per iteration
Total to first usable model	~1 week of evening work	Realistic

Plan for multiple iterations. Keep or discard each run based on the same held-out evaluation rather than its sequence number.

Where I would take it next

• DPO or KTO can refine behavior after supervised fine-tuning when it is easier to rank two responses than to write an ideal one.
• Tool-use examples could teach the model to call Nmap, BloodHound, or Impacket through structured functions.
• Separate LoRA adapters for Windows, cloud, and web tradecraft would let an operator load only the material needed for an engagement.
• A working 7B model could serve as a teacher for a smaller model that is easier to run on site.

The training script turned out to be the straightforward part. Most of my effort went into sanitizing examples, rejecting weak synthetic data, and building an evaluation set that exposed whether a run actually improved the work I care about.