README.md

---
base_model: Qwen/Qwen2.5-VL-3B-Instruct
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- qwen2.5-vl
- lora
- sft
- context-classification
- out-of-context-detection
- coinco
license: cc-by-4.0
---

# COinCO Context Classification Models

**Authors:** Tianze Yang\*, Tyson Jordan\*, Ruitong Sun\*, Ninghao Liu, Jin Sun
\*Equal contribution
**Affiliation:** University of Georgia

## Overview

Fine-grained context classification models for detecting **out-of-context objects** in images. Each model is a fully merged Qwen2.5-VL-3B-Instruct fine-tuned via LoRA on the [COinCO dataset](https://huggingface.co/datasets/COinCO/COinCO-dataset).

The models classify whether an object (marked by a red bounding box) is **in-context** or **out-of-context** based on three criteria:

| Model | Criterion | Description |
|-------|-----------|-------------|
| `co_occurrence/` | Co-occurrence | Whether the object can reasonably appear together with other objects in the scene |
| `location/` | Location | Whether the object is placed in a physically and contextually reasonable position |
| `size/` | Size | Whether the object's size is proportional and realistic relative to other objects |

## How to Use

```python
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
import torch

# Choose a model: "co_occurrence", "location", or "size"
model_id = "COinCO/Context_Classification_Models"
subfolder = "co_occurrence"  # or "location" or "size"

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    subfolder=subfolder,
    torch_dtype=torch.float16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id, subfolder=subfolder)
```

## Training Details

- **Base Model:** [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct)
- **Method:** LoRA fine-tuning (merged into base model)
- **Dataset:** [COinCO](https://huggingface.co/datasets/COinCO/COinCO-dataset) inpainted images with multi-model consensus labels
- **Training Data:** ~5,000 samples per criterion from the training split
- **Epochs:** 3
- **Learning Rate:** 2e-4
- **LoRA Rank:** See adapter config for details

## Evaluation Results

### Inpainted Test Set (binary classification: In-context vs Out-of-context)

| Criterion | Baseline (Qwen2.5-VL-3B) | Fine-tuned | Improvement |
|-----------|--------------------------|------------|-------------|
| Co-occurrence | 75.54% | **80.82%** | +5.28% |
| Location | 74.43% | 71.05% | -3.38% |
| Size | 50.21% | **66.01%** | +15.80% |

### Real COCO Images (shortcut learning detection, higher = less shortcut reliance)

| Criterion | Baseline | Fine-tuned | Improvement |
|-----------|----------|------------|-------------|
| Co-occurrence | 88.95% | 87.00% | -1.95% |
| Location | 47.55% | **91.35%** | +43.80% |
| Size | 52.55% | **83.20%** | +30.65% |

## Related Resources

- **Paper:** "Common Inpainted Objects In-N-Out of Context"
- **Dataset:** [COinCO/COinCO-dataset](https://huggingface.co/datasets/COinCO/COinCO-dataset)
- **Code:** [YangTianze009/COinCO](https://github.com/YangTianze009/COinCO)

## Citation

```bibtex
@article{yang2025coinco,
  title={Common Inpainted Objects In-N-Out of Context},
  author={Tianze Yang and Tyson Jordan and Ruitong Sun and Ninghao Liu and Jin Sun},
  year={2025}
}
```