Gecko: Guidance from Keyword for Identifying Small Details in High-Resolution Images with Vision-Language Models

Undergraduate Thesis

Farros Alferro, Masanori Suganuma, Takayuki Okatani,
▶ Tohoku University
Thesis Code Model Benchmark

🔥 We propose plug-and-play modules that can be attached to a Vision Language Model, enabling the model to effectively identify small objects in high-resolution images. Our models achieve competitive results on the V* Bench compared to both open-source and proprietary models.

Method Attribute (%)↑ Spatial (%)↑ Overall (%)↑ Inference Time (sample/s)↓ Training External Models
Human 98.26 100.00 98.95 – – –
Random Guess 26.73 50.00 35.99 – – –
Open‑source models
LLaVA‑1.5 44.35 52.63 47.64 0.314 ✓ ×
Mantis‑CLIP 35.65 61.84 46.07 0.257 ✓ ×
Mantis‑SigLIP 33.04 59.21 43.46 0.295 ✓ ×
High resolution models
Mini‑Gemini 8B 60.87 63.16 61.78 1.218 ✓ ×
Dragonfly 8B 49.57 52.63 50.79 2.865 ✓ ×
Idefics2 8B 54.78 56.58 55.49 1.657 ✓ ×
LLaVA‑NeXT 8B 60.87 61.84 61.26 1.149 ✓ ×
LLaVA‑HR 46.09 52.63 48.69 0.554 ✓ ×
Tool‑using models
SEAL (SOTA) 77.39 77.63 77.48 94.25 âś“ âś“
Private models
GPT‑4V Preview 42.61 46.05 43.98 – – –
GPT‑4 Turbo 46.09 53.95 49.21 – – –
GPT‑4o 69.57 52.63 62.83 – – –
GPT‑4o Mini 49.57 63.16 54.97 – – –
Ours‑CLIP 52.17 53.95 52.88 5.268 × ×
Ours‑SigLIP 63.48 61.84 62.83 5.479 × ×

Abstract

Recent vision and language models (VLMs) have demonstrated remarkable success, even surpassing human performance in certain tasks. However, these models struggle with recognizing small objects in high-resolution images. To address this limitation, a previous study introduced a visual search method where VLMs iteratively search for clues to identify small target objects using fine-tuned vision modules. While this approach improves small object recognition, it incurs high inference times due to its iterative operations, extensive use of vision modules, and significant computational costs from training.

We propose a plug-and-play method that leverages pre-trained model weights without requiring external tools or additional training. This method capitalizes on the vision encoder’s inherent ability to contrast images and text, enabling it to selectively focus on image areas relevant to the queries. Consequently, this approach streamlines the process and reduces resource consumption.

Gecko Framework

Gecko is a plug-and-play framework that allows Vision Language Model (VLM) to identify small objects in high-resolution images. As shown in below figure, Gecko can be summarized as follows:

  1. Image Cropping: Images are divided into patches that match the native resolution of the vision encoder. This ensures the preservation of fine details, especially those of small objects.
  2. Keyword Extraction: We use in-context learning with a native LLM to extract the "queried object" from the provided prompt.
  3. Patch Selection: The vision encoder and its text encoder counterpart leverage contrastive ability to identify and select the most relevant patches.
  4. LLM Inference: The selected patches are then passed to the LLM to generate a response to the given prompt. We use Mantis for its ability to handle multiple images effectively.
Proposed Framework
Figure: Overview of the proposed Gecko framework. For more details, please refer to the Thesis.

Multi V* Bench

We adapted the Gecko framework to allow the model to find the small objects across multiple images. To test this, we curated a new benchmark called Multi V* Bench, which includes multi-image, high-resolution visual question and answer tasks with an average resolution of 1920 Ă— 2475 pixels. The benchmark consists of four tasks, each comprising 20 samples:

  • Attribute: Identify attributes differences of an object between images. There are 43 images in total with resolution of 1920 Ă— 2622 pixels.
  • Existence: Determine if an object exists between the images. There are 47 images in total with resolution of 1920 Ă— 2419 pixels.
  • Counting: Count the number of objects among the images. There are 56 images in total with resolution of 1919 Ă— 2467 pixels.
  • OCR: Extract or identify text from the images. There are 55 images in total with resolution of 1920 Ă— 2414 pixels.
Multi Image Setup
(a) Figure with resolution of (1920, 2880) px.
Patch Selection and Reasoning
(b) Figure with resolution of (1920, 2880) px.
Figure: One example of attribute task from Multi-V* Bench. The prompt: ”What is the color difference of the motorcycle between these images?” For more details, please refer to the Thesis.

Result

Method Attribute (%)↑ OCR (%)↑ Existence (%)↑ Count (%)↑ Overall (%)↑ Inference Time (sample/s)↓
Open‑source models
Mantis‑CLIP ‡ 35.0 20.0 35.0 30.0 30.0 0.749
Mantis‑SigLIP ‡ 50.0 50.0 25.0 50.0 43.75 0.774
High resolution models
LLaVA‑NeXT‑Interleave 50.0 45.0 45.0 35.0 43.75 0.852
Private models
GPT‑4V Preview 30.0 60.0 45.0 25.0 40.0 –
GPT‑4 Turbo 35.0 60.0 45.0 20.0 40.0 –
GPT‑4o 70.0 65.0 55.0 55.0 61.25 –
GPT‑4o Mini 45.0 65.0 50.0 15.0 43.75 –
Ours‑CLIP 60.0 50.0 45.0 45.0 50.0 8.099
Ours‑SigLIP 45.0 75.0 45.0 40.0 51.25 7.194

BibTeX


  @misc{alferro2024gecko,
          author={Alferro, Farros and Suganuma, Masanori and Okatani, Takayuki},
          title={Guidance from Keyword for Identifying Small Details in High-Resolution Images with Vision-Language Models}, 
          publisher={https://github.com/farrosalferro/gecko},
          year={2024},
  }

  @inproceedings{Jiang2024MANTISIM,
    title={MANTIS: Interleaved Multi-Image Instruction Tuning},
    author={Dongfu Jiang and Xuan He and Huaye Zeng and Cong Wei and Max W.F. Ku and Qian Liu and Wenhu Chen},
    publisher={arXiv2405.01483}
    year={2024},
  }

Acknowledgement

This website is adapted from Nerfies, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. We thank the Mantis team for open-sourcing such wonderful models and LLaMA team for giving us access to their models.