MoI-Bench assesses mobile image generation and editing models
MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating image editing and generation tools that run well on mobile phones is tricky because phones have less power than big computers. The authors created MoI-Bench, a test set designed around common mobile user needs, with many small tasks combining different editing actions and image types. They also made MoI-Eval, which carefully checks how well models follow instructions, keep image quality, and preserve details. By testing various models, they found specific areas where mobile models lag behind, like editing text in images. This helps developers focus improvements on what matters most for phone apps.
What this means in practice
- •For mobile app developers: Improve on-device image editing apps by targeting specific capability gaps like text replacement identified by MoI-Bench.
- •For mobile device engineers: Balance image model quality and deployment costs using the Quality-Deployability map to optimize resource use on mobile hardware.
Authors
Kailai Feng, Haoxiang Rao, Yuxiang Wei, Bo Chen, Junsong Li, Yang Pan, Chengkun Cai, Jagmohan Chauhan
Abstract
Developing on-device image models requires considering practical user needs, task types and device resource constraints. However, existing benchmarks provide limited guidance on identifying capability gaps between on-device models and server-side models. We introduce MoI-Bench, a benchmark grounded in mobile user needs, comprising 64 subtasks and 1150 samples. It defines 16 operations, such as removal and replacement, and eight visual target categories covering content such as portraits and text. These form two orthogonal axes whose applicable combinations define specific tasks, such as removing facial makeup. To better guide model improvement, we further propose MoI-Eval, which introduces task-specific criteria covering instruction following, visual quality and detail preservation. We further construct sample-level evaluation questions and aggregate their scores into overall quality and capability profiles along the operation, target and criteria axes. We evaluate representative on-device and server-side models and measure mobile deployment costs. Results show that this decoupled Operation x Target design provides finer-grained guidance for improving compact models. For example, it highlights the need to prioritize specific capability gaps (e.g., text-replacement) over broad application-level goals (e.g., improving text editing). Unlike conventional benchmarks that rank models using a single score, we introduce a two-dimensional Quality-Deployability capability map to assess models for practical mobile use. The map shows that current on-device models can achieve efficient generation or editing but expanding their task coverage deserves greater attention than further improving deployment efficiency. Further progress requires improving editing quality, particularly identity preservation and text rendering, while broadening task coverage.