Papers for

digital content producers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

ProtoDiffusion speeds up layered graphic design generation and editing

ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images

Abstract: Graphic designs are ordered compositions of RGBA layers whose separation enables selection, movement, and reuse. Existing layered models represent each layer with a full-canvas or region-sized spatial latent grid, so representation size grows with the cumulative spatial extent of the layers. We present ProtoDiffusion, a latent-diffusion framework built on ProtoVAE, a multi-layer variational autoencoder that encodes a design into one shared spatial prototype and one candidate vector per layer. Each candidate conditions a shared decoder to reconstruct its corresponding RGBA layer. For $N$ layers, the representation contains $HW+N$ tokens before backbone-specific packing, compared with $NHW$ for full-grid-per-layer representations. ProtoDiffusion jointly denoises the prototype and a fixed set of candidate slots, conditioned on an image or text, with bounding boxes as optional layout guidance. The model leaves unneeded slots empty and determines the number of layers and, in absence of boxes, their layout, requiring neither as input. The same architecture supports both image-to-design and text-to-design. ProtoVAE achieves high-fidelity reconstruction from its compact shared latent representation, reaching 40.43 dB PSNR on layer crops. On the LayerD evaluation split, image-to-design achieves RGB L1 scores of 0.040 with bounding boxes and 0.057 without, with box-guided alpha soft IoU (0.850) comparable to the strongest fine-tuned baseline (0.871). On merged images, text-to-design reaches 66.41 FID versus 86.84 for FLUX.2 [klein], achieves a 0.860 text-accuracy score, and runs $19.1\times$ faster than the same-backbone generation-then-decomposition cascade.

Thu 19 NovArtificial Intelligence
The gist
Creating graphics often involves stacking layers of images and colors, which can get large and complex. The authors developed ProtoDiffusion, a new way to represent and generate these layers more efficiently by sharing some parts of the image and only separately handling the parts that change. This approach makes it faster to create, edit, and rearrange layers from images or text descriptions. It also figures out how many layers are needed and where they go without extra instructions, making design generation more flexible.
Open → 2611.26586v1

JointREPA improves audio and video matching in underspecified prompts

Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation

Abstract: Our aim is to advance the understanding of genuine audio-visual coordination in joint text-to-audio-video (T2AV) generation, with particular emphasis on semantic consistency. Evaluating audio-visual consistency in T2AV generation is nontrivial, as detailed text prompts obscure whether such consistency stems from genuine cross-modal coordination or merely from independent text following. To isolate the former, we introduce intrinsic jointness: the ability of audio and video to produce the same realization for details left underspecified by the prompt. Using semantically underspecified prompts (e.g., "musical instrument playing" rather than "electric guitar playing"), we investigate four recent models and reveal their limited intrinsic jointness. To address this limitation, we propose JointREPA, a simple representation alignment-based post-training method to facilitate cross-modal coordination. Since existing benchmarks use semantically detailed prompts, we introduce UNSAIDBench, a benchmark designed to evaluate intrinsic jointness under semantic underspecification. Experiments across two distinct models and three benchmarks, including UNSAIDBench, demonstrate that JointREPA consistently improves semantic consistency.

Tue 17 NovComputer Vision and Pattern Recognition
The gist
Generating videos with matching sounds based on text descriptions is tricky when the description is vague. The authors studied how current AI models struggle to truly link the sounds and visuals when details are missing from the text. They introduced a new way to measure this 'jointness' and created a method called JointREPA to help improve it. Their new benchmark, UNSAIDBench, tests how well models can match audio and video when the text doesn’t specify everything. Results show JointREPA helps models make sounds and images that fit together better.
Open → 2611.22857v1

Compensating creators for AI training can increase their replacement

Paying Creators for Training Data Can Increase AI Displacement

Abstract: Generative AI increasingly competes with creators while using their content as training data. In response, policymakers have called for platforms to compensate creators for training data use and allow opting out. We study how such policies affect creator displacement in a model of a platform where creators choose whether to contribute training data and consumers choose between purchasing from creators and prompting generative AI trained on their content. When compensation is required, the low-quality creator chooses to remain solely as a training data supplier while AI fully displaces her demand, despite AI quality and the price the platform charges for AI-generated content being identical with and without compensation. Without compensation, the risk of losing creators and their training data constrains the platform to set the price of creators' content to preserve their demand. This constraint also prevents further AI displacement as AI improves, with the platform raising AI prices while keeping creators' prices and demand unchanged to prevent exit. Compensation increases creators' payoffs and total surplus but reduces consumer surplus. Our findings highlight how compensation policies for training data can shape creators' economic roles in a future with more capable generative AI.

Sun 15 NovComputer Science and Game TheoryArtificial Intelligence
The gist
Generative AI systems learn from content created by people, but sometimes this AI competes with those same creators. The authors show that if platforms pay creators for using their work to train AI, some creators might stop selling their own work and only provide data for AI training, letting AI take over their customers. Without paying for training data, platforms keep creators’ prices low so they keep selling, limiting AI’s takeover. Paying creators more helps them earn more but means customers might pay more and get less overall benefit.
Open → 2611.20204v1

Nawe combines neural networks with signal processing for better digital watermarking

NAWE: Digital Watermarking with Neural-Assisted Watermark Extraction

Abstract: NAWE (Neural-Assisted Watermark Extraction) combines an explicit signal-processing watermarking construction with a pretrained neural host predictor. A periodic, perceptually masked watermark carrier provides synchronization, Polar coding supplies redundancy, and denoising followed by subtraction extracts the embedded watermark. The denoiser remains frozen, without watermark-specific training. A one-factor-at-a-time study compares Wiener, BM3D, DRUNet, and GS-DRUNet host estimators. Comparisons with TrustMark, SSL Watermarking, PixelSeal, and WAM show NAWE's lowest geometric and photometric class BER and strong message recovery, while filtering and noise remain limitations consistent with the non-adaptive selection of the watermark extractor. The comparison retains the systems' different payloads and coding.

Tue 22 SeptComputer Vision and Pattern RecognitionMultimedia
The gist
Digital watermarking hides secret information in images to prove ownership or authenticity. The authors propose NAWE, a method that mixes traditional signal processing with a neural network trained to predict the original image, helping to better extract the hidden watermark. Their approach uses a special pattern and error-correcting codes to reliably recover the message, even after the image is altered. They tested several neural network models and found their method recovers watermarks more accurately than some existing systems, though noise and filtering still create challenges.
Open → 2609.25972v1

Human AI co creativity highlights progress and future challenges

Human-AI Co-Creativity: Advances, Opportunities, and Challenges

Abstract: This survey article has grown out of the human-AI co-creativity workshop organized by the authors at the ICML 2026 conference. We organized this workshop as part of a community-building effort to bring together researchers and practitioners interested in topics of generative AI, creativity, and human-AI co-creation. This article aims to provide an overview of the workshop activities and highlight several future research directions in the area of human-AI co-creativity.

Mon 7 SeptComputers and Society
The gist
Making art or creative work together with AI is becoming more common, but it can be tricky to do well. The authors held a workshop to bring people interested in this topic together and then wrote this article summarizing what was learned. They explain the current progress in human-AI collaboration for creativity and point out areas that need more work. This helps everyone understand what’s possible now and what challenges remain.
Open → 2609.07711v1