Submit to ImagenMuseum

To ensure the transparency of the evaluation experiment, we strongly recommend researchers to submit their methods on ImagenMuseum.

ImagenMuseum allows researchers and general public to understand the actual performance of models beyond the metrics.

You can run your method according to our ImagenHub dataset and save the image as the sample_{uid}.jpg (or sample_{img_id}_{turn_index} for Mask/Text-guided Editing).

Text-guided Image Generation Benchmark Dataset

https://huggingface.co/datasets/ImagenHub/Text_to_Image

from datasets import load_dataset
infer_dataset = load_dataset("ImagenHub/Text_to_Image")['eval']

Mask-guided Image Editing Benchmark Dataset

https://huggingface.co/datasets/ImagenHub/Mask_Guided_Image_Editing

from datasets import load_dataset
infer_dataset = load_dataset("ImagenHub/Mask_Guided_Image_Editing")['filtered']

Text-guided Image Editing Benchmark Dataset

https://huggingface.co/datasets/ImagenHub/Text_Guided_Image_Editing

from datasets import load_dataset
infer_dataset = load_dataset("ImagenHub/Text_Guided_Image_Editing")['filtered']

Subject-driven Image Generation Benchmark Dataset

https://huggingface.co/datasets/ImagenHub/Subject_Driven_Image_Generation

from datasets import load_dataset
train_dataset = load_dataset("ImagenHub/DreamBooth_Concepts")['train']
infer_dataset = load_dataset("ImagenHub/Subject_Driven_Image_Generation")['eval']

Subject-driven Image Editing Benchmark Dataset

https://huggingface.co/datasets/ImagenHub/Subject_Driven_Image_Editing

from datasets import load_dataset
train_dataset = load_dataset("ImagenHub/DreamBooth_Concepts")['train']
infer_dataset = load_dataset("ImagenHub/Subject_Driven_Image_Editing")['eval']

Multi-concept Image Composition Benchmark Dataset

https://huggingface.co/datasets/ImagenHub/Multi_Subject_Driven_Image_Generation

from datasets import load_dataset
train_dataset = load_dataset("ImagenHub/Multi_Subject_Concepts")['train']
infer_dataset = load_dataset("ImagenHub/Multi_Subject_Driven_Image_Generation")['train']

Control-guided Image Generation Benchmark Dataset

https://huggingface.co/datasets/ImagenHub/Control_Guided_Image_Generation

from datasets import load_dataset
infer_dataset = load_dataset("ImagenHub/Control_Guided_Image_Generation")['eval']

Submission Guide

You can run your method according to our ImagenHub dataset and save the image as the sample_{uid}.jpg (or sample_{img_id}_{turn_index} for Mask/Text-guided Editing).

Your submission format has to be a zip file that contains:

YourMethod
--- id.jpg
...

Example:

DALLE3
--- sample_0.jpg
--- sample_100.jpg
--- sample_101.jpg
--- sample_102.jpg
--- sample_103.jpg
--- sample_104.jpg
...

All the experiments has to be done with the requirements:

Seed fixed to 42
No hand-crafted prompt engineering
Person who submit the method has to be one of the authors of corresponding work, unless you integrated the code on ImagenHub codebase.

Feel free to submit your authored method results to m3ku@uwaterloo.ca with Title “ImagenMuseum Submission: {Your Method Name}”. We will be updating your result within 1 week.