Fine-Tuning InstructPix2Pix for Advanced Image Colorization

December 08, 2023 · Entered Twilight · 🏛 arXiv.org

Repo contents: .gitignore, README.md, W&B Chart 2023_11_29 23_04_53.png, W&B Chart 2023_11_29 23_05_08.png, arguments.py, dataset.py, export_to_hub.py, finetune.py, finetune_instruct_pix2pix.py, generate_dataset.py, ori_instructpix2pix.py, prompt.txt, requirements.txt, run_finetune.bat, run_finetune.sh, sample.py, stages.png, test.m, test_data, train_dreambooth_lora.py

Authors Zifeng An, Zijing Xu, Eric Fan, Qi Cao arXiv ID 2312.04780 Category cs.CV: Computer Vision Cross-listed cs.AI Citations 1 Venue arXiv.org Repository https://github.com/AllenAnZifeng/DeepLearning282 ⭐ 17 Last Checked 3 months ago

Abstract

This paper presents a novel approach to human image colorization by fine-tuning the InstructPix2Pix model, which integrates a language model (GPT-3) with a text-to-image model (Stable Diffusion). Despite the original InstructPix2Pix model's proficiency in editing images based on textual instructions, it exhibits limitations in the focused domain of colorization. To address this, we fine-tuned the model using the IMDB-WIKI dataset, pairing black-and-white images with a diverse set of colorization prompts generated by ChatGPT. This paper contributes by (1) applying fine-tuning techniques to stable diffusion models specifically for colorization tasks, and (2) employing generative models to create varied conditioning prompts. After finetuning, our model outperforms the original InstructPix2Pix model on multiple metrics quantitatively, and we produce more realistically colored images qualitatively. The code for this project is provided on the GitHub Repository https://github.com/AllenAnZifeng/DeepLearning282.