Picsart Introduces Open-Source AI Art Tool With Firefly-Like Abilities

[Community]

Video screenshots via Picsart

Picsart’s AI Research team has introduced HD Painter, an open-source artificial intelligence project aimed at enhancing the precision and quality of image inpainting with high-resolution, realistic results through text-to-image diffusion models.

Image inpainting, a computer vision technique, helps reconstruct missing or unwanted parts of images in a seamless manner. Whether it’s a spot, a tear, or an object in a photograph, inpainting intelligently fills these gaps using the surrounding image data. 

However, Picsart asserts that existing methods by software like Stable Diffusion, Google Imagen, and DALL-E 2 often lack precision and might not always align with the user’s intentions. For example, if you tell the program to add a “cat” to an empty part of the photo, it might somehow miss that—engaging in something called prompt neglect—and just fill in with more background or other objects from the image.

HD Painter steps up to this challenge by being more sensitive to and executing text instructions or prompts, thus completing the missing parts of an image accurately and in high resolution.

Those who have experimented with Generative Fill in Photoshop, powered by Adobe’s generative AI model Firefly, would be familiar with this concept. You first select or draw over an area of the image you want to edit or improve. This could be removing an unwanted object, filling in missing parts, or changing something existing in the shot. Then, enter a text prompt to describe what you want in the selected area. The AI analyzes the surrounding area of the selected part to understand the texture, color, and patterns of the nearby image data—crucial for making the new part blend in naturally with the rest.

Video screenshots via Picsart

Central to HD Painter are two innovative technologies: Prompt-Aware Introverted Attention (PAIntA) and Reweighting Attention Score Guidance (RASG). PAIntA improves the model’s attention to user’s text prompts, ensuring that the reconstructed area closely matches the instructions. Meanwhile, RASG adjusts the inpainting process to align precisely with the text prompts, leading to higher quality and accurate results.

A notable advantage of HD Painter is its capability to achieve up to 2K resolution inpainting, offering a substantial improvement in clarity and detail over other methods. This feature ensures that the final images are not only complete but also sharp and refined.

Interestingly, HD Painter employs a “training-free” methodology—meaning it only utilizes pre-trained models and adaptable components, with no additional data needed. This minimizes the need for additional training and computational costs and time, while enabling the tool to be applied to any existing diffusion-based inpainting model.

Find more about the feature here.