Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models

Abstract

We introduce Playground v3 (PGv3), our latest text-to-image model that achieves state-of-the-art (SoTA) performance across multiple testing benchmarks, excels in graphic design abilities and introduces new capabilities. Unlike traditional text-to-image generative models that rely on pre-trained language models like T5 or CLIP text encoders, our approach fully integrates Large Language Models (LLMs) with a novel structure that leverages text conditions exclusively from a decoder-only LLM. Additionally, to enhance image captioning quality—we developed an in-house captioner, capable of generating captions with varying levels of detail, enriching the diversity of text structures. We also introduce a new benchmark CapsBench to evaluate detailed image captioning performance. Experimental results demonstrate that PGv3 excels in text prompt adherence, complex reasoning, and accurate text rendering. User preference studies indicate the super-human graphic design ability of our model for common design applications, such as stickers, posters, and logo designs. Furthermore, PGv3 introduces new capabilities, including precise RGB color control and robust multilingual understanding.

1 Introduction

Great progress has been made in the text-to-image generative models since last year [41, 50, 52, 15, 54]. The model architecture for large-scale text-to-image model is shifting from a traditional UNet-based[41] model to a transformer-based model[44] due to its scalability and simplicity. In this work, we built on this progress to develop a new DiT-based diffusion model and scale it to 24B parameters. We further propose a Deep-Fusion architecture to leverage the knowledge of the modern decoder-only large-language models [13] for the task of text-to-image generation.

In section 2 we describe the simplified model architecture, noise scheduling, and the new Variational Autoencoder (VAE). Our novel text-to-image model structure features a deep integration of an LLM, fully leveraging the LLM’s internal prompt understanding to achieve state-of-the-art prompt-following performance. Section 3 discusses training details, including the use of multi-level captions and model merging during the post-training phase. Section 4 introduces our in-house captioner and a new captioning benchmark CapsBench.

In section 5, we present qualitative examples of our model’s capabilities across five key aspects: photo-realism, prompt-following, text-rendering, RGB color control, and multilingual understanding. Most of the images displayed are generated from our evaluation testing prompt set, which was curated through our human-in-the-loop pipeline and excluded from our training set.

In section 6, we quantitatively evaluate our image models across four categories. Notably, we demonstrate that PGv3 exhibits superior graphic design capabilities, often surpassing even real human designers. We address image-text alignment and reasoning using the LLM-aided DPG-benchmark [24]. To further validate this, we developed an in-house version of the DPG-benchmark with more complex testing prompts and used GPT-4o as the VQA evaluator,; this new benchmark demonstrates PGv3’s SoTA prompt-following performance. Additionally, we briefly showcase the reconstruction quality of our new VAE. Finally, we assess image quality using standard benchmarks such as ImageNet and MSCOCO. For our in-house captioner model, details and results of the CapsBench are discussed in section 7.

2 Methods

2.1 Model

Playground-v3 (PGv3) is a Latent Diffusion Model (LDM) [52] trained using the EDM [27] formulation. Like other models such as DALL-E 3 [4], Imagen 2 [54], and Stable Diffusion 3 [15], PGv3 is designed for text-to-image (t2i) generation tasks. However, it distinguishes itself through a novel model structure, diverging from the commonly-used text encoders like T5 [49] or CLIP [48], which we consider to be insufficient. Instead, PGv3 fully integrates a large language model (LLM), specifically Llama3-8B [13], to enhance its capabilities in prompt understanding and following.

There have been a lot of research efforts aimed at leveraging LLMs to improve t2i models’ prompt-following abilities. These efforts typically involve either using LLMs as the text encoder to replace the commonly-used T5 and CLIP, or employing LLMs to adapt or revise prompts for a pretrained T5/CLIP-based t2i model [19, 56, 47, 24, 38, 3]. In PGv3, we adopt the former approach, i.e. we don’t use any NLP encoders like T5 and CLIP, and solely rely on Llama3-8B to provide text conditioning to our diffusion model, from the initial stage of the model training.

2.1.1 Text Encoder

We take a notably different approach to using a pre-trained text model. The standard practice has been to use the output from the final layer of the T5 encoder or CLIP text encoder, with some research also finding value in incorporating the penultimate layer’s output. It is well-established that each layer within a transformer model captures different representations, containing varying levels of word-level and sentence-level information [14, 12, 51]. However, we found that selecting the optimal layers for conditioning the t2i model can be cumbersome, particularly when using a decoder-style LLM, which arguably possesses superior generative power due to its more complex internal representations.

We believe that the continuity of information flow through every layer of the LLM is what enables its generative power and that the knowledge within the LLM spans across all its layers, rather than being encapsulated by the output of any single layer. Motivated by this, we designed our t2i model structure to replicate all the transformer blocks of an LLM. This design allows our model to take the hidden embedding outputs from each corresponding layer from LLM as conditioning at each respective layer of our t2i model. We argue that this approach fully leverages the complete "thinking process" of the LLM, guiding our t2i model to mirror the LLM’s reasoning and generative process. As a result, our model achieves unparalleled prompt-following and prompt-coherence abilities when generating images.

2.1.2 Model Structure

Figure 2: PGv3 model structure overview simplified

PGv3 employs a DiT-style [44] model structure. Each transformer block in our image model is set up as identical to the corresponding block in the LLM we utilize, Llama3-8B in this case, including matching parameters such as hidden dimension size, number of attention heads, and attention heads dimensions. We trained only the image model component, leaving the LLM untouched. Note that during the diffusion sampling process, the LLM part of the network only needs to run once to generate all the intermediate hidden embeddings required.

The simplified model block is illustrated in Fig.2. Each transformer block consists of only one attention layer and one feed-forward layer, mirroring the structure of Llama3. Unlike most traditional CNN-based diffusion models [50, 51], which typically separate self-attention on image features from cross-attention between image and text features, we performed a single joint attention operation. In this approach, the image query attends to a concatenation of image keys and text keys, allowing it to extract relevant features from a combined pool of image and text values. This joint attention design is primarily motivated by the desire to reduce computational costs and inference time.

Although we finalized our model design and began training before the publication of the SD3 paper [15], we have observed that the PGv3 transformer block shares a similar spirit with the MMDiT proposed in SD3. When considering the corresponding LLM block and image block as a single layer in PGv3, our approach reflects a similar philosophy to MMDiT—enabling integrated and continuous interaction between image and text features across all model layers. However, our design is more minimalist and leverages the text encoder more thoroughly.

There have been many research efforts aimed at exploring better model structures, such as [35, 28, 8]. We experimented with several structural tweaks for PGv3 but did not observe significant benefits from most of them. Here we report on some designs that we found useful and incorporated into our final model design.

U-Net skip connections between transformer blocks. We applied U-Net [53] skip connection across all the image blocks, as previously tried in U-DiT[58].

Token down-sampling at middle layers. Among the 32 layers, we reduced the sequence length of the image keys and values by four times in middle layers making the whole network resembles a traditional convolution U-Net with only one level of down sampling. This adjustment slightly speeds up both training and inference time, and we did not observe any performance degradation as a result.

Positional embedding. We used the traditional Rotary Position Embedding (RoPE) [57], similar to what is used in Llama3. Since we work with 2-dimensional image features, we employed a 2D version of RoPE. Several prior works have explored variations of 2D-RoPE; in our study, we primarily experimented with two types, which we refer to as “interpolating-PE" and “expand-PE."

The “interpolating-PE" approach keeps the start and end positional IDs fixed, regardless of sequence length, and interpolates the positional IDs in between—an idea employed by models like SD3 and others. The “expand-PE" method, on the other hand, increases the positional ID proportionally with sequence length, without applying any tricks or normalization (i.e., vanilla positional embedding).

We found that the “interpolating-PE" had a significant downside: it caused the model to overfit heavily on training resolutions and failed to generalize to unseen aspect ratios. In contrast, the “expand-PE" method performed well, showing no signs of resolution overfitting. Therefore, we opted to use the traditional positional embedding in the end.

2.1.3 New VAE

The Variational Autoencoder (VAE) [30] used in Latent Diffusion Models (LDM) plays a crucial role in determining the model’s upper bound for fine-grained image quality. In the work of Emu [11], they increased the VAE latent channels from 4 to 16, which resulted in an enhanced ability to synthesize finer details, especially for smaller faces and words. Following this approach, we trained our in-house VAE with 16 channels. Additionally, instead of training solely at a $256*256$ resolution, we extended the training to a $512*512$ resolution, which further enhanced reconstruction performance.

We attempted to incorporate the frequency method proposed in Emu’s VAE training but did not observe any performance improvement. Additionally, we explored more aggressive VAE modifications, including using pretrained image encoders like the CLIP image encoder or the DINOv2 encoder [42]. Unfortunately, although these approaches yielded VAEs with comparable reconstruction abilities, the latent spaces generated by these pretrained image encoders, whether frozen or fine-tuned, resulted in unstable diffusion training, therefore we did not pursue this exploration further.

Figure 3: Multi-level caption example, randomly sampled from our training data set.

3 Model Training

Noise schedule. Despite the recent rise of flow-matching models [36, 39, 15], we continue to use the EDM schedule [27], as we did in PGv2.5. There is no specific reason for not transitioning to flow-matching; we simply have not encountered any performance limitations with the EDM schedule in our new PGv3 model structure. In the mean time, we did experiment with the dynamic timestamp weighting technique proposed in EDM2 [28], which is tailored to the EDM schedule. Unfortunately, we did not observe any significant performance improvements. We suspect this may be due to differences in model size and data, as our training scale is much larger and more diverse than the ImageNet training setup used in EDM2.

Multiple Aspect-Ratio Support. Following our previous work in PGv2.5 [33], we adopted the multi-aspect ratio training strategy. We started with square image training at $256*256$ low-resolution and used the online bucketing strategy at the higher resolution $512*512$ and $1024*1024$ pixel scales.

Multi-Level Captions per Image. We developed an in-house Vision Large Language Model (VLM) captioning system capable of generating highly detailed descriptions of images, including small objects, lighting, style, and visual effects. We ran benchmarks on this in-house captioner (PG Captioner) against other state-of-the-art systems, such as GPT-4o. Our results show that our model outperforms others across multiple image-specific categories, which will be discussed in detail later.

Previous studies [4, 15] have proposed re-captioning datasets and using machine-generated captions as text conditions to enhance model performance. In our work, we further improved these captioning conditions by generating multi-level captions to reduce dataset bias and prevent model overfitting. As shown in Fig.3, for each image, we synthesized captions of six different lengths, ranging from fine-grained details to coarse concepts. During training, we randomly sampled one of these six captions for each image at different iterations.

The idea behind using multi-level captions is to help the model learn a better linguistic concept hierarchy. This allows the model to not only learn the relationship between words and images but to also understand semantic relationships and hierarchies among words, by viewing the same image with varying levels of conceptual detail. As a result, our model demonstrates greater diversity when responding to short prompts involving general concepts, and adheres more closely to prompts when provided with highly detailed descriptions. Moreover, when training on datasets with fewer data samples, such as during the supervised fine-tuning (SFT) stage, multi-level captions helped prevent overfitting and significantly enhanced the model’s ability to generalize the good image properties of the SFT dataset to other image domains. As shown in Fig.4, when given a short and simple prompt, our model can generate diverse content, varying in style, color tone, ethnicity, and other attributes.

Figure 4: Image generation from PGv3 with a simple short prompt

Training stability. We observed loss spikes during the later stages of model training. This behavior manifested as the training loss becoming unusually large without resulting in NaN values, causing the model to enter a state where it could no longer generate meaningful content. We tried several strategies to address this issue, including lowering the betas of the Adam optimizer [29], using gradient clipping, and reducing the learning rate. However, none of these methods proved effective. Ultimately, we found that the following approach helped mitigate the issue.

During training, we looped through the gradients of all model parameters and counted how many gradients exceeded a specific gradient-value threshold. If this count surpassed a predefined count threshold, we discarded the training iteration, meaning we do not update the model weights using the gradients. We developed this solution after observing training behavior around the time of a loss spike. Specifically, we noticed that the number of large-valued gradients began to increase a few iterations before a spike occurred, peaking at the spike iteration. By recording the count of large-valued gradients many iterations before a spike, we set that value as a threshold to trigger the abandonment of a training iteration.

This approach differs from gradient clipping, which we found insufficient to prevent the model weights from being adversely affected by problematic gradients. Gradient clipping only reduces the magnitude of the gradients but does not alter the direction of the weight updates. As a result, the weights could still move toward collapse. By discarding the entire batch of gradients, we avoided this risk and ensured the model remained on a stable training trajectory.

A breathtaking night landscape photograph captures a lone figure standing atop a jagged rock formation on a dark sandy beach with scattered pebbles, gazing up at the magnificent Milky Way galaxy spanning diagonally across the sky. The foreground features a rough-textured rock with visible striations and crevices, its weathered surface in shades of brown and gray. The midground showcases the person dressed in a checkered shirt and pants, silhouetted against the soft glow of ambient light from an unseen horizon. To the left stands a massive, rugged cliff with uneven surfaces, visible cracks, and patches of green vegetation clinging to its slopes. On the right side of the midground, another distinctive rock structure extends out into the calm sea, partially submerged and silhouetted against the starry backdrop. The background is dominated by the Milky Way’s vibrant display of colors, including deep navy blue, emerald green, lavender purple, burnt orange, and white hues. A subtle aurora-like effect in pale yellow and green adds depth to the scene below the stars. The water appears smooth and glassy due to a long exposure technique, creating a serene atmosphere with gentle ripples reflecting the faint light above. Small rocks are scattered across the surface near the shore. Distant lights hint at nearby human activity along the coastline. The overall mood is awe-inspiring and tranquil, enhanced by the high contrast between the dark elements and the brightly illuminated stars, creating a harmonious blend of natural beauty and cosmic wonder.

Figure 5: Qualitative example of our in-house captioner.

4 Captioning Model

4.1 PG Captioner

For the captioning model, we used a standard vision-language architecture that consists of a vision encoder, a vision-language adapter, and a decoder-only language model [37, 34, 9]. Following recent works [5], we confirmed that a high-resolution vision encoder is very important for detailed captioning of the images. We also found an inverse relationship between the model size and the data volume required to train it to a reasonable quality: small models require much more data to produce coherent long captions compared to larger models. For model training, we used synthetic data generation workflows [55, 21, 61] and iterative self-improvement [17].

4.2 CapsBench

Captioning evaluation is a complex problem. There are two main classes of captioning evaluation metrics. The first one is reference-based metrics - BLEU [43], CIDEr [59], METEOR [2], SPICE [1]. These metrics use a ground truth caption or a set of captions to calculate similarity as a quality metric. The problem with such methods is that the score is bound to the reference format. Recent works, like FAIEr [60] propose a combination of visual and textual scene graphs to create better metrics. The second class of evaluation metrics is reference-free metrics - CLIPScore [20], InfoMetIC [23], TIGEr [26]. These methods use semantic vectors from the reference image or multiple regions of the image to calculate a similarity metric for the proposed captions. The drawback of these methods is that for dense images and long and detailed captions, semantic vectors will be not be representative as they will include too many concepts.

A new type of evaluation method is a question-based metric. The authors of QACE [32] proposed a framework for generating questions from captions and evaluating proposed captions using these questions. A similar approach was recently proposed for evaluation of image generation models [10]. Davidsonian Scene Graph (DSG) allows generation of questions organized in dependency graphs that help to comprehensively evaluate text-to-image models.

Inspired by DSG and DPG-bench [24], we propose a reversed approach for image captioning evaluation. Based on the image, we generate "yes-no" question-answer pairs across 17 categories: general, image type, text, color, position, relation, relative position, entity, entity size, entity shape, count, emotion, blur, image artifacts, proper noun (world knowledge), color palette, and color grading. The answer to the majority of questions is "yes", as this provides a more clear signal of correctness. During evaluation, we use an LLM to answer the questions based solely on the candidate caption. Available answer options are "yes", "no", and "n/a" (when it is impossible to answer the question). We then compare the answers with the reference answers to calculate the resulting accuracy.

We are releasing CapsBench , a set of 200 images and 2471 questions for them, resulting in 12 questions per image on average. Images represent a wide variety of types: film scenes, cartoon scenes, movie posters, invitations, advertisements, casual photography, street photography, landscape photography, and interior photography. A diversity of questions and images enables comprehensive evaluation of image captioning systems. We describe evaluation results for PG Captioner and other state-of-the-art models in Section 7.

5 Image qualitative results

5.1 Photo-realism

Figure 6: Photo-realistic qualitative comparison: the top-left is Ideogram-2, the top-right is PGv3, the bottom-left is Flux-pro, and the bottom-right is the prompt. Zoom in for better comparison on details and textures.

Figure 7: Photo-realistic qualitative comparison cont.: the top-left is Ideogram-2, the top-right is PGv3, the bottom-left is Flux-pro, and the bottom-right is the prompt. Zoom in for better comparison.

Figure 8: Photo-realistic qualitative results from PGv3. Non-curated random samples from our evaluation prompt set.

Fig. 6, 7, and 8 show image generation results from PGv3, alongside other top-tier models. In each panel from Fig. 6 and 7, the images are arranged as follows: the top-left is Ideogram-2, the top-right is PGv3, the bottom-left is Flux-pro, and the bottom-right is the prompt. When viewed as thumbnails, the images from all 3 models appear similar with minimal qualitative differences.

However, upon zooming in to examine details and textures, notable distinctions become apparent. Flux-pro consistently produces images with overly smooth skin textures, resembling oily or 3D renderings, rather than realistic depictions. Ideogram-2 offers more realistic skin textures compared to Flux-pro but struggles with prompt adherence, often missing key details as prompts grow longer. In contrast, PGv3 excels in both prompt-following and generating realistic images. Furthermore, PGv3 demonstrates a significantly better cinematic quality than the other models.

5.2 Prompt-Following

Figure 9: Prompt-following qualitative comparison. Highlighted text in bright colors indicate instances where Flux-pro or Ideogram-2 failed to follow the prompts, while PGv3 consistently adheres to all of the details in the prompt. The examples shown are selected samples from our evaluation prompt set.

Fig.9 illustrates three examples highlighting the superior prompt-following ability of our PGv3 model compared to other models, including Ideogram-2 and Flux-pro. In each prompt, the colored text highlights the specific details that other models fail to capture, while PGv3 consistently follows those details. There is no particular meaning to specific colors. The advantage of PGv3 becomes particularly evident as the testing prompts become longer and contain more detailed information. We attribute this performance to our LLM-integrated model structure and our advanced VLM captioning system.

5.3 Text Rendering

Figure 10: Text rendering qualitative results. PGv3 can generate text-rich content across various categories, ranging from professional designs like advertisements and logos to playful creations such as memes and greeting cards.

Fig.10 demonstrates the text rendering capabilities of PGv3. The model can generate images across a wide range of design categories, including posters, logos, stickers, book covers, and presentation slides. Notably, PGv3 can reproduce trending memes with customized text and create entirely new memes with limitless characters and compositions, thanks to its powerful prompt-following and text rendering abilities.

5.4 RGB Color Control

PGv3 achieves exceptionally fine-grained color control in generated content, surpassing standard color palettes. With its robust prompt-following capabilities and specialized training, PGv3 enables users to precisely control the color of each object or area within an image using exact RGB values, making it ideal for professional design scenarios that require precise color matching. As shown in Fig. 11 and 12, PGv3 accepts both overall color palettes, automatically applying the specified colors to appropriate objects and areas, and dedicated color values for specific objects as defined by the user.

Figure 11: RGB color control qualitative results. Prompts are omitted due to space limitation, the color bar under each image indicates the specified items and colors in the prompt

Figure 12: RGB color palette control qualitative results. PGv3 accepts an overall color palettes, automatically applying the specified colors to appropriate objects and areas.

5.5 Multilingual Prompt Input

Figure 13: Multilingual qualitative results. In each panel, images are generated from prompts in English, Spanish, Filipino, French, Portuguese, and Russian, arranged from top left to bottom right. For each panel, we display the prompt in one of the languages used, and all languages are represented across the panels.

Thanks to the powerful LLM used as our text encoder, which inherently understands multiple languages and forms well-related word representations across them, PGv3 can naturally interpret prompts in various languages. We found that this multilingual capability is fully realized with just a small dataset of multilingual text-and-image pairs—tens of thousands of images are sufficient.

Fig. 13 illustrates the model’s performance across various languages. In each panel, images are generated from prompts in English, Spanish, Filipino, French, Portuguese, and Russian, arranged from top left to bottom right. PGv3 consistently produces content that adheres to the prompts in each respective language. It is important to note that we did not train our model on non-English text rendering data; therefore, PGv3 typically does not recognize or generate text in other languages.

6 Image Model Quantitative Evaluation

6.1 Evaluation on Graphic Design Ability

Figure 14: Graphic design ability evaluation. We performed user preference studies on common use cases requiring graphic design skills. We compared our model Playground v3 and high-quality ground truth data created by designers that can be used to represent the average graphic design ability of humans. From this study, users preferred designs generated by our model in all categories, especially for stickers, art, and mobile wallpaper.

We collected around 4k high-quality images created by human designers in common graphic design applications like monograms, stickers, t-shirt, mockups, mobile wallpaper, logo, cards, posters etc. We captioned the images referred as Human in Fig. 14 with our in-house captioner, PG Captioner. We then used the same text caption to generate images using Playground v3. We conducted user preference studies on the paired data and asked users to select which image is preferred. Each pair got at least 7 votes, and we used majority voting to determine the winning image.

6.2 Image-text Alignment Evaluation

6.2.1 DPG-bench

Table 1: DPG-bench evaluation. We compared our method with SoTA open-source models. From the table, our model achieved high text-image alignment performance.

Method	Overvall	Global	Entity	Attribute	Relation	Other
SDXL [46]	74.65	83.27	82.43	80.91	86.76	80.41
Playground v2.5 [33]	75.47	83.06	82.59	81.20	84.08	83.50
Lumina-Next [62]	74.63	82.82	88.65	86.44	80.53	81.82
Hunyuan-DiT [35]	78.87	84.59	80.59	88.01	74.36	86.41
PixArt-Sigma [7]	80.54	86.89	82.89	88.94	86.59	87.68
DALLE 3 [4]	83.50	90.97	89.61	88.39	90.58	89.83
SD3-Medium [15]	84.08	87.90	91.01	88.83	80.70	88.68
Playground v3 (ours)	87.04	91.94	85.71	90.90	90.00	92.72

In order to access the text following ability (text-image consistency), we compared our models with SoTA models on two benchmarks. We measured DPG-bench follows [24] in Tab. 1, and our model achieved an overall score of 87.04, which is higher than DALLE 3 and SD3-medium.

6.2.2 DPG-bench Hard

However, upon closer inspection of the results, we found that the Visual-Question-Answering (VQA) model used in the open-sourced DPG-bench often makes errors, providing incorrect answers even when models correctly generate the content specified by the testing prompts. To more accurately assess the true prompt-following performance, we developed an upgraded version called "DPG-bench Hard." The following outlines the steps used to construct this benchmark.

First, we applied k-means clustering on our internally curated dataset, which consists of images rated by real users. This resulted in 120 clusters covering diverse image domains. From each cluster, we randomly selected 20 images near the cluster centers, yielding a testing set of 2,400 images in total. We used GPT-4o to caption these images, ensuring the captions were detailed and comprehensive, with an average length of about 256 tokens. To avoid potential prompt-style bias favoring our model, we particularly choose to not use our in-house Captioner for this task. After generating the captions, we asked GPT-4o to create yes-or-no questions based on each caption, then categorized these questions in line with the original DPG-bench. Each prompt generated an average of 30 questions. This process parallels how we constructed our CapsBench, which will be described in detail in Sec.7

To evaluate each model, we generated four images per testing prompt using random seeds, then used GPT-4o to answer the corresponding questions for each image. We summarized the "yes" responses from GPT-4o as the final accuracy rate. Despite the possibility of occasional errors by GPT-4o, we found the error margin to be below 5%, making the testing scores more closely aligned with human judgment as compared to the original DPG-bench.

Fig.16 and Tab. 2 present our testing results. PGv3 achieved an overall accuracy score of 88.62%, outperforming other tested models, including Flux-pro and Ideogram-2, across all sub-categories. Notably, a significant portion (40%) of the testing images in our DPG-bench Hard set belong to graphic design-related categories, such as posters, ads, greeting cards, and book covers, all of which require the model to be able to follow the complicated object arrangements and layout requirements from the testing prompts. This benchmark not only reflects PGv3’s leading prompt-following performance but also effectively highlights PGv3’s superior performance in graphical design domains.

On text synthesis accuracy, PGv3 scores 82%, outperforming Flux-pro’s 69% and Ideogram-2’s 75%. A closer analysis reveals that, besides common mistakes like misspellings and corrupted glyph generation, another primary reason behind the lower scores of Flux-pro and Ideogram-2 is their tendency to omit required text elements as the prompts increase in length and complexity. This issue highlights PGv3’s superior prompt-following ability, which ensures more accurate text rendering even with longer, more detailed prompts. Examples on text rendering image results from PGv3 together with Ideogram-2 can be found in Fig.15. While it primarily shows failed cases from Ideogram-2, we want to emphasize that Ideogram-2 generally performs well, and both models have good and bad testing cases. Tab. 2 provides the non-cherry-picked quantitative performance scores for these models.

Figure 15: Text rendering comparison between Ideogram-2 and PGv3 using testing prompts from our DPG-bench Hard. In each panel, the left two images are random samples from Ideogram-2, while the right two images are random samples from PGv3. Due to space constraints, testing prompts are abbreviated. Text highlighted in bold red indicates areas where Ideogram-2 makes errors, while PGv3 performs accurately.

We believe that our DPG-bench Hard offers a comprehensive evaluation of text-to-image model performance. As such, we focused our efforts on collecting images from Flux-pro and Ideogram-2 specifically for DPG-bench Hard. For subsequent evaluations in this section, we did not collect additional images from these models, as the extensive image collection process requires significant effort, which we deemed unnecessary.

6.3 Image-Text Reasoning Evaluation

We evaluated the model’s reasoning ability using a popular benchmark, GenEval [18]. In Tab 3, our model achieved a 0.76 overall score, higher than SD3’s 0.74. In particular, our model has a much better score on object and position reasoning.

Figure 16: Visual presentation of DPG-bench Hard, our internal benchmark powered by GPT4-o

Table 2: Quantitative results from DPG-bench Hard, our internal benchmark powered by GPT4-o

Method	Overvall	Entity	Global	Relation	Position	Count	Text
SD3-Medium [15]	53.13	61.57	73.27	49.25	50.00	51.82	18.10
PixArt-Sigma [7]	62.89	76.85	83.46	59.33	73.98	59.71	11.97
Midjourney v6.0	64.63	76.86	82.86	61.29	78.12	65.36	14.30
DALLE 3 [4]	67.71	79.11	85.94	64.48	57.35	62.82	28.84
Flux-pro [31]	78.69	88.68	90.62	75.15	70.58	81.13	69.47
Ideogram-1.0 [25]	79.75	89.32	90.21	78.73	78.12	65.36	47.53
Ideogram-2.0 [25]	80.12	89.81	91.04	79.93	83.33	83.72	75.32
Playground v3 (ours)	88.62	94.45	93.64	86.47	87.21	84.30	82.09

Table 3: GenEval evaluation benchmark. We measured Flux-dev by using their public model API, since there are no official numbers reported. For the rest of the baselines, we used their official numbers in the paper.

Method	Overall	Single	Two	Counting	Colors	Position	Attribution
SDXL [46]	0.55	0.98	0.74	0.39	0.85	0.15	0.23
DALLE 3 [4]	0.67	0.96	0.87	0.47	0.83	0.43	0.45
SD3 [15]	0.74	0.99	0.94	0.72	0.89	0.33	0.60
Flux-dev [31]	0.68	0.99	0.85	0.74	0.79	0.21	0.48
Playground v3	0.76	0.99	0.95	0.72	0.82	0.50	0.54

6.4 Text Synthesis Evaluation

Following previous work [6], we selected 1k samples from Mario-eval, which includes text images in Laion, TMDB and OpenLibrary. We calculated a clip score and used Kosmos 2.5 [40] OCR model to detect text, and calculated text rendering accuracy. In Tab. 4, our model has much better text accuracy with an OCR-Fscore of 40.35, higher than Flux-pro’s 35.28. We also show qualitative examples in Fig. 17, where our model excels at small text.

Table 4: Mario-text-1k text rendering evaluation benchmark. We generate Flux-dev/pro samples using their official APIs; for SD3-Medium, we used the API from Diffusers.

Method	CLIP Score $\uparrow$	OCR-Precision $\uparrow$	OCR-Recall $\uparrow$	OCR-Fscore $\uparrow$
SD3-Medium [15]	31.26	15.99	21.36	18.29
Flux-dev [31]	33.41	32.86	37.38	34.98
Flux-pro [31]	33.65	33.12	37.74	35.28
Playground v3	33.89	39.23	41.54	40.35

A close-up photograph of a black T-shirt with a bold and urban design. The T-shirt features a prominent white print in the center. The text ’LET THE TRAP SAY’ is written in bold, uppercase, sans-serif font and is positioned at the top of the print, slightly tilted. Below this, the word ’amen’ is displayed in a large, bold, blackletter font. The entire text is outlined by a thick, white, cuban link chain design, which forms a rectangular frame around the text. The T-shirt is made of a plain black fabric, and the neckline label shows ’Champion’ in small, black, uppercase letters. The background is plain white, providing a stark contrast to the black T-shirt and white print. The lighting is even and soft, highlighting the details of the print without casting harsh shadows.

Figure 17: Qualitative comparison of text rendering accuracy in Mario-text-1k. From left to right: Ground Truth, Flux-pro and Playground V3. PGv3 successfully render the word "LET" and also make sure "amen" is in lower case following the prompt.

6.5 Evaluation on Image Quality

We present results on two standard benchmarks, ImageNet and MSCOCO, to compare our model’s performance against other state-of-the-art models.

However, we maintain our position that ImageNet and MSCOCO may not effectively capture the comprehensive performance of large-scale text-to-image models, especially given the progress of models like PGv3 in prompt-following.

6.5.1 ImageNet Benchmark

Table 5: ImageNet image quality evaluation. All the samples are evaluated at 256 resolution.

	train		val
Method	FID	FDDINOv2	FID	FDDINOv2
Class condition models
DiT-XL/2 [45]	9.62			79.36
Simple Diffusion [22]	2.77		3.75
Text condition models
Playground v2.5 [33]			21.18	231.05
EDM2-xxl-CLIP [28]			16.37	118.74
Playground v3 (ours)			14.67	102.91

In Tab. 5, we compare PGv3 with our previous model PGv2.5. Since our model takes text as a condition, we follow previous work [15] to use a photo of a {class} format to convert ImageNet class labels into text conditions. We also compared a SoTA method EDM2 [28] model in the same setting. We augmented it with a clip text conditioner for a fair comparison. From the table, PGv3 performs better in terms of FID and FD DINOv2. We also show SoTA class conditioned model performance as a reference; usually, those models take the class label as a condition to achieve better performance.

6.5.2 MSCOCO Benchmark

Table 6: MSCOCO evaluation. Here we compare our previous method PG2.5 at resolutions 256 and 512, and SoTA open-source methods in 1024 resolution. We compare with original caption (usually short) and PG caption (longer with more details).

	Original caption		PG Captioner caption
Method	FID	FDDINOv2	FID	FDDINOv2
256 $\times$ 256 resolution
Playground v2.5 [33]	11.41	225.80	8.64	191.17
Playground v3 (ours)	11.94	172.34	7.06	58.82
512 $\times$ 512 resolution
Playground v2.5 [33]	10.63	182.00	14.16	180.78
Playground v3 (ours)	13.79	160.84	7.14	58.41
1024 $\times$ 1024 resolution
PixArt-Sigma [7]	21.60	291.29	15.29	192.58
Hunyuan-DiT [35]	22.97	339.42	15.98	237.73
AuraFlow-0.2 [16]	22.29	314.66	15.79	189.91
Lumina-Next [62]	19.04	342.93	15.32	243.16
SD3-Medium [15]	19.77	194.97	14.42	128.28
Flux-dev [31]	22.25	281.77	13.54	124.83
Playground v3 (ours)	18.38	238.07	8.58	82.59

We also evaluated our models in MSCOCO validation set with 30k prompts. In table 6 we evaluated both the original prompt (short) and also a reception prompt using PG Captioner (long). We measure the model performance in FID and FD DINOv2 [42], we found FD DINOv2 is a better metrics that is more sensitive to object shapes. We measured our model at different training phases, and compared with the PGv2.5 model for 256 and 512 resolutions. We also compared other SoTA models at the 1024$\times$1024 resolution.

From the table, our model achieves great performance in both short and long prompts and show significant improvement in FD DINOv2 which indicates much better geometry and object shape. Here, we only compare to SoTA open-source models as this evaluation needs to sample 30k images for both the original caption and captions from PG Captioner. From the results, we can see PGv3 achieves the best FID and FD DINOv2 for both caption sets. Notably with PG Captioner captions, our 1024px model significantly outperforms Flux-dev, indicating strong image quality in terms of color and shape generalization.

6.6 Evaluation on VAE Reconstruction

Table 7: VAE reconstruction evaluation.

Method	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
256 $\times$ 256 resolution
SDXL-vae-ch4	0.116	0.797	24.96
PGv3-vae-ch16	0.075	0.914	29.76
512 $\times$ 512 resolution
SDXL-vae-ch4	0.112	0.821	26.52
PGv3-vae-ch16	0.068	0.926	31.60
1024 $\times$ 1024 resolution
SDXL-vae-ch4	0.116	0.840	28.11
PGv3-vae-ch16	0.060	0.939	33.44

Following previous works [11, 15], we increased the VAE latent channel size from 4 to 16, and we trained the VAE in a progressive manner from lower resolution to higher resolution. In Tab. 7, we report reconstruction performance on a validation set on multiple resolutions. From the table, we improve significantly compared to SDXL channel 4 VAE; specifically, at resolution 1024x1024, we achieve 33.44 PSNR.

7 Captioning Evaluation

We used CapsBench to evaluate PG Captioner, along with leading proprietary models GPT-4o and Claude-3.5 Sonnet. When generating captions with proprietary models, we used detailed instructions with output schema, few-shot prompting (by providing three examples of high-quality detailed captions) and chain-of-thought reasoning (perform written analysis of the image prior to generating the caption) to achieve the best results.

For question answering, we used Claude-3.5 Sonnet. To work around stochastic outputs of Sonnet and ensure consistency, we run question answering three times for every caption and use the consensus answer. As a result, combined score of PG Captioner is 72.19%, Claude-3.5 Sonnet’s 71.78%, GPT-4o’s 70.66%. Fig. 18 shows the results split by categories. We can see that CapsBench helps to reveal shortcomings of existing models for captioning. Captions usually don’t contain descriptions of object shapes (e.g., they likely produce "brown wooden table," instead of "brown rectangular wooden table"), sizes in the image (e.g., "the man appears big in the image"), and visual artifacts (like noise or lens flare). This may be a problem for using these captions in systems heavily reliant on completeness of the captions, such as image generation models or systems for visually impaired users.

Figure 18: Captioning evaluation results split by category

Additionally, we evaluated the length of the captions to understand whether longer captions correlate with higher evaluation scores. Fig. 19 shows histograms of caption lengths for evaluated models. Interestingly, all three models have different length distributions, where GPT-4o generates the shortest captions and Claude-3.5 Sonnet produces captions that are twice as long on average.

Figure 19: Caption length distributions

To further study the pros and cons of different models, we performed qualitative analysis of captions. In Fig. 20, we show one of the images from CapsBench. This image contains a significant number of subjects and a lot of small details, which makes it very challenging for VLMs. Tab. 8 shows the questions for this image along with the answers generated from the captions. Captions themselves are presented below.

We can see that PG Captioner describes each knight and their horses in detail. Interestingly, the correctness of descriptions decreases as the model goes from the left knight to the right. PG Captioner also hallucinates the text on the flag in the background, which is not visible due to blur effects. Sonnet, on the other hand, doesn’t provide detailed description of each knight and describes them as a group. It also provides more detailed description of the crowd and trees in the background, which makes the caption noticeably longer. GPT-4o generates very concise caption compared to other models. It also describes the knights as a group and incorrectly describes the building on the right as multiple tents.

Figure 20: Example image from CapsBench

Table 8: Questions for the example image from CapsBench

Category	Question	GT	PG	Claude	GPT-4o
color	Is there a golden yellow flag?	yes	yes	n/a	n/a
color	Is the cape of the right-most knight emerald green?	yes	no	n/a	n/a
position	Are all the knights positioned in the center of the image?	yes	n/a	yes	yes
relative position	Is the knight with the golden horns to the left of the knight with the red feathers?	yes	yes	n/a	n/a
blur	Is the background blurred?	yes	no	no	no
entity size	Is the building on the right side of the image large?	yes	n/a	n/a	n/a
count	Are there seven knights in the image?	yes	yes	yes	yes
entity	Is there a horse in the image?	yes	yes	yes	yes
relation	Is the knight with red feathers mounted on a horse?	yes	yes	n/a	n/a
general	Does the image show an outdoor scene?	yes	yes	yes	yes
image type	Is the image a photograph?	yes	yes	yes	yes
color grading	Does the image have high saturation?	yes	n/a	yes	n/a

PG Captioner caption (accuracy 58.3%): "A vibrant photograph captures seven knights riding horses through an outdoor medieval reenactment event, positioned in front of a large crowd and tall trees under clear blue skies. The scene is bathed in natural daylight, creating high contrast between the brightly colored costumes and the verdant surroundings. Each knight wears ornate armor and carries distinctive flags with various crests and emblems, including symbols such as lions, dragons, eagles, and heraldic designs. From left to right, the first knight rides a white horse adorned with black and gold armor and a cape featuring circular patterns, holding a black flag with a red lion emblem on a wooden staff. Next is a knight on a gray horse wearing golden armor with intricate designs and a horned silver helmet, carrying a yellow flag with a brown eagle. The third knight, mounted on a brown horse with a white blaze, dons black chainmail armor with gold accents and a white cape with small triangular cut-outs, bearing a white banner displaying a black eagle or dragon silhouette. The fourth knight rides a dark brown steed with a white blaze, clad in elaborate red and gold armor with a flowing crimson cloak, holding a red flag with golden dragon motifs. The fifth knight, atop a light brown horse with a white blaze, sports navy blue armor with red and green trimmings and a forest green cape, bearing a purple flag with a red lion design. The sixth knight, mounted on a black horse, wears metallic silver armor with prominent shoulder plates, a closed visor helmet, and a cape with intricate embroidery, carrying a black flag with a white stag or bull symbol. The seventh knight, on a black horse, completes the lineup with charcoal gray armor embellished with brass details, a unique horn-like helmet, and a cape featuring alternating vertical stripes of royal blue and forest green, topped with a large feather plume in vivid red, yellow, and green hues. Behind them, spectators fill wooden benches and stand areas, attentively watching the procession. Large banners with various designs and text are visible among the crowd, one reading ’Deutscher Ritterturnier’ in white serif font on a blue and white striped background. In the background, tall trees provide a lush backdrop, while large tents with conical roofs display colorful shields with various heraldry symbols, including blue and white triangles over red lines on beige backgrounds. One tent features a shield with red and white stripes. The overall mood is festive, celebratory, and historical, capturing the essence of a lively medieval renaissance festival where the past meets present in a picturesque setting."

Claude-3.5 Sonnet caption (accuracy 50%): "This photograph captures a grand medieval-themed event, likely a renaissance fair or historical reenactment. The image is dominated by a symmetrical arrangement of seven knights on horseback, positioned in the foreground of a sunny outdoor arena. Each knight is fully outfitted in elaborate medieval armor, complete with helmets, plumes, and colorful regalia that spans a vibrant spectrum from silver and gold to rich reds, blues, and greens. The knights are mounted on well-groomed horses of various colors - white, brown, and black - each adorned with decorative barding that matches their riders’ colors and heraldry. The knights proudly display an array of flags and banners, each bearing unique heraldic designs such as dragons, lions, and geometric patterns. These banners add vertical elements to the composition and infuse the scene with a sense of pageantry and historical authenticity. Behind the knights, a vast crowd of spectators fills the background, suggesting this is a popular and well-attended event. The audience creates a sea of indistinct faces and colors, providing depth to the image. Beyond the crowd, a line of green trees is visible, framing the upper portion of the photograph and adding a natural backdrop to the medieval spectacle. On the right side of the image, a partial view of what appears to be a medieval-style structure can be seen. It has a conical roof and might be a decorative tower or large tent, further enhancing the period atmosphere of the event. The lighting is bright and natural, indicative of a sunny day, which allows for high detail visibility and vivid color reproduction. This lighting emphasizes the metallic shine of the armor, the rich hues of the fabrics, and the overall festive mood of the occasion. The composition is centered and symmetrical, drawing the viewer’s eye to the line of knights as the main focal point. The image has a high level of saturation, bringing out the bold colors of the costumes and flags, while maintaining good contrast that helps distinguish the intricate details of the armor and heraldic designs. Overall, this photograph captures a moment of historical reenactment that combines educational value with entertainment, showcasing the pageantry and spectacle of medieval tournaments and celebrations. The scale of the event, evident from the large crowd and elaborate costumes, suggests a significant and popular cultural gathering celebrating historical heritage and craftsmanship."

GPT-4o caption (accuracy 41.7%): "The image captures a medieval-themed performance or reenactment, featuring seven knights in full, elaborate medieval armor riding horses. The photograph is taken from a front angle, directly facing the knights, with a well-exposed, bright natural lighting highlighting the scene. The knights are central to the composition, positioned in a symmetrical row. Each knight is dressed in highly detailed armor, including full-body armaments with intricate designs, and each carries a different colored flag adorned with unique heraldic symbols. The knights are mounted on variously colored horses, with some in white, brown, and black. Their helmets are unique, showcasing different designs and colorful plumes, adding to the dramatic and historical mood of the image. The midground displays a large audience dressed in modern-day clothing, attentively watching the performance. They are situated behind a secured viewing area, bordered by fencing. Several colorful banners and flags are visible in the midground, attached to poles and adding to the festive and vibrant atmosphere. The background features lush green trees, giving a natural backdrop to the scene, with medieval-styled tents visible in the far corners, completing the historical setting. The lighting emphasizes the bright, vivid colors and the intricate details of both the knights’ armor and their surroundings. The overall composition, lighting, and vivid colors create a lively and historically immersive scene, capturing the essence of a medieval reenactment event."

8 Conclusion

In this work, we introduce Playground v3, a state-of-the-art text-to-image generation model with strong prompt adherence, reasoning and text rendering ability. It shows superhuman graphic design abilities from a user preference study on various downstream design applications, where we compare samples generated by our model and a high quality curated image dataset created by human designers. We also release to the community a new carefully prepared image captioning benchmark CapsBench, designed for detailed long captions, to facilitate the research and benchmarking of the progress of image understanding ability.