Microsoft's - LLM2CLIP Llama3.2-1B-EVA02-L-14-336

About Model

The LLM2CLIP-Llama3.2-1B-EVA02-L-14-336 model is part of the LLM2CLIP series, aiming to extend the capabilities of CLIP models by combining Large Language Models (LLMs) with advanced visual encoders. This integration allows the model to process more detailed and extended textual descriptions, overcoming the context window limitations of traditional CLIP text encoders. By fine-tuning the LLM in the caption space using contrastive learning, the model enhances the textual discriminability of output embeddings. This approach leads to substantial improvements in cross-modal tasks, such as image-text retrieval and zero-shot image classification. Experiments have demonstrated that this method boosts performance significantly, transforming a CLIP model trained solely on English data into a state-of-the-art cross-lingual model. Moreover, when integrated into multimodal training with models like Llava 1.5, it consistently outperforms traditional CLIP models across nearly all benchmarks, showcasing comprehensive performance enhancements.