Jinghong Chen

Sign in Subscribe

my work

PreFLMR: SoTA Open-sourced Multi-modal Knowledge Retriever from Scaling Up FLMR

[1,087 words, 5-minute read] Three products emerged from our study in scaling up multi-modal late-interaction retrievers: * The Multi-task Multi-modal Knowledge Retrieval benchmark (M2KR) totaling 4.4M training examples for training and comprehensively evaluating knowledge retrievers on question-to-doc, image-to-doc, and question+image-to-doc tasks. * The Pretrained Fine-grained Late-interaction Multi-modal Retriever (PreFLMR)

3-minute Pitch: Retrieval Guided Contrastive Learning for Hateful Memes Detection

[898 words, 3-minute read] Hateful memes are captioned images promoting hostility towards specific social groups. Most hateful memes detection systems are logistic classifiers built on the embedding space of pre-trained visual-langauge model (e.g., CLIP). However, we find that under these embedding spaces, hateful memes and belign memes are located

3-minute Pitch: Late-Interaction Knowledge Retriever (FLMR) for Visual Question Answering @NeurIPS 2023

Knowledge-based Visual Question Answering (KBVQA) aims to answer a question related to an image that requires some world knowledge. Here's an example. Our NeurIPS paper takes the retrieval-augmented approach to tackles KBVQA. We first retrieve relevant documents from an external database and then generate answers based on the

3-minute Pitch: Learning from MBR decoding using Direct Preference Optimization

Minimum Bayes Risk (MBR) decoding generally outperforms temperature sampling and beam search. But it is expensive computationally. We can train the model on the MBR decoding outputs so that cheaper decoding methods perform on par with MBR. Google calls it "MBR fine-tuning". Our recent work introduces a more