CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
Andreas Fürst* 1,
Elisabeth Rumetshofer* 1,
Viet Tran1,
Hubert Ramsauer1,
Fei Tang3,
Johannes Lehner1,
David Kreil2,
Michael Kopp2,
Günter Klambauer1,
Angela Bitto-Nemling1,
Sepp Hochreiter1 2
1 ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria
2 Institute of Advanced Research in Artificial Intelligence (IARAI)
3 HERE Technologies
* Equal contribution
Detailed blog post on this paper at this link.
The full paper is available here.
This repository contains the implemenation of CLOOB used to obtain the results reported in the paper.
The implementation is based on OpenCLIP, an open source implementation of OpenAI’s CLIP.
Setup
We provide an ‘environment.yml’ file to set up a conda environment with all required packages.
Run the following command to clone the repository