The MIRFLICKR Retrieval Evaluation

Current Organizers: Mark Huiskes, Bart Thomee and Michael Lew

NEWS
In 2011, the MIRFLICKR-1M collection will be used in ImageCLEF 2011 for the visual concept detection and annotation task. Please take a look at the Photo Annotation task description for further details.

In 2010, the MIRFLICKR-25000 collection will be used in ImageCLEF 2010 for the visual concept detection and annotation task. Please take a look at the Photo Annotation task description for further details.

In 2009, the MIRFLICKR-25000 collection will be used in ImageCLEF 2009 for the visual concept detection and annotation task. Please take a look at the Photo Annotation task description for further details.

Update
The MIRFLICKR collection has been extended to 1 million images! In addition, various content-based visual descriptors will be made available. See Extension below for further details.  Bart Thomee has been invited to  be an organizer for his dedicated work on MIRFLICKR.

Introduction    Copyright    Tags    EXIF    Annotations    Download    Publications    Extension   

by Silke Gerstenkorn	by Dave Wild	by Hugo A.B. Olivas
by Martin P. Szymczak	by Mani Babbar	by Lee Otis

Introduction

The new MIRFLICKR-25000 collection consists of 25000 images downloaded from the social photography site Flickr through its public API.

We are doing our best to make sure the image collection is going to be:

• OPEN
  Access to the collection is simple and reliable, with image copyright clearly established. This is realized by selecting only images offered under the Creative Commons license. See the copyright section below.
  
  
• INTERESTING
  Images are also selected based on their high interestingness rating. As a result the image collection is representative for the domain of original and high-quality photography.
  
  
• PRACTICAL
  In particular for the research community dedicated to improving image retrieval. We have collected the user-supplied image Flickr tags as well as the EXIF metadata and make it available in easy-to-access text files. Additionally we provide manual image annotations on the entire collection suitable for a variety of benchmarks.
  
  

MIRFLICKR-25000 is an evolving effort with many ideas for extension. So far the image collection, metadata and annotations can be downloaded below. If you enter your email address before downloading, we will keep you posted of the latest updates.

Copyright and Licenses

Although most images on Flickr are published with all rights reserved, there is also a large number of images offered under Creative Commons copyright licenses. The Creative Commons attribution licenses allow for image use as long as the photographer is credited for the original creation. Possibly, use is granted under additional restrictions, but none of these preclude the use of the images for benchmarking purposes.

While compiling the MIRFLICKR-25000 collection we have made sure only Creative Commons images were included and took care to collect as much information possible about the creators of the image. The creator information as well as the exact license type and image title are collected in image license metafiles, which are distributed together with the images.

We would like to take the opportunity here to express our gratitude to the image photographers for allowing us to use their pictures: we greatly appreciate this and gladly acknowledge your work. Your names and license details are also listed in this credit document. Please let us know if you have special wishes on how you would like to be credited or have additional details that must be incorporated.

Flickr Tags

One of the great attractions of Flickr is the platform it offers its users to search and share their pictures based on image tags. We also supply these image tags in two forms: the raw form in which they are obtained from the users and in processed form with raw data cleaned up (a bit) by Flickr.

For retrieval research we are mainly interested in concrete visual concepts. The most common tags of this type are listed below (colors, seasons and place names were left out):

The average number of tags per image is 8.94. In the collection there are 1386 tags which occur in at least 20 images. Most tags are in English, but foreign terms occur as well.

EXIF

EXIF (Exchangable image file format) metadata represents a number of properties and settings of the digital camera at the time of taking a picture. This includes information on:

• the camera itself: brand, manufacturer, ...
  
• camera settings: exposure, aperture, focal length, ISO speed, metering mode, ...
  
• image settings: orientation, resolution, compression, ...
  
• time, date (and location)

Flickr separates the EXIF data from the images: the information is no longer embedded in the image files! For about 85% of the images in the collection, EXIF data are available and permission is granted by the creator to access this data through the API. For these images we have collected the data (with exception of binary data such as for thumbnails) and made them available in plain text files. Note that even when EXIF data was collected, not all fields are always present. The table below shows the possession for a number of common fields.

EXIF geolocation fields are particularly scarce and are available for only 152 images.

Annotations

The annotation scheme has been set up in a way to make it easy to extend it with new keywords without having to go through all 25000 images again. This is possible by stepwise refinement along two dimensions:

1. Relevance level: from possibly relevant to actually relevant
   We first annotate concepts or topics by interpreting them in a very wide sense. For an image to receive a label, the concept does not need to appear prominently; as long as it visible or applicable at least to some extent, this is already sufficient. We call the labels resulting from this level of annotation potential labels. These labels should capture all images that could possibly apply to the concept in real searches. In this way they can act as a sort of greatest common denominator for the concept, with the goal of making subsequent annotation of more narrow interpretations a lot faster.

   Next, using the potential labels, we annotate the images with relevant labels. These are annotations for a specific interpretation of a concept by a single annotator. A label is supplied only if the annotator found the image really relevant to his interpretation. Our goal is to supply several of such annotations for each concept. Note that so far however, most concepts only have single interpretations each: one where the concept/topic is "salient" to the annotator in a general sense.

2. Abstraction level: from general to specific categories
   We have first annotated the image collection with labels for the general topics listed in the table below. These topics were chosen in such a way that they cover a lot of interesting topics as proper subtopics. To annotate such subtopics we only have to consider images that have a potential label for the more general topic.

Download

Please proceed to the download page. (has both 25K and 1M)

Publications

If you use the MIRFLICKR-25000 image collection in your work, please cite:

M. J. Huiskes, M. S. Lew (2008). The MIR Flickr Retrieval Evaluation. ACM International Conference on Multimedia Information Retrieval (MIR'08), Vancouver, Canada (bib)

Extension

The MIR Flickr collection has been extended in two ways. First, the number of images has been extended to 1 million images. Second, we now supply a number of content-based visual descriptors for the entire new set of images.

The new images are obtained in the same way as the original images. All images are made available under a Creative Commons Attribution Licence. To obtain high quality photography, the images are also selected based on their Flickr interestingness score. Note that the new images are not manually annotated like the core set of 25000 images, but all original Flickr user tag data, as well as the EXIF metadata, are again made available.

The content-based visual descriptors that are supplied for the new images are the MPEG-7 Edge Histogram and Homogeneous Texture descriptors, and the ISIS Group color descriptors.

All original images are made available through BitTorrent. Since, for many, the full collection may prove too large to download, we also provide 64x64 pixel jpeg-thumbnails. For further details, see the download page.(has both 25K and 1M)

The extension is described in:

M. J. Huiskes, B. Thomee, M. S. Lew (2010). New Trends and Ideas in Visual Concept Detection. ACM International Conference on Multimedia Information Retrieval (MIR'10), Philadelphia, USA (bib)