The Caliph&Emir CBIRS is the second best system in this test behind VIPER/GIFT, as shown by the ranking and scoring (see figure 2).
This clearly marks VIPER/GIFT as the best CBIRS in this benchmark.
Unfortunately this CBIRS is also affected by the missing-images-problem.
The Oracle Intermedia CBIRS shows very good results with the exception of one big outlier.
It is nowadays an even more vivid research area than when Veltkamp and Tanase provided an overview of the CBIR landscape [VT02], which showed the state-of-the-art at that time.
Most of the efforts try to address the problem of CBIR in a general fashion, they don't focus on specific domains.
We don't review the development of CBIR as a whole.
Seven CBIR systems have been chosen to for our benchmark:
Finally, our performance measure is introduced, as well as two new methods to compare CBIR systems based on these measures.
In recent years efforts were made to analyse the possibilities for benchmarking CBIR systems and to propose standard methods and data sets ([MMS+01b, MMW06, MMS01a, ben]).
The students had no prior knowledge of CBIR techniques and were not familiar with the image sets (other than one could expect from fairly normal photographs).14 queries have been defined, for each the students had to compare the query image with all images from the query's domain.
In [MMS+01b] the Normalized Average Rank (NAR) is proposed as a performance measure for CBIR systems.