ICDAR 2013 CROHME: Third International Competition on Recognition of Online Handwritten Mathematical Expressions
CROHME-III is the continuation of its earlier versions:
- CROHME 2011 which was held during ICDAR 2011,
- CROHME 2012 which was held during ICFHR 2012.
We hereby present the CROHME package which contains the data and tools from all CROHME competition.
All data and tools provided here are freely available only for research purpose without any commercial use.
Data
This package provides training and test data from the competitions CROHME 2011, 2012 and 2013. Furthermore, thanks to the participants' authorization, we are allowed to distribute the results files from the majority of the submitted systems in 2012. Here is the description of all data directories:
- CROHME2011_data : all data from the CROHME 2011 competition:
- CROHME_test : inkml test files without ground truth
- CROHME_testGT : inkml test files with ground truth
- CROHME_train : inkml train files with ground truth
- gram : xml grammars and symbol lists for parts I and II
- CROHME2012_data : all data from the CROHME 2012 competition:
- testData : inkml test files without ground truth
- testDataGT : inkml test files with ground truth
- trainData : inkml train files with ground truth
- gram : xml grammars and symbol lists for parts I, II and III
- lists : lists of inkml files and latex expressions for parts I, II and III
- CROHME2013_data : all data from the CROHME 2013 competition:
- TrainINKML : all training inkml files sorted by origin
- TestINKML : inkml test files without ground-truth, used to run the participants systems.
- TestINKMLGT : inkml test files with ground-truth, used to evaluate the participants systems with the evalinkml tool.
- Test_LG/Test2012LG Test_LG/Test2013LG: label graph version of the test files for 2012 and 2013 dataset, using inherited edges (so the graphs are DAGs).
- Test_LG/Test2012LG_TREE Test_LG/Test2013LG_TREE: label graph version of the test files for 2012 and 2013 data set, without inherited edges (so the graphs are trees).
- ParticipantsResults2012 : results files from the participants of the CROHME 2012 competition (note that the result files from VisionObjects are restricted to the participants):
- ResultsTest : inkml results for the test part for each participant (parts I, II, and III)
- ResultsTrain : inkml results for the train part for each participant (only part III)
Tools
CROHME Organizers provide some tools for math expression selection, running the test phase, evaluation and visualisation :
- CROHMELib : librairy of scripts to
- to filter latex expressions with regards to xml grammars
- to run a recognizer over a list of inkml files
- to compare ground-truthed inkml files with inkml results (evalInkml_v1.7.pl in 2012 and evalInkml_v1.10.pl in 2013)
- to convert the inkml files in lg files
CROHME Results
CROHME results are summed up in the three papers available in the directory CROHME_papers.
CROHME 2013
In the ICDAR paper 671 expressions are used in the test dataset. Two metrics are used : expression level computed on inkml files and weigthed error mean based on hamming distance computed on LG graphs. Systems with an * do not use only the provided train data.
| Exp. rate | Mean LG Error | |
|---|---|---|
| Czech Technical Univ. (Czech Rep.) | 2.68 | 41.63 |
| Tokyo Univ. of Agr. & Tech. (Japan) | 19.97 | 22.04 |
| *Univ. of Sao Paulo (Brazil) | 9.39 | 26.21 |
| Univ. of Valencia (Spain) | 23.40 | 18.31 |
| Rochester Institute of Techn. (USA) | 14.31 | 22.80 |
| Sabanci Univ. (Turkey) | 8.35 | 36.67 |
| *Vision Objects (France) | 60.36 | 4.27 |
| Univ. of Nantes (France) | 18.33 | 19.30 |
CROHME 2012
After CROHME 2012 competition two errors have been detected in the test data after the final version of the paper. In the ICFHR paper 488 expressions are used in the test dataset, thus here are the updated results on the 486 valid inkml files:
| STR_REC | SEG | REC/SEG | EXP | EXP1 | EXP2 | EXP3 | STRUCT | |
|---|---|---|---|---|---|---|---|---|
| FAlvaro | 79.91 | 91.94 | 86.35 | 22.84 | 34.77 | 41.15 | 42.39 | 42.39 |
| FS | 55.81 | 71.21 | 85.08 | 3.70 | 6.38 | 9.05 | 10.49 | 9.88 |
| IVC | 79.02 | 87.79 | 91.45 | 25.72 | 36.21 | 39.30 | 40.12 | 37.04 |
| RIT | 72.17 | 87.57 | 87.67 | 9.47 | 18.52 | 23.25 | 24.90 | 23.87 |
| Sabanci | 45.45 | 59.16 | 84.40 | 4.94 | 10.70 | 14.20 | 15.02 | 14.81 |
| UWaterloo | 86.47 | 95.55 | 91.27 | 40.33 | 54.32 | 59.26 | 62.14 | 61.52 |
| VO | 95.79 | 98.84 | 96.91 | 62.96 | 79.42 | 82.30 | 82.51 | 80.86 |
Organizers
Contact Information:
- Dr. Harold Mouchère - harold.mouchere@univ-nantes.fr
- Pr. Christian Viard-Gaudin - christian.viard-gaudin@univ-nantes.fr
- Dr. Richard Zanibbi - rlaz@cs.rit.edu
- Pr. J. H. Kim
- Mr. Dae Hwan Kim - ofreunde.edward@gmail.com
- Dr. Utpal Garain - utpal@isical.ac.in
License and Copyright
These CROHME 2013 (Competition on Recognition of Handwritten Mathematical Expressions 2013 dataset) is Copyright © 2013, Université de Nantes / CNRS. These CROHME 2013 is free software and data; you can redistribute it and/or modify it under the terms of the Creative Commons CC BY-NC-SA 3.0 (Attribution-NonCommercial-ShareAlike 3.0 Unported)
The CROHME 2013 Train set merges several existing data sets which keep their original copyrigths:
- expressmatch: University of Sao Paulo
- MathBrush: University of Waterloo
- KAIST: KAIST lab
- MfrDB: CzechTechnical University
- HAMEX: University of Nantes
The CROHME 2012 (Competition on Recognition of Handwritten Mathematical Expressions 2012 dataset) is Copyright © 11/05/2011, Université de Nantes / CNRS. These CROHME 2012 is free software and data; you can redistribute it and/or modify it under the terms of the Creative Commons CC BY-NC-SA 3.0 (Attribution-NonCommercial-ShareAlike 3.0 Unported)
These data and software are distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Creative Commons License CC BY-NC-SA 3.0 for more details. You should have received a copy of the Creative Commons License along with this program; if not, you can also find the Creative Commons licence on the Creative Commons web site.
Non-free versions of this dataset are available under terms different from those of the Creative Commons. For these alternative terms you must purchase a license from one of the authors' laboratory. Users interested in such a license should contact them for more information.
References
In addition, we kindly ask you to acknowledge CROHME and its authors in any program or publication in which you use these CROHME. (You are not required to do so; it is up to your common sense to decide whether you want to comply with this request or not.) For general publications, we suggest referencing one of these references:
- ICDAR 2013 CROHME: Third International Competition on Recognition of Online Handwritten Mathematical Expressions.
H. Mouchere, C. Viard-Gaudin, R. Zanibbi, U. Garain, D.H. Kim and J.H. Kim
Proceedings of the International Conference on Document Analysis and Recognition, Washington, DC (2013). - ICFHR 2012 - Competition on Recognition of Online Mathematical Expressions (CROHME2012),
Mouchère H., Viard-Gaudin C., Garain U., Kim D. H. and Kim J. H.
Proceedings of the International Conference on Frontiers in Handwriting Recognition, ICFHR 2012, Italy (2012)