3.6.5 Performance, Evaluation Issues

Chapter Contents (Back)
Performance. Evaluation.

Reproducible Research,
OnlineJanuary 2009.
WWW Link. Survey, Evaluation. Evaluation, General. 0906
A site for gathering information regarding reproducible research, which while common in some fields has been lacking in many parts of signal and iamge processing. BibRef

Nagy, G.,
Candide's Practical Principles of Experimental Pattern Recognition,
PAMI(5), No. 2, March 1983, pp. 199-200. BibRef 8303

Cantoin, V., Guerra, C., Levialdi, S.,
Towards an Evaluation of an Image Processing System,
CSIP83(43-56). BibRef 8300

Preston, Jr., K.[Kendall],
The Abingdon Cross Benchmark Survey,
Computer(22), No. 7, July 1989, pp. 9-18. Benchmarks. BibRef 8907

Preston, Jr., K.[Kendall],
Benchmark results: the Abingdon cross,
EvalMulti1986, pp. 23-54. BibRef 8600

Duff, M.J.B.,
How not to Benchmark Image Processors,
EvalMulti1986, pp. 23-54. BibRef 8600

Evaluation of Multicomputers in Image Processing,
Academic PressOrlando, FL, USA, 1986. Apparently a conference in Tuscon, AZ. BibRef 8600

Haralick, R.M.[Robert M.],
Performance Assessment of Near-Perfect Machines,
MVA(2), No. 1, 1989, pp. 1-16. BibRef 8900

Haralick, R.M.[Robert M.],
Overview: Computer Vision Performance Characterization,
ARPA94(I:663-665). BibRef 9400
And:
Performance Characterization Protocol in Computer Vision,
ARPA94(I:667-673). BibRef
And: Tech report, Univ. of Washington1990. BibRef

Haralick, R.M.,
Propagating Covariance in Computer Vision,
PRAI(10), 1996, pp. 561-572. BibRef 9600
And: AIU96(142-156). BibRef
Earlier: ICPR94(A:493-498).
IEEE DOI 9410
Covariance Propagation.
See also Performance Characterization in Computer Vision. BibRef

Haralick, R.M.,
Covariance Propagation in Computer Vision,
PERF96(XX-YY).
HTML Version. BibRef 9600

Haralick, R.M.,
Detection Performance Methodology,
ARPA96(981-983). BibRef 9600

Liu, X.F.[Xu-Fei], Kanungo, T.[Tapas], Haralick, R.M.[Robert M.],
On the Use of Error Propagation for Statistical Validation of Computer Vision Software,
PAMI(27), No. 10, October 2005, pp. 1603-1614.
IEEE DOI 0509
BibRef
Earlier:
Error Propagation and Statistical Validation of Computer Vision Software,
UMD--TR4213, February 2001.
WWW Link. Error analysis for complex computer vision software. Statistically validate the implementation for building extraction programs. BibRef

Liu, X.F.[Xu-Fei], Kanungo, T.[Tapas], Haralick, R.M.[Robert M.],
Statistical Validation of Computer Vision Software,
ARPA96(1533-1540). BibRef 9600

Kanungo, T.[Tapas], and Haralick, R.M.[Robert M.],
Multivariate Hypothesis Testing for Gaussian Data: Theory and Software,
Tech report, Univ. of WashingtonISL-TR-95-05, 23 October, 1995 BibRef 9510

Yi, S., Haralick, R.M., Shapiro, L.G.,
Error Propagation in Machine Vision,
MVA(7), 1994, pp. 93-114.
See also Performance Characterization in Computer Vision. BibRef 9400

Price, K.E.,
Anything You Can Do, I Can Do Better (No You Can't),
CVGIP(36), No. 2-3, November-December 1986, pp. 387-391.
Elsevier DOI BibRef 8611 USC Computer Vision BibRef
Earlier:
I've seen your demo; so what?,
CVWS85(122-124). Some rules on how to make work more transportable and judgeable. BibRef

Kittler, J.V., and Devijver, P.A.,
The Probability Distribution of the Conditional Classification Error,
PAMI(2), 1980, pp. 259-261. BibRef 8000

Kittler, J.V., and Devijver, P.A.,
Statistical Properties of Error Estimators in Performance Assessment of Recognition Systems,
PAMI(4), No. 2, March 1982, pp. 215-220. BibRef 8203

Wu, Z.,
Homogeneity Testing for Unlabeled Data: A Performance Evaluation,
GMIP(55), No. 5, 1993, pp. 370-380. BibRef 9300

Logan, B.J.,
At What Price Inaccuracy,
PhEngRS(62), No. 6, June 1996, pp. 685-685. 9606
BibRef

Jiang, B.C., Shiau, M.Y.R.,
A Systematic Methodology for Determining/Optimizing a Machine Vision System's Capability,
MVA(3), 1990, pp. 169-182. BibRef 9000

Aksyutov, L.N.,
Prediction of Object Recognition Probability from Space Photographs,
EORS(13), No. 2, 1995, pp. 163-175. 9607
BibRef

Diaspro, A., Parodi, G., Zunino, R.,
A Performance Analysis of an Associative System for Image Classification,
PRL(14), 1993, pp. 801-868. BibRef 9300

Shanbehzadeh, J., Ogunbona, P.O.,
On the Computational-Complexity of the LBG and PNN Algorithms,
IP(6), No. 4, April 1997, pp. 614-616.
IEEE DOI 9704
BibRef

Christensen, H.I., Förstner, W.,
Performance-Characteristics Of Vision Algorithms,
MVA(9), No. 5-6, 1997, pp. 215-218.
Springer DOI 9705
Introduction to the special issue derived from the conference. BibRef

Förstner, W.,
Diagnostics and Performance Evaluation in Computer Vision,
Robust94(XX-YY). Seattle, USA. BibRef 9400

Venetianer, P.L., Large, E.W., Bajcsy, R.,
A Methodology for Evaluation of Task-Performance in Robotic Systems: A Case-Study in Vision-Based Localization,
MVA(9), No. 5-6, 1997, pp. 304-320.
Springer DOI 9705
BibRef

Sheinvald, J., Kiryati, N.,
On The Magic Of Slide,
MVA(9), No. 5-6, 1997, pp. 251-261.
Springer DOI 9705
BibRef

Hay, G.J., Niemann, K.O., Goodenough, D.G.,
Spatial Thresholds, Image-Objects, and Upscaling: A Multiscale Evaluation,
RSE(62), No. 1, October 1997, pp. 1-19. 9709
BibRef

Sunar, F., Kaya, S.,
An Assessment of the Geometric Accuracy of Remotely-Sensed Images,
JRS(18), No. 14, September 20 1997, pp. 3069-3074. 9710
BibRef

Förstner, W.,
Reliability Analysis of Parameter Estimation in Linear Models with Applications to Mensuration Problems in Computer Vision,
CVGIP(40), No. 3, December 1987, pp. 273-310.
Elsevier DOI Measurement. Geometric Features, Evaluation. This paper discusses the general ideas behind error analysis and discusses how measurements from images should be done. BibRef 8712

Förstner, W.,
A Framework For Low Level Feature Extraction,
ECCV94(B:383-394).
Springer DOI BibRef 9400

Ho, C.S.,
Precision of Digital Vision Systems,
PAMI(5), No. 6, November 1983, pp. 593-601. Digital Accuracy, Evaluation. Studies of possible variation in digital measurements of object features. BibRef 8311

Kamgar-Parsi, B.[Behrooz], and Kamgar-Parsi, B.[Behzad],
Evaluation of Quantization Error in Computer Vision,
PAMI(11), No. 9, September 1989, pp. 929-940.
IEEE DOI BibRef 8909
Earlier: CVPR88(52-60).
IEEE DOI BibRef
Earlier: DARPA88(720-730). BibRef

Kamgar-Parsi, B.[Behzad], and Kamgar-Parsi, B.[Behrooz],
Quantization Error in Hexagonal Sensory Configurations,
PAMI(14), No. 6, June 1992, pp. 665-671.
IEEE DOI Quantization Error, Evaluation. BibRef 9206

Kamgar-Parsi, B., Kamgar-Parsi, B.,
Quantization Error in Regular Grids: Triangular Pixels,
IP(7), No. 10, October 1998, pp. 1496-1500.
IEEE DOI BibRef 9810

Kamgar-Parsi, B.[Behzad], Kamgar-Parsi, B.[Behrooz], and Sander, III, W.A.,
Quantization Error in Spatial Sampling: Comparison between Square and Hexagonal Pixels,
CVPR89(604-611).
IEEE DOI BibRef 8900

Wong, P.W.,
On Quantization Errors in Computer Vision,
PAMI(13), No. 9, September 1991, pp. 951-956.
IEEE DOI BibRef 9109

Bunch, J.R., Leborne, R.C., Proudler, I.K.,
Tracking Ill-Conditioning for the RLS-Lattice Algorithms,
VISP(145), No. 1, February 1998, pp. 1-5. 9804
BibRef

Pearson, J.J., Oddo, L.A.,
A Testbed for the Evaluation of Feature Extraction Techniques in a Time Constrained Environment,
Ascona97(13-22). BibRef 9700

Müller, J.P., Ourzik, C., Kim, T., Dowman, I.J.,
Assessment of the Effects of Resolution on Automated DEM and Building Extraction,
Ascona97(233-242). Building Recognition. BibRef 9700

Courtney, P., Thacker, N.A., Clark, A.F.,
Algorithmic Modeling For Performance Evaluation,
MVA(9), No. 5-6, 1997, pp. 219-228.
Springer DOI 9705
BibRef
Earlier:
Algorithmic Modelling for Performance Evaluation,
PERF96(XX-YY).
HTML Version. BibRef

Marik, R., Petrou, M., Kittler, J.V.,
Error Sensitivity Assessment of Vision Algorithms,
VISP(145), No. 2, April 1998, pp. 124-130. 9806
BibRef
Earlier:
Error Sensitivity Assessment of Vision Algorithms Based on Direct Error Propagation,
PERF96(XX-YY).
HTML Version. BibRef

Bowyer, K.W., Phillips, P.J.,
Empirical Evaluation Techniques in Computer Vision,
CS-Press1998. ISBN: 0818684011. Indexed as: BibRef 9800 EEMTV98
WWW Link.
See also Workshop on Empirical Evaluation Methods in Computer Vision. Mostly seems to be from the conference, but not all of them are. BibRef

Phillips, P.J., Bowyer, K.W.,
Introduction to the Special Section on Empirical Evaluation of Computer Vision Algorithms,
PAMI(21), No. 4, April 1999, pp. 289-290.
IEEE DOI BibRef 9904

Bowyer, K.W.[Kevin W.], and Phillips, P.J.[P. Jonathon],
Overview of Work in Empirical Evaluation of Computer Vision Algorithms,
EEMTV98(xx) BibRef 9800

Flynn, P.J.[Patrick J.], Hoover, A.[Adam], Phillips, P.J.[P. Jonathon],
Special Issue on Empirical Evaluation of Computer Vision Algorithms,
CVIU(84), No. 1, October 2001, pp. 1-4.
DOI Link 0203
BibRef

Hoppin, J.W., Kupinski, M.A., Kastis, G.A., Clarkson, E., Barrett, H.H.,
Objective comparison of quantitative imaging modalities without the use of a gold standard,
MedImg(21), No. 5, May 2002, pp. 441-449.
IEEE Top Reference. 0206
BibRef

Klette, R.[Reinhard], Stiehl, H.S.[H. Siegfried], Viergever, M.A.[Max A.], Vincken, K.L.[Koen L.],
Performance Characterization in Computer Vision,
KluwerAugust 2000, ISBN 0-7923-6374-4
WWW Link. Proceedings for the 9th Theoretical Foundations of Computer Vision. Buy this book: Performance Characterization In Computer Vision (Computational Imaging and Vision) BibRef 0008

Min, J.[Jaesik], Powell, M.W.[Mark W.], Bowyer, K.W.[Kevin W.],
Automated Performance Evaluation of Range Image Segmentation Algorithms,
SMC-B(34), No. 1, February 2004, pp. 263-271.
IEEE Abstract. 0403
BibRef
Earlier:
Automated Performance Evaluation of Range Image Segmentation,
WACV00(163-168).
IEEE DOI 0010
Code, Segmenation Evaluation.
HTML Version. BibRef

Min, J.[Jaesik], Powell, M.W.[Mark W.], Bowyer, K.W.[Kevin W.],
Progress in Automated Evaluation of Curved Surface Range Image Segmentation,
ICPR00(Vol I: 644-647).
IEEE DOI 0009
BibRef

Vandewalle, P.[Patrick], Kovacevic, J.[Jelena], Vetterli, M.[Martin],
Reproducible research in signal processing,
SPMag(26), No. 3, May 2009, pp. 37-47.
IEEE DOI A discussion of the need to raise evaluation quality for signal processing. I.e. compare to established algorithms, use large data sets, make code available online. BibRef 0905

Liu, W.Y.[Wen-Yin], Valveny, E.[Ernest],
Special Issue on Performance Evaluation,
IJDAR(14), No. 1, March 2011, pp. 1-2.
WWW Link. 1103
In Character and document analysis. BibRef

Andreopoulos, A.[Alexander], Tsotsos, J.K.[John K.],
On Sensor Bias in Experimental Methods for Comparing Interest-Point, Saliency, and Recognition Algorithms,
PAMI(34), No. 1, January 2012, pp. 110-126.
IEEE DOI 1112
Other effects on algorithm performance from camera settings. BibRef

Zendel, O.[Oliver], Murschitz, M.[Markus], Humenberger, M.[Martin], Herzner, W.[Wolfgang],
How Good Is My Test Data? Introducing Safety Analysis for Computer Vision,
IJCV(125), No. 1-3, December 2018, pp. 95-109.
Springer DOI 1711
BibRef
Earlier:
CV-HAZOP: Introducing Test Data Validation for Computer Vision,
ICCV15(2066-2074)
IEEE DOI 1602
Apply to stereo. Check list of 900 hazards. Benchmark testing. Find the tests that are a problem. BibRef

Lapin, M.[Maksim], Hein, M.[Matthias], Schiele, B.[Bernt],
Analysis and Optimization of Loss Functions for Multiclass, Top-k, and Multilabel Classification,
PAMI(40), No. 7, July 2018, pp. 1533-1554.
IEEE DOI 1806
BibRef
Earlier:
Loss Functions for Top-k Error: Analysis and Insights,
CVPR16(1468-1477)
IEEE DOI 1612
Algorithm design and analysis, Benchmark testing, Calibration, Loss measurement, Optimization, Support vector machines, Training, top-k error. Evaluation of evaluation techniques. Large test datasets may have other criteria. BibRef

Eisenmann, M., Reinke, A., Weru, V., Tizabi, M.D., Isensee, F., Adler, T.J., Ali, S., Andrearczyk, V., Aubreville, M., Baid, U., Bakas, S., Balu, N., Bano, S., Bernal, J., Bodenstedt, S., Casella, A., Cheplygina, V., Daum, M., de Bruijne, M., Depeursinge, A., Dorent, R., Egger, J., Ellis, D.G., Engelhardt, S., Ganz, M., Ghatwary, N., Girard, G., Godau, P., Gupta, A., Hansen, L., Harada, K., Heinrich, M., Heller, N., Hering, A., Huaulmé, A., Jannin, P., Kavur, A.E., Kodym, O., Kozubek, M., Li, J., Li, H., Ma, J., Martín-Isla, C., Menze, B., Noble, A., Oreiller, V., Padoy, N., Pati, S., Payette, K., Rädsch, T., Rafael-Patiño, J., Bawa, V.S.[V. Singh], Speidel, S., Sudre, C.H., van Wijnen, K., Wagner, M., Wei, D., Yamlahi, A., Yap, M.H., Yuan, C., Zenk, M., Zia, A., Zimmerer, D., Aydogan, D., Bhattarai, B., Bloch, L., Brüngel, R., Cho, J., Choi, C., Dou, Q., Ezhov, I., Friedrich, C.M., Fuller, C., Gaire, R.R., Galdran, A., Faura, Á.G.[Á. García], Grammatikopoulou, M., Hong, S., Jahanifar, M., Jang, I., Kadkhodamohammadi, A., Kang, I., Kofler, F., Kondo, S., Kuijf, H., Li, M., Luu, M., Martincic, T., Morais, P., Naser, M.A., Oliveira, B., Owen, D., Pang, S., Park, J., Park, S., Plotka, S., Puybareau, E., Rajpoot, N., Ryu, K., Saeed, N., Shephard, A., Shi, P., Štepec, D., Subedi, R., Tochon, G., Torres, H.R., Urien, H., Vilaça, J.L., Wahid, K.A., Wang, H., Wang, J., Wang, L., Wang, X., Wiestler, B., Wodzinski, M., Xia, F., Xie, J., Xiong, Z., Yang, S., Yang, Y., Zhao, Z., Maier-Hein, K., Jäger, P.F., Kopp-Schneider, A., Maier-Hein, L.,
Why is the Winner the Best?,
CVPR23(19955-19966)
IEEE DOI 2309
BibRef

Sahu, P.[Pritish], Sikka, K.[Karan], Divakaran, A.[Ajay],
Challenges in Procedural Multimodal Machine Comprehension: A Novel Way To Benchmark,
WACV22(526-535)
IEEE DOI 2202
Visualization, Systematics, Correlation, Statistical analysis, Benchmark testing, Transformers, Cognition, Datasets, Analysis and Understanding BibRef