3.6.5 Performance, Evaluation Issues

Chapter Contents (Back)
Performance. Evaluation.

Reproducible Research,
OnlineJanuary 2009.
WWW Link. Survey, Evaluation. Evaluation, General. 0906
A site for gathering information regarding reproducible research, which while common in some fields has been lacking in many parts of signal and iamge processing. BibRef

Nagy, G.,
Candide's Practical Principles of Experimental Pattern Recognition,
PAMI(5), No. 2, March 1983, pp. 199-200. BibRef 8303

Cantoin, V., Guerra, C., Levialdi, S.,
Towards an Evaluation of an Image Processing System,
CSIP83(43-56). BibRef 8300

Preston, Jr., K.[Kendall],
The Abingdon Cross Benchmark Survey,
Computer(22), No. 7, July 1989, pp. 9-18. Benchmarks. BibRef 8907

Preston, Jr., K.[Kendall],
Benchmark results: the Abingdon cross,
EvalMulti1986, pp. 23-54. BibRef 8600

Duff, M.J.B.,
How not to Benchmark Image Processors,
EvalMulti1986, pp. 23-54. BibRef 8600

Evaluation of Multicomputers in Image Processing,
Academic PressOrlando, FL, USA, 1986. Apparently a conference in Tuscon, AZ. BibRef 8600

Haralick, R.M.[Robert M.],
Performance Assessment of Near-Perfect Machines,
MVA(2), No. 1, 1989, pp. 1-16. BibRef 8900

Haralick, R.M.[Robert M.],
Overview: Computer Vision Performance Characterization,
ARPA94(I:663-665). BibRef 9400
Performance Characterization Protocol in Computer Vision,
ARPA94(I:667-673). BibRef
And: Tech report, Univ. of Washington1990. BibRef

Haralick, R.M.,
Propagating Covariance in Computer Vision,
PRAI(10), 1996, pp. 561-572. BibRef 9600
And: AIU96(142-156). BibRef
Earlier: ICPR94(A:493-498).
Covariance Propagation.
See also Performance Characterization in Computer Vision. BibRef

Haralick, R.M.,
Covariance Propagation in Computer Vision,
HTML Version. BibRef 9600

Haralick, R.M.,
Detection Performance Methodology,
ARPA96(981-983). BibRef 9600

Liu, X.F.[Xu-Fei], Kanungo, T.[Tapas], Haralick, R.M.[Robert M.],
On the Use of Error Propagation for Statistical Validation of Computer Vision Software,
PAMI(27), No. 10, October 2005, pp. 1603-1614.
Error Propagation and Statistical Validation of Computer Vision Software,
UMD--TR4213, February 2001.
WWW Link. Error analysis for complex computer vision software. Statistically validate the implementation for building extraction programs. BibRef

Liu, X.F.[Xu-Fei], Kanungo, T.[Tapas], Haralick, R.M.[Robert M.],
Statistical Validation of Computer Vision Software,
ARPA96(1533-1540). BibRef 9600

Kanungo, T.[Tapas], and Haralick, R.M.[Robert M.],
Multivariate Hypothesis Testing for Gaussian Data: Theory and Software,
Tech report, Univ. of WashingtonISL-TR-95-05, 23 October, 1995 BibRef 9510

Yi, S., Haralick, R.M., Shapiro, L.G.,
Error Propagation in Machine Vision,
MVA(7), 1994, pp. 93-114.
See also Performance Characterization in Computer Vision. BibRef 9400

Price, K.E.,
Anything You Can Do, I Can Do Better (No You Can't),
CVGIP(36), No. 2-3, November-December 1986, pp. 387-391.
Elsevier DOI BibRef 8611 USC Computer Vision BibRef
I've seen your demo; so what?,
CVWS85(122-124). Some rules on how to make work more transportable and judgeable. BibRef

Kittler, J.V., and Devijver, P.A.,
The Probability Distribution of the Conditional Classification Error,
PAMI(2), 1980, pp. 259-261. BibRef 8000

Kittler, J.V., and Devijver, P.A.,
Statistical Properties of Error Estimators in Performance Assessment of Recognition Systems,
PAMI(4), No. 2, March 1982, pp. 215-220. BibRef 8203

Wu, Z.,
Homogeneity Testing for Unlabeled Data: A Performance Evaluation,
GMIP(55), No. 5, 1993, pp. 370-380. BibRef 9300

Logan, B.J.,
At What Price Inaccuracy,
PhEngRS(62), No. 6, June 1996, pp. 685-685. 9606

Jiang, B.C., Shiau, M.Y.R.,
A Systematic Methodology for Determining/Optimizing a Machine Vision System's Capability,
MVA(3), 1990, pp. 169-182. BibRef 9000

Aksyutov, L.N.,
Prediction of Object Recognition Probability from Space Photographs,
EORS(13), No. 2, 1995, pp. 163-175. 9607

Diaspro, A., Parodi, G., Zunino, R.,
A Performance Analysis of an Associative System for Image Classification,
PRL(14), 1993, pp. 801-868. BibRef 9300

Shanbehzadeh, J., Ogunbona, P.O.,
On the Computational-Complexity of the LBG and PNN Algorithms,
IP(6), No. 4, April 1997, pp. 614-616.

Christensen, H.I., Förstner, W.,
Performance-Characteristics Of Vision Algorithms,
MVA(9), No. 5-6, 1997, pp. 215-218.
Springer DOI 9705
Introduction to the special issue derived from the conference. BibRef

Förstner, W.,
Diagnostics and Performance Evaluation in Computer Vision,
Robust94(XX-YY). Seattle, USA. BibRef 9400

Venetianer, P.L., Large, E.W., Bajcsy, R.,
A Methodology for Evaluation of Task-Performance in Robotic Systems: A Case-Study in Vision-Based Localization,
MVA(9), No. 5-6, 1997, pp. 304-320.
Springer DOI 9705

Sheinvald, J., Kiryati, N.,
On The Magic Of Slide,
MVA(9), No. 5-6, 1997, pp. 251-261.
Springer DOI 9705

Hay, G.J., Niemann, K.O., Goodenough, D.G.,
Spatial Thresholds, Image-Objects, and Upscaling: A Multiscale Evaluation,
RSE(62), No. 1, October 1997, pp. 1-19. 9709

Sunar, F., Kaya, S.,
An Assessment of the Geometric Accuracy of Remotely-Sensed Images,
JRS(18), No. 14, September 20 1997, pp. 3069-3074. 9710

Förstner, W.,
Reliability Analysis of Parameter Estimation in Linear Models with Applications to Mensuration Problems in Computer Vision,
CVGIP(40), No. 3, December 1987, pp. 273-310.
Elsevier DOI Measurement. Geometric Features, Evaluation. This paper discusses the general ideas behind error analysis and discusses how measurements from images should be done. BibRef 8712

Förstner, W.,
A Framework For Low Level Feature Extraction,
Springer DOI BibRef 9400

Ho, C.S.,
Precision of Digital Vision Systems,
PAMI(5), No. 6, November 1983, pp. 593-601. Digital Accuracy, Evaluation. Studies of possible variation in digital measurements of object features. BibRef 8311

Kamgar-Parsi, B.[Behrooz], and Kamgar-Parsi, B.[Behzad],
Evaluation of Quantization Error in Computer Vision,
PAMI(11), No. 9, September 1989, pp. 929-940.
IEEE DOI BibRef 8909
Earlier: CVPR88(52-60).
Earlier: DARPA88(720-730). BibRef

Kamgar-Parsi, B.[Behzad], and Kamgar-Parsi, B.[Behrooz],
Quantization Error in Hexagonal Sensory Configurations,
PAMI(14), No. 6, June 1992, pp. 665-671.
IEEE DOI Quantization Error, Evaluation. BibRef 9206

Kamgar-Parsi, B., Kamgar-Parsi, B.,
Quantization Error in Regular Grids: Triangular Pixels,
IP(7), No. 10, October 1998, pp. 1496-1500.
IEEE DOI BibRef 9810

Kamgar-Parsi, B.[Behzad], Kamgar-Parsi, B.[Behrooz], and Sander, III, W.A.,
Quantization Error in Spatial Sampling: Comparison between Square and Hexagonal Pixels,
IEEE DOI BibRef 8900

Wong, P.W.,
On Quantization Errors in Computer Vision,
PAMI(13), No. 9, September 1991, pp. 951-956.
IEEE DOI BibRef 9109

Bunch, J.R., Leborne, R.C., Proudler, I.K.,
Tracking Ill-Conditioning for the RLS-Lattice Algorithms,
VISP(145), No. 1, February 1998, pp. 1-5. 9804

Pearson, J.J., Oddo, L.A.,
A Testbed for the Evaluation of Feature Extraction Techniques in a Time Constrained Environment,
Ascona97(13-22). BibRef 9700

Müller, J.P., Ourzik, C., Kim, T., Dowman, I.J.,
Assessment of the Effects of Resolution on Automated DEM and Building Extraction,
Ascona97(233-242). Building Recognition. BibRef 9700

Courtney, P., Thacker, N.A., Clark, A.F.,
Algorithmic Modeling For Performance Evaluation,
MVA(9), No. 5-6, 1997, pp. 219-228.
Springer DOI 9705
Algorithmic Modelling for Performance Evaluation,
HTML Version. BibRef

Marik, R., Petrou, M., Kittler, J.V.,
Error Sensitivity Assessment of Vision Algorithms,
VISP(145), No. 2, April 1998, pp. 124-130. 9806
Error Sensitivity Assessment of Vision Algorithms Based on Direct Error Propagation,
HTML Version. BibRef

Bowyer, K.W., Phillips, P.J.,
Empirical Evaluation Techniques in Computer Vision,
CS-Press1998. ISBN: 0818684011. Indexed as: BibRef 9800 EEMTV98
WWW Link.
See also Workshop on Empirical Evaluation Methods in Computer Vision. Mostly seems to be from the conference, but not all of them are. BibRef

Phillips, P.J., Bowyer, K.W.,
Introduction to the Special Section on Empirical Evaluation of Computer Vision Algorithms,
PAMI(21), No. 4, April 1999, pp. 289-290.
IEEE DOI BibRef 9904

Bowyer, K.W.[Kevin W.], and Phillips, P.J.[P. Jonathon],
Overview of Work in Empirical Evaluation of Computer Vision Algorithms,
EEMTV98(xx) BibRef 9800

Flynn, P.J.[Patrick J.], Hoover, A.[Adam], Phillips, P.J.[P. Jonathon],
Special Issue on Empirical Evaluation of Computer Vision Algorithms,
CVIU(84), No. 1, October 2001, pp. 1-4.
DOI Link 0203

Hoppin, J.W., Kupinski, M.A., Kastis, G.A., Clarkson, E., Barrett, H.H.,
Objective comparison of quantitative imaging modalities without the use of a gold standard,
MedImg(21), No. 5, May 2002, pp. 441-449.
IEEE Top Reference. 0206

Klette, R.[Reinhard], Stiehl, H.S.[H. Siegfried], Viergever, M.A.[Max A.], Vincken, K.L.[Koen L.],
Performance Characterization in Computer Vision,
KluwerAugust 2000, ISBN 0-7923-6374-4
WWW Link. Proceedings for the 9th Theoretical Foundations of Computer Vision. Buy this book: Performance Characterization In Computer Vision (Computational Imaging and Vision) BibRef 0008

Min, J.[Jaesik], Powell, M.W.[Mark W.], Bowyer, K.W.[Kevin W.],
Automated Performance Evaluation of Range Image Segmentation Algorithms,
SMC-B(34), No. 1, February 2004, pp. 263-271.
IEEE Abstract. 0403
Automated Performance Evaluation of Range Image Segmentation,
Code, Segmenation Evaluation.
HTML Version. BibRef

Min, J.[Jaesik], Powell, M.W.[Mark W.], Bowyer, K.W.[Kevin W.],
Progress in Automated Evaluation of Curved Surface Range Image Segmentation,
ICPR00(Vol I: 644-647).

Vandewalle, P.[Patrick], Kovacevic, J.[Jelena], Vetterli, M.[Martin],
Reproducible research in signal processing,
SPMag(26), No. 3, May 2009, pp. 37-47.
IEEE DOI A discussion of the need to raise evaluation quality for signal processing. I.e. compare to established algorithms, use large data sets, make code available online. BibRef 0905

Liu, W.Y.[Wen-Yin], Valveny, E.[Ernest],
Special Issue on Performance Evaluation,
IJDAR(14), No. 1, March 2011, pp. 1-2.
WWW Link. 1103
In Character and document analysis. BibRef

Andreopoulos, A.[Alexander], Tsotsos, J.K.[John K.],
On Sensor Bias in Experimental Methods for Comparing Interest-Point, Saliency, and Recognition Algorithms,
PAMI(34), No. 1, January 2012, pp. 110-126.
Other effects on algorithm performance from camera settings. BibRef

Zendel, O.[Oliver], Murschitz, M.[Markus], Humenberger, M.[Martin], Herzner, W.[Wolfgang],
How Good Is My Test Data? Introducing Safety Analysis for Computer Vision,
IJCV(125), No. 1-3, December 2018, pp. 95-109.
Springer DOI 1711
CV-HAZOP: Introducing Test Data Validation for Computer Vision,
Apply to stereo. Check list of 900 hazards. Benchmark testing. Find the tests that are a problem. BibRef

Lapin, M.[Maksim], Hein, M.[Matthias], Schiele, B.[Bernt],
Analysis and Optimization of Loss Functions for Multiclass, Top-k, and Multilabel Classification,
PAMI(40), No. 7, July 2018, pp. 1533-1554.
Loss Functions for Top-k Error: Analysis and Insights,
Algorithm design and analysis, Benchmark testing, Calibration, Loss measurement, Optimization, Support vector machines, Training, top-k error. Evaluation of evaluation techniques. Large test datasets may have other criteria. BibRef

Eisenmann, M., Reinke, A., Weru, V., Tizabi, M.D., Isensee, F., Adler, T.J., Ali, S., Andrearczyk, V., Aubreville, M., Baid, U., Bakas, S., Balu, N., Bano, S., Bernal, J., Bodenstedt, S., Casella, A., Cheplygina, V., Daum, M., de Bruijne, M., Depeursinge, A., Dorent, R., Egger, J., Ellis, D.G., Engelhardt, S., Ganz, M., Ghatwary, N., Girard, G., Godau, P., Gupta, A., Hansen, L., Harada, K., Heinrich, M., Heller, N., Hering, A., Huaulmé, A., Jannin, P., Kavur, A.E., Kodym, O., Kozubek, M., Li, J., Li, H., Ma, J., Martín-Isla, C., Menze, B., Noble, A., Oreiller, V., Padoy, N., Pati, S., Payette, K., Rädsch, T., Rafael-Patińo, J., Bawa, V.S.[V. Singh], Speidel, S., Sudre, C.H., van Wijnen, K., Wagner, M., Wei, D., Yamlahi, A., Yap, M.H., Yuan, C., Zenk, M., Zia, A., Zimmerer, D., Aydogan, D., Bhattarai, B., Bloch, L., Brüngel, R., Cho, J., Choi, C., Dou, Q., Ezhov, I., Friedrich, C.M., Fuller, C., Gaire, R.R., Galdran, A., Faura, Á.G.[Á. García], Grammatikopoulou, M., Hong, S., Jahanifar, M., Jang, I., Kadkhodamohammadi, A., Kang, I., Kofler, F., Kondo, S., Kuijf, H., Li, M., Luu, M., Martincic, T., Morais, P., Naser, M.A., Oliveira, B., Owen, D., Pang, S., Park, J., Park, S., Plotka, S., Puybareau, E., Rajpoot, N., Ryu, K., Saeed, N., Shephard, A., Shi, P., Štepec, D., Subedi, R., Tochon, G., Torres, H.R., Urien, H., Vilaça, J.L., Wahid, K.A., Wang, H., Wang, J., Wang, L., Wang, X., Wiestler, B., Wodzinski, M., Xia, F., Xie, J., Xiong, Z., Yang, S., Yang, Y., Zhao, Z., Maier-Hein, K., Jäger, P.F., Kopp-Schneider, A., Maier-Hein, L.,
Why is the Winner the Best?,

Sahu, P.[Pritish], Sikka, K.[Karan], Divakaran, A.[Ajay],
Challenges in Procedural Multimodal Machine Comprehension: A Novel Way To Benchmark,
Visualization, Systematics, Correlation, Statistical analysis, Benchmark testing, Transformers, Cognition, Datasets, Analysis and Understanding BibRef

Lou, Y.J.[Yu-Jing], You, Y.[Yang], Li, C.K.[Cheng-Kun], Cheng, Z.J.[Zhou-Jun], Li, L.W.[Liang-Wei], Ma, L.Z.[Li-Zhuang], Wang, W.M.[Wei-Ming], Lu, C.W.[Ce-Wu],
Human Correspondence Consensus for 3d Object Semantic Understanding,
Springer DOI 2011
Human consensus varies on different aspects of the problem. BibRef

Zendel, O., Honauer, K., Murschitz, M., Humenberger, M., Domínguez, G.F.,
Analyzing Computer Vision Data: The Good, the Bad and the Ugly,
Algorithm design and analysis, Cameras, Measurement, Roads, Simultaneous localization and mapping, Stereo, vision BibRef

Rivera-Rubio, J.[Jose], Idrees, S.[Saad], Alexiou, I.[Ioannis], Hadjilucas, L.[Lucas], Bharath, A.A.[Anil A.],
A dataset for Hand-Held Object Recognition,
Dataset, Object Recognition. BibRef
Small Hand-Held Object Recognition Test (SHORT),
Mobile Visual Assistive Apps: Benchmarks of Vision Algorithm Performance,
Springer DOI 1309
Computer vision Cameras BibRef

Gala, A.[Apurva], Shah, S.[Shishir],
Joint Modeling of Algorithm Behavior and Image Quality for Algorithm Performance Prediction,
HTML Version. 1009

Scrapper, C., Madhavan, R., Balakirsky, S.,
Using a High-Fidelity Simulation Framework for Performance Singularity,
Without ground truth, algorithm evaluation to find singularities in results. BibRef

Fraedrich, D.,
Validation Techniques for Image-Based Simulations,

Zanibbi, R.[Richard], Blostein, D.[Dorothea], Cordy, J.R.[James R.],
White-Box Evaluation of Computer Vision Algorithms through Explicit Decision-Making,
Springer DOI 0910

Tesser, H.[Herbert], Trout, T.[Theron],
A Note on Evaluation of Image Recognition Systems,
Springer DOI 0310

Eberst, C.[Christof], Herbig, T.[Thomas],
On the Application of the Concept of Dependability for Design and Analysis of Vision Systems,
CVS03(290 ff).
Springer DOI 0306

Lucas, S., Sarampalis, K.,
Automatic Evaluation of Algorithms Over the Internet,
ICPR00(Vol II: 471-474).

Tu, P., Hartley, R.,
Statistical Significance as an Aid to System Performance Evaluation,
ECCV00(II: 366-378).
Springer DOI 0003

Yachik, T.R.[Theodore R.], Gilfillan, L.[Lynne],
Evaluations of Large, Complex Research and Development Programs: Theory and Practice,
DARPA97(1291-1304). BibRef 9700

Jensen, E.S.[Eric S.], Thompson, W.B.[William B.],
Quantitative Comparison of IU Algorithms,
DARPA97(1007-1010). BibRef 9700

Appenzeller, G., Crowley, J.L.,
Experimental Performance Characterization of Low Level Vision Components in Vision Systems: Theory and Application,
IMAG-PRIMA1995. BibRef 9500

Stoica, P.,
Performance Evaluation of Some Methods for Off-Line Detection of Changes In Autoregressive Signals,
PESIPS93(XX-YY). BibRef 9300

Vogt, R.C.,
The Role of Performance Evaluation in Automated Image Algorithm Generation,
See also Automatic Generation of Simple Morphological Algorithms. BibRef 9300

Weber, W.G., Ulich, E.,
Psychological Criteria for the Evaluation of Different Forms of Group Work in Advanced Manufacturing Systems,
HCI93(26-31). BibRef 9300

Kirsch, C.,
Evaluation of Communication Methods for User Participation in Data Modeling,
HCI93(558-63). BibRef 9300

Nielsen, J.,
Characterization of Vision Algorithms: An Experimental Approach,
Bench95(XX-YY). BibRef 9500

Wu, H.R., Paplinski, A.P., Jian, Q.X., Yuen, M.,
Performance Evaluation of Spatial Dynamic Motion Compensation Algorithms,
SPIE(2419), 1995, pp. xx-yy. San Jose, California. BibRef 9500

Courtney, P., Skordas, T.,
Caracterisation De Performances Des Algorithmes De Vision: Un Exemple: Le Detecteur De Coins,
Proceedings RFIA 10 1996(953-962). Rennes, France. BibRef 9600

Hori, O., Doermann, D.S.,
Quantitative Measurement of the Performance of Raster-to-Vector Conversion Algorithms,
GRMA951996, pp. 57-68. BibRef 9600

Christmas, W.J., Kittler, J.V., and Petrou, M.,
Error Propagation for 2D-to-3D Matching with Application to Underwater Navigation,
BMVC96(Poster Session 2). 9608
University of Surrey BibRef

Förstner, W.,
10 Pros and Cons Against Performance Characterisation of Vision Algorithms,
HTML Version.
HTML Version. BibRef 9600

Ramesh, V., Haralick, R.M.,
Random perturbation models and performance characterization in computer vision,

Petkovic, D.,
The Need for Accuracy Verification of Machine Vision Algorithms and Systems,
IEEE DOI BibRef 8900

Chapter on Books, Collections, Overviews, General, and Surveys continues in
Education Issues, Instructional Media .

Last update:Nov 30, 2023 at 15:51:27