Model families ๐พ

Model architectures ๐

Metrics ๐

Solvers / Optimizers ๐งฎ

Training parameters

Augmentations

Deployment

mAP (mean Average Precision)

Balancing the precision <> recall tradeoff

Mean average precision (mAP) is the most meaningful metric for object detectors, instance, and semantic segmentors. It incorporates the trade-off between precision and recall, and in doing so, takes into account both types of errors, false positives (FP) and false negatives (FN). This property makes mAP applicable for most use cases.

Calculation / interpretation

mAP is based on the IoU, precision, and recall metrics, as well as a confusion matrix. So, make sure to read the entries if you're not familiar with them.

Quick recap: how to calculate a confusion matrix

As the mAP is based on precision and recall, it also relies on the confusion matrix. For object detectors, instance, and semantic segmentors, the confusion matrix is computed by checking if the class of the prediction fits the class from the ground truth and checking if the IoU is above a certain threshold, most of the time above 0.5.

Average Precision (AP) is computed for one class

Don't let the term Average Precision fool you. It's not simply an average of a few precision values, but the result if you average out the **Precision-Recall Curve **for one class.

https://github.com/ultralytics/yolov3/issues/898

The Precision-Recall Curve** **depicts the conflict between precision and recall. Typically,

It is constructed by computing the precision and recall values for different confidence thresholds. *I.e., if the confidence threshold is at 0.9, only predictions where the classifier has a confidence of over 90% are counted as positive predictions. The **precision** will be pretty high. Vice versa, if the threshold is at 0.1, much more predictions will be included in the positive ones and **recall** will be large. *

The Average Precision is the area under the curve. Perfect models would have an AP of 1.0.

Mean Average Precision (mAP) is computed for many classes

Once you know how to calculate the AP, computing the mAP is easy:

The mAP is the mean of the AP over all classes.

Often, you see the mAP cited as **[email protected]** or **[email protected]**. This notation depicts which IoU threshold has been used to calculate the confusion matrix. This is important to note because the IoU threshold can influence the mAP substantially: a low IoU will boost your mAP.

Sometimes, for example, for the COCO dataset, the mAP is benchmarked by averaging it out over IoUs from [.5,.95] in .05 steps.

Object detection results on COCO

Research Paper by Yi Lin et al.

Code implementation

Numpy

TensorFlow

1

import numpy as np

2

โ

3

def apk(actual, predicted, k=10):#Computes the average precision at k.

4

#This function computes the average prescision at k between two lists of items.

5

โ

6

if not actual:

7

return 0.0

8

โ

9

return score / min(len(actual), k)

10

โ

11

def mapk(actual, predicted, k=10):#Computes the mean average precision at k.

12

#This function computes the mean average prescision at k between two lists of lists of items.

13

14

return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])

15

โ

Copied!

1

!pip install tensorflow==1.15 #Make sure you have updated the Tensorflow version for tf.metrics.average_precision_at_k to work

2

import tensorflow as tf

3

import numpy as np

4

โ

5

y_true = np.array([[2], [1], [0], [3], [0]]).astype(np.int64)

6

y_true = tf.identity(y_true)

7

โ

8

y_pred = np.array([[0.1, 0.2, 0.6, 0.1],

9

[0.8, 0.05, 0.1, 0.05],

10

[0.3, 0.4, 0.1, 0.2],

11

[0.6, 0.25, 0.1, 0.05],

12

[0.1, 0.2, 0.6, 0.1]

13

]).astype(np.float32)

14

y_pred = tf.identity(y_pred)

15

โ

16

_, m_ap = tf.metrics.average_precision_at_k(y_true, y_pred, 3)

17

โ

18

sess = tf.Session()

19

sess.run(tf.local_variables_initializer())

20

โ

21

stream_vars = [i for i in tf.local_variables()]

22

โ

23

tf_map = sess.run(m_ap)

24

print(tf_map)

25

โ

26

print((sess.run(stream_vars)))

27

โ

28

tmp_rank = tf.nn.top_k(y_pred,3)

29

โ

30

print(sess.run(tmp_rank))

Copied!

Related entries

Last modified 3mo ago