Explainable Artificial Intelligence (VTR UGE 21), day 3 of 5

Explaining Deep Networks

Prof. Dr. Utku Kose, Süleyman Demirel University

Saliency lab

Part A shows maps of six methods computed by the network of the notebook on real test images. Part B asks whether a map came from the trained network or from random weights. Part C computes Grad-CAM, short for gradient-weighted class activation mapping, by hand.

Part A: Six methods, trained and random

The maps below were computed by the NumPy network of the notebook on real test images: six positive images, all with the corner marker, and two negative images without it. Choose an image and a method, and compare the trained network with the same architecture with random weights. A faithful map should depend on the weights; a map that looks the same for random weights is showing the image, not the model.

Open in ColabContinue in Colab, section 3: compute the maps yourself and choose any test image.

Part B: Trained or random?

Each map below was computed by the network of the notebook, either with its trained weights or with random weights. A map that explains the model should look different when the weights are random. Decide for each map and press Check my answers.

Open in ColabContinue in Colab, section 5: the full sanity check with rank correlations.

Part C: Grad-CAM by hand

A convolutional layer produced two maps of 3 by 3 cells. Grad-CAM weights each map by the average of its gradient, 0.5 for map 1 and -0.25 for map 2, adds the weighted maps and keeps only positive values. Compute three cells.

map 1        map 2
0  1  2      2  0  0
1  3  1      0  1  0
0  1  0      0  0  3

Open in ColabContinue in Colab, section 4: Grad-CAM and its variant on real images.