Abstract
We propose Deep Feature Factorization (DFF), a method capable of localizing similar semantic concepts within an image or a set of images. We use DFF to gain insight into a deep convolutional neural network’s learned features, where we detect hierarchical cluster structures in feature space. This is visualized as heat maps, which highlight semantically matching regions across a set of images, revealing what the network ‘perceives’ as similar. DFF can also be used to perform co-segmentation and co-localization, and we report state-of-the-art results on these tasks.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
- Neural network interpretability
- Part co-segmentation
- Co-segmentation
- Co-localization
- Non-negative matrix factorization
1 Introduction
As neural networks become ubiquitous, there is an increasing need to understand and interpret their learned representations [25, 27]. In the context of convolutional neural networks (CNNs), methods have been developed to explain predictions and latent activations in terms of heat maps highlighting the image regions which caused them [31, 37].
In this paper, we present Deep Feature Factorization (DFF), which exploits non-negative matrix factorization (NMF) [22] applied to activations of a deep CNN layer to find semantic correspondences across images. These correspondences reflect semantic similarity as indicated by clusters in a deep CNN layer feature space. In this way, we allow the CNN to show us which image regions it ‘thinks’ are similar or related across a set of images as well as within a single image. Given a CNN, our approach to semantic concept discovery is unsupervised, requiring only a set of input images to produce correspondences. Unlike previous approaches [2, 11], we do not require annotated data to detect semantic features. We use annotated data for evaluation only.
We show that when using a deep CNN trained to perform ImageNet classification [30], applying DFF allows us to obtain heat maps that correspond to semantic concepts. Specifically, here we use DFF to localize objects or object parts, such as the head or torso of an animal. We also find that parts form a hierarchy in feature space, e.g., the activations cluster for the concept body contains a sub-cluster for limbs, which in turn can be broken down to arms and legs. Interestingly, such meaningful decompositions are also found for object classes never seen before by the CNN.
In addition to giving an insight into the knowledge stored in neural activations, the heat maps produced by DFF can be used to perform co-localization or co-segmentation of objects and object parts. Unlike approaches that delineate the common object across an image set, our method is also able to retrieve distinct parts within the common object. Since we use a pre-trained CNN to accomplish this, we refer to our method as performing weakly-supervised co-segmentation.
Our main contribution is introducing Deep Feature Factorization as a method for semantic concept discovery, which can be used both to gain insight into the representations learned by a CNN, as well as to localize objects and object parts within images. We report results on several datasets and CNN architectures, showing the usefulness of our method across a variety of settings.
2 Related Work
2.1 Localization with CNN Activations
Methods for the interpretation of hidden activations of deep neural networks, and in particular of CNNs, have recently gained significant interest [25]. Similar to DFF, methods have been proposed to localize objects within an image by means of heat maps [31, 37].
In these works [31, 37], localization is achieved by computing the importance of convolutional feature maps with respect to a particular output unit. These methods can therefore be seen as supervised, since the resulting heat maps are associated with a designated output unit, which corresponds to an object class from a predefined set. With DFF, however, heat maps are not associated with an output unit or object class. Instead, DFF heat maps capture common activation patterns in the input, which additionally allows us to localize objects never seen before by the CNN, and for which there is no relevant output unit.
2.2 CNN Features as Part Detectors
The ability of DFF to localize parts stems from the CNN’s ability to distinguish parts in the first place. In Gonzales et al. [11] and Bau et al. [2] the authors attempt to detect learned part-detectors in CNN features, to see if such detectors emerge, even when the CNN is trained with object-level labels. They do this by measuring the overlap between feature map activations and ground truth labels from a part-level segmentation dataset. The availability of ground truth is essential to their analysis, yielding a catalog of CNN units that sufficiently correspond to labels in the dataset.
We confirm their observations that part detectors do indeed emerge in CNNs. However, as opposed to these previous methods, our NMF-based approach does not rely on ground truth labels to find the parts in the input. We use labeled data for evaluation only.
2.3 Non-negative Matrix Factorization
Non-negative matrix factorization (NMF) has been used to analyze data from various domains, such as audio source separation [12], document clustering [36], and face recognition [13].
There has been work extending NMF to multiple layers [6], implementing NMF using neural networks [9] and using NMF approximations as input to a neural network [34]. However, to the best of our knowledge, the application of NMF to the activations of a pre-trained neural network, as is done in DFF, has not been previously proposed.
3 Method
3.1 CNN Feature Space
In the context of CNNs, an input image \(\mathcal {I}\) is seen as a tensor of dimension \(h_\mathcal {I}\times w_\mathcal {I}\times c_\mathcal {I}\), where the first two dimensions are the height and the width of the image, respectively, and the third dimension is the number of color channels, e.g., 3 for RGB. Viewed this way, the first two dimensions of \(\mathcal {I}\) can be seen as a spatial grid, with the last dimension being a \(c_\mathcal {I}\)-dimensional feature representation of a particular spatial position. For an RGB image, this feature corresponds to color.
As the image gets processed layer by layer, the hidden activation at the \(\ell \)th layer of the CNN is a tensor we denote \(\mathcal {A}_\mathcal {I}^\ell \) of dimension \(h_\ell \times w_\ell \times c_\ell \). Notice that generally \(h_\ell< h_\mathcal {I},~w_\ell < w_\mathcal {I}\) due to pooling operations commonly used in CNN pipelines. The number of channels \(c_\ell \) is user-defined as part of the network architecture, and in deep layers is often on the order of 256 or 512.
The tensor \(\mathcal {A}_\mathcal {I}^\ell \) is also called a feature map since it has a spatial interpretation similar to that of the original image \(\mathcal {I}\): the first two dimensions represent a spatial grid, where each position corresponds to a patch of pixels in \(\mathcal {I}\), and the last dimension forms a \(c_\ell \)-dimensional representation of the patch. The intuition behind deep learning suggests that the deeper layer \(\ell \) is, the more abstract and semantically meaningful are the \(c_\ell \)-dimensional features [3].
Since a feature map represents multiple patches (depending on the size of image \(\mathcal {I}\)), we view them as points inhabiting the same \(c_\ell \)-dimensional space, which we refer to as the CNN feature space. Having potentially many points in that space, we can apply various methods to find directions that are ‘interesting’.
3.2 Matrix Factorization
Matrix factorization algorithms have been used for data interpretation for decades. For a data matrix A, these methods retrieve an approximation of the form:
where \(\hat{A}\) is a low-rank matrix of a user-defined rank k. A data point, i.e., a row of A, is explained as a weighted combination of the factors which form the rows of W.
A classical method for dimensionality reduction is principal component analysis (PCA) [18]. PCA finds an optimal k-rank approximation (in the \(\ell ^2\) sense) by solving the following objective:
where \(\Vert .\Vert _F\) denotes the Frobenius norm and \(V_k\in \mathcal {R}^{m\times k}\). For the form of Eq. (1), we set \(H=AV_k,~W=V_k^\top \). Note that the PCA solution generally contains negative values, which means the combination of PCA factors (i.e., principal components) leads to the canceling out of positive and negative entries. This cancellation makes intuitive interpretation of individual factors difficult.
On the other hand, when the data A is non-negative, one can perform non-negative matrix factorization (NMF):
where \(H\in \mathcal {R}^{n\times k}\) and \(W\in \mathcal {R}^{k\times m}\) enforce the dimensionality reduction to rank k. Capturing the structure in A while forcing combinations of factors to be additive results in factors that lend themselves to interpretation [22].
3.3 Non-negative Matrix Factorization on CNN Activations
Many modern CNNs make use of the rectified linear activation function, \(\max (x, 0)\), due to its desirable gradient properties. An obvious property of this function is that it results in non-negative activations. NMF is thus naturally applicable in this case.
Recall the activation tensor for image \(\mathcal {I}\) and layer \(\ell \):
where \(\mathbb {R}\) refers to the set of non-negative real numbers. To apply matrix factorization, we partially flatten \(\mathcal {A}\) into a matrix whose first dimension is the product of h and w:
Note that the matrix \(A_\mathcal {I}^\ell \) is effectively a ‘bag of features’ in the sense that the spatial arrangement has been lost, i.e., the rows of \(A_\mathcal {I}^\ell \) can be permuted without affecting the result of factorization. We can naturally extend factorization to a set of n images, by vertically concatenating their features together:
For ease of notation we assumed all images are of equal size, however, there is no such limitation as images in the set may be of any size. By applying NMF to A we obtain the two matrices from Eq. 1, \(H\in \mathbb {R}^{(n\cdot h\cdot w)\times k}\) and \(W\in \mathbb {R}^{k\times c}\).
3.4 Interpreting NMF Factors
The result returned by the NMF consists of k factors, which we will call DFF factors, where k is the predefined rank of the approximation.
The W Matrix. Each row \(W_j\) (\(1\le j\le k\)) forms a c-dimensional vector in the CNN feature space. Since NMF can be seen as performing clustering [8], we view a factor \(W_j\) as a centroid of an activation cluster, which we show corresponds to coherent object or object-part.
The H Matrix. The matrix H has as many rows as the activation matrix A, one corresponding to every spatial position in every image. Each row \(H_i\) holds coefficients for the weighted sum of the k factors in W, to best approximate the c-dimensional \(A_i\).
Each column \(H_j\) (\(1\le j\le k\)) can be reshaped into n heat maps of dimension \(h\times w\), which highlight regions in each image that correspond to the factor \(W_j\). These heat maps have the same spatial dimensions as the CNN layer which produced the activations, often low. To match the size of the heat map with the input image, we upsample it with bilinear interpolation.
4 Experiments
In this section we first show that DFF can produce a hierarchical decomposition into semantic parts, even for sets of very few images (Sect. 4.3). We then move on to larger-scale, realistic datasets where we show that DFF can perform state-of-the-art weakly-supervised object co-localization and co-segmentation, in addition to part co-segmentation (Sects. 4.4 and 4.5).
4.1 Implementation Details
NMF. NMF optimization with multiplicative updates [23] relies on dense matrix multiplications, and can thus benefit from fast GPU operations. Using an NVIDIA Titan X, our implementation of NMF can process over 6 K images of size \(224\times 224\) at once with \(k=5\), and requires less than a millisecond per image. Our code is available online.
Neural Network Models. We consider five network architectures in our experiments, namely VGG-16 and VGG-19 [32], with and without batch-normalization [17], as well as ResNet-101 [16]. We use the publicly available models from [26].
4.2 Segmentation and Localization Methods
In addition to gaining insights into CNN feature space, DFF has utility for various tasks with subtle but important differences in naming:
-
Segmentation vs. Localization is the difference between predicting pixel-wise binary masks and predicting bounding boxes, respectively.
-
Segmentation vs. co-segmentation is the distinction between segmenting a single image into regions and jointly segmenting multiple images, thereby producing a correspondence between regions in different images (e.g., cats in all images belong to the same segment).
-
Object co-segmentation vs. Part co-segmentation. Given a set of images representing a common object, the former performs binary background-foreground separation where the foreground segment encompasses the entirety of the common object (e.g., cat). The latter, however, produces k segments, each corresponding to a part of the common object (e.g., cat head, cat legs, etc.).
When applying DFF with \(k=1\) can we compare our results against object co-segmentation (background-foreground separation) methods and object co-localization methods.
In Sect. 4.3 we compare DFF against three state-of-the-art co-segmentation methods. The supervised method of Vicente et al. [33] chooses among multiple segmentation proposals per image by learning a regressor to predict, for pairs of images, the overlap between their proposals and the ground truth. Input to the regressor included per-image features, as well as pairwise features. The methods Rubio et al. [29] and Rubinstein et al. [28] are unsupervised and rely on a Markov random field formulation, where the unary features are based on surface image features and various saliency heuristics. For pairwise terms, the former method uses a per-image segmentation into regions, followed by region-matching across images. The latter approach uses a dense pairwise correspondence term between images based on local image gradients.
In Sect. 4.4 we compare against several state-of-the-art object co-localization methods. Most of these methods operate by selecting the best of a set of object proposals, produced by a pre-trained CNN [24] or an object-saliency heuristic [5, 19]. The authors of [21] present a method for unsupervised object co-localization that, like ours, also makes use of CNN activations. Their approach is to apply k-means clustering to globally max-pooled activations, with the intent of clustering all highly active CNN filters together. Their method therefore produces a single heat map, which is appropriate for object co-segmentation, but cannot be extended to part co-segmentation.
When \(k>1\), we use DFF to perform part co-segmentation. Since we have not come across examples of part co-segmentation in the literature, we compare against a method for supervised part segmentation, namely Wang et al. [35] (Table 3 in Sect. 4.5). Their method relies on a compositional model with strong explicit priors w.r.t to part size, hierarchy and symmetry. We also show results for two baseline methods described in [35]: PartBB+ObjSeg where segmentation masks are produced by intersecting part-bounding-boxes [4] with whole-object segmentation masks [14]. The method PartMask+ObjSeg is similar, but here bounding-boxes are replaced with the best of 10 pre-learned part masks.
4.3 Experiments on iCoseg
Dataset. The iCoseg dataset [1] is a popular benchmark for co-segmentation methods. As such, it consists of 38 sets of images, where each image is annotated with a pixel-wise mask encompassing the main object common to the set. Images within a set are uniform in that they were all taken on a single occasion, depicting the same objects. The challenging aspect of this datasets lies in the significant variability with respect to viewpoint, illumination, and object deformation.
We chose five sets and further labeled them with pixel-wise object-part masks (see Table 1). This process involved partitioning the given ground truth mask into sub-parts. We also annotated common background objects, e.g., camel in the Pyramids set (see Fig. 1). Our part-annotation for iCoseg is available online. The number of images in these sets ranges from as few as 5 up to 41. When comparing against [33] and [29] in Table 1, we used the subset of iCoseg used in those papers.
Part Co-segmentation. For each set in iCoseg, we obtained activations from the deepest convolutional layer of VGG19 (conv5_4), and applied NMF to these activations with increasing values of k. The resulting heat maps can be seen in Figs. 1 and 3.
Qualitatively, we see a clear correspondence between DFF factors and coherent object-parts, however, the heat maps are coarse. Due to the low resolution of deep CNN activations, and hence of the heat map, we get blobs that do not perfectly align with the underlying region of interest. We therefore also report additional results with a post-processing step to refine the heat maps, described below.
We notice that when \(k=1\), the single DFF factor corresponds to a whole object, encompassing multiple object-parts. This, however, is not guaranteed, since it is possible that for a set of images, setting \(k=1\) will highlight the background rather than the foreground. Nonetheless, as we increase k, we get a decomposition of the object or scene into individual parts. This behavior reveals a hierarchical structure in the clusters formed in CNN feature space.
For instance, in Fig. 3(a), we can see that \(k=1\) encompasses most of gymnast’s body, \(k=2\) distinguished her midsection from her limbs, \(k=3\) adds a finer distinctions between arms and legs, and finally \(k=4\) adds a new component that localizes the beam. This observation also indicates the CNN has learned representation that ‘explains’ these concepts with invariance to pose, e.g., leg positions in the 2nd, 3rd, and 4th columns.
A similar decomposition into legs, torso, back, and head can be seen for the elephants in Fig. 3(b). This shows that we can localize different objects and parts even when they are all common across the image set. Interestingly, the decompositions shown in Fig. 1 exhibit similar high semantic quality in spite of their dissimilarity to the ImageNet training data, as neither pyramids nor the Taj Mahal are included as class labels in that dataset. We also note that as some of the given sets contain as few as 5 images (Fig. 1(b) comprises the whole set), our method does not require many images to find meaningful structure.
Object and Part Co-segmentation. We operationalize DFF to perform co-segmentation. To do so we have to first annotate the factors as corresponding to specific ground-truth parts. This can be done manually (as in Table 3) or automatically given ground truth, as described below. We report the intersection-over-union (IoU) score of each factor with its associated parts in Table 1.
Since the heat maps are of low-resolution, we refine them with post processing. We define a dense conditional random field (CRF) over the heat maps. We use the filter-based mean field approximate inference [20], where we employ guided filtering [15] for the pairwise term, and use the biliniearly upsampled DFF heat maps as unary terms. We refer to DFF with post-processing ‘DFF-CRF.
Each heat map is converted to a binary mask using a thresholding procedure. For a specific DFF factor f (\(1\le f\le k\)), let \(\{H(f, 1),\cdots , H(f,n)\}\) be the set of n heat maps associated with n input images, The value of a pixel in the binary map B(f, i) of factor f and image i is 0 if its intensity is lower than the 75th percentile of entries in the set of heat maps \(\{H(f,j) | 1\le j\le n\}\).
We associate parts with factors by considering how well a part is covered by a factor’s binary masks. We define the coverage of part p by factor f as:
The coverage is the percentage of pixels belonging to p that are set to 1 in the binary maps\(\{B(f,i) | 1\le i\le n\}\). We associate the part p with factor f when \(Cov_{f,p}>Cov_{\text {th}}\). We experimentally set the threshold \(Cov_{\text {th}}=0.5\).
Finally, we measure the IoU between a DFF factor f and its m associated ground-truth parts \(\{p^{(f)}_1,\cdots ,p^{(f)}_m\}\) similarly to [2], specifically by considering the dataset-wide IoU :
In the top of Table 1 we report results for object co-segmentation (\(k=1\)) and show that our method is comparable with the supervised approach of [33] and domain-specific methods of [28, 29].
The bottom of Table 1 shows the labels and IoU-scores for part co-segmentation on the five image sets of iCoseg that we have annotated. These scores correspond to the visualizations of Figs. 1 and 3 and confirm what we observe qualitatively.
We can characterize the quality of a factorization as the average IoU of each factor with its single best matching part (which is not the background). In Fig. 4(a) we show the average IoU for different layer of VGG-19 on iCoseg as the value of k increases. The variance shown is due to repeated trials with different NMF initializations. There is a clear gap between convolutional blocks. Performance with in a block does not strictly follow the linear order of layers.
We also see that the optimal value for k is between 3 and 5. While this naturally varies for different networks, layers, and data batches, another deciding factor is the resolution of the part ground truth. As k increases, DFF heat maps become more localized, highlighting regions that are beyond the granularity of the ground truth annotation, e.g., a pair of factors that separates leg into ankle and thigh. In Fig. 4(b) we show that DFF performs similarly within the VGG family of models. For ResNet-101 however, the average IoU is distinctly lower.
4.4 Object Co-Localization on PASCAL VOC 2007
Dataset. PASCAL VOC 2007 has been commonly used to evaluate whole object co-localization methods. Images in this dataset often comprise several objects of multiple classes from various viewpoints, making it a challenging benchmark. As in previous work [5, 19, 21], we use the trainval set for evaluation and filter out images that only contain objects which are marked as difficult or truncated. The final set has 20 image sets (one per class), with 69 to 2008 images each.
Evaluation. The task of co-localization involves fitting a bounding box around the common object in a set of image. With \(k=1\), we expect DFF to retrieve a heat map which localizes that object.
As described in the previous section, after optionally filtering DFF heat maps using a CRF, we convert the heat maps to binary segmentation masks. We follow [31] and extract a single bounding box per heat map by fitting a box around the largest connected component in the binary map.
We report the standard CorLoc score [7] of our localization. The CorLoc score is defined as the percentage of predicted bounding boxes for which there exists a matching ground truth bounding box. Two bounding boxes are deemed matching if their IoU score exceeds 0.5.
The results of our method are shown in Table 2, along with previous methods (described in Sect. 4.2). Our method compares favorably against previous approaches. For instance, we improve co-localization for the class dog by 16% higher CorLoc and achieve better co-localization on average, in spite of our approach being simpler and more general.
4.5 Part Co-segmentation in PASCAL-Parts
Dataset. The PASCAL-Part dataset [4] is an extension of PASCAL VOC 2010 [10] which has been further annotated with part-level segmentation masks and bounding boxes. The dataset decomposes 16 object classes into fine grained parts, such as bird-beak and bird-tail etc. After filtering out images containing objects marked as difficult and truncated, the final set consists of 16 image sets with 104 to 675 images each.
Evaluation. In Table 3 we report results for the two classes, cow and horse, which are also part-segmented by Want et al. as described in Sect. 4.2. Since their method relies on strong explicit priors w.r.t to part size, hierarchy, and symmetry, and its explicit objective is to perform part-segmentation, their results serve as an upper bound to ours. Nonetheless we compare favorably to their results and even surpass them in one case, despite our method not using any hand-crafted features or supervised training.
For this experiment, our strategy for mapping DFF factors (\(k=3\)) to their appropriate part labels was with semi-automatic labeling, i.e., we qualitatively examined the heat maps of only five images, out of approximately 140 images, and labeled factors as corresponding to the labels shown in Table 3.
In Table 4 we give IoU results for five additional classes from PASCAL-Parts, which have been automatically mapped to parts as in Sect. 4.3. In Fig. 5 we visualize these DFF heat maps for \(k=3\), as well as for cow from Table 3. When comparing the heat maps against their corresponding IoU-scores, several interesting conclusions can be made. For instance, in the case of motorbike, the first and third factors for \(k=3\) in Table 4 both seems to correspond with wheel. The visualization in Fig. 5(e) reveals that these factors in fact sub-segment the wheel into top and bottom, which is beyond the resolution of the ground truth data.
We can see also that while the first factor of the class aeroplane (Fig. 5(a)) consistently localizes airplane wheels, it does not to achieve high IoU due to the coarseness of the heat map.
Returning to Table 4, when \(k=4\), a factor emerges that localizes instances of the class person, which occur in 60% of motorbike images. This again shows that while most co-localization methods only describe objects that are common across the image set, our DFF approach is able to find distinctions within the set of common objects.
5 Conclusions
In this paper, we have presented Deep Feature Factorization (DFF), a method that is able to locate semantic concepts in individual images and across image sets. We have shown that DFF can reveal interesting structures in CNN feature space, such as hierarchical clusters which correspond to a part-based decomposition at various levels of granularity.
We have also shown that DFF is useful for co-segmentation and co-localization, achieving results on challenging benchmarks which are on par with state-of-the-art methods, and can be used to perform semi-automatic image labeling. Unlike previous approaches, DFF can also perform part co-segmentation as well, making fine distinction within the common object, e.g. matching head to head and torso to torso.
References
Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: Icoseg: interactive co-segmentation with intelligent scribble guidance. In: Computer Vision and Pattern Recognition (CVPR), pp. 3169–3176. IEEE (2010)
Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A.: Network dissection: quantifying interpretability of deep visual representations. In: Computer Vision and Pattern Recognition (CVPR), pp. 3319–3327. IEEE (2017)
Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 35(8), 1798–1828 (2013)
Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., Yuille, A.: Detect what you can: detecting and representing objects using holistic models and body parts. In: Computer Vision and Pattern Recognition (CVPR), pp. 1971–1978 (2014)
Cho, M., Kwak, S., Schmid, C., Ponce, J.: Unsupervised object discovery and localization in the wild: part-based matching with bottom-up region proposals. In: Computer Vision and Pattern Recognition (CVPR) (2015)
Cichocki, A., Zdunek, R.: Multilayer nonnegative matrix factorisation. Electron. Lett. 42(16), 1 (2006)
Deselaers, T., Alexe, B., Ferrari, V.: Weakly supervised localization and learning with generic knowledge. Int. J. Comput. Vis. (IJCV) 100(3), 275–293 (2012)
Ding, C., He, X., Simon, H.D.: On the equivalence of nonnegative matrix factorization and spectral clustering. In: Proceedings of the 2005 SIAM International Conference on Data Mining, pp. 606–610. SIAM (2005)
Dziugaite, G.K., Roy, D.M.: Neural network matrix factorization. arXiv preprint arXiv:1511.06443 (2015)
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal-network.org/challenges/VOC/voc2010/workshop/index.html
Gonzalez-Garcia, A., Modolo, D., Ferrari, V.: Do semantic parts emerge in convolutional neural networks? Int. J. Comput. Vis. (IJCV) 126(5), 1–19 (2017). https://link.springer.com/article/10.1007/s11263-017-1048-0
Grais, E.M., Erdogan, H.: Single channel speech music separation using nonnegative matrix factorization and spectral masks. In: Digital Signal Processing (DSP), pp. 1–6. IEEE (2011)
Guillamet, D., Vitrià, J.: Non-negative matrix factorization for face recognition. In: Escrig, M.T., Toledo, F., Golobardes, E. (eds.) CCIA 2002. LNCS (LNAI), vol. 2504, pp. 336–344. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-36079-4_29
Hariharan, B., Arbeláez, P., Girshick, R., Malik, J.: Simultaneous detection and segmentation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 297–312. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_20
He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 35(6), 1397–1409 (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (ICML), pp. 448–456 (2015)
Jolliffe, I.T.: Principal component analysis and factor analysis. In: Principal Component Analysis, pp. 115–128. Springer, NewYork (1986). https://doi.org/10.1007/0-387-22440-8_7
Joulin, A., Tang, K., Fei-Fei, L.: Efficient image and video co-localization with frank-wolfe algorithm. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 253–268. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_17
Krähenbühl, P., Koltun, V.: Efficient inference in fully connected CRFS with gaussian edge potentials. In: Advances in Neural Information Processing Systems (NIPS), pp. 109–117 (2011)
Le, H., Yu, C.P., Zelinsky, G., Samaras, D.: Co-localization with category-consistent features and geodesic distance propagation. In: Computer Vision and Pattern Recognition (CVPR), pp. 1103–1112 (2017)
Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401(6755), 788 (1999)
Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Advances in Neural Information Processing Systems, pp. 556–562 (2001)
Li, Y., Liu, L., Shen, C., van den Hengel, A.: Image co-localization by mimicking a good detector’s confidence score distribution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 19–34. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_2
Montavon, G., Samek, W., Müller, K.: Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 73, 1–15 (2018) https://doi.org/10.1016/j.dsp.2017.10.011
Paszke, A., et al.: Automatic differentiation in pytorch (2017)
Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you?: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)
Rubinstein, M., Joulin, A., Kopf, J., Liu, C.: Unsupervised joint object discovery and segmentation in internet images. In: Computer Vision and Pattern Recognition (CVPR), June 2013
Rubio, J.C., Serrat, J., López, A., Paragios, N.: Unsupervised co-segmentation through region matching. In: Computer Vision and Pattern Recognition (CVPR), pp. 749–756. IEEE (2012)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization, vol. 37(8) (2016). See arxiv:1610.02391
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Vicente, S., Rother, C., Kolmogorov, V.: Object cosegmentation. In: Computer Vision and Pattern Recognition (CVPR), pp. 2217–2224. IEEE (2011)
Vu, T.T., Bigot, B., Chng, E.S.: Combining non-negative matrix factorization and deep neural networks for speech enhancement and automatic speech recognition. In: Acoustics, Speech and Signal Processing (ICASSP), pp. 499–503. IEEE (2016)
Wang, J., Yuille, A.L.: Semantic part segmentation using compositional model combining shape and appearance. In: CVPR (2015)
Xu, W., Liu, X., Gong, Y.: Document clustering based on non-negative matrix factorization. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 267–273. ACM (2003)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Collins, E., Achanta, R., Süsstrunk, S. (2018). Deep Feature Factorization for Concept Discovery. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds) Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science(), vol 11218. Springer, Cham. https://doi.org/10.1007/978-3-030-01264-9_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-01264-9_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01263-2
Online ISBN: 978-3-030-01264-9
eBook Packages: Computer ScienceComputer Science (R0)