Since the prototypes of different modalities are extracted separately, there is no pairwise correspondence information between two different prototype subgraphs. Moreover, because of different natures of prototypes in different modalities, we cannot directly compare them. In such cases, a proper solution is to investigate the local geometry of prototypes and measure the similarity of two prototypes using a local matching algorithm. To this bibr 1532
end, a good algorithm was proposed in . This algorithm computes local patterns around samples
and matches them by comparing their local patterns. However, running time of this algorithm is extremely large, making it challenging to apply on large-scale annotation datasets. In our framework, we can easily capture local geometry of each prototype based on its connections to feature vectors
(sample nodes) in the same modality. Since there are complete pairwise correspondences between feature vectors of different modalities, the following approach is used to match two prototypes based on their connections to feature vectors.