Bi-directional Contextual Attention
for 3D Dense Captioning

Minjung Kim1,4,*, Hyung Suk Lim3, Soonyoung Lee2, Bumsoo Kim2, Gunhee Kim1,
1Seoul National University, 2LG AI Research, 3Diquest, 4SNU-LG AI Research Center
ECCV 2024
*Work done during an internship at LG AI Research
Paper arXiv Video

Bi-directional Contextual Attention (BiCA) generates object-aware contexts, where the contexts relevant to each object is summarized, and context-aware objects, where the objects relevant to the summarized object-aware contexts are aggregated.

Abstract

3D dense captioning is a task involving the localization of objects and the generation of descriptions for each object in a 3D scene. Recent approaches have attempted to incorporate contextual information by modeling relationships with object pairs or aggregating the nearest neighbor features of an object. However, the contextual information constructed in these scenarios is limited in two aspects: first, objects have multiple positional relationships that exist across the entire global scene, not only near the object itself. Second, it faces with contradicting objectives--where localization and attribute descriptions are generated better with tight localization, while descriptions involving global positional relations are generated better with contextualized features of the global scene.

To overcome this challenge, we introduce BiCA, a transformer encoder-decoder pipeline that engages in 3D dense captioning for each object with Bi-directional Contextual Attention. Leveraging parallelly decoded instance queries for objects and context queries for non-object contexts, BiCA generates object-aware contexts, where the contexts relevant to each object is summarized, and context-aware objects, where the objects relevant to the summarized object-aware contexts are aggregated. This extension relieves previous methods from the contradicting objectives, enhancing both localization performance and enabling the aggregation of contextual features throughout the global scene; thus improving caption generation performance simultaneously. Extensive experiments on two of the most widely-used 3D dense captioning datasets demonstrate that our proposed method achieves a significant improvement over prior methods.

Model

Model Architecture

The overall pipeline of BiCA. We parallelly generate and decode two sets of queries (i.e., Instance Query and Context Query) that encodes the instance features and the non-object context features throughout the global scene, respectively. The object-aware contexts are calculated per each object by the weighted sum of the context queries, where the weights are calculated by the attention between the decoded instance query and context query. Then, with the object-aware context feature, the context-aware object feature is obtained by the weighted sum of the instances, which is weighted by the attention between the object-aware contexts.

Experimental Results

Table 1: Experimental results on the ScanRefer. C, B-4, M, and R represent the captioning metrics CIDEr, BLEU-4, METEOR ,and ROUGE-L, respectively. A higher score for each indicates better performance.
Table 2: Experimental results on the Nr3D with IoU threshold at 0.5. C, B-4, M, and R represent the captioning metrics CIDEr, BLEU-4, METEOR ,and ROUGE-L, respectively. A higher score for each indicates better performance.
Table 3: Ablation study on the ScanRefer.

Visualization

Visualization of (a) the input 3D scene, (b) input point cloud without color, (c) visualization of instance queries, (d) one sampled instance query, (e) contextual attention of (d), and (f) contextual attention of (e) on the ScanRefer

BibTeX

If you find our work useful in your research, please consider citing: 

          @@InProceedings{,
            author    = {Kim, Minjung and Lim, Hyung Suk and Lee, Soonyoung and Kim, Bumsoo and Kim, Gunhee},
            title     = {Bi-directional Contextual Attention for 3D Dense Captioning},
            journal   = {ECCV},
            year      = {2024},
}