Abstract:

According to the characteristics of human cognitive processes, a multimodal fusion method of semantic concept selection and importance measure is proposed. Semantic concepts are first obtained in single mode, and a correlation detection method is used to decide semantic concept of fusion. Correlation detection can reduce the influence of diffusion error due to false detection semantic. Since the method of probability fusion cannot effectively handle the temporal characteristics of the semantic, an importance measure is introduced for high level semantic fusion. Experimental results show that the proposed method using temporal characteristics and relativity between concepts can better extract high-level semantic contents as compared to other fusion methods.

Key words: semantic information, multimodal fusion, importance measure, relativity, concept selection