Summary: We present a novel scheme for object-based video analysis and interpretation based on automatic video object extraction, video object abstraction, and semantic event modeling. In our proposed scheme, video objects (VOs) are first segmented, followed by a video object abstraction algorithm for identifying key frames to reduce data redundancy and provide reliable feature data for next stage of the algorithm. Semantic feature modeling scheme is based on temporal variation of low-level features extracted from VOs. More specifically, the Dynamic Bayesian Network (DBN) is used to characterize the spatio-temporal nature of the semantic objects. Comparing to a hidden Markov model (HMM), the DBN framework can provide a detailed description of the characteristics of VOs, which can be readily interfaced to MPEG-7 application. Experimental results that demonstrate the effective performance of the proposed approach are also presented.