Summary: Future generations of automated video surveillance systems must be scalable and evolvable, and they demand sophisticated middleware support. This middleware approach partitions systems according to an activity topology-a graph describing activity observed by the surveillance camera network. Processing within topological partitions uses well-known architectural styles such as blackboards, and pipes and filters. Communication between partitions uses a service-oriented architecture. This middleware enables building intelligent video surveillance systems at a far larger scale than was previously possible.