A key challenge for vgES is to monitor the health and status of applications once they are instantiated across the VG. A fundamental building block of these systems is the ability to continually query application-defined data streams produced across nodes. For example, application controllers must coordinate data delivery and job execution, monitor for faults, and manage resource consumption across compute environments that are inherently failure prone. To this end we have developed Mortar, a lightweight distributed stream-processing engine that allows applications to manage, transform, and acquire data generated across the virtual grid.

Users specify “expectations” of system behavior, and these form the basis for higher-level software components. For example, a monitoring system may write an expectation for some threshold of failed processes or an event-driven application controller may write one to detect workflow stage completion. Mortar verifies expectations using in-network aggregating functions and scalable instance overlays, static trees of nodes that orchestrate the in-network computation. In particular, we have developed techniques to make these computations accurate and responsive even during node and link failures.  

We have compared Mortar to DHT-based systems that also support in-network aggregates (SDIMS).   In several failure scenarios, Mortar’s instance overlays can reduce error by 50%, with an acceptable increase in total network load. We continue to explore refinements that further improve accuracy while reducing network load.