Performance Evaluation Suite for Semantic Publish-Subscribe Message-oriented Middlewares

The emerging Internet of Things paradigm is driving the industry and the research towards Information and Communication Technologies (ICT) scenarios supporting high heterogeneity and interoperability. We claim that software architectures based on Semantic Publish-Subscribe Message-Oriented Middlewares (SPS-MoMs) are a powerful approach to address the requirements of such scenarios. While benchmarks and frameworks are available to evaluate the performance of MoMs, Semantic Web tools (i.e., SPARQL endpoints and RDF stores) and publish-subscribe systems, there are still no de-facto standards for the evaluation of SPS-MoMs, due to the novelty of this approach. In this paper, we propose Performance Evaluation Suite (PES), a benchmarking framework aimed at retrieving relevant performance indicators about a generic SPS-MoM. The feasibility of the proposed approach is proved by using PES to compare different implementations of a Semantic Information Broker (SIB), the core component of a SPS-MoM named Smart-M3.