Flit-Level InfiniBand Network Simulations of the DAQ System of the LHCb Experiment for Run-3

The Large Hadron Collider beauty (LHCb) experiment is designed to study the differences between particles and antiparticles as well as very rare decays in the charm and beauty sector at the (LHC). The detector will be upgraded in 2019, and a new trigger-less readout system will be implemented in order to significantly increase its efficiency and fully take advantage of the provided machine luminosity at the LHCb collision point. In the upgraded system, both event building and event filtering will be performed in software for all the data produced in every bunch-crossing of the LHC. In order to transport the full data rate of 32 Tb/s, we will use custom field-programmable gate array (FPGA) readout boards (PCIe40) and the state-of-the-art off-the-shelf network technologies. The full-event-building system will require around 500 servers interconnected together. From a networking point of view, event building traffic has an all-to-all pattern, requiring careful design of the network architecture to avoid congestion at the data rates foreseen. In order to maximize link utilization, different techniques can be adopted in various areas like traffic shaping, network topology, and routing optimization. The size of the system makes it very difficult to test at production scale, before the actual procurement. We resort, therefore, to network simulations as a powerful tool for finding the optimal configuration. We will present an accurate low-level description of an InfiniBand-based network with event building like traffic. We will show a comparison between simulated and reduced scale systems and how changes in the input parameters affect the performance.