Augmented reality in the control tower: a rendering pipeline for multiple head-tracked head-up displays

The purpose of the air traffic management system is to accomplish the safe and efficient flow of air traffic. However, the primary goals of safety and efficiency are to some extent conflicting. In fact, to deliver a greater level of safety, separation between aircrafts would have to be greater than it currently is, but this would negatively impact the efficiency. In an attempt to avoid the trade-off between these goals, the long-range vision for the Single European Sky includes objectives for operating as safely and efficiently in Visual Meteorological Conditions as in Instrument Meteorological Conditions. In this respect, a wide set of virtual/augmented reality tools has been developed and effectively used in both civil and military aviation for piloting and training purposes (e.g., Head-Up Displays, Enhanced Vision Systems, Synthetic Vision Systems, Combined Vision Systems, etc.). These concepts could be transferred to air traffic control with a relatively low effort and substantial benefits for controllers’ situation awareness. Therefore, this study focuses on the see-through, head-tracked, head-up display that may help controllers dealing with zero/low visibility conditions and increased traffic density at the airport. However, there are several open issues associated with the use of this technology. One is the difficulty of obtaining a constant overlap between the scene-linked symbols and the background view based on the user’s viewpoint, which is known as ‘registration’. Another one is the presence of multiple, arbitrary oriented Head-Up Displays (HUDs) in the control tower, which further complicates the generation of the Augmented Reality (AR) content. In this paper, we propose a modified rendering pipeline for a HUD system that can be made out of several, arbitrary oriented, head-tracked, AR displays. Our algorithm is capable of generating a constant and coherent overplay between the AR layer and the outside view from the control tower. However a 3D model of the airport and the airport’s surroundings is needed, which must be populated with all the necessary AR overlays (both static and dynamic). We plan to use this concept as a basis for further research in the field of see-through HUDs for the control tower.