This shot is based on 1 billion passages of taxi users in New York.
Two astronomers from the University of Groningen (The Netherlands) created a virtual library capable of generating visualizations based on hundreds of millions of data points. Initially, the software was created to process a huge amount of Gaia mission data. But it can also display templates in other large data files. It is open source and free to use.
The software was called Vaex - “visualization and study of large tabular data sets”. It can create a visualization of billions of data points in just one second and behaves like Google maps. But Google Maps uses fast and powerful servers for work, and Vaex costs a laptop. The power of the Vaex is based on several clever methods. It uses an intelligent algorithm that maximizes all available computing power. Then it counts only the required data from the hard disk and sends it to the main memory of the computer. In addition, the working memory does not store unnecessary copies of data.
The power of the Vaex was demonstrated live on several conferences. Used a data set of 1 billion records related to taxis in New York. It turned out that you can see which trips bring great profits and where taxis can be expected in any part of the day. This suggests that Vaex should be used outside of astronomy.
