Three Cyfronet supercomputers among the fastest and most energy-efficient computers in the world

For the second time in history, the TOP500 list of supercomputers with the highest computing power in the world simultaneously included three supercomputers operating in the same supercomputing centre in Poland, it the Academic Computer Centre CYFRONET at the AGH University. These are: Athena, Helios, and Ares. The Cyfronet Centre’s machines are also high on the list of the most energy-efficient supercomputers, Green500.

The supercomputer provides Polish scientific community and economy with state-of-the-art computing resources based on CPUs and on the latest generation of GPGPU accelerators, along with the necessary data storage subsystem based on ultra-fast flash memories. Athena's configuration includes: servers with AMD EPYC processors (a total of 6,144 CPU computing cores) and 384 GPGPU NVIDIA A100 cards. An essential element for utilizing such high computing power efficiently is the provision of a high-performance internal network for the supercomputer (Infiniband HDR with 4 × 200 Gbps per server) and a very fast disk subsystem.  It has been created with the use of the open source Luster software and dedicated disk servers equipped with NVMe flash memory.

Athena's architecture meets the needs of Cyfronet's supercomputer users, who take advantage of the computing infrastructure both to perform standard high-performance scientific simulations and to apply artificial intelligence (AI) and machine learning (ML) methods in research in fields such as medicine, pharmacology, biology, chemistry, and physics. Athena’s computing power for AI calculations amounts to almost 240 PetaFlops!

Helios – new light for innovation in Polish science and economy

The Helios supercomputer is a new system in the Cyfronet Centre, resulting from work carried out within the National Supercomputing Infrastructure project for EuroHPC, EuroHPC PL, coordinated by Cyfronet. The supercomputer was built according to the Cyfronet's design by Hewlett-Packard Enterprise based on the HPE Cray EX4000 platform and will eventually consist of three computing partitions:

1. CPU equipped with 75,264 AMD Zen4 computing cores and 200 TB of DDR5 working memory,
2. GPU equipped with 440 NVIDIA Grace Hopper GH200 superchips,
3. INT for interactive work, equipped with 24 NVIDIA H100 accelerators and a fast local memory NVMe.

Due to the current installation of the system, only a partition of general-purpose processors (CPUs) with a computing power of 3.4 PetaFlops was submitted for testing for the TOP500 list. Despite this fact, Helios has been ranked 291st on the TOP500 list. After a full system installation, when all partitions are running, Helios is expected to reach about 35 PFlops of theoretical computing power with all partitions launched. On the current list, such power would allow Helios to rank among the world's 50 fastest supercomputers.

Helios' disk subsystem consists of two types of Luster file systems: scratch with 1.5 PB and speed of more than 1.8 TB/s, and project with 16 PB and speed of nearly 200 GB/s. All components of the supercomputer are interconnected by a 200 GB/s Slingshot network. The HPE Cray EX4000 platform, based on which Helios is built, is also used in the fastest supercomputers in the world (Frontier) and in Europe (LUMI). With direct liquid cooling of the CPU and GPU partitions, it is possible for the system to achieve a very low PUE (Power Usage Effectiveness), resulting in increased energy efficiency and lower operating costs. What is more, by recovering the waste heat produced by Helios, it will be possible to use it for heating.

As early as in the first half of 2024, after the completion of the installation process and thorough testing phase of the infrastructure, Helios will be available to both scientific units as well as economic and public administration entities. The expected result is the development of innovative research and acceleration of the introduction of the said research-based technologies on the market.

Ares – a warrior for science

The Ares supercomputer offers a total computing power of more than 4 PetaFlops (the theoretical performance of the CPU part is more than 3.5 PetaFlops, and the GPU part is more than 0.5 PetaFlops). Ares consists of computing servers with Intel processors (Xeon Platinum model), divided into three groups:

• 532 servers, each equipped with 192 GB of RAM,
• 256 servers, each equipped with 384 GB of RAM,
• 9 servers, each with 8 NVIDIA Tesla V100 cards.

This supercomputer has 37.824 computing cores. Ares is also supported by a disc system with 11 PB capacity. An InfiniBand EDR type of network is used for transferring data. On the TOP500 list published in November, Ares is on the 404th position.

World’s best in energy efficiency

Along with the announcement of the TOP500 list of the most powerful supercomputers, another list is published, namely the Green500 list of supercomputers with the highest energy efficiency, calculated as the ratio of the number of floating point operations per second (supercomputer’s computing power) to energy consumption. Athena's 22nd place on the Green500 list of the world's most energy-efficient supercomputers in the current ranking is a major success. This position testifies to the excellent ratio of the available computing power to electricity consumption. The Green500 list also includes Ares, which is highly  ranked at 114th place. In turn, Helios – CPU is 344th on the Green500 list.

Scientific computing in various fields

Scientists conducting research with the use of Cyfronet’s infrastructure represent multiple fields. Advanced modelling and numerical computations are used mainly in: chemistry, biology, physics, medicine, and materials engineering, but also in astronomy, geology, and environmental protection. Supercomputers available in Cyfronet as part of the PLGrid infrastructure also have applications in high energy physics (projects such as ATLAS, LHCb, ALICE, and CMS), astrophysics (CTA, LOFAR), Earth sciences (EPOS), European Spallation Source (ESS), biology (WeNMR), as well as humanities and social studies (CLARIN).

Poland compared to Europe and world

The fastest supercomputer in the world is once again Frontier in the Oak Ridge National Laboratory (ORNL), USA. It is the first supercomputer to have surpassed the barrier of 1 ExaFlops (ExaFlops – 1018 floating point operations per second), and its theoretical computing power is 1,68 ExaFlops. In the world’s TOP10 are now six machines from the USA, three from Europe, and one from Japan.

The fastest European supercomputer is once again LUMI, built as a result of collaboration between 10 European countries, including Poland. On November Top500 list, the LUMI supercomputer is on the 5th position. The current theoretical computing power of LUMI amounts 531 PetaFlops. Polish scientists can obtain access to the LUMI supercomputer by means of the PLGrid infrastructure. Actions in this regard are coordinated by AGH University Academic Computer Centre Cyfronet.

Polish supercomputers on TOP500 list

The newest TOP500 list includes four supercomputers from Poland. The complete list with positions along with the values of the theoretical computing power is as follows:

• 155 – Academic Computer Centre CYFRONET at the AGH University, Athena (7710 TeraFlops),
• 221 – Poznan Supercomputing and Networking Center, Altair (5880 TeraFlops),
• 291 – Academic Computer Centre CYFRONET at the AGH University, Helios CPU (3400 TeraFlops),
• 404 – Academic Computer Centre CYFRONET at the AGH University, Athena (3510 TeraFlops),

Use the power of supercomputers in your research

Supercomputers make it possible to significantly reduce the time required to perform computing to a few hours or weeks that would often take plenty of years (in specific real-world cases, more than 150, 700, or even 1,000 years) with single computers. Cyfronet users may benefit from professional support, from trainings to individual consultations with experts.

Information on Cyfronet’s resources and undertaken measures are available at the website of AGH University Cyfronet, whereas the list of services within the PLGrid infrastructure may be found in PLGrid guides.