We would like to thank Dr. Evangelos Panos from the Paul Scherrer Institute, Switzerland for providing the example GAMS model described below, and for his contributions to this article.
Background
With GAMS 37, we now support the generation of model instances with more than 2³¹ non-zeros. Only a handful of our customers use models of this size, but one of the consequences of Big Data is Big Models. We therefore expect massive models to occur more often in the future, and perhaps even become commonplace in some application areas. If you are brave enough, you can test the old limit for yourself by running the following simple GAMS code on a machine with a lot of RAM (we suggest around 200GB):
set i / 1*46341 /; alias (i,j);
variable x(j), z; equation e(i), obj;
obj.. z =e= 0;
e(i).. sum(j, x(j)) =e= 0;
model m /all/;
m.limrow=0; m.limcol=0; option solvelink=0;
solve m min z us lp;
With GAMS < 37, this will cause a segmentation fault in the solver link after 15-20 minutes, depending on the speed of the machine. The reason for this is that much of the GAMS codebase was developed at a time when 32-bit was the norm, so counters, offsets, and indices were typically signed 32-bit values. With the shift from a 32-bit to a 64-bit paradigm (see also this blog article ), these limits are relaxed and we can work with much larger models. The number of non-zeros in the constraints is the natural place to start.
The Update
Changing the counter variables to 64-bit integers had quite a few ripple effects throughout the code base. Fortunately, the internal data structures used by GAMS did not need to be adjusted, so the memory footprint of the GAMS process is essentially unchanged as a result of this update. Our team has carefully worked through all of the potential consequences and pitfalls - you can probably imagine that running software tests that generate TB-sized models requires some patience and is not much fun :)
The effort has paid off, and GAMS 37 now offers a maximum number of non-zeros in generated model instances of 2^63 bit, which should be sufficient in the future. The solver links GAMS/Cplex, GAMS/Gurobi, GAMS/Xpress, and GAMS/ODHCplex have already been updated to support the new limit, and GAMS/Mosek will follow soon. Currently LPs and MIPs (without SOS constraints) are supported.
A Real World Example from Energy Systems Research
Energy systems analysis research has evolved from simple accounting tools in the 1970s to sophisticated energy systems models in the 2000s to respond to the increased complexity of energy and climate change mitigation policies. It is well embedded in energy strategy deliberations and has helped inform the climate and energy strategy dialogue in the past decade. However, the energy systems increasingly become more complex due to megatrends (e.g. globalisation and digitalisation), decentralisation and sector coupling, combined with ambitious climate change targets and increasing reliance on weather-driven energy supply.
Why is it important to the energy systems community to be able to solve large models
Policymakers and stakeholders are in high need of support in making informed decisions about investments and policy instruments that would lead to a future energy system that secures the supply and provides reliable and affordable services with a low environmental footprint. This necessitates modelling the future energy system at fine spatial and temporal scales; and with increased sectoral and technical detail.
For example, higher spatial and temporal details allow modelling results to reflect local conditions and constraints when calculating energy demand flexibility, storage, interconnection or other flexibility needed to accommodate increasing renewable penetration. The higher spatial resolution also allows assessing best-fit local decarbonisation solutions and strategies at a sub-national level, such as states, regions and municipalities. Further, the emergence of local energy markets requires increased technical and sectoral detail in energy systems analysis.
To accommodate these challenges in providing informed decision making support to policymakers and stakeholders, and supported by the availability of Big Data, there has been an exponential increase in the last years in the size of the matrices of the energy systems models and the number of non-zero coefficients in them. With the increasing importance of higher spatial, temporal, sectoral and technical resolution in energy systems analysis, it is expected that the exponential growth of model matrix sizes and the number of non-zeros in them will continue in the coming years.
EUSTEM model of the Paul Scherrer Institute
In response to the need for higher detail in energy systems analysis, the large-scale EUSTEM model was developed at the Paul Scherrer Institute (PSI) in Switzerland, based on the TIMES modelling framework of the International Energy Agency Energy Technology Systems Analysis Program (IEA-ETSAP) [1]. The model represents the energy systems of the European countries and combines a long-term horizon (2050+) for investment decisions, a high intrannual resolution to capture short-term operating constraints, and a high spatial resolution for local resource constraints.
In its first version of 2016, the model included 11 European regions with 288 timeslices in a year (typical operating hours) and 8 time periods from 2010 to 2050.
In 2021, five years later than its development, the model has been expanded to represent up to 30 European countries and to include up to 8760 timeslices, to assess questions from policymakers and stakeholders related to future storage needs, electricity load profiles (accounting for electrification of transport), deployment of renewable energy sources and their local constraints, digitalization, demand-side flexibility, and more.
The development of EUSTEM is one example of the increasing need in the energy systems community to perform the analysis at high spatial and temporal resolution. However, until GAMS version 37, it was impossible to solve EUSTEM with more than 2016 timeslices per year and 11 regions. For example, trying to solve the model with 4032 timeslices and 11 regions, GAMS aborted with an error caused by the integer overflow due to the large number of non-zeros in the model matrix:
Trade-offs and alternative approaches in solving EUSTEM
Hence, there was a trade-off between the resolution in spatial and temporal analysis with EUSTEM: either a few regions with a high number of timeslices or many regions with fewer timeslices. The alternative was to couple EUSTEM with other simulation tools, operating at high spatial or temporal details, to provide the insights required by the analysis and for targeted years only.
Such model coupling approaches are commonly used in the energy systems modelling community to cope with the challenge of not being able to solve very large models. However, different model coupling techniques can lead to different solutions. The way the model coupling is designed and performed is critical to avoid losing consistency in the analysis. As a result, the model coupling is associated with an increased effort to ensure computational efficiency and conceptual robustness.
With GAMS 37, it is possible for the first time to solve EUSTEM with 4032 timeslices, get new insights because of the higher temporal resolution, and address questions which so far were only partially tackled with the model. Below are some statistics from the run:
Model Generation
- ~4.5 hours
- 171,349,732 rows, 116,118,348 columns, 2,680,768,359 non-zeros
- ~238 GB of memory used
Solve
- Cplex (Barrier, 16 threads, Crossover disabled)
- ~ 65 hours / 316 iterations
Hardware
- 4 Sockets
- 88 physical cores (but only 16 threads were used)
- 2 TB of Memory
Peak Memory consumption: 821.70 GB
Advancing energy systems modelling
The ability to handle a huge number of non-zeros in the model matrix lifts a major obstacle in model-based energy systems analysis. It enables new designs and approaches, which also use Big Data, without trade-offs in model size.
The latest development from GAMS makes it possible to advance energy systems modelling and ensure that it remains a state-of-the-art tool in informing national energy policy.
References
[1] Pattupara, R. (2016). Long Term Evolution of the Swiss Electricity System under a European Electricity Market, Ph.D. Thesis, ETH Zurich, Nr. 23234. DOI:10.3929/ethz-a-010635090