GAMS supports the teaching and application of optimization within academic institutions.
We provide complimentary access to GAMS through our community and course licenses in order
to offer professors and students the best options to teach and learn about optimization.
Request a complimentary, time-limited, and fully featured GAMS license that enables
you to evaluate and explore the different solvers from our portfolio
before making a purchasing decision.
Stadtwerke München Transitioned Their Large GAMS Based Energy Market Model to the Amazon Cloud
Background
Stadtwerke Munich (SWM) is one of the biggest municipal electricity providers in Germany, with an annual revenue for electricity of 2.8 billion EUR in 2019, corresponding to 37 TWh1. For comparison, the total net electricity produced in Germany each year is around 600 TWh2.
SWM is strongly focused on their own renewable power generators (geothermal, hydropower, on- and offshore wind parks, photovoltaic, and biogas) and operate them both within Germany and in other European countries (Belgium, Croatia, Finland, France, Norway, Poland, and Sweden). The renewable energy contributes to a total of more than 3000 individual generators connected to the grid.
Fig 1. Overview of SWM participation in the European grid. Source: https://www.swm.de/english/company
Because of their big investments in renewables SWM is highly interested in the long-term development of the European energy prices and the resulting economy of the assets.
Consequently, SWM has developed and implemented a fundamental model in GAMS for the European energy market, focused on prices of electrical power, carbon and the value of renewables. The model calculates the most effective way to meet the demand of electrical power at all given points in time by taking all existing generators and all generators build by the model into account.
The long-term prices given by the fundamental model are being used for all investment decisions in generators (solar, onshore wind, offshore wind, gas fired CHP power plants and the geothermal sector). Furthermore, the long-term prices are basis for impairments and finally the corporate planning.
Current Situation
SWM has been operating their 2-part GAMS fundamental model for some years.
The first part of the model (the invest part) aims to calculate how the general electricity landscape will look in Europe up to the year 2050. This model is large, with approx. 70 million non-zero matrix elements.
The model inputs are 52 Excel-files with up to 60.000 data per sheet, e.g.:
all existing European power plants,
power demand and net transfer capacities,
typical days for generation from renewable energy sources,
fuel prices.
The major outputs are
the available generators of each technology (renewable, conventional)
the capacity and generation mix and
the baseline market price for electricity for each year.
Using the output from the invest model with yearly data, the second part (the dispatch model) can run in 20-30 parallel jobs for the different sets of assumptions. One job corresponds to one year in hourly resolution. SWM have been running this model on in-house hardware (192 cores, 1.5TB of RAM).
With the recent changes in the legal framework within the EU, the guaranteed price for renewable energy has been abolished, and renewable energy now underlies the same market fluctuations and risks as the other forms of energy. This change adds several degrees of freedom to the scenarios to be solved and answers to questions like: “What happens to the value of renewable energy if the gas price suddenly drops?” are needed. Beside of varying fuel prices a couple of legal regulations and challenges coming from national and European decarbonization targets must be considered.
Because of the fact that the resulting market prices are scenario-based under different assumptions and not a prognosis with a specific probability of occurrence, a lot of scenarios have to be calculated in order to figure out what the leverage on prices is if premises change.
Complete simulation runs must be performed about 200 times per year and the results of each run should be available within 24 hours resulting in extremely high demand in computational power during those times. In between those peak periods, the demand is low. In practice this means that the required compute capacity cannot economically be met with in-house hardware any longer, and a solution using cloud computing appeared to be the best fit.
Technical Challenges
The following technical challenges had to be addressed while developing the cloud solution:
Invest model
The invest model is memory bound and requires approximately 700 GB RAM for one scenario. It must be solved in one closed run without the possibility of partitioning and parallelization. Two cores are sufficient.
Dispatch model
Running the dispatch part of the model requires automatic orchestration of 20-30 independent GAMS workers, feeding them with input data and collecting the results of each worker after the run completes. Here, the GAMS workers can be executed in parallel because one year is one run and the years can be calculated independently from each other. Each worker for the dispatch model requires approximately 30 GB RAM and two cores.
The Solution
In 2020, the SWM have started a strategic cloud program. One stream of this program is the evaluation of cloud-based hyperscaling and for the proof-of-concept Amazon Web Services (AWS) have been chosen. The designed solution is a composition of different cloud services and state of the art infrastructure technology: The GAMS application is supplied as a docker image and the workflow to solve the GAMS model is orchestrated using serverless lambda functions (python).
At the heart of the solution is AWS Batch, which allows running hundreds or thousands of batch jobs and dynamically provisions the needed compute resources per job, based on the resource requirements of the submitted batch jobs. The result is a simple solution that is extensible and scalable and requires next to none hosting cost, while in idle. A strategic decision was to implement every cloud resource using Infastructure as Code (IaC) – for this SWM use the open source product terraform and therefore benefit from the typical software development lifecycle (git, pull request reviews, versioning).
Fig 2. Schematic overview of the developed cloud solution
To use the application, the user uploads the GAMS fundamental model and parameter files to an S3 bucket, which automatically triggers a step function to start the flow. The first step submits an AWS Batch job containing the invest model, AWS Batch manages the required virtual machines and starts the docker image with the GAMS application. After the first calculation, the dispatch models are automatically submitted and run concurrently by AWS Batch. As the dispatch models require less CPU and memory, AWS Batch uses the already existing machines to run the models most of the time. If needed, more computation instances are created. Finally, when all jobs are finished, the resources are terminated automatically.
Currently, on-demand instances are used, which already results in a vast cost reduction over the existing solution. Further cost-optimization by using spot instances is planned in the future: Spot instances can be much cheaper than equivalent on-demand instances, however there is no guarantee that they run continuously, so this will require the implementation of a retry mechanism, if a premature termination occurs.
Cost
AWS Batch dynamically combines multiple smaller instances on a single big machine and allocates intelligently the most appropriate EC2 instances to the GAMS worker.
Example for costs:
Invest model: requires approx. 700 GB RAM, but the number of cores is not relevant
best fit EC2 instance (768 RAM, 96 cores) < 10 USD/h
Dispatch model: each job requires ca. 30 GB RAM, the number of cores is not relevant
best fit EC2 instance (32 RAM, 8 cores) < 0,5 USD/h
(multiplied by 20 years results into the same instance as the invest model concerning RAM and costs, which is automatically detected by AWS batch.)
Total: ~ 50-70 USD per complete run, depending on complexity; up to 200 runs p.a.
The software license is a GAMS application license that allows an unlimited number of runs of this particular application for a flat yearly fee.
Benefits of the new solution
SWM can better react to changes in electricity market conditions. This is particularly true for flexibility due to the EU Green Deal, changes in carbon taxation and fuel prices. Long term fundamental analysis does not mean to provide a price prognosis, but it shows the effects of changes in the premises on long term prices. It is the only way to understand and to quantify market risks that can occur. Combining all changes on the input side one obtains a lot of scenarios to be calculated. Using a hyper scale approach, the results are now available within one day, while using a physical server on premise it would take weeks. On top of the saved time, the cloud-based solution cut costs by up to 40 percent compared to on-premise solution.
The improved analytical skills in the evaluation of long-term energy prices gained by the ability to run many scenarios of the fundamental model help minimize cost in the acquisition process and make risks of the energy market manageable.
About SWM
Stadtwerke München (SWM),
Munich’s municipal utilities company, is one of the largest energy and infrastructure companies in Germany. SWM supplies Munich 24/7 with energy (electricity, natural gas, district heating/cooling), fresh drinking water, mobility and advanced, cutting-edge telecommunication services. Sustainability and climate protection are essential cornerstones of their corporate policy in all sectors.
