Table of Contents
- Note
- This feature is currently in beta status.
Introduction
GAMS Jupyter Notebooks allow to use notebook technology in combination with GAMS. If you just want to learn GAMS there are probably better ways doing this. Notebooks allow you to combine GAMS and Python. The former works great with well structured data and optimization models, while the latter is very rich in features to retrieve, manipulate, and visualize data that comes in all sort of ways. Combining GAMS and Python in a notebook it is relatively easy to tell an optimization story with text, data, graphs, math, and models.
Getting Started
The first step in getting started with GAMS Jupyter Notebooks is to make your Python 3 installation aware of the GAMS Python API described in the Getting Started section of the API tutorial. We recommend to follow the steps below which are specifically tailored for getting started with GAMS Jupyter notebooks. While any Python 3.9 to 3.13 installation is supported, we recommend the use of miniconda
Python distributions.
- Attention
- All core third party dependencies will be installed if the user supplies the optional
pip
syntax (pip install gamsapi[magic]
).
In addition to the GAMS Python API collection (and the core magic
dependencies), the examples located in [PATH TO GAMS]/api/python/examples/magic
will require the additional packages: jupyterlab
, matplotlib
, and tabulate
. The following code section shows how to create and set up a conda
environment for GAMS Jupyter notebooks:
The notebooks Millco.ipynb and Introduction.ipynb located in api/python/examples/magic
are good starting points to get familiar with Jupyter notebooks and GAMS. The remainder of this section gives the dialog of the Introduction.ipynb
notebook. Please see also the other notebook examples:
- PickStock Example
- Filling grids with (distinct) polyominos
- DICE Model from 2018 Nobel laureate William D. Nordhaus
- PickStock Analysis using R
Tutorial
GAMS Jupyter Notebooks¶
GAMS Jupyter Notebooks allow to use notebook technology in combination with GAMS. If you just want to learn GAMS there are probably better ways doing this. Notebooks allow you to combine GAMS and Python. The former works great with well structured data and optimization models, while the latter is very rich in features to retrieve, manipulate, and visualize data that comes in all sort of ways. Combining GAMS and Python in a notebook it is relatively easy to tell an optimization story with text, data, graphs, math, and models.
The GAMS Jupyter Notebook builds on top of the Python 3 kernel. So by default the notebook cells are Python cells. Cells can be turned into GAMS cells, i.e. cells with GAMS syntax, using the Jupyter magic facility (first line in a cell is %%gams
). GAMS magic commands enable GAMS support in Python Jupyter notebooks. Beside running GAMS code, it is possible to transfer data between GAMS and Python. In order to enable the GAMS magic commands, it is required to (re)load the extension gams.magic
:
%reload_ext gams.magic
Running GAMS code¶
Running GAMS code can be done by using either %gams
(line magic) or %%gams
(cell magic). While %gams
can be used for running a single line of GAMS code, %%gams
makes the whole cell a GAMS cell. While %gams
can appear on any line, %%gams
is only valid in the first line of a cell.
%gams set i(*);
There are a few other useful commands in connection with running GAMS in a Jupyter notebook. Some transformation functions for pandas data frames (e.g. from2dim
(see below)) are available through the gams
instance that becomes available right after the %reload_ext gams.magic
.
%%gams
set j(*);
parameter p(i,j);
parameter p2(i,j);
The GAMS compiler and execution system has been adjusted so one can run a GAMS cell multiple time, even if it contains a declaration or an equation definition, which is normally not possible in the GAMS system. The execution of the next two cells does not create a problem, which mimics the execution, modification, and reexecution of a cell.
%%gams
set i / peter,paul,mary /, j / A,B,C /;
parameter p2(i,j) / set.i.set.j 1 /;
%%gams
set i / i1*i5 /, j /j1*j5 /;
parameter p2(i,j) / set.i.set.j 1 /;
You won't see any output from a GAMS cell (unless there is a solve executed in the cell, see below). All output goes to the log and lst file. If you really need to see this you can use magic command %gams_log
and %gams_lst
to display the content of the log and listing file of the most recent GAMS execution. The next cell displays the content of listing file of the last run GAMS cell or line magic. The -e
only display the section of the listing file associated with the execution:
%gams display p2;
%gams_lst -e
E x e c u t i o n ---- 35 PARAMETER p2 j1 j2 j3 j4 j5 i1 1.000 1.000 1.000 1.000 1.000 i2 1.000 1.000 1.000 1.000 1.000 i3 1.000 1.000 1.000 1.000 1.000 i4 1.000 1.000 1.000 1.000 1.000 i5 1.000 1.000 1.000 1.000 1.000
There is a limit to the execution, modification, and reexecution of GAMS cells. If the type or the dimensionality of a symbol changes, you will need to execute the notebook from scratch and do a controlled reset of the entire GAMS database via %gams_reset
. For example, since we declared parameter p2
already over (i,j)
we cannot change our mind and redeclare p2
as parameter p2(i,i,j)
:
%gams parameter p2(i,i,j);
This will give you a compilation error and an exception in the cell execution (uncomment the line in the next cell to do so):
#%gams parameter p2(i,i,j);
%gams_reset
With a %gams_reset
we can reset the GAMS database and can declare symbols with a different type and domain/dimension. All other things in the GAMS database are gone, too. So we need to redeclare the sets i and j, too. Furthermore %gams_reset
provides an optional argument --system_directory=<path/to/gams>
that allows to explicitly specify a GAMS system directory if the automatic detection of such is not wanted. Once a system directory has been specified, all %gams_reset
and gams.reset()
calls will take that one into account. Existing environments other than the active one are not affected automatically. Therefore a %gams_reset
directly after %reload_ext gams.magic
is helpful for specifying a non-default system directory for all subsequent environments. It is possible to switch back to the default behavior by providing the empty string as system directory (%gams_reset --system_directory=
). The state of the GAMS database is kept in various files that can easily clutter your directory. The %gams_cleanup
call helps you to clean the directory of temporary files of your current environment. The option -k
keeps the most recent GAMS database, hence the %gams_cleanup -k
is a save call anywhere in your notebook. The option -a
removes also files from other environments in your notebook. This option can be combined with -k
. The option -c
or --closedown
is to totally cleanup and leaves you with a fresh base
environment only.
Exchanging data between Python and GAMS¶
The property gams.exchange_container
references a GAMS Transfer container that organizes the data exchange between Python and GAMS. Symbols created in gams.exchange_container
will be available in GAMS as well as symbols declared in GAMS will be available in gams.exchange_container
:
import numpy as np
m = gams.exchange_container
i = m.addSet('i', records=["i1","i2","i3"])
j = m.addSet('j', records=["j1","j2"])
p = m.addParameter('p', [i,j], records=np.array([[1.1,2.2],
[3.3,4.4],
[5.5,6.6]]))
%gams display i, j, p;
%gams_lst -e
E x e c u t i o n ---- 13 SET i i1, i2, i3 ---- 13 SET j j1, j2 ---- 13 PARAMETER p j1 j2 i1 1.100 2.200 i2 3.300 4.400 i3 5.500 6.600
The next cell declares a symbol pp
in GAMS and multiplies the value of the parallel p
by 2. The second line is again Python code to display the data frame of pp
using the symbol method <symbol>.pivot()
.
%gams parameter pp(i,j); pp(i,j) = 2*p(i,j)
m['pp'].pivot()
j1 | j2 | |
---|---|---|
i1 | 2.2 | 4.4 |
i2 | 6.6 | 8.8 |
i3 | 11.0 | 13.2 |
The data frames GAMS Transfer uses to store data have a very specific layout. The setRecords
method or records=
argument on a symbol constructor allow to provide data from many different formats and Python data structures. See the GAMS Transfer documentation for details.
The GAMS line or cell magic will only synchronize symbols from gams.exchange_container
to GAMS that have been written to since the last GAMS line or cell magic execution. GAMS magic uses the Transfer attribute modified
in order to decide if the symbol needs to be synchronized with GAMS. In exceptional situations the setting of gams.write_all = True
bypasses this logic and updates all symbols from gams.exchange_container
in GAMS. In a similar way, only symbols that are new or have been assigned to, explicit or implicit (e.g. via a solve) in the GAMS code will be updated in the gams.exchange_container
. With a few exceptions, e.g. implicit loading in execution time embedded code section, and execute_loadpoint "file.gdx";
the reference feature of GAMS identifies such symbols. In case this does not identify all symbols modified by the GAMS the setting of gams.read_all = True
bypasses this logic and updates all symbols from GAMS in gams.exchange_container
.
While you can remove individual symbols from a container, via m.removeSymbols('i')
, the container might be in an unstable condition if the removed symbol is used as a domain in other symbols. Symbol constructors that add a symbol to a GAMS Transfer container m
, e.g. i = gams.transfer.Set(m, 'i')
or their corrresponding container methods (m.addSet('i')
) can be repeated for already existing symbols, but there is a limitation. Similar to repeated GAMS declarations, it continues to work as long as one does not change the structure (e.g. dimension or domain) of the symbol. If such changes to the symbol are necessary, one needs to first delete the symbol (extra care needs to be given to sets that are used as domain sets in other symbols) or start with a fresh exchange container by executing gams.reset()
or %gams_reset
. Unfortunately, symbols and data created in previous cells will also be gone after the gams.reset()
.
Some Data Transformation routines¶
from2dim¶
This transforms a data frame like this
Index | Mill A | Mill B | Mill C |
---|---|---|---|
1 | 8 | 15 | 50 |
2 | 10 | 17 | 20 |
3 | 30 | 26 | 15 |
into the data frame with GAMS readable format
Index | level_0 | level_1 | 0 |
---|---|---|---|
0 | 1 | Mill A | 8 |
1 | 1 | Mill B | 15 |
2 | 1 | Mill C | 50 |
3 | 2 | Mill A | 10 |
4 | 2 | Mill B | 17 |
5 | 2 | Mill C | 20 |
6 | 3 | Mill A | 30 |
7 | 3 | Mill B | 26 |
8 | 3 | Mill C | 15 |
call: gams.from2dim(df)
With the additional argument the default columns names can be renamed:
Index | sites | mills | value |
---|---|---|---|
0 | 1 | Mill A | 8 |
1 | 1 | Mill B | 15 |
2 | 1 | Mill C | 50 |
3 | 2 | Mill A | 10 |
4 | 2 | Mill B | 17 |
5 | 2 | Mill C | 20 |
6 | 3 | Mill A | 30 |
7 | 3 | Mill B | 26 |
8 | 3 | Mill C | 15 |
call: gams.from2dim(df,['sites','mills','value'])
The following cells show some examples:
import pandas as pd
Sites = ['1', '2', '3']
Mills = ['Mill A','Mill B', 'Mill C']
Dist = pd.DataFrame(index=Sites, columns=Mills, data = [[ 8, 15, 50],
[10, 17, 20],
[30, 26, 15]])
display(Dist)
display(gams.from2dim(Dist,['sites','mills','value']))
Mill A | Mill B | Mill C | |
---|---|---|---|
1 | 8 | 15 | 50 |
2 | 10 | 17 | 20 |
3 | 30 | 26 | 15 |
sites | mills | value | |
---|---|---|---|
0 | 1 | Mill A | 8 |
1 | 1 | Mill B | 15 |
2 | 1 | Mill C | 50 |
3 | 2 | Mill A | 10 |
4 | 2 | Mill B | 17 |
5 | 2 | Mill C | 20 |
6 | 3 | Mill A | 30 |
7 | 3 | Mill B | 26 |
8 | 3 | Mill C | 15 |
gams.reset() # we will use some different i and j below, so we better reset
m = gams.exchange_container
dist = m.addParameter('dist', ['sites','mills'])
dist.setRecords(gams.from2dim(Dist))
dist.records
level_0 | level_1 | value | |
---|---|---|---|
0 | 1 | Mill A | 8.0 |
1 | 1 | Mill B | 15.0 |
2 | 1 | Mill C | 50.0 |
3 | 2 | Mill A | 10.0 |
4 | 2 | Mill B | 17.0 |
5 | 2 | Mill C | 20.0 |
6 | 3 | Mill A | 30.0 |
7 | 3 | Mill B | 26.0 |
8 | 3 | Mill C | 15.0 |
# Now display Transfer data frame as table
dist.pivot()
Mill A | Mill B | Mill C | |
---|---|---|---|
1 | 8.0 | 15.0 | 50.0 |
2 | 10.0 | 17.0 | 20.0 |
3 | 30.0 | 26.0 | 15.0 |
# For 3 and more dimensional symbols we get
%gams set a / a1*a3 /, b /b1*b2 /, c /c1*c5 /; parameter abc(a,b,c); abc(a,b,c) = uniform(0,1)
m["abc"].pivot()
c1 | c2 | c3 | c4 | c5 | ||
---|---|---|---|---|---|---|
a1 | b1 | 0.171747 | 0.843267 | 0.550375 | 0.301138 | 0.292212 |
b2 | 0.224053 | 0.349831 | 0.856270 | 0.067114 | 0.500211 | |
a2 | b1 | 0.998118 | 0.578733 | 0.991133 | 0.762250 | 0.130692 |
b2 | 0.639719 | 0.159518 | 0.250081 | 0.668929 | 0.435356 | |
a3 | b1 | 0.359700 | 0.351441 | 0.131492 | 0.150102 | 0.589114 |
b2 | 0.830893 | 0.230816 | 0.665734 | 0.775858 | 0.303658 |
# Works also to display the level value (default) of a 2-dim variable
%gams set i /i1*i5/, j / j1*j3 /; variable ship(i,j); ship.l(i,j) = uniform(0,1);
m["ship"].pivot()
j1 | j2 | j3 | |
---|---|---|---|
i1 | 0.110492 | 0.502385 | 0.160173 |
i2 | 0.872462 | 0.265115 | 0.285814 |
i3 | 0.593956 | 0.722719 | 0.628249 |
i4 | 0.463798 | 0.413307 | 0.117695 |
i5 | 0.314212 | 0.046552 | 0.338550 |
# Can pivot the marginal value as well
%gams ship.m(i,j) = uniform(0,1);
m["ship"].pivot(value="marginal")
j1 | j2 | j3 | |
---|---|---|---|
i1 | 0.182100 | 0.645727 | 0.560746 |
i2 | 0.769962 | 0.297806 | 0.661106 |
i3 | 0.755822 | 0.627447 | 0.283864 |
i4 | 0.086425 | 0.102515 | 0.641251 |
i5 | 0.545309 | 0.031525 | 0.792361 |
GAMS Environment¶
The execution of GAMS lines and cells happens in a particular environment. At the start of the notebook no environment exists, but the base environment is created automatically as soon as required. In some situations it can be useful to have more than one GAMS environment, e.g. you have two models that have the same name for different symbols. In a Python cell you can create and/or activate a GAMS environment so that a GAMS cell is executed in this environment. The particular GAMS Transfer exchange container is also part of the environment. An environment is identified by its name. This name is also part of the scratch files created by executing GAMS cells and lines, e.g. gj_e6aff890_base_cp.g00
. The name consists of common start gj_
(for GAMS Jupyter), a unique id je6aff890
that is assigned when the gams
object is instantiated and is different in different notebooks. The next part _base_
contains the name of the environment this file belongs to. The _cp
and the extension .g00
identify this as the file that captures the state of the GAMS virtual machine.
The following examples, demonstrate the use of environments:
gams.active
'base'
gams.reset() # reset the base environment
%gams parameter p / i1.i2 3 /;
gams.exchange_container['p'].records
uni_0 | uni_1 | value | |
---|---|---|---|
0 | i1 | i2 | 3.0 |
# Create a new environment "new" that contains a 1-dimensional variable p
gams.create("new") # create has an optional argument activate=boolean with default "True"
print(f'Active environment: {gams.active}') # hence we see that the active environment is "new"
%gams variable p(*) / i1.L 3, i1.M 2 /;
gams.exchange_container['p'].records
Active environment: new
uni | level | marginal | lower | upper | scale | |
---|---|---|---|---|---|---|
0 | i1 | 3.0 | 2.0 | -inf | inf | 1.0 |
gams.activate('base') # this allows to switch back to the "base" environment
print(f'Active environment: {gams.active}') # hence we see that the active environment is "new"
gams.exchange_container['p'].records # with 2-dim parameter p
Active environment: base
uni_0 | uni_1 | value | |
---|---|---|---|
0 | i1 | i2 | 3.0 |
Troubleshooting and Hints¶
- Paths to notebooks must not contain whitespaces. A notebook file itself (*.ipynb) can.
- As soon as an error occurs while running GAMS code (the notebook exception is a GamsExecption), it can be useful to examine the listing file (*.lst) using
%cat path/to/listing.lst
or%gams_lst
. The path of the listing file can be found in the last line of the output of a failing cell.
%gams_cleanup --closedown
Converting GAMS Jupyter Notebooks into Python Scripts
Sometimes it can be useful to execute the logic of a GAMS Jupyter notebook in a standalone Python script. This can be achieved by using the gams.magic.GamsInteractive
class which implements the back-end logic of gams.magic
and does require neither IPython
nor jupyter
. Translating GAMS magic commands into Python method equivalents is stright forward. First, the Python equivalent of %reload_ext gams.magic
needs to be added to the beginning of a Python script:
Afterwards, each magic command or method available in GAMS Jupyter notebooks has an equivalent in GamsInteractive
with the same name. GAMS magic commands like %gams
or %gams_reset
can be translated into methods of the exact same name - in this case GamsInteractive.gams()
and GamsInteractive.gams_reset()
. Methods and properties which are accessed directly (without GAMS magic command) in a Jupyter notebook like gams.exchange_container
or gams.activate()
can be used in the exact same way from within GamsInteractive
. Options and parameters of GAMS magic commands can be used with corresponding arguments of the equivalent methods.
While GAMS magic commands can be translated very easily, certain interactive functionality like chart plotting might require a different mechanism when being translated. Also display()
might need to be changed into print()
or similar to be working in a standalone Python script. In addition, Jupyter notebooks display data which is returned by the last command of a Python cell. In Python we need to use another print()
of the return value to get it as output.
For a complete example of a translated GAMS Jupyter notebook see [PATH TO GAMS]/api/python/examples/magic/millco.py
which is a translation of the Millco.ipynb notebook.