Introduction

GAMS Jupyter Notebooks allow to use notebook technology in combination with GAMS. If you just want to learn GAMS there are probably better ways doing this. Notebooks allow you to combine GAMS and Python. The former works great with well structured data and optimization models, while the latter is very rich in features to retrieve, manipulate, and visualize data that comes in all sort of ways. Combining GAMS and Python in a notebook it is relatively easy to tell an optimization story with text, data, graphs, math, and models.

Getting Started

The first step in getting started with GAMS Jupyter Notebooks is to make your Python 3 installation aware of the GAMS Python API described in the Getting Started section of the API tutorial. We recommend to follow the steps below which are specifically tailored for getting started with GAMS Jupyter notebooks. While any Python 3.8 to 3.12 installation is supported, we recommend the use of miniconda Python distributions.

Attention: All core third party dependencies will be installed if the user supplies the optional pip syntax (pip install gamsapi[magic]).

In addition to the GAMS Python API collection (and the core magic dependencies), the examples located in [PATH TO GAMS]/api/python/examples/magic will require the additional packages: jupyterlab, matplotlib, and tabulate. The following code section shows how to create and set up a conda environment for GAMS Jupyter notebooks:

Windows
Unix/macOS
Linux

$ conda create --name gams python=3.10

$ conda activate gams

(gams)$ pip install gamsapi[magic] --find-links [PATH TO GAMS]\api\python\bdist

$ conda install jupyter

(gams)$ jupyter notebook

$ conda create --name gams python=3.10

$ conda activate gams

(gams)$ pip install gamsapi[magic] --find-links /Library/Frameworks/GAMS.framework/Versions/[GAMS MAJOR VERSION]/Resources/api/python/bdist

$ conda install jupyter

(gams)$ jupyter notebook

$ conda create --name gams python=3.10

$ conda activate gams

(gams)$ pip install gamsapi[transfer,magic] --find-links [PATH TO GAMS]/api/python/bdist

$ conda install jupyter

(gams)$ jupyter notebook

The notebooks Millco.ipynb and Introduction.ipynb located in api/python/examples/magic are good starting points to get familiar with Jupyter notebooks and GAMS. The remainder of this section gives the dialog of the Introduction.ipynb notebook. Please see also the other notebook examples:

Tutorial

Introduction

GAMS Jupyter Notebooks¶

GAMS Jupyter Notebooks allow to use notebook technology in combination with GAMS. If you just want to learn GAMS there are probably better ways doing this. Notebooks allow you to combine GAMS and Python. The former works great with well structured data and optimization models, while the latter is very rich in features to retrieve, manipulate, and visualize data that comes in all sort of ways. Combining GAMS and Python in a notebook it is relatively easy to tell an optimization story with text, data, graphs, math, and models.

The GAMS Jupyter Notebook builds on top of the Python 3 kernel. So by default the notebook cells are Python cells. Cells can be turned into GAMS cells, i.e. cells with GAMS syntax, using the Jupyter magic facility (first line in a cell is %%gams). GAMS magic commands enable GAMS support in Python Jupyter notebooks. Beside running GAMS code, it is possible to transfer data between GAMS and Python. In order to enable the GAMS magic commands, it is required to (re)load the extension gams.magic:

In [1]:

%reload_ext gams.magic

Running GAMS code¶

Running GAMS code can be done by using either %gams (line magic) or %%gams (cell magic). While %gams can be used for running a single line of GAMS code, %%gams makes the whole cell a GAMS cell. While %gams can appear on any line, %%gams is only valid in the first line of a cell.

In [2]:

%gams set i(*);

There are a few other useful commands in connection with running GAMS in a Jupyter notebook. Some transformation functions for pandas data frames (e.g. from2dim (see below)) are available through the gams instance that becomes available right after the %reload_ext gams.magic.

In [3]:

%%gams
set j(*);
parameter p(i,j);
parameter p2(i,j);

The GAMS compiler and execution system has been adjusted so one can run a GAMS cell multiple time, even if it contains a declaration or an equation definition, which is normally not possible in the GAMS system. The execution of the next two cells does not create a problem, which mimics the execution, modification, and reexecution of a cell.

In [4]:

%%gams
set i / peter,paul,mary /, j / A,B,C /;
parameter p2(i,j) / set.i.set.j 1 /;

In [5]:

%%gams
set i / i1*i5 /, j /j1*j5 /;
parameter p2(i,j) / set.i.set.j 1 /;

You won't see any output from a GAMS cell (unless there is a solve executed in the cell, see below). All output goes to the log and lst file. If you really need to see this you can use magic command %gams_log and %gams_lst to display the content of the log and listing file of the most recent GAMS execution. The next cell displays the content of listing file of the last run GAMS cell or line magic. The -e only display the section of the listing file associated with the execution:

In [6]:

%gams display p2;
%gams_lst -e

E x e c u t i o n


----     35 PARAMETER p2  

            j1          j2          j3          j4          j5

i1       1.000       1.000       1.000       1.000       1.000
i2       1.000       1.000       1.000       1.000       1.000
i3       1.000       1.000       1.000       1.000       1.000
i4       1.000       1.000       1.000       1.000       1.000
i5       1.000       1.000       1.000       1.000       1.000

There is a limit to the execution, modification, and reexecution of GAMS cells. If the type or the dimensionality of a symbol changes, you will need to execute the notebook from scratch and do a controlled reset of the entire GAMS database via %gams_reset. For example, since we declared parameter p2 already over (i,j) we cannot change our mind and redeclare p2 as parameter p2(i,i,j):

%gams parameter p2(i,i,j);

This will give you a compilation error and an exception in the cell execution (uncomment the line in the next cell to do so):

In [7]:

#%gams parameter p2(i,i,j);

In [8]:

%gams_reset

With a %gams_reset we can reset the GAMS database and can declare symbols with a different type and domain/dimension. All other things in the GAMS database are gone, too. So we need to redeclare the sets i and j, too. Furthermore %gams_reset provides an optional argument --system_directory=<path/to/gams> that allows to explicitly specify a GAMS system directory if the automatic detection of such is not wanted. Once a system directory has been specified, all %gams_reset and gams.reset() calls will take that one into account. Existing environments other than the active one are not affected automatically. Therefore a %gams_reset directly after %reload_ext gams.magic is helpful for specifying a non-default system directory for all subsequent environments. It is possible to switch back to the default behavior by providing the empty string as system directory (%gams_reset --system_directory=). The state of the GAMS database is kept in various files that can easily clutter your directory. The %gams_cleanup call helps you to clean the directory of temporary files of your current environment. The option -k keeps the most recent GAMS database, hence the %gams_cleanup -k is a save call anywhere in your notebook. The option -a removes also files from other environments in your notebook. This option can be combined with -k. The option -c or --closedown is to totally cleanup and leaves you with a fresh base environment only.

Exchanging data between Python and GAMS¶

The property gams.exchange_container references a GAMS Transfer container that organizes the data exchange between Python and GAMS. Symbols created in gams.exchange_container will be available in GAMS as well as symbols declared in GAMS will be available in gams.exchange_container:

In [9]:

import numpy as np
m = gams.exchange_container
i = m.addSet('i', records=["i1","i2","i3"])
j = m.addSet('j', records=["j1","j2"])
p = m.addParameter('p', [i,j], records=np.array([[1.1,2.2],
                                                 [3.3,4.4],
                                                 [5.5,6.6]]))

In [10]:

%gams display i, j, p;
%gams_lst -e

E x e c u t i o n


----     13 SET i  

i1,    i2,    i3


----     13 SET j  

j1,    j2


----     13 PARAMETER p  

            j1          j2

i1       1.100       2.200
i2       3.300       4.400
i3       5.500       6.600

The next cell declares a symbol pp in GAMS and multiplies the value of the parallel p by 2. The second line is again Python code to display the data frame of pp using the symbol method <symbol>.pivot().

In [11]:

%gams parameter pp(i,j); pp(i,j) = 2*p(i,j)
m['pp'].pivot()

Out[11]:

	j1	j2
i1	2.2	4.4
i2	6.6	8.8
i3	11.0	13.2

The data frames GAMS Transfer uses to store data have a very specific layout. The setRecords method or records= argument on a symbol constructor allow to provide data from many different formats and Python data structures. See the GAMS Transfer documentation for details.

The GAMS line or cell magic will only synchronize symbols from gams.exchange_container to GAMS that have been written to since the last GAMS line or cell magic execution. GAMS magic uses the Transfer attribute modified in order to decide if the symbol needs to be synchronized with GAMS. In exceptional situations the setting of gams.write_all = True bypasses this logic and updates all symbols from gams.exchange_container in GAMS. In a similar way, only symbols that are new or have been assigned to, explicit or implicit (e.g. via a solve) in the GAMS code will be updated in the gams.exchange_container. With a few exceptions, e.g. implicit loading in execution time embedded code section, and execute_loadpoint "file.gdx"; the reference feature of GAMS identifies such symbols. In case this does not identify all symbols modified by the GAMS the setting of gams.read_all = True bypasses this logic and updates all symbols from GAMS in gams.exchange_container.

While you can remove individual symbols from a container, via m.removeSymbols('i'), the container might be in an unstable condition if the removed symbol is used as a domain in other symbols. Symbol constructors that add a symbol to a GAMS Transfer container m, e.g. i = gams.transfer.Set(m, 'i') or their corrresponding container methods (m.addSet('i')) can be repeated for already existing symbols, but there is a limitation. Similar to repeated GAMS declarations, it continues to work as long as one does not change the structure (e.g. dimension or domain) of the symbol. If such changes to the symbol are necessary, one needs to first delete the symbol (extra care needs to be given to sets that are used as domain sets in other symbols) or start with a fresh exchange container by executing gams.reset() or %gams_reset. Unfortunately, symbols and data created in previous cells will also be gone after the gams.reset().

Some Data Transformation routines¶

from2dim¶

This transforms a data frame like this

Index	Mill A	Mill B	Mill C
1	8	15	50
2	10	17	20
3	30	26	15

into the data frame with GAMS readable format

Index	level_0	level_1	0
0	1	Mill A	8
1	1	Mill B	15
2	1	Mill C	50
3	2	Mill A	10
4	2	Mill B	17
5	2	Mill C	20
6	3	Mill A	30
7	3	Mill B	26
8	3	Mill C	15

call: gams.from2dim(df)

With the additional argument the default columns names can be renamed:

Index	sites	mills	value
0	1	Mill A	8
1	1	Mill B	15
2	1	Mill C	50
3	2	Mill A	10
4	2	Mill B	17
5	2	Mill C	20
6	3	Mill A	30
7	3	Mill B	26
8	3	Mill C	15

call: gams.from2dim(df,['sites','mills','value'])

The following cells show some examples:

In [12]:

import pandas as pd
Sites = ['1', '2', '3']
Mills = ['Mill A','Mill B', 'Mill C']
Dist = pd.DataFrame(index=Sites, columns=Mills, data = [[ 8, 15, 50], 
                                                        [10, 17, 20],
                                                        [30, 26, 15]])
display(Dist)
display(gams.from2dim(Dist,['sites','mills','value']))

	Mill A	Mill B	Mill C
1	8	15	50
2	10	17	20
3	30	26	15

	sites	mills	value
0	1	Mill A	8
1	1	Mill B	15
2	1	Mill C	50
3	2	Mill A	10
4	2	Mill B	17
5	2	Mill C	20
6	3	Mill A	30
7	3	Mill B	26
8	3	Mill C	15

In [13]:

gams.reset() # we will use some different i and j below, so we better reset
m = gams.exchange_container
dist = m.addParameter('dist', ['sites','mills'])
dist.setRecords(gams.from2dim(Dist))
dist.records

Out[13]:

	level_0	level_1	value
0	1	Mill A	8.0
1	1	Mill B	15.0
2	1	Mill C	50.0
3	2	Mill A	10.0
4	2	Mill B	17.0
5	2	Mill C	20.0
6	3	Mill A	30.0
7	3	Mill B	26.0
8	3	Mill C	15.0

In [14]:

# Now display Transfer data frame as table
dist.pivot()

Out[14]:

	Mill A	Mill B	Mill C
1	8.0	15.0	50.0
2	10.0	17.0	20.0
3	30.0	26.0	15.0

In [15]:

# For 3 and more dimensional symbols we get
%gams set a / a1*a3 /, b /b1*b2 /, c /c1*c5 /; parameter abc(a,b,c); abc(a,b,c) = uniform(0,1)
m["abc"].pivot()

Out[15]:

		c1	c2	c3	c4	c5
a1	b1	0.171747	0.843267	0.550375	0.301138	0.292212
a1	b2	0.224053	0.349831	0.856270	0.067114	0.500211
a2	b1	0.998118	0.578733	0.991133	0.762250	0.130692
a2	b2	0.639719	0.159518	0.250081	0.668929	0.435356
a3	b1	0.359700	0.351441	0.131492	0.150102	0.589114
a3	b2	0.830893	0.230816	0.665734	0.775858	0.303658

In [16]:

# Works also to display the level value (default) of a 2-dim variable
%gams set i /i1*i5/, j / j1*j3 /; variable ship(i,j); ship.l(i,j) = uniform(0,1);
m["ship"].pivot()

Out[16]:

	j1	j2	j3
i1	0.110492	0.502385	0.160173
i2	0.872462	0.265115	0.285814
i3	0.593956	0.722719	0.628249
i4	0.463798	0.413307	0.117695
i5	0.314212	0.046552	0.338550

In [17]:

# Can pivot the marginal value as well
%gams ship.m(i,j) = uniform(0,1);
m["ship"].pivot(value="marginal")

Out[17]:

	j1	j2	j3
i1	0.182100	0.645727	0.560746
i2	0.769962	0.297806	0.661106
i3	0.755822	0.627447	0.283864
i4	0.086425	0.102515	0.641251
i5	0.545309	0.031525	0.792361

GAMS Environment¶

The execution of GAMS lines and cells happens in a particular environment. At the start of the notebook no environment exists, but the base environment is created automatically as soon as required. In some situations it can be useful to have more than one GAMS environment, e.g. you have two models that have the same name for different symbols. In a Python cell you can create and/or activate a GAMS environment so that a GAMS cell is executed in this environment. The particular GAMS Transfer exchange container is also part of the environment. An environment is identified by its name. This name is also part of the scratch files created by executing GAMS cells and lines, e.g. gj_e6aff890_base_cp.g00. The name consists of common start gj_ (for GAMS Jupyter), a unique id je6aff890 that is assigned when the gams object is instantiated and is different in different notebooks. The next part _base_ contains the name of the environment this file belongs to. The _cp and the extension .g00 identify this as the file that captures the state of the GAMS virtual machine.

The following examples, demonstrate the use of environments:

In [18]:

gams.active

Out[18]:

'base'

In [19]:

gams.reset() # reset the base environment
%gams parameter p / i1.i2 3 /;
gams.exchange_container['p'].records

Out[19]:

	uni_0	uni_1	value
0	i1	i2	3.0

In [20]:

# Create a new environment "new" that contains a 1-dimensional variable p
gams.create("new") # create has an optional argument activate=boolean with default "True"
print(f'Active environment: {gams.active}') # hence we see that the active environment is "new"
%gams variable p(*) / i1.L 3, i1.M 2 /;
gams.exchange_container['p'].records

Active environment: new

Out[20]:

	uni	level	marginal	lower	upper	scale
0	i1	3.0	2.0	-inf	inf	1.0

In [21]:

gams.activate('base') # this allows to switch back to the "base" environment
print(f'Active environment: {gams.active}') # hence we see that the active environment is "new"
gams.exchange_container['p'].records # with 2-dim parameter p

Active environment: base

Out[21]:

	uni_0	uni_1	value
0	i1	i2	3.0

Troubleshooting and Hints¶

Paths to notebooks must not contain whitespaces. A notebook file itself (*.ipynb) can.
As soon as an error occurs while running GAMS code (the notebook exception is a GamsExecption), it can be useful to examine the listing file (*.lst) using %cat path/to/listing.lst or %gams_lst. The path of the listing file can be found in the last line of the output of a failing cell.

In [22]:

%gams_cleanup --closedown

Converting GAMS Jupyter Notebooks into Python Scripts

Sometimes it can be useful to execute the logic of a GAMS Jupyter notebook in a standalone Python script. This can be achieved by using the gams.magic.GamsInteractive class which implements the back-end logic of gams.magic and does require neither IPython nor jupyter. Translating GAMS magic commands into Python method equivalents is stright forward. First, the Python equivalent of %reload_ext gams.magic needs to be added to the beginning of a Python script:

from gams.magic import GamsInteractive

gams = GamsInteractive()

Afterwards, each magic command or method available in GAMS Jupyter notebooks has an equivalent in GamsInteractive with the same name. GAMS magic commands like %gams or %gams_reset can be translated into methods of the exact same name - in this case GamsInteractive.gams() and GamsInteractive.gams_reset(). Methods and properties which are accessed directly (without GAMS magic command) in a Jupyter notebook like gams.exchange_container or gams.activate() can be used in the exact same way from within GamsInteractive. Options and parameters of GAMS magic commands can be used with corresponding arguments of the equivalent methods.

While GAMS magic commands can be translated very easily, certain interactive functionality like chart plotting might require a different mechanism when being translated. Also display() might need to be changed into print() or similar to be working in a standalone Python script. In addition, Jupyter notebooks display data which is returned by the last command of a Python cell. In Python we need to use another print() of the return value to get it as output.

For a complete example of a translated GAMS Jupyter notebook see [PATH TO GAMS]/api/python/examples/magic/millco.py which is a translation of the Millco.ipynb notebook.

Table of Contents

Introduction

Getting Started

Tutorial

GAMS Jupyter Notebooks¶

Running GAMS code¶

Exchanging data between Python and GAMS¶

Some Data Transformation routines¶

from2dim¶

GAMS Environment¶

Troubleshooting and Hints¶

Converting GAMS Jupyter Notebooks into Python Scripts