How to use GAMS in Jupyter Notebooks

Posted on: 21 Aug, 2020 Features News

For quite a while we have been providing the option to play around with GAMS in Jupyter notebooks in a hosted environment at https://jupyterhub.gams.com/ . That place is still a good resource for getting started and provides a few examples of how to use GAMS in this environment. [Note: jupyterhub.gams.com has been shut down in March 2023]

We have received a lot of requests to make this available for local installation, and we listened. Without further ado, here is how to get started:

Installation

Windows

  1. Install a python environment, if you do not have one already. We will use miniconda, which you can download at https://docs.conda.io/en/latest/miniconda.html .
  2. Open the Anaconda Prompt
  3. run conda create -n gmsjupyter python=3.8. This will create a new environment for you and install needed dependencies.
  4. Activate the new environment with conda activate gmsjupyter.
  5. Install jupyterlab with conda install jupyterlab
  6. Install pandas: conda install pandas
  7. Install tabulate: conda install tabulate

Now it is time to integrate GAMS into your new Python environment:

cd c:\GAMS\32\apifiles\Python\api_38
python setup.py build -b %TEMP%\build install

Create a directory for your Jupyter notebooks (I will use C:\Users\manns\Documents\GAMS\Jupyter here), and then cd into the directory: cd C:\Users\manns\Documents\GAMS\Jupyter.

Now you can start Jupyter with jupyter notebook. Once the notebook is loaded in your webbrowser, create a new Python notebook with New > Python3.

Linux

We assume you have an up-to-date python installation.

  1. Install pip if you have not done so before: sudo apt-get install -y python3-pip
  2. Install the venv module, which will enable us to create an isolated GAMS environment: sudo apt-get install -y python3-venv
  3. Create a directory to store your python environments in:
mkdir environments
cd environments
  1. Create a dedicated GAMS environment: python3 -m venv gmsjupyter
  2. Activate the environment: source gmsjupyter/bin/activate
  3. Install Jupyterlab: pip3 install jupyterlab
  4. Install pandas: pip3 install pandas
  5. Install tabulate: pip3 install tabulate

Now it is time to integrate GAMS into your new Python environment:

cd ~/gams/gams32.1_linux_x64_64_sfx/apifiles/Python/api_38$
python setup.py build -b %TEMP%/build install

Create a directory for your Jupyter notebooks (I will use ~/gams/jupyter here), and then cd into the directory.

Now you can start Jupyter with jupyter notebook. Once the notebook is loaded in your webbrowser, create a new Python notebook with New > Python3. The examples below have been done under Windows, but should run the same under linux.

Setting up your jupyter notebook for GAMS

The GAMS Jupyter Notebook builds on top of the Python 3 kernel. So by default the notebook cells are Python cells. Cells can be turned into GAMS cells, i.e. cells with GAMS syntax, using the Jupyter magic facility (first line in a cell is %%gams). GAMS magic commands enable GAMS support in Python Jupyter notebooks. Beside running GAMS code, it is possible to transfer data between GAMS and Python. In order to enable the GAMS magic commands, it is required to load the extension gams_magic:

%load_ext gams_magic

There are a few other useful commands in connection with running GAMS in a Jupyter notebook. Some transformation functions for pandas dataframes useful for exchange with GAMS have been collected in the notebook DataTransform.ipynb. To use these, download DataTransform.ipynb and copy the file into your working directory.

The next cell will execute that notebook and make such data transformation functions, e.g. gt_from2dim (see below) available in this notebook. %%capture captures the output from the execution of the notebook and does not clutter your output.

%%capture
%run DataTransform.ipynb

One output from a cell is sometimes not enough, e.g. if you want to display a couple of tables. The display function allows you to do this but needs to imported. As an example, we display a Python list:

from IPython.display import display
display([1,2,3])
[1, 2, 3]

Running GAMS code

Running GAMS code can be done by using either %gams (line magic) or %%gams (cell magic). While %gams can be used for running a single line of GAMS code, %%gams makes the whole cell a GAMS cell.

%gams set i;
%%gams
set j;
parameter p(i,j);
parameter p2(i,j);

The GAMS compiler and execution system has been adjusted so one can run a GAMS cell multiple times, even if it contains a declaration or an equation definition, which is normally not possible in the GAMS system. The execution of the next two cells does not create a problem, which mimics the execution, modification, and reexecution of a cell.

%%gams
set i / peter,paul,mary /, j / A,B,C /;
parameter p2(i,j) / set.i.set.j 1 /;
%%gams
set i / i1*i5 /, j /j1*j5 /;
parameter p2(i,j) / set.i.set.j 1 /;

You won’t see any output from a GAMS cell (unless there is a solve executed in the cell, see below). All output goes to the log and lst file. If you really need to see this you can use magic command %gams_log and %gams_lst to display the content of the log and listing file of the most recent GAMS execution. The next cell displays the content of listing file of the last run GAMS cell or line magic. The -e only display the section of the listing file associated with the execution:

%gams display p2;
%gams_lst -e
E x e c u t i o n


----     20 PARAMETER p2  

            j1          j2          j3          j4          j5

i1       1.000       1.000       1.000       1.000       1.000
i2       1.000       1.000       1.000       1.000       1.000
i3       1.000       1.000       1.000       1.000       1.000
i4       1.000       1.000       1.000       1.000       1.000
i5       1.000       1.000       1.000       1.000       1.000

When things go wrong

There is a limit to the execution, modification, and reexecution of GAMS cells. If the type or the dimensionality of a symbol changes, you will need to execute the notebook from scratch and do a controlled reset of the entire GAMS database via %gams_reset. For example, since we declared parameter p2 already over (i,j) we cannot change our mind and redeclare p2 as parameter p2(i,i,j):

This will give you a compilation error and an exception in the cell execution:

%gams parameter p2(i,i,j);
---------------------------------------------------------------------------

GamsExceptionExecution                    Traceback (most recent call last)

<ipython-input-9-49112f96e0c3> in <module>
----> 1 get_ipython().run_line_magic('gams', 'parameter p2(i,i,j);')


~\miniconda3\envs\gmsjupyter\lib\site-packages\IPython\core\interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2324                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2325             with self.builtin_trap:
-> 2326                 result = fn(*args, **kwargs)
   2327             return result
   2328


<decorator-gen-133> in gams(self, line, cell)


~\miniconda3\envs\gmsjupyter\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188
    189         if callable(arg):


~\miniconda3\envs\gmsjupyter\lib\site-packages\gams_magic\gams_magic.py in gams(self, line, cell)
    451             opt.traceopt = 3
    452             with open(jobName + ".log", "w") as logFile:
--> 453                 self.job.run(opt, checkpoint=self.cp, output=logFile)
    454             solveSummary = self.parseTraceFile(trcFilePath)
    455


~\miniconda3\envs\gmsjupyter\lib\site-packages\gams\execution.py in run(self, gams_options, checkpoint, output, create_out_db, databases)
    905                 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), set the debug flag of the GamsWorkspace constructor to DebugLevel.KeepFiles or higher or define a working_directory to receive a listing file with more details", exitcode)
    906             else:
--> 907                 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), check " + self._workspace._working_directory + os.path.sep + tmp_opt.output + " for more details", exitcode)
    908         self._p = None
    909


GamsExceptionExecution: GAMS return code not 0 (2), check C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter7.lst for more details

To find out what went wrong, we can use %gams_lst, which tells us that we tried to redefine the domain list:

%gams_lst
GAMS 32.1.0  r75a5b5d Released Jul 31, 2020 WEX-WEI x86 64bit/MS Windows - 08/21/20 12:39:07 Page 9
G e n e r a l   A l g e b r a i c   M o d e l i n g   S y s t e m
C o m p i l a t i o n


  23  parameter p2(i,i,j);
****                  $184,184
**** LINE      3 INPUT       C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter7.gms
**** 184  Domain list redefined

**** 2 ERROR(S)   0 WARNING(S)


COMPILATION TIME     =        0.000 SECONDS      3 MB  32.1.0 r75a5b5d WEX-WEI


USER: GAMS Evaluation License                        S200819/0001CO-GEN
      GAMS Software GmbH, Frechen Office                         DCE839


**** FILE SUMMARY

Restart    C:\Users\manns\Documents\GAMS\Jupyter\_gams_py_gcp0.g00
Input      C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter7.gms
Output     C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter7.lst
Save       C:\Users\manns\Documents\GAMS\Jupyter\_gams_py_gcp6.g0?

**** USER ERROR(S) ENCOUNTERED

With a %gams_reset we can reset the GAMS database and can declare symbols with a different type and domain/dimension. All other things in the GAMS database are gone, too. So we need to redeclare the sets i and j, too.

%gams_reset
%gams set i,j; parameter p(i,j), p2(i,i,j);

Pushing Data from Python to GAMS

%gams_push transfers data from Python to GAMS. Supported data types for pushing data are lists, pandas.DataFrame and numpy arrays:

# Define Python lists with data
i = ['i1', 'i2', 'i3']
j = ['j1', 'j2']
p = [('i1', 'j1', 1.1), ('i1', 'j2', 2.2), ('i2', 'j1', 3.3), ('i2','j2', 4.4), ('i3','j1', 5.5), ('i3', 'j2', 6.6)]
%gams_push i j p

As mentioned above the execution of a %%gams cell or %gams and %gams_push line magic does not produce output. If one wants to verify that the data ended up in GAMS we can display the symbols in GAMS and output the corresponding part of the listing file:

%gams display i,j,p;
%gams_lst -e
E x e c u t i o n


----     15 SET i  

i1,    i2,    i3


----     15 SET j  

j1,    j2


----     15 PARAMETER p  

            j1          j2

i1       1.100       2.200
i2       3.300       4.400
i3       5.500       6.600

The next cell turns a Python list into a pandas.DataFrame, multiplies the value by 2 and displays the dataframe with IPythons’s display. We actually display the transformed p2 (via function gt_pivot2d found in the DataTransformation notebook run at the top of the notebook), so the table looks nicer. Next, we sends the pandas.DataFrame down to GAMS via the %gams_push command. Via the GAMS display and the output of the relevant part of the listing file we see that the %gams_push succeeded:

import pandas as pd
# turn the Python list p into a pandas.Dataframe p2 and send this down to GAMS
pp = pd.DataFrame(p)
# multiply the value by 2:
pp[2] = 2*pp[2]
# display a nicer version of the dataframe:
display(gt_pivot2d(pp))
%gams parameter pp(i,j)
%gams_push pp
%gams display pp;
%gams_lst -e

j1 j2
i1 2.2 4.4
i2 6.6 8.8
i3 11.0 13.2
E x e c u t i o n


----     25 PARAMETER pp  

            j1          j2

i1       2.200       4.400
i2       6.600       8.800
i3      11.000      13.200

When using numpy arrays in order to push data into GAMS, the data is assumed to be dense. The corresponding sets are defined automatically from 1..n, 1..m, etc depending on the data that is pushed.

import numpy as np
data = [[[1.1,-1.1], [2.2,-2.2]], [[3.3,-3.3], [4.4,-4.4]], [[5.5,-5.5], [6.6,-6.6]]]
p3 = np.array(data)
%gams set i, j, k; parameter p3(i,j,k);
%gams_push p3
%gams display i,j,k,p3;
%gams_lst -e
E x e c u t i o n


----     34 SET i  

1,    2,    3


----     34 SET j  

1,    2


----     34 SET k  

1,    2


----     34 PARAMETER p3  3-dim Matrix

              1           2

1.1       1.100      -1.100
1.2       2.200      -2.200
2.1       3.300      -3.300
2.2       4.400      -4.400
3.1       5.500      -5.500
3.2       6.600      -6.600

Pulling Data from GAMS to Python

The line magic %gams_pull transfers data from GAMS to Python in different formats. Supported formats are lists (default), pandas.DataFrame and numpy arrays. The following example pulls the sets i, j, and parameter p3 from GAMS into lists. For multi-dimensional symbols the records become Python tuples. Currently, the renaming functionality %gams_pull gamsSym=pySymbol is not yet supported.

%gams_pull p3 i j
display(i,j,p3)
['1', '2', '3']



['1', '2']



[('1', '1', '1', 1.1),
 ('1', '1', '2', -1.1),
 ('1', '2', '1', 2.2),
 ('1', '2', '2', -2.2),
 ('2', '1', '1', 3.3),
 ('2', '1', '2', -3.3),
 ('2', '2', '1', 4.4),
 ('2', '2', '2', -4.4),
 ('3', '1', '1', 5.5),
 ('3', '1', '2', -5.5),
 ('3', '2', '1', 6.6),
 ('3', '2', '2', -6.6)]

The switch -d will populate pandas.DataFrames instead of lists with the GAMS data. The dataframes that are pushed into or pulled from GAMS have a very specific layout. There is a record index and the GAMS domains show up as columns in the dataframe. For parameters, there is an extra value column. For variables and equations we find extra columns level, marginal, lower, upper, and scale. The method head() used in the IPython display provides only the first 5 records of a pandas.DataFrame:

%gams variable x(i) / 1.L 1, 2.M 3 /;
%gams_pull -d i j p3 x
display(i, j, p3.head(), x)

i
0 1
1 2
2 3

j
0 1
1 2

i j k value
0 1 1 1 1.1
1 1 1 2 -1.1
2 1 2 1 2.2
3 1 2 2 -2.2
4 2 1 1 3.3

i level marginal lower upper scale
0 1 1.0 0.0 -inf inf 1.0
1 2 0.0 3.0 -inf inf 1.0

The data transformation functions available from DataTransformations.ipynb help to convert between this format and formats more suitable for display of other transformations in Python. The following lines give a quick overview of the transformation functionality:

%gams parameter r(i,j); r(i,j) = uniformInt(1,10);
%gams_pull -d r
display(r,gt_pivot2d(r),gt_from2dim(gt_pivot2d(r),['i','j','value']))

i j value
0 1 1 2.0
1 1 2 9.0
2 2 1 6.0
3 2 2 4.0
4 3 1 3.0
5 3 2 3.0

1 2
1 2.0 9.0
2 6.0 4.0
3 3.0 3.0

i j value
0 1 1 2.0
1 1 2 9.0
2 2 1 6.0
3 2 2 4.0
4 3 1 3.0
5 3 2 3.0

The switch -n will populate numpy arrays instead of lists with the GAMS parameters. This format works with parameters only! The GAMS data will be dropped into a dense numpy array:

%gams parameter p4(i,j) / 1.1 1, 2.2 2 /;
%gams_pull -n p4
display(p4)
array([[1., 0.],
       [0., 2.],
       [0., 0.]])

Troubleshooting and Hints

  • Paths to notebooks must not contain whitespaces. A notebook file itself (*.ipynb) can.
  • The temporary files created in your working directory are useful for debugging, see below. The naming of the temporary files is not very sophisticated, so it can come to file nameing conflicts if you run two notebooks in the same directory at the same time (in different browser tabs). Create subdirectories and move the notebook into the subdirectories if you run into this problem.
  • As soon as an error occurs while running GAMS code (the notebook exception is a GamsExecption), it can be useful to examine the listing file (*.lst) using %gams_lst. The path of the listing file can be found in the last line of the output of a failing cell.
%gams Parameter pt(i,j,l)
---------------------------------------------------------------------------

GamsExceptionExecution                    Traceback (most recent call last)

<ipython-input-25-4c01a25a5766> in <module>
----> 1 get_ipython().run_line_magic('gams', 'Parameter pt(i,j,l)')


~\miniconda3\envs\gmsjupyter\lib\site-packages\IPython\core\interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2324                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2325             with self.builtin_trap:
-> 2326                 result = fn(*args, **kwargs)
   2327             return result
   2328


<decorator-gen-133> in gams(self, line, cell)


~\miniconda3\envs\gmsjupyter\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188
    189         if callable(arg):


~\miniconda3\envs\gmsjupyter\lib\site-packages\gams_magic\gams_magic.py in gams(self, line, cell)
    451             opt.traceopt = 3
    452             with open(jobName + ".log", "w") as logFile:
--> 453                 self.job.run(opt, checkpoint=self.cp, output=logFile)
    454             solveSummary = self.parseTraceFile(trcFilePath)
    455


~\miniconda3\envs\gmsjupyter\lib\site-packages\gams\execution.py in run(self, gams_options, checkpoint, output, create_out_db, databases)
    905                 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), set the debug flag of the GamsWorkspace constructor to DebugLevel.KeepFiles or higher or define a working_directory to receive a listing file with more details", exitcode)
    906             else:
--> 907                 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), check " + self._workspace._working_directory + os.path.sep + tmp_opt.output + " for more details", exitcode)
    908         self._p = None
    909


GamsExceptionExecution: GAMS return code not 0 (2), check C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter19.lst for more details
%gams_lst
GAMS 32.1.0  r75a5b5d Released Jul 31, 2020 WEX-WEI x86 64bit/MS Windows - 08/21/20 12:54:55 Page 28
G e n e r a l   A l g e b r a i c   M o d e l i n g   S y s t e m
C o m p i l a t i o n


  49  Parameter pt(i,j,l)
****                   $120
**** LINE      3 INPUT       C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter19.gms
**** 120  Unknown identifier entered as set

**** 1 ERROR(S)   0 WARNING(S)


COMPILATION TIME     =        0.000 SECONDS      3 MB  32.1.0 r75a5b5d WEX-WEI


USER: GAMS Evaluation License                        S200819/0001CO-GEN
      GAMS Software GmbH, Frechen Office                         DCE839


**** FILE SUMMARY

Restart    C:\Users\manns\Documents\GAMS\Jupyter\_gams_py_gcp0.g00
Input      C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter19.gms
Output     C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter19.lst
Save       C:\Users\manns\Documents\GAMS\Jupyter\_gams_py_gcp16.g0?

**** USER ERROR(S) ENCOUNTERED