For quite a while we have been providing the option to play around with GAMS in Jupyter notebooks in a hosted environment at https://jupyterhub.gams.com/
. That place is still a good resource for getting started and provides a few examples of how to use GAMS in this environment. [Note: jupyterhub.gams.com has been shut down in March 2023]
We have received a lot of requests to make this available for local installation, and we listened. Without further ado, here is how to get started:
Installation
Windows
- Install a python environment, if you do not have one already. We will use miniconda, which you can download at https://docs.conda.io/en/latest/miniconda.html .
- Open the Anaconda Prompt
- run
conda create -n gmsjupyter python=3.8. This will create a new environment for you and install needed dependencies. - Activate the new environment with
conda activate gmsjupyter. - Install jupyterlab with
conda install jupyterlab - Install pandas:
conda install pandas - Install tabulate:
conda install tabulate
Now it is time to integrate GAMS into your new Python environment:
cd c:\GAMS\32\apifiles\Python\api_38
python setup.py build -b %TEMP%\build install
Create a directory for your Jupyter notebooks (I will use C:\Users\manns\Documents\GAMS\Jupyter here), and then cd into the directory: cd C:\Users\manns\Documents\GAMS\Jupyter.
Now you can start Jupyter with jupyter notebook. Once the notebook is loaded in your webbrowser, create a new Python notebook with New > Python3.
Linux
We assume you have an up-to-date python installation.
- Install pip if you have not done so before:
sudo apt-get install -y python3-pip - Install the venv module, which will enable us to create an isolated GAMS environment:
sudo apt-get install -y python3-venv - Create a directory to store your python environments in:
mkdir environments
cd environments
- Create a dedicated GAMS environment:
python3 -m venv gmsjupyter - Activate the environment:
source gmsjupyter/bin/activate - Install Jupyterlab:
pip3 install jupyterlab - Install pandas:
pip3 install pandas - Install tabulate:
pip3 install tabulate
Now it is time to integrate GAMS into your new Python environment:
cd ~/gams/gams32.1_linux_x64_64_sfx/apifiles/Python/api_38$
python setup.py build -b %TEMP%/build install
Create a directory for your Jupyter notebooks (I will use ~/gams/jupyter here), and then cd into the directory.
Now you can start Jupyter with jupyter notebook. Once the notebook is loaded in your webbrowser, create a new Python notebook with New > Python3. The examples below have been done under Windows, but should run the same under linux.
Setting up your jupyter notebook for GAMS
The GAMS Jupyter Notebook builds on top of the Python 3 kernel. So by default the notebook cells are Python cells. Cells can be turned into GAMS cells, i.e. cells with GAMS syntax, using the Jupyter magic facility (first line in a cell is %%gams). GAMS magic commands enable GAMS support in Python Jupyter notebooks. Beside running GAMS code, it is possible to transfer data between GAMS and Python. In order to enable the GAMS magic commands, it is required to load the extension gams_magic:
%load_ext gams_magic
There are a few other useful commands in connection with running GAMS in a Jupyter notebook. Some transformation functions for pandas dataframes useful for exchange with GAMS have been collected in the notebook DataTransform.ipynb. To use these, download DataTransform.ipynb
and copy the file into your working directory.
The next cell will execute that notebook and make such data transformation functions, e.g. gt_from2dim (see below) available in this notebook. %%capture captures the output from the execution of the notebook and does not clutter your output.
%%capture
%run DataTransform.ipynb
One output from a cell is sometimes not enough, e.g. if you want to display a couple of tables. The display function allows you to do this but needs to imported. As an example, we display a Python list:
from IPython.display import display
display([1,2,3])
[1, 2, 3]
Running GAMS code
Running GAMS code can be done by using either %gams (line magic) or %%gams (cell magic). While %gams can be used for running a single line of GAMS code, %%gams makes the whole cell a GAMS cell.
%gams set i;
%%gams
set j;
parameter p(i,j);
parameter p2(i,j);
The GAMS compiler and execution system has been adjusted so one can run a GAMS cell multiple times, even if it contains a declaration or an equation definition, which is normally not possible in the GAMS system. The execution of the next two cells does not create a problem, which mimics the execution, modification, and reexecution of a cell.
%%gams
set i / peter,paul,mary /, j / A,B,C /;
parameter p2(i,j) / set.i.set.j 1 /;
%%gams
set i / i1*i5 /, j /j1*j5 /;
parameter p2(i,j) / set.i.set.j 1 /;
You won’t see any output from a GAMS cell (unless there is a solve executed in the cell, see below). All output goes to the log and lst file. If you really need to see this you can use magic command %gams_log and %gams_lst to display the content of the log and listing file of the most recent GAMS execution. The next cell displays the content of listing file of the last run GAMS cell or line magic. The -e only display the section of the listing file associated with the execution:
%gams display p2;
%gams_lst -e
E x e c u t i o n
---- 20 PARAMETER p2
j1 j2 j3 j4 j5
i1 1.000 1.000 1.000 1.000 1.000
i2 1.000 1.000 1.000 1.000 1.000
i3 1.000 1.000 1.000 1.000 1.000
i4 1.000 1.000 1.000 1.000 1.000
i5 1.000 1.000 1.000 1.000 1.000
When things go wrong
There is a limit to the execution, modification, and reexecution of GAMS cells. If the type or the dimensionality of a symbol changes, you will need to execute the notebook from scratch and do a controlled reset of the entire GAMS database via %gams_reset. For example, since we declared parameter p2 already over (i,j) we cannot change our mind and redeclare p2 as parameter p2(i,i,j):
This will give you a compilation error and an exception in the cell execution:
%gams parameter p2(i,i,j);
---------------------------------------------------------------------------
GamsExceptionExecution Traceback (most recent call last)
<ipython-input-9-49112f96e0c3> in <module>
----> 1 get_ipython().run_line_magic('gams', 'parameter p2(i,i,j);')
~\miniconda3\envs\gmsjupyter\lib\site-packages\IPython\core\interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
2324 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
2325 with self.builtin_trap:
-> 2326 result = fn(*args, **kwargs)
2327 return result
2328
<decorator-gen-133> in gams(self, line, cell)
~\miniconda3\envs\gmsjupyter\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, *a, **k: f(*a, **k)
188
189 if callable(arg):
~\miniconda3\envs\gmsjupyter\lib\site-packages\gams_magic\gams_magic.py in gams(self, line, cell)
451 opt.traceopt = 3
452 with open(jobName + ".log", "w") as logFile:
--> 453 self.job.run(opt, checkpoint=self.cp, output=logFile)
454 solveSummary = self.parseTraceFile(trcFilePath)
455
~\miniconda3\envs\gmsjupyter\lib\site-packages\gams\execution.py in run(self, gams_options, checkpoint, output, create_out_db, databases)
905 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), set the debug flag of the GamsWorkspace constructor to DebugLevel.KeepFiles or higher or define a working_directory to receive a listing file with more details", exitcode)
906 else:
--> 907 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), check " + self._workspace._working_directory + os.path.sep + tmp_opt.output + " for more details", exitcode)
908 self._p = None
909
GamsExceptionExecution: GAMS return code not 0 (2), check C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter7.lst for more details
To find out what went wrong, we can use %gams_lst, which tells us that we tried to redefine the domain list:
%gams_lst
GAMS 32.1.0 r75a5b5d Released Jul 31, 2020 WEX-WEI x86 64bit/MS Windows - 08/21/20 12:39:07 Page 9
G e n e r a l A l g e b r a i c M o d e l i n g S y s t e m
C o m p i l a t i o n
23 parameter p2(i,i,j);
**** $184,184
**** LINE 3 INPUT C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter7.gms
**** 184 Domain list redefined
**** 2 ERROR(S) 0 WARNING(S)
COMPILATION TIME = 0.000 SECONDS 3 MB 32.1.0 r75a5b5d WEX-WEI
USER: GAMS Evaluation License S200819/0001CO-GEN
GAMS Software GmbH, Frechen Office DCE839
**** FILE SUMMARY
Restart C:\Users\manns\Documents\GAMS\Jupyter\_gams_py_gcp0.g00
Input C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter7.gms
Output C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter7.lst
Save C:\Users\manns\Documents\GAMS\Jupyter\_gams_py_gcp6.g0?
**** USER ERROR(S) ENCOUNTERED
With a %gams_reset we can reset the GAMS database and can declare symbols with a different type and domain/dimension. All other things in the GAMS database are gone, too. So we need to redeclare the sets i and j, too.
%gams_reset
%gams set i,j; parameter p(i,j), p2(i,i,j);
Pushing Data from Python to GAMS
%gams_push transfers data from Python to GAMS. Supported data types for pushing data are lists, pandas.DataFrame and numpy arrays:
# Define Python lists with data
i = ['i1', 'i2', 'i3']
j = ['j1', 'j2']
p = [('i1', 'j1', 1.1), ('i1', 'j2', 2.2), ('i2', 'j1', 3.3), ('i2','j2', 4.4), ('i3','j1', 5.5), ('i3', 'j2', 6.6)]
%gams_push i j p
As mentioned above the execution of a %%gams cell or %gams and %gams_push line magic does not produce output. If one wants to verify that the data ended up in GAMS we can display the symbols in GAMS and output the corresponding part of the listing file:
%gams display i,j,p;
%gams_lst -e
E x e c u t i o n
---- 15 SET i
i1, i2, i3
---- 15 SET j
j1, j2
---- 15 PARAMETER p
j1 j2
i1 1.100 2.200
i2 3.300 4.400
i3 5.500 6.600
The next cell turns a Python list into a pandas.DataFrame, multiplies the value by 2 and displays the dataframe with IPythons’s display. We actually display the transformed p2 (via function gt_pivot2d found in the DataTransformation notebook run at the top of the notebook), so the table looks nicer. Next, we sends the pandas.DataFrame down to GAMS via the %gams_push command. Via the GAMS display and the output of the relevant part of the listing file we see that the %gams_push succeeded:
import pandas as pd
# turn the Python list p into a pandas.Dataframe p2 and send this down to GAMS
pp = pd.DataFrame(p)
# multiply the value by 2:
pp[2] = 2*pp[2]
# display a nicer version of the dataframe:
display(gt_pivot2d(pp))
%gams parameter pp(i,j)
%gams_push pp
%gams display pp;
%gams_lst -e
| j1 | j2 | |
|---|---|---|
| i1 | 2.2 | 4.4 |
| i2 | 6.6 | 8.8 |
| i3 | 11.0 | 13.2 |
E x e c u t i o n
---- 25 PARAMETER pp
j1 j2
i1 2.200 4.400
i2 6.600 8.800
i3 11.000 13.200
When using numpy arrays in order to push data into GAMS, the data is assumed to be dense. The corresponding sets are defined automatically from 1..n, 1..m, etc depending on the data that is pushed.
import numpy as np
data = [[[1.1,-1.1], [2.2,-2.2]], [[3.3,-3.3], [4.4,-4.4]], [[5.5,-5.5], [6.6,-6.6]]]
p3 = np.array(data)
%gams set i, j, k; parameter p3(i,j,k);
%gams_push p3
%gams display i,j,k,p3;
%gams_lst -e
E x e c u t i o n
---- 34 SET i
1, 2, 3
---- 34 SET j
1, 2
---- 34 SET k
1, 2
---- 34 PARAMETER p3 3-dim Matrix
1 2
1.1 1.100 -1.100
1.2 2.200 -2.200
2.1 3.300 -3.300
2.2 4.400 -4.400
3.1 5.500 -5.500
3.2 6.600 -6.600
Pulling Data from GAMS to Python
The line magic %gams_pull transfers data from GAMS to Python in different formats. Supported formats are lists (default), pandas.DataFrame and numpy arrays. The following example pulls the sets i, j, and parameter p3 from GAMS into lists. For multi-dimensional symbols the records become Python tuples. Currently, the renaming functionality %gams_pull gamsSym=pySymbol is not yet supported.
%gams_pull p3 i j
display(i,j,p3)
['1', '2', '3']
['1', '2']
[('1', '1', '1', 1.1),
('1', '1', '2', -1.1),
('1', '2', '1', 2.2),
('1', '2', '2', -2.2),
('2', '1', '1', 3.3),
('2', '1', '2', -3.3),
('2', '2', '1', 4.4),
('2', '2', '2', -4.4),
('3', '1', '1', 5.5),
('3', '1', '2', -5.5),
('3', '2', '1', 6.6),
('3', '2', '2', -6.6)]
The switch -d will populate pandas.DataFrames instead of lists with the GAMS data. The dataframes that are pushed into or pulled from GAMS have a very specific layout. There is a record index and the GAMS domains show up as columns in the dataframe. For parameters, there is an extra value column. For variables and equations we find extra columns level, marginal, lower, upper, and scale. The method head() used in the IPython display provides only the first 5 records of a pandas.DataFrame:
%gams variable x(i) / 1.L 1, 2.M 3 /;
%gams_pull -d i j p3 x
display(i, j, p3.head(), x)
| i | |
|---|---|
| 0 | 1 |
| 1 | 2 |
| 2 | 3 |
| j | |
|---|---|
| 0 | 1 |
| 1 | 2 |
| i | j | k | value | |
|---|---|---|---|---|
| 0 | 1 | 1 | 1 | 1.1 |
| 1 | 1 | 1 | 2 | -1.1 |
| 2 | 1 | 2 | 1 | 2.2 |
| 3 | 1 | 2 | 2 | -2.2 |
| 4 | 2 | 1 | 1 | 3.3 |
| i | level | marginal | lower | upper | scale | |
|---|---|---|---|---|---|---|
| 0 | 1 | 1.0 | 0.0 | -inf | inf | 1.0 |
| 1 | 2 | 0.0 | 3.0 | -inf | inf | 1.0 |
The data transformation functions available from DataTransformations.ipynb help to convert between this format and formats more suitable for display of other transformations in Python. The following lines give a quick overview of the transformation functionality:
%gams parameter r(i,j); r(i,j) = uniformInt(1,10);
%gams_pull -d r
display(r,gt_pivot2d(r),gt_from2dim(gt_pivot2d(r),['i','j','value']))
| i | j | value | |
|---|---|---|---|
| 0 | 1 | 1 | 2.0 |
| 1 | 1 | 2 | 9.0 |
| 2 | 2 | 1 | 6.0 |
| 3 | 2 | 2 | 4.0 |
| 4 | 3 | 1 | 3.0 |
| 5 | 3 | 2 | 3.0 |
| 1 | 2 | |
|---|---|---|
| 1 | 2.0 | 9.0 |
| 2 | 6.0 | 4.0 |
| 3 | 3.0 | 3.0 |
| i | j | value | |
|---|---|---|---|
| 0 | 1 | 1 | 2.0 |
| 1 | 1 | 2 | 9.0 |
| 2 | 2 | 1 | 6.0 |
| 3 | 2 | 2 | 4.0 |
| 4 | 3 | 1 | 3.0 |
| 5 | 3 | 2 | 3.0 |
The switch -n will populate numpy arrays instead of lists with the GAMS parameters. This format works with parameters only! The GAMS data will be dropped into a dense numpy array:
%gams parameter p4(i,j) / 1.1 1, 2.2 2 /;
%gams_pull -n p4
display(p4)
array([[1., 0.],
[0., 2.],
[0., 0.]])
Troubleshooting and Hints
- Paths to notebooks must not contain whitespaces. A notebook file itself (*.ipynb) can.
- The temporary files created in your working directory are useful for debugging, see below. The naming of the temporary files is not very sophisticated, so it can come to file nameing conflicts if you run two notebooks in the same directory at the same time (in different browser tabs). Create subdirectories and move the notebook into the subdirectories if you run into this problem.
- As soon as an error occurs while running GAMS code (the notebook exception is a GamsExecption), it can be useful to examine the listing file (*.lst) using %gams_lst. The path of the listing file can be found in the last line of the output of a failing cell.
%gams Parameter pt(i,j,l)
---------------------------------------------------------------------------
GamsExceptionExecution Traceback (most recent call last)
<ipython-input-25-4c01a25a5766> in <module>
----> 1 get_ipython().run_line_magic('gams', 'Parameter pt(i,j,l)')
~\miniconda3\envs\gmsjupyter\lib\site-packages\IPython\core\interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
2324 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
2325 with self.builtin_trap:
-> 2326 result = fn(*args, **kwargs)
2327 return result
2328
<decorator-gen-133> in gams(self, line, cell)
~\miniconda3\envs\gmsjupyter\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, *a, **k: f(*a, **k)
188
189 if callable(arg):
~\miniconda3\envs\gmsjupyter\lib\site-packages\gams_magic\gams_magic.py in gams(self, line, cell)
451 opt.traceopt = 3
452 with open(jobName + ".log", "w") as logFile:
--> 453 self.job.run(opt, checkpoint=self.cp, output=logFile)
454 solveSummary = self.parseTraceFile(trcFilePath)
455
~\miniconda3\envs\gmsjupyter\lib\site-packages\gams\execution.py in run(self, gams_options, checkpoint, output, create_out_db, databases)
905 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), set the debug flag of the GamsWorkspace constructor to DebugLevel.KeepFiles or higher or define a working_directory to receive a listing file with more details", exitcode)
906 else:
--> 907 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), check " + self._workspace._working_directory + os.path.sep + tmp_opt.output + " for more details", exitcode)
908 self._p = None
909
GamsExceptionExecution: GAMS return code not 0 (2), check C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter19.lst for more details
%gams_lst
GAMS 32.1.0 r75a5b5d Released Jul 31, 2020 WEX-WEI x86 64bit/MS Windows - 08/21/20 12:54:55 Page 28
G e n e r a l A l g e b r a i c M o d e l i n g S y s t e m
C o m p i l a t i o n
49 Parameter pt(i,j,l)
**** $120
**** LINE 3 INPUT C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter19.gms
**** 120 Unknown identifier entered as set
**** 1 ERROR(S) 0 WARNING(S)
COMPILATION TIME = 0.000 SECONDS 3 MB 32.1.0 r75a5b5d WEX-WEI
USER: GAMS Evaluation License S200819/0001CO-GEN
GAMS Software GmbH, Frechen Office DCE839
**** FILE SUMMARY
Restart C:\Users\manns\Documents\GAMS\Jupyter\_gams_py_gcp0.g00
Input C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter19.gms
Output C:\Users\manns\Documents\GAMS\Jupyter\gamsJupyter19.lst
Save C:\Users\manns\Documents\GAMS\Jupyter\_gams_py_gcp16.g0?
**** USER ERROR(S) ENCOUNTERED