Table of Contents
- Recommended Import
- Important Classes of the API
- How to use the API
- How to import packages/modules from the GAMS control API (transport1.py)
- How to choose the GAMS system (transport1.py)
- How to export data to GDX (transport_gdx.py)
- How to import data from GDX (transport_gdx.py)
- How to run a GamsJob from file (transport1.py)
- How to retrieve a solution from an output database (transport1.py)
- How to specify the solver using GamsOptions (transport1.py)
- How to run a job with a solver option file and capture its log output (transport1.py)
- How to use include files (transport2.py)
- How to read data from string and export to GDX (transport3.py)
- How to run a job using data from GDX (transport3.py)
- How to run a job using implicit database communication (transport3.py)
- How to define data using Python data structures (transport4.py)
- How to prepare a GamsDatabase from Python data structures (transport4.py)
- How to initialize a GamsCheckpoint by running a GamsJob (transport5.py)
- How to initialize a GamsJob from a GamsCheckpoint (transport5.py)
- How to run multiple GamsJobs in parallel using a GamsCheckpoint (transport6.py)
- How to create a GamsModelInstance from a GamsCheckpoint (transport7.py)
- How to modify a parameter of a GamsModelInstance using GamsModifier (transport7.py)
- How to modify a variable of a GamsModelInstance using GamsModifier (transport7.py)
- How to use a queue to solve multiple GamsModelInstances in parallel (transport8.py)
- How to fill a GamsDatabase by reading from MS Access (transport9.py)
- How to fill a GamsDatabase by reading from MS Excel (transport10.py)
- How to create and use a save/restart file (transport11.py)
GAMS control
is a sub-module of the Python API that allows for full control over the GAMS system (data, model instances, and solving). It can be used in conjunction with other Python API sub-modules (numpy
and transfer
) to boost performance when pushing/pulling data to/from a GAMS model. The gams.control
package provides objects to interact with the General Algebraic Modeling System (GAMS). Objects in this package allow convenient exchange of input data and model results (GamsDatabase
), help to create and run GAMS models (GamsJob
), that can be customized by GAMS options (GamsOptions
). Furthermore, it introduces a way to solve a sequence of closely related model instances in the most efficient way (GamsModelInstance
).
A GAMS program can include other source files (e.g. $include), load data from GDX files (e.g. $GDXIN or execute_load), and create PUT files. All these files can be specified with a (relative) path and therefore an anchor into the file system is required. The base class GamsWorkspace manages the anchor to the file system. If external file communication is not an issue in a particular Python application, temporary directories and files will be managed by objects in the namespace.
With the exception of GamsWorkspace the objects in the gams.control
package cannot be accessed across different threads unless the instance is locked. The classes themself are thread safe and multiple objects of the class can be used from different threads (see below for restrictions on solvers that are not thread safe within the GamsModelInstance class).
- Note
- If you use multiple instances of the GamsWorkspace in parallel, you should avoid using the same working directory. Otherwise you may end up with conflicting file names.
The GAMS control
Python API lacks support for the following GAMS components: Acronyms, support for GAMS compilation/execution errors (GamsJob.run just throws an exception), structured access to listing file, and proper support for solver options.
Currently only Cplex, Gurobi, and SoPlex fully utilize the power of solving GamsModelInstances. Some solvers will not even work in a multi-threaded application using GamsModelInstances. For some solvers this is unavoidable because the solver library is not thread safe (e.g. MINOS), other solvers are in principle thread safe but the GAMS link is not (e.g. SNOPT). Moreover, GamsModelInstances are not available for quadratic model types (QCP, MIQCP, RMIQCP).
Recommended Import
GAMS control
is available with the following import statement once the API has been installed:
>>> import gams
Other sub-modules must be imported with separate import statements.
Specifying a GAMS System Directory
There are several ways to specify which system directory should be used. On all platforms, the system directory can be specified in the GamsWorkspace constructor. If no system directory is specified by the user, The API tries to find one automatically:
- Windows: Try to find a system directory in the Windows registry.
- Linux: Try to find a system directory in the
PATH
first. If none was found, searchLD_LIBRARY_PATH
. - OS X: Try to find a system directory in the
PATH
first. If none was found, searchDYLD_LIBRARY_PATH
.
The environment variable PATH
can be set as follows on Linux and macOS:
- Note
- On Linux and macOS it is recommended to specify the
PATH
only instead of(DY)LD_LIBRARY_PATH
since this might cause problems loading the correct version of certain modules (e.g. gdx).
Important Classes of the API
This section provides a quick overview of some fundamental classes of the GAMS control
API. Their usage is demonstrated by an extensive set of examples in the How to use the API section.
- gams::control::workspace::GamsWorkspace Class
- gams::control::execution::GamsJob Class
- gams::control::database::GamsDatabase Class
- gams::control::options::GamsOptions Class
- gams::control::execution::GamsModelInstance Class
How to use the API
The GAMS distribution provides several examples that illustrate the usage of the API. [GAMSDIR]\api\python\examples\control
contains multiple examples dealing with the well-known transportation problem. In further course of this tutorial we discuss these examples step by step and introduce new elements of the API in detail.
We recommend to open the aforementioned files to gain a complete overview of the examples. Down below we explain the examples with the help of selected code snippets.
- How to import packages/modules from the GAMS control API (transport1.py)
- How to choose the GAMS system (transport1.py)
- How to export data to GDX (transport_gdx.py)
- How to import data from GDX (transport_gdx.py)
- How to run a GamsJob from file (transport1.py)
- How to retrieve a solution from an output database (transport1.py)
- How to specify the solver using GamsOptions (transport1.py)
- How to run a job with a solver option file and capture its log output (transport1.py)
- How to use include files (transport2.py)
- How to read data from string and export to GDX (transport3.py)
- How to run a job using data from GDX (transport3.py)
- How to run a job using implicit database communication (transport3.py)
- How to define data using Python data structures (transport4.py)
- How to prepare a GamsDatabase from Python data structures (transport4.py)
- How to initialize a GamsCheckpoint by running a GamsJob (transport5.py)
- How to initialize a GamsJob from a GamsCheckpoint (transport5.py)
- How to run multiple GamsJobs in parallel using a GamsCheckpoint (transport6.py)
- How to create a GamsModelInstance from a GamsCheckpoint (transport7.py)
- How to modify a parameter of a GamsModelInstance using GamsModifier (transport7.py)
- How to modify a variable of a GamsModelInstance using GamsModifier (transport7.py)
- How to use a queue to solve multiple GamsModelInstances in parallel (transport8.py)
- How to fill a GamsDatabase by reading from MS Access (transport9.py)
- How to fill a GamsDatabase by reading from MS Excel (transport10.py)
- How to create and use a save/restart file (transport11.py)
How to import packages/modules from the GAMS control API (transport1.py)
Before we can start using the GAMS control
API, it needs to be installed by following the instructions from the Getting Started section. Afterwards we can use the API by importing the GamsWorkspace
class like this:
Conventional Python packages/modules can by imported like that:
How to choose the GAMS system (transport1.py)
By default the GAMS system is determined automatically. In case of having multiple GAMS systems on your machine, the desired system can be specified via an additional argument when the workspace is created. If we type python transport1.py
C:/GAMS/42 we use GAMS 42
to run transport1.py
even if our default GAMS system might be a different one. This is managed by the following code:
How to export data to GDX (transport_gdx.py)
Although the GAMS control
Python API offers much more than exchanging data between Python and GDX, a common use case is the export and import of GDX files. The central class for this purpose is GamsDatabase. We assume that the data to be exported is available in Python data structures.
Different GAMS symbols are represented using different Python data structures. The data for the GAMS sets is represented using Python lists of strings (e.g. plants
and markets
). On the other hand, GAMS parameters are represented by Python dictionaries (e.g. capacity
and demand
). Note that the representation of the two dimensional parameter distance
uses Python tuples for storing the keys. The choice of data structures can also be different, but the used structures in this example fit well for representing GAMS data with standard Python data structures.
A new GamsDatabase instance can be created using GamsWorkspace.add_database.
We start adding GAMS sets using the method GamsDatabase.add_set which takes the name and the dimension as arguments. The third argument is an optional explanatory text. A for-loop iterates through plants
and adds new records to the recently created GamsSet instance i
using GamsSet.add_record.
GamsParameter instances can be added by using the method GamsDatabase.add_parameter. It has the same signature as GamsDatabase.add_set. Anyhow, in this example we use GamsDatabase.add_parameter_dc instead which takes a list of GamsSet instances instead of the dimension for creating a parameter with domain information.
As soon as all data is prepared in the GamsDatabase, the method GamsDatabase.export can be used to create a GDX file.
How to import data from GDX (transport_gdx.py)
Data can be imported from a GDX file using GamsWorkspace.add_database_from_gdx. The method takes a path to a GDX file and creates a GamsDatabase instance.
Reading the data from the GamsSet i
into a list can be done as follows:
A Python list is created using list comprehensions. i
is retrieved by querying the GamsDatabase db2
. The returned GamsSet object can be iterated using a for-loop to access the records of the set. Each record is of type GamsSetRecord and can be asked for its keys.
You can do the same for GamsParameters. Instead of creating a Python list, we want to have the data in the form of a Python dictionary. GamsParameterRecords can not only be asked for their keys, but also for their value. The following code snippet shows how to read the one dimensional parameter a
into a Python dictionary using dict comprehensions.
For multi dimensional symbols, we choose the Python dictionary keys to be tuples instead of string. We access the keys as usual, but do not address a specific key. Instead, we take the whole list of keys and turn it into a tuple.
Scalars can be read into a Python identifier by accessing the value of the first and only record.
How to run a GamsJob from file (transport1.py)
At first we create our workspace using ws = GamsWorkspace()
. Afterwards we load the trnsport
model from the GAMS model library which puts the corresponding gms file in our working directory. Note that you can create a GamsJob with any other gms file you might have created on your own as long as it is located in the current working directory. Then the GamsJob job
can be defined using the add_job_from_file method and afterwards we run the job.
How to retrieve a solution from an output database (transport1.py)
The following lines create the solution output and illustrate the usage of the GamsJob.out_db property to get access to the GamsDatabase created by the run method. To retrieve the content of variable x
we use squared brackets that internally call the get_symbol method.
Note that instead of using the squared brackets we could also use
How to specify the solver using GamsOptions (transport1.py)
The solver can be specified via the GamsOptions class and the GamsWorkspace.add_options method. The GamsOptions.all_model_types property sets xpress as default solver for all model types that can be handled by the solver. Then we run our GamsJob job
with the new GamsOption.
How to run a job with a solver option file and capture its log output (transport1.py)
At first we create the file xpress.opt
with content algorithm=barrier
which will be used as solver option file and is stored in the current working directory. Afterwards we use a GamsOption just like in the preceding example and set GamsOption.optfile property to 1 to tell the solver to look for a solver option file. In addition, we specify the argument output
in order to write the log of the GamsJob into the file transport1_xpress.log
.
Instead of writing the log output to a file, any object that provides the functions write
and flush
can be used. In order to write the log directly to stdout
, we can use the following code:
How to use include files (transport2.py)
In this example, as in many succeeding, the data text and the model text are separated into two different strings. Note that these strings accessed via GAMS_DATA
and GAMS_MODEL
are using GAMS syntax. At first we write an include file tdata.gms
that contains the data but not the model text and save it in our current working directory.
Afterwards we create a GamsJob using the GamsWorkspace.add_job_from_string method. GamsOptions.defines is used like the 'double dash' GAMS parameters, i.e. it corresponds to --incname=tdata
on the command line where incname
is used as name for the include file in GAMS_MODEL
as shown below.
The string GAMS_MODEL
contains the following lines to read in the data.
How to read data from string and export to GDX (transport3.py)
We read the data from the string GAMS_DATA
. Note that this contains no model but only data definition in GAMS syntax. By running the corresponding GamsJob a GamsDatabase is created that is available via the GamsJob.out_db property. We can use the GamsDatabase.export method to write the content of this database to a GDX file tdata.gdx
in the current working directory.
How to run a job using data from GDX (transport3.py)
This works quite similar to the usage of an include file explained in How to use include files (transport2.py).
Note that there are some minor changes in GAMS_MODEL
compared to preceding examples due to the usage of a GDX instead of an include file.
How to run a job using implicit database communication (transport3.py)
This example does basically the same as the two preceding examples together. We create two GamsJobs job_data
and job_model
where the first one contains only the data and the second one contains only the model without data. After running job_data
the corresponding out_db
can be read in directly just like a GDX file. Note that the database needs to be passed to the GamsJob.run method as additional argument.
How to define data using Python data structures (transport4.py)
We use Python lists to define the sets and Python dictionaries for the parameter definition.
How to prepare a GamsDatabase from Python data structures (transport4.py)
At first we create an empty GamsDatabase db using the GamsWorkspace.add_database method. Afterwards we prepare the database. To add a set to the database we use the GamsSet class and the GamsDatabase.add_set method with arguments describing the identifier, dimension and explanatory text. To add the records to the database we iterate over the elements of our Python data structure and add them by using the GamsSet.add_record method.
For parameters the procedure is pretty much the same. Note that the table that specifies the distances in GAMS can be treated as parameter with dimension 2.
The GamsJob can be run like explained in the preceding example How to run a job using implicit database communication (transport3.py).
How to initialize a GamsCheckpoint by running a GamsJob (transport5.py)
The following lines of code conduct several operations. While the first line simply creates a GamsCheckpoint, the second one uses the GamsWorkspace.add_job_from_string method to create a GamsJob containing the model text and data but no solve statement. Afterwards the run method gets the GamsCheckpoint as argument. That means the GamsCheckpoint cp
captures the state of the GamsJob.
How to initialize a GamsJob from a GamsCheckpoint (transport5.py)
Note that the string GAMS_MODEL
contains the entire model and data definition plus an additional demand multiplier and scalars for model and solve status but no solve statement:
In transport5.py we create a list with eight different values for this demand multiplier.
For each entry of that list we create a GamsJob using the GamsWorkspace.add_job_from_string method. Besides another string which resets the demand multiplier bmult
, specifies the solve statement and assigns values to the scalars ms
and ss
we pass the checkpoint cp
as additional argument. This results in a GamsJob combined from the checkpoint plus the content provided by the string.
We run the GamsJob and print some interesting data from the out_db.
NOTE: Some of the demand multipliers cause infeasibility. Nevertheless, GAMS keeps the incumbent objective function value. Therefore the model status and the solve status provide important information for a correct solution interpretation.
How to run multiple GamsJobs in parallel using a GamsCheckpoint (transport6.py)
With the exception of GamsWorkspace the objects in the gams.control
package cannot be accessed across different threads unless the instance is locked. The classes themselves are thread safe and multiple objects of the class can be used from different threads (see below for restrictions on solvers that are not thread safe within the GamsModelInstance
class).
- Note
- If you use multiple instances of the
GamsWorkspace
in parallel, you should avoid using the same working directory. Otherwise you may end up with conflicting file names.
Currently only Cplex, Gurobi, and SoPlex fully utilize the power of solving GamsModelInstances. Some solvers will not even work in a multi-threaded application using GamsModelInstances. For some solvers this is unavoidable because the solver library is not thread safe (e.g. MINOS), other solvers are in principle thread safe but the GAMS link is not (e.g. SNOPT). Moreover, GamsModelInstances are not available for quadratic model types (QCP, MIQCP, RMIQCP).
This example illustrates how to run the jobs known from transport5.py in parallel. We initialize the GamsCheckpoint cp
and introduce a demand multiplier as we did before:
Furthermore, we introduce a lock object io_lock
that will be used to avoid mixed up output from the parallel jobs. We create one scenario for each entry of bmultlist
and cause a thread to begin execution.
In function run_scenario
a GamsJob is created and run just like in the preceding example of transport5.py. The output section is also the same except for the fact that it is 'locked' by the object io_lock
which means that the output section cannot be executed simultaneously for multiple demand multipliers.
While the output in transport5.py is strictly ordered subject to the order of the elements of bmult
in transport6.py the output blocks might change their order but the blocks describing one scenario are still appearing together due to the io_lock
object.
How to create a GamsModelInstance from a GamsCheckpoint (transport7.py)
In transport7.py the usage of GamsModelInstance is demonstrated.
At first checkpoint cp
is created as in the preceding examples. Note that the GamsJob job
again contains no solve statement and the demand multiplier is already included with default value 1. We create the GamsModelInstance mi
using the GamsCheckpoint.add_modelinstance method.
How to modify a parameter of a GamsModelInstance using GamsModifier (transport7.py)
A GamsModelInstance uses a sync_db to maintain the data. We define bmult
as GamsParameter using the GamsDatabase.add_parameter method and specify gurobi as solver. Afterwards the GamsModelInstance is instantiated with 3 arguments, the solve statement, GamsModifier bmult
and GamsOptions opt
. The GamsModifier means that bmult
is modifiable while all other parameters, variables and equations of GamsModelInstance mi
stay unchanged. We use the GamsParameter.add_record method to assign a value to bmult
. That value can be varied afterwards using the GamsParameter.first_record method to reproduce our well-known example with different demand multipliers.
How to modify a variable of a GamsModelInstance using GamsModifier (transport7.py)
We create a GamsModelInstance using the GamsCheckpoint.add_modelinstance method. Afterwards we define x
as GamsVariable and a GamsParameter xup
that will be used as upper bound for x
. At the following instantiate method GamsModifier has 3 arguments. The first one says that x
is modifieable, the second determines which part of the variable (lower bound, upper bound or level) can be modified and the third specifies the GamsParameter that holds the new value, in this case xup
.
In the following loops we set the upper bound of one link of the network to zero, which means that no transportation between the corresponding plant and market is possible, and solve the modified transportation problem.
How to use a queue to solve multiple GamsModelInstances in parallel (transport8.py)
We initialize a GamsCheckpoint cp
from a GamsJob. Then we define a list that represents the different values of the demand multiplier. That list will be used like a queue where we extract the last element first. The objects list_lock
and io_lock
are used later to avoid multiple reading of one demand multiplier and messed up output. Then we call function scen_solve
multiple times in parallel. The number of parallel calls is specified by nr_workers
.
In function scen_solve
we create and instantiate a GamsModelInstance as in the preceding examples and make parameter bmult
modifiable. Note that we choose cplex
as solver because it is thread safe (gurobi would also be possible).
We have two critical sections that are locked by the objects list_lock
and io_lock
. Note that the pop method removes and returns the last element from the list and deletes it. Once the list is empty the loop terminates.
How to fill a GamsDatabase by reading from MS Access (transport9.py)
This example illustrates how to import data from Microsoft Access to a GamsDatabase. There are a few prerequisites required to run transport9.py successfully.
- We import
pyodbc
. - Note that an architecture mismatch might cause problems. The bitness of your MS Access, Python, pyodbc and GAMS should be identical (64 bit).
We call a function read_data_from_access
that finally returns a GamsDatabase as shown below.
The function read_from_access
begins with the creation of an empty database. Afterwards we set up a connection to the MS Access database transport.accdb
which can be found in [GAMSDIR]\apifiles\Data
. To finally read in GAMS sets and parameters we call the functions read_set
and read_parameter
that are explained down below.
The function read_set
adds a set to the GamsDatabase that is filled with the data from the MS Access file afterwards. The function read_parameter
works quite similar.
Once we read in all the data we can create a GamsJob from the GamsDatabase and run it as usual.
How to fill a GamsDatabase by reading from MS Excel (transport10.py)
This example illustrates how to read data from Excel, or to be more specific, from [GAMSDIR]\apifiles\Data\transport.xlsx
.
At first you have to download the openpyxl
package:
pip install openpyxl
Now you should be able to run transport10.py.
In transport10.py the model is given as string without data like in many examples before and the Excel file transport.xlsx
is located at [GAMSDIR]\apifiles\Data
. At first we define the workbook to read from and the different sheet names. To ensure to have the same number of markets and plants in all spreadsheets, we conduct a little test that checks for the number of rows and columns. Our workspace is only created if this test yields no errors.
Now we can create a GamsDatabase and read in the data contained in the different worksheets. We iterate over the columns and read in the set names and the corresponding parameter values.
Note that we can name sets and parameters just like in the database but we don't have to. Now we can run our GamsJob as usual.
How to create and use a save/restart file (transport11.py)
In transport11.py we demonstrate how to create and use a save/restart file. Usually such a file should be supplied by an application provider but in this example we create one for demonstration purpose. Note that the restart is launched from a GamsCheckpoint.
We create a directory tmp
with internal identifier w_dir
in the directory we are currently in. This file will be used as working directory later. From the main function we call the function create_save_restart
giving it directory tmp
and the desired name for the save/restart file (tbase
) as arguments.
In function create_save_restart
we create a workspace with the given working directory (w_dir
refers to tmp
). Then we create a GamsJob from a string. Note that the string given via get_base_model_text
contains the basic definitions of sets without giving them a content (that is what $onempty
is used for). Afterwards we specify a GamsOption to only compile the job but do not execute it. Then we create a checkpoint cp
that is initialized by the following run of the GamsJob and stored in the file given as argument to the function, in our case tbase
. This becomes possible because the add_checkpoint method accepts identifiers as well as file names as argument.
So what you should keep in mind before we return to further explanations of the main function is, that the file tbase
is now in the current working directory and contains a checkpoint that will work exactly like a restart file.
In the main function we define some data using Python data structures as we already did in [transport4.py] (How to define data using Python data structures (transport4.py)) before we create the GamsWorkspace and a GamsDatabase.
Afterwards we set up the GamsDatabase like we already did in [transport4.py] (How to prepare a GamsDatabase from Python data structures (transport4.py)). Once this is done we run a GamsJob using this data plus the checkpoint stored in file tbase
.
Note that the string from which we create job
is different to the one used to prepare the checkpoint stored in tbase
and is only responsible for reading in the data from the GamsDatabase correctly. The entire model definition is delivered by the checkpoint cp_base
which is equal to the one we saved in tbase
.