Motivation

GAMS uses relational data tables as a basic data structure. With these, GAMS code for parallel assignment and equation definition is compact, elegant, and efficient. However, traditional data structures (arrays, lists, dictionaries, trees, graphs, ...) are not natively or easily available in GAMS. Though it is possible to represent such data structures in GAMS, the GAMS code working with such structures can easily become unwieldy, obfuscating, or inefficient. Also, in the past it was not easy to connect libraries from other systems for special algorithms (e.g. graph algorithms, matrix operations, ...) to GAMS without some data programming knowledge and a deep understanding of internal representation of GAMS data (e.g. the GDX API).

The Embedded Code Facility addresses this need and extends the connectivity of GAMS to other programming languages. It allows the use of external code (e.g. Python) during compile and execution time. GAMS symbols are shared with the external code, so no communication via disk is necessary. It utilizes an API (this API will be published in one of the next releases so users can extend the embedded code to other languages/systems) and additional source code for common tasks so that the user can concentrate on the task at hand and not the mechanics of moving data in and out of GAMS.

Concept

As pointed out in section Motivation, the main idea of the Embedded Code Facility is to enable the use of external code in GAMS and give this code direct in-memory access to the GAMS database (or better: to GAMS symbols, namely sets, parameters, variables, and equations). This can be done by defining sections in the GAMS code which contain no GAMS code, but code written in a defined external code. These sections can be used at both GAMS compile and execution time (compare section GAMS Compile Time and Execution Time Phase). Details about how to do this can be found in the Syntax section.

Note: It is planned to extend this feature to different programming languages. However, at the moment only Python is supported.

Also, the system provides some help to develop and debug the external code independent of GAMS first. More about this topic can be found in section Troubleshooting Embedded Python Code.

The communication of the data between the GAMS part of the program and the embedded code part was inspired by the existing interface to GDX in many ways. So the system allows to access specific symbol records by both labels and label indices. Also, it is possible to decide if data changed in the embedded code should replace the GAMS data or get merged with it and just like loading data from GDX, one can decide if data from embedded code should change the GAMS database filtered or domain checked. How to do these things is explained in detail in section Python.

Simple Example

In a very first example we look into some Python code that helps to split some label names that are already present in GAMS. We do this at compile time in order to read the broken up pieces as individual sets (country and city) into GAMS plus some mapping sets (mccCountry and mccCity) between the original labels and the new labels. The compile-time embedded code starts with a $onEmbeddedCode followed by the type of code to expect (Python:). Python is currently the only code that works. GAMS plans to add other (even compiled) languages in the future. The lines between $onEmbeddedCode and $offEmbeddedCode is Python code. We do not want to go into Python details, but the first few lines initialize some empty Python list objects (mccCountry and mccCity) as well as some empty Python set objects. In the Python for loop that follows we iterate over all individual labels of the GAMS set cc. Python gets access to the GAMS set cc via the member function get of the implicitly defined Python object gams. get returns an object that is iterable and can be used in a Python for loop. The type of the records one gets depend on the dimensionality and type of the GAMS symbol. In the loop body we use the Python split function to extract the first (r[0] is country) and second (r[1] is city) part of the label. The three strings cc, r[0], r[1] are used to build up the Python list objects that store the information for the maps mccCountry and mccCity and the Python set objects that store the labels for the new sets country and city. The Python set has the advantage to store a label just once even if we add it multiple times. Python prepares to send items back to GAMS via the gams member function set that can deal with both Python list and Python set objects. The command $offEmbeddedCode is followed by a list (without separating commas) of GAMS symbols that instructs the GAMS compiler to read these symbols back.

Note that GAMS syntax is case insensitive while Python is case sensitive. Hence, the strings that represent the GAMS symbol names in the Python code can have any casing (e.g. gams.get("cc") or gams.get("CC")), while the corresponding Python objects need to have consistent casing throughout the Python code.

The following code presents the entire embeddedSplit example from the GAMS Data Utilities Library:

Set cc      / "France - Paris", "France - Lille", "France - Toulouse"
              "Spain - Madrid", "Spain - Cordoba", "Spain - Seville", "Spain - Bilbao"
              "USA - Washington DC", "USA - Houston", "USA - New York",
              "Germany - Berlin", "Germany - Munich", "Germany - Bonn" /
    country 
    city
    mccCountry(cc,country<)  Mapping between country and related elements in set cc
    mccCity(cc,city<)        Mapping between city and related elements in set cc;

$onEmbeddedCode Python:
  mccCountry = []
  mccCity = []
  for cc in gams.get("cc"):
    r = str.split(cc, " - ", 1)
    mccCountry.append((cc,r[0]))
    mccCity.append((cc,r[1]))
  gams.set("mccCountry",mccCountry)
  gams.set("mccCity",mccCity)
$offEmbeddedCode mccCountry mccCity

Option mccCountry:0:0:1, mccCity:0:0:1;
Display country, city, mccCountry ,mccCity;

The display in the listing file looks as follows:

----     25 SET country
Spain  ,    USA    ,    Germany,    France 

----     25 SET city
Berlin       ,    Bilbao       ,    Cordoba      ,    Madrid
New York     ,    Washington DC,    Paris        ,    Houston
Munich       ,    Lille        ,    Seville      ,    Bonn
Toulouse

----     25 SET mccCountry
France - Paris     .France 
France - Lille     .France 
France - Toulouse  .France 
Spain - Madrid     .Spain
Spain - Cordoba    .Spain
Spain - Seville    .Spain
Spain - Bilbao     .Spain
USA - Washington DC.USA
USA - Houston      .USA
USA - New York     .USA
Germany - Berlin   .Germany
Germany - Munich   .Germany
Germany - Bonn     .Germany

----     25 SET mccCity
France - Paris     .Paris
France - Lille     .Lille
France - Toulouse  .Toulouse
Spain - Madrid     .Madrid
Spain - Cordoba    .Cordoba
Spain - Seville    .Seville
Spain - Bilbao     .Bilbao
USA - Washington DC.Washington DC
USA - Houston      .Houston
USA - New York     .New York
Germany - Berlin   .Berlin
Germany - Munich   .Munich
Germany - Bonn     .Bonn

The second example demonstrates the use of embedded code at execution time. The syntax for the execution time embedded code in this example is identical to the compile time variant with the exception of the keywords that start and end the embedded code section: embeddedCode and endEmbeddedCode. It is important to understand that the execution of the code happens at GAMS execution time, so e.g. no new labels can be produced and send back to GAMS. In this example we use some Python code to generate a random permutation of set elements of set i and store this in a two dimensional set p. In this example we do not use a loop to iterate through the elements of a GAMS system but make use of the fact that the Python object returned by gams.get("i") is iterable and can in its entirety be stored in the Python list with name i with the short and powerful command i = list(gams.get("i")). The permutation of elements in list p which is a copy of list i is created by the Python statement random.shuffle(p). The following code presents the entire example:

Set i /i1*i10/
    p(i,i) "permutation";

embeddedCode Python:
  import random
  i = list(gams.get("i"))
  p = list(i)
  random.shuffle(p)
  for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
  gams.set("p", p)
endEmbeddedCode p

option p:0:0:1; 
display p;

The display in the listing file looks as follows:

----     11 SET p  permutation
i1 .i1 
i2 .i7 
i3 .i5 
i4 .i2 
i5 .i10
i6 .i6 
i7 .i9 
i8 .i4 
i9 .i8 
i10.i3

Syntax

This section explains the GAMS functions/keywords which were introduced to enable the Embedded Code Facility. The first subsection deals with the syntax for compile time, the second with the syntax for execution time (compare section GAMS Compile Time and Execution Time Phase).

Compile Time

There are three dollar control options to start an embedded code section for Python at compile time:

$onEmbeddedCode  Python: [arguments]
$onEmbeddedCodeS Python: [arguments]
$onEmbeddedCodeV Python: [arguments]

Lines following one of the above statements are passed on to a Python interpreter until this dollar control option, which ends the embedded code section at compile time, is hit:

$offEmbeddedCode {symbol[<[=]embSymbol[.dimX]]}

These dollar control options are explained here in more detail. An example which uses them can be seen above.

Note

The optional arguments from the $onEmbeddedCode[S|V] statement can be accessed as gams.arguments in the Python code.
The optional output symbols from the $offEmbeddedCode statement need to be set using the function gams.set in the Python code.
More about the specific GAMS Python syntax can be found below.

Execution Time

At execution time an embedded code section is started with one of these statements:

embeddedCode  Python: [arguments]
embeddedCodeS Python: [arguments]
embeddedCodeV Python: [arguments]

Similar to the compile time alternatives $onEmbeddedCode[S|V], the first two variants are synonyms which allow parameter substitution in the Python code that follows, while the last variant does not allow this but passes the code verbatim to the Python interpreter. The optional arguments in all three variants can be accessed in the Python code that follows as gams.arguments (see section Python for more details). Lines following one of the above statements are passed on to the Python interpreter until one of the following two statements, which end the embedded code section at execution time, is hit:

endEmbeddedCode   {output symbols}
pauseEmbeddedCode {output symbols}

Both statements end the embedded code section and switch back to GAMS syntax in the following lines. Also, both statements can be followed by a GAMS symbol or a list of GAMS symbols which would get updated in the GAMS database after the Python code got executed. If output symbols are specified, they need to be set in the embedded code before using the function gams.set (see section Python for more details). And by default, they really behave exactly the same. However, when the command line parameter freeEmbeddedPython is set to 1, there is a difference between the two statements: Then, as the names suggest, endEmbeddedCode ends the embedded code section, while pauseEmbeddedCode pauses it. If it ends, it cannot be continued at a later point, because some internal resources get freed, it could just be re-initiated (e.g. by another embeddedCode statement). If the embedded code section got paused, it could also be continued at a later stage in the GAMS program, which means that no reinitialization is needed and Python symbols which were defined before the pause are still available when continuing. With the default of freeEmbeddedPython set to 0 endEmbeddedCode behaves like pauseEmbeddedCode (and pauseEmbeddedCode always behaves the same, so it is independent of the setting of freeEmbeddedPython). Note that since GAMS does not stay in memory under the default settings that continuing any Python code after GAMS executes a solve causes the code to fail with the following error message:

    Error executing "continueEmbeddedCode" section
--- (Hint: "Solve" with SolveLink=0 frees previously initialized embedded libraries):
    Error at line 71: No embedded library initialized

This happens because under default conditions GAMS passes out of memory when the solver starts up. This causes the loss of the state of the embedded code environment. Whether GAMS is retained in memory is controlled by SolveLink (default=0). By using a different value (e.g. SolveLink=2 (solveLink.callModule%) or SolveLink=5 (solveLink.loadLibrary%)), GAMS stays in memory and a paused embedded code environment can be continued after the solve statement has carried out.

To continue a previously paused embedded code section one of the following statements is used:

continueEmbeddedCode  [handle]: [arguments]
continueEmbeddedCodeS [handle]: [arguments]
continueEmbeddedCodeV [handle]: [arguments]

As seen before, the first two variants are synonyms which allow parameter substitution in the embedded code that follows, while the last variant does not allow this but passes the code verbatim to the interpreter. Also as seen above when discussing EmbeddedCode[S|V], the optional arguments in all three variants can be accessed in the embedded code that follows as gams.arguments (see section Python for more details). New in these statements is the optional handle. If omitted, the last code section that was paused will be continued. However, sometimes one might need to maintain different embedded code sections active in parallel and independent of each other. In order to use this facility, it is required to set PyMultInst to 1. Note that this setting might cause problems when using third party modules and packages (e.g. numpy or modules that make use of it) and might also impact the performance. There is a new function to store a handle of the last embedded code section that was executed which could then later be used to continue a specific paused code section:

handle = embeddedHandle;

An example of how this can be used can be seen in the GAMS Datalib model [embeddedMultiInstance]. A simplified use of just embeddedCode and endEmbeddedCode can also be seen in a simple example above.

Note: Keeping the Python environment alive with pauseEmbeddedCode and continueEmbeddedCode[S|V] or endEmbeddedCode and freeEmbeddedPython=0 can be particularly useful, when there is an expensive initialization (e.g. to import other code parts) which can be done only once but further execution needs to be done many times (e.g. inside a loop).

Python

GAMS and Python complement each other in many different ways. GAMS has a compact but readable syntax for data assignment and equation definition statements. Python is a great scripting language to do more traditional type programming especially string manipulation which is completely missing in GAMS. Moreover, the vast number of packages help solving scientific problems outside the scope of GAMS. In order to open this world to our customers in an easy way, GAMS comes with a Python 3 installation on the major platforms Windows, macOS, and Linux. By default, this installation is used in the Embedded Code Facility for Python and is ready to be used with the GAMS Python API. The Python installation is located in [GAMS directory]/GMSPython. In contrast to earlier versions this installation comes without the Python package manager pip and therefore is not easily extendable. GMSPython comes with a selection of third party packages and modules that are either required by the APIs of the GAMS Python API collection or that are particularly helpful from within embedded Python code. Those packages are located in [GAMS directory]/apifiles/Python/thirdparty. Running the GMSPython interpreter like follows will display all available modules:

[GAMS directory]/GMSPython/python -c"help('modules')"

The list of redistributed third party packages is subject to constant changes and it is not recommended to rely on their availability. If you need packages that do not come with GMSPython, we recommend that you install your own version of Python and follow a few steps to connect GAMS with your Python installation. It is also possible to install pip for GMSPython using get-pip. While any Python installation should work in combination with GAMS we can recommend the Anaconda or Miniconda Python distributions. These Python distributions work with environments which allow you to have many different Python installations at the same time that don't get in each other's way.

The Python class ECGamsDatabase is the interface between GAMS and Python. An instance of this class is automatically created when an embedded code section is entered and can be accessed using the identifier gams. The following methods can be used in order to interact with GAMS:

gams.get(symbolName, keyType=KeyType.STRING, keyFormat=KeyFormat.AUTO, valueFormat=ValueFormat.AUTO, recordFormat=RecordFormat.AUTO)

This method retrieves an iterable object representing the symbol identified with symbolName. Typically there are two possibilities to access the records. Iterating using e.g. a for loop provides access to the individual records. By calling list() on the iterable object, a list containing all the data is created. Several optional parameters can be used in order to modify the format of the retrieved data:

keyType: Determines the data type of the keys. It can be either KeyType.STRING (labels) or KeyType.INT (label indexes). The default setting is KeyType.STRING.

keyFormat: Specifies the representation of the keys. Possible values are as follows:

KeyFormat.TUPLE: Encapsulate keys in a tuple

KeyFormat.FLAT: No encapsulation

KeyFormat.SKIP: Keys are skipped and do not appear in the retrieved data

KeyFormat.AUTO (default): Depending on the dimension of the GAMS symbol, a default format is applied:

Zero dimensional/scalar: KeyFormat.SKIP

One dimensional: KeyFormat.FLAT

Multi dimensional: KeyFormat.TUPLE

valueFormat: Specifies the representation of the values. Possible values are as follows:

ValueFormat.TUPLE: Encapsulate values in a tuple

ValueFormat.FLAT: No encapsulation

ValueFormat.SKIP: Values are skipped and do not appear in the retrieved data

ValueFormat.AUTO (default): Depending on the type of the GAMS symbol, a default formats is applied:

Set: ValueFormat.SKIP

Parameter: ValueFormat.FLAT

Variable/Equation: ValueFormat.TUPLE

recordFormat Specifies the encapsulation of records into tuples. Possible values are as follows:

RecordFormat.TUPLE: Encapsulates every record in a tuple

RecordFormat.FLAT: No encapsulation. Throws an exception if it can not be applied. It is guaranteed that the length of a retrieved Python list is equal to the number of records of the corresponding GAMS symbol. This principle leads to an incompatibility of RecordFormat.FLAT whenever a record consists of more than one item (e.g. multi dimensional symbols, variables and equations which have five numeric values).

RecordFormat.AUTO (default): Depending on the number of items that represent a record, a default format is applied. If possible this is always RecordFormat.FLAT.

GAMS special values NA, INF, and -INF will be mapped to IEEE special values float('nan'), float('inf'), and float('-inf'). GAMS special value EPS will be either mapped to 0 or to the small numeric value 4.94066E-324 depending on the setting of flag gams.epsAsZero.

The following Python code shows some examples of gams.get and illustrates the use of different formats:
Set i / i1 text 1, i2 text 2 /
    j / j1*j2 /

Scalar p0 /3.14/;
Parameter p1(i)   / #i 3.14 /
          p2(i,j) / i1.#j 3.14 /
Variable v0       / fx 3.14 /;
equation e1(i)    / #i.fx 3.14 /
         e2(i,j)  / i1.#j.fx 3.14 /;

$onEmbeddedCode Python:
  # scalar parameter
  l = list(gams.get('p0'))
  assert l == [3.14], "error"
  l = list(gams.get('p0', recordFormat=RecordFormat.TUPLE))
  assert l == [(3.14,)], "error"

  # one dimensional parameters:
  l = list(gams.get('p1'))
  assert l == [("i1", 3.14), ("i2", 3.14)], "error"
  l = list(gams.get('p1', keyFormat=KeyFormat.TUPLE))
  assert l == [(("i1",), 3.14), (("i2",), 3.14)], "error"
  l = list(gams.get('p1', valueFormat=ValueFormat.TUPLE))
  assert l == [("i1", (3.14,)), ("i2", (3.14,))], "error"
  l = list(gams.get('p1', keyFormat=KeyFormat.TUPLE, valueFormat=ValueFormat.TUPLE))
  assert l == [(("i1",), (3.14,)), (("i2",), (3.14,))], "error"

  # two dimensional parameter:
  l = list(gams.get('p2'))
  assert l == [(('i1', 'j1'), 3.14), (('i1', 'j2'), 3.14)], "error"
  l = list(gams.get('p2', keyFormat=KeyFormat.FLAT))
  assert l == [('i1', 'j1', 3.14), ('i1', 'j2', 3.14)], "error"

  # one dimensional sets:
  l = list(gams.get('i'))
  assert l == ['i1', 'i2'], "error"
  l = list(gams.get('i',valueFormat=ValueFormat.FLAT))
  assert l == [('i1', "text 1"), ('i2', "text 2")], "error"

  # scalar variables/equations
  l = list(gams.get('v0'))
  assert l == [(3.14, 0, 3.14, 3.14, 1)], "error"

  # one dimensional variables/equations:
  l = list(gams.get('e1'))
  assert l == [("i1", (3.14, 0, 3.14, 3.14, 1)), ("i2", (3.14, 0, 3.14, 3.14, 1))], "error"
  l = list(gams.get('e1', valueFormat=ValueFormat.FLAT))
  assert l == [("i1", 3.14, 0, 3.14, 3.14, 1), ("i2", 3.14, 0, 3.14, 3.14, 1)], "error"
  l = list(gams.get('e1', keyFormat=KeyFormat.TUPLE))
  assert l == [(("i1",), (3.14, 0, 3.14, 3.14, 1)), (("i2",), (3.14, 0, 3.14, 3.14, 1))], "error"

  # two dimensional variables/equations:
  l = list(gams.get('e2'))
  assert l == [(("i1", "j1"), (3.14, 0, 3.14, 3.14, 1)), (("i1", "j2"), (3.14, 0, 3.14, 3.14, 1))], "error"
  l = list(gams.get('e2', keyFormat=KeyFormat.FLAT, valueFormat=ValueFormat.FLAT))
  assert l == [("i1", "j1", 3.14, 0, 3.14, 3.14, 1), ("i1", "j2", 3.14, 0, 3.14, 3.14, 1)], "error"

  # using label indexes instead of labels
  l = list(gams.get('p1', keyType=KeyType.INT))
  assert l == [(1, 3.14), (2, 3.14)], "error"
  l = list(gams.get('i', keyFormat=KeyFormat.TUPLE, valueFormat=ValueFormat.TUPLE, keyType=KeyType.INT))
  assert l == [((1,), ("text 1",)), ((2,), ("text 2",))], "error"
  l = list(gams.get('e2', keyType=KeyType.INT))
  assert l == [((1, 3), (3.14, 0, 3.14, 3.14, 1)), ((1, 4), (3.14, 0, 3.14, 3.14, 1))], "error"
$offEmbeddedCode
gams.set(symbolName, data, mergeType=MergeType.DEFAULT, domCheck=DomainCheckType.DEFAULT, mapKeys=lambda x:x)
This method sets the data for the GAMS symbol identified with symbolName. The parameter data takes a Python list or set containing items that represent the records of the symbol. It is also possible to pass an instance of a subclass of _GamsSymbol (e.g. GamsParameter or GamsSet) when using the Object-oriented GAMS Python API in an embedded code section. In case of a Python list or set, depending on the type and the dimension of the symbol, different formats can be used in order to specify the data. Different formats can not be mixed within one list. In general each record needs to be represented as a tuple containing the keys and the value field(s). Keys and/or values can also be enclosed in a tuple. Keys can be entered as labels (string) or label indexes (int). The argument mapKeys allows to pass a callable to remap the elements of the key (e.g. turn them explicitely into strings via mapKeys=str). Value fields depend on the type of the symbol:

Parameters: One numerical value

Sets: explanatory text (optional)

Variable/Equations: Five numerical values: level, marginal, lower bound, upper bound, scale/prior/stage

IEEE special values float('nan'), float('inf'), and float('-inf') will be remapped to GAMS special values NA, INF, and -INF. The small numeric value 4.94066E-324 will be mapped into GAMS special value EPS.

The following Python code gives some examples on different valid formats for different symbol types and dimensions:
$onEmbeddedCode Python:
  # scalar parameter
  data = [3.14]
  data = [(3.14,)]

  # one dimensional parameters:
  data = [("i1", 3.14), ("i2", 3.14)]
  data = [(("i1",), 3.14), (("i2",), 3.14)]
  data = [("i1", (3.14,)), ("i2", (3.14,))]
  data = [(("i1",), (3.14,)), (("i2",), (3.14,))]

  # two dimensional parameter:
  data = [('i1', 'j1', 3.14), ('i1', 'j2', 3.14)]
  data = [(('i1', 'j1'), (3.14,)), (('i1', 'j2'), (3.14,))]

  # one dimensional sets:
  data = ['i1', 'i2']
  data = [('i1',), ('i2',)]

  # one dimensional sets with explanatory text
  data = [('i1', "text 1"), ('i2', "text 2")]
  data = [(('i1',), ("text 1",)), (('i2',), ("text 2",))]

  # scalar variables/equations
  data = [(3.14, 0, 0, 10, 1)]

  # one dimensional variables/equations:
  data = [("i1", 3.14, 0, 0, 10, 1), ("i2", 3.14, 0, 0, 10, 1)]
  data = [("i1", (3.14, 0, 0, 10, 1)), ("i2", (3.14, 0, 0, 10, 1))]
  data = [(("i1",), (3.14, 0, 0, 10, 1)), (("i2",), (3.14, 0, 0, 10, 1))]

  # two dimensional variables/equations:
  data = [("i1", "j1", 3.14, 0, 0, 10, 1), ("i1", "j2", 3.14, 0, 0, 10, 1)]
  data = [(("i1", "j1"), (3.14, 0, 0, 10, 1)), (("i1", "j2"), (3.14, 0, 0, 10, 1))]

  # using label indexes instead of labels
  data = [((1,), (3.14,)), ((2,), (3.14,))] # one dimensional parameter
  data = [((1,), ("text 1",)), ((2,), ("text 2",))] # one dimensional set
  data = [((1, 3), (3.14, 0, 0, 10, 1)), ((1, 4), (3.14, 0, 0, 10, 1))] # two dimensional equation/variable
$offEmbeddedCode
The optional parameter mergeType specifies if data in a GAMS symbol is merged (MergeType.MERGE) or replaced (MergeType.REPLACE). If left at MergeType.DEFAULT it depends on the setting of $on/offMulti[R] if GAMS does a merge, replace, or trigger an error during compile time. During execution time MergeType.DEFAULT is the same as MergeType.MERGE. The optional parameter domCheck specifies if Domain Checking is applied (DomainCheckType.CHECKED) or if records that would cause a domain violation are filtered (DomainCheckType.FILTERED). If left at DomainCheckType.DEFAULT it depends on the setting of $on/offFiltered if GAMS does a filtered load or checks the domains during compile time. During execution time DomainCheckType.DEFAULT is the same as DomainCheckType.FILTERED.

Note
When calling gams.set() in an embedded code section during execution time, new labels that are not known to the current GAMS program can not be added. The attempt will result in an execution error.
gams.getUel(idx)
Returns the label corresponding to the label index idx
gams.mergeUel(label)
Adds label to the GAMS universe if it was unknown and returns the corresponding label index.
Note
When calling gams.mergeUel() in an embedded code section during execution time, new labels that are not known to the current GAMS program can not be added. The attempt will result in an execution error.
gams.getUelCount()
Returns the number of labels.
gams.printLog(msg)
Print msg to log.
gams.arguments
Contains the command line that was passed to the Python interpreter of the embedded code section. The syntax for passing arguments to the Python interpreter can be seen above.
gams.epsAsZero
Flag to read GAMS EPS as 0 (True) or as a small number, 4.94066E-324, when set to False. Default is True.
gams.ws
Property to retrieve an instance of GamsWorkspace that allows to use the Object-oriented GAMS Python API. The instance is created when the property is read for the first time using a temporary working directory. A different working directory can be specified using gams.wsWorkingDir. For debug output, the property gams.debug can be set to a value from DebugLevel
gams.wsWorkingDir
Property that can be specified before accessing gams.ws for the first time in order to set the working directory. Setting the property after the first call to gams.ws will have no effect. of the created GamsWorkspace.
gams.db
Property to retrieve an instance of GamsDatabase. The instance is created when the property is read for the first time and allows to access the GAMS symbols using the methods of the Object-oriented GAMS Python API.
gams.debug
Property that can be set to a value from DebugLevel for debug output. Default is DebugLevel.Off (0). Setting this property affects both the debug output from embedded code and the debug output from the Object-oriented API. The property needs to be changed before accessing gams.ws for the first time in order to take effect in the Object-oriented API. Setting the property after the first call to gams.ws will have no effect on the GamsWorkspace.

For more examples on how to use the interface in Python see the following examples and tests:

embeddedSplit (GAMS Data Utilities Library)

embeddedSort (GAMS Data Utilities Library)

embeddedMultiInstance (GAMS Data Utilities Library)

EMBPY01 (GAMS Test Library)

EMBPY02 (GAMS Test Library)

Using the Object-oriented API

The ECGamsDatabase class provides mechanisms for using the Object-oriented GAMS Python API in an embedded code section. The property gams.ws can be used to get an instance of GamsWorkspace. The property gams.db allows to access an instance of GamsDatabase that can be used to read and write data from the internal GAMS database like it can be done using gams.get and gams.set but using the access mechanisms of the GamsDatabase class.

Exchange via Files and Environment Variables

The Python class ECGamsDatabase provides read and write access to GAMS symbols. There are two other communication methods that can be used at GAMS compile time: files and environment variables. At compile time the Python code can produce a text file that can be included into GAMS via $include as in the following example:

$onEmbeddedCode Python: 10
  f = open('i.txt', 'w')
  for i in range(int(gams.arguments)):
    f.write('i'+str(i+1)+'\n')
  f.close()
$offEmbeddedCode

Set i /
$include i.txt
/;
display i;

Here the Python code received the number of elements to write to a text file via the argument after Python:. This text file is then included in the data statement of the GAMS set i. The display in the listing file looks as follows:

----     20 SET i
i1 ,    i2 ,    i3 ,    i4 ,    i5 ,    i6 ,    i7 ,    i8 ,    i9 ,    i10

Python provides many packages to read input files for many different formats and hence can be used to transform such formats to a GAMS compatible input format, as an alternative to providing the data via list objects and the gams.set functionality.

The second alternative to exchange information at compile time are environment variables. GAMS and Python allow to get and set environment variables and hence can be conveniently used to exchange small pieces of information. The following code provides an example where the maximum value of a parameter b is needed to build a set k:

Set       i    / i1*i5 /;
Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /;

$onEmbeddedCode Python:
  import os
  kmax = int(max([b[1] for b in list(gams.get("b"))]))
  gams.printLog('max value in b is ' + str(kmax))
  os.environ["MAXB"] = str(kmax)
$offEmbeddedCode

$if x%sysEnv.MAXB%==x $abort MAXB is not set
Set k "from 0 to max(b)" / k0*k%sysEnv.MAXB% /;
Scalar card_k; 
card_k = card(k); 
Display card_k;

It is important to understand that os.environ is initialized when Python imports the os module. Hence, if we later set an environment variable in GAMS (via $senEnv ABC abc or put_utility "setenv" / "ABC" / "abc") and access this in some embedded code via os.environ["ABC"], this will not provide the expected value "abc". In such a case the environment variable needs to be accessed via gams.get_env("ABC") (and can be stored in os.environ["ABC"]).

Alternatively in this example, we could build the GAMS set k in Python and send to GAMS via gams.set:

Set       i    / i1*i5 /;
Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /;
Set k "from 0 to max(b)" / system.empty /;

$onEmbeddedCode Python:
  kmax = int(max([b[1] for b in list(gams.get("b"))]))
  gams.printLog('max value in b is ' + str(kmax))
  gams.set("k",list(map(lambda k: 'k'+str(k), range(kmax + 1))))
$offEmbeddedCode k

Scalar card_k; 
card_k = card(k); 
Display card_k;

In both cases the resulting GAMS symbol k is the same and the display in the listing file looks as follows:

----     10 PARAMETER card_k               =       60.000

Multiple Independent Python Sessions

At execution time the user has the ability to pause and continue an embedded code segment. Besides some performance aspects this also allows to work with multiple independent Python sessions. Due to some Python module incompatibilities (e.g. numpy) the independent Python sessions have to be enabled with a command line option pyMultInst set to 1. After the pauseEmbeddedCode we can obtain and store the handle of the last embedded code execution via function embeddedHandle. The handle needs to be supplied when we continue the Python session via continueEmbeddedCode. The different Python sessions are fairly separate as shown in the example below. Here we save the GAMS scalar ord_i in a Python object i five times. The value for i that we store in the five different Python sessions is 1 to 5. In the subsequent loop we activate the Python session with the appropriate handle and print the value of i:

$if not %sysEnv.GMSPYTHONMULTINST%==1 $abort.noError Start with command line option pyMultInst=1
Set       i     / i1*i5 /; 
Parameter h(i) 
          ord_i / 0 /;
loop(i,
  ord_i = ord(i);
  embeddedCode Python:
    i = int(list(gams.get("ord_i"))[0])
    gams.printLog(str(i))
  pauseEmbeddedCode
  h(i) = embeddedHandle;
);
loop(i,
  continueEmbeddedCode h(i):
    gams.printLog(str(i))
  endEmbeddedCode
);

The GAMS log shows the value of i in the different Python sessions:

--- Starting execution: elapsed 0:00:00.002
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 1
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 2
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 3
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 4
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 5
--- Execute embedded library embpycclib64.dll
--- 1
--- Execute embedded library embpycclib64.dll
--- 2
--- Execute embedded library embpycclib64.dll
--- 3
--- Execute embedded library embpycclib64.dll
--- 4
--- Execute embedded library embpycclib64.dll
--- 5
*** Status: Normal completion

Troubleshooting Embedded Python Code

The GAMS compiler ensures that the number of errors during execution time is minimized. While the logic of the GAMS program might be flawed there is nothing (with a few exceptions) that the GAMS system cannot execute. This is different if we embed foreign code in a GAMS program. The GAMS compiler does not understand the foreign code syntax and just skips over it. Only when the code is executed we will find out if everything works as expected. If the embedded code contains some (Python) syntax errors the Python parser will inform us about this and the message will appear in the GAMS log. For example, the following Python code using the gams.printLog function two times will generate a syntax error:

$onEmbeddedCode Python:
  gams.printLog('hello')
   gams.printLog('world...')
$offEmbeddedCode

The GAMS log will provide some guidance:

--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll File "C:\tmp\gamsdir\225a\myPy.dat.dat", line 8
    gams.printLog('world...')
    ^
IndentationError: unexpected indent

--- Python error! Return code from Python execution: -1
*** Error executing Python embedded code section:
*** Check log above

Moreover, if the Python code raises an exception which is not handled within the code this will also lead to a compilation or execution error in GAMS depending at what phase the embedded code is executed.

embeddedCode Python:
  raise Exception('something is wrong')
endEmbeddedCode

will produce the following GAMS log and an execution time error:

--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- Exception from Python: something is wrong
*** Error at line 1: Error executing "embeddedCode" section: Check log above

Note: It is good practice to raise a Python exception if an error occurs. In any case using exit() needs to be avoided since it terminates the executable in an uncontrolled way.

The Python code is executed as part of the GAMS process and GAMS gives control to the Python interpreter when executing the embedded code. So in the worst case if the Python interpreter crashes, the entire GAMS process will crash. Therefore, it is important to be able to test and debug the embedded Python code independent of GAMS. In the following examples we call the Python interpreter executable as part of a GAMS job. In principle this can be tested and debugged completely independent of GAMS where a GDX file represents the content of the GAMS database.

In the first example we mimic the embedded code facility at compile time by exporting the entire GAMS database to a GDX file debug.gdx. With $on/offEcho we write the embedded code with a few extra lines at the top and bottom surrounded by a try/except block and execute the Python interpreter via $call. One of the extra lines at the end of the embedded code triggers the creation of a GDX result file debugOut.gdx which can be imported in subsequent $gdxin/$load commands.

Set i /i1*i10/
    p(i,i) permutation;

$gdxOut debug.gdx
$unload
$gdxOut

$onEcho > debug.py
from gamsemb import *
gams = ECGAMSDatabase('debug.gdx')
try:
  import random
  i = list(gams.get("i"))
  p = list(i)
  random.shuffle(p)
  for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
  gams.set("p", p)
  gmdWriteGDX(gams._gmd,'debugOut.gdx',1);
except Exception as e:
  print(str(e))
$offEcho

$call ="%gams.sysdir%GMSPython/python" debug.py
$if errorlevel 1 $abort Problems running Python

$gdxIn debugOut.gdx
$loadDC p

Option p:0:0:1;
Display p;

In the second example we mimic the embedded code facility at execution time by exporting the entire GAMS database to a GDX file debug.gdx via execute_unload 'debug.gdx';. We write the embedded code with a few extra lines at the top and bottom surrounded by a try/except block via the put facility and execute the Python interpreter via execute. Identical to the compile time example, we export the result to the GDX file debugOut.gdx which can be imported via the execute_load statement.

Set i /i1*i10/
    p(i,i) permutation;

execute_unload 'debug.gdx';
file fpy / 'debug.py' /; put fpy;

$onPutS
from gamsemb import *
gams = ECGAMSDatabase(r'%gams.sysdir% '[:-2],'debug.gdx')
gams.arguments = '-a c -b -db abc'
try:
 import random
 i = list(gams.get("i"))
 p = list(i)
 random.shuffle(p)
 for idx in range(len(i)):
     p[idx] = (i[idx], p[idx])
 gams.set("p", p)
 gmdWriteGDX(gams._gmd,'debugOut.gdx',1);
except Exception as e:
 print(str(e))
$offPut

putClose fpy;
execute '="%gams.sysdir%GMSPython/python" debug.py';
abort$errorlevel 'problems running python';

execute_load 'debugOut.gdx', p;
Option p:0:0:1;
Display p;

Automatic Indentation

All embedded Python code is automatically wrapped in a try-except block that requires an indentation (two spaces). While this additional indentation makes no difference for most Python code, it does make a difference when it comes to multi-line strings. Therefore, multi-line strings created with triple quotes (either single or double quotes) are excluded from this rule to avoid unintentional spaces, e.g. when writing to a file:

$onEmbeddedCode Python:
f = open("file.txt", "w")
f.write('''some
multi
line
string
''')
f.close()
$offEmbeddedCode

Still, when using the line continuation character (\) in combination with strings, one has to be aware of the extra spaces that are added automatically:

$onEmbeddedCode Python:
s = "some\
multi\
line\
string"
print(s)
$offEmbeddedCode

This will print some multi line string instead of somemultilinestring due to the automatically applied indentation.

Performance Considerations of Embedded Python Code

If the same embedded code section (e.g. in a loop) is executed many times there are a few considerations to be taken into account in order to get the best performance. For this we will experiment with the example from the introduction. We look for a random permutation of a set i. In addition we have a cost matrix c(i,ii) and we are looking for the least cost permutation. We should just formulate this as matching in a bi-partite graph but in order to demonstrate some performance considerations we will repeatedly call the Python code that provides a random permutation and we will evaluate the cost of the permutation in GAMS and store the value of the cheapest one. Here is the naive implementation using embedded code:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation; 
c(i,ii) = uniform(-50,50);

Set    iter / 1*100 /;
Scalar tcost
       minTCost / +inf /;
loop(iter,
  embeddedCode Python:
    import random
    i = list(gams.get("i"))
    p = list(i)
    random.shuffle(p)
    for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
    gams.set("p", p)
  endEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
Display minTCost;

In the code we start and stop the Python interpreter and with every execution of the embedded code we need to make the setup and initialization which takes a significant amount of time. The entire GAMS job executes in about 16 seconds. We can avoid the repeated setup and initialization by using pause and continue:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation;
c(i,ii) = uniform(-50,50);

embeddedCode Python:
  import random
pauseEmbeddedCode

Set    iter / 1*1000 /;
Scalar tcost
       minTCost / +inf /;
loop(iter,
  continueEmbeddedCode:
    i = list(gams.get("i"))
    p = list(i)
    random.shuffle(p)
    for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
    gams.set("p", p)
  pauseEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
continueEmbeddedCode:
  pass
endEmbeddedCode
Display minTCost;

The last embedded code execution of the Python pass statement is to clean up and terminate the Python session. As you can see from the set iter we had to increase this from 100 to 1000 to measure the timing properly. This run takes about 1.179 secs (we ran this 20 times and build the average). This is the biggest improvement, the other two following enhancements are just icing on the cake. We can actually extract the set i and store this in Python list i just once:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation; 
c(i,ii) = uniform(-50,50);

embeddedCode Python:
  import random
  i = list(gams.get("i"))
pauseEmbeddedCode

set    iter / 1*1000 /;
scalar tcost
       minTCost / +inf /;
loop(iter,
  continueEmbeddedCode:
    p = list(i)
    random.shuffle(p)
    for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
    gams.set("p", p)
  pauseEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
continueEmbeddedCode:
pass
endEmbeddedCode
Display minTCost;

The total running time of this is 1.005 secs. In addition, we can work with label indexes rather than the labels itself. Indexes are integers and are often faster than labels that are stored as strings. The only difference to the code above is the extraction method of the Python list i by i = list(gams.get("i",keyType=KeyType.INT)). The resulting running time is 0.993 secs.

Extending GMSPython

While we recommend to use your own Python installation if you need additional packages, there are ways to extend the Python system that comes with GAMS in GMSPython. Here are the steps:

Get pip via get-pip (https://pip.pypa.io/en/stable/installing/)
Update pip
Install additional packages

We highly recommend installing pip and additional packages in the user site (use --user when running get-pip and pip). This is especially important for macOS users. Installing files in the GAMS system directory may interfere with the file notarization and may prevent GAMS from starting properly. Here is a typical dialog for Linux/macOS using curl to download get-pip.py. If you don't have curl or wget, use your web browser for downloading.

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
/path/to/gams/GMSPython/python get-pip.py --user
~/.local/bin/pip install -U pip
~/.local/bin/pip install geocoder --user

Using an External Python 3 Installation

Let's assume you don't want to work with the GAMS distributed GMSPython but you want to connect GAMS with a Python 3 installation of your liking. In order for the Embedded Code Facility to work, you need to make the alternative Python interpreter aware of the required modules provided by GAMS by following the Getting started steps in the Python API tutorial.

In addition to this you need to do one more step to allow GAMS to find your Python installation when executing Embedded Python code: you need to point to the Python dynamic load library or shared object. On Windows this file is called python3X.dll, on Linux libpython3.X.so, and on macOS libpython3.X.dylib. The X stands for the minor version number of Python 3. While embedded code has been built and tested for Python 3.8, chances are that you can also point to other Python 3 libraries. Since the expert level API is required, the set of versions is limited to what GAMS supports (currently GAMS ships these for 3.6, 3.7, and 3.8). Limited experiments with the embedded code library for Python 3.8 with Python 3.6 and 3.7 libraries were successful. You need to set the environment variable GMSPYTHONLIB to point GAMS to the Python library:

Windows:
set GMSPYTHONLIB=c:\path\to\python\python38.dll
Linux:
export GMSPYTHONLIB=/path/to/python/lib/libpython3.8.so
Mac OSX:
export GMSPYTHONLIB=/path/to/python/lib/libpython3.8.dylib

GAMS will extract the Python home from the location of the Python library. In case this extraction does not work (because of an unconventional Python installation), you can instruct GAMS to set the Python home via the additional environment variable GMSPYTHONHOME. Please note, that this environment variable only needs to be set for GAMS Embedded Python code, so it is sufficient to add these variables to the GAMS configuration YAML file.

Note: When using environment-based Python distributions like Anaconda or Miniconda, it is not sufficient to set GMSPYTHONLIB (and GMSPYTHONHOME). The environment to be used needs to be activated properly before (e.g. conda activate myEnv). Packages like numpy which have been installed using conda (conda install numpy) won't work otherwise. In addition to setting GMSPYTHONLIB we therefore recommend to start GAMS itself and also tool like GAMS Studio from an activated conda environment in order to make sure that the environment and all installed packages work as expected.

Although it is recommended to use a self-contained Python installation, it is possible to use a virtual environment instead (e.g. via venv). In this case, GMSPYTHONLIB has to be set to the Python library of the installation from which the environment was derived from. In additon, the PYTHONPATH environment variable needs to be set to the site-packages directory of the virtual environment in order to make its packages available. Note that with this setup the site-packages directory of the original Python installation will also be found by GAMS Embedded Python code.

Building your own Embedded Python Code Library

Although the Embedded Code Facility and its binary components are part of the GAMS distribution, it is possible to build it manually from source using the following commands. The exact command line might change depending on the compiler and the operating system in use:

Windows:

cd [GAMS directory]/apifiles/C/api
icl.exe -Feembpycclib64.dll -IC:\path\to\python\include embpyoo.c gmdcc.c gclgms.c -LD -link -nodefaultlib:python38.lib

Linux:

cd [GAMS directory]/apifiles/C/api
gcc -o libembpycclib64.so -nostartfiles -shared -Wl,-Bsymbolic -m64 -pthread -fPIC -Ipath/to/your/python/include/python3.8 embpyoo.c gmdcc.c gclgms.c -lm -ldl

Mac OS X:

cd [GAMS directory]/apifiles/C/api
gcc -o libembpycclib64.dylib -dynamiclib -shared -m64 -install_name @rpath/libembpycclib64.dylib -Ipath/to/your/python/include/python3.8 embpyoo.c gmdcc.c gclgms.c -lm -ldl

Table of Contents