Motivation

GAMS uses relational data tables as a basic data structure. With these, GAMS code for parallel assignment and equation definition is compact, elegant, and efficient. However, traditional data structures (arrays, lists, dictionaries, trees, graphs, ...) are not natively or easily available in GAMS. Though it is possible to represent such data structures in GAMS, the GAMS code working with such structures can easily become unwieldy, obfuscating, or inefficient. Also, in the past it was not easy to connect libraries from other systems for special algorithms (e.g. graph algorithms, matrix operations, ...) to GAMS without some data programming knowledge and a deep understanding of internal representation of GAMS data (e.g. the GDX API).

The Embedded Code Facility addresses this need and extends the connectivity of GAMS to other programming languages. It allows the use of external code (e.g. Python) during compile and execution time. GAMS symbols are shared with the external code, so no communication via disk is necessary. It utilizes an API (this API can be found in apifiles/C/api/emblib*.h together with the code for the Python and Connect embedded code library apifiles/C/api/embpyoo.c/h) and additional source code for common tasks so that the user can concentrate on the task at hand and not the mechanics of moving data in and out of GAMS.

Concept

As pointed out in section Motivation, the main idea of the Embedded Code Facility is to enable the use of external code in GAMS and give this code direct in-memory access to the GAMS database (or better: to GAMS symbols, namely sets, parameters, variables, and equations). This can be done by defining sections in the GAMS code which contain no GAMS code, but code written in a defined external code. These sections can be used at both GAMS compile and execution time (compare section GAMS Compile Time and Execution Time Phase). Details about how to do this can be found in the Syntax section.

Note: We will continuously extend this feature to different programming languages. At the moment only Python and Connect are supported.

Also, the system provides some help to develop and debug the external code independent of GAMS first. More about this topic can be found in section Troubleshooting Embedded Python Code.

The communication of the data between the GAMS part of the program and the embedded code part was inspired by the existing interface to GDX in many ways. For example, using dollar control options $on/offMulti, $onMultiR, and $on/offFiltered it is possible to decide if data changed in the embedded code should replace the GAMS data or get merged with it and just like loading data from GDX, one can decide if data from embedded code should change the GAMS database filtered or domain checked.

Simple Example

Python

In a very first example we look into some Python code that helps to split some label names that are already present in GAMS. We do this at compile time in order to read the broken up pieces as individual sets (country and city) into GAMS plus some mapping sets (mccCountry and mccCity) between the original labels and the new labels. The compile-time embedded code starts with a $onEmbeddedCode followed by the type of code to expect (Python:). The lines between $onEmbeddedCode and $offEmbeddedCode is Python code. We do not want to go into Python details, but the first few lines initialize some empty Python list objects (mccCountry and mccCity) as well as some empty Python set objects. In the Python for loop that follows we iterate over all individual labels of the GAMS set cc. Python gets access to the GAMS set cc via the member function get of the implicitly defined Python object gams. get returns an object that is iterable and can be used in a Python for loop. The type of the records one gets depend on the dimensionality and type of the GAMS symbol. In the loop body we use the Python split function to extract the first (r[0] is country) and second (r[1] is city) part of the label. The three strings cc, r[0], r[1] are used to build up the Python list objects that store the information for the maps mccCountry and mccCity and the Python set objects that store the labels for the new sets country and city. The Python set has the advantage to store a label just once even if we add it multiple times. Python prepares to send items back to GAMS via the gams member function set that can deal with both Python list and Python set objects. The command $offEmbeddedCode is followed by a list (without separating commas) of GAMS symbols that instructs the GAMS compiler to read these symbols back. This list is optional and can be left out. GAMS will import all symbols set by the Python code. The GAMS compiler (without executing execution-time embedded code) does not know which symbols will be implicitly loaded, hence it cannot define symbols. If such a symbol is referenced further down in the compilation process, the compiler will complain with error $141: Symbol declared but no values have been assigned. One can avoid this situation by explicitly list the symbols on the end/pauseEmbeddedCode line.

Note that GAMS syntax is case insensitive while Python is case sensitive. Hence, the strings that represent the GAMS symbol names in the Python code can have any casing (e.g. gams.get("cc") or gams.get("CC")), while the corresponding Python objects need to have consistent casing throughout the Python code.

The following code presents the entire embeddedSplit example from the GAMS Data Utilities Library:

Set cc      / "France - Paris", "France - Lille", "France - Toulouse"
              "Spain - Madrid", "Spain - Cordoba", "Spain - Seville", "Spain - Bilbao"
              "USA - Washington DC", "USA - Houston", "USA - New York",
              "Germany - Berlin", "Germany - Munich", "Germany - Bonn" /
    country 
    city
    mccCountry(cc,country<)  Mapping between country and related elements in set cc
    mccCity(cc,city<)        Mapping between city and related elements in set cc;

$onEmbeddedCode Python:
mccCountry = []
mccCity = []
for cc in gams.get("cc"):
  r = str.split(cc, " - ", 1)
  mccCountry.append((cc,r[0]))
  mccCity.append((cc,r[1]))
gams.set("mccCountry",mccCountry)
gams.set("mccCity",mccCity)
$offEmbeddedCode mccCountry mccCity

Option mccCountry:0:0:1, mccCity:0:0:1;
Display country, city, mccCountry, mccCity;

The display in the listing file looks as follows:

----     25 SET country
Spain  ,    USA    ,    Germany,    France 

----     25 SET city
Berlin       ,    Bilbao       ,    Cordoba      ,    Madrid
New York     ,    Washington DC,    Paris        ,    Houston
Munich       ,    Lille        ,    Seville      ,    Bonn
Toulouse

----     25 SET mccCountry
France - Paris     .France 
France - Lille     .France 
France - Toulouse  .France 
Spain - Madrid     .Spain
Spain - Cordoba    .Spain
Spain - Seville    .Spain
Spain - Bilbao     .Spain
USA - Washington DC.USA
USA - Houston      .USA
USA - New York     .USA
Germany - Berlin   .Germany
Germany - Munich   .Germany
Germany - Bonn     .Germany

----     25 SET mccCity
France - Paris     .Paris
France - Lille     .Lille
France - Toulouse  .Toulouse
Spain - Madrid     .Madrid
Spain - Cordoba    .Cordoba
Spain - Seville    .Seville
Spain - Bilbao     .Bilbao
USA - Washington DC.Washington DC
USA - Houston      .Houston
USA - New York     .New York
Germany - Berlin   .Berlin
Germany - Munich   .Munich
Germany - Bonn     .Bonn

The second example demonstrates the use of embedded code at execution time. The syntax for the execution time embedded code in this example is identical to the compile time variant with the exception of the keywords that start and end the embedded code section: embeddedCode and endEmbeddedCode. It is important to understand that the execution of the code happens at GAMS execution time, so e.g. no new labels can be produced and send back to GAMS. In this example we use some Python code to generate a random permutation of set elements of set i and store this in a two dimensional set p. In this example we do not use a loop to iterate through the elements of a GAMS system but make use of the fact that the Python object returned by gams.get("i") is iterable and can in its entirety be stored in the Python list with name i with the short and powerful command i = list(gams.get("i")). The permutation of elements in list p which is a copy of list i is created by the Python statement random.shuffle(p). The following code presents the entire example:

Set i /i1*i10/
    p(i,i) "permutation";

embeddedCode Python:
import random
i = list(gams.get("i"))
p = list(i)
random.shuffle(p)
for idx in range(len(i)):
    p[idx] = (i[idx], p[idx])
gams.set("p", p)
endEmbeddedCode p

option p:0:0:1; 
display p;

The display in the listing file looks as follows:

----     11 SET p  permutation
i1 .i1 
i2 .i7 
i3 .i5 
i4 .i2 
i5 .i10
i6 .i6 
i7 .i9 
i8 .i4 
i9 .i8 
i10.i3

Connect

In the following example, we need to process a CSV file that cannot be read directly by GAMS under $onDelim. So we instruct Connect to read the CSV file into the Connect database and then write to GAMS:

Set dates, stocks;
Parameter stockprice(dates<,stocks<);

$onText
Read a CSV file that looks as follows:

date;symbol;price
2016/01/04;AAPL;105,35
2016/01/04;AXP;67,59
2016/01/04;BA;140,50
...

and can't be read directly by GAMS under $onDelim because some labels
include '/' and are not quoted, use ';' as field separator, and ',' as
decimal character.
$offText

$onEmbeddedCode Connect:
- CSVReader:
    file: stockprice.csv
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
    fieldSeparator: ';'
    decimalSeparator: ','
- GAMSWriter:
    symbols:
      - name: stockprice
$offEmbeddedCode

display stockprice;

We leave the CSV file processing to the CSVReader and then export the stockprice symbol to GAMS. Please note that the symbols written to GAMS by the GAMSWriter do not have to be explicitly listed on the $offEmbeddedCode line, this is optional. Moreover, the Connect agents GAMSReader and GAMSWriter are only available under embedded code.

We might use the data to calculate an optimal portfolio. The share of each stock in this optimal portfolio is represented by the parameter share(stocks) and can be exported to Excel using the PandasExcelWriter. Before exporting to Excel, the data first needs to be transferred from GAMS to Connect using the GAMSReader:

* Continues the code from the previous example

Parameter share(stocks);
* For demonstration we fill share with random numbers
share(stocks) = uniform(0,1);

embeddedCode Connect:
- GAMSReader:
    symbols:
      - name: share
- PandasExcelWriter:
    file: share.xlsx
    symbols:
      - name: share
        range: Share!A1
endEmbeddedCode

Syntax

This section explains the GAMS functions/keywords which were introduced to enable the Embedded Code Facility. The first subsection deals with the syntax for compile time, the second with the syntax for execution time (compare section GAMS Compile Time and Execution Time Phase).

Compile Time

There are three dollar control options to start an embedded code section at compile time:

$onEmbeddedCode[.tag]  [Connect|Python]: [arguments]
$onEmbeddedCodeS[.tag] [Connect|Python]: [arguments]
$onEmbeddedCodeV[.tag] [Connect|Python]: [arguments]

Lines following one of the above statements are passed on to the embedded code engine (e.g. Connect or the Python interpreter) until this dollar control option, which ends the embedded code section at compile time, is hit:

$offEmbeddedCode[.tag] {symbol[<[=]embSymbol[.dimX]]}

These dollar control options are explained here in more detail. The usage of the optional arguments by the embedded code is explained in the specific embedded code engine section below.

Note

The optional arguments from the $onEmbeddedCode[S|V][.tag] statement can be accessed as gams.arguments in the Python code.
The optional output symbols from the $offEmbeddedCode[.tag] statement need to be written to by the embedded code.
More about the specific GAMS Python syntax can be found below.

Execution Time

At execution time an embedded code section is started with one of these statements:

embeddedCode[.tag]  [Connect|Python]: [arguments]
embeddedCodeS[.tag] [Connect|Python]: [arguments]
embeddedCodeV[.tag] [Connect|Python]: [arguments]

Similar to the compile time alternatives $onEmbeddedCode[S|V][.tag], the first two variants are synonyms which allow parameter substitution in the code that follows, while the last variant does not allow this but passes the code verbatim to the embedded code engine. The usage of the optional arguments by the embedded code is explained in the specific embedded code engine section below. The arguments passed to the embedded code engine at execution time can be extended by the put_utility command ECArguments. For example,

put_utility 'ECArguments' / ' world...';
embeddedCode  Python: Hello
gams.printLog(gams.arguments)
endEmbeddedCode

prints Hello world....

Lines following one of the statements embeddedCode[S|V] are passed on to the embedded code engine (e.g. the Python interpreter) until one of the following two statements, which end the embedded code section at execution time, is hit:

endEmbeddedCode[.tag]   {output symbols}
pauseEmbeddedCode[.tag] {output symbols}

When the optional tag suffix (which can be any string) is used to start an embedded code section, the same one has to be used to end/pause that section. Both statements end the embedded code section and switch back to GAMS syntax in the following lines. Also, both statements can be followed by a GAMS symbol or a list of GAMS symbols which would get updated in the GAMS database after the embedded code got executed. If output symbols are specified, they need to be written to (e.g. by gams.set for embedded Python code) by the embedded code before.

To continue a previously paused embedded code section one of the following statements is used:

continueEmbeddedCode  [handle]: [arguments]
continueEmbeddedCodeS [handle]: [arguments]
continueEmbeddedCodeV [handle]: [arguments]

As seen before, the first two variants are synonyms which allow parameter substitution in the embedded code that follows, while the last variant does not allow this but passes the code verbatim to the interpreter. Again, the usage of the optional arguments by the embedded code is explained in the specific embedded code engine section below.

New in these statements is the optional handle. If omitted, the last code section that was paused will be continued. However, sometimes one might need to maintain different embedded code sections active in parallel and independent of each other. There is a new function to store a handle of the last embedded code section that was executed which could then later be used to continue a specific paused code section:

handle = embeddedHandle;

What it exactly means to end or pause/continue an embedded code section depends on the embedded code engine and is explained in more detail with the code engine below. Independent of the embedded code engine it is important to note that if GAMS stops and restarts while executing a solve statement (with solveLink=0) continuing any embedded code section causes the code to fail with the following error message:

    Error executing "continueEmbeddedCode" section
--- (Hint: "Solve" with SolveLink=0 frees previously initialized embedded libraries):
    Error at line 71: No embedded library initialized

This happens because under solvelink=0 the GAMS process terminates when the solver starts up. This causes the loss of the state of the embedded code environment that is linked into the GAMS process as a shared library. Under all other settings of solveLink the GAMS process is not stopped and restarted and hence a paused embedded code environment can be continued after the solve statement has carried out.

Python

GAMS and Python complement each other in many different ways. GAMS has a compact but readable syntax for data assignment and equation definition statements. Python is a great scripting language to do more traditional type programming especially string manipulation which is completely missing in GAMS. Moreover, the vast number of packages help solving scientific problems outside the scope of GAMS. In order to open this world to our customers in an easy way, GAMS comes with a Python 3 installation. By default, this installation is used in the Embedded Code Facility for Python and is ready to be used with the GAMS Python API. The Python installation is located in [GAMS directory]/GMSPython. In contrast to earlier versions this installation comes without the Python package manager pip and therefore is not easily extendable. GMSPython comes with a selection of third party packages and modules that are either required by the APIs of the GAMS Python API collection or that are particularly helpful from within embedded Python code. Those packages are located in [GAMS directory]/apifiles/Python/thirdparty. Running the GMSPython interpreter like follows will display all available modules:

[GAMS directory]/GMSPython/python -c"help('modules')"

The list of redistributed third party packages is subject to constant changes and it is not recommended to rely on their availability. If you need packages that do not come with GMSPython, we recommend that you install your own version of Python and follow a few steps to connect GAMS with your Python installation. It is also possible to install pip for GMSPython using get-pip. While any Python installation should work in combination with GAMS we can recommend the Anaconda or Miniconda Python distributions. These Python distributions work with environments which allow you to have many different Python installations at the same time that don't get in each other's way.

During the lifetime of the GAMS process a single Python interpreter is used and the code "lives" in Python's global scope. This means that objects defined in one embedded Python code section are still there in another embedded Python code section:

$onEmbeddedCode Python:
s = 'Hello world'
$offEmbeddedCode
$onEmbeddedCode Python:
gams.printLog(s)
$offEmbeddedCode

will produce the following output in the GAMS log:

--- Starting compilation
--- main.gms(3) 2 Mb
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- main.gms(6) 2 Mb
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
Hello world
--- main.gms(7) 2 Mb

If the GAMS process stops and restarts during execution, e.g. because of a solve with solveLink=0, the Python interpreter will be started new. This can cause complications if GAMS is in a sequence of pause/continueEmbeddedCode as discussed above.

Interface between GAMS and Python

The Python class ECGamsDatabase is the interface between GAMS and Python. An instance of this class is automatically created when an embedded code section is entered and can be accessed using the identifier gams. The following methods can be used in order to interact with GAMS:

gams.get(symbolName, keyType=KeyType.STRING, keyFormat=KeyFormat.AUTO, valueFormat=ValueFormat.AUTO, recordFormat=RecordFormat.AUTO)

This method retrieves an iterable object representing the symbol identified with symbolName. Typically there are two possibilities to access the records. Iterating using e.g. a for loop provides access to the individual records. By calling list() on the iterable object, a list containing all the data is created. Several optional parameters can be used in order to modify the format of the retrieved data:

keyType: Determines the data type of the keys. It can be either KeyType.STRING (labels) or KeyType.INT (label indexes). The default setting is KeyType.STRING.

keyFormat: Specifies the representation of the keys. Possible values are as follows:

KeyFormat.TUPLE: Encapsulate keys in a tuple

KeyFormat.FLAT: No encapsulation

KeyFormat.SKIP: Keys are skipped and do not appear in the retrieved data

KeyFormat.AUTO (default): Depending on the dimension of the GAMS symbol, a default format is applied:

Zero dimensional/scalar: KeyFormat.SKIP

One dimensional: KeyFormat.FLAT

Multi dimensional: KeyFormat.TUPLE

valueFormat: Specifies the representation of the values. Possible values are as follows:

ValueFormat.TUPLE: Encapsulate values in a tuple

ValueFormat.FLAT: No encapsulation

ValueFormat.SKIP: Values are skipped and do not appear in the retrieved data

ValueFormat.AUTO (default): Depending on the type of the GAMS symbol, a default formats is applied:

Set: ValueFormat.SKIP

Parameter: ValueFormat.FLAT

Variable/Equation: ValueFormat.TUPLE

recordFormat Specifies the encapsulation of records into tuples. Possible values are as follows:

RecordFormat.TUPLE: Encapsulates every record in a tuple

RecordFormat.FLAT: No encapsulation. Throws an exception if it can not be applied. It is guaranteed that the length of a retrieved Python list is equal to the number of records of the corresponding GAMS symbol. This principle leads to an incompatibility of RecordFormat.FLAT whenever a record consists of more than one item (e.g. multi dimensional symbols, variables and equations which have five numeric values).

RecordFormat.AUTO (default): Depending on the number of items that represent a record, a default format is applied. If possible this is always RecordFormat.FLAT.

GAMS special values NA, INF, and -INF will be mapped to IEEE special values float('nan'), float('inf'), and float('-inf'). GAMS special value EPS will be either mapped to 0 or to the small numeric value 4.94066E-324 depending on the setting of flag gams.epsAsZero.

The following Python code shows some examples of gams.get and illustrates the use of different formats:
Set i / i1 text 1, i2 text 2 /
    j / j1*j2 /

Scalar p0 /3.14/;
Parameter p1(i)   / #i 3.14 /
          p2(i,j) / i1.#j 3.14 /
Variable v0       / fx 3.14 /;
equation e1(i)    / #i.fx 3.14 /
         e2(i,j)  / i1.#j.fx 3.14 /;

$onEmbeddedCode Python:
# scalar parameter
gams.printLog(f"{list(gams.get('p0'))}")                                  # prints [3.14]
gams.printLog(f"{list(gams.get('p0', recordFormat=RecordFormat.TUPLE))}") # prints [(3.14,)]

# one dimensional parameters:
gams.printLog(f"{list(gams.get('p1'))}")                                  # prints [('i1', 3.14), ('i2', 3.14)]
gams.printLog(f"{list(gams.get('p1', keyFormat=KeyFormat.TUPLE))}")       # prints [(('i1',), 3.14), (('i2',), 3.14)]
gams.printLog(f"{list(gams.get('p1', valueFormat=ValueFormat.TUPLE))}")   # prints [('i1', (3.14,)), ('i2', (3.14,))]
gams.printLog(f"{list(gams.get('p1', keyFormat=KeyFormat.TUPLE, valueFormat=ValueFormat.TUPLE))}")   # prints [('i1', (3.14,)), ('i2', (3.14,))]

# two dimensional parameter:
gams.printLog(f"{list(gams.get('p2'))}")                                  # prints [(('i1', 'j1'), 3.14), (('i1', 'j2'), 3.14)]
gams.printLog(f"{list(gams.get('p2', keyFormat=KeyFormat.FLAT))}")        # prints [('i1', 'j1', 3.14), ('i1', 'j2', 3.14)]

# one dimensional sets:
gams.printLog(f"{list(gams.get('i'))}")                                   # prints ['i1', 'i2']
gams.printLog(f"{list(gams.get('i',valueFormat=ValueFormat.FLAT))}")      # prints [('i1', 'text 1'), ('i2', 'text 2')]

# scalar variables/equations
gams.printLog(f"{list(gams.get('v0'))}")                                  # prints [(3.14, 0.0, 3.14, 3.14, 1.0)]

# one dimensional variables/equations:
gams.printLog(f"{list(gams.get('e1'))}")                                  # prints [('i1', (3.14, 0.0, 3.14, 3.14, 1.0)), ('i2', (3.14, 0.0, 3.14, 3.14, 1.0))]
gams.printLog(f"{list(gams.get('e1', valueFormat=ValueFormat.FLAT))}")    # prints [('i1', 3.14, 0.0, 3.14, 3.14, 1.0), ('i2', 3.14, 0.0, 3.14, 3.14, 1.0)]
gams.printLog(f"{list(gams.get('e1', keyFormat=KeyFormat.TUPLE))}")       # prints [(('i1',), (3.14, 0.0, 3.14, 3.14, 1.0)), (('i2',), (3.14, 0.0, 3.14, 3.14, 1.0))]

# two dimensional variables/equations:
gams.printLog(f"{list(gams.get('e2'))}")                                  # prints [(('i1', 'j1'), (3.14, 0.0, 3.14, 3.14, 1.0)), (('i1', 'j2'), (3.14, 0.0, 3.14, 3.14, 1.0))]
gams.printLog(f"{list(gams.get('e2', keyFormat=KeyFormat.FLAT, valueFormat=ValueFormat.FLAT))}") # prints [('i1', 'j1', 3.14, 0.0, 3.14, 3.14, 1.0), ('i1', 'j2', 3.14, 0.0, 3.14, 3.14, 1.0)]

# using label indexes instead of labels
gams.printLog(f"{list(gams.get('p1', keyType=KeyType.INT))}")             # prints [(1, 3.14), (2, 3.14)]
gams.printLog(f"{list(gams.get('i', keyFormat=KeyFormat.TUPLE, valueFormat=ValueFormat.TUPLE, keyType=KeyType.INT))}") # prints [((1,), ('text 1',)), ((2,), ('text 2',))]
gams.printLog(f"{list(gams.get('e2', keyType=KeyType.INT))}")             # prints [((1, 3), (3.14, 0.0, 3.14, 3.14, 1.0)), ((1, 4), (3.14, 0.0, 3.14, 3.14, 1.0))]
$offEmbeddedCode  
gams.set(symbolName, data, mergeType=MergeType.DEFAULT, domCheck=DomainCheckType.DEFAULT, mapKeys=lambda x:x)
This method sets the data for the GAMS symbol identified with symbolName. The parameter data takes a Python list or set containing items that represent the records of the symbol. It is also possible to pass an instance of a subclass of _GamsSymbol (e.g. GamsParameter or GamsSet) when using the Object-oriented GAMS Python API in an embedded code section. In case of a Python list or set, depending on the type and the dimension of the symbol, different formats can be used in order to specify the data. Different formats can not be mixed within one list. In general each record needs to be represented as a tuple containing the keys and the value field(s). Keys and/or values can also be enclosed in a tuple. Keys can be entered as labels (string) or label indexes (int). The argument mapKeys allows to pass a callable to remap the elements of the key (e.g. turn them explicitly into strings via mapKeys=str). Value fields depend on the type of the symbol:

Parameters: One numerical value

Sets: explanatory text (optional)

Variable/Equations: Five numerical values: level, marginal, lower bound, upper bound, scale/prior/stage

IEEE special values float('nan'), float('inf'), and float('-inf') will be remapped to GAMS special values NA, INF, and -INF. The small numeric value 4.94066E-324 will be mapped into GAMS special value EPS.

The following Python code gives some examples on different valid formats for different symbol types and dimensions:
Set i / i1 text 1, i2 text 2 /
    j / j1*j2 /
    ii(i);
Scalar p0;
Parameter p1(i)
          p2(i,j);
equation e1(i)
         e2(i,j);
$onEmbeddedCode Python:
# scalar parameter
gams.set('p0', [3.14])
gams.set('p0', [(3.14,)])

# one dimensional parameters:
gams.set('p1', [("i1", 3.14), ("i2", 3.14)])
gams.set('p1', [(("i1",), 3.14), (("i2",), 3.14)])
gams.set('p1', [("i1", (3.14,)), ("i2", (3.14,))])
gams.set('p1', [(("i1",), (3.14,)), (("i2",), (3.14,))])

# two dimensional parameter:
gams.set('p2', [('i1', 'j1', 3.14), ('i1', 'j2', 3.14)])
gams.set('p2', [(('i1', 'j1'), (3.14,)), (('i1', 'j2'), (3.14,))])

# one dimensional sets:
gams.set('ii', ['i1', 'i2'])
gams.set('ii', [('i1',), ('i2',)])

# one dimensional sets with explanatory text
gams.set('ii', [('i1', "text 1"), ('i2', "text 2")])
gams.set('ii', [(('i1',), ("text 1",)), (('i2',), ("text 2",))])

# one dimensional variables/equations:
gams.set('e1', [("i1", 3.14, 0, 0, 10, 1), ("i2", 3.14, 0, 0, 10, 1)])
gams.set('e1', [("i1", (3.14, 0, 0, 10, 1)), ("i2", (3.14, 0, 0, 10, 1))])
gams.set('e1', [(("i1",), (3.14, 0, 0, 10, 1)), (("i2",), (3.14, 0, 0, 10, 1))])

# two dimensional variables/equations:
gams.set('e2', [("i1", "j1", 3.14, 0, 0, 10, 1), ("i1", "j2", 3.14, 0, 0, 10, 1)])
gams.set('e2', [(("i1", "j1"), (3.14, 0, 0, 10, 1)), (("i1", "j2"), (3.14, 0, 0, 10, 1))])

# using label indexes instead of labels
gams.set('p1', [((1,), (3.14,)), ((2,), (3.14,))]) # one dimensional parameter
gams.set('ii', [((1,), ("text 1",)), ((2,), ("text 2",))]) # one dimensional set
gams.set('e2', [((1, 3), (3.14, 0, 0, 10, 1)), ((1, 4), (3.14, 0, 0, 10, 1))]) # two dimensional equation/variable
$offEmbeddedCode p0 p1 p2 ii e1 e2
The optional parameter mergeType specifies if data in a GAMS symbol is merged (MergeType.MERGE) or replaced (MergeType.REPLACE). If left at MergeType.DEFAULT it depends on the setting of $on/offMulti[R] if GAMS does a merge, replace, or trigger an error during compile time. During execution time MergeType.DEFAULT is the same as MergeType.MERGE. The optional parameter domCheck specifies if Domain Checking is applied (DomainCheckType.CHECKED) or if records that would cause a domain violation are filtered (DomainCheckType.FILTERED). If left at DomainCheckType.DEFAULT it depends on the setting of $on/offFiltered if GAMS does a filtered load or checks the domains during compile time. During execution time DomainCheckType.DEFAULT is the same as DomainCheckType.FILTERED.

Note
When calling gams.set() in an embedded code section during execution time, new labels that are not known to the current GAMS program can not be added. The attempt will result in an execution error.
gams.getUel(idx)
Returns the label corresponding to the label index idx
gams.mergeUel(label)
Adds label to the GAMS universe if it was unknown and returns the corresponding label index.
Note
When calling gams.mergeUel() in an embedded code section during execution time, new labels that are not known to the current GAMS program can not be added. The attempt will result in an execution error.
gams.getUelCount()
Returns the number of labels.
gams.printLog(msg)
Print msg to log.
gams.arguments
Contains the command line that was passed to the Python interpreter of the embedded code section. The syntax for passing arguments to the Python interpreter can be seen above.
gams.epsAsZero
Flag to read GAMS EPS as 0 (True) or as a small number, 4.94066E-324, when set to False. Default is True.
gams.ws
Property to retrieve an instance of GamsWorkspace that allows to use the Object-oriented GAMS Python API. The instance is created when the property is read for the first time using a temporary working directory. A different working directory can be specified using gams.wsWorkingDir. For debug output, the property gams.debug can be set to a value from DebugLevel
gams.wsWorkingDir
Property that can be specified before accessing gams.ws for the first time in order to set the working directory. Setting the property after the first call to gams.ws will have no effect. of the created GamsWorkspace.
gams.db
Property to retrieve an instance of GamsDatabase. The instance is created when the property is read for the first time and allows to access the GAMS symbols using the methods of the Object-oriented GAMS Python API.
gams.debug
Property that can be set to a value from DebugLevel for debug output. Default is DebugLevel.Off (0). Setting this property affects both the debug output from embedded code and the debug output from the Object-oriented API. The property needs to be changed before accessing gams.ws for the first time in order to take effect in the Object-oriented API. Setting the property after the first call to gams.ws will have no effect on the GamsWorkspace.

Using the Object-oriented API

The ECGamsDatabase class provides mechanisms for using the Object-oriented GAMS Python API in an embedded code section. The property gams.ws can be used to get an instance of GamsWorkspace. The property gams.db allows to access an instance of GamsDatabase that can be used to read and write data from the internal GAMS database like it can be done using gams.get and gams.set but using the access mechanisms of the GamsDatabase class. In addition to the different reading and writing mechanisms offered by the OO-API other classes like GamsModelInstance can be used inside embedded Python code. This combination provides programmatic access to a generated model instance and exchange of update parameters and results between Python and GAMS. The GAMS Library model embmiex1 gives an example of how to combine embedded Python code and a GamsModelInstance which utilized the library include $libInclude pyEmbMI.

Moreover, the GAMS Transfer Python API can be combine with the OO-API in the embedded code context as the following example demonstrates:

Set i / i1*i1000 /; alias(i,j);
Parameter a(i,j), aInv(i,j);
a(i,j) = 1 / (ord(i)+ord(j) - 1);
a(i,i) = 1;

embeddedCode Python:
import gamstransfer as gt
import numpy as np

m = gt.Container(gams.db)
A = m.data["a"].toDense()
m.data['aInv'].setRecords(np.linalg.inv(A))
m.write(gams.db, ["aInv"])
endEmbeddedCode aInv

Exchange via Files and Environment Variables

The Python class ECGamsDatabase provides read and write access to GAMS symbols. There are two other communication methods that can be used at GAMS compile time: files and environment variables. At compile time the Python code can produce a text file that can be included into GAMS via $include as in the following example:

$onEmbeddedCode Python: 10
with open('i.txt', 'w') as f:
  for i in range(int(gams.arguments)):
    f.write(f'i{i+1}\n')
$offEmbeddedCode

Set i /
$include i.txt
/;
display i;

Here the Python code received the number of elements to write to a text file via the argument after Python:. This text file is then included in the data statement of the GAMS set i. The display in the listing file looks as follows:

----     20 SET i
i1 ,    i2 ,    i3 ,    i4 ,    i5 ,    i6 ,    i7 ,    i8 ,    i9 ,    i10

Python provides many packages to read input files for many different formats and hence can be used to transform such formats to a GAMS compatible input format, as an alternative to providing the data via list objects and the gams.set functionality.

The second alternative to exchange information at compile time are environment variables. GAMS and Python allow to get and set environment variables and hence can be conveniently used to exchange small pieces of information. The following code provides an example where the maximum value of a parameter b is needed to build a set k:

Set       i    / i1*i5 /;
Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /;
$onEmbeddedCode Python:
import os
os.environ["MAXB"] = str(int(max([b for i,b in gams.get("b")])))
gams.printLog(f'max value in b is {os.environ["MAXB"]}')
$offEmbeddedCode

$if x%sysEnv.MAXB%==x $abort MAXB is not set
Set k "from 0 to max(b)" / 0*%sysEnv.MAXB% /;
$eval CARDK card(k)
$log card(k)=%CARDK%

It is important to understand that os.environ is initialized when Python imports the os module. Hence, if we later set an environment variable in GAMS (via $setEnv ABC abc or put_utility "setEnv" / "ABC" / "abc") and access this in some embedded code via os.environ["ABC"], this will not provide the expected value "abc". In such a case the environment variable needs to be accessed via gams.get_env("ABC") (and can be stored in os.environ["ABC"]) as demonstrated by the following code:

$onEmbeddedCode Python:
import os
os.environ["TEST"] = 'Hello world'
$offEmbeddedCode
$log TEST=%sysEnv.TEST%
$setEnv ABC abc
$onEmbeddedCode Python:
gams.printLog(f'ABC={os.environ.get("ABC", "ABC not set")}')
gams.printLog(f'ABC={gams.get_env("ABC")}')
$offEmbeddedCode

which produces the following output:

--- Starting compilation
--- main.gms(4) 2 Mb
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
TEST=Hello world
--- main.gms(10) 2 Mb
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
ABC=ABC not set
ABC=abc

Obviously, the set k can also be filled directly with data from within Python:

Set       i    / i1*i5 /;
Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /;
Set       k(*) "from 0 to max(b)";

$onEmbeddedCode Python:
kmax = max([b for i,b in gams.get("b")])
gams.set('k',[str(k) for k in range(int(kmax+1))])
$offEmbeddedCode k

In both cases the resulting GAMS symbol k is a set with elements 0 to 59.

Note that the (*) in the declaration of k is necessary to inform the compiler of dimensionality of the symbol k. GAMS symbol declarations can be done without domain list, so a set k does not necessarily mean that this is a one dimensional set. The compiler will learn the dimensionality from the use of k. For example, if k shows up as a domain set in a declaration (parameter p(k)) it is clear that k is a one dimensional set. Without knowing the dimensionality k will not be available with embedded code or exported (at compile-time) to GDX and the following code

Set k;
$onEmbeddedCode Python:
gams.set('k',['0','1','2'])
$offEmbeddedCode k

will fail with compilation error Cannot load/unload symbol with unknown dimension.

Multiple Independent Python Sessions

At execution time the user has the ability to pause and continue an embedded code segment. As explained at the beginning, the user Python code lives in Python's global scope. Nevertheless, besides some performance aspects one can also allows work with multiple independent Python sessions. Due to some Python module incompatibilities (e.g. numpy) the independent Python sessions have to be enabled with a command line option pyMultInst set to 1. After the pauseEmbeddedCode we can obtain and store the handle of the last embedded code execution via function embeddedHandle. The handle needs to be supplied when we continue the Python session via continueEmbeddedCode. The different Python sessions are fairly separate as shown in the example below. Here we save the GAMS scalar ord(i) in a Python object i five times. The value for i that we store in the five different Python sessions is 1 to 5. In the subsequent loop we activate the Python session with the appropriate handle and print the value of i:

$if not %sysEnv.GMSPYTHONMULTINST%==1 $abort.noError Start with command line option pyMultInst=1
Set i / i1*i5 /; 
Parameter h(i) 'session handle';
loop(i,
  put_utility 'setEnv' / 'ORDI' / ord(i):0:0;
  embeddedCode Python:
  i = int(gams.get_env("ORDI"))
  gams.printLog(f'{i}')
  pauseEmbeddedCode
  h(i) = embeddedHandle;
);
loop(i,
  continueEmbeddedCode h(i):
  gams.printLog(f'{i}')
  endEmbeddedCode
);

The GAMS log shows the value of i in the different Python sessions:

--- Starting execution: elapsed 0:00:00.002
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
1
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
2
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
3
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
4
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
5
--- Execute embedded library embpycclib64.dll
1
--- Execute embedded library embpycclib64.dll
2
--- Execute embedded library embpycclib64.dll
3
--- Execute embedded library embpycclib64.dll
4
--- Execute embedded library embpycclib64.dll
5
*** Status: Normal completion

Troubleshooting Embedded Python Code

The GAMS compiler ensures that the number of errors during execution time is minimized. While the logic of the GAMS program might be flawed there is nothing (with a few exceptions) that the GAMS system cannot execute. This is different if we embed foreign code in a GAMS program. The GAMS compiler does not understand the foreign code syntax and just skips over it. Only when the code is executed we will find out if everything works as expected. If the embedded code contains some (Python) syntax errors the Python parser will inform us about this and the message will appear in the GAMS log. For example, the following Python code using the gams.printLog function two times will generate a syntax error:

$onEmbeddedCode Python:
gams.printLog('hello')
 gams.printLog('world...')
$offEmbeddedCode

The GAMS log will provide some guidance:

--- Execute embedded library libembpycclib64.so
  File "<string>", line 7
    gams.printLog('world...')
    ^
IndentationError: unexpected indent
Python error! Return code from Python execution: -1
*** Error executing embedded code section:
*** Check log above

Moreover, if the Python code raises an exception which is not handled within the code this will also lead to a compilation or execution error in GAMS depending at what phase the embedded code is executed.

embeddedCode Python:
raise Exception('something is wrong')
endEmbeddedCode

will produce the following GAMS log and an execution time error:

--- Initialize embedded library libembpycclib64.so
--- Execute embedded library libembpycclib64.so
Exception from Python (line 0): something is wrong
*** Error at line 1: Error executing "embeddedCode" section: Check log above

Note: It is good practice to raise a Python exception if an error occurs. In any case using exit() needs to be avoided since it terminates the executable in an uncontrolled way.

The Python code is executed as part of the GAMS process and GAMS gives control to the Python interpreter when executing the embedded code. So in the worst case if the Python interpreter crashes, the entire GAMS process will crash. Therefore, it is important to be able to test and debug the embedded Python code independent of GAMS. In the following examples we call the Python interpreter executable as part of a GAMS job. In principle this can be tested and debugged completely independent of GAMS where a GDX file represents the content of the GAMS database.

In the first example we mimic the embedded code facility at compile time by exporting the entire GAMS database to a GDX file debug.gdx. With $on/offEcho we write the embedded code with a few extra lines at the top and bottom surrounded by a try/except block and execute the Python interpreter via $call. One of the extra lines at the end of the embedded code triggers the creation of a GDX result file debugOut.gdx which can be imported in subsequent $gdxin/$load commands.

Set i /i1*i10/
    p(i,i) permutation;

$gdxOut debug.gdx
$unload
$gdxOut

$onEcho > debug.py
from gamsemb import *
gams = ECGAMSDatabase(r'%gams.sysdir% '[:-2], 'debug.gdx')
try:
   import random
   i = list(gams.get("i"))
   p = list(i)
   random.shuffle(p)
   for idx in range(len(i)):
       p[idx] = (i[idx], p[idx])
   gams.set("p", p)
   gmdWriteGDX(gams._gmd,'debugOut.gdx',1);
except Exception as e:
   print(str(e))
$offEcho

$call.checkErrorLevel ="%gams.sysdir%GMSPython/python" debug.py

$gdxIn debugOut.gdx
$loadDC p

Option p:0:0:1;
Display p;

In the second example we mimic the embedded code facility at execution time by exporting the entire GAMS database to a GDX file debug.gdx via execute_unload 'debug.gdx';. We write the embedded code with a few extra lines at the top and bottom surrounded by a try/except block via the put facility and execute the Python interpreter via execute. Identical to the compile time example, we export the result to the GDX file debugOut.gdx which can be imported via the execute_load statement.

Set i /i1*i10/
    p(i,i) permutation;

execute_unload 'debug.gdx';
file fpy / 'debug.py' /; put fpy;

$onPutS
from gamsemb import *
gams = ECGAMSDatabase(r'%gams.sysdir% '[:-2],'debug.gdx')
try:
   import random
   i = list(gams.get("i"))
   p = list(i)
   random.shuffle(p)
   for idx in range(len(i)):
       p[idx] = (i[idx], p[idx])
   gams.set("p", p)
   gmdWriteGDX(gams._gmd,'debugOut.gdx',1);
except Exception as e:
   print(str(e))
$offPut

putClose fpy;
execute.checkErrorLevel '="%gams.sysdir%GMSPython/python" debug.py';

execute_load 'debugOut.gdx', p;
Option p:0:0:1;
Display p;

Automatic Indentation

All embedded Python code is automatically wrapped in a try-except block that requires an indentation (two spaces). While this additional indentation makes no difference for most Python code, it does make a difference when it comes to multi-line strings. Therefore, multi-line strings created with triple quotes (either single or double quotes) are excluded from this rule to avoid unintentional spaces, e.g. when writing to a file:

$onEmbeddedCode Python:
f = open("file.txt", "w")
f.write('''some
multi
line
string
''')
f.close()
$offEmbeddedCode

Still, when using the line continuation character (\) in combination with strings, one has to be aware of the extra spaces that are added automatically:

$onEmbeddedCode Python:
s = "some\
multi\
line\
string"
gams.printLog(s)
$offEmbeddedCode

This will print some multi line string instead of somemultilinestring due to the automatically applied indentation.

Performance Considerations of Embedded Python Code

If the same embedded code section (e.g. in a loop) is executed many times there are a few considerations to be taken into account in order to get the best performance. For this we will experiment with the example from the introduction. We look for a random permutation of a set i. In addition we have a cost matrix c(i,ii) and we are looking for the least cost permutation. We should just formulate this as matching in a bi-partite graph but in order to demonstrate some performance considerations we will repeatedly call the Python code that provides a random permutation and we will evaluate the cost of the permutation in GAMS and store the value of the cheapest one. Here is the naive implementation using embedded code:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation; 
c(i,ii) = uniform(-50,50);

Set    iter / 1*100 /;
Scalar tcost
       minTCost / +inf /;
loop(iter,
  embeddedCode Python:
  import random
  i = list(gams.get("i"))
  p = list(i)
  random.shuffle(p)
  gams.set("p", [ (i[idx], p[idx]) for idx in range(len(i)) ])
  endEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
Display minTCost;

Even though the Python interpreter stays alive the fresh start and end of an embedded code section goes together with the setup and initialization of multiple data structure, including the gams object of type ECGAMSDatabase. We can avoid the repeated setup and initialization by using pause and continue:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation;
c(i,ii) = uniform(-50,50);

embeddedCode Python:
import random
pauseEmbeddedCode

Set    iter / 1*100 /;
Scalar tcost
       minTCost / +inf /;
loop(iter,
  continueEmbeddedCode:
  i = list(gams.get("i"))
  p = list(i)
  random.shuffle(p)
  gams.set("p", [ (i[idx], p[idx]) for idx in range(len(i)) ])
  pauseEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
continueEmbeddedCode:
pass
endEmbeddedCode
Display minTCost;

The last embedded code execution of the Python pass statement is to clean up and terminate the Python session. Since the amount of data is relatively small, there is little difference in the running time between these variants. The performance impact of independent Python session (command line option pyMultInst) is significant. While the pause/continue code runs at the same speed (roughly 0.2 seconds) independent of the pyMultInst setting, the original code runs much slower (about 6.6 seconds) because every iteration results in a newly created session.

There are other performance considerations. We can actually extract the set i once before the GAMS loop starts. Moreover, we can work with label indexes rather than the labels itself. Indexes are integers and are often faster than labels that are stored as strings. The only difference to the code above is the extraction method of the Python list i by i = list(gams.get("i",keyType=KeyType.INT)).

Extending GMSPython

While we recommend to use your own Python installation if you need additional packages, there are ways to extend the Python system that comes with GAMS in GMSPython. Here are the steps:

Get pip via get-pip (https://pip.pypa.io/en/stable/installing/)
Update pip
Install additional packages

We highly recommend installing pip and additional packages in the user site (use --user when running get-pip and pip). This is especially important for macOS users. Installing files in the GAMS system directory may interfere with the file notarization and may prevent GAMS from starting properly. Here is a typical dialog for Linux/macOS using curl to download get-pip.py. If you don't have curl or wget, use your web browser for downloading.

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
/path/to/gams/GMSPython/python get-pip.py --user
~/.local/bin/pip install -U pip
~/.local/bin/pip install geocoder --user

Using an External Python 3 Installation

Let's assume you don't want to work with the GAMS distributed GMSPython but you want to connect GAMS with a Python 3 installation of your liking. In order for the Embedded Code Facility to work, you need to make the alternative Python interpreter aware of the required modules provided by GAMS by following the Getting started steps in the Python API tutorial.

In addition to this you need to do one more step to allow GAMS to find your Python installation when executing Embedded Python code: you need to point to the Python dynamic load library or shared object. On Windows this file is called python3X.dll, on Linux libpython3.X.so, and on macOS libpython3.X.dylib. The X stands for the minor version number of Python 3. While embedded code has been built and tested for Python 3.8, chances are that you can also point to other Python 3 libraries. Since the expert level API is required, the set of versions is limited to what GAMS supports (currently GAMS ships these for Python >=3.6). Limited experiments with the embedded code library for Python 3.8 with Python 3.6, 3.7, and 3.9 libraries were successful. Python 3.10 is not supported yet. You need to set the environment variable GMSPYTHONLIB to point GAMS to the Python library:

Windows:
set GMSPYTHONLIB=c:\path\to\python\python38.dll
Linux:
export GMSPYTHONLIB=/path/to/python/lib/libpython3.8.so
Mac OSX:
export GMSPYTHONLIB=/path/to/python/lib/libpython3.8.dylib

GAMS will extract the Python home from the location of the Python library. In case this extraction does not work (because of an unconventional Python installation), you can instruct GAMS to set the Python home via the additional environment variable GMSPYTHONHOME. Please note, that this environment variable only needs to be set for GAMS Embedded Python code, so it is sufficient to add these variables to the GAMS configuration YAML file.

Note: When using environment-based Python distributions like Anaconda or Miniconda, it is not sufficient to set GMSPYTHONLIB (and GMSPYTHONHOME). The environment to be used needs to be activated properly before (e.g. conda activate myEnv). Packages like numpy which have been installed using conda (conda install numpy) won't work otherwise. In addition to setting GMSPYTHONLIB we therefore recommend to start GAMS itself and also tool like GAMS Studio from an activated conda environment in order to make sure that the environment and all installed packages work as expected.

Although it is recommended to use a self-contained Python installation, it is possible to use a virtual environment instead (e.g. via venv). In this case, GMSPYTHONLIB has to be set to the Python library of the installation from which the environment was derived from. In additon, the PYTHONPATH environment variable needs to be set to the site-packages directory of the virtual environment in order to make its packages available. Note that with this setup the site-packages directory of the original Python installation will also be found by GAMS Embedded Python code.

Building your own Embedded Python Code Library

Although the Embedded Code Facility and its binary components are part of the GAMS distribution, it is possible to build it manually from source using the following commands. The exact command line might change depending on the compiler and the operating system in use:

Windows:

cd [GAMS directory]/apifiles/C/api
icl.exe -Feembpycclib64.dll -IC:\path\to\python\include embpyoo.c gmdcc.c gclgms.c -LD -link -nodefaultlib:python38.lib

Linux:

cd [GAMS directory]/apifiles/C/api
gcc -o libembpycclib64.so -nostartfiles -shared -Wl,-Bsymbolic -m64 -pthread -fPIC -Ipath/to/your/python/include/python3.8 embpyoo.c gmdcc.c gclgms.c -lm -ldl

Mac OS X:

cd [GAMS directory]/apifiles/C/api
gcc -o libembpycclib64.dylib -dynamiclib -shared -m64 -install_name @rpath/libembpycclib64.dylib -Ipath/to/your/python/include/python3.8 embpyoo.c gmdcc.c gclgms.c -lm -ldl

Connect

The Connect framework gives unified and platform independent access to data exchange with different formats (e.g. CSV and Excel). The instructions how to access the various data sources are given in YAML syntax. These YAML instructions are the code for the embedded Connect code. The embedded Connect code engine will parse the YAML instructions and use Connect agents to act upon the instructions. The agents GAMSReader and GAMSWriter send data back and forth to GAMS. A simple example is given above.

Embedded code Connect is build on top of embedded code Python. Similar considerations (e.g. Python interpreter is loaded once) carry over from Python to Connect. Moreover, the agent PythonCode has access to the object gams of type ECGamsDatabase which is used for communication of data with GAMS. In contrast to embedded Python code, the code of the agent PythonCode does not execute in the global scope but in a separate block. This means that variables set in the code are not available in another PythonCode section. The following code ends up in the except clause and prints Cannot access my_i.

embeddedCode Connect:
- PythonCode:
    code: |
        my_i = 1
- PythonCode:
    code: |
        try:
          gams.printLog(f'{my_i}')
        except:
          gams.printLog('Cannot access my_i')
endEmbeddedCode

Since Python allows to add attributes to an object at runtime, the my_i variable can be added to the gams object and hence becomes available in different PythonCode sections and even in different embedded Connect code parts (when pause/continue is used):

embeddedCode Connect:
- PythonCode:
    code: |
        gams.my_i = 1
- PythonCode:
    code: |
        gams.printLog(f'{gams.my_i}')
pauseEmbeddedCode
continueEmbeddedCode:
- PythonCode:
    code: |
        gams.printLog(f'{gams.my_i}')
endEmbeddedCode

Even though embedded code Python and Connect share the same basis, they run in different instantiations of the Python interpreter and hence don't share e.g. the same gams object.

Substitutions in Embedded Connect Code

The GAMS compiler has mechanisms to substitute compile-time variables to parameterize GAMS code including embedded code (unless the V, e.g. $onEmbeddedCodeV, of embedded code is used). Here is an example how to use double-dash parameters to allow the user to set the CSV file name via the GAMS command line (gams mymodel.gms --csvfilename=mystockprice.csv):

$if not set CSVFILENAME $set CSVFILENAME stockprice.csv
$onEmbeddedCode Connect:
- CSVReader:
    file: %CSVFILENAME%
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
- GAMSWriter:
    symbols:
      - name: stockprice
$offEmbeddedCode

Not only user defined compile-time variables will be substituted, but also all other compile time variables, e.g. %gams.scrDir% or %system.dirSep%. Another way to pass substitution instructions down to embedded Connect code are the embedded code arguments:

$if not set MYCSVFILENAME $set MYCSVFILENAME stockprice.csv
$onEmbeddedCodeV Connect: --CSVFILENAME=%MYCSVFILENAME%
- CSVReader:
    file: %CSVFILENAME%
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
- GAMSWriter:
    symbols:
      - name: stockprice
$offEmbeddedCode

Together with the put_utility command ECArguments that appends some arbitrary text to the embedded code arguments this can be used to parameterize the embedded Connect code at execution time, e.g. to create files based on label names:

Set i /1*5/; Parameter p(i);
loop(i,
   p(i) = uniform(0,1);
   put_utility 'ECArguments' / '--CSVFILENAME=' i.tl:0 '.csv';
   EmbeddedCodeV Connect:
   - GAMSReader:
       symbols:
         - name: p
   - CSVWriter:
       name: p
       file: %CSVFILENAME%
   endEmbeddedCode
);

Please note that while the GAMS command line parameters allow various formats to specify double-dash parameters, the embedded code Connect substitution arguments must be in the form --key=value, so a = character to separate key and value with no spaces. The value might be a (double) quoted string though. Please also note that the embedded code arguments set with put_utility command ECArguments will be applied to any subsequent embedded code section. So one might want to clear the argument by adding a put_utility "ECArguments" / ""; after the relevant embedded code section.

Table of Contents

Motivation

Concept

Simple Example

Python

Connect

Syntax

Compile Time

Execution Time

Python

Interface between GAMS and Python

Using the Object-oriented API

Exchange via Files and Environment Variables

Multiple Independent Python Sessions

Troubleshooting Embedded Python Code

Automatic Indentation

Performance Considerations of Embedded Python Code

Extending GMSPython

Using an External Python 3 Installation

Building your own Embedded Python Code Library

Connect

Substitutions in Embedded Connect Code