Motivation

GAMS uses relational data tables as a basic data structure. With these, GAMS code for parallel assignment and equation definition is compact, elegant, and efficient. However, traditional data structures (arrays, lists, dictionaries, trees, graphs, ...) are not natively or easily available in GAMS. Though it is possible to represent such data structures in GAMS, the GAMS code working with such structures can easily become unwieldy, obfuscating, or inefficient. Also, in the past it was not easy to connect libraries from other systems for special algorithms (e.g. graph algorithms, matrix operations, ...) to GAMS without some data programming knowledge and a deep understanding of internal representation of GAMS data (e.g. the GDX API).

The Embedded Code Facility addresses this need and extends the connectivity of GAMS to other programming languages. It allows the use of external code (e.g. Python) during compile and execution time. GAMS symbols are shared with the external code, so no communication via disk is necessary. It utilizes an API (this API can be found in apifiles/C/api/emblib*.h together with the code for the Python and Connect embedded code library apifiles/C/api/embpyoo.c/h) and additional source code for common tasks so that the user can concentrate on the task at hand and not the mechanics of moving data in and out of GAMS.

Concept

As pointed out in section Motivation, the main idea of the Embedded Code Facility is to enable the use of external code in GAMS and give this code direct in-memory access to the GAMS database (or better: to GAMS symbols, namely sets, parameters, variables, and equations). This can be done by defining sections in the GAMS code which contain code written in a defined external code. These sections can be used at both GAMS compile and execution time (compare section GAMS Compile Time and Execution Time Phase). Details about how to do this can be found in the Syntax section.

Note: We will continuously extend this feature to different programming languages. At the moment only Python, Connect, and GAMS are supported.

Also, the system provides some help to develop and debug the external code independent of GAMS first. More about this topic can be found in section Troubleshooting Embedded Python Code.

The communication of the data between the GAMS part of the program and the embedded code part was inspired by the existing interface to GDX in many ways. For example, using dollar control options $on/offMulti, $onMultiR, and $on/offFiltered it is possible to decide if data changed in the embedded code should replace the GAMS data or get merged with it and just like loading data from GDX, one can decide if data from embedded code should change the GAMS database filtered or domain checked.

Simple Example

Python

In a very first example we look into some Python code that helps to split some label names that are already present in GAMS. We do this at compile time in order to read the broken up pieces as individual sets (country and city) into GAMS plus some mapping sets (mccCountry and mccCity) between the original labels and the new labels. The compile-time embedded code starts with a $onEmbeddedCode followed by the type of code to expect (Python:). The lines between $onEmbeddedCode and $offEmbeddedCode is Python code. We do not want to go into Python details, but the first few lines initialize some empty Python list objects (mccCountry and mccCity) as well as some empty Python set objects. In the Python for loop that follows we iterate over all individual labels of the GAMS set cc. Python gets access to the GAMS set cc via the member function get of the implicitly defined Python object gams. get returns an object that is iterable and can be used in a Python for loop. The type of the records one gets depend on the dimensionality and type of the GAMS symbol. In the loop body we use the Python split function to extract the first (r[0] is country) and second (r[1] is city) part of the label. The three strings cc, r[0], r[1] are used to build up the Python list objects that store the information for the maps mccCountry and mccCity and the Python set objects that store the labels for the new sets country and city. The Python set has the advantage to store a label just once even if we add it multiple times. Python prepares to send items back to GAMS via the gams member function set that can deal with both Python list and Python set objects. The command $offEmbeddedCode is followed by a list (without separating commas) of GAMS symbols that instructs the GAMS compiler to read these symbols back. This list is optional and can be left out. GAMS will import all symbols set by the Python code. The GAMS compiler (without executing execution-time embedded code) does not know which symbols will be implicitly loaded, hence it cannot define symbols. If such a symbol is referenced further down in the compilation process, the compiler will complain with error $141: Symbol declared but no values have been assigned. One can avoid this situation by explicitly list the symbols on the end/pauseEmbeddedCode line.

Another caveat with implicit loading is related to aliases. GAMS does not allow to load data into an alias, so the following code will fail with a compilation error 493 Alias cannot receive data:

Set i(*); Alias (i,j);
$onEmbeddedCode Python:
gams.set('j',['i1','i2','i3'])
$offEmbeddedCode j

By leaving off the symbol j from the $offEmbeddedCode line, the code works fine and actually imports the set i. The gams.set and other write methods in embedded code to an alias symbol, here j, actually fills the aliased set, here i, and hence the implicit load will load the modified symbol i into GAMS.

Implicit loading also needs special consideration in the context of external input. Since embedded code inside an $on/offExternalInput section is not executed if the command line parameter IDCGDXInput has been set, there will be no symbols to be loaded implicitly. So make sure the symbols to load are given explicitly in such a situation.

Note that GAMS syntax is case insensitive while Python is case sensitive. Hence, the strings that represent the GAMS symbol names in the Python code can have any casing (e.g. gams.get("cc") or gams.get("CC")), while the corresponding Python objects need to have consistent casing throughout the Python code.

The following code presents the entire embeddedSplit example from the GAMS Data Utilities Library:

Set cc      / "France - Paris", "France - Lille", "France - Toulouse"
              "Spain - Madrid", "Spain - Cordoba", "Spain - Seville", "Spain - Bilbao"
              "USA - Washington DC", "USA - Houston", "USA - New York",
              "Germany - Berlin", "Germany - Munich", "Germany - Bonn" /
    country 
    city
    mccCountry(cc,country<)  Mapping between country and related elements in set cc
    mccCity(cc,city<)        Mapping between city and related elements in set cc;

$onEmbeddedCode Python:
mccCountry = []
mccCity = []
for cc in gams.get("cc"):
  r = str.split(cc, " - ", 1)
  mccCountry.append((cc,r[0]))
  mccCity.append((cc,r[1]))
gams.set("mccCountry",mccCountry)
gams.set("mccCity",mccCity)
$offEmbeddedCode mccCountry mccCity

Option mccCountry:0:0:1, mccCity:0:0:1;
Display country, city, mccCountry, mccCity;

The display in the listing file looks as follows:

----     25 SET country
Spain  ,    USA    ,    Germany,    France 

----     25 SET city
Berlin       ,    Bilbao       ,    Cordoba      ,    Madrid
New York     ,    Washington DC,    Paris        ,    Houston
Munich       ,    Lille        ,    Seville      ,    Bonn
Toulouse

----     25 SET mccCountry
France - Paris     .France 
France - Lille     .France 
France - Toulouse  .France 
Spain - Madrid     .Spain
Spain - Cordoba    .Spain
Spain - Seville    .Spain
Spain - Bilbao     .Spain
USA - Washington DC.USA
USA - Houston      .USA
USA - New York     .USA
Germany - Berlin   .Germany
Germany - Munich   .Germany
Germany - Bonn     .Germany

----     25 SET mccCity
France - Paris     .Paris
France - Lille     .Lille
France - Toulouse  .Toulouse
Spain - Madrid     .Madrid
Spain - Cordoba    .Cordoba
Spain - Seville    .Seville
Spain - Bilbao     .Bilbao
USA - Washington DC.Washington DC
USA - Houston      .Houston
USA - New York     .New York
Germany - Berlin   .Berlin
Germany - Munich   .Munich
Germany - Bonn     .Bonn

The second example demonstrates the use of embedded code at execution time. The syntax for the execution time embedded code in this example is identical to the compile time variant with the exception of the keywords that start and end the embedded code section: embeddedCode and endEmbeddedCode. It is important to understand that the execution of the code happens at GAMS execution time, so e.g. no new labels can be produced and send back to GAMS. In this example we use some Python code to generate a random permutation of set elements of set i and store this in a two dimensional set p. In this example we do not use a loop to iterate through the elements of a GAMS system but make use of the fact that the Python object returned by gams.get("i") is iterable and can in its entirety be stored in the Python list with name i with the short and powerful command i = list(gams.get("i")). The permutation of elements in list p which is a copy of list i is created by the Python statement random.shuffle(p). The following code presents the entire example:

Set i /i1*i10/
    p(i,i) "permutation";

embeddedCode Python:
import random
i = list(gams.get("i"))
p = list(i)
random.shuffle(p)
for idx in range(len(i)):
    p[idx] = (i[idx], p[idx])
gams.set("p", p)
endEmbeddedCode p

option p:0:0:1; 
display p;

The display in the listing file looks as follows:

----     11 SET p  permutation
i1 .i1 
i2 .i7 
i3 .i5 
i4 .i2 
i5 .i10
i6 .i6 
i7 .i9 
i8 .i4 
i9 .i8 
i10.i3

Connect

In the following example, we need to process a CSV file that cannot be read directly by GAMS under $onDelim. So we instruct Connect to read the CSV file into the Connect database and then write to GAMS:

Set dates, stocks;
Parameter stockprice(dates<,stocks<);

$onText
Read a CSV file that looks as follows:

date;symbol;price
2016/01/04;AAPL;105,35
2016/01/04;AXP;67,59
2016/01/04;BA;140,50
...

and can't be read directly by GAMS under $onDelim because some labels
include '/' and are not quoted, use ';' as field separator, and ',' as
decimal character.
$offText

$onEmbeddedCode Connect:
- CSVReader:
    file: stockprice.csv
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
    fieldSeparator: ';'
    decimalSeparator: ','
- GAMSWriter:
    symbols:
      - name: stockprice
$offEmbeddedCode

display stockprice;

We leave the CSV file processing to the CSVReader and then export the stockprice symbol to GAMS. Please note that the symbols written to GAMS by the GAMSWriter do not have to be explicitly listed on the $offEmbeddedCode line, this is optional. Moreover, the Connect agents GAMSReader and GAMSWriter are only available under embedded code.

We might use the data to calculate an optimal portfolio. The share of each stock in this optimal portfolio is represented by the parameter share(stocks) and can be exported to Excel using the ExcelWriter. Before exporting to Excel, the data first needs to be transferred from GAMS to Connect using the GAMSReader:

* Continues the code from the previous example

Parameter share(stocks);
* For demonstration we fill share with random numbers
share(stocks) = uniform(0,1);

embeddedCode Connect:
- GAMSReader:
    symbols:
      - name: share
- ExcelWriter:
    file: share.xlsx
    symbols:
      - name: share
endEmbeddedCode

GAMS

In the following example, we need some execution time GAMS power to create data needed at compilation. We want to solve the bin packing problem. In order to solve this we need the set of bins which is not part of the input data. There are different ways of dealing with this not so uncommon issue and they are all given in the GAMS Model Library model binpacking, but we want to concenrate here on a solution that utilizes embedded code GAMS:

Set i 'items';
Parameter s(i<) 'item sizes';
Scalar B 'bin capacity';
$gdxLoad data.gdx s B

* Randomize the item sizes:
s(i) = uniform(0.9,1.1)*s(i);

Scalar nj 'number of bins required to pack all items';
$save.keepCode bp

$onEmbeddedCode GAMS: restart=bp
scalar size /0/;
nj = 1; loop(i, size = size+s(i); if (size>B, nj = nj+1; size = s(i)));
$offEmbeddedCode nj

$eval NJ nj
Set j 'bins' / b1*b%NJ% /;

* Simple optimization model to minimize number of bins:
Binary variable y(i,j) 'assignment of item to bin', z(j) 'bin open';
Variable open_bins 'number of open bins';

Equations
    defopen(j) 'allow item in open bins only'
    defone(i)  'assign each item to one bin'
    defobj     'count number of open bins';

defopen(j)..   sum(i, s(i)*y(i,j)) =l= z(j)*B;
defone(i)..    sum(j, y(i,j)) =e= 1;
defobj..       open_bins =e= sum(j, z(j));

model bp 'bin packing' /all/;
solve bp min open_bins using mip;

In this example we read the item sizes and the bin capacity from a GDX file and need to create the set of bins j. We even randomize the item sizes (at GAMS execution time). In the embedded code GAMS section we utilize a simple way to estimate the number of bins: we pack the items (in data entry order) in a bin and as soon as the bin capacity is exceeded, we open the next bin. Obviously, this is far from being an optimal strategy, but it allows based on the actual sizes to quickly overestimate the number of bins required for the optimization model. The scalar of the number of bin (nj) is communicated back to the outer GAMS process and we use this to set a compile time variable NJ that is used to build the set of bins j. The embedded code GAMS that calculates nj can access the items and its sizes without declaration or data statements. This is possible because the embedded code continues via embedded code arguments restart=bp from a save file created one line above with $save.keepCode bp. Here the save file is created at compile time via $save and the execution code compiled so far is kept in the save file due to suffix .keepCode. So when the embedded code GAMS section runs, it first executes the code from the save file (here s(i) = uniform(0.9,1.1)*s(i);) before it executes its own code (nj=1; loop(i, ...);). Without the suffix .keepCode the embedded code GAMS would have used the item sizes available at compile time, i.e. the sizes as we find them in the GDX file. The remaining part of the model is a straight forward binary linear model for the bin packing problem.

Syntax

This section explains the GAMS functions/keywords which were introduced to enable the Embedded Code Facility. The first subsection deals with the syntax for compile time, the second with the syntax for execution time (compare section GAMS Compile Time and Execution Time Phase).

Compile Time

There are three dollar control options to start an embedded code section at compile time:

$onEmbeddedCode[.tag]  [Connect|Python]: [arguments]
$onEmbeddedCodeS[.tag] [Connect|Python]: [arguments]
$onEmbeddedCodeV[.tag] [Connect|Python]: [arguments]

Lines following one of the above statements are passed on to the embedded code engine (e.g. Connect or the Python interpreter) until this dollar control option, which ends the embedded code section at compile time, is hit:

$offEmbeddedCode[.tag] {symbol[<[=]embSymbol[.dimX]]}

These dollar control options are explained here in more detail. The usage of the optional arguments by the embedded code is explained in the specific embedded code engine section below.

Note

The optional arguments from the $onEmbeddedCode[S|V][.tag] statement can be accessed as gams.arguments in the Python code.
The optional output symbols from the $offEmbeddedCode[.tag] statement need to be written to by the embedded code.
More about the specific GAMS Python syntax can be found below.

Execution Time

At execution time an embedded code section is started with one of these statements:

embeddedCode[.tag]  [Connect|Python]: [arguments]
embeddedCodeS[.tag] [Connect|Python]: [arguments]
embeddedCodeV[.tag] [Connect|Python]: [arguments]

Similar to the compile time alternatives $onEmbeddedCode[S|V][.tag], the first two variants are synonyms which allow parameter substitution in the code that follows, while the last variant does not allow this but passes the code verbatim to the embedded code engine. The usage of the optional arguments by the embedded code is explained in the specific embedded code engine section below. The arguments passed to the embedded code engine at execution time can be extended by the put_utility command ECArguments. For example,

put_utility 'ECArguments' / ' world...';
embeddedCode  Python: Hello
gams.printLog(gams.arguments)
endEmbeddedCode

prints Hello world....

Lines following one of the statements embeddedCode[S|V] are passed on to the embedded code engine (e.g. the Python interpreter) until one of the following two statements, which end the embedded code section at execution time, is hit:

endEmbeddedCode[.tag]   {output symbols}
pauseEmbeddedCode[.tag] {output symbols}

When the optional tag suffix (which can be any string) is used to start an embedded code section, the same one has to be used to end/pause that section. Both statements end the embedded code section and switch back to GAMS syntax in the following lines. Also, both statements can be followed by a GAMS symbol or a list of GAMS symbols which would get updated in the GAMS database after the embedded code got executed. If output symbols are specified, they need to be written to (e.g. by gams.set for embedded Python code) by the embedded code before.

To continue a previously paused embedded code section one of the following statements is used:

continueEmbeddedCode[.tag]  [handle]: [arguments]
continueEmbeddedCodeS[.tag] [handle]: [arguments]
continueEmbeddedCodeV[.tag] [handle]: [arguments]

As seen before, the first two variants are synonyms which allow parameter substitution in the embedded code that follows, while the last variant does not allow this but passes the code verbatim to the interpreter. Again, the usage of the optional arguments by the embedded code is explained in the specific embedded code engine section below.

New in these statements is the optional handle. If omitted, the last code section that was paused will be continued. However, sometimes one might need to maintain different embedded code sections active in parallel and independent of each other. There is a new function to store a handle of the last embedded code section that was executed which could then later be used to continue a specific paused code section:

handle = embeddedHandle;

What it exactly means to end or pause/continue an embedded code section depends on the embedded code engine and is explained in more detail with the code engine below. Independent of the embedded code engine it is important to note that if GAMS stops and restarts while executing a solve statement (with solveLink=0) continuing any embedded code section causes the code to fail with the following error message:

    Error executing "continueEmbeddedCode" section
--- (Hint: "Solve" with SolveLink=0 frees previously initialized embedded libraries):
    Error at line 71: No embedded library initialized

This happens because under solvelink=0 the GAMS process terminates when the solver starts up. This causes the loss of the state of the embedded code environment that is linked into the GAMS process as a shared library. Under all other settings of solveLink the GAMS process is not stopped and restarted and hence a paused embedded code environment can be continued after the solve statement has carried out.

Python

GAMS and Python complement each other in many different ways. GAMS has a compact but readable syntax for data assignment and equation definition statements. Python is a great scripting language to do more traditional type programming especially string manipulation which is completely missing in GAMS. Moreover, the vast number of packages help solving scientific problems outside the scope of GAMS. In order to open this world to our customers in an easy way, GAMS comes with a Python 3 installation. By default, this installation is used in the Embedded Code Facility for Python and is ready to be used with the GAMS Python API. The Python installation is located in [GAMS directory]/GMSPython. In contrast to earlier versions this installation comes without the Python package manager pip and therefore is not easily extendable. GMSPython comes with a selection of third party packages and modules that are either required by the APIs of the GAMS Python API collection or that are particularly helpful from within embedded Python code. Those packages are located in the site-packages directory of GMSPython. Running the GMSPython interpreter like follows will display all available modules:

[GAMS directory]/GMSPython/python -c"help('modules')"

The list of redistributed third party packages is subject to constant changes and it is not recommended to rely on their availability. If you need packages that do not come with GMSPython, we recommend that you install your own version of Python and follow a few steps to connect GAMS with your Python installation. It is also possible to install pip for GMSPython using get-pip. While any Python installation should work in combination with GAMS we can recommend the Anaconda or Miniconda Python distributions. These Python distributions work with environments which allow you to have many different Python installations at the same time that don't get in each other's way.

During the lifetime of the GAMS process a single Python interpreter is used and the code "lives" in Python's global scope. This means that objects defined in one embedded Python code section are still there in another embedded Python code section:

$onEmbeddedCode Python:
s = 'Hello world'
$offEmbeddedCode
$onEmbeddedCode Python:
gams.printLog(s)
$offEmbeddedCode

will produce the following output in the GAMS log:

--- Starting compilation
--- main.gms(3) 2 Mb
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- main.gms(6) 2 Mb
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
Hello world
--- main.gms(7) 2 Mb

If the GAMS process stops and restarts during execution, e.g. because of a solve with solveLink=0, the Python interpreter will be started new. This can cause complications if GAMS is in a sequence of pause/continueEmbeddedCode as discussed above.

Note: Sometimes it is desirable to move Python code to separate files which get imported from within an embedded Python code section. By default the current working directory is not considered by Pythons import statement. In order to achieve this, one can add sys.path.append(".") before the actual import.

Interface between GAMS and Python

The Python class ECGamsDatabase is the interface between GAMS and Python. An instance of this class is automatically created when an embedded code section is entered and can be accessed using the identifier gams. The following methods can be used in order to interact with GAMS:

gams.get(symbolName, keyType=KeyType.STRING, keyFormat=KeyFormat.AUTO, valueFormat=ValueFormat.AUTO, recordFormat=RecordFormat.AUTO)

This method retrieves an iterable object representing the symbol identified with symbolName. Typically there are two possibilities to access the records. Iterating using e.g. a for loop provides access to the individual records. By calling list() on the iterable object, a list containing all the data is created. Several optional parameters can be used in order to modify the format of the retrieved data:

keyType: Determines the data type of the keys. It can be either KeyType.STRING (labels) or KeyType.INT (label indexes). The default setting is KeyType.STRING.

keyFormat: Specifies the representation of the keys. Possible values are as follows:

KeyFormat.TUPLE: Encapsulate keys in a tuple

KeyFormat.FLAT: No encapsulation

KeyFormat.SKIP: Keys are skipped and do not appear in the retrieved data

KeyFormat.AUTO (default): Depending on the dimension of the GAMS symbol, a default format is applied:

Zero dimensional/scalar: KeyFormat.SKIP

One dimensional: KeyFormat.FLAT

Multi dimensional: KeyFormat.TUPLE

valueFormat: Specifies the representation of the values. Possible values are as follows:

ValueFormat.TUPLE: Encapsulate values in a tuple

ValueFormat.FLAT: No encapsulation

ValueFormat.SKIP: Values are skipped and do not appear in the retrieved data

ValueFormat.AUTO (default): Depending on the type of the GAMS symbol, a default formats is applied:

Set: ValueFormat.SKIP

Parameter: ValueFormat.FLAT

Variable/Equation: ValueFormat.TUPLE

recordFormat Specifies the encapsulation of records into tuples. Possible values are as follows:

RecordFormat.TUPLE: Encapsulates every record in a tuple

RecordFormat.FLAT: No encapsulation. Throws an exception if it can not be applied. It is guaranteed that the length of a retrieved Python list is equal to the number of records of the corresponding GAMS symbol. This principle leads to an incompatibility of RecordFormat.FLAT whenever a record consists of more than one item (e.g. multi dimensional symbols, variables and equations which have five numeric values).

RecordFormat.AUTO (default): Depending on the number of items that represent a record, a default format is applied. If possible this is always RecordFormat.FLAT.

GAMS special values NA, INF, and -INF will be mapped to IEEE special values float('nan'), float('inf'), and float('-inf'). GAMS special value EPS will be either mapped to 0 or to the small numeric value 4.94066E-324 depending on the setting of flag gams.epsAsZero.

The following Python code shows some examples of gams.get and illustrates the use of different formats:
Set i / i1 text 1, i2 text 2 /
    j / j1*j2 /

Scalar p0 /3.14/;
Parameter p1(i)   / #i 3.14 /
          p2(i,j) / i1.#j 3.14 /
Variable v0       / fx 3.14 /;
equation e1(i)    / #i.fx 3.14 /
         e2(i,j)  / i1.#j.fx 3.14 /;

$onEmbeddedCode Python:
# scalar parameter
gams.printLog(f"{list(gams.get('p0'))}")                                  # prints [3.14]
gams.printLog(f"{list(gams.get('p0', recordFormat=RecordFormat.TUPLE))}") # prints [(3.14,)]

# one dimensional parameters:
gams.printLog(f"{list(gams.get('p1'))}")                                  # prints [('i1', 3.14), ('i2', 3.14)]
gams.printLog(f"{list(gams.get('p1', keyFormat=KeyFormat.TUPLE))}")       # prints [(('i1',), 3.14), (('i2',), 3.14)]
gams.printLog(f"{list(gams.get('p1', valueFormat=ValueFormat.TUPLE))}")   # prints [('i1', (3.14,)), ('i2', (3.14,))]
gams.printLog(f"{list(gams.get('p1', keyFormat=KeyFormat.TUPLE, valueFormat=ValueFormat.TUPLE))}")   # prints [('i1', (3.14,)), ('i2', (3.14,))]

# two dimensional parameter:
gams.printLog(f"{list(gams.get('p2'))}")                                  # prints [(('i1', 'j1'), 3.14), (('i1', 'j2'), 3.14)]
gams.printLog(f"{list(gams.get('p2', keyFormat=KeyFormat.FLAT))}")        # prints [('i1', 'j1', 3.14), ('i1', 'j2', 3.14)]

# one dimensional sets:
gams.printLog(f"{list(gams.get('i'))}")                                   # prints ['i1', 'i2']
gams.printLog(f"{list(gams.get('i',valueFormat=ValueFormat.FLAT))}")      # prints [('i1', 'text 1'), ('i2', 'text 2')]

# scalar variables/equations
gams.printLog(f"{list(gams.get('v0'))}")                                  # prints [(3.14, 0.0, 3.14, 3.14, 1.0)]

# one dimensional variables/equations:
gams.printLog(f"{list(gams.get('e1'))}")                                  # prints [('i1', (3.14, 0.0, 3.14, 3.14, 1.0)), ('i2', (3.14, 0.0, 3.14, 3.14, 1.0))]
gams.printLog(f"{list(gams.get('e1', valueFormat=ValueFormat.FLAT))}")    # prints [('i1', 3.14, 0.0, 3.14, 3.14, 1.0), ('i2', 3.14, 0.0, 3.14, 3.14, 1.0)]
gams.printLog(f"{list(gams.get('e1', keyFormat=KeyFormat.TUPLE))}")       # prints [(('i1',), (3.14, 0.0, 3.14, 3.14, 1.0)), (('i2',), (3.14, 0.0, 3.14, 3.14, 1.0))]

# two dimensional variables/equations:
gams.printLog(f"{list(gams.get('e2'))}")                                  # prints [(('i1', 'j1'), (3.14, 0.0, 3.14, 3.14, 1.0)), (('i1', 'j2'), (3.14, 0.0, 3.14, 3.14, 1.0))]
gams.printLog(f"{list(gams.get('e2', keyFormat=KeyFormat.FLAT, valueFormat=ValueFormat.FLAT))}") # prints [('i1', 'j1', 3.14, 0.0, 3.14, 3.14, 1.0), ('i1', 'j2', 3.14, 0.0, 3.14, 3.14, 1.0)]

# using label indexes instead of labels
gams.printLog(f"{list(gams.get('p1', keyType=KeyType.INT))}")             # prints [(1, 3.14), (2, 3.14)]
gams.printLog(f"{list(gams.get('i', keyFormat=KeyFormat.TUPLE, valueFormat=ValueFormat.TUPLE, keyType=KeyType.INT))}") # prints [((1,), ('text 1',)), ((2,), ('text 2',))]
gams.printLog(f"{list(gams.get('e2', keyType=KeyType.INT))}")             # prints [((1, 3), (3.14, 0.0, 3.14, 3.14, 1.0)), ((1, 4), (3.14, 0.0, 3.14, 3.14, 1.0))]
$offEmbeddedCode  
gams.set(symbolName, data, mergeType=MergeType.DEFAULT, domCheck=DomainCheckType.DEFAULT, mapKeys=lambda x:x)
This method sets the data for the GAMS symbol identified with symbolName. The parameter data takes a Python list or set containing items that represent the records of the symbol. It is also possible to pass an instance of a subclass of _GamsSymbol (e.g. GamsParameter or GamsSet) when using the GAMS control API in an embedded code section. In case of a Python list or set, depending on the type and the dimension of the symbol, different formats can be used in order to specify the data. Different formats can not be mixed within one list. In general each record needs to be represented as a tuple containing the keys and the value field(s). Keys and/or values can also be enclosed in a tuple. Keys can be entered as labels (string) or label indexes (int). The argument mapKeys allows to pass a callable to remap the elements of the key (e.g. turn them explicitly into strings via mapKeys=str). Value fields depend on the type of the symbol:

Parameters: One numerical value

Sets: explanatory text (optional)

Variable/Equations: Five numerical values: level, marginal, lower bound, upper bound, scale/prior/stage

IEEE special values float('nan'), float('inf'), and float('-inf') will be remapped to GAMS special values NA, INF, and -INF. The small numeric value 4.94066E-324 will be mapped into GAMS special value EPS.

The following Python code gives some examples on different valid formats for different symbol types and dimensions:
Set i / i1 text 1, i2 text 2 /
    j / j1*j2 /
    ii(i);
Scalar p0;
Parameter p1(i)
          p2(i,j);
equation e1(i)
         e2(i,j);
$onEmbeddedCode Python:
# scalar parameter
gams.set('p0', [3.14])
gams.set('p0', [(3.14,)])

# one dimensional parameters:
gams.set('p1', [("i1", 3.14), ("i2", 3.14)])
gams.set('p1', [(("i1",), 3.14), (("i2",), 3.14)])
gams.set('p1', [("i1", (3.14,)), ("i2", (3.14,))])
gams.set('p1', [(("i1",), (3.14,)), (("i2",), (3.14,))])

# two dimensional parameter:
gams.set('p2', [('i1', 'j1', 3.14), ('i1', 'j2', 3.14)])
gams.set('p2', [(('i1', 'j1'), (3.14,)), (('i1', 'j2'), (3.14,))])

# one dimensional sets:
gams.set('ii', ['i1', 'i2'])
gams.set('ii', [('i1',), ('i2',)])

# one dimensional sets with explanatory text
gams.set('ii', [('i1', "text 1"), ('i2', "text 2")])
gams.set('ii', [(('i1',), ("text 1",)), (('i2',), ("text 2",))])

# one dimensional variables/equations:
gams.set('e1', [("i1", 3.14, 0, 0, 10, 1), ("i2", 3.14, 0, 0, 10, 1)])
gams.set('e1', [("i1", (3.14, 0, 0, 10, 1)), ("i2", (3.14, 0, 0, 10, 1))])
gams.set('e1', [(("i1",), (3.14, 0, 0, 10, 1)), (("i2",), (3.14, 0, 0, 10, 1))])

# two dimensional variables/equations:
gams.set('e2', [("i1", "j1", 3.14, 0, 0, 10, 1), ("i1", "j2", 3.14, 0, 0, 10, 1)])
gams.set('e2', [(("i1", "j1"), (3.14, 0, 0, 10, 1)), (("i1", "j2"), (3.14, 0, 0, 10, 1))])

# using label indexes instead of labels
gams.set('p1', [((1,), (3.14,)), ((2,), (3.14,))]) # one dimensional parameter
gams.set('ii', [((1,), ("text 1",)), ((2,), ("text 2",))]) # one dimensional set
gams.set('e2', [((1, 3), (3.14, 0, 0, 10, 1)), ((1, 4), (3.14, 0, 0, 10, 1))]) # two dimensional equation/variable
$offEmbeddedCode p0 p1 p2 ii e1 e2
The optional parameter mergeType specifies if data in a GAMS symbol is merged (MergeType.MERGE) or replaced (MergeType.REPLACE). If left at MergeType.DEFAULT it depends on the setting of $on/offMulti[R] if GAMS does a merge, replace, or trigger an error during compile time. During execution time MergeType.DEFAULT is the same as MergeType.MERGE. The optional parameter domCheck specifies if Domain Checking is applied (DomainCheckType.CHECKED) or if records that would cause a domain violation are filtered (DomainCheckType.FILTERED). If left at DomainCheckType.DEFAULT it depends on the setting of $on/offFiltered if GAMS does a filtered load or checks the domains during compile time. During execution time DomainCheckType.DEFAULT is the same as DomainCheckType.FILTERED.

Note
When calling gams.set() in an embedded code section during execution time, new labels that are not known to the current GAMS program can not be added. The attempt will result in an execution error.
gams.getUel(idx)
Returns the label corresponding to the label index idx
gams.mergeUel(label)
Adds label to the GAMS universe if it was unknown and returns the corresponding label index.
Note
When calling gams.mergeUel() in an embedded code section during execution time, new labels that are not known to the current GAMS program can not be added. The attempt will result in an execution error.
gams.getUelCount()
Returns the number of labels.
gams.printLog(msg, end="\n")
Print msg followed by end to log.
gams.arguments
Contains the command line that was passed to the Python interpreter of the embedded code section. The syntax for passing arguments to the Python interpreter can be seen above.
gams.epsAsZero
Flag to read GAMS EPS as 0 (True) or as a small number, 4.94066E-324, when set to False. Default is True.
gams.ws
Property to retrieve an instance of GamsWorkspace that allows to use the GAMS control API. The instance is created when the property is read for the first time using a temporary working directory. A different working directory can be specified using gams.wsWorkingDir. For debug output, the property gams.debug can be set to a value from DebugLevel
gams.wsWorkingDir
Property that can be specified before accessing gams.ws for the first time in order to set the working directory. Setting the property after the first call to gams.ws will have no effect. of the created GamsWorkspace.
gams.db
Property to retrieve an instance of GamsDatabase. The instance is created when the property is read for the first time and allows to access the GAMS symbols using the methods of the GAMS control API.
gams.debug
Property that can be set to a value from DebugLevel for debug output. Default is DebugLevel.Off (0). The property needs to be changed before accessing gams.ws for the first time in order to take effect in the GAMS control API. Setting the property after the first call to gams.ws will have no effect on the GamsWorkspace.

Using the Control API

The ECGamsDatabase class provides mechanisms for using the GAMS control API in an embedded code section. The property gams.ws can be used to get an instance of GamsWorkspace. The property gams.db allows to access an instance of GamsDatabase that can be used to read and write data from the internal GAMS database like it can be done using gams.get and gams.set but using the access mechanisms of the GamsDatabase class. In addition to the different reading and writing mechanisms offered by the GAMS control API other classes like GamsModelInstance can be used inside embedded Python code. This combination provides programmatic access to a generated model instance and exchange of update parameters and results between Python and GAMS. The GAMS Library model embmiex1 gives an example of how to combine embedded Python code and a GamsModelInstance which utilized the library include $libInclude pyEmbMI.

Moreover, the GAMS Transfer Python API can be combine with the GAMS control API in the embedded code context as the following example demonstrates:

Set i / i1*i1000 /; alias(i,j);
Parameter a(i,j), aInv(i,j);
a(i,j) = 1 / (ord(i)+ord(j) - 1);
a(i,i) = 1;

embeddedCode Python:
import gams.transfer as gt
import numpy as np

m = gt.Container(gams.db)
A = m.data["a"].toDense()
m.data['aInv'].setRecords(np.linalg.inv(A))
m.write(gams.db, ["aInv"])
endEmbeddedCode aInv

Exchange via Files and Environment Variables

The Python class ECGamsDatabase provides read and write access to GAMS symbols. There are two other communication methods that can be used at GAMS compile time: files and environment variables. At compile time the Python code can produce a text file that can be included into GAMS via $include as in the following example:

$onEmbeddedCode Python: 10
with open('i.txt', 'w') as f:
  for i in range(int(gams.arguments)):
    f.write(f'i{i+1}\n')
$offEmbeddedCode

Set i /
$include i.txt
/;
display i;

Here the Python code received the number of elements to write to a text file via the argument after Python:. This text file is then included in the data statement of the GAMS set i. The display in the listing file looks as follows:

----     20 SET i
i1 ,    i2 ,    i3 ,    i4 ,    i5 ,    i6 ,    i7 ,    i8 ,    i9 ,    i10

Python provides many packages to read input files for many different formats and hence can be used to transform such formats to a GAMS compatible input format, as an alternative to providing the data via list objects and the gams.set functionality.

The second alternative to exchange information at compile time are environment variables. GAMS and Python allow to get and set environment variables and hence can be conveniently used to exchange small pieces of information. The following code provides an example where the maximum value of a parameter b is needed to build a set k:

Set       i    / i1*i5 /;
Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /;
$onEmbeddedCode Python:
import os
os.environ["MAXB"] = str(int(max([b for i,b in gams.get("b")])))
gams.printLog(f'max value in b is {os.environ["MAXB"]}')
$offEmbeddedCode

$if x%sysEnv.MAXB%==x $abort MAXB is not set
Set k "from 0 to max(b)" / 0*%sysEnv.MAXB% /;
$eval CARDK card(k)
$log card(k)=%CARDK%

It is important to understand that os.environ is initialized when Python imports the os module. Hence, if we later set an environment variable in GAMS (via $setEnv ABC abc or put_utility "setEnv" / "ABC" / "abc") and access this in some embedded code via os.environ["ABC"], this will not provide the expected value "abc". In such a case the environment variable needs to be accessed via gams.get_env("ABC") (and can be stored in os.environ["ABC"]) as demonstrated by the following code:

$onEmbeddedCode Python:
import os
os.environ["TEST"] = 'Hello world'
$offEmbeddedCode
$log TEST=%sysEnv.TEST%
$setEnv ABC abc
$onEmbeddedCode Python:
gams.printLog(f'ABC={os.environ.get("ABC", "ABC not set")}')
gams.printLog(f'ABC={gams.get_env("ABC")}')
$offEmbeddedCode

which produces the following output:

--- Starting compilation
--- main.gms(4) 2 Mb
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
TEST=Hello world
--- main.gms(10) 2 Mb
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
ABC=ABC not set
ABC=abc

Obviously, the set k can also be filled directly with data from within Python:

Set       i    / i1*i5 /;
Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /;
Set       k(*) "from 0 to max(b)";

$onEmbeddedCode Python:
kmax = max([b for i,b in gams.get("b")])
gams.set('k',[str(k) for k in range(int(kmax+1))])
$offEmbeddedCode k

In both cases the resulting GAMS symbol k is a set with elements 0 to 59.

Note that the (*) in the declaration of k is necessary to inform the compiler of dimensionality of the symbol k. GAMS symbol declarations can be done without domain list, so a set k does not necessarily mean that this is a one dimensional set. The compiler will learn the dimensionality from the use of k. For example, if k shows up as a domain set in a declaration (parameter p(k)) it is clear that k is a one dimensional set. Without knowing the dimensionality k will not be available with embedded code or exported (at compile-time) to GDX and the following code

Set k;
$onEmbeddedCode Python:
gams.set('k',['0','1','2'])
$offEmbeddedCode k

will fail with compilation error Cannot load/unload symbol with unknown dimension.

Encodings

Python 3 expects source code to be encoded in UTF-8 per default. Therefore all embedded Python code has to be encoded in UTF-8 as well. However, this can be customized by adding a comment in the format # coding=<encoding name> or # -*- coding: <encoding name> -*- as first line of an Embedded Python Code section. The encoding can be changed from any Embedded Python Code section and will affect all subsequent sections. So in case of a GAMS source file which is encoded using a non-UTF-8 encoding like cp1252 or others, it is sufficient to change the encoding only once in the very first Embedded Code section. Note that UTF-16 encoding is not supported.

Note: Changing the source code encoding has no effect on the default encoding that is used by the open() function since it uses whatever locale.getpreferredencoding(False) returns. Furthermore the encoding used for stdin/stdout/stderr has to be controlled separately. This can be achieved by setting the environment variable PYTHONIOENCODING.

Multiple Independent Python Sessions

At execution time the user has the ability to pause and continue an embedded code segment. As explained at the beginning, the user Python code lives in Python's global scope.

Troubleshooting Embedded Python Code

The GAMS compiler ensures that the number of errors during execution time is minimized. While the logic of the GAMS program might be flawed there is nothing (with a few exceptions) that the GAMS system cannot execute. This is different if we embed foreign code in a GAMS program. The GAMS compiler does not understand the foreign code syntax and just skips over it. Only when the code is executed we will find out if everything works as expected. If the embedded code contains some (Python) syntax errors the Python parser will inform us about this and the message will appear in the GAMS log. For example, the following Python code using the gams.printLog function two times will generate a syntax error:

$onEmbeddedCode Python:
gams.printLog('hello')
 gams.printLog('world...')
$offEmbeddedCode

The GAMS log will provide some guidance:

--- Execute embedded library libembpycclib64.so
  File "<string>", line 7
    gams.printLog('world...')
    ^
IndentationError: unexpected indent
Python error! Return code from Python execution: -1
*** Error executing embedded code section:
*** Check log above

Moreover, if the Python code raises an exception which is not handled within the code this will also lead to a compilation or execution error in GAMS depending at what phase the embedded code is executed.

embeddedCode Python:
raise Exception('something is wrong')
endEmbeddedCode

will produce the following GAMS log and an execution time error:

--- Initialize embedded library libembpycclib64.so
--- Execute embedded library libembpycclib64.so
Exception from Python (line 0): something is wrong
*** Error at line 1: Error executing "embeddedCode" section: Check log above

Note: It is good practice to raise a Python exception if an error occurs. In any case using exit() needs to be avoided since it terminates the executable in an uncontrolled way.

The Python code is executed as part of the GAMS process and GAMS gives control to the Python interpreter when executing the embedded code. So in the worst case if the Python interpreter crashes, the entire GAMS process will crash. Therefore, it is important to be able to test and debug the embedded Python code independent of GAMS. In the following examples we call the Python interpreter executable as part of a GAMS job. In principle this can be tested and debugged completely independent of GAMS where a GDX file represents the content of the GAMS database.

In the first example we mimic the embedded code facility at compile time by exporting the entire GAMS database to a GDX file debug.gdx. With $on/offEcho we write the embedded code with a few extra lines at the top and bottom surrounded by a try/except block and execute the Python interpreter via $call. One of the extra lines at the end of the embedded code triggers the creation of a GDX result file debugOut.gdx which can be imported in subsequent $gdxin/$load commands.

Set i /i1*i10/
    p(i,i) permutation;

$gdxOut debug.gdx
$unload
$gdxOut

$onEcho > debug.py
from gams.core.embedded import *
gams = ECGAMSDatabase(r'%gams.sysdir% '[:-2], 'debug.gdx')
try:
   import random
   i = list(gams.get("i"))
   p = list(i)
   random.shuffle(p)
   for idx in range(len(i)):
       p[idx] = (i[idx], p[idx])
   gams.set("p", p)
   gmdWriteGDX(gams._gmd,'debugOut.gdx',1);
except Exception as e:
   print(str(e))
$offEcho

$call.checkErrorLevel ="%gams.sysdir%GMSPython/python" debug.py

$gdxIn debugOut.gdx
$loadDC p

Option p:0:0:1;
Display p;

In the second example we mimic the embedded code facility at execution time by exporting the entire GAMS database to a GDX file debug.gdx via execute_unload 'debug.gdx';. We write the embedded code with a few extra lines at the top and bottom surrounded by a try/except block via the put facility and execute the Python interpreter via execute. Identical to the compile time example, we export the result to the GDX file debugOut.gdx which can be imported via the execute_load statement.

Set i /i1*i10/
    p(i,i) permutation;

execute_unload 'debug.gdx';
file fpy / 'debug.py' /; put fpy;

$onPutS
from gams.core.embedded import *
gams = ECGAMSDatabase(r'%gams.sysdir% '[:-2],'debug.gdx')
try:
   import random
   i = list(gams.get("i"))
   p = list(i)
   random.shuffle(p)
   for idx in range(len(i)):
       p[idx] = (i[idx], p[idx])
   gams.set("p", p)
   gmdWriteGDX(gams._gmd,'debugOut.gdx',1);
except Exception as e:
   print(str(e))
$offPut

putClose fpy;
execute.checkErrorLevel '="%gams.sysdir%GMSPython/python" debug.py';

execute_load 'debugOut.gdx', p;
Option p:0:0:1;
Display p;

Automatic Indentation

All embedded Python code is automatically wrapped in a try-except block that requires an indentation (two spaces). While this additional indentation makes no difference for most Python code, it does make a difference when it comes to multi-line strings. Therefore, multi-line strings created with triple quotes (either single or double quotes) are excluded from this rule to avoid unintentional spaces, e.g. when writing to a file:

$onEmbeddedCode Python:
f = open("file.txt", "w")
f.write('''some
multi
line
string
''')
f.close()
$offEmbeddedCode

Still, when using the line continuation character (\) in combination with strings, one has to be aware of the extra spaces that are added automatically:

$onEmbeddedCode Python:
s = "some\
multi\
line\
string"
gams.printLog(s)
$offEmbeddedCode

This will print some multi line string instead of somemultilinestring due to the automatically applied indentation.

SSL Certificate Error

When doing HTTPS requests from within Embedded Python code using packages like urllib or urllib3, one might get an error indicating that no certificate was found:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>

error

GamsInteractive error

GMSPython comes with the certifi package that provides certificates to be used for validating the trustworthiness of SSL certificates. In order to make use of those, one can set the environment variable SSL_CERT_FILE at the beginning of the Embedded Code section, before making the actual request:

$onEmbeddedCode Python:
import certifi
import os
import urllib.request
os.environ['SSL_CERT_FILE'] = certifi.where()
urllib.request.urlretrieve('<url>', '<file>')
$offEmbeddedCode

Performance Considerations of Embedded Python Code

If the same embedded code section (e.g. in a loop) is executed many times there are a few considerations to be taken into account in order to get the best performance. For this we will experiment with the example from the introduction. We look for a random permutation of a set i. In addition we have a cost matrix c(i,ii) and we are looking for the least cost permutation. We should just formulate this as matching in a bi-partite graph but in order to demonstrate some performance considerations we will repeatedly call the Python code that provides a random permutation and we will evaluate the cost of the permutation in GAMS and store the value of the cheapest one. Here is the naive implementation using embedded code:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation; 
c(i,ii) = uniform(-50,50);

Set    iter / 1*100 /;
Scalar tcost
       minTCost / +inf /;
loop(iter,
  embeddedCode Python:
  import random
  i = list(gams.get("i"))
  p = list(i)
  random.shuffle(p)
  gams.set("p", [ (i[idx], p[idx]) for idx in range(len(i)) ])
  endEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
Display minTCost;

Even though the Python interpreter stays alive the fresh start and end of an embedded code section goes together with the setup and initialization of multiple data structure, including the gams object of type ECGAMSDatabase. We can avoid the repeated setup and initialization by using pause and continue:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation;
c(i,ii) = uniform(-50,50);

embeddedCode Python:
import random
pauseEmbeddedCode

Set    iter / 1*100 /;
Scalar tcost
       minTCost / +inf /;
loop(iter,
  continueEmbeddedCode:
  i = list(gams.get("i"))
  p = list(i)
  random.shuffle(p)
  gams.set("p", [ (i[idx], p[idx]) for idx in range(len(i)) ])
  pauseEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
continueEmbeddedCode:
pass
endEmbeddedCode
Display minTCost;

The last embedded code execution of the Python pass statement is to clean up and terminate the Python session. Since the amount of data is relatively small, there is little difference in the running time between these variants.

There are other performance considerations. We can actually extract the set i once before the GAMS loop starts. Moreover, we can work with label indexes rather than the labels itself. Indexes are integers and are often faster than labels that are stored as strings. The only difference to the code above is the extraction method of the Python list i by i = list(gams.get("i",keyType=KeyType.INT)).

Extending GMSPython

While we recommend to use your own Python installation if you need additional packages, there are ways to extend the Python system that comes with GAMS in GMSPython. Here are the steps:

Get pip via get-pip (https://pip.pypa.io/en/stable/installation/)
Update pip
Install additional packages

We highly recommend installing pip and additional packages in the user site (use --user when running get-pip and pip). This is especially important for macOS users. Installing files in the GAMS system directory may interfere with the file notarization and may prevent GAMS from starting properly. Here is a typical dialog for Linux/macOS using curl to download get-pip.py. If you don't have curl or wget, use your web browser for downloading.

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
/path/to/gams/GMSPython/python get-pip.py --user
~/.local/bin/pip install -U pip
~/.local/bin/pip install geocoder --user

Using an External Python 3 Installation

Let's assume you don't want to work with the GAMS distributed GMSPython but you want to connect GAMS with a Python 3 installation of your liking. In order for the Embedded Code Facility to work, you need to make the alternative Python interpreter aware of the required modules provided by GAMS by following the Getting Started steps in the Python API documentation.

In addition to this you need to do one more step to allow GAMS to find your Python installation when executing Embedded Python code: you need to point to the Python dynamic load library or shared object. On Windows this file is called python3X.dll, on Linux libpython3.X.so, and on macOS libpython3.X.dylib. The X stands for the minor version number of Python 3. While embedded code has been built and tested for Python 3.12, chances are that you can also point to other Python 3 libraries. Since the expert level API is required, the set of versions is limited to what GAMS supports (currently GAMS ships these for Python 3.8 to 3.12). Limited experiments with the embedded code library for Python 3.12 with Python 3.8, 3.9, 3.10, and 3.11 libraries were successful. You need to set the environment variable GMSPYTHONLIB to point GAMS to the Python library:

Windows:
set GMSPYTHONLIB=c:\path\to\python\python312.dll

c
GamsInteractive c
Linux:
export GMSPYTHONLIB=/path/to/python/lib/libpython3.12.so
macOS:
export GMSPYTHONLIB=/path/to/python/lib/libpython3.12.dylib

GAMS will extract the Python home from the location of the Python library. In case this extraction does not work (because of an unconventional Python installation), you can instruct GAMS to set the Python home via the additional environment variable GMSPYTHONHOME. Please note, that this environment variable only needs to be set for GAMS Embedded Python code, so it is sufficient to add these variables to the GAMS configuration YAML file.

Note: When using environment-based Python distributions like Anaconda or Miniconda, it is not sufficient to set GMSPYTHONLIB (and GMSPYTHONHOME). The environment to be used needs to be activated properly before (e.g. conda activate myEnv). Packages like numpy which have been installed using conda (conda install numpy) won't work otherwise. In addition to setting GMSPYTHONLIB we therefore recommend to start GAMS itself and also tool like GAMS Studio from an activated conda environment in order to make sure that the environment and all installed packages work as expected.

Although it is recommended to use a self-contained Python installation, it is possible to use a virtual environment instead (e.g. via venv). In this case, GMSPYTHONLIB has to be set to the Python library of the installation from which the environment was derived from. In addition, the PYTHONPATH environment variable needs to be set to the site-packages directory of the virtual environment in order to make its packages available. Note that with this setup the site-packages directory of the original Python installation will also be found by GAMS Embedded Python code.

Building your own Embedded Python Code Library

Although the Embedded Code Facility and its binary components are part of the GAMS distribution, it is possible to build it manually from source using the following commands. The exact command line might change depending on the compiler and the operating system in use:

Windows:

cd [GAMS directory]/apifiles/C/api
icl.exe -Feembpycclib64.dll -IC:\path\to\python\include embpyoo.c gmdcc.c gclgms.c -LD -link -nodefaultlib:python38.lib

Linux:

cd [GAMS directory]/apifiles/C/api
gcc -o libembpycclib64.so -nostartfiles -shared -Wl,-Bsymbolic -pthread -fPIC -Ipath/to/your/python/include/python3.12 embpyoo.c gmdcc.c gclgms.c -lm -ldl

macOS on x86_64 CPUs:

cd [GAMS directory]/apifiles/C/api
gcc -o libembpycclib64.dylib -dynamiclib -shared -install_name @rpath/libembpycclib64.dylib -Ipath/to/your/python/include/python3.12 embpyoo.c gmdcc.c gclgms.c -lm -ldl

macOS on arm64 CPUs:

cd [GAMS directory]/apifiles/C/api
clang -o libembpycclib64.dylib -dynamiclib -shared -install_name @rpath/libembpycclib64.dylib -Ipath/to/your/python/include/python3.12 embpyoo.c gmdcc.c gclgms.c -lm -ldl

Connect

The Connect framework gives unified and platform independent access to data exchange with different formats (e.g. CSV and Excel). The instructions how to access the various data sources are given in YAML syntax. These YAML instructions are the code for the embedded Connect code. The embedded Connect code engine will parse the YAML instructions and use Connect agents to act upon the instructions. The agents GAMSReader and GAMSWriter send data back and forth to GAMS. A simple example is given above.

Embedded code Connect is build on top of embedded code Python. Similar considerations (e.g. Python interpreter is loaded once) carry over from Python to Connect. Moreover, the agent PythonCode has access to the object gams of type ECGamsDatabase which is used for communication of data with GAMS. In contrast to embedded Python code, the code of the agent PythonCode does not execute in the global scope but in a separate block. This means that variables set in the code are not available in another PythonCode section. The following code ends up in the except clause and prints Cannot access my_i.

embeddedCode Connect:
- PythonCode:
    code: |
        my_i = 1
- PythonCode:
    code: |
        try:
          gams.printLog(f'{my_i}')
        except:
          gams.printLog('Cannot access my_i')
endEmbeddedCode

Since Python allows to add attributes to an object at runtime, the my_i variable can be added to the gams object and hence becomes available in different PythonCode sections and even in different embedded Connect code parts (when pause/continue is used):

embeddedCode Connect:
- PythonCode:
    code: |
        gams.my_i = 1
- PythonCode:
    code: |
        gams.printLog(f'{gams.my_i}')
pauseEmbeddedCode
continueEmbeddedCode:
- PythonCode:
    code: |
        gams.printLog(f'{gams.my_i}')
endEmbeddedCode

Even though embedded code Python and Connect share the same basis, they run in different instantiations of the Python interpreter and hence don't share e.g. the same gams object.

Substitutions in Embedded Connect Code

The GAMS compiler has mechanisms to substitute compile-time variables to parameterize GAMS code including embedded code (unless the V, e.g. $onEmbeddedCodeV, of embedded code is used). Here is an example how to use double-dash parameters to allow the user to set the CSV file name via the GAMS command line (gams mymodel.gms --csvfilename=mystockprice.csv):

$if not set CSVFILENAME $set CSVFILENAME stockprice.csv
$onEmbeddedCode Connect:
- CSVReader:
    file: %CSVFILENAME%
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
- GAMSWriter:
    symbols:
      - name: stockprice
$offEmbeddedCode

Not only user defined compile-time variables will be substituted, but also all other compile time variables, e.g. %gams.scrDir% or %system.dirSep%. Another way to pass substitution instructions down to embedded Connect code are the embedded code arguments:

$if not set MYCSVFILENAME $set MYCSVFILENAME stockprice.csv
$onEmbeddedCodeV Connect: --CSVFILENAME=%MYCSVFILENAME%
- CSVReader:
    file: %CSVFILENAME%
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
- GAMSWriter:
    symbols:
      - name: stockprice
$offEmbeddedCode

Together with the put_utility command ECArguments that appends some arbitrary text to the embedded code arguments this can be used to parameterize the embedded Connect code at execution time, e.g. to create files based on label names:

Set i /1*5/; Parameter p(i);
loop(i,
   p(i) = uniform(0,1);
   put_utility 'ECArguments' / '--CSVFILENAME=' i.tl:0 '.csv';
   EmbeddedCodeV Connect:
   - GAMSReader:
       symbols:
         - name: p
   - CSVWriter:
       name: p
       file: %CSVFILENAME%
   endEmbeddedCode
);

Please note that while the GAMS command line parameters allow various formats to specify double-dash parameters, the embedded code Connect substitution arguments must be in the form --key=value, so a = character to separate key and value with no spaces. The value might be a (double) quoted string though. Please also note that the embedded code arguments set with put_utility command ECArguments will be applied to any subsequent embedded code section. So one might want to clear the argument by adding a put_utility "ECArguments" / ""; after the relevant embedded code section.

GAMS

The embedded code GAMS differs in various ways from the other embedded code languages. It's definitely not a new language one gets access to or does things not easily possible in GAMS. Moreover, it does the data communication not in-memory but via disk files. Mostly it adds some syntactical sugar to methods to call GAMS from within GAMS that are available for a long time. So a compile time embedded code GAMS section

$onEmbeddedCode GAMS: args
* ...
$offEmbeddedCode list_of_symbols

essentially can be done via some file creation via $on/offEcho, spawning of a GAMS process via $call, and loading of symbols via $gdxLoad:

$onEcho > sub.gms
* ...
$offEcho
$call.checkErrorLevel gams sub.gms args gdx=sub.gdx
$gdxLoad sub.gdx list_of_symbols

Similar at execution time an embedded code GAMS section

embeddedCode GAMS: args
* First part ...
pauseEmbeddedCode list_of_symbols
continueEmbeddedCode GAMS:
* Second part ...
endEmbeddedCode list_of_symbols

essentially can be done via some file creation via a put file, spawning of a GAMS process via execute, and loading of symbols via execute_loadpoint:

file fgms / 'sub.gms' /; put fgms;
$onPut
* First part ...
$offPut
putclose;
execute.checkErrorLevel 'gams sub.gms args gdx=sub.gdx save=sub'
execute_loadpoint 'sub.gdx', list_of_symbols;

put fgms;
$onPut
* Second part ...
$offPut
putclose;
execute.checkErrorLevel 'gams sub.gms args gdx=sub.gdx restart=sub'
execute_loadpoint 'sub.gdx', list_of_symbols;

In addition to the more readable syntax provided by the embedded code syntax, embedded code GAMS takes care of files (source, log, listing, GDX etc) and options related to these files, e.g. logFile. In case of execution-time pauseEmbeddedCode/continueEmbeddedCode embedded code handles the management of the save and restart operations which is especially convenient when working with multiple independent embedded code sections in combination with function embeddedHandle. Moreover, the log and listing streams of the embedded code GAMS section become part of the outer GAMS log and listing stream in case the GAMS subprocess returns with an error. For example, the following code with a syntax error (symbol names can't start with _) in the embedded code section:

embeddedCode GAMS:
* This produces an error
set _i /1*10/;
endEmbeddedCode

results in the following relevant part of the log:

...
--- Initialize embedded library embgamscclib64.dll
--- Execute embedded library embgamscclib64.dll
*** Error running gams (return code 2)
--- Start of log file ecgamslog.dat:
    --- Job myEmb.dat Start 01/18/23 12:24:00 42.0.0 3dcbfd09 WEX-WEI x86 64bit/MS Windows
    ...
    --- Starting compilation
    --- myEmb.dat(2) 3 Mb 1 Error
    *** Error   2 in C:\Users\mbuss\Downloads\225e\myEmb.dat
        Identifier expected
    --- GDX File C:\Users\mbuss\Downloads\225e\ecgamsgdx.dat
    --- myEmb.dat(2) 3 Mb 1 Error
    *** Status: Compilation error(s)
    --- Job myEmb.dat Stop 01/18/23 12:24:00 elapsed 0:00:00.007
--- End of log file ecgamslog.dat
*** Error at line 1: Error executing "embeddedCode" section: Inspect listing file for details
...

and following relevant part of the listing file:

...
E x e c u t i o n

--- Start of include of file ecgamslst.dat
GAMS 42.1.0  3dcbfd09 Jan 30, 2023   WEX-WEI x86 64bit/MS Windows - 01/31/23 12:24:00 Page 1
G e n e r a l   A l g e b r a i c   M o d e l i n g   S y s t e m
C o m p i l a t i o n


   1  * This produces an error
   2  set _i /1*10/;
****      $2
****   2  Identifier expected

**** 1 ERROR(S)   0 WARNING(S)

...
**** USER ERROR(S) ENCOUNTERED
--- End of include of file ecgamslst.dat
**** Exec Error at line 1: Error executing "embeddedCode" section: Inspect listing file for details
...

Best Practices

Embedded code GAMS at compile time can be used to build up data, e.g. domain sets, required at compile time inside an embedded code GAMS at execution time. Data from the embedded code GAMS section can be moved via GDX (see example above) or file ASCII input files into the outer GAMS program. The following two examples complements the bin packing example by demonstrating two ways to communicate the data via an ASCII input file.

The following solution creates a global compile time variable for the size of set j:

...
$save.keepCode bp

$onEmbeddedCode GAMS: restart=bp
scalar size /0/;
nj = 1; loop(i, size = size+s(i); if (size>B, nj = nj+1; size = s(i)));
file fnj / 'nj.gms' /; put fnj '$setGlobal NJ ' nj:0:0;
$offEmbeddedCode
$include nj
Set j 'bins' / b1*b%NJ% /;
...

In the next solution the embedded code GAMS writes already the data statement for set j:

...
$save.keepCode bp

$onEmbeddedCode GAMS: restart=bp
scalar jcnt, size /0/;
nj = 1; loop(i, size = size+s(i); if (size>B, nj = nj+1; size = s(i)));
file fnj / 'nj.gms' /; put fnj '* Set j:';
* Either as short hand b1*bN or all individual elements
if (1=0,
  put / 'b1*b' nj:0:0;
else
  for (jcnt=1 to nj, put / 'b' jcnt:0:0);
);
$offEmbeddedCode
Set j 'bins' /
$include nj
/;
...

Embedded code GAMS works best in combination with restarting from a save file. This makes the embedded code short because it can utilize the symbol declarations and definitions from the restart file. Save files can be created at compile time via $save[.keepCode] fileName and at execution time via put_utility 'save' / 'fileName';.

Execution time embedded code GAMS can be useful if at execution time GAMS needs to deal with new labels as in the following sophisticated example. Here we build on top of the GAMS Model Library trnsport model and represent the Jacobian matrix of the LP model utilizing the Convert option dumpGDX in the original name space inside GAMS:

...
Model transport / all /;

* Generate the Jacobian (t.gdx) with e1,e2,e3,... and x1,x2,x3,... and a mapping (tdm.gdx) to map the
* scalar elements e1,e2,e3,... and x1,x2,x3,... to the original name space:
$onEcho > convert.opt
dumpGDX t.gdx
dictMap tdm.gdx
$offEcho
transport.optfile = 1; option lp=convert;
solve transport using lp minimizing z;

Sets
    v 'variables block names' / x, z /,
    e 'equation block names' / cost, supply, demand /
    empty 'the empty string' / '' /;
Parameter jac(e,*,*,v,*,*) 'Jacobian of the generated LP model';

embeddedCode GAMS:
$call.checkErrorLevel gdxdump tdm.gdx noData > tdm.gms
$include tdm
Parameter A(i,j) 'Jacobian in e1,e2,.. (set i) and x1,x2,... (set j) space', jac;
$gdxLoad t.gdx A
Alias (*,ti,tii,tj,tjj);
loop((cost_EM(i)      ,x_VM(j,ti,tj)), jac('cost'  ,'' ,'','x',ti,tj) = A(i,j));
loop((cost_EM(i)      ,z_VM(j)      ), jac('cost'  ,'' ,'','z','','') = A(i,j));
loop((supply_EM(i,tii),x_VM(j,ti,tj)), jac('supply',tii,'','x',ti,tj) = A(i,j));
loop((supply_EM(i,tii),z_VM(j)      ), jac('supply',tii,'','z','','') = A(i,j));
loop((supply_EM(i,tjj),x_VM(j,ti,tj)), jac('demand','',tjj,'x',ti,tj) = A(i,j));
loop((supply_EM(i,tjj),z_VM(j)      ), jac('demand','',tjj,'z','','') = A(i,j));
endEmbeddedCode jac

option jac:3:3:3; display jac;

The display of the last line in the code provides the following output in the listing file:

----     87 PARAMETER jac
                                     x           x           x           x           x           x           z
                               seattle     seattle     seattle   san-diego   san-diego   san-diego
                              new-york     chicago      topeka    new-york     chicago      topeka

cost  .         .               -0.225      -0.153      -0.162      -0.225      -0.162      -0.126       1.000
supply.seattle  .                1.000       1.000       1.000
supply.san-diego.                                                    1.000       1.000       1.000
demand.         .seattle         1.000       1.000       1.000
demand.         .san-diego                                           1.000       1.000       1.000

Other embedded code languages allow inside the language to set instructions to either filter/domain check or replace/merge data that comes back into GAMS. At compile time the dollar control options $on/offFiltered and $on/offMulti[R] impact the load of symbols from embedded code sections, including GAMS. At execution time embedded code GAMS filters and merges the data. If a different behavior is required, the symbols to be loaded should not be listed on the end/pauseEmbeddedCode line but should be loaded with the appropriate execute_load*** call like execute_load, execute_loaddc, or execute_loadpoint. The GDX file to load can be set via put_utility "gdxIn" / gams.scrDir "ecgamsgdx." gams.scrExt;:

Set ii / i0*i10 /, i / i1*i5 /; Parameter p(i) / i1 1, i3 1, i5 1 /;
embeddedCode GAMS:
Parameter p / i2 1, i4 1, i6 1 /;
endEmbeddedCode p
display p;
embeddedCode GAMS:
Parameter p / i2 1, i4 1, i6 1 /;
endEmbeddedCode
put_utility 'gdxIn' / gams.scrDir 'ecgamsgdx.' gams.scrExt;
execute_load p;
display p;

GAMS' first load of p filters and merges the symbol into the existing p resulting in i1 1, i2 1, i3 1, i4 1, i5 1 while the second load replaces the symbol resulting in i2 1, i4 1. An execute_loaddc p; would have resulted in a domain violation (i6) and would have triggered an execution error.

ReSHOP

The embedded code ReSHOP is used to communicate the EMP structure to the ReSHOP solver. Functionally, this is close to the %emp.info% mechanism. The advantage of using this embedded code is the access to the GAMS sets and parameters. This facilitates the use of the declarative empinfo syntax, as existing GAMS sets and parameters can be use in the embedded code statements. The used sets and parameters are then communicated to ReSHOP as the model instance is solved.

Note: The embedded code ReSHOP can only be use at execution time, with EmbeddedCode/endEmbeddedCode and not at compile time.

A small example follows:

set       a / a1*a2 /;
variables x(a) 
          obj(a);
equations defobj(a)
          cons(a);

...
* Skipping equation definitions
...

model m / all /;

EmbeddedCode ReSHOP:
n(a): min obj(a) x(a) defobj(a) cons(a)
root: Nash(n(a))
endEmbeddedCode

option emp=reshop;

solve m using emp;

Warning: ReSHOP and this embedded code feature are under development. Feedback is highly appreciated.

Table of Contents

Motivation

Concept

Simple Example

Python

Connect

GAMS

Syntax

Compile Time

Execution Time

Python

Interface between GAMS and Python

Using the Control API

Exchange via Files and Environment Variables

Encodings

Multiple Independent Python Sessions

Troubleshooting Embedded Python Code

Automatic Indentation

SSL Certificate Error

Performance Considerations of Embedded Python Code

Extending GMSPython

Using an External Python 3 Installation

Building your own Embedded Python Code Library

Connect

Substitutions in Embedded Connect Code

GAMS

Best Practices

ReSHOP