GAMS Transfer is a package to maintain GAMS data outside a GAMS script in a programming language like Python or Matlab. It allows the user to add GAMS symbols (Sets, Parameters, Variables and Equations), to manipulate GAMS symbols, as well as read/write symbols to different data endpoints. GAMS Transfer’s main focus is the highly efficient transfer of data between GAMS and the target programming language, while keeping those operations as simple as possible for the user. In order to achieve this, symbol records – the actual and potentially large-scale data sets – are stored in native data structures of the corresponding programming languages. The benefits of this approach are threefold: (1) The user is usually very familiar with these data structures, (2) these data structures come with a large tool box for various data operations, and (3) optimized methods for reading from and writing to GDX can transfer the data as a bulk – resulting in the high performance of this package. This tutorial describes, in detail, the use of GAMS Transfer within a Python environment.

Data within GAMS Transfer will be stored as Pandas DataFrame. The flexible nature of Pandas DataFrames makes them ideal for storing/manipulating sparse data. Pandas includes advanced operations for indexing and slicing, reshaping, merging and even visualization.

Pandas also includes a number of advanced data I/O tools that allow users to generate DataFrames directly from CSV (.csv), JSON (.json), HTML (.html), Microsoft Excel (.xls, .xlsx), SQL , pickle (.pkl), SPSS (.sav, .zsav), SAS (.xpt, .sas7bdat), etc.

Centering GAMS Transfer around the Pandas DataFrame gives GAMS users (on a variety of platforms – MacOS, Windows, Linux) access to tools to move data back and forth between their favorite environments for use in their GAMS models.

The goal of this tutorial is to introduce the user to GAMS Transfer and its functionality. This tutorial is not designed to teach the user how to effectively manipulate Pandas DataFrames; users seeking a deeper understanding of Pandas are referred to the extensive documentation.

Install

The user must download and install the latest version of GAMS in order to install GAMS Transfer. GAMS Transfer is installed when the GAMS Python API is built and installed. The user is referred HERE for instructions on how to install the Python API files. GAMS Transfer and all GAMS Python API files are compatible with environment managers such as Anaconda.

Design

Storing, manipulating, and transforming sparse data requires that it live within an environment – this data can then be linked together to enable various operations. In GAMS Transfer we refer to this "environment" as the Container, it is the main repository for storing and linking our sparse data. Symbols can be added to the Container from a variety of GAMS starting points but they can also be generated directly within the Python environment using convenient function calls that are part of the GAMS Transfer package; a symbol can only belong to one container at a time.

The process of linking symbols together within a container was inspired by typical GAMS workflows but leverages aspects of object oriented programming to make linking data a natural process. Linking data enables data operations like implicit set growth, domain checking, data format transformations (to dense/sparse matrix formats), etc. All of these details will be discussed in the following sections. For now, we highlight a few key aspects of GAMS Transfer that will help lay the foundation. Users can skip directly to examples using the Table of Contents.

GAMS Special Values

The GAMS system contains five special values: UNDEF (undefined), NA (not available), EPS (epsilon), +INF (positive infinity), -INF (negative infinity). These special values must be mapped to their Python equivalents. GAMS Transfer follows the following convention to generate the 1:1 mapping:

• +INF is mapped to float("inf")
• -INF is mapped to float("-inf")
• EPS is mapped to -0.0 (mathematically identical to zero)
• NA is mapped to a special NaN
• UNDEF is mapped to float("nan")

GAMS Transfer syntax is designed to quickly get data into a form that is usable in further analyses or visualization; this mapping also highlights the preference for data that is of type float, which offers performance benefits within Pandas/NumPy. The user does not need to remember these constants as they are provided within the class SpecialValues as SpecialValues.POSINF, SpecialValues.NEGINF, SpecialValues.EPS, SpecialValues.NA, and SpecialValues.UNDEF. The SpecialValues class also contains methods to test for these special values. Some examples are shown below; already, we, begin to introduce some of the GAMS Transfer syntax.

Example (special values in a parameter)

The following DataFrame for x would look like:

The user can now easily test for specific special values in the value column of the DataFrame (returns a boolean array):

Other data structures can be passed into these methods as long as these structures can be converted into a numpy array with dtype=float. It follows that:

Pandas DataFrames allow data columns to exist with mixed type (dtype=object). As such, GAMS Transfer allows special string representations ("+inf", "-inf", "eps", "na", or "undef") of these special values when setting symbol records as seen in the following example.

Example (special values defined by strings)

These special strings will be immediately mapped to their float equivalents from the SpecialValues class in order to ensure that all data entries are float types.

Note

Python is case sensitive. GAMS Transfer is strict such that EPS will not map to -0.0 and an error will be generated; the user is responsible for transforming their data to use the lower case representations of special values.

GAMS Transfer Standard Data Formats

This section is meant to introduce the standard format that GAMS Transfer expects for symbol records. It has already been mentioned that we store data as a Pandas DataFrame, but there is an assumed structure to the column headings and column types that will be important to understand. GAMS Transfer includes convenience functions in order to ease the burden of converting data from a user-centric format to one that is understood by GAMS Transfer. However, advanced users will want to convert their data first and add it directly to the Container to avoid making extra copies of (potentially large) data sets.

Set Records Standard Format

All set records (including singleton sets) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 1. The first n-1 columns include the domain elements while the last column includes the set element explanatory text. Records are organized such that there is one record per row.

The names of the domain columns follow a pattern of <set_name>_<index_position>; a symbol dimension that is referenced to the universe is labeled uni_<index position>. The explanatory text column is called element_text and must take the last position in the DataFrame.

All domain columns must be a categorical data type and the element_text column must be a object type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however GAMS Transfer requires that all these are type str. All rows in the element_text column must be type str.

Some examples:

Parameter Records Standard Format

All parameter records (including scalars) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 1. The first n-1 columns include the domain elements while the last column includes the numerical value of the records. Records are organized such that there is one record per row. Scalar parameters have zero dimension, therefore they only have one column and one row.

The names of the domain columns follow a pattern of <set_name>_<index_position>; a symbol dimension that is referenced to the universe is labeled uni_<index_position>. The value column is called value and must take the last position in the DataFrame.

All domain columns must be a categorical data type and the value column must be a float type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however GAMS Transfer requires that all these are type str.

Some examples:

Variable/Equation Records Standard Format

Variables and equations share the same standard data format. All records (including scalar variables/equations) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 5. The first n-5 columns include the domain elements while the last five columns include the numerical values for different attributes of the records. Records are organized such that there is one record per row. Scalar variables/equations have zero dimension, therefore they have five columns and one row.

The names of the domain columns follow a pattern of <set_name>_<index position>; a symbol dimension that is referenced to the universe is labeled uni_<index_position>. The attribute columns are called level, marginal, lower, upper, and scale. These attribute columns must appear in this order. Attributes that are not supplied by the user will be assigned the default GAMS values for that variable/equation type; it is possible to not pass any attributes, GAMS Transfer would then simply assign default values to all attributes.

All domain columns must be a categorical data type and all the attribute columns must be a float type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however GAMS Transfer requires that all these are type str.

Some examples:

Core Functionality

The reader has already had glimpses of the GAMS Transfer syntax, but in this section we will present more details about the core functionalty and dig further into the syntax. Specifically, we will discuss:

1. How to create a Container
2. How to add symbols to a Container
3. How to validate the data that is in the Container
4. How to defined domains implicitly with domain_forwarding
5. How to describe the data that is in the Container
6. How to transform data that is in the Container
7. Understand what the universe set is (UEL list)
8. How to reorder symbols in the Container
9. How to remove symbols from the Container
10. Tips for debugging data

All of these operations pertain specifically to the Container and the symbols within the Container. We will leave a detailed discussion about reading/writing data from/to GDX files until the next section.

Creating a Container

The main object class within GAMS Transfer is called Container. The Container is the vessel that allows symbols to be linked together (through their domain definitions), it enables implicit set definitions, it enables structural manipulations of the data (matrix generation), and it allows the user to perform different read/write operations. Creating a container is a simple matter of initializing an object. For example:

This new Container object, here called m, contains a number of convenient properties and methods that allow the user to interact with the symbols that are in the Container. Some of these properties are simply used to filter out different types of symbols, other methods are used to numerically characterize the data within each symbol.

Symbols are organized in the container under the data container attribute. The dot notation (m.data) is used to access the underlying dict; symbols in this dict can then be retrieved with the standard bracket notation (m.data[<symbol_name>]). For convenience, there are list* methods that will filter the different types of symbols and return a list. Those methods and their arguments are described below:

A valid symbol is one that has passed a number of internal quality checks to ensure that this data is in the GAMS Transfer standard format and is self-consistent. More details on data validity will be provided in Validating Data.

As an example, a Container that has been populated with symbol data from the trnsport.gms (we will discuss how to exchange data with GDX in Data Exchange with GDX) model would look like:

We can then call the list* methods to explore the different types of symbols. The order of these lists mirror the order of appearance in the m.data dict.

Creating a Symbol

There are two different ways to create a symbol and add it to a Container.

• Use Container methods addSet, addAlias, addParameter, addVariable, and addEquation
• Use symbol constructors Set, Alias, Parameter, Variable and Equation The following tables detail the arguments used to create each symbol type and their default values.

The following table detail the argument and their default values for the symbol constructors.

Set constructor

Alias constructor

Parameter constructor

Variable constructor

Equation constructor

The other symbol properties are implied by the arguments that the user specifies. For example, dimension is the number of elements in the domain list. All the above symbol properties can be modified at any time after the addition to the container, however, any changes could result in an invalid symbol. Care must be taken when modifying data directly.

Note

If any domain element is passed as string except for "*" (universe set), the domain_type of the symbol will be labeled as a relaxed symbol rather than regular. This means that domain checking will not be performed.

When using the Container methods to create new symbols (addSet, addParameter, addAlias, addVariable, addEquation) it is not necessary to pass the container reference as an argument. All other arguments have the same structure as in the tables above.

Adding Set Records

Three possibilities exist to assign symbol records to a set:

• Setting the property records directly
• Using the symbol method setRecords
• Setting the argument records in a symbol constructor/container method (internally calls setRecords)

If data is assigned to the records property directly, the user must take care that the data follow the standard format. If the user does not follow this format the symbol will be marked as invalid and therefore cannot be written to the data endpoint. Users that are concerned about memory use may want to set the records directly because this avoids creating a copy of the data in memory.

The symbol method setRecords is a convenience method that transforms the given data into an approved Pandas DataFrame format. Many native python data types can be easily transformed into DataFrames, so the setRecords method for Set objects will accept a number of different types for input. The setRecords method is called internally on any data structure that is passed through the records argument.

Example (create a 1D set from a list)

Example (create a 1D set from a tuple)

Example (create a 2D set from a list of tuples)

Example (create a 1D set from a DataFrame slice + unique())

Note

The .unique() method preserves the order of appearance, unlike set().

Set element text is very handy when labeling specific set elements within a set. A user can add a set element text directly with a set element. Note that it is not required to label all set elements, as can be seen in the following example.

Example (adding set element text)

Note

It is not possible to "add records" to an alias because an alias simply points to another set. If an alias is made from another alias, GAMS Transfer will recursively search for the root set and point directly to this set.

Adding Parameter Records

Three possibilities exist to assign symbol records to a parameter:

• Setting the property records directly
• Using the symbol method setRecords
• Setting the argument records in a symbol constructor/container method (internally calls setRecords)

If data is assigned to the records property directly, the user must take care that the data follow the standard format. If the user does not follow this format the symbol will be marked as invalid and therefore cannot be written to the data endpoint. Users that are concerned about memory use may want to set the records directly because this avoids creating a copy of the data in memory.

The symbol method setRecords is a convenience method that transforms the given data into an approved Pandas DataFrame format. Many native python data types can be easily transformed into DataFrames, so the setRecords method for Parameter objects will accept a number of different types for input. The setRecords method is called internally on any data structure that is passed through the records argument.

Example (create a 1D parameter from a list of tuples)

Example (create a 2D parameter from a tuple)

Example (create a 2D set from a DataFrame)

Note

When passing a DataFrame in the records argument a deep copy of the data will be made. This copy will be converted to the GAMS Transfer standard format leaving the other copy untouched.

Adding Variables

Three possibilities exist to assign symbol records to a variable:

• Setting the property records directly
• Using the symbol method setRecords
• Setting the argument records in a symbol constructor/container method (internally calls setRecords)

If data is assigned to the records property directly, the user must take care that the data follow the standard format. If the user does not follow this format the symbol will be marked as invalid and therefore cannot be written to the data endpoint. Users that are concerned about memory use may want to set the records directly because this avoids creating a copy of the data in memory.

The symbol method setRecords is a convenience method that transforms the given data into an approved Pandas DataFrame format. The setRecords method is called internally on any data structure that is passed through the records argument. The structure for Variable symbols is more complicated so the user must present setRecords with a DataFrame, however the user does not need to supply all necessary columns (i.e., "level", "marginal", "lower", "upper" and "scale"). Any columns that are not supplied will be created and filled with their GAMS default values.

Example (create a 1D free variable, specifying only level values)

Example (create a scalar variable)

Example (create a 2D positive variable, specifying no numerical data)

Adding Equations

Three possibilities exist to assign symbol records to an equation:

• Setting the property records directly
• Using the symbol method setRecords
• Setting the argument records in a symbol constructor/container method (internally calls setRecords)

If data is assigned to the records property directly, the user must take care that the data follow the standard format. If the user does not follow this format the symbol will be marked as invalid and therefore cannot be written to the data endpoint. Users that are concerned about memory use may want to set the records directly because this avoids creating a copy of the data in memory.

The symbol method setRecords is a convenience method that transforms the given data into an approved Pandas DataFrame format. The setRecords method is called internally on any data structure that is passed through the records argument. The structure for Equation symbols is more complicated so the user must present setRecords with a DataFrame, however the user does not need to supply all necessary columns (i.e., "level", "marginal", "lower", "upper" and "scale"). Any columns that are not supplied will be created and filled with their GAMS default values.

Example (create a 1D =L= equation, specifying only level values)

Example (create a scalar =E= equation)

Validating Data

GAMS Transfer requires that the records for all symbols exist in a standard format (GAMS Transfer Standard Data Formats) in order for them to be understood and written successfully. It is certainly possible that the data could end up in a state that is inconsistent with the standard format (especially if setting symbol attributes directly). GAMS Transfer includes the .isValid() method in order to determine if a symbol is valid and ready for writing; this method returns a bool. For example, we create two valid sets and then check them with .isValid() to be sure.

Note

It is possible to run .isValid() on both the Container as well as the symbol object – .isValid() will also return a bool if there are any invalid symbols in the Container object.

Example (valid data)

Now we create some data that is invalid due to domain violations in the set j.

Example (intentionally create domain violations)

In this example, we know that the validity of the data is compromised by the domain violations, but there could be other subtle descrepancies that must be remedied before writing data. The user can get more detailed error reporting if the verbose argument is set to True. For example:

Domain Forwarding

GAMS includes the ability to define sets directly from data using the implicit set notation (see: Implicit Set Definition (or: Domain Defining Symbol Declarations)). This notation has an analogue in GAMS Transfer called domain_forwarding.

Note

It is possible to recursively update a subset tree in GAMS Transfer.

Domain forwarding is available as an argument to all symbol object constructors; the user would simply need to pass domain_forwarding=True.

In this example we have raw data that in the dist DataFrame and we want to send the domain information into the i and j sets – we take care to pass the set objects as the domain for parameter c.

Note

The element order in the sets i and j mirrors that in the raw data.

In this example we show that domain forwarding will also work recursively to update the entire set lineage – the domain forwarding occurs at the creation of every symbol object. The correct order of elements in set i is [z, a, b, c] because the records from j are forwarded first, and then the records from k are propagated through (back to i).

Describing Data

The methods describeSets, describeParameters, describeVariables, and describeEquations allow the user to get a summary view of key data statistics. The returned DataFrame aggregates the output for a number of other methods (depending on symbol type). Column descriptions are tabulated below:

The methods used to generate these tables are also available directly to the user. These methods (and their arguments) are tabulated:

Transforming Data

GAMS Transfer stores data in a "flat" format, that is, one record entry per DataFrame row. However, it is often necessary to convert this data format into a matrix format – GAMS Transfer enables users to do this with relative ease using the toDense and the toSparseCoo symbol methods. The toDense method will return a dense N-dimensional numpy array with each dimension corresponding to the GAMS symbol dimension; it is possible to output an array up to 20 dimensions (a GAMS limit). The toSparseCoo method will return the data in a sparse scipy COOrdinate format, which can then be efficiently converted into other sparse matrix formats.

Attention

Both the toDense and toSparseCoo methods do not transform the underlying DataFrame in any way, they only return the transformed data.

Note

toSparseCoo will only convert 2-dimensional data to the scipy COOrdinate format. A user interested in sparse data for an N-dimensional symbol will need to decide how to reshape the dense array in order to generate the 2D sparse format.

Attention

In order to use the toSparseCoo method the user will need to install the scipy package. Scipy is not provided with GMSPython.

Both the toDense and toSparseCoo method leverage the indexing that comes along with using categorical data types to store domain information. This means that linking symbols together (by passing symbol objects as domain information) impacts the size of the matrix. This is best demonstrated by a few examples.

Example (1D data w/o domain linking (i.e., a relaxed domain))

Note that the parameter a is not linked to another symbol, so when converting to a matrix, the indexing is referenced to the data structure in a.records. Defining a sparse parameter a over a set i allows us to extract information from the i domain and construct a very different dense matrix, as the following example shows:

Example (1D data w/ domain linking (i.e., a regular domain))

Example (2D data w/ domain linking)

The Universe Set

A Unique Element List (UEL) (aka the "universe" or "universe set") is an (i,s) pair where i is an identification number for a string s. GAMS uses UELs to efficiently store domain entries of a record by storing the UEL ID i of a domain entry instead of the actual string s. This avoids storing the same string multiple times. The concept of UELs also exists in Python/Pandas and is called a "categorical array". GAMS Transfer leverages these types in order to efficiently store strings and enable domain checking within the Python environment.

Each domain column in a DataFrame can be assigned a unique categorical type, the effect is that each symbol maintains its own list of UELs per dimension. The UEL IDs are numbered from 1 to the number of UELs stored independently for each dimension. It is possible to convert a categorical column to its ID number representation by using the categorical accessor x.records[<domain_column_label>].cat.codes; however, this type of data manipulation is not necessary within GAMS Transfer, but could be handy when debugging data.

Pandas offers the possibility to create categorical column types that are ordered or not; GAMS Transfer relies exclusively on ordered categorical data types. By using ordered categories, GAMS Transfer will order the UEL such that elements appear in the order in which they appeared in the data (which is how GAMS defines the UEL). There is a feature to GAMSTransfer that allows the user to make last minute modifications to the UEL order, should they desire.

GAMS Transfer does not actually keep track of the UEL separately from other symbols in the Container, it will be created at the last minute based on the order in which data is added to the container. However, the user can access the current state of the UEL with the .getUniverseSet() container method. For example, we set a two dimensional set:

Pandas also includes a number of methods that allow categories to be renamed, appended, etc. These methods may be useful for advanced users, but most users will probably find that modifying the original data structures and reseting the symbol records provides a simpler solution. The design of GAMS Transfer should enable to user to quickly move data back and forth, without worrying about the deeper mechanics of categorical data.

Reordering Symbols

The order of the Container file requires the symbols to be sorted such that, for example, a Set used as domain of another symbol appears before that symbol. The Container will try to establish a valid ordering when writing the data. However, this operation can also be invoked manually by calling the method reorderSymbols.

Rename Symbols

It is possible to rename a symbol even after it has been added to a Container. There are two methods that can be used to achieve the desired outcome:

• using the container method renameSymbol
• directly changing the name symbol property

We create a Container with two sets:

Example (changing the name of a symbol with the container method)

Example (changing the name of a symbol with the container method)

Note

Note that the renamed symbols maintain the original symbol order, this will prevent unnecessary reordering operations later in the workflow.

Removing Symbols

Removing symbols from a container is easy when using the removeSymbols container method; this method accepts either a str or a list of str.

Attention

Once a symbol has been removed, it is possible to have hanging references as domain links in other symbols. The user will need to repair these other symbols with the proper domain links in order to avoid validity errors.

Debugging Data

Several methods are provided as part of GAMS Transfer that enable operations that could be classified as "data debugging" rather than "describe" operations. These are: findDomainVolations, findEps, findNA, findUndef, findPosInf, and findNegInf. These methods and their arguments are tabulated below:

Full Example

It is possible to use everything we now know about GAMS Transfer to recreate the trnsport.gms results in GDX form. As part of this example we also introduce the write method (and generate new.gdx). We will discuss in more detail in the following section: Data Exchange with GDX.

Data Exchange with GDX

Up until now, we have been focused on using GAMS Transfer to create symbols in an empty Container using the symbol constructors (or their corresponding container methods). These tools will enable users to ingest data from many different formats and add them to a Container – however, it is also possible to read in symbol data directly from GDX files using the read container method. In the following sections, we will discuss this method in detail as well as the write method, which allows users to write out to new GDX files.

Reading from GDX

There are two main ways to read in GDX based data.

• Pass the file path directly to the Container constructor (will read all symbols and records)
• Pass the file path directly to the read method (default read all symbols, but can read partial files)

The first option here is provided for convenience and will, internally, call the read method. This method will read in all symbols as well as their records. This is the easiest and fastest way to get data out of a GDX file and into your Python environment. For the following examples we leverage the GDX output generated from the trnsport.gms model file.

Example (reading full data w/ Container constructor)

A user could also read in data with the read method as shown in the following example.

Example (reading full data w/ read method)

It is also possible to read in a partial GDX file with the read method, as shown in the following example:

This syntax assumes that the user will always want to read in both the metadata as well as the actual data records, but it is possible to skip the reading of the records by passing the argument values=False.

Attention

The read method attempts to link the domain objects together (in order to have a "regular" domain_type) but if domain sets are not part of the read operation there is no choice but to default to a "relaxed" domain_type. This can be seen in the last example where we only read in the variable x and not the domain sets (i and j) that the variable is defined over. All the data will be available to the user, but domain checking is no longer possible. The symbol x will remain with "relaxed" domain type even if the user were to read in sets i and j in a second read call.

Writing to GDX

A user can write data to a GDX file by simply passing a file path (as a string). The write method will then create the GDX and write all data in the Container.

Note

It is not possible to write the Container when any of its symbols are invalid. If any symbols are invalid an error will be raised and the user will need to inspect the problematic symbols (perhaps using a combination of the listSymbols(isValid=False) and isValid(verbose=True) methods).

Example

Example (write a compressed GDX file)

Advanced users might want to specify an order to their UEL list (i.e., the universe set); recall that the UEL ordering follows that dictated by the data. As a convenience, it is possible to prepend the UEL list with a user specified order using the uel_priority argument.

Example (change the order of the UEL)

The original UEL order for this GDX file would have been ["a", "b", "c"], but at the last minute, the UEL order was changed to swap the positions of b and c. This can be verified with the gdxdump utility (using the uelTable argument):