Validating Data

transfer requires that the records for all symbols exist in a standard format (Standard Data Formats) in order for them to be understood by the Container. It is certainly possible that the data could end up in a state that is inconsistent with the standard format (especially if setting symbol attributes directly). transfer includes the .isValid() method in order to determine if a symbol is structurally valid – this method returns a bool. This method does not guarantee that a symbol will be successfully written to either GDX or GMD, other data errors (duplicate records, long UEL names, or domain violations) could exist that are not tested in .isValid().

For example, we create two valid sets and then check them with .isValid() to be sure.

Note

It is possible to run .isValid() on both the Container as well as the symbol object – .isValid() will also return a bool if there are any invalid symbols in the Container object.

Example (valid data)

The .isValid() method checks:

1. If the symbol belongs to a Container
2. If all domain set symbols exist in the Container
3. If all domain set symbols objects are valid
4. If any domain set is also a singleton set (not allowed in GAMS)
5. If records are a DataFrame (or None)
6. If the records DataFrame is the right number of columns (based on symbol dimension)
7. If the symbol is a scalar, then ensure there is only one record (row) in the DataFrame
8. If records column headings are unique
9. If any symbol attribute columns are missing or out of order
10. If all domain columns are type category
11. If all domain categories are type str
12. If all data columns are type float

Custom Column Headings

The names of the domain columns are flexible, but transfer requires unique column names. Users are encouraged to change the column headings of the underlying dataframe by using the domain_labels property. Using this property will ensure that unique column names are generated by adding a _<dimension> tag to the end of any user supplied column names. The following examples show this behavior.

Attention

All * domains are recast as uni. This allows users to access the column data with both the Pandas bracket and/or dot notation (i.e., df["uni"] or df.uni).

Column heading behavior at symbol instantiation

The setRecords (which is called internally at symbol instantiation) method will set default domain_labels if they were not provided by the user. The only way for a user to provide domain labels with setRecords is by passing in a Pandas DataFrame object. The _<dimension> tag will be added to all domain labels in order to make all domain names unique – this tag is added to all dimensions if any subset of the domain names are non-unique.

Customizing column headings

Many users may want to output the GAMS DataFrame directly to another format (CSV, etc.) and may wish to create customized DataFrame column headings for readability. User can do this by directly setting the domain_labels property, as seen in the following example.

Attention

Users are encouraged to use the <symbol>.domain_labels property instead of setting the <DataFrame>.columns directly. The avoids the possibility of out-of-sync symbol validity. The domain_labels property does not store anything, calling this property simply returns the exact domain labels from the DataFrame.

Converting Records

All data in transfer will be stored as a Pandas DataFrame – however, it is desirable to have easy access to data without the additional infrastructure that comes with the DataFrame object. We include to* methods (available for all symbol types) that will return other data structures. The following examples show the behavior of toValue, toList, and toDict (previous examples showed examples of toDense and toSparseCoo).

Examples of toList

Examples of toValue

Examples of toDict

Comparing Symbols

Sparse GAMS data is inherently unordered. The concept of order is GAMS is governed by the order of the UELs in the universe set not the order of the records. This differs from the sparse data structures that we use in transfer (Pandas DataFrames) because each record (i.e., DataFrame row) has an index (typically 0..n) and can be sorted by this index. Said a slightly different way, two GDX files will be equivalent if their universe order is the same and the records are the same, however when creating the GDX file, it is of no consequence what order the records are written in. Therefore, in order to calculate an equality between two symbols in transfer we must perform a merge operation on the symbol domain labels – an operation that could be computationally expensive for large symbols.

Attention

The nature of symbol equality in transfer means that a potentially expensive merge operation is performed, we do not recommend that the equals method be used inside loops or when speed is critical. It is, however, very useful for data debugging.

A quick example shows the syntax of equals:

By default, equals takes the strictest view of symbol "equality" – everything must be the same. In this case, the symbol names and descriptions differ between the two sets i and j. We can relax the view of equality with a combination of argument flags. Comparing the two symbols again, but ignoring the meta data (i.e., ignoring the symbol name, description and type (if a Variable or Equation)):

It is also possible to ignore the set element text in equals:

The check_uels argument will ensure that the symbol "universe" is the same (in order and content) between two symbols, as illustrated in the following example:

Clearly, the two sets i and ip have the same records, but the UEL order is different. If check_uels=True the resulting symbols will not be considered equal – turning this flag off results in equality.

Numerical comparisons are enabled for Parameters, Variables and Equations – equality can be flexibly defined through the equals method arguments. Again, the strictest view of equality is taken as the default behavior of equals (no numerical tolerances, some limitations exist – see: numpy.isclose for more details).

Attention

The numerical comparison is handled by numpy.isclose, more details can be found in the Numpy documentation.

In the case of variables and equations, it is possible for the user to confine the numerical comparison to certain certain attributes (level, marginal, lower, upper and scale) by specifying the columns argument as the following example illustrates:

Domain Forwarding

GAMS includes the ability to define sets directly from data using the implicit set notation (see: Implicit Set Definition (or: Domain Defining Symbol Declarations)). This notation has an analogue in transfer called domain_forwarding.

Note

It is possible to recursively update a subset tree in transfer.

Domain forwarding is available as an argument to all symbol object constructors; the user would simply need to pass domain_forwarding=True.

In this example we have raw data that in the dist DataFrame and we want to send the domain information into the i and j sets – we take care to pass the set objects as the domain for parameter c.

Note

The element order in the sets i and j mirrors that in the raw data.

In this example we show that domain forwarding will also work recursively to update the entire set lineage – the domain forwarding occurs at the creation of every symbol object. The correct order of elements in set i is [z, a, b, c] because the records from j are forwarded first, and then the records from k are propagated through (back to i).

It is also possible to forward to specific domain sets by passing a list of bool to the domain_forwarding property, as seen in the following example:

Domain Violations

Domain violations occur when domain labels appear in symbol data but they do not appear in the parent set which the symbol is defined over – attempting to execute a GAMS model when there domain violations will lead to compilation errors. Domain violations are found dynamically with the <Symbol>.findDomainViolations() method.

Note

the findDomainViolations method can be computationally expensive – UELs in GAMS are case preserving (just like symbol names); additionally, GAMS ignores all trailing white space in UELs (leading white space is considered significant). As a result, transfer must lowercase all UELs and then strip any trailing white space before doing the set comparison to locate (and create) any DomainViolation objects. findDomainViolations should not be used in a loop (nor should any of its related methods: hasDomainViolations, countDomainViolations, getDomainViolations, or dropDomainViolations).

In the following example we intentionally create data with domain violations in the a parameter:

Dynamically locating domain violations allows transfer to return a view of the underlying pandas dataframe with the problematic domain labels still intact – at this point the user is free to correct issues in the UELs with any of the *UELs methods or by simply dropping any domain violations from the dataframe completely (the dropDomainViolations method is a convenience function for this operation).

Attention

It is not possible to create a GDX file if symbols have domain violations.

Unused UELs will not result in domain violations.

Attempting to write this container to a GDX file will result in an exception.

Duplicate Records

Duplicate records can easily appear in large datasets – locating and fixing these records is straightforward with transfer. transfer includes find*, has*, count* and drop* methods for duplicate records, just as it has for domain violations.

Note

the findDuplicateRecords method can be computationally expensive – UELs in GAMS are case preserving (just like symbol names); additionally, GAMS ignores all trailing white space in UELs (leading white space is considered significant). As a result, transfer must lowercase all UELs and then strip any trailing white space before doing the set comparison to locate duplicate records. findDuplicateRecords should not be used in a loop (nor should any of its related methods: hasDuplicateRecords, countDuplicateRecords, or dropDuplicateRecords).

Dynamically locating duplicate records allows transfer to return a view of the underlying pandas dataframe with the problematic domain labels still intact – at this point the user is free to correct issues in the UELs with any of the *UELs methods or by simply dropping any duplicate records from the dataframe completely (the dropDuplicateRecords method is a convenience function for this operation).

Note

The user can decide which duplicate records they would like keep with keep="first" (default), keep="last", or keep=False (which returns all duplicate records)

Attention

It is not possible to create a GDX file if symbols have duplicate records.

Attempting to write this container to a GDX file will result in an exception.

Pivoting Data

It might be convenient to pivot data into a multi-dimensional data structure rather than maintaining the flat structure in records. A convenience method called pivot is provided for all symbol classes and will return a pivoted pandas.DataFrame. Pivoting is only available for symbols with more than one dimension.

Example #1 - Pivot a 2D Set

Example #2 - Pivot a 3D Set

Note

When pivoting symbols with >2 dimensions, the first [0..(dimension-1)] dimensions will be set to the index and the last dimension will be pivoted into the columns. This behavior can be customized with the index and columns arguments.

Example #3 - Pivot a 3D Parameter w/ a fill_value

Example #4 - Pivot (only the marginal values) of a 3D Variable

Describing Data

The methods describeSets, describeParameters, describeVariables, and describeEquations allow the user to get a summary view of key data statistics. The returned DataFrame aggregates the output for a number of other methods (depending on symbol type). A description of each Container method is provided in the following subsections:

describeSets

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return.

Example #1

Example #2 – with aliases

describeParameters

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return.

Example

describeVariables

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return.

Example

describeEquations

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return.

Example

describeAliases

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return. All data is referenced from the parent set that the alias is created from.

Example

Matrix Generation

transfer stores data in a "flat" format, that is, one record entry per DataFrame row. However, it is often necessary to convert this data format into a matrix format – transfer enables users to do this with relative ease using the toDense and the toSparseCoo symbol methods. The toDense method will return a dense N-dimensional numpy array with each dimension corresponding to the GAMS symbol dimension; it is possible to output an array up to 20 dimensions (a GAMS limit). The toSparseCoo method will return the data in a sparse scipy COOrdinate format, which can then be efficiently converted into other sparse matrix formats.

Attention

Both the toDense and toSparseCoo methods do not transform the underlying DataFrame in any way, they only return the transformed data.

Note

toSparseCoo will only convert 2-dimensional data to the scipy COOrdinate format. A user interested in sparse data for an N-dimensional symbol will need to decide how to reshape the dense array in order to generate the 2D sparse format.

Attention

In order to use the toSparseCoo method the user will need to install the scipy package. Scipy is not provided with GMSPython.

Both the toDense and toSparseCoo method leverage the indexing that comes along with using categorical data types to store domain information. This means that linking symbols together (by passing symbol objects as domain information) impacts the size of the matrix. This is best demonstrated by a few examples.

Example (1D data w/o domain linking (i.e., a relaxed domain))

Note that the parameter a is not linked to another symbol, so when converting to a matrix, the indexing is referenced to the data structure in a.records. Defining a sparse parameter a over a set i allows us to extract information from the i domain and construct a very different dense matrix, as the following example shows:

Example (1D data w/ domain linking (i.e., a regular domain))

Example (2D data w/ domain linking)

The Universe Set

A Unique Element (UEL) is an (i,s) pair where i (or index) is an identification number for a (string) label s. GAMS uses UELs to efficiently store domain entries of a record by storing the UEL ID i of a domain entry instead of the actual string s. This avoids storing the same string multiple times. The concept of UELs also exists in Python/Pandas and is called a "categorical series". transfer leverages these types in order to efficiently store strings and enable domain checking within the Python environment.

Each domain column in a DataFrame can be assigned a unique categorical type, the effect is that each symbol maintains its own list of UELs per dimension. It is possible to convert a categorical column to its ID number representation by using the categorical accessor x.records[<domain_column_label>].cat.codes; however, this type of data manipulation is not necessary within transfer, but could be handy when debugging data.

Pandas offers the possibility to create categorical column types that are ordered or not; transfer relies exclusively on ordered categorical data types (in order for a symbol to be valid it must have only ordered categories). By using ordered categories, transfer will order the UEL such that elements appear in the order in which they appeared in the data (which is how GAMS defines the UEL). transfer allows the user to reorder the UELs with the uel_priority argument in the .write() method.

transfer does not actually keep track of the UEL separately from other symbols in the Container, it will be created internal to the .write() method and is based on the order in which data is added to the container. The user can access the current state of the UEL with the .getUELs() container method. For example, we set a two dimensional set:

Pandas also includes a number of methods that allow categories to be renamed, appended, etc. These methods may be useful for advanced users, but most users will probably find that modifying the original data structures and resetting the symbol records provides a simpler solution. The design of transfer should enable the user to quickly move data back and forth, without worrying about the deeper mechanics of categorical data.

Customize the Universe Set

The concept of a universe set is fundamental to GAMS and has consequences in many areas of GAMS programming including the order of loop execution. For example:

set final_model_year / 2030 /;
set t "all model years" / 2022*2030 /;

singleton set my(t) "model solve year";


loop(t,
  my(t) = yes;
  display my;
  );

The loop will execute model solve year 2030 first because the UEL 2030 was defined in the set final_model_year before it was used again in the definition of set t. This could lead to some surprising behavior if model time periods are linked together. Many GAMS users would create a dummy set (perhaps the first line of their model file) that contained all the UELs that had a significant order tom combat this behavior. transfer allows for full control (renaming as well as ordering) over the universe set through the *UELS methods, briefly described here:

Quick summary table of UELs functions

These tools are extremely useful when data is arriving at a model from a variety of data sources. We will describe each of these functions in detail and provide examples in the following sections.

Attention

GAMS is insensitive to trailing whitespaces, the *UELs methods will automatically trim any trailing whitespace when creating the new UELs.

getUELs Examples

getUELs is a method of all GAMS symbol classes as well as the Container class. This allows the user to retrieve (ordered) UELs from the entire container or just a specific symbol dimension. For example:

addUELs Examples

addUELs is a method of all GAMS symbol classes. This method allows the user to add in new UELs labels to a specific dimension of a symbol – the user can add UELs that do not exist in the symbol records. For example:

In this example we have added three new (unused) UELs: ham, and, cheese. These three UELs will now appear in the GAMS universe set (accessible with m.getUELs()). The addition of unused UELs does not impact the validity of the symbols (i.e., unused UELs will not trigger domain violations).

removeUELs Examples

removeUELs is a method of all GAMS symbol classes as well as the Container class. As a result, this method allows the user to clean up unwanted or simply unused UELs in a symbol dimension(s), over several symbols, or over the entire container. The previous example added three unused UELs (ham, and, cheese), but now we want to remove these UELs in order to clean up the GAMS universe set. We can accomplish this several ways:

In all cases the resulting universe set will be:

If a user removes a UEL that appears in data, that data will be lost permanently. The domain label will be transformed into an NaN as seen in this example:

Attention

A container cannot be written if there are NaN entries in any of the domain columns (in any symbol) – an Exception is raised if there are missing domain labels.

renameUELs Examples

renameUELs is a method of all GAMS symbol classes as well as the Container class. This method allows the user to rename UELs in a symbol dimension(s), over several symbols, or over the entire container. This particular method is very handy when attempting to harmonize labeling schemes between data structures that originated from different sources. For example:

...results in the following records:

However, two different data sources were used to generate the parameters a and b – one data source used the uppercase postal abbreviation of the state name and the other source used a lowercase full state name as the unique identifier. With the following syntax the user would be able to harmonize to a mixed case postal code labeling scheme (without losing any of the original UEL ordering).

...results in the following records (and the universe set):

The universe set will now be:

It is possible that some data needs to be cleaned and multiple UELs need to be mapped to a single label (within a single dimension). This is not allowed under default behavior because transfer assumes that the provided UELs are truly unique (logically and lexicographically) – however, it might be necessary recreate the underlying categorical object to combine n (previously unique) UELs into one to establish the necessary logical set links. For example:

The records are unique for a, but logically, there might be a need to rename WI to WISCONSIN.

In order achieve the desired behavior it is necessary to pass allow_merge=True to renameUELs:

reorderUELs Examples

reorderUELs is a method of all GAMS symbol classes. This method allows the user to reorder UELs of a specific symbol dimension – reorderUELs will not all any new UELs to be create nor can they be removed. For example:

But perhaps we want to reorder the UELs i1, i2, i3 to i3, i2, i1.

Note

This example does not change the indexing scheme of the Pandas DataFrame at all, it only changes the underlying integer numbering scheme for the categories. We can see this by looking at the Pandas codes:

setUELs Examples

reorderUELs is a method of all GAMS symbol classes. This method allows the user to create new UELs, rename UELs, and reorder UELs all in one method. For example:

A user could accomplish a UEL reorder operation with setUELs:

A user could accomplish a UEL reorder + add UELs operation with setUELs:

A user could accomplish a UEL reorder + add + rename with setUELs:

Note

This example does not change the indexing scheme of the Pandas DataFrame at all, but the rename=True flag means that the records will get updated just as if a renameUELs call had been made.

If a user wanted to set new UELs on top of this data, without renaming, they would need to be careful to include the current UELs in the UELs being set. It is possible to loose these labels if they are not included (which will prevent the data from being written to GDX/GMD).

String Manipulation on UELs

It is easy to perform common string manipulations on UELs at the dimension, symbol and container levels with a series of convenience functions: lowerUELs, upperUELs, lstripUELs, rstripUELs, stripUELs, capitalizeUELs, casefoldUELs, titleUELs, ljustUELs, rjustUELs. These methods are wrappers around Python's built in string methods and are designed to efficiently perform bulk UEL transformations on your GAMS data.

The following example shows operations on the entire container:

Note

The ljustUELs and rjustUELs methods require the user to specify the final string length and the fill_character used to pad the string to achieve the final length.

Similar operations can be performed at the dimension and symbol levels as can be seen in the following examples:

Note

Symbol dimension is indexed from zero (per Python convention)

Reordering Symbols

The order of the Container file requires the symbols to be sorted such that, for example, a Set used as domain of another symbol appears before that symbol. The Container will try to establish a valid ordering when writing the data. This type of situation could be encountered if the user is adding and removing many symbols (and perhaps rewriting symbols with the same name) – users should attempt to only add symbols to a Container once, and care must be taken when creating symbol names. The method reorderSymbols attempts to fix symbol ordering problems. The following example shows how this can occur:

Example Symbol reordering

The symbols are now out of order in .data and must be reordered:

Rename Symbols

It is possible to rename a symbol even after it has been added to a Container. There are two methods that can be used to achieve the desired outcome:

• using the container method renameSymbol
• directly changing the name symbol property

We create a Container with two sets:

Example #1 - Change the name of a symbol with the container method

Example #2 - Change the name of a symbol with the .name attribute

Note

Note that the renamed symbols maintain the original symbol order, this will prevent unnecessary reordering operations later in the workflow.

Removing Symbols

Removing symbols from a container is easy when using the removeSymbols container method; this method accepts either a str or a list of str.

Attention

Once a symbol has been removed, it is possible to have hanging references as domain links in other symbols. The user will need to repair these other symbols with the proper domain links in order to avoid validity errors.

GAMS Special Values

The GAMS system contains five special values: UNDEF (undefined), NA (not available), EPS (epsilon), +INF (positive infinity), -INF (negative infinity). These special values must be mapped to their Python equivalents. transfer follows the following convention to generate the 1:1 mapping:

• +INF is mapped to float("inf")
• -INF is mapped to float("-inf")
• EPS is mapped to -0.0 (mathematically identical to zero)
• NA is mapped to a special NaN
• UNDEF is mapped to float("nan")

transfer syntax is designed to quickly get data into a form that is usable in further analyses or visualization; this mapping also highlights the preference for data that is of type float, which offers performance benefits within Pandas/NumPy. The user does not need to remember these constants as they are provided within the class SpecialValues as SpecialValues.POSINF, SpecialValues.NEGINF, SpecialValues.EPS, SpecialValues.NA, and SpecialValues.UNDEF. The SpecialValues class also contains methods to test for these special values. Some examples are shown below; already, we, begin to introduce some of the transfer syntax.

Example (special values in a parameter)

The following DataFrame for x would look like:

The user can now easily test for specific special values in the value column of the DataFrame (returns a boolean array):

Other data structures can be passed into these methods as long as these structures can be converted into a numpy array with dtype=float. It follows that:

Pandas DataFrames allow data columns to exist with mixed type (dtype=object) – transfer leverages this convenience feature to enable users to import string representations of EPS, NA, and UNDEF (or UNDF). transfer is tolerant of any mixed-case special value string representation. Python offers additional flexibility when representing negative/positive infinity. Any string x where float(x) == float("inf") evaluates to True can be used to represent positive infinity. Similarly, any string x where float(x) == float("-inf") evaluates to True can be used to represent negative infinity. Allowed values include inf, +inf, INFINITY, +INFINITY, -inf, -INFINITY and all mixed-case equivalents.

Example (special values defined by strings)

These special strings will be immediately mapped to their float equivalents from the SpecialValues class in order to ensure that all data entries are float types.

Standard Data Formats

This section is meant to introduce the standard format that transfer expects for symbol records. It has already been mentioned that we store data as a Pandas DataFrame, but there is an assumed structure to the column headings and column types that will be important to understand. transfer includes convenience functions in order to ease the burden of converting data from a user-centric format to one that is understood by transfer. However, advanced users will want to convert their data first and add it directly to the Container to avoid making extra copies of (potentially large) data sets.

Set Records Standard Format

All set records (including singleton sets) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 1. The first n-1 columns include the domain elements while the last column includes the set element explanatory text. Records are organized such that there is one record per row.

The names of the domain columns are flexible, but transfer requires unique column names. Users are encouraged to change the column headings of the underlying dataframe by using the domain_labels property. Using this property will ensure that unique column names are generated by adding a _<dimension> tag to the end of any user supplied column names. The explanatory text column is called element_text and must take the last position in the DataFrame.

All domain columns must be a categorical data type and the element_text column must be a object type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however transfer requires that all these are type str. All rows in the element_text column must be type str.

Some examples:

Parameter Records Standard Format

All parameter records (including scalars) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 1. The first n-1 columns include the domain elements while the last column includes the numerical value of the records. Records are organized such that there is one record per row. Scalar parameters have zero dimension, therefore they only have one column and one row.

By default, the names of the domain columns follow a pattern of <set_name>; a symbol dimension that is referenced to the universe is labeled uni. The domain labels can be customized. Users are encouraged to change the column headings of the underlying dataframe by using the domain_labels property. Using this property will ensure that unique column names are generated (if not currently unique) by adding a _<dimension> tag to the end of any user supplied column names. The value column is called value and must take the last position in the DataFrame.

All domain columns must be a categorical data type and the value column must be a float type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however transfer requires that all these are type str.

Some examples:

Variable/Equation Records Standard Format

Variables and equations share the same standard data format. All records (including scalar variables/equations) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 5. The first n-5 columns include the domain elements while the last five columns include the numerical values for different attributes of the records. Records are organized such that there is one record per row. Scalar variables/equations have zero dimension, therefore they have five columns and one row.

By default, the names of the domain columns follow a pattern of <set_name>; a symbol dimension that is referenced to the universe is labeled uni. The domain labels can be customized. Users are encouraged to change the column headings of the underlying dataframe by using the domain_labels property. Using this property will ensure that unique column names are generated (if not currently unique) by adding a _<dimension> tag to the end of any user supplied column names. The attribute columns are called level, marginal, lower, upper, and scale. These attribute columns must appear in this order. Attributes that are not supplied by the user will be assigned the default GAMS values for that variable/equation type; it is possible to not pass any attributes, transfer would then simply assign default values to all attributes.

All domain columns must be a categorical data type and all the attribute columns must be a float type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however transfer requires that all these are type str.

Some examples:

GDX Read/Write

Up until now, we have been focused on using transfer to create symbols in an empty Container using the symbol constructors (or their corresponding container methods). These tools will enable users to ingest data from many different formats and add them to a Container – however, it is also possible to read in symbol data directly from GDX files using the read container method. In the following sections, we will discuss this method in detail as well as the write method, which allows users to write out to new GDX files.

Read GDX

There are two main ways to read in GDX based data.

• Pass the file path directly to the Container constructor (will read all symbols and records)
• Pass the file path directly to the read method (default read all symbols, but can read partial files)

The first option here is provided for convenience and will, internally, call the read method. This method will read in all symbols as well as their records. This is the easiest and fastest way to get data out of a GDX file and into your Python environment. For the following examples we leverage the GDX output generated from the `trnsport.gms` model file.

Example (reading full data w/ Container constructor)

A user could also read in data with the read method as shown in the following example.

Example (reading full data w/ read method)

It is also possible to read in a partial GDX file with the read method, as shown in the following example:

This syntax assumes that the user will always want to read in both the metadata as well as the actual data records, but it is possible to skip the reading of the records by passing the argument records=False.

Attention

The read method attempts to link the domain objects together (in order to have a "regular" domain_type) but if domain sets are not part of the read operation there is no choice but to default to a "relaxed" domain_type. This can be seen in the last example where we only read in the variable x and not the domain sets (i and j) that the variable is defined over. All the data will be available to the user, but domain checking is no longer possible. The symbol x will remain with "relaxed" domain type even if the user were to read in sets i and j in a second read call.

Write GDX

A user can write data to a GDX file by simply passing a file path (as a string). The write method will then create the GDX and write all data in the Container.

Example

Example (write a compressed GDX file)

By default, all symbols in the Container will be written, however it is possible to write a subset of the symbols to a GDX file with the symbols argument. If a domain set is not included in the symbols list then the symbol will automatically be relaxed (but will retain the domain set's name as a string label – it does not get relaxed to *). This behavior can be seen in the following example.

In line 4 we can see that the auto-relaxation of the domain for a is only temporary for writing (in this case, from Container object m) and will be restored so as not to disturb the Container state.

Advanced users might want to specify an order to their UEL list (i.e., the universe set); recall that the UEL ordering follows that dictated by the data. As a convenience, it is possible to prepend the UEL list with a user specified order using the uel_priority argument.

Example (change the order of the UEL)

The original UEL order for this GDX file would have been ["a", "b", "c"], but this example reorders the UEL with uel_priority – the positions of b and c have been swapped. This can be verified with the gdxdump utility (using the uelTable argument):

gdxdump foo.gdx ueltable=foo

Set foo /
  'a' ,
  'c' ,
  'b' /;
$onEmpty

Set i(*) /
'a',
'c',
'b' /;

$offEmpty

GamsDatabase Read/Write

We have discussed how to create symbols in an empty Container and we have discussed how to exchange data with GDX files, however it is also possible to read and write data directly in memory by interacting with a GamsDatabase/GMD object – this allows transfer to be used to read/write data within an Embedded Python Code environment or in combination with the Python OO API. There are some important differences when compared to data exchange with GDX since we are working with data representations in memory.

Read GamsDatabases

Just as with a GDX, there are two main ways to read in data that is in a GamsDatabase/GMD object.

• Pass the GamsDatabase/GMD object directly to the Container constructor (will read all symbols and records)
• Pass the GamsDatabase/GMD object directly to the read method (default read all symbols, but can read partial files)

The first option here is provided for convenience and will, internally, call the read method. This method will read in all symbols as well as their records. This is the easiest and fastest way to get data out of a GamsDatabase/GMD object and into your Python environment. While it is possible to generate a custom GamsDatabase/GMD object from scratch (using the gmdcc API), most users will be interacting with a GamsDatabase/GMD object that has already been instantiated internally when he/she is using Embedded Python Code or the GamsDatabase class in the Python OO API. Our examples will show how to access the GamsDatabase/GMD object – we leverage the some of the data from the `trnsport.gms` model file.

Example (reading full data w/ Container constructor)

Note

Embedded Python Code users will want pass the GamsDatabase object that is part of the GAMS Database object – this will always be referenced as gams.db regardless of the model file.

The following example uses embedded Python code to create a new Container, read in all symbols, and display some summary statistics as part of the gams log output.

Set
   i 'canning plants' / seattle,  san-diego /
   j 'markets'        / new-york, chicago, topeka /;

Parameter
   a(i) 'capacity of plant i in cases'
        / seattle    350
          san-diego  600 /

   b(j) 'demand at market j in cases'
        / new-york   325
          chicago    300
          topeka     275 /;

Table d(i,j) 'distance in thousands of miles'
              new-york  chicago  topeka
   seattle         2.5      1.7     1.8
   san-diego       2.5      1.8     1.4;

$onembeddedCode Python:
import gams.transfer as gt

m = gt.Container(gams.db)
print(m.describeSets())

print(m.describeParameters())

$offEmbeddedCode

The gams log output will then look as such (the extra print calls are just providing nice spacing for this example):

GAMS 43.1.0   Copyright (C) 1987-2023 GAMS Development. All rights reserved
--- Starting compilation
--- matrix.gms(29) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
  name  is_singleton domain domain_type  dimension  number_records sparsity
0    i         False    [*]        none          1               2     None
1    j         False    [*]        none          1               3     None
  name  domain domain_type  dimension  number_records      min     mean      max            where_min            where_max sparsity
0    a     [i]     regular          1               2  350.000  475.000  600.000            [seattle]          [san-diego]      0.0
1    b     [j]     regular          1               3  275.000  300.000  325.000             [topeka]           [new-york]      0.0
2    d  [i, j]     regular          2               6    1.400    1.950    2.500  [san-diego, topeka]  [seattle, new-york]      0.0

--- Starting execution - empty program
*** Status: Normal completion

[3 rows x 16 columns]

--- Starting execution - empty program
*** Status: Normal completion

A user could also read in a subset of the data located in the GamsDatabase object with the read method as shown in the following example. Here we only read in the sets i and j, as a result the .describeParameters() method will return None.

Example (reading subset of full data w/ read method)

Set
   i 'canning plants' / seattle,  san-diego /
   j 'markets'        / new-york, chicago, topeka /;

Parameter
   a(i) 'capacity of plant i in cases'
        / seattle    350
          san-diego  600 /

   b(j) 'demand at market j in cases'
        / new-york   325
          chicago    300
          topeka     275 /;

Table d(i,j) 'distance in thousands of miles'
              new-york  chicago  topeka
   seattle         2.5      1.7     1.8
   san-diego       2.5      1.8     1.4;

$onembeddedCode Python:
import gams.transfer as gt

m = gt.Container()
m.read(gams.db, symbols=["i","j"])
gams.printLog("")
print(m.describeSets())
print(m.describeParameters())

$offEmbeddedCode

GAMS 43.1.0   Copyright (C) 1987-2023 GAMS Development. All rights reserved
--- Starting compilation
--- matrix.gms(29) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- name  is_singleton domain domain_type  dimension  number_records sparsity
0    i         False    [*]        none          1               2     None
1    j         False    [*]        none          1               3     None
None

--- Starting execution - empty program
*** Status: Normal completion

All the typical functionality of the Container exists when working with GamsDatabase/GMD objects. This means that domain linking, matrix conversion, and other more advanced options are available to the user at either compilation time or execution time (depending on the Embedded Code syntax being used, see: Syntax). The next example generates a 1000x1000 matrix and then takes its inverse using the Numpy linalg package.

Example (Matrix Generation and Inversion)

set i / i1*i1000 /;
alias(i,j);

parameter a(i,j);
a(i,j) = 1 / (ord(i)+ord(j) - 1);
a(i,i) = 1;


embeddedCode Python:
import gams.transfer as gt
import numpy as np
import time

gams.printLog("")
s = time.time()
m = gt.Container(gams.db)
gams.printLog(f"read data: {round(time.time() - s, 3)} sec")

s = time.time()
A = m["a"].toDense()
gams.printLog(f"create matrix A: {round(time.time() - s, 3)} sec")

s = time.time()
invA = np.linalg.inv(A)
gams.printLog(f"generate inv(A): {round(time.time() - s, 3)} sec")

endEmbeddedCode

Note

In this example, the assignment of the a parameter is done during execution time so we must use the execution time syntax for embedded code in order to get the numerical records properly.

GAMS 43.1.0   Copyright (C) 1987-2023 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(27) 3 Mb
--- Starting execution: elapsed 0:00:00.003
--- test.gms(9) 36 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---
--- read data: 1.1 sec
--- create matrix A: 0.02 sec
--- generate inv(A): 0.031 sec
*** Status: Normal completion

We will extend this example in the next section to write the inverse matrix A back into a GAMS parameter.

Write to GamsDatabases

A user can write to a GamsDatabase/GMD object with the .write() method just as he/she would write a GDX file – however there are some important differences. When a user writes a GDX file the entire GDX file represents a complete data environment (all domains have been resolved, etc.) thus, transfer does not need to worry about merge/replace operations. It is possible to merge/replace symbol records when a user is writing data to in-memory data representations with GamsDatabase/GMD. We show a few examples to illustrate this behavior.

Example (Populating a set in GAMS)

* note that we need to declare the set i over "*" in order to provide hints about the symbol dimensionality
set i(*);

$onembeddedCode Python:
import gams.transfer as gt

m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
m.write(gams.db)

$offEmbeddedCode i


embeddedCode Python:
import gams.transfer as gt

m = gt.Container(gams.db)
gams.printLog("")
print(m["i"].records)

endEmbeddedCode

Note

In general, it is possible to use transfer to create new symbols in a GamsDatabase and GMD object (and not necessarily merge symbols) but embedded code best practices necessitate the declaration of any GAMS symbols on the GAMS side first, then the records can be filled with transfer.

If we break down this example we can see that the set i is declared within GAMS (with no records) and then the records for i are set by writing a Container to the gams.db GamsDatabase object (we do this at compile time). The second embedded Python code block runs at execution time and is simply there to read all the records on the set i – printing the sets this way adds the output to the .log file (we could also use the more common display i; operation in GAMS to display the set elements in the LST file).

GAMS 43.1.0   Copyright (C) 1987-2023 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(10) 2 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- test.gms(20) 3 Mb
--- Starting execution: elapsed 0:00:01.464
--- test.gms(13) 4 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   uni   element_text
0    i0
1    i1
2    i2
3    i3
4    i4
5    i5
6    i6
7    i7
8    i8
9    i9

*** Status: Normal completion

Example (Merging set records)

set i / i1, i2 /;

$onmulti
$onembeddedCode Python:
import gams.transfer as gt

m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
m.write(gams.db, merge_symbols="i")

$offEmbeddedCode i
$offmulti

embeddedCode Python:
import gams.transfer as gt

m = gt.Container(gams.db)
gams.printLog("")
print(m["i"].records)

endEmbeddedCode

In this example we need to make use of $onMulti/$offMulti in order to merge new set elements into the the set i (the same would be true if we were merging other symbol types) – any symbol that already has records defined (in GAMS) and is being added to with Python (and transfer) must be wrapped with $onMulti/$offMulti. As with the previous example, the second embedded Python code block runs at execution time and is simply there to read all the records on the set i. Note that the UEL order will be different in this case (i1 and i2 come before i0).

GAMS 43.1.0   Copyright (C) 1987-2023 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(11) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- test.gms(21) 3 Mb
--- Starting execution: elapsed 0:00:01.535
--- test.gms(14) 4 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   uni   element_text
0    i1
1    i2
2    i0
3    i3
4    i4
5    i5
6    i6
7    i7
8    i8
9    i9

*** Status: Normal completion

Example (Replacing set records)

set i / x1, x2 /;

$onmultiR
$onembeddedCode Python:
import gams.transfer as gt

m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
m.write(gams.db)

$offEmbeddedCode i
$offmulti

embeddedCode Python:
import gams.transfer as gt

m = gt.Container(gams.db)
gams.printLog("")
print(m["i"].records)

endEmbeddedCode

In this example we want to replace the x1 and x2 set elements and built up a totally new element list with set elements from the Container. Instead of $onMulti/$offMulti we must use $onMultiR/$offMulti to ensure that the replacement happens in GAMS; we also need to remove the set i from the merge_symbols argument.

Attention

If the user seeks to replace all records in a symbol they must use the $onMultiR syntax. It is not sufficient to simply remove them from the merge_symbols argument in transfer. If the user mistakenly uses $onMulti the symbols will end up merging without total replacement.

GAMS 43.1.0   Copyright (C) 1987-2023 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(11) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- test.gms(21) 3 Mb
--- Starting execution: elapsed 0:00:01.482
--- test.gms(14) 4 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   uni   element_text
0    i0
1    i1
2    i2
3    i3
4    i4
5    i5
6    i6
7    i7
8    i8
9    i9

*** Status: Normal completion

Example (Merging parameter records)

set i;
parameter a(i<) /
i1 1.23
i2 5
/;

$onmulti
$onembeddedCode Python:
import gams.transfer as gt

m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
a = gt.Parameter(m, "a", domain=i, records=[("i"+str(i),i) for i in range(10)])
m.write(gams.db, merge_symbols="a")

$offEmbeddedCode i, a
$offmulti

embeddedCode Python:
import gams.transfer as gt

m = gt.Container(gams.db)
gams.printLog("")
print(m["a"].records)
endEmbeddedCode

In this example we also need to make use of $onMulti/$offMulti in order to merge new set elements into the the set i, however the set i also needs to contain the elements that are defined in the parameter – here we make use of the < operator that will add the set elements from a(i) into the set i

Note

It would also be possible to run this example by explicitly defining the set i /i1, i2/; before the parameter declaration.

Attention

transfer will overwrite all duplicate records when merging. The original values of a("i1") and a("i2") have been replaced with their new values when writing the Container in this example (see output below).

GAMS 43.1.0   Copyright (C) 1987-2023 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(16) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- test.gms(25) 3 Mb
--- Starting execution: elapsed 0:00:01.467
--- test.gms(19) 4 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   i    value
0  i1    1.0
1  i2    2.0
2  i3    3.0
3  i4    4.0
4  i5    5.0
5  i6    6.0
6  i7    7.0
7  i8    8.0
8  i9    9.0

*** Status: Normal completion

Example (Advanced Matrix Generation and Inversion w/ Write Operation)

set i / i1*i1000 /;
alias(i,j);

parameter a(i,j);
a(i,j) = 1 / (ord(i)+ord(j) - 1);
a(i,i) = 1;

parameter inv_a(i,j);
parameter ident(i,j);

embeddedCode Python:
import gams.transfer as gt
import numpy as np
import time

gams.printLog("")
gams.printLog("")

s = time.time()
m = gt.Container(gams.db)
gams.printLog(f"read data: {round(time.time() - s, 3)} sec")

s = time.time()
A = m["a"].toDense()
gams.printLog(f"create matrix A: {round(time.time() - s, 3)} sec")

s = time.time()
invA = np.linalg.inv(A)
gams.printLog(f"calculate inv(A): {round(time.time() - s, 3)} sec")

s = time.time()
m["inv_a"].setRecords(invA)
gams.printLog(f"convert matrix to records for inv(A): {round(time.time() - s, 3)} sec")

s = time.time()
I = np.dot(A,invA)
tol = 1e-9
I[np.where((I<tol) & (I>-tol))] = 0
gams.printLog(f"calculate A*invA + small number cleanup: {round(time.time() - s, 3)} sec")

s = time.time()
m["ident"].setRecords(I)
gams.printLog(f"convert matrix to records for I: {round(time.time() - s, 3)} sec")

s = time.time()
m.write(gams.db, ["inv_a","ident"])
gams.printLog(f"write to GamsDatabase: {round(time.time() - s, 3)} sec")

gams.printLog("")
endEmbeddedCode inv_a, ident

display ident;

In this example we extend the example shown in Read GamsDatabases to read data from GAMS, calculate a matrix inversion, do the matrix multiplication, and then write both the A^-1 and A*A^-1 (i.e., the identity matrix) back to GAMS for display in the LST file. This data round trip highlights the benefits of using a transfer Container (and the linked symbol structure) as the mechanism to move data – converting back and forth from a records format to a matrix format can be cumbersome, but here, transfer takes care of all the indexing for the user.

The first few lines of GAMS code generates a 1000x1000 A matrix as a parameter (at execution time), we then define two more parameters that we will fill with results of the embedded Python code – specifically we want to fill a parameter with the matrix A^-1 and we want to verify that another parameter (ident) contains the identity matrix (i.e., I). Stepping through the code:

1. We start the embedded Python code section (execution time) by importing both transfer and Numpy and by reading all the symbols that currently exist in the GamsDatabase. We must read in all this information in order to get the domain set information – transfer needs these domain sets in order to generate matricies with the proper size.
2. Generate the matrix A by calling .toDense() on the symbol object in the Container.
3. Take the inverse of A with np.linalg.inv().
4. The Parameter symbol for inv_a already exists in the Container, but it does not have any records (i.e., m["inv_a"].records is None will evaluate to True). We use .setRecords() to convert the invA back into a records format.
5. We continue the computations by performing the matrix multiplication using np.dot() – we must clean up a lot of small numbers in I.
6. The Parameter symbol for ident already exists in the Container, but it does not have any records. We use .setRecords() to convert I back into a records format.
7. Since we are calculating these parameter values at execution time, it is not possible to modify the domain set information (or even merge/replace it). Therefore we only want to write the parameter values to GAMS. We achieve this by writing a subset of the Container symbols out with the m.write(gams.db, ["inv_a","ident"]) call. This partial write preserves symbol validity in the Container and it does not violate other GAMS requirements.
8. Finally, we can verify that the (albeit large) identity matrix exists in the LST file (or in another GDX file).

Note

It was not possible to just use np.round because small negative numbers that round to -0.0 will be interpreted by transfer as the GAMS EPS special value.

The output for this example is shown below:

GAMS 43.1.0   Copyright (C) 1987-2023 GAMS Development. All rights reserved
--- Starting compilation
--- matrix.gms(52) 3 Mb
--- Starting execution: elapsed 0:00:00.004
--- matrix.gms(11) 36 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---
---
--- read data: 1.083 sec
--- create matrix A: 0.016 sec
--- calculate inv(A): 0.032 sec
--- convert matrix to records for inv(A): 0.176 sec
--- calculate A*invA + small number cleanup: 0.027 sec
--- convert matrix to records for I: 0.17 sec
--- write to GamsDatabase: 1.937 sec
---
--- matrix.gms(52) 68 Mb
*** Status: Normal completion

Container Read

Containers can read from other Container instances. The syntax and behavior is much the same as reading from GDX and GMD sources. It is important to note that a deepcopy of all data is made when reading from these sources. The container object can be passed into the constructor (to be consistent with the shorthand notation) or the object can be passed as a argument to the .read() method.

Combining two containers

In this example we create two containers (which could have been populated from GDX files) and add in all symbol that do not currently exist in the first Container