Table of Contents
- Validating Data
- Custom Column Headings
- Converting Records
- Comparing Symbols
- Domain Forwarding
- Domain Violations
- Duplicate Records
- Pivoting Data
- Describing Data
- Matrix Generation
- The Universe Set
- Customize the Universe Set
- Reordering Symbols
- Rename Symbols
- Removing Symbols
- GAMS Special Values
- Standard Data Formats
- GDX Read/Write
- GamsDatabase Read/Write
- Container Read
Validating Data
transfer
requires that the records for all symbols exist in a standard format (Standard Data Formats) in order for them to be understood by the Container
. It is certainly possible that the data could end up in a state that is inconsistent with the standard format (especially if setting symbol attributes directly). transfer
includes the .isValid()
method in order to determine if a symbol is structurally valid – this method returns a bool
. This method does not guarantee that a symbol will be successfully written to either GDX or GMD, other data errors (duplicate records, long UEL names, or domain violations) could exist that are not tested in .isValid()
.
For example, we create two valid sets and then check them with .isValid()
to be sure.
- Note
- It is possible to run
.isValid()
on both theContainer
as well as the symbol object –.isValid()
will also return abool
if there are any invalid symbols in theContainer
object.
- Example (valid data)
The .isValid()
method checks:
- If the symbol belongs to a Container
- If all domain set symbols exist in the Container
- If all domain set symbols objects are valid
- If any domain set is also a singleton set (not allowed in GAMS)
- If records are a DataFrame (or
None
) - If the records DataFrame is the right number of columns (based on symbol dimension)
- If the symbol is a scalar, then ensure there is only one record (row) in the DataFrame
- If records column headings are unique
- If any symbol attribute columns are missing or out of order
- If all domain columns are type
category
- If all domain categories are type
str
- If all data columns are type
float
Custom Column Headings
The names of the domain columns are flexible, but transfer
requires unique column names. Users are encouraged to change the column headings of the underlying dataframe by using the domain_labels
property. Using this property will ensure that unique column names are generated by adding a _<dimension>
tag to the end of any user supplied column names. The following examples show this behavior.
- Attention
- All
*
domains are recast asuni
. This allows users to access the column data with both the Pandas bracket and/or dot notation (i.e.,df["uni"]
ordf.uni
).
- Column heading behavior at symbol instantiation
The setRecords
(which is called internally at symbol instantiation) method will set default domain_labels
if they were not provided by the user. The only way for a user to provide domain labels with setRecords
is by passing in a Pandas DataFrame object. The _<dimension>
tag will be added to all domain labels in order to make all domain names unique – this tag is added to all dimensions if any subset of the domain names are non-unique.
- Customizing column headings
Many users may want to output the GAMS DataFrame directly to another format (CSV, etc.) and may wish to create customized DataFrame column headings for readability. User can do this by directly setting the domain_labels
property, as seen in the following example.
- Attention
- Users are encouraged to use the
<symbol>.domain_labels
property instead of setting the<DataFrame>.columns
directly. The avoids the possibility of out-of-sync symbol validity. Thedomain_labels
property does not store anything, calling this property simply returns the exact domain labels from the DataFrame.
Converting Records
All data in transfer
will be stored as a Pandas DataFrame – however, it is desirable to have easy access to data without the additional infrastructure that comes with the DataFrame object. We include to*
methods (available for all symbol types) that will return other data structures. The following examples show the behavior of toValue
, toList
, and toDict
(previous examples showed examples of toDense
and toSparseCoo
).
- Examples of toList
- Examples of toValue
- Examples of toDict
Comparing Symbols
Sparse GAMS data is inherently unordered. The concept of order is GAMS is governed by the order of the UELs in the universe set not the order of the records. This differs from the sparse data structures that we use in transfer
(Pandas DataFrames) because each record (i.e., DataFrame row) has an index (typically 0..n
) and can be sorted by this index. Said a slightly different way, two GDX files will be equivalent if their universe order is the same and the records are the same, however when creating the GDX file, it is of no consequence what order the records are written in. Therefore, in order to calculate an equality between two symbols in transfer
we must perform a merge operation on the symbol domain labels – an operation that could be computationally expensive for large symbols.
- Attention
- The nature of symbol equality in
transfer
means that a potentially expensive merge operation is performed, we do not recommend that theequals
method be used inside loops or when speed is critical. It is, however, very useful for data debugging.
A quick example shows the syntax of equals
:
By default, equals
takes the strictest view of symbol "equality" – everything must be the same. In this case, the symbol names and descriptions differ between the two sets i
and j
. We can relax the view of equality with a combination of argument flags. Comparing the two symbols again, but ignoring the meta data (i.e., ignoring the symbol name, description and type (if a Variable or Equation)):
It is also possible to ignore the set element text in equals
:
The check_uels
argument will ensure that the symbol "universe" is the same (in order and content) between two symbols, as illustrated in the following example:
Clearly, the two sets i
and ip
have the same records, but the UEL order is different. If check_uels=True
the resulting symbols will not be considered equal – turning this flag off results in equality.
Numerical comparisons are enabled for Parameters
, Variables
and Equations
– equality can be flexibly defined through the equals
method arguments. Again, the strictest view of equality is taken as the default behavior of equals
(no numerical tolerances, some limitations exist – see: numpy.isclose
for more details).
- Attention
- The numerical comparison is handled by
numpy.isclose
, more details can be found in the Numpy documentation.
In the case of variables and equations, it is possible for the user to confine the numerical comparison to certain certain attributes (level
, marginal
, lower
, upper
and scale
) by specifying the columns
argument as the following example illustrates:
Domain Forwarding
GAMS includes the ability to define sets directly from data using the implicit set notation (see: Implicit Set Definition (or: Domain Defining Symbol Declarations)). This notation has an analogue in transfer
called domain_forwarding
.
- Note
- It is possible to recursively update a subset tree in
transfer
.
Domain forwarding is available as an argument to all symbol object constructors; the user would simply need to pass domain_forwarding=True
.
In this example we have raw data that in the dist
DataFrame and we want to send the domain information into the i
and j
sets – we take care to pass the set objects as the domain for parameter c
.
- Note
- The element order in the sets
i
andj
mirrors that in the raw data.
In this example we show that domain forwarding will also work recursively to update the entire set lineage – the domain forwarding occurs at the creation of every symbol object. The correct order of elements in set i
is [z, a, b, c]
because the records from j
are forwarded first, and then the records from k
are propagated through (back to i
).
It is also possible to forward to specific domain sets by passing a list
of bool
to the domain_forwarding
property, as seen in the following example:
Domain Violations
Domain violations occur when domain labels appear in symbol data but they do not appear in the parent set which the symbol is defined over – attempting to execute a GAMS model when there domain violations will lead to compilation errors. Domain violations are found dynamically with the <Symbol>.findDomainViolations()
method.
- Note
- the
findDomainViolations
method can be computationally expensive – UELs in GAMS are case preserving (just like symbol names); additionally, GAMS ignores all trailing white space in UELs (leading white space is considered significant). As a result,transfer
must lowercase all UELs and then strip any trailing white space before doing the set comparison to locate (and create) anyDomainViolation
objects.findDomainViolations
should not be used in a loop (nor should any of its related methods:hasDomainViolations
,countDomainViolations
,getDomainViolations
, ordropDomainViolations
).
In the following example we intentionally create data with domain violations in the a
parameter:
Dynamically locating domain violations allows transfer
to return a view of the underlying pandas dataframe with the problematic domain labels still intact – at this point the user is free to correct issues in the UELs with any of the *UELs
methods or by simply dropping any domain violations from the dataframe completely (the dropDomainViolations
method is a convenience function for this operation).
- Attention
- It is not possible to create a GDX file if symbols have domain violations.
- Unused UELs will not result in domain violations.
Attempting to write this container to a GDX file will result in an exception.
Duplicate Records
Duplicate records can easily appear in large datasets – locating and fixing these records is straightforward with transfer
. transfer
includes find*
, has*
, count*
and drop*
methods for duplicate records, just as it has for domain violations.
- Note
- the
findDuplicateRecords
method can be computationally expensive – UELs in GAMS are case preserving (just like symbol names); additionally, GAMS ignores all trailing white space in UELs (leading white space is considered significant). As a result,transfer
must lowercase all UELs and then strip any trailing white space before doing the set comparison to locate duplicate records.findDuplicateRecords
should not be used in a loop (nor should any of its related methods:hasDuplicateRecords
,countDuplicateRecords
, ordropDuplicateRecords
).
Dynamically locating duplicate records allows transfer
to return a view of the underlying pandas dataframe with the problematic domain labels still intact – at this point the user is free to correct issues in the UELs with any of the *UELs
methods or by simply dropping any duplicate records from the dataframe completely (the dropDuplicateRecords
method is a convenience function for this operation).
- Note
- The user can decide which duplicate records they would like
keep
withkeep="first"
(default),keep="last"
, orkeep=False
(which returns all duplicate records)
- Attention
- It is not possible to create a GDX file if symbols have duplicate records.
Attempting to write this container to a GDX file will result in an exception.
Pivoting Data
It might be convenient to pivot data into a multi-dimensional data structure rather than maintaining the flat structure in records
. A convenience method called pivot
is provided for all symbol classes and will return a pivoted pandas.DataFrame
. Pivoting is only available for symbols with more than one dimension.
- Example #1 - Pivot a 2D Set
- Example #2 - Pivot a 3D Set
- Note
- When pivoting symbols with >2 dimensions, the first [0..(dimension-1)] dimensions will be set to the index and the last dimension will be pivoted into the columns. This behavior can be customized with the
index
andcolumns
arguments.
- Example #3 - Pivot a 3D Parameter w/ a fill_value
- Example #4 - Pivot (only the marginal values) of a 3D Variable
Describing Data
The methods describeSets
, describeParameters
, describeVariables
, and describeEquations
allow the user to get a summary view of key data statistics. The returned DataFrame aggregates the output for a number of other methods (depending on symbol type). A description of each Container
method is provided in the following subsections:
describeSets
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , str , NoneType | A list of sets in the Container to include in the output. describeSets will include aliases if they are explicitly passed by the user. | No | None (if None specified, will assume all sets – not aliases) |
Returns: pandas.DataFrame
The following table includes a short description of the column headings in the return.
Property / Statistic | Description |
---|---|
name | name of the symbol |
is_singleton | bool if the set/alias is a singleton set (or an alias of a singleton set) |
alias_with | [OPTIONAL if users passes an alias name as part of symbols ] name of the parent set (for alias only), None otherwise |
domain | domain labels for the symbol |
domain_type | none , relaxed or regular depending on the symbol state |
dimension | dimension |
number_records | number of records in the symbol |
sparsity | 1 - number_records/cardinality |
- Example #1
- Example #2 – with aliases
describeParameters
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , str , NoneType | A list of parameters in the Container to include in the output | No | None (if None specified, will assume all parameters) |
Returns: pandas.DataFrame
The following table includes a short description of the column headings in the return.
Property / Statistic | Description |
---|---|
name | name of the symbol |
domain | domain labels for the symbol |
domain_type | none , relaxed or regular depending on the symbol state |
dimension | dimension |
number_records | number of records in the symbol |
min | min value in data |
mean | mean value in data |
max | max value in data |
where_min | domain of min value (if multiple, returns only first occurrence) |
where_max | domain of max value (if multiple, returns only first occurrence) |
sparsity | 1 - number_records/cardinality |
- Example
describeVariables
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , str , NoneType | A list of variables in the Container to include in the output | No | None (if None specified, will assume all variables) |
Returns: pandas.DataFrame
The following table includes a short description of the column headings in the return.
Property / Statistic | Description |
---|---|
name | name of the symbol |
type | type of variable (i.e., binary , integer , positive , negative , free , sos1 , sos2 , semicont , semiint ) |
domain | domain labels for the symbol |
domain_type | none , relaxed or regular depending on the symbol state |
dimension | dimension |
number_records | number of records in the symbol |
sparsity | 1 - number_records/cardinality |
min_level | min value in the level |
mean_level | mean value in the level |
max_level | max value in the level |
where_max_abs_level | domain of max(abs(level )) in data |
- Example
describeEquations
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , str , NoneType | A list of equations in the Container to include in the output | No | None (if None specified, will assume all equations) |
Returns: pandas.DataFrame
The following table includes a short description of the column headings in the return.
Property / Statistic | Description |
---|---|
name | name of the symbol |
type | type of variable (i.e., binary , integer , positive , negative , free , sos1 , sos2 , semicont , semiint ) |
domain | domain labels for the symbol |
domain_type | none , relaxed or regular depending on the symbol state |
dimension | dimension |
number_records | number of records in the symbol |
sparsity | 1 - number_records/cardinality |
min_level | min value in the level |
mean_level | mean value in the level |
max_level | max value in the level |
where_max_abs_level | domain of max(abs(level )) in data |
- Example
describeAliases
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , str , NoneType | A list of alias (only) symbols in the Container to include in the output | No | None (if None specified, will assume all aliases – not sets) |
Returns: pandas.DataFrame
The following table includes a short description of the column headings in the return. All data is referenced from the parent set that the alias is created from.
Property / Statistic | Description |
---|---|
name | name of the symbol |
alias_with | name of the parent set (for alias only), None otherwise |
is_singleton | bool if the set/alias is a singleton set (or an alias of a singleton set) |
domain | domain labels for the symbol |
domain_type | none , relaxed or regular depending on the symbol state |
dimension | dimension |
number_records | number of records in the symbol |
sparsity | 1 - number_records/cardinality |
- Example
Matrix Generation
transfer
stores data in a "flat" format, that is, one record entry per DataFrame row. However, it is often necessary to convert this data format into a matrix format – transfer
enables users to do this with relative ease using the toDense
and the toSparseCoo
symbol methods. The toDense
method will return a dense N
-dimensional numpy array with each dimension corresponding to the GAMS symbol dimension; it is possible to output an array up to 20 dimensions (a GAMS limit). The toSparseCoo
method will return the data in a sparse scipy COOrdinate format, which can then be efficiently converted into other sparse matrix formats.
- Attention
- Both the
toDense
andtoSparseCoo
methods do not transform the underlying DataFrame in any way, they only return the transformed data.
- Note
toSparseCoo
will only convert 2-dimensional data to the scipy COOrdinate format. A user interested in sparse data for an N-dimensional symbol will need to decide how to reshape the dense array in order to generate the 2D sparse format.
- Attention
- In order to use the
toSparseCoo
method the user will need to install the scipy package. Scipy is not provided with GMSPython.
Both the toDense
and toSparseCoo
method leverage the indexing that comes along with using categorical
data types to store domain information. This means that linking symbols together (by passing symbol objects as domain information) impacts the size of the matrix. This is best demonstrated by a few examples.
- Example (1D data w/o domain linking (i.e., a relaxed domain))
Note that the parameter a
is not linked to another symbol, so when converting to a matrix, the indexing is referenced to the data structure in a.records
. Defining a sparse parameter a
over a set i
allows us to extract information from the i
domain and construct a very different dense matrix, as the following example shows:
- Example (1D data w/ domain linking (i.e., a regular domain))
- Example (2D data w/ domain linking)
The Universe Set
A Unique Element (UEL) is an (i,s)
pair where i
(or index) is an identification number for a (string) label s
. GAMS uses UELs to efficiently store domain entries of a record by storing the UEL ID i
of a domain entry instead of the actual string s
. This avoids storing the same string multiple times. The concept of UELs also exists in Python/Pandas and is called a "categorical series". transfer
leverages these types in order to efficiently store strings and enable domain checking within the Python environment.
Each domain column in a DataFrame can be assigned a unique categorical type, the effect is that each symbol maintains its own list of UELs per dimension. It is possible to convert a categorical column to its ID number representation by using the categorical accessor x.records[<domain_column_label>].cat.codes
; however, this type of data manipulation is not necessary within transfer
, but could be handy when debugging data.
Pandas offers the possibility to create categorical column types that are ordered
or not; transfer
relies exclusively on ordered
categorical data types (in order for a symbol to be valid it must have only ordered
categories). By using ordered categories, transfer
will order the UEL such that elements appear in the order in which they appeared in the data (which is how GAMS defines the UEL). transfer
allows the user to reorder the UELs with the uel_priority
argument in the .write()
method.
transfer
does not actually keep track of the UEL separately from other symbols in the Container
, it will be created internal to the .write()
method and is based on the order in which data is added to the container. The user can access the current state of the UEL with the .getUELs()
container method. For example, we set a two dimensional set:
Pandas also includes a number of methods that allow categories to be renamed, appended, etc. These methods may be useful for advanced users, but most users will probably find that modifying the original data structures and resetting the symbol records provides a simpler solution. The design of transfer
should enable the user to quickly move data back and forth, without worrying about the deeper mechanics of categorical data.
Customize the Universe Set
The concept of a universe set is fundamental to GAMS and has consequences in many areas of GAMS programming including the order of loop execution. For example:
set final_model_year / 2030 /;
set t "all model years" / 2022*2030 /;
singleton set my(t) "model solve year";
loop(t,
my(t) = yes;
display my;
);
The loop will execute model solve year 2030
first because the UEL 2030
was defined in the set final_model_year
before it was used again in the definition of set t
. This could lead to some surprising behavior if model time periods are linked together. Many GAMS users would create a dummy set (perhaps the first line of their model file) that contained all the UELs that had a significant order tom combat this behavior. transfer
allows for full control (renaming as well as ordering) over the universe set through the *UELS
methods, briefly described here:
Quick summary table of UELs functions
Method | Brief Description |
---|---|
addUELs | Adds UELS to a symbol dimension(s). This function does not have a container level implementation. |
capitalizeUELs | Capitalize all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
casefoldUELs | Casefold all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
getUELs | Gets the UELs in a over either a symbol dimension, the entire symbol or the entire container. Unused UELs do not show up in symbol data but will show up in the GAMS UEL list. |
ljustUELs | Left justify all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
lowerUELs | Lowercase all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
lstripUELs | Left strip whitespace from all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
removeUELs | Removes UELs from a symbol dimension, the entire symbol, the entire container (or just a subset of symbols). If a used UEL is removed the DataFrame record will show a NaN . |
renameUELs | Renames UELs in a symbol dimension, the entire symbol, the entire container (or just a subset of symbols). Very handy for harmonizing UEL labeling of data that might have originated from different sources. |
reorderUELs | Reorders UELs in a symbol dimension(s). This function does not have a container level implementation. |
rjustUELs | Right justify all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
rstripUELs | Right strip whitespace from all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
setUELs | Sets UELs for a symbol dimension(s). Equivalent results could be obtained with a combination of renameUELs and reorderUELs , but this one call may have some performance advantage. |
stripUELs | Strip whitespace from all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
titleUELs | Title (capitalize all individual words) in all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
upperUELs | Uppercase all UELs in the symbol or a subset of specified dimensions , can be chained with other *UELs string operations |
These tools are extremely useful when data is arriving at a model from a variety of data sources. We will describe each of these functions in detail and provide examples in the following sections.
- Attention
- GAMS is insensitive to trailing whitespaces, the
*UELs
methods will automatically trim any trailing whitespace when creating the new UELs.
getUELs Examples
getUELs
is a method of all GAMS symbol classes as well as the Container
class. This allows the user to retrieve (ordered) UELs from the entire container or just a specific symbol dimension. For example:
addUELs Examples
addUELs
is a method of all GAMS symbol classes. This method allows the user to add in new UELs labels to a specific dimension of a symbol – the user can add UELs that do not exist in the symbol records
. For example:
In this example we have added three new (unused) UELs: ham
, and
, cheese
. These three UELs will now appear in the GAMS universe set (accessible with m.getUELs()
). The addition of unused UELs does not impact the validity of the symbols (i.e., unused UELs will not trigger domain violations).
removeUELs Examples
removeUELs
is a method of all GAMS symbol classes as well as the Container
class. As a result, this method allows the user to clean up unwanted or simply unused UELs in a symbol dimension(s), over several symbols, or over the entire container. The previous example added three unused UELs (ham
, and
, cheese
), but now we want to remove these UELs in order to clean up the GAMS universe set. We can accomplish this several ways:
In all cases the resulting universe set will be:
If a user removes a UEL that appears in data, that data will be lost permanently. The domain label will be transformed into an NaN
as seen in this example:
- Attention
- A container cannot be written if there are
NaN
entries in any of the domain columns (in any symbol) – an Exception is raised if there are missing domain labels.
renameUELs Examples
renameUELs
is a method of all GAMS symbol classes as well as the Container
class. This method allows the user to rename UELs in a symbol dimension(s), over several symbols, or over the entire container. This particular method is very handy when attempting to harmonize labeling schemes between data structures that originated from different sources. For example:
...results in the following records:
However, two different data sources were used to generate the parameters a
and b
– one data source used the uppercase postal abbreviation of the state name and the other source used a lowercase full state name as the unique identifier. With the following syntax the user would be able to harmonize to a mixed case postal code labeling scheme (without losing any of the original UEL ordering).
...results in the following records (and the universe set):
The universe set will now be:
It is possible that some data needs to be cleaned and multiple UELs need to be mapped to a single label (within a single dimension). This is not allowed under default behavior because transfer
assumes that the provided UELs are truly unique (logically and lexicographically) – however, it might be necessary recreate the underlying categorical object to combine n
(previously unique) UELs into one to establish the necessary logical set links. For example:
The records are unique for a
, but logically, there might be a need to rename WI
to WISCONSIN
.
In order achieve the desired behavior it is necessary to pass allow_merge=True
to renameUELs
:
reorderUELs Examples
reorderUELs
is a method of all GAMS symbol classes. This method allows the user to reorder UELs of a specific symbol dimension – reorderUELs
will not all any new UELs to be create nor can they be removed. For example:
But perhaps we want to reorder the UELs i1
, i2
, i3
to i3
, i2
, i1
.
- Note
- This example does not change the indexing scheme of the Pandas DataFrame at all, it only changes the underlying integer numbering scheme for the categories. We can see this by looking at the Pandas
codes
:
setUELs Examples
reorderUELs
is a method of all GAMS symbol classes. This method allows the user to create new UELs, rename UELs, and reorder UELs all in one method. For example:
A user could accomplish a UEL reorder operation with setUELs
:
A user could accomplish a UEL reorder + add UELs operation with setUELs
:
A user could accomplish a UEL reorder + add + rename with setUELs
:
- Note
- This example does not change the indexing scheme of the Pandas DataFrame at all, but the
rename=True
flag means that the records will get updated just as if arenameUELs
call had been made.
If a user wanted to set new UELs on top of this data, without renaming, they would need to be careful to include the current UELs in the UELs being set. It is possible to loose these labels if they are not included (which will prevent the data from being written to GDX/GMD).
String Manipulation on UELs
It is easy to perform common string manipulations on UELs at the dimension, symbol and container levels with a series of convenience functions: lowerUELs
, upperUELs
, lstripUELs
, rstripUELs
, stripUELs
, capitalizeUELs
, casefoldUELs
, titleUELs
, ljustUELs
, rjustUELs
. These methods are wrappers around Python's built in string methods and are designed to efficiently perform bulk UEL transformations on your GAMS data.
The following example shows operations on the entire container:
- Note
- The
ljustUELs
andrjustUELs
methods require the user to specify the final string length and thefill_character
used to pad the string to achieve the final length.
Similar operations can be performed at the dimension and symbol levels as can be seen in the following examples:
- Note
- Symbol dimension is indexed from zero (per Python convention)
Reordering Symbols
The order of the Container file requires the symbols to be sorted such that, for example, a Set used as domain of another symbol appears before that symbol. The Container will try to establish a valid ordering when writing the data. This type of situation could be encountered if the user is adding and removing many symbols (and perhaps rewriting symbols with the same name) – users should attempt to only add symbols to a Container
once, and care must be taken when creating symbol names. The method reorderSymbols
attempts to fix symbol ordering problems. The following example shows how this can occur:
- Example Symbol reordering
The symbols are now out of order in .data
and must be reordered:
Rename Symbols
It is possible to rename a symbol even after it has been added to a Container
. There are two methods that can be used to achieve the desired outcome:
- using the container method
renameSymbol
- directly changing the
name
symbol property
We create a Container
with two sets:
- Example #1 - Change the name of a symbol with the container method
- Example #2 - Change the name of a symbol with the .name attribute
- Note
- Note that the renamed symbols maintain the original symbol order, this will prevent unnecessary reordering operations later in the workflow.
Removing Symbols
Removing symbols from a container is easy when using the removeSymbols
container method; this method accepts either a str
or a list
of str
.
- Attention
- Once a symbol has been removed, it is possible to have hanging references as domain links in other symbols. The user will need to repair these other symbols with the proper domain links in order to avoid validity errors.
GAMS Special Values
The GAMS system contains five special values: UNDEF
(undefined), NA
(not available), EPS
(epsilon), +INF
(positive infinity), -INF
(negative infinity). These special values must be mapped to their Python equivalents. transfer
follows the following convention to generate the 1:1
mapping:
+INF
is mapped tofloat("inf")
-INF
is mapped tofloat("-inf")
EPS
is mapped to-0.0
(mathematically identical to zero)NA
is mapped to a specialNaN
UNDEF
is mapped tofloat("nan")
transfer
syntax is designed to quickly get data into a form that is usable in further analyses or visualization; this mapping also highlights the preference for data that is of type float
, which offers performance benefits within Pandas/NumPy. The user does not need to remember these constants as they are provided within the class SpecialValues
as SpecialValues.POSINF
, SpecialValues.NEGINF
, SpecialValues.EPS
, SpecialValues.NA
, and SpecialValues.UNDEF
. The SpecialValues
class also contains methods to test for these special values. Some examples are shown below; already, we, begin to introduce some of the transfer
syntax.
- Example (special values in a parameter)
The following DataFrame for x
would look like:
The user can now easily test for specific special values in the value
column of the DataFrame (returns a boolean array):
Other data structures can be passed into these methods as long as these structures can be converted into a numpy array with dtype=float
. It follows that:
Pandas DataFrames allow data columns to exist with mixed type (dtype=object
) – transfer
leverages this convenience feature to enable users to import string representations of EPS
, NA
, and UNDEF
(or UNDF
). transfer
is tolerant of any mixed-case special value string representation. Python offers additional flexibility when representing negative/positive infinity. Any string x
where float(x) == float("inf")
evaluates to True can be used to represent positive infinity. Similarly, any string x
where float(x) == float("-inf")
evaluates to True can be used to represent negative infinity. Allowed values include inf
, +inf
, INFINITY
, +INFINITY
, -inf
, -INFINITY
and all mixed-case equivalents.
- Example (special values defined by strings)
These special strings will be immediately mapped to their float
equivalents from the SpecialValues
class in order to ensure that all data entries are float types.
Standard Data Formats
This section is meant to introduce the standard format that transfer
expects for symbol records. It has already been mentioned that we store data as a Pandas DataFrame, but there is an assumed structure to the column headings and column types that will be important to understand. transfer
includes convenience functions in order to ease the burden of converting data from a user-centric format to one that is understood by transfer
. However, advanced users will want to convert their data first and add it directly to the Container to avoid making extra copies of (potentially large) data sets.
- Set Records Standard Format
All set records (including singleton sets) are stored as a Pandas DataFrame with n
number of columns, where n
is the dimensionality of the symbol + 1. The first n-1
columns include the domain elements while the last column includes the set element explanatory text. Records are organized such that there is one record per row.
The names of the domain columns are flexible, but transfer
requires unique column names. Users are encouraged to change the column headings of the underlying dataframe by using the domain_labels
property. Using this property will ensure that unique column names are generated by adding a _<dimension>
tag to the end of any user supplied column names. The explanatory text column is called element_text
and must take the last position in the DataFrame.
All domain columns must be a categorical data type and the element_text
column must be a object
type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however transfer
requires that all these are type str
. All rows in the element_text
column must be type str
.
Some examples:
- Parameter Records Standard Format
All parameter records (including scalars) are stored as a Pandas DataFrame with n
number of columns, where n
is the dimensionality of the symbol + 1. The first n-1
columns include the domain elements while the last column includes the numerical value of the records. Records are organized such that there is one record per row. Scalar parameters have zero dimension, therefore they only have one column and one row.
By default, the names of the domain columns follow a pattern of <set_name>
; a symbol dimension that is referenced to the universe is labeled uni
. The domain labels can be customized. Users are encouraged to change the column headings of the underlying dataframe by using the domain_labels
property. Using this property will ensure that unique column names are generated (if not currently unique) by adding a _<dimension>
tag to the end of any user supplied column names. The value column is called value
and must take the last position in the DataFrame.
All domain columns must be a categorical data type and the value
column must be a float
type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however transfer
requires that all these are type str
.
Some examples:
- Variable/Equation Records Standard Format
Variables and equations share the same standard data format. All records (including scalar variables/equations) are stored as a Pandas DataFrame with n
number of columns, where n
is the dimensionality of the symbol + 5. The first n-5
columns include the domain elements while the last five columns include the numerical values for different attributes of the records. Records are organized such that there is one record per row. Scalar variables/equations have zero dimension, therefore they have five columns and one row.
By default, the names of the domain columns follow a pattern of <set_name>
; a symbol dimension that is referenced to the universe is labeled uni
. The domain labels can be customized. Users are encouraged to change the column headings of the underlying dataframe by using the domain_labels
property. Using this property will ensure that unique column names are generated (if not currently unique) by adding a _<dimension>
tag to the end of any user supplied column names. The attribute columns are called level
, marginal
, lower
, upper
, and scale
. These attribute columns must appear in this order. Attributes that are not supplied by the user will be assigned the default GAMS values for that variable/equation type; it is possible to not pass any attributes, transfer
would then simply assign default values to all attributes.
All domain columns must be a categorical data type and all the attribute columns must be a float
type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however transfer
requires that all these are type str
.
Some examples:
GDX Read/Write
Up until now, we have been focused on using transfer
to create symbols in an empty Container
using the symbol constructors (or their corresponding container methods). These tools will enable users to ingest data from many different formats and add them to a Container
– however, it is also possible to read in symbol data directly from GDX files using the read
container method. In the following sections, we will discuss this method in detail as well as the write
method, which allows users to write out to new GDX files.
Read GDX
There are two main ways to read in GDX based data.
- Pass the file path directly to the Container constructor (will read all symbols and records)
- Pass the file path directly to the
read
method (default read all symbols, but can read partial files)
The first option here is provided for convenience and will, internally, call the read
method. This method will read in all symbols as well as their records. This is the easiest and fastest way to get data out of a GDX file and into your Python environment. For the following examples we leverage the GDX output generated from the `trnsport.gms` model file.
- Example (reading full data w/ Container constructor)
A user could also read in data with the read
method as shown in the following example.
- Example (reading full data w/
read
method)
It is also possible to read in a partial GDX file with the read
method, as shown in the following example:
This syntax assumes that the user will always want to read in both the metadata as well as the actual data records, but it is possible to skip the reading of the records by passing the argument records=False
.
- Attention
- The
read
method attempts to link the domain objects together (in order to have a "regular"domain_type
) but if domain sets are not part of the read operation there is no choice but to default to a "relaxed"domain_type
. This can be seen in the last example where we only read in the variablex
and not the domain sets (i
andj
) that the variable is defined over. All the data will be available to the user, but domain checking is no longer possible. The symbolx
will remain with "relaxed" domain type even if the user were to read in setsi
andj
in a secondread
call.
Write GDX
A user can write data to a GDX file by simply passing a file path (as a string). The write
method will then create the GDX and write all data in the Container
.
- Example
- Example (write a compressed GDX file)
By default, all symbols in the Container will be written, however it is possible to write a subset of the symbols to a GDX file with the symbols
argument. If a domain set is not included in the symbols
list then the symbol will automatically be relaxed (but will retain the domain set's name as a string label – it does not get relaxed to *
). This behavior can be seen in the following example.
In line 4
we can see that the auto-relaxation of the domain for a
is only temporary for writing (in this case, from Container object m
) and will be restored so as not to disturb the Container state.
Advanced users might want to specify an order to their UEL list (i.e., the universe set); recall that the UEL ordering follows that dictated by the data. As a convenience, it is possible to prepend the UEL list with a user specified order using the uel_priority
argument.
- Example (change the order of the UEL)
The original UEL order for this GDX file would have been ["a", "b", "c"]
, but this example reorders the UEL with uel_priority
– the positions of b
and c
have been swapped. This can be verified with the gdxdump
utility (using the uelTable
argument):
gdxdump foo.gdx ueltable=foo Set foo / 'a' , 'c' , 'b' /; $onEmpty Set i(*) / 'a', 'c', 'b' /; $offEmpty
GamsDatabase Read/Write
We have discussed how to create symbols in an empty Container
and we have discussed how to exchange data with GDX files, however it is also possible to read and write data directly in memory by interacting with a GamsDatabase/GMD object – this allows transfer
to be used to read/write data within an Embedded Python Code environment or in combination with the Python OO API. There are some important differences when compared to data exchange with GDX since we are working with data representations in memory.
Read GamsDatabases
Just as with a GDX, there are two main ways to read in data that is in a GamsDatabase/GMD object.
- Pass the GamsDatabase/GMD object directly to the Container constructor (will read all symbols and records)
- Pass the GamsDatabase/GMD object directly to the
read
method (default read all symbols, but can read partial files)
The first option here is provided for convenience and will, internally, call the read
method. This method will read in all symbols as well as their records. This is the easiest and fastest way to get data out of a GamsDatabase/GMD object and into your Python environment. While it is possible to generate a custom GamsDatabase/GMD object from scratch (using the gmdcc
API), most users will be interacting with a GamsDatabase/GMD object that has already been instantiated internally when he/she is using Embedded Python Code or the GamsDatabase class in the Python OO API. Our examples will show how to access the GamsDatabase/GMD object – we leverage the some of the data from the `trnsport.gms` model file.
- Example (reading full data w/ Container constructor)
- Note
- Embedded Python Code users will want pass the GamsDatabase object that is part of the GAMS Database object – this will always be referenced as
gams.db
regardless of the model file.
The following example uses embedded Python code to create a new Container, read in all symbols, and display some summary statistics as part of the gams log output.
Set
i 'canning plants' / seattle, san-diego /
j 'markets' / new-york, chicago, topeka /;
Parameter
a(i) 'capacity of plant i in cases'
/ seattle 350
san-diego 600 /
b(j) 'demand at market j in cases'
/ new-york 325
chicago 300
topeka 275 /;
Table d(i,j) 'distance in thousands of miles'
new-york chicago topeka
seattle 2.5 1.7 1.8
san-diego 2.5 1.8 1.4;
$onembeddedCode Python:
import gams.transfer as gt
m = gt.Container(gams.db)
print(m.describeSets())
print(m.describeParameters())
$offEmbeddedCode
The gams log output will then look as such (the extra print
calls are just providing nice spacing for this example):
GAMS 43.1.0 Copyright (C) 1987-2023 GAMS Development. All rights reserved --- Starting compilation --- matrix.gms(29) 3 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib name is_singleton domain domain_type dimension number_records sparsity 0 i False [*] none 1 2 None 1 j False [*] none 1 3 None name domain domain_type dimension number_records min mean max where_min where_max sparsity 0 a [i] regular 1 2 350.000 475.000 600.000 [seattle] [san-diego] 0.0 1 b [j] regular 1 3 275.000 300.000 325.000 [topeka] [new-york] 0.0 2 d [i, j] regular 2 6 1.400 1.950 2.500 [san-diego, topeka] [seattle, new-york] 0.0 --- Starting execution - empty program *** Status: Normal completion [3 rows x 16 columns] --- Starting execution - empty program *** Status: Normal completion
A user could also read in a subset of the data located in the GamsDatabase object with the read
method as shown in the following example. Here we only read in the sets i
and j
, as a result the .describeParameters()
method will return None
.
- Example (reading subset of full data w/
read
method)
Set
i 'canning plants' / seattle, san-diego /
j 'markets' / new-york, chicago, topeka /;
Parameter
a(i) 'capacity of plant i in cases'
/ seattle 350
san-diego 600 /
b(j) 'demand at market j in cases'
/ new-york 325
chicago 300
topeka 275 /;
Table d(i,j) 'distance in thousands of miles'
new-york chicago topeka
seattle 2.5 1.7 1.8
san-diego 2.5 1.8 1.4;
$onembeddedCode Python:
import gams.transfer as gt
m = gt.Container()
m.read(gams.db, symbols=["i","j"])
gams.printLog("")
print(m.describeSets())
print(m.describeParameters())
$offEmbeddedCode
GAMS 43.1.0 Copyright (C) 1987-2023 GAMS Development. All rights reserved --- Starting compilation --- matrix.gms(29) 3 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- name is_singleton domain domain_type dimension number_records sparsity 0 i False [*] none 1 2 None 1 j False [*] none 1 3 None None --- Starting execution - empty program *** Status: Normal completion
All the typical functionality of the Container exists when working with GamsDatabase/GMD objects. This means that domain linking, matrix conversion, and other more advanced options are available to the user at either compilation time or execution time (depending on the Embedded Code syntax being used, see: Syntax). The next example generates a 1000x1000 matrix and then takes its inverse using the Numpy linalg
package.
- Example (Matrix Generation and Inversion)
set i / i1*i1000 /;
alias(i,j);
parameter a(i,j);
a(i,j) = 1 / (ord(i)+ord(j) - 1);
a(i,i) = 1;
embeddedCode Python:
import gams.transfer as gt
import numpy as np
import time
gams.printLog("")
s = time.time()
m = gt.Container(gams.db)
gams.printLog(f"read data: {round(time.time() - s, 3)} sec")
s = time.time()
A = m["a"].toDense()
gams.printLog(f"create matrix A: {round(time.time() - s, 3)} sec")
s = time.time()
invA = np.linalg.inv(A)
gams.printLog(f"generate inv(A): {round(time.time() - s, 3)} sec")
endEmbeddedCode
- Note
- In this example, the assignment of the
a
parameter is done during execution time so we must use the execution time syntax for embedded code in order to get the numerical records properly.
GAMS 43.1.0 Copyright (C) 1987-2023 GAMS Development. All rights reserved --- Starting compilation --- test.gms(27) 3 Mb --- Starting execution: elapsed 0:00:00.003 --- test.gms(9) 36 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- --- read data: 1.1 sec --- create matrix A: 0.02 sec --- generate inv(A): 0.031 sec *** Status: Normal completion
We will extend this example in the next section to write the inverse matrix A
back into a GAMS parameter.
Write to GamsDatabases
A user can write to a GamsDatabase/GMD object with the .write()
method just as he/she would write a GDX file – however there are some important differences. When a user writes a GDX file the entire GDX file represents a complete data environment (all domains have been resolved, etc.) thus, transfer
does not need to worry about merge/replace operations. It is possible to merge/replace symbol records when a user is writing data to in-memory data representations with GamsDatabase/GMD. We show a few examples to illustrate this behavior.
- Example (Populating a set in GAMS)
* note that we need to declare the set i over "*" in order to provide hints about the symbol dimensionality
set i(*);
$onembeddedCode Python:
import gams.transfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
m.write(gams.db)
$offEmbeddedCode i
embeddedCode Python:
import gams.transfer as gt
m = gt.Container(gams.db)
gams.printLog("")
print(m["i"].records)
endEmbeddedCode
- Note
- In general, it is possible to use
transfer
to create new symbols in a GamsDatabase and GMD object (and not necessarily merge symbols) but embedded code best practices necessitate the declaration of any GAMS symbols on the GAMS side first, then the records can be filled withtransfer
.
If we break down this example we can see that the set i
is declared within GAMS (with no records) and then the records for i
are set by writing a Container
to the gams.db
GamsDatabase object (we do this at compile time). The second embedded Python code block runs at execution time and is simply there to read all the records on the set i
– printing the sets this way adds the output to the .log
file (we could also use the more common display i;
operation in GAMS to display the set elements in the LST file).
GAMS 43.1.0 Copyright (C) 1987-2023 GAMS Development. All rights reserved --- Starting compilation --- test.gms(10) 2 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- test.gms(20) 3 Mb --- Starting execution: elapsed 0:00:01.464 --- test.gms(13) 4 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- uni element_text 0 i0 1 i1 2 i2 3 i3 4 i4 5 i5 6 i6 7 i7 8 i8 9 i9 *** Status: Normal completion
- Example (Merging set records)
set i / i1, i2 /;
$onmulti
$onembeddedCode Python:
import gams.transfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
m.write(gams.db, merge_symbols="i")
$offEmbeddedCode i
$offmulti
embeddedCode Python:
import gams.transfer as gt
m = gt.Container(gams.db)
gams.printLog("")
print(m["i"].records)
endEmbeddedCode
In this example we need to make use of $onMulti/$offMulti in order to merge new set elements into the the set i
(the same would be true if we were merging other symbol types) – any symbol that already has records defined (in GAMS) and is being added to with Python (and transfer
) must be wrapped with $onMulti/$offMulti. As with the previous example, the second embedded Python code block runs at execution time and is simply there to read all the records on the set i
. Note that the UEL order will be different in this case (i1
and i2
come before i0
).
GAMS 43.1.0 Copyright (C) 1987-2023 GAMS Development. All rights reserved --- Starting compilation --- test.gms(11) 3 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- test.gms(21) 3 Mb --- Starting execution: elapsed 0:00:01.535 --- test.gms(14) 4 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- uni element_text 0 i1 1 i2 2 i0 3 i3 4 i4 5 i5 6 i6 7 i7 8 i8 9 i9 *** Status: Normal completion
- Example (Replacing set records)
set i / x1, x2 /;
$onmultiR
$onembeddedCode Python:
import gams.transfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
m.write(gams.db)
$offEmbeddedCode i
$offmulti
embeddedCode Python:
import gams.transfer as gt
m = gt.Container(gams.db)
gams.printLog("")
print(m["i"].records)
endEmbeddedCode
In this example we want to replace the x1
and x2
set elements and built up a totally new element list with set elements from the Container
. Instead of $onMulti
/$offMulti
we must use $onMultiR
/$offMulti
to ensure that the replacement happens in GAMS; we also need to remove the set i
from the merge_symbols
argument.
- Attention
- If the user seeks to replace all records in a symbol they must use the
$onMultiR
syntax. It is not sufficient to simply remove them from themerge_symbols
argument intransfer
. If the user mistakenly uses$onMulti
the symbols will end up merging without total replacement.
GAMS 43.1.0 Copyright (C) 1987-2023 GAMS Development. All rights reserved --- Starting compilation --- test.gms(11) 3 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- test.gms(21) 3 Mb --- Starting execution: elapsed 0:00:01.482 --- test.gms(14) 4 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- uni element_text 0 i0 1 i1 2 i2 3 i3 4 i4 5 i5 6 i6 7 i7 8 i8 9 i9 *** Status: Normal completion
- Example (Merging parameter records)
set i;
parameter a(i<) /
i1 1.23
i2 5
/;
$onmulti
$onembeddedCode Python:
import gams.transfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
a = gt.Parameter(m, "a", domain=i, records=[("i"+str(i),i) for i in range(10)])
m.write(gams.db, merge_symbols="a")
$offEmbeddedCode i, a
$offmulti
embeddedCode Python:
import gams.transfer as gt
m = gt.Container(gams.db)
gams.printLog("")
print(m["a"].records)
endEmbeddedCode
In this example we also need to make use of $onMulti
/$offMulti
in order to merge new set elements into the the set i
, however the set i
also needs to contain the elements that are defined in the parameter – here we make use of the <
operator that will add the set elements from a(i)
into the set i
- Note
- It would also be possible to run this example by explicitly defining the
set i /i1, i2/;
before the parameter declaration.
- Attention
transfer
will overwrite all duplicate records when merging. The original values ofa("i1")
anda("i2")
have been replaced with their new values when writing the Container in this example (see output below).
GAMS 43.1.0 Copyright (C) 1987-2023 GAMS Development. All rights reserved --- Starting compilation --- test.gms(16) 3 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- test.gms(25) 3 Mb --- Starting execution: elapsed 0:00:01.467 --- test.gms(19) 4 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- i value 0 i1 1.0 1 i2 2.0 2 i3 3.0 3 i4 4.0 4 i5 5.0 5 i6 6.0 6 i7 7.0 7 i8 8.0 8 i9 9.0 *** Status: Normal completion
- Example (Advanced Matrix Generation and Inversion w/ Write Operation)
set i / i1*i1000 /;
alias(i,j);
parameter a(i,j);
a(i,j) = 1 / (ord(i)+ord(j) - 1);
a(i,i) = 1;
parameter inv_a(i,j);
parameter ident(i,j);
embeddedCode Python:
import gams.transfer as gt
import numpy as np
import time
gams.printLog("")
gams.printLog("")
s = time.time()
m = gt.Container(gams.db)
gams.printLog(f"read data: {round(time.time() - s, 3)} sec")
s = time.time()
A = m["a"].toDense()
gams.printLog(f"create matrix A: {round(time.time() - s, 3)} sec")
s = time.time()
invA = np.linalg.inv(A)
gams.printLog(f"calculate inv(A): {round(time.time() - s, 3)} sec")
s = time.time()
m["inv_a"].setRecords(invA)
gams.printLog(f"convert matrix to records for inv(A): {round(time.time() - s, 3)} sec")
s = time.time()
I = np.dot(A,invA)
tol = 1e-9
I[np.where((I<tol) & (I>-tol))] = 0
gams.printLog(f"calculate A*invA + small number cleanup: {round(time.time() - s, 3)} sec")
s = time.time()
m["ident"].setRecords(I)
gams.printLog(f"convert matrix to records for I: {round(time.time() - s, 3)} sec")
s = time.time()
m.write(gams.db, ["inv_a","ident"])
gams.printLog(f"write to GamsDatabase: {round(time.time() - s, 3)} sec")
gams.printLog("")
endEmbeddedCode inv_a, ident
display ident;
In this example we extend the example shown in Read GamsDatabases to read data from GAMS, calculate a matrix inversion, do the matrix multiplication, and then write both the A^-1
and A*A^-1
(i.e., the identity matrix) back to GAMS for display in the LST file. This data round trip highlights the benefits of using a transfer
Container (and the linked symbol structure) as the mechanism to move data – converting back and forth from a records format to a matrix format can be cumbersome, but here, transfer
takes care of all the indexing for the user.
The first few lines of GAMS code generates a 1000x1000 A
matrix as a parameter (at execution time), we then define two more parameters that we will fill with results of the embedded Python code – specifically we want to fill a parameter with the matrix A^-1
and we want to verify that another parameter (ident
) contains the identity matrix (i.e., I
). Stepping through the code:
- We start the embedded Python code section (execution time) by importing both
transfer
and Numpy and by reading all the symbols that currently exist in the GamsDatabase. We must read in all this information in order to get the domain set information –transfer
needs these domain sets in order to generate matricies with the proper size. - Generate the matrix
A
by calling.toDense()
on the symbol object in the Container. - Take the inverse of
A
withnp.linalg.inv()
. - The Parameter symbol for
inv_a
already exists in the Container, but it does not have any records (i.e.,m["inv_a"].records is None
will evaluate to True). We use.setRecords()
to convert theinvA
back into a records format. - We continue the computations by performing the matrix multiplication using
np.dot()
– we must clean up a lot of small numbers inI
. - The Parameter symbol for
ident
already exists in the Container, but it does not have any records. We use.setRecords()
to convertI
back into a records format. - Since we are calculating these parameter values at execution time, it is not possible to modify the domain set information (or even merge/replace it). Therefore we only want to write the parameter values to GAMS. We achieve this by writing a subset of the Container symbols out with the
m.write(gams.db, ["inv_a","ident"])
call. This partial write preserves symbol validity in the Container and it does not violate other GAMS requirements. - Finally, we can verify that the (albeit large) identity matrix exists in the LST file (or in another GDX file).
- Note
- It was not possible to just use
np.round
because small negative numbers that round to-0.0
will be interpreted bytransfer
as the GAMS EPS special value.
The output for this example is shown below:
GAMS 43.1.0 Copyright (C) 1987-2023 GAMS Development. All rights reserved --- Starting compilation --- matrix.gms(52) 3 Mb --- Starting execution: elapsed 0:00:00.004 --- matrix.gms(11) 36 Mb --- Initialize embedded library libembpycclib64.dylib --- Execute embedded library libembpycclib64.dylib --- --- --- read data: 1.083 sec --- create matrix A: 0.016 sec --- calculate inv(A): 0.032 sec --- convert matrix to records for inv(A): 0.176 sec --- calculate A*invA + small number cleanup: 0.027 sec --- convert matrix to records for I: 0.17 sec --- write to GamsDatabase: 1.937 sec --- --- matrix.gms(52) 68 Mb *** Status: Normal completion
Container Read
Containers can read from other Container
instances. The syntax and behavior is much the same as reading from GDX and GMD sources. It is important to note that a deepcopy of all data is made when reading from these sources. The container object can be passed into the constructor (to be consistent with the shorthand notation) or the object can be passed as a argument to the .read()
method.
- Combining two containers
In this example we create two containers (which could have been populated from GDX files) and add in all symbol that do not currently exist in the first Container