Container

The main object class within transfer is called Container. The Container is the vessel that allows symbols to be linked together (through their domain definitions), it enables implicit set definitions, it enables structural manipulations of the data (matrix generation), and it allows the user to perform different read/write operations.

Constructor

Constructor Arguments

Creating a Container is a simple matter of initializing an object. For example:

This new Container object, here called m, contains a number of convenient properties and methods that allow the user to interact with the symbols that are in the Container. Some of these methods are used to filter out different types of symbols, other methods are used to numerically characterize the data within each symbol.

Properties

Symbols are organized in the Container under the data Container attribute. The dot notation (m.data) is used to access the underlying dictionary. Symbols in this dictionary can then be retrieved with the standard bracket notation (m.data[<symbol_name>]). The Container is also subscriptable (i.e., m["i"] will return the i Set object just as if the user called m.data["i"]). The behavior of the data dictionary is has been customized to be case-insensitive (which mimics the behavior of GAMS) – m["i"] and m["I"] will return the same object.

Symbol existence in the Container can be tested with an overloaded Python in operator. The following (case-insensitive) syntax is possible:

Note

The final example assumes the existence of a separate symbol object called i.

The Container can also iterate through the symbols (will return a tuple of (symbol_name, symbol_object) using a for-loop as in the following example. The list and dict methods can be useful in creating lists and dictionaries of all symbols in the Container.

Note

The len (length) function can be used to quickly find out how many symbols exist in the Container (len(m)).

Methods

Set

There are two different ways to create a GAMS set and add it to a Container.

1. Use Set constructor
2. Use the Container method addSet (which internally calls the Set constructor)

Constructor

Note

Set records can be updated through the object constructor (a new object will not be created) if a symbol of the same name already exists in the container, has the same domain, and has the same is_singleton and domain_forwarding state. The symbol description will only be updated if new text is provided.

Properties

Methods

Adding Set Records

Three possibilities exist to assign symbol records to a set (roughly ordered in complexity):

1. Setting the argument records in the set constructor/container method (internally calls setRecords) - creates a data copy
2. Using the symbol method setRecords - creates a data copy
3. Setting the property records directly - does not create a data copy

If the data is in a convenient format, a user may want to pass the records directly within the set constructor. This is an optional keyword argument and internally the set constructor will simply call the setRecords method. The symbol method setRecords is a convenience method that transforms the given data into an approved Pandas DataFrame format (see Standard Data Formats). Many native Python data types can be easily transformed into DataFrames, so the setRecords method for Set objects will accept a number of different types for input. The setRecords method is called internally on any data structure that is passed through the records argument. We show a few examples of ways to create differently structured sets:

Example #1 - Create a 1D set from a list

Example #2 - Create a 1D set from a tuple

Example #3 - Create a 2D set from a list of tuples

Example #4 - Create a 1D set from a DataFrame slice + .unique()

Note

The .unique() method preserves the order of appearance, unlike set().

Set element text is very handy when labeling specific set elements within a set. A user can add a set element text directly with a set element. Note that it is not required to label all set elements, as can be seen in the following example.

Example #5 - Add set element text

Example #6 - Create a 1D set from a Pandas Series

Example #7 - Create a 2D set from a Pandas Series

Attention

The order of the set element could be surprising depending on how the pandas MultiIndex is created. Users should take care that the order of the elements returned from the getUELs method is correct.

Example #8 - Create a 2D set from a Pandas DataFrame with uels_on_axes=True

Note

It not possible to set element_text when uels_on_axes=True

Directly Set Records

The primary advantage of the setRecords method is that transfer will convert many different (and convenient) data types into the standard data format (a Pandas DataFrame). Users that require higher performance will want to directly pass the Container a reference to a valid Pandas DataFrame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process and it avoids making a copy of the data in memory. In this section we walk the user through an example of how to set records directly.

Example #1 - Directly set records (1D set)

Stepping through this example we take the following steps:

1. Create an empty Container
2. Create a GAMS set i in the Container, but do not set the records
3. Create a Pandas DataFrame (manually, in this example) taking care to follow the standard format
4. The DataFrame has the right shape and column labels so we can proceed to set the records.
5. We need to cast the uni column as a categorical data type, so we create a custom ordered categorty type using pandas.CategoricalDtype
6. Finally, we set the records directly by passing a reference to df_i into the symbol records attribute. The setter function of .records checks that a DataFrame is being set, but does not check validity. Thus, as a final step we call the .isValid() method to verify that the symbol is valid.

Attention

Users can debug their DataFrames by running <symbol_name>.isValid(verbose=True) to get feedback about their data.

Example #2 - Directly set records (1D subset)

This example is more subtle in that we want to create a set j that is a subset of i. We create the set i using the setRecords method but then set the records directly for j. There are two important details to note: 1) the column labels in df_j now reflect the standard format for a symbol with a domain set (as opposed to the universe) and 2) we create the categorical dtype by referencing the parent set (i) for the categories (instead of referencing itself).

Generate Set Records

Generating the initial pandas.DataFrame object could be difficult for Set symbols that have a large number of records and a small number of UELs – these higher dimensional symbols will benefit from the generateRecords convenience function. Internally, generateRecords computes the dense Cartesian product of all the domain sets that define a symbol (generateRecords will only work on symbols where <symbol>.domain_type == "regular").

Example #1 - Create a large (dense) 4D set

It is also possible to generate a sparse set (randomly selected rows are removed from the dense dataframe) with the density argument to generateRecords.

Example #2 - Create a large (sparse) 4D set

Example #3 - Create a large 4D set w/ only 1 sparse dimension

Parameter

There are two different ways to create a GAMS parameter and add it to a Container.

1. Use Parameter constructor
2. Use the Container method addParameter (which internally calls the Parameter constructor)

Constructor

Constructor Arguments

Note

Parameter records can be updated through the object constructor (a new object will not be created) if a symbol of the same name already exists in the container, has the same domain, and has the same domain_forwarding state. The symbol description will only be updated if new text is provided.

Properties

Methods

Adding Parameter Records

Three possibilities exist to assign symbol records to a parameter (roughly ordered in complexity):

1. Setting the argument records in the set constructor/container method (internally calls setRecords) - creates a data copy
2. Using the symbol method setRecords - creates a data copy
3. Setting the property records directly - does not create a data copy

If the data is in a convenient format, a user may want to pass the records directly within the parameter constructor. This is an optional keyword argument and internally the parameter constructor will simply call the setRecords method. The symbol method setRecords is a convenience method that transforms the given data into an approved Pandas DataFrame format (see Standard Data Formats). Many native Python data types can be easily transformed into DataFrames, so the setRecords method for Set objects will accept a number of different types for input. The setRecords method is called internally on any data structure that is passed through the records argument. We show a few examples of ways to create differently structured parameters:

Example #1 - Create a GAMS scalar

Note

transfer will still convert scalar values to a standard format (i.e., a Pandas DataFrame with a single row and column).

Example #2 - Create a 1D parameter (defined over *) from a list of tuples

Example #3 - Create a 1D parameter (defined over a set) from a list of tuples

Example #4 - Create a 2D parameter (defined over a set) from a DataFrame slice

Note

The original indexing is preserved when a user slices rows out of a reference dataframe.

Example #5 - Create a 2D parameter (defined over a set) from a matrix

Example #6 - Create a 2D parameter from an array using setRecords

Example #7 - Create a 1D parameter from a pandas Series

Example #8 - Create a 2D parameter from a pandas Series

Example #9 - Create a 2D parameter from a DataFrame (uels_on_axes=True)

Directly Set Records

As with sets, the primary advantage of the setRecords method is that transfer will convert many different (and convenient) data types into the standard data format (a Pandas DataFrame). Users that require higher performance will want to directly pass the Container a reference to a valid Pandas DataFrame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process and it avoids making a copy of the data in memory. In this section we walk the user through an example of how to set records directly.

Example #1 - Correctly set records (directly)

In this example we create a large parameter (31,536,000 records and 8880 unique domain elements – we mimic data that is labeled for every second in one year) and assign it to a parameter with a.records. transfer requires that all domain columns must be a categorical data type, furthermore, this categorical must be ordered. The records setter function does very little work other than checking if the object being set is a DataFrame. This places more responsibility on the user to create a DataFrame that complies with the standard format. In Example #1 we take care to properly reference the categorical data types from the domain sets – and in the end a.isValid() = True.

Users will need to use the .isValid(verbose=True) method to debug any structural issues. As an example we incorrectly generate categorical data types by passing the DataFrame constructor the generic dtype="category" argument. This creates categorical column types but they are not ordered and they do not reference the underlying domain set. These errors result in a being invalid.

Example #2 - Incorrectly set records (directly)

Generate Parameter Records

Generating the initial pandas.DataFrame object could be difficult for Parameter symbols that have a large number of records and a small number of UELs – these higher dimensional symbols will benefit from the generateRecords convenience function. Internally, generateRecords computes the dense Cartesian product of all the domain sets that define a symbol (generateRecords will only work on symbols where <symbol>.domain_type == "regular").

Example #1 - Create a large (dense) 4D parameter

Note

In Example #1 a large 4D parameter was generated – by default, the value of these records are randomly drawn numbers from the interval [0,1] (uniform distribution).

As with Sets, it is possible to generate a sparse parameter with the density argument to generateRecords. We extend this example by passing our own custom func argument that will control the behavior of the value columns. The func argument accepts a callable (i.e., a reference to a function).

Example #2 - Create a large (sparse) 4D parameter with normally distributed values

Note

The custom callable function reference must expose a size argument. It might be tedious to know the exact number of the records that will be generated, especially if a fractional density is specified; therefore, the generateRecords method will pass in the correct size automatically. Users are encouraged to use the Numpy suite of random distributions when generating samples – custom functions have the potential to be computationally burdensome if a symbol has a large number of records.

Example #3 - Create a large 4D parameter with 1 sparse dimension

Example #4 - Create a large 4D parameter with a random number seed

Note

The seed is an int that will set the random number generator state (enables reproducible sequences of random numbers).

Variable

There are two different ways to create a GAMS variable and add it to a Container.

1. Use Variable constructor
2. Use the Container method addVariable (which internally calls the Variable constructor)

Constructor

Constructor Arguments

Note

Variable records can be updated through the object constructor (a new object will not be created) if a symbol of the same name already exists in the container, has the same domain, has the same type, and has the same domain_forwarding state. The symbol description will only be updated if new text is provided.

Properties

Methods

Adding Variable Records

Three possibilities exist to assign symbol records to a variable (roughly ordered in complexity):

1. Setting the argument records in the set constructor/container method (internally calls setRecords) - creates a data copy
2. Using the symbol method setRecords - creates a data copy
3. Setting the property records directly - does not create a data copy

If the data is in a convenient format, a user may want to pass the records directly within the variable constructor. This is an optional keyword argument and internally the variable constructor will simply call the setRecords method. In contrast to the setRecords methods in in either the Set or Parameter classes the setRecords method for variables will only accept Pandas DataFrames and specially structured dict for creating records from matrices. This restriction is out of necessity because to properly set a record for a Variable the user must pass data for the level, marginal, lower, upper and scale attributes. That said, any missing attributes will be filled in with the GAMS default record values (see: Variable Types), default scale value is always 1, and the default level and marginal values are 0 for all variable types). We show a few examples of ways to create differently structured variables:

Example #1 - Create a GAMS scalar variable

Example #2 - Create a 1D variable (defined over *) from a list of tuples

In this example we only set the marginal values.

Example #3 - Create a 1D variable (defined over a set) from a list of tuples

Example #4 - Create a 2D positive variable, specifying no numerical data

Example #5 - Create a 2D variable (defined over a set) from a matrix

Example #6 - Create a 1D variable from a pandas Series

Note

transfer will assume that the level value is being set if attributes cannot be found in the axes

Example #7 - Create a 1D variable from a pandas Series (set only the marginal)

Note

transfer will search the axes for any variable attributes and set those attributes provided. It is necessary to include all attributes into the same axes level (if a MultiIndex) without any other UELs – if other UELs are found in the same level then all element will be interpreted as UELs and could result in GAMS domain violations.

Example #8 - Create a 2D variable from a DataFrame (uels_on_axes=True)

Directly Set Records

As with sets, the primary advantage of the setRecords method is that transfer will convert many different (and convenient) data types into the standard data format (a Pandas DataFrame). Users that require higher performance will want to directly pass the Container a reference to a valid Pandas DataFrame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process and it avoids making a copy of the data in memory. In this section we walk the user through an example of how to set records directly.

Example #1 - Correctly set records (directly)

Attention

All numeric data in the records will need to be type float in order to maintain a valid symbol.

In this example we create a large variable (31,536,000 records and 8880 unique domain elements – we mimic data that is labeled for every second in one year) and assign it to a variable with a.records. transfer requires that all domain columns must be a categorical data type, furthermore this categorical must be ordered. The records setter function does very little work other than checking if the object being set is a DataFrame. This places more responsibility on the user to create a DataFrame that complies with the standard format. In Example #1 we take care to properly reference the categorical data types from the domain sets – and in the end a.isValid() = True. As with Set and Parameters, users can use the .isValid(verbose=True) method to debug any structural issues.

Generate Variable Records

Generating the initial pandas.DataFrame object could be difficult for Variable symbols that have a large number of records and a small number of UELs – these higher dimensional symbols will benefit from the generateRecords convenience function. Internally, generateRecords computes the dense Cartesian product of all the domain sets that define a symbol (generateRecords will only work on symbols where <symbol>.domain_type == "regular").

Example #1 - Create a large (dense) 4D variable

Note

In Example #1 a large 4D variable was generated – by default, only the level value of these records are randomly drawn from the interval [0,1] (uniform distribution). Other variable attributes take the default record value.

As with Parameters, it is possible to generate a sparse variable with the density argument to generateRecords. We extend this example by passing our own custom func argument that will control the behavior of the value columns. The func argument accepts a dict of callables (i.e., a reference to a function).

Example #2 - Create a large (sparse) 4D variable with normally distributed values

Note

The custom callable function reference must expose a size argument. It might be tedious to know the exact number of the records that will be generated, especially if a fractional density is specified; therefore, the generateRecords method will pass in the correct size automatically. Users are encouraged to use the Numpy suite of random distributions when generating samples – custom functions have the potential to be computationally burdensome if a symbol has a large number of records.

Equation

There are two different ways to create a GAMS equation and add it to a Container.

1. Use Equation constructor
2. Use the Container method addEquation (which internally calls the Equation constructor)

Constructor

Constructor Arguments

Note

Equation records can be updated through the object constructor (a new object will not be created) if a symbol of the same name already exists in the container, has the same domain, has the same type, and has the same domain_forwarding state. The symbol description will only be updated if new text is provided.

Properties

Methods

Adding Equation Records

Adding equation records mimics that of variables – three possibilities exist to assign symbol records to an equation (roughly ordered in complexity):

1. Setting the argument records in the set constructor/container method (internally calls setRecords) - creates a data copy
2. Using the symbol method setRecords - creates a data copy
3. Setting the property records directly - does not create a data copy

Setting equation records require the user to be explicit with the type of equation that is being created; in contrast to setting variable records (where the default variable is considered to be free).

If the data is in a convenient format, a user may want to pass the records directly within the equation constructor. This is an optional keyword argument and internally the equation constructor will simply call the setRecords method. In contrast to the setRecords methods in in either the Set or Parameter classes the setRecords method for variables will only accept Pandas DataFrames and specially structured dict for creating records from matrices. This restriction is out of necessity because to properly set a record for an Equation the user must pass data for the level, marginal, lower, upper and scale attributes. That said, any missing attributes will be filled in with the GAMS default record values (level = 0.0, marginal = 0.0, lower = -inf, upper = inf, scale = 1.0). We show a few examples of ways to create differently structured variables:

Example #1 - Create a GAMS scalar equation

Example #2 - Create a 1D Equation (defined over *) from a list of tuples

In this example we only set the marginal values.

Example #3 - Create a 1D Equation (defined over a set) from a list of tuples

Example #4 - Create a 2D equation, specifying no numerical data

Example #5 - Create a 2D equation (defined over a set) from a matrix

Example #6 - Create a 1D equation from a pandas Series

Note

transfer will assume that the level value is being set if attributes cannot be found in the axes (same behavior when setting Variables)

Example #7 - Create a 1D equation from a pandas Series (set only the marginal)

Example #8 - Create a 2D equation from a DataFrame (uels_on_axes=True)