Table of Contents
GAMS Transfer is a package to maintain GAMS data outside a GAMS script in a programming language like Python, Matlab, or R. It allows the user to add GAMS symbols (Sets, Aliases, Parameters, Variables and Equations), to manipulate GAMS symbols, as well as read/write symbols to different data endpoints. GAMS Transfer's main focus is the highly efficient transfer of data between GAMS and the target programming language, while keeping those operations as simple as possible for the user. In order to achieve this, symbol records - the actual and potentially large-scale data sets - are stored in native data structures of the corresponding programming languages. The benefits of this approach are threefold: (1) The user is usually very familiar with these data structures, (2) these data structures come with a large tool box for various data operations, and (3) optimized methods for reading from and writing to GAMS can transfer the data as a bulk - resulting in the high performance of this package.
Getting Started
Install
The user must download and install the latest version of GAMS in order to install GAMS Transfer R. GAMS Transfer R can then be installed from either the source package or from the binary package.
The installation from the source will install gamstransfer
and all of its dependencies.
The users can also install gamstransfer
without compiling the package source code using gamstransfer
binary package. The binary packages are platform dependent and the instructions for each supported platform are shown below.
Windows
Linux
Mac OS X
GAMS Transfer R depends on packages R6
, assertthat
, R.utils
, and Rcpp
.
Examples
GDX Read
Reading in all symbols can be accomplished with one line of code (we reference data from the trnsport.gms example).
All symbol data is organized in the data field (a list) – m$data[<symbol_name>]$records
– records are stored as data frames.
GDX Write
There are five symbol classes within GAMS Transfer R: Sets, Parameters, Variables, Equations, and Aliases. For purposes of this quick start, we show how to recreate the distance
data structure from the trnsport.gms model (the parameter d
). This brief example shows how users can achieve "GAMS-like" functionality, but within an R environment – GAMS Transfer R leverages the object oriented programming to simplify the syntax.
This example shows a few fundamental features of GAMS Transfer R:
- A Container is analogous to a GDX file
- Symbols will always be linked to a Container (notice that we always pass the Container reference
m
to the symbol constructor) - Records can be added to a symbol with the
setRecords()
method, through therecords
constructor argument (internally callssetRecords()
), or through directly setting therecords
field. GAMS Transfer R will convert many common R data structures into a standard format. - Domain linking is possible by passing domain set objects to other symbols – this will also enable domain checking (violations will show up as
<NA>
as shown in this example) - Writing a GDX file can be accomplished in one line with the write() method.
Full Example
It is possible to use GAMS Transfer R to recreate the trnsport.gms results in GDX form. As part of this example, we also introduce the Container methods write()
method (and generate new.gdx
). We will discuss it in more detail in the following section: Data Exchange with GDX.
It can be observed from the above example that a typical work flow for writing using GAMS Transfer R is creating a container, filling it with symbols (Sets, Parameters, Variables, Equations, and Aliases), and write it to a GDX file. To read a GDX file, a Container can simply be initialized with the GDX file name as an argument.
These examples introduced the reader to the GAMS Transfer R syntax. In the remaining sections, we will present details about the core functionality and dig further into the syntax.
Manual
Container
Storing, manipulating, and transforming sparse data requires that it lives within an environment – this data can then be linked together to enable various operations. In GAMS Transfer R, we refer to this "environment" as the Container
, it is the main repository for storing and linking our data. Linking data enables data operations like implicit set growth, domain checking, data format transformations (to dense/sparse matrix formats), etc –. A Container also enables different read/write operations.
Argument | Type | Description | Required | Default |
---|---|---|---|---|
loadFrom | string | Name of the GDX file being read into the Container | No | NULL |
systemDirectory | string | Absolute path to GAMS system directory | No | Attempts to find the GAMS installation by checking path system variable and looking for the GAMS executable. |
Creating a Container
is a simple matter of initializing an object. For example:
This new Container
object, here called m
, contains a number of convenient fields and methods that allow the user to interact with the symbols that are in the Container
. Some of these methods are simply used to filter out different types of symbols, other methods are used to numerically characterize the data within each symbol.
Container fields
Field | Description | Type | Special Setter Behavior |
---|---|---|---|
data | main list that is used to store all symbol data | list | - |
Symbols are organized in the Container
under the data
field. The $
notation (m$data
) is used to access the underlying named list; symbols in this named list can then be retrieved with the standard bracket (or double bracket) notation (m$data[<symbol_name>]
).
Container methods
Method | Description | Arguments/Defaults | Returns |
---|---|---|---|
addAlias | add an Alias | name (string ) aliasWith (Set ,Alias ) | Alias object |
addEquation | add an Equation | name (string ) type (string ) domain=NULL (string ,list ) records=NULL (data frame ,vector ,NULL ) domainForwarding=FALSE (logical ) description="" (string ) | Equation object |
addParameter | add a Parameter | name (string ) domain=NULL (string ,list ) records=NULL (data frame ,array , matrix ) domainForwarding=FALSE (logical ) description="" (string ) | Parameter object |
addSet | add a Set | name (string ) domain="*" (string ,list ) isSingleton=FALSE (logical ) records=NULL (data frame ,array ,matrix ) domainForwarding=FALSE (logical ) description="" (string ) | Set object |
addVariable | add a Variable | name (string ) type="free" (string ) domain=NULL (string ,list ) records=NULL (data frame ,array ,matrix ) domainForwarding=FALSE (logical ) description="" (string ) | Variable object |
describeAliases | create a summary table with descriptive statistics for Aliases | symbols=NULL (string ,list ) - if NULL , assumes all Aliases | data frame |
describeParameters | create a summary table with descriptive statistics for Parameters | symbols=NULL (string ,list ) - if NULL , assumes all parameters | data frame |
describEquations | create a summary table with descriptive statistics for Equations | symbols=NULL (string ,list ) - if NULL , assumes all equations | data frame |
describeSets | create a summary table with descriptive statistics for Sets | symbols=NULL (string ,list ) - if NULL , assumes all sets | data frame |
describeVariables | create a summary table with descriptive statistics for Variables | symbols=NULL (string ,list ) - if NULL , assumes all variables | data frame |
getSymbols | returns a list of object refernces for symbols | symbols (character , vector , list ) | list |
getUniverseSet | provides a universe for all symbols | - | list |
isValid | TRUE if all symbols in the Container are valid | verbose=FALSE (logical ) force=FALSE (logical ) | logical |
listAliases | list all aliases (isValid=NULL ), list all valid aliases (isValid=TRUE ), list all invalid aliases (isValid=FALSE ) in the container | isValid=NULL (logical ) | vector |
listEquations | list all equations (isValid=NULL ), list all valid equations (isValid=TRUE ), list all invalid equations (isValid=FALSE ) in the container | isValid=NULL (logical ) types=NULL (list or vector of eqaution types) - if NULL , assumes all types | vector |
listParameters | list all parameters (isValid=NULL ), list all valid parameters (isValid=TRUE ), list all invalid parameters (isValid=FALSE ) in the container | isValid=NULL (logical ) | vector |
listSets | list all sets (isValid=NULL ), list all valid sets (isValid=TRUE ), list all invalid sets (isValid=FALSE ) in the container | isValid=NULL (logical ) | vector |
listSymbols | list all symbols (isValid=NULL ), list all valid symbols (isValid=TRUE ), list all invalid symbols (isValid=FALSE ) in the container | isValid=NULL (logical ) | vector |
listVariables | list all variables (isValid=NULL ), list all valid variables (isValid=TRUE ), list all invalid variables (isValid=FALSE ) in the container | isValid=NULL (logical ) types=NULL (list or vector of variable types) - if NULL , assumes all types | vector |
read | main method to read loadFrom , can be provided with a list of symbols to read in subsets, records controls if symbol records are loaded or just metadata | loadFrom (string ) symbols="all" (string , list ) records=TRUE (logical ) | NULL |
removeSymbols | symbols to remove from the Container, also sets the symbols' refContainer to NULL | symbols (string ,list ) | NULL |
renameSymbol | rename a symbol in the Container | oldName (string ), newName (string ) | NULL |
reorderSymbols | reorder symbols in order to avoid domain violations | - | NULL |
write | main bulk write method to a writeTo target | writeTo (string ) compress=FALSE (logical ) uelPriority=NULL (string ,list ) | NULL |
Symbols
In GAMS Transfer R, a symbol is either a GAMS Set, Parameter, Variable, Equation, Alias. In GAMS Transfer R, a symbol cannot live on its own, but is always part of a container.
- Create a symbol
There are two different ways to create a GAMS set and add it to a Container.
- Use the Container methods addSet, addParameter, addVariable, addEquation, addAlias (which internally calls the symbol constructor) m = Container$new()p = Parameter$new(m, "p")
- Add Symbol Records
- Three possibilities exist to assign symbol records:
- Setting the argument
records
in the symbol constructor/Container method (internally callssetRecords
) - Using the symbol method
setRecords
- Setting the field
records
directly
If the data is in a convenient format, a user may want to pass the records directly within the set constructor. This is an optional keyword argument and internally the symbol constructor will simply call the setRecords
method. The symbol method setRecords
is a convenience method that transforms the given data into an approved data frame format (see Standard Data Formats). Many native R data types can be easily transformed into data frames, so the setRecords
method will accept a number of different types for input. The setRecords
method is called internally on any data structure that is passed through the records
argument.
The examples of setting records using the above-mentioned methods can be found in the respective documentation for each symbol ( Set, Parameter, Variable, Equation, and Alias ).
Set
- Set Constructor
Argument | Type | Description | Required | Default |
---|---|---|---|---|
container | Container | A reference to the Container object that the symbol is being added to | Yes | - |
name | string | Name of symbol | Yes | - |
domain | list | String, List of domains given either as a string ("*" for universe set) or as a reference to a Set object | No | "*" |
isSingleton | logical | Indicates if set is a singleton set (TRUE ) or not (FALSE ) | No | FALSE |
records | string vector, data frame | Symbol records | No | NULL |
domainForwarding | logical | Flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) | No | FALSE |
description | string | Description of symbol | No | "" |
- Set Fields
Field | Description | Type |
---|---|---|
description | description of symbol | string |
dimension | dimension of symbol, setting dimension is a shorthand notation to set domain to a list of size n containing "*" | integer |
domainForwarding | flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) | logical |
domainLabels | column headings for the records data frame | list of string |
domainNames | string version of domain names | list of string |
domainType | none , relaxed or regular depending on state of domain links | string |
isSingleton | logical if symbol is a singleton set | logical |
name | name of symbol | string |
numberRecords | number of symbol records (i.e., returns nrow(self$records) if not NULL ) | integer |
records | the main symbol records | data frame |
refContainer | reference to the Container that the symbol belongs to | Container |
summary | output a list of only the metadata | list |
- Set Methods
Method | Description | Arguments/Defaults | Returns |
---|---|---|---|
getCardinality | get the full cartesian product of the domain | - | integer |
getSparsity | get the sparsity of the symbol w.r.t the cardinality | - | numeric |
findDomainViolations | get the index of records that contain any domain violations | - | vector |
isValid | checks if the symbol is in a valid format, throw exceptions if verbose=TRUE , recheck a symbol if force=TRUE | verbose=FALSE force=FALSE | logical |
setRecords | main convenience method to set standard data frame formatted records | records (string vector, list, data frame) | NULL |
- Adding Set Records
Three possibilities exist to assign symbol records to a set: We show a few examples of ways to create differently structured sets.
- Example #1 - Create a 1D set from a vector
- m = Container$new()i = Set$new(m, "i", records = c("seattle", "san-diego"))# NOTE: the above syntax is equivalent to -# i = Set$new(m, "i")# i$setRecords(c("seattle", "san-diego"))# NOTE: the above syntax is also equivalent to -# m$addSet("i", records= c("seattle", "san-diego"))# NOTE: the above syntax is also equivalent to -# i = m$addSet("i")# i$setRecords(c("seattle", "san-diego"))# NOTE: the above syntax is also equivalent to -# m$addSet("i")# m$data[["i"]]$setRecords(c("seattle", "san-diego"))> i$recordsuni_1 element_text1 seattle2 san-diego
- Example #2 - Create a 2D set from a list
- m = Container$new()k = Set$new(m, "k", c("*", "*"), records=list("seattle", "san-diego"))# NOTE: the above syntax is equivalent to -# k = Set$new(m, "k", c("*", "*"))# k$setRecords(list("seattle", "san-diego"))# NOTE: the above syntax is also equivalent to -# m$addSet("k", c("*","*"), records=list("seattle", "san-diego"))# NOTE: the above syntax is also equivalent to -# k = m$addSet("k", c("*","*"))# k$setRecords(list("seattle", "san-diego"))# NOTE: the above syntax is also equivalent to -# m$addSet("k", c("*", "*"))# m$data[["k"]]$setRecords(list("seattle", "san-diego"))> k$recordsuni_1 uni_2 element_text1 seattle san-diego
- Example #3 - Create a 1D set from a data frame slice
- m = Container$new()dist = data.frame(from = c("seattle", "seattle", "seattle","san-diego", "san-diego", "san-diego"),to = c("new-york", "chicago", "topeka","new-york", "chicago", "topeka"),thousand_miles = c(2.5, 1.7, 1.8, 2.5, 1.8, 1.4))l = Set$new(m, "l", records = unique(dist[["from"]]))# NOTE: the above syntax is equivalent to -# l = Set$new(m, "l")# l$setRecords(unique(dist[["from"]]))# NOTE: the above syntax is also equivalent to -# m$addSet("l", records=unique(dist[["from"]]))# NOTE: the above syntax is also equivalent to -# l = m$addSet("l")# l$setRecords(unique(dist[["from"]]))# NOTE: the above syntax is also equivalent to -# m$addSet("l")# m$data[["l"]]$setRecords(unique(dist[["from"]]))> l$recordsuni_1 element_text1 seattle2 san-diego
Set element text is very handy when labeling specific set elements within a set. A user can add a set element text directly with a set element. Note that it is not required to label all set elements, as can be seen in the following example.
- Example #4 - Add set element text
The primary advantage of the setRecords
method is that GAMS Transfer R will convert many different (and convenient) data types into the standard data format (a data frame). Users that require higher performance will want to directly pass the Container a reference to a valid data frame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process.
In this section, we walk the user through an example of how to set records directly.
- Example #5 - Directly set records (1D set)
Stepping through this example we take the following steps:
- Create an empty Container
- Create a GAMS set
i
in the Container, but do not set therecords
- Create a data frame (manually, in this example) taking care to follow the standard format
- The data frame has the right shape and column labels so we can proceed to set the records.
- We need to cast the
uni_1
column as afactor
, so we create a custom ordered category type usingfactor
- Finally, we set the records directly by passing a reference to
df_i
into the symbol records attribute. The setter function ofrecords
checks that a data frame is being set, but does not check validity. Thus, as a final step, we call the$isValid()
method to verify that the symbol is valid.
- Note
- Users can debug their data frames by running
<symbol_name>$isValid(verbose=TRUE)
to get feedback about their data.
- Example #6 - Directly set records (1D subset)
This example is more subtle in that we want to create a set j
that is a subset of i
. We create the set i
using the setRecords
method but then set the records directly for j
. There are two important details to note: 1) the column labels in df_j
now reflect the standard format for a symbol with a domain set (as opposed to the universe) and 2) we create the factors by referencing the parent set (i
) for the levels
(instead of referencing itself).
Parameter
- Parameter Constructor
Argument Type Description Required Default container
Container
A reference to the Container object that the symbol is being added to Yes - name
string
Name of symbol Yes - domain
list
List of domains given either as a string ("*" for universe set) or as a reference to a Set object, an empty domain list will create a scalar parameter No NULL records
many Symbol records No NULL domainForwarding
logical
Flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) No FALSE description
string
Description of symbol No ""
- Parameter Fields
Field | Description | Type |
---|---|---|
description | description of symbol | string |
dimension | dimension of symbol, setting dimension is a shorthand notation to set domain to a list of size n containing "*" | integer |
domainForwarding | flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) | logical |
domainLabels | column headings for the records data frame | list of string |
domainNames | string version of domain names | list of string |
domainType | none , relaxed or regular depending on state of domain links | string |
isScalar | TRUE if the length(self$domain) = 0 | logical |
name | name of symbol | string |
numberRecords | number of symbol records (i.e., returns nrow(self$records) if not NULL ) | integer |
records | the main symbol records | data frame |
refContainer | reference to the Container that the symbol belongs to | Container |
shape | a vector describing the array dimensions if records were converted with $toDense() | vector |
summary | output a list of only the metadata | list |
- Parameter Methods
Method | Description | Arguments/Defaults | Returns |
---|---|---|---|
countEps | total number of SpecialValues$EPS in value column | - | integer |
countNA | total number of SpecialValues[["NA"]] in value column | - | integer |
countNegInf | total number of SpecialValues$NEGINF in value column | - | integer |
countPosInf | total number of SpecialValues$POSINF in value column | - | integer |
countUndef | total number of SpecialValues$UNDEF in value column | - | integer |
findDomainViolations | get the index of records that contain any domain violations | - | vector |
getCardinality | get the full cartesian product of the domain | - | integer |
getSparsity | get the sparsity of the symbol w.r.t the cardinality | - | numeric |
getMaxValue | get the maximum value in value column | - | numeric |
getMinValue | get the minimum value in value column | - | numeric |
getMeanValue | get the mean value in value column | - | numeric |
getMaxAbsValue | get the maximum absolute value in value column | - | numeric |
isValid | checks if the symbol is in a valid format, throw exceptions if verbose=TRUE , recheck a symbol if force=TRUE | verbose=FALSE force=FALSE | logical |
setRecords | main convenience method to set standard data frame records | records (many types) | NULL |
toDense | convert symbol to a dense matrix or array format | column = value | array or matrix |
whereMax | find the row number in records data frame with a maximum value (return the first instance only) | - | integer |
whereMaxAbs | find the row number in records data frame with a maximum absolute value (return the first instance only) | - | integer |
whereMin | find the row number in records data frame with a minimum value (return the first instance only) | - | integer |
- Adding Parameter Records
- Three possibilities exist to assign symbol records to a parameter: We show a few examples of ways to create differently structured parameters:
- Example #1 - Create a GAMS scalar
- m = Container$new()pi = Parameter$new(m, "pi", records = 3.14159)# NOTE: the above syntax is equivalent to -# pi = Parameter$new(m, "pi")# pi$setRecords(3.14159)# NOTE: the above syntax is also equivalent to -# m$addParameter("pi", records=3.14159)# NOTE: the above syntax is also equivalent to -# pi = m$addParameter("pi")# pi$setRecords(3.14159)# NOTE: the above syntax is also equivalent to -# m$addParameter("pi")# m$data[["pi"]]$setRecords(3.14159)> pi$recordsvalue1 3.14159
- Note
- GAMS Transfer R will still convert scalar values to a standard format (i.e., a data frame with a single row and column).
- Example #2 - Create a 2D parameter (defined over a set) from a
- data frame slice
- Example #3 - Create a 2D parameter from an array using setRecords
As with Sets, the primary advantage of the setRecords
method is that GAMS Transfer will convert many different (and convenient) data types into the standard data format (data frame). Users that require higher performance will want to directly pass the Container a reference to a valid data frame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process. In this section, we walk the user through an example of how to set records directly.
- Example #4 - Correctly set records (directly)
In this example, we create a large parameter (31,536,000 records and 8880 unique domain elements – we mimic data that is labeled for every second in one year) and assign it to a parameter with a$records
. GAMS Transfer R requires that all domain columns must be ordered factors. The records
setter function does very little work other than checking if the object being set is a data frame. This places more responsibility on the user to create a data frame that complies with the standard format. In Example #1 we take care to properly reference the factor from the domain sets – and in the end a$isValid() = TRUE
.
Users will need to use the $isValid(verbose=TRUE)
method to debug any structural issues. As an example, we incorrectly generate categorical data types by passing the data frame constructor the generic factor
argument. This creates factor columns, but they are not ordered and they do not reference the underlying domain set. These errors result in a
being invalid.
- Example #5 - Incorrectly set records (directly)
Variable
- Variable Constructor
Argument | Type | Description | Required | Default |
---|---|---|---|---|
container | Container | A reference to the Container object that the symbol is being added to | Yes | - |
name | string | Name of symbol | Yes | - |
type | string | Type of variable being created [binary , integer , positive , negative , free , sos1 , sos2 , semicont , semiint ] | No | free |
domain | list , vector | List, vector of domains given either as a string ("*" for universe set) or as a reference to a Set object, an empty domain list will create a scalar variable | No | NULL |
records | many | Symbol records | No | NULL |
domainForwarding | logical | Flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) | No | FALSE |
description | string | Description of symbol | No | "" |
- Variable Fields
Field | Description | Type |
---|---|---|
description | description of symbol | string |
dimension | dimension of symbol, setting dimension is a shorthand notation to set domain to a list of size n containing "*" | integer |
domainForwarding | flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) | logical |
domainLabels | column headings for the records data frame | list of string |
domainNames | string version of domain names | list of string |
domainType | none , relaxed or regular depending on state of domain links | string |
name | name of symbol | string |
numberRecords | number of symbol records (i.e., returns nrow(self$records) if not NULL ) | integer |
records | the main symbol records | data frame |
refContainer | reference to the Container that the symbol belongs to | Container |
shape | a vector describing the array dimensions if records were converted with $toDense() | vector |
summary | output a list of only the metadata | list |
type | string type of variable | list |
- Variable Methods
Method | Description | Arguments/Defaults | Returns |
---|---|---|---|
countEps | total number of SpecialValues$EPS across all columns | columns = "level" | integer |
countNA | total number of SpecialValues[["NA"]] across all columns | columns = "level" | integer |
countNegInf | total number of SpecialValues$NEGINF across all columns | columns = "level" | integer |
countPosInf | total number of SpecialValues$POSINF across all columns | columns = "level" | integer |
countUndef | total number of SpecialValues$UNDEF across all columns | columns = "level" | integer |
findDomainViolations | get the index of records that contain any domain violations | - | vector |
getCardinality | get the full cartesian product of the domain | - | integer |
getSparsity | get the sparsity of the symbol w.r.t the cardinality | - | numeric |
getMaxValue | get the maximum value across all columns | columns = "level" | numeric |
getMinValue | get the minimum value across all columns | columns = "level" | numeric |
getMeanValue | get the mean value across all columns | columns = "level" | numeric |
getMaxAbsValue | get the maximum absolute value across all columns | columns = "level" | numeric |
isValid | checks if the symbol is in a valid format, throw exceptions if verbose=TRUE , recheck a symbol if force=TRUE | verbose=FALSE force=FALSE | logical |
setRecords | main convenience method to set standard data frame records | records (many types) | NULL |
toDense | convert symbol to a dense matrix or array format | column = value | array or matrix |
whereMax | find the row number in records data frame with a maximum value (return the first instance only) | - | integer |
whereMaxAbs | find the row number in records data frame with a maximum absolute value (return the first instance only) | - | integer |
whereMin | find the row number in records data frame with a minimum value (return the first instance only) | - | integer |
- Adding Variable Records
- Three possibilities exist to assign symbol records to a variable: We show a few examples of ways to create differently structured variables:
- Example #1 - Create a GAMS scalar variable
- m = Container$new()pi = Variable$new(m, "pi", records = data.frame(level = 3.14159))# NOTE: the above syntax is equivalent to -# pi = Variable$new(m, "pi", "free")# pi$setRecords(data.frame(level = 3.14159))# NOTE: the above syntax is also equivalent to -# m$addVariable("pi", "free", records=data.frame(level = 3.14159))> pi$recordslevel marginal lower upper scale1 3.14159 0 -Inf Inf 1
- Example #2 - Create a 2D positive variable, specifying no numerical data
- Example #3 - Create a 2D variable (defined over a set) from a matrix
- m = Container$new()i = Set$new(m, "i", "*", records = paste0("i", 1:5))j = Set$new(m, "j", "*", records = paste0("j", 1:5))# creating records for parameter aij = list(i_1 = paste0("i", 1:5), j_2 = paste0("j", 1:5))df = rev(expand.grid(rev(ij)))df$value = 1:25a = Parameter$new(m, "a", c(i, j), records = df)# create a free variable and set the level and marginal attributes from matriciesv = Variable$new(m, "v", domain = c(i, j), records = list(level = a$toDense(), marginal = a$toDense()))# if not specified, the toDense() method will convert the level values to a matrix> v$toDense()[,1] [,2] [,3] [,4] [,5][1,] 1 2 3 4 5[2,] 6 7 8 9 10[3,] 11 12 13 14 15[4,] 16 17 18 19 20[5,] 21 22 23 24 25
As with Sets, the primary advantage of the setRecords
method is that GAMS Transfer will convert many different (and convenient) data types into the standard data format (data frame). Users that require higher performance will want to directly pass the Container a reference to a valid data frame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process. In this section, we walk the user through an example of how to set records directly.
- Example #4 - Correctly set records (directly)
In this example, we create a large variable (31,536,000 records and 8880 unique domain elements – we mimic data that is labeled for every second in one year) and assign it to a variable with a$records
. GAMS Transfer R requires that all domain columns must be ordered factors. The records
setter function does very little work other than checking if the object being set is a data frame. This places more responsibility on the user to create a data frame that complies with the standard format. In Example #1 we take care to properly reference the factor from the domain sets – and in the end a$isValid() = TRUE
.
As with Set and Parameters, users can use the $isValid(verbose=TRUE)
method to debug any structural issues.
Equation
- Equation Constructor
Argument | Type | Description | Required | Default |
---|---|---|---|---|
container | Container | A reference to the Container object that the symbol is being added to | Yes | - |
name | string | Name of symbol | Yes | - |
type | string | Type of equation being created [eq (or E /e ), geq (or G /g ), leq (or L /l ), nonbinding (or N /n ), external (or X /x )] | No | free |
domain | list , vector | List, vector of domains given either as a string ("*" for universe set) or as a reference to a Set object, an empty domain list will create a scalar equation | No | NULL |
records | many | Symbol records | No | NULL |
domainForwarding | logical | Flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) | No | FALSE |
description | string | Description of symbol | No | "" |
- Equation Fields
Field | Description | Type |
---|---|---|
description | description of symbol | string |
dimension | dimension of symbol, setting dimension is a shorthand notation to set domain to a list of size n containing "*" | integer |
domainForwarding | flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) | logical |
domainLabels | column headings for the records data frame | list of string |
domainNames | string version of domain names | list of string |
domainType | none , relaxed or regular depending on state of domain links | string |
name | name of symbol | string |
numberRecords | number of symbol records (i.e., returns nrow(self$records) if not NULL ) | integer |
records | the main symbol records | data frame |
refContainer | reference to the Container that the symbol belongs to | Container |
shape | a vector describing the array dimensions if records were converted with $toDense() | vector |
summary | output a list of only the metadata | list |
type | string type of equation | list |
- Equation Methods
Method | Description | Arguments/Defaults | Returns |
---|---|---|---|
countEps | total number of SpecialValues$EPS across all columns | columns = "level" | integer |
countNA | total number of SpecialValues[["NA"]] across all columns | columns = "level" | integer |
countNegInf | total number of SpecialValues$NEGINF across all columns | columns = "level" | integer |
countPosInf | total number of SpecialValues$POSINF across all columns | columns = "level" | integer |
countUndef | total number of SpecialValues$UNDEF across all columns | columns = "level" | integer |
findDomainViolations | get the index of records that contain any domain violations | - | vector |
getCardinality | get the full cartesian product of the domain | - | integer |
getSparsity | get the sparsity of the symbol w.r.t the cardinality | - | numeric |
getMaxValue | get the maximum value across all columns | columns = "level" | numeric |
getMinValue | get the minimum value iacross all columns | columns = "level" | numeric |
getMeanValue | get the mean value across all columns | columns = "level" | numeric |
getMaxAbsValue | get the maximum absolute value across all columns | columns = "level" | numeric |
isValid | checks if the symbol is in a valid format, throw exceptions if verbose=TRUE , recheck a symbol if force=TRUE | verbose=FALSE force=FALSE | logical |
setRecords | main convenience method to set standard data frame records | records (many types) | NULL |
toDense | convert symbol to a dense matrix or array format | column = value | array or matrix |
whereMax | find the row number in records data frame with a maximum value (return the first instance only) | - | integer |
whereMaxAbs | find the row number in records data frame with a maximum absolute value (return the first instance only) | - | integer |
whereMin | find the row number in records data frame with a minimum value (return the first instance only) | - | integer |
- Adding Equation Records
- Three possibilities exist to assign symbol records to an equation: We show a few examples of ways to create differently structured equations:
- Example #1 - Create a GAMS scalar equation
- m = Container$new()pi = Equation$new(m, "pi", records = data.frame(level = 3.14159))# NOTE: the above syntax is equivalent to -# pi = Equation$new(m, "pi", "free")# pi$setRecords(data.frame(level = 3.14159))# NOTE: the above syntax is also equivalent to -# m$addEquation("pi", "free", records=data.frame(level = 3.14159))> pi$recordslevel marginal lower upper scale1 3.14159 0 -Inf Inf 1
- Example #2 - Create a 2D positive equation, specifying no numerical data
- Example #3 - Create a 2D equation (defined over a set) from a matrix
- m = Container$new()i = Set$new(m, "i", "*", records = paste0("i", 1:5))j = Set$new(m, "j", "*", records = paste0("j", 1:5))# creating records for parameter aij = list(i_1 = paste0("i", 1:5), j_2 = paste0("j", 1:5))df = rev(expand.grid(rev(ij)))df$value = 1:25a = Parameter$new(m, "a", c(i, j), records = df)# create a free variable and set the level and marginal attributes from matricese = Equation$new(m, "e", "nonbinding", domain = c(i, j), records = list(level = a$toDense(), marginal = a$toDense()))> e$recordsi_1 j_2 level marginal lower upper scale1 i1 j1 1 1 -Inf Inf 12 i2 j1 6 6 -Inf Inf 13 i3 j1 11 11 -Inf Inf 14 i4 j1 16 16 -Inf Inf 15 i5 j1 21 21 -Inf Inf 16 i1 j2 2 2 -Inf Inf 17 i2 j2 7 7 -Inf Inf 18 i3 j2 12 12 -Inf Inf 19 i4 j2 17 17 -Inf Inf 110 i5 j2 22 22 -Inf Inf 111 i1 j3 3 3 -Inf Inf 112 i2 j3 8 8 -Inf Inf 113 i3 j3 13 13 -Inf Inf 114 i4 j3 18 18 -Inf Inf 115 i5 j3 23 23 -Inf Inf 116 i1 j4 4 4 -Inf Inf 117 i2 j4 9 9 -Inf Inf 118 i3 j4 14 14 -Inf Inf 119 i4 j4 19 19 -Inf Inf 120 i5 j4 24 24 -Inf Inf 121 i1 j5 5 5 -Inf Inf 122 i2 j5 10 10 -Inf Inf 123 i3 j5 15 15 -Inf Inf 124 i4 j5 20 20 -Inf Inf 125 i5 j5 25 25 -Inf Inf 1# if not specified, the toDense() method will convert the level values to a matrix> e$toDense()[,1] [,2] [,3] [,4] [,5][1,] 1 2 3 4 5[2,] 6 7 8 9 10[3,] 11 12 13 14 15[4,] 16 17 18 19 20[5,] 21 22 23 24 25
As with sets, parameters, and variables the primary advantage of the setRecords
method is that GAMS Transfer will convert many different (and convenient) data types into the standard data format (data frame). Users that require higher performance will want to directly pass the Container a reference to a valid data frame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process. In this section, we walk the user through an example of how to set records directly.
- Example #4 - Correctly set records (directly)
In this example, we create a large equation (31,536,000 records and 8880 unique domain elements – we mimic data that is labeled for every second in one year) and assign it to an equation with a$records
. GAMS Transfer R requires that all domain columns must be ordered factors. The records
setter function does very little work other than checking if the object being set is a data frame. This places more responsibility on the user to create a data frame that complies with the standard format. In Example #1 we take care to properly reference the factor from the domain sets – and in the end a$isValid() = TRUE
.
As with sets, parameters, and variables, users can use the $isValid(verbose=TRUE)
method to debug any structural issues.
Alias
- Alias Constructor
Argument | Type | Description | Required | Default |
---|---|---|---|---|
container | Container | A reference to the Container object that the symbol is being added to | Yes | - |
name | string | Name of symbol | Yes | - |
aliasWith | Set object | set object to create an alias for | Yes | - |
- Example - Creating an alias from a set
GAMS Transfer R only stores the reference to the parent set as part of the alias structure – most properties that are called from an alias object simply point to the properties of the parent set (with the exception of refContainer
, name
, and aliasWith
). It is possible to create an alias from another alias object. In this case, a recursive search will be performed to find the root parent set – this is the set that will ultimately be stored as the aliasWith
field. We can see this behavior in the following example:
- Alias Fields
Field | Description | Type |
---|---|---|
description | description of symbol | string |
dimension | dimension of symbol, setting dimension is a shorthand notation to set domain to a list of size n containing "*" | integer |
domainForwarding | flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) | logical |
domainLabels | column headings for the records data frame | list of string |
domainNames | string version of domain names | list of string |
domainType | none , relaxed or regular depending on state of domain links | string |
isSingleton | logical if symbol is a singleton set | logical |
name | name of symbol | string |
numberRecords | number of symbol records (i.e., returns nrow(self$records) if not NULL ) | integer |
records | the main symbol records | data frame |
refContainer | reference to the Container that the symbol belongs to | Container |
summary | output a list of only the metadata | list |
- Alias Methods
Method | Description | Arguments/Defaults | Returns |
---|---|---|---|
getCardinality | get the full cartesian product of the domain | - | integer |
getSparsity | get the sparsity of the symbol w.r.t the cardinality | - | numeric |
isValid | checks if the symbol is in a valid format, throw exceptions if verbose=TRUE , recheck a symbol if force=TRUE | verbose=FALSE force=FALSE | logical |
setRecords | main convenience method to set standard data frame formatted records | records (string vector, list, data frame) | NULL |
- Adding Alias Records
The linked structure of Aliases offers some unique opportunities to access some of the setter functionality of the parent set. Specifically, GAMS Transfer allows the user to change the domain
, description
, dimension
, and records
of the underlying parent set as a shorthand notation. We can see this behavior if we look at a modified Example #1 from R_GAMSTRANSFER_ADD_SET_RECORDS.
- Example - Creating set records through an alias link
- Note
- An alias
$isValid()=TRUE
when the underlying parent set is also valid – if the parent set is removed from the Container the alias will no longer be valid.
Additional Features
Validating Data
GAMS Transfer R requires that the records for all symbols exist in a standard format (Standard Data Formats) in order for them to be understood and written successfully. It is certainly possible that the data could end up in a state that is inconsistent with the standard format (especially if setting symbol attributes directly). GAMS Transfer R includes the $isValid()
method in order to determine if a symbol is valid and ready for writing; this method returns a logical
. For example, we create two valid sets and then check them with $isValid()
to be sure.
- Note
- It is possible to run
$isValid()
on both theContainer
as well as the symbol object –$isValid()
will also return alogical
if there are any invalid symbols in theContainer
object.
- Example (valid data)
- m = Container$new()i = Set$new(m, "i", records = c("seattle", "san-diego", "washington_dc"))j = Set$new(m, "j", i, records = c("san-diego", "washington_dc"))> i$isValid()[1] TRUE> j$isValid()[1] TRUE> m$isValid()[1] TRUE
Now we create some data that is invalid due to domain violations in the set j
.
In this example, we know that the validity of the data is compromised by the domain violations, but there could be other subtle discrepancies that must be remedied before writing data. The user can get more detailed error reporting if the verbose
argument is set to TRUE
. For example:
The $isValid()
method checks:
- If the symbol belongs to a Container
- If all domain set symbols exist in the Container
- If all domain set symbols objects are valid
- If records are a data frame (or
None
) - The shape of the records is congruent with the dimensionality of the symbol
- If records column headings are in standard format
- If all domain columns are
factors
and also ordered - If all domain factors are referenced properly (
$records
for referenced domain sets cannot beNULL
in order to create categoricals properly) - If there are any domain violations
- If there are any duplicate domain members
- That all data columns are type
numerical
- To make sure that all domain categories are type
string
- Note
- Calling
$isValid()
too often may have a significant impact on the performance.
Domain Forwarding
GAMS includes the ability to define sets directly from data using the implicit set notation (see: Implicit Set Definition (or: Domain Defining Symbol Declarations)). This notation has an analogue in GAMS Transfer R called domainForwarding
.
- Note
- It is possible to recursively update a subset tree in GAMS Transfer R.
Domain forwarding is available as an argument to all symbol object constructors; the user would simply need to pass domainForwarding=TRUE
.
In this example, we have raw data that is in the dist
data frame, and we want to send the domain information into the i
and j
sets – we take care to pass the set objects as the domain for parameter c
.
- Note
- The element order in the sets
i
andj
mirrors that in the raw data.
In this example, we show that domain forwarding will also work recursively to update the entire set lineage – the domain forwarding occurs at the creation of every symbol object. The correct order of elements in set i
is (z, a, b, c)
because the records from j
are forwarded first, and then the records from k
are propagated through (back to i
).
Describing Data
The methods describeSets, describeParameters, describeVariables, and describeEquations allow the user to get a summary view of key data statistics. The returned data frame aggregates the output for a number of other methods (depending on symbol type). A description of each Container method is provided in the following subsections:
describeSets
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , string | A list of sets in the Container to include in the output. | No | NULL (if NULL specified, will assume all sets) |
Returns: data frame
The following table includes a short description of the column headings in the return.
Field / Statistic | Description |
---|---|
name | name of the symbol |
isAlias | logical if the symbol is an Alias |
isSingleton | logical if the set is a singleton set |
domain | domain labels for the symbol |
domainType | none , relaxed or regular depending on the symbol state |
dim | dimension |
numberRecs | number of records in the symbol |
cardinality | cartesian product of the domain information |
sparsity | 1 - numberRecs/cardinality |
- Example #1
- m = Container$new("trnsport.gdx")> m$describeSets()name isAlias isSingleton domain domainType dim numberRecs cardinality1 i FALSE FALSE * none 1 2 NA2 j FALSE FALSE * none 1 3 NAsparsity1 NA2 NA
describeParameters
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , string | A list of parameters in the Container to include in the output | No | NULL (if NULL specified, will assume all parameters) |
Returns: data frame
The following table includes a short description of the column headings in the return.
Field / Statistic | Description |
---|---|
name | name of the symbol |
isScalar | logical if the symbol is a scalar (i.e., dimension = 0) |
domain | domain labels for the symbol |
domainType | none , relaxed or regular depending on the symbol state |
dim | dimension |
numRecs | number of records in the symbol |
minValue | min value in data |
meanValue | mean value in data |
maxValue | max value in data |
whereMin | row number min value (if multiple, returns only first occurrence) |
whereMax | row number of max value (if multiple, returns only first occurrence) |
countEps | number of SpecialValues$EPS in data |
countNa | number of SpecialValues[["NA"]] in data |
countUndef | number of SpecialValues$UNDEF in data |
cardinality | cartesian product of the domain information |
sparsity | 1 - numRecs/cardinality |
describeVariables
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , string | A list of variables in the Container to include in the output | No | NULL (if NULL specified, will assume all variables) |
Returns: data frame
The following table includes a short description of the column headings in the return.
Field / Statistic | Description |
---|---|
name | name of the symbol |
type | type of variable (i.e., binary ,integer ,positive ,negative ,free ,sos1 ,sos2 ,semicont ,semiint ) |
domain | domain labels for the symbol |
domainType | none , relaxed or regular depending on the symbol state |
dim | dimension |
numRecs | number of records in the symbol |
cardinality | cartesian product of the domain information |
sparsity | 1 - numRecs/cardinality |
minLevel | min value in the level |
meanLevel | mean value in the level |
maxLevel | max value in the level |
whereMaxAbsLevel | row number of max(abs(level )) in data |
countEpsLevel | number of SpecialValues$EPS in level |
minMarginal | min value in the marginal |
meanMarginal | mean value in the marginal |
maxMarginal | max value in the marginal |
whereMaxAbsMarginal | row number of max(abs(marginal )) in data |
countEpsMarginal | number of SpecialValues$EPS in marginal |
describeEquations
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , string | A list of equations in the Container to include in the output | No | NULL (if NULL specified, will assume all variables) |
Returns: data frame
The following table includes a short description of the column headings in the return.
Field / Statistic | Description |
---|---|
name | name of the symbol |
type | type of variable (i.e., binary ,integer ,positive ,negative ,free ,sos1 ,sos2 ,semicont ,semiint ) |
domain | domain labels for the symbol |
domainType | none , relaxed or regular depending on the symbol state |
dim | dimension |
numRecs | number of records in the symbol |
cardinality | cartesian product of the domain information |
sparsity | 1 - numRecs/cardinality |
minLevel | min value in the level |
meanLevel | mean value in the level |
maxLevel | max value in the level |
whereMaxAbsLevel | row number of max(abs(level )) in data |
countEpsLevel | number of SpecialValues$EPS in level |
minMarginal | min value in the marginal |
meanMarginal | mean value in the marginal |
maxMarginal | max value in the marginal |
whereMaxAbsMarginal | row number of max(abs(marginal )) in data |
countEpsMarginal | number of SpecialValues$EPS in marginal |
describeAliases
Argument | Type | Description | Required | Default |
---|---|---|---|---|
symbols | list , string | A list of aliases in the Container to include in the output. | No | NULL (if NULL specified, will assume all aliases) |
Returns: data frame
The following table includes a short description of the column headings in the return.
Field / Statistic | Description |
---|---|
name | name of the symbol |
aliasWith | name of the parent set |
isSingleton | logical if an alias of a singleton set |
domain | domain labels for the symbol |
domainType | none , relaxed or regular depending on the symbol state |
dim | dimension |
numberRecs | number of records in the symbol |
cardinality | cartesian product of the domain information |
sparsity | 1 - numberRecs/cardinality |
- Example #1
- m = Container$new()i = Set$new(m, "i", records = paste0("i",1:5))j = Set$new(m, "j", records = paste0("j",1:10))ip = Alias$new(m, "ip", i)ipp = Alias$new(m, "ipp", ip)jp = Alias$new(m, "jp", j)> m$describeAliases()name aliasWith isSingleton domain domainType dim numberRecs cardinality1 ip i FALSE * none 1 5 NA2 ipp i FALSE * none 1 5 NA3 jp j FALSE * none 1 10 NAsparsity1 NA2 NA3 NA
Matrix Generation
GAMS Transfer R stores data in a "flat" format, that is, one record entry per data frame row. However, it is often necessary to convert this data format into a matrix/array format – GAMS Transfer R enables users to do this with relative ease using the toDense
symbol methods This method will return a dense N
-dimensional array (matrix for 2-Dimensions) with each dimension corresponding to the GAMS symbol dimension; it is possible to output an array up to 20 dimensions (a GAMS limit).
- Example (1D data w/o domain linking (i.e., a relaxed domain))
Note that the parameter a
is not linked to another symbol, so when converting to a matrix, the indexing is referenced to the data structure in a$records
. Defining a sparse parameter a
over a set i
allows us to extract information from the i
domain and construct a very different dense matrix, as the following example shows:
- Example (2D data w/ domain linking)
The Universe Set
A Unique Element List (UEL) (aka the "universe" or "universe set") is an (i,s)
pair where i
is an identification number for a string s
. GAMS uses UELs to efficiently store domain entries of a record by storing the UEL ID i
of a domain entry instead of the actual string s
. This avoids storing the same string multiple times. The concept of UELs also exists in R and is called a "factor". GAMS Transfer R leverages these types in order to efficiently store strings and enable domain checking within the R environment.
Each domain column in a data frame can be a factor, the effect is that each symbol maintains its own list of UELs per dimension. R lets the user choose if levels in a factor should be regarded as ordered. GAMS Transfer R relies exclusively on ordered
factors (in order for a symbol to be valid it must have only ordered
factors). By using ordered factors, GAMS Transfer R will order the UEL such that elements appear in the order in which they appeared in the data (which is how GAMS defines the UEL). GAMSTransfer
allows the user to reorder the UELs with the uelPriority
argument in the $write()
method.
GAMS Transfer R does not keep track of the UEL separately from other symbols in the Container, it will be created internal to the $write()
method and is based on the order in which data is added to the container. The user can access the current state of the UEL with the $getUniverseSet()
container method. For example, we set a two dimensional set:
R also allows the user to modify (rename, reorder, etc.) factor levels. These methods may be useful for advanced users, but most users will probably find that modifying the original data structures and resetting the symbol records provides a simpler solution. The design of GAMS Transfer R should enable the user to quickly move data back and forth, without worrying about the deeper mechanics of factors.
Removing Symbols
Removing symbols from a Container is easy when using the removeSymbols
container method; this method accepts either a string
or a list
of string
.
- Attention
- Once a symbol has been removed, it is possible to have hanging references as domain links in other symbols. The user will need to repair these other symbols with the proper domain links in order to avoid validity errors.
Reordering Symbols
In order to write the contents of the Container, it is required that the symbols are sorted such that, for example, a Set used as a domain of another symbol appears before that symbol. The Container will try to establish a valid ordering when writing the data. This type of situation could be encountered if the user is adding and removing many symbols (and perhaps rewriting symbols with the same name) – users should attempt to only add symbols to a Container
once, and care must be taken when naming. The method reorderSymbols
attempts to fix symbol ordering problems. The following example shows how this can occur:
- Example Symbol reordering
Since the link to i
is broken, Set j
is now invalid. The user has to manually set the domain again.
Now even though j
is valid, the symbols are out of order. The order can be fixed as follows.
GAMS Special Values
The GAMS system contains five special values: UNDEF
(undefined), NA
(not available), EPS
(epsilon), +INF
(positive infinity), -INF
(negative infinity). These special values must be mapped to their R equivalents. GAMS Transfer R follows the following convention to generate the 1:1
mapping:
+INF
is mapped toInf
-INF
is mapped to-Inf
EPS
is mapped to-0.0
(mathematically identical to zero)NA
is mapped to aNA
UNDEF
is mapped toNaN
GAMS Transfer R syntax is designed to quickly get data into a form that is usable in further analyses or visualization. The user does not need to remember these constants as they are provided within the class SpecialValues
as SpecialValues$POSINF
, SpecialValues$NEGINF
, SpecialValues$EPS
, SpecialValues[["NA"]]
, and SpecialValues$UNDEF
. Some examples are shown below.
The following data frame for x
would look like:
- Note
- The syntax SpecialValues$NA is not allowed in R. Therefore, to access NA, one has to use SpecialValues[["NA"]]. As shown in the example above, double bracket syntax works for other special values too.
Standard Data Formats
This section is meant to introduce the standard format that GAMS Transfer R expects for symbol records. It has already been mentioned that we store data as a data frame, but there is an assumed structure to the column headings and column types that will be important to understand. GAMS Transfer R includes convenience functions in order to ease the burden of converting data from a user-centric format to one that is understood by GAMS Transfer R. However, advanced users will want to convert their data first and add it directly to the Container.
- Set Records Standard Format
- All set records (including singleton sets) are stored as a data frame with
n
number of columns, wheren
is the dimensionality of the symbol + 1. The firstn-1
columns include the domain elements while the last column includes the set element explanatory text. Records are organized such that there is one record per row.
The names of the domain columns follow a pattern of <domain_set_name>_<index_position>
; a symbol dimension that is referenced to the universe is labeled uni_<index position>
. The explanatory text column is called element_text
and must take the last position in the data frame.
All domain columns must be factors
and the element_text
column must be a string
type.
Some examples:
- Parameter Records Standard Format
- All parameter records (including scalars) are stored as a data frame with
n
number of columns, wheren
is the dimensionality of the symbol + 1. The firstn-1
columns include the domain elements while the last column includes the numerical value of the records. Records are organized such that there is one record per row. Scalar parameters have zero dimension, therefore they only have one column and one row.
The names of the domain columns follow a pattern of <domain_set_name>_<index_position>
; a symbol dimension that is referenced to the universe is labeled uni_<index_position>
. The value column is called value
and must take the last position in the data frame.
All domain columns must be factors
and the value
column must be a numeric
type. GAMS Transfer R requires that all the factor levels are of type string
.
Some examples:
- Variable/Equation Records Standard Format
Variables and equations share the same standard data format. All records (including scalar variables/equations) are stored as a data frame with n
number of columns, where n
is the dimensionality of the symbol + 5. The first n-5
columns include the domain elements while the last five columns include the numerical values for different attributes of the records. Records are organized such that there is one record per row. Scalar variables/equations have zero dimension, therefore they have five columns and one row.
The names of the domain columns follow a pattern of <domain_set_name>_<index position>
; a symbol dimension that is referenced to the universe is labeled uni_<index_position>
. The attribute columns are called level
, marginal
, lower
, upper
, and scale
. These attribute columns must appear in this order. Attributes that are not supplied by the user will be assigned the default GAMS values for that variable/equation type. It is possible to not pass any attributes, GAMS Transfer R would then simply assign default values to all attributes.
All domain columns must be factors
and the attribute columns must be a numeric
type. GAMS Transfer R requires that all the factor levels are of type string
.
Some examples:
Data Exchange with GDX
Up until now, we have been focused on using GAMS Transfer R to create symbols in an empty Container using the symbol constructors (or their corresponding container methods). These tools will enable users to ingest data from many different formats and add them to a Container
– however, it is also possible to read in symbol data directly from GDX files using the read
container method. In the following sections, we will discuss this method in detail as well as the write
method, which allows users to write out to new GDX files.
Reading from GDX
There are two main ways to read in GDX based data.
- Pass the file path directly to the Container constructor (will read all symbols and records)
- Pass the file path directly to the
read
method (default read all symbols, but can read partial files)
The first option here is provided for convenience and will, internally, call the read
method. This method will read in all symbols as well as their records. This is the easiest and fastest way to get data out of a GDX file and into your R environment. For the following examples, we leverage the GDX output generated from the trnsport.gms model file.
A user could also read in data with the read method as shown in the following example.
- Example (reading full data w/ read method)
It is also possible to read in a partial GDX file with the read method, as shown in the following example:
This syntax assumes that the user will always want to read in both the metadata as well as the actual data records, but it is possible to skip the reading of the records by passing the argument records=FALSE
.
- Attention
- The
read
method attempts to link the domain objects together (in order to have a "regular"domainType
) but if domain sets are not part of the read operation there is no choice but to default to a "relaxed"domain_type
. This can be seen in the last example where we only read in the variablex
and not the domain sets (i
andj
) that the variable is defined over. All the data will be available to the user, but domain checking is no longer possible. The symbolx
will remain with "relaxed" domain type even if the user were to read in setsi
andj
in a secondread
call.
Writing to GDX
A user can write data to a GDX file by simply passing a file path (as a string). The write
method will then create the GDX and write all data in the Container
.
- Note
- It is not possible to write the Container when any of its symbols are invalid. If any symbols are invalid an error will be raised and the user will need to inspect the problematic symbols (perhaps using a combination of the
listSymbols(isValid=FALSE)
andisValid(verbose=TRUE)
methods).
- Example
- m$write("path/to/file.gdx")
- Example (write a compressed GDX file)
- m$write("path/to/file.gdx", compress = TRUE)
Advanced users might want to specify an order to their UEL list (i.e., the universe set); recall that the UEL ordering follows that dictated by the data. As a convenience, it is possible to prepend the UEL list with a user specified order using the uelPriority
argument.
- Example (change the order of the UEL)
The original UEL order for this GDX file would have been c("a", "b", "c")
, but this example reorders the UEL with uelPriority
– the positions of b
and c
have been swapped. This can be verified with the gdxdump
utility (using the uelTable
argument):
gdxdump foo.gdx ueltable=foo Set foo / 'a' , 'c' , 'b' /; $onEmpty Set i(*) / 'a', 'c', 'b' /; $offEmpty