Table of Contents
Symbol records are the actual data.
In GAMS Transfer Matlab they are stored in Matlab native data structures: structs, tables, dense or sparse matrices (see Records Format for more information). In GDX, a record is the combination of domain entry data and value data. Domain entries are given as UELs. The values a symbol stores per record depends on the symbol type. Sets have element_text
, Parameters have value
and Equations and Variables have level
, marginal
, lower
, upper
and scale
. If some of these value fields are not provided, then default values are used. A record format is chosen for each symbol and not for each of these values independently. Hence note, a container can store different symbols with different record formats.
When working with symbol records, there are two things that can go wrong:
- Record data does not satisfy the chosen format.
- Simply ask whats wrong with symbol.Abstract.isValid, see Validate Symbol Records.
- Record data contains invalid domain entries, so called domain violations.
- Simply ask for domain violations with symbol.Abstract.[has|count|get|find]DomainViolations, see Domain Violations.
Records Format
GAMS Transfer Matlab can read and maintain the symbol records in four different formats: struct
, table
, dense_matrix
and sparse_matrix
. Both struct
and table
are table-like formats and the dense_matrix
and sparse_matrix
– obviously – matrix-like formats. The default is table
as it allows for a good data display and overview. However note that table
is not the most efficient format.
- Table-Like Formats:
The formats
table
andstruct
store the domain entries in the first dimension columns followed by value columns (element_text
for Set,value
for Parameter andlevel
,marginal
,lower
,upper
,scale
for Variable and Equation) and the records as rows. In case ofstruct
, the columns are given as struct fields. The column names for domain entry columns can be shown and altered by symbol.Abstract.domain_labels.For example,
x
in Example astable
:>> x.recordsans =6×7 tablei j level marginal lower upper scale_________ ________ _____ ________ _____ _____ _____seattle new-york 50 0 0 Inf 1seattle chicago 300 0 0 Inf 1seattle topeka 0 0.036 0 Inf 1san-diego new-york 275 0 0 Inf 1san-diego chicago 0 0.009 0 Inf 1san-diego topeka 275 0 0 Inf 1For example,
x
in Example asstruct
:>> x.recordsans =struct with fields:i: [6×1 categorical]j: [6×1 categorical]level: [6×1 double]marginal: [6×1 double]lower: [6×1 double]upper: [6×1 double]scale: [6×1 double]>> x.records.levelans =5030002750275
- Note
- Sets can only be maintained in table-like formats
struct
andtable
.
- Matrix-Like Formats:
The formats
dense_matrix
andsparse_matrix
store the record values individually as matrices with dimensionmax(2,d)
, whered
is the symbol dimension, and shape size. If size is undefined (see Symbol Domain), a matrix-like format is not possible. Domain entries cannot be stored in the matrix, but can be queried using the symbol method getUELs (see also Unique Elements (UELs)). Assume a symbols
has two dimensions. Then, a(row,col)
matrix entry corresponds to the domain entrys.getUELs(1, [row, col])
. The logic is analogue for different dimensions.For example,
x
in Example asdense_matrix
:>> x.recordsans =struct with fields:level: [2×3 double]marginal: [2×3 double]lower: [2×3 double]upper: [2×3 double]scale: [2×3 double]>> x.records.levelans =50 300 0275 0 275For example,
x
in Example assparse_matrix
:>> x.recordsans =struct with fields:level: [2×3 double]marginal: [2×3 double]lower: [2×3 double]upper: [2×3 double]scale: [2×3 double]>> x.records.levelans =(1,1) 50(2,1) 275(1,2) 300(2,3) 275In order to get the domain entries for matrix elements, note that in the above examples the following holds:
x.getUELs(1, 1:2) equals {'seattle', 'san-diego'}x.getUELs(2, 1:3) equals {'new-york', 'chicago', 'topeka'}
- Note
- For scalar symbols (dimension equals 0), the formats
struct
anddense_matrix
are equivalent.
Each format has its advantages and disadvantages, see the following table. There, ratings are ++
(very good), +
, o
, -
, --
(rather bad), rated relatively for each category.
Record Format | Max Dimension | Efficiency | Memory (General) | Memory (Dense Data) | Display |
---|---|---|---|---|---|
struct | 20 (GAMS limit) | ++ | + | - | - |
table | 20 (GAMS limit) | -- | o | -- | ++ |
dense_matrix | 20 (GAMS limit) | + | -- | ++ | - |
sparse_matrix | 2 (Matlab limit) | o | ++ | + | -- |
The possibilities to chose or change a record format are:
- Specify the format when reading from GDX.
- Let GAMS Transfer Matlab choose the record format based on the data provided to symbol.Abstract.setRecords.
- Transform the records to a different format using symbol.Abstract.transformRecords.
- Change the record format (without transforming the records) using the property symbol.Abstract.format.
Validate Symbol Records
GAMS Transfer Matlab requires the symbol records to be stored in one of the supported record formats in order to understand and write them to GDX. However, it can easily happen that a certain criteria of the format is not met and the symbol is marked as invalid, i.e., the symbol method symbol.Abstract.isValid returns false
. In that case setting the argument verbose
of symbol.Abstract.isValid to true
will print the reason for invalidity and can thus help to resolve the issue.
For example, take x
of Example, which is of course valid:
Let's invalidate this symbol by storing it in an incorrect shape:
Unique Elements (UELs)
A Unique ELement (UEL) is an (i,s)
pair where i
is an identification number (or index) for a (string) label s
. GDX uses UELs to efficiently store domain entries of a record by storing the UEL index i
of a domain entry instead of the actual string s
. This avoids storing the same string multiple times. The concept of UELs also exists in Matlab and is called a categorical
. If you are new to categorical
: Work with categorical
as you would work with strings – it's just a way to save memory. Converting a categorical
to uint64
(or similar) reveals the identification number. In GAMS Transfer Matlab you can specify the identification numbers either by categorical
(usually default) or by providing the numbers directly.
- Note
categorical
will trim the strings and hence prefixed or trailing whitespaces are not supported. In case of reading a GDX file with those, GAMS Transfer Maltab will use identification numbers instead.
For example, note the categorical
in x
of Example
- Attention
- In the very unlikely case that your Matlab (or Octave) version does NOT support
categorical
(Matlab earlier than R2013b or any Octave version), please go on and read carefully about UELs. You must store the UEL indices in the domain columns of the symbol.Abstract.records. Storing strings is not supported. For looking up the corresponding string, use the method symbol.Abstract.getUELs. Otherwise, if you can usecategorical
– very likely –, you won't need the Symbol methods regarding adding or modifying UELs, explained below.
Each symbol maintains its own list of UELs per dimension, which can be accessed and modified via the methods symbol.Abstract.getUELs, symbol.Abstract.setUELs, symbol.Abstract.addUELs, symbol.Abstract.removeUELs, symbol.Abstract.renameUELs and symbol.Abstract.reorderUELs (or the Matlab functions for modifying categorical
directly). The UEL indices are numbered from 1 to the number of UELs stored independently for each dimension.
- Attention
- The methods symbol.Abstract.setUELs and symbol.Abstract.removeUELs may reassign different UEL indices to the UEL labels. These changes are applied to the indices used in symbol.Abstract.records. This implies that removing a UEL that is used in the symbol.Abstract.records will then lead to an invalid domain entry (displayed as
<undefined>
in Matlab). For symbol.Abstract.setUELs, these updates can be disabled by passing the arguments ‘'rename’, true`.
For example, continuing the Example from above, the UELs are:
Changing these can invalidate some records:
- Advanced Users Only:
- Even with support of
categorical
it can be useful to explicitly add a UEL with symbol.Abstract.addUELs, although simply using a new UEL in symbol.Abstract.records will add it automatically to the UEL list. However, it is possible to store more UELs than those actually used in the records. Advanced users can use this fact to sort the universe set of a GDX file to their needs, see also Writing To GDX.
Domain Violations
Domain violations occur when a symbol uses other Sets as domain(s) – and is thus of domain type regular
, see Symbol Domain – and uses a domain entry in its records that is not present in the corresponding referenced domain set. Such a domain violation will lead to a GDX error when writing the data!
- Note
- Checking for domain violations is not part of symbol.Abstract.isValid for performance reasons.
For example, altering x
in Example – remember that x
has domains {i,j}
, where i
and j
are Sets and madison
is not part of set i
– as follows:
doesn't update the domain set i
:
Trying to write this to a GDX file will fail:
To ask for domain violations, call the method symbol.Abstract.getDomainViolations. It returns a list of DomainViolation objects w.r.t. each dimension of the symbol which can then be used to resolve the domain violations: The GAMS Transfer Matlab methods symbol.Abstract.resolveDomainViolations and DomainViolation.resolve offer an automatic expansion of the domain sets with the violated entries in order to eliminate domain violations.
For example, continuing the example from above,
shows the domain violation:
Calling either of the following
resolves it:
This resolving feature can further be triggered automatically by setting the symbol property symbol.Abstract.domain_forwarding to true
. If records are updated by direct access (see Assigning Symbol Records), the domain update will happen delayed for improved efficiency, but can be forced by calling isValid or the resolving methods mentioned above.
- Note
- The method for automatically resolving the domain violations can be convenient, but it effectively disables domain checking, which is a valuable tool for error detection. We encourage to use symbol.Abstract.resolveDomainViolations, DomainViolation.resolve or symbol.Abstract.domain_forwarding enabled as rarely as possible. The same holds for using
relaxed
domain information whenregular
domain information would be possible, see Symbol Domain.