GAMS [ Home | Downloads | Documentation | Solvers | APIs | Tools | Model Libraries | Resources | Sales | Support | Contact Us | Search ]

GAMS Data eXchange (GDX)

This document describes the GDX (GAMS Data eXchange) facilities available in GAMS. In addition to these facilities, there are a few utilities to work with GDX files.

A GDX file is a file that stores the values of one or more GAMS symbols such as sets, parameters variables and equations. GDX files can be used to prepare data for a GAMS model, present results of a GAMS model, store results of the same model using different parameters etc. A GDX file does not store a model formulation or executable statements.

GDX files are binary files that are portable between different platforms. They are written using the byte ordering native to the hardware platform they are created on, but can be read on a platform using a different byte ordering.

Users can also write their own programs using GDX files by using the gdxdclib library. The interface and usage for this library is described in a separate document; see gdxioapi.chm or gdxioapi.pdf.

Compression:

Starting with version 22.3 of GAMS, gdx files can be written in a compressed format. Compression is controlled by the environment variable GDXCOMPRESS. A value of 1 indicates compression.

GDX files can be converted to a compressed format or an older format too; see GDXCOPY.

Using the GDX facilities in GAMS

Reading and writing of GDX files in a GAMS model can be done during the compile phase or the execution phase. A GDX file can also be written as the final step of GAMS compile or execute sequence.

The GAMSIDE can read a GDX file and display its contents.

Compile Phase

During compilation, we can use dollar control options to specify the gdx file and the symbols to read or write. Reading during the compilation phase also allows us to define the elements of a set and the subsequent use of such a set as a domain.

Compile Phase Reading Data

The following directives are available for reading data from a GDX file into GAMS during compilation of a GAMS model:

  • $GDXIN - Close the current GDX input file
    Parameter(s) Description
    Filename Specify the GDX file to be used for reading
  • $LOAD - List all symbols in the GDX file
    Parameter(s) Description
    id=* Loads all unique elements from the gdx file into set id
  • $LOAD and $LOADDC - List all symbols in the GDX file
    Parameter(s) Description
    id1 id2 ... idn Read GAMS symbols id1, id2, … idn from the GDX file
    id1=gdxid1 id2=gdxid2 Read GAMS symbols id1, id2 with corresponding names gdxid1, gdxid2 in the GDX file
    id1<gdxid1 id2<gdxid2.dim3 Reads GAMS one dimensional set id1 and id2 from the GDX parameter or set gdxid1 and gdxid2. WIthout the dimN suffix, GAMS tries to match the domains from the right (<). If no domain information is available for the GDX symbol, the dimN suffix determines the index position that should be read into the GAMS set. For more details see the fourth example.
    id1<=gdxid1 id2<=gdxid2.dim3 Reads GAMS one dimensional set id1 and id2 from the GDX parameter or set gdxid1 and gdxid2. WIthout the dimN suffix, GAMS tries to match the domains from the left (<=). If no domain information is available for the GDX symbol, the dimN suffix determines the index position that should be read into the GAMS set. For more details see the fourth example.

    Note: $LOAD simply ignores elements that are not in the domain. $LOADDC will cause a compilation error when the data read causes a domain violation.

  • $LOADM, $LOADR, $LOADDCM, and $LOADDCR - Additional forms of the $LOAD and $LOADDC directives. The M indicating a merge and the R indicating a full replacement.
    Parameter(s) Description
    id1 id2 ... idn Read GAMS symbols id1, id2, … idn from the GDX file
    id1=gdxid1 id2=gdxid2 Read GAMS symbols id1, id2 with corresponding names gdxid1, gdxid2 in the GDX file
    id1<gdxid1 id2<gdxid2.dim3 Reads GAMS one dimensional set id1 and id2 from the GDX parameter or set gdxid1 and gdxid2. WIthout the dimN suffix, GAMS tries to match the domains from the right (<). If no domain information is available for the GDX symbol, the dimN suffix determines the index position that should be read into the GAMS set. For more details see the fourth example.
    id1<=gdxid1 id2<=gdxid2.dim3 Reads GAMS one dimensional set id1 and id2 from the GDX parameter or set gdxid1 and gdxid2. WIthout the dimN suffix, GAMS tries to match the domains from the left (<=). If no domain information is available for the GDX symbol, the dimN suffix determines the index position that should be read into the GAMS set. For more details see the fourth example.
  • $LOADIDX - Read GAMS symbols from the GDX file. Each symbol should have been written using an indexed write.
    Parameter(s) Description
    id1 id2 ... idn Read GAMS symbols id1 id2 ... idn from the GDX file. Each symbol should have been written using an indexed write; see Execute_UnloadIDX
    id1=gdxid1 id2=gdxid2 Read GAMS symbols id1, id2 with corresponding names gdxid1, gdxid2 in the GDX file. Each symbol should have been written using an indexed write; see Execute_UnloadIDX
Attention
  • Only one GDX file can be open at the same time.
  • When reading data, the symbol to be read has to be defined in GAMS already

Compile Phase Writing Data

Writing to a GDX file during compilation

  • $GDXOUT - Close the current GDX output file
    Parameter(s) Description
    Filename Specify the GDX file to be used for writing
  • $UNLOAD - Write GAMS symbols from the GDX file.
    Parameter(s) Description
    (no identifiers) Write all symbols to the gdx file
    id1 id2 ... idn Write GAMS symbols id1, id2, … idn to the GDX file
    id1=gdxid1 id2=gdxid2 Write the GAMS symbol id1 to the GDX file with name gdxid1
Attention
  • Only one GDX file can be open at the same time.
  • When writing data, an existing GDX file will be overwritten with the new data; there is no merge or append option.

Example 1

The trnsport.gms model ([TRNSPORT] model from the GAMS Model Library) has been modified to use the demand data from an external source. Only the relevant declarations are shown.

The parameter B is read from the GDX file using the name 'demand', and only those elements that are in the domain J will be used. Values for parameter B that are outside the domain J will be ignored without generating any error messages.

*Example 1
Set  
   j    markets  / new-york, chicago, topeka / ;
Parameter 
   B(j) demand at market j in cases ;
$GDXIN demanddata.gdx  
$LOAD  b=demand 
$GDXIN

Example 2

In this example, the set J is also read from the GDX file, and is used as the domain for parameter B. All elements read for the set J will be used. Values for the parameter B that are outside the domain J will be ignored. Note that the dimension of set J is set to one by specifying its domain.

*Example 2
$GDXIN demanddata.gdx  
Set  
   J(*)    markets;
$LOAD j=markets
Parameter 
   B(j) demand at market j in cases ;
$LOAD  b=demand 
$GDXIN

Example 3

Using $LOAD to get a listing of all symbols

*Example 3
$GDXIN trnsport.gdx
$LOAD

Writes the following to the listing file:

Content of GDX C:\XLSFUN\TRNSPORT.GDX

umber Type      Dim  Count  Name
 
    1 Set         1      2  i         canning plants
    2 Set         1      3  j         markets
    3 Parameter   1      2  a         capacity of plant i in cases
    4 Parameter   1      3  b         demand at market j in cases
    5 Parameter   2      6  d         distance in thousands of miles
    6 Parameter   0      1  f         freight in dollars per case per thousand miles
    7 Parameter   2      6  c         transport cost in thousands of dollars per case
    8 Variable    2      6  x         shipment quantities in cases
    9 Variable    0      1  z         total transportation costs in thousands of dollars
   10 Equation    0      1  cost      define objective function
   11 Equation    1      2  supply    observe supply limit at plant i
   12 Equation    1      3  demand    satisfy demand at market j

Example 4

Sometimes, a set is implicitly given by the elements of a parameter symbol. For example,

parameter a(i) / seattle 350, san-diego 600 /

in trnsport.gms implicitly defines the set of plants i. GAMS does not allow us to provide domain checked data, if the data for domain sets is unknown. So this code produces a compilation error:

Set i plant;
Parameter a(i) capacity / seattle 350, san-diego 600 /;

When entering data directly in the GAMS source adding the domain sets before the actual parameter declarations is usually not a problem, but when data comes from external sources (e.g. spreadsheets, databases, etc), this often results in an additional query to the database, spreadsheet etc. Nowadays, such data exchange happens mostly via the GD facility. With the domain load capability of the compile time load instructions ($load, $loadDC, $loadR, $loadM, $loadDCM, and $loadDCR) one can project an index position from a parameter or set symbol in the GDX container and load this slice into a one dimensional set. Here is a simple example:

Set i plant;
Parameter a(i) capacity;
$gdxin data
$load i<adata a=adata

This will try to load set elements from the GDX parameter symbol adata into the set i and next load the GDX parameter adata into the GAMS parameter a. The latter one is no problem anymore, since the data for set i is known when loading symbol a. GAMS will use the domain information stored in GDX of parameter adata to identify the index position to project on. If no appropriate domain information can be found in GDX, the GAMS compiler will generate an error. In such case the user can explicitly select an index position (here first index position) from the GDX symbol:

$load i<adata.dim1 a=adata

The automatic index position matching (i.e. no .dimN) using the domain information stored in GDX matches on the name of the set to be loaded and the domain set names stored in GDX for the symbol. The domain in GDX are searched from right to left (start with n=symbol dimension, then n-1, n-2, ...) and stops at the first match. With the projection symbol <=, the domain in GDX is searched from left to right. This follows the style of the GAMS run time projection operation:

option sym1<sym2, sym1<=sym2;

Here is an example how to load. The network is defined by the capacity parameter cap contained in a GDX container net.gdx:

parameter cap(n,n) / (1*3).4 5, 4.(5*9) 3 /;

The following code loads the entire node set n of the network as well as the nodes with outgoing (out) and incoming (in) arcs and the capacity c.

set n nodes, out(n), in(n);
parameter c(n,n) capacity
$gdxin net
$loadM n<=cap n<cap
$loadDC out<cap.dim1 in<cap.dim2 c=cap
display n, out, in;

The listing file looks as follows:

----      6 SET n  nodes
1,    2,    3,    4,    5,    6,    7,    8,    9
 
----      6 SET out  Domain loaded from cap position 1
1,    2,    3,    4
 
----      6 SET in  Domain loaded from cap position 2
4,    5,    6,    7,    8,    9

There is a potential issue with loading domains from parameters that have a zero value for some record. Since GAMS works with sparse data, it is sometime difficult to distinguish between a record with value zero (0) and the non-existence of a record. This is usually not a problem since we know the domain of a parameter and hence know all potential records. In case of using a parameter to define the domain this represents a source of confusion. Moreover, GDX has the capability of storing true zeros (most GDX utilities like gdxxrw have options (Squeeze=Y or N) to either write a true 0 or squeeze the 0s when writing GDX). So in case GDX has a zero record, a domain load from such a parameter will include this record. Here is an example. The spreadsheet Book1.xlsx contains the following data:

embim1.jpg

The GDX utility GDXXRW with the following command line

gdxxrw Book1.xlsx Squeeze=N par=dat rng=Sheet1!a1 rdim=1

Reads the Excel data and produces a GDX container Book1.gdx with a one dimensional parameter dat(*) which can be viewed in the GDX browser in the GAMSIDE:

embim2.jpg

Notice that label a4 is present while label a3 is not part of GDX symbol dat. Without the Squeeze=N (the default is Squeeze=Y) we also would not have seen a4. If we load dat to define the domain (remember we need to use $load i<dat.dim1 since gdxxrw does not write domain information to GDX), we will miss out on a3 but have a4 (assuming Squeeze=N). Please also note that the zero record disappears on regular loading and is turned into an EPS when loading under $OnEps:

set i;
parameter a(i);
$gdxin Book1
$load i<dat.dim1 a=dat
display i,a;
parameter a0(i);
$OnEps
$load a0=dat
display a0;

This results in a listing file

----      5 SET i  Domain loaded from dat position 1
a1,    a2,    a4
 
----      5 PARAMETER a
a1 5.000,    a2 1.000
 
----      9 PARAMETER a0
a1 5.000,    a2 1.000,    a4   EPS

With gdxxrw parameter Squeeze=Y the listing file would look as follows:

----      5 SET i  Domain loaded from dat position 1
a1,    a2
 
----      5 PARAMETER a
a1 5.000,    a2 1.000
 
----      9 PARAMETER a0
a1 5.000,    a2 1.000

Execution phase

During execution, we can read and write GDX files with the following statements:

To read data

execute_load   'filename',id1,id2=gdxid2,..;
execute_loaddc 'filename',id1,id2=gdxid2,..;

The execute_load statement acts like an assignment statement, except that it does not merge the data read with the current data; it is a full replacement. The same restrictions apply as in an assignment statement: we cannot assign to a set that is used as a domain, or to a set used as a loop control. With execute_loaddc any domain violation will be reported and flagged as execution error. In contrast, execute_load ignores all domain violations and loads only data that meets the domain restrictions. In addition to loading data for sets, parameters and variables, we can load a field of a variable into a parameter. Warning: when loading a single field, all other fields are reset to their default value.

To write data

execute_unload    'filename',id1,id2=gdxid,..;
execute_unloaddi  'filename',id1,id2=gdxid,..;
execute_unloadidx 'filename',id1,id2=gdxid,..;

The execute_unload statement replaces an existing file with that name; it does not add symbols to or replace symbols in an existing GDX file. Without specifying any identifier, all sets, parameters, variables and equations will be written to the GDX file. Execute_unloaddi does the same as execute_unload, but also writes the domains of all unloaded symbols to the same file.

The execute_unloadidx statement requires that each symbol written is a parameter; each parameter must have a domain specified for each index position. These domains have the requirement that they are formed using an integer sequence for the UELs that starts at 1 (one). The domain names are changed to indicate the size of each domain. This information is used when reading the data back from the GDX file using $LoadIDX during compilation. Using the special domain names, the UELs for the domains can be recovered without writing the domains to the GDX file; see example below.

The GAMS option gdxUELs controls which UELs are registered in filename. With option gdxUELs = squeezed; (default) only the UELs that are required by the exported symbols are registered while all known UELs are registered if we set option gdxUELs = full;.

Write a solution point

save_point = n

This is an option, specified on the command line, using an option statement or a model attribute, to write the current model solution to a GDX file. The option values are:

0:   do not write a point file (default)
1:   write the solution to <workdir><modelname>_p.gdx
2:   write the solution to <workdi><modelname>_p<solvenumber>.gdx

Read a solution

execute_loadpoint 'filename';
execute_loadpoint 'filename',id1,id2=gdxid2,..;

The execute_loadpoint allows you to merge solution points into any GAMS database. Loading the data acts like an assignment statement and it merges/replaces data with current data. For variables and equations, only level values and marginal values (.L and .M) are used. If no symbols are specified, all symbols that match in type and dimensionality will be loaded into the GAMS database.

The gdx file that can be used is not limited to files created with SAVE_POINT; any gdx file can be used.

Example: Execution Phase Writing Data

This example again uses the trnsport.gms model. After solving the model, we write the sets I and J and the variables Z and X to the GDX file:

*Example 5
Set I /. . ./,
    J / . . . /;
Variable X(I,J),
         Z;
. . .

Solve trnsport using LP minimizing Z;
Execute_Unload 'results.gdx',I,J,Z,X;

Example: Indexed writing and reading of data

This example shows the use of the indexed write and read data:

Set I /1*100/,
    J /1*50 /;
parameter A(I,J) /1.1=11, 1.9=19, 10.1=101/;
 
Execute_UnloadIDX 'data.gdx', A;

Viewing the file data.gdx in the gamside shows the modified domain information:

clip0015.gif
gamside showing data.gdx note the modified domains

To read from data.gdx, we use the indexed read:

Set I,J;
parameter A(I,J);
*load the data
$gdxin data.gdx
$LoadIDX A
$gdxin
*write all symbols so we can inspect in the gamside
$gdxout test.gdx
$unload
$gdxout
 
Execute_UnloadIDX 'data.gdx', A;

Viewing the file test.gdx in the gamside shows that the domains have been populated:

clip0016.gif
View test.gdx in the gamside

Writing a GDX file after compilation or execution

Using the gdx option in the GAMS call, will cause all sets, parameters, variables and equations to be written to the GDX file.

For example:

Gams trnsport gdx=trnsport

Or

Gams trnsport a=c gdx=trnsport

Using the gdx parameter when running the model using the GAMSIDE, the process window will show the GDX filename in blue indicating that the file can be opened using a double-click with the mouse.

Inspecting a GDX file

After creating a GDX file there are a few ways to look at its contents:

  • The $LOAD directive without any parameters will show a listing of all symbols in the file.
  • The GAMSIDE can be used to view the contents of a GDX file by opening the file as any other file. The IDE only recognizes the .gdx file extension.
  • The GDXDUMP utility can list the symbols in the file and it also can write sets and parameters formatted as a GAMS data statement.
  • The GDXDIFF utility can be used to compare two GDX files by creating a third GDX file containing the differences between all symbols with the same name, type and dimension.

GDX Utilities

This section describes an alphabetical list of GDX based tools and GDX related tools, included in any GAMS distribution and maintained by GAMS. See also Tools included in the distribution.

  • CHOLESKY Matrix decomposition A=LL^T
  • CSV2GDX converts a .CSV file (comma separated values) to a GDX file.
  • EIGENVALUE calculates eigenvalues of a symmetric matrix
  • EIGENVECTOR calculates eigenvalues/vectors of a symmetric matrix
  • GDX2ACCESS dumps the contents of a GDX file to an MS Access file (.mdb file). Every identifier gets its own table in the .MDB file.
  • GDX2HAR and HAR2GDX translate between HAR and GDX file formats.
  • GDX2SQLITE dumps GDX contents into SQLite database file.
  • GDX2XLS converts an entire gdx data container to a Microsoft Excel spread sheet.
  • GDXCOPY copies/converts one or more GDX files to a different format.
  • GDXDIFF compares the data of symbols with the same name, type and dimension in two GDX files and writes the differences to a third GDX file.
  • GDXDUMP writes the contents of a GDX file as a GAMS formatted text file.
  • GDXMERGE combines multiple GDX files into one file. Symbols with the same name, dimension and type are combined into a single symbol of a higher dimension. The added dimension has the file name of the combined file as its unique element.
  • GDXMRW imports and exports data between GAMS and MATLAB and to call GAMS models from MATLAB and get results back into MATLAB.
  • GDXRANK reads one or more one dimensional parameters from a GDX file, sorts each parameter and writes the sorted indices as a one dimensional parameters to the output GDX file.
  • GDXRENAME renames the unique elements in a GDX file using the unique elements from second GDX file.
  • GDXXRW allows reading and writing of an Excel spreadsheet. This utility requires the presence of Microsoft Excel and therefore can only be used on a PC running the Windows operating system with Microsoft Excel installed.
  • GDXVIEWER views and converts data contained in GDX files. Besides inspecting a GDX file, gdxviewer allows you to export to a large number of data formats, including ASCII text, CSV, HTML, XML, database, and spreadsheet formats.
  • INVERT performs a matrix inversion.
  • MCFILTER filters duplicate and dominated points from a solution set
  • MDB2GMS converts data from an Microsoft Access database into GAMS readable format. The source is an MS Access database file (*.MDB) and the target is a GAMS Include File or a GAMS GDX File.
  • SQL2GMS converts data from an SQL database into GAMS readable format. The source is any data source accessible through Microsoft's Data Access components including ADO, ODBC and OLEDB. The target is a GAMS Include File or a GAMS GDX File.
  • XLS2GMS converts spreadsheet data from a Microsoft Excel spreadsheet into GAMS readable format. The source is a MS Excel spreadsheet file (*.XLS) and the target is a GAMS Include File.
  • XLSDump writes all worksheets of a MS Excel workbook to a GDX file. Unlike gdxxrw, the program does not require that Excel is installed.
  • XLSTalk opens/closes/runs macro in MS Excel.

Some tools listed above are MS Windows based tools for data exchange between different applications, which are included in the current GAMS Distribution and are maintained by GAMS. These tools are designed as interactive Windows programs, but most of them can also be operated through command line parameters. See Support Platforms for more details.