Table of Contents
The program GDXMERGE
combines multiple GDX files into a single GDX file. Symbols with the same name, dimension and type are combined into a single symbol of a higher dimension. The added dimension has the file name of the combined file as its unique element.
Usage
gdxmerge filepattern1 filepattern2 .... filepatternN {options}
Each file pattern represents a file name or a wildcard representation using ? and *. A parameter of the form @filename will process the commands from the text file specified. The result of the GDXMERGE
execution will be written to a file called merged.gdx, unless this default is overwritten by the output option.
The .gdx file extension can be omitted. Files without a full path name are assumed to be in the current directory when using a command prompt. When using the GAMS IDE, these files are assumed to be in the current project directory.
Options
The following options can be specified:
id = <ident1>, <ident2>... (default = none)
Only merge the symbols
ident1
,ident2
, ...exclude = <ident1>, <ident2>... (default = none)
Merge all symbols except for
ident1
,ident2
`, ...big = <integer>
The size for big symbols.
strict = true|false (default = false)
The program terminates with an error (non-zero return code) if input files specified cannot be found or a file pattern results in no files. An already existing output file also triggers an error if strict is set to true.
output = fileName (default = merged.gdx)
The optional output file name.
All symbols with matching type and dimension will be merged. By specifying the parameter id=ident1
the merge process will only be performed for the identifier(s) specified, while exclude=ident1
indicates that all symbols should be merged except for the ones specified in the exclude
list. Note that the two options id
and exclude
are mutually exclusive.
By default, the program reads all GDX once and stores all data in memory before writing the merged.gdx file. The big
parameter is used to specify a cutoff for symbols that will be written one at a time. Each symbol that exceeds the size will be processed by reading each GDX file and only process the data for that symbol. This can lead to reading the same GDX file many times, but it allows the merging of large data sets. The formula used to calculate the cutoff is:
dimension * totalNumberOfElements
.
The calculated value is doubled for variables and equations.
In addition to the symbols written, a set is added to the GDX file representing all the files processed during the merge operation. The name of the set is Merged_set_1
, and is made unique by changing the number. The explanatory text for each set element contains the date and time of the GDX file processed.
- Note
- The file merged.gdx, or the file specified with the output parameter, will never be used in a merge operation even if the name matches a file pattern.
- Symbols with dimension 20 cannot be merged, because the resulting symbol will have dimension 21 which exceeds the maximum dimension allowed by GAMS.
Examples
Merging several GDX Files
In this example, we solve the [trnsport] model from the GAMS Model Library using different LP solvers. After each run, we write all symbols to a GDX file and merge the files into one file. The variable X
is read from the merged file and displayed.
$call gamslib trnsport
$call gams trnsport lp=cplex gdx=cplex
$call gams trnsport lp=xpress gdx=xpress
$call gams trnsport lp=conopt gdx=conopt
$call gams trnsport lp=minos gdx=minos
$call gams trnsport lp=snopt gdx=snopt
$call gdxmerge *.gdx
Variable AllX(*,*,*);
$gdxIn merged.gdx
$load AllX=X
$gdxIn
option AllX:5:1:2;
display AllX.L;
The display statement generates the following output in the listing file:
---- 22 VARIABLE AllX.L shipment quantities in cases seattle seattle san-diego san-diego new-york chicago new-york topeka conopt 300.00000 325.00000 275.00000 cplex 50.00000 300.00000 275.00000 275.00000 minos 50.00000 300.00000 275.00000 275.00000 snopt 50.00000 300.00000 275.00000 275.00000 xpress 300.00000 325.00000 275.00000
Note that the different solutions are combined into a single symbol of a higher dimension. The filenames (resp. the solver used) are added as unique elements.
Instead of using the display statement, we can also use the GAMS IDE or GAMS Studio to display the merged.gdx file. The following figure shows the contents of merged.gdx after selecting the level subfield of the variable to be displayed and arranging the display:
This example is also part of the GAMS Data Utilities Library, see model [GDXMERGEExample17] for reference.
Recursively Merging GDX Files
Some users generate data scenarios for a model in a hierarchical way and store the results in a directory tree:
The following example demonstrates how to use three independent scenario scalars a, b, and c with values low, medium, and high to generate 3*3*3=27 scenarios, store these in a directory tree, and merge them recursively. The resulting GDX file has for each scenario scalar a new dimension which makes this data especially suitable for viewing in a pivot table:
---- 39 PARAMETER p commodity price new-york chicago topeka a_high.b_high.c_high.report 0.25 0.17 0.14 a_high.b_high.c_low .report 0.20 0.14 0.11 a_high.b_high.c_med .report 0.23 0.15 0.13 a_high.b_low .c_high.report 0.25 0.17 0.14 a_high.b_low .c_low .report 0.20 0.14 0.11 a_high.b_low .c_med .report 0.23 0.15 0.13 a_high.b_med .c_high.report 0.25 0.17 0.14 a_high.b_med .c_low .report 0.20 0.14 0.11 a_high.b_med .c_med .report 0.23 0.15 0.13 a_low .b_high.c_high.report 0.25 0.17 0.14 a_low .b_high.c_low .report 0.20 0.14 0.11 a_low .b_high.c_med .report 0.23 0.15 0.13 ...
The GAMS program that creates the directory structure and runs the model for all 27 scenarios looks as follows:
set a,b,c;
parameter
ascen(a<) / a_low 0.9, a_med 1.0, a_high 1.1 /
bscen(b<) / b_low 0.9, b_med 1.0, b_high 1.1 /
cscen(c<) / c_low 0.9, c_med 1.0, c_high 1.1 /;
loop((a,b,c),
put_utility 'shell' / 'mkdir ' a.tl:0 '\' b.tl:0 '\' c.tl:0
put_utility 'exec' / 'gams t.gms lo=2 -idir "%gams.wdir%" -curdir ' a.tl:0 '/' b.tl:0 '/' c.tl:0 ' --a ' ascen(a):3:1 ' --b ' bscen(b):3:1 ' --c ' cscen(c):3:1
);
The program creates the directory tree (by creating the leave nodes of the tree), changes into the leave directory (curDir
) and calls GAMS (using iDir
to point to the starting location). The program t.gms
is almost an identical copy of the [trnsport] model with the last few lines changed:
* ...
a(i) = %a%*a(i);
b(j) = %b%*b(j);
c(i,j) = %c%*c(i,j);
solve transport using lp minimizing z;
scalar tmodelstat;
tmodelstat = transport.modelstat;
parameter p(j) 'commodity price';
p(j) = demand.m(j);
execute 'test -d results || mkdir results';
execute_unload 'results/input', a, b, c;
execute_unload 'results/report', x, z, p, tmodelstat;
This code saves the results of a run in to GDX files input.gdx
and report.gdx
in a results
subdirectory. After the program has executed the 27 runs of t.gms
it executes some Python code that walks recursively through the directory tree and merges GDX files it finds in a results
folder in the leaf nodes percolates the results up in the tree by calling gdxmerge
on each high node level. The Python function os.walk
with argument topdown=False
ensure the walking of the tree in the correct order. The complete example can be found in the GAMS Model Library in model [scenmerge]:
embeddedCode Python: import os gd = -1 for root, dirs, files in os.walk(os.path.normpath(r"C:\Users\test\allscen\ ".rstrip()), topdown=False): if 'results' in dirs: if gd == -1: gd = root.count(os.path.sep) gdr = root; elif not gd == root.count(os.path.sep): raise NameError('results subdirs found at different depths in the tree: ' + root + ' and ' + gdr) cmd = 'cd "' + root + '" && gdxmerge results' + os.path.sep + '* output=' + root.split(os.path.sep)[-1] print('Merging results in ' + root) if not 0 == os.system(cmd + '>' + os.devnull): raise NameError('problems running: ' + cmd) elif root.count(os.path.sep) < gd: input = '' for d in dirs: if os.path.isfile(os.path.join(root, d, d + '.gdx')): input = input + ' ' + os.path.join(d, d + '.gdx') if len(input): cmd = 'cd "' + root + '" && gdxmerge' + input + ' output=' + root.split(os.path.sep)[-1] print('Merging' + input + ' in ' + root) print(cmd) if not 0 == os.system(cmd + '>' + os.devnull): raise NameError('problems running: ' + cmd) print('All done. Final result file: ' + root.split(os.path.sep)[-1] + '.gdx') endEmbeddedCode