Description
This test ensures that the substitution mechanism for Connect works properly for the three uses of Connect: GAMS CLP connectIn/Out, standalone gamsconnect, and embedded code. In addition it tests that the interaction of connectIn/IDCGDXInput and connectOut/IDCGDXOutput works as expected. Contributor: Michael Bussieck, February 2022
Small Model of Type : GAMS
Category : GAMS Test library
Main file : connectsub.gms
$title 'Test substitution for Connect' (CONNECTSUB,SEQ=900)
$onText
This test ensures that the substitution mechanism for Connect works properly
for the three uses of Connect: GAMS CLP connectIn/Out, standalone gamsconnect,
and embedded code. In addition it tests that the interaction of connectIn/IDCGDXInput
and connectOut/IDCGDXOutput works as expected.
Contributor: Michael Bussieck, February 2022
$offText
$log --- Using Python library %sysEnv.GMSPYTHONLIB%
$onEcho > in.csv
date,symbol,price
2016/01/04,AAPL,105.35
2016/01/04,AXP,67.59
2016/01/04,BA,140.50
$offEcho
$onEcho > sp.gms
Set dates, stocks;
$onExternalInput
Parameter stockprice(dates<,stocks<) / /;
$offExternalInput
abort$(card(stockprice)<>3) 'wrong parameter stockprice', stockprice;
$onExternalOutput
Parameter numData; numData = card(stockprice);
$offExternalOutput
$offEcho
$onEchoV > cin.yaml
- CSVReader:
file: %CSVInFile%
name: stockprice
indexColumns: [1, 2]
valueColumns: [3]
- GDXWriter:
file: %gams.input%.gdx
symbols:
- name: stockprice
$offEcho
$onEchoV > cout.yaml
- GDXReader:
file: %GDXOUT%
symbols:
- name: numData
- CSVWriter:
file: %gams.input%.csv
name: numData
header: False
$offEcho
$log Test connectIn/Out connection with IDCGDXInput/Output and substitution at connectIn/Out level
$call.checkErrorLevel gams sp.gms lo=2 connectIn=cin.yaml --GDXOUT=cout.gdx --CSVINFILE=in.csv IDCGDXInput=sp.gms.gdx connectOut=cout.yaml IDCGDXOutput=cout.gdx
$echo 3.0 > out_expected.csv
$call.checkErrorLevel diff -bwi sp.gms.csv out_expected.csv
$log Test substitution at gamsconnect level
$call rm -f sp.gms.csv
$call.checkErrorLevel gamsconnect cin.yaml CSVINFILE=in.csv GAMS.INPUT=sp.gms
$call.checkErrorLevel gams sp.gms lo=2 IDCGDXInput=sp.gms.gdx IDCGDXOutput=cout.gdx
$call.checkErrorLevel gamsconnect cout.yaml GDXOUT=cout.gdx GAMS.INPUT=sp.gms
$call.checkErrorLevel diff -bwi sp.gms.csv out_expected.csv
Set dates, stocks;
Parameter stockprice(dates<,stocks<);
$if not set CSVINPUT $set CSVINPUT in
$onEmbeddedCodeV Connect: --CSVINPUT=%CSVINPUT%.csv
- CSVReader:
file: %CSVINPUT%
name: stockprice
indexColumns: [1, 2]
valueColumns: [3]
- GAMSWriter:
symbols:
- name: stockprice
$offEmbeddedCode
abort$(card(stockprice)<>3) 'wrong parameter stockprice', stockprice;
Parameter numData; numData = card(stockprice);
put_utility 'ECArguments' / ' --NUMDATA=numData';
embeddedCodeV Connect: --CSVOUTPUT=sp.gms
- GAMSReader:
symbols:
- name: %NUMDATA%
- CSVWriter:
file: %CSVOUTPUT%.csv
name: numData
header: False
endEmbeddedCode
execute.checkErrorLevel 'diff -bwi sp.gms.csv out_expected.csv';
$log Test EC arguments containing quotes (compile time)
$onEmbeddedCode Connect: argument containing 'single' and "double" quotes
- PythonCode:
code: |
args = gams.arguments
if args != '''argument containing 'single' and "double" quotes''':
raise Exception("gams.arguments not as expected")
$offEmbeddedCode
$log Test EC arguments containing quotes (run time)
put_utility "ecArguments" / "argument containing 'single' and " '"double" quotes';
embeddedCode Connect:
- PythonCode:
code: |
args = gams.arguments
if args != '''argument containing 'single' and "double" quotes''':
raise Exception("gams.arguments not as expected")
endEmbeddedCode
$log Test error message preserving "%"
$onEchoV > print_log_percent.gms
$onEmbeddedCode Connect:
- GDXReader:
file: %file%
symbols: all
$offEmbeddedCode
$offEcho
$call gams print_log_percent.gms > print_log_percent.log
$onEmbeddedCode Python:
with open("print_log_percent.log", "r")as f:
s = f.read()
if s.count("%file%") != 1 or s.count("%") != 3:
raise Exception('Expected one occurrence of "%file%" and three occurrences of "%" in log output')
$offEmbeddedCode
$log Test parsing Connect code with different Python versions
$onEcho > connect_ast.py
import ast
import os
import gams
connect_dir = os.path.join(os.path.dirname(gams.__file__), "connect")
for dirpath, dirnames, filenames in os.walk(connect_dir):
for file in filenames:
if file.endswith(".py"):
with open(os.path.join(dirpath, file), "r") as f:
ast.parse(f.read())
$offEcho
$ifThen setenv PYTHON39
$ call.checkErrorLevel cd . && "%sysenv.PYTHON39%" "connect_ast.py"
$endIf
$ifThen setenv PYTHON310
$ call.checkErrorLevel cd . && "%sysenv.PYTHON310%" "connect_ast.py"
$endIf
$ifThen setenv PYTHON311
$ call.checkErrorLevel cd . && "%sysenv.PYTHON311%" "connect_ast.py"
$endIf
$ifThen setenv PYTHON312
$ call.checkErrorLevel cd . && "%sysenv.PYTHON312%" "connect_ast.py"
$endIf
$ifThen setenv PYTHON313
$ call.checkErrorLevel cd . && "%sysenv.PYTHON313%" "connect_ast.py"
$endIf
$log Test ConnectDatabase.execute with both list and dict (list iteration)
$onEmbeddedCode Python:
from contextlib import redirect_stdout
import yaml
from gams.connect import ConnectDatabase
cdb = ConnectDatabase(system_directory=r"%gams.sysdir% ".strip())
inst_str = """
- PythonCode:
code: |
print("agent 1")
instructions.append({"PythonCode": {"code": "print('agent 2')"}})
instructions.append({"PythonCode": {"code": "print('agent 3')"}})
- PythonCode:
code: |
print("agent 4")
"""
instructions = yaml.safe_load(inst_str)
with open("execute_list.txt", "w") as f:
with redirect_stdout(f):
cdb.execute(instructions)
with open("execute_dict.txt", "w") as f:
with redirect_stdout(f):
for inst in instructions:
cdb.execute(inst)
$offEmbeddedCode
$call.checkErrorLevel diff execute_list.txt execute_dict.txt
$log Test instruction trace output for some Connect agents
$onEchoV > test_trace.yaml
- CSVReader:
trace: 1
file: in.csv
name: stockprice
indexColumns: [1, 2]
valueColumns: [3]
- ExcelWriter:
trace: 1
file: test_trace.xlsx
columnDimension: 5
tableOfContents: True
valueSubstitutions: {1: 2, 3: 4}
clearSheet: True
symbols:
- name: stockprice
range: sp!A1
columnDimension: 1
valueSubstitutions: {1: 4}
clearSheet: null
- PythonCode:
trace: 3
code: |
instructions.append(
{
'ExcelWriter':
{
'trace': 1,
'file': 'test_trace.xlsx',
'columnDimension': 5,
'tableOfContents': False,
'valueSubstitutions': {1: 2, 3: 4},
'clearSheet': True,
'symbols': [{'name': 'stockprice', 'columnDimension': 1}]
}
})
$offEcho
$call.checkErrorLevel gamsconnect test_trace.yaml > test_trace_log.txt
$onEmbeddedCode Python:
with open("test_trace_log.txt", "r") as f:
lines = f.readlines()
new_lines = []
for idx, l in enumerate(lines.copy()):
if l.startswith("file:"):
new_lines.append("####\n")
else:
new_lines.append(l)
with open("test_trace_log.txt", "w") as f:
f.writelines(new_lines)
$offEmbeddedCode
$onEcho > test_trace_ref.txt
------------------------------------------------------------------------------------------
CSVReader:
Option Input Processed Input
------------------------------------------------------------------------------------------
trace: >1< >1<
####
name: >stockprice< >stockprice<
indexColumns:
(1) >1< >1<
(2) >2< >2<
valueColumns:
(1) >3< >3<
type: >< >par<
indexSubstitutions: >< >None<
valueSubstitutions: >< >None<
autoColumn: >< >None<
autoRow: >< >None<
decimalSeparator: >< >.<
fieldSeparator: >< >,<
thousandsSeparator: >< >None<
header: >< >infer<
names: >< >None<
quoting: >< >0<
readCSVArguments: >< >None<
skipRows: >< >None<
stack: >< >infer<
------------------------------------------------------------------------------------------
Connect Container (before):
Connect Container (after):
Parameters:
name domain domain_type dimension number_records min \
0 stockprice [date, symbol] relaxed 2 3 67.59
mean max where_min where_max sparsity
0 104.48 140.5 [2016/01/04, AXP] [2016/01/04, BA] NaN
------------------------------------------------------------------------------------------
ExcelWriter:
Option Input Processed Input
------------------------------------------------------------------------------------------
trace: >1< >1<
####
columnDimension: >5< >5<
tableOfContents: >True<
sheetName: >Table Of Contents<
sort: >False<
emptySymbols: >False<
valueSubstitutions:
>1:2< >1:2<
>3:4< >3:4<
clearSheet: >True< >True<
symbols:
(1)
name: >stockprice< >stockprice<
range: >sp!A1< >sp!A1<
columnDimension: >1< >1<
valueSubstitutions:
>1:4< >1:4<
clearSheet: >None< >None<
mergedCells: >< >None<
mergedCells: >< >False<
index: >< >None<
------------------------------------------------------------------------------------------
Connect Container:
Parameters:
name domain domain_type dimension number_records min \
0 stockprice [date, symbol] relaxed 2 3 67.59
mean max where_min where_max sparsity
0 104.48 140.5 [2016/01/04, AXP] [2016/01/04, BA] NaN
------------------------------------------------------------------------------------------
Write symbol >stockprice<:
Option Input Processed Input
------------------------------------------------------------------------------------------
name: >stockprice< >stockprice<
range: >sp!A1< >sp!A1<
columnDimension: >1< >1<
valueSubstitutions:
>1:4< >1:4<
clearSheet: >None< >True<
mergedCells: >None< >False<
------------------------------------------------------------------------------------------
------------------------------------------------------------
PythonCode:
Option Processed Input
------------------------------------------------------------
trace: >3<
code: >instructions.append(
{
'ExcelWriter':
{
'trace': 1,
'file': 'test_trace.xlsx',
'columnDimension': 5,
'tableOfContents': False,
'valueSubstitutions': {1: 2, 3: 4},
'clearSheet': True,
'symbols': [{'name': 'stockprice', 'columnDimension': 1}]
}
})
<
------------------------------------------------------------
Connect Container (before):
Parameters:
name domain domain_type dimension number_records min \
0 stockprice [date, symbol] relaxed 2 3 67.59
mean max where_min where_max sparsity
0 104.48 140.5 [2016/01/04, AXP] [2016/01/04, BA] NaN
Number of generated instructions: 1
List of generated instructions:
------------------------------------------------------------
(1) ExcelWriter:
Option Processed Input
------------------------------------------------------------
trace: >1<
####
columnDimension: >5<
tableOfContents: >False<
valueSubstitutions:
>1:2<
>3:4<
clearSheet: >True<
symbols:
(1)
name: >stockprice<
columnDimension: >1<
------------------------------------------------------------
Connect Container (after):
Parameters:
name domain domain_type dimension number_records min \
0 stockprice [date, symbol] relaxed 2 3 67.59
mean max where_min where_max sparsity
0 104.48 140.5 [2016/01/04, AXP] [2016/01/04, BA] NaN
------------------------------------------------------------------------------------------
ExcelWriter:
Option Input Processed Input
------------------------------------------------------------------------------------------
trace: >1< >1<
####
columnDimension: >5< >5<
tableOfContents: >False< >False<
valueSubstitutions:
>1:2< >1:2<
>3:4< >3:4<
clearSheet: >True< >True<
symbols:
(1)
name: >stockprice< >stockprice<
columnDimension: >1< >1<
range: >< >None<
clearSheet: >< >None<
mergedCells: >< >None<
valueSubstitutions: >< >None<
mergedCells: >< >False<
index: >< >None<
------------------------------------------------------------------------------------------
Connect Container:
Parameters:
name domain domain_type dimension number_records min \
0 stockprice [date, symbol] relaxed 2 3 67.59
mean max where_min where_max sparsity
0 104.48 140.5 [2016/01/04, AXP] [2016/01/04, BA] NaN
------------------------------------------------------------------------------------------
Write symbol >stockprice<:
Option Input Processed Input
------------------------------------------------------------------------------------------
name: >stockprice< >stockprice<
columnDimension: >1< >1<
range: >None< >stockprice!A1<
clearSheet: >None< >True<
mergedCells: >None< >False<
valueSubstitutions: >None<
>1:2<
>3:4<
------------------------------------------------------------------------------------------
$offEcho
$call.checkErrorLevel diff test_trace_log.txt test_trace_ref.txt