connectsub.gms : Test substitution for Connect

Description

This test ensures that the substitution mechanism for Connect works properly
for the three uses of Connect: GAMS CLP connectIn/Out, standalone gamsconnect,
and embedded code. In addition it tests that the interaction of connectIn/IDCGDXInput
and connectOut/IDCGDXOutput works as expected.

Contributor: Michael Bussieck, February 2022


Small Model of Type : GAMS


Category : GAMS Test library


Main file : connectsub.gms

$title 'Test substitution for Connect' (CONNECTSUB,SEQ=900)

$onText
This test ensures that the substitution mechanism for Connect works properly
for the three uses of Connect: GAMS CLP connectIn/Out, standalone gamsconnect,
and embedded code. In addition it tests that the interaction of connectIn/IDCGDXInput
and connectOut/IDCGDXOutput works as expected.

Contributor: Michael Bussieck, February 2022
$offText


$log --- Using Python library %sysEnv.GMSPYTHONLIB%

$onEcho > in.csv
date,symbol,price
2016/01/04,AAPL,105.35
2016/01/04,AXP,67.59
2016/01/04,BA,140.50
$offEcho
$onEcho > sp.gms
Set dates, stocks;
$onExternalInput
Parameter stockprice(dates<,stocks<) / /;
$offExternalInput
abort$(card(stockprice)<>3) 'wrong parameter stockprice', stockprice;
$onExternalOutput
Parameter numData; numData = card(stockprice);
$offExternalOutput
$offEcho
$onEchoV > cin.yaml
- CSVReader:
    file: %CSVInFile%
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
- GDXWriter:
    file: %gams.input%.gdx
    symbols:
      - name: stockprice
$offEcho
$onEchoV > cout.yaml
- GDXReader:
    file: %GDXOUT%
    symbols:
      - name: numData
- CSVWriter:
    file: %gams.input%.csv
    name: numData
    header: False
$offEcho

$log Test connectIn/Out connection with IDCGDXInput/Output and substitution at connectIn/Out level
$call.checkErrorLevel gams sp.gms lo=2 connectIn=cin.yaml --GDXOUT=cout.gdx --CSVINFILE=in.csv IDCGDXInput=sp.gms.gdx connectOut=cout.yaml IDCGDXOutput=cout.gdx
$echo 3.0 > out_expected.csv
$call.checkErrorLevel diff -bwi sp.gms.csv out_expected.csv

$log Test substitution at gamsconnect level
$call rm -f sp.gms.csv
$call.checkErrorLevel gamsconnect cin.yaml CSVINFILE=in.csv GAMS.INPUT=sp.gms
$call.checkErrorLevel gams sp.gms lo=2 IDCGDXInput=sp.gms.gdx IDCGDXOutput=cout.gdx
$call.checkErrorLevel gamsconnect cout.yaml GDXOUT=cout.gdx GAMS.INPUT=sp.gms
$call.checkErrorLevel diff -bwi sp.gms.csv out_expected.csv

Set dates, stocks;
Parameter stockprice(dates<,stocks<);
$if not set CSVINPUT $set CSVINPUT in
$onEmbeddedCodeV Connect: --CSVINPUT=%CSVINPUT%.csv
- CSVReader:
    file: %CSVINPUT%
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
- GAMSWriter:
    symbols:
      - name: stockprice
$offEmbeddedCode
abort$(card(stockprice)<>3) 'wrong parameter stockprice', stockprice;
Parameter numData; numData = card(stockprice);
put_utility 'ECArguments' / ' --NUMDATA=numData';
embeddedCodeV Connect: --CSVOUTPUT=sp.gms
- GAMSReader:
    symbols:
      - name: %NUMDATA%
- CSVWriter:
    file: %CSVOUTPUT%.csv
    name: numData
    header: False
endEmbeddedCode  
execute.checkErrorLevel 'diff -bwi sp.gms.csv out_expected.csv';

$log Test EC arguments containing quotes (compile time)
$onEmbeddedCode Connect: argument containing 'single' and "double" quotes
- PythonCode:
    code: |
      args = gams.arguments
      if args != '''argument containing 'single' and "double" quotes''':
        raise Exception("gams.arguments not as expected")
$offEmbeddedCode

$log Test EC arguments containing quotes (run time)
put_utility "ecArguments" / "argument containing 'single' and " '"double" quotes';
embeddedCode Connect:
- PythonCode:
    code: |
      args = gams.arguments
      if args != '''argument containing 'single' and "double" quotes''':
        raise Exception("gams.arguments not as expected")
endEmbeddedCode

$log Test error message preserving "%"
$onEchoV > print_log_percent.gms
$onEmbeddedCode Connect:
- GDXReader:
    file: %file%
    symbols: all
$offEmbeddedCode
$offEcho
$call gams print_log_percent.gms > print_log_percent.log
$onEmbeddedCode Python:
with open("print_log_percent.log", "r")as f:
  s = f.read()
  if s.count("%file%") != 1 or s.count("%") != 3:
    raise Exception('Expected one occurrence of "%file%" and three occurrences of "%" in log output')
$offEmbeddedCode

$log Test parsing Connect code with different Python versions
$onEcho > connect_ast.py
import ast
import os
import gams

connect_dir = os.path.join(os.path.dirname(gams.__file__), "connect")
for dirpath, dirnames, filenames in os.walk(connect_dir):
    for file in filenames:
        if file.endswith(".py"):
            with open(os.path.join(dirpath, file), "r") as f:
                ast.parse(f.read())
$offEcho

$ifThen setenv PYTHON39
$  call.checkErrorLevel cd . && "%sysenv.PYTHON39%" "connect_ast.py"
$endIf

$ifThen setenv PYTHON310
$  call.checkErrorLevel cd . && "%sysenv.PYTHON310%" "connect_ast.py"
$endIf

$ifThen setenv PYTHON311
$  call.checkErrorLevel cd . && "%sysenv.PYTHON311%" "connect_ast.py"
$endIf

$ifThen setenv PYTHON312
$  call.checkErrorLevel cd . && "%sysenv.PYTHON312%" "connect_ast.py"
$endIf

$ifThen setenv PYTHON313
$  call.checkErrorLevel cd . && "%sysenv.PYTHON313%" "connect_ast.py"
$endIf


$log Test ConnectDatabase.execute with both list and dict (list iteration)
$onEmbeddedCode Python:
from contextlib import redirect_stdout
import yaml
from gams.connect import ConnectDatabase

cdb = ConnectDatabase(system_directory=r"%gams.sysdir% ".strip())
inst_str = """
- PythonCode:
    code: |
      print("agent 1")
      instructions.append({"PythonCode": {"code": "print('agent 2')"}})
      instructions.append({"PythonCode": {"code": "print('agent 3')"}})
- PythonCode:
    code: |
      print("agent 4")
"""

instructions = yaml.safe_load(inst_str)

with open("execute_list.txt", "w") as f:
    with redirect_stdout(f):
        cdb.execute(instructions)

with open("execute_dict.txt", "w") as f:
    with redirect_stdout(f):
        for inst in instructions:
            cdb.execute(inst)
$offEmbeddedCode
$call.checkErrorLevel diff execute_list.txt execute_dict.txt


$log Test instruction trace output for some Connect agents
$onEchoV > test_trace.yaml
- CSVReader:
    trace: 1
    file: in.csv
    name: stockprice
    indexColumns: [1, 2]
    valueColumns: [3]
- ExcelWriter:
    trace: 1
    file: test_trace.xlsx
    columnDimension: 5
    tableOfContents: True
    valueSubstitutions: {1: 2, 3: 4}
    clearSheet: True
    symbols:
      - name: stockprice
        range: sp!A1
        columnDimension: 1
        valueSubstitutions: {1: 4}
        clearSheet: null
- PythonCode:
    trace: 3
    code: |
      instructions.append(
      {
        'ExcelWriter':
        {
          'trace': 1,
          'file': 'test_trace.xlsx',
          'columnDimension': 5,
          'tableOfContents': False,
          'valueSubstitutions': {1: 2, 3: 4},
          'clearSheet': True,
          'symbols': [{'name': 'stockprice', 'columnDimension': 1}]
        }
      })
$offEcho
$call.checkErrorLevel gamsconnect test_trace.yaml > test_trace_log.txt

$onEmbeddedCode Python:
with open("test_trace_log.txt", "r") as f:
    lines = f.readlines()
    new_lines = []
    for idx, l in enumerate(lines.copy()):
        if l.startswith("file:"):
            new_lines.append("####\n")
        else:
            new_lines.append(l)
with open("test_trace_log.txt", "w") as f:
    f.writelines(new_lines)
$offEmbeddedCode

$onEcho > test_trace_ref.txt

------------------------------------------------------------------------------------------
CSVReader:
Option                        Input                         Processed Input
------------------------------------------------------------------------------------------
trace:                        >1<                           >1<
####
name:                         >stockprice<                  >stockprice<
indexColumns:                 
  (1)                         >1<                           >1<
  (2)                         >2<                           >2<
valueColumns:                 
  (1)                         >3<                           >3<
type:                         ><                            >par<
indexSubstitutions:           ><                            >None<
valueSubstitutions:           ><                            >None<
autoColumn:                   ><                            >None<
autoRow:                      ><                            >None<
decimalSeparator:             ><                            >.<
fieldSeparator:               ><                            >,<
thousandsSeparator:           ><                            >None<
header:                       ><                            >infer<
names:                        ><                            >None<
quoting:                      ><                            >0<
readCSVArguments:             ><                            >None<
skipRows:                     ><                            >None<
stack:                        ><                            >infer<
------------------------------------------------------------------------------------------

Connect Container (before):

Connect Container (after):

Parameters:
         name          domain domain_type  dimension  number_records    min  \
0  stockprice  [date, symbol]     relaxed          2               3  67.59   

     mean    max          where_min         where_max  sparsity  
0  104.48  140.5  [2016/01/04, AXP]  [2016/01/04, BA]       NaN  


------------------------------------------------------------------------------------------
ExcelWriter:
Option                        Input                         Processed Input
------------------------------------------------------------------------------------------
trace:                        >1<                           >1<
####
columnDimension:              >5<                           >5<
tableOfContents:              >True<
  sheetName:                                                >Table Of Contents<
  sort:                                                     >False<
  emptySymbols:                                             >False<
valueSubstitutions:           
                              >1<: >2<                      >1<: >2<
                              >3<: >4<                      >3<: >4<
clearSheet:                   >True<                        >True<
symbols:                      
  (1)                         
    name:                     >stockprice<                  >stockprice<
    range:                    >sp!A1<                       >sp!A1<
    columnDimension:          >1<                           >1<
    valueSubstitutions:       
                              >1<: >4<                      >1<: >4<
    clearSheet:               >None<                        >None<
    mergedCells:              ><                            >None<
mergedCells:                  ><                            >False<
index:                        ><                            >None<
------------------------------------------------------------------------------------------

Connect Container:

Parameters:
         name          domain domain_type  dimension  number_records    min  \
0  stockprice  [date, symbol]     relaxed          2               3  67.59   

     mean    max          where_min         where_max  sparsity  
0  104.48  140.5  [2016/01/04, AXP]  [2016/01/04, BA]       NaN  


------------------------------------------------------------------------------------------
Write symbol >stockprice<:
Option                        Input                         Processed Input
------------------------------------------------------------------------------------------
name:                         >stockprice<                  >stockprice<
range:                        >sp!A1<                       >sp!A1<
columnDimension:              >1<                           >1<
valueSubstitutions:           
                              >1<: >4<                      >1<: >4<
clearSheet:                   >None<                        >True<
mergedCells:                  >None<                        >False<
------------------------------------------------------------------------------------------


------------------------------------------------------------
PythonCode:
Option                        Processed Input
------------------------------------------------------------
trace:                        >3<
code:                         >instructions.append(
{
  'ExcelWriter':
  {
    'trace': 1,
    'file': 'test_trace.xlsx',
    'columnDimension': 5,
    'tableOfContents': False,
    'valueSubstitutions': {1: 2, 3: 4},
    'clearSheet': True,
    'symbols': [{'name': 'stockprice', 'columnDimension': 1}]
  }
})
<
------------------------------------------------------------

Connect Container (before):

Parameters:
         name          domain domain_type  dimension  number_records    min  \
0  stockprice  [date, symbol]     relaxed          2               3  67.59   

     mean    max          where_min         where_max  sparsity  
0  104.48  140.5  [2016/01/04, AXP]  [2016/01/04, BA]       NaN  

Number of generated instructions: 1
List of generated instructions:

------------------------------------------------------------
(1) ExcelWriter:
Option                        Processed Input
------------------------------------------------------------
trace:                        >1<
####
columnDimension:              >5<
tableOfContents:              >False<
valueSubstitutions:           
                              >1<: >2<
                              >3<: >4<
clearSheet:                   >True<
symbols:                      
  (1)                         
    name:                     >stockprice<
    columnDimension:          >1<
------------------------------------------------------------


Connect Container (after):

Parameters:
         name          domain domain_type  dimension  number_records    min  \
0  stockprice  [date, symbol]     relaxed          2               3  67.59   

     mean    max          where_min         where_max  sparsity  
0  104.48  140.5  [2016/01/04, AXP]  [2016/01/04, BA]       NaN  


------------------------------------------------------------------------------------------
ExcelWriter:
Option                        Input                         Processed Input
------------------------------------------------------------------------------------------
trace:                        >1<                           >1<
####
columnDimension:              >5<                           >5<
tableOfContents:              >False<                       >False<
valueSubstitutions:           
                              >1<: >2<                      >1<: >2<
                              >3<: >4<                      >3<: >4<
clearSheet:                   >True<                        >True<
symbols:                      
  (1)                         
    name:                     >stockprice<                  >stockprice<
    columnDimension:          >1<                           >1<
    range:                    ><                            >None<
    clearSheet:               ><                            >None<
    mergedCells:              ><                            >None<
    valueSubstitutions:       ><                            >None<
mergedCells:                  ><                            >False<
index:                        ><                            >None<
------------------------------------------------------------------------------------------

Connect Container:

Parameters:
         name          domain domain_type  dimension  number_records    min  \
0  stockprice  [date, symbol]     relaxed          2               3  67.59   

     mean    max          where_min         where_max  sparsity  
0  104.48  140.5  [2016/01/04, AXP]  [2016/01/04, BA]       NaN  


------------------------------------------------------------------------------------------
Write symbol >stockprice<:
Option                        Input                         Processed Input
------------------------------------------------------------------------------------------
name:                         >stockprice<                  >stockprice<
columnDimension:              >1<                           >1<
range:                        >None<                        >stockprice!A1<
clearSheet:                   >None<                        >True<
mergedCells:                  >None<                        >False<
valueSubstitutions:           >None<
                                                            >1<: >2<
                                                            >3<: >4<
------------------------------------------------------------------------------------------

$offEcho

$call.checkErrorLevel diff test_trace_log.txt test_trace_ref.txt