GAMS provides a number of built-in or intrinsic functions for use in equations. Still, the extremely diverse set of application areas in which GAMS is used can create demand for the addition of new and often sophisticated and specialized functions. There is a trade-off between satisfying these requests and avoiding complexity not needed by most users. The GAMS External Equations Facility provides one means for managing this trade-off, since it allows users to import functions from an external library to define equations in a GAMS model. However, these external libraries can currently only provide functionality for the evaluation of functions (incl. their first derivatives) in a point. Solvers that need to analyze the algebraic structure of the model instance are therefore not able to work with external equations. This includes the class of deterministic global solvers, see column "Global" in this table, while, for example, stochastic global solvers can work with external equations.

Note

Both external equations and extrinsic functions aim to provide possibilities to extend GAMS by user-provided mathematical functions. However, there are fundamental differences in the use and implementation of both. For most situations, extrinsic functions should be preferred over external equations. See also Extrinsic Functions vs. External Equations.

Attention

Building external equation libraries requires the knowledge of a regular programming language (like C/C++, FORTRAN, ...) and experience with handling compilers and linkers to build dynamically linked libraries.

Note

The external equation interface is not intended as a way to bypass some of the very useful model checking done by GAMS for models that are solved with NLP solvers. External equations are still assumed to be continuous with accurate and smooth first derivatives. The continuity assumption implies that external equations must have very low noise levels, considerably below the feasibility tolerance used by the solver. The assumption about accurate derivatives implies that derivatives must be computed more accurately than can be done with standard finite differences. If these assumptions are not satisfied, then there is no guarantee that the NLP solver can find a solution that has the mathematical properties of a local optimum, i.e., a solution that satisfies the Karush-Kuhn-Tucker conditions within the standard tolerances used by the solver.

In the following, connecting code written in FORTRAN, C, Delphi, or some other programming language to equations and variables in a GAMS model is described. These GAMS equations will be referred to as external equations and the compiled version of the programming routines will be referred to as the external module that defines the external functions. The form of the external module depends on the operating system that is used. The external module under Windows is a Dynamic Link Library (.dll) and the external module under Unix is a shared object (.so or .dylib). In principle, any language or system may be used to build the DLL or shared object that defines the external module, as long as the interface conventions are not changed.

The GAMS Test Library provides examples of external equations consisting of GAMS models and C, Delphi, Java, and FORTRAN code. For more details, see Section Examples in the GAMS Test Library.

The basic mechanism of external equations is to declare all the equations and variables using the usual GAMS syntax. The interpretation of the external equations is done in a special way. Instead of the usual semantic content, the external equations specify the mapping between the equation and variable names used in GAMS and the function and variable indices used in the external module. This mapping is described in Section Model Interface. The external module may be written in C, FORTRAN, or most other programming languages. Section Programming Interface describes the general definitions for an external module for C, Delphi, and FORTRAN from a programming language perspective. Note that the way the program is compiled and converted into an external module is system and compiler specific. The following Section Implementation gives detailed advice on various aspects of the implementation of the external module.

Examples in the GAMS Test Library

Model [TESTEXEQ] gives an overview of all examples in the GAMS Test Library and may be used to compile and run them. Note that the remainder of this chapter will reference examples that are listed in this model. Further, model [COMPLINK] may be used as a script to compile and link external equation libraries. Note that these models hardcode the path to the Java compiler and libraries and these paths will need to be adapted by the user when running the Java examples.

Observe that regardless of how external libraries are built, the examples (e.g. [EX1]) will by default solve a model without using external equations. To solve the example models with all kinds of different external equation libraries, they may be run with the argument

--runall=1

Alternatively, only selected libraries may be used by using one or more of the following command line parameters:

--runC=1
--runC_cb=1
--runD=1
--runD_cb=1
--runF=1
--runF_cb=1
--runJ=1

Model Interface

External Equation Syntax

External equations that are used to specify the interface to the external module are declared in GAMS like any other equation. The syntax for the external equation definition statement is as follows:

eqn_name(index_list)[$logical_condition(s)].. expression =x= expression ;

Note that the only difference to the usual equation definition is the use of the equation type =x=.

The equations defined by an external module are always interpreted as equality constraints with zero right-hand sides. Thus inequalities have to be converted to equalities by adding explicit slack variables, which will serve as additional external variables. A nonzero right-hand side need to be taken care of in the external equation implementation.

Mapping of external equations and variables to indices

Some mappings must be specified to link an external module to a GAMS model. External equations are assumed to be defined in terms of indices i = 1...m. These indices must be mapped to GAMS equation names. Similarly, the variables used inside the external functions are assumed to be defined in terms of indices j = 1...n. These indices must be mapped to GAMS variable names. Finally, the name of the external module must be specified. Note that GAMS solvers are typically designed for large models and rely on sparsity. The last part of the specification of a set of external equations is therefore the sparsity pattern of the external equations, i.e., which variables appear in which equations.

The value of the constant term of the external equation must be an integer, since the value of the constant maps the row of the GAMS equation to the index (in 1...m) of the external equation. Several blocks of GAMS equations may be mapped to external equations using the =x= notation. The mapping between GAMS equations of type =x= and indices 1...m must be bijective (one-to-one). This means that two GAMS equations may not be mapped into the same external equation index and that there may not be any holes in the list of external equation indices. Although there may be any number of blocks of GAMS external equations, they must all map into and be implemented by one single external module.

The variable part of each external equation defines both the sparsity pattern of the external equation and the mapping from GAMS variables to the indices of the external variables. The variable part must be a sum of terms where each term is an integer times a variable. The existence of the term indicates that the variable involved is used in the external equation and that there is a corresponding derivative. The value of the coefficient defines the index of the external variable (in 1...n) that the GAMS variable is mapped to. For example, the term 5*Y indicates that the external equation depends on the GAMS variable Y and that Y is mapped to the 5th element in the vector of external variables. Clearly, if a variable appears in more than one external equation, then the value of its coefficient must be the same in each case.

Note that several blocks of GAMS variables may be used in external equations. In contrast to equations, where all rows in an equation block are either external or not, some columns in a variable block may be external while others are not. The mapping between GAMS variables that appear in equations of type =x= and external variable indices 1...n must be bijective (one-to-one). This means that two GAMS columns may not be mapped into the same external variable index and that there may not be any holes in the list of external variable indices. Although there may be any number of blocks of GAMS variables mapped to external variables, they must all map into one single vector passed to the subroutine in the external module.

Observe that while some GAMS variables are external, there is no syntax provided to mark them as external variables. They may be used in non-external GAMS equations as well as external equations. Indeed, without this capability the model would be separable and the external equations and the functions they map to would be of little use.

Note

• As the coefficients and right-hand sides in the GAMS definition of external equations are interpreted as indices, users are not allowed to scale external equations and variables.
• External equations are treated in a special way, therefore the command line parameter and model attribute HoldFixed will not treat any fixed external variables as constants.

Name of external module

The name of the external module in which the external equations are implemented may be defined in a number of ways. By default, the external module is assumed to have the same name as the GAMS model with an extension that is operating system dependent. The extension is .dll for Windows, .dylib for macOS, and .so for any other Unix.

A custom name for the external module may be specified with a file statement. In this case the file name has to be listed as an additional item in the model statement. If the library extension is omitted in the file statement, GAMS will add the system-dependent extension automatically. This helps to make the model portable between different operating systems.

Consider the following simple example:

File  myextfile / extern /;
Model mymodel   / all, myextfile /;

When model mymodel is solved, GAMS will try to load the an external module file named extern.so, extern.dylib, or extern.dll, depending on the current operating system.

By default, the external module is assumed to be located in the directory external_equations in the GAMS standard locations or in the directory from which GAMS is called. A different location may be specified with an added path in the file statement.

Programming Interface

This section discusses C, Delphi, and FORTRAN interfaces to the GAMS external equations facility.

The external equation module need to provide a function called GEFUNC. The beginning of the external equation module typically looks as follows:

C:

#define GE_EXPORT

#include "geheader.h"

GE_API int GE_CALLCONV

gefunc(int* icntr, double* x, double* f, double* d, msgcb_t msgcb)

The header file geheader.h can be found in the testlib_ml subdirectory of the GAMS distribution. It defines GE_API and GE_CALLCONV and the signature of the function gefunc. GE_API is used to indicate to the compiler whether the function should be exported or imported. Due to defining GE_EXPORT before including geheader.h, GE_API is defined such that the function will be marked for export (__declspec(dllexport) on Windows and __visibility__("default") with GCC). Further, GE_CALLCONV indicates the calling convention that should be used on Windows. Currently, this is defined to be __stdcall. On other operating systems, it is empty.

FORTRAN:

Integer Function gefunc (icntr, x, f, d, msgcb)

C Control Buffer:

Integer icntr(*)

C Numerical Input and Output:

Double Precision x(*), f, d(*)

C Message Callback Routine

External msgcb

Delphi:

uses

geheader_d;

Function GeFunc(var Icntr: ticntr;

var x: tarray;

var F: double;

var D: tarray;

MsgFunc: tMsgCallBack): integer; stdcall;

The unit file geheader_d.pas can be found in the testlib_ml subdirectory of the GAMS distribution.

In the following, the arguments of GEFUNC are described in detail.

Control vector icntr

The array icntr is a control vector that is used to pack and communicate control information between GAMS and the external module. Some helpful definitions to work with the icntr array are provided by the files geheader.h (C), geheader_d.pas (Delphi), and gehelper.f90 (Fortran 90). The array elements are the following:

Observe that FORTRAN programmers will have to replace the square brackets [] with parentheses ().

Evaluation point x

Argument x is an array with icntr[I_Nvar] elements and is provided by GAMS if GEFUNC is called in function evaluation mode (icntr[I_Mode] = DOEVAL). Typically, GAMS and the solvers ensure that the individual elements of x are in between, or very close to, the variable bounds defined in the GAMS model. During initialization and termination calls, x is not defined and the external module must not reference x. C programmers should index this array starting at zero, i.e., the first external variable is referenced as x[0].

Function value f

If icntr[I_Mode] = DOEVAL and icntr[I_Dofunc] = 1, then the external module must return the value of the external equation icntr[I_Eqno] in the scalar f. During initialization and termination calls, f must not be referenced.

Derivative vector d

If icntr[I_Mode] = DOEVAL and icntr[I_Dodrv] = 1, then the external module must return the values of the derivatives of external function icntr[I_Eqno] with respect to all variables in the array d. The derivative with respect to variable x[i] is returned in d[i]. It is sufficient to set only those positions in d that correspond to variables actually appearing in equation icntr[I_Eqno]. Other positions are not being used by GAMS and may be left undefined. During initialization and termination calls, d must not be referenced.

Message callback msgcb

This argument is the address of a message callback routine that can be used to write messages to the status and/or log files of the GAMS process. Its type definition in C is as follows:

The argument mode is used to point to an integer which indicates where messages should be written to. This integer can be set to the following values:

• LOGFILE (1): Write the message to the log file only.
• STAFILE (2): Write the message to the status file only.
• LOGFILE | STAFILE (3): Write the message to both the log file and the status file. Observe that the symbol | denotes the bitwise logical OR in C.

The argument nchars points to an integer that specifies the number of bytes contained in the message (excluding the \0-terminator if there is one present). Thus, in C, nchars is typically set to strlen(buf). The argument buf is a pointer to the character array containing the message to be printed. Finally, len is the size or length of the string buf, thus it is typically the same as *nchars.

Calling the message callback msgcb from C is straightforward. Note that the arguments mode, nchars, and buf are all call-by-reference and that addresses, not values, must be used. However, the argument len is call-by-value and *nchars should be passed as its value.

If the implementation is done in Delphi or Visual Basic, observe that pointers of all types are 4-byte quantities on a 32bit system and 8-byte quantities on a 64bit system. Integers are 4 bytes.

Calling this routine from a FORTRAN environment is a bit more complicated due to the different ways that FORTRAN compilers handle strings. The Unix convention - at least the convention observed on all systems for which GAMS is built - is that strings are passed by reference. In addition, the length of the string is passed by value as a hidden 4-byte quantity appended to the end of the argument list. This is the reason for including len as the last argument in msgcb. The argument len facilitates making FORTRAN callbacks in a Unix environment like the following:

Return code

The function GEFUNC must return one of the following status codes:

Implementation

After describing the function GEFUNC in Section Programming Interface above, this section offers some practical comments on implementing GEFUNC.

Compiling and Linking

The examples for GAMS external equations contain a set of GAMS models for compiling the code on various systems using various compilers. Note that the compiler and linker flags shown in these examples should be used to ensure that the modules conform to the interface standard. In addition, the appropriate include file (geheader.h, geheader_d.pas, gehelper.f90) should be used.

Initialization Mode

The initialization mode should always check whether the external equations have the expected size: icntr[I_Neq], icntr[I_Nvar] and icntr[I_Nz] have to be tested against fixed expected values or values derived from some external data set.

The initialization mode may be used for several purposes like allocating memory and initializing numerical information or mapping information needed by the function evaluations that will follow. Data can be computed or read from external data sources or it can be derived from calls to an external database. Note that data that is shared with GAMS may be written to a file from GAMS using the put statement and then read in GEFUNC. Note further, that users must close the put file with a putclose statement before the solve statement. Observe that memory used to hold information from one invocation of GEFUNC to the next should be static. For FORTRAN it should either be in a Common block or it should be included in a Save statement.

Termination Mode

The termination mode may be used to perform some clean-up tasks like computing statistics, closing files, and returning memory.

Evaluation Mode

The bulk of the computational work will usually be in evaluation mode. Observe that GEFUNC only works with one equation at a time. One of the reasons for this choice is that the addressing of derivatives becomes very simple: there is one derivative for each variable and they have the same index in d and x, respectively.

In some applications several functions are naturally evaluated together, for example, because all functions are computed in some joint integration routine. The icntr[I_Newpt] flag is included for these applications. Using this flag, an implementation could evaluate all functions using a common routine when icntr[I_Newpt] equals 1 and saves the function and derivative values. Additionally, it returns the values corresponding to equation icntr[I_Eqno]. In subsequent calls to GEFUNC, icntr[I_Newpt] will likely be zero and the function and derivative values can quickly be extracted from the previously computed (and saved) information.

Evaluation Errors

It is good modeling practice to add bounds to the variables in such a way that all nonlinear functions are defined for all values of the variables within the bounds. Most solvers will also guarantee that nonlinear functions are called only when all entries of the vector x are between the bounds. However, it may not be practical to add all the necessary bounds and the implementation of GEFUNC should therefore capture evaluation errors such as division by zero, taking the logarithm of non-positive numbers, overflow in exponentiation, etc. If an equation cannot be evaluated at the given point, function GEFUNC should simply let the solver know about this situation by returning the value 1. The solver may then be able to backtrack to a safe point and continue the optimization from there.

Attention

System-default or user-defined functions that handle evaluation errors (for example, the C library function matherr()) will sometimes not work in the same way inside a DLL or a shared object as they do in a self-contained program or a static library.

Message Output

External modules can send messages to the GAMS status file (usually the listing file) and the GAMS log file (usually the screen). Messages to be included in the GAMS status file can be buffered using the GEstat utility routine described below and messages to be included in the GAMS log file can be buffered using the GElog utility routine. Note that it is not possible to open these files for writing in the external module since GAMS or the solver process controls them.

Moreover, messages may be sent to both the status and log file without buffering, using the message callback msgcb. This removes the limit imposed by the size of the message buffer and may also make debugging somewhat simpler, since there is no need to worry about messages that never got flushed from the buffer. As it may be difficult or impossible to use the message callback from some environments, both the buffered and unbuffered techniques are provided.

Note that the two techniques for sending messages (buffered via GEstat and GElog and unbuffered via the message callback msgcb) are complementary. Either one or the other may be used, but if both are used in the same external module, the buffered messages will be printed after the unbuffered messages.

GEstat: The Utility Routine for Writing Messages to the Status File

GEstat is provided in the appropriate include file (Fortran 90: gehelper.f90, C: geheader.h, Delphi: geheader_d.pas). It is used to communicate messages that should be written to the GAMS status file. The function definition follows:

FORTRAN:

C:

Delphi:

Note that the first argument, icntr, must be passed through from the call of the function GEFUNC. The content of the argument line (or s in Delphi) is packed into the control buffer as one line. When GEFUNC returns, the content of the buffer will be written to the GAMS status file. GEstat may be called several times, each time with one line. Observe that line should not be longer than 132 characters and the overall amount of information written in one call to GEFUNC should not exceed 1000 characters. Further, line should not contain any special characters such as new-line or tab.

In practice, GEstat is often used with calls like the following:

FORTRAN:

C:

Delphi:

GElog: The Utility Routine for Writing Messages to the Log File

Like GEstat, GElog is provided in the appropriate include file. It is used to communicate messages that should be written to the GAMS log file. Note that by default, the log file is the screen. Alternatively, log may be written to a file that is specified with the GAMS command line parameter LogFile. The function definition of GElog follows.

FORTRAN:

C:

Delphi:

Note that GElog behaves exactly like GEstat, with the status file replaced by the log file. The content of line is written to a buffer that in turn is written to the log file when GEFUNC returns.

Observe that it is not possible to write directly to the screen with some combinations of operating system and compiler. This may also depend on the options or flags that are used to build the external module.

Attention

On some systems writing directly to the screen may cause the external module to crash. Therefore, it is advised not to write to the screen as a method for debugging, unless it is clear that it works. Otherwise the module may continue to crash because of the debugging statements after all other errors have been removed. Writing to a file and flushing the buffer is recommended as a safe alternative.

Communicating Data to the External Module via Files

Some external equations will need data from the GAMS program. This data may be passed on via one or more files written using put statements. Usually, such put files will be written in the current directory and the external module will look for them in the current directory. However, if users need to run multiple copies of the same model at the same time, data files should be written in the GAMS scratch directory and the external module should be directed to look for the data files in the scratch directory.

Note that a put file may be defined to be located in the scratch directory with the following file statement in the GAMS model:

File f / '%gams.scrdir%filename' /;

Observe that if the extension .dat is used, GAMS will remove the file from the scratch directory after the run. If another extension is used and the file is not deleted, GAMS will complain about an unexpected file when it cleans up after the run. The external module can receive the name of the scratch directory from GAMS during initialization by setting icntr[I_Getfil] to I_scr and returning immediately. GAMS will then store the name of the scratch directory and length of the name in the communication buffer and call GEFUNC in initialization mode again. Note that GEFUNC will now be called with the sub-mode icntr[I_Smode] set to I_Scr. Then the name may be extracted using the following FORTRAN call:

Here, Scrdir (declared as character*255) will receive the scratch directory and Scrlen (declared as integer) will receive the actual length of Scrdir. In C, the call takes the following form:

Here the routine will return the number or characters transferred to the buffer scratchDir if successful and the value -1 otherwise. If there is space, a terminating '\0'-byte will be written to scratchDir. If the value returned is equal to sizeof(scratchDir), then the string returned will not be '\0'-terminated and may have been truncated as well.

Observe that it is possible to get other directory or file names by specifying other values in icntr[I_Getfil]. After setting this flag, GEFUNC must always return immediately.

For examples, see models [EX5] and [EXMCP3] and their respective FORTRAN and C source files.

Constant Derivatives

Some solvers, like the CONOPT solvers, can take advantage of the knowledge about constant derivatives in equations, which are a result of linear terms. This can be especially useful if an external equation represents an equation like Y=f(X), where Y is unbounded, since variable Y can then be substituted by f(X).

However, with the external module interface as described so far, the solver cannot know which variables appear linearly in the external equations. An optional extension allows to indicate that some of the relationships are linear. This can be activated by returning a 1 in icntr[I_ConstDeriv] during the call of GEFUNC in initialization mode.

GEFUNC will be called again repetitively with icntr[I_Mode] set to DOCONSTDERIV, once for each external equation, with its index specified as usual in icntr[I_Eqno]. For each of these calls, values of all constant derivatives must be specified in the array d. The remaining elements of d, both those corresponding to varying derivatives and to zeros, must be left untouched. These special calls will take place after the initialization call and before the first function evaluation call. Note that in these calls other flags like icntr[I_Dofunc] and icntr[I_Dodrv] and the array x will not be defined.

For an example, see model [EX4X] with the corresponding Fortran 90 and C source files ex4xf_cb.f90 and ex4xc_cb.c, respectively. It is instructive to compare these files to the corresponding files without constant derivatives: ex4f_cb.f90 and ex4c_cb.c.

Second Derivatives: Hessian times Vector

External modules cannot provide a solver with the Hessian matrix of external equations. However, some solvers have particular options for internally approximating the Hessian. For example, see the hessian_approximation option for IPOPT or hessopt for KNITRO. Further, the solver CONOPT can take advantage of second order information in the form of the product Hessian matrix \(\nabla^2 f(x)\) times a vector \(v\). This special form can be used for external equations by setting icntr[I_HVprod] to 1 during the call of GEFUNC in initialization mode.

If the solver can use this information (not all solvers will), then GEFUNC may be called with icntr[I_Mode] set to DOHVPROD to request this operation. icntr[I_Eqno] will hold the equation number and the array x will hold the values of the variables ( \(x\)) in its first icntr[I_NVar] positions and a vector \(v\) in the following icntr[I_Nvar] positions. GEFUNC should evaluate and return \(d = \nabla^2 f(x)\, v\) for the particular external equation \(f\) at the particular point \(x\). Note that \(d\) (which is otherwise used for the derivative vector) will have been initialized to zero by GAMS.

\(\nabla^2 f(x)\, v\) will often be needed for several vectors v at the same point x. Therefore, icntr[I_Newpt] will be used to indicate changes in x in the usual way.

Note that model [EX1X] with the corresponding Fortran 90 source file shows how to use both constant and second derivatives.

Debugging

Implementing external equations brings a number of new potential error sources which GAMS cannot protect against as well as with pure GAMS models. For example, the argument lists in the C or FORTRAN code may be incorrect or the linking process may create an incorrect external module. There is little GAMS can do to help users with this type of errors. It is recommended to carefully follow the examples and output debug messages during the setup calls, for example using the utility routines GEstat and GElog.

Once the overall setup is correct and GAMS can establish proper communication with the external module, there may still be numerical errors where the function values and the derivatives do not match.

Note that the solver CONOPT will by default call its Function and Derivative Debugger in the initial point, if a model has any external equations. The debugger will check that the functions only depend on the variables that are defined in the sparsity pattern and that derivatives computed by numerical perturbation are consistent with the derivatives computed by the external module. If an error is found, CONOPT will stop immediately with an appropriate message. For examples, see the GAMS Test Library models [er1], [er2], and [er3], which illustrate different types of errors. The respective error messages will appear if CONOPT is used as the NLP solver. Note that comments about the errors may be found in the C or FORTRAN source code.

Observe that several types of errors cannot be detected. Derivatives that are computed in the external module and are returned in positions that were not defined in the sparsity pattern in GAMS will be filtered out by the interface and will therefore not be detected. Similarly, derivatives that should be computed but are forgotten, may inherit values from the same derivatives in another equation computed earlier. Finally, fixed variables cannot be perturbed, thus errors related to these variables will usually not be detected.