Engine Clients

Introduction

With the GAMS Engine UI, GAMS MIRO Desktop and GAMS Studio, GAMS already provides three interfaces to Engine. However, if you want to use your own application with Engine, you can do so with ease. The use of OpenAPI allows applications of all kinds to connect to Engine via a RESTful API. Learn more about custom clients here.

Registration

To use Engine, you must first register with the system. For this, you need an invitation code.

  1. Open Engine UI
  2. Click 'Register' below the Login button
  3. Put your invitation code, username and password
  4. Click 'Register'
  5. You will automatically be logged in

GAMS Engine UI

GAMS Engine comes with a web user interface that implements most of the available features. Via this interface you can manage users, models and data, but also submit new jobs and download the results. Depending on your role ("User", "Inviter" or "Administrator"), different elements of the UI are visible.

Components of the GAMS Engine web user interface

The UI consists of six views (the minimum user role required to see this view is appended in brackets):

  • Jobs (User)
  • Namespaces (User)
  • Users (Inviter)
  • Webhooks (User)
  • Cleanup (User)
  • Administration (Administrator)

The Jobs view lists all the jobs you submitted (or all jobs submitted by all users if you are an administrator). You can sort your submissions by clicking on the desired column. From this view, you can also interact with your jobs (view more details, cancel running jobs or download results) as well as submit new ones.

The Namespaces view lists all the namespaces as well as all the models registered in each namespace. To view the models registered in a particular namespace, click on this namespace and the table that lists the models will update. Just as in the Jobs view you can sort the table by clicking on the desired column. You can add or remove namespaces from this view as well as register new models.

The Users view is only visible to inviters and administrators. If you are an administrator, you can see all users registered in the system; if you are an inviter, only users invited either by you or by one of your invitees are displayed. You can edit user roles and permissions as well as remove a user. Additionally, new invitation codes can be created from here. Note that inviters can only invite people who have the same or less privileges as themselves. To learn more about user roles and permissions click here.

Webhooks can be added/updated and removed via the Webhooks view. This view may not be visible to you if you are a user or an inviter and the administrator has not enabled this feature for you. As an administrator, you can also enable/disable the Webhook feature from this view.

To clean up the results of your model runs and free up space in the database, use the Cleanup view. You can either remove files one by one or clean up multiple files at once by clicking the "Run housekeeping" button. This "housekeeping" dialog allows you to remove all files created more than x days ago and/or files created by users who have been removed from the system.

The Administration view can be used to add/update the GAMS Engine license as well as the system-wide GAMS license. In addition, identity providers can be managed here.

GAMS MIRO integration

With the deployment tool GAMS MIRO you can take any existing GAMS model and turn it into a deployable web application with a few lines of additional code. MIRO comes with a rich set of graphical output formats, scenario management, and much more. It is designed for people looking for an easy and automated way to make their GAMS models available to others.

GAMS MIRO is seamlessly linked to Engine and it is very easy to get their connection up and running. Once set up, you can run your CPU intense optimization problems from a tablet or even a smartphone.

A detailed description for this is available in the MIRO documentation.

GAMS MIRO and Engine

GAMS Studio integration

GAMS Studio has built-in functionality of some basic interactions with GAMS Engine. Starting with version 1.4.2 Studio can be used to perform GAMS jobs with Engine. The feature set is constantly being extended.

To use Engine for solving a GAMS job, select Run GAMS Engine in the Quick Access Toolbar or via the entry GAMS in the main menu.

Run GAMS Engine menu

In the dialog that appears, you will first have to authenticate with an Engine server:

  • Engine URL: URL to the GAMS Engine server.
  • Login via: Login method (either "Username/Password" or "JWT")
  • Username (if "Username/Password" is selected): Username of the Engine user account
  • Password (if "Username/Password" is selected): Password of the Engine user account
  • JWT (if "JWT" is selected): JSON Web Token of the Engine user account
When editing the Engine URL Studio tries to reach the server and requests version information. On success the version of GAMS and Engine on the server are displayed. If the local GAMS version is newer, the command line parameter previousWork=1 is automatically added.

When you are logging in to Engine using an identity providers such as Okta or Microsoft Azure Active Directory, you will have to use a JSON Web Token for logging in to GAMS Studio. A JSON Web Token can be generated by logging into the Engine UI and navigating to the Create Authentication Token interface. You get there by clicking on your username in the top left corner and clicking on Create auth token:

Engine UI: Create authentication token button

After clicking the Login button, another form will appear where you can select the namespace in which to solve your model, whether to create a GDX file of the entire GAMS database at the end of your run, and which instance to run your model on (the latter only if you are using GAMS Engine SaaS).

With a click on OK the job is submitted. Note that the model is compiled locally and only the execution phase is performed on GAMS Engine. The results are written into a sub-folder named by the base name (i.e. main file name) of the model. Links in the log output are replaced by their local counterpart.

Note:

GAMS jobs processed via Engine are executed on a Linux operating system. Thus, the UNIX path separator ('/') is used. This is relevant for existing paths in the GAMS model as well as for paths on the command line when submitting a job. You should therefore ensure that the UNIX path separator is always used!

Custom clients

GAMS Engine provides a REST API which follows a specification called OpenAPI version 2.0. This specification defines a standard, language-agnostic interface to RESTful APIs which allows both humans and computers to easily discover and understand the abilities of the software. One main advantage of this technology is that client libraries can easily be autogenerated for several programming languages using the OpenAPI Generator. This includes languages like Python, Java, C, C#, C++, R, Node.js/JavaScript, Ruby, and many more. A complete list of supported API clients can be found here. GAMS Engine can be accessed via any of these clients. A typical workflow could be to take an autogenerated client and use it as a starting point for a custom solution.

Tip:

For the communication between the client and Engine, only the REST API is relevant. You don't have to care about anything that happens inside Engine. For example, to submit a GAMS job, only the corresponding endpoint of the API must be addressed. All subsequent steps (e.g. sending the job to a free worker or queue it, calling GAMS, etc.) are taken care of by Engine.

Tip:

The (technical) API documentation can be found here. You can download the definition file for the latest version of GAMS Engine here.

Let's say we want to submit GAMS jobs via a python client using an autogenerated client as a starting point. On https://openapi-generator.tech/ you can find several ways on how to get such a client. One approach is via npm. First, install the OpenAPI generator tool:

> npm install @openapitools/openapi-generator-cli -g
and set the version to version 5.4.0:
> npx @openapitools/openapi-generator-cli version-manager set 5.4.0

Next, we can generate the python client. For this we need a valid definition file for the Engine API. You can download this file for the latest version of GAMS Engine here:

> npx @openapitools/openapi-generator-cli generate -i path/to/gams_engine.json -g python -o path/to/output/files --package-name=gams_engine

In the directory specified in the above command we will then find all the client and (also auto-generated) documentation files containing a lot of examples. In ./docs/JobsApi.md we can find an example for the job submission:

import time
import gams_engine
from gams_engine.api import jobs_api
from gams_engine.model.message import Message
from gams_engine.model.quota_exceeded import QuotaExceeded
from gams_engine.model.message_and_token import MessageAndToken
from pprint import pprint
# Defining the host is optional and defaults to http://localhost
# See configuration.py for a list of all supported configuration parameters.
configuration = gams_engine.Configuration(
    host = "http://localhost/api"
)

# The client must configure the authentication and authorization parameters
# in accordance with the API server security policy.
# Examples for each auth method are provided below, use the example that
# satisfies your auth use case.

# Configure HTTP basic authorization: BasicAuth
configuration = gams_engine.Configuration(
    username = 'YOUR_USERNAME',
    password = 'YOUR_PASSWORD'
)

# Enter a context with an instance of the API client
with gams_engine.ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = jobs_api.JobsApi(api_client)
    model = "model_example" # str | Name of the model
    namespace = "namespace_example" # str | Namespace containing(or will contain) the model
    run = "run_example" # str | Name of the main gms file with its extension. Will use model + '.gms' if not provided. (optional)
    inex_string = "inex_string_example" # str | Optional JSON string to filter the contents of the result zip file (inex_file takes precedence if specified) (optional)
    text_entries = [
        "text_entries_example",
    ] # [str] |  (optional)
    stream_entries = [
        "stream_entries_example",
    ] # [str] |  (optional)
    stdout_filename = "log_stdout.txt" # str | Name of the file that captures stdout (optional) if omitted the server will use the default value of "log_stdout.txt"
    arguments = [
        "arguments_example",
    ] # [str] | Arguments that will be passed to GAMS call (optional)
    dep_tokens = [
        "dep_tokens_example",
    ] # [str] | Tokens of jobs on which this job depends. The order defines the order in which the results of dependent jobs are extracted. (optional)
    labels = [
        "labels_example",
    ] # [str] | Labels that will be attached to the job in key=value. Currently supported labels are: cpu_request, memory_request, workspace_request, node_selectors, tolerations, instance (optional)
    model_data = open('/path/to/file', 'rb') # file_type | Zip file containing model files, if model is not registered (optional)
    data = open('/path/to/file', 'rb') # file_type | File containing data in zip (optional)
    inex_file = open('/path/to/file', 'rb') # file_type | Optional JSON file to filter the contents of the result zip file (optional)

    # example passing only required values which don't have defaults set
    # and optional values
    try:
        # Submits a new job to be solved
        api_response = api_instance.create_job(model, namespace, run=run, inex_string=inex_string, text_entries=text_entries, stream_entries=stream_entries, stdout_filename=stdout_filename, arguments=arguments, dep_tokens=dep_tokens, labels=labels, model_data=model_data, data=data, inex_file=inex_file)
        pprint(api_response)
    except gams_engine.ApiException as e:
        print("Exception when calling JobsApi->create_job: %s\n" % e)

Using this as a basis, we can now write our own python client. In the following code some of the above elements are used to submit a job for the model trnsport. The GAMS log is shown, the ZIP file containing the results is unpacked and a few results are displayed. Note that the GAMS Python API is also used in this example.

import sys
import time
import os
import zipfile

from gams import *
import gams_engine
from gams_engine.api import jobs_api


if __name__ == "__main__":
    if len(sys.argv) > 1:
        ws = GamsWorkspace(system_directory = sys.argv[1])
    else:
        ws = GamsWorkspace()

    model = 'trnsport' # str | Name of the main .gms file
    ws.gamslib(model)

    model_data_path = os.path.join(ws.working_directory, model + '.zip')

    with zipfile.ZipFile(model_data_path, 'w', zipfile.ZIP_DEFLATED) as model_data:
        model_data.write(os.path.join(ws.working_directory, model+'.gms'), arcname=model+'.gms')

    stdout_filename = 'log_stdout.txt' # str | Name of the file that captures stdout (default to 'log_stdout.txt')
    arguments = ['gdx=default'] # list[str] | Arguments that will be passed to GAMS call (optional)

    data = None  # file | File containing data in zip (optional)
    pf_file_name = None # str | Name of the pf file in the zip, if there is (optional)
    text_entries = []  # list[str] |  (optional)
    stream_entries = []  # list[str] |  (optional)
    inex_file = None # file | Optional file to filter what will be inside the result zip file (optional)

    configuration = gams_engine.Configuration(
        host     = os.environ['ENGINE_URL'],
        username = os.environ['ENGINE_USER'],
        password = os.environ['ENGINE_PASSWORD']
    )
    namespace = os.environ['ENGINE_NAMESPACE']  # str | Namespace containing(or will contain) the model

    # Enter a context with an instance of the API client
    with gams_engine.ApiClient(configuration) as api_client:
        # Create an instance of the API class
        job_api_instance = jobs_api.JobsApi(api_client)
        try:
            print('Posting ' + model)
            with open(model_data_path, 'rb') as model_data:
                create_job_response = job_api_instance.create_job(model, namespace,
                    stdout_filename=stdout_filename, model_data=model_data, arguments=arguments)

            token = create_job_response.token
            print('Job token: %s' % token)

        except gams_engine.ApiException as e:
            print("Exception when calling JobsApi.create_job(): %s\n" % e)
            sys.exit(1)

        finished = False
        time_spent = 0
        while not finished:
            try:
                resp = job_api_instance.pop_job_logs(token)
                print(resp.message, end='')
                if resp.queue_finished:
                    finished = True
                time.sleep(0.5)
            except gams_engine.ApiException as e:
                if e.status == 403:
                    print('Job still in queue. Wait 0.5 seconds.')
                    time.sleep(0.5)
                    time_spent += 0.5
                    if time_spent > 120:
                        print("The Engine instance seems to be busy. Please try again later.")
                        sys.exit(1)

        if job_api_instance.get_job(token).process_status != 0:
            print("Job did not terminate successfully.")

        try:
            print('Fetching results of model: ' + model)
            with zipfile.ZipFile(job_api_instance.get_job_zip(token)) as zf:
                gdxfile = zf.extract(model+".gdx", path=ws.working_directory)
        except gams_engine.ApiException as e:
            print("Exception when calling JobsApi.get_job_zip(): %s\n" % e)
            sys.exit(1)

        try:
            # remove results from server
            job_api_instance.delete_job_zip(token)
        except gams_engine.ApiException as e:
            print("Exception when calling JobsApi.delete_job_zip(): %s\n" % e)
            sys.exit(1)

        result_db = ws.add_database_from_gdx(os.path.join(ws.working_directory, gdxfile))

        for rec in result_db["x"]:
            print("x(" + rec.key(0) + "," + rec.key(1) + "): level=" + str(rec.level) + " marginal=" + str(rec.marginal))

The job submission is done here:


create_job_response = job_api_instance.create_job(model,
                                                  namespace,
                                                  model_data=model_data,
                                                  arguments=arguments)

The log is streamed here:


token = create_job_response.token
[...]
resp = job_api_instance.pop_job_logs(token)

The zip file with the results is downloaded here:


result_zip = job_api_instance.get_job_zip(token)
Did you know...

...that the interface from GAMS Studio to Engine was developed in the same way? It's based on an autogenerated C++ client which was used as a starting point for an integrated solution in Studio.