Engine Clients

Introduction

With the GAMS Engine UI, GAMS MIRO Desktop and GAMS Studio, GAMS already provides three interfaces to Engine. However, if you want to use your own application with Engine, you can do so with ease. The use of OpenAPI allows applications of all kinds to connect to Engine via a RESTful API. Learn more about custom clients here.

Registration

To use Engine, you must first register with the system. For this, you need an invitation code.

  1. Open Engine UI
  2. Click 'Register' below the Login button
  3. Put your invitation code, username and password
  4. Click 'Register'
  5. You will automatically be logged in

GAMS Engine UI

GAMS Engine comes with a web user interface that implements most of the available features. Via this interface you can manage users, models and data, but also submit new jobs and download the results. Depending on your role ("User", "Inviter" or "Administrator"), different elements of the UI are visible.

Components of the GAMS Engine web user interface

The UI consists of six views (the minimum user role required to see this view is appended in brackets):

  • Jobs (User)
  • Namespaces (User)
  • Users (User)
  • Pools (config-dependent)
  • Webhooks (User)
  • Cleanup (User)
  • Administration (Administrator)

Furthermore, options can be reached by clicking on your username in the top left corner. Here you find Settings, Create auth token and Change password. Also your quota is displayed here.

Engine UI: Create authentication token button

In the Settings dialog you can define the quota unit, the default table length per page and set your default instance, if at least one instance exist for you. There are two options for the quota unit, the default is mults, which means that the quota is expressed as multiplier * seconds. The second option is multh, in which case all quota values will be divided by 3600 (multiplier * hours). In addition to these general settings, you also have the option of activating push notifications to be notified when certain events occur, e.g. when a job is completed. For this feature to be available, an administrator must have activated the webhooks feature.

Engine settings dialog

Jobs

The Jobs view lists all the jobs you submitted and all jobs shared with you via groups (User groups). Furthermore, an administrator sees all jobs submitted by all users and inviters can see all jobs submitted by their invitees. You can sort your submissions by clicking on the desired column. From this view, you can also interact with your jobs (view more details, cancel running jobs or download results) as well as submit new ones.

Namespaces

The Namespaces view lists all the namespaces as well as all the models registered in each namespace. To view the models registered in a particular namespace, click on this namespace and the table that lists the models will update. Just as in the Jobs view you can sort the table by clicking on the desired column. You can add or remove namespaces from this view as well as register new models.

Users

If you open the Users view as a user, you can only see yourself here. If you are an administrator, you can see all users registered in the system; if you are an inviter, only users invited either by you or by one of your invitees are displayed. The table displays the role, inviter and date of creation of the users. Additionally, here you can delete users or create new invitation codes. Note that inviters can only invite people who have the same or less privileges as themselves. To learn more about user roles and permissions click here. To edit user roles, permissions and quota as well as view the usage, and for inviters their invitees, click on the underlined usernames to access the User Page of the selected user.

User Page

The User Page contains all information and settings for the seleced user. This page is reachable wherever a username is mentioned (if you have the permission to see that user's page). Depending on your permissions the page contains a different amount of tabs. If you are a user you have only the permission to view your own page and will find your Usage here. As an inviter you can view the User Page of all your invited users, and the users they invited. For them you have the option to edit the password, license, instance, quota, identity provider, permissions, and view their usage. For yourself you will find your own usage and an invitee tree. As an admin you will find the same options as an inviter, but for all existing users.

If you select the Usage tab a more detailed view of the usage of that user is displayed. Here you will find the user's remaining quota and an overview of the jobs generated by that user. First, select the timeframe you are interested in and if you are an admin or an inviter, whether you want to include invitees in the data. The Dashboard allows a more detailed view of the individual jobs. The table will by default be grouped by username. It is possible to change this to no grouping at all, or by instance, or by pool label. If at least two users/instances/pool labels exist in the selected timeframe, the data will also be displayed in pie diagrams. If Instance Pools are used, the second table will display the idle times of these pools. Under the second tab Timeline a graphical visualization shows either the jobs weighted by their multipliers, or just the number of parallel jobs over time.

Usage dialog

Pools

If GAMS Engine is used with a Kubernetes backend (e.g. Engine SaaS), Instance Pools can be generated and modified under the Pools view. As an administrator this view is always visible to you, for Users/Inviters it depends on the configuration. As an administrator, you can also enable/disable the Instance Pools feature from this view.

Webhooks

Webhooks can be added/updated and removed via the Webhooks view. This view may not be visible to you if you are a user or an inviter and the administrator has not enabled this feature for you. As an administrator, you can also enable/disable the Webhook feature from this view.

Cleanup

To clean up the results of your model runs and free up space in the database, use the Cleanup view. You can either remove files one by one or clean up multiple files at once by clicking the "Run Housekeeping" button. This "Housekeeping" dialog allows you to remove all files created more than x days ago and/or files created by users who have been removed from the system.

Administration

The Administration view can be used to add/update the GAMS Engine license as well as the system-wide GAMS license. Here it is also possible for administrators to manage identity providers and instances. Furthermore, it is possible for the administrator to set a password policy for all users who use Engine as an identity provider.

GAMS MIRO integration

With the deployment tool GAMS MIRO you can take any existing GAMS model and turn it into a deployable web application with a few lines of additional code. MIRO comes with a rich set of graphical output formats, scenario management, and much more. It is designed for people looking for an easy and automated way to make their GAMS models available to others.

GAMS MIRO is seamlessly linked to Engine and it is very easy to get their connection up and running. Once set up, you can run your CPU intense optimization problems from a tablet or even a smartphone.

You can find more details for the MIRO Server setup here. Logging into GAMS Engine from MIRO Desktop is described here.

GAMS MIRO and Engine

GAMS Studio integration

GAMS Studio has built-in functionality of some basic interactions with GAMS Engine. Starting with version 1.4.2 Studio can be used to perform GAMS jobs with Engine. The feature set is constantly being extended.

To use Engine for solving a GAMS job, select Run GAMS Engine in the Quick Access Toolbar or via the entry GAMS in the main menu.

Run GAMS Engine menu

In the dialog that appears, you will first have to authenticate with an Engine server:

  • Engine URL: URL to the GAMS Engine server.
  • Login via: Login method (either "Username/Password" or "JWT")
  • Username (if "Username/Password" is selected): Username of the Engine user account
  • Password (if "Username/Password" is selected): Password of the Engine user account
  • JWT (if "JWT" is selected): JSON Web Token of the Engine user account
  • Login via OpenID Connect with hidden/puplic OpenID Connect providers
Note:

GAMS Studio only supports OpenID Connect providers which inlcude Device Authorization Flow.

When editing the Engine URL Studio tries to reach the server and requests version information. On success the version of GAMS and Engine on the server are displayed. If the local GAMS version is newer, the command line parameter previousWork=1 is automatically added.

When you are logging in to Engine using an identity providers such as Okta or Microsoft Azure Active Directory, you will have to use a JSON Web Token for logging in to GAMS Studio. A JSON Web Token can be generated by logging into the Engine UI and navigating to the Create Authentication Token interface. You get there by clicking on your username in the top left corner and clicking on Create auth token:

Engine UI: Create authentication token button

After clicking the Login button, another form will appear where you can select the namespace in which to solve your model, whether to create a GDX file of the entire GAMS database at the end of your run, and which instance to run your model on (the latter only if you are using GAMS Engine SaaS).

With a click on OK the job is submitted. Note that the model is compiled locally and only the execution phase is performed on GAMS Engine. The results are written into a sub-folder named by the base name (i.e. main file name) of the model. Links in the log output are replaced by their local counterpart.

Note:

GAMS jobs processed via Engine are executed on a Linux operating system. Thus, the UNIX path separator ('/') is used. This is relevant for existing paths in the GAMS model as well as for paths on the command line when submitting a job. You should therefore ensure that the UNIX path separator is always used!

Custom clients

GAMS Engine provides a REST API which follows a specification called OpenAPI version 2.0. This specification defines a standard, language-agnostic interface to RESTful APIs which allows both humans and computers to easily discover and understand the abilities of the software. One main advantage of this technology is that client libraries can easily be autogenerated for several programming languages using the OpenAPI Generator. This includes languages like Python, Java, C, C#, C++, R, Node.js/JavaScript, Ruby, and many more. A complete list of supported API clients can be found here. GAMS Engine can be accessed via any of these clients. A typical workflow could be to take an autogenerated client and use it as a starting point for a custom solution.

Tip:

For the communication between the client and Engine, only the REST API is relevant. You don't have to care about anything that happens inside Engine. For example, to submit a GAMS job, only the corresponding endpoint of the API must be addressed. All subsequent steps (e.g. sending the job to a free worker or queue it, calling GAMS, etc.) are taken care of by Engine.

Tip:

The (technical) API documentation can be found here. You can download the definition file for the latest version of GAMS Engine here.

Let's say we want to submit GAMS jobs via a python client using an autogenerated client as a starting point. On https://openapi-generator.tech/ you can find several ways on how to get such a client. One approach is via npm. First, install the OpenAPI generator tool:

> npm install @openapitools/openapi-generator-cli -g
and set the version to version 5.4.0:
> npx @openapitools/openapi-generator-cli version-manager set 5.4.0

Next, we can generate the python client. For this we need a valid definition file for the Engine API. You can download this file for the latest version of GAMS Engine here:

> npx @openapitools/openapi-generator-cli generate -i path/to/gams_engine.json -g python -o path/to/output/files --package-name=gams_engine

In the directory specified in the above command we will then find all the client and (also auto-generated) documentation files containing a lot of examples. In ./docs/JobsApi.md we can find an example for the job submission:

import time
import gams_engine
from gams_engine.api import jobs_api
from gams_engine.model.message import Message
from gams_engine.model.quota_exceeded import QuotaExceeded
from gams_engine.model.message_and_token import MessageAndToken
from pprint import pprint
# Defining the host is optional and defaults to http://localhost
# See configuration.py for a list of all supported configuration parameters.
configuration = gams_engine.Configuration(
    host = "http://localhost/api"
)

# The client must configure the authentication and authorization parameters
# in accordance with the API server security policy.
# Examples for each auth method are provided below, use the example that
# satisfies your auth use case.

# Configure HTTP basic authorization: BasicAuth
configuration = gams_engine.Configuration(
    username = 'YOUR_USERNAME',
    password = 'YOUR_PASSWORD'
)

# Enter a context with an instance of the API client
with gams_engine.ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = jobs_api.JobsApi(api_client)
    model = "model_example" # str | Name of the model
    namespace = "namespace_example" # str | Namespace containing(or will contain) the model
    run = "run_example" # str | Name of the main gms file with its extension. Will use model + '.gms' if not provided. (optional)
    inex_string = "inex_string_example" # str | Optional JSON string to filter the contents of the result zip file (inex_file takes precedence if specified) (optional)
    text_entries = [
        "text_entries_example",
    ] # [str] |  (optional)
    stream_entries = [
        "stream_entries_example",
    ] # [str] |  (optional)
    stdout_filename = "log_stdout.txt" # str | Name of the file that captures stdout (optional) if omitted the server will use the default value of "log_stdout.txt"
    arguments = [
        "arguments_example",
    ] # [str] | Arguments that will be passed to GAMS call (optional)
    dep_tokens = [
        "dep_tokens_example",
    ] # [str] | Tokens of jobs on which this job depends. The order defines the order in which the results of dependent jobs are extracted. (optional)
    labels = [
        "labels_example",
    ] # [str] | Labels that will be attached to the job in key=value. Currently supported labels are: cpu_request, memory_request, workspace_request, node_selectors, tolerations, instance (optional)
    model_data = open('/path/to/file', 'rb') # file_type | Zip file containing model files, if model is not registered (optional)
    data = open('/path/to/file', 'rb') # file_type | File containing data in zip (optional)
    inex_file = open('/path/to/file', 'rb') # file_type | Optional JSON file to filter the contents of the result zip file (optional)

    # example passing only required values which don't have defaults set
    # and optional values
    try:
        # Submits a new job to be solved
        api_response = api_instance.create_job(model, namespace, run=run, inex_string=inex_string, text_entries=text_entries, stream_entries=stream_entries, stdout_filename=stdout_filename, arguments=arguments, dep_tokens=dep_tokens, labels=labels, model_data=model_data, data=data, inex_file=inex_file)
        pprint(api_response)
    except gams_engine.ApiException as e:
        print("Exception when calling JobsApi->create_job: %s\n" % e)

Using this as a basis, we can now write our own python client. In the following code some of the above elements are used to submit a job for the model trnsport. The GAMS log is shown, the ZIP file containing the results is unpacked and a few results are displayed. Note that the GAMS Python API is also used in this example.

import sys
import time
import os
import zipfile

from gams import *
import gams_engine
from gams_engine.api import jobs_api


if __name__ == "__main__":
    if len(sys.argv) > 1:
        ws = GamsWorkspace(system_directory = sys.argv[1])
    else:
        ws = GamsWorkspace()

    model = 'trnsport' # str | Name of the main .gms file
    ws.gamslib(model)

    model_data_path = os.path.join(ws.working_directory, model + '.zip')

    with zipfile.ZipFile(model_data_path, 'w', zipfile.ZIP_DEFLATED) as model_data:
        model_data.write(os.path.join(ws.working_directory, model+'.gms'), arcname=model+'.gms')

    stdout_filename = 'log_stdout.txt' # str | Name of the file that captures stdout (default to 'log_stdout.txt')
    arguments = ['gdx=default'] # list[str] | Arguments that will be passed to GAMS call (optional)

    data = None  # file | File containing data in zip (optional)
    pf_file_name = None # str | Name of the pf file in the zip, if there is (optional)
    text_entries = []  # list[str] |  (optional)
    stream_entries = []  # list[str] |  (optional)
    inex_file = None # file | Optional file to filter what will be inside the result zip file (optional)

    configuration = gams_engine.Configuration(
        host     = os.environ['ENGINE_URL'],
        username = os.environ['ENGINE_USER'],
        password = os.environ['ENGINE_PASSWORD']
    )
    namespace = os.environ['ENGINE_NAMESPACE']  # str | Namespace containing(or will contain) the model

    # Enter a context with an instance of the API client
    with gams_engine.ApiClient(configuration) as api_client:
        # Create an instance of the API class
        job_api_instance = jobs_api.JobsApi(api_client)
        try:
            print('Posting ' + model)
            with open(model_data_path, 'rb') as model_data:
                create_job_response = job_api_instance.create_job(model, namespace,
                    stdout_filename=stdout_filename, model_data=model_data, arguments=arguments)

            token = create_job_response.token
            print('Job token: %s' % token)

        except gams_engine.ApiException as e:
            print("Exception when calling JobsApi.create_job(): %s\n" % e)
            sys.exit(1)

        finished = False
        time_spent = 0
        while not finished:
            try:
                resp = job_api_instance.pop_job_logs(token)
                print(resp.message, end='')
                if resp.queue_finished:
                    finished = True
                time.sleep(0.5)
            except gams_engine.ApiException as e:
                if e.status == 403:
                    print('Job still in queue. Wait 0.5 seconds.')
                    time.sleep(0.5)
                    time_spent += 0.5
                    if time_spent > 120:
                        print("The Engine instance seems to be busy. Please try again later.")
                        sys.exit(1)

        if job_api_instance.get_job(token).process_status != 0:
            print("Job did not terminate successfully.")

        try:
            print('Fetching results of model: ' + model)
            with zipfile.ZipFile(job_api_instance.get_job_zip(token)) as zf:
                gdxfile = zf.extract(model+".gdx", path=ws.working_directory)
        except gams_engine.ApiException as e:
            print("Exception when calling JobsApi.get_job_zip(): %s\n" % e)
            sys.exit(1)

        try:
            # remove results from server
            job_api_instance.delete_job_zip(token)
        except gams_engine.ApiException as e:
            print("Exception when calling JobsApi.delete_job_zip(): %s\n" % e)
            sys.exit(1)

        result_db = ws.add_database_from_gdx(os.path.join(ws.working_directory, gdxfile))

        for rec in result_db["x"]:
            print("x(" + rec.key(0) + "," + rec.key(1) + "): level=" + str(rec.level) + " marginal=" + str(rec.marginal))

The job submission is done here:


create_job_response = job_api_instance.create_job(model,
                                                  namespace,
                                                  model_data=model_data,
                                                  arguments=arguments)

The log is streamed here:


token = create_job_response.token
[...]
resp = job_api_instance.pop_job_logs(token)

The zip file with the results is downloaded here:


result_zip = job_api_instance.get_job_zip(token)
Did you know...

...that the interface from GAMS Studio to Engine was developed in the same way? It's based on an autogenerated C++ client which was used as a starting point for an integrated solution in Studio.