Administration

Introduction

GAMS Engine comes with a highly flexible model- and user management system that allows you to restrict the activities of your users according to your organizational hierarchy. To understand this in its entirety, we need to address a few concepts in more detail: models, namespaces, permissions and user types.

Note:

To illustrate some of the aspects, the Engine UI is used here. However, each of the steps shown below can also be achieved with the help of custom clients.

Models

When submitting a GAMS job, it usually consists of model files and data files. While the latter usually differ with each job, the actual GAMS model remains mostly unchanged. In order to avoid submitting the same files to Engine with each job, models can be registered with Engine. Files of registered models are stored by Engine and used for subsequent jobs. You only need to provide the name under which the model is registered and send the dynamic data.

  • Job with unregistered model
    The user provides both the model files as well as the data to run the model with.

Unregistered model

  • Job with registered model
    The GAMS model is registered with Engine and the user only provides the model name and dynamic data to run the model with.

Registered model
Note:

If a job submit includes files that are already stored as part of the registered model at Engine, the files submitted by the user will be used.

Namespaces and Permissions

Models are organized in namespaces, which you can think of as file directories. Models are always registered in a namespace and submitted GAMS jobs are also executed there. Each model name can only occur once within a namespace. If two models with the same name are to be registered, this must be done in different namespaces. Apart from the ability to manage multiple models with the same name, namespaces are particularly useful for representing hierarchical (user) structures. This is due to an important property of namespaces, namely the concept of permissions.

Namespaces are similar to directories in the UNIX file system: you can specify which users have access to which namespaces. Just like the UNIX file system, GAMS Engine has three types of permissions on namespaces:
  • Read Permission
    Users can download GAMS models registered in this namespace
  • Write Permission
    Users can register new models in this namespace
  • Execute Permission
    Users can execute models in this namespace
The table below shows the mapping of these permissions to the permission codes used by GAMS Engine.

Permission Code Read Permission Write Permission Execute Permission
0 - - -
1 - -
2 - -
3 -
4 - -
5 -
6 -
7

In the following example, two namespaces are used by three people with different permissions to represent the structure and workflow of a company with a customer.

  • Developer:
    The Model Developer is responsible for maintaining existing models and developing future models. He only accesses the namespace R&D provided for this purpose, in which he has all permissions.
  • Manager:
    The manager has read and execute permissions on the namespace R&D. There she checks the models for customer suitability. If a model (update) is ready, then the manager registers it in the namespace production.
  • Customer:
    The customer has read and execute permissions on the namespace production. With these permissions she can submit jobs with her data and receive the results.

Permission hierarchy
Warning:

By default, the result files to which the customer has access to also contain the model files. In order to restrict a user from accessing your model files, you have to specify a proper INEX file in addition to revoking read permission from this user!

Tip:

After setting up GAMS Engine, there is a single namespace available with the name: global. Adding namespaces as well as registering models can easily be done via the Engine UI in the Models view.

Add namespace and register model

User Management

After first installing the system, there is a single default user available with the name: admin and password: admin.

Note:

We strongly recommend that you change the default password for the admin user immediately after installing Engine to prevent unauthorized users from accessing your system!

If you are administrator, you can invite other users. If you are using the Engine UI, you will find an invite user button in the upper right corner of the Users section.

Invite a new user

GAMS Engine distinguishes three types of users:

  • Users
  • Inviters
    Inviters are users with the additional privilege to invite new users to use GAMS Engine. However, inviters can only invite users who have the same or fewer permissions than themselves. For example, if an inviter has read and execute permissions on the namespace global, she is not allowed to invite a user with write permission on this namespace. Inviters can invite other inviters, but not admins.
  • Administrators
    Administrators are the most privileged users. They have full permissions for each namespace and are allowed to invite new users, even new administrators. Administrators can also add new namespaces or remove existing ones.

In order to add a new user to GAMS Engine, you need to generate an invitation code. Users can then register themselves by providing this invitation code, a username and a password. Invitation codes can be generated by administrators and inviters. When creating an invitation code, permissions to namespaces can be assigned so that the new user can start interacting with the system directly. Note that the invitee's permissions may be lower than the inviter's, but not higher. Furthermore, inviters are able to manage - i.e. modify permissions and delete - their children (invitees) as well as any grandchildren. In this way, you can set up several hierarchy levels.

User hierarchy example:

User hierarchy
Note:

Users only have access to jobs that they have submitted themselves. Inviters can see and interact with their own jobs as well as those of all users invited by them directly or indirectly. Administrators have access to the jobs of all users.

User groups

Users can be assigned to user groups. A user group is a collection of users with any role (user, iviter, or administrator). A user group is always assigned to a namespace and has a unique label/name within that namespace. To create or remove a user group, you must be an inviter with write permission in that namespace (or an administrator).

Any member of a group can see all other members within that group. In addition, inviters can see the members of all their invitees' groups. Only inviters and admins can add and remove members from a group. While admins can remove any member from any group, inviters can only remove their invitees. Also, inviters can only remove members from groups that they are either a member of themselves or that belong to an invitee.

Future plans are to extend the functionality of groups to allow a more fine-grained access control system not only by namespace, but by model.

Quotas

GAMS Engine allows you to limit the solve time as well as the disk usage of individual users or groups of users.

Let's take the user hierarchy from the previous section on user management to give you an example of how quotas work:

User hierarchy example

Only admins and inviters can update quotas of users. While admins can update the quotas of all users, inviters can only update the quotas of direct/indirect invitees (we also call this the "subtree" of an inviter). An inviter cannot assign a quota larger than his own

There are three types of quotas that can be assigned: volume_quota, disk_quota and parallel_quota (only used in the Engine Kubernetes version).

If "Admin" sets the volume_quota of "Student 1" to 60 seconds, the student can only run jobs for one minute. If the quota is exceeded, new jobs submitted by "Student 1" are rejected by Engine. In addition, currently active jobs are canceled (using SIGKILL) when the volume quota of the user who submitted the job is exhausted.

If "Admin" sets the volume_quota of "Teacher" to 1800 seconds, it means that the "Teacher", "Student 1", "Student 2", "Assistant" and anyone invited by the "Assistant" can run GAMS for a total of 1800 seconds. This allows administrators to invite new "entities" by creating a new inviter that has quotas assigned to it. Both the inviter and anyone invited by that inviter are now limited by those quotas.

Quotas allow further restrictions. Imagine the case where "Admin" has assigned 1800 seconds of volume_quota to "Teacher". As an inviter, "Teacher" can give students any volume_quota that is less than or equal to 1800 seconds. "Teacher" could give "Student 1" and "Student 2" each 600 seconds and "Assistant" 800 seconds. In this case, "Student 1" can use up to 600 seconds, "Student 2" can use up to 600 seconds, and "Assistant" can use up to 800 seconds.

Quotas allow for overbooking. An attentive reader may have noticed that in the last example, the teacher had 1800 seconds, but she provided a total of 2000 seconds for her invitees. This does not mean that the teacher's subtree had a total of more than 1800 seconds. When "Student 1" and "Student 2" have used up their entire allotment, the "Assistant" still has 600 seconds left. Therefore, he can no longer use up the full 800 seconds allocated to him by the teacher.

disk_quota is used to limit the disk usage of a user as well as the user's invitees (if any). The following files contribute to disk usage:

  • Temporary/unregistered models
  • Additional data provided with the (Hypercube) job
  • Inex files provided with the (Hypercube) job
  • (Hypercube) job results

If the disk_quota is exceeded, GAMS Engine rejects all new jobs submitted by users who have exceeded their quotas. When a worker receives a job from a user whose disk_quota is exceeded, it is cancelled.

The disk space used during solving does not contribute to the disk_quota. A job could use more disk space during solving, but only upload a small portion (perhaps a result GDX file) and would not violate the quota.

Unlike volume_quota, disk_quota has a grace space. When a job is solved, it might be undesirable to refuse to upload the result because the disk_quota is exceeded. In this scenario, the result will be uploaded if it does not exceed disk_quota + 1GB. If the grace space is also exceeded, the job is marked as "Corrupted" and the results are not available.

disk_quota allows you to limit unregistered files. To limit registered files, such as registered models, inex files, etc. admins can assign a disk_quota to namespaces (available via the /namespaces/{namespace}/disk-quota API).

Similar to the volume_quota, the disk_quota allows further restrictions for invited users as well as overbooking.

Cleanup

Over time, the amount of disk space required on the server can grow accordingly. It is the responsibility of the admin to ensure that the result data is deleted when required. The Engine API provides a cleanup endpoint for this purpose.

When working with the Engine UI, you can reduce the amount of disk space used by GAMS Engine using the Cleanup view. You can either remove result files of individual jobs one by one or clean up multiple files at once using the Run housekeeping dialog. This allows you to remove all files created more than x days ago and/or files created by users who have been removed from the system.

Run housekeeping
Pro Tip for Administrators:

If you notice a significant difference between the disk usage report and the actual disk usage, you can manually examine the files. GAMS Engine has its own container for housekeeping purposes. In case of network interruptions or other unusual events, automatic housekeeping may be disrupted. In this case, you can always force the process manually.

Assuming you have access to the cleaner container, you can get a status report using the commands: docker exec cleaner_container_name kill -10 1 and docker logs cleaner_container_name.

You can force the cleaner to manually check and delete the files that should be deleted. To do this, run: docker exec cleaner_container_name kill -12 1