Smart Executor Service Documentation¶

The Smart Executor is a gCube service that runs tasks on behalf of the infrastructure: the management, monitoring and harvesting activities that must run once, after a delay or on a recurring schedule. Each kind of task is a plugin, a jar deployed with the service; the service exposes the plugins it holds and lets a client launch them, follow their executions and stop them, through a REST API.

The REST API is described by the OpenAPI document of the service, shown by ReDoc and Swagger UI from the menu of these pages.

Concepts¶

Plugins¶

A plugin is identified by its name and version, and declares the inputs it accepts and its capabilities. GET /plugins lists the plugins the instance holds.

Launching an Execution¶

A client launches a plugin with POST /plugins/{name}/executions, sending a launch parameter:

Field	Meaning
`pluginName`	The plugin to run; it must be the one named in the URL
`pluginVersion`	The version of the plugin; absent means any version
`pluginCapabilities`	The capabilities the plugin must declare, or it is not launched; absent means no constraint
`inputs`	The inputs passed to the plugin
`pluginStateNotifications`	The notifiers to call at every change of state
`scheduling`	Absent for a single immediate run; otherwise the scheduling below

The service answers with the UUID of the scheduled task, which identifies its executions in every later request.

Scheduling¶

Field	Meaning
`cronExpression`	A cron-like expression for a recurring task; not used together with `delay`
`delay`	The seconds between subsequent executions; not used together with `cronExpression`
`schedulingTimes`	How many times the pattern is applied; `0` means indefinitely
`firstStartTime`	The first instant the scheduling can start; absent or `0` means immediately
`endTime`	The instant after which no execution starts; absent or `0` means never
`previuosExecutionsMustBeCompleted`	With a cron expression, an execution fired while the previous one is still running is discarded
`global`	The task is released in the database when its instance stops, instead of staying assigned to it

States of an Execution¶

State	Final	Meaning
`CREATED`	No	The execution is created and not yet running
`RUNNING`	No	The execution is running
`STOPPED`	Yes	The execution was stopped
`DONE`	Yes	The execution terminated successfully
`FAILED`	Yes	The execution failed
`DISCARDED`	Yes	The scheduler discarded the execution because the previous one was still running

GET /plugins/{name}/executions/{uuid} returns the state of the last execution of a task, or of the one given by iteration.

Stopping and Unscheduling¶

DELETE /plugins/{name}/executions/{uuid} stops the running execution of a task and keeps its scheduling; with unschedule=true, or with the PURGE method on the same URL, it also removes the scheduling, the global one included.

Tasks of an Instance¶

One instance of the Smart Executor holds a plugin. The scheduled tasks are kept in the database of the service: an instance that starts takes in charge every scheduled task of its plugins in its Context.

Statistics¶

The node gives the public statistics of the service, as totals only, in the group executor at /smartgears/smart-executor/statistics, with no token. They are computed every 15 minutes:

Figure	What it counts
`plugins`	The plugins installed on the node
`scheduledTasks`	The scheduled tasks
`scheduledTasksByKind`	The scheduled tasks, recurrent (cron expression or delay) or one-time
`executions`	The executions in the database that reached a final state; purged executions are not counted
`executionsLast24h`	The executions that reached a final state in the last 24 hours

The outcome of the executions is not public: it is in the metrics of the node, /smartgears/metrics, which the load balancers serve to the VPN only. The counter smartexecutor_executions_total counts the executions by plugin, context and final state (DONE, FAILED, STOPPED, DISCARDED). A failed execution is also written at level ERROR in the log of the service, with its plugin, UUID, iteration, Context and error.

Authorization¶

D4Science uses OpenID Connect for authentication and UMA 2 for authorization. Every request carries a JWT access token in the Authorization: Bearer header; the Context of the token is the Context the task runs in. A Bearer token can be obtained following How to access resources.

What a caller can do depends on its roles in the Context of the token:

Request	Who
`GET /plugins`	Anybody with a token of the Context
`POST /plugins/{name}/executions`	A user with the role `SmartExecutor-Manager` or `Infrastructure-Manager`
`GET /plugins/{name}/executions`	A manager gets every task of the Context; anybody else only the tasks they launched
`GET /plugins/{name}/executions/{uuid}`	A manager, or who launched the task
`DELETE` and `PURGE /plugins/{name}/executions/{uuid}`	A manager

A request never reaches a task of another Context.

The task keeps the Context of the request and the identifier of its caller, never its token. Every execution, immediate or scheduled, runs in that Context: the plugin runs with the token of its own client, and the service records the states of the execution with the token of the node.

Configuration of a Plugin¶

Every plugin has its configuration in WEB-INF/lib/plugins/<plugin name>/<plugin name>.yaml, written by the provisioning. SmartGears reads it when the service starts, and the Smart Executor gives it to the plugin at every execution (Plugin.getConfiguration()): a plugin never looks for its configuration in the classpath, which the plugins of a node share. The file holds at least the client of the plugin:

client_id: "smart-executor-plugin-hello-world"
client_secret: "..."

and then the keys of the plugin itself. An execution of a plugin whose file is missing, or holds no client, ends FAILED.

HTTP Statuses¶

Status	When
`200 OK`	The request succeeded and the response carries content
`204 No Content`	The execution was stopped
`400 Bad Request`	Missing or invalid inputs, e.g. a launch parameter naming another plugin
`401 Unauthorized`	No token, or a token that is not valid
`404 Not Found`	The plugin or the execution does not exist
`500 Internal Server Error`	A failure of the service

A 500 Internal Server Error that persists after retrying the request is reported in the gCube ticketing system, with the request, how to reproduce it and the time it happened.

Service Discovery¶

The service is published in the Information System as an EService. It is found with this JSON query:

{
  "type": "EService",
  "consistsOf": [
    {
      "type": "IsIdentifiedBy",
      "target": {
        "type": "SoftwareFacet",
        "group": "org.gcube.vremanagement",
        "name": "smart-executor"
      }
    }
  ]
}

SmartGears publishes every plugin the instance holds as a RunningPlugin related to the EService by Activates, with a stable identifier and the name of the plugin (its Maven artifactId). /smartgears/smart-executor/plugins shows the plugins SmartGears found.

The service keeps its scheduled tasks and the states of their executions in a PostgreSQL database of its own. It finds the address and the credentials of the database in the Information System, in the Context of the task: the AccessPointFacet of the EService identified by the SoftwareFacet org.postgresql / postgresql and classified by the ClassificationFacet Database / SmartExecutor. The endpoint of the access point is the JDBC URL of the database and its password is encrypted with the key of the Context. The service creates its two tables, scheduled_task and plugin_state_evolution, when they are missing, so the user of the database needs the right to create tables in its schema. The service carries the PostgreSQL driver: a plugin that uses PostgreSQL takes it as provided from smart-executor-bom.

Once a day the service deletes, in each Context, the states of the executions older than 90 days, except the last state of each task: the state of a task can always be read. The history of the executions is in the accounting, where every execution is a JobUsageRecord.

Maven Coordinates¶

<groupId>org.gcube.vremanagement</groupId>
<artifactId>smart-executor</artifactId>