** Notice: all internal pipelines have been restricted to 30 minute timeouts, if your CI/CD runs take longer, either use your own runners or read below **
We would remind everyone that the CI/CD services provided by the university Gitlab servers are intended for CI/CD use only (ie. delivering new versions of your software to your production servers, etc).They are not intended for data analysis or -production use, or creating, building and running servers to do heavy data calculation or visualization.
As a general guideline; if your jobs take more than 15 minutes, you might be running something that is better suited for a dedicated system or a virtual server - in this case, please contact the helpdesk for a solution to your needs.
Redis streams are used to inform about update events, typically "create", "delete" and "update" for various resource types. The data itself should be stored within regular Redis keys and not included in the event, the event just details which resource has changed.
[Redis streams](https://redis.io/docs/data-types/streams-tutorial/) are used to inform about update events, typically "create", "delete" and "update" for various resource types. Relevant information about the state changes is included with the event. Events are sent using the `redisSendEvent` function defined in `@ftl/common` package.
The owner of events is the generator of that event, that services defines the structures.
The owner of events is the generator of that event, that services defines the structures inside a corresponding directory of the `@ftl/api` package. The API package should only contain typescript definitions for the events, no actual Javascript code.
Should each event type have it's own stream? The stream keys can then be specific to that type. The downside is consumers need to watch for many different streams and ordering might become a problem.
Events usually include common information such as session, request and user identifiers so that the origin of the event can be traced. Typically, each general type of event has its own Redis stream key (eg. `events:stream`), but there might be subcategories of event within that stream such as `start` / `stop` or `create` / `destroy`.
Other services can this listen for these events by registering a callback with `redisSetStreamCallback`. Each service type is a "consumer group" which means that only one instance of that service type will receive the event, not all instances. The idea is that the receiving instance will make some database update or take some action and that other instances of the same service can become aware of the consequences by some other means.
# Commands
Command streams instruct other services to perform a CRUD operation or issue an instruction of another kind. As an example, one service might wish to query a resource but finds that Redis does not contain that data yet (or it expired). To obtain the data it needs to send a query request and then respond to an update event.
Command streams instruct other services to perform some action. These are the inverse of an event and are owned by the service which responds to the command. Using events requires that a service becomes dependent and knowledgeable about the service generating the event, whereas using commands it need not know about other services who may issue the command. Keeping inter-service dependencies to a minimum and directed (a directed rather than cyclic graph of dependencies) is important.
Examples of commands might be to:
* log a user in
* request information events from another service
* provide an inter-service equivalent to a http endpoint CRUD style operation
# Metrics
One additional stream type is for metric data. These are not strictly "events" because the data is not critical to the operation of the system and no actions are expected to occur as a result of this data. Metrics are also not typically user or client generated, but are periodic and automated. Nonetheless, `redisSendEvent` is used to send to metric stream keys.
