** Notice: all internal pipelines have been restricted to 30 minute timeouts, if your CI/CD runs take longer, either use your own runners or read below **
We would remind everyone that the CI/CD services provided by the university Gitlab servers are intended for CI/CD use only (ie. delivering new versions of your software to your production servers, etc).They are not intended for data analysis or -production use, or creating, building and running servers to do heavy data calculation or visualization.
As a general guideline; if your jobs take more than 15 minutes, you might be running something that is better suited for a dedicated system or a virtual server - in this case, please contact the helpdesk for a solution to your needs.
This project provides a script able to recursively copy/synchronize a GitLab group from one GitLab server to another.
This project allows to recursively copy/synchronize the [to be continuous](https://gitlab.com/to-be-continuous) project from gitlab.com to your self-managed GitLab instance.
It can be run manually (command line) and also as scheduled CI/CD job to regularly synchronize a GitLab group mirror.
It can be run manually (manual pipeline) and also as scheduled CI/CD pipeline to regularly retrieve upstream changes.
## Pre-requisites
Under the hood, it uses the [GitLab Copy CLI](https://gitlab.com/to-be-continuous/tools/gitlab-cp) tool as a Docker image, [directly pulled from gitlab.com](https://gitlab.com/to-be-continuous/tools/gitlab-cp/container_registry/6261230).
The GitLab Synchronization Script has the following requirements:
## Usage: CLI (first-time copy)
* Bash interpreter <br/>_Trivial on Linux or MacOS, tested with [Git Bash](https://www.atlassian.com/git/tutorials/git-bash) on Windows (available in [Git for Windows](https://gitforwindows.org/))_
The GitLab Copy CLI tool can be used directly from your computer (this is the recommended way to clone _to be continuous_ for the first time).
*[curl tool](https://curl.se/) installed and accessible as `curl` command from the Bash interpreter
*[jq tool](https://stedolan.github.io/jq/download/) installed and accessible as `jq` command from the Bash interpreter
## Usage: script
More info is available in [_to be continuous_ documentation](https://to-be-continuous.gitlab.io/doc/self-managed/basic/#copy-tbc-to-your-gitlab).
```bash
## Usage: CI/CD
gitlab-sync.sh \
[--src-api{GitLab source API url}]\
Once copied _to be continuous_ to your GitLab server, you shall then schedule a pipeline in this project (`to-be-continuous/tools/gitlab-sync`) - for instance every night - to keep synchronized with the upstream project.
[--src-token{GitLab source token}]\
[--src-sync-path{GitLab source root group path to synchronize}]\
The job only requires a GitLab token, that shall be configured declaring a `$GITLAB_TOKEN` CI/CD project variable.
--dest-api{GitLab destination API url}\
All the other parameters/variables are automatically determined by the job but might be overwritten with the appropriate project variables if need be:
--dest-token{GitLab destination token}\
[--dest-sync-path{GitLab destination root group path to synchronize}]\
[--max-visibility{max visibility}]\
[--exclude{coma separated list of project/group path(s) to exclude}]\
[--insecure]\
[--update-release]\
[--no-group-description{do not synchronise group description}]\
[--no-project-description{do not synchronise project description}]
| `--dest-token` | `$DEST_TOKEN` or `$GITLAB_TOKEN` | GitLab destination token with at least scopes `api,read_repository,write_repository` and `Owner` role (**mandatory**) | _none_ |
| `--dest-token` | `$DEST_TOKEN` or `$GITLAB_TOKEN` | GitLab destination token with at least scopes `api,read_repository,write_repository` and `Owner` role | _none_ (**mandatory**) |
| `--dest-sync-path` | `$DEST_SYNC_PATH` | GitLab destination root group path to synchronize | guessed from GitLab CI env or `to-be-continuous` |
| `--dest-sync-path` | `$DEST_SYNC_PATH` | GitLab destination root group path to synchronize (defaults to `--src-sync-path`) | determined from `$CI_PROJECT_NAMESPACE` |
| `--max-visibility` | `$MAX_VISIBILITY` | maximum visibility of projects in destination group | `$CI_PROJECT_VISIBILITY` (when running in GitLab CI) or `public` |
| `--max-visibility` | `$MAX_VISIBILITY` | maximum visibility of projects in destination group (defaults to `public`) | `$CI_PROJECT_VISIBILITY` |
| `--exclude` | `$EXCLUDE` | coma separated list of project/group path(s) to exclude | _none_ |
| `--exclude` | `$EXCLUDE` | project/group path(s) to exclude (multiple CLI option; env. variable is a coma separated list) | `samples,custom` |
| `--insecure` | `$INSECURE` | set to skip TLS check on curl requests | `false` |
| `--update-release` | `$UPDATE_RELEASE` | set to update the releases even if they exists | `false` |
| `--update-release` | `$UPDATE_RELEASE` | set to force the update of the latest release (in order to trigger GitLab CI/CD catalog publication) | `false` |
| `--no-group-description` | `$GROUP_DESCRIPTION_DISABLED` | do not synchronise group description | _none_ |
| `--update-avatar` | `$UPDATE_AVATAR` | force update the avatar images even when they exist and look the same | `false` |
| `--no-project-description` | `$PROJECT_DESCRIPTION_DISABLED` | do not synchronise project description | _none_ |
You shall use this script to copy the _to be continuous_ project to your own GitLab server for the first time with the following command:
| `--dry-run` | _none_ | dry run (don't execute any write action) | `false` |
| `--halt-on-error` | _none_ | halt synchronizing when an error occurs | `false` |
```bash
| `--cache-dir` | `$CACHE_DIR` | cache directory (used to download resources such as images and Git repositories) (defaults to `.work`) | `.work` |
curl -s https://gitlab.com/to-be-continuous/tools/gitlab-sync/-/raw/master/gitlab-sync.sh | bash /dev/stdin --dest-api{your GitLab server API url}--dest-token{your GitLab token}--exclude samples,custom
```
:warning: Each CLI option may alternately be specified with an environment variable (see in the table above). This might be useful to configure the CI/CD job.
## CI/CD Catalog
## Usage: CI/CD
Once copied _to be continuous_ to your GitLab server, you shall then schedule a pipeline in this project (`to-be-continuous/tools/gitlab-sync`) - for instance every night - to keep synchronized with source project.
To use _to be continuous_ templates as CI/CD Catalog resources in your GitLab server, you'll have to manually activate the _CI/CD Catalog resource_ option (`Settings > General > Visibility > Project Features > Permissions`) in each template project.
Then you'll have to re-publish a release to make the template appear in the CI/CD Catalog.
The script will only require a GitLab token, that shall be configured declaring a `$GITLAB_TOKEN` CI/CD project variable. (`--dest-api` will be implicitly retrieved using predefined `$CI_API_V4_URL`).
:information_source: You may re-run the GitLab Copy CLI tool with the `--update-release` (or `$UPDATE_RELEASE` variable) set to force re-publishig the latest release of each template.
You can define a custom alpine image to use by setting `$GITLAB_SYNC_IMAGE` (default to `registry.hub.docker.com/alpine:latest`)
## Legacy `gitlab-sync.sh` script
## CI/CD Catalog
This project was previously using the `gitlab-sync.sh` script to perform the recursive copy/sync.
It has now been replaced by the [GitLab Copy CLI](https://gitlab.com/to-be-continuous/tools/gitlab-cp) tool that has numerous advantages over it.
To use _to be continuous_ templates as CI/CD Catalog resources on a your GitLab Server, you have to manually activate the _CI/CD Catalog resource_ option in `Settings`/`General`/`Visibility, project features, permissions`.
Anyhow, the legacy script can still be reactivated by setting the `SYNC_TOOL` CI/CD project variable to `legacy`.
