The Broker, an open-source tool composed of a client and a server-side, acts as a proxy between Snyk and your on-premise hosted Git repositories (e.g. GitHub Enterprise or BitBucket Server) and with your on-premise Jira installation when installed behind a firewall.
The Broker client is a service to be installed and hosted on your private infrastructure. Together with the Broker server, hosted on the Snyk backend, it establishes a secure and moderated tunnel with Snyk servers.
With this secure connection, the Broker client:
keeps sensitive data such as your access tokens, within the perimeter of your private network—never sharing that information with Snyk.
limits access to your network by Snyk to the bare minimum required in order to scan and monitor vulnerabilities and offer automated remediation.
Installed within your organization’s private network, the Broker client maintains an explicit approved data list, for inbound and outbound data requests, in order to provide maximum protection for your network.
The default approved list provided by Snyk limits bi-directional communication as follows:
Inbound—Snyk.io is only allowed to fetch and view dependency manifest files and the Snyk policy file. No other source code is viewed, extracted, or modified under any circumstances. Additional files (.snyk files) might be checked in to support our patch mechanism and for any ignore instructions included in your vulnerability policy. For more information about these, see Manifest files.
Outbound—Git repo webhooks are set when you initially configure your Broker setup and are necessary to enable automatic Snyk scans that are triggered when new pull requests or merge events are submitted by your developers. Webhook notifications are delivered to Snyk via the Broker client for only events relevant to Snyk actions (push to branch, pull request opened), AND only when the event data also includes a dependency manifest file or a Snyk policy file.
All data, both in transit and at rest, is encrypted. Communication between the Broker client and the Broker server takes place over a secure WebSocket connection.
All requests and webhooks not included in this approved list are dropped.
Broker can also be used as a proxy between Snyk and your publically-accessible Git repos themselves. When implemented for public repos, every interaction between the Broker client and your Git repos is logged, thereby increasing your visibility and control over Snyk activity in those repositories and increasing data security.
The communication between Snyk, the Broker and your repo, as well as the data shared with Snyk, are further described step-by-step in How it works.
The Broker workflow comprises these main components:
Snyk SaaS backend (on Google Cloud Platform)/Broker server
Snyk Broker client (a Docker image running on a virtual machine behind your firewall)
Either a Git repo instance (Github Enterprise, Gitlab, Bitbucket Server) or Jira
Once implemented the components communicate similarly to the following diagram:
The Broker workflow is as described in the following steps:
Your organization’s IT administrator installs Snyk Broker client, with a Docker image, on a host that has access to the on-prem Git repo as well as outbound access to https://broker.snyk.io (the Broker server). It is preferable to install at least two instances for every integration planned for your deployment to support redundancy.
Your organization’s admin configures the Broker client by providing a personal access token or credentials (from your repo or Jira account as described in Retrieve a unique Broker client token). The Broker client uses those credentials to create webhooks on particular events performed against your Git repos (for example, when a PR is created), which trigger events on the Snyk backend to run the appropriate tests (scans). The token or credentials must have the appropriate scope (also as described in Retrieve a unique Broker client token) and be tied to a user or service account that has enough privileges to create these webhooks. These credentials remain within your perimeter at all times and are never sent to the Snyk SaaS backend. They are also obfuscated in the Broker client logs.
Snyk support enables the Broker functionality, exposing a Snyk token in the Project settings area of the snyk.io UI (and as further described in Retrieve a unique Broker client token). This token is used by the Broker client to authenticate itself whenever communicating with Snyk.
Your admin initiates the Broker with a simple Docker command, which “activates” the container/s.
When initiating, the Broker client container/s calls “home” (establishing contact with the Broker server at https://broker.snyk.io) using the TLS protocol (through port 443 out/HTTPS, which must be allowed by your firewalls) and establishes a WebSocket (2-way connection) over that open pipe. Each running Broker client container opens a persistent bi-directional WebSocket connection to the Snyk servers that facilitates ongoing communication with your on-premise repo. The Broker client also establishes communication with your Git repo with an HTTPS connection.
Once the connection is established, your developers can each access snyk.io to commence work, as outlined in the remainder of this process.
When a developer clicks Add projects from the Integrations area of snyk.io, snyk.io sends a request to the Broker server to fetch the list of repositories available (HTTPS).
The Broker server tunnels this request to the Broker client. The Broker client validates this request type against its approved list, and then relays that request to the customer’s Git repository only if included in that list.
The response is the list of repositories to which Snyk has access. Access is determined by the token that the admin created from the Git repo, and is also based on the permissions of the account itself that was used to create that token. The response list is then returned to the Snyk backend for display to the end user from the UI.
The user selects a repo (or repos) to import from the snyk.io UI and clicks Import.
Via the Broker server, snyk.io then sends the Broker client a request to fetch a manifest file/s from the repositories that the developer selected. As before, the Broker server tunnels this request to the customer's Broker client. The Broker client validates the request against the approved list, and then sends that request to the customer's Git repo.
The manifest file/s is then returned to the Snyk SaaS backend over a secure encrypted connection (Snyk uses TLS 1.2 protocol and above with strongest ciphers).
Snyk extracts the dependency tree for the repo and matches the dependencies against our vulnerability database.
In addition to the import and analysis, webhooks are also set (using HTTPS) on the specific repo to trigger on future events for new Pull Requests.
The dependency tree and findings are displayed in the Snyk UI, and the notification mechanisms, monitoring, reporting, Slack integration and more are enabled.
Whenever your developers submit new pull or merge requests, your Git notifies (HTTP) Snyk of that Pull Request by using the webhook that was set upon Import. The extraction mechanism is identical, but this time against the manifest file(s) in the PR, which is used to compare the delta with the base/branch.
The Broker client can be installed and configured for use with one or more Git repos, and also with your on-prem Jira when installed behind a firewall.
The connectivity to an on-prem Git repo is achieved by running the Snyk Broker (“Broker client”) inside the customer’s network on a host that has access to the on-prem SCM as well as outbound access to https://Broker.snyk.io.
The Broker is an open source project hosted at GitHub, published as a set of Docker images, and each configured for a specific integration. It is best to install the Broker client as a Docker image. Furthermore, you can customize the configuration by using the environment variables offered in the Docker images, also specific to each integration. For this reason, you should install separate, multiple instances of the client for different integration types to ensure proper configuration as well as redundancy (two GitHub instances in one deployment should be separate from a Bitbucket or Jira deployment for example).
To install, configure and roll out your Broker client and repo integration, follow these steps:
Prepare for deployment
Enable permissions for Snyk from your repo/Jira
Retrieve a unique Broker client token
Install and configure the client
Import projects to Snyk
Snyk Broker currently integrates with these systems:
Snyk.io fetches and views dependency manifest files in order to analyze and deliver vulnerability results. To get proper test results and to create Snyk projects, at least one of the following manifest files must be present in the tested folder (for integration with CLI), or in the repository (for integration with Git).
Gopkg.* &amp; vendor/vendor.json
package.json &amp; package-lock.json
package.json & yarn.lock
*.csproj, *.vbproj, *.fsproj
Your Broker client must be the most recent version in order to ensure that all of these manifest files are supported and found when importing projects to snyk.io.
In this article, you find the prerequisites and all the information you need to prepare Broker and configure your network for deployment.
Create a Snyk account and contact email@example.com to enable Broker integration for the organization for which you would like to implement the proxy. Each Broker is identified by a UUID token. Once Broker support has been enabled for your organization, you can access your unique Broker token on the organization’s Settings page from the Snyk UI, as described in Retrieve a unique Broker client token. This token is private, and must not be shared.
Install and set up Docker Hub from the client machine on which you want to install the Broker.
We recommend configuring:
at least two separate instances of the Broker client for every integration you plan on deploying, either on different hosts or installed via a Kubernetes system to ensure that you always have two (or more) instances running; for example, if you plan on deploying both GitHub and Jira, you should plan on installing for 4 instances, 2 of each to ensure redundancy.
separate Broker clients for every unique integration
Ensure each host designated for the Broker client installation has:
access to the on-prem Git deployment
an outbound TLS (443) connection to https://broker.snyk.io that is also allowed by any firewalls installed on your network. The Broker client needs to be accessible for webhooks coming from the Git repo. Alternatively, you can specify which port the Broker client listens on using the --port command line argument.
If you use a proxy server, ensure you configure it, and any firewalls, to allow the Broker client inbound and outbound access.
Traffic (very light) initiated from your network (i.e. webhooks) should be load-balanced across the broker instances for redundancy purposes; payloads transiting through WebSocket connection used to proxy the request to the broker server are small as well.
Traffic initiated from the Snyk (server) side always uses the latest Broker connection—we maintain a stack of the connections and always use the top one. If that Broker connection fails for whatever reason, Snyk removes it from the stack and uses the previous one for connection, and so on. All activity from our side (e.g. driven by recurring tests) appears on only one of your replicas at a time (the latest one to connect to us).
In practical terms, this means the amount of Snyk activity is proportional to the activity in your repos (or Jira) as that activity generates webhooks, which is distributed across all replicas.
In order to enable Snyk to find, monitor, fix and prevent, Snyk needs credentials to communicate with and send information to your repo.
Modern Gits grant various access rights by scoped authorization (auth) tokens to ensure that access to repo data and actions are managed. The various scopes associated to tokens grant specific ranges of action on repos.
snyk.io access rights to the Git repos are regulated by such Git access mechanisms and scopes. Therefore, you need to generate auth tokens or credentials for Snyk to perform tasks such as submitting fix pull requests or merge requests. Because tokens are generated by and associated to users (or accounts) that are then added as members of that given repo, the credentials must be generated from a user or service account that also has enough privileges to create the necessary webhooks and pull requests (usually admin-level).
Once you have generated a token, you set it only in the Broker configuration file during installation. Since the client runs from your private network, the credentials are never shared or sent to Snyk. Credentials (or tokens) never leave your network.
read access on a repo to perform an initial test (Find) and recurring tests (Monitor).
Note that while *not* recommended, Snyk can function in read-only mode, but would then not able to offer Prevent and Fix functionality.
write access on repo including webhooks to also perform pull request checks (Prevent—checking code whenever one of your developers submits a new pull request) and Pull Request creation (Fix and AutoFix—creating a pull request in order to apply a security fix). Snyk never writes directly to your repositories, but rather opens Pull Requests with suggested changes. Explicit approval of the Pull Request is always required before said suggestions are taken.
Assign permissions based on your integration as follows:
GitHub/GitHub Enterprise—see GitHub documentation for assistance in creating the necessary token: https://github.com/settings/tokens
Bitbucket server—Snyk needs to use the username and password credentials of a Bitbucket user in your organization. See Bitbucket documentation for assistance in creating a user with the necessary credentials: https://confluence.atlassian.com/bitbucket/grant-repository-access-to-users-and-groups-221449716.html
GitLab—Snyk needs to use an access token with API scope. See GitLab documentation for assistance in creating an access token with the necessary credentials: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html
Jira—Snyk needs user credentials with API access. When installing the client, if you wish to authenticate with an API token, then provide the username (usually an email address) as the JIRA_USERNAME and the API token as the JIRA_PASSWORD in the command line arguments. See Jira documentation for assistance in creating a user with the necessary credentials: https://confluence.atlassian.com/cloud/api-tokens-938839638.html
Create the token or user account with the permissions for the specific integration, as described above.
Copy and paste the credentials/token in a file on the desktop to use in the environment variables of the command line argument when you install the client: Install and configure the client.
In order to enable the Broker client to contact the Broker server (Snyk servers), you need to provide the client with Snyk credentials. Snyk credentials are granted with a Broker UUID token, which you can enable through our Support team.
For each organization in Snyk, you can only configure one instance of each integration type. For example, a single organization can integrate with one GitHub repo and also one Bitbucket repo. Similarly, you can run only one Broker for each integration type per organization. This means that any organization can have one integration for each of the different integration types from a single Snyk organization.
At the same time, a second, separate, integration can only be associated with a different organization in Snyk. For example, organization X can integrate with GitHub X and Bitbucket X. If you also work with GitHub Y repos, then you must associate those repos to any organization in Snyk other than organization X.
On the other hand, you can choose to use the same Git across multiple organizations in Snyk by simply using the same Broker token for all of the different organizations. For example, you can integrate Snyk organizations X, Y and Z with your single Git repo X.
To do this, first ensure the token has been created for the desired organization, and then create a new additional organization based on the original. When you do this, the token is cloned for the new organization and you can now enable the Broker for it.
Contact Snyk support to enable Broker functionality.
Once the Support team enables Broker for you, the fields in the Settings => Integrations area for that specific integration update and now display the Broker token. You need this token to install the Broker client.
To retrieve the token, from your snyk.io account navigate to Settings =>Integrations.
From the row for the integration you’re setting up, click Edit settings.
Since you have not yet installed and configured the client, the notification from this screen correctly displays “Could not connect to…”.
Copy and paste the credentials/token in a file on the desktop to use in the environment variables of the command line argument when you install the client: Install and configure the client.
Install the Broker client using the Docker image designed for the specific repo (or Jira). The relevant command line argument and environment variables are fully detailed per integration in the following location: https://hub.docker.com/r/snyk/broker/.
At this Docker Hub location you can find the following details and instructions:
Unique Docker images for each of the different available integrations. Use the command line arguments and environment variables relevant to the integration you are deploying.
A full description of the mandatory environment variables per integration type—from the README file, in the opening section for each of the different integrations, the mandatory variables are listed.
In addition, take special note of the following: For the hostname of your private repo/Jira deployment, use the URL of the repo without the https:// prefix, such as your.ghe.domain.com.
For the Broker client URL, enter the full URL of the Broker client that you installed on your network including the HTTP prefix and the port suffix. For example https://gitlab-broker.companyname:7341
Advanced installation instructions in cases when you prefer to build your own Docker image as part of the installation, providing an additional layer of security
Advanced configuration instructions for HTTPs, SCM certificates, and customized approved lists
Instructions for monitoring and troubleshooting, such as /healthcheck and /systemcheck
When installing the client, if you wish to authenticate with an API token, then provide the username (usually an email address) as the JIRA_USERNAME and the API token as the JIRA_PASSWORD.
Here’s an example of the parameters and values for Bitbucket:
docker run --restart=always \ -p 8000:8000 \ -e BROKER_TOKEN=secret-broker-token \ -e BITBUCKET_USERNAME=username \ -e BITBUCKET_PASSWORD=password \ -e BITBUCKET=your.bitbucket-server.domain.com \ -e BITBUCKET_API=your.bitbucket-server.domain.com/rest/api/1.0 \ -e BROKER_CLIENT_URL=https://my.broker.client:8000 \ -e PORT=8000 \ snyk/broker:bitbucket-server
Save the command-line arguments and values to copy and paste when you need to upgrade in the future.
The Broker token is as sensitive as user credentials. Do not share it with others or distribute it electronically.
Download the Broker Docker image from here: https://hub.docker.com/r/snyk/broker/ with the docker pull command. You must indicate a specific tag for the pull to succeed; indicate the integration platform as the snyk/broker tag, similar to the following example:
docker pull snyk/broker:github-com
Use the command line arguments for the integration you want to set up as detailed in our Docker README (same location), entering the correct values for the different environment variables, including the values you prepared and saved on the side: the token/credentials you set up for Snyk (Enable permissions for Snyk from your repo/Jira) and the Broker token (Retrieve a unique Broker client token).
Press Enter to run the command arguments.
As part of the output, “identifying Broker server” appears towards the end of the log results, indicating connection has been established.
Refresh the Snyk app.
If this is an installation for upgrade, you can continue working as before.
Select the organization with which you are integrating and navigate to Integrations where you can now see the Add projects button is enabled. For example, if you asked Support for GitLab integration, the relevant button is now activated.
To add projects for scanning, see Import projects to Snyk.
Snyk regularly updates the Broker client in order to provide new features, bug fixes and more. Our full list of versions and their release notes are available from GitHub as part of our Broker project. We also encourage you to subscribe to the RSS feed for that page to easily receive info about versions as they are released.
To upgrade any container, which is a running copy of its image, you need to kill it, delete its instance, and then replace its image (with the docker pull command) and run it again. Similarly, to upgrade your Broker client, which is also installed as a container, you must do the same.
Obtain a list of all Docker containers currently running on this host by running the command:
docker psA full list of all Docker containers that are currently running on this host and their container IDs is returned, including actively running Broker containers.
Kill and delete running the Broker containers: To kill all running Broker containers, run the following command, separating multiple container IDs with a space only:
docker kill <container_ID> <container_ID>
Now that the containers have stopped running, delete them by running the following command, separating multiple container IDs with a space only:
docker rm <container_ID> <container_ID>
Now, delete the Docker image that was originally downloaded when installing the Broker by running the following command, separating multiple image IDs with a space only:
docker rmi <image_ID> <image_ID>
Once you have killed all running containers and deleted the original image, install a new version of the Broker client. To do so, follow the instructions in Install and configure the client.
Once your Broker client is up and running, you can import projects and begin finding, monitoring, fixing and preventing vulnerabilities for your private repo deployment.
Log in to https://app.snyk.io
Select the organization that you’re using with your Broker.
Navigate to the Projects page and click on Add Projects.
When prompted, select the Git repo with which you integrated the Broker.
Repositories from your Brokered Git appear and you can now select the relevant projects for import.