Overview
In order to monitor query performance within Firebolt effectively, it is convenient to have this data made available via a dashboard. In order to help customers set up a monitoring dashboard, we have provided a means to connect to Firebolt via Grafana and a sample dashboard that can be used to perform monitoring of the system using the information_schema.query_history and information_schema.running_queries tables.
Getting Started
Clone the firebolt-qh-monitor repository.
Configure a Grafana instance that you wish to use with a user and API key and install the Infinity plugin.
Run the provided Python script to create Grafana datasource
Clone the repository
Clone the repository using the following command:
## httpsgit clone https://github.com/firebolt-db/firebolt-qh-monitor.git## sshgit clone git@github.com:firebolt-db/firebolt-qh-monitor.git
Configure Grafana Instance
The dashboard code uses the Grafana Infinitiy Plugin . The Infinity plugin allows us to easily create visuals via the Firebolt api's JSON response.
Instructions for installation to an existing Grafana instance are found under the installation page.
Alternatively, or if you do not have a Grafana instance, you can create one using Docker in order to test configuring the dashboard:-
docker run -d -p 3000:3000 --name=grafana -e "GF_INSTALL_PLUGINS=yesoreyeram-infinity-datasource" grafana/grafana-ossOnce you have Grafana running, you will need to create a Service Account with the appropriate permissions to create a datasource and dashboard, and generate a Service Account Token in order to use the Python scripts provided. Documentation for setting this up can be found here.
Once you have created the Service Account Token, this needs to be set as the GRAFANA_API_TOKEN environment variable when running the Python set up scripts.
Running the Python scripts to create the Grafana Dashboard
A python script has been provided in the repository in order to allow you to configure Grafana data sources and dashboards to monitor Firebolt. Prerequisites are in the requirements.txt file and so can be installed using pip from the root of the repository:
pip install -r ./grafana_monitor/datasource_config/requirements.txtThe script requires the following environment variables to be set:-
export FIREBOLT_USER=[your-firebolt-user]export FIREBOLT_PASSWORD=[your-firebolt-password]export GRAFANA_BASE_URL=[your-grafana-instance-url]export GRAFANA_API_TOKEN=[your-grafana-service-account-token]
N.B. If you already have generated a Firebolt Auth Token, this can be passed into the scripts via the --firebolt-access-token parameter.
Datasources
To configure a data source through the UI you will need to enter the following:
Name - whatever you wish to call the connection
-
Authentication:
Auth Type - Bearer Token
Bearer Token - A valid token from the Firebolt auth api (more on this below)
Allowed Hosts - the engine url(s) for running queries; these can be retrieved from the Firebolt UI Databases page as shown below.
To simplify this, we have created a module within the Python script to create this for you.
python ./grafana_monitor/datasource_config/datasource_config.py create-datasource --helpUsage: datasource_config.py create-datasource [OPTIONS] DATASOURCE_NAMEDATABASE_NAMEENGINE_ENDPOINTS...Arguments:DATASOURCE_NAME [required]DATABASE_NAME [required]ENGINE_ENDPOINTS... [required]Options:--is-default / --no-is-default [default: no-is-default]--firebolt-access-token TEXT--help Show this message and exit.
As an example, passing in two engines as a list (N.B. do not pass in the http prefix.)
export FIREBOLT_USER=[your-firebolt-user]export FIREBOLT_PASSWORD=[your-firebolt-password]export GRAFANA_BASE_URL=[your-grafana-instance-url]export GRAFANA_API_TOKEN=[your-grafana-service-account-token]python ./grafana_monitor/datasource_config/datasource_config.py create-datasource MyFirebolt my_firebolt_database your-engine-name_1.firebolt-account-name.region-id.app.firebolt.io your-engine-name_2.firebolt-account-name.region-id.app.firebolt.io --is-default
Once created the data source should be visible within the data sources.
N.B. sometimes when creating a data source via the Grafana api, these are not enabled until you manually go in to the data source and click save; if the dashboard is not working, this is a good first test. Once manually saved you will need to rerun the dashboard creation.
Refreshing the Firebolt Auth Token
As we are using the API to connect, and passing authentication via an Auth Token stored in the data source, this will need refreshing within 12 hours.
To do this, we can run the python script as follows:-
export FIREBOLT_USER=[your-firebolt-user]export FIREBOLT_PASSWORD=[your-firebolt-password]export GRAFANA_BASE_URL=[your-grafana-instance-url]export GRAFANA_API_TOKEN=[your-grafana-service-account-token]python grafana_monitor/datasource_config/datasource_config.py update-datasource-bearer-token [DATA_SOURCE_NAME] [DATABASE_NAME]
This can then be scheduled to run on a schedule via any mechanism you choose, such as an AWS Lambda function invoked by AWS Eventbride, or an Airflow DAG etc.
N.B. the Auth Token is stored securely as part of the Grafana data source, but anyone who has access to the data source will be able to run queries against Firebolt.
Creating the Dashboard
A sample dashboard has been configured and included in the repository as a JSON file. You can use the python script to deploy this dashboard with the previously configured data source and the engine endpoint you want to deploy the dashboard for; by default if you do not pass in the --dashboard-name parameter, the dashboard will be name Firebolt_Monitor.
export GRAFANA_BASE_URL=[your-grafana-instance-url]export GRAFANA_API_TOKEN=[your-grafana-service-account-token]python grafana_monitor/datasource_config/datasource_config.py create-dashboard [DATA_SOURCE_NAME] [ENGINE_ENDPOINT] --dashboard-name [YOUR_DASHBOARD_NAME]
This will create the dashboard, which will be visible in your Grafana instance:-