Visualise what's trending in your build process

This project contains the files that offer Buildtime Trend as a service. The Buildtime Trend library powers this service.

• production : https://buildtimetrend.herokuapp.com/ (Currently discontinued)

  Learn how to setup your own service on Heroku.

• development : https://buildtimetrend-dev.herokuapp.com/

  Used for development, expect frequent changes, accepts only a limited number of projects. Contact us if you'd like to test the development version. It's recommended to use the production version.

See instructions on the wiki on how to configure the service on Heroku.

• Launch a CherryPy instance hosting the service :

python service.py

Remark : By default it will attach to port 5000, use environment variable PORT to override this.

More configuration options are available.

• To enable a worker to handle the task queue (optional) :

utils/start_worker.sh

A Celery compatible task queue service is required to use the task queue, see the task_queue config parameter in the configuration options section.

• Index page
• Dashboard
• Service usage statistics
• Shield badges
• Process Travis CI build log

• path : /

Display a dashboard with Buildtime Trend charts

• path : /dashboard
• usage : /dashboard/repo_owner/repo_name?refresh=<refresh_rate>&timeframe=<timeframe_setting>
• parameters :
  • repo_owner : name of the Github repo owner, fe. buildtimetrend
  • repo_name : name of the Github repo, fe. service
  • refresh (optional) : enables auto-refreshing the dashboard charts. 0 = disabled (default), a positive integer is the number of minutes after which the charts are refreshed, a value of 10 will refresh the charts every 10 minutes. The refresh rate should at least be equal to maximum age of the Query cache, if it is less, the cache max age value will be used (typically, 10 min).
  • timeframe (optional) : the default timeframe can be set with this url parameter. Possible values : day, week, month, year. If it is not defined the default timeframe is used (week)
  • filter_* : set the default filter value. When this url parameter is defined, the corresponding filter dropdown will be set to this value and it will be applied to the queries of all charts and metrics. Available filters :
    • filter_build_matrix : Build matrix environment settings, usually a combination of language, language version and operating system : fe. python 2.7 linux
    • filter_build_result : Build job result, possible values : passed, errored, failed, ...
    • filter_build_trigger : What triggered the build job, possible values : pull_request, push (git push)
    • filter_branch : Repository branch that was build : fe. master

Remark : When visiting /dashboard (without parameters), an overview of all hosted projects is displayed.

Display a dashboard with Buildtime Trend service usage statistics

• path : /stats
• usage : /stats?timeframe=<timeframe_setting>
• parameters :
  • timeframe (optional) : the default timeframe can be set with this url parameter. Possible values : day, week, month, year. If it is not defined the default timeframe is used (week)

Generate a shield badges

• path : /badge
• usage : /badge/repo_owner/repo_name/badge_type/interval
• parameters :
  • repo_owner : name of the Github repo owner, fe. buildtimetrend
  • repo_name : name of the Github repo, fe. service
  • badge_type : type of badge, options :
    • latest : buildtime of last build job
    • avg : average buildtime of buildjobs in period set by interval (default)
    • jobs : number of build jobs in period set by interval
    • builds : number of builds in period set by interval
    • passed : percentage of successful build jobs during interval
    • last_fail : days since last failed build job
  • interval : time interval, options :
    • week (default) : events of last week (last 7 days)
    • month : events of last month (last 30 days)
    • year : events of last year (last 52 weeks)

• : /badge/buildtimetrend/service/latest
• : /badge/buildtimetrend/service/avg/week
• : /badge/buildtimetrend/service/avg/month
• : /buildtimetrend/service/avg/year
• : /badge/buildtimetrend/service/jobs/week
• : /badge/buildtimetrend/service/jobs/month
• : /buildtimetrend/service/jobs/year
• : /badge/buildtimetrend/service/builds/week
• : /badge/buildtimetrend/service/builds/month
• : /buildtimetrend/service/builds/year
• : /badge/buildtimetrend/service/passed/week
• : /badge/buildtimetrend/service/passed/month
• : /badge/buildtimetrend/service/passed/year
• : /badge/buildtimetrend/service/last_fail

Loads a Travis CI build log file, processes it and sends the data to Keen.io.

• path : /travis
• usage : /travis/<repo_owner>/<repo_name>/<first_build>/<last_build>
• parameters :
  • repo_owner : name of the Github repo owner, fe. buildtimetrend
  • repo_name : name of the Github repo, fe. service
  • first_build : Travis CI build number of the first build to be processed
  • last_build : (optional) Travis CI build number of the last build to be processed

If last_build is not defined, only a single build will be processed, namely the one defined by first_build. If last_build is defined, it turns into batch mode : multiple builds will be processed : all builds from first_build to and including last_build will be scheduled to be processed. To limit the load on the worker, a delay is added to every next build. F.e. build #1 will be executed immediately, build #2 after x seconds, build #3 after x*2 seconds, build #n after x*(n-1) seconds. By default the delay is 3 seconds, but this can be configured with the multi_import.delay setting in config_service.yml or BTT_MULTI_DELAY env variable. The maximum number of builds that can be imported at once in batch mode is by default limited to 100. This can be configured with the multi_import.max_builds setting in config_service.yml or BTT_MULTI_MAX_BUILDS env variable.

OR

• payload : Travis CI notification payload, more info in the Travis CI documentation

To trigger the service at the end of a Travis CI build, add this to your .travis.yml file:

    notifications:
      webhooks:
        # trigger Buildtime Trend Service to parse Travis CI log and send result to Keen.io
        - https://buildtimetrend.herokuapp.com/travis

When Buildtime Trend Service is triggered by a Travis CI notification, it will get the necessary parameters (repo name and build number) from the payload that is passed by Travis CI. This will trigger loading and parsing the Travis CI log of the corresponding build, the analysed data is stored in the Keen.io database.

Add a configfile named config_service.yml based on config_sample.yml to configure the way the service behaves.

• denied_repo : defines which repos are not allowed by the service. If the denied_repo setting is not defined, all repos are allowed (unless allowed_repo is defined, see below). If a substring matches the repo name, it is denied, so fe. my_name will disallow my_name/my_first_repo and my_name/another_repo. A complete repo name is valid as well.

Remark : Setting denied_repo takes precedence over allowed_repo, if a repo name matches a rule in denied_repo setting, the repo will be denied, even if it matches a rule in allowed_repo. Multiple entries are allowed, fe. :

buildtimetrend:
  denied_repo:
    - "my_name" # denying all repo names that contain my_name
    - "another_name/some_repo" # denies this specific repo

• allowed_repo : defines which repos are allowed by the service. If the allowed_repo setting is not defined, all repos are allowed (unless denied by the denied_repo setting). If substring matches the repo name, it is allowed, so fe. my_name will allow my_name/my_first_repo and my_name/another_repo. A complete repo name is allowed as well.

Remark : Setting denied_repo takes precedence over allowed_repo, if a repo name matches a rule in denied_repo setting, the repo will be denied, even if it matches a rule in allowed_repo. Multiple entries are allowed, fe. :

buildtimetrend:
  allowed_repo:
    - "my_name" # allowing all repo names that contain my_name
    - "another_name/some_repo" # allows this specific repo

• travis_account_token : define to enable checking Travis CI notification Authorization header. More info on Travis CI Webhook Authorization and where to find the Account token : http://docs.travis-ci.com/user/notifications/#Authorization-for-Webhooks

Remark : the account token should be the one of the user who created the repo on Travis CI.

It can also be defined with the TRAVIS_ACCOUNT_TOKEN environment variable.

• loglevel : defines loglevel, possible values : DEBUG, INFO, WARNING (default), ERROR It can also be defined with the BTT_LOGLEVEL environment variable.

• task_queue : configures using a task queue, powered by Celery, to process background tasks by a worker. Celery supports different kinds of backend task queues, which can be configured :

buildtimetrend:
    task_queue:
        backend: "" # backend type fe. amqp, redis, ...
        broker_url: "" # backend instance url, format : transport://userid:password@hostname:port/virtual_host

If task_queue is defined, this will be enable using a worker to execute heavy tasks. To disable using a worker, remove the task_queue directive from the config file.

The broker_url can also be defined with several environment variables :

• BTT_AMQP_URL : set a AMQP based task queue instance. (fe. amqp://localhost/)
• BTT_REDIS_URL : set a Redis based task queue instance. (fe. redis://localhost/)
• RABBITMQ_BIGWIG_URL : environment variable set when creating a RabbitMQ Bigwig instance on Heroku
• CLOUDAMQP_URL : environment variable set when creating a Cloud AMQP RabbitMQ instance on Heroku
• REDISGREEN_URL : environment variable set when creating a Redis Green instance on Heroku

The environment variables are checked in this order, the first one that is defined, will be used. The backend type (amqp, redis, ...) associated with the environment variable will be set automatically.

• multi_import : configures batch mode when importing build timing data.

buildtimetrend:
    multi_import:
        max_builds: 100 # maximum number of builds allowed in one batch
        delay: 3 # number of seconds between the start of each build

Setting max_builds is 100 by default, and can also be set with the BTT_MULTI_MAX_BUILDS environment variable. Setting delay is 3 seconds by default, and can also be set with the BTT_MULTI_DELAY environment variable.

• data_detail : level of detail when storing build job data, possible values :
  • "minimal" : only get build(job) data, with total time, no detailed (sub)stages data is stored
  • "basic" : minimal + substages grouped into stages (for future use : currently equals minimal)
  • "full": basic + substages and stages details are stored (default)
  • "extended" : full + also store data of custom timestamps (for future use : currently equals full)

It can also be defined with the BTT_DATA_DETAIL environment variable.

• repo_data_detail : level of detail when storing build job data, but defined per repository. See data_detail for possible values. If a setting matches the repository name, the corresponding setting is used. Substrings are allowed, if a substring is part of a repo name, the corresponding setting is used. If no (sub)string matches a repo, the default setting data_detail is used.

Be carefull with substrings, if a repository name matches more than one entry, only the first one that matches will be used (not necessarily in the order as listed in the config file)

Don't use '-' to prepend the different repositories

buildtimetrend:
  repo_data_detail:
    "my_name": "minimal" # sets `data_detail` to `minimal` for all repo names that contain my_name
    "another_name/some_repo": "basic # set `data_detail` to `basic` for this repositiry

• buildtimetrend : Buildtime Trend library
• cherrypy : CherrPy A Minimalist Python Web Framework, making the API available as a web service
• celery : Celery : Distributed Task Queue

See requirements.txt for package names and versions.

Please report bugs and add feature requests in the Github issue tracker.

If you want to contribute to make Buildtime Trend even better, check out the contribution page. We are looking for testers, developers, designers, ... and what more. Contact us if you want to help out.

You can support the project by making a donation. The donations will help pay for the hosting and support further development.

For an overview of who contributed to create Buildtime trend, see Credits.

Website : https://buildtimetrend.github.io/service

Mailinglist : Buildtime Trend Community

Follow us on Twitter, Github and OpenHub.

Copyright (C) 2014-2016 Dieter Adriaenssens ruleant@users.sourceforge.net

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see http://www.gnu.org/licenses/.