Skip to main content

Test 🧪

Run tests in a Dart or Flutter project with very_good test.

By default, this command will optimize your tests to run more efficiently by grouping them into a single entrypoint (see this issue).

Usage​

very_good test [arguments]
-h, --help                            Print this usage information.
    --coverage                        Whether to collect coverage information.
-r, --recursive                       Run tests recursively for all nested packages.
    --[no-]optimization               Whether to apply optimizations for test performance.
                                      Automatically disabled when --platform is specified.
                                      Add the `skip_very_good_optimization` tag to specific test files to disable them individually, or use --exclude-optimization to exclude them by path.
                                      (defaults to on)
    --exclude-optimization=<glob>     A glob which will be used to exclude matching test files from the optimized bundle (e.g. 'test/integration'). Excluded files still run, as their own test suites. Can be passed multiple times.
-j, --concurrency                     The number of concurrent test suites run.
                                      (defaults to "4")
-t, --tags                            Run only tests associated with the specified tags.
    --exclude-coverage                One or more space-separated globs, relative to the package root, which will be used to exclude files that match from the coverage (e.g. '**/*.g.dart **/*.freezed.dart').
-x, --exclude-tags                    Run only tests that do not have the specified tags.
    --min-coverage                    Whether to enforce a minimum coverage percentage.
                                      Implicitly enables coverage collection when used alone.
    --show-uncovered                  Whether to show uncovered lines when coverage is below 100%.
                                      Implicitly enables coverage collection when used alone.
    --test-randomize-ordering-seed    The seed to randomize the execution order of test cases within test files.
    --update-goldens                  Whether "matchesGoldenFile()" calls within your test methods should update the golden files.
    --force-ansi                      Whether to force ansi output. If not specified, it will maintain the default behavior based on stdout and stderr.
    --platform                        The platform to run tests on. For Flutter tests, this can be "chrome", "vm", "android", "ios", etc. For Dart tests, this can be "chrome", "vm", etc.
    --dart-define=<foo=bar>           Additional key-value pairs that will be available as constants from the String.fromEnvironment, bool.fromEnvironment, int.fromEnvironment, and double.fromEnvironment constructors. Multiple defines can be passed by repeating "--dart-define" multiple times.
    --dart-define-from-file=<use-define-config.json|.env>
                                      The path of a .json or .env file containing key-value pairs that will be
                                      available as environment variables. Multiple defines can be passed by
                                      repeating "--dart-define-from-file" multiple times. Entries from
                                      "--dart-define" with identical keys take precedence.
    --collect-coverage-from=<imports|all>
                                      Whether to collect coverage from imported files only or all files.
                                      (defaults to "imports")
    --report-on=<lib/>                Optional file paths to report coverage information to.
                                      This should be paths relative to the current working directory.
                                      Can be passed multiple times.
    --run-skipped                     Run skipped tests instead of skipping them.
    --flavor                          Build a custom app flavor as defined by platform-specific build setup. Supports the use of product flavors in Android Gradle scripts, and the use of custom Xcode schemes.
    --fail-fast                       Stop running tests after the first failure.
    --timeout=<seconds>               Maximum seconds to let tests run before killing the process.
                                      Useful when tests hang due to an unbounded pumpAndSettle() call.
    --file-reporter=<name:path>       Enable an additional reporter writing test results to a file.
                                      Should be in the form <name>:<path> (e.g. "json:reports/tests.json").
    --shard-index=<index>             The 1-based index of the shard to run.
                                      Must be used together with --total-shards.
                                      Requires optimization to be enabled.
                                      When omitted, no sharding is applied.
    --total-shards=<count>            Split the test suite into this many shards and run only the one
                                      selected by --shard-index.
                                      Useful to parallelize tests across multiple CI runners.
                                      When omitted, no sharding is applied.

Run "very_good help" to see global options.

Machine-readable test reports​

If you need a machine-readable summary of your test results (for example to feed a CI tool or convert to JUnit XML), use --file-reporter. It maps directly to the --file-reporter flag from flutter test and dart test, and writes the report to a file instead of stdout — so it stays compatible with very_good test's optimization and progress rendering.

# Emits reports/tests.json alongside the normal test output.
very_good test --coverage --file-reporter json:reports/tests.json
info

--reporter and --machine are intentionally not exposed: they replace stdout with a machine-readable stream, which prevents very_good test from rendering progress and picking up failures. --file-reporter gives you the same machine-readable content without that trade-off.

tip

For Dart projects, use very_good dart test instead.

All test parameters and options work identically with this command.

Sharding tests across CI runners​

When the test phase is your CI bottleneck, split the suite across several runners with --shard-index and --total-shards. Each runner generates its own optimized entrypoint containing only its slice of the test files, so the shards together run every test exactly once.

jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3]
    steps:
      - run: very_good test --shard-index ${{ matrix.shard }} --total-shards 3

Test files are sorted and then dealt out round-robin, which keeps the partition deterministic: the same suite always produces the same shards, on every machine. Files kept out of the bundle, whether tagged skip_very_good_optimization or matched by --exclude-optimization, are sharded as well, so they do not run on every runner.

A shard with no test files (more shards than test files) succeeds without running anything, so an oversized matrix will not fail your build. The same applies to a shard whose tests are all filtered out, for example by --exclude-tags golden on a shard that only holds golden tests.

When combined with --recursive, each package is sharded independently, so a given runner executes a slice of every package. Balance therefore degrades when a workspace contains many packages with few tests each.

caution

Sharding cannot be combined with --min-coverage, and a min_coverage set in very_good.yaml is ignored while sharding. Each shard only exercises a subset of the codebase, so its coverage is not representative of the whole suite. Collect coverage per shard with --coverage, merge the resulting lcov reports once every shard has finished, and enforce the threshold on the merged report in a separate job. A shard without tests still writes an empty coverage/lcov.info; pass --ignore-errors empty to lcov when merging so it is accepted.

info

Sharding requires the test optimizer, so it is rejected whenever the optimizer is off: with --no-optimization, --platform, --update-goldens, or when targeting specific test files.

Passing Flutter specific arguments​

The dart test and flutter test commands expose more arguments than those available through very_good dart test or very_good test. However, you can use the argument terminator -- to pass additional arguments directly to the underlying test commands. Any arguments after -- will be forwarded to dart test or flutter test, making all their options available.

For example, if you wish to run flutter test --no-track-widget-creation you can simply use very_good test -- --no-track-widget-creation.

Tests without pub install​

Unlike dart test or flutter test, very_good test will always run your tests without installing the project's dependencies (i.e. --no-pub flag).

This is an optimization done by the CLI because dependency installation is usually run once after cloning the repository. Conversely, running tests locally is usually done many times and it's often unnecessary to re-install dependencies prior to each test run.

If you need to install dependencies before running the tests with very_good_cli, be sure to run very_good packages get first.

Golden tests​

Golden tests are tests that compare the output of a test to a "golden" file. If the output of the test does not match the golden file, the test will fail. These usually invoke the matchesGoldenFile matcher, which supports specifying a GoldenFileComparator to customize the comparison by setting the top-level property goldenFileComparator, for example to accept a certain amount of difference.

Very Good CLI supports golden tests out of the box, with and without a custom goldenFileComparator. This means that no change is required to your test code to run them with very_good test.

For an example on specifying a custom GoldenFileComparator that accepts a certain amount of difference (toleration threshold), refer to the goldenFileComparator Flutter documentation.

File-level annotations​

package:test reads @Skip, @Tags, @Timeout, @TestOn, @OnPlatform and @Retry only from the metadata of a file's first directive. The optimizer forwards them to the group that wraps each bundled file, so an optimized run behaves the way the same suite does under plain dart test or flutter test.

@Tags(['integration'])
@Timeout(Duration(minutes: 5))
library;

import 'package:test/test.dart';

The file above keeps its five minute timeout inside the bundle, is skipped when dart_test.yaml marks integration as skip:, and is left out by --exclude-tags integration.

Arguments are forwarded as source text into a file that imports only package:test and dart:core, so an annotation naming anything else, such as @Timeout(kSlowSuite), is left out with a warning instead of breaking the bundle. Inline the value to keep the annotation.

@TestOn cannot save a file that imports something platform-specific, such as dart:html, since every optimized file is imported regardless of platform. Keep those files out with the skip_very_good_optimization tag or --exclude-optimization.

Skip optimization for specific tests​

By default, all tests run with optimizations enabled; use the --no-optimization flag to disable globally, or add the skip_very_good_optimization tag to specific test files to disable them individually.

The tag needs a directive such as library; beneath it, the only place package:test reads metadata from.

@Tags(['skip_very_good_optimization'])
library;

import 'package:test/test.dart';

To exclude tests by path instead of tagging each file, pass one or more globs to --exclude-optimization, or list them under optimization.exclude in very_good.yaml:

very_good test --exclude-optimization test/integration
# very_good.yaml
test:
  optimization:
    exclude:
      - test/integration

Globs are matched against each test file's path relative to the package root, and they match everything nested underneath what they name, so test/integration excludes every test in that directory. See optimization for the glob syntax details.

Excluded files are left out of the optimized bundle but still run, each as its own test suite, in their own isolate. Exclude a file when it cannot share an isolate with the rest of the suite, for example because it mutates global state or installs its own bindings. File-level annotations do not need an exclusion, as the bundle forwards them.

Prefer narrow globs. Every excluded file becomes its own suite with its own VM startup, so excluding hundreds of files gives up most of what the optimization buys you.

Exclusions only matter while optimization is on. --no-optimization, --platform, --update-goldens, and targeting specific test files all disable optimization entirely, which makes the globs a no-op.

Configuring defaults with very_good.yaml​

To avoid repeating flags every time you run very_good test locally or on CI, you may create a very_good.yaml file at the root of your project. The test section accepts the same names as the CLI flags in snake_case (e.g. --min-coverage becomes min_coverage). Values from very_good.yaml are used as defaults; anything you pass on the command line takes precedence.

# very_good.yaml
test:
  min_coverage: 100
  exclude_coverage: '**/*.g.dart'
  report_on:
    - lib/
  dart_define:
    - FLAVOR=development
  file_reporter: json:reports/tests.json
  optimization:
    exclude:
      - test/integration

With the file above, running very_good test behaves the same as running very_good test --min-coverage 100 --exclude-coverage '**/*.g.dart' --report-on lib/ --dart-define=FLAVOR=development. You can still override any of these values on the command line, for example very_good test --min-coverage 90 to lower the coverage threshold for a single run.

To customize the defaults for very_good dart test, use the dart.test section. It accepts the same options as test (minus the Flutter-only flags such as update_goldens, flavor, timeout, dart_define, and dart_define_from_file) plus check_ignore.

# very_good.yaml
dart:
  test:
    min_coverage: 100
    exclude_coverage: '**/*.g.dart'
    report_on:
      - lib/
    file_reporter: json:reports/tests.json

The very_good.yaml file is looked up starting from the directory where the command runs and walking up through its ancestors. The closest file wins; configuration from ancestor directories is not merged. This lets a single very_good.yaml at the repository root apply to commands run from any nested package.