Add ability to run performance benchmarks on engine PR

This issue has been tracked since 2022-09-22.

We should add and document the ability to run the full gamut of performance benchmarks on an engine PR.

This came up when I created a potential performance improving PR that works indiscriminately across the whole codebase (https://github.com/flutter/engine/pull/36221/files). We'd like to see the results of benchmark testing before landing it since its benefit is complicated by optimizations the compiler can take.

I talked with @dnfield and @christopherfujino and it sounds like there may be something close to this available where it is launched with led, but it may need to be cleaned up and documented.

keyonghan wrote this answer on 2022-09-23

Related bugs:

  1. recipes support to run a devicelab test: #111119
  2. framework tests that should run in engine: #83731
christopherfujino wrote this answer on 2022-09-23

So, we do have it wired up that you can run a "local" engine from an arbitrary engine commit:

The download is happening via the recipe_engine/cas module: https://flutter.googlesource.com/recipes/+/refs/heads/main/recipe_modules/flutter_deps/api.py#34, I'm not sure exactly how this works (@whesse do you know?)

When triggering an engine build via led, we upload to an experimental namespace in our storage bucket: https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine/engine.py#94. If we could add another property that told the flutter_deps.flutter_engine module code to download from the experimental namespace, I think we could have a workflow like:

  1. trigger a led build on your PR for the particular engine target you are interested in
  2. trigger a led build of the benchmark you are interested in, specifying you want to download experimental binaries and the commit from your engine PR, which would then download the PR binaries and pass them to the tool with --local-engine
More Details About Repo
Owner Name flutter
Repo Name flutter
Full Name flutter/flutter
Language Dart
Created Date 2015-03-06
Updated Date 2022-10-04
Star Count 145499
Watcher Count 3567
Fork Count 23393
Issue Count 11206

YOU MAY BE INTERESTED

Issue Title Created Date Updated Date