Hardware tests are often more expensive to scale, so tests are targeted accordingly

• Hardware tests are often more expensive to scale, so tests are targeted accordingly
• Some tests are too expensive/too long to run as presubmit/CQ, so they are run in CI
• If a CI run fails, it notifies the list of Cls that were included. Running CI more often means fewer people get bundled in that warning Email, and probably confused.
• Running tests on real hardware is essential for projects like ChromeOS or Fuchsia, and useful for some projects like Chrome that use hardware to find some bugs, timing or power use regressions, but requires a dedicated team to roll out and maintain the hardware
• You now also need to deal with both software and hardware flakes. Retrying tests once helps, after that you need people to look into why they flake or failed.
• Aggressively go after flaky software tests and get them fixed or removed. A flaky test is worse than no test.

CQ/CI costs and issues