chromium_build_stats/

This is an application designed to collect and analyze build/compile stats.

go/cbs-doc

Deign Doc: Chromium build time profiler

How to:

See infra/go/README.md for preparation.

to compile

  $ make build

to run locally with dev_appserver (note: no service account available, so you couldn't fetch file from gs://chrome-goma-log)

   $ (cd app; dev_appserver.py app.yaml)

to deploy to production

  $ make deploy-prod

and need to migrate traffic.

NOTE: Check ninja trace data after deploy. If it's not accessible, you must forget to generate trace-viewer contents (See the first item of this how-to). Re-generate it and deploy again.

to run test

  $ make test

to read go documentation

  $ godoc <package>
  $ godoc <package> <symbol>

(or

  $ godoc -http :6060

and go to http://localhost:6060 )

Operation for BigQuery Table

Setup

1. Make Dataset

$ bq --project_id=$PROJECT mk ninjalog

1. Make table

# Set 2 year expiration.
# This is for log table from buildbot.
$ bq --project_id=$PROJECT mk --time_partitioning_type=DAY \
    --time_partitioning_expiration=$((3600 * 24 * 365 * 2)) ninjalog.ninjalog

# This is for log table from chromium developer.
# Set ***540 days*** expiration.
$ bq --project_id=$PROJECT mk --time_partitioning_type=DAY \
    --time_partitioning_expiration=$((3600 * 24 * 30 * 18)) ninjalog.user

1. Update schema

$ make update-prod # or `make update-staging`

ninja log upload from user

Ninja log is uploaded from user too. Upload script is located in depot_tools.

example query

link to query editor

1. Find time consuming build tasks in a day per target_os, build os and outputs

SELECT
  (
  SELECT
    value
  FROM
    UNNEST(build_configs)
  WHERE
    key = "target_os") target_os,
  os,
  SUBSTR(ARRAY_TO_STRING(log_entry.outputs, ", "), 0, 128) outputs,
  TRUNC(AVG(log_entry.end_duration_sec - log_entry.start_duration_sec), 2) task_duration_avg,
  TRUNC(SUM(log_entry.end_duration_sec - log_entry.start_duration_sec), 2) task_duration_sum,
  TRUNC(SUM(weighted_duration_sec), 2) weighted_duration_sum,
  COUNT(1) cnt
FROM
  `chromium-build-stats.ninjalog.user`
WHERE
  (_PARTITIONTIME >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
    -- This is for streaming buffer.
    OR _PARTITIONTIME IS NULL)
GROUP BY
  target_os,
  os,
  outputs
ORDER BY
  weighted_duration_sum DESC