AOM: User Manual for doing AOM CTC development in AWCY
Author: vibhoothi <mindfreeze at videolan dot org>
This wiki page serves as a guide on how to use AWCY for AOMedia use cases, specifically,
- Submitting a CTC run with a CTC configurations
- Submitting a run with a different build configuration
- Generation of CTC results for existing or submitted AWCY runs
- Downloading a job for browsing/analysing locally
- Common FAQs of AWCY usage
AWCY (AreWeCompressedYet) is a platform which is currently designed for doing compression evaluation with specific configurations to compare Codec A with Codec B.
The canonical URL of AWCY is located at https://beta.arewecompressedyet.com
The source code is split as frontend and backend,
- Frontend. https://github.com/xiph/awcy
- Backend. https://github.com/xiph/rd_tool
The current AOM-CTC[1][2] consists of videos spanning from 270p to 4K HDR and with multiple coding configurations (e.g, Random-Access, Low-delay, All-Intra, adaptive-streaming).
- Log in to AWCY with the given API Password [Refer to FAQ-Q1 if you want to obtain one]
- Click on "Submit New Job" under the "Active Runs" tab.
- Fill-in run-information
- Submit the job
- Make sure the target commit hash is inside the tracking repositories in AWCY [Refere to FAQ-Q2 to add your upstream to AWCY tracker for a repository].
Note: Currently, the commit hash by default is empty, and on-submission of no-commit reference will cause a runtime failure.
- Add the required CLI and build options to configure the run. (Refer #submitting-a-custom-build-run for more information)
- Enter either your IRC nick, or name, or email address.
Note: Email notification will be coming from
<[email protected]>; Make sure you whitelist this email. For more information on the current email notification frequency, please refer to FAQ-Q3.
- If native CTC QPs are preferred, leave the "custom qualities column" blank. If you prefer a custom set of QP value/values, please provide a "space-separated values" [e.g.,
120 130 140 150 160] - Choose the desired CTC-Sets one by one. If the mandatory job is for a given configuration(s), please choose
mandatoryas the set value. While, if All sets in the AOM-CTC are required for testing, please chooseAll.
- Choose the desired CTC configuration [RA, LD, AI] one by one. If All configuration is chosen then the RA job will be Sequential GOP instead of parallel GOP. If Parallel-GOP is preferred, then manual selection would be required.
For submitting an AOM-CTC regular run, users would need to utilise the CTC configuration section during the submission of the run. This includes selecting the required CTC configurations along with target sets (whether it is subset of video-sets, or mandatory sets, or all the videos).
For submitting an AOM-CTC Adaptive streaming run, users would need to be careful and need to do two things,
a) Choose "set" as aomctc-a1-4k-as
b) Choose "Encoder" as AVM Adaptive Streaming (AS)/AVM Adaptive Streaming GOP Parallel.
Later, provide the commit ID, runID, and other parameters as regular runs.
Currently, it is not possible to submit both Regular and Adaptive-streaming together. We require two separate AWCY submissions to generate Regular and AS results.
For submitting an AWCY run with custom encode specification (e.g., disabling a tool or changing CMAKE config). Please specify the Build options during the run submission.
Example of extra build options
For submitting an AWCY run with custom runtime options (CLI flags). Please specify the options during the run submission.
Example of extra runtime options
In the current AWCY, anyone can generate CTC results [Please refer to FAQ-Q4 to know the current CTC version deployed, and respective anchor runs to be used]. AWCY can also generate partial CTC-XLSM results.
In the current deployment, a lightweight CSV export is also possible for a given AWCY run.
For generating the results, it is not mandatory to wait till the job is finished, the AWCY is capable of producing partial CTC results and the XLSM/CSV version of the result will be available on request.
Note: Currently CTC Excel file generation takes ~3 minutes to generate and download the file, please keep the current AWCY window opened during generation.
For a given run, users can download the full run data (even when running), as a zip.
Currently, the AWCY supports the generation of Convex-hull in the Frontend for quick visualisation for current AOM-CTC.
- Select the current AS Anchor Run.
- Choose the desired target job.
- Wait for a minute to complete generation of the Convex-hull Curves
- Once the BD-Rate is generated in the Table, try to Select the Convex-hull variant of the clip from the Clip drop-down.
In AWCY, we have a dedicated service which creates a nighty run for at every night ~00:30 UTC+0:00.
- Current HEAD commit from AVM on that day
- AOM-CTC Mandatory Set
- 17 Frames
- For videos >2K resolution (A1/A2/B1), we run the encodes with
--tile-columns=2 --threads=3 --row-mt=0[3 Threads]
- ((8 A1 (4K) Videos + 20 A2 (2K) Videos + 10 B1 (2K) Videos) * 3 threads + (9 B2 (~2K) + 4 A5 (270p) + 6 A4 (360p) + 8 A3 (720p)) * 1 threads) * 6 QPs ~ 846 Slots of the AWCY machine, around 7.5 machines in-use.
- Currently, it is floating between 15-20 hours for the nightly run depending on the CTC Version in-use. [Updated on 20231217]
TO-Be-Filled
Please email with official AOMedia ID to mindfreeze at videolan dot org, and CC thomas.daede at vimeo dot com
Please email with official AOMedia ID to mindfreeze at videolan dot org, and CC thomas.daede at vimeo dot com
Currently, in AWCY, we support notifying the user when a) A set for a given configuration is completed. b) A configuration for a given configuration is completed. c) When all the run is completed. d) When a job is failed or cancelled.
AOM-CTC version: CTCv4.0
Regular CTCv4.0 Anchor: https://beta.arewecompressedyet.com/?job=AVM-Research-v4.0.0-Regular-All-CTCv4
Adaptive-Streaming CTCv4.0 Anchor: https://beta.arewecompressedyet.com/?job=AOM-CTCv4.0-AS-Anchor
Please navigate to Debug Menu and click on get_machine_usage.json to get the exact specifics on a slot-by-slot with a machine-by-machine basis.
Currently AWCY have 12 Runners in production, All the runners are made of the same configuration.
They are listed below,
CPU: Intel Xeon Platinum 8280 (112) @ 2.700GHz [56 Cores, 112 Threads]
RAM: 755GB
BIOS Configuration,
Hyper-Threading : Enabled
Hardware P-States : Disabled
Package C-State : C0/C1 state
Processor C6 : Disabled
Kernel Arguments: intel_pstate=disable intel_idle.max_cstate=0 numa_balance=disable nml_watchdog=0 nosoftlockup hpet=disablemce=off tsc=reliable memory_corruption_check=0
OS: Ubuntu 22.04.3 LTS x86_64 with 5.15.0-78-generic
Serverboard: https://www.intel.com/content/www/us/en/products/sku/89015/intel-server-board-s2600wft/specifications.html
Disk: 512GB SSD [INTEL SSDSCKKI51]
[1]: CTCv4.0, CWG-C097, https://groups.aomedia.org/g/wg-codec/files/InputDocuments/C2022/CWG-C097/CWG-C097_AV2_CTC_v4.pdf [Protected];
[2]: CTCv2.0, https://aomedia.org/docs/CWG-B075o_AV2_CTC_v2.pdf [Public]