github.com/llvm/llvm-test-suite commits | Ecosyste.ms: OpenCollective

cmake: Add support for PREPARE: steps and use it where apropriate.

llvm-svn: 269247

e648b9e342e3832dc1fa28ad738fe05fedb3cf7b authored over 8 years ago

litsupport: Introduce a PREPARE: part for test files.

PREPARE: allows to specify steps that should happen before the benchmark
runs. While you should ...

1e67a8d002af04871085a934947e1abc76b7ae9d authored over 8 years ago

cmake: Revert accidental removal of PROFILE_GENERATE normalization in last commit

llvm-svn: 269159

0dd1484d17c91777db5ee5ba2a176f7a40d772b9 authored over 8 years ago

cmake: remote lit module should be used after profilegen

llvm-svn: 269158

e934759599223bb1616f21ce3bbc85b27f658a41 authored over 8 years ago

Revert r269071 "Add Bitcode tests from Halide suite"

svn crashed during the commit and it was somehow partial only apparently.

llvm-svn: 269078

98d756c770108fd62129101637b6d0bf4d67b980 authored over 8 years ago

Add Bitcode tests from Halide suite

These are unittests checking mostly for simd operations.
They are compiled with and without opti...

3ad71d2bedfd6f0c91e6548304af1c02c7d7c5de authored over 8 years ago

lit: Use a custom error code in case the executable is missing

llvm-svn: 268584

8d0a7dcdd898bb020d5f85d7858432f19076457b authored almost 9 years ago

cmake/caches: Use CMAKE_C_FLAGS_RELEASE to pass flags.

Do not set CMAKE_C_FLAGS etc. in the cache files because these often get
overridden by the user....

07779bc2d7c6f038fa4c7af6de5c27e3e5d22416 authored almost 9 years ago

cmake: Do not reset CPPFLAGS/LDFLAGS in per-benchmark files.

The per-benchmark files should only append to CPPFLAGS/LDFLAGS and never
override them. Otherwis...

c88be613774c7de3602aecbd2759a29f68821899 authored almost 9 years ago

cmake: Remove non-working files from Examples dir.

The Examples are not used in the normal test-suite run so nobody noticed
they were broken. Also ...

81d96144ab62526733a4b86df76c796f2d159035 authored almost 9 years ago

lit/remote: Fix typo in ssh invocation

llvm-svn: 267627

dd92a82e273e535fd3f97130c567e04f0db9319b authored almost 9 years ago

cmake/caches: Improve/rename iphoneos caches.

- Rename the caches to iphoneos to better match possible watchos/tvos
additions.
- Do not chan...

b554470355e585af1980409abff6fbc863f13c5d authored almost 9 years ago

[test-suite] There is no need to check for name clashes explicitly.

cmake_minimum_required(VERSION 3.2) sets CMP0002 policy to NEW
which will produce an error when ...

fce90c92df4504f223a2bd7a8e9e977522e930e9 authored almost 9 years ago

[test-suite] Set unique prefix for Vector and Vector/NEON

This prevents name clashes for test targets CMake generates on arm.

llvm-svn: 267439

baa138909ea277dc685d55630e41d5d6a73b55c3 authored almost 9 years ago

[test-suite] Added llvm_target_prefix() command to set unique target prefix.

Differential Revision: http://reviews.llvm.org/D19423

llvm-svn: 267243

cd9b4f237cb40bdb511eb6539bbb5efaf73289fa authored almost 9 years ago

cmake: Cleanup xcode_sdk cache file, set defaults for c/c++ compiler

llvm-svn: 267213

b0b883bec5367edf5f090c9a2a37037ed196c5b2 authored almost 9 years ago

Add new cmake caches for iOS setups

These are similar in spirit like iOS.cmake from the llvm repository. The
main difference is that...

6c78c29dbe1db8c0355ce743f5ff3fda8331bd61 authored almost 9 years ago

litsupport-tests: Cleanup, do not hardcode lit path/name

llvm-svn: 266785

7dcf54381f37b42fa97f518208cfd605bdf00877 authored almost 9 years ago

Create a testsuite for litsupport

llvm-svn: 266782

934ce547b0206e82350d7d30da7e3da83974e5ad authored almost 9 years ago

lit: Set test_exec_root and create the logfile there

Also remove the unused test_suite_root setting.

llvm-svn: 266781

71fae8cfc127c36edca3bfa90bd6811adcbcab23 authored almost 9 years ago

lit: Dynamically load test modules

You can now specify a list of test modules in lit.site.cfg.
This allows to add external test mod...

26f24d56201c58eb0c2a17846494b6ed255a4ae6 authored almost 9 years ago

litsupport: Refactor parsing code, add 'run' module

The run module runs the benchmark. The trick is that you can leave it
out to not run the benchm...

62098481296aeea75fcf5c90d9b3d95833cca742 authored almost 9 years ago

litsupport: Make hash module more robust

llvm-svn: 266778

f9d1e4a2cb4dc6d3ce14b64da30adc0cfa6ee521 authored almost 9 years ago

Add 'check' target that runs lit after building the tests

llvm-svn: 266777

b054bc074c9381ded9abf6a3117075366b5bb68c authored almost 9 years ago

cmake: Prefer llvm tools from compiler directory

We should preferably take llvm-size and llvm-profdata at the same path
as the clang binary.

llv...

cc1973096fcfacd93bad8b1a2e9ac1f2e6d73090 authored almost 9 years ago

TEST_SUITE_DIAGNOSE_XXX should be the cached vars

CMAKE_C_FLAGS etc. already are cached and need no alternate description.

llvm-svn: 266769

e5087ab28cabde30d1918c2ac24942cd1f9c5240 authored almost 9 years ago

Implement Matthias' feedback for TEST_SUITE_DIAGNOSE flags

llvm-svn: 266762

13306e6a2fe29652b8bdd6f8f69032609cbcda89 authored almost 9 years ago

Update TEST_SUITE_DIAGNOSE to work with more than one flag

llvm-svn: 266678

e4cfad533ddb7b0bf39ef7913bf7cbeb82243ee1 authored almost 9 years ago

Add a flag passthough for LNT diagnosis tool

LNT will now collected details about a run to help triage and diagnose
performance problems. LN...

dfbbb4bb2c189011a1c08c5f6fa5f267d7afa943 authored almost 9 years ago

Recognize *-*-macosx* as Darwin

After r256038, we pass --target to configure from LNT. LNT detects the
OS based on the target t...

7ffc492f048cfb0db4a5acf3f113f7801f41c860 authored almost 9 years ago

lit: Fix python3 compatibility

llvm-svn: 265071

9adc494311c36bac3218075ea4d3bc7af8e5e97e authored almost 9 years ago

cmake: PROGRAMS_TO_SKIP must be set before llvm_singlesource()

llvm-svn: 264907

dffa86e47a068002170d4bfc561e78b14eda3f99 authored almost 9 years ago

cmake: The ms_xxx tests only work on x86

llvm-svn: 264900

8939617690fef45ac52b8289ccfe8581915b28cc authored almost 9 years ago

Fix one more diff-line I missed in the previous commit.

llvm-svn: 264622

4bed566c1a475fce5f3de35b9884f9340923c622 authored almost 9 years ago

cmake/lit: Make lit aware of SingleSource tests.

This reverts most of the cmake hackery from r264499, instead
communicate to lit which tests are ...

41c021437a93d89de14a169b8e5c02fb6bb6e129 authored almost 9 years ago

Use fpcmp instead of plain diff for checking results in SPECs.

llvm-svn: 264613

014fd0dc535a4141ba04b0ddadc43827b4e0470b authored almost 9 years ago

lit: Log all executed commands in test.log

This should make all steps transparent and reproducible without using
the lit tool.

This also c...

67ee2f53c74feab0f32b52d3f3f6d2925456258c authored almost 9 years ago

lit/codesize: Fix warning reports

llvm-svn: 264500

fe95622c225c61d7c9e7d42a8dac2b8288f0f016 authored almost 9 years ago

cmake/lit: Put SingleSource executables into separate directories

This fixes the problem of compiletimes not getting collected corrected.

llvm-svn: 264499

82446dd7f6ea2e5e963f25f1bf956073a75ff43c authored almost 9 years ago

Add missing reference_output files

llvm-svn: 264498

4bee7343a83dcd7b297041373a6369dd00d5488d authored almost 9 years ago

cmake: Cleanup ClamAV CMakeLists.txt.

This way we can get rid of the llvm_prepend() function.

llvm-svn: 264497

894f27c3d9e9f83909f8960528d2f8bbfb5c983d authored almost 9 years ago

cmake: Do not check for x86_64(h?)

The cmake build has never set ARCH to x86_64 or x86_64h, so no need to
check for it (things work...

2468cec2e36f52394df999ac5bb5116d28c34681 authored almost 9 years ago

cmake: Remove unused CMakeLists.txt file

This was probably automatically generated by accident when we generated
the CMakeLists.txt files...

84c9a336ed3c2f0a86916fd5533e1d97850c1b3a authored almost 9 years ago

cmake: Cleanup comments to not repeat the function name

llvm-svn: 264494

b3e0739bb558a19145e437b6fac96dd3891f4d34 authored almost 9 years ago

cmake: Remove unused file FindYACC.cmake

llvm-svn: 264493

de9fd35382f5d3d8c2eaa1f0f9cb6e54c7bf426e authored almost 9 years ago

cmake: Cleanup, move .test file related functions into TestFile.cmake

llvm-svn: 264492

905efe464f12eac4c31d8da92623b62360536e73 authored almost 9 years ago

cmake: Check that TEST_SUITE_RUN_TYPE has a supported value

llvm-svn: 264242

0c6d32d7ff2c57895397b15d9225ab51d5075a81 authored almost 9 years ago

lit: Collect detailed codesize information via llvm-size

llvm-svn: 264240

82cbc3fef5a6e25c174275f78511b7a0d3f753ae authored almost 9 years ago

Makefile: Update small reference results for SPEC2000 v1.3

llvm-svn: 264220

e4c88bc093f08d86fa619043f5700e572121ad11 authored almost 9 years ago

cmake/lit: Specify work directories explicitely.

Specify the working directory explicitely (if the benchmark depends on
it) instead of relying on...

137201e91b88a4f02ae13087353044c483f574bd authored almost 9 years ago

cmake: Avoid broad -lm for SingleSource/Benchmarks

llvm-svn: 264101

6b4e0618ccf46b92725a547e737837c52fc27d2f authored almost 9 years ago

lit: Fix for pep8 style, fix whitespace; NFC

llvm-svn: 264100

5a920472386e01aa0daf594c943e2abe20924376 authored almost 9 years ago

[litsupport] Make sure perf is run under "rununder"!

When profiling, we were only mutating the *original* runscript. But the run_under argument is es...

735ad30c1a6ac2cbb3946ce61fc14084a4208926 authored almost 9 years ago

[cmake] Properly convert custom metrics to Lit metric types

Lit complains if you give it metrics as floats.

llvm-svn: 264045

4b12b1be7944db6db624d0b078da4e754351a82a authored almost 9 years ago

[timeit] Properly shift only one item for a boolean option

This fixes parsing for any subsequent options.

llvm-svn: 264044

5d608f7162f5e37169f7342b4998cf972c027b3b authored almost 9 years ago

cmake/lit: Use CMAKE_STRIP variable to provide strip tool name to hash.py

Host `strip` tool cannot recognize binary format of files built for
another target architecture....

7671f74664e409de1c23942405e3ce9b76fc2a34 authored almost 9 years ago

lit: Add space between "run under" script and its command line arguments.

llvm-svn: 263901

1dc441c98647def304f1ce2a8f56e6548e876569 authored almost 9 years ago

[LNTBased/SPEC] Only populate <test>.exec.status upon failure

This is what the rest of the testsuite does which makes it more
convenient to grep for errors di...

6b644795548496f5c556c9d24bf1d8ef13a75ea2 authored almost 9 years ago

cmake: Use variables in if() expression directly where possibke; NFC

llvm-svn: 263615

68319ea08ea38d47d82c3f3382c78e0d7afc1d01 authored almost 9 years ago

cmake: Fix BUILD_TYPE default not working

llvm-svn: 263614

a6a221e4948dd004f2be154bc9367224284b22ce authored almost 9 years ago

cmake: Add pregenerated bison output and drop yacc/bison dependency

llvm-svn: 263613

6df63d30608d33040f67e13bf1bd3de782cc24e5 authored almost 9 years ago

litsupport: Fix bug in environment variable parsing

llvm-svn: 263612

337f2f80202fe59d74b641cd908a25ec949c6e4e authored almost 9 years ago

cmake/lit: Collect link_times

Previously we only collected compile_times, this adds a new metric
link_time. This currently has...

e0cac2842d83ed9edd8599d30635dc4ab636cb35 authored almost 9 years ago

lit: Skip RunSafely.sh and invoke timeit directly

This also allows us to split run_under and remove device handling into
separate modules.

llvm-s...

11048a9d297f9b09efd066efd5db622493dcb490 authored almost 9 years ago

timeit: Add append-exitstatus switch

Appending the last line with "exit XXX" is usually done by RunSafely. I
added the same capabilit...

6c95b4c432d1097ddd55c9fdc854e509a1d42abb authored almost 9 years ago

lit: Big refactoring, introduce TestPlan object

The testplan object describes the sequence of steps that will get
executed to run the benchmark ...

c58ba9de2bc9e432b7430201efb5d4c706f4947a authored almost 9 years ago

litsupport/shellcommand: Add toCommandline() and wrap() convenience functions, support envvars

llvm-svn: 263531

9e2fd679cf5e7c77a796a3cc8124b09d0f0b8166 authored almost 9 years ago

tools/timeit.sh: Fix space/tab inconsistencies

Use an llvm style 2-space indentation.

llvm-svn: 263492

089b03425a94a379eafa2da8e208d7b0461c2d0d authored almost 9 years ago

cmake: Mark the release/debug/... flags as non-advanced, fix typo in Release.cmake

llvm-svn: 263484

bf35b6d9deb03bac03b390392054ce96ecc202d9 authored almost 9 years ago

cmake/caches: -march=native is a bad idea for reproducability

llvm-svn: 263465

d6bc5aa8faa39d76c1b6ca97e5159c73f8d086f8 authored almost 9 years ago

cmake/caches: Add some comments, more aggressive release mode

llvm-svn: 263456

a9b0341e7327389d8ed5a61a0e7b6c62e2d3d1d6 authored almost 9 years ago

[cmake] Fix typo in hash strip command

It's meant to be '.note*', not .note! We weren't stripping all the note sections, causing hashes...

c7cc5529b3581bfe0b0ec1325158033357ba6170 authored almost 9 years ago

[cmake] Add support for --redirect-stdout and --chdir to timeit.sh

These options were added to timeit.c by Matthias and are required for running SPEC with --use-pe...

51f8d8382e14c61a2a3adc51ed1c9e811d0331f1 authored almost 9 years ago

[LNTBased/SPEC] Fix 403.gcc and 471.omnetpp checking

llvm-svn: 263315

f493a2105c2a4166869966e8d07930f2ae3a5e71 authored almost 9 years ago

Update test-suite caches with jbogner's cmake feedback

llvm-svn: 263306

5cd8118e0e5a5245cdf8cb75ed2cfd7634e2bcee authored almost 9 years ago

Makefile/SPEC: Further adjustments to SPEC2000 v1.3

llvm-svn: 263284

7f8f4622fba7f6d826944d8d58fb385add447904 authored almost 9 years ago

[test-suite] Append LDFLAGS to LINK_LIBRARIES

r261397 changed how LDFLAGS were added to the target under cmake from
using target_link_librarie...

68c51bf554c78aa774f2fe159707177d4dad1008 authored almost 9 years ago

Makefile/179.art: Fix reference output for SPEC 2000 v1.3

llvm-svn: 263189

32270908b9b6d57ae052a99b0772a89a1524548f authored almost 9 years ago

cmake: Make compile time measurements optional

llvm-svn: 263188

bdaa89a691e158048a0017f7c78c94b592b9a68a authored almost 9 years ago

Add cmake caches for common configurations

There are some common configurations of the test-suite which really
should be encoded right into...

ace63fd8df1cc2ff707fe97f59b0c4d53cd1e0e6 authored almost 9 years ago

cmake: Skip Regression subdirectory in BENCHMARKING_ONLY mode

Instead of trying to list every benchmark in the subdirectories in
PROGRAMS_TO_SKIP (and failing...

86b809131bf05410ccb538236a09cec33cdcdd97 authored almost 9 years ago

cmake: Simplify BENCHMARKING_ONLY handling.

No need to an internal variable BENCHMARKING_ONLY that is just set
depending on TEST_SUITE_BENCH...

2eb6dacdea9d01b24f9eccf357b29bbdad79208f authored almost 9 years ago

Makefile/176.gcc: Add -std=gnu89 CFLAG

The -std=gnu89 flag is necessary to build this benchmark with SPEC 2000 v1.3
(we need traditiona...

0b01e374632c0d8f5797736304c197a55ff985fe authored almost 9 years ago

lit: Add 'cp' to the list of ignored executables

llvm-svn: 263147

0233c6bbc923da1d3e480a76b0d2845b354a4019 authored almost 9 years ago

lit: Fix detection of non-zero exit codes in run-lines

This went undetected before, because:
- RunSafely sets exitcodes to zero anyway and only puts th...

8915b986fb714137bd08ed74337e7984675107f8 authored almost 9 years ago

175.vpr: Use 'rm -f' to avoid non-zero exit status in preparation step

llvm-svn: 263145

f8bd468d64f8cd685bac50ff00af1d8669665ba2 authored almost 9 years ago

lit: Option to skip test if hash did not change.

You can now use "llvm-lit -Dprevious=xxx.json" to make lit load the
results of a previous run an...

b21d2800e89ed63c488192ad6b3b86b7ec4b265c authored almost 9 years ago

lit: Add simple codesize module

This simply records the filesize of the executable.

llvm-svn: 262306

cc5c7a3da784a4afa4b35c262caf23fa2a7e26b5 authored almost 9 years ago

lit: Determine executable name only once

We need it for several modules so it makes sense to determine the name
of the main executable on...

0985d5baed2bbfeeb3610267bf91d40bcd6dde9f authored almost 9 years ago

lit: Fix a bunch of pep8 violations

llvm-svn: 262304

2e790992cf52c96e4e286d4f719a0f7dd17a10ff authored almost 9 years ago

lit: Improve executable hashing

I had a similar hashing support lying around in my personal repository
for weeks now, thinking t...

67e88dc4b952cd95ff51f21f01ca80adbaa61b1f authored almost 9 years ago

[cmake] hash support: make sure we run strip

Stripping out certain sections of the binary under test is important to get a stable hash, and i...

12b18088aa482834ca4a06acea8639e69fb1ad88 authored almost 9 years ago

[cmake] Add support for hashing test binaries

This adds the "hash" metric as computed by the HashProgramOutput.sh script (used by the Makefile...

bc0c36633a4421b1d071806f896b9371fa5344dd authored almost 9 years ago

[cmake] Add support for arbitrary metrics

This allows a .test script to specify a command to get a metric for the test. For example:

METR...

0f69f202a64fea9fd774517210052d53aa162540 authored almost 9 years ago

[cmake] Add support for timeit.sh (--use-perf) and --benchmarking-only.

llvm-svn: 261854

8380d4e7b38ca008c9c4b544b44f86298a769da8 authored almost 9 years ago

cmake/lit: Add SPEC CPU2006 floating point benchmarks

This does not include the fortran benchmarks.

llvm-svn: 261816

3ff8a7f8fa4b0264ee83fadd92435605e84b47fd authored almost 9 years ago

cmake: Tweak which options are advanced/basic

Mark some options that are untypical for the testsuite as advanced, and
clear the advanced statu...

37903fd683e44a54c793e6a91c5700362b6e47c4 authored almost 9 years ago

[test-suite][mips] Deduce using MIPS architecture automatically

Use `check_symbol_exists`() to check macros defined by compiler and deduce
enabled MIPS features...

041deaa57f70008cdfabd9f92c6955ab155f07ef authored almost 9 years ago

A number of cmake configuration fixes for SPEC.

These changes were needed to make the spec 2000 and spec 2006 benchmark
compile correctly on the...

16ddea4de6e60d355775b7e3b80f1b4275a04246 authored almost 9 years ago

lit: Fix typo in error handling code

llvm-svn: 261400

8d9e685caa59df19324324f39657814aef71249c authored almost 9 years ago