github.com/ooni/pipeline commits | Ecosyste.ms: OpenCollective

Remove the --progress-bar off option from pip

ad1f36056ea41d5a55d8da322fc32e96d48de010 authored over 5 years ago

Merge pull request #218 from ooni/fixture-based-testing-pytest

Fixture based testing pytest

b907e25f7039257b9a887aa617e71422e4b7634a authored over 5 years ago

Disable pip progress bar

ca579770d0f36ed383dc8ef2d3ec8543e145d66e authored over 5 years ago

Switch to Pytest

2f698c41b8810a1301a1f72504e0339bc3099988 authored over 5 years ago

Reformat with Black

8b60f18d16a106f71adaeb56ea370d2c192e8879 authored over 5 years ago

Fix typo

8f1c728b718afb25ce91b900c5418aefb365e185 authored over 5 years ago

Unset the confirmed

d556ad658411b1e9a7f42831c697f19ed89c21db authored over 5 years ago

don't cache

c0b4ff166172ea71026635f9ea7dd8a264e375a8 authored over 5 years ago

Change file

e0de51a5c41ae6c42be6e58fca25c0e09ba63b30 authored over 5 years ago

Create bridge_db.json file

70edecdd909f7210f5d8b0b6168380d71a10c6bc authored over 5 years ago

Fix bugs in the complete workflow

57659619591438cd6f78679f8eec938a9f857eb1 authored over 5 years ago

Only download files if they don't exist

80e5ca2dc721fec370bfbd55cae80ab99795b534 authored over 5 years ago

Fix missing docker_client

32458288017911f655c386139c7f5d45af94881a authored over 5 years ago

More debug output

b8330d548d656ead40976a2e6a4c512dce977a65 authored over 5 years ago

WIP on full integration tests

665342d50920c266822d514d387db4611271b458 authored over 5 years ago

Merge pull request #207 from ooni/fingerprint-by-cc

Match fingerprints by country and globally

342da8921fec4bdc2a7f4d09d75d86fb87598f14 authored over 5 years ago

Match fingerprints by country and globally

5e491d720abe9d6f3f649926738d69a5ab62dc94 authored over 5 years ago

Run Black --fast on Python files

0b6ff05f275dcad305c5a98bb3a3755d34860562 authored over 5 years ago

INSERT filtering on bucket_date

7b7b166d44f659333b65fddb82c73016e34d16e9 authored over 5 years ago

Move from ooexpl_wc_confirmed view to table

Insert one day at a time to improve efficiency.
In case of duplicate runs of centrifugation with...

4c4a121f64539be736c98cc1610ffa3297ae2d58 authored over 5 years ago

Add one-shot INSERT to comment of migration script

08a17f269fc515800ef968dccb7a8082d5430bbe authored over 5 years ago

Give better column names

187644683ef050d1369196d33a305eaa535481b6 authored over 5 years ago

Fix ooexpl_wc_confirmed query

Add JOIN on autoclaved, do GROUP BY on test_day, use colum names in
GROUP BY

0a2af2d47877fb18dd81120d585aea494d09b9ac authored over 5 years ago

Add migration scripts for the mat view

350a24de75d406215b45b111abb0c60352608ee6 authored over 5 years ago

Add task for refreshing the ooni-explorer web_connectivity mat view

e73ddd340b655fc079794991deca6e68ecd1695e authored over 5 years ago

Drop query from comment

0b39ccc283df711091b2354daa1cd62c9019353b authored over 5 years ago

Merge pull request #183 from ooni/doc-reprocessing

Doc on reprocessing & reingestion

274eeecbb6ac64c99cdd89e0407e69314cd81e85 authored over 5 years ago

Merge branch 'master' into doc-reprocessing

* master: (67 commits)
Fix typo in function name
Run check for reingestion as well
Check f...

3d0ea58e8ab9a6ff7e0ff2160ec076f71a85702a authored over 5 years ago

Add example of editing PRJ

5208ff2c144cf09e66e510ea279e55f44702e9fc authored over 5 years ago

Add example on code_ver bump

2d075dbe31b6a09c1c6e3beca3297ce056247df3 authored over 5 years ago

Expand more ellipses in doc

e0265dd56c2a192977160883542ea8ea969fa0d4 authored almost 6 years ago

Expand ellipses in doc

dbd4ea463efc2137490e18812185b0a7849712b0 authored almost 6 years ago

Address @hellais comment on document date

ef9c2c9120b6ed4933eeda4703861f24e6148cd7 authored almost 6 years ago

Docs on code_ver, CODE_VER, CODE_VER_REPROCESS and min_compat_code_ver

debe822a43eeba3a6ddbd6229b31a5cdee6faf72 authored almost 6 years ago

Docs on http-body-based fingerprint case and pipeline-reprocess script

f69f154ed5190afa9e46b94e3cdbbc183d79372f authored almost 6 years ago

More docs on "feature-based fingerprint" case

790984a325c64da2c31a189b136276c56881fdf5 authored almost 6 years ago

Merge pull request #151 from ooni/unconfirm-probe_cc-not-origin_cc

Make the confirmed flag more accurate

065cccdfeb531e93a22d2aacc05ec05e990f99ee authored almost 6 years ago

Fix typo in function name

353547fc93ebd35b23e0b533d672998a34905857 authored almost 6 years ago

Run check for reingestion as well

f22d95d1a65ca97150ce753e4a11cbb9c0821dd9 authored almost 6 years ago

Check for anomaly counts as well

3ec18616eb9d9ff6ea12d14a6a3573dbabefadfe authored almost 6 years ago

Fix missing import & typo

fbc0c579dcc8f9f100d8001679822177ea8b658d authored almost 6 years ago

Write a simple test case to check if the reprocessing works as expected

54e819b315803bef7e82ab2c10bb3cdf0b503bea authored almost 6 years ago

Bump the fingeprint table hash

13a9d284e4b924353b3ece7cf71b93ea0415b870 authored almost 6 years ago

The country code for the UK is actually `GB`

It's safe to blindly convert UK origin_cc to GB, because UK is specially
reserved by the UK.

d895c6f6132b16185a5442bc43424f980da549ee authored almost 6 years ago

Even better readability :sparkles:

c83811c2e6782f9f9eb70a9e55ff4299b2679462 authored almost 6 years ago

Stylistic improvements to query

9688f51545307f544bd37df976685e3b97e2757f authored almost 6 years ago

Disable canning and autoclaving tests

caca4ca57b6ea5cedc6e617912b13429c20329bf authored almost 6 years ago

Fix ambiguity in msm_no

eaf5728b39e4de2ef912134ff060bf74679ddbdf authored almost 6 years ago

Fix typo

582f1a90740035b689d6a13158956ea644359723 authored almost 6 years ago

Fix syntax of JOIN operations

b6257c198f42caf61754819407a8fd8ef6cbaea8 authored almost 6 years ago

Make the confirmed flag more accurate

Limit marking measurement with the confirmed flag only when the country
of the blockpage fingerp...

5d975c9811170c1ba0c3a8af0f99be26347f5487 authored almost 6 years ago

Merge pull request #149 from moar-docs

Add small notes in headers of shovel files

ecbbf477b37798f80b62fffcf8ae149350c68ed5 authored almost 6 years ago

Merge pull request #144 from ooni/ooexplr-meta

OONI Pipeline workflow for API needed by OONI Explorer

2861fe495e56d5ef120ec2108dfd83f8cfa0563a authored almost 6 years ago

Fix rollback script

4a837e10deaac9f0752b7ea577fdd686d814943e authored almost 6 years ago

Save some bytes

de69d52e4133c72b1cd8fca96e339ac19f400f61 authored almost 6 years ago

Update Readme.md on docker-build

437f63f9f365efdb5ce90ddacbd444d5646ef460 authored almost 6 years ago

centrifugation: update stub `anomaly` and `confirmed` flags

5fac5cf96b4c7416f5ab2c4b41f709d3bca0a9b6 authored almost 6 years ago

Drop website related tables

a63257915c094e37e5b4e4698af68bc09343d6a6 authored almost 6 years ago

Use a smaller bucket

13bf13a72d3440ef69742f2b41fd2cc20a86479c authored almost 6 years ago

Add small notes in headers of shovel files

24410c7cd0a9cdd43a6924bbe62472cc8854e1fe authored almost 6 years ago

Add shovel unittests to travis

c5b20661c6c5164987ffba473e41e48e61d638ce authored almost 6 years ago

Hack to get the unittests to run

5e69199567495b7db1effb8aa4d88984b898ee85 authored almost 6 years ago

Update the table updates in the centrifugation script

11518dc1cf9c0937c777d8650768c9f7791219a1 authored almost 6 years ago

Split the various DB scripts into separate ones to aid testing and debugging

ef5a456de501a16e8e2a86a6891ce8c827b2e586 authored almost 6 years ago

Replace != NULL with IS NOT NULL

df4ffec038f60292a1a9a5a225569f4a15b38c0e authored almost 6 years ago

Add test_name != NULL to the ooexpl_recent_msm_count queries

f2ec4b5c5ff237f4bdf91f383de6bf261f6b1cd2 authored almost 6 years ago

Use a more recent bucket to test the recent_msm tables

a251586f73a805675210c7252356c81a365cc257 authored almost 6 years ago

Fix pg-wtf

`GROUP BY` with aliases produces a different result than GROUP BY with
column index

72d683a86ebe9b83a1dc4cfd2868d5d54b8b4918 authored almost 6 years ago

Whitespace fix

671f9aebb111d829b7fe6ba8a495c64278b0101f authored almost 6 years ago

Refactor integration tests to use unittest

3b4420887a7eca66074eccb87c5b382f44de70b6 authored almost 6 years ago

Fix some bugs in the 013-ooexplr-meta and centrifugation workflows

2f6464b10bb7dcef96a749a4f9dd1bf94a04fd4f authored almost 6 years ago

Add more checks to the integration tests

4c37ddd1e9a009634d799c071ab472dae37786f1 authored almost 6 years ago

Use a smaller bucket for the CI tests

2393247f6e8f9f76ce420b57dbf021eab8909f04 authored almost 6 years ago

Create the ooexpl tables even when the DB is empty

Since it was being created based on a CREATE AS SELECT query it would
only create the table if t...

804e8fb8976390112f75a9d08521065f8c5d54b4 authored almost 6 years ago

Merge pull request #145 from ooni/centrifugation-docs

Centrifugation docs

5b77efc24fa32db8dd6b9499e1312e21f2bdbe34 authored almost 6 years ago

Add some notes about buckets to use for testing

256e0cf9aa2f39640b6380d3d9f18eeb286be0fd authored almost 6 years ago

Check the status code of the container

9518a5ec04421b48f25973e06e85846230bc0b3a authored almost 6 years ago

Fix bug in update_explorer_metrics workflow

729130f89a4d9402f13ef701e29a900e279f184a authored almost 6 years ago

Use a smaller bucket as a test case for the integration

4f821a5d265990f9cd885c0ef8e78d93363c1b68 authored almost 6 years ago

Avoid too many newlines in output

32b1e3888fdf82d7ead7310d715722989024b501 authored almost 6 years ago

Handle output from centrifugation.py script to prevent travis from stalling

84a42710b1497863a167279fb5c67871a8a06035 authored almost 6 years ago

Refactoring of the centrifugation runner

2038724fd1e1aff6050c5bb30eb2ee1e24ba102e authored almost 6 years ago

Fix typo in comment on 013 sql script

63ce99f1cab4a155a5a850e8adfa3e95248a2df2 authored almost 6 years ago

Calculate the centrifugation runtime and cache the testdata dir

353b96ab52a671e066cc3b52f98e0ec12709c1c2 authored almost 6 years ago

Merge branch 'ooexplr-meta' of github.com:ooni/pipeline into ooexplr-meta

* 'ooexplr-meta' of github.com:ooni/pipeline:
Commit missing SQL files

abb0344beb362d6ec89a01e3b04a3a6a556d0a3b authored almost 6 years ago

Fix exception

3a262644e3d0d959b63794b52bf23d7d64eced40 authored almost 6 years ago

Commit missing SQL files

ba754ebec68c5ec715fba13310ed06acb2b40c9d authored almost 6 years ago

Make the tests work with python2

ecc43915a70027cb096465e802ba3360ceadf0a1 authored almost 6 years ago

Enforce python3.7

4f22113e95cd933ef5071ce5a3d2806b3f2f6b9b authored almost 6 years ago

Add mention to tests in the Readme.md

1255c2db7bdf76cd8cadd6d4c35335dc2b57e1b0 authored almost 6 years ago

Add .travis.yml

69d5dc09e0ade7d094196499483d026ce8650ce9 authored almost 6 years ago

Implement integration test for centrifugation

9361fb4e8e0cd6dcb1ebbadca3ad0657de3a715d authored almost 6 years ago

Start writing integration testing for the pipeline

95dc49e9aaaec3673b3a27ca3cc86df552a52101 authored almost 6 years ago

Vendor Versioning from https://gitlab.com/depesz/Versioning/2dbcdf94e93a68a6f273375408a92d42c69dd5a5

61cb4e323270e0cf246fea581971ef929d51a1c7 authored almost 6 years ago

Drop Versioning from the submodules

4d36770e2c3156e0d5be0ffed5d05b603997b692 authored almost 6 years ago

Add update_explorer_metrics PG task

bcf9500478861c494fef00472935289ed426b071 authored almost 6 years ago

Fix wrapping of docs

2cf05d2281bf9f0be0d319a4d93eca24188c926a authored almost 6 years ago

Add docstrings to some steps of the centrifugation workflow

* load_global_duplicate_reports()
* load_autoclaved_index()
* load_autoclaved_db_todo()

842cd09622d4c749efc37a7aab7a43f9e6695b7b authored almost 6 years ago

Store input_no instead of input

68ea84d84b4a9549107be1d61d8448e8007e61ae authored almost 6 years ago

Add rollback script and implement mat view for websites table

3ce4073b85152dc5e0280e3c76d483b3020991fe authored almost 6 years ago