github.com/ooni/pipeline commits | Ecosyste.ms: OpenCollective

Merge pull request #231 from ooni/fastpath-requirements

Fastpath requirements

6ab44b37b5ffa19624cdea8460d5b76bae916e7c authored over 5 years ago

Merge pull request #213 from ooni/fastpath

Implement fastpath

bdb76abf2aba41074226210e1282e17aeee64602 authored over 5 years ago

Version 0.1

f73f462f3c766edaf7336965ea26f1a34fa8dd22 authored over 5 years ago

Read collector hostnames from conffile

211ca8f9a338e5e75c296a2355edcf98d736c595 authored over 5 years ago

Update table permissions

2cfd6319e2d8cb67ca51a01913ed8a5deb38f1a4 authored over 5 years ago

Bump urllib3 from 1.24.1 to 1.24.2 in /tests

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.24.1 to 1.24.2.
- [Release notes](htt...

bc7b86231204b08e8d78feea6e7d75a38290dfdb authored over 5 years ago

Fix file creation permissions

a1470d7836a1101c7ce600285d6eb7eb19de0fb9 authored over 5 years ago

Switch to simpler queue implementation

5027244d9378f49d45d599033adb438734b473e0 authored over 5 years ago

Cleanup

ca2e1f41419efdacbfd950f50ec22b5c9cf196ce authored over 5 years ago

Normalize: add tests, disable on fastpath

ea660edc353e01801449350d4eef4e41a3a45c34 authored over 5 years ago

Drop fingerprints CSV

63103dac63c4898f99320b0c12da2d6118cb858e authored over 5 years ago

Cleanup

fab45fff1b259d0bc1a376d782d2347c806f72d3 authored over 5 years ago

Run only on web_connectivity

b7ed9f696876e306f0d0bc378db3cb73de31bbf1 authored over 5 years ago

Cleanup normalize, improve tests

be6294da79cadcfb246a09747b9dc5468f9f2712 authored over 5 years ago

Handle ZZ fingerprints, add tests

6f3634948595f231f07740b9d7310b792f8efffe authored over 5 years ago

Update dependencies, minor cleanup

ec66d34fdd84b520481dbb30c8dd88db312b9e31 authored over 5 years ago

Add support for writing into the platform column in the fastpath table

cfad9ff11c3a54018d3a74ded801ab242580840b authored over 5 years ago

Add a portable Queue implementation to work on macOS

See: https://github.com/vterron/lemon/commit/9ca6b4b

125eeeb701f86ec7cb56bd19e01231a5a12a3d48 authored over 5 years ago

Make the import of systemd conditional to make it work on macOS

ba39ca09da1f78e10ee3de91eebcf2ea0ce15344 authored over 5 years ago

Add requirements.txt file

36278fe617416835f7753bf5dab4f95e9e40a2aa authored over 5 years ago

Leave more free space

d6374f5ce0401c26a7ae4f812b6334a39ee87074 authored over 5 years ago

Move unused code out

755bcb7c34f5b99b6b879de69eaefe4868726d43 authored over 5 years ago

Send NOTIFY, increase trimming, minor cleanup

06dac0da5ce3d06a80e85d6266257da2a83fcb0f authored over 5 years ago

Trim old measurements

3245c78bddc21e5b195e7a394e831b85d0982e23 authored over 5 years ago

Handle exceptions in sshfeeder

Simplify code

a78c7593b3612715d34883f4470e1643edc3bb10 authored over 5 years ago

Bugfix

724db577074f9c8dcce4f8f7a4d16171ca5ed1f8 authored over 5 years ago

Improve SFTP metrics

ffdd2bb9ba395bbc3b76a4dbb2f6985d7d414cfa authored over 5 years ago

Update README

f2496a078049d6274188c1978b92b1fb7f6665da authored over 5 years ago

Enable DB ping

6bd07aee956bae476645789ae5785397ed38f3f7 authored over 5 years ago

Drop unneded SSH caching

ea1b5a0cdb28861883b3990821b2620d10ece3b4 authored over 5 years ago

Handle exceptions, improve reliability and logging

be99b482a26906778786cf459d05e803528c95db authored over 5 years ago

Update YAML handling

880f6e8f9d215807c23b2d6d8a7ce08080747808 authored over 5 years ago

Add comment

b97fd0a3600e675182c0d9c7cfd900f1fef167e7 authored over 5 years ago

Handle missing archive directory

adbaa1f32a0fce793cc9c16b3d0596a59e55a36c authored over 5 years ago

Add comment

7774d1ed8e278f2e78fc6c658ec546561f49e420 authored over 5 years ago

Add YAML support

Better exception logging
Rename report -> measurement
Handle body_proportion being None

172eca329ee2022d8c069d919a0350e23668c8d6 authored over 5 years ago

Sort keys and unit test

aa85a7958af9f1ac64300c7f71aef7cc5af9c1dc authored over 5 years ago

Count exceptions

a0dda6ab1a2375800f3a251fa63eacb8f225f872 authored over 5 years ago

Update test

9a97a31ca35d81b34e1da2d411cf84391b2b8759 authored over 5 years ago

Add trivial ID, prevent race in writing files

Improve logging

18b2a46ffd23194c7ee7b5074247e01ebd720ef8 authored over 5 years ago

Fix body_proportion bug

7173b2efba3927bdb2a596a67eadaccc141a35bd authored over 5 years ago

Consolidate tables, add flag to overwrite outputs

Cast NULL input into ""
Detect, log and report collisions

1f15622789a46749314a18e559de1550ede3a743 authored over 5 years ago

Remove JSON parsing from sshfeeder as well

bc7dde8b619f97726c64f7241535be29b28fce2f authored over 5 years ago

Add Postgres integration

Implement multiprocessing wth 15 workers and a queue
Write measurement to disk
s3feeder: move JS...

0f77bf7e91aa35b1ec6a2808896c2954aa5019d5 authored over 5 years ago

Rename module

053346e6260871bab6bc50e22e6a406bc7088486 authored over 5 years ago

Minor change

04266d958e392a689bc8faa34875a71ae717a29b authored over 5 years ago

Restructure to allow uploading msmt to S3

bb3a25361a09557f6427cf1cf577ad5229483191 authored over 5 years ago

Rename module

5744ccf9b59c3910bf15e13e732b6f70b8dffe90 authored over 5 years ago

Rename report -> measurement where needed

9936a31bb5abd2073c428437ca4a2688798e71d0 authored over 5 years ago

Cleanup status reset and add test

ac048978aa2c817fd7ac53dd8ada2c2fb1e7fd40 authored over 5 years ago

Bugfix

15d02bd268e1c8277652eaa8f21bd4a27681094e authored over 5 years ago

Estimate run ETA

ab83287ee258d6f7bd229924f3020bee6bb65986 authored over 5 years ago

Bugfix

c369f0c4179aa1989c62cc96a533422e2b5e0071 authored over 5 years ago

Add comments

68bd9177b00f97e88b562a6f0c1548bb5f4c6be3 authored over 5 years ago

Add S3 file uploader

3b67afc452591ca143389eef305a9ddf0ee2b2ef authored over 5 years ago

Minor cleanup

bbd97090db6aa087adf4ba9019fb1f7d40d6d024 authored over 5 years ago

Sort cans before loading

b920a425c54402938c4048abe833c618ae5df374 authored over 5 years ago

Switch fingerprinting and away from Pandas

Improve performance
Preprocess fingerprints
Improve metrics

Comment out WIP/unused code

f5314b99ab483602808206386f088213cb8b95ee authored over 5 years ago

Merge pull request #222 from ooni/ooexpl_wc_input_counts

Add materialized table to improve OONI Explorer performance

4867a2060333dc28c5333380659eb7ab5bc0df5f authored over 5 years ago

Add checks for the ooexpl_wc_input_counts query

33e93ee7db7a81de795621adb7d25c4e33f0e871 authored over 5 years ago

Explicit the column names in the INSERT query comment

1cf2cb08c6c1ca682a57ea85f58459211631ea79 authored over 5 years ago

Rename constraint to avoid clash

4f690db33cc85eb8feb3c4d71832c5d9cb2dc211 authored over 5 years ago

Add index drop to rollback

603a2cbdf247357dfa6c41f0e67f6f18a8b2860b authored over 5 years ago

Fastpath: initial commit ooni/spec/issues/145

742885fd7f2dc3293168ad4b2b0198451ca2ab4b authored over 5 years ago

Merge pull request #221 from ooni/unit

Enable basic unit tests and coverage on Travis

5544e5e4abd3869a87666d099de2449ca48faad8 authored over 5 years ago

Fix the centrifugation update query

25ce207ee99d74bd6595cbe2c6447583c4734782 authored over 5 years ago

Add failure_count column

94868494b722b19b3ef417cc2e8a4d0d8060b81e authored over 5 years ago

Fix missing column

01e66dc602de6ba9789a0b828a2ad7e5b0047631 authored over 5 years ago

Add materialized table to improve OONI Explorer performance

8fb0dd9b3b63d52a89917de81864d61451689b28 authored over 5 years ago

Merge pull request #220 from ooni/tox

Use Tox from Travis

512ca5126046bebf3ce22123679d8298b53d58b9 authored over 5 years ago

Enable basic unit tests and coverage on Travis

69f90c08772e8aa6bd306dead6d5bed3713a88c6 authored over 5 years ago

Extract Postgres port

105b8c89e2576155b0ecdb700627b82f742f9190 authored over 5 years ago

Use Tox from Travis

0fca8db16e1547076cafe551824c1c66a001f11e authored over 5 years ago

Merge pull request #215 from ooni/check-confirmed

Add assert to check if confirmed is > 0

51c98f79019095526f33c367170932631445d5b0 authored over 5 years ago

Add assertion on confirmed count

293acaaa1ce2cfca698f57ce4c41056287536250 authored over 5 years ago

Merge pull request #208 from ooni/metrics

Add pipeline metrics

d4874cf1384eaf70dad38290c43f740d86ebfdfe authored over 5 years ago

Improve test_integration

Add tox.ini

31bcfbcf2af1a293b5bf2421e94811d7d1f942eb authored over 5 years ago

Add some extra comments suggested by @bassosimone

fb84d3105b52ab3aedf920a4f9539dfffb006af2 authored over 5 years ago

Add comment

85d96e398bc08ed77b027e135ad75dcfa743702c authored over 5 years ago

Add metrics support

83d1f4735f875c65e56828b9f0338c62aaf98065 authored over 5 years ago

Add more metrics

042322a10387e40f0746c12272e5bc8484213299 authored over 5 years ago

Add docstrigs

fb8f265f4caeba6debf38c67bd85c103cdcc22de authored over 5 years ago

Add comment

bb4021f271813eb9ed734d0ef28129f8a144b6bd authored over 5 years ago

Merge pull request #216 from ooni/fixture-based-testing

Fixture based testing

18165f06468fc3d8b905372b3b74651d96e3fed2 authored over 5 years ago

Use repr instead of str

4bb42e317e7b86da0a27822647dafeed3b0d0541 authored over 5 years ago

Add comment and drop print debug line

3091336101ad51985838cfb3107187500710667e authored over 5 years ago

Fix bug in marking of confirmed and anomaly flags

The measurement_meta and report_blob tables are only filled with data on
ingestion and re-ingest...

24d4424e1f63ac45337baff309da7b817181102d authored over 5 years ago

Merge pull request #219 from ooni/shovel/readme

Add some notes about the shovel

d0be977b7db30de83f7bbfcd9640348b86fbb6cf authored over 5 years ago

Print the http_request_fp table count

f301e1d4c5a88d3a9386bf6969b9b8fc2ec5a909 authored over 5 years ago

Fix bug in centrifugation

b1d55580bd7cab08f2af313ef3f006d9f2b5c956 authored over 5 years ago

Handle header values which are base64 encoded

75db59c375c85027b7188625cbcae04f87f40e4f authored over 5 years ago

Add some notes about the shovel

de34c445905135c34adc680fef709bc103c5ead2 authored over 5 years ago

Add some more debug output

fd8d20e82ae937f0cdddad31933cc5aab4f954ab authored over 5 years ago

Add a fast path for the fingerprints

6d0e92963519b754f1b1bcf21f1f59f10bf97ed6 authored over 5 years ago

Utility functions

* Add function to print query output
* Create working_dir inside of the repo root

d368bc9377beb64fc0e179300e1645e550abf173 authored over 5 years ago

Only set confirmed = NULL

b6c4d5a836c194f91426489f422bcb285bef5db8 authored over 5 years ago

Fix typo in centrifugation

f8e357a4878143f84fe846db0979d5035a2f95e1 authored over 5 years ago

Move tests into Makefile

ecb42518f01b83cecaa6b82ef3f533cf20624dbe authored over 5 years ago

Refactor the integration tests to use a temporary directory

ef9799658c2b3c9f392895c633c3dea0666746e2 authored over 5 years ago

Drop caching as travis fetches from s3 anyways

See: https://docs.travis-ci.com/user/caching/#how-does-caching-work

9ff24d2b1edf169fa444d1cd1add59f08c04b7d4 authored over 5 years ago