github.com/ooni/backend commits | Ecosyste.ms: OpenCollective

encode binary as hex, stringify list keys

0455bf9c030a75f2f05622c4d196d9a6a570d93f authored about 9 years ago

streams_to_db task

3626885e6b55aaa1c1bec41c53e78d952d6dd9f6 authored about 9 years ago

pull out aggregator/sanitiser

8263a28d18a9c01338f668332b99a108ef4bc7d3 authored about 9 years ago

fix logging

86e9ec1ce72126fa325fe4ac5f0326f57311d43f authored about 9 years ago

bugfixes in directory walker

1301b51947afdf3658c77ca8c79c36b7a792baf7 authored about 9 years ago

rm redundant move_reports (awscli does it better)

2d7afe897e41bd18d0885c98323375bce0d15931 authored about 9 years ago

rm ansible/start_computer stuff (goes to *-sysadmin)

fe26ab124e946f62ec865c17f51f73663f39e01c authored about 9 years ago

remove remote_syslog cruft

7136fb5852968b98876dfc5ad80881d3bc74fe67 authored about 9 years ago

bin by upload date

786ddcb6833bbbb8b9544ce96d02157e0c2a6c5d authored about 9 years ago

simplify sync_reports, decouple from start_computer

219b3e9945494c4b8e957504c6373bd0410fef1c authored about 9 years ago

TODO: get timer decorator to play with pyinvoke

c10134ccc0fcf482df7ce168fbbcbdcce5d40388 authored about 9 years ago

rename sync_reports to move_reports

02d06e0f4dcf7996c0adb9fb938f38853ef79f58 authored about 9 years ago

Add fix for the start_time key

* Remove unneeded imports

74da814f3b81b9f546e65d49e3677eaf71fba555 authored about 9 years ago

remove unused limit from upload_reports

f2ed4538f5d05b3984c5a9c016e7a9d55351d204 authored about 9 years ago

Do normalisation of test_name in report parsing.

Expand the keys of the DNS Consistency test

6409a211182a3003788512cba106cd01fd31f415 authored about 9 years ago

gitignore

0d359e50e3dfa3642d6177e2115fa28108174ca0 authored about 9 years ago

report body in postgres

29ee64f9cdd56c11416398aa751efeb329d86cfd authored about 9 years ago

schema change

8b699d6c9f443a982155bf32f27628abcedc7f03 authored about 9 years ago

Normalise some other test_names as well

10b53d953b2a423931947cf9605c073362f92791 authored about 9 years ago

simplify upload_reports and decouple from add_headers

6213d1010036200c3970a4d3b6f53236800ba688 authored about 9 years ago

fixes and incomplete test for move_and_bin_reports

9444e072cc8cb5c439124a27ae5121c60f0a65de authored about 9 years ago

rename upload_reports to move_and_bin_reports

f51a87cbfc62dbd46db7facfbea6647df98857be authored about 9 years ago

Insert the star_time using TIMESTAMP field type

dd1db1c955815d74ccc3b9608be86fb6a8387b43 authored about 9 years ago

Start adding support for reporting via JSON

a592986dfa5a79b8d4f0d11c44a1e18cdf46938b authored about 9 years ago

Implement fix to failing import of psycopg2

e6fe45734bc74c390e6f0912004a6f28c72979a5 authored about 9 years ago

Delete Procfile since we don't use heroku anymore

bfa6da08f04a55fd728396d2b7dc7743a322a89a authored about 9 years ago

Merge branch 'master' of github.com:thetorproject/ooni-pipeline-ng

* 'master' of github.com:thetorproject/ooni-pipeline-ng:
s/ooni-pipeline-ng/ooni-pipeline/g
...

32993d3a49bbcb69bea8a44c0ed3e9218edb9365 authored about 9 years ago

Hotfix to missing test_name attribute

8411b8b6f141e39185173803708b7c958f24a496 authored about 9 years ago

Merge pull request #4 from TheTorProject/bugfix/remove-ng-suffix

s/ooni-pipeline-ng/ooni-pipeline/g

2daf29e03add0d5ef426cf2159637c2b5e25a5ce authored about 9 years ago

s/ooni-pipeline-ng/ooni-pipeline/g

9857b2014e064dd2450e5ddbefce24230f712b85 authored about 9 years ago

Merge pull request #3 from TheTorProject/fix/ooni-pipeline-ng

Fixes to support new versions of luigi and aws. Explicitly install gcc and python-devel

d6c7bc427ac8ee62dc206cc850bb6e8e0e90e21d authored about 9 years ago

Fix some more typos

c495a30e2d65ba937417988381862c11bb6977c3 authored about 9 years ago

Fix command with --date-interval option

2ba6df60a78746ebc92ee7e98c4a12d95ba5584d authored about 9 years ago

Fix typo in invoke-command

03dd89b5bcf4206e2a5d1cfcf5e26b7ef207b4e6 authored about 9 years ago

Update docs detailing how to re-run the pipeline from scratch

924a043a796fbe9da7ac66f5500b3621fdddecf5 authored about 9 years ago

Add multiprocess flag to use multiple cores in new version of luigi

453c339697e531b303f6e4ddd096fd1ddc3482c3 authored about 9 years ago

Make mkdocs a development requirement only

6123df3ed918cc2a42d1e7aa5f36d7d0b50bd33a authored about 9 years ago

Explicitly install gcc and python-devel

Replace ooni-pipeline-ng with ooni-pipeline

72c71e3f68b8974ca9b140c2099ba0521a62be74 authored about 9 years ago

Merge branch 'feature/docs'

* feature/docs:
Add description of the invoke.yaml config file

fdeff7f57278019f72d1f66c37aa8a0e3410844a authored about 9 years ago

Add description of the invoke.yaml config file

89b12492505ff2cb3b2caa4500abe7e6eca18947 authored about 9 years ago

Merge pull request #2 from TheTorProject/feature/docs

Feature/docs

ec0c586275d38126347157d6050276671c35c98a authored about 9 years ago

Make Readme a symlink

bd098dfac083d42281d1e9372ed8ea87b032952f authored about 9 years ago

Merge branch 'master' of github.com:thetorproject/ooni-pipeline-ng into feature/docs

* 'master' of github.com:thetorproject/ooni-pipeline-ng:
Catch missing invoke.yaml earlier; fi...

74655e8576c23598ddbfd8842c658fbb566146ed authored about 9 years ago

Add mkdocs based documentation for ooni-pipeline

8065e50ea4f0613e4de092dc5358b47426ffe6ec authored about 9 years ago

Catch missing invoke.yaml earlier; fix name error

95e8ae91df8cbbeaef77633a053f5606d0e7ce81 authored about 9 years ago

Add more documentation to the ooni-pipeline

29852f5360eccacbd243683546420a94a2c7ad45 authored about 9 years ago

Merge pull request #58 from dstufft/patch-1

Update the pip recommendations

479a1bb154037b834292ccc4b3d593d1472b44de authored over 9 years ago

Update the pip recommendations

* Recommend at least pip 7 because versions since 1.3 have changed
the defaults to better pro...

af27fb6f11731f4124813a9bce8953a9e886f3f5 authored over 9 years ago

Terminate the ec2 instance after we are done with it

740338b2a834b1775708081595082dc2b242e9a6 authored over 9 years ago

Run the invoke command directly

11e2430226c1b912bf8b58fd286a500065d19ef2 authored over 9 years ago

Return uploaded_dates in task too

1c8891497a27dcc5d60a50941f98559391ddd854 authored over 9 years ago

Fix typo

b4c0942309b5770c9214114007281c6d26a4a921 authored over 9 years ago

It's takes 2 os to make one dirname

07532394e54e5af553565f2092591fafddc4086b authored over 9 years ago

Re-run the add_headers_to_db command on all the uploaded reports (including older ones)

920856bf600a2e0ea375ccc370ea0acbb853911c authored over 9 years ago

Fix typo and a very dangerous bug

517bcf92873701724630fd6383b79423d9311b23 authored over 9 years ago

Run commands to setup remote_syslog as root

0d3a36ab27e86256d0100491c3fdca6c2ddc5b9d authored over 9 years ago

Start remote_syslog from the ansible playbook

10ac8c1c9137afe104632b84d9a58e60b03cc6b7 authored over 9 years ago

Run the run.sh command directly inside of VM

fcbfa9bee6c3d9fac326726da56b0d675b1bcdd3 authored over 9 years ago

Only sync today

e618c79573a6cfe6c717ad4e000aa6b2789f7cca authored over 9 years ago

Pass the move argument

717cf5e156f72671a170de06213ad69b96158142 authored over 9 years ago

Add debug output of command

4e1d7a00b5f0521103f91893256aee040bb8991a authored over 9 years ago

Halt also in case of failure

03189cc22321f34314c2fa23b5b281bed4e8cd0b authored over 9 years ago

Refactor command

640430207c1dca327087d6b6ec18b0a2bfdea0f9 authored over 9 years ago

Pass extra_vars as dict

6103be6c7c5d3366614fd9c4c2c27142240dda97 authored over 9 years ago

Move the upload_reports command to the compute node

16d093c0b9850365a618fe581dd726310177a00e authored over 9 years ago

Merge branch 'master' of github.com:thetorproject/ooni-pipeline-ng

* 'master' of github.com:thetorproject/ooni-pipeline-ng:
Add diagram to reamde

a986cfc07f485e5e139509354855a7f0b45d772e authored over 9 years ago

Add support for passing a invoke command to start_computer

72959e170dd7ce83a9115039be10e51ec9dea246 authored over 9 years ago

Add diagram to reamde

83820603e10a9691de188ee0907c36203364066c authored over 9 years ago

Add ooni-pipeline-ng architecture diagram

be2442d84caa263b2a7f1f995d0cdedcecd402b0 authored over 9 years ago

Use relative paths

8cbed8f69bfa1747a3ce30b94165ef2302deaff9 authored over 9 years ago

Add task that sets up logging before running any task

6477b4e9ea67310e0924a39eb019164619eb4cf0 authored over 9 years ago

Move reports when sync

* Consider also alternative test_names

ccff7d0b42775af95b3419b46b7a236224f83d91 authored over 9 years ago

Also consider synonyms

3e1e9c6c94c34cbff516406d15d1ee16aa3f8985 authored over 9 years ago

fix typo

7e1858e67c1342322c17da212a9b5dcc96e24362 authored over 9 years ago

Spark tasks should use less workers

b3f8fc041ff94b29310bab7c889a751c497a1884 authored over 9 years ago

Improvements to spark tasks

46071fe0786a0aa0d08be3d90be0396ab49a5101 authored over 9 years ago

Do not magically detect the schema

fefed8ac404556d6fd32f8db5da6abd044ffc95d authored over 9 years ago

Get imported dir elsewhere

7db811d390d2238fbcc8721a457d1922c4ee66ad authored over 9 years ago

Add support for specifying timerange and dumping to DB

Make the OONI tasks cleaner

e314a97587eebef1c786f91d762ace4a3786543b authored over 9 years ago

Set environment variable python path

086bdbbb7b45e2fe5f382b381d9aefbfbfd5b48f authored over 9 years ago

Pass pypackages as attr

e3ec8a6cecc8c54f6aa1274667ad3a31b70037c2 authored over 9 years ago

Pass the pipeline package to spark

ff87346cb2042ee473e745afd1447babfa497959 authored over 9 years ago

Fixes to spark task

57ac02f0fd3eec55603830e479321ebade768394 authored over 9 years ago

add spark apps task

c030f434fa9791fcb6cdfcf838558522aa641b4b authored over 9 years ago

Actually move the report files on copy

80305f09f45db673cb0e5abf73a7d795b4f03262 authored over 9 years ago

Pass src array with comma separated lsit

a1ec3720eb77c4d5ab16c14f6c8c8711acd3906e authored over 9 years ago

Remove unneeded imports, pass recursive argument

808b82a38f33151c3af0db7caeea35c195473b3f authored over 9 years ago

You actually need a little bit more

28a9eff2f0edcff3493229788e0ec1a5f3e3b68e authored over 9 years ago

No need to say true

a9cce32cdbc46c33581135e5e40fcb989280815b authored over 9 years ago

Add support for halting machine after the task is run

0450c84c3372afc5d7d09d7b29e08e23a911497c authored over 9 years ago

Fix init script and run template

2fcadc117e992d6a78ecf6a987eef29e6b464b1a authored over 9 years ago

Convert run.sh into a template

f720d6482951a416e150620f2ecf34c7dc19864b authored over 9 years ago

Copy the run.sh over

452749e7a89b4a1958f16b67d84b4944f9d4888b authored over 9 years ago

Fix init script for centos

8c443626bb48b0430b648d7fb1fa157c5c5ce848 authored over 9 years ago

It appears I can't type..

df40da9d5ef4ffce9918dca3d82aa40a2454b1ea authored over 9 years ago

fix playbook

d33f682663ca099a58a83c5aa737414ddbf7b920 authored over 9 years ago

Fix playbook

281a0cfa70bb3f8371c0091b0aee26a3b14952cb authored over 9 years ago

Fix playbook and make start_computer task simpler to use

c4a96854d714d3b1dff62e99e5cb93d1d6de5b7a authored over 9 years ago

Use a custom run.sh script

cca99e10460507c96b31d5d7eab9431bc45ef464 authored over 9 years ago

Add support for running as a Heroku worker

Fixes to playbook.yaml, add template for init script and fix path to requirements
Add procfile
A...

c075cf05ce19204d7b3a7a576e97d48bb2a6f109 authored over 9 years ago