github.com/ooni/pipeline commits | Ecosyste.ms: OpenCollective

Fix setting of the default value

c1d0ac22bf91e55748b25477ad323f859a3f834d authored about 9 years ago

Set the ID column to DEFAULT

3f6f7631cc6161ae9ca8b392e96ecd024f00dd2c authored about 9 years ago

Fix typas

f8f4b3944d1423b9f3403dc2893e207c704f859a authored about 9 years ago

Fix multiple primary key issue

d3472cfe6f395fcb50ca7a20d7677aa6f74ccedf authored about 9 years ago

Add support for creating indexes and refreshing the materialised views

f96b6a8a448ba29fff265a67ee6957bf18c79fbd authored about 9 years ago

Merge branch 'joe/redo' of github.com:thetorproject/ooni-pipeline into joe/redo

* 'joe/redo' of github.com:thetorproject/ooni-pipeline:
clean ooni-public/json

99a6267f275299005fe51e28d5cb2c16c6ac588e authored about 9 years ago

Refactoring of the streams to db function

f9a73c779e50fe5bc9e206e09e316b7ce5a0fa41 authored about 9 years ago

clean ooni-public/json

2891cd09cfc8e4700e0c27a447e9ca9c2d7e4e97 authored about 9 years ago

Do some re-ordering of the http_requests tests so that the experiment is always the first in the list

ba13e21b0298e3572425c0b8e804f28a53b5003d authored about 9 years ago

Add measurement id primary key

03781b3326a3718e8b9ed9be118b521b0a01993b authored about 9 years ago

Fix bug in poping of response body

* Replace tabs with whitespace

ede73218c9ae94278c50b5e082176acca9367655 authored about 9 years ago

learn to python

7f99dcd83567e798e8e0d587a599dde99c87c11b authored about 9 years ago

don't stick body in db if it has null bytes

daa5f9d8f93ed8eac6d1db41c35549af283928b3 authored about 9 years ago

add some more columns

6ff6a55e14283a2301a4ac22ae5a8ee456231202 authored about 9 years ago

catch an early StopIteration exception

d5e2e48a027155c5fee5234fe717851677b65108 authored about 9 years ago

make content-length more accessible

4261ab1aa7e0e671cca4f2649b8c3c734308d0f0 authored about 9 years ago

string encoding handling

749a0471dd49163313ea0c1d87f04d7b3dcee60f authored about 9 years ago

better programming

09c698a83c65b30552cf3f9514057edca460a6af authored about 9 years ago

tweak streams_to_db

4ba84002427ed84de14d9f9ebb62073325ff2ab8 authored about 9 years ago

encode binary as hex, stringify list keys

0455bf9c030a75f2f05622c4d196d9a6a570d93f authored about 9 years ago

streams_to_db task

3626885e6b55aaa1c1bec41c53e78d952d6dd9f6 authored about 9 years ago

pull out aggregator/sanitiser

8263a28d18a9c01338f668332b99a108ef4bc7d3 authored about 9 years ago

fix logging

86e9ec1ce72126fa325fe4ac5f0326f57311d43f authored about 9 years ago

bugfixes in directory walker

1301b51947afdf3658c77ca8c79c36b7a792baf7 authored about 9 years ago

rm redundant move_reports (awscli does it better)

2d7afe897e41bd18d0885c98323375bce0d15931 authored about 9 years ago

rm ansible/start_computer stuff (goes to *-sysadmin)

fe26ab124e946f62ec865c17f51f73663f39e01c authored about 9 years ago

remove remote_syslog cruft

7136fb5852968b98876dfc5ad80881d3bc74fe67 authored about 9 years ago

bin by upload date

786ddcb6833bbbb8b9544ce96d02157e0c2a6c5d authored about 9 years ago

rename sync_reports to move_reports

02d06e0f4dcf7996c0adb9fb938f38853ef79f58 authored about 9 years ago

simplify sync_reports, decouple from start_computer

219b3e9945494c4b8e957504c6373bd0410fef1c authored about 9 years ago

TODO: get timer decorator to play with pyinvoke

c10134ccc0fcf482df7ce168fbbcbdcce5d40388 authored about 9 years ago

Normalise some other test_names as well

10b53d953b2a423931947cf9605c073362f92791 authored about 9 years ago

simplify upload_reports and decouple from add_headers

6213d1010036200c3970a4d3b6f53236800ba688 authored about 9 years ago

remove unused limit from upload_reports

f2ed4538f5d05b3984c5a9c016e7a9d55351d204 authored about 9 years ago

Do normalisation of test_name in report parsing.

Expand the keys of the DNS Consistency test

6409a211182a3003788512cba106cd01fd31f415 authored about 9 years ago

rename upload_reports to move_and_bin_reports

f51a87cbfc62dbd46db7facfbea6647df98857be authored about 9 years ago

gitignore

0d359e50e3dfa3642d6177e2115fa28108174ca0 authored about 9 years ago

Insert the star_time using TIMESTAMP field type

dd1db1c955815d74ccc3b9608be86fb6a8387b43 authored about 9 years ago

fixes and incomplete test for move_and_bin_reports

9444e072cc8cb5c439124a27ae5121c60f0a65de authored about 9 years ago

Add fix for the start_time key

* Remove unneeded imports

74da814f3b81b9f546e65d49e3677eaf71fba555 authored about 9 years ago

schema change

8b699d6c9f443a982155bf32f27628abcedc7f03 authored about 9 years ago

report body in postgres

29ee64f9cdd56c11416398aa751efeb329d86cfd authored about 9 years ago

Implement fix to failing import of psycopg2

e6fe45734bc74c390e6f0912004a6f28c72979a5 authored about 9 years ago

Delete Procfile since we don't use heroku anymore

bfa6da08f04a55fd728396d2b7dc7743a322a89a authored about 9 years ago

Merge branch 'master' of github.com:thetorproject/ooni-pipeline-ng

* 'master' of github.com:thetorproject/ooni-pipeline-ng:
s/ooni-pipeline-ng/ooni-pipeline/g
...

32993d3a49bbcb69bea8a44c0ed3e9218edb9365 authored about 9 years ago

Hotfix to missing test_name attribute

8411b8b6f141e39185173803708b7c958f24a496 authored about 9 years ago

Merge pull request #4 from TheTorProject/bugfix/remove-ng-suffix

s/ooni-pipeline-ng/ooni-pipeline/g

2daf29e03add0d5ef426cf2159637c2b5e25a5ce authored about 9 years ago

s/ooni-pipeline-ng/ooni-pipeline/g

9857b2014e064dd2450e5ddbefce24230f712b85 authored about 9 years ago

Merge pull request #3 from TheTorProject/fix/ooni-pipeline-ng

Fixes to support new versions of luigi and aws. Explicitly install gcc and python-devel

d6c7bc427ac8ee62dc206cc850bb6e8e0e90e21d authored about 9 years ago

Fix some more typos

c495a30e2d65ba937417988381862c11bb6977c3 authored about 9 years ago

Fix command with --date-interval option

2ba6df60a78746ebc92ee7e98c4a12d95ba5584d authored about 9 years ago

Fix typo in invoke-command

03dd89b5bcf4206e2a5d1cfcf5e26b7ef207b4e6 authored about 9 years ago

Update docs detailing how to re-run the pipeline from scratch

924a043a796fbe9da7ac66f5500b3621fdddecf5 authored about 9 years ago

Add multiprocess flag to use multiple cores in new version of luigi

453c339697e531b303f6e4ddd096fd1ddc3482c3 authored about 9 years ago

Make mkdocs a development requirement only

6123df3ed918cc2a42d1e7aa5f36d7d0b50bd33a authored about 9 years ago

Explicitly install gcc and python-devel

Replace ooni-pipeline-ng with ooni-pipeline

72c71e3f68b8974ca9b140c2099ba0521a62be74 authored about 9 years ago

Merge branch 'feature/docs'

* feature/docs:
Add description of the invoke.yaml config file

fdeff7f57278019f72d1f66c37aa8a0e3410844a authored about 9 years ago

Add description of the invoke.yaml config file

89b12492505ff2cb3b2caa4500abe7e6eca18947 authored about 9 years ago

Merge pull request #2 from TheTorProject/feature/docs

Feature/docs

ec0c586275d38126347157d6050276671c35c98a authored about 9 years ago

Make Readme a symlink

bd098dfac083d42281d1e9372ed8ea87b032952f authored about 9 years ago

Merge branch 'master' of github.com:thetorproject/ooni-pipeline-ng into feature/docs

* 'master' of github.com:thetorproject/ooni-pipeline-ng:
Catch missing invoke.yaml earlier; fi...

74655e8576c23598ddbfd8842c658fbb566146ed authored about 9 years ago

Add mkdocs based documentation for ooni-pipeline

8065e50ea4f0613e4de092dc5358b47426ffe6ec authored about 9 years ago

Catch missing invoke.yaml earlier; fix name error

95e8ae91df8cbbeaef77633a053f5606d0e7ce81 authored about 9 years ago

Add more documentation to the ooni-pipeline

29852f5360eccacbd243683546420a94a2c7ad45 authored about 9 years ago

Terminate the ec2 instance after we are done with it

740338b2a834b1775708081595082dc2b242e9a6 authored over 9 years ago

Run the invoke command directly

11e2430226c1b912bf8b58fd286a500065d19ef2 authored over 9 years ago

Return uploaded_dates in task too

1c8891497a27dcc5d60a50941f98559391ddd854 authored over 9 years ago

Fix typo

b4c0942309b5770c9214114007281c6d26a4a921 authored over 9 years ago

It's takes 2 os to make one dirname

07532394e54e5af553565f2092591fafddc4086b authored over 9 years ago

Re-run the add_headers_to_db command on all the uploaded reports (including older ones)

920856bf600a2e0ea375ccc370ea0acbb853911c authored over 9 years ago

Fix typo and a very dangerous bug

517bcf92873701724630fd6383b79423d9311b23 authored over 9 years ago

Run commands to setup remote_syslog as root

0d3a36ab27e86256d0100491c3fdca6c2ddc5b9d authored over 9 years ago

Start remote_syslog from the ansible playbook

10ac8c1c9137afe104632b84d9a58e60b03cc6b7 authored over 9 years ago

Run the run.sh command directly inside of VM

fcbfa9bee6c3d9fac326726da56b0d675b1bcdd3 authored over 9 years ago

Only sync today

e618c79573a6cfe6c717ad4e000aa6b2789f7cca authored over 9 years ago

Pass the move argument

717cf5e156f72671a170de06213ad69b96158142 authored over 9 years ago

Add debug output of command

4e1d7a00b5f0521103f91893256aee040bb8991a authored over 9 years ago

Halt also in case of failure

03189cc22321f34314c2fa23b5b281bed4e8cd0b authored over 9 years ago

Refactor command

640430207c1dca327087d6b6ec18b0a2bfdea0f9 authored over 9 years ago

Pass extra_vars as dict

6103be6c7c5d3366614fd9c4c2c27142240dda97 authored over 9 years ago

Move the upload_reports command to the compute node

16d093c0b9850365a618fe581dd726310177a00e authored over 9 years ago

Merge branch 'master' of github.com:thetorproject/ooni-pipeline-ng

* 'master' of github.com:thetorproject/ooni-pipeline-ng:
Add diagram to reamde

a986cfc07f485e5e139509354855a7f0b45d772e authored over 9 years ago

Add support for passing a invoke command to start_computer

72959e170dd7ce83a9115039be10e51ec9dea246 authored over 9 years ago

Add diagram to reamde

83820603e10a9691de188ee0907c36203364066c authored over 9 years ago

Add ooni-pipeline-ng architecture diagram

be2442d84caa263b2a7f1f995d0cdedcecd402b0 authored over 9 years ago

Use relative paths

8cbed8f69bfa1747a3ce30b94165ef2302deaff9 authored over 9 years ago

Add task that sets up logging before running any task

6477b4e9ea67310e0924a39eb019164619eb4cf0 authored over 9 years ago

Move reports when sync

* Consider also alternative test_names

ccff7d0b42775af95b3419b46b7a236224f83d91 authored over 9 years ago

Also consider synonyms

3e1e9c6c94c34cbff516406d15d1ee16aa3f8985 authored over 9 years ago

fix typo

7e1858e67c1342322c17da212a9b5dcc96e24362 authored over 9 years ago

Spark tasks should use less workers

b3f8fc041ff94b29310bab7c889a751c497a1884 authored over 9 years ago

Improvements to spark tasks

46071fe0786a0aa0d08be3d90be0396ab49a5101 authored over 9 years ago

Do not magically detect the schema

fefed8ac404556d6fd32f8db5da6abd044ffc95d authored over 9 years ago

Get imported dir elsewhere

7db811d390d2238fbcc8721a457d1922c4ee66ad authored over 9 years ago

Add support for specifying timerange and dumping to DB

Make the OONI tasks cleaner

e314a97587eebef1c786f91d762ace4a3786543b authored over 9 years ago

Set environment variable python path

086bdbbb7b45e2fe5f382b381d9aefbfbfd5b48f authored over 9 years ago

Pass pypackages as attr

e3ec8a6cecc8c54f6aa1274667ad3a31b70037c2 authored over 9 years ago

Pass the pipeline package to spark

ff87346cb2042ee473e745afd1447babfa497959 authored over 9 years ago

Fixes to spark task

57ac02f0fd3eec55603830e479321ebade768394 authored over 9 years ago

add spark apps task

c030f434fa9791fcb6cdfcf838558522aa641b4b authored over 9 years ago