github.com/ooni/backend commits | Ecosyste.ms: OpenCollective

Add minimal flask HTTP API to support synching reports

Refactor requirements.txt

377516fe1972fb140278cbcb3e0b4cf6eb58b1c2 authored over 9 years ago

Implement service for pipeline

Fix some bugs in the sanitise task

87d97d1b90b14a2f5d4e0b632b2bc33aed42d215 authored over 9 years ago

Fix missing import

cb84225159d786374bd6a1cae35db5a19737a947 authored over 9 years ago

Add aws configuration

9f3627bfc3287d1e79b79a3ceaaee73efc901575 authored over 9 years ago

Add support for moving when uploading reports

490eb3e23ecbd671b7e2ec337f9eb15777d81497 authored over 9 years ago

Fix private key permissions

080ade1f598b1df45070db4214af2c16ddc24d87 authored over 9 years ago

Fix some problems with ssh:// targets

b5d016dd94619c90bb60af889cdc97fd8ee6362d authored over 9 years ago

Ignore verification of host key for collectors

8f67031fa308bd460fea4cb61687f1020872b99b authored over 9 years ago

Add task to sync reports from collectors

1e189626807ccf889bf06efa7955ec47ccb7ba1e authored over 9 years ago

Add script to add reports to elastic search

Add links.txt
Cleanup gitignore

d245ed5576a2a39b0aa5be72f930ac09e841467b authored over 9 years ago

Add private directory

1fe9a78b43b935dc637fdbc68e082585d16a056e authored over 9 years ago

Refactor to avoid passing bridge_db to class

* Refactoring of the AWS compute node task runner

ec5a044a278851ff686a2d5bf8698e0391daaed3 authored over 9 years ago

Remove my fork

1461b0e0c17fe5bb1ec2dda8e3162bd7ac128b0c authored over 9 years ago

Remove unneeded options add needed

70c0088c27ccb0e349c612e3b39083c53171e417 authored over 9 years ago

Add support for launching aws compute node to add reports to database

1fc6527289bd012b1eb20ecf5f02ee7f5dcaeb53 authored over 9 years ago

Implement batch task to add report headers to Postgres

d249173ae5cac8bbe3ff3eaf376d3ded4eb79b16 authored over 9 years ago

More robust exception handling

e982a494ef0720df5988c032b1e86057c83050cf authored over 9 years ago

No more need for ad-hoc http_request fixing method

975ce90b951f8dd03deead510529bed2024b4ef8 authored over 9 years ago

Ignore utf-8 errors

065e0ad1c42de86555b7f70a5975c5dfbabf3a40 authored over 9 years ago

Add init to lib

5695038c58886dcc1d96369ed806322b7496d513 authored over 9 years ago

Use custom simplejson with overriden encoder

5dd06f2af06c17b296649a5755fedac224441c97 authored over 9 years ago

Add simplejson from c2190efda0e6fdd6783a56a6aeab818b072350a1

623581e75fee7dd336eb9345dfc8eb7b16d8a330 authored over 9 years ago

Fix entries at the root of the problem

d10ffc35683ef8a9ab37984a1948965ffbd65555 authored over 9 years ago

Remove terrible hack

d1c01a7b92cd2dc4b6677dd8f82f6da001a462e3 authored over 9 years ago

fix path

af702a71d757569bebffa4589c02910b9c15569c authored over 9 years ago

s/name/path/

f534ce27f568e9fa3461b1f5dd51fa6fd53b8b8c authored over 9 years ago

More robust sanitising

Add report_filename key

ea61d0a6e77c053f51cb5be6a90fadf136798272 authored over 9 years ago

All you need is log

46c65d4384dc642d049f3d71c13466deefd38528 authored over 9 years ago

Simple json will be faster? A little bit.

5314be5a3dfd865f6aca46d2bfb1c42ae351c129 authored over 9 years ago

Use ujson for serializing json

32015d244a7216b87d0d5e0744e29fef6e180977 authored over 9 years ago

Add a task to clean the generated streams

d80c92e2cf0b2ebb9e77190bf2fc8fb91779779e authored over 9 years ago

Add support for logging and timer

5d4dc8a95fb7b4eecfb10199cdf3fc6c022c62a5 authored over 9 years ago

Fix get_luigi_target

23b57aac85a2a836c6355ff34775b8c4cfeadb9a authored over 9 years ago

Refactoring of sanitisation task

9e17cc86f661bf5cd370ff08803052a759db38b7 authored over 9 years ago

Add default bridge_db file

a5cde439ad8d2e48db9fbc7f4477540167914cef authored over 9 years ago

Fix parsing of some legacy filename formats

4bdea9cba871cbda444bffbd3707dac3c07539ca authored over 9 years ago

Add generate streams to the namespace

cf7cd2e76ea622643aa41b131f320d55174ca8a2 authored over 9 years ago

A filesystem is actually sort of a like a database...

b437774d14c25fd76199485ccec542879de0b056 authored over 9 years ago

Cast limit to int

66af75f28ae9af2f6ba1903b44a79e89dfd69883 authored over 9 years ago

Add support for parsing old report file format

aeda1e839507fc9aabf0e268216606677d1c40f7 authored over 9 years ago

Other approach to config file parsing

7875d25fc68d6e753ef87edde8833246b9dbcff3 authored over 9 years ago

Implement walking of s3 directories

Write luigi config from invoke

adc9a36b427951af8aa61a3f6c0ed348f611d17d authored over 9 years ago

Add support for generating sanitised versions of the reports

fac36b171f6d4f9291b5f83b5951c831e7e02f33 authored over 9 years ago

Add support for parsing some older formatted reports

6ccd59877f1771ab2c0c9717d2f0a97b34a8a423 authored over 9 years ago

Fix parsing of the ASN part for old reports

Fix config file parsing

3d62941fcbed3500a5deac82e6c2bdfe4a7e045d authored over 9 years ago

Pass ctx to create_luigi_config

9afbe6a3a089dfb0181e061e24a4cc5b2b7fe87c authored over 9 years ago

Increment index immediately after the check

bd324d5c231b8328559b1c3f3cbef0b12c0777cd authored over 9 years ago

Write file that failed to run the complete method to a file prefixed with ERROR-

ca39ad825fff4e951ebef475d355c132cb9d0e72 authored over 9 years ago

Pass all arguments with keywords

0b1e34891f33dc653a6ac087119887e05ca3f8e1 authored over 9 years ago

Fix generation of parts

b1cd127ec9e8b98122a80f72eef6e0ed3fb192bb authored over 9 years ago

Add script to add files to hadoop

a9b62bf8eb5a374786d36610ad596ac5ddeb50b2 authored over 9 years ago

Properly call run_process

31f334f466afef1d3f8f4249346d7207ccd5661c authored over 9 years ago

Remove the reports prefix

75651d3cb4df9abc1e90449245f96f3db11ca6fc authored over 9 years ago

It's a method of key

8c867a83037879fcf24ae7d86e50025aa3e838f7 authored over 9 years ago

I actually want a newline

d8db40247ca6fe1cc932b97bc4682e44ddb40957 authored over 9 years ago

Add s3 workflow to copy reports

1270953e111a768c625e80c450312bb446092da5 authored over 9 years ago

Place files into subdirectories

762956bd4caee721359922d6a27ba421c426fb5a authored over 9 years ago

Need to seek to 0

dea3730f36bb55b586637b2e95936b684f8acb90 authored over 9 years ago

Re-ordering of directory structure

1687d20d4402b020c21fda8bdf8f61ba1964b2a3 authored over 9 years ago

Implement invoke based task to upload raw reports in a certain directory to s3

2a6569cc2a0ef8deda27fea433c1c2890508b32d authored over 9 years ago

Don't limit it to only 16 reports

1f0e1273e06556155a2f61829252d3d0738fb51d authored over 9 years ago

Add unittests for workflow engine

cfaa34a36fda54d68bb0f87d971d69cb27d96f64 authored over 9 years ago

Call upload

b09a330252390d391da1be39c9e83571562947de authored over 9 years ago

Add workflow to upload reports to s3

bdaba1e64434416b1a64556c1936e24e1e786566 authored over 9 years ago

Append to file instead of creating a new one per entry

2129078b3cdbb36b6e251528f4da6170bad38c1d authored over 9 years ago

s/cached_bucket/get_bucket/

e94ef8937b69ce27fc157f8c9bec55823d6bae26 authored over 9 years ago

Add minimal logging

7b658db1f1c2d8d0fdc64dc4510fe62d594e8719 authored over 9 years ago

Also process raw data

369d575d75e94864fef8725e43f98551bb5b5749 authored over 9 years ago

Improve binario

73cbf2816fb6e9eb97562ee4145c17c432e1c22c authored over 9 years ago

Write to .sanitised

ece22c3aeb34acf63b7a234fea86b57e527639a9 authored over 9 years ago

Add fail log

0d6390ff5df27e09a13f9122a94f42b1054a2240 authored over 9 years ago

Don't add it twice

c0153a83dfee6927e5caffb7b69ead0395321b16 authored over 9 years ago

Make binario use 1 process per concurrency option

7ee28dbf31c9ace438cbb8e517c158aaf0f4b535 authored over 9 years ago

Fix calling if output

9c2ebc86ec5390aef2532a23b002d3e9c6762ef5 authored over 9 years ago

More robust iteration

638c82fe088d852228add809ab5a5859fd502ff9 authored over 9 years ago

No printing like that

80441c3144d10ef37651c7eef9933ab299e3c72b authored over 9 years ago

We actually yield

a2215469c587fc44d2d618fe616160ba44fe6d76 authored over 9 years ago

Kafka commits

23cb2b6daeb1bfda9cd318321f529c481635e059 authored over 9 years ago

Fix it

8823fe21f84682442b5d9f84af0e8c91852e7200 authored over 9 years ago

Could this be it?

8e9567e2aa3cde1e0baa58358dc27176e3675ab7 authored over 9 years ago

No double generators

396567807754b54ac6d612282a31ebd34ab8ef1a authored over 9 years ago

Add missing function

4b80f5f309d69a509d1c62239788587d9becaab2 authored over 9 years ago

Start it already

36a5bdb76c80ac58c3f5952e4b62d6f4094f6a72 authored over 9 years ago

Iterations on the workflow engine

8148d5ce8bf52b6a92f1b7785faa0249ff828585 authored over 9 years ago

Add binario workflow

4cd191e35f0bbcfaa6ba0d4c730e8439201d8cdb authored over 9 years ago

Import traceback

0de2b2c4b7d02e0b640dfe734c8a2a479275d568 authored over 9 years ago

Adjust parallelism. Print the traceback.

d77c9ca9692da6d269980df49fc3513d6eae7753 authored over 9 years ago

Fix bug in sanitisation procedure

19fe34bc2748494df235835d2d9a0f8fa969012e authored over 9 years ago

Print the traceback..

9196365ac78f2f7e3924fabe78528646693fe59f authored over 9 years ago

Carry the exception along..

85e956d344d79c35847f7f114cdbf6de7b1f0fd2 authored over 9 years ago

Add function to skip over misformatted YAML reports

686a2192622a32e446eeaf43c4b3a8df38f8f506 authored over 9 years ago

Debug various problems with the pipeline by running it on the test cluster.

* Need clj in path
* Set all the dependencies
* Wrong topology
* Fix imports and names
* Add src...

72ef69711dbf6b1e77bd83f20e2a5bd5ddf1b207 authored over 9 years ago

Rename to report (singular)

9ada3384338330103beef11d4aa4978f7815b469 authored over 9 years ago

Update README

15ad5a05584c15c7ec815b3f86d430b67b794751 authored over 9 years ago

Add aggregate task chart

b6e504efb15fc0beb5bbaeaa70ef8c472e380dd3 authored over 9 years ago

Refactor pipeline to use a KafkaSpout and 2 Bolts

18e849be6ab0ec5320d56d885f1906c481172d1b authored over 9 years ago

Fix typos

6f73b236b733c23f94572cdc63b360deba66ad27 authored over 9 years ago

Properly move kafka messages

1869221ffee25861c5df000e398c664cf45172e2 authored over 9 years ago

Update project.clj to use the hortonworks repo

Thanks to @dan-blanchard for the swift reply and fix

8df70710a485365a2657285b4c398d33d69860e2 authored over 9 years ago

Also add footer to report bucket

6efdae7d9e2eba85438600ca50401c36246a5355 authored over 9 years ago