github.com/ooni/backend commits | Ecosyste.ms: OpenCollective

Query object only exists in master branch of luigi

b3fd6a6a23e35d3a3ac68f0af7ea4877b2c08eb8 authored almost 9 years ago

Remove trailing s

79d92894b88d2bf1721a7ad80240dad0aa599702 authored almost 9 years ago

Add task to compress reports

20d72b74afb7b525533b1b3d3fd272cef98dff3c authored almost 9 years ago

Add tasks related to creating and updating the materialised views

2bac7a3e8b9c90387664a2abff23a646d219e7bd authored almost 9 years ago

Update requirements.txt

* Delete templates

846e9c6837f1b8d9ea9df7bcb421e5f26938ba12 authored almost 9 years ago

Update README with the most up to date version of the data pipeline.

* Delete unused code.

* Add logging configuration file.

af648a8317816ecbd2e549654ff9d7518069e6c9 authored almost 9 years ago

Delete more dead code

743681310f75e62486a604b0ba40d9037b8e243e authored almost 9 years ago

Delete all dead code

b95178b6aa668d0c19da81695097fbef27d07cc7 authored almost 9 years ago

Update README.rst

f556a9adb329a0a16858b25bb04b701ed59766e3 authored about 9 years ago

Fix sanitisation of input key

af54a0a21c26caf41e54917e7ea46d661927e214 authored about 9 years ago

Also hash the fingerprints of bridges not in bridge_db

96bb024477181766156b68462987b7b0935dd28e authored about 9 years ago

Set the report_id when it is not set.

35886543d7ee2f838989859bd791020b5f66cfd5 authored about 9 years ago

Normalise issuer and subject

a90a28debed5c2f594c3e8a07be5255030087eff authored about 9 years ago

Make getting the serial n. attribute lazy

13269664aa1347c97f7c02bcd25873617d8ac446 authored about 9 years ago

Encode serial number with hex

b6971761f01e32e6413f0d22e6f1aaf121b0e00a authored about 9 years ago

Encode serial number as hex

6c0c836c54f247a9d494b8b2a15f855149275dac authored about 9 years ago

Normalise double nested headers in legacy reports

193827a85a76cdecc62957b1d0a63ca45f241c4d authored about 9 years ago

Extra subargs to normalise the options of legacy tests

2210c544a86029ed5e3e53d209de4514781a9eca authored about 9 years ago

Make the handling of captive portal and tls_handshake tests more robust

2a418d64e232b7a0115a9538943ee22519c2d5d0 authored about 9 years ago

Sanitise tls_handshake test

c7bc01ee2830b50f88a11340bf6071bf8a55b6e8 authored about 9 years ago

Add tor_http_requests_test to httpt tests

58c27da009ba4840279da56887f6b8ec90c99df2 authored about 9 years ago

Add support for fixing tor_http_requests test

e3c2ee411593adefbaf4e3a748e9b4fa76073ff9 authored about 9 years ago

Encode unicode data as UTF-8

1a100678fea5d6ce68ec84b2a951b6c0f1369253 authored about 9 years ago

tabularize

7b32f5c5f8b32c689f63d815efc50b95a599d324 authored about 9 years ago

if regex not found, empty string

f66d120dfab8108c48c786095776662aa32e6d10 authored about 9 years ago

Do not perform the conversions to JSON in the backend, but only write to JSON files when it's of file type JSON

1f2ec237404c7bc5fa0f5d6d8375256e07bda9f8 authored about 9 years ago

log when trying fallback yaml loader

4dd67bb8710b42521e166ade0f18eabcd3a1049d authored about 9 years ago

Do the conversion of YAML to JSON on the submission stage

973297a0ad8f5093ec9c08954a4d8419f50bd5d7 authored about 9 years ago

squash another runtime exception

3e55b7212ed7c723372d6ec03e33305b0605b3c2 authored about 9 years ago

generator excretes NoneTypes...

5b3539004e356c2e5f7c3d23dfabe73579ee7e32 authored about 9 years ago

r2's yaml hack

1503ef7fa90b3b0a2765058e3056d99551e36754 authored about 9 years ago

revert 2de23459c

63e4fcb67766b4e664c76a84993d949fd70cb3b0 authored about 9 years ago

Normalise the sets in the captive portal test

3cfdeb86d0eb5e914c446a9545b79d4ad9bd46bb authored about 9 years ago

catch exceptions while scanning yaml

2de23459c5a1e2ec1a8eac5a401917f137e5e977 authored about 9 years ago

Fix normalisation of dns_consistency tests

7aaa07d0c630d7b2b52effbebf27de7966cb8436 authored about 9 years ago

Remove start_time key

beb64d6db5fb0d4b46b4bd22d13e266b5022a520 authored about 9 years ago

Set exit_ip and exit_name to null by default

236d218f17366bf06923cdbfe0c79dc3859ac31e authored about 9 years ago

Add input_hashes to the base schema

d6d7fbb760e83fb0826be2af8871955b0079eaf3 authored about 9 years ago

Add probe_city and backend_version to the base schema

1d5f0f05107290af24f2c76ef195e6c4bcfa9540 authored about 9 years ago

Perform better normalisation of MX and SOA records

* Do not fail when the tampering key is not present

deb44e709e0cc427c4d974c80b00ba5a2f074aea authored about 9 years ago

re-arrange tor_{exit,name} finding, exception stuff

68ef01d1cecb5124f8e20d7e75edbea2f28079dd authored about 9 years ago

http test options

c036d03ef10dffc15826628c21b9b5bd5b341d35 authored about 9 years ago

daily_workflow tweaks

27ac98c959a6f7d08abaa4ddcfc93a356bb87e2c authored about 9 years ago

multi_protocol_traceroute normalisation tweaks

aa44b6012dcdb42116af345855496a18fde55af0 authored about 9 years ago

raise exception when encountering unknown extension

151d6067f65f12335e6d65ac70d3ed4a5176b3a2 authored about 9 years ago

dns_consistency normalisation tweaks

2f1766922d60a7dd993e4316fea7fdeab71f31fb authored about 9 years ago

fix regex in dns_consistency normalization

445d51f3fee326f2ff24cbf6a0b97fea9985f9bd authored about 9 years ago

Do normalisation of the Headers as well

fbf247209a9d6200ae934413c225e0fad789e944 authored about 9 years ago

Don't swallow exceptions but re-raise them

087b1d2d25b9886f4959c0defe6aa9d82328f858 authored about 9 years ago

Fix bug in setting is_tor variable

b21c43535158b9bc1fda04b62b3be24531488086 authored about 9 years ago

Headers_diff can also be none

4a12fcb304c7aeb73eef06287eeb7fbafd0ea30a authored about 9 years ago

Convert set to list and fix tor entry

77d7b17aad7c804d819024fa8f19b32435994be5 authored about 9 years ago

Remember to write newlines

7af3f0ad92f50aaa070488d070473292896f354a authored about 9 years ago

Only cast to unicode when needed

6af36f4bf0973ec07cba039213a8b798bb2f56fc authored about 9 years ago

Integrate @TylerJFisher DNS normalisation code into the daily workflow

72aa0e00f8ac5bfb9f92cb7db7f94c978b35d011 authored about 9 years ago

Improvements to body sanitisation function

* Fix bug in bridge_reachability sanitisation

79e67dcf650a1f6e6ce8033a623cde0696a89486 authored about 9 years ago

Add example client.cfg file

* Progress in implementing luigi based daily workflow

63aea9c31261ee6e8a3515cf6241f612e69e46b9 authored about 9 years ago

Delete all dead code

78052bd210f378be745951ba2151779a2317bfaa authored about 9 years ago

Implement the daily workflow all in one luigi workflow

32037c8b76f87d29d16bff7b33e812bf0eb9ed9f authored about 9 years ago

Don't base64 encode the url

f5bfa10151bdfd6142cc7937c0fa29a3d52061a1 authored about 9 years ago

Also print out the failed entry

6758b1581c7d123bf9f26b7514896cd9aff8cb33 authored about 9 years ago

Skip over invalid reports

d90f263846216f7a38a5ba2e853d127de9d223fe authored about 9 years ago

Skip over entries that are not dictionaries

07e51da3a86375110ec7e65a9566c4eb6050da03 authored about 9 years ago

Implement performance boost by using CLoader

40c484bb7ed445d0151df9d108602b82ddaa2fec authored about 9 years ago

Fix bug in sanitise of scapy tests

c09abb8c56540b4d6afb8c1c3274c0a17e7fed90 authored about 9 years ago

Use the pgcrypto function since Postgres 9.4 is a requirement anyways (JSONB)

1cd35bc226bac318482702060466d2e5201f91b2 authored about 9 years ago

Move the unicode decode catch down

9f484117a17dbd15ee0f1e24e81060ef1e2f5e68 authored about 9 years ago

Implement base64 encoding of body

* Add bucket date to DB

1f2bbdb9638164ab9305045ee5467236b348177c authored about 9 years ago

Only strip null characters from response bodies.

* Properly remove the body of the response when failed to insert

b04bc6815ba299221267627c437631cfca6b90da authored about 9 years ago

Also create indexes for test_start_time and refresh the new materialised views

af70f30d7b35f4a7c50a969f245e29b4256f2473 authored about 9 years ago

Move the creation of indexes out of the Streams class

a5e0ef2fa7d912f794de0095092f2a90b8b94506 authored about 9 years ago

Implement hack to avoid decoding/encoding errors in URLs

6e5997c128e2bb50d35cec47e50fc9bc4b62f6a1 authored about 9 years ago

Fix typo

abeac9e37574cbca05eeb5ca5944d607c1f868b3 authored about 9 years ago

Add an empty response when it's missing

a462dcffb8ee5207a83617c5c9919ded898d6995 authored about 9 years ago

Fixing of body and headers

be80ba44fc13dea16cb1d3d73df5e88c737b131a authored about 9 years ago

Fix tor detection logic

59c00efba943ac4e015e3587bd101035f4fd0b09 authored about 9 years ago

Fix request key nesting

* Specify default values for bins_to_sanitised_streams task

da13382a7bf23f32c2fc3241e71f6d5ce01c9df3 authored about 9 years ago

Fix logic of detection if the request is done over Tor or not

1fa21a548c4baa035bc2c53dd75c91b4e0b19086 authored about 9 years ago

Headers should be inside of a dictionary instead of a list

78110213f8ebc795fdb3f6bd162a9ebff6cf905a authored about 9 years ago

Un-ravel the headers to not be a list within a list

96f2d106d62ed436539b4a2e8785c0aebd27afb1 authored about 9 years ago

Fix some bugs in the stream to DB logic

* Fix bugs in the sanitisation procedure

e866cb356a1b8abd320a2f85082dc3a17087d9cd authored about 9 years ago

Fix setting of the default value

c1d0ac22bf91e55748b25477ad323f859a3f834d authored about 9 years ago

Set the ID column to DEFAULT

3f6f7631cc6161ae9ca8b392e96ecd024f00dd2c authored about 9 years ago

Fix typas

f8f4b3944d1423b9f3403dc2893e207c704f859a authored about 9 years ago

Fix multiple primary key issue

d3472cfe6f395fcb50ca7a20d7677aa6f74ccedf authored about 9 years ago

Add support for creating indexes and refreshing the materialised views

f96b6a8a448ba29fff265a67ee6957bf18c79fbd authored about 9 years ago

Merge branch 'joe/redo' of github.com:thetorproject/ooni-pipeline into joe/redo

* 'joe/redo' of github.com:thetorproject/ooni-pipeline:
clean ooni-public/json

99a6267f275299005fe51e28d5cb2c16c6ac588e authored about 9 years ago

Refactoring of the streams to db function

f9a73c779e50fe5bc9e206e09e316b7ce5a0fa41 authored about 9 years ago

clean ooni-public/json

2891cd09cfc8e4700e0c27a447e9ca9c2d7e4e97 authored about 9 years ago

Do some re-ordering of the http_requests tests so that the experiment is always the first in the list

ba13e21b0298e3572425c0b8e804f28a53b5003d authored about 9 years ago

Add measurement id primary key

03781b3326a3718e8b9ed9be118b521b0a01993b authored about 9 years ago

Fix bug in poping of response body

* Replace tabs with whitespace

ede73218c9ae94278c50b5e082176acca9367655 authored about 9 years ago

learn to python

7f99dcd83567e798e8e0d587a599dde99c87c11b authored about 9 years ago

don't stick body in db if it has null bytes

daa5f9d8f93ed8eac6d1db41c35549af283928b3 authored about 9 years ago

add some more columns

6ff6a55e14283a2301a4ac22ae5a8ee456231202 authored about 9 years ago

catch an early StopIteration exception

d5e2e48a027155c5fee5234fe717851677b65108 authored about 9 years ago

make content-length more accessible

4261ab1aa7e0e671cca4f2649b8c3c734308d0f0 authored about 9 years ago

string encoding handling

749a0471dd49163313ea0c1d87f04d7b3dcee60f authored about 9 years ago

better programming

09c698a83c65b30552cf3f9514057edca460a6af authored about 9 years ago

tweak streams_to_db

4ba84002427ed84de14d9f9ebb62073325ff2ab8 authored about 9 years ago