github.com/ooni/pipeline commits | Ecosyste.ms: OpenCollective

shovel:0.0.12 - skip LZ4 frames while skipping duplicates

4b6762871cc2e4aa2126cf9987b31ecd10b448e5 authored over 7 years ago

shovel:0.0.11 - skip cross-bucket duplicates

830ab169dd20f8b0725893b069bad96c21cc2477 authored over 7 years ago

shovel:0.0.10 - capture one more postgres exception during COPY

6e8462fdeb1543c958b5e245f4d99b3d61a25e1a authored over 7 years ago

shovel:0.0.9 - fix missing encoding header for check_sanitised.py

bc2936db3a7f61da95cab1190bd04ada60fe5185 authored over 7 years ago

shovel:0.0.8 - bad data rows are not fatal anymore during COPY, HTTP headers follow unicode "taste" of postgres

0602fe11640947b3892997c78139792cd2a38da3 authored over 7 years ago

centrifugation: pass HTTP headers through pg_strquote to strip zero bytes

ab977d1956bddfc826c28ae0c0971a9ffed018d3 authored over 7 years ago

check_sanitised: fix sanitised_check pipeline task broken since 45d12b1a, thanks pylint

fd66ecec9e80cde43fd5f280b7aa16ae1c391d0b authored over 7 years ago

centrifugation: make bad rows during COPY non-fatal

abac7a37b8618b26c82228c8915f548ccea2e02d authored over 7 years ago

centrifugation: use ' instead of " where possible

c53699f14443f22ea12154013a42b7a4e02f6db4 authored over 7 years ago

Add indexes for TheTorProject/ooni-measurements#22

64aa029151daa74b0aedbc8dd8a807a735b26e0d authored over 7 years ago

fix typo

fec53deb5f6b14a5dc6ca02875ab3034e07cb87a authored over 7 years ago

shovel:0.0.7 - extract known fingerprints, fix HTTP code, DNS TTL

3706643ce29be6c9d18dfdbe52d17fb57154530f authored over 7 years ago

centrifugation: add `msm_no` indexes to support partial updates, fix NOT NULLs

8088298607ad5150171615958f8b5c56733bbf64 authored over 7 years ago

Merge branch 'master' into pipeline-16.10

Merging to get new blockpage signatures into the branch.

72d2e2aef14a258f12aff8222f31003e0773a555 authored over 7 years ago

Merge pull request #58 from TheTorProject/fingerprint/gf

Add French Guiana block page fingerprint

8c1a372126a350677f98ae1e3ad09e6ea2f22146 authored over 7 years ago

Merge branch 'master' into fingerprint/gf

74739f0dd3207ddd6684f06637b7fe3b1b6fee0c authored over 7 years ago

Merge pull request #67 from TheTorProject/fingerprint/ro

Add blockpage body fingerprint for Romania

9df2b1001e12c181a2a89bebd80c8f3c45c4807c authored over 7 years ago

Merge branch 'master' into fingerprint/ro

0950d24b5f4d9e47b1d4bf05b721ea46759adf58 authored over 7 years ago

Merge pull request #68 from TheTorProject/fingerprint/my

Add blockpage body fingerprint for Malaysia

cc886aef29ec028898c3f4e16a25d34ddca22e33 authored over 7 years ago

Merge pull request #69 from TheTorProject/fingerprint/ae

Add blockpage header fingerprint for United Arab Emirates

43db476b23afee6cce7352fa19bc4afec851834d authored over 7 years ago

shovel:0.0.6 - fix centrifugation postgres COPY wbuf

4a8471e45d132b3c7fa170198314f86dc69362d1 authored over 7 years ago

shovel:0.0.5 - refactor centrifugation to extract more features and be more robust against bad data

45d12b1a524874658f7a0a41cb135eb2c6a72571 authored over 7 years ago

centrifugation: add unique constraint to `input` table & add comments

85ffd20091102a74fc5b3a886c8ecfd841c956e8 authored over 7 years ago

Add blockpage header fingerprint for United Arab Emirates

9a72b16c5c3f9049ca56a99c8b3c3ce2a1d75f2f authored over 7 years ago

Add blockpage body fingerprint for Malaysia

48a10f7b9a383ae203ee30c0b9bb7ff0d7e5854e authored over 7 years ago

Add blockpage body fingerprint for Romania

fdc056adaf231fb94e1b4cdcc1f5679841499aca authored over 7 years ago

Add French Guiana block page fingerprint

0ed504f71f283a5b18a29f7120ba4f546244877d authored almost 8 years ago

shovel:0.0.4 - log okayish diff between sanitised and autoclaved for future reference

dcd5b9e37031fddd6d85a134fd3ac78b32c92041 authored almost 8 years ago

shovel:0.0.3 - checked cleanup of sanitised and reports-raw

881dd7c9c5177ffb4a7178e84fab54e748b6fe04 authored almost 8 years ago

Mark openobservatory/shovel:0.0.2

fd8cc29ebc5644b1c04b82ceda9f1bb4ee970f27 authored almost 8 years ago

airflow-dags and Dockerfile for airflow, airflow-worker migrated to TheTorProject/ooni-sysadmin#105

4728cb2c84957a3dcc2ebdceadeb753223b88af0 authored almost 8 years ago

centrifugation: batching does NOT reduce number of Queue.get lines in pyflame

ae3d4557aa98955dc6af907da433f3c9d95728e4 authored almost 8 years ago

pipeline-16.10 doc: some highlights

82ad892e00b18d42f9d4e79ccba69276b4d59496 authored almost 8 years ago

Pin depesz/Versioning that is used to bootstrap schema versioning

2a71fdaccb28b9730c5fbdf31c46af6767e5c4db authored almost 8 years ago

originas2pg: tool to load snapshot of originas.bz2 to postgres

The table does not store whole BGP stream of migrations of IP subnets
between ASes across time, ...

a702f728d45a03aeb374badf0522dc1d077fc98b authored almost 8 years ago

centrifugation: extract more metadata from autoclaved reports

- TCP failures
- A records for DNS
- verdicts for HTTP requests

9e37fa81f632e57baea746913cabc3f779e6e731 authored almost 8 years ago

canning: comment on ARG_MAX

8176dc3df71819ac115a9662169df78857a6780a authored almost 8 years ago

centrifugation: load indexing metadata to postgres

fe345a6383f80f175084bdb221dff2cd013290c0 authored almost 8 years ago

pipeline-16.10 doc: some gotchas

2a3da7c0ed4f3f09cc990bfb176f6a8c93626fca authored almost 8 years ago

centrifugation: converting http bodies to simhash (text dump for a while)

80e4602aaacb3406b9f46465521b681c792d8c41 authored almost 8 years ago

autoclaving: fix bridge_db loading to autoclave `tcp_connect` & `bridge_reachability`

a484ee834414228130bba2a72d31b5116c12fcd5 authored almost 8 years ago

autoclaving: slicing files into independent reports

4ddc40b6ab5eafc5759eeaf4f23dcf05bdbf6e65 authored almost 8 years ago

Support Reykjavík(sic!), great source of privacy geeks and TZ=UTC datetimes

8490691e8eb3e3fdda5fa17e48b0dcd6480da741 authored almost 8 years ago

Script to cleanup private/reports-raw (checking against S3 backup) after canning

9b488b513242732dbfe1c5fe2a36973e613f90a4 authored almost 8 years ago

shovel: strip python eggs

ce95c4f6198fb5252ff52f1db30cb609085eceb7 authored almost 8 years ago

Canning, pt.2

3222d571095efd0472119d896dac639215eb7919 authored almost 8 years ago

Some mu-benchmarks to justify questionable lines of code

cc10f7ef6de3f92ccc7280961963ee440af207c1 authored almost 8 years ago

Merge pull request #52 from TheTorProject/rebuild-notes

Add pipeline rebuild notes, updated client.cfg, add requirement

c6faeeaa8fede9fb2c075d40fe5002998797f348 authored almost 8 years ago

Add pipeline rebuild notes, updated client.cfg, add requirement

bd9adf3b3a3828bc55c018f5e47be549635e964f authored almost 8 years ago

Merge pull request #43 from TheTorProject/update-concurrently

Refresh materialized views concurrently

0a62bd4f2a17dc470a3974ec8b500877f63f684d authored almost 8 years ago

Merge pull request #51 from TheTorProject/fix/id-generation

Always set the id in the pipeline

51fdd878a7b3ad18dc54e43a5fe807820ae6899f authored almost 8 years ago

Always set the id in the pipeline

6e3bd1df1275f84f2d5048d0cba9890f50ad9610 authored almost 8 years ago

airflow: step #1 (raw -> canned)

Compression ratio ~21% & some hard-coded stuff.

064e9a053977f7ef2aafb4cba11f302de386692b authored almost 8 years ago

Merge pull request #44 from VEinteligente/temp-pull-req

Add psycopg2 to requirements.txt #1

e57c5c2d137ef00b6772e167d49a5f93dbc7fd26 authored about 8 years ago

Update pipeline-16.10 doc

ae941acf37da075ed587d5741fd5664dfdf1de43 authored about 8 years ago

Add psycopg2 to requirements.txt #1

cc5b8116dab4bfb6c48d506c77590290a4e5a49f authored over 8 years ago

Refresh materialized views concurrently

Prior to this materialized view updates lead to the database tables
related to these views will ...

32ae1b346b6bc485418a9b1857bffdb427df2e82 authored over 8 years ago

Merge pull request #42 from TheTorProject/optimize/indexes

Create a multicolumn index on test_start_time and probe_cc

5ec56d9d6960c906833842100d32be6667e12750 authored over 8 years ago

Merge pull request #39 from TheTorProject/fingerprint/ru

Add Russia blockpage header fingerprint

76135154f3979b58949f5201bb21fc509426541f authored over 8 years ago

Create a multicolumn index on test_start_time and probe_cc

This leads to a significant performance boost to queries on countries
with few measurements (que...

f926cc873823639037ec4d3be8fc8f6a490f44c3 authored over 8 years ago

Merge pull request #41 from TheTorProject/feature/pipeline-16.10-ideas

Add some comments on next generation of ooni-pipeline-ng

391c62fe7b933e75b9af0b50fe2897fb1899f324 authored over 8 years ago

Add some comments on next generation of ooni-pipeline-ng