github.com/ooni/backend commits | Ecosyste.ms: OpenCollective

Fix paths in unittests and adjust error message checks

7785abdbccb30670ec3a0b9459a6e87f2db9dded authored over 7 years ago

Implement spiffy new docs UI

b4eb493d73af9dda73fa17725e75700cbfca7a4e authored over 7 years ago

Refactor measurements API to use OpenAPI 2.0 (Swagger) to generate the

public API endpoints

This achieves the following goals:

* It's possible to programmatically gen...

bcba8a645dfbca5f0221f8ca29637e7fcc7471db authored over 7 years ago

Use yarn instead of npm for more reproducible building

5ea291c6b97c3af63de22b752b41a747fcfb8c12 authored over 7 years ago

Merge pull request #27 from TheTorProject/fix/pipeline-integration-darkk-bugs

Fix/pipeline integration darkk bugs

4a294fe6a6e6829fa52200f101f8ba900e369724 authored over 7 years ago

Add connexion, drop unneded deps

d8ad2df86259d5316bdb41bd9f6de7f703031684 authored over 7 years ago

Perform join on input table only when input is defined

3fa44f6ab0647aa0f4e17b06bebca3e1c8b58070 authored over 7 years ago

shovel:0.0.9 - fix missing encoding header for check_sanitised.py

bc2936db3a7f61da95cab1190bd04ada60fe5185 authored over 7 years ago

centrifugation: pass HTTP headers through pg_strquote to strip zero bytes

ab977d1956bddfc826c28ae0c0971a9ffed018d3 authored over 7 years ago

check_sanitised: fix sanitised_check pipeline task broken since 45d12b1a, thanks pylint

fd66ecec9e80cde43fd5f280b7aa16ae1c391d0b authored over 7 years ago

shovel:0.0.8 - bad data rows are not fatal anymore during COPY, HTTP headers follow unicode "taste" of postgres

0602fe11640947b3892997c78139792cd2a38da3 authored over 7 years ago

centrifugation: make bad rows during COPY non-fatal

abac7a37b8618b26c82228c8915f548ccea2e02d authored over 7 years ago

centrifugation: use ' instead of " where possible

c53699f14443f22ea12154013a42b7a4e02f6db4 authored over 7 years ago

Implement download_url based on new ooni-pipeline

852f04331b9d052f82abc440d14490122aa36697 authored over 7 years ago

Add create-tables Makefile target

Run create-tables on travis to populate database

425c0c616d322849666c6d987fa993bc89e89e13 authored over 7 years ago

Do not log DATABASE_URL

suggest by @darkk in #24

817b30713eae6d99266c9fec5e41889eb3d51623 authored over 7 years ago

Remove all s3 related dead code

e4b9ae8cedeecd774c50d83a6a12123df6b1a5c8 authored over 7 years ago

Set app["DEBUG"] when APP_ENV == "development"

suggested by @darkk in #24

d2ccb1d85b8a00362c2cc19bf2a5de8ff156617a authored over 7 years ago

Do not create tables, do not init database

d1f7c996c14283ab3552089e93d5cf4679d95bc8 authored over 7 years ago

Remove dead variables

suggested by @darkk in #24

b200006b0b6cfad044e91e3435c6aab7a6129a81 authored over 7 years ago

Merge pull request #22 from TheTorProject/feature/pipeline-integration

Feature/pipeline integration

bef3ec6b3196bc19fd3c7b8f0c4cdd8ba6ac8e6d authored over 7 years ago

Remove old docker-compose containers

fe3a7ba231616cf0c5b9b3195819bcf9afb09af5 authored over 7 years ago

Add changelog

d7a320bf8b1ac21fe001b379aa6e73654b9c23de authored over 7 years ago

Add version API.

Bump version to beta.3

0eed3db2d83e0cadffd09b158f4ae88ce70f5aa9 authored over 7 years ago

Fix makefile version parsing

0039f89e67a0200f2a1ad135f66e4926a4641d68 authored over 7 years ago

Bump to version beta.2

* Add parsing of version in Makefile

285a87581dc5c212fd1df9dbb7a95c886f368bdb authored over 7 years ago

Fix bug not allowing you to use since_index option

* Bump REPORT_INDEX_OFFSET
* Fix typo spotted by @darkk in config `AUTOCLAVED_BASE_URL`

6563c74b6ba16af7ab53cdb1215c36f04a847bca authored over 7 years ago

Add indexes for TheTorProject/ooni-measurements#22

64aa029151daa74b0aedbc8dd8a807a735b26e0d authored over 7 years ago

Bump version.

Update dockerignore

e9d9e6a7852f7952c188c8ea9594cb9ef14c3487 authored over 7 years ago

Rip out all unneeded docker-compose stuff

5d5815d96ea82aecdb68c31943a33dc6fd185150 authored over 7 years ago

Move REPORT_INDEX_OFFSET into config

92fd908156d3aedf8493f97449f94b76df8efd88 authored over 7 years ago

Add support for building and pushing docker image

5677f00d781181b1285178cd3314ba8c1cf20ae7 authored over 7 years ago

Rip out all celery and redis related functionality

817e1fd468071b8c4ec32f8ffd1ac132d11227a2 authored over 7 years ago

fix typo

fec53deb5f6b14a5dc6ca02875ab3034e07cb87a authored over 7 years ago

shovel:0.0.7 - extract known fingerprints, fix HTTP code, DNS TTL

3706643ce29be6c9d18dfdbe52d17fb57154530f authored over 7 years ago

centrifugation: add `msm_no` indexes to support partial updates, fix NOT NULLs

8088298607ad5150171615958f8b5c56733bbf64 authored over 7 years ago

Merge branch 'master' into pipeline-16.10

Merging to get new blockpage signatures into the branch.

72d2e2aef14a258f12aff8222f31003e0773a555 authored over 7 years ago

Merge pull request #58 from TheTorProject/fingerprint/gf

Add French Guiana block page fingerprint

8c1a372126a350677f98ae1e3ad09e6ea2f22146 authored over 7 years ago

Merge branch 'master' into fingerprint/gf

74739f0dd3207ddd6684f06637b7fe3b1b6fee0c authored over 7 years ago

Merge pull request #67 from TheTorProject/fingerprint/ro

Add blockpage body fingerprint for Romania

9df2b1001e12c181a2a89bebd80c8f3c45c4807c authored over 7 years ago

Merge branch 'master' into fingerprint/ro

0950d24b5f4d9e47b1d4bf05b721ea46759adf58 authored over 7 years ago

Merge pull request #68 from TheTorProject/fingerprint/my

Add blockpage body fingerprint for Malaysia

cc886aef29ec028898c3f4e16a25d34ddca22e33 authored over 7 years ago

Merge pull request #69 from TheTorProject/fingerprint/ae

Add blockpage header fingerprint for United Arab Emirates

43db476b23afee6cce7352fa19bc4afec851834d authored over 7 years ago

Split out measurements/api

8a3bc0477951417016861af05717a4d18c35b10c authored over 7 years ago

Split out blueprints/api and blueprints/pages

e4ffb926a61e763b33838cf009c62443705a4865 authored over 7 years ago

shovel:0.0.6 - fix centrifugation postgres COPY wbuf

4a8471e45d132b3c7fa170198314f86dc69362d1 authored over 7 years ago

shovel:0.0.5 - refactor centrifugation to extract more features and be more robust against bad data

45d12b1a524874658f7a0a41cb135eb2c6a72571 authored over 7 years ago

centrifugation: add unique constraint to `input` table & add comments

85ffd20091102a74fc5b3a886c8ecfd841c956e8 authored over 7 years ago

Implement single measurement retrieval

01173c924bc710261f4a0d535665d1319a1b4870 authored over 7 years ago

Do some improvements to the measurement listing

cf97cf0864134f80636f8c86ac7e67c8cbebb03e authored over 7 years ago

Implement measurement listing API endpoint

TODO: Implement fetching of individual measurements

68219418c48064cef117da03f6598c146e355a34 authored over 7 years ago

Rip out all the unneeded periodic tasks

75a83dac6717688fc077b7c80f7f83ad83f86ccc authored over 7 years ago

Feature parity with current measurement API

fd3f6570127f99f79449b03b357547f855983f89 authored over 7 years ago

Swap out measurements database with the pipeline generated database

TODO: missing how to determine the index

bca88a848068f002b4ec1a357826e762a910c776 authored over 7 years ago

Add blockpage header fingerprint for United Arab Emirates

9a72b16c5c3f9049ca56a99c8b3c3ce2a1d75f2f authored over 7 years ago

Add blockpage body fingerprint for Malaysia

48a10f7b9a383ae203ee30c0b9bb7ff0d7e5854e authored over 7 years ago

Add blockpage body fingerprint for Romania

fdc056adaf231fb94e1b4cdcc1f5679841499aca authored over 7 years ago

Add French Guiana block page fingerprint

0ed504f71f283a5b18a29f7120ba4f546244877d authored almost 8 years ago

shovel:0.0.4 - log okayish diff between sanitised and autoclaved for future reference

dcd5b9e37031fddd6d85a134fd3ac78b32c92041 authored almost 8 years ago

shovel:0.0.3 - checked cleanup of sanitised and reports-raw

881dd7c9c5177ffb4a7178e84fab54e748b6fe04 authored almost 8 years ago

Mark openobservatory/shovel:0.0.2

fd8cc29ebc5644b1c04b82ceda9f1bb4ee970f27 authored almost 8 years ago

airflow-dags and Dockerfile for airflow, airflow-worker migrated to TheTorProject/ooni-sysadmin#105

4728cb2c84957a3dcc2ebdceadeb753223b88af0 authored almost 8 years ago

Merge pull request #89 from TheTorProject/add-note-to-script

Add note about limitations of the gen-ssl-key-cert

03d738b8daa86ed103e435a23adc16e9bac64127 authored almost 8 years ago

Merge pull request #19 from anadahz/restart-docker

Set restart policy to always restart docker containers

1e9b79f04b98a98c5564a0743afde5e8d97237bd authored almost 8 years ago

centrifugation: batching does NOT reduce number of Queue.get lines in pyflame

ae3d4557aa98955dc6af907da433f3c9d95728e4 authored almost 8 years ago

pipeline-16.10 doc: some highlights

82ad892e00b18d42f9d4e79ccba69276b4d59496 authored almost 8 years ago

Pin depesz/Versioning that is used to bootstrap schema versioning

2a71fdaccb28b9730c5fbdf31c46af6767e5c4db authored almost 8 years ago

Merge pull request #21 from TheTorProject/feature/cors

Add CORS support

b5921059b635df0984d2c242348c1562d880a698 authored almost 8 years ago

originas2pg: tool to load snapshot of originas.bz2 to postgres

The table does not store whole BGP stream of migrations of IP subnets
between ASes across time, ...

a702f728d45a03aeb374badf0522dc1d077fc98b authored almost 8 years ago

Add CORS support

Implements: https://github.com/TheTorProject/ooni-measurements/issues/20

750816d62cee6cebcd87982b5c862689e1c40295 authored almost 8 years ago

centrifugation: extract more metadata from autoclaved reports

- TCP failures
- A records for DNS
- verdicts for HTTP requests

9e37fa81f632e57baea746913cabc3f779e6e731 authored almost 8 years ago

canning: comment on ARG_MAX

8176dc3df71819ac115a9662169df78857a6780a authored almost 8 years ago

centrifugation: load indexing metadata to postgres

fe345a6383f80f175084bdb221dff2cd013290c0 authored almost 8 years ago

pipeline-16.10 doc: some gotchas

2a3da7c0ed4f3f09cc990bfb176f6a8c93626fca authored almost 8 years ago

centrifugation: converting http bodies to simhash (text dump for a while)

80e4602aaacb3406b9f46465521b681c792d8c41 authored almost 8 years ago

Set restart policy to always restart docker containers

5ffea798881fc081adb5913f63bd18232fa5e6f8 authored almost 8 years ago

Merge pull request #18 from TheTorProject/fix/duplicates

Fix crontab scheduling to execute every 6 hours

e95de6e21b396b5aff0c06a968072a494c165445 authored almost 8 years ago

Run the updatefiles task every 30 minutes

ace848806958317f28c770928ebc599080eec6db authored almost 8 years ago

Pin the versions of redis, rabbitmq. Update to postgres 9.6

343baf561c85bc34ab86cf81ca8c2f249f7749aa authored almost 8 years ago

Fix crontab scheduling to execute every 6 hours

9794bb29e5c9a6e447fc3b1d6c6bc872207aa881 authored almost 8 years ago

autoclaving: fix bridge_db loading to autoclave `tcp_connect` & `bridge_reachability`

a484ee834414228130bba2a72d31b5116c12fcd5 authored almost 8 years ago

autoclaving: slicing files into independent reports

4ddc40b6ab5eafc5759eeaf4f23dcf05bdbf6e65 authored almost 8 years ago

Support Reykjavík(sic!), great source of privacy geeks and TZ=UTC datetimes

8490691e8eb3e3fdda5fa17e48b0dcd6480da741 authored almost 8 years ago

shovel: strip python eggs

ce95c4f6198fb5252ff52f1db30cb609085eceb7 authored almost 8 years ago

Canning, pt.2

3222d571095efd0472119d896dac639215eb7919 authored almost 8 years ago

Script to cleanup private/reports-raw (checking against S3 backup) after canning

9b488b513242732dbfe1c5fe2a36973e613f90a4 authored almost 8 years ago

Some mu-benchmarks to justify questionable lines of code

cc10f7ef6de3f92ccc7280961963ee440af207c1 authored almost 8 years ago

Merge pull request #16 from TheTorProject/hotfix/race-condition

Reduce the chance of race conditions by committing on every iteration

99ce058f20d4d688636d578b85b6bf5258324b89 authored almost 8 years ago

Reduce the chance of race conditions by committing on every iteration

31df0f51baad71fb46e043782e8ac568791e89da authored almost 8 years ago

Merge pull request #15 from TheTorProject/fix/timetravel

Ignore measurements from time travelers

e52e178ceea7c8913288db49bcd11d24215a3067 authored almost 8 years ago

Ignore measurements from time travelers

c8bc45c3d7769a58a2177a4ee5af13d0f623e0ba authored almost 8 years ago

Merge pull request #14 from TheTorProject/fix/cc-keyerror

Handle exception when country code is invalid

86de8339b68d3f038e72e1416db723635ee0faf8 authored almost 8 years ago

Handle exception when country code is invalid

3a8a2e8cb6e39eb1329fb20dd41f35a08224688d authored almost 8 years ago

Merge pull request #52 from TheTorProject/rebuild-notes

Add pipeline rebuild notes, updated client.cfg, add requirement

c6faeeaa8fede9fb2c075d40fe5002998797f348 authored about 8 years ago

Add pipeline rebuild notes, updated client.cfg, add requirement

bd9adf3b3a3828bc55c018f5e47be549635e964f authored about 8 years ago

Merge pull request #43 from TheTorProject/update-concurrently

Refresh materialized views concurrently

0a62bd4f2a17dc470a3974ec8b500877f63f684d authored about 8 years ago

Bump version, update changelog

5395ce8275c87526b443cfc63c4db37e4b1f3ce8 authored about 8 years ago

Fix/redirect (#104)

* Update txextra to fix issue in redirect handling

* Ignore private redirects

* Pass the e...

6f92669ab75b286aaad25b6cf02c9d3637764a65 authored about 8 years ago

Merge pull request #51 from TheTorProject/fix/id-generation

Always set the id in the pipeline

51fdd878a7b3ad18dc54e43a5fe807820ae6899f authored about 8 years ago

Always set the id in the pipeline

6e3bd1df1275f84f2d5048d0cba9890f50ad9610 authored about 8 years ago