github.com/ooni/backend commits | Ecosyste.ms: OpenCollective

Start writing integration testing for the pipeline

95dc49e9aaaec3673b3a27ca3cc86df552a52101 authored almost 6 years ago

Vendor Versioning from https://gitlab.com/depesz/Versioning/2dbcdf94e93a68a6f273375408a92d42c69dd5a5

61cb4e323270e0cf246fea581971ef929d51a1c7 authored almost 6 years ago

Drop Versioning from the submodules

4d36770e2c3156e0d5be0ffed5d05b603997b692 authored almost 6 years ago

Add country_overview API endpoint

8d42a7f168d32261ebe9c93979c4adb7d86f4072 authored almost 6 years ago

Mock the middlebox_detected field

e5da7f3eaaab366f902155f580b1d2cf642e4077 authored almost 6 years ago

Add support for network_stats

6118c4dce4721427e6057936a00a67969333a219 authored almost 6 years ago

Add update_explorer_metrics PG task

bcf9500478861c494fef00472935289ed426b071 authored almost 6 years ago

Reshape the stats in the test_coverage as well

be649a6ddbb2f204bfe215764e5c1fbd65cf7e4d authored almost 6 years ago

Prevent there being "holes" in the data for the charts

Rename date to test_day

f67ab0ac81de16f90eaab03148295aec32a66c8e authored almost 6 years ago

Implement im_stats and im_networks endpoints

465fed4dc97ff722fda0c70a3ef32a18e43214f8 authored almost 6 years ago

Fix wrapping of docs

2cf05d2281bf9f0be0d319a4d93eca24188c926a authored almost 6 years ago

Add docstrings to some steps of the centrifugation workflow

* load_global_duplicate_reports()
* load_autoclaved_index()
* load_autoclaved_db_todo()

842cd09622d4c749efc37a7aab7a43f9e6695b7b authored almost 6 years ago

Store input_no instead of input

68ea84d84b4a9549107be1d61d8448e8007e61ae authored almost 6 years ago

Add rollback script and implement mat view for websites table

3ce4073b85152dc5e0280e3c76d483b3020991fe authored almost 6 years ago

Replace msmt with msm. Leave the plural as msmts

* Add comments to tables

ed351f8e4d3a20fe495bddf1eadfeaf9c7264e14 authored almost 6 years ago

Fix bug in limit field parsing

48c4070b5304f793fdda1d59fedfe43719d2bea0 authored almost 6 years ago

Add support for showing the total number of tested URLs

8b4d32e4f1b62d01de985584d2427221e1920f4f authored almost 6 years ago

Refactor recent_msm table based on input from @darkk

- GROUP BY bucket_date and truncated measurement_start_time in recent_measurement table
- Replac...

ac2bd3f870180a25d12c83aeeffbf0929386596f authored almost 6 years ago

Fix default websites list limit

d7902b2be284407e2bd790f965e743ffcee3ed3e authored almost 6 years ago

Implement websites related API endpoints

* /website_networks
* /website_stats
* /website_urls

a66d07183e54f0cf950129679719ead3e15cf678 authored almost 6 years ago

Implement websites related API endpoints

* /website_network_tests
* /website_stats
* /website_test_urls

2ce82aeeac032f136b6df9e92520f36a217eebaa authored almost 6 years ago

Drop pycountry requirement

d2e6442a60af17c369707a20e45f3968f0cfdd33 authored almost 6 years ago

Update country list

c79232b589b81664bebaf93b8f13b320f9e612cc authored almost 6 years ago

Add support for looking up countries using country-util

8752be42124f7be715d4c0d7c6fb0fbd48d72ef5 authored almost 6 years ago

Add initial schema for some of the tables needed by OONI Explorer

652282ad47071c9f7f1aec2795de86b594173a8b authored almost 6 years ago

Merge pull request #117 from TheTorProject/release/1.3.5

Bump version and add changelog entry

97e4a129510cb0913a2926a330788d898e8d9af5 authored almost 6 years ago

ooni-uuid.md: fix obvious typo

4a3a014256e4c2b46fc693d732da4e16aba3f0dd authored almost 6 years ago

Implement the test_coverage views

6eedf2efb168d283d57485fc37da4debcc0fa739 authored almost 6 years ago

Merge pull request #143 from ooni/drop-legacy

Delete all legacy pipeline code

0eab0a52c5dfce17a1fd4203c5bdf1749e4470ef authored almost 6 years ago

Readme.md: fix obvious typo

b6f43c0eaa98f6774535e7cfe06cdffcd7560eef authored almost 6 years ago

Revert "Drop daily_workflow.py"

This reverts commit 7ac04de98103b8e3eca6bc5c456d4e66b2da3425.

83bafc99f7e3b19d6cb72e79cbf191e5566969e7 authored almost 6 years ago

Drop daily_workflow.py

7ac04de98103b8e3eca6bc5c456d4e66b2da3425 authored almost 6 years ago

Drop private dir

36f4c46be4e93a3f3e71edd88a10fe903f918d15 authored almost 6 years ago

WIP: document report reprocesing, see #133, #134, #135

ee8258e654224d3656c8c5af8dbe57ce624a323f authored almost 6 years ago

Delete all legacy pipeline code

9a62ef13906032551f475af7d0fd70d2c32ab2e0 authored almost 6 years ago

Merge pull request #76 from ooni/fix/75

Fix/75

9d10fdeff30e2797430f7b3b99e93c26b5fdb5b6 authored almost 6 years ago

Merge pull request #74 from ooni/capture-timeouts

Capture timeouts

f7a5a468ed63765345b39d34e2ad54ce2a929fa6 authored almost 6 years ago

Use full path for api stats request

Fixes: https://github.com/ooni/api/issues/75

2adb49b72acb138c0754ca584f8c6dc8cfe6f0f4 authored almost 6 years ago

Update yarn.lock

95bc35986be3ef73bf1f6f7254813dab5fb4ae1b authored almost 6 years ago

Pin d3-selection to overcome https://github.com/d3/d3-selection/issues/185

389affbc0b66516782eda918714a55b8b084e75e authored almost 6 years ago

Update bootstrap-sass

53efaeae6550de08d391dedbe8fbe0105c335400 authored almost 6 years ago

Bump version and add changelog entry

0980b82a63d9728f9b3dc93d39fdd0d89fb155b7 authored almost 6 years ago

Merge pull request #116 from TheTorProject/android-2.0.0-block

Android 2.0.0 block

1c3547246fccabf24cc920704bbb4fc47699eaae authored almost 6 years ago

Also block incoming reports from ooniprobe-android 2.0.0-alpha,beta

7b3c4e670248b4f723eb368fff2607fa880b77e9 authored almost 6 years ago

The smoke test is really flaky

c3536606604bcdabdc2a6445370daa503994891d authored almost 6 years ago

Don't use the tor deb mirror

27bc90c15d2e2071cdc63eb6872f4dc3e653178f authored almost 6 years ago

Add unittest to validate that probe-android measurements are not accepted

7b9f7af7a75f4f082e3a56ea8a5e01bca89a809e authored almost 6 years ago

Add stop-gap solution to not accept new reports from OONI Probe Android 2.0.0

7ba4030e2071382cb5069848b4c9a38dd3058232 authored almost 6 years ago

Merge pull request #142 from gabelula/documentation

A few small changes for pipeline documentation.

1b2688d75a7abc09e446a7d965dd8011f5b5564d authored almost 6 years ago

A few small changes for pipeline documentation.

15f935d0fef4df3e8c234aacaaaa506d686bc937 authored almost 6 years ago

Merge pull request #128

31dd687563d7125460e863f39f054c74521a22b9 authored almost 6 years ago

Add docs/delete-report note & debugging scripts used during data cleanup

6108671bb92fd67164886433a32bc49181f005a5 authored almost 6 years ago

Ignore empty reports on reingestion, closes #141

Empty reports may be result of yaml to json conversion: the yaml report
header was there, but no...

220417f8aa57360fcb94eae2e1d6caf7d48a36b2 authored about 6 years ago

Fix unnecessary Cartesian product during reingestion

f8e15241dffe707cce59a2cc2361f972c018ac59 authored about 6 years ago

Fix ZeroDivisionError

3e9506ca3425a648e09552cbe5291558258c56cd authored about 6 years ago

Fix chmod typo

ee816b71c550fca6519dd1c75bc9b42901bb78b0 authored about 6 years ago

Add `--missing` to autoclaving to handle modified canned files

5ab3e90c28161c81e30a25f9a3b6ded2ce0cbca3 authored about 6 years ago

Move tempfile wrappers to oonipl.tmp

39c9c1970ff6cd26ce33e03fe1e6a81e4cac8d3a authored about 6 years ago

Add `delete_canned_report.py` to re-compressed canned and reports-tgz

69cd5575f54516b5a9a41567436504c7c42ad8f4 authored about 6 years ago

Move two more canning subroutines to `oonipl`

04cb81adc1d6ed0277726cc9dacc9887b671c5d4 authored about 6 years ago

Move `filename` CLI validator to oonipl.cli

1e114bd7101c71e35564db1a3cebc0648db7d507 authored about 6 years ago

Change uniq(input) to uniq(sha256(input)), closes #139

abef3b80e1b91c754781152c59ff691855b454f8 authored about 6 years ago

Capture the QueryTimeout exception so it's clustered in the right group

76e2c29d2a8f31377930e5df2956013dd47a86f9 authored about 6 years ago

Explicitly capture query timeouts

85d10ae3680de6831f3cceefb0085cbe0023f896 authored about 6 years ago

Explicit import paths

1a83cf5c5e5b32c7771c11f458ae3be8d6845d47 authored about 6 years ago

[hotfix] replace exc with e to avoid conflict with sqlalchemy

bb2633eafceac50e912dff4a956481a91ecec9d0 authored about 6 years ago

Update date in changelog

a758e08c904b23900aa2683f7a2ff551ca173822 authored about 6 years ago

Add new version entry

94c2ecd876ef5f92d3fffbe21b489f27e8994aac authored about 6 years ago

Update yarn.lock and package.json

6787aab1b58d712aa35a66169f24e6ab96533575 authored about 6 years ago

Merge pull request #70 from ooni/feature/sentry-context

Better query timeout exception capturing

b5f96d7fa052bdf2b3590813d0e7d5648d08abf0 authored about 6 years ago

Better query timeout exception capturing

Also collect the sentry sql_query context

18ca575614097600c01432cc6b743c050b2132c7 authored about 6 years ago

Merge pull request #69 from ooni/file-download

Make single measurements trigger a download in browser when ?download…

251063e424578a44ff21bcc909011055f74d7936 authored about 6 years ago

Make single measurements trigger a download in browser when ?download=true

This implements: https://github.com/ooni/api/issues/68

be80ea39188a60ed827f7f1a4d82975a8cd0c0ef authored about 6 years ago

Merge pull request #130 from ooni/add-license

Add standard OONI license file. It seems simple and good to me.

8de708cf3512c79cc2ab773e57bc32dfa6488d40 authored about 6 years ago

Edit changelog entry

5dec58447ad280f0f2913e49c699bd069c5e2d83 authored about 6 years ago

Merge pull request #60 from ooni/more-anomalies

API counterpart of Pipeline `more-anomalies` branch

29794221b856a3e1bdb32bad2c82f65635c95bc5 authored about 6 years ago

Release/1.0.5 (#67)

* Add sentry integration

* Bump version

367fa4de29e215ada6aa451248de7f3cf6ef860d authored about 6 years ago

Add sentry integration (#66)

628fb3aca131d1039cee16eeaf9abd4d37138352 authored about 6 years ago

Merge pull request #63 from ooni/query-timeout

Handle query timeout

1d7d210db30ae224246d766ed54500c99f10b4d4 authored about 6 years ago

Merge pull request #65 from ooni/update-deps

Bump versions of requests and pyyaml

b96336830d99fe4e41d57df4da529854c8d7df28 authored about 6 years ago

Rename header in Readme

2cb1b3c2391631fcbe7f8e0ea9500aca7d3d56dd authored about 6 years ago

Bump versions of requests and pyyaml

Fixes:
* https://nvd.nist.gov/vuln/detail/CVE-2017-18342
* https://nvd.nist.gov/vuln/detail/CVE-...

72f5570f6875de27e5cf5ace76ddaf061ac0344f authored about 6 years ago

Handle query timeout

Fixes: https://github.com/ooni/api/issues/55

904abc45d3f96b55fc8fdd96482ec7cbf3fef852 authored about 6 years ago

Merge pull request #64 from ooni/fix-travis

Fix travis build

d95ee6e0d5fac20ffc64560a72e1ef542578fba6 authored about 6 years ago

Fix travis build

* pip freeze dependencies to last known working version
* Add .python-version and remove unneede...

80c938d6d51fbaff9988671f6226e8ff80eea356 authored about 6 years ago

Add standard OONI license file

eafa16743c7eb624f9797e2bd263b8dfeaf1d671 authored about 6 years ago

Add a LICENSE file

2a06ebf7f8187d42e399917105e8a7786352da54 authored about 6 years ago

Table/columns/attributes cleanup

a62726a7b861d153c61c2245d47247f21c96f4d3 authored about 6 years ago

Drop `label` table, it is going away

6ab45a565afaabf2aedcb7531fa2edeab573d332 authored about 6 years ago

Add ShouldCreateEmptyBucket image

ed16eed6cd842251c78f82da57333ce509e83ddd authored about 6 years ago

Enrich empty bucket text

dd0da9c4f92f68e556524f9f2868a4fbcc6bf429 authored about 6 years ago

Merge pull request #129 from ooni/doc-empty-bucket

Docuemnt empty bucket creation

19f9d28e5f0a7bd206b1157a4b72d8561d6f3baa authored about 6 years ago

Docuemnt empty bucket creation

4b2a40e1258e0974c6eeaddc42498485e57bcddb authored about 6 years ago

Merge pull request #127 from ooni/dag-failures

Add docs on investigating DAG failures

e9db58c071dc0515e842bc9bf9488f699e8fb1ea authored about 6 years ago

Add clear task screenshot

bde8114a4939a7ddf9add91173adc85553b2197c authored about 6 years ago

s/DAG/Task/

5b1c3b44b4622d52c5642887c7cd636353870392 authored about 6 years ago

Add docs on investigating DAG failures

51628c5cbb1073fcbefadd5dca9fb3c85668254e authored about 6 years ago

Add estimates of SimhashCache hit-rate and explanation for not-so-great speedup

3ba73981a38f92df7bd69fd4babd5c9aafa3e5d3 authored about 6 years ago

Log SimhashCache hit-rate both in items and in bytes, these value differ a lot

5fb852ef01a5200bea456bdd9213ed220636007c authored about 6 years ago

Preloading simhash cache from previous bucket to speedup ingestion

101d5ee8ea47a238bf1c587a84f31005309bcf86 authored about 6 years ago