github.com/ooni/wcthcheck commits | Ecosyste.ms: OpenCollective

fix(compare): continue improving comparison

12253c0e8f1d2c05a11661c92cf888ba5f1330c4 authored about 3 years ago by Simone Basso <[email protected]>

compare: finish improving code to check status codes

See https://github.com/ooni/probe/issues/1707#issuecomment-944480601

8e990074950c5a594253bb497ea21aa165d1c513 authored about 3 years ago by Simone Basso <[email protected]>

compare: handle 308 => 200

This is quite annoying. It means the old TH does not follow 308.

9ec1325a2a009f6a57ef5600bea349fdeb656497 authored about 3 years ago by Simone Basso <[email protected]>

compare: handle 403=>503 and special-case for cloudflare

See https://github.com/ooni/probe/issues/1707#issuecomment-944480601

0ab86e577e8d6140c7173d347832de3613dcea18 authored about 3 years ago by Simone Basso <[email protected]>

feat(compare): find 403->200 and 403->200+cloudflare

See https://github.com/ooni/probe/issues/1707#issuecomment-944480601

02b0947b8a0ef5c8c40f5e88331ca406e44453c6 authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): account for failures when checking at status

See https://github.com/ooni/probe/issues/1707

63c7882d4f34aa9d8905f8539eebe83dbb566ac4 authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): better code for headers comparison

See https://github.com/ooni/probe/issues/1707#issuecomment-944472370

76cb73944133525ef8f29e732708805520eae8e9 authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): special dns case for url hostname being IP address

47862e84541ec3deb6c69638d97ff493373f9ba5 authored about 3 years ago by Simone Basso <[email protected]>

feat(reprocessurl): script to mark single URL as not processed

c478665c35783693aa5cb78ff5f6f15e1403b350 authored about 3 years ago by Simone Basso <[email protected]>

rename: reprocess => reprocesscategory

I found the name confusing. It should be clear what it does.

7d666103ddd98309a01a7297bdfab05fc88bffb6 authored about 3 years ago by Simone Basso <[email protected]>

fix(diffurl): print directly command to see what the TH does

84853552d43c2eb649b3f7313e3ce798bd2db3bf authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): one more header to the ignore set

(I have noticed that some websites we are testing are really not
good and may randomly return 50...

98a5d015a55549e9ef1a9439804d2bd1a5b0515d authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): less confusing label when both are still missing

ad3b7852f5d496428cca0adb9ff10ba099885371 authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): make sure dns error comparison is sane

1cff4ea9f14a1f9e31e7b85162df77e919ff195c authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): refine title length analysis

e5fe9be663f04ee8d987cdb7958d234a6890beb7 authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): also account for dns failures

8004b7637c04e84fcba1b96be6595aa54174a5a5 authored about 3 years ago by Simone Basso <[email protected]>

feat(diffurl): tool to view differences between old and new TH

54f1834499ff151f76f0879705fcbc8a56a87d7c authored about 3 years ago by Simone Basso <[email protected]>

fix(showcategory): emit urls on stdout

15b45af915b0024184408ef3aa1190fc0bf8c18f authored about 3 years ago by Simone Basso <[email protected]>

feat: add script to show URLs inside a category

000d343e9cd95bcfa32ee681e6942eaf7a52acf2 authored about 3 years ago by Simone Basso <[email protected]>

feat: add script to mark URLs for reprocessing

By using this script, I can delete some URLs from the results
database for which I should have f...

a5277b154c63eb43f779783013ff96cc0ab0155e authored about 3 years ago by Simone Basso <[email protected]>

feat(compare): store interesting URLs into a database

6d06aca002ea85783fc0abe878e0342ae8bc4068 authored about 3 years ago by Simone Basso <[email protected]>

feat(compare): further break down dns/mismatch/other

Introduce the case in which we have possible poisoning.

9d5945e160e47e89f63ef5b79872a0f7cdcf23fb authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): consider case where both dns are none

This should not happen in practice. The old TH should not
do that, maybe the new one can still h...

b03e29e83b7e826f9f2d7f09be4ded3660184b14 authored about 3 years ago by Simone Basso <[email protected]>

doc(compare): better explain headers comparison

36fb9726701780efc33f5d63a7207588b0437c31 authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): remove workarounds for bugs

We've now fixed those bugs in the new TH implementation.

6486359a26ec9377043a3c97e8d7218996bdbe8a authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): correctly report number of available entries

07db667650078c9ed270e42d9809e686630bc90e authored about 3 years ago by Simone Basso <[email protected]>

fix(run): zap unused import and raise parallelism

46b5a3d96eee847abb5f7d05f39d05b3f3a678e8 authored about 3 years ago by Simone Basso <[email protected]>

fix(compare): update docstring

fd496db753b0c1cbddc505d4819c278a3e3f5aec authored about 3 years ago by Simone Basso <[email protected]>

compare: rename diff => mismatch for consistency

3f43d5d2dd790e16608ffab2ddb80a4efbe43464 authored about 3 years ago by Simone Basso <[email protected]>

fix(run): used pre-determined default parallelism

fb3ff105aa071af8ad31efa1bbdb3496c35fa5c8 authored about 3 years ago by Simone Basso <[email protected]>

fix(run): use multiprocessing to go faster

c58e6f828c028475f44098004351e83e95007d97 authored about 3 years ago by Simone Basso <[email protected]>

compare: fix bugs and improve analysis

694e013ab60c978955b3f1b4e820a96d9e333385 authored about 3 years ago by Simone Basso <[email protected]>

fix: use recent version of oohelper

aa548eb6be533b2037018bb7a90c738fb1d86b3a authored about 3 years ago by Simone Basso <[email protected]>

feat(compare): significantly improve metrics

e0fa478267669ca0408581422b4be87006f215bd authored over 3 years ago by Simone Basso <[email protected]>

fix(compare): improve our DNS comparison fu

While there, document a very interesting oddity I came across
regarding lies from upstream serve...

eb5558c7bcc5e19ffcb3388194da4d42eb43e39a authored over 3 years ago by Simone Basso <[email protected]>

feat(compare): better processing of DNS diffs

1dd1b4508835d618b147bfcc2efe3416c7337f37 authored over 3 years ago by Simone Basso <[email protected]>

chore(.gitignore): ignore .mmdb files

15027ffdc96833b6a89e26a495fc344700d1db28 authored over 3 years ago by Simone Basso <[email protected]>

feat(compare): run with specific database

2bf6a188ca60cf7f82b941a3f17e27605c9e7032 authored over 3 years ago by Simone Basso <[email protected]>

fix(run): ensure we can actually run an oohelper client

390b4ae169ca8f18bf73055919049939fc4d4d8e authored over 3 years ago by Simone Basso <[email protected]>

feat(compare): allow to ~easily extract URL w/ tcp_connect diffs

2c02fca67559812e54d6c2e3d34f09bea0b6b4ae authored over 3 years ago by Simone Basso <[email protected]>

feat(run): allow checking a subset of inputs

0f80dd26f1824354aef676fec15f66e7e04957db authored over 3 years ago by Simone Basso <[email protected]>

fix: improve comparison of http title

c6836eff2b7cd4170a21d28d0ccdc3cb21bbf3f3 authored over 3 years ago by Simone Basso <[email protected]>

fix(compare): improve processing of body length

a0952c7901bb99090ede6f10fb1b41e1d4e7e4c3 authored over 3 years ago by Simone Basso <[email protected]>

fix(compare): compare each IPv4 endpoint in TCP connect

5fb1985b08b73afe805745370cbb1362ab229d67 authored over 3 years ago by Simone Basso <[email protected]>

fix(compare): process only difference between headers

The comparison algorithm was dangerously wrong before, while now
at least it seems it's starting...

6574e2611edcc28fd665c351ef233494994f2417 authored over 3 years ago by Simone Basso <[email protected]>

feat(compare): choose which logs to show

025d3912e2caa44f8680a3f31ec9526f1406eadb authored over 3 years ago by Simone Basso <[email protected]>

fix(compare): sort DNS entries

f36267bcc2eb7b27270dcd820f493723530c4e43 authored over 3 years ago by Simone Basso <[email protected]>

fix(compare): always log to stderr

4ea0b89f5ce8408c7e1beb129e34fd1525784f38 authored over 3 years ago by Simone Basso <[email protected]>

fix(compare): reduce number of exceptions processing headers

22236c5ee2629dc4e56a8a1b429116ce9751e5d2 authored over 3 years ago by Simone Basso <[email protected]>

feat(compare): print statistics

0a12c023525cb7901a01203b19e200fe0ec2b42e authored over 3 years ago by Simone Basso <[email protected]>

refactor(compare): load JSON early

8ddf1b472b78a662410ae28a26b6327c88a70b9a authored over 3 years ago by Simone Basso <[email protected]>

fix(compare): correct the spelling of "begin"

ee575bbdcbe727b6ea5cda7904e0d17043a93b26 authored over 3 years ago by Simone Basso <[email protected]>

doc: add license file

775e638cdb4f05f7aa4caaef8da6033c630f0dd2 authored over 3 years ago by Simone Basso <[email protected]>

doc: add initial README.md file

dc92bce45d2135ef8331407deb313c69a0e6462d authored over 3 years ago by Simone Basso <[email protected]>

fix(run): remove unused import

While there, add documentation string.

b8b4c553e5c7950161ae041842017520c91bb4ef authored over 3 years ago by Simone Basso <[email protected]>

feat(run): shuffle input

de084033720097180d9b88fffb0d6176a09b154d authored over 3 years ago by Simone Basso <[email protected]>

feat: add script for comparing the results

e8029bdfa5c9cda402e60e1797ed0cd81ef62113 authored over 3 years ago by Simone Basso <[email protected]>

chore: add database.sqlite3 to .gitignore

7863a03f8f3ac9ba0abebeae8d1dbc72bfd0529a authored over 3 years ago by Simone Basso <[email protected]>

feat: introduce script to incrementally update the database

64de3a49bd7735658c0ab5d70acec7305a22f19a authored over 3 years ago by Simone Basso <[email protected]>

chore: add all the urls from the test list

ee299ea97ef106f11ef59f9cd290f0a7a26cad18 authored over 3 years ago by Simone Basso <[email protected]>

chore: add script to build oohelper

ade881b2c937503198e26fa2733c29278de2f731 authored over 3 years ago by Simone Basso <[email protected]>

fix(buildoohelper): make idempotent

4f81c713aef25d4d03911acee2d6821658270274 authored over 3 years ago by Simone Basso <[email protected]>