Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/wretch-grab


https://github.com/ArchiveTeam/wretch-grab

Remove the Yahoo! US IP address that has a corresponding IPv6 address; it messed up every WARC grabbed on a machine with IPv6 resolution

87ff3b586809cad2cf5098dd869672800137acf7 authored almost 11 years ago by Ivan Kozik <[email protected]>
Bump version

937d0f087d548f5d905a3df69de1c7ee852b20ec authored almost 11 years ago by Ivan Kozik <[email protected]>
Accept exit code 1 from zgrep, which it returns when there are no matches

6a273da68e3da96165e4836a11ef4441d7c8cbc6 authored almost 11 years ago by Ivan Kozik <[email protected]>
Add missing import

2d31e0ce86b34f2c86149746148cd69484f2fee2 authored almost 11 years ago by Ivan Kozik <[email protected]>
Write out and upload a .friends file

341c697b2fa55adac2afd1dd51579e3b08be2df7 authored almost 11 years ago by Ivan Kozik <[email protected]>
Skip 5XX. #3.

ff4284b34d79458a14b4ae5bb7e17698e3f920d1 authored almost 11 years ago by Christopher Foo <[email protected]>
Merge branch 'give-up'

Conflicts:
pipeline.py

b076ad66f0c656abd341d8006c4cab940931f64f authored almost 11 years ago by Christopher Foo <[email protected]>
Ignore exit code 4.

a1cb048d8c2914dfadc64081a50097fb901a3ea7 authored almost 11 years ago by Christopher Foo <[email protected]>
Use hard coded IP

e799cca6e446b0d1a1216ce3963e7db50a3cd11e authored almost 11 years ago by Christopher Foo <[email protected]>
Use random user agent on each invoke of wget.

ef8569a7ec3fa6c5f31abf05d1860e11a14030ca authored almost 11 years ago by Christopher Foo <[email protected]>
Decrease sleep to 0.1sec.

37ec99598bccdca74a1c67e5544fec28049e9314 authored almost 11 years ago by Christopher Foo <[email protected]>
Flag grabs done in the give-up branch with a version.

127c37c109a772dec7f3f39fcfb8f5b8e8fd485b authored almost 11 years ago by David Yip <[email protected]>
Enabled --no-cookies

38e190216aa262c89db967e99efdabf6003c44f5 authored almost 11 years ago by Christopher Foo <[email protected]>
Give up on a URL if it 5xxs/999s too many times.

7fa8dd2df7665619d32ad1652a4f5d07465ddbf7 authored almost 11 years ago by David Yip <[email protected]>
Layout tweak in readme.

335018675759bde9fd9d19aec3a2d6a10927c737 authored almost 11 years ago by Preston Maness <[email protected]>
readme.md: ~/.local/bin appears to be standard (Closes #2).

Also removes the statement about wget-lua

18f9e20efecda2ae2d94dc0ae12782cf3cb610ed authored about 11 years ago by Christopher Foo <[email protected]>
Merge branch 'patch-1' of https://github.com/nemobis/wretch-grab into nemobis-patch-1

057ae35c39a9b5a536ebced6d63c77c37325fde4 authored about 11 years ago by Christopher Foo <[email protected]>
Skip broken d.yimg.com (Closes #1)

577dd4bdd2ed631ca14f3c7eb1a751d1896e31a2 authored about 11 years ago by Christopher Foo <[email protected]>
Update README.md

Clarify instructions for unlucky users with no root access

97a7ec3dcd05d97547650342766874cf1e7691b2 authored about 11 years ago by nemobis <[email protected]>
Fixes issue related to ArchiveTeam/yahooblog2-grab#1

7d63ae2b51206185637f1306bafb4a8efdf1135c authored about 11 years ago by Christopher Foo <[email protected]>
hyves-grab -> wretch-grab.

63f87fbf744d638d3e7dd6ece193b85d8d6f541d authored about 11 years ago by David Yip <[email protected]>
Increase sleep to 5sec. Pick random user agent.

fdfa841b7233ae4a3f8bcf5cc15d4ff58d45fba0 authored about 11 years ago by Christopher Foo <[email protected]>
Increase request sleep time to 3 sec.

009e36e902833ac6961e071a9e833003e506db96 authored about 11 years ago by Christopher Foo <[email protected]>
Also increase request sleep time to 2sec.

f6669f15e91937ca9b4f81445bffed555648e890 authored about 11 years ago by Christopher Foo <[email protected]>
Increase sleep time.

d877b819f9256fe3d11541c0513770ad012c1853 authored about 11 years ago by Christopher Foo <[email protected]>
Take out debugging output.

222d31905e529ce5c0a5346b91b46071057c329e authored about 11 years ago by Christopher Foo <[email protected]>
Add instructions to readme

4c5666c46f91541075ab81b50e6f103b859f8229 authored about 11 years ago by Christopher Foo <[email protected]>
Add pipeline files.

10becf14683beeb02176ab85fdb86c7d8a7012e5 authored about 11 years ago by Christopher Foo <[email protected]>
Initial commit

24ce577b9aac8dc07a0bb05e82c11944d02e0fab authored about 11 years ago by Christopher Foo <[email protected]>