Ecosyste.ms: OpenCollective
An open API service for software projects hosted on Open Collective.
github.com/ArchiveTeam/wretch-grab
https://github.com/ArchiveTeam/wretch-grab
Remove the Yahoo! US IP address that has a corresponding IPv6 address; it messed up every WARC grabbed on a machine with IPv6 resolution
87ff3b586809cad2cf5098dd869672800137acf7 authored almost 11 years ago by Ivan Kozik <[email protected]>
87ff3b586809cad2cf5098dd869672800137acf7 authored almost 11 years ago by Ivan Kozik <[email protected]>
Bump version
937d0f087d548f5d905a3df69de1c7ee852b20ec authored almost 11 years ago by Ivan Kozik <[email protected]>
937d0f087d548f5d905a3df69de1c7ee852b20ec authored almost 11 years ago by Ivan Kozik <[email protected]>
Accept exit code 1 from zgrep, which it returns when there are no matches
6a273da68e3da96165e4836a11ef4441d7c8cbc6 authored almost 11 years ago by Ivan Kozik <[email protected]>
6a273da68e3da96165e4836a11ef4441d7c8cbc6 authored almost 11 years ago by Ivan Kozik <[email protected]>
Add missing import
2d31e0ce86b34f2c86149746148cd69484f2fee2 authored almost 11 years ago by Ivan Kozik <[email protected]>
2d31e0ce86b34f2c86149746148cd69484f2fee2 authored almost 11 years ago by Ivan Kozik <[email protected]>
Write out and upload a .friends file
341c697b2fa55adac2afd1dd51579e3b08be2df7 authored almost 11 years ago by Ivan Kozik <[email protected]>
341c697b2fa55adac2afd1dd51579e3b08be2df7 authored almost 11 years ago by Ivan Kozik <[email protected]>
Skip 5XX. #3.
ff4284b34d79458a14b4ae5bb7e17698e3f920d1 authored almost 11 years ago by Christopher Foo <[email protected]>
ff4284b34d79458a14b4ae5bb7e17698e3f920d1 authored almost 11 years ago by Christopher Foo <[email protected]>
Merge branch 'give-up'
Conflicts:
pipeline.py
Ignore exit code 4.
a1cb048d8c2914dfadc64081a50097fb901a3ea7 authored almost 11 years ago by Christopher Foo <[email protected]>
a1cb048d8c2914dfadc64081a50097fb901a3ea7 authored almost 11 years ago by Christopher Foo <[email protected]>
Use hard coded IP
e799cca6e446b0d1a1216ce3963e7db50a3cd11e authored almost 11 years ago by Christopher Foo <[email protected]>
e799cca6e446b0d1a1216ce3963e7db50a3cd11e authored almost 11 years ago by Christopher Foo <[email protected]>
Use random user agent on each invoke of wget.
ef8569a7ec3fa6c5f31abf05d1860e11a14030ca authored almost 11 years ago by Christopher Foo <[email protected]>
ef8569a7ec3fa6c5f31abf05d1860e11a14030ca authored almost 11 years ago by Christopher Foo <[email protected]>
Decrease sleep to 0.1sec.
37ec99598bccdca74a1c67e5544fec28049e9314 authored almost 11 years ago by Christopher Foo <[email protected]>
37ec99598bccdca74a1c67e5544fec28049e9314 authored almost 11 years ago by Christopher Foo <[email protected]>
Flag grabs done in the give-up branch with a version.
127c37c109a772dec7f3f39fcfb8f5b8e8fd485b authored almost 11 years ago by David Yip <[email protected]>
127c37c109a772dec7f3f39fcfb8f5b8e8fd485b authored almost 11 years ago by David Yip <[email protected]>
Enabled --no-cookies
38e190216aa262c89db967e99efdabf6003c44f5 authored almost 11 years ago by Christopher Foo <[email protected]>
38e190216aa262c89db967e99efdabf6003c44f5 authored almost 11 years ago by Christopher Foo <[email protected]>
Give up on a URL if it 5xxs/999s too many times.
7fa8dd2df7665619d32ad1652a4f5d07465ddbf7 authored almost 11 years ago by David Yip <[email protected]>
7fa8dd2df7665619d32ad1652a4f5d07465ddbf7 authored almost 11 years ago by David Yip <[email protected]>
Layout tweak in readme.
335018675759bde9fd9d19aec3a2d6a10927c737 authored almost 11 years ago by Preston Maness <[email protected]>
335018675759bde9fd9d19aec3a2d6a10927c737 authored almost 11 years ago by Preston Maness <[email protected]>
readme.md: ~/.local/bin appears to be standard (Closes #2).
Also removes the statement about wget-lua
18f9e20efecda2ae2d94dc0ae12782cf3cb610ed authored about 11 years ago by Christopher Foo <[email protected]>
Merge branch 'patch-1' of https://github.com/nemobis/wretch-grab into nemobis-patch-1
057ae35c39a9b5a536ebced6d63c77c37325fde4 authored about 11 years ago by Christopher Foo <[email protected]>
057ae35c39a9b5a536ebced6d63c77c37325fde4 authored about 11 years ago by Christopher Foo <[email protected]>
Skip broken d.yimg.com (Closes #1)
577dd4bdd2ed631ca14f3c7eb1a751d1896e31a2 authored about 11 years ago by Christopher Foo <[email protected]>
577dd4bdd2ed631ca14f3c7eb1a751d1896e31a2 authored about 11 years ago by Christopher Foo <[email protected]>
Update README.md
Clarify instructions for unlucky users with no root access
97a7ec3dcd05d97547650342766874cf1e7691b2 authored about 11 years ago by nemobis <[email protected]>
Fixes issue related to ArchiveTeam/yahooblog2-grab#1
7d63ae2b51206185637f1306bafb4a8efdf1135c authored about 11 years ago by Christopher Foo <[email protected]>
7d63ae2b51206185637f1306bafb4a8efdf1135c authored about 11 years ago by Christopher Foo <[email protected]>
hyves-grab -> wretch-grab.
63f87fbf744d638d3e7dd6ece193b85d8d6f541d authored about 11 years ago by David Yip <[email protected]>
63f87fbf744d638d3e7dd6ece193b85d8d6f541d authored about 11 years ago by David Yip <[email protected]>
Increase sleep to 5sec. Pick random user agent.
fdfa841b7233ae4a3f8bcf5cc15d4ff58d45fba0 authored about 11 years ago by Christopher Foo <[email protected]>
fdfa841b7233ae4a3f8bcf5cc15d4ff58d45fba0 authored about 11 years ago by Christopher Foo <[email protected]>
Increase request sleep time to 3 sec.
009e36e902833ac6961e071a9e833003e506db96 authored about 11 years ago by Christopher Foo <[email protected]>
009e36e902833ac6961e071a9e833003e506db96 authored about 11 years ago by Christopher Foo <[email protected]>
Also increase request sleep time to 2sec.
f6669f15e91937ca9b4f81445bffed555648e890 authored about 11 years ago by Christopher Foo <[email protected]>
f6669f15e91937ca9b4f81445bffed555648e890 authored about 11 years ago by Christopher Foo <[email protected]>
Increase sleep time.
d877b819f9256fe3d11541c0513770ad012c1853 authored about 11 years ago by Christopher Foo <[email protected]>
d877b819f9256fe3d11541c0513770ad012c1853 authored about 11 years ago by Christopher Foo <[email protected]>
Take out debugging output.
222d31905e529ce5c0a5346b91b46071057c329e authored about 11 years ago by Christopher Foo <[email protected]>
222d31905e529ce5c0a5346b91b46071057c329e authored about 11 years ago by Christopher Foo <[email protected]>
Add instructions to readme
4c5666c46f91541075ab81b50e6f103b859f8229 authored about 11 years ago by Christopher Foo <[email protected]>
4c5666c46f91541075ab81b50e6f103b859f8229 authored about 11 years ago by Christopher Foo <[email protected]>
Add pipeline files.
10becf14683beeb02176ab85fdb86c7d8a7012e5 authored about 11 years ago by Christopher Foo <[email protected]>
10becf14683beeb02176ab85fdb86c7d8a7012e5 authored about 11 years ago by Christopher Foo <[email protected]>
Initial commit
24ce577b9aac8dc07a0bb05e82c11944d02e0fab authored about 11 years ago by Christopher Foo <[email protected]>
24ce577b9aac8dc07a0bb05e82c11944d02e0fab authored about 11 years ago by Christopher Foo <[email protected]>