Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/wpull

Wget-compatible web downloader and crawler.
https://github.com/ArchiveTeam/wpull

changelog: Update latest to 0.1008

943a5ecb1ef178b3ee2ee2cf9d652db065cbdec2 authored almost 10 years ago by Christopher Foo <[email protected]>
builder: Check for psutil before using.

19e5e0bfce6868f8ed4e8027f18030d0a3f15979 authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Add resmon.rst

[ci skip]

cb16e737373d292f59f9c9ad8ed14f2ca74896e4 authored almost 10 years ago by Christopher Foo <[email protected]>
option: Fix wrong group label for youtube-dl

a4f1c6e45bc10de2914130461d41e5d70835b983 authored almost 10 years ago by Christopher Foo <[email protected]>
option: Fix args not added in their own group.

95a20a434c0f23dc294c761d546cb479309baa10 authored almost 10 years ago by Christopher Foo <[email protected]>
Move ResourceMontior into resmon. Check if psutil imported.

psutil may not work under pypy

d36ad31b8608a055c4f553512af4da8b7bc6f315 authored almost 10 years ago by Christopher Foo <[email protected]>
Implement --monitor-disk, --monitor-memory

952c719a219faddd316284cde0818c0d5ceae3db authored almost 10 years ago by Christopher Foo <[email protected]>
app: Add ResourceMonitor. Require psutil.

3a6b016a2163998bcd24c65151290bc69597bc07 authored almost 10 years ago by Christopher Foo <[email protected]>
engine: Fix up documentation.

d0308be183850f24eb5feb10716db2d4e5d65ff4 authored almost 10 years ago by Christopher Foo <[email protected]>
url.flatten_path: Handle '.' & '.'-related paths.

Closes chfoo/wpull#241

4f35c514b6b57c4b6d134a1037c2d25e1b61e5a9 authored almost 10 years ago by Christopher Foo <[email protected]>
ftp: Percent-decode filename from URL before passing to server

Closes chfoo/wpull#242

e48af67aa98b37dfca238dc7e563d50d26d4a4e1 authored almost 10 years ago by Christopher Foo <[email protected]>
Move filename/path functions from writer to path module.

f2dba0773207b4093366962c6ce65b630fe3a498 authored almost 10 years ago by Christopher Foo <[email protected]>
ca-bundle.pem: Update to Thu Sep 4 06:31:22 2014

91c62034327a536dd0b7d2fe5878bf3de0c32824 authored almost 10 years ago by Christopher Foo <[email protected]>
travis.yml: Use travis_retry instead of a for loop ourselves

f6aaf3b015945d6101f1d18d0a71e97157970a9b authored almost 10 years ago by Christopher Foo <[email protected]>
travis.yml: Use quiet options and retry pip errors

7cd43c5d94bbdd27d981003949b4541f7e718c67 authored almost 10 years ago by Christopher Foo <[email protected]>
options: Add help strings to unimplemented options

fa850f29c5d37d2bda1318568effefdfbe1f62c9 authored almost 10 years ago by Christopher Foo <[email protected]>
app_test: Fix early timeout on test_immediate_robots_error

6c3be0e568e56a26f4e2c9abe7c270423c80e038 authored almost 10 years ago by Christopher Foo <[email protected]>
string.format_size: Simplify if expression.

bdc0aca73ef4b002b3968b0923baf9a4f09bf331 authored almost 10 years ago by Christopher Foo <[email protected]>
Change fuzzers to check for specific exit codes.

1c02476a48dbd9299c5e4ea91c2ad300fb2510b0 authored almost 10 years ago by Christopher Foo <[email protected]>
scraper.html: Don't call with None. Don't identify on meta refresh.

Don't call identify_link_type with None. Treat meta refresh as redirects
by not attempting to id...

ef5e19c826c882eaeee775f9345912ba131318c8 authored almost 10 years ago by Christopher Foo <[email protected]>
fixup! http.client_test: Handle case with multiple warnings

d9fc3933fff5e90ea97802d7542150f0155b7592 authored almost 10 years ago by Christopher Foo <[email protected]>
http.client_test: Handle case with multiple warnings

dd9dd041b81635e0231ac8751d633f10deb126a6 authored almost 10 years ago by Christopher Foo <[email protected]>
Implement --report-speed

d171aa619d8886a52e5925b3c9508a1233c72b67 authored almost 10 years ago by Christopher Foo <[email protected]>
Implement --no-cache

dad10832d9f711de87b2bcfc98349e1f4e00269d authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1007

b547d0b62badac628633d43f7e82813226d65d5f authored almost 10 years ago by Christopher Foo <[email protected]>
Merge commit '0717f36b22242f9b9da9ffdc2778455a0984eda8'

ad06324e00c03c9d4fa9a3384d3bbdc5f30258fa authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1008a1

fd5f9604690283a7f34a65fb3b2b73ac7c5a3f0e authored almost 10 years ago by Christopher Foo <[email protected]>
changelog: Update latest to 0.1007

0717f36b22242f9b9da9ffdc2778455a0984eda8 authored almost 10 years ago by Christopher Foo <[email protected]>
http.client_test: Print out warnings for debugging

90f149bc4057593a4689a8a97aeb9bec7c2e7e1c authored almost 10 years ago by Christopher Foo <[email protected]>
Add --session-timeout.

Closes chfoo/wpull#211

ae8c70194675ab3ec7ee400ad136627222ff29de authored almost 10 years ago by Christopher Foo <[email protected]>
ftp,http: Add duration_timeout arg, DurationTimeout.

re chfoo/wpull#211

927ec1ca90ed1bac9afc710d8ff4ac6414ec0ad8 authored almost 10 years ago by Christopher Foo <[email protected]>
app_test: Add TestAppFTP.test_file_vs_directory todo

ac93f7e6b1783db78b21475bac61683e7815bfaa authored almost 10 years ago by Christopher Foo <[email protected]>
ftp.util: Fix ReplyCodes constants redeclaration

a8f9848cc6ef21784cbc4d41a37335accde57d19 authored almost 10 years ago by Christopher Foo <[email protected]>
scraper.util,util: Add lru_cache to identify_link_type, urljoin

297e6c5c57b3c0d0d7c163ea1c18353f40fda078 authored almost 10 years ago by Christopher Foo <[email protected]>
Add profile helper script.

eb9aeb7575fe939d8bcafdc5738326006b3019a2 authored almost 10 years ago by Christopher Foo <[email protected]>
collections_test: Add more tests for OrderedDefaultDict

re chfoo/wpull#189

668b710d6b34063a9664ac3dd3ea82e16f7efaf2 authored almost 10 years ago by Christopher Foo <[email protected]>
namevalue: Use str.splitlines instead of guess

401eca0485b5a834060c8ec66f9f1900593f1f82 authored almost 10 years ago by Christopher Foo <[email protected]>
http.client: Handle and warn incomplete sessions instead of assert.

Closes chfoo/wpull#177

20e051a42cfa7dade017738b70056f30078b8fe9 authored almost 10 years ago by Christopher Foo <[email protected]>
abstract.client: Rename close to abort, clean to recycle.

2e4f5b519764e67e505919b821892d9fee3e7506 authored almost 10 years ago by Christopher Foo <[email protected]>
phantomjs,youtubedl: Assert version string has no space

181adc6717ed20f71285751fe2eb275e880151f7 authored almost 10 years ago by Christopher Foo <[email protected]>
Clean up imports.

e2c5ac55845797725f42155d985db4499be52e4d authored almost 10 years ago by Christopher Foo <[email protected]>
abstract,coprocessor: Fix var declaration outside __init__

ee696ef08136b9a986eb3fb0719336141287e892 authored almost 10 years ago by Christopher Foo <[email protected]>
ftp: Track connection logins in WeakKeyDictionary

chfoo/wpull#222

217e7d830702f15bc47e001f5340a17a0d2199d8 authored almost 10 years ago by Christopher Foo <[email protected]>
recorder: Add connection_reused and connection_closed args.

772b2d980b3757f530f6972dee9bf5a930e7e8fe authored almost 10 years ago by Christopher Foo <[email protected]>
ftp.client: Convert results from MLSD to FileEntry

16e131acc3184570ddfeb3c2bf06e763c9bf4d74 authored almost 10 years ago by Christopher Foo <[email protected]>
ftp.util: Add machine_listings_to_file_entries

78aa4720de043eb5bc74dd9825971875f35877b1 authored almost 10 years ago by Christopher Foo <[email protected]>
scraper: Identify link_type in HTML scraper. Identify swf links.

Closes chfoo/wpull#226

551da576c6249c0133bd30626c3d6a26ab3e8a19 authored almost 10 years ago by Christopher Foo <[email protected]>
scraper: Return link context from JS and CSS scrapers.

Closes chfoo/wpull#150

17eb862f8154af14137f4202d9a6e2ad670c2156 authored almost 10 years ago by Christopher Foo <[email protected]>
scraper.util: Add identify_link_type

70ef83ffa7f5af2cf73e916dc069252b506ca347 authored almost 10 years ago by Christopher Foo <[email protected]>
document: Add VeryFalse and short-circuit is_supported if VeryFalse

re: chfoo/wpull#150

b8dce30177ee7a4d2b9a7d4e15a0b811ffa89b92 authored almost 10 years ago by Christopher Foo <[email protected]>
Implement --retr-symlinks=off

re: chfoo/wpull#222

5f39dbcfacae87ba6a5f0345548106c47b13c8b4 authored almost 10 years ago by Christopher Foo <[email protected]>
Implement --retr-symlinks=on.

re: chfoo/wpull#222

e7c598360805386117ec1c956327939ce4dec32a authored almost 10 years ago by Christopher Foo <[email protected]>
ftp.ls.listing: Support parsing symlink

re: chfoo/wpull#222

21791ee236db80c751c51ea5b402958c7e9faeaa authored almost 10 years ago by Christopher Foo <[email protected]>
ftp.ls.listing: Use namedlist.namedtuple to allow defaults

163778105ecfc92d51eec2c50013e324520357e2 authored almost 10 years ago by Christopher Foo <[email protected]>
app_test: Skip test_app_phantomjs under PyPy

f46643da169bcb676ac5b12e9721b2194d2b8789 authored almost 10 years ago by Christopher Foo <[email protected]>
url: Sanitize invalid URL before logging

c0eed52cfea3fbbb08c82473e570ddc586e4ca1f authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1006.1

bb93b414242bb81ee202d609f78786fdab0388c1 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge branch 'develop'

Conflicts:
wpull/version.py

5520e76a5d2006c78238a9feeb8f66584bc9de47 authored almost 10 years ago by Christopher Foo <[email protected]>
changelog: Update latest to 0.1006.1

[ci skip]

e47fac81c009c44c4ce751dda48ae9bdd7dff77e authored almost 10 years ago by Christopher Foo <[email protected]>
processor.web: Ensure --referer has priority. Don't set referer from HTTPS.

Ensure --referer option is not overwritten on subsequent requests.
Prohibit setting referer when...

8c456fd47b0b1a1a63dbc6e762ffafbd1de6a8db authored almost 10 years ago by Christopher Foo <[email protected]>
app_test: Add tests for --referer option

9d31d2a7e57740e95fa7d9a6a7c9b1cd4a3d7128 authored almost 10 years ago by Christopher Foo <[email protected]>
app_test: Skip test_app_phantomjs_scroll under PyPy

d3a6ccaac1959fc6c71cb6d2221aff7842a0e0bc authored almost 10 years ago by Christopher Foo <[email protected]>
app: Set logging.raiseExceptions False if force closed

Closes chfoo/wpull#162

d07cf0e171bdd08218d0c4d5ede226bdb7b93ee0 authored almost 10 years ago by Christopher Foo <[email protected]>
app: Add debug log in _start_servers

6cb2a9b6d20013d982ffee20a31df94eaa98d679 authored almost 10 years ago by Christopher Foo <[email protected]>
processor: Use consistent set of server exception classes

Closes chfoo/wpull#233

7d04018745cd42ac1e7034bdca1162c6fff4cfd0 authored almost 10 years ago by Christopher Foo <[email protected]>
app_test: Use IP address to avoid querying nameserver

4a7c95ba87e9b4bc2021a4d96e6c72fdc61f3fd8 authored almost 10 years ago by Christopher Foo <[email protected]>
dns: Use get_exception_message to avoid AttributeError on errors

Closes chfoo/wpull#234

6b6faa9e78ab189c41d07e0914e41b640828c1b7 authored almost 10 years ago by Christopher Foo <[email protected]>
util: Add get_exception_message

a44bcee466f5b9a985c466e0076a38d421748499 authored almost 10 years ago by Christopher Foo <[email protected]>
{ftp,http}.request: Check if body has to_dict attr before calling it.

Closes chfoo/wpull#236

f8f0eedc687496c4d7844fab91a62229228f9956 authored almost 10 years ago by Christopher Foo <[email protected]>
app_test: Increase timeout for test_app_phantomjs_scroll on pypy

d13dbfa3e19f2fb87995c2e7697c706fea0a1245 authored almost 10 years ago by Christopher Foo <[email protected]>
proxy: Set request.body to None as it is not a Body instance

Re: chfoo/wpull#236

277ea4bf35cb5f11db8eaf2ae13831cbab2d3bd3 authored almost 10 years ago by Christopher Foo <[email protected]>
proxy: Raise ConnectionAbortedError instead of return in failed start TLS

656f32c2617f75de7cd28ff4b49cbb0f2d2275bd authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1006

5ab01d8d341420da9b6d32b872b0b85ea8b770a7 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge commit 'da2861eeac61ddd3e7661a79966da0c1fca71b6c'

Conflicts:
wpull/version.py

806630fe98a92423c19168f01b291d8cb484698e authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1007a1

[ci skip]

d32a034925d2702d3780bc14748172c89abf29ee authored almost 10 years ago by Christopher Foo <[email protected]>
changelog: Update latest to 0.1006

[ci skip]

da2861eeac61ddd3e7661a79966da0c1fca71b6c authored almost 10 years ago by Christopher Foo <[email protected]>
ftp.ls.date: Subtract year only when ambigous and in the future

6b233a308ca057f720dfb6b9c0ad9e7698af8b5f authored almost 10 years ago by Christopher Foo <[email protected]>
driver,proxy: Add some more debugging logs

2d59b987fd33d9d828fae7add1edc336682b46c3 authored almost 10 years ago by Christopher Foo <[email protected]>
proxy: Catch ValueError: Invalid file descriptor

Closes chfoo/wpull#231

acd02812a44cd622c6ccd0381605826e178a6f62 authored almost 10 years ago by Christopher Foo <[email protected]>
Add PluginEnvironment and --plugin-script option.

4335e05549dc5ee645db6f416df70380b0aa1481 authored almost 10 years ago by Christopher Foo <[email protected]>
Pass user-agent to YoutubeDlCoprocessor

6be81ab9c07773b5214b0c0d9e44df0202e9e161 authored almost 10 years ago by Christopher Foo <[email protected]>
app,builder: yield from server tasks before running

e8ffe6aee7ad2d40157710026a96b3e8e206bcc6 authored almost 10 years ago by Christopher Foo <[email protected]>
doc,readme: Update note about youtube-dl integration

17d2f05d88e3794185596dcc5740defd3435f173 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge branch 'issue/22-flash_videos' into develop

re: chfoo/wpull#22

0239a613ea09eb2c7c8a005511189b0457fb710f authored almost 10 years ago by Christopher Foo <[email protected]>
app_test: Skip test_youtube_dl

7ff79c1c015f2b948f22e90dea8b9bdd9226b8a6 authored almost 10 years ago by Christopher Foo <[email protected]>
.travis.yml: pip install youtube-dl

396c9c195e748b90bd064118261579919c4a5e4a authored almost 10 years ago by Christopher Foo <[email protected]>
Add youtube-dl tests. Include youtube-dl version in WARC.

40f8d2a3db9c72a0a800ff24d34c05d780d09dd6 authored almost 10 years ago by Christopher Foo <[email protected]>
coprocessor.youtubedl: Download annotations/metadata. Log process output

19dc771cc72dc8a539737361f86690966ad0a256 authored almost 10 years ago by Christopher Foo <[email protected]>
proxy: Add debug log for request after possible CONNECT

cfa5527bcc62efd754cb72deb221b652b9654ffb authored almost 10 years ago by Christopher Foo <[email protected]>
app,builder: Undo and make server Task to allow use under unit tests

f2a19a9a2106be276cd5af1e15bed0cdcbca88bd authored almost 10 years ago by Christopher Foo <[email protected]>
http.stream: Check for file, check for data before drain

Add check for file is not None before writing to it.
Check if content_data is not empty to avoid...

bf206ffd286056c084a3d088baa46d69fe8461ae authored almost 10 years ago by Christopher Foo <[email protected]>
app,builder: Gracefully close proxy server and wait for close

01c0eaa79784e164f9f676aff3ba51447bcb32e2 authored almost 10 years ago by Christopher Foo <[email protected]>
Add WIP YoutubeDlCoprocessor stub.

163fb44d4e67add596ec2a751f012d0b5e04b528 authored almost 10 years ago by Christopher Foo <[email protected]>
builder,options: Add --proxy-server option

re: chfoo/wpull#22

cdb236bcd945f46cd48ab55141eabf82a7fa8cdc authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Regen coprocessor.proxy rst file.

[ci skip]

ded142c679e0bf68e831738106e571fbb1ba065a authored almost 10 years ago by Christopher Foo <[email protected]>
options: Remove commented non-canon truncate-document option

[ci skip]

30debbf4b4677dacaed5c0a5cd5c150b7596f50e authored almost 10 years ago by Christopher Foo <[email protected]>
proxy_test: Fix pre_response_callback

fb8c4fa4be126ce498d66f0a92b588e95f5181e2 authored almost 10 years ago by Christopher Foo <[email protected]>
coprocessor.proxy: Call FetchRule and ResultRule with mock instances

re: chfoo/wpull#228 chfoo/wpull#123

7abea0f13f9abeabb167b3513044c891245d38f9 authored almost 10 years ago by Christopher Foo <[email protected]>
coprocessor.proxy: Add fetch logging

re: chfoo/wpull#228

d59102bd708f3e4bc33c133b11e5bfe6b4fddcc1 authored almost 10 years ago by Christopher Foo <[email protected]>
proxy: Add pre_response_callback

baf340bc22d5038d923b97dc080969b32d1aa1ea authored almost 10 years ago by Christopher Foo <[email protected]>