github.com/ArchiveTeam/ludios_wpull commits | Ecosyste.ms: OpenCollective

option: Fix args not added in their own group.

95a20a434c0f23dc294c761d546cb479309baa10 authored almost 10 years ago

Move ResourceMontior into resmon. Check if psutil imported.

psutil may not work under pypy

d36ad31b8608a055c4f553512af4da8b7bc6f315 authored almost 10 years ago

Implement --monitor-disk, --monitor-memory

952c719a219faddd316284cde0818c0d5ceae3db authored almost 10 years ago

app: Add ResourceMonitor. Require psutil.

3a6b016a2163998bcd24c65151290bc69597bc07 authored almost 10 years ago

engine: Fix up documentation.

d0308be183850f24eb5feb10716db2d4e5d65ff4 authored almost 10 years ago

url.flatten_path: Handle '.' & '.'-related paths.

Closes chfoo/wpull#241

4f35c514b6b57c4b6d134a1037c2d25e1b61e5a9 authored almost 10 years ago

ftp: Percent-decode filename from URL before passing to server

Closes chfoo/wpull#242

e48af67aa98b37dfca238dc7e563d50d26d4a4e1 authored almost 10 years ago

Move filename/path functions from writer to path module.

f2dba0773207b4093366962c6ce65b630fe3a498 authored almost 10 years ago

ca-bundle.pem: Update to Thu Sep 4 06:31:22 2014

91c62034327a536dd0b7d2fe5878bf3de0c32824 authored almost 10 years ago

travis.yml: Use travis_retry instead of a for loop ourselves

f6aaf3b015945d6101f1d18d0a71e97157970a9b authored almost 10 years ago

travis.yml: Use quiet options and retry pip errors

7cd43c5d94bbdd27d981003949b4541f7e718c67 authored almost 10 years ago

options: Add help strings to unimplemented options

fa850f29c5d37d2bda1318568effefdfbe1f62c9 authored almost 10 years ago

app_test: Fix early timeout on test_immediate_robots_error

6c3be0e568e56a26f4e2c9abe7c270423c80e038 authored almost 10 years ago

string.format_size: Simplify if expression.

bdc0aca73ef4b002b3968b0923baf9a4f09bf331 authored almost 10 years ago

Change fuzzers to check for specific exit codes.

1c02476a48dbd9299c5e4ea91c2ad300fb2510b0 authored almost 10 years ago

scraper.html: Don't call with None. Don't identify on meta refresh.

Don't call identify_link_type with None. Treat meta refresh as redirects
by not attempting to id...

ef5e19c826c882eaeee775f9345912ba131318c8 authored almost 10 years ago

fixup! http.client_test: Handle case with multiple warnings

d9fc3933fff5e90ea97802d7542150f0155b7592 authored almost 10 years ago

http.client_test: Handle case with multiple warnings

dd9dd041b81635e0231ac8751d633f10deb126a6 authored almost 10 years ago

Implement --report-speed

d171aa619d8886a52e5925b3c9508a1233c72b67 authored almost 10 years ago

Implement --no-cache

dad10832d9f711de87b2bcfc98349e1f4e00269d authored almost 10 years ago

Bump version 0.1007

b547d0b62badac628633d43f7e82813226d65d5f authored almost 10 years ago

Merge commit '0717f36b22242f9b9da9ffdc2778455a0984eda8'

ad06324e00c03c9d4fa9a3384d3bbdc5f30258fa authored almost 10 years ago

Bump version 0.1008a1

fd5f9604690283a7f34a65fb3b2b73ac7c5a3f0e authored almost 10 years ago

changelog: Update latest to 0.1007

0717f36b22242f9b9da9ffdc2778455a0984eda8 authored almost 10 years ago

http.client_test: Print out warnings for debugging

90f149bc4057593a4689a8a97aeb9bec7c2e7e1c authored almost 10 years ago

Add --session-timeout.

Closes chfoo/wpull#211

ae8c70194675ab3ec7ee400ad136627222ff29de authored almost 10 years ago

ftp,http: Add duration_timeout arg, DurationTimeout.

re chfoo/wpull#211

927ec1ca90ed1bac9afc710d8ff4ac6414ec0ad8 authored almost 10 years ago

app_test: Add TestAppFTP.test_file_vs_directory todo

ac93f7e6b1783db78b21475bac61683e7815bfaa authored almost 10 years ago

ftp.util: Fix ReplyCodes constants redeclaration

a8f9848cc6ef21784cbc4d41a37335accde57d19 authored almost 10 years ago

scraper.util,util: Add lru_cache to identify_link_type, urljoin

297e6c5c57b3c0d0d7c163ea1c18353f40fda078 authored almost 10 years ago

Add profile helper script.

eb9aeb7575fe939d8bcafdc5738326006b3019a2 authored almost 10 years ago

collections_test: Add more tests for OrderedDefaultDict

re chfoo/wpull#189

668b710d6b34063a9664ac3dd3ea82e16f7efaf2 authored almost 10 years ago

namevalue: Use str.splitlines instead of guess

401eca0485b5a834060c8ec66f9f1900593f1f82 authored almost 10 years ago

http.client: Handle and warn incomplete sessions instead of assert.

Closes chfoo/wpull#177

20e051a42cfa7dade017738b70056f30078b8fe9 authored almost 10 years ago

abstract.client: Rename close to abort, clean to recycle.

2e4f5b519764e67e505919b821892d9fee3e7506 authored almost 10 years ago

phantomjs,youtubedl: Assert version string has no space

181adc6717ed20f71285751fe2eb275e880151f7 authored almost 10 years ago

Clean up imports.

e2c5ac55845797725f42155d985db4499be52e4d authored almost 10 years ago

abstract,coprocessor: Fix var declaration outside __init__

ee696ef08136b9a986eb3fb0719336141287e892 authored almost 10 years ago

ftp: Track connection logins in WeakKeyDictionary

chfoo/wpull#222

217e7d830702f15bc47e001f5340a17a0d2199d8 authored almost 10 years ago

recorder: Add connection_reused and connection_closed args.

772b2d980b3757f530f6972dee9bf5a930e7e8fe authored almost 10 years ago

ftp.client: Convert results from MLSD to FileEntry

16e131acc3184570ddfeb3c2bf06e763c9bf4d74 authored almost 10 years ago

ftp.util: Add machine_listings_to_file_entries

78aa4720de043eb5bc74dd9825971875f35877b1 authored almost 10 years ago

scraper: Identify link_type in HTML scraper. Identify swf links.

Closes chfoo/wpull#226

551da576c6249c0133bd30626c3d6a26ab3e8a19 authored almost 10 years ago

scraper: Return link context from JS and CSS scrapers.

Closes chfoo/wpull#150

17eb862f8154af14137f4202d9a6e2ad670c2156 authored almost 10 years ago

scraper.util: Add identify_link_type

70ef83ffa7f5af2cf73e916dc069252b506ca347 authored almost 10 years ago

document: Add VeryFalse and short-circuit is_supported if VeryFalse

re: chfoo/wpull#150

b8dce30177ee7a4d2b9a7d4e15a0b811ffa89b92 authored almost 10 years ago

Implement --retr-symlinks=off

re: chfoo/wpull#222

5f39dbcfacae87ba6a5f0345548106c47b13c8b4 authored almost 10 years ago

Implement --retr-symlinks=on.

re: chfoo/wpull#222

e7c598360805386117ec1c956327939ce4dec32a authored almost 10 years ago

ftp.ls.listing: Support parsing symlink

re: chfoo/wpull#222

21791ee236db80c751c51ea5b402958c7e9faeaa authored almost 10 years ago

ftp.ls.listing: Use namedlist.namedtuple to allow defaults

163778105ecfc92d51eec2c50013e324520357e2 authored almost 10 years ago

app_test: Skip test_app_phantomjs under PyPy

f46643da169bcb676ac5b12e9721b2194d2b8789 authored almost 10 years ago

url: Sanitize invalid URL before logging

c0eed52cfea3fbbb08c82473e570ddc586e4ca1f authored almost 10 years ago

Bump version 0.1006.1

bb93b414242bb81ee202d609f78786fdab0388c1 authored almost 10 years ago

Merge branch 'develop'

Conflicts:
wpull/version.py

5520e76a5d2006c78238a9feeb8f66584bc9de47 authored almost 10 years ago

changelog: Update latest to 0.1006.1

[ci skip]

e47fac81c009c44c4ce751dda48ae9bdd7dff77e authored almost 10 years ago

processor.web: Ensure --referer has priority. Don't set referer from HTTPS.

Ensure --referer option is not overwritten on subsequent requests.
Prohibit setting referer when...

8c456fd47b0b1a1a63dbc6e762ffafbd1de6a8db authored almost 10 years ago

app_test: Add tests for --referer option

9d31d2a7e57740e95fa7d9a6a7c9b1cd4a3d7128 authored almost 10 years ago

app_test: Skip test_app_phantomjs_scroll under PyPy

d3a6ccaac1959fc6c71cb6d2221aff7842a0e0bc authored almost 10 years ago

app: Set logging.raiseExceptions False if force closed

Closes chfoo/wpull#162

d07cf0e171bdd08218d0c4d5ede226bdb7b93ee0 authored almost 10 years ago

app: Add debug log in _start_servers

6cb2a9b6d20013d982ffee20a31df94eaa98d679 authored almost 10 years ago

processor: Use consistent set of server exception classes

Closes chfoo/wpull#233

7d04018745cd42ac1e7034bdca1162c6fff4cfd0 authored almost 10 years ago

app_test: Use IP address to avoid querying nameserver

4a7c95ba87e9b4bc2021a4d96e6c72fdc61f3fd8 authored almost 10 years ago

dns: Use get_exception_message to avoid AttributeError on errors

Closes chfoo/wpull#234

6b6faa9e78ab189c41d07e0914e41b640828c1b7 authored almost 10 years ago

util: Add get_exception_message

a44bcee466f5b9a985c466e0076a38d421748499 authored almost 10 years ago

{ftp,http}.request: Check if body has to_dict attr before calling it.

Closes chfoo/wpull#236

f8f0eedc687496c4d7844fab91a62229228f9956 authored almost 10 years ago

app_test: Increase timeout for test_app_phantomjs_scroll on pypy

d13dbfa3e19f2fb87995c2e7697c706fea0a1245 authored almost 10 years ago

proxy: Set request.body to None as it is not a Body instance

Re: chfoo/wpull#236

277ea4bf35cb5f11db8eaf2ae13831cbab2d3bd3 authored almost 10 years ago

proxy: Raise ConnectionAbortedError instead of return in failed start TLS

656f32c2617f75de7cd28ff4b49cbb0f2d2275bd authored almost 10 years ago

Bump version 0.1006

5ab01d8d341420da9b6d32b872b0b85ea8b770a7 authored almost 10 years ago

Merge commit 'da2861eeac61ddd3e7661a79966da0c1fca71b6c'

Conflicts:
wpull/version.py

806630fe98a92423c19168f01b291d8cb484698e authored almost 10 years ago

Bump version 0.1007a1

[ci skip]

d32a034925d2702d3780bc14748172c89abf29ee authored almost 10 years ago

changelog: Update latest to 0.1006

[ci skip]

da2861eeac61ddd3e7661a79966da0c1fca71b6c authored almost 10 years ago

ftp.ls.date: Subtract year only when ambigous and in the future

6b233a308ca057f720dfb6b9c0ad9e7698af8b5f authored almost 10 years ago

driver,proxy: Add some more debugging logs

2d59b987fd33d9d828fae7add1edc336682b46c3 authored almost 10 years ago

proxy: Catch ValueError: Invalid file descriptor

Closes chfoo/wpull#231

acd02812a44cd622c6ccd0381605826e178a6f62 authored almost 10 years ago

Add PluginEnvironment and --plugin-script option.

4335e05549dc5ee645db6f416df70380b0aa1481 authored almost 10 years ago

Pass user-agent to YoutubeDlCoprocessor

6be81ab9c07773b5214b0c0d9e44df0202e9e161 authored almost 10 years ago

app,builder: yield from server tasks before running

e8ffe6aee7ad2d40157710026a96b3e8e206bcc6 authored almost 10 years ago

doc,readme: Update note about youtube-dl integration

17d2f05d88e3794185596dcc5740defd3435f173 authored almost 10 years ago

Merge branch 'issue/22-flash_videos' into develop

re: chfoo/wpull#22

0239a613ea09eb2c7c8a005511189b0457fb710f authored almost 10 years ago

app_test: Skip test_youtube_dl

7ff79c1c015f2b948f22e90dea8b9bdd9226b8a6 authored almost 10 years ago

.travis.yml: pip install youtube-dl

396c9c195e748b90bd064118261579919c4a5e4a authored almost 10 years ago

Add youtube-dl tests. Include youtube-dl version in WARC.

40f8d2a3db9c72a0a800ff24d34c05d780d09dd6 authored almost 10 years ago

coprocessor.youtubedl: Download annotations/metadata. Log process output

19dc771cc72dc8a539737361f86690966ad0a256 authored almost 10 years ago

proxy: Add debug log for request after possible CONNECT

cfa5527bcc62efd754cb72deb221b652b9654ffb authored almost 10 years ago

app,builder: Undo and make server Task to allow use under unit tests

f2a19a9a2106be276cd5af1e15bed0cdcbca88bd authored almost 10 years ago

http.stream: Check for file, check for data before drain

Add check for file is not None before writing to it.
Check if content_data is not empty to avoid...

bf206ffd286056c084a3d088baa46d69fe8461ae authored almost 10 years ago

app,builder: Gracefully close proxy server and wait for close

01c0eaa79784e164f9f676aff3ba51447bcb32e2 authored almost 10 years ago

Add WIP YoutubeDlCoprocessor stub.

163fb44d4e67add596ec2a751f012d0b5e04b528 authored almost 10 years ago

builder,options: Add --proxy-server option

re: chfoo/wpull#22

cdb236bcd945f46cd48ab55141eabf82a7fa8cdc authored almost 10 years ago

doc: Regen coprocessor.proxy rst file.

[ci skip]

ded142c679e0bf68e831738106e571fbb1ba065a authored almost 10 years ago

options: Remove commented non-canon truncate-document option

[ci skip]

30debbf4b4677dacaed5c0a5cd5c150b7596f50e authored almost 10 years ago

proxy_test: Fix pre_response_callback

fb8c4fa4be126ce498d66f0a92b588e95f5181e2 authored almost 10 years ago

coprocessor.proxy: Call FetchRule and ResultRule with mock instances

re: chfoo/wpull#228 chfoo/wpull#123

7abea0f13f9abeabb167b3513044c891245d38f9 authored almost 10 years ago

coprocessor.proxy: Add fetch logging

re: chfoo/wpull#228

d59102bd708f3e4bc33c133b11e5bfe6b4fddcc1 authored almost 10 years ago

proxy: Add pre_response_callback

baf340bc22d5038d923b97dc080969b32d1aa1ea authored almost 10 years ago

changelog: Note PhantomJS improvement

0aca4d7041dee4d57671a11071c6226f5f929562 authored almost 10 years ago

Merge branch 'issue/230-phantomjs-junk' into develop

55594e468ff42aec4e40af6684293a902316c515 authored almost 10 years ago

driver: Check if resourceReceived event stage is 'end'

6dc12be62d789aa9c8a6829cc610b6f786f1e80e authored almost 10 years ago

Change phantomjs-scroll default to 20. Add phantomjs-max-time option.

318fb124de1e674902bb17a90a1d0dbde9230a39 authored almost 10 years ago