github.com/ArchiveTeam/wpull commits | Ecosyste.ms: OpenCollective

coprocessor.phantomjs: Time out wait_load()

dcdd53d03886e01790dbe4021679f20d4051cf9a authored about 10 years ago by Christopher Foo <[email protected]>

coprocessor.phantomjs: Remove HTTPS rewrite hack

f06ebfd57a3cc38d3386bbf2a6f5db9df78e638e authored about 10 years ago by Christopher Foo <[email protected]>

proxy: Support MITM SSL proxying.

890e392b885acfa129c3c850978b8c1c79aab488 authored about 10 years ago by Christopher Foo <[email protected]>

proxy: Add another rewrite case for referrer

f6c78ed5909082394ec9efbaea514207aa620b8a authored about 10 years ago by Christopher Foo <[email protected]>

coprocessor.phantomjs: Better screenshotting logic.

Always fetch resources. Scroll to top when screenshot. Click corner of
page to get rid of any ov...

0d7df2333a97121516811a16aef6a9d63e1b2760 authored about 10 years ago by Christopher Foo <[email protected]>

driver: Add send_key and send_click RPC calls.

d01663391b993cb53ad35495ec44f1841fb2f3a2 authored about 10 years ago by Christopher Foo <[email protected]>

driver.resource: Catch and discard KeyErrors

08e26f4e5e448d1a32bcee3255b761e73d45dce8 authored about 10 years ago by Christopher Foo <[email protected]>

coprocessor.phantomjs: Reimplement timeout and retry handling.

e3e7897d08fc1ec8d47d9156a5ec25db8dc2bfa4 authored about 10 years ago by Christopher Foo <[email protected]>

driver.phantomjs: Implement timeout.

422ab330a5b35db05f6684bbe602d7884c24c6b5 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.client: Move async call outside exception handler

Closes chfoo/wpull#215

To see this problem quickly, put raise ValueError inside processor.ftp
_...

821afce940c1536460064c5892fa5785515bab02 authored about 10 years ago by Christopher Foo <[email protected]>

setup.py: Update cx_freeze config for moved phantomjs.js file.

[ci skip]

bae851b749bcb754bb0b8cf6a551565f40804103 authored about 10 years ago by Christopher Foo <[email protected]>

fpt.command: Reset connection before use

Closes chfoo/wpull#214

ae8a5ad548bb09d075c27aba5bd15ff31ca4f085 authored about 10 years ago by Christopher Foo <[email protected]>

fixup! driver.phantomjs: Fix process reuse

9821b526f4fae514664b2ff71727d414ff082cad authored about 10 years ago by Christopher Foo <[email protected]>

coprocessor.phantomjs: Fix up url rewriting hacks. Clean temp files.

516218034118d96f8c631c2a586f7a594620fa48 authored about 10 years ago by Christopher Foo <[email protected]>

driver.phantomjs: Fix process reuse

1875ce2dba1bedd7b8eb1961ef31f43a312d9103 authored about 10 years ago by Christopher Foo <[email protected]>

Fix up phantomjs events and resource tracker events

0f52e6f7a9258656640141861581a5ae5a429261 authored about 10 years ago by Christopher Foo <[email protected]>

WIP coprocessor.phantomjs rewrite.

a8c3bdc30122c9f147be097f3aa50214cecc6cbc authored about 10 years ago by Christopher Foo <[email protected]>

driver.process: Wrap try-except and log on error reading stdout/stderr

f72f8454a1697a640d06a895ee1f3748c2baf350 authored about 10 years ago by Christopher Foo <[email protected]>

travis.yml: Blacklist topic/rewrite_phantomjs_2

8ef2b6205210c6084e0a33821201996a3f4e573e authored about 10 years ago by Christopher Foo <[email protected]>

Merge branch 'develop' into topic/rewrite_phantomjs_2

Conflicts:
wpull/processor/web.py

f1926e2f7b44a3dc9ca2127da2a934b147fedd87 authored about 10 years ago by Christopher Foo <[email protected]>

Bump version 0.1003

f67d12daa95dacfec59805836d7bf20bb480d482 authored about 10 years ago by Christopher Foo <[email protected]>

Merge commit '8cc79c24b4900710310d34392603d9b0d536b55b'

Conflicts:
wpull/version.py

bc0bd3b1096e68f0df35d686c3d59a0b9bfc3e65 authored about 10 years ago by Christopher Foo <[email protected]>

Bump version to 0.1004a1

86ae89fbe5f031dd228a7aa712e7ac3bf21e1d4f authored about 10 years ago by Christopher Foo <[email protected]>

changlog: Update latest to 0.1003

[ci skip]

8cc79c24b4900710310d34392603d9b0d536b55b authored about 10 years ago by Christopher Foo <[email protected]>

Fix missing response scripting hook callback during redirects.

fde26fae4d7f1b3e9709dfe320680ca712c85e4a authored about 10 years ago by Christopher Foo <[email protected]>

fixup! ftp.ls: Better detection of msdos ls w/o dir

Fix unit test

59c9d8f574fcdabeef0b3a1b92129f5fef62239b authored about 10 years ago by Christopher Foo <[email protected]>

stats: Add debug log

4016177170a68600b8a8e651ca8fd1ca0a0648c7 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.ls: Better detection of msdos ls w/o dir

c5507b4fa7d2799526be7a904c96f0ae23ad169c authored about 10 years ago by Christopher Foo <[email protected]>

http.stream_test,scraper.html_test: Add pragma no cover

960651ea83def23b928f958736bbc45b8c37cf85 authored about 10 years ago by Christopher Foo <[email protected]>

ftp: Catch exception when passive address cannot be parsed.

81bb66a91c40a2b58e17fff119fbf8121aa75878 authored about 10 years ago by Christopher Foo <[email protected]>

driver: WIP add scroller module.

0d0d2f323626b121fb69f25c362a9bb592777d15 authored about 10 years ago by Christopher Foo <[email protected]>

driver: Clean up a bit and add get_page_url(), resource_timeout

8536eba08fb670dc717fb84b8cb787e12a18ed3f authored about 10 years ago by Christopher Foo <[email protected]>

driver: WIP Add resource unit tests

6ae7472b0ba2193ab97785eed3881fd4d4901298 authored about 10 years ago by Christopher Foo <[email protected]>

url: Use original string instead of lowercased on schemeless URL

Closes chfoo/wpull#213

05458bc7b0ca058a3bbd7fbaaac882cbc38f9ccc authored about 10 years ago by Christopher Foo <[email protected]>

driver: Fix up phantomjs pool unit tests

2c93b361f277d6bb0ab6d0fc55808750a391f713 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.ls.date.parse_datetime: Fix 12 AM/PM to 24hr logic.

1e2b4bb32a75320b94d88bc6a7099a0ff731ab5b authored about 10 years ago by Christopher Foo <[email protected]>

ftp.client: Add some debug logging

7a919d85299b391be8db48c9133f1e78c3d18ff5 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.command: Accept FTP code 125 on data connection

80c7c8275010cccdc41206760cf52510b13582df authored about 10 years ago by Christopher Foo <[email protected]>

Rewrite ftp.client interface to align with http.client interface

ftp.client: fetch() pairs with read_contnet(), fetch_file_listing()
pairs with read_listing_cont...

94d4c1f4a92359a3486b27f9e98490352a612704 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.command: Split read_stream into begin_stream & read_stream

e6f4960a1ab5c92295c8fce75228e6fc686c506e authored about 10 years ago by Christopher Foo <[email protected]>

http.client: Better docstring.

4d27da0e87fe788301f437049f21ee5a260cc875 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.client: Remove outdated TODO

800ce05974a981ac2ad038c8026535ac28c01dca authored about 10 years ago by Christopher Foo <[email protected]>

abstract.client,ftp.util: Add some docstrings.

1378ebb956406c1af388cabf6a54012420176911 authored about 10 years ago by Christopher Foo <[email protected]>

http.robots: Add missing import.

ed07f7d8ec74b463546a7689ca61e0d1c0eb3949 authored about 10 years ago by Christopher Foo <[email protected]>

driver: WIP Rewrite PhantomJSClient into PhantomJSPool

e830c768e86d6393c1ec893601055be576e710c2 authored about 10 years ago by Christopher Foo <[email protected]>

WIP Rewrite PhantomJSRemote to PhantomJSDriver

de30c9b89daae74c5ade82824f1264786232b657 authored about 10 years ago by Christopher Foo <[email protected]>

WIP Rewrite phantomjs script with Haxe version with RPC polling

c604276e31935b33719301b98af79a8beaa2b509 authored about 10 years ago by Christopher Foo <[email protected]>

coprocessor: Fix html/pdf snapshot logic

67244905ff279cc6297d618df00f539d32205753 authored about 10 years ago by Christopher Foo <[email protected]>

Add missing imports. Change tuple return code style.

63743e267d4aafa8735c9d9a9a49a988dc356e9b authored about 10 years ago by Christopher Foo <[email protected]>

url.PercentEncoderMap: Fix missing super() call

de62cf4ad80196ae75ffac799b5ff95d5d0e1188 authored about 10 years ago by Christopher Foo <[email protected]>

warc: Fix 'separator' spelling.

2702abc2c758416105941aa12a0461a6a8fb80f7 authored about 10 years ago by Christopher Foo <[email protected]>

STASH

33698168df0a62e73aa238dca3f62cf264d0db73 authored about 10 years ago by Christopher Foo <[email protected]>

Implement username/password support.

Closes chfoo/wpull#30

f3c4cc57220b87efd9944b1451d34bc9218a7ab8 authored about 10 years ago by Christopher Foo <[email protected]>

url: Include username and password as part of URL

0734bec1fde3d535ae766720101d2fbb0a90ae28 authored about 10 years ago by Christopher Foo <[email protected]>

Split off PhantomJSCoprocessor

3b8a8ebc58d60155e3c19f3e84ba15ce01502b2a authored about 10 years ago by Christopher Foo <[email protected]>

driver.resource: Stub in start(), end(), error()

[ci skip]

dab13307eea8d9740f16b4f5f2295597e7692a56 authored about 10 years ago by Christopher Foo <[email protected]>

Merge branch 'develop' into topic/rewrite_phantomjs

140b949dd5d16f6393a459f59a89535e0bec8fb3 authored about 10 years ago by Christopher Foo <[email protected]>

Bump version to 0.1002

03100702f79fdb2d989b79d97bd6e73560a27e73 authored about 10 years ago by Christopher Foo <[email protected]>

Merge commit 'fcb98e675a9d8e26324de1027417abf66566218f'

a9145309b36b033475b4f111bb48524a7ba504ea authored about 10 years ago by Christopher Foo <[email protected]>

Add 'none' as choice to --progress

329e90f85aa2a9d3878d2bdcb5802a19f8d467a3 authored about 10 years ago by Christopher Foo <[email protected]>

recorder.progress: Add some spacing to output

58f4d5b129485fa3b401a9ba6f84a85be057e136 authored about 10 years ago by Christopher Foo <[email protected]>

Add --database-uri option.

Closes chfoo/wpull#152

8dc4818de1aac7db1afdda11583b14b032425b32 authored about 10 years ago by Christopher Foo <[email protected]>

version: Bump version to 0.1003a1

[ci skip]

824bcd86a4099626d5ce44113b14802558694361 authored about 10 years ago by Christopher Foo <[email protected]>

changelog: Update latest to 0.1002

[ci skip]

fcb98e675a9d8e26324de1027417abf66566218f authored about 10 years ago by Christopher Foo <[email protected]>

doc: Regenerate api files.

[ci skip]

f096704f20216c72f0b15c62e340281276c3afd5 authored about 10 years ago by Christopher Foo <[email protected]>

http.robots: Properly format the new robots URL's hostname

Closes chfoo/wpull#210

8c03bbc0c6ea47838eee318c2fb312838f52d539 authored about 10 years ago by Christopher Foo <[email protected]>

url: Flatten slashes in path, more specific scheme-rel urljoin.

Closes chfoo/wpull#205

7072d5895060be94421db5247a10c799425c40e0 authored about 10 years ago by Christopher Foo <[email protected]>

connection, http.proxy: Handle IPv6 4-tuple addresses

4a11a5d3815887137101379bb6f7d1f482c03538 authored about 10 years ago by Christopher Foo <[email protected]>

url: Normalize IPv6 address regardless for calls that later check IDNA.

Closes chfoo/wpull#207

1320c5cc531e53b796eea8949798fd98b167d832 authored about 10 years ago by Christopher Foo <[email protected]>

Update fusil runners

[ci skip]

1bb1d87507888e1665cf73d4c9f2cb806ceb85c3 authored about 10 years ago by Christopher Foo <[email protected]>

setup.py: Drop lxml as hard requirement

ab0f040f894e0fa2c3783045e111a93e795002f4 authored about 10 years ago by Christopher Foo <[email protected]>

Rename SSLVerficationError to SSLVerificationError

98e80faef66c9809f684d0754c59d6029c03f177 authored about 10 years ago by Christopher Foo <[email protected]>

Downgrade SSLVerficationError to ProtocolError if --no-check-certificate

Closes chfoo/wpull#209

df7d72cc54b2e10a37f153ad7a74a1f39115dce2 authored about 10 years ago by Christopher Foo <[email protected]>

app: Whitelist exceptions that are not crashes for crash message

7445cd116edcfa4d7d286768432b934139c955c6 authored about 10 years ago by Christopher Foo <[email protected]>

document.util: Add missing import

5437180895b875a28ee2a62819e8bf0736983348 authored about 10 years ago by Christopher Foo <[email protected]>

options: Use --html-parser html5lib as default

Closes chfoo/wpull#208

06fd37ddbb01204a220566859b9f5a265477396d authored about 10 years ago by Christopher Foo <[email protected]>

cookie: Handle cookies without "=".

Fix regression introduced in 38a4152. Closes chfoo/wpull#206

2b03864d6865d3cba5116f259ed6e56e12ee5d76 authored about 10 years ago by Christopher Foo <[email protected]>

Update setup.py packages

e8b350a2be0eec3b90c3973ab3ac738aeab78483 authored about 10 years ago by Christopher Foo <[email protected]>

coprocessor.phantomjs: Fix up merge.

43eb042426006bfcdc641fdcbb51ea1e6147ef35 authored about 10 years ago by Christopher Foo <[email protected]>

Merge commit '712265f831cc' into develop

Conflicts:
wpull/builder.py
wpull/driver/phantomjs_test.py
wpull/processor/web.py

b794eb34cbcc0906942db51d58f222e8b5ab1a4b authored about 10 years ago by Christopher Foo <[email protected]>

stream: Keep data and close connnection on content overrun.

5bdcb985c195a870216e57c93d3015b16e38ecb1 authored about 10 years ago by Christopher Foo <[email protected]>

Add --sitemaps URL as child instead of as root.

Closes chfoo/wpull#199

c8c18bf7c203bf346fe11b1c2c6eb955614d9381 authored about 10 years ago by Christopher Foo <[email protected]>

Moved web processor doc scraping concerns into ProcessingRule.

74b8cd189494735a9b83d20d20482eb35dbea289 authored about 10 years ago by Christopher Foo <[email protected]>

Scripting: Add handle_pre_response callback hook.

Closes chfoo/wpull#188

cc37c0a96b9897a3e75d160d69d00b1c9ecdcdf8 authored about 10 years ago by Christopher Foo <[email protected]>

connection: Fix ConnectionPool max_host_count arg not used.

f6de1a12cd8645717217cb04772d733559881a3d authored about 10 years ago by Christopher Foo <[email protected]>

travis.yml: Reintroduce Python 3.4

Re: chfoo/wpull#125

53c47868fb6d9d35a2139b2e82841c6b09ccb3d3 authored about 10 years ago by Christopher Foo <[email protected]>

app_test.TestAppFTP: Hush the logging output

9f648d25a9d585bb8f0ca1b86d68fdef9a932481 authored about 10 years ago by Christopher Foo <[email protected]>

phantomjs.py: Catch ESRCH on atexit handler

0f4d63bb91ee632f641e4c6ce04edaa94e7c4fba authored about 10 years ago by Christopher Foo <[email protected]>

Update app_test, ftp.ls.parse_test to match test FTP server

99120a5e3095bac534f542672f5cbace25591f66 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.client: Consider empty listings as no files.

eb53d69aaaa36e997fc9e708df14f9f769507eb5 authored about 10 years ago by Christopher Foo <[email protected]>

testing.ftp: Implement path routing logic.

65919674af15dabcbc5f3c010772b16e0a09fbe0 authored about 10 years ago by Christopher Foo <[email protected]>

doc: Update changelog & terse_options w/ experimental FTP support

Closes chfoo/wpull#23

60f3d2ac06318fe77dd325a3ffb2b44d3eed6a56 authored about 10 years ago by Christopher Foo <[email protected]>

Merge branch 'issue/23_ftp_2' into develop

60d65f68651207e70cfa240cffedb5bd4ac74741 authored about 10 years ago by Christopher Foo <[email protected]>

Merge branch 'develop' into issue/23_ftp_2

7856dabc1230e069048018eabee7ff84d488a24d authored about 10 years ago by Christopher Foo <[email protected]>

phantomjs_test: Ignore 'undefined' behavior in PhantomJS 1.9.8

Ignore PhantomJSRPCError: TypeError: 'undefined' is not a function
introduced in PhantomJS 1.9.8

01058a6c6630c000839a854eb04ca4f8403cfb81 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.client_test: Update test due to updated test FTP server

e725ff040bc3bea9e1e54fefe0f8ba7d53b021bf authored about 10 years ago by Christopher Foo <[email protected]>

recorder.warc: Backport textwrap.indent

bc1d44767956be8540fb472c06df7f383885eaae authored about 10 years ago by Christopher Foo <[email protected]>

Implement WARC logging data connection events

21d26fdb06ad42d2a4c9e42fdc0eb3db70ed85cf authored about 10 years ago by Christopher Foo <[email protected]>

ftp: Handle bad directory listing case.

f228c0a8763e109f6b3253de26421d5fdfdf9262 authored about 10 years ago by Christopher Foo <[email protected]>

ftp.client: Adjust FTP password to match Wget

7902f8dc137bf8318aa32e196ac8a13b99fbf233 authored about 10 years ago by Christopher Foo <[email protected]>