github.com/ArchiveTeam/ludios_wpull commits | Ecosyste.ms: OpenCollective

coprocessor.phantomjs: Better screenshotting logic.

Always fetch resources. Scroll to top when screenshot. Click corner of
page to get rid of any ov...

0d7df2333a97121516811a16aef6a9d63e1b2760 authored about 10 years ago

driver: Add send_key and send_click RPC calls.

d01663391b993cb53ad35495ec44f1841fb2f3a2 authored about 10 years ago

driver.resource: Catch and discard KeyErrors

08e26f4e5e448d1a32bcee3255b761e73d45dce8 authored about 10 years ago

coprocessor.phantomjs: Reimplement timeout and retry handling.

e3e7897d08fc1ec8d47d9156a5ec25db8dc2bfa4 authored about 10 years ago

driver.phantomjs: Implement timeout.

422ab330a5b35db05f6684bbe602d7884c24c6b5 authored about 10 years ago

ftp.client: Move async call outside exception handler

Closes chfoo/wpull#215

To see this problem quickly, put raise ValueError inside processor.ftp
_...

821afce940c1536460064c5892fa5785515bab02 authored about 10 years ago

setup.py: Update cx_freeze config for moved phantomjs.js file.

[ci skip]

bae851b749bcb754bb0b8cf6a551565f40804103 authored about 10 years ago

fpt.command: Reset connection before use

Closes chfoo/wpull#214

ae8a5ad548bb09d075c27aba5bd15ff31ca4f085 authored about 10 years ago

fixup! driver.phantomjs: Fix process reuse

9821b526f4fae514664b2ff71727d414ff082cad authored about 10 years ago

coprocessor.phantomjs: Fix up url rewriting hacks. Clean temp files.

516218034118d96f8c631c2a586f7a594620fa48 authored about 10 years ago

driver.phantomjs: Fix process reuse

1875ce2dba1bedd7b8eb1961ef31f43a312d9103 authored about 10 years ago

Fix up phantomjs events and resource tracker events

0f52e6f7a9258656640141861581a5ae5a429261 authored about 10 years ago

WIP coprocessor.phantomjs rewrite.

a8c3bdc30122c9f147be097f3aa50214cecc6cbc authored about 10 years ago

driver.process: Wrap try-except and log on error reading stdout/stderr

f72f8454a1697a640d06a895ee1f3748c2baf350 authored about 10 years ago

travis.yml: Blacklist topic/rewrite_phantomjs_2

8ef2b6205210c6084e0a33821201996a3f4e573e authored about 10 years ago

Merge branch 'develop' into topic/rewrite_phantomjs_2

Conflicts:
wpull/processor/web.py

f1926e2f7b44a3dc9ca2127da2a934b147fedd87 authored about 10 years ago

Bump version 0.1003

f67d12daa95dacfec59805836d7bf20bb480d482 authored about 10 years ago

Merge commit '8cc79c24b4900710310d34392603d9b0d536b55b'

Conflicts:
wpull/version.py

bc0bd3b1096e68f0df35d686c3d59a0b9bfc3e65 authored about 10 years ago

Bump version to 0.1004a1

86ae89fbe5f031dd228a7aa712e7ac3bf21e1d4f authored about 10 years ago

changlog: Update latest to 0.1003

[ci skip]

8cc79c24b4900710310d34392603d9b0d536b55b authored about 10 years ago

Fix missing response scripting hook callback during redirects.

fde26fae4d7f1b3e9709dfe320680ca712c85e4a authored about 10 years ago

fixup! ftp.ls: Better detection of msdos ls w/o dir

Fix unit test

59c9d8f574fcdabeef0b3a1b92129f5fef62239b authored about 10 years ago

stats: Add debug log

4016177170a68600b8a8e651ca8fd1ca0a0648c7 authored about 10 years ago

ftp.ls: Better detection of msdos ls w/o dir

c5507b4fa7d2799526be7a904c96f0ae23ad169c authored about 10 years ago

http.stream_test,scraper.html_test: Add pragma no cover

960651ea83def23b928f958736bbc45b8c37cf85 authored about 10 years ago

ftp: Catch exception when passive address cannot be parsed.

81bb66a91c40a2b58e17fff119fbf8121aa75878 authored about 10 years ago

driver: WIP add scroller module.

0d0d2f323626b121fb69f25c362a9bb592777d15 authored about 10 years ago

driver: Clean up a bit and add get_page_url(), resource_timeout

8536eba08fb670dc717fb84b8cb787e12a18ed3f authored about 10 years ago

driver: WIP Add resource unit tests

6ae7472b0ba2193ab97785eed3881fd4d4901298 authored about 10 years ago

url: Use original string instead of lowercased on schemeless URL

Closes chfoo/wpull#213

05458bc7b0ca058a3bbd7fbaaac882cbc38f9ccc authored about 10 years ago

driver: Fix up phantomjs pool unit tests

2c93b361f277d6bb0ab6d0fc55808750a391f713 authored about 10 years ago

ftp.ls.date.parse_datetime: Fix 12 AM/PM to 24hr logic.

1e2b4bb32a75320b94d88bc6a7099a0ff731ab5b authored about 10 years ago

ftp.client: Add some debug logging

7a919d85299b391be8db48c9133f1e78c3d18ff5 authored about 10 years ago

ftp.command: Accept FTP code 125 on data connection

80c7c8275010cccdc41206760cf52510b13582df authored about 10 years ago

Rewrite ftp.client interface to align with http.client interface

ftp.client: fetch() pairs with read_contnet(), fetch_file_listing()
pairs with read_listing_cont...

94d4c1f4a92359a3486b27f9e98490352a612704 authored about 10 years ago

ftp.command: Split read_stream into begin_stream & read_stream

e6f4960a1ab5c92295c8fce75228e6fc686c506e authored about 10 years ago

http.client: Better docstring.

4d27da0e87fe788301f437049f21ee5a260cc875 authored about 10 years ago

ftp.client: Remove outdated TODO

800ce05974a981ac2ad038c8026535ac28c01dca authored about 10 years ago

abstract.client,ftp.util: Add some docstrings.

1378ebb956406c1af388cabf6a54012420176911 authored about 10 years ago

http.robots: Add missing import.

ed07f7d8ec74b463546a7689ca61e0d1c0eb3949 authored about 10 years ago

driver: WIP Rewrite PhantomJSClient into PhantomJSPool

e830c768e86d6393c1ec893601055be576e710c2 authored about 10 years ago

WIP Rewrite PhantomJSRemote to PhantomJSDriver

de30c9b89daae74c5ade82824f1264786232b657 authored about 10 years ago

WIP Rewrite phantomjs script with Haxe version with RPC polling

c604276e31935b33719301b98af79a8beaa2b509 authored about 10 years ago

coprocessor: Fix html/pdf snapshot logic

67244905ff279cc6297d618df00f539d32205753 authored about 10 years ago

Add missing imports. Change tuple return code style.

63743e267d4aafa8735c9d9a9a49a988dc356e9b authored about 10 years ago

url.PercentEncoderMap: Fix missing super() call

de62cf4ad80196ae75ffac799b5ff95d5d0e1188 authored about 10 years ago

warc: Fix 'separator' spelling.

2702abc2c758416105941aa12a0461a6a8fb80f7 authored about 10 years ago

STASH

33698168df0a62e73aa238dca3f62cf264d0db73 authored about 10 years ago

Implement username/password support.

Closes chfoo/wpull#30

f3c4cc57220b87efd9944b1451d34bc9218a7ab8 authored about 10 years ago

url: Include username and password as part of URL

0734bec1fde3d535ae766720101d2fbb0a90ae28 authored about 10 years ago

Split off PhantomJSCoprocessor

3b8a8ebc58d60155e3c19f3e84ba15ce01502b2a authored about 10 years ago

driver.resource: Stub in start(), end(), error()

[ci skip]

dab13307eea8d9740f16b4f5f2295597e7692a56 authored about 10 years ago

Merge branch 'develop' into topic/rewrite_phantomjs

140b949dd5d16f6393a459f59a89535e0bec8fb3 authored about 10 years ago

Bump version to 0.1002

03100702f79fdb2d989b79d97bd6e73560a27e73 authored about 10 years ago

Merge commit 'fcb98e675a9d8e26324de1027417abf66566218f'

a9145309b36b033475b4f111bb48524a7ba504ea authored about 10 years ago

Add 'none' as choice to --progress

329e90f85aa2a9d3878d2bdcb5802a19f8d467a3 authored about 10 years ago

recorder.progress: Add some spacing to output

58f4d5b129485fa3b401a9ba6f84a85be057e136 authored about 10 years ago

Add --database-uri option.

Closes chfoo/wpull#152

8dc4818de1aac7db1afdda11583b14b032425b32 authored about 10 years ago

version: Bump version to 0.1003a1

[ci skip]

824bcd86a4099626d5ce44113b14802558694361 authored about 10 years ago

changelog: Update latest to 0.1002

[ci skip]

fcb98e675a9d8e26324de1027417abf66566218f authored about 10 years ago

doc: Regenerate api files.

[ci skip]

f096704f20216c72f0b15c62e340281276c3afd5 authored about 10 years ago

http.robots: Properly format the new robots URL's hostname

Closes chfoo/wpull#210

8c03bbc0c6ea47838eee318c2fb312838f52d539 authored about 10 years ago

url: Flatten slashes in path, more specific scheme-rel urljoin.

Closes chfoo/wpull#205

7072d5895060be94421db5247a10c799425c40e0 authored about 10 years ago

connection, http.proxy: Handle IPv6 4-tuple addresses

4a11a5d3815887137101379bb6f7d1f482c03538 authored about 10 years ago

url: Normalize IPv6 address regardless for calls that later check IDNA.

Closes chfoo/wpull#207

1320c5cc531e53b796eea8949798fd98b167d832 authored about 10 years ago

Update fusil runners

[ci skip]

1bb1d87507888e1665cf73d4c9f2cb806ceb85c3 authored about 10 years ago

setup.py: Drop lxml as hard requirement

ab0f040f894e0fa2c3783045e111a93e795002f4 authored about 10 years ago

Rename SSLVerficationError to SSLVerificationError

98e80faef66c9809f684d0754c59d6029c03f177 authored about 10 years ago

Downgrade SSLVerficationError to ProtocolError if --no-check-certificate

Closes chfoo/wpull#209

df7d72cc54b2e10a37f153ad7a74a1f39115dce2 authored about 10 years ago

app: Whitelist exceptions that are not crashes for crash message

7445cd116edcfa4d7d286768432b934139c955c6 authored about 10 years ago

document.util: Add missing import

5437180895b875a28ee2a62819e8bf0736983348 authored about 10 years ago

options: Use --html-parser html5lib as default

Closes chfoo/wpull#208

06fd37ddbb01204a220566859b9f5a265477396d authored about 10 years ago

cookie: Handle cookies without "=".

Fix regression introduced in 38a4152. Closes chfoo/wpull#206

2b03864d6865d3cba5116f259ed6e56e12ee5d76 authored about 10 years ago

Update setup.py packages

e8b350a2be0eec3b90c3973ab3ac738aeab78483 authored about 10 years ago

coprocessor.phantomjs: Fix up merge.

43eb042426006bfcdc641fdcbb51ea1e6147ef35 authored about 10 years ago

Merge commit '712265f831cc' into develop

Conflicts:
wpull/builder.py
wpull/driver/phantomjs_test.py
wpull/processor/web.py

b794eb34cbcc0906942db51d58f222e8b5ab1a4b authored about 10 years ago

stream: Keep data and close connnection on content overrun.

5bdcb985c195a870216e57c93d3015b16e38ecb1 authored about 10 years ago

Add --sitemaps URL as child instead of as root.

Closes chfoo/wpull#199

c8c18bf7c203bf346fe11b1c2c6eb955614d9381 authored about 10 years ago

Moved web processor doc scraping concerns into ProcessingRule.

74b8cd189494735a9b83d20d20482eb35dbea289 authored about 10 years ago

Scripting: Add handle_pre_response callback hook.

Closes chfoo/wpull#188

cc37c0a96b9897a3e75d160d69d00b1c9ecdcdf8 authored about 10 years ago

connection: Fix ConnectionPool max_host_count arg not used.

f6de1a12cd8645717217cb04772d733559881a3d authored about 10 years ago

travis.yml: Reintroduce Python 3.4

Re: chfoo/wpull#125

53c47868fb6d9d35a2139b2e82841c6b09ccb3d3 authored about 10 years ago

app_test.TestAppFTP: Hush the logging output

9f648d25a9d585bb8f0ca1b86d68fdef9a932481 authored about 10 years ago

phantomjs.py: Catch ESRCH on atexit handler

0f4d63bb91ee632f641e4c6ce04edaa94e7c4fba authored about 10 years ago

Update app_test, ftp.ls.parse_test to match test FTP server

99120a5e3095bac534f542672f5cbace25591f66 authored about 10 years ago

ftp.client: Consider empty listings as no files.

eb53d69aaaa36e997fc9e708df14f9f769507eb5 authored about 10 years ago

testing.ftp: Implement path routing logic.

65919674af15dabcbc5f3c010772b16e0a09fbe0 authored about 10 years ago

doc: Update changelog & terse_options w/ experimental FTP support

Closes chfoo/wpull#23

60f3d2ac06318fe77dd325a3ffb2b44d3eed6a56 authored about 10 years ago

Merge branch 'issue/23_ftp_2' into develop

60d65f68651207e70cfa240cffedb5bd4ac74741 authored about 10 years ago

Merge branch 'develop' into issue/23_ftp_2

7856dabc1230e069048018eabee7ff84d488a24d authored about 10 years ago

phantomjs_test: Ignore 'undefined' behavior in PhantomJS 1.9.8

Ignore PhantomJSRPCError: TypeError: 'undefined' is not a function
introduced in PhantomJS 1.9.8

01058a6c6630c000839a854eb04ca4f8403cfb81 authored about 10 years ago

ftp.client_test: Update test due to updated test FTP server

e725ff040bc3bea9e1e54fefe0f8ba7d53b021bf authored about 10 years ago

recorder.warc: Backport textwrap.indent

bc1d44767956be8540fb472c06df7f383885eaae authored about 10 years ago

Implement WARC logging data connection events

21d26fdb06ad42d2a4c9e42fdc0eb3db70ed85cf authored about 10 years ago

ftp: Handle bad directory listing case.

f228c0a8763e109f6b3253de26421d5fdfdf9262 authored about 10 years ago

ftp.client: Adjust FTP password to match Wget

7902f8dc137bf8318aa32e196ac8a13b99fbf233 authored about 10 years ago

Implement --follow-ftp option.

70492a75e1c1a5080c312fa46a29bf83d35850cf authored about 10 years ago

urlfilter.FollowFTPFilter: Handle case where referrer is None

7580b858304445df6917f91b0b46c5aa88821369 authored about 10 years ago

recorder.warc: Append newline to log as needed.

df6b09864cb5f83a0de40667a372cab440f51b52 authored about 10 years ago

ftp.client: Implement recorder on data stream read

206cbfcf82220acc376375c2772832f9b320e0a8 authored about 10 years ago