github.com/ArchiveTeam/ludios_wpull commits | Ecosyste.ms: OpenCollective

driver: Track pending resources and scrollHeight

9ee5c6d2b0e499633dc3568712aba48b8eee6ed5 authored about 10 years ago

proxy: Give up starttls only after a minute

885fb0c85afa910a2240b760d2d9e68e9d94f1bd authored about 10 years ago

Revert "driver: Wait before rendering snapshots"

This reverts commit d547837b88dc0c1b5e585dc0ae5f339dd418270e.

0ee6b1989ac21dc5a42c09828c6a937966ba05ab authored about 10 years ago

driver: Wait before rendering snapshots

d547837b88dc0c1b5e585dc0ae5f339dd418270e authored about 10 years ago

driver: Check for stderr for slimerjs compatibility

re: chfoo/wpull#229

5e459ba539c56203c0ac85f68d3b3271e0d37c8d authored about 10 years ago

phantomjs test: Increase wait time to 0.4

For slow Travis CI environment

5fe9337fbca2810e7490527ad70edba673cdbe3b authored about 10 years ago

phantomjs test: Increase wait time to 0.2

For slow Travis CI environment

bcc03233813cda9bdf6f1af071a3e445f22c7028 authored about 10 years ago

coprocessor: Fix up phantomjs args.

Closes chfoo/wpull#230

270b2435b6c6bdbdd30dfb0bdb26d91f62cf1bfb authored about 10 years ago

driver: Add global error handler. Fix cast error.

45b04d0e95c8b4711eec8ec10a637204e0cb0a72 authored about 10 years ago

driver: Rewrite PhantomJS into use-once tool. Remove any RPC logic.

re: chfoo/wpull#230

[ci skip]

85212e60099f661fb21d532ca84c57c5cf258430 authored about 10 years ago

Move cookie jar concerns into ProxyCoprocessor

52ef4e87c972792503afbe06fba2fc1771b0ee69 authored about 10 years ago

proxy: Refactor into Session class. Return 501 before rejecting.

1453441b767c803326c1a3d8ec4fbf0d207aba78 authored about 10 years ago

Implement --limit-rate.

Closes chfoo/wpull#43

685ff2471068cb86c8c5a8b2ac7c5c15214749d5 authored about 10 years ago

bandwidth: Fix bytes_transfered spelling

06a923bcccde9069a8d6b9d86dd0e6d8b7225aab authored about 10 years ago

dns: Fix up -4 and -6 PythonResolver fallback cases

68fc5da393873c8ec56ed615989b119169dd119b authored about 10 years ago

Rename dns-resolver option to no-skip-getaddrinfo

e01117eaed39b171ab5b6f58194bb81190ed98d8 authored about 10 years ago

dns: Undo Resolver abstraction. PythonResolver falls back C resolver

75074090c25a89ab53a415ac49dad3f222f231cf authored about 10 years ago

dns: Raise approprate exceptions at lower level. Fix Py3.2 compatibility.

33edef78ab94dd53ff246752b65979f5e14d1b67 authored about 10 years ago

Add dns.PythonResolver and --dns-resolver option.

requirements: Add dnspython3
dns: Abstract Resolver and add PythonResolver
builder,options: Defa...

0fad75040e7caf63480204c342378641a077e152 authored about 10 years ago

Add WIP link_type expected content type checking tests

RE: chfoo/wpull#150

49f55b04c3b463ecd23818a9685505dc674e9d85 authored about 10 years ago

scraper: Add link_type arg to scrape for checking expected content type

6be95b410908f3aba27d18ba282cd65c12f6395b authored about 10 years ago

processor.rule: Let script hooks scrape the document unconditionally

973bcbb7f9971272c1d2dc27f4acc09ec3f2afd9 authored about 10 years ago

item: Add add_child_url to replace add_{inline,linked}_url_infos

6e41bed09c28e71917510351ff2ad875f476f572 authored about 10 years ago

database: Change add_many to use dict argument

9bb239bb3027875ea2bed1774827bec2d683448f authored about 10 years ago

scraper: Add LinkContext, ScrapeResult and rewrite to return them.

94cc734fdd2841bc858c2ac7ba02f2ecd8a8609e authored about 10 years ago

document,scraper: Provide more link context for CSS

327c7915a2240b3c4c85069cc91f6289c352c3a5 authored about 10 years ago

Add --strip-session-id option.

Closes chfoo/wpull#203

f3b244f8858197df52213ca7f728c47736d5ba7c authored about 10 years ago

readme,doc/usage.rst: Update archive example.

[ci skip]

9b807ab7694817b8c6a0d27b8f5446fea7a6e7b6 authored about 10 years ago

Add urlrewrite.URLRewriter and --escaped-fragment option.

Closes chfoo/wpull#34

b397bddfc5466fea6a15b655fa13a6ef56947573 authored about 10 years ago

recorder.progress,writer: Support FTP continued download

Re: chfoo/wpull#222

6ce43806876e7dcd95f4b7f1a09c6798509670c7 authored about 10 years ago

ftp: Support requesting REST

Re: chfoo/wpull#222

09f454f4c103d4d8ebc1dfb7e2e4bf418b8396ec authored about 10 years ago

Bump version 0.1005

7c9cc5e03570756d7e5c4f80dfaba60cbe99b9e4 authored about 10 years ago

Merge commit '490fc555179a056ddef961dadc19acb60b698a5c'

91d93adf398d531ba98bb4790ffa3866cb3c959d authored about 10 years ago

Bump version 0.1006a1

[ci skip]

2e10639b388bba15aa2542c8d1a5aaa5f263b52b authored about 10 years ago

changelog: Update latest to 0.1005

[ci skip]

490fc555179a056ddef961dadc19acb60b698a5c authored about 10 years ago

driver.phantomjs: Don't listen to page.onerror callback

74eeb1d0db7193bdb253683b2550a5121509d3a5 authored about 10 years ago

http.request,writer: Move the Range field setting into Request

5fea7ca8e19e44ae5ae7e2d7d441265a8f438c15 authored about 10 years ago

recorder.progress: Handle FTP transfer size

re: chfoo/wpull#222

762b16d9d7c4e24839c7543f618579c2a294f920 authored about 10 years ago

ftp: Use SIZE command and move pre_response after it

re: chfoo/wpull#222

354ca122d15e4aa20eece1bce153b864ccab3579 authored about 10 years ago

database.sqlmodel: Add column doc.

c2d4692731955fd40a2881fd70412e4c7c5aac6f authored about 10 years ago

item.LinkType: Expand additional types for future use.

07fd39570d7fe89c1df589cfc051db42bdc4fda4 authored about 10 years ago

builder: Refactor proxy server creation out of phantomjs function.

fdb79ee5b83fecf3c415d50291d9c39a42d99f6c authored about 10 years ago

driver.process: Abstract RPCProcess into Process/RPCProcess

bbee84329c9520d50459fe03824d4ab152b30b11 authored about 10 years ago

builder: Add console log filter to suppress user-irrelevant errors

Closes chfoo/wpull#178
Closes chfoo/wpull#164
Closes chfoo/wpull#163

abd44340cba0787aa99e7e05b008bb2ca44ac1e6 authored about 10 years ago

builder,options: Fix --no-verbose to align with Wget. Add --very-quiet.

86ddeb92bd4d19e7cf89a1138f353658308f1b2f authored about 10 years ago

driver/PhantomJS.hx: Add doc comments. Fix inconsistent style.

b3b4b9fa6723d28b625404aafc5358c63569ed0d authored about 10 years ago

item, urlfilter: Fix inline logic

item: Fix inline stuck on 1.
urlfilter: Fix inline case during finite level.
Add more unit tests...

bd1130b95048b0f94fe03bd16b3dad7d648bf52d authored about 10 years ago

scraper.util: Consider js files inline. Reject paths starting with domain.

1644dbc2044fb4deb47ba8fbbdb2deb7c1f121cc authored about 10 years ago

Bump copyright year to 2015

[ci skip]

c21dc3ed61239d8a80295fdd9c1369f6cd5e635b authored about 10 years ago

recorder/warc_test: Validate WARCs with Warcat.

a65df061c8fbcf189ac3902fa0168d70e570a75b authored about 10 years ago

Disable SSLv2/3 in auto protocol. Add --no-strong-crypto.

Closes chfoo/wpull#198

71d26aaea03bf264aa62dd2ff85217dad243928a authored about 10 years ago

driver.phantomjs_test: Wait for load before testing snapshot

4f91e129c4ae2616a41631067975cb0f35c6c92c authored about 10 years ago

coprocessor.phantomjs: Time out waiting load on separate value not wait-time

12d02e48ab38acc037afc330f144d3ef510c1449 authored about 10 years ago

proxy: Use exceptions compatible with Python 3.2

4d80214da2c4f2056c4db436ff75e950730f8445 authored about 10 years ago

driver.resource: Suppress logging resource tracking errors.

98f8d442cffcc4aac7c283d468537806dec57723 authored about 10 years ago

app_test: Fix unit tests to channel phantomjs through proxy.

ce2e48db754ac2ac021a7012873d145d06a240c5 authored about 10 years ago

Merge branch 'hotfix/0.1004.2' into develop

Conflicts:
doc/changelog.rst

43ed96a656d679c852a2081585661b567f7029df authored about 10 years ago

Merge branch 'hotfix/0.1004.2'

e764cb922c12e15bfef52fa67ebd6eaf4251ea1c authored about 10 years ago

changelog: Update latest. Bump version to 0.1004.2

a55c60628afd6a9927a6111edc73f971539b8f07 authored about 10 years ago

proxy: Fix AttributeError on certificate filename

b83a1e4eefe5c740baf54c26da3dc6c9f558dc1e authored about 10 years ago

Add --page-requisites-level

Closes chfoo/wpull#151

f6fa0bf8d631c450c8462d43b0b1ea35d29d47a8 authored about 10 years ago

Change URL inline column and LevelFilter to use Integer type.

urlfilter.LevelFilter: Add inline_max_depth arg

4b8d0de83d01c33bb1f384d077e13362108f474e authored about 10 years ago

urlfilter.LevelFilter: Match pseudo-level inline behavior in wget.

Re: chfoo/wpull#151

10dfe56b36d683c742c7dff8c05304a7b4508608 authored about 10 years ago

Bump version 0.1004.1

4292f79ca9035897cb8bd5e23dd9973c327303bd authored about 10 years ago

Merge branch 'develop'

Conflicts:
wpull/version.py

f4f8f080178b11b1fc7441c88f2215edf28f4284 authored about 10 years ago

changelog: Update latest to 0.1004.1

[ci skip]

15fc881433726000eb67be36fe5db24ce33c0170 authored about 10 years ago

ftp.ls.date_test: Fix new year before feb test.

56a46cc4aa5c7cec098a261873825725b50d1540 authored about 10 years ago

app_test: Run phantomjs tests under https too.

7b76c42174b6026ebac07c917021db7572ef4260 authored about 10 years ago

setup.py: Add missing MITM proxy SSL certs. proxy: Add filename asserts.

Closes chfoo/wpull#224

e347d20ff0fcdabcaf686de3b62dc56cd7ab1982 authored about 10 years ago

url: Add missing docstrings.

123f384f271df1898ba463440f28274c1a40ca28 authored about 10 years ago

url: Lower case the hostnames

Closes chfoo/wpull#223

125847188b75caad154604e99d5e69db514b2a86 authored about 10 years ago

cookie: Handle cookiejar TypeError

Closes chfoo/wpull#220

ba9d3fc64543305bda1d2a1b7db7329344194620 authored about 10 years ago

Bump version 0.1004

7b07c0ce5073b9b7c52d797c2e50880216e7e47c authored about 10 years ago

Merge commit '51106'

Conflicts:
wpull/version.py

5eac519f0247ad09bd1eaf4b7ffba65e79e9b57c authored about 10 years ago

Bump version 0.1005a1

[ci skip]

d11aa43cf73115d797faea3ac33576236c30fd55 authored about 10 years ago

changelog: Update latest to 0.1004

[ci skip]

51106d368f43be061c9155f43f41852068c3e4ee authored about 10 years ago

coprocessor.phantomjs: Comment out should_fetch code again.

e03c0a88306d007f04011d8912129a4c45dfd2c4 authored about 10 years ago

util: Rewrap certificate contents for better deduplication.

Fixes "cert already in hash table" raised from ssl.

503873ad2428e2c61d813b2d223f597be5a55b97 authored about 10 years ago

proxy: Handle response is None

828a3c98e2a2ab75b46edc0b552e70c217c0f4ba authored about 10 years ago

Stub in code needed for PhantomJS resource & database item interaction.

aa242d4b2bd313a42702d70210555c02a9fe6bb0 authored about 10 years ago

Implement --trust-server-names

Closes chfoo/wpull#35

4977aec2606d5d5447e433ff28a8d42aadf49760 authored about 10 years ago

Implement --content-disposition.

RE: chfoo/wpull#35

3f25ff2f71a874a2cfd5c52f04692e7177f60fed authored about 10 years ago

Implement --adjust-extension.

RE: chfoo/wpull#35

ea0f25f857c77930ac83f0f4ddc258aa67f83007 authored about 10 years ago

coprocessor.phantomjs: Check if page is dynamic before scrolling

0b6c95daecdc2d5dad4d3e9a26283f8f7d239949 authored about 10 years ago

driver.phantomjs: Add is_page_dynamic

12d442ee4862fdc274f29e6b1128d2218aa12a03 authored about 10 years ago

doc: Add WARC specifications

Closes chfoo/wpull#217

[ci skip]

1a2ad846227a4f31d70d62da3f383de9ebe7b110 authored about 10 years ago

Add some docstrings

[ci skip]

ae54abae865d51c46fe9691b8cf8371bd254da30 authored about 10 years ago

doc: Add sphinx-argparse for automatic options generation.

7527103b75547e03703253a0cf1fd11af74ad88f authored about 10 years ago

proxy: Movie cookie jar concerns out using callback functions

0c09e59ff84fa403742faeedac1828195f5cd522 authored about 10 years ago

proxy: Support SSL tunnel reuse.

76866ddbc9b850f66df2350fec113a9606804936 authored about 10 years ago

proxy: Fix syntax for Py 3.2

f1478aedd26e850fa8ee24e18770d2bda4f2bd8e authored about 10 years ago

changelog: Note phantomjs improvement.

e0006193aa188b940ad8b6ef8d5659b1fd23055d authored about 10 years ago

Merge branch 'topic/rewrite_phantomjs_2' into develop

6163f6a9e8a47fafc34d0ab36ba09cca007591df authored about 10 years ago

processor.web: Add status 204 as (empty) document ok.

7b81b5669eaa254923585d2413967ca72b63ca09 authored about 10 years ago

ftp.ls.date: Adjust IS8601 regex pattern to stop eager match of digits

Closes chfoo/wpull#216

cced19cf5cf66bd0cc0f9304bab455dd34673d7c authored about 10 years ago

ftp.ls.listing: Handle UnboundLocalError on start_index

re: chfoo/wpull#216

cd3a94ce6d6c8583633c8ba53ab6bf1de84162dd authored about 10 years ago

coprocessor.phantomjs: Time out wait_load()

dcdd53d03886e01790dbe4021679f20d4051cf9a authored about 10 years ago

coprocessor.phantomjs: Remove HTTPS rewrite hack

f06ebfd57a3cc38d3386bbf2a6f5db9df78e638e authored about 10 years ago

proxy: Support MITM SSL proxying.

890e392b885acfa129c3c850978b8c1c79aab488 authored about 10 years ago

proxy: Add another rewrite case for referrer

f6c78ed5909082394ec9efbaea514207aa620b8a authored about 10 years ago