github.com/ArchiveTeam/wpull commits | Ecosyste.ms: OpenCollective

stream_test.py: Log less on repeated calls.

b7f7dd956b429796b752487b032bad97cd0e3add authored over 10 years ago by Christopher Foo <[email protected]>

app_test.TestAppHTTPS: Fix wrong class.

3308a7df0c3bf179c4a427a44f558bf8e6c0ee46 authored over 10 years ago by Christopher Foo <[email protected]>

http.web: Check for None before adding cookies.

0d9eb421303e1912e1911f58d295e01bd975789f authored over 10 years ago by Christopher Foo <[email protected]>

app_test.py: Fix TestAppHTTPS with asyncio support.

c870960533dd4d95d8a09cf3a997e9a481901606 authored over 10 years ago by Christopher Foo <[email protected]>

builder.py: Put back ignore-length and no-http-keep-alive opts.

114932a55d494290d67845f13b2267c280c9ce71 authored over 10 years ago by Christopher Foo <[email protected]>

app_test.py: Fix up erroring code.

24c271b3333808eb314a86e378d6311526a75c3c authored over 10 years ago by Christopher Foo <[email protected]>

engine.py: Count items out using the queue.

f18283f62047ecd54aadfdaafb75f46058c6fe12 authored over 10 years ago by Christopher Foo <[email protected]>

connection.py: Catch SSL verfication error by checking error msg str.

3ac9718ff3dcc58ef0e12e68b1d417414ddba2bc authored over 10 years ago by Christopher Foo <[email protected]>

Continously force feed phantomjs stdin data to not block.

bf76b469d64ca56f132f1056d94862a505c9fabe authored over 10 years ago by Christopher Foo <[email protected]>

Fix up attributes and calls. Add back in robots.txt handling.

11519fc0f9d0cd760ed8a67193760fc72d4dbf31 authored over 10 years ago by Christopher Foo <[email protected]>

Fix up proxy and unit tests.

92e427a2484ab87ba4172484ee8a7c1717555bb6 authored over 10 years ago by Christopher Foo <[email protected]>

Remove obsolete actor, iostream, async modules.

186ca925d92bc5a807187958b9083c9c1d6a772b authored over 10 years ago by Christopher Foo <[email protected]>

phantomjs: Use stdin/stdout for communication instead of WebSocket.

38f435f2858d0449b163b400979965a60ef748cc authored over 10 years ago by Christopher Foo <[email protected]>

Misc attribute fix ups.

6075685aacd891e21e484df907030c32e5295bed authored over 10 years ago by Christopher Foo <[email protected]>

document_test.py: Fix up Response args

21416c6079910b28b87974778f409fc30662efb1 authored over 10 years ago by Christopher Foo <[email protected]>

Merge branch 'develop' into topic/asyncio

Conflicts:
wpull/http/client.py
wpull/http/connection.py
wpull/phantomjs.py

4ccc25ab369fabcc0062fe6efc4a890f7ce447aa authored over 10 years ago by Christopher Foo <[email protected]>

Rewrite proxy.py. http: Allow raw read of chunked content.

fc0b4056daaeb98aa788dc127f0c81338c55f4c6 authored over 10 years ago by Christopher Foo <[email protected]>

Fix up attributes.

14bb875a4799c03ebbb369f723d2e3dc331c2c2b authored over 10 years ago by Christopher Foo <[email protected]>

WIP fix up WebProcessor

50b9d105fbe0b60e200872b12c8f54e567365b63 authored over 10 years ago by Christopher Foo <[email protected]>

Split processor module into package.

958828196a61a058d3f2bd2025ce87121270f55b authored over 10 years ago by Christopher Foo <[email protected]>

Fix up engine.

f23917bdaf21ee7306dc858e08f7bba40a2af688 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up builder.

c61e6103ad8c9c2458c0a624909d01eced02e979 authored over 10 years ago by Christopher Foo <[email protected]>

travis.yml: Blacklist topic/asyncio branch.

So don't have to write ci skip so many times

6d2fbf020690fabc1063035ebb89ae82afa79c54 authored over 10 years ago by Christopher Foo <[email protected]>

Fix up recorder_test.py and related

[ci skip]

64dda8a156fd4a4c5877fbd8a99290fb9d3b9284 authored over 10 years ago by Christopher Foo <[email protected]>

Fix up wrapper_test

[ci skip]

ab1f8c09f123d3bb1d679931ee05f51b17af9926 authored over 10 years ago by Christopher Foo <[email protected]>

Fix up scraper

[ci skip]

6eefc8b4b58d5b62b29a6668951f719309b99516 authored over 10 years ago by Christopher Foo <[email protected]>

http.request: Split Request into Request and RawRequest

[ci skip]

0eceb812c5ae267eeb22e32f992d5feb61c58739 authored over 10 years ago by Christopher Foo <[email protected]>

Bump version to 0.36.4

b20cbc95e77aed1494906ad0b5dc668f7f48ebb4 authored over 10 years ago by Christopher Foo <[email protected]>

Merge branch 'develop'

af7e2b3deb2a03cab38973749efdd3d4f524baa6 authored over 10 years ago by Christopher Foo <[email protected]>

changelog: Update latest to 0.36.4

[ci skip]

90201850984ab166d4ea4b21c10aa835f9bfc458 authored over 10 years ago by Christopher Foo <[email protected]>

changelog: Add entry: Tornado 4.0 support.

Closes chfoo/wpull#142

189943e469604575cb31803affa19c1e6b65640f authored over 10 years ago by Christopher Foo <[email protected]>

Merge branch 'issue/142-tornado4' into develop

2fffc154316e2b4dc1b13bafd0a63c87ef3995db authored over 10 years ago by Christopher Foo <[email protected]>

backport.conf: Add websocket.py to nofix=str

7494f6d948628157eb921b331e2ce454b9a66c46 authored over 10 years ago by Christopher Foo <[email protected]>

app_test.py: Use coroutine version of fetch

Avoid tornado getting confused and stopping/starting the ioloop
incorrectly

017db5932d8285ea4e5aa4a0fe5f2923f43b179c authored over 10 years ago by Christopher Foo <[email protected]>

phantomjs: Don't call set_exit_callback

Avoid signal/ioloop/atexit/AttributeError problems.

b7dae043d2ae338122d535f0b113c88944bff7fc authored over 10 years ago by Christopher Foo <[email protected]>

requirements.txt: Set tornado>=3.1.1,<5.0

6d1be19c7b9b47ae7bc7a2c540844406a64da708 authored over 10 years ago by Christopher Foo <[email protected]>

backport.conf: Update for .thirdparty.tornado

1d22ed7b7499bd579506ac1f2a65315e8e4cb4b0 authored over 10 years ago by Christopher Foo <[email protected]>

phnatomjs: Copy and use websocket.py

re: chfoo/wpull#142

websocket.py is from
tornadoweb/tornado@1c36307463b1e8affae100bf9386948e6c1...

65c04ff679de0150f06aeddb958107e325dc7829 authored over 10 years ago by Christopher Foo <[email protected]>

http.connection: Don't early log all exceptions.

Let handlers log exceptions at the expense of missing original traceback
info.

0e1f46944fc88f5ef407af5ffa9edc767e4c4cb0 authored over 10 years ago by Christopher Foo <[email protected]>

async: Ensure future is consumed after timeout.

455a28b0f090acc6604536164a884839a13b438e authored over 10 years ago by Christopher Foo <[email protected]>

.http.connection: Don't leak tracebacks across exception contexts

Python 3 seems to want to store the traceback on the exception object.

Side effect is that exce...

243ba0d52b98e7df79246f77f5136f1abdc1da10 authored over 10 years ago by Christopher Foo <[email protected]>

decompression: Check for gzip using only first byte

Closes chfoo/wpull#155

4e2d5fb8885b3c36106d6ee7d712d241a1a26d2c authored over 10 years ago by Christopher Foo <[email protected]>

decompression: Copy and mod tornado.util.GzipDecompressor

[ci skip]

8de02bd469e2d27e50473f61649cb3769a96b726 authored over 10 years ago by Christopher Foo <[email protected]>

http.connection: Don't use bare except.

b24510f729be1a859c2a7e00a9a793b1a7630a4a authored over 10 years ago by Christopher Foo <[email protected]>

dns.py: Add missing .async import.

22de80e89fd4d18b58c02e09c33c54fc3a71c3d5 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Rewrite RobotsTxtRichClientSession into new .http.robots module.

[ci skip]

bde05c02d6bb64d5e19a60d7e17fff3680c57eec authored over 10 years ago by Christopher Foo <[email protected]>

WIP http.web asyncio support

[ci skip]

ca78db76248bbe2402e13c317fbb147807b029ac authored over 10 years ago by Christopher Foo <[email protected]>

testing: Support goodapp with asyncio/trollius

[ci skip]

28b93872ed0e5bc365bc4b0ef1c8c6b25027b19e authored over 10 years ago by Christopher Foo <[email protected]>

http.request: Add copy method

[ci skip]

912b15ce30af0ce63413ba600a7d9d6af4967a87 authored over 10 years ago by Christopher Foo <[email protected]>

http.redirect: Fix up response attribute access

[ci skip]

0f15c325f7f0ee75a4ef3b9d7f2be50c5438a8aa authored over 10 years ago by Christopher Foo <[email protected]>

cookie: Discard non-ASCII cookie values

Closes chfoo/wpull#145

85d7af952bc379b54d24e89c73a6a1941f50e1c5 authored over 10 years ago by Christopher Foo <[email protected]>

cookie: Rename CookieLimitsPolicy to DeFactoCookiePolicy

25fa23663c919843980d01b41e0963675f03d2ff authored over 10 years ago by Christopher Foo <[email protected]>

Bump version to 0.36.3

0c79110bc1e948178beda1475778f9b7462bff67 authored over 10 years ago by Christopher Foo <[email protected]>

Merge branch 'develop'

e64554f4655f08135693cab8903b641b4d23c2a8 authored over 10 years ago by Christopher Foo <[email protected]>

changelog: Update latest to 0.36.3.

[ci skip]

7a20f9e7aec6f14497e3687ede534974a9e750b1 authored over 10 years ago by Christopher Foo <[email protected]>

database: Use synchronous=NORMAL.

Closes chfoo/wpull#144

f13bf4674c52f0801a73d352a1b01b78b02a50ba authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up .http.redirect unit tests

[ci skip]

5acec712f26fa1a30ebabd3a083a3ae92cdba7cc authored over 10 years ago by Christopher Foo <[email protected]>

WIP .http.client: Populate requests objects

170e4a125069d74c87dc08c7c72a21c32f809d91 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up .http.client.Client recorder and add unit tests

[ci skip]

9f8335a5410537fd844be3a544bf1930eb6738a6 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up connection pool unit tests.

[ci skip]

362648aefd5f664ae2ddd0835977e2d36a94f1c2 authored over 10 years ago by Christopher Foo <[email protected]>

Add testing/test.pem

[ci skip]

ff82a0f2cdd0af72cb0f57ebc3858c5743aa0661 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up http Client unit test

[ci skip]

d328e0ff4b27628f6d729d0e0271c3b3b43b9f47 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up SSL stream unit tests

[ci skip]

e8565fa0cab78de692d54a20458db683363eef3f authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up .http.request unit tests

[ci skip]

d226127edb6f64fa180d7270f5e0c99706f7ac94 authored over 10 years ago by Christopher Foo <[email protected]>

WIP .http.stream: Wire in connection close logic.

[ci skip]

41988c63723156515aa5455f6344ca08b928f0d0 authored over 10 years ago by Christopher Foo <[email protected]>

Merge branch 'develop' into topic/asyncio

[ci skip]

634cc7e8c19c865110aa2f171805f595a043895f authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up .http.stream and unit tests.

[ci skip]

9bf89b652fcc7621115f8c869498bbb1ceeba5cf authored over 10 years ago by Christopher Foo <[email protected]>

WIP better .http.request Request objects

[ci skip]

0e9b836d2a3d4eeaa0f0f8ba45b48184389b631d authored over 10 years ago by Christopher Foo <[email protected]>

WIP Move Body into new body module.

aa0e96936d88cbba2488c42aa4d653db91e3bccd authored over 10 years ago by Christopher Foo <[email protected]>

Bump version to 0.36.2.

85a3909941548208f44da3faa07e638d99a5eadd authored over 10 years ago by Christopher Foo <[email protected]>

Merge branch 'develop'

84d485a39aef239ba0d4662ed1744e65c1a344b8 authored over 10 years ago by Christopher Foo <[email protected]>

changelog.txt: Update latest to 0.36.2.

[ci skip]

7b4ee6621ac4848f7f507c034ed90dbf6d853906 authored over 10 years ago by Christopher Foo <[email protected]>

requirements.txt: Set tornado>=3.1.1,<4.0

a2ee9d3ddc0790190ba1c1f30a38194790db8a41 authored over 10 years ago by Christopher Foo <[email protected]>

Bump version to 0.36.1

Merge branch 'develop'

Conflicts:
wpull/version.py

76d5021267257c58718256255f65839779701082 authored over 10 years ago by Christopher Foo <[email protected]>

changelog: Update latest to 0.36.1.

3e33e84eef25bd379549f6f797ebb8d3708ccc0f authored over 10 years ago by Christopher Foo <[email protected]>

Close Processor and URLTable only after stats are printed.

Closes chfoo/wpull#139.

9efb53a2be6508fa5c330084292508af33ec3df6 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Rewrite .http.client with asyncio and .http.stream.

[ci skip]

ac455d1e786eba52e42b1fab9b0e728a869b4cfe authored over 10 years ago by Christopher Foo <[email protected]>

WIP Move RedirectTracker to .http.redirect

[ci skip]

9faf4074f304dd7780d794a14c990913de335157 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Add HTTP stream abstraction. Remove observer from chunked reader.

86491d98e0418350a73a5c13184fdca4e69adafd authored over 10 years ago by Christopher Foo <[email protected]>

WIP Move HTTP chunked transfer reader to chunked module.

[ci skip]

34962afa52121188df57ee8ccd6618e3da35f1de authored over 10 years ago by Christopher Foo <[email protected]>

Add contributing file.

[ci skip]

e52074773dcd90fc96bc159fff932de5a52cf643 authored over 10 years ago by Christopher Foo <[email protected]>

Add WIP connection module for async stream abstraction

[ci skip]

08dd3a246a4a2df24efe6ab520a6723e05fb4332 authored over 10 years ago by Christopher Foo <[email protected]>

WIP Fix up dns.py.

[ci skip]

2d0dafbfb2907ada92b2a4030dbbad447e5ba8f1 authored over 10 years ago by Christopher Foo <[email protected]>

Merge commit '54e0b9ec47694f641e2d59f2e305c47553346034' into topic/asyncio

Conflicts:
wpull/phantomjs.py
wpull/processor.py

[ci skip]

d00521df84b01a5b010538b4b5cc56af79449691 authored over 10 years ago by Christopher Foo <[email protected]>

Bump version to 0.36.

Merge commit '7f47914ff781dccfee18ecf3fde966a2809f1b69'

Conflicts:
wpull/version.py

6d8c25544f6517d830cc11754da4fe434687531e authored over 10 years ago by Christopher Foo <[email protected]>

Bump version to 0.37a1.

54e0b9ec47694f641e2d59f2e305c47553346034 authored over 10 years ago by Christopher Foo <[email protected]>

changelog: Update latest to 0.36.

[ci skip]

7f47914ff781dccfee18ecf3fde966a2809f1b69 authored over 10 years ago by Christopher Foo <[email protected]>

scraper.py: Add srcset link scrape.

Closes chfoo/wpull#134

32d9354eb82b75e485ab1cc3602f49961f7261b5 authored over 10 years ago by Christopher Foo <[email protected]>

observer: Allow handlers to be removed during notify.

Re: https://github.com/chfoo/wpull/commit/b12d77dd551c640a1f99c6ca4ac549e16dd6f533#commitcomment...

a862e8cc32c3aae6e99aac7f639f807f854f6491 authored over 10 years ago by Christopher Foo <[email protected]>

Add --phantomjs-exe option.

d376ec0fa45aff12c33bc3a689deff9f396a56d5 authored over 10 years ago by Christopher Foo <[email protected]>

processor.py: Workaround PhantomJS timeouts.

Re: chfoo/wpull#137

01ae1b40cbb9b4d1ea091086d6b795bafff52a1a authored over 10 years ago by Christopher Foo <[email protected]>

Move stop callback concerns from Engine to Application.

ba60ec29414d5942c8d41b91da2890f765f5223d authored over 10 years ago by Christopher Foo <[email protected]>

WIP Port tornado async to asyncio/trollius.

[ci skip]

078e3e31f8fc7fb9e5c1d429d4a172ef66762c79 authored over 10 years ago by Christopher Foo <[email protected]>

phantomjs: Rename page_event→page_observer and use Observer.

99edac21331b6c35e8c3fc3c4c04895f2791ee65 authored over 10 years ago by Christopher Foo <[email protected]>

Add observer module.

This module uses more conventional names. It will replace the 'actor'
module to avoid confusing ...

b12d77dd551c640a1f99c6ca4ac549e16dd6f533 authored over 10 years ago by Christopher Foo <[email protected]>

Format with autopep8.

d84624051f3d56b9bf94784117027dc97b06b7cb authored over 10 years ago by Christopher Foo <[email protected]>

Add and use BraceMessage for log formatting.

56064e249cd1ef2f68ff1960064fbcaf522e6625 authored over 10 years ago by Christopher Foo <[email protected]>

Split 'network' module into 'bandwidth' and 'dns'

5725908cc7911a672f87d472fe4bb20621f42591 authored over 10 years ago by Christopher Foo <[email protected]>

Move exit code logic from Engine to Application.

828e42ec7066856d42c07da178aca9435d9863ac authored over 10 years ago by Christopher Foo <[email protected]>

Builder.build(): Return Application instead of Engine

03974c7110a833b5c38a6e740a5c6a0368aebb12 authored over 10 years ago by Christopher Foo <[email protected]>