Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/ludios_wpull

wpull fork with fixes and faster parsing using html5-parser; used by grab-site; should go away when wpull is similarly improved
https://github.com/ArchiveTeam/ludios_wpull

Remove unused FrozenDict and EmptyFrozenDict

30df162dc1bef94fcd8a3ae57a2a72be912c6650 authored about 6 years ago by Ivan Kozik <[email protected]>
asyncio.async -> asyncio.ensure_future for Python 3.7 which makes `async` a syntax error in this context

b50ba63317395123176d88a540fb0d1fd14d4e05 authored about 6 years ago by Ivan Kozik <[email protected]>
Load plugins early for improved flexibility (fixes chfoo/wpull#383)

Plugins are attached to existing HookableMixin instances as before. In addition, they are also k...

1f635ee49354df3486c9652d88d6c773385255b3 authored about 6 years ago by JustAnotherArchivist <[email protected]>
Remove Python 2 comment

626607de0e82585cd4c8d7cb1cb2ba749014d44d authored about 6 years ago by Ivan Kozik <[email protected]>
Fix error in be96abc86d5f45a3ff91e6567594736176b08c27

f30a776761407d47f1718400cc8c56aeb673544f authored over 7 years ago by Falcon Darkstar Momot <[email protected]>
Fix crash introduced by removal of FrozenDict in 561380774baf5fd44990d16d64f545259c7385a1, but without constantly reallocating dicts

be96abc86d5f45a3ff91e6567594736176b08c27 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>
Bump to 2.0.3. Less crashy than ever!

f946d3fd430cb8edf5a9d14b7c1ceda5ce479e58 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>
Remove:

- Non-performant FrozenDict, which doesn't appear necessary based on testing
- HTTP proxy CONNECT...

561380774baf5fd44990d16d64f545259c7385a1 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>
Handle invalid URLs during parsing loop, which will avoid recursive wget halting on certain invalid URLs

582fd260def114f2a421c357d7af7c4a1adae247 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>
Give hostname we choked on when raising IDN hostname exceptions in url.py

bf0f9cc8fed1451465c838131e0e017a5d99ff58 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
2.0.1a not valid; use 2.0.2

2c7f18ede892d5773f82ebef6f5891cdcdfe580c authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Version bump 2.0.1a

f8761ce683203a3d0866846a38e5e03eed3c8f44 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Have wpull.url.URLInfo return None if given a None for url, to support numerous checks against root and parent URLs that follow this form

f4dd12c23d46d5f165a002704ff155448dc69c1b authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Raise correct exception type while sock becomes invalid in TLS establishment

7a31b08de33f99b6b7efba3953b0dc1b003768d4 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Attempt to fix RuntimeError('Event loop is closed') issue

2778442cb94c708a9cd520bdc3460a249952400b authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Remove unnecessary reference loss that prevents WebSession.__exit__() from having any effect on its _recorder_session (WebSession going out of scope after __exit__() will prevent lingering references)

ff8e99cadd7e490c3e35ad401dcefca2b90a1ea6 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Fix intermittent issue wherein self.writer.transport.get_extra_info crashes with AttributeError while operating on NoneType

71a4299415fadfa2e23cf59c10acda6c4ddb88b2 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Fix issue whereby WebSession never fires SessionEvent.end_session in its __exit__, thereby causing --warc-max-size to be ignored

a4c6e92d41507e9b1494e1ca68235c20bb96bf3c authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Handle AttributeError coming from asyncio/streams.py on what appears to be an attempt to read from a closed connection

0c517bab510ff9555bff055b4e1e78a807e0bd90 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>
Bump version 2.0.1

d0a9dffd2f79e62242e5ddb603c5e77e5b012e91 authored over 8 years ago by Christopher Foo <[email protected]>
Merge branch 'develop'

4ac84c3bd51cf929ce63a02a7f8764f5196867ea authored over 8 years ago by Christopher Foo <[email protected]>
changelog: Update latest to 2.0.1

89ecd35fc5c62398a9e04f0239242f4632c58ac4 authored over 8 years ago by Christopher Foo <[email protected]>
proxy.server: Actually write response body to a file for use later

Closes chfoo/wpull#329

bc55c6b53528ac56c5d56d2e66605f4b711b212b authored over 8 years ago by Christopher Foo <[email protected]>
github: Format and reword issue template a bit

[ci skip]

b2a23d15c1c20a2cc6009a96e9a7e8cf28678da1 authored over 8 years ago by Christopher Foo <[email protected]>
tasks.resmon: Use get() instead of index access

Closes chfoo/wpull#327

2bc5ca8b62c37bc042b94374c07b70bc56547bd1 authored over 8 years ago by Christopher Foo <[email protected]>
Bump version 2.1a0

6f222e0410ff2a8e222e5c3cf31a52ac7c119e37 authored over 8 years ago by Christopher Foo <[email protected]>
Bump version 2.0

0c20cb186d78ed4ac6ea8cd678d557e84afa3689 authored over 8 years ago by Christopher Foo <[email protected]>
Merge branch 'develop'

b84ba51c4ae3f3cdd786c6fa903bb6b6bd0116d3 authored over 8 years ago by Christopher Foo <[email protected]>
doc: Update latest version to 2.0

[ci skip]

e987fe2d267a99405031342d022c9197c34bfa86 authored over 8 years ago by Christopher Foo <[email protected]>
doc: Reword database resume line

[ci skip]

9ea1f6b433e77048f2366090490a12f587d3811e authored over 8 years ago by Christopher Foo <[email protected]>
freezer: Install specific version of Pyinstaller. Add plain entry point file.

15b260e522f595cc3c1b66a1067fbfbc15007c7f authored over 8 years ago by Christopher Foo <[email protected]>
freezer: Include wpull plugin files using pyinstaller spec file

9d68e5f440577e91ef52d2b43514c6c71f42def2 authored over 8 years ago by Christopher Foo <[email protected]>
setup.py: Use entry_points instead of providing scripts ourselves

8b4985574ea18d963e58c4d55a0b57deaa47c614 authored over 8 years ago by Christopher Foo <[email protected]>
freezer: Update to support pyinstaller 3.1.1+

fcf5573e6c07b71059bbf0fded42f34508dbc889 authored over 8 years ago by Christopher Foo <[email protected]>
doc: Add testing notes, how to help, update current year

[ci skip]

c634ebf6ef5dbc051954068677b25d1ff47ca941 authored over 8 years ago by Christopher Foo <[email protected]>
app: InputURLTask: Add TODOs and stub code for progress

215000ad5d48ef39555d599b726f09aa52414136 authored over 8 years ago by Christopher Foo <[email protected]>
app: DatabaseSetupTask: Add missing release()

265bcdedb49b2c619496b6aa37f645941511573a authored over 8 years ago by Christopher Foo <[email protected]>
db,app: Add QueuedFile and make LinkConversionTask support items

b1cd09efa60b1503831c4dcfa547a6c089fa89ad authored over 8 years ago by Christopher Foo <[email protected]>
database: Add priority column for future use

88fde26b2a20a7e3a2ff75615b0b944a29c36768 authored over 8 years ago by Christopher Foo <[email protected]>
converter: Handle NotFound error on converting CSS links not in database

Closes chfoo/wpull#312

131a221024ed1212a051990fb2de0e27a1f48d8b authored over 8 years ago by Christopher Foo <[email protected]>
doc: Add note about switch from trollius to asnycio

[ci skip]

a4623ca407a6cb842dd4ba7b94733ac96b07c477 authored over 8 years ago by Christopher Foo <[email protected]>
doc: Remove should_activate() from example

should_activate() is usually used for internal or advanced plugins

[ci skip]

870cde72a05c9ed4538eaa43679da2545597ac56 authored over 8 years ago by Christopher Foo <[email protected]>
doc: Update hook/event functions

[ci skip]

9eed1a4a1c64223421c690e8b6259c8537db3831 authored over 8 years ago by Christopher Foo <[email protected]>
dns: Reuse original host if hook callback returns None

b97409ec96553b68b588cdccc4e37e751a6dc03a authored over 8 years ago by Christopher Foo <[email protected]>
doc: Add note that proxy requests/respondes now hooked

re: #123

[ci skip]

2f470f36ba0e135affa2042d6d280ac4b9d6f010 authored over 8 years ago by Christopher Foo <[email protected]>
doc: Add some versions to dependencies

[ci skip]

36f4dcc19a2d03b2bcb06a1551d4b6bbeb9c1234 authored over 8 years ago by Christopher Foo <[email protected]>
Add stricter ranges to requirements.txt.

a86eb88f301bdb15d3c2704c63cd70f6fa0514ee authored over 8 years ago by Christopher Foo <[email protected]>
doc: Regenerate API templates

a55ebc5140be318fbe8285b0ee56f1f6f7053d9a authored over 8 years ago by Christopher Foo <[email protected]>
doc: Remove autodoc on private members. Expand plugin example. Add category to hook interfaces.

6f41bcd065c78bea76d75578ab0bebb7b37ea26c authored over 8 years ago by Christopher Foo <[email protected]>
doc: Use built-in Napoleon module for Sphinx 1.3+

2be4733b4f03fd872191e418164637a41c7776c6 authored over 8 years ago by Christopher Foo <[email protected]>
stats: Remove obsolete temp_dir and db doc. Add typing

4d43db06932d1065f7e498558d8282d6663341bc authored over 8 years ago by Christopher Foo <[email protected]>
Change some modules to use logger StyleAdapter

67de1852850ec815e09691fd6bce1f4c6c850948 authored over 8 years ago by Christopher Foo <[email protected]>
application.plugins: Add missing super call for activate()

44d076ac9647d612d781a3eaa8d93c626d34ae30 authored over 8 years ago by Christopher Foo <[email protected]>
tasks.plugin: Fix wrong variable used

7609f7fe6d85ef7bc6726920c25c0dec0753a5e4 authored over 8 years ago by Christopher Foo <[email protected]>
Add logging StyleAdapter

7e4da21a8ecea00f7803593e74305b86dd835b99 authored over 8 years ago by Christopher Foo <[email protected]>
robotstxt,urlfilter,urlrewrite: Add typing

934dd9661adb06eca5626b630a6fd925c790e42c authored over 8 years ago by Christopher Foo <[email protected]>
app: Move arg warning to a plugin

1a7675000e1e85a666aa4256fbf4e302f15a5dee authored over 8 years ago by Christopher Foo <[email protected]>
app: Move debug console to a plugin

9bc4738c9018de0c19c735302b24d286a496731e authored over 8 years ago by Christopher Foo <[email protected]>
app: Rename serverresponse plugin to server_response

8803e99ebb9e1d4f54ab270e4a3972e6b7107a8e authored over 8 years ago by Christopher Foo <[email protected]>
app: Move download progress to a plugin

d78570b44868fa68763e8b64018b26a57de727f2 authored over 8 years ago by Christopher Foo <[email protected]>
tasks: Log internal plugins as debug level

163d4ec02f9c561f80fa6ec49831bee6343a1856 authored over 8 years ago by Christopher Foo <[email protected]>
progress: Fix not updating. Remove redundant stats already in logging

192436c7165574f1ea7789746709645c61b14a6b authored over 8 years ago by Christopher Foo <[email protected]>
app: Close the event loop after run() finishes

Avoids python issue 23548

2f69428605cfc2d914ba97f81be60556f03f689e authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Remove logging.basicConfig

41cdc389fa9d4d0d3c7f74eb75e6b23b3581630d authored over 8 years ago by Christopher Foo <[email protected]>
github: Add instruction to attach files

[ci skip]

b214c7ec8b43e8dff10cdb8dc303efb2c3dccbad authored over 8 years ago by Christopher Foo <[email protected]>
ftp.ls.date: Seperate elif cases so scores can accumulate properly

Closes chfoo/wpull#294

8dbc358c6e2d79f3527e2a4324faf630faba19af authored over 8 years ago by Christopher Foo <[email protected]>
doc: Add autogen plugin hook/event function interfaces

[ci skip]

0b7d25103c73674c52f6517f57686baf77bd7b85 authored over 8 years ago by Christopher Foo <[email protected]>
Use create_task() instead of ensure_future() for backward compatiblity

Backward compatibility for Python 3.4.3

e611513f85b294931eb40aa532e4c154da343328 authored over 8 years ago by Christopher Foo <[email protected]>
setup.py: Readd pem file include

31acf9cdd60e04557a2d870e313bbc471940f5b5 authored over 8 years ago by Christopher Foo <[email protected]>
github: Fill in more contributing doc. Add issue/PR templates.

[ci skip]

cherry picks 3d6309d8 a317ea09

b1aa5095cab1f41693f69c86e3469d061e4b811c authored over 8 years ago by Christopher Foo <[email protected]>
fixup! github: Fill in more contributing doc. Add issue/PR templates.

Fix syntax errors

a317ea095ced945907baba72d537f0c7d86672b0 authored over 8 years ago by Christopher Foo <[email protected]>
doc: Remove bullet list which is already included in github template

[ci skip]

da091a01bfc21e3d6f1b618b06ed365248ad7983 authored over 8 years ago by Christopher Foo <[email protected]>
github: Fill in more contributing doc. Add issue/PR templates.

[ci skip]

3d6309d8b5620519ed7bd6c6ba58a80a8c738548 authored over 8 years ago by Christopher Foo <[email protected]>
changelog: Add python-script removal and plugin-script entries

51f7e6e510c573b0cfc489b8b9fc2cddd6f7e39e authored over 8 years ago by Christopher Foo <[email protected]>
doc: Update WIP scripting docs

257982b97a3c614578e45252176df6ca5e7755bc authored over 8 years ago by Christopher Foo <[email protected]>
doc: Add info on differences and intended use cases, update links

Closes #282

249c195f5c99d9bd3c8b2c492944ff498d75c7a3 authored over 8 years ago by Christopher Foo <[email protected]>
doc: Regenerate API templates, add undoc-members for now

ac33a0483a5fb9b067c50e5783f8d792447b5b48 authored over 8 years ago by Christopher Foo <[email protected]>
setup.py: Update package names

daf5c12b677f9afa656cdb9c3bb5a5bbbc5456b9 authored over 8 years ago by Christopher Foo <[email protected]>
Merge branch 'topic/modernize_2016' into develop

4d6b668e801bc4e6b205f94884c3fe254e474214 authored over 8 years ago by Christopher Foo <[email protected]>
Reimplement quota checking by using get_root_url_todo_count()

52321b02e80f2a9b90c8bd498e8d935190392f70 authored over 8 years ago by Christopher Foo <[email protected]>
database: Add get_root_url_todo_count()

c6af9bef99a362858a63a03a28ef9d8e7fec9c72 authored over 8 years ago by Christopher Foo <[email protected]>
writer: Add MuxBody for output-document for cases where it is write-only

For cases such as stdout, MuxBody will use a temporary file so it
reading can be done on the dow...

95d9e5a67fa78e44a36ea9b3f16ff2afae60427c authored over 8 years ago by Christopher Foo <[email protected]>
Rename wrapper to cookiewrapper

87482a3d7ff80b601a2eb4fe97f3c5efdf8f5038 authored over 8 years ago by Christopher Foo <[email protected]>
Move tasks subpackage to the application subpackage

19a1784f74ebc6ce367eaeb48eaaa751796155ef authored over 8 years ago by Christopher Foo <[email protected]>
testing: Fix cookie jar and robots.txt pool tests

9be67b7bfd835ab65d0877ec25cc04f402691c2c authored over 8 years ago by Christopher Foo <[email protected]>
processor: Add context manager needed on web client session to fix conn pool

d4cdbe94d162db5d8dc91abfc48ca96a3ec41ffd authored over 8 years ago by Christopher Foo <[email protected]>
tasks: Fix FileWriter init args passed to wrong function

026af156b784c3d14e4b29f7764ffbb648954dc5 authored over 8 years ago by Christopher Foo <[email protected]>
application: Fix cookie jar not saved at exit

92ec57440d0a00a7d41e98024ff319ee2c726e94 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Fix link type not used

bd65cf71f57c22e998b165ec43d360ec45e8b964 authored over 8 years ago by Christopher Foo <[email protected]>
testing: Change file types for youtube-dl, comment out cert absence check

ef0efb20d2e85713071e3e0b613aafe3628243f0 authored over 8 years ago by Christopher Foo <[email protected]>
http.client: Fix wrong StreamReader/StreamWriter typing for download()

6d2cacdf96e74c856016d62072ada2aae0f25776 authored over 8 years ago by Christopher Foo <[email protected]>
coprocessor.proxy: Support calling hooks, aborting requests, add url to database

Re: chfoo/wpull#123, chfoo/wpull#228

dc638ea1051f30e5b7a27a102554ba6c38dd158b authored over 8 years ago by Christopher Foo <[email protected]>
proxy: Rewrite server to use hook/event dispatcher. Sniff for SSL/plain tunnel.

7c61c95e04e76d045ee91ed5c13c3d3f6ff551b8 authored over 8 years ago by Christopher Foo <[email protected]>
hook: Fix event dispatcher transclusion not being called

81757d6cdf7137f0ab98c96f4f39375e76d3f1b2 authored over 8 years ago by Christopher Foo <[email protected]>
warc: Fix extra arg passed on session close event

9e82b7203a8a4b3d3107c4823eeb9fa7ffe852e1 authored over 8 years ago by Christopher Foo <[email protected]>
phantomjs,youtubedl: Add coroutine decorator to callbacks avoid returning None

9cbffc8770d17e7c8e856ab24dd6d064c9e58704 authored over 8 years ago by Christopher Foo <[email protected]>
proxy: Fix error classes referring to old Trollius aliases

fc528f351b279747fe48ccad005eb7eb41698dd6 authored over 8 years ago by Christopher Foo <[email protected]>
connection: Use new ssl_object key for Python 3.5.1 to get ssl socket

13cd44301178761e2ea380492989913f3d37f2bb authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Fix proxy server not created

3dd1fdceb0fe531970302ef5a55b89d64bbb6209 authored over 8 years ago by Christopher Foo <[email protected]>
warc: Fix TempDirMixin and import

ab7ae45b8ebec64c6e9acb8ab563da4b9d5ef5a0 authored over 8 years ago by Christopher Foo <[email protected]>