Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/ludios_wpull

wpull fork with fixes and faster parsing using html5-parser; used by grab-site; should go away when wpull is similarly improved
https://github.com/ArchiveTeam/ludios_wpull

converter: Fix enum usage in tests

fbf925ff7184bf4fd4e75e2b8864ff4abe7ae847 authored over 8 years ago by Christopher Foo <[email protected]>
processor: Fix request not set on item_session

08a8864701b49ac488ba2235451a40ed30b6c407 authored over 8 years ago by Christopher Foo <[email protected]>
urlfilter: Fix tests to use root_url instead of top_url

f6bd2b05dfee1198383824b3f6b410b7e38ae888 authored over 8 years ago by Christopher Foo <[email protected]>
proxy: Fix up tests and update signature call to http client

4a3a005124bd88dc81533241bf97191ae4b1894f authored over 8 years ago by Christopher Foo <[email protected]>
setup.py/requirements.txt: Add yapsy

3d2a765be9f7dfe035da3de02d55ddfae45eb769 authored over 8 years ago by Christopher Foo <[email protected]>
database: Ensure all row values have the same keys

21c1f41252556b2b3a1bd63bd7d68908704c1e45 authored over 8 years ago by Christopher Foo <[email protected]>
session ItemSession: Ignore invalid URLs

eeabd0dba436d85f367ae0e8a9e644d0e6c14896 authored over 8 years ago by Christopher Foo <[email protected]>
testing: Fix up plugin script tests

9965ab5f4bf74142f72ddd0ef5833ebf07d4b48b authored over 8 years ago by Christopher Foo <[email protected]>
rule: Modify URL testing to use current request instead of db url record

0fcd1efe4228c9a42f63b27b9fffc70500bb51e2 authored over 8 years ago by Christopher Foo <[email protected]>
session ItemSession: Support replace param for add_child_url

bdac75552144a7d6cb88108a772a066785881d3f authored over 8 years ago by Christopher Foo <[email protected]>
tasks PluginSetupTask: Connect other tasks in the pipeline

59d36c867d422d7a9f3185b80b2b1dbd3b3fdd86 authored over 8 years ago by Christopher Foo <[email protected]>
tasks: Split URLFilters task so SpanHostsFilter created database is loaded

04f79864bc8871a5045983be3ccc078ef48594d6 authored over 8 years ago by Christopher Foo <[email protected]>
Reimplement PrintServerResponse as a internal plugin

6ce7d35a454d23ad863fff8d1bccc74eeb332656 authored over 8 years ago by Christopher Foo <[email protected]>
hook: Support listening to hooks as events

ce61cf6436e1f65bff407f836550df40aee05200 authored over 8 years ago by Christopher Foo <[email protected]>
Reimplement --output-document as a writer

0773fd5001af6b6a12c377cccb3574aa366b8a09 authored over 8 years ago by Christopher Foo <[email protected]>
testing: Fix syntax and option for phantomjs tests

b0ae891ab14787628c4f0032d5c0303f4d11f8d8 authored over 8 years ago by Christopher Foo <[email protected]>
Rewrite plugin scripts for tests

86764924deeba932188f92c777e3b017e3c5424a authored over 8 years ago by Christopher Foo <[email protected]>
Fix some missing use of PluginFunctions enum

7836495b0b36713c1a6f58e37d723fe1d612124d authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Add PipelineSeries

328ba196fecb587575467bf0022d4bb6c2967e0d authored over 8 years ago by Christopher Foo <[email protected]>
Update hook/plugin names to use Enum

b802e323e69f67c5b2f9ad7ea6aa078154ddc4bf authored over 8 years ago by Christopher Foo <[email protected]>
plugin: Add InterfaceRegistry and decorators for callback connecting

0e10f0457123576493ae68d6130306bc2242dd5a authored over 8 years ago by Christopher Foo <[email protected]>
testing: Move sample scripts/plugins into integration subdirectory

cfa840c6e117096f35d341af1915c58d77195739 authored over 8 years ago by Christopher Foo <[email protected]>
builder: Update imports

1681a7045c486325499eea0268eaec7f872ac52b authored over 8 years ago by Christopher Foo <[email protected]>
coprocessor.proxy: Update to new URLRecord

53f9fe1a2b1c53ea51b33e548ab5eac82458882a authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Add missing call to finish() and add is_processed assert

3e34c97d5ade18ea05bc925ed9a91a5081839bc1 authored over 8 years ago by Christopher Foo <[email protected]>
database: Fix hostname returned wronly as tuple

52d58f7c63ccf2ffe6da4501b8426b8a363f8b9a authored over 8 years ago by Christopher Foo <[email protected]>
processor.web: Fix request not set on ItemSession. Check for None on coprocessor

e64d71867a87d26e1d35fd33d5918a481a2a453b authored over 8 years ago by Christopher Foo <[email protected]>
Fix member name typos, add missing imports

cda357a9bf75a395617fc91396cce73ff6d0bd32 authored over 8 years ago by Christopher Foo <[email protected]>
Fix WARC recorder event function signatures to match

bf7169b477e2627234a4a2d496811dd189b63db2 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline.tasks: Split tasks into smaller files

c54cfeb847181f7bc91d2a7a0cf9d44cc1d02c5d authored over 8 years ago by Christopher Foo <[email protected]>
testing: Fix timeout param to async_test

6164d9abff47579c5b6fc1fc910314b9a8a6ea08 authored over 8 years ago by Christopher Foo <[email protected]>
Fix import WARCRecord

57d14e7baccb7f530ee6a42d84f659896fc60888 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline.session: Fix handling check out of url record

7168a98bc7e263c614568c344487e8f148933ea0 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline.tasks: Fix syntax, member, function arg errors

8f8d1a115121ee07ef9e62e5a63bc0b95fa4ddee authored over 8 years ago by Christopher Foo <[email protected]>
network: Fix connection test checking wrong connection

df5270a2d778eec127e45f773bbb8f9d05aec416 authored over 8 years ago by Christopher Foo <[email protected]>
database: Fix func arg and member name errors

80e6ed3c81e729d0e9d0945dc9a1ebec75057750 authored over 8 years ago by Christopher Foo <[email protected]>
converter: Update tests to use AddURLInfo

7ac0a684e2b4feb68c279331b1892d0ee8a16125 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Add missing worker ready condition notify. Add debug logs

00ec17a80460c9d94990d861a9532ce74c75e5dc authored over 8 years ago by Christopher Foo <[email protected]>
Remove obsolete engine module

8fa301bfcacdb8d748dfdf8f22dd3b643d96ba93 authored over 8 years ago by Christopher Foo <[email protected]>
Split integration tests into multiple files

1fd3da22460ac5b6524bd62d7fd9d8dcaaa741b5 authored over 8 years ago by Christopher Foo <[email protected]>
Move options testing into options_test

a097b5fc69bd12fa32400568a4c9d50914a6aeb7 authored over 8 years ago by Christopher Foo <[email protected]>
builder: Add ProgressSetupTask to pipeline

44c231cc340e80b276566759918624d36eb54ea3 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Rewrite progress bar into a class and task

88daedb025afea4541c65809479d019323553df8 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline WARCRecorderSetupTask: Connect listeners to recorder

b40aea38c34a5267d80a39c4b895ca623f900760 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Move WARC recorder setup task into seperate module

4744675bddfb1fd3b54e3a88f8e776e113a9903b authored over 8 years ago by Christopher Foo <[email protected]>
warc: Rewrite WARC recorder to listen to client events

11dd8697c42a9400940686ee44990026dab5f63d authored over 8 years ago by Christopher Foo <[email protected]>
protocol: Move Client Session Event callbacks into abstract

379b8d37fbcb03256784bfdc20418757cf4d10c1 authored over 8 years ago by Christopher Foo <[email protected]>
Move warc modules into a subpackage

754fc4666570f2a24b7902a28db92903768cc9e2 authored over 8 years ago by Christopher Foo <[email protected]>
protocol: Use enums for client and session events

cf861dbf00612ae4eebb3b7db082cd8e53302f46 authored over 8 years ago by Christopher Foo <[email protected]>
protocol: Add DataEventDispatcher class, add events to FTP & HTTP streams

e0dd83540757fe2a446b254d6148ee4ddaf2b5f7 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline.tasks: Remove instances class to Web/FTP processor init

485c144c9e3870fef31d8511e2eba618ddff05e1 authored over 8 years ago by Christopher Foo <[email protected]>
Move exit_status finishing_statistics hooks. Remove old dispatch hook code.

da357c6a30ed233e06825d12c367d4c4e0fe7182 authored over 8 years ago by Christopher Foo <[email protected]>
coprocessor: WIP update to use ItemSession

5d4e41ea76418230f84063a0ef3a09e76fe0637b authored over 8 years ago by Christopher Foo <[email protected]>
processor.web: Update to use ItemSession. Add typing.

f4074f1b9aab4587383ac10d38986697ff12f755 authored over 8 years ago by Christopher Foo <[email protected]>
processor.ftp: Update to use ItemSession. Add typing.

7964b3ce3cd343afe41c2ee28d78c02970ba36c2 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline.session: Add ItemSession.update_record_value()

543fb2ee07a7aea319516aeb24bf0e5f0ce4561b authored over 8 years ago by Christopher Foo <[email protected]>
processor,rule: Accept ItemSession instances, update signatures, add tests

2ea5d8aa04285f6c7d3917d1720a2165d69fa15f authored over 8 years ago by Christopher Foo <[email protected]>
http.robots: Update mock object signatures

59e5c8d4793b534eb0e19d187ce4d7e3a9f40b22 authored over 8 years ago by Christopher Foo <[email protected]>
testing.util: Log changed directory

10590a7cf5adb2b29c7349c3152d7772ec476825 authored over 8 years ago by Christopher Foo <[email protected]>
pipeline: Fix wrong member name

bf74ed27e284f0acf2230b221eef26fd03a990d1 authored over 8 years ago by Christopher Foo <[email protected]>
writer: Rewrite unit tests to not rely on application test

ad10a2fb014696cd340f22c937a54cf44a745a20 authored over 8 years ago by Christopher Foo <[email protected]>
writer: Add typing

0eee1d6d068effa6203a3d4220b62bd036b729d3 authored over 8 years ago by Christopher Foo <[email protected]>
writer: Handle case where changing filename of an absolute path

Absolute such as /tmp/asdf

9b7292447c16389d76d12187d5379c2ab507163c authored over 8 years ago by Christopher Foo <[email protected]>
path: Handle case where path to anti-clobber is absolute

For example, /tmp/asdf/a/b/c/d would mistakenly be changed to
tmp/asdf/a/b/c/d

53d2e6fee7eb46060a744a956d38d80989a0f625 authored over 8 years ago by Christopher Foo <[email protected]>
protocol: Add set_continue to BaseRequest. Add body & request to BaseResponse

1e3820ff974016a7c1330e37e81558eb39dc2cb7 authored over 8 years ago by Christopher Foo <[email protected]>
Move coprocessor package under processor package

6e62a0e99551a295952f44dcd90d8d09481ac6d0 authored almost 9 years ago by Christopher Foo <[email protected]>
pipeline: Fix call to URLTable. Adjust path of CA bundle

a9378088ae860a4da9891db04353250d50284542 authored almost 9 years ago by Christopher Foo <[email protected]>
pipeline: Fix imports.

a3ca3f92fc834e5e36571deba086b118b02a434d authored almost 9 years ago by Christopher Foo <[email protected]>
pipeline: Add stderr to AppSession

2523716a099f33c4492a8bc12c8dc884ff9af22f authored almost 9 years ago by Christopher Foo <[email protected]>
Use get_hostnames() for creating SpanHostsFilter

7ec435041f18adad1f0debb0aa4ae038e6fb46ef authored almost 9 years ago by Christopher Foo <[email protected]>
dns: Add back rotate parameter with cache

41f7dcf8b6409268aab1e695f582accce6881301 authored almost 9 years ago by Christopher Foo <[email protected]>
database: Add get_hostnames()

3146197171dd4e60172118b4c09794ae1b5258a7 authored almost 9 years ago by Christopher Foo <[email protected]>
urlfilter: Update arguments to match new URLItem properties

bc476710d5ed17647265adb4ad7f532a640c2837 authored almost 9 years ago by Christopher Foo <[email protected]>
database.wrap: Update class to use parameters and hook functions

3e889fa3e38b9b1cb4fc75bf53947cbff09aa799 authored almost 9 years ago by Christopher Foo <[email protected]>
Add missing colon

b5003f481b8dc08d514526787bb37e9b99d6ca08 authored almost 9 years ago by Christopher Foo <[email protected]>
dns: Add hook callback

ead0cad2c194326aec541f945a536c802287aae5 authored almost 9 years ago by Christopher Foo <[email protected]>
Implement connecting hooks from plugins

7079d343c55cf939da6eaf49047c34af86c9c5b0 authored almost 9 years ago by Christopher Foo <[email protected]>
processor: Rewrite DelegateProcessor to register indivdual schemes

96e490e28059a1297d8ea43cceeb5ce1c1088a46 authored almost 9 years ago by Christopher Foo <[email protected]>
pipeline: Stub in download and conversion tasks. Fix imports

15359fbbded8b8678883478a80a040bb5a1008c4 authored almost 9 years ago by Christopher Foo <[email protected]>
pipeline: Rewrite URLItem into ItemSession

79d24695c1a2fd4683d31a569314817593471ea4 authored almost 9 years ago by Christopher Foo <[email protected]>
pipeline.tasks: Use ItemSession for ItemTask

3675bde938ffb5d678b3294e6012717a96dd8848 authored almost 9 years ago by Christopher Foo <[email protected]>
Move tasks package under pipeline package

7263c24f93869bb0e681e0d1f1958405256590a5 authored almost 9 years ago by Christopher Foo <[email protected]>
Add plugin stub code

1135d2c73239afd8fb69100adf551ec59df818ac authored almost 9 years ago by Christopher Foo <[email protected]>
Rewrite basic hook logic to HookDispatcher and add EventDispatcher

5c78942caea04d5a1807c3367f3e856509c317b6 authored almost 9 years ago by Christopher Foo <[email protected]>
Move functions in builder module into tasks package

5a07eaa05e2efeb7af7ac6b84ad46021c887ea6f authored almost 9 years ago by Christopher Foo <[email protected]>
util: Add grouper()

48619f8da30db27795ed02f5c758ac40428e193b authored almost 9 years ago by Christopher Foo <[email protected]>
database: Rename URL to QueuedURL. Clean up and rename columns.

b54007c78a8d3eccdfde4547f1c72eb1873fb667 authored almost 9 years ago by Christopher Foo <[email protected]>
database.base: Add typing to functions

214dc73a7acb2520171685391f77125c19034ba7 authored almost 9 years ago by Christopher Foo <[email protected]>
item: Add in function to convert values to dict for database

98f9d1b8cc5124f644daa34db3e423d50a2b76fc authored almost 9 years ago by Christopher Foo <[email protected]>
item: Split URLRecord into seperate data structures.

Move URLItem into session module

4ebdd0afa3759a393a555196ecf89042268f9b33 authored almost 9 years ago by Christopher Foo <[email protected]>
Rewrite Application task to work only on Pipelines

27ac860929c20ce950a8605c5d59aa08574ceac1 authored almost 9 years ago by Christopher Foo <[email protected]>
WIP Move Application class specific code into tasks package

70bdfe59d26be63afd58d42ed8acdb18060f9c65 authored almost 9 years ago by Christopher Foo <[email protected]>
Move code from __main__ into application.main

b7183d3bc514dbd919dbc6e57b1066aba1293de4 authored almost 9 years ago by Christopher Foo <[email protected]>
Rename app_test

311cdf8a4bd1acb8ba7cd3197eac49e54bbfad33 authored almost 9 years ago by Christopher Foo <[email protected]>
Move factory module into application package

3d7ff4bc5664f2f5bb1599cf700213e91946b4c4 authored almost 9 years ago by Christopher Foo <[email protected]>
Move app, builder, hook, options module into application package

c9835481132d10b99aaca4148d0e4bb25d637af9 authored almost 9 years ago by Christopher Foo <[email protected]>
Add pipeline module as generic replacement for engine class

54ee69b15ea24fe0c7211de20a49f52eaf76d92d authored almost 9 years ago by Christopher Foo <[email protected]>
Move engine & item modules into pipeline package

badf0ae5deabc8aab37bd4723081b591cbdc31ec authored almost 9 years ago by Christopher Foo <[email protected]>
ftp.client: Rename ambiguous "fetch" functions. Use SessionState enum.

Also add typing.

a30fe9513875f6f530f2671a2e0eea43d7580420 authored almost 9 years ago by Christopher Foo <[email protected]>
ftp: Add typing to command, stream, util modules

305dcea649fbf7ecf7f47c712c48bcdf14bddef8 authored almost 9 years ago by Christopher Foo <[email protected]>