Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/wpull

Wget-compatible web downloader and crawler.
https://github.com/ArchiveTeam/wpull

recorder.warc: Assert request address is an IP address

43d98a3f8b3f787423dbff4b1c4d8587ff6bb851 authored over 9 years ago by Christopher Foo <[email protected]>
Change log/progresss message "none" to "unspecified" to match Wget

9fcf5011a110627523734a0868f57e32fabf81aa authored over 9 years ago by Christopher Foo <[email protected]>
connection: Refactor Connection to BaseConnection

e774e1b04f66d187ddcad2856cb4e15fdf81bad9 authored almost 10 years ago by Christopher Foo <[email protected]>
dns: Remove global cache and use instance cache.

Use a per instance cache to avoid potential side effects that may
happen, for example, during un...

b90d044faa7175e3fbca543c046e8bfc5f32b66d authored almost 10 years ago by Christopher Foo <[email protected]>
doc/options: Add nodefault directive to hide internal defaults

[ci skip]

4ec014e25cafea96109957b51f32e73b778caa32 authored almost 10 years ago by Christopher Foo <[email protected]>
coprocessor.youtubedl: Save JSON file to WARC file (pywb spec).

re chfoo/wpull#22
Closes chfoo/wpull#252

93ce9ab7177aae7f6e401892ed8b8473cf107e50 authored almost 10 years ago by Christopher Foo <[email protected]>
url docstring: Note path always begins with slash

[ci skip]

b9e07b91765c1cf93349bc9e43acef6c5b8536ae authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 1.1

209b599bc66120b6d3c7acd34e555df300ff18f8 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge commit '1af45d06f021166189bef471dbc10f1f1e1fbf47'

a7ca8c214eb0c55f985faeab157a0159facc6182 authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 1.2a1

[ci skip]

fba51c2b356fef77aa1a7622ae0f14a15536d031 authored almost 10 years ago by Christopher Foo <[email protected]>
chnagelog: Update latest to 1.1

[ci skip]

1af45d06f021166189bef471dbc10f1f1e1fbf47 authored almost 10 years ago by Christopher Foo <[email protected]>
Update CA certs (Fri Apr 3 15:56:53 2015)

248f4c173236bf67d0126855fabfc3bd4debb37c authored almost 10 years ago by Christopher Foo <[email protected]>
stats: Check keys instead of bool value of db.

5ded09dce497ca195cd9449829dcfa3456a53635 authored almost 10 years ago by Christopher Foo <[email protected]>
builder, stats: Use temporary shelve to store input URLs.

Closes chfoo/wpull#259

a20fb78832b90d3b910f9841f9a7214f470a4046 authored almost 10 years ago by Christopher Foo <[email protected]>
Prefix all temp files with "tmp"

9a287a080d2736f6658a006f182f7df580687941 authored almost 10 years ago by Christopher Foo <[email protected]>
recoder.warc: Raise OSError if journal files are found.

Closes chfoo/wpull#253

76cf49c9547c8f647f9dfb26e6e8a6596d5e3611 authored almost 10 years ago by Christopher Foo <[email protected]>
recorder.warc: Write a "-wpullinc" journal file to accompany WARC file.

re chfoo/wpull#253

3b15fcb4beb4984a94d52341059835cf8a4cafa4 authored almost 10 years ago by Christopher Foo <[email protected]>
recorder.warc: Don't append to existing sequential WARC files.

Don't append to existing sequential WARC files and then check the file
size to make the sequence...

ba8181ea94f9fe4569f4411443b492f6382eb696 authored almost 10 years ago by Christopher Foo <[email protected]>
dns: Revert default family arg back to PREFER_IPv4

The unit tests expect to be bound to IPv4 addresses.

f606838ba6b8e968d009a87c17cc6d580a1bbe9c authored almost 10 years ago by Christopher Foo <[email protected]>
dns: Refactor and add resolve_all and resolve_dual

re chfoo/wpull#154

366d45fe29f79525045999955d19ce502ef281a7 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge branch 'yipdw-topic/install-typo' into develop

Conflicts:
wpull/version.py

6335f27032ddf4df1d59e91a516a4ba102ba368a authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Remove stray "update" in install.rst.

024db7f9712581f881553ccb280c17a70e4e9ed6 authored almost 10 years ago by David Yip <[email protected]>
scraper.html: Scrape links from Open Graph meta tags

Closes chfoo/wpull#255

ad12b122764d33da9fd3fe41c07ddcdaa89d6dfc authored almost 10 years ago by Christopher Foo <[email protected]>
options: Change 'pcre' as --regex-type instead of 'posix'

Closes chfoo/wpull#256

ae42a72c8fa073bd8ddc25903b3c09d0056e8083 authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Remove outdated statement about SSL with proxies

Was fixed in 7cf39c835b11fc865b0d40e94cf772cf0076677c

[ci skip]

7079ac27eb5b1d21412415ddba71d328a9d18c8f authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 1.1a1

I meant to put develop's version one minor version greater than master.
This fixes 616a4a0.

[ci...

a88cdcadf72ff0a89a64aec249fb11d287911fe5 authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 1.0

2ff0a52328b6d86166ce0a219edbd209bced92f3 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge commit '7b846902edd57659421cf5aec3339f1244037c4c'

Conflicts:
wpull/version.py

e9f33c86525edc09c6bc06f94c21ad2ac9077042 authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 1.0a1

616a4a0f5ac40982b8af267ae7defe9a7118fef0 authored almost 10 years ago by Christopher Foo <[email protected]>
changelog: Update latest to 1.0.

[ci skip]

7b846902edd57659421cf5aec3339f1244037c4c authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Simplify feature list to 5 points.

[ci skip]

a0962193a2fc1eb98a008366db132760c888a4f6 authored almost 10 years ago by Christopher Foo <[email protected]>
freezer: Append the timestamp to the zip filename.

[ci skip]

97ce30b907b5d5aebdb147049006934a81f57ef3 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge branch 'issue/251-pyinstaller' into develop

Closes chfoo/wpull#251

[ci skip]

9fd7dfd42a55b4cbbc339d23305e5c2e5d5d5840 authored almost 10 years ago by Christopher Foo <[email protected]>
freezer: Use deflate instead of bzip for backwards compatibility

bzip doesn't compress the binaries much anyway

[ci skip]

42ef79feb8f41c2021b299c0666b4f838f888f30 authored almost 10 years ago by Christopher Foo <[email protected]>
freezer: Add a hiddenimports hook for dnspython

[ci skip]

f7dcfb3a1502ea35f4b0c7ebb6ff0f2fdcc3cb6a authored almost 10 years ago by Christopher Foo <[email protected]>
freezer: Prepare a zip file.

[ci skip]

53cf70e6648bff4b8cd6ba3e77c70c03e8325e4f authored almost 10 years ago by Christopher Foo <[email protected]>
freezer: WIP support for Windows and Mac OS X

re chfoo/wpull#251

[ci skip]

1bcdd1f679979b9e9bf70eb49e1ac8a35db596fa authored almost 10 years ago by Christopher Foo <[email protected]>
freeze: Add some readmes

[ci skip]

f675b9ce5b58765e5018370f521d9bf8dc9cebd6 authored almost 10 years ago by Christopher Foo <[email protected]>
freezer: Add WIP pyinstaller runner.

re chfoo/wpull#251

[ci skip]

4209d037ac6a4e71818f3ad7fe890198f573e923 authored almost 10 years ago by Christopher Foo <[email protected]>
freezer: Move cx_freeze scripts to a subdir

[ci skip]

dbe984c9b518dfd78b9b4c9052b77665e2d61eeb authored almost 10 years ago by Christopher Foo <[email protected]>
Implement no_proxy env var support. Added --proxy-domains etc.

Closes chfoo/wpull#244

5f6f89f008d1204261bc410be164ae193798c132 authored almost 10 years ago by Christopher Foo <[email protected]>
Add proxy.hostfilter.HostFilter

re chfoo/wpull#244

dbda328e0a0712c5befb1349d4de0b8636f6c605 authored almost 10 years ago by Christopher Foo <[email protected]>
builder: Add prefix & suffix to temp cert file.

19d03be01cd2f1d27a3c160adad7fcb446175399 authored almost 10 years ago by Christopher Foo <[email protected]>
Add TempDirMixin and use for unit tests in setUp/tearDown

Reduces amount of indent and fixes warc files not cleaned up.

00f8766b8d2c79205acbdf1f6fe4fda3769481da authored almost 10 years ago by Christopher Foo <[email protected]>
Merge branch 'topic/proxy_fix_up' into develop

7cf39c835b11fc865b0d40e94cf772cf0076677c authored almost 10 years ago by Christopher Foo <[email protected]>
proxy: Track and reuse SSL connection instances. Add proxy_test.

dd345f0d7d6b36395c0b9606b7a0672f4cd3de39 authored almost 10 years ago by Christopher Foo <[email protected]>
Remove --no-secure-proxy-tunnel option

242f6d61d829edf097972f864088de8b16e3194e authored almost 10 years ago by Christopher Foo <[email protected]>
Rewrite ProxyAdapter to HTTPProxyConnectionPool

e201cc5dd6cc03b2316d0d4aa29f6304aace5778 authored almost 10 years ago by Christopher Foo <[email protected]>
connection: Add proxied property to Connection

0ed96ff1d6bb779ed18a80367cb96b8ae4e841a6 authored almost 10 years ago by Christopher Foo <[email protected]>
http.stream: Call prepare_for_send only if it exists.

93a297a1a2fd161c29324686fdd9ad872e1f84b5 authored almost 10 years ago by Christopher Foo <[email protected]>
connection: Add host_key arg and store key on Connection instance

d99f844c635083efcb12211aa6a3d39dec2965c8 authored almost 10 years ago by Christopher Foo <[email protected]>
connection: Add sock arg and start_tls().

c9c3882fb2aeb40c9679a35ab1e9d9f99b479c98 authored almost 10 years ago by Christopher Foo <[email protected]>
connection: Rename 'ssl' arg to 'use_ssl'

Avoid shadowing the ssl module.

696614d5b355c63dddc161c3314bda6bb278f5e4 authored almost 10 years ago by Christopher Foo <[email protected]>
proxy.client: Check for None before using authentication var.

Fix up check_out acquire renames.

aebc5dc7045e701339197ac29a4ab32f493b74c4 authored almost 10 years ago by Christopher Foo <[email protected]>
setup.py: Add missing proxy package

0ddeb123e64edf2ecc5063f1da562ad0b163784a authored almost 10 years ago by Christopher Foo <[email protected]>
manifest.in: Add missing test and freezer files.

0a834e681ceaf35cce006c50b6646cea27b90bde authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Regenerate proxy docs

d8a4f6a3f1c456ac003f15f2263187e268466caa authored almost 10 years ago by Christopher Foo <[email protected]>
Move proxy modules into proxy package

8944fde89e80bcaa7cf110e39bf853c7d0ae6805 authored almost 10 years ago by Christopher Foo <[email protected]>
abstract.client: Use dummy context manager to remove duplicate lines

7ae8ab5abadb59ad661663bfa8edfe874545a286 authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Document missing no_proxy

[ci skip]

6d58198382e64bd9d049bcd1687de6ad17690575 authored almost 10 years ago by Christopher Foo <[email protected]>
connection_test: Handle num connections may be lower than max

21b1e1dd8249869fe811c3720070af35032bde43 authored almost 10 years ago by Christopher Foo <[email protected]>
database: Replace question mark with underscore in db path.

There is no mechanism to pass a path containing a question mark to
Sqlalchemy because it use URL...

55f9aa090ac5026ddf3b1b4c82788809d0acc790 authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Note phantomjs console is not logged to WARC. Add link to spec.

170e009435f4bc5dc5453410f9d7c45bb88e0d8c authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Update example with --sitemaps and --session-timeout

[ci skip]

95c2be348e7dac9670483e5f70421fde91c6b6f1 authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Note what http compression is supported

[ci skip]

ce92fee95781854e06e18f5267e38f2597dcb6c0 authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Mention wpull is generally not a library

[ci skip]

46e32952120068b569ad436492178e85eb419de5 authored almost 10 years ago by Christopher Foo <[email protected]>
setup.py, readme: Remove the beta label.

[ci skip]

6a2604f6ad1f374b9e5ddd33c14b0395d464688f authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1009

24863860e093f806bf1478b5a16cfbeb50022be3 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge commit 'f4686f6008a59b40df80a4cd186162e451dbc5e1'

Conflicts:
wpull/version.py

40b8c45b4ef40a371d52de6ab209770392048b3f authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1010a1

cc54f73de5115a51af5cbf64980846d049a6ce2d authored almost 10 years ago by Christopher Foo <[email protected]>
changelog: Update latest to 0.1009

[ci skip]

f4686f6008a59b40df80a4cd186162e451dbc5e1 authored almost 10 years ago by Christopher Foo <[email protected]>
changelog: Fix code syntax

[ci skip]

58fae291d9b0323c405e16e695902b4db8291ae9 authored almost 10 years ago by Christopher Foo <[email protected]>
doc: Add list of "deficiencies" in Wget.

[ci skip]

d2e68251c0d2aba4bea07458d9e3d4c2751304d4 authored almost 10 years ago by Christopher Foo <[email protected]>
doc/scripting: Add link to ArchiveBot as example

[ci skip]

cdfbb43c3ecd9cff244e2a77dfb0f2a95d43dce4 authored almost 10 years ago by Christopher Foo <[email protected]>
changelog: Start using a consistent format

[ci skip]

4adfdb74c134b74982f82de52e6b983e68bb58e1 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge branch 'issue/182-connection_pool_synchronization_2' into develop

Closes chfoo/wpull#182

98b849797e2f6a133f0b1c44aec4f67dd4889097 authored almost 10 years ago by Christopher Foo <[email protected]>
test/fuzz_fusil_2: Use random concurrent

[ci skip]

4bd7b8be287d4d967ba1cc6a6621fdb37c3c0d2d authored almost 10 years ago by Christopher Foo <[email protected]>
Fix up clients for ConnectionPool acquire/release function rename.

5920f136b5b420688be334644acce9c08c361e40 authored almost 10 years ago by Christopher Foo <[email protected]>
connection: Rewrite pools w/ Condition. Rename funcs to acquire/release.

Rewrite the connection pool using Locks and Conditions.

Rename the functions to acquire/release...

faafb741c5da3bc0051917b65a0e43fcfc5f5d5a authored almost 10 years ago by Christopher Foo <[email protected]>
recorder.warc: Compress temp WARC log

239ad69c5ebff918211872d94297e5858537c164 authored almost 10 years ago by Christopher Foo <[email protected]>
util, warc: Add seek_file_end

7d88816b0e9ca816b1baa7356e88e21cfd9e9e15 authored almost 10 years ago by Christopher Foo <[email protected]>
hook: Add v3 and wait_time with more context

ftp, web: Move get_wait_time into fetch so args can be provided

Closes chfoo/wpull#227

531e06da4f8b9e5a291b31fe964cc468f84fc8de authored almost 10 years ago by Christopher Foo <[email protected]>
testing: Rename hook script http_info arg to response_info for clarity

c577e64f674c4fbb77b2af7bb84a2c715bad1461 authored almost 10 years ago by Christopher Foo <[email protected]>
processor: Remove unused action variable

76e6bfd5d632bc2fe309232918f7b515552bf297 authored almost 10 years ago by Christopher Foo <[email protected]>
fixup! app: Remove ValueError returned incorrectly as parser error.

[ci skip]

6ecec1cfdb1eacc58ec51befbede4f02c66eb069 authored almost 10 years ago by Christopher Foo <[email protected]>
app: Print scary tracebacks only on generic errors.

Closes chfoo/wpull#237

77564c9ac8cc3a532a6dd0d2712322ed4c3e51cb authored almost 10 years ago by Christopher Foo <[email protected]>
app: Remove ValueError returned incorrectly as parser error.

bde9a39109cb0d9c0b356078d78dde5fbdcebd43 authored almost 10 years ago by Christopher Foo <[email protected]>
Raise AuthenticationError on failed FTP login.

008f585422cda77af35f82e2c385ff75cd6deb8e authored almost 10 years ago by Christopher Foo <[email protected]>
fixup! Implement --preserve-permissions.

re chfoo/wpull#222

7633538d95466289c6cb37b3500740bde5d792d9 authored almost 10 years ago by Christopher Foo <[email protected]>
Implement --preserve-permissions.

93dcf09009582f24f986b4d2d64a1b25a4d4c465 authored almost 10 years ago by Christopher Foo <[email protected]>
ftp.ls: Support parsing file permissions

d3e0568a21fcfe89281c3ac3eb127028d99f3742 authored almost 10 years ago by Christopher Foo <[email protected]>
body: Fix docstring for accessing file size

5cc1909968595a29503d050365352b139d9da398 authored almost 10 years ago by Christopher Foo <[email protected]>
Add doc for None attributes. Cross ref scripting hooks docs.

Closes chfoo/wpull#235

b00cb3e3fb2171282d7d50d3f2178166df8b76b4 authored almost 10 years ago by Christopher Foo <[email protected]>
hook: Refactor Callbacks into dispatching class with CallbacksV2

b63227ae883513d9faa3bf9ee142036116c8a841 authored almost 10 years ago by Christopher Foo <[email protected]>
fixup! processor.ftp: Implement file vs directory checking.

0af816ef5907abd51704e81ab9bb7703fbd9b2a2 authored almost 10 years ago by Christopher Foo <[email protected]>
processor.ftp: Implement file vs directory checking.

re chfoo/wpull#222

18f1bb053beacbb8f35015c4eda34eda90101d33 authored almost 10 years ago by Christopher Foo <[email protected]>
item,sqlmodel: Add file and directory types.

b9b614bb177a4b55b3b24d37d40766ee1da5c6c8 authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1008

32837d7c5614d7f90b8242e1fbb41f8da9bc7ce7 authored almost 10 years ago by Christopher Foo <[email protected]>
Merge commit '943a5ecb1ef178b3ee2ee2cf9d652db065cbdec2'

Conflicts:
wpull/version.py

cc9501589d93ae30831595fef39bb69b450c6d69 authored almost 10 years ago by Christopher Foo <[email protected]>
Bump version 0.1009a1

20eaa7eeab6179e259cd6b3b27ede469b2bb287f authored almost 10 years ago by Christopher Foo <[email protected]>