github.com/ArchiveTeam/ludios_wpull commits | Ecosyste.ms: OpenCollective

Bumps version to 0.21.

Merge commit '40f0a'

Conflicts:
wpull/version.py

2e8c9598c1ab984a23287187294e7d33942a52a6 authored almost 11 years ago

Bumps version to 0.22a1

83317452951f424d393764c29724c107317fa29a authored almost 11 years ago

changelog: Updates latest to 0.21.

40f0a4de36ae93ca94df3a209d6f1614d30146fb authored almost 11 years ago

Merge branch 'topic/phantomjs_integration' into develop

Conflicts:
doc/changelog.rst

de001d3b0ca91c099234bc7544708c9d88c6e41c authored almost 11 years ago

Updates documentation about PhantomJS.

dcaacd69c9f4bd965b1ed24d221754b11f72424c authored almost 11 years ago

Implements --phantomjs.

0ba02370053fb614346dbcdc7bdf89ef0061aaca authored almost 11 years ago

phantomjs: Supports setting default custom headers.

773347a23856b7600de76c485f6c76b30ebe28d4 authored almost 11 years ago

proxy.py: Adds debug statement.

e423e2caa4cdf976542828a6d9a7d3fd2c0d67e2 authored almost 11 years ago

phantomjs.py: Fixes proxy address. Adds return_code property.

45ed4d0f82dd65084e4dfbac58e41f755f006064 authored almost 11 years ago

processor.py: Fixes request factory not being used.

0108f72545e79088dd0e049acb43f075e84bc961 authored almost 11 years ago

phantomjs.py: Fixes terminate(), remotes sets, termination.

10b2d544f7f5b64ec5530956ecb844b6ad03934c authored almost 11 years ago

phantomjs_test.py: Increases timeout for Travis CI.

0f8fd74d02aa5cd4ab1300f00f70433129a19858 authored almost 11 years ago

phantomjs.py: Adds PhantomJSClient.

fcf5226c562a104ecbcffa865b4e9363063a5f95 authored almost 11 years ago

phantomjs.py: Supports default page settings.

1a2b572e0774557fc9b1ee5d597610efec5d42c1 authored almost 11 years ago

phantomjs.py: Renames PhantomJS to PhantomJSRemote

cf730dede7625f09604cb10eefed2895a9fe5abd authored almost 11 years ago

phantomjs.js: Implements page_reset().

b6d092231da23c6cebb5b27342f40dbd3de526c7 authored almost 11 years ago

phantomjs.js: Fixes setting of variables.

cc4ba0898fc18c894d35f07cc27e289b2117d437 authored almost 11 years ago

Modifies WebProcessorSession._should_fetch to use argument.

9e42d5419118ae585751583eef83302981e7699f authored almost 11 years ago

phantomjs.py: Supports receiving page events.

16aab7d48b2f34f8e7ca58ad6f647ce6a9a97ffe authored almost 11 years ago

phantomjs.py: Allows extra arguments.

1d6193e40b34aaf1e9fc9f67ca8a95f8527c185a authored almost 11 years ago

Fixes docstring references to http module.

0364f95f7cf904174177641564ee8f1f35643710 authored almost 11 years ago

phantomjs.py: Fixes to use tornado.gen.Return.

eea78fb51dfe86019dba5eb23dabf42bee400922 authored almost 11 years ago

Adds phantomjs module.

7250eb660adea84c4c9ad45d692887927e0491bf authored almost 11 years ago

proxy.py: Invokes the future so errors will be logged.

13cfcc51014f7045c267b75d2b95f168a87c1bf8 authored almost 11 years ago

engine,http.py: Invokes the worker futures so errors are logged.

a18ae25d83c46e5156c15b2a9849b1a89707ac04 authored almost 11 years ago

changelog: Updates recent changes.

e8086126013a4e665ae883ff65b8f8178e0ddef1 authored almost 11 years ago

proxy.py: Adds ProgressRecorder on debug runner.

baadc1123fa213af87ca480b999623eede0e8642 authored almost 11 years ago

http/client.py: Allows Recorders on initializer and fetch method.

ab2ba223fc17b6564b6c39eae17bbaf55b511b9a authored almost 11 years ago

Updates backport.conf for http package.

94609b5b31b493288b0c2e9b98513d53aacb1155 authored almost 11 years ago

Splits http module into http submodules.

Splits into modules http.client, http.connection, http.request,
http.util. Moves web module into...

a83bf000b204a72b7c527436b990799abb7374ba authored almost 11 years ago

proxy_test.py: Moves tornado curl_httpclient import to later.

844abf2b6e82a3ff5ffd3ee821995914cc2827dd authored almost 11 years ago

Makes TestProxy tests optional for now.

d4439a2c09f7fdf66ee728b105ed52e9c4f8167e authored almost 11 years ago

Fixes travis config with sudo for pycurl install.

3d28375d16e4d29928690d47e19ef856dc3e9e96 authored almost 11 years ago

Fixes travis config for installing pycurl.

14f989ffb6cbba4088fb7a677f678b61ae7dbc2f authored almost 11 years ago

Adds proxy module.

8986ffa216ead4bea24d66dec0166ccbba735fcb authored almost 11 years ago

engine,http: Adds the worker futures to the IOLoop to avoid losing them.

Will this finally fix chfoo/wpull#27 ?

517b5a27b4f20c3e31db495a6437a305c5e9ba24 authored almost 11 years ago

http.py: Removes unneeded use of Task.

bd784d0eef7758ad69b46f067fb4c88e3e2c1b08 authored almost 11 years ago

extended.py: Makes write a coroutine.

ce220df6d98a8a6599409e32bb943936a2b24cfd authored almost 11 years ago

http.py: Refactors chunked transfer logic into ChunkedTransferStreamReader.

fcab68fe9de55b0e6a2c2756c9a4d6c12f62342d authored almost 11 years ago

Bumps version to 0.20.

d707b471195da306ef1537c3f5539225705a7d80 authored almost 11 years ago

Bumps version to 0.21a1.

dd74d181bdf32577e044b651da79a512dde3cd3e authored almost 11 years ago

Merge branch 'develop'

02844aec01c2e5a182d2dd3465e28dff5099c146 authored almost 11 years ago

changelog: Updates latest to 0.20.

e5058fc7d5b223097f7fc6fbecc4695147abca0a authored almost 11 years ago

http.py: Don't raise exceptions if not active (chfoo/wpull#27).

c5cb958d0ea34f35116d5746a5976e95b2897859 authored almost 11 years ago

hook.py: Fixes handle_error AttributeError.

6b490fa30df362b93316f38833a6976e56c71ea8 authored almost 11 years ago

Updates terse_options.txt

6f2115d66a6b40b8e9a84eca414d0f8b3b2120ff authored almost 11 years ago

Implements --accept/--reject.

52ebb55d93e30e75a0f3c0368007911d7f7bd13e authored almost 11 years ago

Implements --no-dns-cache.

0acb2ef8c5e56224d848de8c6328003444157ab1 authored almost 11 years ago

http.py: Relies on IOStream's closed state instead of ourself.

bf8a7483692db385a4599b72d047314ec1c5ded1 authored almost 11 years ago

Bumps version to 0.19.2

ef9c6d1ae63a7e79779162ed4f905f46738be8ab authored almost 11 years ago

Merge branch 'develop'

8dd3f20dda03cde7b21a55951dab836d0177af95 authored almost 11 years ago

changelog: Updates latest to 0.19.2.

0534c8a5c09fb2b9ba85fa5e94d103e24e8fe37d authored almost 11 years ago

hook.py: Fixes booleans converted to integers on Python 2.

0d772074536962c1516d906b4fa1d54c98dcac6e authored almost 11 years ago

processor.py: Fixes crash on non-HTTP URLs w/o testing filters.

Don't create RichClientSession until filters passed at least once.

940a1cfb492182ee4e136b0934023fae52655292 authored almost 11 years ago

Bumps version to 0.19.1

Merge branch 'develop'

Conflicts:
wpull/version.py

4d9fe275644adf9cd4bbd60143831166d7736261 authored almost 11 years ago

changelog: Updates latest to 0.19.1

d05367b8880db5f715c62291a8a685dcf5e2de77 authored almost 11 years ago

changelog: Adds entries about timestamping and http no content fixes.

77acc6c54bfdeb6415fb533484fd6ce173d5cb8f authored almost 11 years ago

writer.py: Fixes Timestamping writer. Implements checking .orig files.

02c385225be05a563f4612ef12fa94924f8c5b7f authored almost 11 years ago

http.py: Implements handling of no content responses.

e00a0211711f1dc1022217ba6735853dce235188 authored almost 11 years ago

converter.py: Use copy2 instead of copyfile.

8641e2076d95a4fa13876f0e3f4b11a88410f497 authored almost 11 years ago

Bumps version to 0.20a1.

fc7e3bea064211f9cca172e200e5c2fb1b409245 authored almost 11 years ago

Bumps version to 0.19.

4d1676ab9b740b3fdf75f954da128271af28a1f1 authored almost 11 years ago

Merge branch 'develop'

d6032af2e8254c915c9fde2a033f427f7049468c authored almost 11 years ago

changelog: Updates latest to 0.19.

872e96f90b32a7147beb083f6bab6a32e3cad92b authored almost 11 years ago

converter.py: Passes the encoding along to URLInfo.parse.

70e7f82f20bdc002dd304eadcc98fd3e3f90a87f authored almost 11 years ago

converter.py: Decodes strings from lxml before using for Python 2.

005b7fccc36bf3cb0e2c0c0ef803b26fb1f245f4 authored almost 11 years ago

changelog: Updates latest entry (files not written, --convert-links).

Closes chfoo/wpull#4. Updates terse_options.rst

03bb0e6a47e21e7b0c8c61073cd46409c202becb authored almost 11 years ago

Implements --convert-links,--backup-converted.

f03c2585b5c7661cc724cb7bb0a3a722b7c244fa authored almost 11 years ago

converter.py: Detects link_type if not provided.

575c0aed234fadb5e079d97ed418215e7e067432 authored almost 11 years ago

database.py: Fixes __iter__().

97b0228eec53fdc7162982867fc0edf1af753159 authored almost 11 years ago

web.py: Fixes files not written.

30d7560cdc3222c6a3d6e76af70aebb7c80b05cf authored almost 11 years ago

converter.py: Implements HTMLConverter.

0f05dd4fa39946630abdfe2c1b782b358d3b3ae2 authored almost 11 years ago

database.py: Fixes urltable contains.

950837d675518e19fc54a96d0dcd17b525b743df authored almost 11 years ago

scraper.py: Refactors link scrape functions.

ScrapedLink renamed to LinkInfo. LinkInfo adds new element and
value_type attributes. Scrape fun...

4aac48c94ae28237268d6b3a0469eaece095d267 authored almost 11 years ago

converer.py: Changes converts to inherit from scrapers.

0db7a3458b41d2681c7bccc4383dfde232855939 authored almost 11 years ago

document.py: Uses dummy abstract class method decorator on Python 2.

fc43c22f0c038fbb387a53944d5ca8f3b3c656a2 authored almost 11 years ago

converter.py: Implements CSSConverter.

ccea34d4b9c896ab49c0a6ae13fd2a2e3ba00868 authored almost 11 years ago

scraper.py: Moves re patterns for css to CSSReader.

6e55319afff8655db40ecfef5a88c82ad1b18238 authored almost 11 years ago

processor.py: Implements setting link type to css.

1d81d42bf928ef47176758d87bc0023749628015 authored almost 11 years ago

Moves doc type detection from scraper.py to document.py

9b32d986ffedcd6f8e51c3cceb1566ea35502f9c authored almost 11 years ago

Bumps version to 0.18.1

Merge branch 'develop'

Conflicts:
wpull/version.py

1a9822a34af94225dfc64f13c2dcd2e257574de5 authored almost 11 years ago

changelog: Updates latest to 0.18.1.

983ca538239d29e698970b49cefd3d3da215797a authored almost 11 years ago

changelog: Updates entry with note about CDX and database fixes.

4ce4676e8e41efaa1ad949d0682c9b441697b358 authored almost 11 years ago

database.py: Adds close() function. Fixes db not cleaned up.

bf031f59fa5807b2397629eb6c03eb1e65eff895 authored almost 11 years ago

recorder.py: Fixes error when cdx option is not specified.

8a2d93e91d1399de00c42f08d9f42e3d6354cfe6 authored almost 11 years ago

warc.py: Adds attribute docstring. Changes so bytes() can be used.

d2c6d8393638d22fc372d33b47ed92c4ac62e8db authored almost 11 years ago

Bumps version to 0.19a1.

50e0724047a1a6aaa12870eb2a1c12e6d291c55d authored almost 11 years ago

Bumps version to 0.18.

fd6ed32c6ace85e27368674be84f5dfd19f7d872 authored almost 11 years ago

Merge branch 'develop'

e81593dc2707a5a4f41f5b7e6c3f386037b842fd authored almost 11 years ago

changelog: Notes moved WARCRecord.

e1189e688ed55fd7d63a6ad4698cbd10d85e0e45 authored almost 11 years ago

warc.py: Fixes typo in get_http_header() docstring.

152bcd4db1f44cfdc715715df846fa9156d30401 authored almost 11 years ago

changelog: Updates latest to 0.18.

cae13d3f565a248bd99240133754665a1f3468dd authored almost 11 years ago

doc/terse_options.rst: Updates listing.

e9886561c391c5323d6d59b0e38e9671186d6722 authored almost 11 years ago

database.py: Optimize using core queries.

67e6779a54efe659531c2a64d4acf60a0441737f authored almost 11 years ago

scraper.py: Optimize _scrape_links_by_attrib() w/ likely shorter loop.

e122761b832fc5dde9d02d4656782688b0f2967f authored almost 11 years ago

url.py: Optimize is_percent_encoded() with constants.

ec1dd89e19dcdb4f35d9a1845a87aab69b4d7574 authored almost 11 years ago

__main__.py: Supports dumping profile stats.

307e8a87d8552077bdd5398b13eeb62732722b10 authored almost 11 years ago

testing/badapp.py: Fixes silent Python 2 error.

ec151e9c324d3fe0b985a2938338b2f40cb2c4ee authored almost 11 years ago

util.py: Fixes parse_iso8601_str() to use portable calc.

a35a7a87740192fe9761ac71f45bca328df70e14 authored almost 11 years ago

Implements --warc-cdx.

33bbe838d259c9b47a173e00a8f542234063ecc1 authored almost 11 years ago