github.com/ArchiveTeam/ArchiveBot commits | Ecosyste.ms: OpenCollective

Correct dict iteration error

6c5e5ccf701ddc19e93c50eb26af909f3d0827ae authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct compile error

b58c6855c3aa6a03cd31db5c11a4e861a0cfa362 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Version bump

f3f4a000939f23f59630e59d9a5da5d8e57f50eb authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Modify logging thread to use non-transactional redis pipelines, properly accrete incremental values, no longer be daemonic, and attempt to finish sending logs shortly before unregistering the pipeline during clean termination; discard less logs;

batch log commits into about as many as can be generated during one dashboard round trip (guessed...

a90f33d74520187443ea7ab54aca30ec055c8e3b authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Bump pipeline version.

0ba68b1c3c83338f7ed58958281f0f1de2d349ac authored about 8 years ago by yipdw <[email protected]>

Merge pull request #227 from falconkirtaran/master

Limit transmitted filename and transmitted bucket DNS component to 64 bytes each for S3 uploads

608056266c2d5b1a6ea3575a42bd9ab08239d651 authored about 8 years ago by yipdw <[email protected]>

Double length to 64 for uploader filename/DNS components

7b066790818fa5192086845a47a06d172c377798 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Limit DNS names and filenames to 32 characters for IA S3 uploads to prevent 400 errors on uploading

4a5727bc7e26e649e93ebf091f1fcb12af0e3a85 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge remote-tracking branch 'upstream/master'

ad3cf9a88ae0ea6613ac9bcc11f4eb15e535fb4f authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

dashboard3: Regenerate dashboard3 from Haxe code.

caed894f0a334c652c77ebf1152fed97dc888be6 authored about 8 years ago by David Yip <[email protected]>

dashboard: Use scheme-relative URLs and protocol detection for HTTPS/WS.

b263ded8d9bebdec62254173bf7766a8029e7eb1 authored about 8 years ago by David Yip <[email protected]>

dashboard3: Remove Safari-specific onmousewheel case.

https://github.com/HaxeFoundation/haxe/issues/4005 mentions that the
onmousewheel export was rem...

81413cf4ec6acd094ab9d1337e96ca1fd27af742 authored about 8 years ago by David Yip <[email protected]>

A Dockerfile for ArchiveBot's backend.

ed6679062228ee346d1fbbf7688cebe967a12387 authored about 8 years ago by David Yip <[email protected]>

Also update plumbing's version of json.

902276bb74d9bc0b49b6deb0c2084d1c0f9e3a6a authored about 8 years ago by David Yip <[email protected]>

Update json to 1.8.3.

The native extension for 1.8.1 runs into a compile error on the latest
release of Debian jessie.

c631c8b6e7de2b549947a6959be106919bd0ade7 authored about 8 years ago by David Yip <[email protected]>

Set up a skeleton wpull 2.0 plugin for ArchiveBot. #219.

To use in wpull 2.0, run

$ wpull --plugin-script=archive_bot_plugin.py http://example.com

...

23a6712f20d5eba7d2483051a52b57e77f9ccab0 authored about 8 years ago by David Yip <[email protected]>

Opt out of Chrome's misbehaving Scroll Anchoring

(from https://github.com/ludios/grab-site,
commit ec5cc3f287fe67c2437718d3d7e8ddf591d0a76d)

301ea0a7061f5d965988540fe66692a2ac2ad6b2 authored about 8 years ago by Ivan Kozik <[email protected]>

Merge remote-tracking branch 'upstream/master'

4eacdc0518f907222d1afcb919039bd786a5241c authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

Coalesce counts in log shipping to reduce memory and network requirements

3edf6ea2d8617eb8de0c85d58a95f678d1b6db7f authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge pull request #226 from Frogging101/master

uploader: Fix UnboundLocalError when not in s3 mode

9b4048f4d22d4a58aa3b0ef40725dab5bb757b8b authored over 8 years ago by yipdw <[email protected]>

uploader: Fix UnboundLocalError when not in s3 mode

The code tried to use ia_item_prefix even if we weren't in s3 mode, in which
case it was unset.

483bd51eade3e5edcae8367cfd57dc8bb237778c authored over 8 years ago by John Brooks <[email protected]>

Bump pipeline version.

e00d5549e39f2563945d51bfeafad4e034980e9c authored over 8 years ago by David Yip <[email protected]>

Merge pull request #224 from falconkirtaran/master

Async shipping of logs from pipelines to redis

b2f62d22a4f952093b92a34506c4ac3bebd3a694 authored over 8 years ago by yipdw <[email protected]>

Put the async log shipper in charge of maintaining file and byte counts too, and factor away unneeded state

8448e841d667174f11158760a134031e9317b418 authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

Fix syntax from another planet

802606342d105b84916e163dbdde0ab15b994d08 authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

For some reason it is already passing self.

6e79807a38b66236d38a6e101b42a50056de506e authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

typo

4555d8661c918456633fa9b59b4b01d478ed9992 authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

Fix bug

dd85abed23222651fb5e943b5ba440a6ceb2e220 authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

Use threading to make shipping logs into redis async with a shared queue

337238335b76fb87d09159c362c75d00a80bcec8 authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge branch 'master' of https://github.com/ArchiveTeam/ArchiveBot

08bef41055d7424c276d77e8722ca62effa778ac authored over 8 years ago by Falcon Darkstar Momot <[email protected]>

Eat trailing whitespace on bot commands.

faa63b29da9bc385fcb6e7c8a728defb4d6a040c authored over 8 years ago by David Yip <[email protected]>

Pylint: fix variable name typo in assertion

db0977841e127411b9c823a2634ebe1c67a50cce authored over 8 years ago by FalconK <[email protected]>

Merge pull request #209 from falconkirtaran/master

Support for uploading directly to IA

5f534603e2cf39f5c47221a3779ba48186062706 authored over 8 years ago by yipdw <[email protected]>

Merge pull request #220 from Frogging101/master

Expand global tumblr ignore to cover other subdomains

2c7e33038a4cb767986e32ccbaff4688c52fc4f2 authored over 8 years ago by yipdw <[email protected]>

Bump wpull dependency for pipelines to 1.2.2, ref. wpull issue 289 inter alia

09800d612f9cd6cba3a62ccf1e85be7c0666c2f5 authored over 8 years ago by FalconK <[email protected]>

Add option to change WARC size

bbaa19b4b326da88bd899107a42bb20b960b8bae authored over 8 years ago by FalconK <[email protected]>

Merge remote-tracking branch 'upstream/master'

1f1f4218be736d555f973878863b5824b1116d31 authored over 8 years ago by FalconK <[email protected]>

Expand global tumblr ignore to cover other subdomains

"^https?://www\.tumblr\.com/(impixu\?|share(/link/?)?\?|reblog/)" was failing
to match URLs such...

6ef63ab16cfe5d52493b67eb21b65a1e969fdc2e authored over 8 years ago by John Brooks <[email protected]>

pipeline: Allow phantomjs 2.1.1. Bump version. Bump min seesaw version.

Closes #213

890c99d072520d490b2f8f7f71b85dbfe7ee8ebb authored over 8 years ago by Christopher Foo <[email protected]>

Be more aggressive with throttling for direct-to-IA uploading: evaluate whether this particular upload is likely to be throttled. Upload if at least 5 bucket tasks, 10 API key tasks, and 10 global tasks remain in the quota, even if throttling is on.

b8ac1eda69a3e4606b55ba8e2101b4593abc1bc1 authored over 8 years ago by FalconK <[email protected]>

Add exception code to deal with inability to query from IA for

throttling stats

4ee277e43d8bc7dffb9ed26e84a61b493701e9d5 authored almost 9 years ago by Falcon Darkstar Momot <[email protected]>

Uploads to IA wait until rationing is not engaged and we are not above our limit

a8119ed9fcc5aef2eca0b5d4a2e25eb15aa4222d authored almost 9 years ago by Falcon Darkstar Momot <[email protected]>

Minor fixes to upload command, and don't replace . in basename with _

8a136e013f77e11d0031706046f01dc60525e09c authored almost 9 years ago by Falcon Darkstar Momot <[email protected]>

Add subject:archivebot to identify items uploaded to opensource

b194e5ff14a7eba27a1ee9a6746ceea34c2bf927 authored almost 9 years ago by Falcon Darkstar Momot <[email protected]>

Dates comply with IA metadata format

1f681b3a9f5fd9f7673f6aa2827583c03b6cbe55 authored almost 9 years ago by Falcon Darkstar Momot <[email protected]>

Minor change to how target URI is constructed; tests OK now

8199d6a2cfe199075784edba43921509306ffe63 authored almost 9 years ago by Falcon Darkstar Momot <[email protected]>

Preliminary (untested) support for uploading directly to IA

217064195aeb8b43c17a197f5362bce231aea67f authored almost 9 years ago by Falcon Darkstar Momot <[email protected]>

Merge branch 'JesseWeinstein-viewer_pywb-ia_links'

fb4b80229d13f9b9d8b0f02a1f61e7e06477a5d6 authored almost 9 years ago by Christopher Foo <[email protected]>

Add link to archivelab WARC viewer

eb43136343c2934628f437b70dfbf4ca3e37c338 authored almost 9 years ago by Jesse Weinstein <[email protected]>

update safari user agents

0a61df2944a2c1ae5a65d3de4909b11550d10755 authored almost 9 years ago by Start <[email protected]>

readme: Note existence of & recommend grab-site

2ee59a36978e89df08398984376a43a4ee086327 authored almost 9 years ago by yipdw <[email protected]>

Merge pull request #206 from Asparagirl/patch-5

Update forums.json

8521462d459d0cfa57405004878a2d4cc0d6db35 authored almost 9 years ago by yipdw <[email protected]>

Update forums.json

Additions to the ignore set for forums.

a8e6e3896d9a2352713d551f6089baf0431c6e31 authored almost 9 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #205 from JesseWeinstein/tumblr_ignore_sets

Add ignore sets to block tumblr notes & tumblr avatars in notes

39931a6583b4ff137f92a417fc62682fb5ac4515 authored almost 9 years ago by yipdw <[email protected]>

pipeline: Bump pipeline version. #202.

18cfcd57f25f0b3c3a526c1bf8789dfe77d259d1 authored almost 9 years ago by David Yip <[email protected]>

dashboard: Don't print "undefined" for missing ignore sources. #202.

Existing pipelines do not report the ignore source, and pipeline
upgrades do not occur instantan...

6e5a3cf49d6f01910bcba58144b03cdbe7fb6a2f authored almost 9 years ago by David Yip <[email protected]>

Merge pull request #202 from JesseWeinstein/add_ignore_source_display

Add dashboard display of the source of ignores

ee71140ced4e4f78cc7c05c6b60a60c18f71be7c authored almost 9 years ago by yipdw <[email protected]>

Fix typo

3be97866a48380c8a8e89692137a5166055a072c authored almost 9 years ago by Jesse Weinstein <[email protected]>

Add ignore sets to block tumblr notes & tumblr avatars in notes

d3b30c97d67196f653915743a2c4544331525dc6 authored almost 9 years ago by Jesse Weinstein <[email protected]>

Move source into IGNOR span, per ivan

ffa42b87544961a12bc939afb6b81c5e66bd7498 authored almost 9 years ago by Jesse Weinstein <[email protected]>

Put source before url, to avoid having to scroll

6b001a27c55bf44683102edc05baec8f88786ce6 authored almost 9 years ago by Jesse Weinstein <[email protected]>

Make source arg required & more verbose

471770d8b6e970f1a27764f6af2999b8c899e5ee authored almost 9 years ago by Jesse Weinstein <[email protected]>

Add dashboard display of the source of ignores

e052b1348967d173c935b6824a8f95875115faaa authored about 9 years ago by Jesse Weinstein <[email protected]>

Merge pull request #201 from JesseWeinstein/option_aliases

Fix aliases for options with arguments; add tests

a067d0b7fa49b4d4c422ef359de6c7169c97d0c7 authored about 9 years ago by yipdw <[email protected]>

Convert all the job_options_parser tests to expect().to form

5866302258aba6d826382e8c0847251ab9615cb5 authored about 9 years ago by Jesse Weinstein <[email protected]>

Remove leftover

0dd10185228186ef90343973a4a1556109c9ba7c authored about 9 years ago by Jesse Weinstein <[email protected]>

Fix aliases for options with arguments; add tests; fix tests for really old version of rspec

6298b60222c8d45a9308e04fa331fa005eb76923 authored about 9 years ago by Jesse Weinstein <[email protected]>

Merge pull request #200 from JesseWeinstein/option_aliases

Add aliases for some job options

4fd5d79799419f1bc7e5aa730d6851ab7c5c73a7 authored about 9 years ago by yipdw <[email protected]>

ci: We ought to be able to run on Ruby 2.2.3.

I think Ruby 1.9.3 entered The Badlands.

1a78905a3c1558ab49f63020d30491c49628aa82 authored about 9 years ago by David Yip <[email protected]>

Add aliases for some job options

b6765be4eb75dde8787c1908bb9bce065029c644 authored about 9 years ago by Jesse Weinstein <[email protected]>

Merge pull request #195 from JesseWeinstein/viewer_search_warning

Add warning that URL paths are not searched in the viewer

dbf69362122fa25421a044ff365fa2b1214645a3 authored about 9 years ago by Christopher Foo <[email protected]>

Add warning that URL paths are not searched in the viewer

5a31bc24672822b3a512799f9b03844c1dd10326 authored about 9 years ago by Jesse Weinstein <[email protected]>

Added additional ignore patterns for Russian MediaWiki installations.

Ignore patterns for Russian MediaWiki installations. I’m not actually
sure if this (“Uchastnik_b...

c6ddc518d92d9b5be61d28de40c08deb108e5789 authored about 9 years ago by ethus3h <[email protected]>

Merge pull request #190 from JesseWeinstein/avoid_spurious_authorized_check

The hooked call to add_ignore_sets doesn't need to re-check authorization

15d15a87e3ce70f523bca6874b72a5d8417590e5 authored about 9 years ago by yipdw <[email protected]>

The hooked call to add_ignore_sets does not need to re-check authorization

0d8ba1968667753d4f202574d92da304eb40e656 authored about 9 years ago by Jesse Weinstein <[email protected]>

Merge pull request #188 from JesseWeinstein/add_space_around_URL

Add a space between the URL and the surrounding parens

8d403fef7b212392a2bd600b79c00a4d094370db authored about 9 years ago by yipdw <[email protected]>

Switch to < > due to visual preference

b569fc7bb2f2a9298207451d5364fb1787763963 authored about 9 years ago by Jesse Weinstein <[email protected]>

Merge pull request #187 from Asparagirl/patch-4

Update INSTALL.backend

f42864d7792d2147f877a607243fb0471647d628 authored about 9 years ago by yipdw <[email protected]>

Add a space between the URL and the surrounding parens to make various auto-linkers happy

77047c269455a70ff75ab09b83938268c7bdc2e9 authored about 9 years ago by Jesse Weinstein <[email protected]>

Merge pull request #186 from Asparagirl/patch-3

Update INSTALL.pipeline

8512820664015c46958b98e2339bc65abc05d988 authored about 9 years ago by yipdw <[email protected]>

Update INSTALL.backend

ee7a2158dfa61bd4476de8030ebfaf4ce0ca4da7 authored about 9 years ago by Brooke Schreier Ganz <[email protected]>

Update INSTALL.pipeline

Documentation updates and explanations, based on my experiences getting my pipeline set up.

5d5637d05eed22b56309568888d241b50af5388e authored about 9 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #183 from Asparagirl/patch-2

Update INSTALL.pipeline

f53faefe2f0e312c53456d2e1941dcf1e82ba2e9 authored about 9 years ago by yipdw <[email protected]>

Update INSTALL.pipeline

Added youtube-dl installation info, and minor text tweaks.

8a3869587b78ba83e8d07fcd612df122b85c8cad authored about 9 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #181 from JesseWeinstein/doc_fixes

Add links to db/user_agents; a few other minor doc fixes

60a15b0a54079ec61d07a02825ca97d76942018b authored about 9 years ago by yipdw <[email protected]>

Add links to db/user_agents; a few other minor doc fixes

8e7b4b0c3dac97dfc7d5ee7b4623bd5c8581d533 authored about 9 years ago by Jesse Weinstein <[email protected]>

Merge pull request #180 from JesseWeinstein/add_links_to_ignore_patterns

Add links in the documentation to the list of ignore patterns

aede954c417681ec3b4ef6391ebe2b90b560bf2f authored about 9 years ago by yipdw <[email protected]>

Add links in the documentation to the list of ignore patterns in the code (at GitHub)

85a0833f09ddf1062b0841c4e8184e2e39d28df8 authored about 9 years ago by Jesse Weinstein <[email protected]>

Merge pull request #177 from garyrh/twitter_igset

Fix twitter ignore set.

1232cdc3d882bb07a6e54543c8a21cc75bedc8cf authored over 9 years ago by yipdw <[email protected]>

Fix twitter ignore set.

060d0514e6847e56c44f60891d65b679f4fcfc57 authored over 9 years ago by Gary Herreman <[email protected]>

install.backend: Add in the grep steps to match travis config.

[ci skip]

8fd3ab7e05d73edc8828816ac4e72fcb58d3f96b authored over 9 years ago by Christopher Foo <[email protected]>

bot: Use Shellwords.split to handle --opt="a b". #139.

8cb8258a3db386e87f839288315a26fe2f50f924 authored over 9 years ago by David Yip <[email protected]>

Apply comments from pull-request

4d463a4a18fc18db68ba91b3a8c893de70866c10 authored over 9 years ago by Jesse Weinstein <[email protected]>

Add --concurrency option to !archive command

3d796b29b1851a0d0712d5f282e0a2124de56589 authored over 9 years ago by Jesse Weinstein <[email protected]>

Add --delay, --min_delay, and --max_delay options to !archive command

d0d52a42f4d4cf353c693092cecc53888bc8376a authored over 9 years ago by Jesse Weinstein <[email protected]>

Add --explain option to !archive command

(and two trivial comments)

b701a1a85df69afef87f90d198b8ab5923d724ec authored over 9 years ago by Jesse Weinstein <[email protected]>

bot: Require +o to use !expire.

Some people seem to think that this is meant to be used frequently as a
way to get crawls on the...

28f3014fa3ec58073dc68f894193ba49d176029e authored over 9 years ago by David Yip <[email protected]>

Remove questionable /(.*)/(\1/){3,} ignore

51b650e9a1b74724f858f65db84a195a71239117 authored over 9 years ago by Ivan Kozik <[email protected]>

Import the dashboard killer

971c8c9b3f470a0e73e44dfb35e3368b9ffb6454 authored over 9 years ago by Ivan Kozik <[email protected]>

Unbreak archivebot.seesaw.wpullargs_test.TestWpullArgs

b6699abaf4e9fcfe5c4889cc21e56139d25223ec authored over 9 years ago by Ivan Kozik <[email protected]>