github.com/ArchiveTeam/ArchiveBot commits | Ecosyste.ms: OpenCollective

Don't forget to apt-get phantomjs too, because it's required by pipeline

73b46d48eafae1d93534761c6461caace06ad2bc authored over 10 years ago by Brooke Schreier Ganz <[email protected]>

Merge branch 'master' into next

09c8305e44241a7c011dea70ebd067fe59397850 authored over 10 years ago by David Yip <[email protected]>

bot: Support password-protected IRC servers. #112.

2159cde7df44e495a2948467319ae0fd047ecfe9 authored over 10 years ago by David Yip <[email protected]>

Minor text stuff in INSTALL

ae87d7bef18005b8f3dc1c7d5ef8bcea9fac09ff authored over 10 years ago by Brooke Schreier Ganz <[email protected]>

Update INSTALL to add instructions about IRC server setup

e6d5ccf0bd13f763e22fefed00ce6ce8e737e163 authored over 10 years ago by Brooke Schreier Ganz <[email protected]>

Update INSTALL to add instructions on CouchDB setup

0443dfa8cd51ff745eac32a78eee471b0f0239c0 authored over 10 years ago by Brooke Schreier Ganz <[email protected]>

Update the INSTALL directions with some more information

b114a67e07544cd8163d6d907fb72253b81bca32 authored over 10 years ago by Brooke Schreier Ganz <[email protected]>

plumbing: Update INSTALL.backend for cogs/plumbing split.

4f13a510eb365ec0a46424e3184c47d9d7c7a4a8 authored over 10 years ago by David Yip <[email protected]>

plumbing: Scripts to start the log analyzer, recorder, and trimmer.

e6c385430c97d8b88e409700acb7bdb7c94d1e84 authored over 10 years ago by David Yip <[email protected]>

Ignore more JavaScript non-URLs

140c1f620afaf5a00e459a06407a312fbf7b9954 authored over 10 years ago by Ivan Kozik <[email protected]>

plumbing: Split recent-logs functionality out of log-firehose.

This simplifies log-firehose quite a bit.

1758e7b4de9749680fa633af10caf18561695b1d authored over 10 years ago by David Yip <[email protected]>

Reformat another long line

4240ce3dd93dc00ad6ec4d50c26c2ba0d7496742 authored over 10 years ago by Ivan Kozik <[email protected]>

Remove the use of ionice

I'm not sure -c 2 -n 0 did anything even with the cfq scheduler.
Most people have the deadline s...

1d2cc660cc528420c481a0b4feea27b727166500 authored over 10 years ago by Ivan Kozik <[email protected]>

Refactor some long lines

d3d12a00f2684a9a38590a7e962cb3eec47605e5 authored over 10 years ago by Ivan Kozik <[email protected]>

Bump pipeline version.

0863c31e5b78bb6b63e93de29e5fc88c7eac9522 authored over 10 years ago by David Yip <[email protected]>

Link to the correct Python docs

fabfc7e0e85a06330288b7d69ef6451d50a30fa9 authored over 10 years ago by Ivan Kozik <[email protected]>

Lock the upload directory so that another uploader can't start

uploading from it.

d115d14f36fd094e42ff9ff6c500a2c78831d82c authored over 10 years ago by Ivan Kozik <[email protected]>

uploader: Read WARC directory from FINISHED_WARCS_DIR.

This commit keeps the argv[1] behavior as a fallback.

We need to set FINISHED_WARCS_DIR for a p...

4e2105854738d7e1075c0695178ddc4547fc6dfe authored over 10 years ago by David Yip <[email protected]>

Document logging pipeline output with tee

We need to log somewhere to find out about wpull crashes

fb52d10a51f7868d6e15691f22674bd4b109af87 authored over 10 years ago by Ivan Kozik <[email protected]>

ignorednick -> NAME now that it's used

309ae661a9e7e9a0a32333289c53851aed3df283 authored over 10 years ago by Ivan Kozik <[email protected]>

Add note about sharing FINISHED_WARCS_DIR

602ff8dbe200773480325bc9f1387f111992d624 authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'one-uploader' into next

92c1c61bfe1f13fd77ca94d3d4a3441f7d8eb03c authored over 10 years ago by Ivan Kozik <[email protected]>

Make uploader.py executable

5b6b529b569362c9a4fdf9b0367e32a3ea89f843 authored over 10 years ago by Ivan Kozik <[email protected]>

Document one-uploader setup

1c55b950c6d88b4c497ba9e94b6966a31e379648 authored over 10 years ago by Ivan Kozik <[email protected]>

Use ludios/wpull hax-12

33d8fc1d30a20243dc81fd704f08733070292f2d authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'master' into one-uploader

c4d02db5482e33d59705bce1d6205b5008892a81 authored over 10 years ago by Ivan Kozik <[email protected]>

dashboard: Add links to pipeline and job reports.

Better wording and layout is totally up for grabs.

0b49481d23086c4d2ff1b8e21047e18fe0df8e9f authored over 10 years ago by David Yip <[email protected]>

Merge branch 'master' into next

1a746e34f23144c732ec5c8f7c1af7b81d0f4cbe authored over 10 years ago by Ivan Kozik <[email protected]>

Very minor dashboard status fix

2c1b37517bd5efc73314c5b3c9100244977cb169 authored over 10 years ago by Ivan Kozik <[email protected]>

dashboard: Close reports with </html>, not </head>.

80e735fa2e82fa19edfc3a75d6e925f5770dd516 authored over 10 years ago by David Yip <[email protected]>

dashboard: Explicitly request /logs/recent as JSON.

Firefox's default Accept headers trigger the HTML response.

1d93e55502ed2b0238560a6888262d4e903d521d authored over 10 years ago by David Yip <[email protected]>

Don't allow setting count to 0, which gets all of the logs

71add1747640fed63235fd1fd3ee8fd76ec5e741 authored over 10 years ago by Ivan Kozik <[email protected]>

Don't get all of the log messages by default

count was being incorrectly set to 0

4312e38a51a197994870e7d12828664d7f4e00cf authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'chfoo-topic-job-monitor' into next

b6d0e589a3e96c0cd6ec4cdc2fd68fadf9c30ab7 authored over 10 years ago by David Yip <[email protected]>

Add recent job log monitor page for dashboard.

ae1abb8ce7ac06070f7bad13b1f4ca4fce22ceed authored over 10 years ago by Christopher Foo <[email protected]>

Bump pipeline version.

Normally I wouldn't care about pipeline versions on next, but I'll be
running this pipeline in t...

3ae080a4491eaa8cf420006c3ecc78ac0d49a95e authored over 10 years ago by David Yip <[email protected]>

pipeline: Move to ludios/wpull@hax-12.

bc36e904f59c8dc25386dd625780d6ad7206fad9 authored over 10 years ago by David Yip <[email protected]>

dashboard: Remove stray newline.

5093f94274123da5bdeb5018a8bb27c078ca0826 authored over 10 years ago by David Yip <[email protected]>

dashboard: Display load averages on pipeline page. #106.

1f5504bc12bcbd820a35f6b2e0afe9e1990726c8 authored over 10 years ago by David Yip <[email protected]>

Merge branch 'master' into next

fd98b65eb6bceafa06952a2a563d386617afc06f authored over 10 years ago by David Yip <[email protected]>

Bump pipeline version.

37554e2dc1bcc53aa5e996ad983ed68d1e6dba94 authored over 10 years ago by David Yip <[email protected]>

pipeline: Report load averages. #106.

3a806c125d091d7f2102b330b79529b2913d0f09 authored over 10 years ago by David Yip <[email protected]>

pipeline: Factor out expected PhantomJS version.

No reason to have the expected version in two places, especially since
they've been out-of-sync ...

657c2a764c16166c44f153418bc932501747d075 authored over 10 years ago by David Yip <[email protected]>

doc: Use golden ratio for ArchiveBot version.

At some point we will have a way to compute ArchiveBot's version based
on number of commits; how...

fc299225185b156ebec223e6ad6a044a672cde38 authored over 10 years ago by David Yip <[email protected]>

plumbing: Fix comment.

9a5310253c3c7721e88db4a7672fef51e97b738b authored over 10 years ago by David Yip <[email protected]>

plumbing: Factor out common Redis connection code.

09f11d42face53e865c458ebd68480f06cdf81eb authored over 10 years ago by David Yip <[email protected]>

Ignore more mp3 streaming sites

da5575342ec5947f7330cf7b92d61f3d0c794532 authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore *.corp.ne1.yahoo.com - drops traffic

312c8477bb8f090f3b8c9948699a65c022f3af4e authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore more mp3 streaming sites

807c43a9d36254f7f132eabf761495564fb0427c authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore more mp3 streaming sites

55d5bd096ddaa258512ff0da17757fe4d4ac20d7 authored over 10 years ago by Ivan Kozik <[email protected]>

pipeline: Switch to ludios/wpull@hax-9.

This branch contains:

- a recent performance improvement in wpull
(f3eabbe4ce32df3395785741db...

b1c1c519a01e35238af388af3f7a0e62eaa2f831 authored over 10 years ago by David Yip <[email protected]>

pipeline: Bump to Redis 2.10.3.

bf7648cb857b5f9faf14205426767b5212522810 authored over 10 years ago by David Yip <[email protected]>

Merge branch 'master' into next

9d7c267484617d67da845ae6c162abe65a804d33 authored over 10 years ago by David Yip <[email protected]>

Ignore more mp3 streaming sites

f04433dd3edc68b3427380edb158756f840dc08d authored over 10 years ago by Ivan Kozik <[email protected]>

Update global.json

e016459ad407fc63ae8162a2dfd1e2d6b6dc2d7a authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore another Icecast site

0d581e278e89ba5ac36ca965e5498252967b4204 authored over 10 years ago by Ivan Kozik <[email protected]>

Remove unnecessary ignore

" is quoted

753df7e9a57850e7d3f9f670c8d67d02bed6754b authored over 10 years ago by Ivan Kozik <[email protected]>

Add ignores for wpull@develop

It does not quote as many URLs

c1d17aa7faa2f23750b71586475bc2c2f9745927 authored over 10 years ago by Ivan Kozik <[email protected]>

Fail instead of hang when FINISHED_WARCS_DIR is not set

162ca84712336887a3dede50360ed040a43e670a authored over 10 years ago by Ivan Kozik <[email protected]>

Upload .json files as well

Useful when uploading failed broken jobs

534b2e8ae74ddd3a1e98f9be37c02a3b3056024b authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'dupes-db' into one-uploader

4a6cbafbf36cd4b6859c7f953234c2aba8d09d3e authored over 10 years ago by Ivan Kozik <[email protected]>

Grab newer patched wpull

b8c0132c75df784593d9987d0157ed321f82b503 authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'master' into dupes-db

d0a15d587553eac35a869130573a08ccf954ba06 authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore bad /js/chartbeat.js links

bceb26a42ef7e8038a79b05737f6b8bc9d7da553 authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore bad linkedin URLs found by wpull

650b52af3fce4c38b9badb068252c4d9d5a314db authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore more twitter share links

1ce9961d77fd7754dc859849e05e5628e402f30d authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore &action=edit&section=new

4ca0b9d2378072c214283f40bca6fbe7775a415f authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore more mp3 streaming sites

ece1e381a3b5fd4da81297bca89cb8016074e548 authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore weibo share links

986da90e2daf5858cdd0dae9f9231f1e9f9ef38a authored over 10 years ago by Ivan Kozik <[email protected]>

Remove anti-loop patterns that may result in false positives

7442efaf87ac61a3e73a91d17a633c9158cfff55 authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore share links on IPB

d229031aaddda1f61cd1ea693a84a5e0f4847512 authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore /?view=getlastpost

33060f4ad9001c6eb7672a58d9823723ace72950 authored over 10 years ago by Ivan Kozik <[email protected]>

Reformat code

4d7175dcbf45d33e730c81fd5ba8d1bea28b2b5f authored over 10 years ago by Ivan Kozik <[email protected]>

Tweak # queued tooltip, refactor

ad6de1f25d9d0587add944513070004f4d5e05d3 authored over 10 years ago by Ivan Kozik <[email protected]>

Add more forum ignores

213743f918febd9664f6494bfa33c819a04ff3f9 authored over 10 years ago by Ivan Kozik <[email protected]>

Merge remote-tracking branch 'origin/master' into one-uploader

a02ed2c9e85dc09d2696cee739f4d9a98ffdd3f1 authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'master' into next

Conflicts:
pipeline/wpull_hooks.py

04eb101c940556b010c61c1ac6ac60bd4315c656 authored over 10 years ago by David Yip <[email protected]>

Merge branch 'master' into dupes-db

b4b8a7edb0fcf7fb14cbe7cc66c98834cf28b467 authored over 10 years ago by David Yip <[email protected]>

pipeline: Batch items queued/downloaded updates.

Previously, when wpull processed a page containing many URLs, we'd fire
off many HINCRBYs. Furt...

c57e0fcc105f995e543f532db27ab9d54a47f018 authored over 10 years ago by David Yip <[email protected]>

bot: Tweak note-presentation message.

c75b6343f20dedd212c0f203e3fc3c28397c0e78 authored over 10 years ago by David Yip <[email protected]>

bot: Show job note in !status output.

90eb6a683c32b645229b8d5c212651b314d3f4c7 authored over 10 years ago by David Yip <[email protected]>

dashboard: JSLint happiness, one === at a time.

b8e139150f1941596086c80a7053e32970b8a337 authored over 10 years ago by David Yip <[email protected]>

Ignore loops on gdcvault.com

999c458c3e1773b932a076eef430ea6cf7158054 authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'master' into one-uploader

c1122de3dda62d505998ad37a298ee9ea03720ce authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'master' into dupes-db

87e6aabdd0f37f496b6b865e42a1bdff7ceb80c8 authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'master' into next

889a25d57aea71cebec09c5c0c9533866fce202d authored over 10 years ago by David Yip <[email protected]>

dashboard: Don't show blank notes.

600abac4f206a8daf71a86bddd78def49f93fcf4 authored over 10 years ago by David Yip <[email protected]>

Merge branch 'master' into next

67e530478df57e0712a3ba16126d4cec7a2883b7 authored over 10 years ago by David Yip <[email protected]>

dashboard: Remove extraneous space when note not present.

54d504c212fff3a8f5d30b6e79396963ff233668 authored over 10 years ago by David Yip <[email protected]>

Merge branch 'master' into next

eaa1ac4f44d90cb0dbd4e245704f3c645c4bcfe5 authored over 10 years ago by David Yip <[email protected]>

dashboard: Show short notes to justify jobs.

We're starting to have jobs that run to many (hundreds) of gigabytes,
and the question "why are ...

f00a9d52a6c94279ff181e950b563a1c4ffb25e9 authored over 10 years ago by David Yip <[email protected]>

Merge branch 'master' into next

bebf5978deac9bc121eeebe4c4f92fe109c74b6f authored over 10 years ago by David Yip <[email protected]>

Shorten several things in the status lines

d7006db1870380f4d78e93a564ae17d29ec7d5c1 authored over 10 years ago by Ivan Kozik <[email protected]>

Add ?host= URL param for testing dashboard.html locally

aac230e3504caf71179b0ab8da0ac24159a40808 authored over 10 years ago by Ivan Kozik <[email protected]>

bot: Patch critical deficiency in command set.

c11adcefae0e8cdc681a4e194839857f6e69a42d authored over 10 years ago by David Yip <[email protected]>

Ignore more mp3 streaming sites

404d8cd2ecf11d7d001aa279bbadd70bdea7ec4c authored over 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'master' into next

Conflicts:
pipeline/wpull_hooks.py

1c68f31ab56b4abc51e37ad091338846b20fc593 authored over 10 years ago by David Yip <[email protected]>

Bump pipeline version.

fc352c7d3a122d745771e3a490cd0e5bf889090f authored over 10 years ago by David Yip <[email protected]>

Show remaining queue length in dashboard. #96.

Also show total number of queued and downloaded items as a title
attribute.

456070547321060d8b2d75c9b2e287f73a6b21bc authored over 10 years ago by David Yip <[email protected]>

Merge branch 'mback2k-topic/queue_counter'

Conflicts:
bot/job_status_generation.rb
dashboard/assets/javascripts/models.js.coffee
dashboa...

81b0ec0d4d70185f4a4486fcc4bb91b32b21d332 authored over 10 years ago by David Yip <[email protected]>