github.com/ArchiveTeam/ArchiveBot commits | Ecosyste.ms: OpenCollective

travis: Disable integration test

It is failing severely and I'm not able to fix it.

c387de2242163b87f30c962509f1876d1790a969 authored over 7 years ago by Sanqui <[email protected]>

Fix or skip failing tests for pipeline/archivebot

4d3e4fc73bb0a21046f0d589f4ef02a41abd9c41 authored over 7 years ago by Sanqui <[email protected]>

dashboard3: Fix bug in counting number of responses

1097fbfa6440ff76ee161f3b7746322c25912f98 authored over 7 years ago by Sanqui <[email protected]>

dashboard3: Add pipeline identifiers

55d3e66c252aa4444e40239de90d33d353ccba4f authored over 7 years ago by Sanqui <[email protected]>

dashboard3: Add filtering by nick

Note to self: to recompile the Dashboard, do

haxe -js dashboard.js Dashboard.hx

and then a...

2a97fafc5f44266679de75b5bf75c32a53c3484b authored over 7 years ago by Sanqui <[email protected]>

dashboard3: Remove duplicate 300s row from code table

4b8d91edd72ed726191983b1814c5f53c2ea08c6 authored over 7 years ago by Sanqui <[email protected]>

dashboard3: Add nick as a column in order to enable sorting

This is useful for people who want to track their own jobs and
see them all in order.

11fa35d90f2a1c39657540ddbc484805579a7f81 authored over 7 years ago by Sanqui <[email protected]>

dashboard3: Change date format to near-ISO 8601

35d7f283aa10f44609179b02dd4fe91bbda4d2e2 authored over 7 years ago by Sanqui <[email protected]>

dashboard3: Add handy links to jobs

a3ed6cfce3532638672f78157a4afa34f49047db authored over 7 years ago by Sanqui <[email protected]>

More forums ignore patterns

The important addition here is ignoring individual post URLs.
These could easily increase the nu...

590714cd5e506f79e307a75f71652e418d4ba7c4 authored over 7 years ago by Sanqui <[email protected]>

docs: Note that !expire is restricted to ops

8d9610d9705cf20f66553e78ec01cdf8bf5f9f65 authored over 7 years ago by Sebastian Blunt <[email protected]>

Merge pull request #247 from falconkirtaran/master

Add doubleclick tracking pixel to global igset

d51500cfcb5710587d4f31e09c602d52efa28cc8 authored over 7 years ago by David Yip <[email protected]>

Add doubleclick tracking pixel to global igset

f2d56b892452121a4e6b4da8f5b1744a2f74e568 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

Merge pull request #246 from falconkirtaran/master

Correct JSON syntax in mediawiki igset

7c9cccfba5c110d2f1325f0d6ed408dd7b802953 authored over 7 years ago by David Yip <[email protected]>

Merge remote-tracking branch 'upstream/master'

cecf546b414e853a475924635c1a52ea0e1c3f6e authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

Correct JSON syntax in mediawiki igset

5eb5cfb57e7bb5ac46e545002f253f67feb0e000 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

Merge pull request #245 from falconkirtaran/master

Add --large option to job options parser

07e61242331f59f29fcaae3c04018a2845851a62 authored over 7 years ago by David Yip <[email protected]>

Add --large option to job options parser

81ba36a4c8013ceca8dabd37e1f21e38c5e9e749 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

pipeline: add missing comma; bump to 20170510.02.

f83ce45c8585e0864830a4a9a68b836fabf1b5bc authored over 7 years ago by David Yip <[email protected]>

Merge pull request #244 from falconkirtaran/master

Add --large option for !a command, and LARGE option for corresponding pipelines

f30d801110a0fb7d0e3d2ab4f09aea19868a1a76 authored over 7 years ago by David Yip <[email protected]>

Document --large pipeline option

c0991e7b4e5f76d388678a47de514778b227577f authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

Add --large functionality to IRC bot

b551901d6075d1c4d6089f5ef0134995c256b73b authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

Merge remote-tracking branch 'upstream/master'

4eb19a957a2733f33fb75c2fa57a4301acfb3e87 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

Version bump to 20170510.01; add --large option to check queue pending-large

7d3a135b789d499fc31e04a0c58223d64950df43 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

Merge pull request #192 from ethus3h/master

Added additional ignore patterns for Russian MediaWiki installations.

4d9c478a54d28d49d170c73ab5d24617597d6c45 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #243 from falconkirtaran/master

Change IA S3 uploader to add metadata "sponsor:Internet Archive"

3d558e9b53e2569b07ca0f7cb9ef265c2d83ab90 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #242 from Asparagirl/master

Installation instructions update

74ab3eee7ff7f57c39367722fe729a8892349d84 authored over 7 years ago by David Yip <[email protected]>

Change IA S3 uploader to add metadata "sponsor:Internet Archive" per

request from SketchCow

46b44ccfc6f897f20f4a8e09bc0c660fadc31248 authored over 7 years ago by Falcon Darkstar Momot <[email protected]>

Edits! We got edits!

Edits! We got edits!

d47bccf35f919d616b975235575fa8336f94ce06 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge remote-tracking branch 'ArchiveTeam/master'

1ea86ed9708687aa4154efec6378c636bfe7f170 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #241 from Sanqui/master

Add Twitter analytics ignore patterns

90e6fa31673fa7f0f2344659b396ae36746cfe6c authored almost 8 years ago by David Yip <[email protected]>

Add Twitter analytics ignore patterns

022c997465dc7c4c7afe6898c290989c66a7ac1b authored almost 8 years ago by Sanqui <[email protected]>

Merge pull request #240 from Sanqui/master

Add Semantic MediaWiki ignore patterns

c6d6d9c1c3c7acaf1d9c77667476148d7d1d66bf authored almost 8 years ago by David Yip <[email protected]>

Add Semantic MediaWiki ignore patterns

List of Special pages lifted from
https://www.semantic-mediawiki.org/wiki/Help:Blocking_robots_f...

e8d4872c16510679c03af9bcbe0fe1e76c12ac63 authored almost 8 years ago by Sanqui <[email protected]>

Merge pull request #239 from falconkirtaran/master

Fix rsync uploader error and add facebook captcha to global igset

9bfd98693b3e3a9124086c6560961e7ba5e55a7d authored almost 8 years ago by David Yip <[email protected]>

Add /wp-admin to blogs igset

eaf99e30d3c0fb4f2cc7871944677595b5135b91 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct error in global igset re. Facebook captcha

ce35dcef31da3a12945a432493203f9b7d6e0c80 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Add wordfence_logHuman=1 to global ignores

f709ecdeff5304e78edf86d8dbd96a70c016d18b authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct error in tmp_cleaner.sh path

fbad4efbc1e21ae6a50283e4f88de290ec9dab38 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Make new tmp_cleaner script +x

1dc8b18572b571e704c3a12157064f561c9b0566 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Add lsof, which is useful for cleaning up dead tmp files, to the

pipeline install

1fd8c2c9d1eb60f0f3612a8ce50595c21450507b authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Add some parts of doubleclick tracking to igsets. Add script to clean

up leftover tmp files in the pipeline directory.

af42462bdce26a0ded790272683bb8b8d5fed3a6 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge remote-tracking branch 'upstream/master'

086711980f45b070860f47aa6a7de311c0719832 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct error whereby uploader wait param is not initialized in rsync mode

f61095bf4f4a75675f786269be9aa685289d487d authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Add facebook captcha page to global ignores

e61af08cc7df62c9e413ca664a83732d4fea068b authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge pull request #238 from falconkirtaran/master

Add to igsets and improve pipeline documentation

d895c2840a3a9346aed4c52b784ee1693e6b3b5c authored almost 8 years ago by David Yip <[email protected]>

Correct escaping in addition to reddit igset

51077aa4960af61edb81ccfdd3970d2ce0d86524 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Add archive.is to global ignores

1170dfbed62bd4a46f406a6564c7233d2dc9a17c authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct pinterest global igset typo

8f671373b0346d2f0db13d95568aa60bcc0f3c48 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Broaden default ignore for pintrest join to include all TLDs

6b3009a6aa17209225addb39cc4895ab083d375c authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Add *.pintrest.com/join/ to global ignores

b317e153f1ef9d791c669efef4bd20ee009bdcea authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge remote-tracking branch 'upstream/master'

c8d01333b4d550fcf281856fb5152f72d0e9d314 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Minor documentation update to pipeline deployment. Add out.reddit.com

to reddit ignoreset.

01a36dea4b7ee7c5bdc3c3e5bc6ca6b51d936f5a authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge pull request #237 from falconkirtaran/master

Prevent filenames in IA uploader from starting with . and _

40f17817ecd9d64bc456375342f1bcece1432a7b authored almost 8 years ago by David Yip <[email protected]>

Prevent filenames in IA uploader from starting with . and _, per sketchcow's request

d93199a0f3ce8ec5bd4bfb993e5a46e898f115f6 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge pull request #235 from falconkirtaran/master

Fix partial removal of archivebot.control.ConnectionError

f8877ecc597d75dc0027dec4b9e6eef2604b3c97 authored almost 8 years ago by David Yip <[email protected]>

Fix partial removal of archivebot.control.ConnectionError

71cd783da07f2cdc7722feb961fdb3a64d4db079 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

[pipeline] Bump version to 20170106.02

141d7a81698c27b49d3da75d98f7a09b83966005 authored almost 8 years ago by David Yip <[email protected]>

[pipeline] Lock to falconkirtaran/wpull

This fork contains https://github.com/chfoo/wpull/pull/348, which is
needed to resolve some cras...

25ee0d4b925d5824f17fb0f83dfc314cbc59d6b5 authored almost 8 years ago by David Yip <[email protected]>

Bump pipeline version to 20170106.01

6292947ed37ed2c1669311cf297c2e69c05396d4 authored almost 8 years ago by David Yip <[email protected]>

Merge pull request #234 from falconkirtaran/master

Fix: high memory consumption on shipping many log actions; add Content-MD5 to S3 uploader; use w...

0506b6cc6cebd5d77958ae2dfce80ecfb699103a authored almost 8 years ago by David Yip <[email protected]>

Simplify and correct Redis exception handling in control.py; pylint fixes to same

597c8073963262e1e703199187fda246741bf180 authored almost 8 years ago by Falcon Darkstar Momot <[email protected]>

Repair mark_done logic in log shipping to not call it on Empty (oops)

6501fbc2d650df9abaf33de1dbe6e457f7bf2e6a authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Remove redundant task_done in control

e95a9cc2b739be45c78d2e8b464fc88332da8e62 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Fix RedisConnectionError aliasing and clean up control logic somewhat. Version bump.

3a6b0507d00ea407ddb03aeb2160774293c555c2 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Prevent crashes on determination of file size by reporting 0 on failure (best-effort)

260f16adf14b50679b8c48a76b202f4d90dff7ad authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Fix error in default ignores (how this worked in test, I do not know)

222dd69f7bc2d3cdc1ccd662eb88cff6c787a9cc authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Fix job aborting. wpull 2.x use should work now. Bump version.

d00425227860508906598c07ec188ac70ffff6ed authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct size reporting code

98780b0f24acf67e24c95539e2a67bc4d2adc974 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

More tweaks to make wpull 2.x work

cc94765558a35aa9efae458093cfcc040e4358d1 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

pylint fixes to uploader

681bff19fbf6e7e4f048e8f44687bb4d4df73615 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Update wpull command line and chain-load previous plugin for dupespotter

67c4a63745262d170ae3a927c88190dfe6441482 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

pylint cleanups

3e481776d9fb5bed8a951686e7fdd8d5f816c83d authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Revert "Revert "Remove obsolete wpull_hooks.py and bump version""

This reverts commit 9c62f0fa43cd9e898e1e0e6a5ded198cc4d75b15.

471781918cf3ac5394c72e1d61a5b5ab4585a5cd authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Revert "Remove obsolete wpull_hooks.py and bump version"

This reverts commit 37e352d09872a442973d492ec4912e1b90edd7d5.

9c62f0fa43cd9e898e1e0e6a5ded198cc4d75b15 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Remove obsolete wpull_hooks.py and bump version

37e352d09872a442973d492ec4912e1b90edd7d5 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

First cut at wpull 2.x plugin porting; many formatting fixes

967d5aa6538976e3899256963e2d779d7b12dcde authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge remote-tracking branch 'upstream/219-upgrade-to-wpull-20' into wpull2

365c10d31623413a616ec3a6ae51b0b736aff6d2 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct typo

4e8e76945b714e97e6675603a668698e8ce369a6 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Add Content-MD5 to IA S3 uploader to reduce corruption

bdab55326dd62ce53d11aa6d70f031b7a368fb84 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Clean up control script

4d801c918e2a980e7b4a5243aa37ff122b19e6f3 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct merge and bump version

fcfe4302c1bbf8c648be266993a021898661b368 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Merge remote-tracking branch 'upstream/master'

11b2c129e7cfcc3bd8fa6c9cb449ca541bef8acd authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

bot: remove reference to undefined variable.

3afb2904fed8ded998e1c8a30788e65e6747dcdf authored about 8 years ago by David Yip <[email protected]>

bot: remove broken smarts in !status URL.

The bot used to contact a CouchDB database of previously-run jobs, but
this database has turned ...

bd08f7dff0868ce67ca952b3de88d97c9fc5b609 authored about 8 years ago by David Yip <[email protected]>

Bump pipeline version.

a9d7893bef400d506de2406a9c1d0b482a2c3d51 authored about 8 years ago by David Yip <[email protected]>

Revert "Coalesce counts in log shipping to reduce memory and network requirements"

This reverts commit 3edf6ea2d8617eb8de0c85d58a95f678d1b6db7f.

This commit introduces the follow...

7f8ebd5f2f41df361dcee0536ca93955e6965614 authored about 8 years ago by David Yip <[email protected]>

dashboard: Add /3 alias for /beta

559f6149ea7f90fce0720a65fc0ab969c7cc99a3 authored about 8 years ago by David Yip <[email protected]>

Merge pull request #230 from Sanqui/master

Fix dashboard3

1df495555c0800bf8581d5f327d99dab75660c4e authored about 8 years ago by yipdw <[email protected]>

dashboard3: Fix dashboard3 by building it with Haxe Compiler 3.3.0

ceceb0468d5b8c05400e21546c9d30a3b80766ea authored about 8 years ago by Sanqui <[email protected]>

Remove !firstworldproblems.

I'm still a jerk; I just think of myself as a slightly less juvenile
jerk.

54fc7530ab9f76c1b83a7329bb2792fc3f23101e authored about 8 years ago by David Yip <[email protected]>

Merge remote-tracking branch 'ArchiveTeam/master'

0163ff29461893f1e9e6e6fba5a5baaffda1c783 authored about 8 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #228 from Asparagirl/ignoreset-dreamwidth

add new ignore pattern for dreamwidth

481c285ad97b1fda9a8854c0698a31241ea50f3c authored about 8 years ago by yipdw <[email protected]>

add new ignore pattern for dreamwidth

44cf7f8f0e9fe336b2f48b0761ac8232e6b802e5 authored about 8 years ago by Brooke Schreier Ganz <[email protected]>

Merge branch 'master' of github.com:falconkirtaran/ArchiveBot

cc94d19def831b22c8a01bc5af26dedde00710c2 authored about 8 years ago by FalconK <[email protected]>

Update to a working use of threading that can drain the log queue quickly enough to not build any up in RAM using redis pipelining after yipdw's example. It still can't join at the end, so it will still drop the last handful of entries sometimes.

3239f916e1c2c44f7e4c484179fd47b1c500c43e authored about 8 years ago by FalconK <[email protected]>

Also stop the logging thread on aborts

3496532c31a81380c28d736a25786e9bc7890440 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Stop logging when a job finishes, I think?

56f6b9ca714f574b34571cc82b49e152e6a528c9 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Actually increment the shipping count when shipping a log line

cf173969cf194a6a997e694bf4cf9c424d7addc0 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>

Correct syntax python 2->3

b6e13e2294459ce21b7d39b6f6a9dd0d0d182d01 authored about 8 years ago by Falcon Darkstar Momot <[email protected]>