github.com/ArchiveTeam/ArchiveBot commits | Ecosyste.ms: OpenCollective

dashboard: enable context menu for all browsers (Safari 10+ has `document.execCommand('copy')`.)

ed8c722ac92989ba18e3d1ea312bb2ba6f313abc authored over 6 years ago by Ivan Kozik <[email protected]>

dashboard: adjust the font stacks

284a06bd924610c5d7d298b6394b29481a8efc20 authored over 6 years ago by Ivan Kozik <[email protected]>

dashboard: for Chrome 63+, use the faster `overscroll-behavior: contain` instead of attaching an onwheel event.

c6ae77ee696e921e82a9510b34bc0ff342dbb99c authored over 6 years ago by Ivan Kozik <[email protected]>

dashboard: keep stats rows aligned when using San Francisco font

4414057e19144c94e00fcb8d6cfa9ae80db47d9c authored over 6 years ago by Ivan Kozik <[email protected]>

dashboard: allow for another digit in the MB stat

d8023c3542cbda01cb7f1c1fc680d2bfbd5c0497 authored over 6 years ago by Ivan Kozik <[email protected]>

dashboard: align the req/s stat properly

a8b82f8a1bf222a0ab334c0881190f7aaf6c5ac3 authored over 6 years ago by Ivan Kozik <[email protected]>

Merge pull request #321 from JustAnotherArchivist/ignore-sinkhole

Ignore sinkhole domain

89a9d91bc25cf3b76c4ec220f1abe2e99861117a authored over 6 years ago by Arkiver2 <[email protected]>

Ignore sinkhole domain for the CBL

68c800e925defec1dfbacf4fbb13fbb25f90c049 authored over 6 years ago by JustAnotherArchivist <[email protected]>

Merge pull request #320 from JustAnotherArchivist/ignore-reddit-update

Update Reddit ignore set

8ebcc59aff1dd04d32301518201119824458e1ad authored over 6 years ago by Arkiver2 <[email protected]>

Merge pull request #316 from JustAnotherArchivist/ignore-radio-france-streams

Add ignores for Radio France's streams (France Info/Inter/Culture/Musique/Bleu, FIP, and Mouv')

d040b0591ec370a72339164066e641d87e8c88c2 authored over 6 years ago by Arkiver2 <[email protected]>

Update Reddit ignore set

- Support old.reddit.com and np.reddit.com subdomains. old is the pre-2018-redesign version of R...

e3f954209780a90b7c410a061a4f46778e1f1917 authored over 6 years ago by JustAnotherArchivist <[email protected]>

Update Radio France CDN ignore

f4b45145e5c473cced8888f2d7fe422fef805ee0 authored over 6 years ago by JustAnotherArchivist <[email protected]>

Add ignores for Radio France's streams (France Info/Inter/Culture/Musique/Bleu, FIP, and Mouv')

312517ef99c52f64c42968389701ec3959b0b024 authored over 6 years ago by JustAnotherArchivist <[email protected]>

Merge pull request #314 from JustAnotherArchivist/remove-uploader-rsync-options

Remove unnecessary rsync options in the uploader due to permission issues in some configurations...

33d2eb3112af19d32428c901d69a91e766d1b4c4 authored almost 7 years ago by David Yip <[email protected]>

Remove unnecessary rsync options in the uploader due to permission issues in some configurations and for simplicity

The uploader used -a, i.e. -rlptgoD. All except -t were removed for the following reasons:

- -r...

6690413f64cfd623aa7c6a5157d8f7b1a304f975 authored almost 7 years ago by JustAnotherArchivist <[email protected]>

Merge pull request #310 from DoomTay/patch-1

Added more tracking URLs

4d909d984ee7ad1f6fd0373be3c3911d74d98244 authored almost 7 years ago by David Yip <[email protected]>

Added missing comma

e58a79d2f6ccce8197a70a39bb4c1195160b6266 authored about 7 years ago by DoomTay <[email protected]>

Appended tracking URL to global igset

Also removed extra URL

602c9359cec8a9bf2358ac25478739d4f566e3c8 authored about 7 years ago by DoomTay <[email protected]>

Added another tracking URL

cef1d78c9711cfe7934af5de3c8ff67cb0464b1e authored about 7 years ago by DoomTay <[email protected]>

Merge pull request #300 from JustAnotherArchivist/ignore-tumblr-avatar-resolutions

Ignore higher resolution of Tumblr note avatars

23974b22ce475440309c7433161126f0b8ae2024 authored about 7 years ago by David Yip <[email protected]>

Merge pull request #299 from JustAnotherArchivist/ignore-youtube-www-recursion

Ignore /www/www/... recursion on recursive YouTube channel jobs

291a6fe9772a31d0c438762c758cc742074d5825 authored about 7 years ago by David Yip <[email protected]>

Merge pull request #307 from Iggersm00/patch-2

Update pipeline.py

4b53e1210972e032bd9dcda30db1b7f8da24a6e4 authored about 7 years ago by David Yip <[email protected]>

Merge pull request #298 from JustAnotherArchivist/ignore-wordfence-lh

Add ignore for new Wordfence URL parameter wordfence_lh

fea45d7037f2fac384cfeafb8d8aecae21e84818 authored about 7 years ago by David Yip <[email protected]>

Merge pull request #301 from JustAnotherArchivist/ignore-whatsapp-send-link

Ignore WhatsApp send link

381888656b1ca4d531e97d3f4123932a9b40d231 authored about 7 years ago by David Yip <[email protected]>

Merge pull request #302 from JustAnotherArchivist/ignore-mediawiki-special-recursions

Ignore PrefixIndex and WhatLinksHere recursions on MediaWiki

a074d0ae881616c0d238a5c1467167305c411b69 authored about 7 years ago by David Yip <[email protected]>

Update pipeline.py

Increased version to 20171212.01

Added wpull version check for issues 306

849b445beceb563d471786b144ce5f97aaceb604 authored about 7 years ago by Iggersm00 <[email protected]>

Ignore PrefixIndex and WhatLinksHere recursions on MediaWiki

97c501b387c3eddc7fe8dfa2790965c99891df93 authored about 7 years ago by JustAnotherArchivist <[email protected]>

Ignore WhatsApp send link

1a45cf72803d0f445ad68ec47850894598163cef authored about 7 years ago by JustAnotherArchivist <[email protected]>

Ignore higher resolution of Tumblr note avatars

0b7bba1013120b19fab7d9618ece0b2254fead26 authored about 7 years ago by JustAnotherArchivist <[email protected]>

Ignore /www/www/... recursion on recursive YouTube channel jobs

f49f52019d23ab6bc2ed571788e696715445959a authored about 7 years ago by JustAnotherArchivist <[email protected]>

Add ignore for new Wordfence URL parameter wordfence_lh

4696e9c0b4cbc8169bd73e49cf48984108673158 authored about 7 years ago by JustAnotherArchivist <[email protected]>

Merge pull request #297 from Asparagirl/master

moar minor text edits

8bc8eb28fa20204ecb9d931baaae4685f10ee36f authored about 7 years ago by David Yip <[email protected]>

moar minor text edits

moar minor text edits

b94dbaa11e6e7f93dad0393bfa4811e85e7f19fd authored about 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge remote-tracking branch 'ArchiveTeam/master'

94b33b024eac18e9922ef52d33c3ad9455d8f689 authored about 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #295 from Asparagirl/master

added new control node documentation, linked redis documentation

bb34ca2992c66221a5dc55ecddc8f10abd613e52 authored about 7 years ago by David Yip <[email protected]>

minor text edits as per yipdw's feedback

minor text edits as per yipdw's feedback

cf79ccd7629201eec8418fed585b5908d5208573 authored about 7 years ago by Brooke Schreier Ganz <[email protected]>

edits to new admin documentation

edits to new admin documentation

ecefa5fce68fcfd1674e61846b0bb817ce4f5ed7 authored about 7 years ago by Brooke Schreier Ganz <[email protected]>

added new control node documentation, linked redis documentation

added new control node documentation, linked redis documentation

ae5db23850923a027ff008f281fc91db44ce4261 authored about 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #294 from Asparagirl/master

updated documentation

1e9c0c3c507c067837913516a1178d5920b0a20d authored about 7 years ago by David Yip <[email protected]>

Merge remote-tracking branch 'ArchiveTeam/master'

f632566861c03a6a60f032a608f6f444a1bcad78 authored about 7 years ago by Brooke Schreier Ganz <[email protected]>

updated documentation

updated documentation

2a71a3f75e23417f77f47da5ddc7467dbf8910ed authored about 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #287 from JustAnotherArchivist/bot-short-unignore-alias

Add !ug as an alias for !unignore

2814e51b3ed014cbf07862f64ccab4ff816298f8 authored about 7 years ago by David Yip <[email protected]>

Merge pull request #286 from Asparagirl/master

Master

04eed2b0cbcef75b74f0c4c127981d1af628c3d2 authored about 7 years ago by David Yip <[email protected]>

Added QQ share to global ignores

Added QQ share to global ignores

20a09393360cc6b241f3d0cd5dfe4023adf0c8a9 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Add !ug as an alias for !unignore

Motivation: IRC crops long lines at a certain number of characters. If you add a too long patter...

7ff494534d269f4d3aa3cb08d9b5b9761a482649 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Added more forum ignores

More forum ignores for Vanilla forum URL patterns

3fb9f30ed0bcb80cc6d836d837899919d42d8c6d authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge remote-tracking branch 'ArchiveTeam/master'

adb0f9675376f16b4332943b7e344b98d9732456 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #281 from JustAnotherArchivist/wordpress-plugin-super-socializer-ignore

Ignore links by the Wordpress plugin "Super Socializer"

a6e6da8ba37e733e4b10b7090b5fc4a6cffc9119 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #285 from valgrind/patch-2

Add some RFC5735 networks that should not be crawled [RFR]

f5a4fa89f67efa22d9014b2ccd32144207a084bb authored over 7 years ago by David Yip <[email protected]>

Merge pull request #283 from Asparagirl/master

Update AddToAny

4458c7631a81871b74140ef7afac48c1408908a8 authored over 7 years ago by David Yip <[email protected]>

Fixed two quote errors that escaped

f66fa7065165fa059eb713a95175f18119a1ea7e authored over 7 years ago by valgrind <[email protected]>

Add some RFC5735 networks that should not be crawled [RFR]

Add the following networks to the global ignore list as listed by RFC5735:

0.0.0.0/8 ...

3a3f21a97ca18ec8171304fdea528956dc3c3ea4 authored over 7 years ago by valgrind <[email protected]>

Update for AddToAny

AddToAny has switched to https, so need to update global ignore list

d362d773a68c4f5bb50ea89e7b9a67038dceaaf0 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge remote-tracking branch 'ArchiveTeam/master'

185b40c37a22307fd263c86262069a6aa36deb10 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Ignore links by the Wordpress plugin "Super Socializer"

359c19d3dbf672871e802f8c5b2f40d11e5726a5 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Merge branch 'JustAnotherArchivist-merge-noonion-into-global-igset'

1abeb717cff899227a23f416993a7409fc2982e8 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #276 from JustAnotherArchivist/vkontakte-share-link-ignore

Ignore VKontakte share link also on vk.com domain

106da9d7ae2036f5322da8e316a4efa60c3405f9 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #277 from JustAnotherArchivist/smf-ignore-verificationcode

Ignore verification code/captcha on SMF which causes an infinite loop

0c15a9cdc81ca82ca641e75cf12241da2a20abff authored over 7 years ago by David Yip <[email protected]>

Merge pull request #278 from JustAnotherArchivist/ignore-oauth-urls

Ignore OAuth URLs by Google and Facebook

72f521488988ceb9a9b64d8dc18fc0282de3105a authored over 7 years ago by David Yip <[email protected]>

Merge pull request #275 from JustAnotherArchivist/reddit-tracking-link-ignores

Added some ignore patterns for tracking links and pixels on Reddit

6a20178dab8fcac7eebd82ae2771a0a8d1e5f4ce authored over 7 years ago by David Yip <[email protected]>

Merge noonion igset into global igset

f00a0269fb893a23a0043a477b9fa2b26a8982bc authored over 7 years ago by JustAnotherArchivist <[email protected]>

Ignore OAuth URLs by Google and Facebook

3bf35b23059b82df1757e7f1b2a2a4413f6205e7 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Ignore verification code/captcha on SMF which causes an infinite loop

10dcc051b09fea9ce6d7ffd92dd3611fc2824788 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Ignore VKontakte share link also on vk.com domain

ba7d9bf192c0362805096d78fdcd0351f9d290b9 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Added some ignore patterns for tracking links and pixels on Reddit

c710b47f91ea55f63a25ae0806aec4730a078448 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Bump pipeline version (see #273).

d912c078d8bb99b2fed396fc19657d4159168e90 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #273 from valgrind/patch-2

Support storing files on ZFS

c2288f9b0070236c803112953d5e194707da52be authored over 7 years ago by David Yip <[email protected]>

Support storing files on ZFS

ZFS (At least the 0.6.5.6 version included with Ubuntu) will return st_blocks == 1 for all sizes...

f7f558096cd8e9752d2b92c14404a5ae766675f5 authored over 7 years ago by valgrind <[email protected]>

Merge pull request #265 from JustAnotherArchivist/unignore-mediawiki-revisions

Replace ignore pattern causing all previous revisions of wiki pages to be ignored

4d156a56102b07accab3edca9b704cbbd3432c40 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #270 from JustAnotherArchivist/smf-ignore-forum-listing-sort-order

Added ignore pattern for sort orders on SMF's forum listings

fa56da8596e6aaed0c1f3dad17a7ce774fed9266 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #271 from valgrind/patch-1

Use autossh in place of ssh

6454406675258fab3f71517611ed7b2db0d620e1 authored over 7 years ago by David Yip <[email protected]>

Use autossh in place of ssh

Makes the setup more robust against network problems and ninjawedding downtime.

b806e98278b12686a69fd7cc0690b1a5dc229f45 authored over 7 years ago by valgrind <[email protected]>

Added ignore pattern for sort orders on SMF's forum listings

96e2f22f175a4bd143af491d1a33a237e6e92a11 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Ignoring some more mediawiki pages with duplicate data which I overlooked in the previous commit

68097006c1f0eba228039350300d1dab8779f9da authored over 7 years ago by JustAnotherArchivist <[email protected]>

Remove ignore pattern causing all previous revisions of wiki pages to be ignored; only ignore the diff pages instead

98495d85f3c7fc17d62f94f84d619bcfa590f8f4 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Merge pull request #264 from DoomTay/patch-1

Added ignore for amp;amp; chains

0a8dc44a429b1684af6b1394f8f3e8611b1c6046 authored over 7 years ago by David Yip <[email protected]>

Added ignore for amp;amp; chains

cf22b539166d3def942b736a212e4fb4865779d0 authored over 7 years ago by DoomTay <[email protected]>

Cache the ignoracle patterns while primary_{url,netloc} stay constant

dea91b3fa0301ad60f2d86f65ab9ac19805631f1 authored over 7 years ago by JustAnotherArchivist <[email protected]>

Merge pull request #260 from Asparagirl/master

Added Tumblr share tool to global ignore-set

412936b917d24bbe646e70aba067e05777d144dd authored over 7 years ago by David Yip <[email protected]>

Added Tumblr share tool to global ignore-set

Added Tumblr share tool to global ignore-set

654c28be3e587f25619df2508ead02f15429b7c0 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge remote-tracking branch 'ArchiveTeam/master'

31666380205eae7c53d3ca6b09ff4023ce9a9ba2 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge pull request #256 from Sanqui/x-dns-prefetch-control

dashboard: opt out of DNS prefetching to avoid making DNS lookups on every host

dc05935086280eda3c4e288f5e3514705cb9dd89 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #257 from Asparagirl/master

Slight update to forums ignore-set

b3ac88e7da38862d2a200c2ae85ea0bfe157da17 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #255 from Sanqui/some-ignore-patterns

Add some forum and facebook ignore patterns

bc7b89c9746d648ab8e49074301b91d4ae6ade73 authored over 7 years ago by David Yip <[email protected]>

Very slight update to forums ignore-set

Added 'reportpost' to forums ignore-set

08594ea1fc9f1d7bc33a1f1fa27b5aee43ea5c71 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

Merge remote-tracking branch 'ArchiveTeam/master'

112a2555b3da6f5483bd51fbeb1fc7d8fe3bced1 authored over 7 years ago by Brooke Schreier Ganz <[email protected]>

dashboard: opt out of DNS prefetching to avoid making DNS lookups on every host

Before this fix, if "Use a prediction service to load pages more quickly" was
enabled in Chrome,...

d5123070c9d7b917b8a704e63e8209b5a197924f authored over 7 years ago by Sanqui <[email protected]>

Add some forum and facebook ignore patterns

b3c55e73d2ef4d963246c9167d9959fc669d4a01 authored over 7 years ago by Sanqui <[email protected]>

Merge pull request #254 from DoomTay/patch-1

Changed TED ignore patterns to include HTTPS

e4687f35f1b7f03eae6bcbabd8b455162c34e7f2 authored over 7 years ago by Justin Kerk <[email protected]>

Changed TED ignore patterns to include HTTPS

159aefb0488254a669aa5f34821c12df05a35ffb authored over 7 years ago by DoomTay <[email protected]>

Merge pull request #253 from Sanqui/fix

Restore the integration test

c90fe7c087d04e29783b40d9712d33402bfb9749 authored over 7 years ago by David Yip <[email protected]>

Fix integration test and re-enable it

Dashboard crashes on shutdown and I can't fix it, but that's
not of much concern.

8f73e66d471d9df69f5d5c91a08522c2062d213f authored over 7 years ago by Sanqui <[email protected]>

travis: Disable IRC notifications

They report into the main channel for forks as well. If desired,
please configure them per-repo.

f4f3beb9c22a6dd70642d3af41cfc5b0d18c8489 authored over 7 years ago by Sanqui <[email protected]>

pipeline: Fix slight bugs with locating wpull and phantomjs

d1bd7803aac19dd1993117dc5e1db119c1d93eb5 authored over 7 years ago by Sanqui <[email protected]>

Merge pull request #251 from Sanqui/dashboard3-improvements

Minor dashboard3 improvements

b08a757e490757ac6e8641706509ad6a083165d4 authored over 7 years ago by David Yip <[email protected]>

Merge pull request #252 from Sanqui/fix-tests

Fix or skip failing tests, add new test validating json

1e820b9c72f612d72c875c3a59522d5d60fd12aa authored over 7 years ago by David Yip <[email protected]>

Merge pull request #249 from C4K3/master

docs: Note that !expire is restricted to ops

73879a9fbde74bfb100066391990e3b77f765e6a authored over 7 years ago by David Yip <[email protected]>

Merge pull request #250 from Sanqui/master

More forums ignore patterns

47ec38a9710af8217b16dc6c58f48839d2e053c1 authored over 7 years ago by David Yip <[email protected]>

Fix `noonion` ignore pattern

9b183ee8f6d23bd45c6656eb02e88d88b1991625 authored over 7 years ago by Sanqui <[email protected]>

tests+travis: Add db/ JSON validation

5760a6a4579b42d4bbacb70c79b691adc5b25908 authored over 7 years ago by Sanqui <[email protected]>