github.com/ArchiveTeam/ArchiveBot commits | Ecosyste.ms: OpenCollective

Merge pull request #125 from PressStartandSelect/Start-socialmedia+useragents

Add social media ignores and safari/mobile safari user agents.

e8bf7b5ec1bbae8484fb1de1bdb1ba3a980a18b8 authored about 10 years ago by yipdw <[email protected]>

maintaining consistency

bc9d066ca340be7a797ac4bb48638d1120e70a37 authored about 10 years ago by Start <[email protected]>

add mobile firefox

c6c9a67443b6f2d72fe2e4008ea40c1c07e51650 authored about 10 years ago by Start <[email protected]>

add mobile chrome

488d4dcff20768b0b7bd5baca4f3f8106ca0e6b9 authored about 10 years ago by Start <[email protected]>

fix ignore

dba262dd3f28ecff9064188b3e628d51781478ff authored about 10 years ago by Start <[email protected]>

add social media ignores and safari user agent

a22332918bb1964dd2b7819e126adda6d920706d authored about 10 years ago by PressStartandSelect <[email protected]>

docs: fix dashboard launch, add UPDATES_CHANNEL environment variable.

4731764489f28a506779e084e8e090a2dcc34363 authored about 10 years ago by David Yip <[email protected]>

docs: Recommend piping trimmer output to /dev/null for now.

638ed198a4eea5be7fb8809b7515eb6cfb44e940 authored about 10 years ago by David Yip <[email protected]>

Forward-port @Asparagirl's documentation improvements (#113).

(This makes me realize just how ridiculously rococo ArchiveBot's backend
is. Time to simplify.)

68e41bc851ca94a827a528ebc2be6156b900f1bc authored about 10 years ago by David Yip <[email protected]>

Merge branch 'next'

Conflicts:
INSTALL

e06c98a86c5b31f9b96390729c5d7915a415da24 authored about 10 years ago by David Yip <[email protected]>

Add blogspot.sg

0e7f7bfd14f670cbc324b2a3595d8ceeb7e26eff authored about 10 years ago by Ivan Kozik <[email protected]>

Merge branch 'topic/archive_viewer' into next

5071e3cf4146fb592059b3ee1643597eec650e3d authored about 10 years ago by Christopher Foo <[email protected]>

bot: Don't process !a handler for !a < FILE.

(cherry picked from commit b8f2e654e85e2f501dbacfc0e7dfe5eab7770815)

a99cb4822b66af21e987be277f559e40840b4066 authored about 10 years ago by David Yip <[email protected]>

bot: Experimental !a < FILE mode.

This is intended for archiving Meetup Everywhere sites. Most of the
sites in Meetup Everywhere ...

b3fa6b028c5ac9a11d13abd705901f7c11f75758 authored about 10 years ago by David Yip <[email protected]>

Add Meetup Everywhere ignore set.

Added to help out with a bunch of Meetup Everywhere jobs.

(cherry picked from commit 6e91c50d12...

2689b8d8c7e37804730d6d5cee01b0d6f1e09ea6 authored about 10 years ago by David Yip <[email protected]>

bot: Don't process !a handler for !a < FILE.

b8f2e654e85e2f501dbacfc0e7dfe5eab7770815 authored about 10 years ago by David Yip <[email protected]>

bot: Experimental !a < FILE mode.

This is intended for archiving Meetup Everywhere sites. Most of the
sites in Meetup Everywhere ...

cd49c7ae9bf083904823f1894ff73ac7a9560ee9 authored about 10 years ago by David Yip <[email protected]>

Add Meetup Everywhere ignore set.

Added to help out with a bunch of Meetup Everywhere jobs.

6e91c50d12169d98c8e6cb744a363da46ab9c29c authored about 10 years ago by David Yip <[email protected]>

viewer: Flush db only before populating data.

ab76636ae4df3cefb33fb5f5116e5ed55d4db433 authored about 10 years ago by Christopher Foo <[email protected]>

viewer: Rewrite to use controls package. Use Column chart for daily.

985c935044578b318c48a9dcc49ca38fa1b84e52 authored about 10 years ago by Christopher Foo <[email protected]>

bot: Suppress ignore reports by default.

4f6c583c48cfe697473aa02b27efb24f5ed8a255 authored about 10 years ago by David Yip <[email protected]>

bot: Suppress ignore reports by default.

69a653d364c6c6d84f7d4cf636b5685ef942df93 authored about 10 years ago by David Yip <[email protected]>

viewer: Flush data every 3 days

18fa2cebbad515a6e1d39040417255ab4039d664 authored about 10 years ago by Christopher Foo <[email protected]>

viewer: Fix stats off by 1 on month

edda388760e477a96daa4c5ae04b10fdd3b3c3e0 authored about 10 years ago by Christopher Foo <[email protected]>

viewer: Aggredate sizes on public date instead of job date.

2bb2208a5685beee0b7f41927b646bfb6fa80b22 authored about 10 years ago by Christopher Foo <[email protected]>

viewer: Add stats view

2015a57ccdca320a09a3c7d1099169bbd164a161 authored about 10 years ago by Christopher Foo <[email protected]>

viewer: Fix regex to work on WARC part files

d5e0d32a06123643db5d49e8e26fac59738c070e authored about 10 years ago by Christopher Foo <[email protected]>

viewer: Add basic audit page

bd0d76016c216bd84485c2864e66b89003118519 authored about 10 years ago by Christopher Foo <[email protected]>

viewer: Structure the model. Support identless files.

31718d812eb27ab0f08a0c186182a7ab3cfacfa6 authored about 10 years ago by Christopher Foo <[email protected]>

Ignore Windows 7 .iso's that we've already grabbed

7bf1c67c52366ce532a400236f149386f5a7f1ec authored about 10 years ago by Ivan Kozik <[email protected]>

Need to install libffi-dev on Debian 7

dc08df56af948d1a43e9ee25dd348cd37a2a76d2 authored about 10 years ago by Ivan Kozik <[email protected]>

Use wpull hax-20, keep using lxml for parsing until html5lib performance is tested acceptable

5a92c96dd4fa80e35891a3a89d39fb45aabf3a1e authored about 10 years ago by Ivan Kozik <[email protected]>

Bump pipeline version

0979ae9c3174b0a2c287b17255f5967fb68b195e authored about 10 years ago by Ivan Kozik <[email protected]>

Create 5GB WARC files instead of 10GB WARC files so that we can run more

jobs on servers without much disk space

3f17bf1d1c5d00ce750afc8698842a0cc9816be3 authored about 10 years ago by Ivan Kozik <[email protected]>

Fix phantomjs version in INSTALL

e5a3f28a447d6c4ce0427030a01f245f7bfd75e1 authored about 10 years ago by Ivan Kozik <[email protected]>

Add initial import for archivebot viewer

b6e76e339f5535a8566905021e0a77a32159e04b authored about 10 years ago by Christopher Foo <[email protected]>

Ignore another Icecast site

f7a5fe90ea1926569158966ac3fadd91df5e8fed authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore another Icecast site

e4b54a66a758026fcaca4d9ed25a13c1fbad5b9e authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore another mp3 streaming site

a19740a41cb98b7a0b28c8d8fcd8ecf6a1ed861d authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore another mp3 streaming site

a8a12b4112d9b09468b4b4c2ba45c8e6faddd9c9 authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore another share link

d6c9cc7392d0e439858ab4bc9ebc466fb2873397 authored about 10 years ago by Ivan Kozik <[email protected]>

Install manhole, needed for wpull --debug-manhole

ad18ec0297da01e6092d8149a3d565971f9dddc9 authored about 10 years ago by Ivan Kozik <[email protected]>

Require PhantomJS 1.9.8

aa2337e8e6c004fa147bff18c650a5663c950a75 authored about 10 years ago by Ivan Kozik <[email protected]>

Document use with tmux instead of screen due to screen reattachment bugs

ad83caa5a3aa3ab59db8da1eda55bc05a32bc1d4 authored about 10 years ago by Ivan Kozik <[email protected]>

Add ftp to list of supported schemes

e349dbafabc9f5995fbcbcb25e6c7f15bc59cdfc authored about 10 years ago by Ivan Kozik <[email protected]>

Bump pipeline version

adceec471161f93e53d8b7a8b784d1223726e946 authored about 10 years ago by Ivan Kozik <[email protected]>

Use wpull hax-19

ea18cbabf1ea7c1d6fdcc351d3dcd4d0cc799837 authored about 10 years ago by Ivan Kozik <[email protected]>

Sleep before first upload

da84fceb2c6ff7291d44f5b587cd9a192776ccf6 authored about 10 years ago by Ivan Kozik <[email protected]>

Print upload target and warning messages

126b747b6a7331a555774d26bc2a024c8a2d27cc authored about 10 years ago by Ivan Kozik <[email protected]>

Don't allow uploading to localhost

3a73421a21ec12513e68dd5d454298faca07325f authored about 10 years ago by Ivan Kozik <[email protected]>

Upload .txt files as well

4016f59ac836dad862e990d4e700849c9e19d26c authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore another mp3 streaming site

11580c63499531a35138e9a088b4aa168b98fc3c authored about 10 years ago by Ivan Kozik <[email protected]>

bot: Experiment with finer !con permissions.

I think that raising job concurrency is something that needs to be
restricted to channel operato...

8d4630541e9246594f0e0e874cd653096b97fdfb authored about 10 years ago by David Yip <[email protected]>

bot: Experiment with finer !con permissions.

I think that raising job concurrency is something that needs to be
restricted to channel operato...

94e17cfcf6b819b5058db9880dc41f33c042f411 authored about 10 years ago by David Yip <[email protected]>

Wait only 10 seconds between uploads

4908bf310345317e4fac90e2a91d771b232798d9 authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore another share link

1d20e8d744e4d6cdbb3df1ecec485233d5e9438f authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore more share links

ea5f805989424065082eff5929d642a36648d566 authored about 10 years ago by Ivan Kozik <[email protected]>

dashboard: Remove CouchDB connection.

775f040621146358b6c56a61b2475fee6d1c3b57 authored about 10 years ago by David Yip <[email protected]>

Ignore more do=markread

46456a97192ca942832b36234af1ce3c4a468a9a authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore another Icecast site

49b827cba5aab425d980329e799d6acff55ce5d5 authored about 10 years ago by Ivan Kozik <[email protected]>

plumbing: Remove redundant watermark == 0 check.

9cc2c267e4aedda368a6b27a79c6f71237317374 authored about 10 years ago by David Yip <[email protected]>

plumbing: Remove noisy watermark logging.

8e6e38fe457663acd8bd30a4bc03516692b7d25d authored about 10 years ago by David Yip <[email protected]>

plumbing: Remove unnecessary queued > 0 check in set-watermarks.

This was added to avoid continuously setting next_watermark to zero,
which occurred in the case ...

939d2f9b100050849f4d795c7dad073514063464 authored about 10 years ago by David Yip <[email protected]>

plumbing: Introduce set-watermarks.

The set-watermarks program watches the number of URLs queued (note: the
total number of URLs que...

2ec6acfab08be9e3e844297b47786cc5f0f00686 authored about 10 years ago by David Yip <[email protected]>

Ignore some vbulletin loops

578c8c9e6374705926f9c57d4e107230b01c53e3 authored about 10 years ago by Ivan Kozik <[email protected]>

Ignore /ucp\.php\?mode=delete_cookies

8a1bfb46f0482e271d3a711140fb9da6d9c839b8 authored about 10 years ago by Ivan Kozik <[email protected]>

Fix licdn.com ignore for new wpull URL encoding behavior

993a8053beaa33183dd7d1b48fe0c6d5dcae6a4e authored over 10 years ago by Ivan Kozik <[email protected]>

Bump user agent strings

5a0e588f31984dcad91588bec90a16dc10ebc2ac authored over 10 years ago by Ivan Kozik <[email protected]>

Use wpull hax-16 and trollius>=1.0.2

0d1beeb148185399a6137e7b06dd292e6acd055a authored over 10 years ago by Ivan Kozik <[email protected]>

Show/hide one log window when its stats line is clicked

148cd66d29fd3019b53dfe98549712ff7a8e0a79 authored over 10 years ago by Ivan Kozik <[email protected]>

filterChanged -> applyFilter for clarity

0a4d253658421839f8f548b56a1ec6899bd2cea8 authored over 10 years ago by Ivan Kozik <[email protected]>

Clean up suppress_ignore_reports checks

bead8b972e6c5340d7317105382dfe34d7ef467e authored over 10 years ago by Ivan Kozik <[email protected]>

Simplify MB stat

13503fadb40e974858c178a7d3a681857a5c65e8 authored over 10 years ago by Ivan Kozik <[email protected]>

Adjust filter box styling

3710ad495765163ce4486756b673d48022707c3f authored over 10 years ago by Ivan Kozik <[email protected]>

s/Filter:/Show:/

1bed3c91728b20aa294b1390014434edb8cb0975 authored over 10 years ago by Ivan Kozik <[email protected]>

s/Clear/None/

bb2e5fa628886184b5d3bab27c15d059f8ee4328 authored over 10 years ago by Ivan Kozik <[email protected]>

Add button to filter out all jobs

de6f0a4af4b343766c5a9c62f66a099026980b83 authored over 10 years ago by Ivan Kozik <[email protected]>

Align and filter new log windows appropriately

14b0fcc00e7a395c9d37f30b09660b819e255eac authored over 10 years ago by Ivan Kozik <[email protected]>

Add Clear button for filter; always show scrollbar

2b257ca79d5d786ae28d840ed983fd50c8cb1f0c authored over 10 years ago by Ivan Kozik <[email protected]>

Emphasize igon for easy visual scanning

451454c707f4fb27792436318492e5592c401f85 authored over 10 years ago by Ivan Kozik <[email protected]>

Adjust styles on job ident

a26e1328f64b3cdbaba7bb75315169eab7d0d27b authored over 10 years ago by Ivan Kozik <[email protected]>

Align more things

31886c3881ef989375c345220a943f1f135af0b3 authored over 10 years ago by Ivan Kozik <[email protected]>

Improve some comments

327163add2c3787b85d1bc13dc8f389360c69f05 authored over 10 years ago by Ivan Kozik <[email protected]>

Automatically adjust alignment based on filter status

dad28e6ba1fc8e0a867bc617101e5697f63fce88 authored over 10 years ago by Ivan Kozik <[email protected]>

Align the nick as well; fix padding when aligned

8bf43a6466ba0b30904e52b80efbb4284f5e4786 authored over 10 years ago by Ivan Kozik <[email protected]>

Clean up alignment feature

05244932629bca24a3a94a6aef77024971ac8bea authored over 10 years ago by Ivan Kozik <[email protected]>

Preliminary implementation of 'Align!' feature

e26da407a8f0508c5408dd113d52e7637f5595e9 authored over 10 years ago by Ivan Kozik <[email protected]>

Make Help! a button because it's not a link

ffc3689a7f4fe033cd5561c0d4b7b70ffcb501f1 authored over 10 years ago by Ivan Kozik <[email protected]>

Autofocus the filter box

2d6a5342cf55645530214edb126fb78cc7c6bd85 authored over 10 years ago by Ivan Kozik <[email protected]>

Implement filter box

bf801543942a25bcc928a3ddec938809b615e75e authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore blogger.com/blog_this.pyra

d116aa84b2ef701b5e8de114abcab52451498ad5 authored over 10 years ago by Ivan Kozik <[email protected]>

Move blogger.com ignore to global

70978f90142b0933a66cbfd6f89b42d0b28f5bb1 authored over 10 years ago by Ivan Kozik <[email protected]>

Refuse to start outside screen or tmux

dba60a42e0586fb140d666407645f95dc6be0778 authored over 10 years ago by Ivan Kozik <[email protected]>

Refuse to start outside screen or tmux

8c5fa15bb14bc36cf929e9e1dc165f8320662db5 authored over 10 years ago by Ivan Kozik <[email protected]>

plumbing: Add ia-list, a script to list all ArchiveBot materials in IA.

Assume ia and jq are present.

6cc3b22937b1d74e444a0dbcce7c61954fcca369 authored over 10 years ago by David Yip <[email protected]>

Enable wpull debug manhole

37bdc26c0d3683f6712bdeea50816a73087b93dc authored over 10 years ago by Ivan Kozik <[email protected]>

Grab sitemaps to find more links

c48a74ce59845c2a5b2be4e7f29def967cf6d034 authored over 10 years ago by Ivan Kozik <[email protected]>

Ignore another mp3 streaming site

ee10bc2e3fc7916db6ecd0058385924fb2a74cc1 authored over 10 years ago by Ivan Kozik <[email protected]>

Remove 32-character note limit.

This was originally added to avoid overflowing the dashboard's job
status line, but 32 character...

5ef4b3f794fd92ef8102c3ffc0ba52a9d3f758bd authored over 10 years ago by David Yip <[email protected]>

Merge pull request #113 from Asparagirl/patch-1

MOAR n00b INSTALL STUFF

25895ad596339a2aaee42251e22af86238aad32f authored over 10 years ago by yipdw <[email protected]>