This requires at least git 2.6.0 due to --date=format:x

Require that the time zone is set to UTC on pipelines

Notify IRC channel on pipeline changes

Register pipeline immediately on startup

The data directory is created by seesaw when it starts running the Pipeline (or specifically, th...

Fix log shipper crash

Previously, only connection errors were caught. Buf if the disk on the control node was full, fo...

cogs: Run the Broadcaster only if necessary, i.e. if tweeting is enabled

Block uploading to an rsync URL that doesn't end in a slash

Ignore a CBL sinkhole domain

Check for local webservers

Verify that inexistent domains do not resolve

Compress the log file if wpull didn't finish cleanly (and didn't write a meta WARC)

Update tests (August 2019)

Remove PhantomJS support

* PhantomJS is no longer maintained. The last stable release was in early 2016, little developme...

Check that rsync exists in pipeline and add it to the installation command

www<dot>cgzxb<dot>com is associated with some malware, and accessing it causes a listing on the ...

The Broadcaster uses a ridiculous amount of CPU, and if tweeting's disabled, that's just a waste...

https://github.com/ArchiveTeam/ArchiveBot/issues/305

Fixes issue 305.

Ignore more WhatsApp, Facebook, LinkedIn, and LINE share links

Ignore WordPress Fastest Cache's minified JS recursion

Self-host the dashboard's JS dependencies and update them

jQuery from 2.1.1 to 3.4.1, DataTables from 1.10.2 to 1.10.19.

Update user agents to current versions

Mostly based on https://techblog.willshouse.com/2012/01/03/most-common-user-agents/

Couldn't fi...

Ignore reverse pagination on custom Blogspot domains as well

Add separate igsets for MediaWiki locales and a script to generate them from MW message files

Add igsets for GitHub

Basic Mastodon igset

A few blogs igset improvements

Ignore Squarespace JS hell

Various improvements for MediaWiki ignores

Update Safari UA

"github" is the general igset that should be used with GitHub grabs. "nogithubcode" ignores the ...

- Fix CSI ignore on Blogspot, and remove search and label ignore since those pages are actually ...

- I've seen a couple jobs recurse through the entire log of a user with limit=1. Nothing good ca...

New dashboard WebSocket server

The old server was unable to keep up with the messages (#333), ate all the CPU and RAM it could ...

Updated Safari UA for Mac OSX with the string for High Sierra

Add pending job list to the dashboard and disable !pending when the queue is long

Allow unauthorised users to add an explanation to their own jobs (fixes #223)

Add alias --concurrent for !a/!ao command and !concurrent for !concurrency

Add pending-ao counter to !status reply

Add some error handling

Add WPULL_MONITOR_DISK/MEMORY env vars

wpull's option is called --concurrent, so it makes sense to support that on ArchiveBot as well.

The defaults might not fit every pipeline and are a bit low when it
comes to large file download...

Updated grab-site repo URL in README

Allow only opped users to add jobs when there is a queue of 5 or more jobs

document the archive --large flag

It's in the source but not in the docs.

Disable youtube-dl until #291 is fixed

This reflects the ignoracle speedups.

make job status less chatty

... and the rest of the universe

The reference is noisy for nothing - it's already in the channel topic
and it's not used in othe...

The job status is currently two verbose and spams the channel with
four lines of log.

So instea...

Import grab-site's dashboard fixes

Cache the ignoracle patterns while primary_url and primary_netloc stay constant