Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/ArchiveBot

ArchiveBot, an IRC bot for archiving websites
https://github.com/ArchiveTeam/ArchiveBot

dashboard: remove unused browser version checking functions

0dc4794bf67765c3b8339420b4fa672450f76d6f authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: remove some old TODO items

We don't really need a dynamic !con; it's easy enough to backspace and change the number yourself.

5508ac65eab208a4d82071949521610217376ccc authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: fix indentation

b1dadfbab1336543ae50c3dc0b19a50906265382 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: context menu: add suggested command to set delay back to default

7373eddba29086ffcee62b7b0a83ba95c865476d authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: remove `overscroll-behavior: contain` emulation

35308c9471f959d2494c263790c3bd3c7a84dd6c authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: use `function X(...) {` instead of `const X = function(...) {`

7fc1752568a080059ba9cfc567f73fb243f6f835 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: use Array.prototype.findIndex instead of our own pre-ES6 implementation

4faa46b1bbf8f5f39eca05fbaa454e76af326ef4 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: use String.prototype.{startsWith,endsWith} instead of our pre-ES6 implementation

e1d5905ba041cb8db435e91bb033e734ab8404a3 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: replace all .bind(this) with arrow functions

2bea325af4d0b79303dd42ee7cd48696985c7e04 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: remove animations for all browsers

Their performance wasn't good enough in Chrome, even on a 7950X3D.

cf8abc619548e975695f16a937d25f9d21e1a3e5 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: reduce default historyLines to 500

The dashboard displays 2-3 times as many jobs as it used to, and we need to do something to miti...

c2cf7849e06f271bc626983436555d6c9a13f983 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: use `for-of` to iterate arrays

6b62e8ff54e42d90635b3bc8d574254f0ffbe9c4 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: replace `var` with `let` or `const`

7946d7fa62ba3f2222a4cdfc9e51056210914260 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: move JavaScript to its own file so that it can be analyzed with rome

7be1a8e46182e02a3669dcf1915a969bfa6b5cf8 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: reduce batchTimeWhenHidden to 1000 because the WebSocket server now sends ~250 messages/sec

227215457811a0a3aa93f79fe1e0a2ee916c0495 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: continue even without /logs/recent data

3d92ccd325619aaa6b43f6a3c78152ba1c760799 authored over 1 year ago by Ivan Kozik <[email protected]>
dashboard: fix whitespace on control flow statements

ab96deb4dfe628cd6f6891491629e9220459d0ae authored over 1 year ago by Ivan Kozik <[email protected]>
Merge pull request #550 from gabldotink/master

Documentation: change "www.bar.org" links to "example.net" links

7ad0d3882c7508d4190dc46fa5e66f1ae937e9ab authored almost 2 years ago by JustAnotherArchivist <[email protected]>
Change "www.bar.org" links to "example.net" links

7a42330b6b16d53db0d312d6901a8ab9c9ff43da authored almost 2 years ago by gabl <[email protected]>
Merge pull request #548 from JustAnotherArchivist/generalise-docs

Overhaul the documentation to remove all references of the ArchiveTeam instance

cddce9230edc42edfdb844d64333e7328f98110e authored about 2 years ago by JustAnotherArchivist <[email protected]>
Overhaul the documentation to remove all references of the ArchiveTeam instance

This makes the docs more relevant to anyone wanting to run their own instance of ArchiveBot.

It...

ca550b992db6e3d8c27a3a3e9fa6b891e6b03987 authored about 2 years ago by JustAnotherArchivist <[email protected]>
Remove FOS from pipeline instructions

f2743e2c3d557b95f3346c131e62fc7dd58979a2 authored about 2 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #546 from Pokechu22/patch-3

Add additional github ignores

38c67e162439fa9dbb82babc65393a5d77f064fe authored about 2 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #547 from Pokechu22/patch-4

Globally ignore s7.addthis.com

aeea17002409ed6bf43ea25f90318b0541dec978 authored about 2 years ago by JustAnotherArchivist <[email protected]>
Globally ignore s7.addthis.com

This appears on a lot of sites, and almost always times out, slowing down jobs. Other addthis.co...

28d1f40b1d31cb11b147d773f644d61972e2fa8e authored about 2 years ago by Pokechu22 <[email protected]>
Add additional github ignores

Both of these are 406.

385bcd795f820400929f7eb22603f9e0f08c30cc authored over 2 years ago by Pokechu22 <[email protected]>
Merge pull request #544 from JustAnotherArchivist/ignore-fc2-blog-2nt

Add 2NT's blogging platform to fc2-blog (same backend software as FC2)

ba5a520c21261f19e469a35555b3a2be93ceef09 authored over 2 years ago by JustAnotherArchivist <[email protected]>
Add 2NT's blogging platform to fc2-blog (same backend software as FC2)

9be05e1e2f82fb83abe318fd73305cbcddf7dddf authored over 2 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #542 from JustAnotherArchivist/ignore-mediawiki-languages-ar

Add Arabic MediaWiki igset

5c8c7ecafa1d4754ca450b4f41a11caccdeec0e4 authored over 2 years ago by JustAnotherArchivist <[email protected]>
Add Arabic MediaWiki igset

970732901f7d41d74d1094fcc03322eb99e7667e authored over 2 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #541 from JustAnotherArchivist/ignore-badvideos-ign-new-host

Update IGN ignore in badvideos igset

fc108288080954c1d7d70ed67acb15d0b138ad5f authored over 2 years ago by JustAnotherArchivist <[email protected]>
Update IGN ignore in badvideos igset

8ccaff69f76f709d5c67f3991cfd039a600927d6 authored over 2 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #531 from ArchiveTeam/ignore-discord-assets

Global ignores: Ignore Discord assets

4a672dbff49597dd8a1f53d95ee60f6ff17a5c87 authored over 2 years ago by Sanqui <[email protected]>
Global ignores: Ignore Discord assets

Discord invites show up in a large portion of the Web nowadays and
hog jobs with 100+ partially ...

63118fa1d65f11390fcb1d0ab9a40e221566f3c9 authored over 2 years ago by Sanqui <[email protected]>
Add further ignores for DSpace 6.x filters

dateIssued_page seems to be entirely broken on many instances (or in general?), but more than 10...

a84a6e93beb16a69299ef8d384b01b8dc871b3ec authored almost 3 years ago by JustAnotherArchivist <[email protected]>
Add igset for DSpace 6.x

/discover is for the XMLUI, /simple-search for the JSPUI.

a857ebddc4e988047b51df3baf6d78b6f37da9da authored almost 3 years ago by JustAnotherArchivist <[email protected]>
Add Ukranian MediaWiki igset

9c9c84e2d6dbdc043ce59cb9d32779aa8d4b520d authored almost 3 years ago by JustAnotherArchivist <[email protected]>
dashboard: mention the Chromium timer throttling issue

ef97da7729ad689dcf625eb1be59c3380927fd4f authored almost 3 years ago by Ivan Kozik <[email protected]>
dashboard: tweak link text

da1bcdd326b2b3f5957eb282e1cef3070cdd5b6f authored almost 3 years ago by Ivan Kozik <[email protected]>
Merge pull request #526 from JustAnotherArchivist/fix-pyyaml-6

Fix compatibility with PyYAML 6.0 (mandatory `Loader`)

1cab84619ffd21a9e3df36b3b0bf829777998af4 authored almost 3 years ago by JustAnotherArchivist <[email protected]>
Fix compatibility with PyYAML 6.0 (mandatory `Loader`)

e050864d333e4b332a21671cb5f08f2ffd9172fe authored almost 3 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #525 from JustAnotherArchivist/ignore-notweets

Add notweets igset

22b3b08e18c5f15113a95c94bdf6cd3921b0dbd6 authored almost 3 years ago by JustAnotherArchivist <[email protected]>
Add notweets igset

880bb95ed13fe0685599a317a5c6e4fed9098499 authored almost 3 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #509 from Sanqui/master

ignoresets: add a few forum URL patterns

16be7656148451cd1d22101986e63b2190222ad6 authored over 3 years ago by Falcon Darkstar Momot <[email protected]>
Merge pull request #517 from JustAnotherArchivist/ignore-tumblr-singletumblr-looser

Be looser on singletumblr igset

c94b1a26862c55475185f46839ce62de1101a140 authored over 3 years ago by JustAnotherArchivist <[email protected]>
Be looser on singletumblr igset

This lets jobs grab video iframes and offsite links that aren't Tumblr blogs (except custom doma...

2b2fde86863003e52aae985d5a80d0e7d86a7cd0 authored over 3 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #512 from JustAnotherArchivist/bot-explain-reason-alias

Add !reason alias for !explain and --reason for --explain

0522ca3b81ce2f418f593620fb2a58bf74cd88c5 authored over 3 years ago by JustAnotherArchivist <[email protected]>
Add !reason alias for !explain and --reason for --explain

aa4e4807dc1528698870fe58e49d1f725729c140 authored over 3 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #511 from JustAnotherArchivist/dashboard-irc-hackint

Fix dashboard references from EFnet to hackint

e964397b48ee0a8f8df4e4ae38f49d0acea25f8b authored over 3 years ago by JustAnotherArchivist <[email protected]>
Fix dashboard references from EFnet to hackint

0da5be4e2e41e1cd1283f3475f263de0c15601bf authored over 3 years ago by JustAnotherArchivist <[email protected]>
ignoresets: add a few forum URL patterns

3dcd873f2f7d741a979aa110cba75a39e308c0dd authored over 3 years ago by Sanqui <[email protected]>
Merge pull request #508 from JustAnotherArchivist/dashboard-finished-response-details

Add counters for status codes and queued/downloaded to finished page

c1020e56bcd3d37596ff7695f6503e76aa4e0bca authored over 3 years ago by JustAnotherArchivist <[email protected]>
Add counters for status codes and queued/downloaded to finished page

48e8b12a923587d4492de46ca43ea1eef46ba5bf authored over 3 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #506 from JustAnotherArchivist/ignore-mediawiki-languages-ja

Add Japanese MediaWiki igset

d3d10ceb2a7258acffe9a81d1ae0e0e54911b1d0 authored over 3 years ago by JustAnotherArchivist <[email protected]>
Add Japanese MediaWiki igset

57c8b061ed709c6cfe21f1d4bf4dfc08cb7c7414 authored over 3 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #503 from JustAnotherArchivist/cookiejar-empty-hack

Add a hacky script for clearing a job's cookie jar

877335ef4b5ebf2913609ce326873df41abeb392 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Add a hacky script for clearing a job's cookie jar

Cf. https://github.com/ArchiveTeam/wpull/issues/448

Based on https://gist.github.com/JustAnothe...

39e097c4edbc0c3eb3b8d0269d85666272531091 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #504 from JustAnotherArchivist/wpull-sqlalchemy-lt-1.4

Pin SQLAlchemy version to <1.4

f163982afa9a0d305ef016c2680e11d5a4221846 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Pin SQLAlchemy version to <1.4

https://github.com/ArchiveTeam/wpull/issues/463

e9354853ad479519b30e2c42b6cba1c12c7ff1ff authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #502 from JustAnotherArchivist/travis-python3.5-pip-2-electric-boogaloo

Fix get-pip.py URL for Python 3.5 tests

3bd500fcb34b9847dac266ec71a10dff4e24097d authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Fix get-pip.py URL for Python 3.5 tests

Cf. https://github.com/pypa/get-pip/issues/61

891ae4f48675b28017c7631a938fb9bbb08aaad1 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #501 from JustAnotherArchivist/useragent-curl

Add curl user agent

2d87c6b7534a62f418cca5b4b857ddf9cfe52f8a authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Add curl user agent

515a6aa50601c69ebe2c6869b68db5682f9962e4 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #500 from JustAnotherArchivist/cogs-start-load-igsets-uas

Load igsets and UAs into CouchDB on cogs start

ed1feffa53a9dec2029ce8a14cd4d20e13673a61 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Load igsets and UAs into CouchDB on cogs start

53872f629472578c1863c761df577772eb03b8f7 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #499 from JustAnotherArchivist/wpull-canonical-repo

Switch back to canonical wpull repository

d21680ebbc32d3eebe25117eefef857f0831806d authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #498 from JustAnotherArchivist/dnspython-1.16.0

Bump dnspython to 1.16.0

f1ba36e2903efe9e8bd5804a39d440eb74f5fcb6 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #497 from ArchiveTeam/all-env-vars

Stop filtering the wpull environment

65aab2462c02f01c2d920848c226057b86e00f2a authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Stop filtering the wpull environment

All set environment variables will now be passed to wpull, including for example LD_LIBRARY_PATH...

7a68211b334307da13f3a94c0e7baa382d057da8 authored almost 4 years ago by Falcon Darkstar Momot <[email protected]>
Switch back to canonical wpull repository

8057afd0dc20b58c0ac5a7fba3ce77d41bdea300 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Test for broken dnspython on pipeline launch

ab15e7a5c2a2fec90abfd446c2281b2bcbbabce8 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Bump dnspython to 1.16.0

wpull depends on dnspython3, which is only a transitional package nowadays and hasn't been updat...

09c0de722e7f6dd5e6c14f4e4518bde4016d769a authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #496 from ArchiveTeam/openssl-ciphers-all

Update openssl-less-secure.cnf

47797be730d1aaf57361497bd516135c164da8f9 authored almost 4 years ago by Falcon Darkstar Momot <[email protected]>
Update openssl-less-secure.cnf

Update ciphers from DEFAULT to ALL since we intend to accept weak ciphers

c605b6eab18cdc81804f13bdd1cf507795d9e8e6 authored almost 4 years ago by Falcon Darkstar Momot <[email protected]>
Merge pull request #493 from JustAnotherArchivist/ignoracle-threading-bug

Fix ignores sometimes not being applied correctly due to thread-related race conditions

3fbf32bb3d2c9348527c700e8ccfab093c80c4f2 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #495 from JustAnotherArchivist/travis-python3.5-pip

Fix Python 3.5 tests due to current pip no longer supporting that version

7e57a6107b7634d0c3cc9bedb73d09427ff99dbe authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Fix Python 3.5 tests due to current pip no longer supporting that version

b37eab3c3c46a9b2540d3522169e9f7f5f5a53df authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #494 from JustAnotherArchivist/ignore-badvideos-ign-mlb

Add IGN and MLB to badvideos

f32be97154451d5de9f42f113708059832fb5969 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Add IGN and MLB to badvideos

6344f443cf27a2a57d7d68aec01d70a4fbd5d9aa authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Fix ignores sometimes not being applied correctly due to thread-related race conditions

`set_patterns` is called by `ListenerWorkerThread` and modifies variables used by the main wpull...

4ce74e6f8894ab271fe91e81e7df2e40f0af666a authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #491 from JustAnotherArchivist/pipeline-tmpdir

Pass TMPDIR to wpull if present to avoid jobs stalling on systems with small /tmp partition

3af42e74b7eeb9aced663cdba0eb700a068ec497 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Pass TMPDIR to wpull if present to avoid jobs stalling on systems with small /tmp partition

e303082ae7d2b6c3ac845698b80e637ccb46e825 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #490 from JustAnotherArchivist/pipeline-preflight

Pipeline preflight test improvements

0353b943cd734a6282c677997c6494c25d31c6f5 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Run preflight test before pipeline registration

cd1d4138d8fcd3460700f1ff1844e09d9c7b4f83 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Use a truly invalid domain for the preflight test

3ce5fc0691d33fc87400b22b51cd75900087b894 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Run preflight test in the pipeline directory

Previously, the item directory would be created in TMPDIR (usually /tmp), which may be on a diff...

b0867451f6c6f41089e6e939fb9672d480f5f187 authored almost 4 years ago by JustAnotherArchivist <[email protected]>
Add ČSFD video files to badvideos ignore set

Typically movie trailers, e.g.

https://video.csfd.cz/files/videos/157/765/157765147/164315717...

80af9d17bc899969d3f23de5c7269b9576c43673 authored about 4 years ago by Sanqui <[email protected]>
Merge branch 'master' of github.com:ArchiveTeam/ArchiveBot

dfdba96f59747fce86ce50377c3ad108971f5afc authored about 4 years ago by Falcon Momot <[email protected]>
Add readme file for how to save off as much data as possible if a

pipeline crashes

71a57558749df02b4c6d9a32f372c259465c9040 authored about 4 years ago by Falcon Momot <[email protected]>
Merge pull request #489 from JustAnotherArchivist/ignore-mediawiki-languages-ko

Add Korean MediaWiki igset

72d2d09cc447fac0a8acc8feff714b84f16f6e56 authored about 4 years ago by JustAnotherArchivist <[email protected]>
Add Korean MediaWiki igset

Closes #488

c818e3a4b983775a64bb6a9e34a6fffac7a056aa authored about 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #487 from JustAnotherArchivist/dashboard-finished-order

Order dashboard finished page by completion date by default

a3fa117a01cc2c4cb24948b2574b829a2ffb990a authored about 4 years ago by JustAnotherArchivist <[email protected]>
Order dashboard finished page by completion date by default

31c13ea408b225b0dd9e8772c062a41c7b5a2514 authored about 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #486 from JustAnotherArchivist/dashboard-finished-perf

Improve performance of dashboard finished endpoint

d06fa2f4f20c4cca927247c1919ad8361537e6f5 authored about 4 years ago by JustAnotherArchivist <[email protected]>
Pipelined HGETALL

a88f27d342d5516301f99172530c9ec9c8282e8b authored about 4 years ago by JustAnotherArchivist <[email protected]>
Improve performance of dashboard finished endpoint by only doing one HGETALL instead of HGET + HGETALL for every job

Job.from_ident first fetches the URL and then retrieves the rest of the data (in amplify). That ...

420891bdaf0ef81b3dcbedd8164adfc42f9fbeda authored about 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #485 from JustAnotherArchivist/dashboard-finished-2

Improvements to the list of recently finished jobs

a889ea682449dd3e2f21eabab9a66b8841c03de1 authored about 4 years ago by JustAnotherArchivist <[email protected]>
Clean up Responses column code

0828ba6a667d02485bcdb16b6f662d53b64fac4a authored about 4 years ago by JustAnotherArchivist <[email protected]>
Add Remaining column to finished job list

8321b96cdcb44f2efaeec8ce70b007afa6f6f6b1 authored about 4 years ago by JustAnotherArchivist <[email protected]>
Merge pull request #484 from JustAnotherArchivist/dashboard-finished

Add list of recently finished jobs to dashboard

52632ec8b47139cefe2453921b96eded3069f01e authored about 4 years ago by JustAnotherArchivist <[email protected]>