Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/NewsGrabber-Warrior


https://github.com/ArchiveTeam/NewsGrabber-Warrior

Update pipeline.py

warrior boot file changes to force dnspython 1.16.0

No changes for pure docker clients.

3de01b0bde452afd2e95272a7dcf57a7080fd625 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

f15009fccb9da29ee5398c22bb4f7dacb1e3e4ff authored over 5 years ago by Matt Iggo <[email protected]>
Update Dockerfile

c6ba4f39be1f94d94f601e3474b9fc829b58bea6 authored over 5 years ago by Matt Iggo <[email protected]>
Create Dockerfile

Initial Dockerfile

ae54fd4db83bfcc61fdabc8af791a0ec76c804c4 authored over 5 years ago by Matt Iggo <[email protected]>
Update README.md

Added webports

4e5410f4261909beebf3e2e290a0b1d965fcee37 authored over 5 years ago by Matt Iggo <[email protected]>
Update README.md

Docker instructions

85d98c273f845eee41521fb11c285b4b4b17b3d7 authored over 5 years ago by Matt Iggo <[email protected]>
Update README.md

Initial comment to add docker awareness.

32934045d8fe57f4c28e7c271b3f10f66d0b1001 authored over 5 years ago by Matt Iggo <[email protected]>
Bind wpull proxy to 127.0.0.1

Fixes issue with binding to v6 address and failing jobs

dbd87e81ddd0d2b5ab8944538fa5aae969940698 authored over 5 years ago by km09 <[email protected]>
Update warrior-install.sh

Undo all warrior changes

aee8833f75ebfa67cd5c7b073868f5f1569bcb98 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Prefer local wpull for warriors

7b1fe1a7363202f1d9154d42600845bd8870ebb8 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Add back in wpull binary for warrior due to python 3.5+ being in the docker image used.

90f4d32b242b58d7ce47fbff8290f453e67ec75c authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Update version to 20190602.01

7a1fe9a49733699774c0558d2c110efd0d4c570a authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Revert changes

33509519e65753f155250954ed1756151f22d550 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

19e5dcef5446f7dffe0a479e8947e3cb8534adb5 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Remove DeduplicateWarcExtProc

ab26a3381e62e8c383b444f7a08e654f07aaec7b authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

make it singlerequest

0a6ae9700384544ecc30481c2ec424c1601f1b7e authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

3a66854a2f11026adc6cf77e25b1c0843a2c0a3a authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

9ed53ca1fafa6fa64cf9b452de87399234444fa8 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

01fd98c94c6f45f04084cb297b70b01c53e673e3 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

fcb1837e49a39ebaa920c793c62a4c39cbf00d5e authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

DeduplicateWarcExtProcArgs not defined anymore.

d1189ed6db76486a246c762aef04e30a96cae003 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

removal of legacy runner

641dec87615549ea2f3a226f5f2e6a09adf86cd1 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

61155d12c8a23c1ab72e0c0e75babdf2f0e6428f authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

da001f3b91f4d554f6d2d0ae9dcc6ca32ef31140 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

b422cf3eb25b16136a0550148d93c6afe34dcfa5 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

fe188685339554cb9e39ec3b5a13dc248d6ecf1f authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

f8decb3079a269b5eeeca1f61183d708bd1b71d6 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

6c0683ed0172caae887d36764c3f2881082bf908 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

3d2a6456aed99f0de353358923f9a7130b59b295 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

81c5f617044cecf16ec52ef47328e5ece40e7a0a authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

ca963d4adfa8b43ffed598ba8d5265d078c28b67 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Removal of wpull binary, tidy up of pip module installs

e282b144ddd23f799a55c981b417adb1b51e16a0 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

986eaa5c2b055bc24f5872153dea7c963b660736 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

755b1fa040a341bc4a41a93a382fa52e3d6de279 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

7964f7c92ad367bc7852379b1eaee3cc62ffd96f authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

9a660cb06b25d1b3077230cfb97c424b3c4d44c2 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

a617a83a7429f3479a36a3bea42c4067c024bb05 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

97f35fcd7104472d24cf2770e807f05aa28be1f3 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

c6ac77b77727fa032cf2f2c1e2b5b12243abbe5e authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

c1f3431bdd2f249b73a142810417e369a686004c authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

90850ab576bb28b5795577fc51d61134534a378e authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

469917912f45ecb477e37fddefaeead399f8e355 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

87f95c30e190da19f6197df6d1329590544dff86 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

10ada567ffac84fde8fdd06379762307d8d8f955 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

d10526653c33a05a05e6163c1121110237d9f036 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

d9e9f824be1981c5a442e719acd0794eb581775e authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

8ab5300e3bc8b727a372cb757937b01143106e7f authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

b1e5ae4bdb6a7360e8cbfcf56bc31890be125bf3 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

313dbc3f62431ea2d4511787479e5c3824b18f3b authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

00f768f6defc98d534408809438809eca175babf authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

6ed764971d348805d62911ba09b979e6e864b823 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

dc6b0105a271e0466440dc2918311af38791e927 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

ee28502afe08c3d8c747a04ee6c0fd246d35a233 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Find CWD - Docker debug.

3dc5bf4727a5c3637937f858708bec04b1eff6a3 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

3f9a75972034ecdc1ac09bb0cf5c9f668b3a19cd authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

866af848fe0c20902a1a007fd8489a81d4bc94d8 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Docker wpull location

ab9d5cb7aa516bbe98dc6875bf7a82c0bda6c14c authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Docker wpull changes

2171b4dad86ff0e1360265475078c93324c0ebae authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

21e5b4a84afb407b039b9ab712d8280f5853e76e authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

51943ae1a0e45efbab7496ad1bc65c9a0dcb4486 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

a5940f6452c1902281f697656d1ef9d6a2503b7a authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Changes to working docker pipeline.py

dee5f7b9268328f17bde89fd72cf8fab0f99c6f9 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

284a6fc5031d8e18f6ca472db62d556d4e386683 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Python3 dedupe

1e066b0715b3b154e4a30a5708d0e63b3d3b18a7 authored over 5 years ago by Matt Iggo <[email protected]>
Update dedupe.py

Python3 dedupe!

dee2ab6245c0499caa3aff410622beffe58725e2 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Rolling back changes as they caused unpredicatable results

8adc732e881583b28450805fc05f728729a3e765 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

8ecb6bae833220c2e2c02e4c1e94e57513e8985e authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Youtube-dl changes

5e92fb17e8ffab08cefdd0c9f1cc638d9aa73e5e authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Warrior boot changes

a198986faf9781766196cd8375355a3650194bb8 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Fixes for warrior not starting if blank

ecd30a4819e29fc31a6c0fb9c8e1109371e3aa8b authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Broke the warrior as the run-warrior and consequent run-pipeline break out of the environment.

26fb58b7391bdf4180079937b6b2b15aacd04efa authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Stop duplicating files

d77ef9886da1e5ed8953c7dadc7d09fbe4e50ee4 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

499761d1a03f4468f75823e5a4aa287e7b96a559 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

venv check

f9578bbc29993a23179a510c337fda5a8a258bd4 authored over 5 years ago by Matt Iggo <[email protected]>
Create checkvenv.py

bd0bb73d796a21f0172dd6342d89d5fe20d4c7ba authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

3d9725c8097ffce2464b3a42ce59a2b7ad15d6cb authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Think i've got this...

75b3ad5ad1e188b96c31aacddd50de8667832cb0 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Unzip overwrite.

be07aeecbd1720d5350972aec5d5c0af73d854d5 authored over 5 years ago by Matt Iggo <[email protected]>
Create wpullsetup.py

To set tornado == 4.5.3 and html5lib==0.9999999

Copied over downloaded version during setup.

b26758680f3b273a0ae9b7c3546d5760fc0d0f1c authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

a55bcd1445cbe57e304e97ae1ee9f817942a6ab4 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

2d3e405263dc68d40c35114fbcff212bb178f8a2 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

72f3bcd7c1926c4e5fabc65983123caca0fb3aa6 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Forgot me some sudo.

a7107d217cb88f568cfac52df8d27d3a2dee19a3 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Version bump

2bd842ab0c7a31cc94466dc5a3cfb83b4b875bec authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Changes to allow python3.4 to run (allowing youtube-dl to work again)

ec2be03138ee7c35047a3be007ef9da4029ca022 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

Fixing issues since the last commits in December 2018 moved to pip3 (and python3). However, Dedu...

6489c00c98a1973b55ecd0139381d29a19ba55b0 authored over 5 years ago by Matt Iggo <[email protected]>
Update warrior-install.sh

075cfad277158ae0671961ec5b6ba1161ddc8e9a authored over 5 years ago by Matt Iggo <[email protected]>
Update bunny CDN domain - update warcio detection in dedupe (#18)

* Update pipeline.py

Update version due to CDN change indedupe

* Update dedupe.py

Updat...

b50db8820cb0696390294b30f52b3e7fdab61936 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py

Increased version from pull #17

adbe1e5e02aec192787e01bbdc0dc361eaaaf76a authored over 5 years ago by Matt Iggo <[email protected]>
Check for the existence of a custom attribute instead of the unreliable path comparison (#17)

ab85f229bba43f771b3b68b1db833881aac2d2ee authored over 5 years ago by JustAnotherArchivist <[email protected]>
Update pipeline.py

Version uplift

b8c02495feb06bf8fc8fec8bea77a9d95494de09 authored over 5 years ago by Matt Iggo <[email protected]>
Update pipeline.py - Added ignore for xx.fbcdn.net

5bd8c78133e96766e695116582e14c802ba2a393 authored over 5 years ago by Matt Iggo <[email protected]>
Merge pull request #16 from MattIggo/patch-4

Update dedupe.py

Verified & committed

dd7f563371234d631ce0efa7d51b21abeb7b4601 authored over 5 years ago by Matt Iggo <[email protected]>
Update dedupe.py

Changed URL due to server replacement (and code change on dedupe side)

4bec6ef912123f5597be79fb374e51da1d517da5 authored over 5 years ago by Matt Iggo <[email protected]>
Explicitly look for py3 versions of six/requests

646b849886d574b5b7897a37978c0f3f1e155458 authored almost 6 years ago by km09 <[email protected]>
Merge pull request #15 from matthusby/patch-1

Update Readme to include the needed pip modules.

ec9f94eb25940a350b0de4c78800fa3eb3358e5e authored almost 6 years ago by km09 <[email protected]>
Add six module

5d4da1e02889a7211b337f4922b6f008add5a109 authored almost 6 years ago by km09 <[email protected]>
Update Readme to include the needed pip modules.

It seems like `youtube-dl` is also needed but I am not sure about the best way to handle that in...

48569224c699df7b49ec441810bd2fb599f5ffb3 authored almost 6 years ago by Matt Husby <[email protected]>
Merge pull request #14 from hutcheon/master

Ignore characters that we can't encode

732f17627b1b9a03bd00c214131f340580a0c3ec authored over 6 years ago by km09 <[email protected]>
Ignore characters that we can't encode

1c58e37e58d31688b846818aa129f0fefcd2b6ba authored over 6 years ago by Tyler Hutcheon <[email protected]>