github.com/ArchiveTeam/reddit-grab commits | Ecosyste.ms: OpenCollective

Version 20240216.01. Use fixed minimum Wget version 1.21.3-at.20231213.03. Use TLSv1.2. Fix check on svc comment content check.

daab40aa6e10194d30b3257fd6f7efb943a43625 authored 11 months ago by arkiver <[email protected]>

Version 20231201.01. Change protocol.

48dc016fafb69d032c056819a74d319b4806bb67 authored about 1 year ago by arkiver <[email protected]>

Version 20231127.02. New --ciphers value.

5f7cee8d3a482b09809c677d397f4f79a714cc15 authored about 1 year ago by arkiver <[email protected]>

Version 20231127.01. Use --ciphers SECURE256.

2b41d8ef429438d02ac9988110167c7ad5d9072e authored about 1 year ago by arkiver <[email protected]>

Version 20231118.01. Switch to gnutls.

7da27ab11036a75bb514378e3b62fc4c4488251f authored about 1 year ago by arkiver <[email protected]>

Version 20231115.01. Change cipher list.

0dc36e31e004f8f31981500c0918c535a3eb6ae7 authored about 1 year ago by arkiver <[email protected]>

Version 20231111.02.

8fc86a11ca8d34ec4d00b1bfa6e23178b0bbe2ac authored about 1 year ago by arkiver <[email protected]>

Version 20231111.01. Switch ciphers again.

6fdf778e19de3826b8cd3922f1f33b9f1abc6eec authored about 1 year ago by arkiver <[email protected]>

Version 20231108.02. Move to another cipher.

e87de8969cf9ec94eb28826fbbb74ab8c65e0ff3 authored about 1 year ago by arkiver <[email protected]>

Version 20231108.01. Do not install utf8 with luarocks, this is now in base parent image.

9c9b59dafdb8ce0ffee2eea04055ebc06d471800 authored about 1 year ago by arkiver <[email protected]>

Version 20231102.01. Do not keep partial files over rsync.

388e4325c5b5520b9211540e60927398ac17d8d9 authored about 1 year ago by arkiver <[email protected]>

Version 20231026.01. Use --ciphers HIGH:+SHA384.

1c2723f9f2895e656924c0fd3fe73e569ce3f05b authored about 1 year ago by arkiver <[email protected]>

Version 20231020.01. Use gnutls. Support new method of serving Reddit comments.

e350e69f898482dc6e250683761e718d76f8c44b authored about 1 year ago by arkiver <[email protected]>

Version 20231019.01. Use --secure-protocol=TLSv1_2.

0e7392acd3b1e586fb142de61fb90fd27966ba93 authored about 1 year ago by arkiver <[email protected]>

Version 20231017.02. Use --secure-protocol=TLSv1_3.

4bcc04734fd5503b6689d4f35669ca70428dc45d authored about 1 year ago by arkiver <[email protected]>

Version 20231017.01. Use --secure-protocol=auto. Use new minimum Wget version checker.

b1bf682030f05070f1a3ec9f5062324df6566bd7 authored about 1 year ago by arkiver <[email protected]>

Version 20230910.05. Install Lua utf8 library through warrior-install.sh.

a0e35bb72d0267fddfd9c56a238613845ca96283 authored over 1 year ago by arkiver <[email protected]>

Version 20230910.04. Install lua utf8 library. Fix converting unicode codepoint to utf8 character support.

3add4f891ce3600b1b8e7328566353ddde1d5981 authored over 1 year ago by arkiver <[email protected]>

Version 20230910.03. Increase hardcoded multi item size to 100, for soft limiting on tracker side.

12abd58d4dbf7496f811039a0c76bc12a58004bb authored over 1 year ago by arkiver <[email protected]>

Version 20230910.02. Remove old Lua files.

8a46824231ca9725939c5e700da9ffc32f83af39 authored over 1 year ago by arkiver <[email protected]>

Version 20230910.01. Use cjson instead of JSON.lua.

a2ffd1f6712fd58861c5b1d1b26ebc74dd59c589 authored over 1 year ago by arkiver <[email protected]>

Version 20230827.01. Use --secure-protocol=TLSv1_3.

e6b1602e31b7f8e1c337a6ab0c6037bba2c6d923 authored over 1 year ago by arkiver <[email protected]>

Merge pull request #18 from imerr/master-1

Extra docker container params

d210e659676399a2989a23e3ea70327760807bd8 authored over 1 year ago by arkiver <[email protected]>

Extra docker container params

watchtower: `--include-restarting` also update if the container is in a crash loop due to a bad ...

b7feddc14708f2950c94f383d6250cbd8c118723 authored over 1 year ago by Robin Rolf <[email protected]>

Version 20230727.03. In the Warrior, do not use GnuTLS compiled Wget-AT.

29a6952edb10f44982cc29dc8c0f0493ff5bd2f1 authored over 1 year ago by arkiver <[email protected]>

Version 20230727.02. Only allow GNU Wget 1.21.3-at.20230623.01. Use Wget-AT option --reject-reserved-subnets. Remove old Wget files. Update README to latest.

6e73452ec52e27eb7d8a267be17f4966968f95fb authored over 1 year ago by arkiver <[email protected]>

Version 20230727.01. Use openssl instead of gnutls.

288c9b731cfba0a2c41467d9c530ad4e8a268de3 authored over 1 year ago by arkiver <[email protected]>

Version 20230627.01. Queue outlinks directly to the urls project.

bb6198cc1a2a4f707ef7b7504aa4517c193e3427 authored over 1 year ago by arkiver <[email protected]>

Version 20230619.02. Accept 404 on mediaembed URL.

f1ef7d169771c785c3182fccb1b04fbf1c3eb663 authored over 1 year ago by arkiver <[email protected]>

Version 20230619.01. Primitive fix to user post verification problems.

d2571cde06a5cb445b0427313d5734a59192a529 authored over 1 year ago by arkiver <[email protected]>

Version 20230617.01. Use --secure-protocol=auto for Wget-AT.

2b19cdcd43ac0d252062cd33b142a0a7cdb0d4ea authored over 1 year ago by arkiver <[email protected]>

Merge pull request #17 from masterX244/master

Ignore fix for certain 404-ing garbage

5a0dcd6dd916b5e9cd8d7da4c32ca8b3f71d2109 authored over 1 year ago by arkiver <[email protected]>

Update pipeline.py

488aaa2181a206a50873b56ecf3af163268af1a8 authored over 1 year ago by masterX244 <[email protected]>

Ignore for some garbge URLs that 404

wget guesses too much and generates bad URLs, ignore needed

520e8b95d6d549a5ea22d3acb1029b0362796694 authored over 1 year ago by masterX244 <[email protected]>

Version 20230614.03. Better check for level error page on svc URL.

bea971f375482bfd3b70f194f03bca50bcedf313 authored over 1 year ago by arkiver <[email protected]>

Version 20230614.02. Extra validity checks.

be6e32cba503b947d3526124634db41c523f8f8b authored over 1 year ago by arkiver <[email protected]>

Version 20230614.01. Fix check for valid data.

e84e804fc5ba1889458668b314c9bfdec5977d59 authored over 1 year ago by arkiver <[email protected]>

Version 20230612.02. Add Reddit problem check for /comments/.../comment/ URL.

4936505b0f7ebdf19d4300206d1e15caeb244343 authored over 1 year ago by arkiver <[email protected]>

Version 20230612.01. Kill grab when reddit seems to have problems.

57adbb381cfbe08ada1b94a45aa0c68a50a4235b authored over 1 year ago by arkiver <[email protected]>

Version 20230611.02. Multi item size 40.

0ef6368945642f593c0944e96f71b515b2135bbe authored over 1 year ago by arkiver <[email protected]>

Version 20230611.01. Extra very simple check on validity of old.reddit.com returned body.

a974b8161893be26aaeee20e83d6bb88d8aae071 authored over 1 year ago by arkiver <[email protected]>

Version 20230607.06. Ignore discovered /r/FIFA URL if coming from a /r/EASportFC parent URL.

15a0a1a6f5d8dee39803af9f6b492c57318771cf authored over 1 year ago by arkiver <[email protected]>

Version 20230607.05. Better checking for video. Abort item if no post is found (during blackout for example).

fe17191306b8ad912031c535d67403848acfea74 authored over 1 year ago by arkiver <[email protected]>

Version 20230607.04. Abort on video for now.

7bb5c394194ca941629d70261874d7804f415350 authored over 1 year ago by arkiver <[email protected]>

Version 20230607.03. Prevent getting URL ending with /". Ignore /message/compose URLs.

f63c8ab69668a3e6eb504d3bd504e2a48a17e895 authored over 1 year ago by arkiver <[email protected]>

Version 20230607.02. Very simple content checks to check if response is complete. Properly prevent writing to WARC in cases and do not abort all items when finding a problematic URL.

393407520b1c48ae388cb39312c672fbe47b9188 authored over 1 year ago by arkiver <[email protected]>

Version 20230607.01. Use GNU Wget 1.21.3-at.20230605.01 and arguments around DNS.

37ba172c61417864b2402a4ae6ffc108bfbb447e authored over 1 year ago by arkiver <[email protected]>

Version 20230531.01. Use --secure-protocol PFS.

da85457aae2818d25414f21162c4c7c0a2735b68 authored over 1 year ago by arkiver <[email protected]>

Version 20230530.01. Queue discovered outlinks to urls-stash-reddit.

48b24323c67c2f29f8c775a9184840e376a28f88 authored over 1 year ago by arkiver <[email protected]>

Version 20230529.01. Correctly extract more comment pages from comment pages in the new design. Print debug infrmation for comment pages on old design.

a3b5bcecc1165977b3dc402033f0e4d3b1c82bf1 authored over 1 year ago by arkiver <[email protected]>

Version 20230509.02. Support new Wget-AT.

1a14af20954f15c109452af475887cfc9a871aeb authored over 1 year ago by arkiver <[email protected]>

Version 20230509.01. Support for new design.

b2654e93171aecbe0e4ebcff1791a842b98d0264 authored over 1 year ago by arkiver <[email protected]>

Version 20221021.01. Ignore /tailwind-build.css URL from comment in HTML.

7f4db173480ae74de46c1a17bf8ebc9b87f92918 authored about 2 years ago by arkiver <[email protected]>

Version 20221005.01. Max tries for backfeed to 10.

8a27002fd33267acdf778a56bd071565f7e20751 authored over 2 years ago by arkiver <[email protected]>

Queue redditstatic.com URLs as outlinks.

35e31af37fa6a78265494c832393ac01ba0df81f authored over 2 years ago by arkiver <[email protected]>

Version 20220729.05. Fix aborting item on bad status code on url: item. Keep old retry code otherwise.

bab4b4dcd2dd4ecb20448b701cce25cd94e3d106 authored over 2 years ago by arkiver <[email protected]>

Version 20220729.04. Queue extra found URLs on media URLs to backfeed.

8c45a263aaccad9ccd1a03a19bea194aa8ab1bc7 authored over 2 years ago by arkiver <[email protected]>

Version 20220729.03. Add url: prefix to url item.

e8fe03fbd047448a637dbd1d2bdd92187fd3b1a4 authored over 2 years ago by arkiver <[email protected]>

Version 20220729.02. Support older Wget versions.

2d8fa4034b81918c233b54376f0893dac4460963 authored over 2 years ago by arkiver <[email protected]>

Version 20220729.01. Queue media URLs back to reddit project and download individually.

f81b2ce97e82725f5bc99161aa79e35fc3e48c9d authored over 2 years ago by arkiver <[email protected]>

Fix README.

edacb2065ae1dd142fb60c95cd41bd822f33e207 authored over 2 years ago by arkiver <[email protected]>

Version 20220605.01. Support GNU Wget 1.21.3-at.20220503.02. Fix killing crawl when items cannot be queued.

cc83009a94ebdba84f40491215352de7cc05ca42 authored over 2 years ago by arkiver <[email protected]>

Version 20220415.02.

7c4cf4548e62fc115a7da4ce7150b923d72c793d authored over 2 years ago by arkiver <[email protected]>

Merge pull request #13 from NGTmeaty/patch-1

Add support for latest change in _options

754fd256cb1585e48697212f819f78d921a329ba authored over 2 years ago by arkiver <[email protected]>

Version 20220415.01. Do not queue /r/undefined/ URLs.

0ce1c59ca4c5cc657ffe7df17f7a887f63159380 authored over 2 years ago by arkiver <[email protected]>

Add support for latest change in _options

a858c33e29751242bf484e70b66ca8c494f13f8c authored almost 3 years ago by Jake L <[email protected]>

Version 20220323.03. Fix items to maxtries variable name. Fix backfeed key name.

da28d3c902c205f2337a31c2e80302e777e26919 authored almost 3 years ago by arkiver <[email protected]>

Version 20220323.02. Fix items to maxtries variable name.

8944cf1fc6c3b8f8ec3fa55fdcc20968ee652823 authored almost 3 years ago by arkiver <[email protected]>

Version 20220323.01. Fix backfeed. Fix maxtries use.

10eaa7c50c6fcde23e10ec16c41c767a1462cd4f authored almost 3 years ago by arkiver <[email protected]>

Version 20220312.01. Fix backfeed.

28f132a0526070405093cd8d69d4f3e57693e9f4 authored almost 3 years ago by arkiver <[email protected]>

Version 20220311.01. Use new backfeed endpoint for queuing.

4f50a0d6995e9cd65b80e22ef717cecbda4c5f16 authored almost 3 years ago by arkiver <[email protected]>

Version 20220109.02. Cut off URL at space when found between brackets without href= in front.

383c101aef0dacae550334dcedfbec55970ff5a0 authored almost 3 years ago by arkiver <[email protected]>

Version 20220109.01. Add codepoint to utf8 support. Percent encode outlinks correctly.

df35317e0c3601b6e5c1ad1c3a5c4112442b9ccd authored almost 3 years ago by arkiver <[email protected]>

Version 20211004.02. Fix incomplete facebook.com fix.

8a3f8cd1de366698e9ed1f5fae7d04d58628ac0f authored over 3 years ago by arkiver <[email protected]>

Version 20211004.01. Do not check facebook.com while down at the moment.

d0070db67a8714fa84455103888b2c9cf6ea8784 authored over 3 years ago by arkiver <[email protected]>

Version 20211001.01. Use GNU Wget 1.20.3-at.20211001.01.

0c5e8cd3bdbb70e105cec07bf740fed8ffc889a0 authored over 3 years ago by arkiver <[email protected]>

Version 20210707.01. Do not get media for cross posts.

ed80cb5a9d6702d359cf6abb07e36fd1ee08baec authored over 3 years ago by arkiver <[email protected]>

Version 20210521.01. Use TLS 1.2.

4b976e2ea70c8931a8d12201622766e39bcc7887 authored over 3 years ago by arkiver <[email protected]>

use onbuild-based image

f4619bb17f5f4d361a7533c80fa3fb09570f10ed authored over 3 years ago by Katie Holly <[email protected]>

New day.. new wget-at 1.20.3-at.20210504.01

e6b876e9e64d04b1a5391beec37257c46347cb89 authored over 3 years ago by km09 <[email protected]>

20210410.01 - New day, new wget-at

1f9e995b4e407b71557f1871f0bcf535710f1e1e authored over 3 years ago by Thomas Glass <[email protected]>

Version 20210407.01. Improve video archiving. Detect if video is still being processed by reddit.

6e1584155001c1948d9c1f3c98f69c7235433480 authored over 3 years ago by arkiver <[email protected]>

Version 20210330.04. Only decode unicode characters in URLs on v.redd.it URLs.

1b3690d994a024c126cd837a850a852e8117243b authored almost 4 years ago by arkiver <[email protected]>

Version 20210330.03. Unescape unicode characters. Do not HLS for video.

ce7fff480d24a9a22cae1d4b6f59ca9809ee6604 authored almost 4 years ago by arkiver <[email protected]>

Fix typo.

ad04f45d4feecde30ad1ef38c2bba568bac86f93 authored almost 4 years ago by arkiver <[email protected]>

Version 20210330.02. Skip images that are only in JSON and not on web page.

adc7f9c6fbeb837da0fda7a6f07cea369d40871d authored almost 4 years ago by arkiver <[email protected]>

Version 20210330.01. Handle 403 on v.redd.it on deleted post.

07ed16c44bb8888785c16c97d92b021cd02ec690 authored almost 4 years ago by arkiver <[email protected]>

Version 20210321.01. Do not get all video sizes.

8849165130f257257905d4b23ba5c794b3172991 authored almost 4 years ago by arkiver <[email protected]>

Version 20210312.01. Get URLs with utm_* and context params.

d3b6659419daf4e27a1af42dbbf39ca3f4a44e02 authored almost 4 years ago by arkiver <[email protected]>

Version 20210306.01. Remove some AppleWebKir user-agents for getting 403s.

a5c798945cacfd73c2e37e32070986415e296ee9 authored almost 4 years ago by arkiver <[email protected]>

add 1.20.3-at.20210212.02 as supported wget-at version

eaad7cd7e7d08952738884aa400729174e92ceb9 authored almost 4 years ago by Katie Holly <[email protected]>

20210225.01: update dict url

3b4a2ef5a7e95fec99d4fee2f725bc6b13d6f62e authored almost 4 years ago by Katie Holly <[email protected]>

Updated warrior support

e6c33f9433baa13ad03a58ec5b00b34a9b92be69 authored almost 4 years ago by Thomas Glass <[email protected]>

Update tracker host

261a7f76d295c3688d7c9ccffbac3d4e18ed98a2 authored almost 4 years ago by Thomas Glass <[email protected]>

Support new wget-at location

3d8f85a08a5dfed3957969acd609ca764d9ba8ed authored almost 4 years ago by km09 <[email protected]>

Version 20210130.02. Fix merge conflict.

ac59cfa3ed19919da1d847c3f6032a491c368081 authored almost 4 years ago by arkiver <[email protected]>

Version 20210130.01. Support &amp; in URL. Properly abort selected items.

11d57773910fe27df4e0ce73ed72dbe3cbaf9c95 authored almost 4 years ago by arkiver <[email protected]>

bump version number

053de362878e77a8db8e0d314f04647a8f465eec authored almost 4 years ago by Katie Holly <[email protected]>

use gnutls base image

68b4dd7a210a476704a7a39118c7a6d30d49bc4b authored almost 4 years ago by Katie Holly <[email protected]>

Version 20210115.01. Use Connection: Close header.

2f96fa399c54b818af97cb3c1cc6ca9355fa7b5d authored almost 4 years ago by arkiver <[email protected]>