github.com/ArchiveTeam/tumblr-grab commits | Ecosyste.ms: OpenCollective

Merge pull request #40 from mutantmonkey/no_match_media

Don't look through media files for URLs

0f8da27aedee935f3693a07a3a89d68f74eb42ee authored about 6 years ago by kiska3 <[email protected]>

Merge pull request #53 from Fusl/excl_tag

Update to 20181216.03

16c7fc9c54bf0d27b8842a7ea8582f6393a11e9a authored about 6 years ago by kiska3 <[email protected]>

bump version num

bafcf811a07b8d4a416398154b22a167aaa12307 authored about 6 years ago by Katie Holly <[email protected]>

Exclude /:tag urls

c29a2ef4e4d53a51fec97ac76061ca17383187bd authored about 6 years ago by Katie Holly <[email protected]>

Update to 20181216.02

Insert ArchiveTeam into UA

0920291952fddb37f3c0f452322bdc0a2b24cf10 authored about 6 years ago by kiska3 <[email protected]>

Update to 20181216.01

7427d92752a3deef03e8c8d3cf2ac8a09cb36864 authored about 6 years ago by kiska3 <[email protected]>

Merge pull request #52 from Fusl/expbackoff

Implement exponential backoff and add higher retries (abort after 2047 seconds of failing)

dae5bfd56a89c8e5fafa5a36f584c70cdab207dc authored about 6 years ago by kiska3 <[email protected]>

Skip media URLs earlier (255 seconds) than others (2047 seconds)

17d5285c2cbe008217a51f43a940e182b9f045ff authored about 6 years ago by Katie Holly <[email protected]>

Merge pull request #51 from Fusl/failon403

Abort grab when IP banned

8b348c59bc4457aac5c08e6ad9189ecb86f45c75 authored about 6 years ago by kiska3 <[email protected]>

Add comment explaining the use for math.floor around math.pow

7b3fcf1b55df97eb07bb1e970fbf7bf34ef48f1e authored about 6 years ago by Katie Holly <[email protected]>

Implement exponential backoff and add higher retries (abort after 2047 seconds of failing)

0ace1b7dad6e5b99902e4dc67b1ee84bec4a3f4c authored about 6 years ago by Katie Holly <[email protected]>

Abort grab when IP banned

44a834b8e1fa638955b278e6476413325e8beb16 authored about 6 years ago by Katie Holly <[email protected]>

Merge pull request #50 from noorus/patch-1

Some formatting fixups and additional info

c23eb95673d07257f41342be689712cc6c4953bf authored about 6 years ago by km09 <[email protected]>

Some formatting fixups and additional info

a39216b6f4b694fb3264571494d17bb953be3f00 authored about 6 years ago by Noora <[email protected]>

Update README.md

84e5dc8d5f83281b5b0760b98cc27b75f9f952ca authored about 6 years ago by km09 <[email protected]>

FAQ tidyup

961d570da263679ee9f55827e442d92cee016eaf authored about 6 years ago by km09 <[email protected]>

Merge pull request #49 from terorie/master

FAQ

4007572a8a27e6d5403ef2f13778389076d1285c authored about 6 years ago by km09 <[email protected]>

FAQ

79bb8f4ce0edbca58978e1bb5c4324f24a43893e authored about 6 years ago by Richard Patel <[email protected]>

Update to 20181215.01

Change UA to original GoogleBot

ffd66c55e0ffefb9188de32f87f1099cc0e7e1b6 authored about 6 years ago by kiska3 <[email protected]>

Change sleep time between errors

075eabdf06550055213735d6cebc5750e556ff6c authored about 6 years ago by kiska3 <[email protected]>

Don't look through media files for URLs

a3ddf6cb7734c3f20a7fd8e2d9ce18ad35955abe authored about 6 years ago by mutantmonkey <[email protected]>

Update version to 20181213.04

2b9be95ddaab1e2920c056eb1168414bff652797 authored about 6 years ago by Arkiver2 <[email protected]>

Merge pull request #34 from marked/patch-10

Nix: /embed, /notes Limit: /likes Combine: various regexs

e66a02f1550c70246672821ffaa2febc5eff8fa1 authored about 6 years ago by Arkiver2 <[email protected]>

Merge pull request #31 from LucasLarson/patch-1

Use HTTPS where possible

da4a1523500bd39f546ac0be14471640b85a1a71 authored about 6 years ago by Arkiver2 <[email protected]>

Re-enabling /likes/ pages , Unlimited

3a0f79c0bd7e3303b8c022eb5d92505669e79eab authored about 6 years ago by marked <[email protected]>

Merge pull request #2 from marked/marked-patch-11

Marked patch 11

db121190af064d0f9a55c1dcb7b87962b7b75e70 authored about 6 years ago by marked <[email protected]>

Block /:page tag also

9ebf09153ee8c16177123d244ffe51792787689b authored about 6 years ago by marked <[email protected]>

Block :tag in URI's which are template

most likely javascript

3432baca449fec24a5466078815f46c6f9744fa1 authored about 6 years ago by marked <[email protected]>

Undo -light tag in VERSION

e40ffe79f588c2ee393fbe2ec2f1efd4717e15bd authored about 6 years ago by marked <[email protected]>

Combine avatar img file regex

977789647ac5f461e568939c9f35a1246157f644 authored about 6 years ago by marked <[email protected]>

Correct /embed regex

fe4b0b0561732cf1253e483a1aa87a3ea2c528c2 authored about 6 years ago by marked <[email protected]>

Combining avatar image size restrictions

712b99294478709d6bab23a18932f0643d30eb01 authored about 6 years ago by marked <[email protected]>

Anchoring /amp and /embed to 4th position in path

ce6b473568d9d816fe7593709d55718431c2b491 authored about 6 years ago by marked <[email protected]>

Removing unused character class

3a73c17dcd7848597f1b530e37e693bfcbfbb332 authored about 6 years ago by marked <[email protected]>

Tag version -light

485c4ec839ecbd64332d7a24f7cfb62d6c935caf authored about 6 years ago by marked <[email protected]>

Bump version to 20181213.03

a1f226b2812fc28bed09b2cc059ceee6383142a8 authored about 6 years ago by marked <[email protected]>

Limiting /likes/pages/* to 99 pages

c67947d3ffce3ce891ae8d9157555e41a648650f authored about 6 years ago by marked <[email protected]>

Better block for /embed

eefaa57733e2b1108e28908fa809e41873c50fca authored about 6 years ago by marked <[email protected]>

Blocking /embed URL. Causing some 404's

a8f304285468ea6def543b1ec719364ea6b76d50 authored about 6 years ago by marked <[email protected]>

Blocking partial /notes to save crawl time

636907bf8a6dd9d742c3666e9f30f6f417062dfe authored about 6 years ago by marked <[email protected]>

Update to 20181213.02; removing random hosts from ignored

78512e27ddc6e65437fce88db678d9db1df61f72 authored about 6 years ago by kiska <[email protected]>

Merge pull request #33 from ArchiveTeam/test

Update to 20181213.01

f3be8a916d00b3565ed9b7a1ff1085b14f8e8672 authored about 6 years ago by kiska3 <[email protected]>

Ignoring results that return >=500 or >=400 that is an image from an external hosting source, or tumblr

d7d32e34acb8c2b326885cc2ef6caaeb6a580c82 authored about 6 years ago by kiska <[email protected]>

Ignoring errors on wielkie-hu215.metal-invest.pl, de05.cdn.z5o.net and adult...

711290569859a5e7e67ad777e8a5c1de7103321e authored about 6 years ago by kiska <[email protected]>

Update to 20181213.01; Ignoring tumblr /services/ endpoint and removing counter.website-hit-counters.com(dead service)

7ece748f82c716a0a2def699ed38d7ac412117cc authored about 6 years ago by kiska <[email protected]>

Merge pull request #32 from Fusl/excl-avatar-sizes

Exclude avatar sizes 16x16, 24x24, 30x30, 40x40, 48x48 and 64x64

dd22eeb9e67ed7b360bd7d399294fdbfa24f912b authored about 6 years ago by Arkiver2 <[email protected]>

Bump version number

2c2369c1202f78561e2367760d6dbffcafdecbb6 authored about 6 years ago by Katie Holly <[email protected]>

Exclude avatar sizes 16x16, 24x24, 30x30, 40x40, 48x48 and 64x64

21:27 <Fusl> so looking at https://github.com/ArchiveTeam/tumblr-grab/blob/master/tumblr.lua#L52...

3c60260405c3af3219fdd3cbdcb4199fe0353d81 authored about 6 years ago by Katie Holly <[email protected]>

Use HTTPS where possible

3a041d278bcc68b06e8fe526e13044466941f093 authored about 6 years ago by Lucas Larson <[email protected]>

Merge pull request #30 from ArchiveTeam/test

Update to 20181212.02
Fixing #22

f44866a6e6056fa493a281e040f6698ea2f9bc59 authored about 6 years ago by kiska3 <[email protected]>

Merge branch 'master' into test

83e49994c2bb9dbfc02b42d10d65ad0a4ec68f89 authored about 6 years ago by kiska3 <[email protected]>

Merge pull request #29 from ArchiveTeam/revert-26-master

Revert "Fast forward test branch"

42efcb90b47f4acb8041ab71505bae9fe4c3f89a authored about 6 years ago by kiska3 <[email protected]>

Revert "Fast forward test branch"

5fc861f2c2d1595813711e9c3ab77ec6ec7b36bd authored about 6 years ago by kiska3 <[email protected]>

Merge pull request #27 from ArchiveTeam/revert-25-fix_ignore_list

Revert "Fix use of ignore-list in download_child_p"

71db8e0715d6d222bea7b3747c28a0dc5b954589 authored about 6 years ago by kiska3 <[email protected]>

Revert "Fix use of ignore-list in download_child_p"

2221f41cc929586c1d2d14de08d17760d52e41c1 authored about 6 years ago by kiska3 <[email protected]>

Update to 20181212.02

Fixing issue #22

f2c70e0db80923d6afb1d1fc70fde3e6caa41134 authored about 6 years ago by kiska <[email protected]>

Merge pull request #26 from ArchiveTeam/master

Fast forward test branch

60d7640a7ac14e25cec6fcfa1b086437a4a9b645 authored about 6 years ago by kiska3 <[email protected]>

Merge pull request #25 from mutantmonkey/fix_ignore_list

Fix use of ignore-list in download_child_p

61a68e921b7c053c2f4d1050cb08b934970bfcb2 authored about 6 years ago by kiska3 <[email protected]>

Update pipeline.py

83fcf028e25377c432373158f4052312b09e906c authored about 6 years ago by kiska3 <[email protected]>

Bump pipeline version

6e38e311403f1ef36e9ee3675b179e2304cad75b authored about 6 years ago by mutantmonkey <[email protected]>

Fix use of ignore-list in download_child_p

6413027913f621d0d41fb77e58d4d12c4eca55fc authored about 6 years ago by mutantmonkey <[email protected]>

Merge pull request #24 from marked/patch-7

More URLs block oembed, amp, rss

9093c097b70b1dbb7c60316216baac6f94dc84f8 authored about 6 years ago by kiska3 <[email protected]>

Update to 20181212.01; Removing Jquery and googleapis fonts

4bbafb539b7b29dcd9b21c2783ebeeb8e3f1b1c0 authored about 6 years ago by kiska <[email protected]>

Bump version number

8f2a3a47eec60cd2422f2c2f1c0b6a36250e6dcd authored about 6 years ago by marked <[email protected]>

More URLs block oembed, amp, rss

bfc71b51dc9f567af8ceecee5a41ec06cc5bbe85 authored about 6 years ago by marked <[email protected]>

Update to 20181210.05

Removing "?route=" analytic

b7adcfc38eb5c39da99b96fd69ca4e481d2def7b authored about 6 years ago by kiska <[email protected]>

Merge pull request #18 from kiska3/master

Update to 20181210.04; Removing reblog links

74e220320612e5aaaca13797b5fd6fa624c26e56 authored about 6 years ago by kiska3 <[email protected]>

Missing OR

9a011152859b1c7f3b91af0eeb928a926cbc4571 authored about 6 years ago by kiska <[email protected]>

Update to 20181210.04; Removing reblog links

c91770ff8261c29e5b08f314bbacc7ae40958fc0 authored about 6 years ago by kiska <[email protected]>

Update to 20181210.03; Add static URLs to ignore

dce2a05bd6e7e84331aa39fe54875a056127441a authored about 6 years ago by Arkiver2 <[email protected]>

Update to 20181210.02; Extract blogs

85071c7a6269bb4281393a2bb5f9062726eeccca authored about 6 years ago by Arkiver2 <[email protected]>

Merge pull request #14 from marked/patch-3

Fix keeping the URL length at a number of / occurences.

d185fd57d90c11186670660415b9d82be7d84902 authored about 6 years ago by Arkiver2 <[email protected]>

Bump version number to 20181210.01

Patch for PR #11

b821bf0d05bcbfe2ada93667fcaea29d3ea62f14 authored about 6 years ago by marked <[email protected]>

Modify Regex for % encoding in URLs

Fixes PR #11
(Banned % encoding in URLs not being detected all the time with URLs getting long...

f0616a4b01d35a17ee01ba4057c799d6c0041061 authored about 6 years ago by marked <[email protected]>

Merge pull request #7 from mutantmonkey/fix_avatar_exclude

Fix ignore of small avatars

4ba0764d025f1628f87c20bd0a1bf931cf890097 authored about 6 years ago by km09 <[email protected]>

Fix ignore of small avatars

Previously, there was a logic error in the condition to ignore small
avatars. Although the patte...

84ecb37569c1fc6334d75852ec432c872074116d authored about 6 years ago by mutantmonkey <[email protected]>

Merge pull request #6 from marked/patch-1

excluding more 16px avatars

44a3371d62e691aec7d755cb50a921bfcea5a987 authored about 6 years ago by km09 <[email protected]>

Bumping version # to 20181209.02

Added Avatar 16 gif exclusion

7ddf1b598ea520b898f9f735e8a4307a1e410f8a authored about 6 years ago by marked <[email protected]>

Merge pull request #5 from joepie91/patch-1

Fix remaining placeholder

a314c4281d4a30f59b1e83355b566919c7282c64 authored about 6 years ago by km09 <[email protected]>

excluding more 16px avatars

Avatars can be .gif

87b1cb595524ba89c749fcad7730ba55b38c04fc authored about 6 years ago by marked <[email protected]>

Ignore _16 avatars

80436f4cd75c1ef225e6a9e155590f78b7f1f3ba authored about 6 years ago by km09 <[email protected]>

Fix remaining placeholder

72dcdaea3c8a7b2dd426ee5897bf151e99d48463 authored about 6 years ago by Sven Slootweg <[email protected]>

Update pipeline.py

402e12233ff6bee527a6489ad793c199e6ad4bef authored about 6 years ago by km09 <[email protected]>

Update pipeline.py

b351806b1277a71285ebb65ee537678364c7f762 authored about 6 years ago by km09 <[email protected]>

Merge pull request #1 from marked/patch-1

Adds support for builds on RedHat systems

7b5522040ee9585c115c018dfc350975be9abe51 authored about 6 years ago by km09 <[email protected]>

Adds support for builds on RedHat systems

Upon a build failure, patches config file for RedHat library path, attempts reconfigure / rebuild

4cf8cc43ec62995cfd1842adad23ea6672b53c68 authored about 6 years ago by marked <[email protected]>

Update to 20181208.12; Update README

cf19f8627197e89eaac790602586851a00b75b18 authored about 6 years ago by kiska <[email protected]>

Revert to 20181208.11

289b6438195dd156f2989e9697f3b01d2f0d3633 authored about 6 years ago by kiska <[email protected]>

Update to 20181208.12; Forcing TLSv1 and IPv4 causing some issues with kiskaAus

d09df155842c9fb6dd5746b7f061b38c7addb545 authored about 6 years ago by kiska <[email protected]>

no message

35759520f13107e055ebaf1fff368b3ba608a345 authored about 6 years ago by kiska <[email protected]>

Update to 20181208.11; Extend sleep to 10 seconds, to allow tumblr server to cope

db808bca23881f72d59e62459fd4b455698a74d9 authored about 6 years ago by kiska <[email protected]>

Update to 20181208.10; Make ios-app url obvious

55f4294edd10e69738c6d3750136a6f66694c2cf authored about 6 years ago by kiska <[email protected]>

Update to 20181208.09; 403'ing on www.tumblr.com

b17a5243b846e0e49acd46f2ccc0673d030da5f8 authored about 6 years ago by kiska <[email protected]>

Update to 20181208.08; Remove 403 issue on vtt.tumblr.com

fd63d78df2c656588f4d3dceca37c60337407177 authored about 6 years ago by kiska <[email protected]>

Update to 20181208.07; removing debug messages

6969b4055a4ed99111d54bb074eef85dfbe21795 authored about 6 years ago by kiska <[email protected]>

Merge pull request #1 from kiska3/test2

Finished scripts

8252feee04db44f0962a447dae7280ded2511518 authored about 6 years ago by kiska3 <[email protected]>

Merge branch 'master' into test2

4bfcce94b626701af7668bd689d7336de911ab97 authored about 6 years ago by kiska3 <[email protected]>

Allowing www.tumblr.com/video blog urls

3f078e69bb073aaf9e384c138ffdc58b64d0d86a authored about 6 years ago by kiska <[email protected]>

Update to 20181208.06

e32766d3928daaa724dde1f3d3b7be6a1a286996 authored about 6 years ago by kiska <[email protected]>

Update to 20181208.05; Include all [a-z]+ media subdomains

cfc4f820da9603b7ff579eb8dc8bda06e6d74024 authored about 6 years ago by kiska <[email protected]>