Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
https://github.com/ArchiveTeam/grab-site

Ignore https as well

23344a0d8c4339a2777d54ddc1a067b36c85f818 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another Icecast site

375f036c11835b8990229e09e20d8d6fb41755fc authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore loops on gdcvault.com

39704b6598db1aad50a9730345d1d3a408494bc1 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another Icecast site

e647876e258b9070748bf48f699579319c5035fe authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore TMZ videos

These have popped up as page requisites twice now

973b2731eeca9616a275bbe2efbc951eb676da02 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore Special:MobileOptions and Special:MobileFeedback

42bc68ce2eeeacae482dc52c3b1bd41c3c8ca166 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another Icecast site

0cb5b0c1c60b01d4377cb35254365ef1d9da60ba authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore www as well

60e89778b3b25cb962ffc8ec5a823e514ab04fa8 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore some mediawiki loops

61eeb41ce1eff91e48fa2a23e3de022a9ffddacf authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore web-beta.archive.org

e7b67e2b60b98af3c5421269e8cf44a742e7c26a authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore blogger.com/email-post.g

dd124c90f505c8492a955b20b97d19ddcd3159b5 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore loop on photobucket.com

7b2244388ed2b9db3d5dc0804dbf1ace71644c44 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore runaway %25252525

1b29d0dde53466e0cc3f695a9111decbc3797c58 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore dead ad/tracking site

5523b6fe53cab78c700c6ba301c798e7bc4c9b90 authored over 9 years ago by Ivan Kozik <[email protected]>
Add nogravatar ignore set

f885656cd074448b980a5af350ff1f114116db31 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore Google Analytics tracking image

b3ad482143cd038ea01dfe5c2021565ae9f3acd4 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore infinite /styles/ and /scripts/ loops

f3a0c218138ce25e4af98a04598a744bdc69d42d authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore dead host

a424eede4157ce09d9d0f9eb905acb5285951821 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore local/private URLs that have a :port

8acffe9b6695dcfeaca0c81dfb3f27cdcb1ceb10 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore add.my.yahoo.com/rss

83c0718adccae3fd20b64b45c0e3352bb8784174 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore /privmsg.php

5871f92710c9f46028605b34509d6e16be329520 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore flickr change-language page

81b268efb9d66805738ce5138c38b645e89fada3 authored over 9 years ago by Ivan Kozik <[email protected]>
Remove ignores moved to the global set

f8396764adef4d3e096833fdd2590eb4a29ebf8e authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore junk on connect.facebook.com as well

bbecff69bd16fbcaeb64f09782594a45ce9c59bf authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another Icecast site

e0db95f935b692333fa10aa45d955123422231b8 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore tumblr.com/share/link/

e49be9d6e2e208ae59f8d4593f45006ce929757d authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore megaupload.com

It's long gone

420bed747ee87eaeaa7b6b0f4ec5c4064eeff1e4 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore two dead file hosting sites

a9e384104b8cba2dcd1827d37c1d1a4a5672cf01 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore some JavaScript cruft that always 404s

c6a66cbfb0140c3e6b24e1306402e6fc4a3e7fcf authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore stumbleupon.com/badge/embed/

4e2f681513b1b2b219d01bc05a67876813064576 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore /%5C%5C%22 and similar

bcd2dca4e47b28265b2be1db01f045a2635fd0d5 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore %5C%27 as well

901093d092eb153dee26f607840b4402e57dcfe2 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore /%5C/%5C/

885d2fb731435dcbb162d60db74009b845c31ba6 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more myspace /PostTo/

6f558ee7cb7bb713cfbe48831ea8e6a77750b61b authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore some mp3 streaming sites

a79d66d5614872cb6bfdf4aeaa74a2a8e9af4ea7 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another mp3 streaming site

15e45be62b28b173a4f796ab89cd8201aed1cf43 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more plus.google.com share links

8eac0ae79c7d8362591383341782765a2cc0b0de authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore wayback.archive.org

d942960435ab0f964fb05c2c8096d6f1f18e2e09 authored over 9 years ago by Ivan Kozik <[email protected]>
Remove a rule obsoleted by one in the global set

44bf0ef7bd8a44130c08e7807f2a6c419c7f10fe authored over 9 years ago by Ivan Kozik <[email protected]>
Copy share link ignores and tracking bugs from the blogs set

e7976c5ace3159af05ecff449fb473545b845dcc authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore &hilit=

beb411e6ff34614cc176a1f22b53ea28c57a41c6 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore del.icio.us/post

b070df4bdcfc7f1f2f4c8f91cc5a250bc24ff898 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more JavaScript non-URLs

d5ff087ccfea9d420b0f0fd8eb7da898f1f3b65f authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another mp3 streaming site

0bd87327c15e8d54b824a548315716278c5c4347 authored over 9 years ago by Ivan Kozik <[email protected]>
Fix del.icio.us/post ignore

c0796ccb3826132c6a4e0b9b4e86558ed53078ef authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore dead Yahoo domain

479cdca7e18a53dc17674f4bb5f04ae995bd33b0 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore /quote-comment-\d+/

838c4b41d6e64ee7085c0ffb51b71440213c1701 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore infinite loop on facebook.com

c0fe88dc62b890936e28ab23d13f09176790e53c authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore some common infinite loops

4f09fc4c7be0399f182a88d3d0ade7e65a56d1d6 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore some more common infinite loops

5eac32ee917723b956ec18358210a65f747b74fd authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore dokuwiki loop

838de0b90e743d14bd22ce13153a6d5b9e227dba authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another default gravatar

3da36a6a1fa443f40c3108d3db095fcbdf443dfb authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore some bad joined URLs

These don't point to real ad trackers, they are joined URL nonsense

00085f616281d1fcf0d34b05b5418f786b163636 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore &hidepatrolled=

cbc884233a7a14bca33fd518e0c96c456adb4419 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore infinite loops on khaleejtimes.com

11fc98a472f3c35d369fee0c04623369afd88916 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more khaleejtimes.com loops

1594f5375b88f070a8e1d695a069c2e85a602130 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another facebook infinite loop

https://www.facebook.com/connect.php/js/FB.SharePro/rsrc.php/v2/yo/r/rsrc.php/v2/yY/r/rsrc.php/v...

aefd9ad866da55eb797c2e0f383520b8f4cc23e3 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore reCAPTCHAs

d5dca3627258172315d8057c59672cf71ebad10d authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another facebook infinite loop

7c6f1b0beb8012b207ff80ba67641e30517ef382 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore some bxSlider loops

f8419014c0a65f298ee345b2a8958bb7cf100fb3 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore blogger.com/comment-iframe.g

ef60f59a0374d6d04d18cd2ff4313a5ec7610272 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore local and private IP ranges

3f0f6f169de5a275ccba39a7b9a9cc9b07cc77f8 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore fbcdn.net URL that causes infinite loop

a067b614ca5db9165e72bed9dc19e57703858541 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore addthis.com/bookmark.php

474efeadaed63c042b95d79b1cf3b410faa2dc1a authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more blogger URLs

45e1af6d35a5e61019e41810e3a0badbdffeb1f0 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more share links

0e9a5143649eaf17362e1a491d4bf0b26ef9076f authored over 9 years ago by Ivan Kozik <[email protected]>
Also ignore /delete-comment.g

c735250fd3233f2d00c046e73ce0c50e2d77a7eb authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore tumblr.com/reblog/

f7952b0b710227c99700205bcb462cefe9de975d authored over 9 years ago by Ivan Kozik <[email protected]>
Also ignore Special:Translate

b76b1cf3e634ed2eb861e5145c021f078ca7d5cb authored over 9 years ago by Ivan Kozik <[email protected]>
Add untested reddit ignore set

4fbeb39dbecd4c94559325b141d4c16426324d88 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore /ljcounter/

6da1690aad53acc7b9daf23e0b598840132ed62f authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore /wp-login.php?

09b954eb6a985e1adff8db8f075cf57b25448873 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore reddit comment links that a ?context= as well

66e605a4b8c6122c8e5f7b7b61e38008d083df9f authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore dead/internal website

7bcc5f2a50ae751841885138158a5f94f6144eac authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore http://b.scorecardresearch.com/

d0cd692882a7dc9419c67c72fe88b5d69aae2d63 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore blogger.com/delete-backlink.g

3bcfbebbc58427bb2883facb990d9a8d7b1babce authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore some netvibes.com and add.my.yahoo.com

04c3eff0bc339fb4877779b0d310b291f1648f96 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore archive.org

e65bb33a14c67773b2877909787c145e10286e8c authored over 9 years ago by Ivan Kozik <[email protected]>
Unbreak blogs ignore set

5d363db982f37edbfd2b0728f2691a2147161abf authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore tumblr.com/share/link

6d0a9c50be70623948eb594de474ab5a6b98a5fd authored over 9 years ago by Ivan Kozik <[email protected]>
Narrow the livejournal ignores

a321e42c09c3141e69170d1ec3d02ef7824c19a7 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore stream.publicradio.org/

It has never-ending responses

0047d11fad98f94882719f6e2fce1926093a9efb authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore ?mode=reply on dreamwidth

5b447aba03aae016727546ba874e6db9f96c7177 authored over 9 years ago by Ivan Kozik <[email protected]>
Delete livejournal ignore set; use blogs instead

ae0875bae9bb6870214f1460bc716f9a1f5be15e authored over 9 years ago by Ivan Kozik <[email protected]>
Copy ignores from livejournal set

4af5b67b4a83b4fc207ceb48883f12291f09c5ad authored over 9 years ago by Ivan Kozik <[email protected]>
Add global ignore patterns

7abcfcce0630e11bd2c0e0cdbe094cef36b090f1 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more tweet buttons; newsvine links

18583e93a39629597657291dc7ef7c2a66d7a7e3 authored over 9 years ago by Ivan Kozik <[email protected]>
Create livejournal ignore set

8d4fffd2edc81a8ed14510e63d5677b826e2f63d authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore blogger.com/rearrange

17418e0d41d7efd8f5eb4c74f1724530c55bb868 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore reddit.com/message/compose

6c601fba044c62ecabbac937088e574363d04461 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore another share link found in latest Sociable plugin

3940af54e0720caf93006dfb62cf1b121cee3e49 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore /editpost.php

Spotted on http://www.ikeafans.com/forums/ which runs vbulletin

916792de8606caba7d9575e2e46edae758cb4f0b authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more reddit share links

bde21305515acb1110cbf6976b122ab5627c975b authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore &printable=yes

0217ca206caf271fc659a769f5654f8dda40c1c3 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore more share links created by the WordPress Sociable plugin

1c97ce697d476defb977a8819586c6f3dd362284 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore livejournal.com/update.bml

303ff1634026676eecccc17dd909f2ccd54c9535 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore ?mode=reply

77d229777ff29069bad22a1fa46fdd77e112df8a authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore /stats.g.doubleclick.net/dc.js

022195f2f3d6d150fec9c1b42d8f6707095a5756 authored over 9 years ago by Ivan Kozik <[email protected]>
Ignore http://pixel.blog.hu/

d63ceee6c93cec62c3918e7b4fa6f699b59e96a6 authored over 9 years ago by Ivan Kozik <[email protected]>
Combine two ignores

eeb21fc3259a7f61199a13a5fcc0f6f6636a9e76 authored over 9 years ago by Ivan Kozik <[email protected]>