Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/tinyback

A tiny web scraper
https://github.com/ArchiveTeam/tinyback

Add default rate-limit of 2 requests / second

Use a default, low rate limit and let the individual services specify a
higher/lower one if nece...

cf9be3fcf1e23a41789211390b0030b863cd4fe8 authored over 12 years ago by David Triendl <[email protected]>
services: Add Owly

501eace5b659e8d9de2d76641ff067cc4cc0eea7 authored over 12 years ago by David Triendl <[email protected]>
services: Use URL as class documentation

d139365ba83bb010a6e71b170b811abaa9d9fa0c authored over 12 years ago by David Triendl <[email protected]>
services: Add Klam

5bf1d0d70f99bd765dbd3510de94455c47fa89e6 authored over 12 years ago by David Triendl <[email protected]>
Tinyurl: Add 2req/s rate limit

44912bae879bd1cfeb579cfb658e6a630cec83b5 authored over 12 years ago by David Triendl <[email protected]>
services.Bitly: Add "rate limit" of 2 requests per second

Try to guess the correct rate limit since bit.ly keeps banning us even though
we obey their robo...

59ece55f615fe8085d156ca4edc48887aa3ce7e2 authored over 12 years ago by David Triendl <[email protected]>
HTTPService: Add default timeout of 30 seconds

c3358effb38378f3253e4302ac0f223e80dac947 authored over 12 years ago by David Triendl <[email protected]>
Update README

87772c5d4df9e86ea9832ad499a77b10a887a7b2 authored over 12 years ago by David Triendl <[email protected]>
pipeline.py: Add correct tracker URL

83ff58a486ff7ac204ec0b9e20499472eeda12c6 authored over 12 years ago by David Triendl <[email protected]>
services: Add HTML unescape

Unescape URLs that were just read from HTML.

89cd11cef444d3794e813bbcc3a48806c52dd778 authored over 12 years ago by David Triendl <[email protected]>
services.Tinyurl: Get real URLs on "advertisement" links

Tinyurl.com inserts it's own affiliate ID into URLs. To get the real URL,
tinyback fetches the p...

9922f0ee0a43f8d322565bf099d1fd09b8013960 authored over 12 years ago by David Triendl <[email protected]>
run.py: More verbose log format

630cc49e7c72891b41340e167b42c8c6d0fe00ab authored over 12 years ago by David Triendl <[email protected]>
generators: Add "sequence" generator

Python port of the shortcode generator used in the ruby version of tinyback.

35800923d1bed6b21d35cb49f82ade8b5f09a4d6 authored over 12 years ago by David Triendl <[email protected]>
run.py: Fix multithreading

fa86e6d1a07c7c61b067bf8859511d6c1e5d3fb4 authored over 12 years ago by David Triendl <[email protected]>
Reaper: Add "backoff" after being blocked

aa0d8f84a3ce922cb64d10e8e74d87ea8d7e2f12 authored over 12 years ago by David Triendl <[email protected]>
HTTPService: Handle http and socket errors

Reraise http exeptions and socket errors as ServiceException

183b7dc93652136837e689b29842f8b191a8ed3d authored over 12 years ago by David Triendl <[email protected]>
Services: Create new HTTP base class

Refactor services to use a common HTTPService base class

ee269c591d8d4d6c31c58d2ce405e9af124a63bc authored over 12 years ago by David Triendl <[email protected]>
services.Bitly: Fix HTTP status for "blocked"

45bf29450bcfb9be8a28e6b597de0d3c1c04e64a authored over 12 years ago by David Triendl <[email protected]>
services.Bitly: Hourly rate-limit

e835ca9f644c7d4df8ebb577ab73763d12025f11 authored over 12 years ago by David Triendl <[email protected]>
Reaper: Use gzip compression for output file

619304d484ab7e4aa1743c420e26a18d5858f953 authored over 12 years ago by David Triendl <[email protected]>
Add "default" rate limit

5a38e55c4f6ba219faf38f3f003848bd79a0617b authored over 12 years ago by David Triendl <[email protected]>
Services: Add Ur1ca

923cadadf9a9e9504c39d4ec74f8be1f9be1b2bc authored over 12 years ago by David Triendl <[email protected]>
test-definitions/tinyurl: Add "errorhelp" that leads to URL with non-ASCII characters

237b1da75d7ccc0e2597a42fd0691c031e52fffa authored over 12 years ago by David Triendl <[email protected]>
Update Readme

27f6d2f706a1406010347d0a500a8cfbf177ac2e authored over 12 years ago by David Triendl <[email protected]>
Services: Add Tinyurl

32b7a112535549c6b279d2e603239b458ffda4f1 authored over 12 years ago by David Triendl <[email protected]>
Add version number

8ae3bf50756e35a5316025e7608e5eb8d9e80122 authored over 12 years ago by David Triendl <[email protected]>
Add "Reaper"

596e00d67a9faba84b06fb978a1534190af563e3 authored over 12 years ago by David Triendl <[email protected]>
Services: Add factory method

bd803f924e213bd717e066896a6ad05437550d9e authored over 12 years ago by David Triendl <[email protected]>
Services: Add Bitly

7dd850167408e105718380358a93453aa19a06e2 authored over 12 years ago by David Triendl <[email protected]>
Services: Add Isdg

675d0735ad8023c6a97e1b22947cc7a6eded7a91 authored over 12 years ago by David Triendl <[email protected]>
Add ServiceTester

59ceee6b5b04fa763cd6a402dd1cdcc00b7e57ca authored over 12 years ago by David Triendl <[email protected]>
Fix tinyurl test defintion

153e2401e4992ddc19dc74c8e94f07c1d3faf8e1 authored over 12 years ago by David Triendl <[email protected]>
Add pipeline

c6da907f11204e34dc194b69276cb48de4ddb7e2 authored over 12 years ago by David Triendl <[email protected]>
Add run.py

707d5ad75b0d0c5389351a2b8d2f876c3dc15dd0 authored over 12 years ago by David Triendl <[email protected]>
Add tracker client

79904b8217f30031252cb47d41550cf7b70cc693 authored over 12 years ago by David Triendl <[email protected]>
Add test-definitions

Converted Ruby unit tests to simple text files

6e747b65bc5199e516c5304b80c5ac7ca958fe39 authored over 12 years ago by David Triendl <[email protected]>
Add exception tree and abstract service class

845b9f9cd6760554f8abbe0731b6cf82db7aef29 authored over 12 years ago by David Triendl <[email protected]>
Remove old tinyback code

9067f2bdbaf8037e8494b4a60c06a4b8cae6cd56 authored about 13 years ago by David Triendl <[email protected]>
Reaper: Don't terminate on newline in URL

825ca34d86cd3fe5bd0122285a766df5af81d3e2 authored over 13 years ago by David Triendl <[email protected]>
Reaper: Don't retry on CodeBlockedError

28c38a7d165d0aa1a20192cd20e161a493a14368 authored over 13 years ago by David Triendl <[email protected]>
Reaper: Fix stupid threading bug

1c06ce70c681bf0bcc879e83a04a29084c7c4bd9 authored over 13 years ago by David Triendl <[email protected]>
Services::Bitly: Add reserved keyword "bundles"

76d9aabdba92ff8cca102a98216fc094c3705071 authored almost 14 years ago by David Triendl <[email protected]>
Services::Bitly: Do not use IPManage anymore

Bit.ly switched to using one IP address exlusively (even for bit.ly aliases).
Using IPManager wo...

902138bca6702636d4cb565418b4552fa9781007 authored almost 14 years ago by David Triendl <[email protected]>
Services::Base.advance: Do not modify string that is given as argument

Instead modify a copy of it

28a1e350d146716c79194ffbd65aa916c3582b21 authored almost 14 years ago by David Triendl <[email protected]>
Officially license the whole code (GPLv3 or newer)

8394897cc3cd5a1764def4ed830c65bef2d78d23 authored almost 14 years ago by David Triendl <[email protected]>
Ditz housekeeping

* Mark "Runable version" as released
* Add v0.2 release for near-future tasks
* Shift some hard ...

e236c347f8e6a04782ac2fd1a324c1d4bee3e835 authored almost 14 years ago by David Triendl <[email protected]>
Merge branch 'rework'

051440b034e1a7980f09139c885fd3a5313e8b51 authored almost 14 years ago by David Triendl <[email protected]>
Restructure Reaper

Removed fetch queue and generator thread. Each fetch thread is now responsible
for generating th...

8a38272d877ad2a972b1316337bb2b2653640a44 authored almost 14 years ago by David Triendl <[email protected]>
to_csv: Add prefix before code

fc7f8c16cba62ce31ae2a26c2b388fade625e846 authored almost 14 years ago by David Triendl <[email protected]>
Add to_csv.rb

aff4a4326b994ad620617a092a165c67750abd27 authored almost 14 years ago by David Triendl <[email protected]>
Remove Services::Trim

Service keeps breaking. Service must have been programmed by an idiot. /me
looks it up. Yep, Eri...

1e691c7a172a8426b5735884577e2146c77d6e10 authored almost 14 years ago by David Triendl <[email protected]>
Services: Don't canonicalize code in fetch, this should be done outside

694badd50493e8dc50286c0560a6cdf0bf80dbba authored almost 14 years ago by David Triendl <[email protected]>
Actually close "Write directly to compressed file" bug

3228d1f3a8d8e2c34c320b7eb2062955d8f169ab authored almost 14 years ago by David Triendl <[email protected]>
test_services.rb: Sort output by service name

37a9d3e087db29c70540a24e8839eee7b32abf61 authored almost 14 years ago by David Triendl <[email protected]>
Use different style for raising errors

Old: raise Error.new "Msg"
New: raise Error, "Msg"

The old style triggered lots of "warning: pa...

99ba4290572e2515ce03e0aee6beb84b3ab39065 authored almost 14 years ago by David Triendl <[email protected]>
Services::Bitly: Remove unnecessary dependency on Hpricot

00c69405981eb4c3c344e915c49192cf34b6f14e authored almost 14 years ago by David Triendl <[email protected]>
Add automatic gzip compression

bd8c047e9969bcfd93d81c50620563c6b21f2a02 authored almost 14 years ago by David Triendl <[email protected]>
Some ditz housekeeping

Add release v0.1 and assign some bugs to it
Add some new bugs

9ec61d849f652b7735c3a4a32916618adc751d9e authored almost 14 years ago by David Triendl <[email protected]>
Remove C tools (sort, verify)

These tools will from now on be tracked in my urlteam-stuff repository

56a8349ddf5352d76a491567ed180d7ef0761818 authored almost 14 years ago by David Triendl <[email protected]>
Fix Services::Trim

f082ccf3a81c126daaa62329d09db247747156d7 authored almost 14 years ago by David Triendl <[email protected]>
Add repair_chronomex.rb

fee99efce66530424ea7b8fd084e84837e0e544d authored almost 14 years ago by David Triendl <[email protected]>
src/shorturl.c: Improve compare_codes

c7d1e36de30361c9037c7629cf1aae8a79e03dae authored almost 14 years ago by David Triendl <[email protected]>
Fix "won't terminate" bug in reaper

2ffa94000c177f5586ced1605085c55f30e9081e authored almost 14 years ago by David Triendl <[email protected]>
Add src/verify.c

5b6a2c71588cfea5f62390f03e51ffe8af5aa512 authored almost 14 years ago by David Triendl <[email protected]>
Services::Bitly: Fix Cookie trick

d8c7fb6a1eab16f1b3565180c23c66cde538ffa3 authored almost 14 years ago by David Triendl <[email protected]>
Services::Bitly: Remove whitespace before/after URL on 302 responses

1e0dda34e44235f60bfdf90948231de67bced4b9 authored almost 14 years ago by David Triendl <[email protected]>
Ditz: Add bug ("Reaper does not terminate")

d7721a326d5bbe434b789a0f2a5c75c9dab89ad1 authored almost 14 years ago by David Triendl <[email protected]>
Services: Fix is.gd

b29b36837b412f3065a3232a640579920a28bba1 authored almost 14 years ago by David Triendl <[email protected]>
Ditz: Add 4 new issues

d8453a5a710627d6c63a8a1adc1353b0984527a4 authored almost 14 years ago by David Triendl <[email protected]>
Services::Bitly: Fix charset (non-critical)

Switch letters 'N' and 'O'

16759800f9067240f45c648d92e0cd94f58ff49c authored about 14 years ago by David Triendl <[email protected]>
Services::TinyURL: Remove no longer needed dependencies ("resolv", "singleton")

cb4b3d74d40758e41b383465e296ae553624d587 authored about 14 years ago by David Triendl <[email protected]>
Add remove_duplicates.rb

64ae308204fe02aa0b2dcdfea5c55e52de895b95 authored about 14 years ago by David Triendl <[email protected]>
Ditz: Add 3 issues

67dcb788387c7055011bf96e23456d0e7d39ac22 authored about 14 years ago by David Triendl <[email protected]>
Reape: Only increase/decrease sleep interval by 1 second a time

6d50da786b07e8f7cc104ee08422b65baa7829e3 authored about 14 years ago by David Triendl <[email protected]>
Add test_services.rb

08a9bee5c91ff61518582a3d1da5c2dab118197b authored about 14 years ago by David Triendl <[email protected]>
Merge branch 'blocked'

881d08bd69a99d8a03139fe8c224485b3b74ed64 authored about 14 years ago by David Triendl <[email protected]>
Reaper: Give up when more than 10% of all fetches result in the service blocking us

15abf0491844471f7265d03ea0fb0e4ec41449dd authored about 14 years ago by David Triendl <[email protected]>
Fix test_nonexistant(TC_Trim)

Closes services-3

34f210cad1ef46965fc13bf316361322bf611e16 authored about 14 years ago by David Triendl <[email protected]>
Add ServiceBlockedError

ae69e6363b49004d4fa8850feff2e468cce42a1f authored about 14 years ago by David Triendl <[email protected]>
Rename BlockedError to CodeBlockedError

3a41b159b62971ea135e7f78d873ac5209e15c3e authored about 14 years ago by David Triendl <[email protected]>
Ditz: Add release "Runable version" and three bugs

655604e6429c100e74da717c6e4475bacb183c95 authored about 14 years ago by David Triendl <[email protected]>
Fix bug in Ruby's Timeout module

1f079542c44b28cadfe7c5037684816cf8e900a3 authored about 14 years ago by David Triendl <[email protected]>
Ditz: Add two bugs

04a1f9a60e81bdd58c77357d8f8e1f9bc046fa0a authored about 14 years ago by David Triendl <[email protected]>
Various Services: Fix Location header regurlar expression

db1bec36c1363d141d185b78fb1903ec1180cb8c authored about 14 years ago by David Triendl <[email protected]>
Add Services::Trim

b25a7fe9f3c046133658ac340eafbeffa9d0b7dc authored about 14 years ago by David Triendl <[email protected]>
Services::Bitly: Recognize some reserved keywords

9e343264a9b00c3e6bb466bd106abff4523680f1 authored about 14 years ago by David Triendl <[email protected]>
Merge branch 'bitly'

20bab04413eb592db1a4c3caa56cc925377e42bf authored about 14 years ago by David Triendl <[email protected]>
Add Services::Bitly

Should be working, according to my Unit tests ;-)

7ff4c5525f634c0014c16adab70c09584b3bb90d authored about 14 years ago by David Triendl <[email protected]>
Add IPManager and use it for Services::TinyURL

727bef6a6cb3339154fb441b96d39af81d30b975 authored about 14 years ago by David Triendl <[email protected]>
Bitly: Add tc_bitly.rb

79005b6c0cfac3040aa84a29c1e55d7a13d0f518 authored about 14 years ago by David Triendl <[email protected]>
Add ditz

4d18450a82f32899d1f2b1d5c289717696fccd4b authored about 14 years ago by David Triendl <[email protected]>
Services::TinyURL: Adjust to yet another change on tinyurl.com

d8ffbb5665a740c3ce7744eafb473e872a35694f authored about 14 years ago by David Triendl <[email protected]>
Services::Isgd: Fix non-critical bug in charset/advance function

5317f127dbdd8e2ab5c0f43d96c362a5683179de authored about 14 years ago by David Triendl <[email protected]>
shorturl.c: Fix compare function

9b0d9106e416b31820190ea326067ce6396b9318 authored about 14 years ago by David Triendl <[email protected]>
Move sort.c to src/ and split out the generic shorturl stuff

edf722d43730a638acb1e320fce1f6104f166002 authored about 14 years ago by David Triendl <[email protected]>
Add sort.c

a65ee49e9318c2d2b90e05cc1a30186e66a8521b authored about 14 years ago by David Triendl <[email protected]>
Reaper: Increase FETCH_QUEUE_MIN_SIZE_PER_THREAD to 250

8a997d5603fa3923ccf02466669969c942328a26 authored about 14 years ago by David Triendl <[email protected]>
Services::TinyURL: Remove additional newline on HTTP 200 response URLs

08274548263613f1ed60884a3c72b9625be949cc authored about 14 years ago by David Triendl <[email protected]>
Reaper: Sleep on fetch stall

d093e27059c370f2e3f4b0cd9d718cc9149c73a7 authored about 14 years ago by David Triendl <[email protected]>
Reaper: Terminate on newline in URL

1efd66b177338ba810629084e725b6ea7883bbc7 authored about 14 years ago by David Triendl <[email protected]>