Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ArchiveTeam/splinder-grab


https://github.com/ArchiveTeam/splinder-grab

You need the Hanzo warctools to run the verification script.

e5f05a74eb95e6e4ea5871dd54ea6bd983d9e099 authored almost 13 years ago by Alard <[email protected]>
Merge branch 'master' of github.com:ArchiveTeam/splinder-grab

03a991ff5f211714141ff497bfe7cac6d58e318b authored almost 13 years ago by Alard <[email protected]>
Add verification script.

840000c1c0c413752b978a41c5b9c0a18bc601d9 authored almost 13 years ago by Alard <[email protected]>
add check for splinder_noconn.html

Splinder occasionally redirects to a maintenance page named
splinder_noconn.html. Profiles conta...

5941168f80f8d410c5126d6189968de97dd636bb authored almost 13 years ago by Thad Ward <[email protected]>
use quotes and encoded username for filepath

The entire assignment into filepath should be quoted, and we should use
the encoded username for...

938362da06b47d015c630ffcb82a3085f7ef5595 authored almost 13 years ago by Thad Ward <[email protected]>
output incompletes on stderr

output incomplete users on stderr so stdout may be used to get a list
of only complete users wit...

1b19d943c5a868d9908978d7ca4e7298330bf9df authored almost 13 years ago by Thad Ward <[email protected]>
add optional list file parameter

add an optional parameter: username list filename
If specified, read usernames from the list rat...

d3e65bbe0c9520199a06c0eeb6d47e059c7d9876 authored almost 13 years ago by Thad Ward <[email protected]>
add timestamps to the output

add timestamps to the output, to make it easier to see how long
it has been since the last messa...

e8b8081a41d942def8386974ecf2558bd9fbdd43 authored almost 13 years ago by Thad Ward <[email protected]>
didn't check for other instances of the log file

I didn't notice the removal of log files on success at the bottom of
the loop. Correct the path ...

3203ccd513c30452a6ebfc4a7190462e1a27af2f authored almost 13 years ago by Thad Ward <[email protected]>
add a couple more items to .gitignore

df571d18910b276e51231738084126be8b5bc683 authored almost 13 years ago by Thad Ward <[email protected]>
output dld-single.sh log files into a logs subdir

e03992bfa542d169fe69f815f7c3a0232a1dfbef authored almost 13 years ago by Thad Ward <[email protected]>
use trap for cleanup on script exit

There are a lot of exit statements. use trap on the "exit" signal to
perform cleanup on all scri...

5a3208129ac6a20d630a50cfa2320659de35d8c9 authored almost 13 years ago by Thad Ward <[email protected]>
add a country level to the temporary directory

add a country level to the ./tmpfs/ temporary directory, to prevent the
(probably rather small) ...

47a18ea84940a6d7fa001b2fffd2a32fe0c731ac authored almost 13 years ago by Thad Ward <[email protected]>
Merge remote branch 'chronomex/master'

8c26ec590722f559ad3a836224cd99c143461baa authored almost 13 years ago by Thad Ward <[email protected]>
add some stuff to .gitignore

a100f905b2354612c1af110dc839b1d4c01af3fd authored almost 13 years ago by Daniel Brooks <[email protected]>
Robustify tmpfs support. Add reinforcement to dld-client.sh.

b6a63644b55d01453930de7ddd2e0b6108a918e6 authored almost 13 years ago by Duncan Smith <[email protected]>
play a sound on underscore's computer for every completed download. silly, but mostly harmless

8161e6ea7219f139f69008ff5ce40f19fdf30d17 authored almost 13 years ago by Daniel Brooks <[email protected]>
Added tmpfs support.

Tmpfs must be mounted on /tmp/tmpfs, or at the very least a directory
has to exist there.

93c9da9f7675100be208ec4361b8d5bf27bde88b authored almost 13 years ago by Duncan Smith <[email protected]>
Simple and stupid tracker-unavailable backoff delay

abaaaac62b9aaeec345d66506053db7ac73cf4c2 authored almost 13 years ago by Duncan Smith <[email protected]>
fix infinite loop

profile download loop has a retry counter, and appears to intend to stop
after 5 tries, but ther...

193659714b1922c1c1f41151809004bba8346df1 authored almost 13 years ago by Thad Ward <[email protected]>
Add third tracker.

8f332cca8ca8b34c01e61699fe84226ebd9d5471 authored almost 13 years ago by Alard <[email protected]>
Balance load between two Heroku-instances.

73c72cbfc75de236eb96f63b35e08223a05fd5da authored almost 13 years ago by Alard <[email protected]>
Bugfix: dld-streamer would take 100% cpu while stopping

6532e1674d9987813c84a0c542a7215e8dbdbf2c authored almost 13 years ago by Duncan Smith <[email protected]>
dld-streamer is a way to run multiple downloaders at once.

d3f6351da08ac2af41f74eec85ddf54b1f248312 authored almost 13 years ago by Duncan Smith <[email protected]>
Correctly report media.warc.gz size to tracker.

478a9cb6486adcf28b0f0d6e333c3192a7a39642 authored almost 13 years ago by Alard <[email protected]>
Retry on network errors (Read error, connection reset etc.).

bc1f2d34113e62451031faa5f757a0d4b2dcb07a authored almost 13 years ago by Alard <[email protected]>
Use find instead of glob for.

c76482c7e20d62d7e474f705d5e2bd6195559674 authored almost 13 years ago by Alard <[email protected]>
Fix 502, 504 errors on the US site.

3244673fcd934579f08da64a3f8fabee16e2afcc authored almost 13 years ago by Alard <[email protected]>
Ignore the /myblog/taglist directory.

25437683b73bb68c3556a3fbfbf90911ed349574 authored almost 13 years ago by Alard <[email protected]>
Ignore the /book directory (the US manual).

16db8ffa8c46c6ad03e77f7fb7bb8b5f5cc77e6a authored almost 13 years ago by Alard <[email protected]>
Make it possible to report users without media.

1dc965464c64c8b80ef1bd62129d5199063e58fe authored almost 13 years ago by Alard <[email protected]>
Small changes to the output of dld-profile.sh.

d4355534237f29c58f86756b5b739b91d251f056 authored almost 13 years ago by Alard <[email protected]>
Merge branch 'master' of github.com:ArchiveTeam/splinder-grab

4cc8c39c7cc36d8768a2d82169038c64db6a18d5 authored almost 13 years ago by Alard <[email protected]>
Check with new directory structure.

4a82fe91d845eea847251062442a8b0199bfbbb4 authored almost 13 years ago by Alard <[email protected]>
Adopt check-dld to work with introduction of country codes.

45aff12650db3981263fd71d40e7bc8b09c8df9d authored almost 13 years ago by David Yip <[email protected]>
Add domain to output.

53ede334199fb0462ac4626fd9ef222bb14e206a authored almost 13 years ago by Alard <[email protected]>
This is more or less a good version.

57906095cd3f26a3360729ae2fe9ff32d9796d88 authored almost 13 years ago by Alard <[email protected]>
Only download owned blogs, but list the others. Download from us.splinder.com as well.

4788c5c7ebaff8914e7f8fd1eed5a651370c7d16 authored almost 13 years ago by Alard <[email protected]>
Oops. : instead of |.

acb0cd7ec0e1b2946f1f8bfc513f6067c464e31b authored almost 13 years ago by Alard <[email protected]>
Report total size of blog warcs.

2f350fcd85cbc37aa0b2fa5c55c7c4e25d172dd6 authored almost 13 years ago by Alard <[email protected]>
Update readme.

e932467c418ef793f0214b4ad6e461f0eb359120 authored almost 13 years ago by Alard <[email protected]>
Don't print answer to server.

df3ac73e3956454fb1585b89bc0cecd4a22f05fd authored almost 13 years ago by Alard <[email protected]>
First version of client scripts.

5a961a123eb57e36194213ff20a5f78a907e8eaf authored almost 13 years ago by Alard <[email protected]>
curl not needed by dld-profile.sh

fcee6de39878632c8108bcb9d6aef8995d680271 authored almost 13 years ago by Alard <[email protected]>
Deal with funny characters.

1ebf31996c07003d35483ea1eda5ef748da398ce authored almost 13 years ago by Alard <[email protected]>
Do not make a profile/ subdirectory. (Everything is in one script.)

0f5d205ee47d0fd3cb4d126bda2dc5e85f52ee28 authored almost 13 years ago by Alard <[email protected]>
Slightly changed the behavior of du-helper.sh.

b8154a5b093d5d4a4060e556dc579995ae5eaf4a authored almost 13 years ago by Alard <[email protected]>
Merge branch 'osx' of github.com:ArchiveTeam/splinder-grab

d34241bc48705c8db96396b9ca9dc744138da16c authored almost 13 years ago by Alard <[email protected]>
Record seen URLs.

782bdcbbc5a46fd481097447b5bac2c4c7facc62 authored almost 13 years ago by David Yip <[email protected]>
More status output.

593021407975a852ba72e0b7b4ff6e1f51abd828 authored almost 13 years ago by David Yip <[email protected]>
Don't try to guess when we're done; instead, just busy-wait.

GirlFriday::WorkQueue#wait_for_empty polls every 1/10th of a second (or
so) for whether the queu...

7a71e8e49d72e8a577afb526fc9bf3c025976795 authored almost 13 years ago by David Yip <[email protected]>
A tool to scrape usernames from www.splinder.com.

10e96e654a348952ee51eed96e00c60273e6bfeb authored almost 13 years ago by David Yip <[email protected]>
Script to paper over du differences on OS X.

The du-helper.sh script first checks whether --apparent-size is a
recognized du option. If it i...

ebed54a2083d49500954cf54ae54dfbe4588b6bd authored almost 13 years ago by David Yip <[email protected]>
Nicer du output. Do not redownload profiles.

82c7d0e72703b7e13452a0b7ae4f67bbb922769e authored almost 13 years ago by Alard <[email protected]>
Initial commit.

d4e4c3871a411aea0ac2babec879ce8f1a232c3d authored almost 13 years ago by Alard <[email protected]>