github.com/ArchiveTeam/ancestry-grab commits | Ecosyste.ms: OpenCollective

Revert "Revert "ancestry.lua: fix typo""

This reverts commit 6fe42f6e603a8b55c45a84eba1dc93aba729edd2.

07602ce50fd1212d7ed44ea14ce20a2df36ef1eb authored over 10 years ago by Christopher Foo <[email protected]>

Revert "Revert "pipeline.py: add support for 'myfamily'""

This reverts commit b7215800792ad2e447c04daece1782e56937e62c.

5bf45bbc99133bbbd070ba41d8a2a2bec51199a9 authored over 10 years ago by Christopher Foo <[email protected]>

Revert "pipeline.py: add support for 'myfamily'"

This reverts commit 50e893d8f474d10c987da1bfea39815ae21375da.

b7215800792ad2e447c04daece1782e56937e62c authored over 10 years ago by Christopher Foo <[email protected]>

Revert "ancestry.lua: add support for 'myfamily'"

This reverts commit 3abac53c5fdaaa52bc2c290f3b888c0f7b51da95.

6baa0f61a3908a18ecd4dedd40c121b86384b00e authored over 10 years ago by Christopher Foo <[email protected]>

Revert "ancestry.lua: fix typo"

This reverts commit eeb188cc4e503fc536213945516fee5901ea73c5.

6fe42f6e603a8b55c45a84eba1dc93aba729edd2 authored over 10 years ago by Christopher Foo <[email protected]>

Revert "pipeline.py: add support for 'myfamily'"

This reverts commit 09f930cf87c87a16b32699cae7f97538bce2fce8.

237dfd4e32ee52d46f781ad16c3929d51c470a0c authored over 10 years ago by Christopher Foo <[email protected]>

Revert "pipeline.py: add support for 'genealogysite'"

This reverts commit 4d51d382a73ff820b257b6aca6e6af95f2b6ef06.

0ba61950138abe349208a9151bfeb05876f05a7c authored over 10 years ago by Christopher Foo <[email protected]>

Revert "ancestry.lua: add support for 'genealogysite'"

This reverts commit c1cc0e338c9de2fbe46298666b3a13d4f9091f1a.

324a133f48a21fd70f2757329ea23c6420a3506e authored over 10 years ago by Christopher Foo <[email protected]>

ancestry.lua: add support for 'genealogysite'

c1cc0e338c9de2fbe46298666b3a13d4f9091f1a authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add support for 'genealogysite'

4d51d382a73ff820b257b6aca6e6af95f2b6ef06 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add support for 'myfamily'

09f930cf87c87a16b32699cae7f97538bce2fce8 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: fix typo

eeb188cc4e503fc536213945516fee5901ea73c5 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: add support for 'myfamily'

3abac53c5fdaaa52bc2c290f3b888c0f7b51da95 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add support for 'myfamily'

50e893d8f474d10c987da1bfea39815ae21375da authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: bump version

48ec7cbd229d1624b310a64ae4f88f02c1306704 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: exclude all links containing "///"

73716533ddbb52eccbf3e0d2c3935aa8df8fbfa7 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: bump version

a65677bfbb7ad35810d8a1b96b8c1327f99b708c authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: don't retry urls with status code

403

93ab64165b6716de81feaa5616ebfc1f08b3dc77 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: bump version

240391a080ff15de384693c2518b6057f6f48ab2 authored over 10 years ago by Arkiver2 <[email protected]>

Revert "ancestry.lua: abort after 5 tries with status code"

This reverts commit 3017d97d37ea4ea51d22ada52d394e1c69b7981a.

c3efa60db62f425b88340fa60240de3cb2dfbce4 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: bump version

4ba5b100a5f3adc351734d5d410f51be0761bdf9 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: abort after 5 tries with status code

403, since status code 403 might mean we are banned

3017d97d37ea4ea51d22ada52d394e1c69b7981a authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: exclude for different site

e4b41a7eb3deeb2bbccb64104668e410e6ac6e83 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua

0aca8488063285cac6bacd84fa085d58e436d60a authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: return false on chat.genealogy.com

cc4fe69d60daf08c73a6b91f8de214ff9c3acf36 authored over 10 years ago by Arkiver2 <[email protected]>

Revert "ancestry.lua"

This reverts commit 8ed88f5d406d49dd17fcd7a376f5cfbeacda61d7.

a1dca152df6f3cc0d60017fcaff1257411551ed0 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua

8ed88f5d406d49dd17fcd7a376f5cfbeacda61d7 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: download urls from cdn sites

e86c75934031712ccfa4a92dd6803fe64ea1fb57 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua

56a2d88259172c42e1c367958085db0ca9a915c0 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py

db521e7544e6f6edd6633867d9d80bd81b8c3c59 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add mundia.com to --domains

2a72282bcb4816d25667ee39fceeedce914a8e8b authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: download all images

d17743613ac13aeefaa6050aae009169d5f40a58 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: fix for mundiasurnames

a02f62978c06ef5f5b96b668fd2eb9102166621c authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: return true on links from mundia

c8532f95a259da027f9fb399c732634e88d08500 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua

5ce62fdb2cc7786f548a6a7492b239cd232cb07e authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: fix for downloading all directories

b79bd6b1f0136775dbe5d3437c6964323242e663 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py

5eb72fce862723b80e3ad974298d8850d82be53c authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: minor fix

9b24ad2b417f70b85b1d0385d4ff8fca335717eb authored over 10 years ago by Arkiver2 <[email protected]>

Revert "ancestry.lua: fix conflict between codes"

This reverts commit 4ed3ec97a580699ff9ac7234953a77ccf90c430a.

5311491a0c0580d8cfd92b027083e80c347edcb1 authored over 10 years ago by Arkiver2 <[email protected]>

Revert "Revert "ancestry.lua: download all directories""

This reverts commit b918cd734c3e6f27b6b975f15d944f43e90db4d4.

6ceee1607b75d90e9f90bc2f6ec154dbf9352920 authored over 10 years ago by Arkiver2 <[email protected]>

Revert "ancestry.lua: download all directories"

This reverts commit 6b3ec1388cdc42aa73bfdb37be0681052398f617.

b918cd734c3e6f27b6b975f15d944f43e90db4d4 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: download all directories

6b3ec1388cdc42aa73bfdb37be0681052398f617 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: delete unused loval variable

b5ebee3e57d03b0c05425db87de8c2c483fd07c6 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: fix in mundia pttern matching

d9c821d15a65c3b80751435be72cbde789ab3080 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: make sure both

genforum.genealogy.com and genforum.com will be downloaded

7f141ebc7e90c5b66519a3442d89fe064626b539 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: fix conflict between codes

4ed3ec97a580699ff9ac7234953a77ccf90c430a authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua

6fd7e1131bd2e8cdc861a23bf550ff059be7f6eb authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua

42fe7697677d2f096a4d2c93075f7ce4a4378c86 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: download "mundiasurnames" recursively

33057f582fc75891f3f135c9e323f9057eaafb72 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: make recursive crawl for mundia

872b4adf1cba7a6bcf8c9dc9943b0894963949ef authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: use cookie to pass shutdown mundia

eb25c3d658859cef88f1a14f52c0d93d9bde6657 authored over 10 years ago by Arkiver2 <[email protected]>

cookies.txt: only shutdownnotice cookie

471f2ee40314b954121146ddb2fa174b06d4a39d authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add cookies specific for item_type

37f9bcff67efca00c7ddc50621fb0f1d44044fd7 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: fix for familyorigins

386c2be38ea7f9483bc58ec843da13a075658e9b authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add domains specific for item_type

170f52ed68be6a267d7acd5c25c54dcef737df17 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: download both /users/ and

/genealogy/users/ urls for item_type "genealogy", "familytreemaker" and ""familyorigins"

b4f1270ba1770f3f378dae2ceab193176f86e73e authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua

a03cabdbab14074f62c7c1258f1161be1b491d98 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: also download "index.html" urls

ec91d8dbbd88dd8c8c539546922d2a683b70345a authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: download external urls one link deep

for item_types "genealogy", "familytreemaker" and "familyorigins"

87efe71150e7132853bdb9808d04a54a696749e2 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: track downloaded urls

33480b6a89b14dd8019c5c1b922903ee55bc2816 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: make sure /genealogy/ is downloaded

for item_type "genealogy"

13e99f7c59520b5801a96d67debb6839c93a377d authored over 10 years ago by Arkiver2 <[email protected]>

ancestry: retries for code 0 up to 100

af356c932ae40fb2a923c6005d897ec2a290bd13 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: retry status code 0 5 times

a859447ef3f69ec89fd1067d0ba5fe718b4f9794 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: remove cookies.txt from args

8c21f5fec61349fa07119163243ad443f771de27 authored over 10 years ago by Arkiver2 <[email protected]>

Update cookies.txt

9edf2cd38d23618654e5964aeb367d1f4d89e918 authored over 10 years ago by Arkiver2 <[email protected]>

Update pipeline.py

0a5fe919a99fade092cc53be6dbe319e6a966a9e authored over 10 years ago by Arkiver2 <[email protected]>

Update pipeline.py

c2554efe756f0774b0d739cddf0579669565d355 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add cookies.txt

9cb8d091d165b40802c0960c39f95a2630f7f1e1 authored over 10 years ago by Arkiver2 <[email protected]>

cookies.txt: add cookie to not show the shutdown

notice

d3b3c8b7811923b0e082908b98f92271056fffa6 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: show list of downloaded files

423233838c346ff94178213c785b37a2872add29 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry: one if too much

6c86dac38f839100c532291f2a5815476fcfd7d9 authored over 10 years ago by Arkiver2 <[email protected]>

Set executable bit for wget-lua

a0d4810280b0a8cd7c70b0b753bd1bda4071ed1b authored over 10 years ago by Christopher Foo <[email protected]>

ancestry.lua: make cgi regex work with all pages

6e793753d3b8460b4bf7d07a56f840ea607bc3ed authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: fix characters in pattern matching

51e9b8233f7d16e451c17b35d5a6245860e70b17 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: add support for cgi url for genforum

bcce1e5f49578a6a2eafc6ac4b2cad91ee163428 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: add TODO

1e6512e8a7a0f032a2c5a9adea2b3d039355ef67 authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: always download non-html files

Download files that are not html according to urlpos["link_expect_html"].

11227d88b7b4a7a15970d4bd151fae726a23f5ff authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add genforum.com to --domains

c5c82494f674e979a44298c5b490cf14dfe1b857 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add support for genforum.com

469d2107243a2ec3cc4e47c592b81ac09f858e8c authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: add support for genforum.com

e520558d7b379ce6dcb314efd4e368debb22113e authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add ancestry warrior panel text

050a60d1dc92ca1857a72799cda5816345aa0dfd authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: only check url when mundiasurnames

4bb448b8a460a7f231e78e9baf8b76ae4665247a authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: add support for genforum

20652e45148ca6458c7f88c2e7d63e132f8ea016 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add support for genforum

b964233808dc8fa477fe7c3c1b150b491c29fca9 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: add recursive depending on item_type

791650529713dd1351110990b34a551abb8492db authored over 10 years ago by Arkiver2 <[email protected]>

ancestry.lua: merge mundia and genealogy

7fcb59e6316c5d93e139fb67894e61d995b74ed4 authored over 10 years ago by Arkiver2 <[email protected]>

pipeline.py: merge mundia and genealogy pipelines

ce9de46dfc28358b1fa79a6e7d7aa97068853b1d authored over 10 years ago by Arkiver2 <[email protected]>

first version

af5683ed4e5825f11829563f92e91a127b9bb020 authored over 10 years ago by Arkiver2 <[email protected]>