github.com/ArchiveTeam/grab-site commits | Ecosyste.ms: OpenCollective

README: use spaces for indentation to avoid tab-completion on Tab paste

447166065631f426e4f92df4a0aa34fa1cd07924 authored about 6 years ago by Ivan Kozik <[email protected]>

README: try to get people to stop installing grab-site as root and explain the pip options

e6502784212eced0b6f03cff5d7200ff69c9e26d authored about 6 years ago by Ivan Kozik <[email protected]>

README: add install steps for NixOS 18.09

d22ea9ad781f93dee166f6eda81fc3b2f00fd874 authored about 6 years ago by Ivan Kozik <[email protected]>

Use ludios/wpull 3.0.7 to fix https://github.com/ludios/grab-site/issues/138

700ea18693bc3efe2a9246950f7265d399c77ebb authored about 6 years ago by Ivan Kozik <[email protected]>

Use ludios/wpull 3.0.6

50a3db6e59a35702fabfc9bcba610ca7a0fd972e authored about 6 years ago by Ivan Kozik <[email protected]>

Alignment

0c18c9c0ea2944e2147c0eb24f7406add81e8855 authored about 6 years ago by Ivan Kozik <[email protected]>

README: mention pkg-config

4f898d2b06c7559c9e8ea0acc1d087632ea55261 authored about 6 years ago by Ivan Kozik <[email protected]>

Fix --which-wpull-args-partial

e541ad5c26f6a137244a50267513f8a12bd80890 authored about 6 years ago by Ivan Kozik <[email protected]>

.travis.yml: try to make the travis build work

795a27121c8605b27d33e8e6633bda7ae5556368 authored about 6 years ago by Ivan Kozik <[email protected]>

README: move pkg-config to the end because it's for grab-site's dependencies and not for Python

9022a57f6ff72681c581b55f640412c5c362aac6 authored about 6 years ago by Ivan Kozik <[email protected]>

.travis.yml: make Python 3.7.0 work, according to https://github.com/travis-ci/travis-ci/issues/9069#issuecomment-425720905

886426c566965c2e9ce9f96ddb10e8cc2f89001f authored about 6 years ago by Ivan Kozik <[email protected]>

README: abbreviate webrecorder.io instructions

db251482cfde86b6f5fe6de413c36031ea67aaea authored about 6 years ago by Ivan Kozik <[email protected]>

README: http:// -> https:// links

73587696f2f943de285a43e39bd0f03a712e1081 authored about 6 years ago by Ivan Kozik <[email protected]>

README: update webrecorder.io instructions

ab7e20eb4ddcd11366781f55783a595b565f1f29 authored over 6 years ago by Ivan Kozik <[email protected]>

README: write more about how to archive Twitter users

20156b1f76081ec3b541d7fcbe93c18eaa1ecd04 authored over 6 years ago by Ivan Kozik <[email protected]>

Bump version

cee2eb977b16fcf7ef38d29496570efe0e693439 authored over 6 years ago by Ivan Kozik <[email protected]>

dashboard: add -apple-system for macOS Firefox

https://bugzilla.mozilla.org/show_bug.cgi?id=1226042

c05987f93a58d3af468b122964ea3a1060ce42d3 authored over 6 years ago by Ivan Kozik <[email protected]>

README: fix macOS install instructions

ea94a3d9257fcbf246efb6a7487e6cb032555bd3 authored over 6 years ago by Ivan Kozik <[email protected]>

README: fix link

d737e09b3b3ad18d74e8959cf043454797263571 authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: mention that re also returns useful errors

c4bbd26779688f20b9fbc64eb28e3357316e6682 authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: validate regexps with `re` before passing to `re2`

9e8ad88703f0c065f0a431fbef06b893a212f475 authored over 6 years ago by Ivan Kozik <[email protected]>

Use facebook/pyre2 because ludios/pyre2 has a buggy PR applied:

https://github.com/facebook/pyre2/pull/11#issuecomment-428256331

e6b1709ef32b762092c806fc88c93a702e04bdab authored over 6 years ago by Ivan Kozik <[email protected]>

README: install instruction fixes

5b67d90cfb78855d0ef62e57c9d81cb5901e6049 authored over 6 years ago by Ivan Kozik <[email protected]>

Use ludios/wpull 3.0.5

e790ff8329201584701d5230ef1154c672281416 authored over 6 years ago by Ivan Kozik <[email protected]>

Use wpull 3.0.4

caf0497b930bde62969608beed060d4ecbe31488 authored over 6 years ago by Ivan Kozik <[email protected]>

Alignment

5bad64563649c52ac2440c27579a05644b9dc2ce authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: update igoff state less frequently; use f" strings

8dad425bc715569759f08b9571cf94d3cc79b8b6 authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: when igon is enabled, print which ignore was responsible

ce80bcc8c10e46a9342f61f8e1c0592c8b58929e authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: re.escape() the netloc during {any_start_netloc} replacement

1aeccf24c5256afbfeaf1772e25663511d0fc63d authored over 6 years ago by Ivan Kozik <[email protected]>

reddit igset: ignore new simple.reddit.com

4d127249af7f16b465ccf7dd578dde99d83451b0 authored over 6 years ago by Ivan Kozik <[email protected]>

README: Ubuntu 14.04 and Debian 8 (jessie) are no longer supported

53370c78db9c7028e65e9ed6ec17ea0ee75bc5dc authored over 6 years ago by Ivan Kozik <[email protected]>

README: fix TOC link

cc839e346886f161395c7f7a6f40e5cbc4d8fd55 authored over 6 years ago by Ivan Kozik <[email protected]>

README: install libssl-dev instead of libssl1.0-dev now that we no longer use Python 3.4

8e6649d144efeca37a9f571be0d77f38df657b80 authored over 6 years ago by Ivan Kozik <[email protected]>

mediawiki igset: use {any_start_netloc} and add [\?&]lqt_method=

c38e931485ac245d6ae0186f817153ef2a976de0 authored over 6 years ago by Ivan Kozik <[email protected]>

gs-dump-urls: add support for old wpull 1.x wpull.db files

20dfd9966bebf5acffbd2e9cc789d42875250226 authored over 6 years ago by Ivan Kozik <[email protected]>

global igset: ignore another endless mp3 stream

https://aechai9hib.cdn.dvmr.fr/franceinfo-midfi.mp3

cbf202fb44e0438ad33c2cb207f2a492e8a02d42 authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: implement support for {any_start_netloc} (previously {primary_netloc})

3316c048b7540d94c409d8b036c9d0af941f0c46 authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: reimplement ignores

8e7ab58c9d4617877ef12ccc8a2c691f52840624 authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: use re2 for icecast checks

bec7f36aa1461a0802b83e656807737a25954338 authored over 6 years ago by Ivan Kozik <[email protected]>

README: update macOS instructions for lxml and pyre2 (untested)

Brew has a Python 3.7.0 right now, so there is no need to compile a Python with pyenv.

2a1f0b9548ca96734d94bdea0ff5e42fd5c18abc authored over 6 years ago by Ivan Kozik <[email protected]>

Thank falconkirtaran

7589d57e75054c89ad1b1b787a3175f848a40d0d authored over 6 years ago by Ivan Kozik <[email protected]>

Use ludios/wpull 3.0.3

737314070951ef44792027f8592bfc8524721613 authored over 6 years ago by Ivan Kozik <[email protected]>

gs-dump-urls: fix for wpull 2.0's new database schema

6ddbf4e3d758dfd33ffcf9a0c6e39938a93efbc3 authored over 6 years ago by Ivan Kozik <[email protected]>

server: move method around

212af01ede22b63804908854aef04147d963ff62 authored over 6 years ago by Ivan Kozik <[email protected]>

' -> "

205d4a5378e465cb07d90b3da6c5a6acbd33c15e authored over 6 years ago by Ivan Kozik <[email protected]>

dupes: code cleanup

3e26d388666ea6a03d15234d81d0024258528adb authored over 6 years ago by Ivan Kozik <[email protected]>

dashboard_client: avoid a potential bug with task_done() not being called if send_object raises an exception

ca6e77e3789ac7f24157e7ea8c8f00437a0f408b authored over 6 years ago by Ivan Kozik <[email protected]>

gs-server: alignment; use f"" strings; use camel_case for non-autobahn methods

f24959ce4a8490998147d145cd301d224dfd4d94 authored over 6 years ago by Ivan Kozik <[email protected]>

' -> "

4403f9fdbb9839f448206617b5f72df04a8bc051 authored over 6 years ago by Ivan Kozik <[email protected]>

Alignment

90307d0e975540357f600979e3eea09d9df11ae1 authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: reimplement the dashboard websocket client

21a1e44196d4f6fde7ccdf396708104fed735c01 authored over 6 years ago by Ivan Kozik <[email protected]>

Alignment

6c69fee73fb579dffdb96e490bc683cd5c2b0154 authored over 6 years ago by Ivan Kozik <[email protected]>

Alignment

d87caabead109d10886d88af47f2258dd3789068 authored over 6 years ago by Ivan Kozik <[email protected]>

Alignment

1f20b5577b911118b6fd8cb92165efae564501b6 authored over 6 years ago by Ivan Kozik <[email protected]>

Bump Firefox UA

59c803bcd8fff2f0e9e929cd6fecf5fb668f60e2 authored over 6 years ago by Ivan Kozik <[email protected]>

Alignment

910734afd0f60a1ed2cf38fa6d646151570a359a authored over 6 years ago by Ivan Kozik <[email protected]>

Alignment

72db6f5582e7ccfbc1b011405f47aa36975ebc77 authored over 6 years ago by Ivan Kozik <[email protected]>

Code formatting

e0334923efe692cf097c3bd74a0271c5517eb864 authored over 6 years ago by Ivan Kozik <[email protected]>

Alignment

c34fa28b379ecfcb53a8519e0f85ce4bb3702999 authored over 6 years ago by Ivan Kozik <[email protected]>

Remove the psutil support that hopefully no one was using

grab-site processes can be kill -STOP / kill -CONT'ed externally

4b199bf423c252e1b29c5b88faf15454f48391b3 authored over 6 years ago by Ivan Kozik <[email protected]>

Code formatting

1123896f45e55596937558e16236452220a8de4a authored over 6 years ago by Ivan Kozik <[email protected]>

Remove commented `libgrabsite import wpull_hooks as _`; import-time errors are viewable in DIR/wpull.log

7b1cabed9ab95a2d5e22cc06ff49a28fe6ad847c authored over 6 years ago by Ivan Kozik <[email protected]>

Remove unsupported --custom-hooks

The plan is to document some replacement later that might involve
modifying and pointing to anot...

b4f67c80e1f2dd859cb3064ab726f61ba36bc719 authored over 6 years ago by Ivan Kozik <[email protected]>

Refactor wpull_hooks functions to methods on GrabSitePlugin

TODO: add back support for ignores
TODO: add back websocket client

7b40c6b946838828f98485be1deb110a76a0cca2 authored over 6 years ago by Ivan Kozik <[email protected]>

README: move security note to the end

08acad3e1c79ddb62f749e0422c291baa47a0db7 authored over 6 years ago by Ivan Kozik <[email protected]>

rm extra_docs/custom_hooks_sample.py

4e0952126e3e939ef9619a7fa8c6b5de984c5597 authored over 6 years ago by Ivan Kozik <[email protected]>

README: wrap lines

fca0eaf432e83df55e95eb8e483eca7416b5bea3 authored over 6 years ago by Ivan Kozik <[email protected]>

README: wrap lines

ea4e4eff748e7374be56c3c0bed938455cc17803 authored over 6 years ago by Ivan Kozik <[email protected]>

README: wrap lines

d3da0899ad229d3859cfe53228fad0084ba77e50 authored over 6 years ago by Ivan Kozik <[email protected]>

README: wrap lines

c186df0617007f6aaf3104aec7c2cbdb8a204639 authored over 6 years ago by Ivan Kozik <[email protected]>

README: remove phantomjs mention because I removed support from ludios/wpull

65b4f5caca61698bebd2d847b63ce554ba49527f authored over 6 years ago by Ivan Kozik <[email protected]>

README: wrap lines

e8a2163dd357b96ffdd260db009acbc81f693075 authored over 6 years ago by Ivan Kozik <[email protected]>

README: wrap lines

4473c7040ef8443301dc55ad47bce9e8a7a8ecf8 authored over 6 years ago by Ivan Kozik <[email protected]>

README: wrap lines

469dfdd0b979a579d654e64a1075e80e1f84f728 authored over 6 years ago by Ivan Kozik <[email protected]>

README: remove BrowserStack mention

e45f6f5b97f6104693c9b35bd6f98d92c9a200c0 authored over 6 years ago by Ivan Kozik <[email protected]>

README: thank JAA

8bf22e410f93ff6163d5cecf41d3c97c5103c492 authored over 6 years ago by Ivan Kozik <[email protected]>

README: don't tell users to file issues on chfoo/wpull

fc54bed43a950be1dce92fd308e6dcc04a893baa authored over 6 years ago by Ivan Kozik <[email protected]>

README: fix list of options

ca10585c45f11e551ae51d688b7657b390d0a2eb authored over 6 years ago by Ivan Kozik <[email protected]>

Add grab-site --debug flag for turning on wpull --debug

8bed843e249ff0dff1bba12c1c1099b2f51c6e55 authored over 6 years ago by Ivan Kozik <[email protected]>

Start porting hooks to wpull 2.0's new plugin system

0d42842fd024f52dd1487b468cd7b9f8abbc0c8f authored over 6 years ago by Ivan Kozik <[email protected]>

Bump version

9c12ae390fa516e543d3e53cb700980ae19a3046 authored over 6 years ago by Ivan Kozik <[email protected]>

README: pip3 -> pip

3f14886435272d1c1a56198339622dfc94692182 authored over 6 years ago by Ivan Kozik <[email protected]>

Require ludios/wpull 3.0.2 and remove the install_requires that are in wpull

e6d81e81c5397dfe43a35f661661c19370596edc authored over 6 years ago by Ivan Kozik <[email protected]>

Bump version

01d0fc28b8bf331192699d2440e8197472e81854 authored over 6 years ago by Ivan Kozik <[email protected]>

Remove unused html5lib dependency

82c94683d45f2a924bbf640df073a7212530b11c authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: remove method confusion

f27c61bdfea6c8562f8916428c5f4ae99680f323 authored over 6 years ago by Ivan Kozik <[email protected]>

Bump version

e1e60c6072a6e8d36e2ec9afd1bc233d65c78070 authored over 6 years ago by Ivan Kozik <[email protected]>

.travis.yml: test only on Python 3.7

8ec6659cc8a9d215c0b222905fd0056360d58195 authored over 6 years ago by Ivan Kozik <[email protected]>

Use wpull 1.3.1

4d8218db7fc82202ab89926f8d87e83c8e52f351 authored over 6 years ago by Ivan Kozik <[email protected]>

Upgrade to and require Python 3.7.0

b7dfb14dd866e5cb65a0a94f64d24331de10185d authored over 6 years ago by Ivan Kozik <[email protected]>

Remove trollius

7ec5accb2203f9b767e7674675a9021d1e41f378 authored over 6 years ago by Ivan Kozik <[email protected]>

Install ludios/pyre2, to be used soon for processing ignores

b32da83a0fdf310479ac95812d0d563f8f894d1d authored over 6 years ago by Ivan Kozik <[email protected]>

wpull_hooks: "Picked up the changes to" -> "Imported"

f7ed0260100c897d60fcbdf29919218eaeed9a04 authored over 6 years ago by Ivan Kozik <[email protected]>

README: link to ludios/wpull

837551c2016c75032193ec47bb3887bdaefb9a88 authored over 6 years ago by Ivan Kozik <[email protected]>

Use ludios/wpull 1.2.5

b6127f20775c9d460c905bc2394df9c5fc9841af authored over 6 years ago by Ivan Kozik <[email protected]>

README: fix pip3 install step

bc512c696da9467cd175d07a4038d91dfea52a6c authored over 6 years ago by Ivan Kozik <[email protected]>

README: Python 3.4.8 -> 3.4.9

7c14a909ef6d402eaae0045947e1b38cda16c368 authored over 6 years ago by Ivan Kozik <[email protected]>

README: fix pip3 install step for new setup.py

ba960e0ea8e10891314650cd815ce8f147ace29d authored over 6 years ago by Ivan Kozik <[email protected]>

Use ludios/wpull for html5-parser support

eaaf0ec06ed1021e8b14c1ab26593d1111d42ce9 authored over 6 years ago by Ivan Kozik <[email protected]>

Bump version

29b9825dc5f49c25f01d93746cfb0638c724c22a authored over 6 years ago by Ivan Kozik <[email protected]>