github.com/webrecorder/browsertrix-crawler issues

Make numRetries configurable

ikreymer opened this pull request 4 days ago

hang protection: wrap remaining evaluate() calls to avoid rare hangs

ikreymer opened this pull request 9 days ago

pages redis: include 'depth', 'seed' and 'favIconUrl' in page data added to redis

ikreymer opened this pull request 9 days ago

if uploading wacz files, compute waczfile name on load to be able to …

ikreymer opened this pull request 11 days ago

Set WACZ URL ahead of time so it can be known when pages are written.

ikreymer opened this issue 11 days ago

Add a create WACZ command after that fact.

ikreymer opened this issue 11 days ago

Apply exclusions to redirects

ikreymer opened this pull request 13 days ago

Ensure exclusions apply to pages that redirect

ikreymer opened this issue 13 days ago

Retry support and additional fixes

ikreymer opened this pull request 22 days ago

set failed URL retry to 5 by default

ikreymer opened this pull request 22 days ago

Improve add exclusion test

stavares843 opened this pull request 23 days ago

clear out core dumps to avoid using up volume space:

ikreymer opened this pull request 23 days ago

Retry Failed Pages + Ignore Hashtags in Redirect Check

ikreymer opened this pull request 23 days ago

Consider disabling core dumps

ikreymer opened this issue 24 days ago

separate fetch api for autofetch bbehavior + additional improvements on partial responses:

ikreymer opened this pull request about 1 month ago

Autofetch behavior results in empty 200 responses

karenhanson opened this issue about 2 months ago

Possible encoding issue with some  

karenhanson opened this issue about 2 months ago

Consistent aspect ratio

SuaYoo opened this issue about 2 months ago

Youtube video format

benoit74 opened this issue 2 months ago

Manually create a profile.tar.gz

djhmateer opened this issue 2 months ago

Add WARC resource containing DOM tree after load

magbb opened this pull request 2 months ago

Autoclick Support

ikreymer opened this pull request 2 months ago

[Feature] Support for clicking on links / other elements.

ikreymer opened this issue 2 months ago

package: pin @novnc/novnc to 1.4.0 to prevent accidental upgrades

ikreymer opened this pull request 3 months ago

Headful interactive profile creation broken after v1.4.0 update

halvir opened this issue 3 months ago

Dependency Update

ikreymer opened this pull request 3 months ago

support removing range from query (via wabac.js 2.20.6):

ikreymer opened this pull request 3 months ago

some links on page not crawled

robert-1043 opened this issue 3 months ago

WARC Record Write Failed

benoit74 opened this issue 3 months ago

Ensure partial responses are not written

ikreymer opened this pull request 3 months ago

add disable-lazy-loading flag, should fix #699

ikreymer opened this pull request 3 months ago

Crawler is not failing when seed page returns an HTTP error code

benoit74 opened this issue 3 months ago

Dependency Update

ikreymer opened this pull request 3 months ago

Support loading custom behaviors from git repo

tw4l opened this pull request 3 months ago

WARC-Protocol + WARC-Cipher-Suite headers

ikreymer opened this pull request 3 months ago

fix indexing of cookie header:

ikreymer opened this pull request 3 months ago

fix cookie not being passed to replay regression: for now, add x-waba…

ikreymer opened this pull request 3 months ago

Support loading behaviors from a Git repo with branch + path

ikreymer opened this issue 3 months ago

Stop crawler when we have been hit by a WAF protection

benoit74 opened this issue 3 months ago

tests: use old.webrecorder.net for testing

ikreymer opened this pull request 3 months ago

various edge-case loading optimizations:

ikreymer opened this pull request 3 months ago

deps: update to latest wabac

ikreymer opened this pull request 4 months ago

Support loading custom behaviors from URLs and/or filepaths

tw4l opened this pull request 4 months ago

Browser disconnected (crashed?)

rgaudin opened this issue 4 months ago

Issue creating profile for intranet

MRLeflei opened this issue 4 months ago

dep: update to wabac.js 2.20

ikreymer opened this pull request 4 months ago

Archiving particular site kills networking on host machine

vitorio opened this issue 4 months ago

Automatically crawl `<form>` URLs when `method` is `get`

benoit74 opened this issue 4 months ago

link extraction promise cleanup:

ikreymer opened this pull request 4 months ago

bump puppeteer core to 23.5.1

ikreymer opened this pull request 4 months ago

Youtube video is not crawled when `loading="lazy"`

benoit74 opened this issue 4 months ago

Tests: disable blockrules test in CI

ikreymer opened this pull request 4 months ago

fix typo in QA exclude check, which resulted in all URLs being excluded

ikreymer opened this pull request 4 months ago

BUG: Create Browser Profile --headless can't render at viewport

gitreich opened this issue 4 months ago

Add documentation for crawl collections

tw4l opened this pull request 4 months ago

ensure extraHops also apply to maxDepth

ikreymer opened this pull request 4 months ago

Crawler never fetches `<img>` source image in `src` when `srcset` is activated

benoit74 opened this issue 4 months ago

[Bug]: Crawl Configuration Inconsistency: Max Depth and Include Any Linked Page

mona-ul opened this issue 4 months ago

Additional exception safety

ikreymer opened this pull request 4 months ago

Include depth in pages JSONL files

tw4l opened this pull request 5 months ago

Include depth in pages jsonl files

tw4l opened this issue 5 months ago

support custom css selectors for extracting links