github.com/mwmbl/crawler-extension commits | Ecosyste.ms: OpenCollective

Merge pull request #43 from mwmbl/bug-fixes-for-curation

Bug fixes for curation

895b69c970c8b22cfaa3cc37ee03e60e985c618e authored about 1 year ago

Compile worker separately so we can inline all modules

646cda16bb94b447f3df75c81855095cc50246fe authored about 1 year ago

WIP: Refactor to share code

33653960686216ad68ab92cfcc6a54b8c17bb365 authored about 1 year ago

Send a message to allow ordering the results after loading

664eff7061c2c3ad5a00657ec46aba2164c4f298 authored about 1 year ago

Merge pull request #42 from mwmbl/query-extra-search-engine

Query extra search engine

4ceb8b1133da5d1222340520df066a7a58a72cba authored about 1 year ago

Update version

0f3aa4de1761e02cda09bc12c28c7e0017d8a35e authored about 1 year ago

Use mwmbl.org instead of localhost

a2c5159637a1b10df2917d4897972cce02275da2 authored about 1 year ago

Only query Google if the user has allowed it

cf3b01d04db05b6601e6206f564e57dd0a389a01 authored about 1 year ago

Add toggle to enable/disable crawling

02ab903f9437c39155e1a16ee5acf54e85b76174 authored about 1 year ago

Enhance results by querying another search engine

8cbf42fd8656ca278c13129e4ef466ef61bd8176 authored about 1 year ago

WIP get results from Google

88e1efa1529d17819fd4ff3c4aed465400701e87 authored over 1 year ago

Run crawling in a web worker

82202e6e255e0df14337d606fd0e7cda68756e2c authored over 1 year ago

Bump version

605bb1a25eb3388c9191eeb0d95376e49e4fb50b authored almost 2 years ago

Allow more extra links; remove logging

701efff450ca162bd7a4912f27fa7f9d53b7462a authored almost 2 years ago

Post links that don't just occur in good paragraphs

e03258f55ee78716249ef63076ddb565dacd3393 authored about 2 years ago

Bump version

4c6d21dae572e814b59c024e67676928174efe7f authored about 2 years ago

Merge pull request #37 from mwmbl/remove-logging

Remove some logging

a1cd8d3ecb842e5cdcaeaed036ee01e352495753 authored about 2 years ago

Remove some logging

822e4e67c81fc449a19685cca06132f6fe6bde25 authored about 2 years ago

Merge pull request #35 from omasanori/accept-language

Specify the Accept-Language field to fetch requests

3cddbe7d0e1b34cfd3bbe780a458114e21a8b3f0 authored about 2 years ago

Specify the Accept-Language field to fetch requests

After multilingual indexing is implemented, the preference should be
configurable. For now, Engl...

6e9f18290dd0445dff79386b97153d420afc5e27 authored about 2 years ago

Merge pull request #30 from AdJaGu/master

Internationalization

4ec1ab9a40d3cdbdfcc80e19e4a1b1b5ff562365 authored over 2 years ago

Update translation key names

6a9e44ebbb134e50224b364cbd26047089abc63f authored over 2 years ago

add: Internationalization

81a62eaa7b3ba1b1630efd53043effab356d409d authored over 2 years ago

Merge pull request #28 from AdJaGu/master

Update README.md

dfa0bdc497f45c43bec5743397275bc9538b6b8a authored over 2 years ago

Merge branch 'mwmbl:master' into master

42cc5392ead20f6176b7cbdb25ed321b513cbdb5 authored over 2 years ago

Merge pull request #29 from mwmbl/send-all-links

Send all links

db7021f152ef7fd6093ffadcfb6b41bc355450ba authored over 2 years ago

Send all links

ffc84a2e93c7389f92cb5528b32949b41c6c34af authored over 2 years ago

Update README.md

Update "How to Build" section.

6505107199e11bb751e5101760c9d41425ef3281 authored over 2 years ago

Merge branch 'master' of github.com:mwmbl/crawler-extension

22a5008db2226e5b50b28ec2d1571ad333e410a2 authored over 2 years ago

Catch some uncaught exceptions

34472c7ddae2d9b0ebc2aee554b92fbf9e992ee9 authored over 2 years ago

Update README.md

7071e5b555feb73b153e3492ea0e6ecf0b0c4fed authored over 2 years ago

Update README.md

a4ed40a1eeb05842d3aeda200bc7b2871a46ea3f authored over 2 years ago

Update README.md

c83c0265728064884a3d55b4bbaa8b87047e8342 authored over 2 years ago

Update README.md

966e0eff7cbb8575f178f0ca78433721ef6a6242 authored over 2 years ago

Update README.md

7f99744f0c63437579ea90e3bc7f4322b9481284 authored over 2 years ago

Merge pull request #23 from mwmbl/detect-404s

Detect status correctly

2bc5d0fdd522cbf2067e52582b1ac270fe7944f1 authored over 2 years ago

Detect status correctly

a6f8418158842b03f2d83c5fe154f4d1b29c3ba7 authored over 2 years ago

Merge pull request #22 from mwmbl/add-timeout

Add timeout

a736412f2972ad645fb41852f7b6140389701d27 authored over 2 years ago

Add timeout

e1dca3de174191dcf6d6cad07ebd7d3c7118911f authored over 2 years ago

Merge pull request #21 from mwmbl/bump-version-to-0.4

Bump version to 0.4

4a2f3536c4b92a96bb0bfab5d36e4ec6e3f7c85a authored over 2 years ago

Bump version to 0.4

e3765b6cc965e04d51e23fe33d4e882dbb5e7bd7 authored over 2 years ago

Merge pull request #20 from mwmbl/prevent-loading-big-pages

Prevent loading big pages

351e58efed187e4e1dac9ea0f62c7d89797f76f1 authored over 2 years ago

Improve presentation of results

275c614086655417a0eb6a1d5f14f54a13d97a99 authored over 2 years ago

Load existing batch from storage

7c21e316924fb2c30c9e283c1dae03e5d3135db1 authored over 2 years ago

Store batch for GUI display

e977b1f77d2cb9e05b4c976341da7272570c3351 authored over 2 years ago

Update GUI

890d6d59b81bed26c356d5859bcbd37ad58cd279 authored over 2 years ago

Return a result whether we succeeded or not

1bcbedcd5eeda33f118f3df6a74a4d2ae7dd7fed authored over 2 years ago

Crawl domains chosen by the API

b58b235122caa1c1d8c913e898e8eddb1904f863 authored over 2 years ago

Bump version

a657418a89db1b206323c6255f273495b7df686f authored almost 3 years ago

Update icons

d783d5141d1d0cb75f1701f7c275b9298b386949 authored almost 3 years ago

Show URLs as links

56989d881f2905d2f66459865691483060e7ee2f authored almost 3 years ago

Use a while loop instead of setInterval

48f4331bc6507626400f8b4c7aa4fd25f5ff7a26 authored almost 3 years ago

Log the link and not its source

234f62bdc4e91bdec92d3b5b590a66fcc42ce76c authored almost 3 years ago

Truncate stream for pages > 1Mb

12f9fc912048567e9f8afaf62ca305be0f3b496e authored almost 3 years ago

feat: added popup to log crawler urls

8662197751eaa6842dce603875ef40b11d45a3db authored about 3 years ago

Bump version

250ce089426e5927e2d9c4e4e29030002e39ec2b authored about 3 years ago

Merge pull request #16 from mwmbl/dont-send-cookies

Don't send cookies

92d63fb0e2e39c61d9e8caacbff110c5d31820cb authored about 3 years ago

Don't send cookies

b2d59b30e1706896a1b0b195456dd3d7a2402400 authored about 3 years ago

Bump version

7b05191309533c62c7a32f487bb095f91788cadf authored about 3 years ago

Merge pull request #14 from mwmbl/store-visited-urls

Remember visited links so we don't visit them multiple times

9c8828743397b68664293a6007d7d6faf72ca173 authored about 3 years ago

Merge pull request #15 from mwmbl/check-for-offline

Don't try and crawl if we're not online

c76474363f0ca876b4ae567c4ddab8d8b34a1944 authored about 3 years ago

Don't try and crawl if we're not online

4547d79074a0a14520dc518884453d20037e3ceb authored about 3 years ago

Remember visited links so we don't visit them multiple times

05c718202a98848690041ec441c8de9b67be05a4 authored about 3 years ago

Bump version

8098d12624c2c39dfec3e42e62a99f577787784b authored about 3 years ago

Fix a bug with deleting links

05f93391660bd72783957f72425384807c2e8c66 authored about 3 years ago

Turn off minification to make deployment easier

ff0a8f265a365e21b068e4e787206a00b6e92ece authored about 3 years ago

Bump version

75cfba07a29fb42e52c8b1561f6acba8dedbbfeb authored about 3 years ago

Merge pull request #13 from mwmbl/detect-loops

Check the number of unique domains to prevent falling into loops

9ea58b24675d8e2414a5d262f02da85847b4d853 authored about 3 years ago

Check the number of unique domains to prevent falling into loops

3c03302540bb696a885e8782ab7bed4b1680944c authored about 3 years ago

Merge pull request #12 from mwmbl/crawl-more-root-domains

Crawl more root domains

91d0ec9c5f89347d068cc7968dc47fbb665c34bd authored about 3 years ago

Crawl more root domains

a083e5eb8fe9fc7e399bed818a8c9879d6d148d5 authored about 3 years ago

Fix some bugs: invalid URLs and running out of links (fix it properly)

47246019214584a404052bf5371facb41d008d93 authored about 3 years ago

Bump version

4b84c961aafd576101e1a2bc7212c2e232b55278 authored about 3 years ago

Merge pull request #11 from mwmbl/adapt-for-firefox

Adapt for firefox

3152132f36da8d4a4aa963e26720cd327c9c66ce authored about 3 years ago

Remove unused things from config

e02f666e9c1a40fb8233623621c995c64d200ff8 authored about 3 years ago

Remove unused stuff

10307ea4b172e97c87fd18018daff959b9ce8d86 authored about 3 years ago

Add instructions on how to build using the README

f48c07d6f0cfc913b2f95666cd18801e906d821e authored about 3 years ago

Merge pull request #10 from mwmbl/handle-exceptions

Handle exceptions

5b38159343e7b289f72ea7d74154f79192b52093 authored about 3 years ago

Copy the manifest to the dist folder so the dist folder now contains the built extension

83634cefcbff1d55d9ec936036ace113989de145 authored about 3 years ago

Update manifest

39b73c7ccc0098bc0c42310b1cc3a556cb3efe7d authored about 3 years ago

Fix bad URL detection, fix another bug in robots parser, delete URLs in the right place

77110e6f4582c2d074d6f5efe6a272fc7b66e9c7 authored about 3 years ago

If the user agent isn't specified, assume it is allowed

99d38106775beb4e117e8a4472cdb6eae9274de0 authored about 3 years ago

Merge pull request #8 from mwmbl/implement-crawl

Implement crawl

314e46c6a9f286dd6fbdd4860bd38aecdfed225a authored about 3 years ago

Avoid race conditions by storing everything in member variables (to prevent saving the same batch multiple times)

d926fd74bb5282729d6f9fc1e4a25476c64a87cd authored about 3 years ago

Links are either from curated domains or (not from curated domains but to curated domains)

742711fa57a20b4495ff8b9f93cd4b2b0481eefa authored about 3 years ago

Only add links to external sites

fc95b1e10ca14a9bcbb01c50d3c51a40d295396b authored about 3 years ago

Send batches to the server

14427e83fe924d4aaa6998be679926295e0a7dfe authored about 3 years ago

Store results of crawling in local storage and batch up for sending to the server

de7a86cb77392f5bc3e6569dfc7d9fdf4627fa16 authored about 3 years ago

Generalise storage functions

ad87c515639d239ab2b6fb1966805d94e8163b06 authored about 3 years ago

Prevent 'undefined' being added as a link

6fa1edbdd746b65e36a340f34521d7b092dee891 authored about 3 years ago

Update the set of links

8535974f5524a34887b2190cb3b3bccd4abf7228 authored about 3 years ago

Check if the current URL is in our set of curated domains

22d9f423475a9cb664a4bc22abcc88a0162419fc authored about 3 years ago

Find links from good paragraphs

9c3d1b5ad32b702e32f8cae0a6b84d56981425b3 authored about 3 years ago

Remove console logging in justext

b699a73d74c0dce56ae619d1fb524096c45020ae authored about 3 years ago

Store links

9e5b7193b43c6fb1f70baabbc21693f73d8483cb authored about 3 years ago

Merge pull request #7 from mwmbl/justext

Justext

62b090fb1b40b1618afc9daa167125d73028a5ca authored about 3 years ago

Tidy tests

9d08b4be196856214ed05801ca03441b56c79323 authored about 3 years ago

Use Colin's suggestion for simplifying response handling

60136443bc2ef5af291031220c3175ff0802901e authored about 3 years ago

Merge branch 'master' into justext

c5ba4905fe99427796ad4404bb29fa65220e2dcd authored about 3 years ago

Remove todo

0d394a8568252510b375444b2bfb74b5d5e28b99 authored about 3 years ago