github.com/tosdr/crawler.tosdr.org commits | Ecosyste.ms: OpenCollective

Delete .github directory

dd393d83921b4d019b3f097d3d8dc8a85e649ee4 authored over 1 year ago

Add healthcheck prefs

5da2809b5969326b73170726347cf016515febc6 authored almost 2 years ago

Update docker-compose

69afa7673a8ab4e0db98a4e7058e13ca49820b2f authored almost 2 years ago

Use new Authentication System

63877d4b3003836207188d62bcc8c08926d04392 authored almost 2 years ago

Aktualisieren crawler/crawler.js

3d80415a3f9a3ac7fab594a697c8f615a85f6121 authored about 2 years ago

Aktualisieren crawler/crawler.js

809f6020b7490c8fe72d07965be37548bb17045e authored about 2 years ago

Aktualisieren crawler/crawler.js

6493ee393b1cdb4cb71a02232e2be4bf76e32b52 authored about 2 years ago

Aktualisieren crawler/crawler.js

356bad2413df6e777caf2a430033d318027a7d05 authored about 2 years ago

Aktualisieren crawler/crawler.js

f6d72d11ffcce128d9cab78211a354db7f1f541f authored about 2 years ago

Aktualisieren crawler/crawler.js

68c34d51dbf850b6e4a16606f239c35bbee8d6ea authored about 2 years ago

Aktualisieren crawler/crawler.js

0de41265e4fd30110cd27be445dc9f04adc7ac7e authored about 2 years ago

Aktualisieren crawler/crawler.js

38d99cfd2eacc7b7e591afca001a41a5184168ba authored about 2 years ago

Aktualisieren crawler/crawler.js, crawler/functions/index.js, crawler/functions/error.js, crawler/.env.example, crawler/functions/success.js, docker-compose.yml

8f7cf9d9beff19877d9b47680820a8edfae45e3e authored about 2 years ago

Aktualisieren docker-compose.yml

0b6a682744a9a801b509c54bf9046f9b13d6eea3 authored about 2 years ago

Aktualisieren Dockerfile

ac32b39cdd55a57dd60ac3ff322e8110714e7c61 authored about 2 years ago

Aktualisieren Dockerfile

883950065be9788d8364e5875a247842e1e37aa8 authored about 2 years ago

Aktualisieren Dockerfile

4e9426785c68bd0a0ae1f77e38b810f36c86f526 authored about 2 years ago

Aktualisieren Dockerfile

23d533d908769a63c5e5d658ff5a5c6d68a70aa0 authored about 2 years ago

Aktualisieren Dockerfile

7a7f05daff96b4f30d971da99b17fbb957c1aad4 authored about 2 years ago

Aktualisieren Dockerfile

d049817afe0711105675f138d6a0568c07c45d18 authored about 2 years ago

Aktualisieren Dockerfile

9b710556bb25f300d2cd8b348329fa88f9393ccd authored about 2 years ago

Aktualisieren Dockerfile

16feb01f845862881657580a03047183772bdd0f authored about 2 years ago

Aktualisieren Dockerfile

43f07a3290d45de8dea4f2a08be4783e0d8836a8 authored about 2 years ago

Aktualisieren Dockerfile

d897d6c6f81beb98429828f9dd517cdfc0e27dd5 authored about 2 years ago

Aktualisieren Dockerfile, .gitlab-ci.yml

Gelöscht .drone.yml

925d123c6ef9df1d96bb951f3305b4af4aa73734 authored about 2 years ago

Fix lang hopefully

09c4b28c389bd676a28198f135a60defecfe1f4f authored almost 3 years ago

Fix setUserPreferences

50b064670e6beaca545388ab29079229b43f9aaa authored almost 3 years ago

Fix missing language

7a2215e0f4581f6b94290bb9335ea95dd10e60a9 authored almost 3 years ago

Fix missing crawl function

1c1eb09bb6d75d9701a992d6df8067cb02674be0 authored almost 3 years ago

Disable chrome options

328a993973a98df21125d9ee0e3caf7dd540140d authored almost 3 years ago

Remove chrome driver, use selenium driver now

28ce136d4a8aecb2a685193651e8074df0a9394b authored almost 3 years ago

Disable api key date check for now

0487d0b32a57906bb47989c641d671655b386e0e authored almost 3 years ago

Disable mandatory API Key. Crawler runs without tosdr api now

b40f312d1b53416143866c45dd94dd9f0d367c31 authored almost 3 years ago

Add Readme

34e0848bab5ed7c57a961cbe03f7cbdbe92402c1 authored almost 3 years ago

Verbose error for api keys

ce4f41b9b3f219dd89922a7587b195ba1890feba authored almost 3 years ago

Log API Key details

9f36888c634837a8084d9d63dbe52e79fe84f3a3 authored almost 3 years ago

Remove JBCDN entries

147ae861d4c65f562b64afb58a34a82d37949692 authored almost 3 years ago

Add idea stuff

7fcb20559878ea270740e32b5d8c1d53ca1de110 authored almost 3 years ago

Add Dockerfile

f0c535fb4c0ada724434aaabf2bfa6b0aa80082a authored almost 3 years ago

TDC-5 Integrate Jira into Drone

ab9260742087a97e39f9bacdb2b99f37133ee4af authored over 3 years ago

Update .drone.yml

48774b73f715357d284adc08fb12711233e633ea authored over 3 years ago

Add better debug to sentry for uncrawlable websites.

e016c122d3f0d3e171493781b9b59bb854b24d1d authored over 3 years ago

Merge pull request #4 from tosdr/EDIT-89_Add_Sentry

EDIT-89 Add Sentry to Crawlers.

83b88088d026cfb97118c54097c65f8db48bf0cb authored over 3 years ago

Add Sentry to Crawlers.

b2f5878ac7e381d495bbcaa819118dc27e61e615 authored over 3 years ago

Add .idea to .gitignore

6dfb1ceb62e328798014a0e88ebe3272e13769a5 authored over 3 years ago

Sign CI

23315f405188bdb4eb6869eb91141a56aed6b79a authored over 3 years ago

Create config.yml

872911e5d960518bdf56129c6c06f9c990ade1bc authored over 3 years ago

Delete SECURITY.md

eb3386322090ac666ba3a1f343784850fb3f80ba authored over 3 years ago

Update .drone.yml

b3146ea4b3c8fdbd7e7181f057523d3fee93219a authored over 3 years ago

Update .drone.yml

324058685dee1e9caa1710fa5053825afadd9932 authored over 3 years ago

Update .drone.yml

446cd7b3e870aa5728621f74ad33c660d8481124 authored over 3 years ago

Update .drone.yml

0434a87219b6aea634969043fe4a6a6f96542367 authored over 3 years ago

Update .drone.yml

8667138e58a4b5be1769fc626a872d48b1103f83 authored over 3 years ago

Update package.json

e46db35982b888511aef3a88a4ad2451662adc4c authored over 3 years ago

Update .drone.yml

938705aa6ac8da74f69628514c1b1c4b8d1222e3 authored over 3 years ago

Update .drone.yml

147ab4e70b59a117250580469e11029a8ccf3296 authored over 3 years ago

Update .drone.yml

7ea4ac01beae343ee81da8f7518950d50b325107 authored over 3 years ago

Update .drone.yml

14d1f21b8bb82877b841b841785733260915274a authored over 3 years ago

Update .drone.yml

3c6abd9ffcbce1766e1e20dae5d02c42c3cec6a6 authored over 3 years ago

Update .drone.yml

b0e324a8bde40b2f331d8c553132594a855cfa6a authored over 3 years ago

Update .drone.yml

b9704bfcba65c0507273ad6971a2c1143c88d2f0 authored over 3 years ago

Update .drone.yml

5837c9999979d38d1b2e9f94319afdc6c511c1e1 authored over 3 years ago

Update .drone.yml

5ad676a7d73a5039d68e49c6b34c97ad078c15a3 authored over 3 years ago

Create .drone.yml

fa618a3a3545b0bad2aefc075cf8f6b663be479b authored over 3 years ago

Follow redirects

cb6157872565756c0eb0c3a75e5da307e912a1d9 authored over 3 years ago

Update async.crawl.js

7d16f7797773540956950fa243982c206fab5068 authored over 3 years ago

Add files via upload

45bf5a9b710b4410afba1aa1a48339eb52ef7f80 authored almost 4 years ago

Respect revoked and expired keys.

f0b3ace9ed8db8ea683389157a04447b3537e981 authored almost 4 years ago

Add API Endpoint to use crisp api system

d40e2abf271e29ddaa841bc4e0fae38a76ace9de authored almost 4 years ago

Add another catch for invalid drivers

1f36032e1e9e77c8485f7e4593d5294a22e1e5bf authored almost 4 years ago

Fixed mimetype bug with charsets

74668226aa648ff65cddccb11e952d9c83ba28a5 authored almost 4 years ago

Redirect to KB on empty request

9ad14bafef94ade0cdda943090fc7ce422631d65 authored almost 4 years ago

Add njsproj file

6d994e0ffe9ceec1356e09e02f9deb9561dc3cbb authored almost 4 years ago

Add Masterserver envs

c1fbc65c6af8d33ea10c6facec0ad7836b14eb2f authored almost 4 years ago

Kill driver in catch

2e6e35263727462793528d2c9bd512ee51e9288b authored almost 4 years ago

Add xwidth calculation

fe65b7616428742aa47bc0e2c738449ac9ea60e2 authored almost 4 years ago

Changed vars to lets

bce21b753a124d8ca3b4cf649ed733bc9855e09e authored almost 4 years ago

Add PDF support

4f5724cd3260120fb0530d31a524cece8f0e59bb authored almost 4 years ago

Block pdfs and octet streams now. Only allow html and plain text.

c1af575301839a8c7758182051c8e6825797ef08 authored almost 4 years ago

Don't download files now, force pdf to open in browser.

337e387af978506b647881a2cc17a56081da5f74 authored almost 4 years ago

Create README.md

d57a4a7f54044941a06a9b3b8b1f15e92fc5c179 authored almost 4 years ago

Add urlshort link for the bot post

b2863718a3c8b5231196f306c1f4500777cdc33c authored almost 4 years ago

Attempt to bypass cloudflare and disable selenium detection

2f0029b209f27d8c258cec9871ad0e9dc4c7e98c authored almost 4 years ago

Fix crash for wait element

6c4eabaf117f9ddb8f954ccb3cca7597ce5a6f43 authored almost 4 years ago

IGNORE_ROBOTS=false

31fda4ce55db9d3c1ee5062971d0a721ee7a843c authored almost 4 years ago

API and xpath not required anymore.

4508dd17e62e7caac928993f8b759f6ba2230c22 authored almost 4 years ago

Scroll into view of the XPath

f875348d013470f3aa7858a299d25cf032525750 authored almost 4 years ago

highlight the xpath now

afe7fded435cc1b81e352c631ebbf87e8c97cc50 authored almost 4 years ago

Added JBCDN uploads to the crawler to see images of the webpage

2a2d30fdfdaaa9e98d53b95fba9d707173095027 authored almost 4 years ago

Added robots.txt honoring and user agent, kewl.

9c8bc8f89a4b55627a31649a750cfa9677efa4dd authored almost 4 years ago

Rewrote to independent functions

210cf657662f36df195c3e50e31ac5518d5f0dc4 authored almost 4 years ago

Rewritten crawler to use asyncs rather than promises.

Now wait until element is loaded, then getAttribute.

fc808b94d870487b37a7277b968139ccce783724 authored almost 4 years ago

Kill everything properly now

54a29c6261320392cfc957ae944fa79c6448c51d authored almost 4 years ago

Kill all sessions after crawling

a7926ee0f6496b8d5420469646176e6f4417c4e2 authored almost 4 years ago

Add default XPath

4d49e93ddf944cb050aa7483dd00c980bffa4ab5 authored almost 4 years ago

Added License

8459e86d901fe5638324dfd8c8e907a715e15d4d authored almost 4 years ago

Initial push

47f5379573810bc35d58cd25026633665ed49dc2 authored almost 4 years ago