github.com/mwmbl/crawler-server commits | Ecosyste.ms: OpenCollective

Typo

e1bdeb95a80d8cdb863b0e32cd93eb4cb95cbebc authored over 2 years ago

It's late so I'm just making more bugs

2750553563e7fc03ed8349cb36cc7930e5bdf569 authored over 2 years ago

Even more bugs

30b03bb4a18e497f0128be4d0bd177f97ed08f61 authored over 2 years ago

Fix the same bug and another one

436bb85d14b96103c5db36f4e94443313e0073fd authored over 2 years ago

Fix bug in crawling: don't overwrite status of crawled URLs

de85938ac4b1061bf8852e71e7303ac4fa6a1bf4 authored over 2 years ago

Merge pull request #2 from mwmbl/url-database

Use a central URL database

d9be475ebaf4fef237eb7fa791d6c17d7be27316 authored over 2 years ago

Create tables on startup

0afaaf3b4bb5d93a9f51f541ae228a12594195c7 authored over 2 years ago

Use a better environment variable

7869b023c58d096f55f1e0c408783c8a4ecc7712 authored over 2 years ago

Update for new batch format

f78e43b2fb3349e51cc8a56faa72f5ec3f6da786 authored over 2 years ago

Add get new batch for user endpoint

39a07a4c7524fbeb230a18faaebc9069655e3142 authored over 2 years ago

Record URL scores

aa7c3ab7b2fdbecbdf95101fd9a48c38e1fdc657 authored over 2 years ago

Create a URL database class to hold the connection

e65ceba454c41265a9c0acfd6fd17ef772f21b5e authored over 2 years ago

Create an endpoint for historical batches

cbad8790114a9c3a117ca4e53bebff9010983b36 authored over 2 years ago

Update the updated date

5d731fe9a5fd20a808c12f1e6b79c1806552c349 authored over 2 years ago

Add a way to update state for found URLs

bc6efc1aaaf3c1a16c620a0235503143f9824d42 authored over 2 years ago

Move urls to the crawler module

b8d3de8a1a549dbd5b1843b4c367d89f3bebda7f authored over 2 years ago

Add code to create URL table in Postgresql

200d80be1696ec2f4e5669ca253949e4ac36cd93 authored over 2 years ago

WIP: store crawled URLs in a database to prevent duplicate effort

6b7c1dc541ffaff543c3dafd3b1a717460e777df authored over 2 years ago

Merge pull request #1 from kodxana/master

Docker-compose integration

f42c3cff0c660e65d071f9964f149dd1527a6a41 authored almost 3 years ago

Update README.md

dbf2e1cc516dec6cc9322c8fd2130cb9b809dc43 authored almost 3 years ago

Update .env

f1b191d3487341631fd6f5803e72cad326025f37 authored almost 3 years ago

Update .gitignore

.env file should not be shared cause it contain private informations :)

06475dcbaa48fe4e355acaf13b536a75816201a0 authored almost 3 years ago

Update app.py

Endpoint and S3 Bucket name are now loaded from ENV variables instead of static addresses in cod...

a4eeeff5ed172a26b15ae4319f036ff19df6e28d authored almost 3 years ago

Create .env

789d48d7c1c78d880b0a84cb68e30eeed3805bfb authored almost 3 years ago

Added docker-compose

Docker compose will build image at first run and load environmental variables from .env file
To...

307f6d426f065d386af681678fcc01a4111d2bc4 authored almost 3 years ago

Add an endpoint to download a batch from a batch ID

309d438c7ea2b0d5eb8b92ac3120fcdc51c10f76 authored almost 3 years ago

Store the last batch and provide an endpoint to retrieve it

78b61ac1d5b63b9405f4330335e23512d770aadc authored almost 3 years ago

Return public user id (hash) and add an endpoint to get user's batches

89de9c3d5818b2dc204be25661ada473942ede39 authored about 3 years ago

Add endpoints for batches and users

2ee16613897745e45e79a126bddc2877748f5224 authored about 3 years ago

Update path

67e0927185ace4c8bdacc0c684549817f1f8be72 authored about 3 years ago

Add some extra fields to stored items, check user ID length

d7dedd8051ed405c52ec3bf4291ac362aa88c571 authored about 3 years ago

Follow @nitred's suggestion for folder structure

9bdd86f241aaac219e5fecb1cad05fcdb323f395 authored about 3 years ago

Specify correct host

6b40c86501f6fd66b843c6ce7d984789d9830929 authored about 3 years ago

Use the correct port (cloud run default)

d705fe4136881c1d713870509e48536e9c112246 authored about 3 years ago

Fix docker build

21221f2c4d1fa49b25f9899b60ed1dd6a470d28f authored about 3 years ago

Copy the right files

82877656abbf682799745f63dfc7139fec7de3fd authored about 3 years ago

Trigger another build

9cfa29c15d604c2655ed08adada91c57b1b06556 authored about 3 years ago

Commit to trigger build on google cloud

1df468d70c819eb21890fe6365e8f42ea4511210 authored about 3 years ago

Add Dockerfile

99c53362278f99f4c71561d4ba318e1e0cd07c16 authored about 3 years ago

Upload to Backblaze

3d11894569966ca7112ebbb6f497b6fe12955539 authored about 3 years ago

Add README

fd2801fcc0e71f005277afd1746cb8f1540c5417 authored about 3 years ago

Initial commit

3a6d779494231f59663720b9a83d71842c981f1d authored about 3 years ago