Mwmbl - a non-profit search engine commits | Ecosyste.ms: OpenCollective

Implement new indexing approach

github.com/mwmbl/mwmbl - 1bceeae3dfb393d6d9a193706f6b39a3400594e0 authored over 2 years ago

Refactor to clarify flow of search-bar component

github.com/mwmbl/front-end - c2c732d434618b524abf72c334db68e7169391ef authored over 2 years ago

Merge pull request #30 from AdJaGu/master

Internationalization

github.com/mwmbl/crawler-extension - 4ec1ab9a40d3cdbdfcc80e19e4a1b1b5ff562365 authored over 2 years ago

Update translation key names

github.com/mwmbl/crawler-extension - 6a9e44ebbb134e50224b364cbd26047089abc63f authored over 2 years ago

add: Internationalization

github.com/mwmbl/crawler-extension - 81a62eaa7b3ba1b1630efd53043effab356d409d authored over 2 years ago

Use URL path to store locally so that we can easily get a local path from a URL

github.com/mwmbl/mwmbl - a8a6c6723985b8a86c9968ccb266b132f6a63941 authored over 2 years ago

Merge pull request #28 from AdJaGu/master

Update README.md

github.com/mwmbl/crawler-extension - dfa0bdc497f45c43bec5743397275bc9538b6b8a authored over 2 years ago

Merge branch 'mwmbl:master' into master

github.com/mwmbl/crawler-extension - 42cc5392ead20f6176b7cbdb25ed321b513cbdb5 authored over 2 years ago

Merge pull request #29 from mwmbl/send-all-links

Send all links

github.com/mwmbl/crawler-extension - db7021f152ef7fd6093ffadcfb6b41bc355450ba authored over 2 years ago

Send all links

github.com/mwmbl/crawler-extension - ffc84a2e93c7389f92cb5528b32949b41c6c34af authored over 2 years ago

Implement a batch cache to store files locally before preprocessing

github.com/mwmbl/mwmbl - 0d1e7d841c57c24dc6f4469b13ed4c8c81940198 authored over 2 years ago

Merge pull request #68 from mwmbl/fix-missing-query

Fix missing query

github.com/mwmbl/mwmbl - 27a4784d08e99b65275b17918396be99838ad43d authored over 2 years ago

Update README.md

Update "How to Build" section.

github.com/mwmbl/crawler-extension - 6505107199e11bb751e5101760c9d41425ef3281 authored over 2 years ago

Log at info level

github.com/mwmbl/mwmbl - 5ce333cc9aaa368413743af22ac1859058c19822 authored over 2 years ago

Allow more tries so that popular terms can be indexed

github.com/mwmbl/mwmbl - a097ec9fbe889285eda5ca47db43ef00e13bcc39 authored over 2 years ago

Enough preprocessing

github.com/mwmbl/mwmbl - cfca015efe93bce190d3b1f0089dce1f8c0cc103 authored over 2 years ago

Run preprocessing

github.com/mwmbl/mwmbl - 003cd217f4dc9c0bc587873b644e2dad8b9fc674 authored over 2 years ago

Just index a single page for now

github.com/mwmbl/mwmbl - bcd31326b8a1bb544f0e2e257c6e3467c9ee2459 authored over 2 years ago

Use a more specific exception in case we're discarding ones we shouldn't

github.com/mwmbl/mwmbl - a471bc2437ded7aea51b2cef9a7abec4c03db1fb authored over 2 years ago

Run update

github.com/mwmbl/mwmbl - ce9f52267a80c9da6223867819bb7b52df15e873 authored over 2 years ago

Catch corrupt data

github.com/mwmbl/mwmbl - 09a9390c92ef03edd69bb2fb8bbfc64da818b750 authored over 2 years ago

Add util script to send batch; add logging

github.com/mwmbl/mwmbl - 93307ad1ec2d31947f683073904cb38b00863ad5 authored over 2 years ago

Merge pull request #66 from mwmbl/fix-unicode-encode-error

Fix unicode encode error; bigger index

github.com/mwmbl/mwmbl - 3c97fdb3a0a6211a8e1d1f0bdbbb9292f0787c88 authored over 2 years ago

Fix unicode encoding error

github.com/mwmbl/mwmbl - 680fe1ca0c9d9b8f3793bbcec5572e5fe843909d authored over 2 years ago

Merge pull request #23 from mwmbl/fix/cd-xml

Change Parcel for Vite when building the app

github.com/mwmbl/front-end - 0b8bc0c4aea1615db3cf0a258a02b7e7cd6278d3 authored over 2 years ago

ci: changed parcel for vite when building the application

github.com/mwmbl/front-end - 11f38028cc18c4ffe5ed3aff59a8419894bf3a0e authored over 2 years ago

fix: manually install xml plugin for parcel

github.com/mwmbl/front-end - 4dbfda4da3b7d92e18f0fe6ecabd8cd4dc55aa50 authored over 2 years ago

Merge pull request #22 from AdJaGu/master

Add OpenSearch to MWMBL

github.com/mwmbl/front-end - e717feeb2c3d3130b12cd27e4c26c78e79cae2a7 authored over 2 years ago

Add OpenSearch

Add link rel="search" for OpenSearch autodiscovery.

github.com/mwmbl/front-end - f953152645303bce022bd59adcc69f7d75387bc5 authored over 2 years ago

opensearch.xml

Add an OpenSearchDescription file.

github.com/mwmbl/front-end - c5d4bc84cc0d12c2f36d3a7ae0c9438bdcf93046 authored over 2 years ago

Merge pull request #21 from sarayourfriend/fix/center-reduced-motion

Fix reduced motion search bar not centering on load

github.com/mwmbl/front-end - f98075a94c4c79e97993701897d8485a157a740c authored over 2 years ago

Fix reduced motion search bar not centering on load

github.com/mwmbl/front-end - 5b2f67f49328ce3093387d2be585b54e726a1c91 authored over 2 years ago

Merge pull request #19 from sarayourfriend/add/reduced-motion

Add reduced motion variant styles

github.com/mwmbl/front-end - b16b22931badc1adfae017ca42931563c4f40856 authored over 2 years ago

Add note about how to run the app locally

github.com/mwmbl/front-end - cd3494fdd12aee22c473219d0f53ae334118ff03 authored over 2 years ago

Add back real time search when not preferring reduced motion

github.com/mwmbl/front-end - 76331618f6080e82a8a855d24ffc1d10c2dc010a authored over 2 years ago

Remove unnecessary extra attributes

github.com/mwmbl/front-end - ccbc0aee4e4c23e190ae003a04bc2cbe4e43b264 authored over 2 years ago

Remove unused module

github.com/mwmbl/front-end - b8e1e674bba2fcfb7c9343ef16066ed7775a55e7 authored over 2 years ago

Respect reduced motion setting

Remove the debounced search behavior and instead wait for the user to
submit the query. We wrap ...

github.com/mwmbl/front-end - 7b91f7839acd64fac18aa3a6363192a1c4a29879 authored over 2 years ago

Merge pull request #61 from milovanderlinden/issue-60-consistent-use-of-env-vars

Fix issue #60

github.com/mwmbl/mwmbl - e1e1b0057bc1312ec6966186427064d595d53454 authored over 2 years ago

10x index size

github.com/mwmbl/mwmbl - fee5cbb40050636d633bd5cd7a23e49672539ea9 authored over 2 years ago

Fix issue #60

github.com/mwmbl/mwmbl - dfd3f3962ec7bcfba6b7b20f1c6f44e114fc1740 authored over 2 years ago

Don't include web.archive.org as a curated domain

github.com/mwmbl/mwmbl - dba50b372fb50c74d31d656db64ba4dacef3a359 authored over 2 years ago

Merge pull request #58 from mwmbl/improve-ranking-for-root-domains

Improve ranking for root domains

github.com/mwmbl/mwmbl - 2e40ae1dcaa8c6716785113ccb408ff554bb9fd0 authored over 2 years ago

Add a URL length penalty

github.com/mwmbl/mwmbl - 43815c73225598fe2d66798adab45f6a9737a960 authored over 2 years ago

Score domain and path, weight components

github.com/mwmbl/mwmbl - a3ff2f537f3988966e14aae270be931f97fc6ccf authored over 2 years ago

Merge pull request #57 from mwmbl/clear-indexed-documents

Delete documents that have been preprocessed from the database to sav…

github.com/mwmbl/mwmbl - 4b5df76ca5c41e8b024136f80a2e6667974805b6 authored over 2 years ago

Delete documents that have been preprocessed from the database to save space

github.com/mwmbl/mwmbl - 9482ae5028d31f9ec48ad7b8b2038a5ed74ef788 authored over 2 years ago

Merge pull request #56 from mwmbl/allow-links-from-unknown-domains

Allow crawling links from unknown domains

github.com/mwmbl/mwmbl - 6fa192daa40faed0e2e06664c1227165dd9d8cba authored over 2 years ago

Record new batches as being local

github.com/mwmbl/mwmbl - f9fefa0b62221a9a39ef780c70ce37874bd59661 authored over 2 years ago

Fix tiny font size

github.com/mwmbl/front-end - 7ac12a0a57e1315c1c70fe8145340b06434b4568 authored over 2 years ago

Allow crawling links from unknown domains

github.com/mwmbl/mwmbl - e578d557891153b780867244c69ea31cd8cedc5f authored over 2 years ago

Merge pull request #55 from mwmbl/index-continuously

Index continuously

github.com/mwmbl/mwmbl - 4967830ae1cbfb53d7eacba91f6e1dde6ea9f273 authored over 2 years ago

Merge pull request #17 from mwmbl/use-monolith

Use new monolithic back-end

github.com/mwmbl/front-end - c1d57bf082419bebb5ffc57e98c6fed508d9d869 authored over 2 years ago

Don't require a / on the search URL

github.com/mwmbl/front-end - bcbf6a2e70d75ac3ac79cb2408cec0f191b787c8 authored over 2 years ago

Don't require a slash for the search URL

github.com/mwmbl/mwmbl - db1aa1a9280db4c3b92100a9facb639dfb5e1164 authored over 2 years ago

Actually used the passed in timestamp

github.com/mwmbl/mwmbl - 24f82a3c2f783056969e6d6c4734c8450446664e authored over 2 years ago

CONFIRMED no longer exists

github.com/mwmbl/mwmbl - d47457b834916ceefe2d2ca41d62f84db0054db9 authored over 2 years ago

Fix log message

github.com/mwmbl/mwmbl - b6f29548dba0e06052397708d9d6f52a56fe2715 authored over 2 years ago

Wrap background tasks in try/except

github.com/mwmbl/mwmbl - e9835edc452f71a8d61c05df40261985cdfaa389 authored over 2 years ago

Allow batches to fail silently

github.com/mwmbl/mwmbl - 6ea3a95684ba601a218e5491f0787b0bcfcc8309 authored over 2 years ago

Queue the right type of batch

github.com/mwmbl/mwmbl - ddc8664c11d8081cd25b582e74153a5d8a882bb0 authored over 2 years ago

Queue new batches for indexing

github.com/mwmbl/mwmbl - 2b52b50569f96b36e94e9f2f59631aad51d35ad4 authored over 2 years ago

Correctly insert new URLs

github.com/mwmbl/mwmbl - b8c495bda860aa916168c5650a4db870dcfae2d3 authored over 2 years ago

Prevent deadlock when inserting URLs

github.com/mwmbl/mwmbl - 955d650cf434527b9c1152d407e73970c84ac209 authored over 2 years ago

Cache batches; start a background process

github.com/mwmbl/mwmbl - 1457cba2c2a931b6d390e0288bd5774b65ce6536 authored over 2 years ago

Use different scores for same domain links

github.com/mwmbl/mwmbl - ff2312a5ca211813b7e98419d6a4263b0eb1f7e3 authored over 2 years ago

Fix logic in found URL logic SQL and allow crawling URLs crawled by one user for now

github.com/mwmbl/mwmbl - 36b168a8f676f2da943ca396a16390a3e1188295 authored over 2 years ago

Temporarily disable startup background processes; add root domains; check for empty batches.

github.com/mwmbl/mwmbl - 5e1ec9ccd54583f9a8cc74dec7b7dc69d51f1132 authored over 2 years ago

Investigate duplication of URLs in batches

github.com/mwmbl/mwmbl - e27d749e18c59ed5d98ca23c357ef63c97141b68 authored over 2 years ago

Merge branch 'master' of github.com:mwmbl/crawler-extension

github.com/mwmbl/crawler-extension - 22a5008db2226e5b50b28ec2d1571ad333e410a2 authored over 2 years ago

Catch some uncaught exceptions

github.com/mwmbl/crawler-extension - 34472c7ddae2d9b0ebc2aee554b92fbf9e992ee9 authored over 2 years ago

Typo

github.com/mwmbl/crawler-server - e1bdeb95a80d8cdb863b0e32cd93eb4cb95cbebc authored over 2 years ago

It's late so I'm just making more bugs

github.com/mwmbl/crawler-server - 2750553563e7fc03ed8349cb36cc7930e5bdf569 authored over 2 years ago

Even more bugs

github.com/mwmbl/crawler-server - 30b03bb4a18e497f0128be4d0bd177f97ed08f61 authored over 2 years ago

Fix the same bug and another one

github.com/mwmbl/crawler-server - 436bb85d14b96103c5db36f4e94443313e0073fd authored over 2 years ago

Fix bug in crawling: don't overwrite status of crawled URLs

github.com/mwmbl/crawler-server - de85938ac4b1061bf8852e71e7303ac4fa6a1bf4 authored over 2 years ago

Add a script to count urls in the index

github.com/mwmbl/mwmbl - eb571fc5fee75e71fd3f44cfaa9c62d968f8f876 authored over 2 years ago

Make more robust

github.com/mwmbl/mwmbl - 1d9b5cb3ca9ca59ca58abdf0392cc5837f7cb861 authored over 2 years ago

Update queued pages in the index

github.com/mwmbl/mwmbl - 30e1e190729eead94916da9a4b06dad66aa9580d authored over 2 years ago

Tokenize documents and store pages to be added to the index

github.com/mwmbl/mwmbl - 4330551e0fe6302914dbfddc8d8c9712b1f51cc0 authored over 2 years ago

Update README.md

github.com/mwmbl/crawler-extension - 7071e5b555feb73b153e3492ea0e6ecf0b0c4fed authored over 2 years ago

WIP: index continuously. Retrieve batches and store in Postgres

github.com/mwmbl/mwmbl - 9594915de12ddffed4ed8278fc585e96933fe309 authored over 2 years ago

Factor out connection code

github.com/mwmbl/mwmbl - b8b605daedd12b6f055ad94155bd354f289af201 authored over 2 years ago

CORS is handled by nginx

github.com/mwmbl/mwmbl - c31cea710f75fd0f1c3bd65c86bb89b18c0cdda9 authored over 2 years ago

Use new monolithic back-end

github.com/mwmbl/front-end - eb0b4e4c774e517f9c6c0ed443beab371a8c1b81 authored over 2 years ago

Don't add CORS on the python side

github.com/mwmbl/mwmbl - 96da534ca561ee80dc3fc04c89921feb9daffdd0 authored over 2 years ago

Use updated CORS settings

github.com/mwmbl/mwmbl - 9dbb724ba9cce1561c58eb0a50cd3bee9af9a76e authored over 2 years ago

Remove seemingly extraneous backslashes

github.com/mwmbl/mwmbl - e3baf879183b5d67f4af0944643b52be199c34d7 authored over 2 years ago

Use an updated template

github.com/mwmbl/mwmbl - c245be775b86a0d845fdfaaa7af94c076aa45a62 authored over 2 years ago

Remove problematic SSL_DIRECTIVES line

github.com/mwmbl/mwmbl - 01772517dae3410ffcaf7dc2239c7848190dac59 authored over 2 years ago

Enable CORS in nginx

github.com/mwmbl/mwmbl - a67ca7b298c870ba9e123d479e94523b03ac5ca8 authored over 2 years ago

Use the dokku app storage

github.com/mwmbl/mwmbl - 866c17f2dcf44da0be22a3416db1a37a3f866575 authored over 2 years ago

Start processing historical data on startup

github.com/mwmbl/mwmbl - 16c26920997f53399a66b033525545fae9d22391 authored over 2 years ago

Add script to process historical data

github.com/mwmbl/mwmbl - d4009506892a6113152d0d300e4c2c4f3cf98d5a authored over 2 years ago

Expose the port

github.com/mwmbl/mwmbl - eb1c59990cc1c2554fd5b2272ab26c1a6a4e44ee authored over 2 years ago

Use the correct port for dokku

github.com/mwmbl/mwmbl - d7c6dcb5c2d414e47125bd92caa6adfe7b7db3ae authored over 2 years ago

Use a database URL env var

github.com/mwmbl/mwmbl - 77088a8a1b14b02d86183c1a29848537dadf4471 authored over 2 years ago

Put the resources in the package

github.com/mwmbl/mwmbl - 476481c5f817ca2b94b148489bd9d5ce2f714908 authored over 2 years ago

Copy the resources

github.com/mwmbl/mwmbl - 505e7521d45e50c6831db30819aebf804ddb2c2c authored over 2 years ago

Fix relative path

github.com/mwmbl/mwmbl - 5ea9efcfa2fd6d3f3ab44846a247c633b592598b authored over 2 years ago