libpostal commits | Ecosyste.ms: OpenCollective

[version] bumping version to 0.3 for consistency

github.com/openvenues/libpostal - fed599ac3974c053a6af60a5079d18785c5d7735 authored about 9 years ago

[tests] Using greatest (https://github.com/silentbicycle/greatest) for automated testing

github.com/openvenues/libpostal - 87899050b2806fbc1cfe282945ce8d1685eff8dd authored about 9 years ago

[docs] Removing the coming soon label from language classification, cleaning up the README a bit

github.com/openvenues/libpostal - 0bad3adf07aa173428a1f7d390cb11b0e28f352f authored about 9 years ago

[build] Adding relevant language_classifier sources to build

github.com/openvenues/libpostal - 95a7978131ac369181ebcfdde8619df0d6233a3f authored about 9 years ago

[api] Making language optional in libpostal cli

github.com/openvenues/libpostal - 93ed2bf15bfa8faf326a7356e9744af8bea6fae0 authored about 9 years ago

[build] Adding language classifier to data file download script. As the current file is rather large, added multipart downloads from S3 to speed things up

github.com/openvenues/libpostal - 789db8f5827df5e66e2930a7358a69e6364545e2 authored about 9 years ago

[api] Libpostal expand API will now detect language automatically using a high accuracy language classifier trained on OSM streets/addresses/toponyms. Hooray batch geocoding!

github.com/openvenues/libpostal - 42d169feeebc573b44f44b01272c95698201de32 authored about 9 years ago

[language_classification] Making directory optional on language_classifier client/test program

github.com/openvenues/libpostal - 71c51f2e45cbcd6d34a29a036e103e9c0c3d5ec2 authored about 9 years ago

[expansion] Regenerated address_expansion_data.c

github.com/openvenues/libpostal - c770468d031c5029e37fb5ed8b2cf421b2c7a6ef authored about 9 years ago

[fix] Removing feature printing

github.com/openvenues/libpostal - 36f52d97072165802d015daed5d88e4d874725e4 authored about 9 years ago

[docs] README updates now that the Python repo is separate

github.com/openvenues/libpostal - 239f8adec699a91472b3754d5eb6a820c35fe38c authored about 9 years ago

[rm] Removing Python bindings from this project, moving to https://github.com/openvenues/pypostal

github.com/openvenues/libpostal - cffc7e1034ddec834a514cc7ffe96dc32f8c5aa7 authored about 9 years ago

[fix] temporary files for language classifier training

github.com/openvenues/libpostal - 507746275441f106c077365d2218da90c154f199 authored about 9 years ago

[language_classification] Simple accuracy-based test program for language classifier.

github.com/openvenues/libpostal - 426edccbf80d7b4aca2718ab000e8af1541381b4 authored about 9 years ago

[language_classifier] Command-line client for language classification

github.com/openvenues/libpostal - 9abbf42bf46dde72ceb0d59896102a5078f27891 authored about 9 years ago

[build] Adding shuffle.c to language_classifier_train

github.com/openvenues/libpostal - 314b65e192e9f4bdbe522d09ccdbe19319646bae authored about 9 years ago

[fix] sign comparison in regularized gradient computation for logistic regression

github.com/openvenues/libpostal - ababb8f2d0bca17982ef5531b2b407f1f276a575 authored about 9 years ago

[build] Adding language classifier train/test/cli programs to the build

github.com/openvenues/libpostal - ae2b839f17d3be6c9b3a22308e1eece91750e964 authored about 9 years ago

[languages] Making Basque the only default in the Basque region.

github.com/openvenues/libpostal - 299998d8b5e4c9e1b3acdca1d49cd5ee5f22146c authored about 9 years ago

[fix] sets of potential languages in case phrase matches multiple dictionaries

github.com/openvenues/libpostal - b4dcb83e1067d9a0cb04be13f0197e0fb3498bfb authored about 9 years ago

[languages] using whole phrase len, not first token, in disambiguation. Using single unambiguous observed default language or unambiguous observed language

github.com/openvenues/libpostal - b713d102d1bb2ab55afc0ae3296d489392992b42 authored about 9 years ago

[languages] If there's a single default language, assume ambiguous abbreviations are the default

github.com/openvenues/libpostal - b3e730d83f14370524a0a52b95ae195fcf24b9dc authored about 9 years ago

[languages] Only count regional defaults when returning languages

github.com/openvenues/libpostal - fffaeecfc616f1943f2bce3e386e41092e8e357a authored about 9 years ago

[languages] Adding Spanish in as a secondary default in Spain to supplement regional language defaults so we're more careful in disambiguation

github.com/openvenues/libpostal - b735c7932658d79ddb40e8b4a879bcf0f40d6942 authored about 9 years ago

[languages] Language disambiguation treats the national languages as non-default

github.com/openvenues/libpostal - f8a0463aa093eda56e60c5484b739bec16dcb4a2 authored about 9 years ago

[dictionaries] Gulch

github.com/openvenues/libpostal - 87aff60a7e70dc6b426987a4d7e4de5f893197f3 authored about 9 years ago

[languages] Single character cannot be sufficient to disambiguate with multiple languages (Avenue A for example)

github.com/openvenues/libpostal - f04360732c4820aaef43bba9f175e04e6906bbfd authored about 9 years ago

[dictionaries] Adding a few terms to English dictionaries for automated disambiguation in the US/Canada

github.com/openvenues/libpostal - cb914ae85ba0ecd391dacc0035daa65b53209a3e authored about 9 years ago

[osm] Using the default probabilities for abbreviations in ways training data

github.com/openvenues/libpostal - 00ce71223fdacf3b0f0b6b89a38e7412d73eb55f authored about 9 years ago

[osm] splitting streets (way names) on semicolons

github.com/openvenues/libpostal - bab7a0f961ce90f86e43aa9bdb31f46d49fce2cd authored about 9 years ago

[fix] regional languages in French Canada

github.com/openvenues/libpostal - 3485738c2b5a78f5e255f688dec0154c5f77e2b5 authored about 9 years ago

[osm] Adding abbreviated street names in addition to the originals

github.com/openvenues/libpostal - 7646adfc0f33b3abc2a230fd1803802c0d7cdca9 authored about 9 years ago

[fix] converting semicolons to commas in OSM house numbers and picking one at random

github.com/openvenues/libpostal - 67130383cefbd65e776e5abae1f71bd786e57d5a authored about 9 years ago

[fix] spacing in phrases

github.com/openvenues/libpostal - 1bb797f7835cb3ad2d1e52c65d5331294ace816a authored about 9 years ago

[fix] spacing in phrases at end of string

github.com/openvenues/libpostal - 3a8c3dfcf64deb830329f72ae8c10a1cd6a2f95c authored about 9 years ago

[fix] Spaces in abbreviation

github.com/openvenues/libpostal - 78450bfad91fbb5da92aa95eb1282d10b2a91e52 authored about 9 years ago

[fix] convert UTF8 slices back to unicode before using with the Python trie

github.com/openvenues/libpostal - 308ceb5a5fb835179522a630e38139ddd9e41923 authored about 9 years ago

[fix] Only adding whitespace back into tokenized strings during abbreviation if it existed in the original string

github.com/openvenues/libpostal - 5eb6bb309b72aa500d99ec114c71e67fb4a0c481 authored about 9 years ago

[fix] var name

github.com/openvenues/libpostal - d61207e95ac1ccd3e7342c1b0d118c5a95ae4c94 authored about 9 years ago

[fix] geonames db not required in OSM training data

github.com/openvenues/libpostal - e44cba1d0632e6c8ef136d64fb940027161efb9a authored about 9 years ago

[osm] Adding abbreviated training examples to ways language training data

github.com/openvenues/libpostal - 4f03711e6014b9d828db4dab605d6783642c7fcb authored about 9 years ago

[osm/formatting] Dropping state more often than not, except in the US and Canada where those fields are more commonly used

github.com/openvenues/libpostal - c9fb4ee69da5c883ee2bf6f835f027c9a2b8f609 authored about 9 years ago

[fix] Abbreviation probabilities should only apply once, not once per dictionary. Also fixing issues where some of the abbreviations were doubled

github.com/openvenues/libpostal - ea9bb3f2d5b19001f2e7177d0dc933f7dca9272c authored about 9 years ago

[fix] simple whitespace field splits for the limited format training data (used for language classification)

github.com/openvenues/libpostal - f9f6558e06b7ba60855bf12f1430b6a2ecc83f2a authored about 9 years ago

[fix] Making sure rare components are dropped first, adding state and country back in

github.com/openvenues/libpostal - cd1db7b288ff41d8ab3b9c77551e5328e42a883e authored about 9 years ago

[fix] var name

github.com/openvenues/libpostal - adc3a00264315d67d23f83ca6b50b79749ab409a authored about 9 years ago

[fix] Actually better to remove country and state from rare components and let them use the standard dropout probabilities

github.com/openvenues/libpostal - 261beffa36020b149aadccd529f23a07cffa0119 authored about 9 years ago

[fix] Adding state to the more frequently dropped components

github.com/openvenues/libpostal - a6cc3d0114a487018d14ece5ef80e71c49d70052 authored about 9 years ago

[fix] state full name probabilities for limited vs. full formatted OSM training sets

github.com/openvenues/libpostal - bca3dae004d864766431b8802a53dbc8e4e20cf8 authored about 9 years ago

[osm/formatting] Higher probability of dropout for rare components like counties, etc.

github.com/openvenues/libpostal - d1cf2530928ffb9932f845f014a3d5692ab62a4b authored about 9 years ago

[fix] removing gazetteer configuration from disambiguation module

github.com/openvenues/libpostal - 9dd965a6faff4b844f4d1fa41c698ecd8b995c2b authored about 9 years ago

[mv] Moving gazetteers into their own module

github.com/openvenues/libpostal - b22646ee306389677fd002a8528aa893b682ace8 authored about 9 years ago

[fix] import

github.com/openvenues/libpostal - 5a68e7aeef26b72d41d37831d887df4225989b20 authored about 9 years ago

[osm/formatting] Randomly abbreviating street names and venue names using all the available libpostal dictionaries. Refactoring OSM formatting into separate methods which can be individually tested. Adding override for special phrases like UK

github.com/openvenues/libpostal - 6ac72576bc281217fb036431e284ffdfe36da50b authored about 9 years ago

[languages] Adding several different types of dictionaries for name expansion/abbreviation in OSM

github.com/openvenues/libpostal - f4995d4f0fe589ec62122fa94c3a09b55c1ae960 authored about 9 years ago

[dictionaries] Adding some Italian month abbreviations

github.com/openvenues/libpostal - 89aa039692bdd7137e42eab36968ef9563ff20c3 authored about 9 years ago

[languages] Fixing multiple expansions in the same dictionary for Python trie, adding length for prefixes/suffixes

github.com/openvenues/libpostal - 26cbb1eb8df25399a5545315de0985995542e5b1 authored about 9 years ago

[languages] Adding canonical string and dictionary type to Python trie, modifying disambiguate_languages accordingly, and adding lists of alternate forms

github.com/openvenues/libpostal - 0269d92e3d039b906162756e300432921c63cff1 authored about 9 years ago

[text] making normalize_string directly callable from Python geodata

github.com/openvenues/libpostal - 2e15db06dde23d17f8e20152be9364e5a504c648 authored about 9 years ago

[fix] prefix/suffix phrase search in Python trie search

github.com/openvenues/libpostal - 71e01e613376040d86731962e6ad585a63a0fbd3 authored about 9 years ago

[build] std=gnu99 in geodata build

github.com/openvenues/libpostal - 39667b73a2fda80258edc5c7940db55e58eca440 authored about 9 years ago

[languages] encoding in language disambiguation

github.com/openvenues/libpostal - 8b94a018e638c982376a8f27f2d7303f67af1ed1 authored about 9 years ago

[fix] arg count

github.com/openvenues/libpostal - 3262d2ccd38baefc2791897067241068073da1be authored about 9 years ago

[transliteration] Regenerating transliterator scripts

github.com/openvenues/libpostal - 5d5d5713cc70ab55981c7495cf1e2de75ed295df authored about 9 years ago

[fix] missing file in geodata

github.com/openvenues/libpostal - fe8f3158f6ca4c974b04fc677f05b5ad61db8123 authored about 9 years ago

[scripts] relative dirs in setup.py for geodata

github.com/openvenues/libpostal - 5fd9dc7e2b52f703d1d400115bec247641c38360 authored about 9 years ago

[transliteration] Fixing Malayalam script

github.com/openvenues/libpostal - da62ff309eb5eb20476f48c7098b2d9eac32c583 authored about 9 years ago

[languages] Adding English dictionaries to Indonesia

github.com/openvenues/libpostal - 5385cb71d64b093fafc739c0a3d4b4e8fbd16928 authored about 9 years ago

[languages] Changing the definition in script languages so only languages that appear on street signs will be used

github.com/openvenues/libpostal - 8030b235e6d1d46fcb820acbde517681a4bcd054 authored about 9 years ago

[language_classification] Adding script feature for any non-Latin script. Even if the script doesn't directly identify the language, it can act as a modified intercept (all Han script addresses will share the Han feature, even if we haven't seen one of the > 80k Han characters)

github.com/openvenues/libpostal - 0dfd8d6439a53d8dfe87c28b411293110cd5901d authored about 9 years ago

[language_classification] Removing the per-country classifier, text-based alone is doing close to 99% accuracy now

github.com/openvenues/libpostal - b9a3230f65d80f4aa1cd0ac13c922863314a11d8 authored about 9 years ago

[language_classification] Automatic hyperparameter optimization using either the cross-validation set or two distinct subsets of the training set

github.com/openvenues/libpostal - f808f74271130e7a49a9139dc50834a67c5f36a8 authored about 9 years ago

[fix] removing unused var

github.com/openvenues/libpostal - af5689ee52121418cd9f854b35e6a565b64823a8 authored about 9 years ago

[optimization] Using adapted learning rate in stochastic gradient descent (if lambda > 0)

github.com/openvenues/libpostal - 7d727fc8f075a9c1d3024605dd30cf37f24f0ca1 authored about 9 years ago

[fix] Trie prefix search tail comparison

github.com/openvenues/libpostal - 7b300639f1157c19021d620b1a3144e8ca2ae1bd authored about 9 years ago

[unicode] Regenerating unicode_script_data.c

github.com/openvenues/libpostal - 70dbfdd560b56b0e507bd1b86d97791993eaf110 authored about 9 years ago

[fix] tokenize_add_tokens respects specified length

github.com/openvenues/libpostal - de240d2b94556d7e7af066fcf6cacc59956a104b authored about 9 years ago

[io] matrix_read using array I/O functions

github.com/openvenues/libpostal - 10cadc67d7f6c10faab8da45a244ab6049e6c263 authored about 9 years ago

[io] Cutting down on system calls in trie_read

github.com/openvenues/libpostal - baba826d212bd44332946bb1758c63067bfb85ba authored about 9 years ago

[io] Sparse matrix using array I/O methods

github.com/openvenues/libpostal - cba2acc21fc46df42aaa4aa405d7437f5e0efdaf authored about 9 years ago

[utils] Adding functions to read numeric arrays from files

github.com/openvenues/libpostal - 46b35c5202c57f27c5b5f18b54c4e6e169603a64 authored about 9 years ago

[languages] Using unicode script in language disambiguation in addition to dictionaries. Eliminating dependency on address_normalizer

github.com/openvenues/libpostal - 3d7dd8966e1d809acab2ca28fa97e947551c3bed authored about 9 years ago

[phrases] Adding Python phrase filter from address_normalizer until a Python wrapper around libpostal's trie_search is available

github.com/openvenues/libpostal - fa32eacdd1950e001b0c6f200ae4a7f4b810add5 authored about 9 years ago

[osm/polygons] Allowing polygons that GEOS claims are invalid in OSM polygon index (there were some glaring omissions from the index like the polygons for the UK or Berlin). For some reason .buffer(0) creates weird multipolygons that no longer contain their centroids, etc. and aren't useful in reverese geocoding

github.com/openvenues/libpostal - f79a3c5bf4518f46dbed201f67e38d64859ffa34 authored about 9 years ago

[polygons] Don't call fix_polygon (force polygon validity) by default

github.com/openvenues/libpostal - 04f251c1cc6154e0a4a7c25bab2c653de542d5a9 authored about 9 years ago

[polygons/osm] append polygon nodes by vertices that connect to each other

github.com/openvenues/libpostal - 19a5541a85c38422ac63c176f715b81b83d6a838 authored about 9 years ago

[parsing] Adding an optimization to the parser API where, if the entire input is a single known geographic phrase like New York, it returns the most likely label from the training data. That way e.g. a search for 'Florida' doesn't get tagged as 'house.' This doesn't affect training, only prediction.

github.com/openvenues/libpostal - d4143c16853a5e661348b82e7e1389ebc96fa5fc authored about 9 years ago

[languages] Adding English dictionaries for Bangladesh

github.com/openvenues/libpostal - 24b4a680c3b43b2712573fc9695baafd3ee2c1ee authored about 9 years ago

[dictionaries] Using long forms as canonical for English degrees as new language models may do some auto-abbreviating

github.com/openvenues/libpostal - edebdf73e045585bcd9b09331b3dae718d6bc1b0 authored about 9 years ago

[scripts] Adding the tokenize/normalize wrappers directly into the internal geodata package so pypostal can be maintained in an independent repo

github.com/openvenues/libpostal - 58e53cab1c39da35f67b93ed30393943100e65fd authored about 9 years ago

[optimization] Adding learning rate to lazy sparse update in stochastic gradient descent

github.com/openvenues/libpostal - 622dc354e7563caa8ba41ec27b33edee1c7caced authored about 9 years ago

[build] Removing source from libpostal shared lib

github.com/openvenues/libpostal - 79f2b7c1927e75316b4298694678d7e3e3acb327 authored about 9 years ago

[build] Adding trie_utils.c to address parser train/test

github.com/openvenues/libpostal - 6a9c1e8c6d117cff433d9e9062dada13000b57a6 authored about 9 years ago

[optimization] Moving gamma_t calculation to the header in SGD

github.com/openvenues/libpostal - 7cc201dec3f812afffe8714b602d50277df361e7 authored about 9 years ago

[unicode] Adding SCRIPT_INHERITED as a common script so diacritics like COMBING CEDILLA don't break the current script and produce false word breaks

github.com/openvenues/libpostal - 25ae5bed33ee23089b184627238c4645a810d3ba authored about 9 years ago

[math] Adding sparse dot sparse given a dense output matrix (suitable for the minibatch use case), fixing sparse dot vector

github.com/openvenues/libpostal - 3260edcf18f4991707a85a201e79c8705aa66fda authored about 9 years ago

[config] language_classifier data dir

github.com/openvenues/libpostal - 736bc7c70d3c0af13343681c4ca5524f231afc55 authored about 9 years ago

[language_classifier] Language classifier training using L2-regularized logistic regression and stochastic gradient descent

github.com/openvenues/libpostal - ebaedb6bcf864f3cea26f1c7cf6239e99feae66c authored about 9 years ago

[language_classifier] Language classifier data set I/O

github.com/openvenues/libpostal - 56710cce216e9b58b3ca232b6f4c071535a1fe1d authored about 9 years ago

[language_classifier] Language classifier structs, I/O and API

github.com/openvenues/libpostal - 0558475a50d61c5dc4aad3d549de3b272dbb9316 authored about 9 years ago