A lot of import pipeline details lived in one place. As many details as
possible are now moved i...

If a user's peliasConfig specified a list of files, the importer was not
properly joining the re...

Count records that don't have a lat, lon, housenumber or streetname and
display the total count ...

It's expected that zipcodes aren't always present, so a try/catch isn't
appropriate.

Both of these fields are required, so if there is an error setting
either of them, the document ...

Add cleanup module with numbered street handling

This cleanup code handles numbered streets with leading zeros. For
example, `05th Street` will b...

This can happen on import when the user is specifying a directory of
files and hasn't set anythi...

migrate from FST to ngrams

resolves #15

lib/import_pipelines.js
-Remove repeated spaces with a single space.

test/(openaddresses_sampl...

This reverts commit bbf81fc9948b6998476d4d4eb8f7abc7a4f30b4b.

package.json
-Link to a version with an attempted memory fix.

lib/import_pipelines.js
-Instead of logging a message per bad CSV record encountered
(which mi...

lib/import_pipelines.js
-Extract the address-validation conditional to its own function,
`isVa...

test/openaddresses_sample.csv
-Make the data more unpredictable by adding extra additional
col...

lib/import_pipelines.js
-A change in the OA schema (the addition of a `POSTCODE`) field
broke ...

README.md, import.js
-Close #13: allow cherry-picking the files to import from
pelias-config.

README.md
-Update the documentation of the `files` property in
`pelias-config` against the cha...

import.js
-When an empty `files` array is specified in the config, import
all files from the t...

README.md, import.js
-Update documentation to reflect changes in the past two
commits.
-Docum...

import.js
-The code that filters out non-CSV files was meant to only run
on files read from a ...

import.js
-Check the `pelias-config` openaddresses configuration object
for a `files` property...

import.js
-Log the total amount of time the import took, as a convenience
(instead of forcing ...

README.md
-Remove the copy-and-pasted `node import.js --help` from the
"usage" section to prev...

import.js, package.json
-Integrate the latest admin-lookup (v2.0.0), which ships with a
pseudo...

import.js
-Concurrent requests don't make a ton of sense for what seems
like a CPU-bound problem.

import.js, package.json
-Update references to the `pelias-deduplicator-stream` package
to refl...

package.json
-Use the `^` in dependency versions to make npm install the
latest compatible ones.

import.js, lib/import_pipelines.js, package.json
-Use the new `pelias-logger` package instead o...

lib/import_pipelines.js, package.json
-OpenAddresses data got trickier to parse now that it con...

import.js, package.json
-Integrate the `pelias-admin-lookup` package, which replaces
`pelias-h...

import.js, package.json
-Use the `minimist` package for argument parsing.
-Make the data path ...

package.json
-Set the `pelias-hierarchy-lookup` package to the latest
version, which should fi...

import.js
-Add two conditionals to `interpretUserArgs()` to check whether
the provided directo...

admin_data/(country_codes, state_codes/(ca, us)).json
-Remove all `admin_data` files, which wer...

lib/import_pipelines.js
-Fix #9.

lib/import_pipelines.js, package.json
-The replacement of `fast-csv` with a dead-simple CSV par...

lib/import_pipelines.js
-`fast-csv`, the package used to parse OpenAddresses CSVs,
appeared to...

import.js, lib/import_pipelines.js, test/test.js
-Fix the few things that `jshint` complained a...

.jshintignore, .jshintrc, package.json
-Add a `.jshintrc`, and the `precommit-hook` package to ...

import.js
-Add some configuration options to Winston.
-Add more diagnostic logging statements ...

import.js, lib/import_pipelines.js, package.json
-Add the `winston` logger, and replace all `co...

README.md
-Add usage information, as copied from `node import.js --help`,
and this repo's trav...

import.js
-Add a `--help` argument, to print the usage message.

import.js, test/
-Improve the module's test coverage substantially, and refactor
some bits to ...

test/test.js
-Add a unit-test for the newly revised `interpretUserArgs()` in
`import.js`.

import.js
-Rewrite `handleUserArgs()` such that it returns an object
containing the interprete...

test/test.js
-Add a test for the interface of
`createPeliasElasticsearchPipeline()`.

test/(test.js, openaddresses_sample.csv)
-Add a thorough test for `createRecordStream()`, to ch...

import.js, lib/import_pipelines.js
-Break up `import.js` into another module, and add some
doc...

lib/import_pipelines.js
-The `try/except` block in `createRecordStream()` performed more
actio...

import.js, lib/import_pipelines.js
-Add documentation comments to all functions that were missi...

import.js, lib/import_pipelines.js
-Move functions that create parts of the import data pipelin...

lib/create_admin_values.js, test/test.js
-Remove the remnants of the old admin-value assignment...

lib/create_admin_values.js, test/test.js
-Remove the `create_admin_values` module, as we're now...

import.js, package.json
-Use `pelias-admin-hierarchy` to perform admin value lookups
against Q...

import.js
-Make address deduplication and admin hierarchy lookup optional
steps of the import....

import.js
-Add a real CLI usage message.

README.md
-Add a warning indicating that this importer is strictly a WIP,
and not entirely rea...

import.js, package.json
-Use the `hierarchy-lookup` package to identify address admin
values, ...