github.com/openaddresses/machine commits | Ecosyste.ms: OpenCollective

Added wrapper around empty house numbers for issue #58

667e5b651706c0164b78490fe03dfa39e501204e authored almost 10 years ago by Michal Migurski <[email protected]>

Added fix for geometry error points in LinearRing like f9dd7cc

2bca555be1e15f1a848fbe44c7cadac8e9293f40 authored almost 10 years ago by Michal Migurski <[email protected]>

Reorganized some test parts

f6eb592954a00721a16f296676399a9aa004d081 authored almost 10 years ago by Michal Migurski <[email protected]>

Added fix for geometry error points in LinearRing for issue #58

f9dd7ccde5386c9e3b546bb63fdbec8f2f0018a1 authored almost 10 years ago by Michal Migurski <[email protected]>

Added awareness of application/x-zip-compressed mime-type for issue #58

d8ca3bb303d9651599e6fd08cab60b440d857c8d authored almost 10 years ago by Michal Migurski <[email protected]>

Raised global job timeout to two hours

aaa4800deb389dd7c246e4105f60b1dfc37dcae7 authored almost 10 years ago by Michal Migurski <[email protected]>

Fixed remaining Python 3 warnings

d6ea591cab004717184bf66c86cb6072d6a281a4 authored almost 10 years ago by Michal Migurski <[email protected]>

Removed all Node recipes

6f9161bc8506f6d542f8d4e9f5181d9c86822b08 authored almost 10 years ago by Michal Migurski <[email protected]>

Merge branch 'master' into ditch-node

9f242fb61a89e1c56960f813a0f97e1798cd1f70 authored almost 10 years ago by Michal Migurski <[email protected]>

TODO was done, removing comment

703c6e960036b222a3c836c9a8f7986775bec317 authored almost 10 years ago by Nelson Minar <[email protected]>

Removed and cleaned up Python 3 warning assertion

763d0a8cf9ceddb37a2c06daa91dd94b848c1ebb authored almost 10 years ago by Michal Migurski <[email protected]>

Corrected binary-related Python 2/3 bugs

0f5a9407df3829df33cf42e096fb8dcfb9963849 authored almost 10 years ago by Michal Migurski <[email protected]>

Corrected string-related Python 2/3 bugs

c855329931553fe57298be485c86b57afc101e4f authored almost 10 years ago by Michal Migurski <[email protected]>

Corrected csv-related Python 2/3 bugs

b585598e6820737a7687bad031a268f8c690b556 authored almost 10 years ago by Michal Migurski <[email protected]>

Corrected a few Python 2/3 bugs

0232f37621735f51c7d7eb33272a80f9946c49ec authored almost 10 years ago by Michal Migurski <[email protected]>

Corrected a few Python 2/3 bugs

3838168ac3149b861a84a88b9509c829e3765f05 authored almost 10 years ago by Michal Migurski <[email protected]>

Trying 3.3 without system packages

68c152abfd959438d07c7280bdc09988f3bbdf50 authored almost 10 years ago by Michal Migurski <[email protected]>

Trying 2.7 without system packages

d2ff00986a31335d5e7134c105384f1e9da58900 authored almost 10 years ago by Michal Migurski <[email protected]>

Replaced python-gdal with pip GDAL in Travis

23d2690961ebf71ddf82e3604d082cbae0a40354 authored almost 10 years ago by Michal Migurski <[email protected]>

Replaced pycairo with cairocffi in Travis, for easier installation

2f737bc645a710efa192da19fa0695d16be99fa3 authored almost 10 years ago by Michal Migurski <[email protected]>

Switched to more-compatible _thread from thread

c952220a0563f1a020891b3b5c4ace3928798349 authored almost 10 years ago by Michal Migurski <[email protected]>

Moved future import to openaddr.compat

d65b280700a6fbc0e7946221ac87cce68eb64f9b authored almost 10 years ago by Michal Migurski <[email protected]>

Created openaddr.compat for 2/3 compatibility imports

46c0701ddd6dc6ffd1f7d1dc5869703e9c3a1e4f authored almost 10 years ago by Michal Migurski <[email protected]>

Switched from unicodecsv to real csv for Python 3

22e9d756d8ce54a58cc4bf383c438a2893126a02 authored almost 10 years ago by Michal Migurski <[email protected]>

Switched to Pythons 2.7 and 3.2 with system-site-package support from Travis

01333f23a448ca2d84595b96c14228cfca6874a1 authored almost 10 years ago by Michal Migurski <[email protected]>

Added explicit Python 3 imports and pip install

2cfa2dfd562806efbc1f71ad911e33cd9bfff301 authored almost 10 years ago by Michal Migurski <[email protected]>

Attempt to make Travis work with patched system paths

58603f825cebcca5f66266370a08eb5cffc969dd authored almost 10 years ago by Michal Migurski <[email protected]>

Skip rows that TypeError

f7f328a6e989bc92abec5b40b07316308bbb1050 authored almost 10 years ago by Ian Dees <[email protected]>

Forgot to add this file

c46f819e6078cda5074a5075b1910aff0f5de0d5 authored almost 10 years ago by Ian Dees <[email protected]>

A bit more error handling on the first query.

Also handle really wide columns.

4094ad44e4a7ff4a928dfcae503f2d12303ffe41 authored almost 10 years ago by Ian Dees <[email protected]>

Port ESRI GeoJSON test to new ESRI CSV format.

c25584e1949e756abbf04e3d61dfd06c893baca8 authored almost 10 years ago by Nelson Minar <[email protected]>

Teach conform that "ESRI/GeoJSON" sources now create cache.csv

Some tests still broken, working on it...

f66a9dfffd579ac477f91fe6ce3a9918ed116adf authored almost 10 years ago by Nelson Minar <[email protected]>

Fix an ESRI test that was breaking in Travis.

This was masked (and I was seeing another failure) because I keep
running a version of the code w...

7bdecbd9c324b9d4d43bd1f9f28fbd4bad8297fa authored almost 10 years ago by Nelson Minar <[email protected]>

Skip an ESRI/GeoJSON test. Source data needs updating to ESRI/CSV.

1ae582697dc911ecbd22726b856f3e33b19f16a9 authored almost 10 years ago by Nelson Minar <[email protected]>

Rewrite the ESRI downloader to output CSV instead of GeoJSON

39742a5b36e54af2232f1576f3214063d9f6353c authored almost 10 years ago by Ian Dees <[email protected]>

Update Carson tests to work with the new ESRI code.

We no longer write a geometry type since it's not an OGR source.
The tests run completely now, no...

d16de6fd8baeaff167bdfc3a59a41159a18dd888 authored almost 10 years ago by Nelson Minar <[email protected]>

Mock the one extra call that was added to ESRI download

68b8c7035711752494f19978adf7a218d2fa21fc authored almost 10 years ago by Ian Dees <[email protected]>

Added m3.2xlarge EC2 instance type to swapfile setup

468d6dd47ed451597817f49cd62bdc4f39b4c9de authored almost 10 years ago by Michal Migurski <[email protected]>

Added --quiet option for logging so default isn't silent

103535061524d9e9b43d79151ef2be76c3b4d327 authored almost 10 years ago by Michal Migurski <[email protected]>

Added PID to log messages instead of thread ID

db7f0a702dda3451487eb6807547675337d4d7c9 authored almost 10 years ago by Michal Migurski <[email protected]>

Made EC2 spot instance big amount stingier

8064837cd443af609a5112b8696f6eb5bcb3ae83 authored almost 10 years ago by Michal Migurski <[email protected]>

Bumped EC2 instance volume size and fixed path to bash

34d4452c18015946737981683e5690bdb1525f90 authored almost 10 years ago by Michal Migurski <[email protected]>

Added EC2 m3.xlarge swap setup

ebfe096a92335671d1c914b38ac9ec8b5f6565b0 authored almost 10 years ago by Michal Migurski <[email protected]>

oops

d245cdcbf39ae4c8287cf0733a499142b574254c authored almost 10 years ago by Michal Migurski <[email protected]>

Increased job timeout from 30min to one hour

a05a8510f7d7ffe3b6a88f00d29a13fc794ced1c authored almost 10 years ago by Michal Migurski <[email protected]>

More reliably removing the temporary job loggers

65ef3c5b32509e4b43e4f52462bf22721a2769c3 authored almost 10 years ago by Michal Migurski <[email protected]>

Join the zombie pool. Long live the new flesh.

pool.join() ensures the worker child processes are cleanly exited.
Before, they were hanging arou...

d2ca719b82d7dcf91c77d4cb1a5a8581d2d9ba62 authored almost 10 years ago by Nelson Minar <[email protected]>

Add a few INFO messages about what process_all is doing.

c5912c8516d582872f2771671d2d062b44db0b77 authored almost 10 years ago by Nelson Minar <[email protected]>

Count weird exceptions as completed jobs.

2a009c6b82da7ed26634cb1cee654589855290d0 authored almost 10 years ago by Nelson Minar <[email protected]>

Capture logging messages from multiprocessing.

Key change here is turning on logging propagation for its messages.

Note that logging and multi...

d92a709e64a6618a64a3538f88a14907deeb08f2 authored almost 10 years ago by Nelson Minar <[email protected]>

Add handler for SIGUSR2 to invoke the debugger.

Note that the worker processes are still running while the
debugger is active, and will be spamm...

dc596c9796def7b70742f9451e495092226d48a9 authored almost 10 years ago by Nelson Minar <[email protected]>

Added support and test for skip tag, issue #54

e772a9cac17d9241e094e8ed21f28317eb03ad0f authored almost 10 years ago by Michal Migurski <[email protected]>

Added support and tests for non-matching cached data

7d6c4d822bd1053cadc550226ba39885e36795af authored almost 10 years ago by Michal Migurski <[email protected]>

Added support and tests for 404 cached data

7ee74681d09490c976515d0287debc101e1fbb96 authored almost 10 years ago by Michal Migurski <[email protected]>

Updated docstring

5c5c0c2196603eb59aef7c35840cf9767e3e0af9 authored almost 10 years ago by Michal Migurski <[email protected]>

Looking for http:// cache URLs to better use existing data for issue #53

55aa8fde761b735bd97553b482d6a53abc195132 authored almost 10 years ago by Michal Migurski <[email protected]>

Replicated second-run bug from issue #53

3423080557a1d59b2ed1ab29c07f5d5204f88d4b authored almost 10 years ago by Michal Migurski <[email protected]>

Job queue should now never abort if a Task throws an exception.

Reworked the exception handling logic for running tasks to catch
all exceptions, also simplify a ...

7c3d0fba8ba0915af2101ace45dfaf5519d5ad85 authored almost 10 years ago by Nelson Minar <[email protected]>

Don't set maxtasksperchild in the pool.

This seems to fix an idle worker problem, see issue #51

6370898b2ff060b5467aaf99175c5ef648820607 authored almost 10 years ago by Nelson Minar <[email protected]>

30 minute job timeouts, cpu_count()*2 jobs

These defaults should be a better match for EC2.

17da30729e154a5f28d34a3e7e97c7ffeff3774b authored almost 10 years ago by Nelson Minar <[email protected]>

Corrected output log keys

748cf2218254fd047d5e6c7e9b2b345b8644a672 authored almost 10 years ago by Michal Migurski <[email protected]>

setproctitle wants python-dev to compile

5d10888a5fd754940a41b666766edabdba8d4c62 authored almost 10 years ago by Michal Migurski <[email protected]>

cosmetic change to status reporting

6d9f65cca850ac877091691b03e88ced615a84ea authored almost 10 years ago by Nelson Minar <[email protected]>

Add a SIGUSR1 handler to shut down all process jobs.

2e5e3e8be6f63573b755e196812545376576eb64 authored almost 10 years ago by Nelson Minar <[email protected]>

Process jobs in parallel using multiprocessing.

Still a work in progress, but basic testing is promising.
Various hard-coded numbers inappropriat...

48306f8a07eae603274fa172ee78d3d5d6946c11 authored almost 10 years ago by Nelson Minar <[email protected]>

Sensible production defaults for job queue.

4082d680fb122eaf15076d76f7abdbdab6103560 authored almost 10 years ago by Nelson Minar <[email protected]>

Temporary changes, short-circuit some work while testing.

710c8bfc7ff48d48da4e26db5a2897e4609cf1b1 authored almost 10 years ago by Nelson Minar <[email protected]>

Set process title to the source being processed.

Discussed in issue #47

f1390386f58e32d3eb7230500c2562ae46346378 authored almost 10 years ago by Nelson Minar <[email protected]>

Hoist config variables to top of file. Better reports.

4c0c077cd10f6d317ba41ec707e6e27166903034 authored almost 10 years ago by Nelson Minar <[email protected]>

Uncommented upload_states in process_all

9bed1232e2e0a244c35fae206307c1b907124e7c authored almost 10 years ago by Michal Migurski <[email protected]>

Undo temporary changes for testing.

S3 uploads happen again, undoing 35487a48f3112

a5db9041787fa802f8e7726a6b3da01ce4487292 authored almost 10 years ago by Nelson Minar <[email protected]>

Added cooperative two hour timeout in Esri download task

f7976f006bf838c4d28095991a7cec3602e5b960 authored almost 10 years ago by Michal Migurski <[email protected]>

Fixing overzealous chef deletion

62ae4b8326a2c8dd5d79e1cd1bd21e63aaf70c1d authored almost 10 years ago by Michal Migurski <[email protected]>

Removed Node dependencies from chef

43fa8670bd525d64c3df72b2a6ca7ee4562d5ea0 authored almost 10 years ago by Michal Migurski <[email protected]>

Moved httmock to setup.py requirements

9b20a1729139e1118a3da4b2948ed11109c70cc8 authored almost 10 years ago by Michal Migurski <[email protected]>

Wrapped process_one in an internal try/catch to log errors

252145d9ad319dffa5cc92000c845130c5bce0fc authored almost 10 years ago by Michal Migurski <[email protected]>

Shapefile reading is now encoding-aware. Fixes issue #42.

This change got complicated, read issue #42 for all the detail.
The strategy here is to hope OGR...

09db1224c53bc39ca6722ed47bbd9ff151ef62ce authored almost 10 years ago by Nelson Minar <[email protected]>

Putting that log file someplace safer

25c6b404e62c269a1441f34eea0423f058fc3471 authored almost 10 years ago by Michal Migurski <[email protected]>

Replaced log stream handler with file handler

a60b73329e259836e4925c247268ad86ca405ba0 authored almost 10 years ago by Michal Migurski <[email protected]>

Removed unneeded result output attributes

6fcdc1d2d52952dc570c23c7c3aede762913caab authored almost 10 years ago by Michal Migurski <[email protected]>

Added log handler so single process log gets complete debug output

6b1f9e4649321dcaca59ac3e26634e9f76477207 authored almost 10 years ago by Michal Migurski <[email protected]>

HTTP timeout added on all download requests.

See issue #47

f81de20ed11278dd24817a99ecaa0f93a350d510 authored almost 10 years ago by Nelson Minar <[email protected]>

Remove unnecessary imports of requests.get()

9007530bd49105e3272eb66ee0901d8154c5abdd authored almost 10 years ago by Nelson Minar <[email protected]>

Attempt to use encoding specified in source conform.

References:

https://github.com/openaddresses/machine/issues/35
https://github.com/openaddresses...

e5099e5a23b8ab6571227c5f8487034c8a8b7cc2 authored almost 10 years ago by Michal Migurski <[email protected]>

Add support for finding GML files in downloads.

See also issue #38 and change bbd215935f3

3dd1f461875d95e74651e40ce93d47139f6cb91a authored almost 10 years ago by Nelson Minar <[email protected]>

Use a smaller slice size

d5ce64a0e4dc5f4459b680f09a517abd719b7a07 authored almost 10 years ago by Ian Dees <[email protected]>

Added test case for missing conform in source

ce9f1f5d3b5aef692686f542936e86864f971e9f authored almost 10 years ago by Michal Migurski <[email protected]>

Captured missing conforms for partial output

78bebea2c3b2d19d6abaa40ad3ba621909538a8f authored almost 10 years ago by Michal Migurski <[email protected]>

Fixed a bytes/unicode mismatch

b4224635448ac547ebc6e57949eafc14fb649d18 authored almost 10 years ago by Michal Migurski <[email protected]>

Forcing non-SSL for S3 URLs

4c76e526bdce1d89d9588491350ed4217ed34304 authored almost 10 years ago by Michal Migurski <[email protected]>

Get all the OIDs as a list and iterate through them rather than using a where clause

20db0f8aa01a374a9bed9a95fcb33570cfc67cf4 authored almost 10 years ago by Ian Dees <[email protected]>

Moved code around for clarity

f28c27f61a90d83329e6acc21a73450ca8ab8163 authored almost 10 years ago by Michal Migurski <[email protected]>

Use an apt cache to speed up build maybe

fc5a66fb031d30c4ec2000e1b6f53d0898b8619a authored almost 10 years ago by Ian Dees <[email protected]>

Fix objectid name in test

40f03a84fb3c174ae5ed3ed885585fb63cabbf59 authored almost 10 years ago by Ian Dees <[email protected]>

Try this to fix the test

096bd07e5c3fc1a490e1abbc6d4c8c3e461ba881 authored almost 10 years ago by Ian Dees <[email protected]>

Fix tests

1d759789d00422ef94ae169f18934706a0b87dc5 authored almost 10 years ago by Ian Dees <[email protected]>

Ditch whitespace I guess?

d927604369307485f2eed8b0c63ff840e571c740 authored almost 10 years ago by Ian Dees <[email protected]>

Query for the oid field and use its name instead of always objectid

0b06eeddab12f361ebff00a772417961109f1731 authored almost 10 years ago by Ian Dees <[email protected]>

Work around Shapefiles with occasonal missing geometry

This failure showed up in us-ga-muscogee, maybe others.
Unfortunately I don't know how to create...

36287b2c1fa5c12ad4fcfb361f633d2ba78617a4 authored almost 10 years ago by Nelson Minar <[email protected]>

More test cases for rounding

aedfd049f0e0569c4ff71c41f96a0b0fb15ae0ca authored almost 10 years ago by Nelson Minar <[email protected]>