github.com/openaddresses/batch-machine commits

Partially added second test of conform behavior

803b0954fa613bc96d36c337e3d721c75d4f70bc authored about 10 years ago

Added first test of conform behavior

Conflicts:
test.py

de060fd0601174472bb8913975be6b17a11e5f45 authored about 10 years ago

Added geometry type to data excerpts

af34bdbbd0872dbf4baf42cd3804df9b4548b9e2 authored about 10 years ago

Added test for ESRI data source

9f498413fbc7ff3ff3a897fb3d3bcbebc207291d authored about 10 years ago

Fixed a file clobbering bug in .zip excerpts

50d5878a276707e893c148be3b6f8f79873482c1 authored about 10 years ago

Moved sample_geojson test to openaddr.sample

3ee7c8bf61f98bfd589ccacda06ef49d71374ab4 authored about 10 years ago

Added streaming JSON samples to openaddr.extract()

14ee8f00e1d9ad8ff69f984059aa8d33ce582ad7 authored about 10 years ago

Added test coverage for JSON sample code

485072c2f13f0cbb5d36ba62c6df72a5af32153b authored about 10 years ago

Switched to returning a JSON literal in sample

320a279ce218378b9dcef4517090d2da647cdce2 authored about 10 years ago

Moved iter.py to openaddr.sample

120aa0e76769fcd4bc3b19badb1bc065145254e3 authored about 10 years ago

Successfully parsing a few features from even vast amounts of GeoJSON

9af3ebc1dcf6f29813649704ea061ed8cb4e0106 authored about 10 years ago

Closing OGR datastore differently

See http://trac.osgeo.org/gdal/wiki/PythonGotchas#Savingandclosingdatasetsdatasources

e1314252743a06828b307126232f0451f1700f2f authored about 10 years ago

Closing OGR datastore to stop eating memory

b374ecc4b5ffdd9ef22868b1a017826ba6798c8c authored about 10 years ago

Added sample data excerpt to ConformResult

Note sure where to put this yet, but it's a start on issue #8.

0840564c44149e1445998d6aeaeb2a2f1e71522e authored about 10 years ago

Added ESRI JSON sample source to tests

2395d58eb17c3c00721311b7ec23414f8db23977 authored about 10 years ago

Minor touch-up and removed various mentions of urllib2 since it's now requests

a9454a4131355e498938c4121e934b2d5ef5a3a8 authored about 10 years ago

Added notes about unpredictable process.join() bugs on Mac

c78f990d2b2305d442b7ce0be90496d4e5e18b55 authored about 10 years ago

Removed mixed variable scopes in cache() and conform() processed

455b63da01e694fd352c41ef8a9adb2e9e6356be authored about 10 years ago

Fixed missing file extension bug for Alameda County data

8a460410ebab0b7f0384f320bea0ee9733eb27a8 authored about 10 years ago

Switched to faster cPickle

Found that non-fast pickle was throwing value errors.

7e1c94f69035b9d17e87b67ee63424d8052ce4a6 authored about 10 years ago

Found it necessary to use matching modes on pickle file

When debugging on Ubuntu 14.04, discovered that mixing 'r+' and 'r'
was generating bad file desc...

110dbba553112cc9b3fcda0c456ee9c7d79540e9 authored about 10 years ago

Cleaned up and commented test script

e18c5abea7153ce28fa4fe7e44c5cc60dbfba0ea authored about 10 years ago

Added Travis CI config

6499e879cb3bef25f3607cc82ff06110755be5f2 authored about 10 years ago

Fixed Travis CI config

See https://github.com/travis-ci/travis-ci/issues/2938

b6dea1380b4168f648441f96fe1c570eb316dc68 authored about 10 years ago

Using hashed URLs for cached data in download tasks

fdb5531e58f043e7f0d3ad136582c54a0a6d62d7 authored about 10 years ago

Made test_process() chattier to figure out why it's failing on Travis

31816ab2846a004499cdc31a29fe6e349bceddd7 authored about 10 years ago

Fixing a few preliminary test failures

21626bacf3ac49b1d57767d1965cde1436b37656 authored about 10 years ago

Replaced urllib2 with requests in openaddr.cache.Urllib2DownloadTask

4e8aa9229adfcf990d52676cf94044e804d08f9c authored about 10 years ago

Replaced upload_to_s3() with S3 wrapper class

3236d587dadeffcfd3485d587303038ab59a9083 authored about 10 years ago

Teach it about esriGeometryPolyline

97f4a07a78fb986960ef35db1d12dec3efb2de72 authored about 10 years ago

First swing at moving to all-python downloading code

e23eeb33041ada2615a2e654f2b8f1d616b485be authored about 10 years ago

CacheResult is in cache.py. Forgot to save json result

bcefc35d0798f3f83829edf6199329db641caae5 authored about 10 years ago

Fixed a few post-rebase bugs

d3cda04be6e3cc40acc0b80ef034c03207bc15f6 authored about 10 years ago

Added initial test of HTTMock

6afc1158c9902e80360e885cf3f25c65c8ea08fb authored about 10 years ago

Add in conform-like stuff

c3c3c21e5efba5ec4933068376c708ef1c51fc86 authored about 10 years ago

There's no stderr/stdout output yet

6a6db70e04eb7396984b9572ff3547b7d0c50ad6 authored about 10 years ago

Introduced HTTMock to prevent network traffic when testing

345f4b0a5577d31c1f145691c90c92daadea1ad2 authored about 10 years ago

Change default S3 bucket names

4f375c861fffdd37d6c6562e929ba266fc8d573e authored about 10 years ago

Trying to get mocks working, failing at process

5fae99fa0107eab575b88c0f1ea6e2463ae1afd4 authored about 10 years ago

Take out the expand/conform stuff and only upload the source to S3

61b7f0c42036d26a99955aa66d104a855660822d authored about 10 years ago

Completed HTTMock to prevent network traffic when testing

de96c7603e8dcb757751a36b8f0b5e5d8b95a6d4 authored about 10 years ago

oops

35830eb1fc475eddeb3f29fdd6b57efb3ec0da63 authored about 10 years ago

Allowing unicode display of sample data

d2389efc21926056e14b96413aece3e284e523bd authored about 10 years ago

Added fake test so Travis-CI would shut up about the main branch

38e96e32cc00580320189f499e288f1220e9feb2 authored about 10 years ago

Cleaned up excerpt code

585dc5f26d58e151a6a684620ed0996aa77af33c authored about 10 years ago

Added JSON sample data to output

26cd614937fc994e60215dabb9726cd627f60300 authored about 10 years ago

Responsively showing additional sample data columns

d21ac2e0c28c3acb4cd2ea43f917530a8b995b6c authored about 10 years ago

Set two-hour limit for cache and conform subprocesses

bbb36a98ad2418bf8e2a3a351ee8b0114f1a615d authored about 10 years ago

Pinned requests to 2.2.1 because of Ubuntu 14.04 weirdness

See also:
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1306991/comments/10
https://...

bc6239648b4f004afa0268a3c1b6d948d2a0d182 authored about 10 years ago

Cleaned up some README details

2d4826628a9445ee322c9460dc3a35c4421eda2d authored about 10 years ago

Preventing cache on primary index.html

30d53a4c40e24cdc6ae89cb07f3bf9f20e66ec04 authored about 10 years ago

Fixed blank sample data in output summary

a5ca3d392c4cc7d2646fd9d0daff4a09fd698a94 authored about 10 years ago

Added inline display of sample data from each source

b8f38ef5d48d67973f34c69390471dce31fda358 authored about 10 years ago

Added output date display to summary page

e4844590031e4df62e828214ead43ff8c9d43f81 authored about 10 years ago

Including bad URLs of sample data in state.txt

ede86475984dc748626757a5bca5015af289bfbc authored about 10 years ago

Bashed in basic data samples

Ultimately, use code from https://github.com/openaddresses/machine/commit/ce43a30447da but
for n...

0d08331710fb4aab3df92e7804e671bb0ae2bac4 authored about 10 years ago

Stubbed out excerpt code for sample datasets

91b4a5eebc3afbab3e2cd0d33ce9a7b9dac30c45 authored about 10 years ago

Finally added a .gitignore

079e4568c42a4f33fc6380f49ef7b7dede11f387 authored about 10 years ago

Tweaked small-screen rendering of output state

ed7472d00009c4da57b1c3d9aae9fcc027539a9c authored about 10 years ago

Updated index output

e52af1660fecb8cd65c740a7e61f0f46348ff494 authored about 10 years ago

Updated summary output

a071ef723b6311df2b8ceb628e313bb1f0903f7e authored about 10 years ago

Allowing for old-style state.txt and new-style linked state.txt

85f5a4ba8d6dfd1c9c5f63ad2ccab7277127bf27 authored about 10 years ago

Added missing package requirement

a8fd06f6038cf8994f0f62816918ff3d890b7205 authored about 10 years ago

Added summary output

ec8c0f01bc26c4ac24bd590e84b5f50eaa4e14b8 authored about 10 years ago

Fixed process test

09f9b89062aba1711de6f4365009081f6635e1b6 authored about 10 years ago

parse-logs.py output is now included in state.txt

b31f4b4c53527840d4a8b35e03b3b70fd45b3c0c authored about 10 years ago

Pushing command output to S3 run directories

2fed840b38f781e27fdd3d82fd5a21d6ade08332 authored about 10 years ago

Started keeping old versions of state.txt around

Closes #5.

ed9bb230c54a54871dda7b9bc61349263afa2d50 authored about 10 years ago

Updated with OA-owend bucket names

ac6b0a919590dd43342564dfcc7bd8b0f18dd359 authored about 10 years ago

Started testing actual openaddr.process code

5df142fa236b6b7395f7cf71ebe70d3696fc5028 authored about 10 years ago

Moved parallel.py inside openaddr module

44cf03d17d2883998ef4962b3a64b896969daa76 authored about 10 years ago

Added primitive HTML summary generator

d0b80111c1ebf66a7be5524ed3e6fd79be8da34b authored about 10 years ago

Added render-us console script to install

0f2a0bc0962e5acdb37ba68ff7b4ee657f753b1a authored about 10 years ago

Preparing render for deployment

9a0e2415da45c3b612aa527f5332102eb45f1549 authored about 10 years ago

Ported render-us.py script from https://github.com/migurski/openaddresses/tree/display-geodata

6cff5f9650a1bb23cfba8994af5697aa38437dd1 authored about 10 years ago

Moved Python dependencies to setup.py

f6be15717e54fcaffb26cbf771b0589c1c812338 authored about 10 years ago

Cleaned up S3 interface slightly

259636f59e8a2549697701f9b3f94dc3f500bbff authored about 10 years ago

Switched to real list of sources

This and efc2501 closes #1.

2077c26b36d692e19e91653ad72862b7020785e5 authored about 10 years ago

Fixed sorting and missing directory weirdnesses

Closes #4.

76801474185749917c7c3665fbee95472ba845a4 authored about 10 years ago

oops

d167f095bb0578df9cd4ee8838a3c4aa6fbab9e0 authored about 10 years ago

Set a smarter thread count for each task type

d5cd113a7ec12898720d588aa5a1c97d905ea065 authored about 10 years ago

Sorting jobs to put the biggest pickles in the jar first

51b034c43ef41588a832f47ca266f6d2348e69d0 authored about 10 years ago

Added S3 configuration to argparse

Closes #2.

f9372c76da23b869d2a29fa1e8af7b0e3d86e74a authored about 10 years ago

Moved tmpjson creation into _tmp_json()

ea4b49cff83eb8b88c2f1eed8f6b4bad84977247 authored about 10 years ago

Created an S3 object to pass around instead of bare bucket names

aa78ac9487155c59cefe344f613945f9ad40a723 authored about 10 years ago

Added elapsed time for each source to output

40888182309e4e459cbca37be3dd10e22d9c4d91 authored about 10 years ago

Moved error-handling out to _run_cache() and _run_conform()

4d2d4b90d74259155d9982bf71d6c71be2082281 authored about 10 years ago

Added ConformResult class for conform() results

1195d8de6abba662f622adb23d827c419eaa1cec authored about 10 years ago

Added CacheResult class for cache() results

b6bc167aab2663d5b96c39498e2c340935284751 authored about 10 years ago

Added simple log parser for parallel.py output

19a6e4c3c82ff053425ff8dbe4b28fed5422c8cf authored about 10 years ago

Fixed a conflict between working directory JSON file names

Some sources are cached as JSON, with identical name to source file.

8b14c6312dfd9113b836bafe53a74f6a81ddb4be authored about 10 years ago

Added more JSON error handling to cache() and conform()

ef25a37344326337b0ac889ad0653463a110d6f4 authored about 10 years ago

Fixed zombie thread bug

4390ba25d708143c2aee909be75f3317abed0428 authored about 10 years ago

Moved limited error-handling inside cache() and conform()

3f9bfe5b1a9fd8bfbd9c20a565f0066bb2b60ccb authored about 10 years ago

Moved chatty thread business into jobs._wait_for_threads()

ae226442d2b2b929a406ce550480d40895471c39 authored about 10 years ago

Logging start and end of complete process

ef674114f657996b363102252c8fb0383ae6413a authored about 10 years ago

Returning stderr/stdout from cache() and conform()

4fbc86216af792a6c24f8853b5408f6a1bf1c612 authored about 10 years ago

Switched to machine-specific tags for OA-cache and OA-conform repos

553bcf5d141153aa2fcfe511f9ad6b8839e0cec5 authored about 10 years ago

Carefully wrapped calls to conform() and cache() in try/except blocks

8720a8589faed9e0fa0ec251f4115d0ce932bd4f authored about 10 years ago

Corrected more bugs in fresh system install

2f12f166b0509cbda54494acce3fb278489ce36d authored about 10 years ago

Ecosyste.ms: OpenCollective

github.com/openaddresses/batch-machine