Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/censusreporter/census-api

The home for the API that powers the Census Reporter project.
https://github.com/censusreporter/census-api

Support the new 2015 1yr release

2fbaaa1db07cb319f7b4f7b24357002206733b09 authored over 8 years ago by Ian Dees <[email protected]>
Forgot to commit these changes to the server templates

48d49dbac8b73e615cef4d3090090ced85506db6 authored over 8 years ago by Ian Dees <[email protected]>
Merge pull request #56 from cliftonmcintosh/patch-1

Correct spelling of "spreadsheet" in API.md

8bab5f713d16747d42c8c9fd175e84fcbdabc9ad authored over 8 years ago by Ian Dees <[email protected]>
Update API.md

Correct spelling of "spreadsheet"

a6f75ca039c565dcf8d0500d6643ce3fc84e833b authored over 8 years ago by Clifton McIntosh <[email protected]>
don't assume table codes are only 5 digits. Fixes censusreporter/censusreporter#195

5e4482ddeed133c232b74d320226a1ed745110da authored over 8 years ago by Joe Germuska <[email protected]>
add 252/254 summary levels

b0aca3efc26050e1349933b940bad285e7b300f5 authored over 8 years ago by Joe Germuska <[email protected]>
Change metro area to CBSA

70b6eaaaffc0e0bae1040ca98396b1152b193436 authored over 8 years ago by Tushar Chandra <[email protected]>
Add metro area as parent of metro divisions

011eb571c34d38f830ed653326cfd9f1760fa795 authored over 8 years ago by Tushar Chandra <[email protected]>
Merge branch 'master' into full-text-search

cc5ef6117308ab2eccb21341c4b2095a66cac6a6 authored over 8 years ago by Joe Germuska <[email protected]>
Merge pull request #52 from censusreporter/fix-sumlevel-250

Add sumlevels 252 and 254 to SUMLEV_NAMES

4f2d40eb47ec1363567cf5ac9a1b9c04573143ab authored over 8 years ago by Joe Germuska <[email protected]>
Merge pull request #49 from censusreporter/fix-error-codes

Change certain error codes from 400 to 404

21e95dc9ebc628f981681d7ab38378b2edd51ef6 authored over 8 years ago by Joe Germuska <[email protected]>
Add sumlevels 252 and 254 to SUMLEV_NAMES

Add sumlevels 252 and 254 to the list of valid sumlevels. This is
related to censusreporter/cens...

2e39135c9c4ce6037b6be57e68ed77b1e876dcba authored over 8 years ago by Tushar Chandra <[email protected]>
Change certain error codes from 400 to 404

Changed certain error codes from 400 to 404 to better reflect what
is actually happening (i.e., ...

614475212fcfc0466aa664d0c6ebae8fa6cf3a59 authored over 8 years ago by Tushar Chandra <[email protected]>
Fix logic for finding dead tables

47fada94a1d6a9836a381e2212f924389d8bf09a authored over 8 years ago by Tushar Chandra <[email protected]>
Lookup annotations for other table versions

This script now reads the annotations next to each table code (thanks
to the modifications to to...

7412ef72f539f51fa5231d8bb09478f199364803 authored over 8 years ago by Tushar Chandra <[email protected]>
Fix order of returned result

0693e8b68668c8f05181328b209625b21052bb3b authored over 8 years ago by Tushar Chandra <[email protected]>
Detect and parse annotations to table codes

Modify topic_scraper.py to detect annotations to table codes. This
searches the characters after...

f7ef676e72f182a5362f254a37711d62b5b81fdb authored over 8 years ago by Tushar Chandra <[email protected]>
Merge branch 'full-text-search'

9a33ab71c59d6bbc26b8ca262a815d32bb93cfcd authored over 8 years ago by Joe Germuska <[email protected]>
Merge branch 'full-text-search' of github.com:censusreporter/census-api into full-text-search

82f75590f2af4887bdac0b9138641ccdf1d5b6c8 authored over 8 years ago by Joe Germuska <[email protected]>
Add HTMLStripper class to remove HTML tags

The HTMLStripper class removes HTML tags from a page, leaving only
the contents of those tags (i...

cc887c5d9f610d3b7befea281d8aa8767b3b85d1 authored over 8 years ago by Tushar Chandra <[email protected]>
Initial commit

table_tester scrapes all topic pages and checks every table referenced
on those pages for existe...

671673f077fb72b3485753cdef975c8506eb20a5 authored over 8 years ago by Tushar Chandra <[email protected]>
Modify TopicPageParser table scraping

Table scraping now happens when the TopicPageParser is initialized,
using a regex search for tab...

f08029300b936b253a8a00b227370ba2ce90ed18 authored over 8 years ago by Tushar Chandra <[email protected]>
Update setup; add table ID and geoid search

Add a note that if the search_metadata table is rebuilt, the topic
scraper must be run again as ...

145e2103efb11ae27317d99b9deedf3032db4103 authored over 8 years ago by Tushar Chandra <[email protected]>
Fix whitespace

109d5ecb42402d4a43296c42dd975b534b61b683 authored over 8 years ago by Tushar Chandra <[email protected]>
Group score computation logic

Group score computation logic into one block, rather than having
three separate blocks for each ...

f9d783b1eb675318ccae01b79f0aa2e0e52a5a73 authored over 8 years ago by Tushar Chandra <[email protected]>
Remove unnecessary apostrophe replacement

e00c95fb5d1411b51b44a58f675e8172ee93bd6d authored over 8 years ago by Tushar Chandra <[email protected]>
Use only geoid in profile URL

The profile URLs returned only use the geoid now, rather than the
geoid and display name earlier...

e50636670b99472e2f26c38aac2de3d48c247b01 authored over 8 years ago by Tushar Chandra <[email protected]>
Tweak control flow

0b441e3776d5465fbe2f90be6a911338b0bd4bf5 authored over 8 years ago by Tushar Chandra <[email protected]>
Use SQLAlchemy query formatting

44f721970467cc29d28d0bc3043b0d12d4504ab6 authored over 8 years ago by Tushar Chandra <[email protected]>
Add code to compute topic score

a3c42fbb9b182ea6678322a42dbf8985b2f4f25c authored over 8 years ago by Tushar Chandra <[email protected]>
Refactor score computation

Refactor compute_profile_score() and compute_table_score() into
one function, compute_score(). T...

5ebe46d2db0bc2721020cd2b267fe862a9d4888b authored over 8 years ago by Tushar Chandra <[email protected]>
Refactor search functions

Refactor table_search(), profile_search(), and topic_search() into
one function to make more sen...

92a2c5e987af3ee658b218ba53066a18d4265c3a authored over 8 years ago by Tushar Chandra <[email protected]>
Add topic search to API

The full text search endpoint now returns topics as well as profiles
and tables. There is work t...

f56727865b04a19e05ef35d7d3018d4a79b271c8 authored over 8 years ago by Tushar Chandra <[email protected]>
Update search_metadata with URL

topic_scraper.py now adds the URL to search_metadata. It was necessary
to include this in the da...

99436991c5922fa3f30df8e4ca2f143866be0cd1 authored over 8 years ago by Tushar Chandra <[email protected]>
Update with instructions for topic pages

c3fcedc62d0035b1b45d8f264c48c9b27a67d837 authored over 8 years ago by Tushar Chandra <[email protected]>
Remove broken query script, update docs

Removed query_script.py, as there have been many changes to the API
and to the database that wer...

09d8a1ad3c7cae125df9e442c150f36c06281160 authored over 8 years ago by Tushar Chandra <[email protected]>
Add scraper for glossary page

The glossary page on Census Reporter is now scraped as well, and
it is indexed as a topic page a...

799049ca0210730569f33d50bacc02ff2d507f30 authored over 8 years ago by Tushar Chandra <[email protected]>
Update database with topics entries

The functions created before now update search_metadata with topic
page entries. Data for the to...

5cc3e2b5faf680d8dd749c504e06e88c79d051ac authored over 8 years ago by Tushar Chandra <[email protected]>
Add functions to update search_metadata

Add functions to remove old topics from search_metadata (it should
be cleaned every time the scr...

69cd3b169a92b74209457c33483f3a11487c0fb8 authored over 8 years ago by Tushar Chandra <[email protected]>
Fix whitespace

24efe3b9d0228d9a8ad762dac472239855bac4db authored over 8 years ago by Tushar Chandra <[email protected]>
Clean code and fix small bugs

Minor bugfixes; a couple of words were getting concatenated together,
which I took care of. The ...

389323463530f8be06cabe61cdf49966af2b5307 authored over 8 years ago by Tushar Chandra <[email protected]>
Add scraper for individual topic pages

TopicPageParser now scrapes individual topics pages, and it gets both
the relevant text on the p...

1e173006bab6ceb836d22769ffd851d3d465d13d authored over 8 years ago by Tushar Chandra <[email protected]>
Initial commit

Topic scraper builds a list of topic pages and their URLs. The next
step is to scrape each page.

e526b3de35dd4c36c1077d2771406ff4d1a64659 authored over 8 years ago by Tushar Chandra <[email protected]>
Allow app to be run without S3 connection.

For local testing purposes, the app.S3 connection initialization block
is now inside a try excep...

01c94738a9eacaee34d4b2f209eb61f614c80467 authored over 8 years ago by Joon Park <[email protected]>
Merge branch 'full-text-search' of https://github.com/censusreporter/census-api into full-text-search

884c7cc28108c40e711c55ac66cb15be29854586 authored over 8 years ago by Tushar Chandra <[email protected]>
Handle queries with apostrophes.

Replacing apostrophes with spaces in the query string, which is
what postgres did when building ...

65b739248005e63b072db60ca20bcf9a2fa2aaf4 authored over 8 years ago by Tushar Chandra <[email protected]>
Merge branch 'full-text-search' of https://github.com/censusreporter/census-api into full-text-search

2c855c57011eee935f2c8aeed11f4139d9150189 authored over 8 years ago by Joon Park <[email protected]>
Add OS X instructions and other misc. visual tweaks to abbreviations.md

8f7a912f86662c33e32fdc47b719098e4e3b71da authored over 8 years ago by Joon Park <[email protected]>
Merge branch 'full-text-search' of github.com:censusreporter/census-api into full-text-search

e6a68c65506e1658a90a5340e77c2b59c6530c8b authored over 8 years ago by Joe Germuska <[email protected]>
Add abbreviation instructions to setup

d796bab637ce1efa80281e2e9c3714910f5e0d6c authored over 8 years ago by Tushar Chandra <[email protected]>
Fix metadata script syntax error.

Sumlevel exclusion vetos were written as numbers when they should have
been strings.

bf1ad6b054244ccca09d9b34c6c485188af75de7 authored over 8 years ago by Joon Park <[email protected]>
Remove sumlevels without maps.

5538438202cba19705dfa67bf84a512da793ed16 authored over 8 years ago by Tushar Chandra <[email protected]>
Update formatting in setup

a72ffbb4524fa6f606d43389856f931e62aef4d5 authored over 8 years ago by Tushar Chandra <[email protected]>
Add basic setup insructions

1de8766c8bbe14553683852d0c512955630ffa2d authored over 8 years ago by Tushar Chandra <[email protected]>
Remove obsolete functions

Remove full text table search and profile search functions that are
entirely obsolete after the ...

c9a124b7239b626742a03cec79654288f963586b authored over 8 years ago by Tushar Chandra <[email protected]>
Change subtable format to list from string

d76ef0895cd1f639616f8827fa5b2871e14e3054 authored over 8 years ago by Tushar Chandra <[email protected]>
Change 'topics' field formatting in API.

The 'topics' field is now a list rather than a string, to stay
faithful to the original API format.

94e6dc02411fd81909b48286e5ddcb9553bb263b authored over 8 years ago by Tushar Chandra <[email protected]>
Make undefined sumlevel names empty strings.

b30da33b49d3680715bbc60c8f30cdeed0e066e6 authored over 8 years ago by Tushar Chandra <[email protected]>
Modify API to return all results at once.

Removed pagination functionality added in the last commit. Removed
the PositiveInteger validatio...

868a2dc975b964fa81842912ffb60aadddc606cb authored over 8 years ago by Tushar Chandra <[email protected]>
Add page parameter to API to allow for more results.

Added another qwarg, "page", to api.py, which allows for one to obtain
more results. Different v...

2fee3987457ca08babda97aba33753db54749418 authored over 8 years ago by Tushar Chandra <[email protected]>
Add option to search for profiles or tables only

Added functionality to search for only profiles or only tables (or,
by default, both). This was ...

f7d57fd65d649d634e234efbb5aec055fc673320 authored over 8 years ago by Tushar Chandra <[email protected]>
Add support for abbreviations.

This support is implemented on the database end. Refer to
abbreviations.md for instructions on h...

31d05c71d578bafa63dce7e881c556021b9552d7 authored over 8 years ago by Tushar Chandra <[email protected]>
Add logic to choose representative table of tabulation.

When there are multiple tables represented by a tabulation, we
need to choose one to link to tha...

577e8cf29eeeab0d777501ae7ab1b821783eb47a authored over 8 years ago by Tushar Chandra <[email protected]>
Modify API and query scripts to search tabulations.

The Python scripts now search tabulations, reflecting the changes to
the metadata search table. ...

5f24b8556ca8555cf039d1a507c2fda3123aa234 authored over 8 years ago by Tushar Chandra <[email protected]>
Modify metadata table to index tabulations, not tables

The search_metadata table will now include tabulations, not tables.
This is done in an effort to...

7a965c154f28c8a399080809d3861236f719c2b5 authored over 8 years ago by Tushar Chandra <[email protected]>
Standardize print formatting across scripts.

0b01cf5fd222d6afb135d390b5aec5f00e95c553 authored over 8 years ago by Tushar Chandra <[email protected]>
Improve profile search behavior.

Eliminated removal of stopwords on profiles, so that names such as
"Columbus, IN" did not get pa...

c4b70330a36c2aa46eadad31848dc14ce66fe844 authored over 8 years ago by Tushar Chandra <[email protected]>
Add profile and table URLs to API.

355059735cb1c19d8227edac2006ba74137b95d8 authored over 8 years ago by Tushar Chandra <[email protected]>
Add prefix matching to combined search.

b36dd98d3958c7e004aa25cf13b23c903942cebd authored over 8 years ago by Tushar Chandra <[email protected]>
Get rid of trailing slash for full text search route

c983fb6c5962612199f4e9f1b8d4bfba2eda4a40 authored over 8 years ago by Joon Park <[email protected]>
Add sumlevel name to API response.

88de0d3d90d98ba3940f6dc743dd127c93f28511 authored over 8 years ago by Tushar Chandra <[email protected]>
Refactor row processing from SQL queries.

27e5a89b5268886e555baafea10cc6b57a6e39de authored over 8 years ago by Tushar Chandra <[email protected]>
Merge branch 'full-text-search' of https://github.com/censusreporter/census-api into full-text-search

ae9f25e2aa43c5a7bca1238c3b9c6b5130663cf6 authored over 8 years ago by Joon Park <[email protected]>
Add drop table if statement to sql script

23193cc81592fd0967b41c61cd7c134944870b6c authored over 8 years ago by Joon Park <[email protected]>
Create endpoint for full text search.

api.py now includes a function to search both tables and profiles,
implementing the functionalit...

7db524c3f51054b207b62896386b3046ed6cf9b5 authored over 8 years ago by Tushar Chandra <[email protected]>
Fix null handling on populations.

e0c2c1a24debf1334ed70483f98f53385d8727db authored over 8 years ago by Tushar Chandra <[email protected]>
Add normalization for table query.

In addition, modified query_script.py to take into account the
new data shapes from table_ and p...

ce2206f2e6e511f22aa6329f9435966c70ea1841 authored over 8 years ago by Tushar Chandra <[email protected]>
Remove unnecessary JOIN from SQL query.

9554607e4440bc4e0268c20800e0a34074e0da06 authored over 8 years ago by Tushar Chandra <[email protected]>
Add population and priority columns to metadata table.

In addition, modified the table querying script to target the combined
metadata table, simplifyi...

c4947a290245c345c5fbb2bf9ebd721a7f3cb73f authored over 8 years ago by Tushar Chandra <[email protected]>
Adjust normalization of table search results.

8ab33aa4ac0971a7bd49781fadf052591ef4e651 authored over 8 years ago by Tushar Chandra <[email protected]>
Refactor querying scripts and combine search results.

The first change is that profile_query_script.py and table_query_script.py
were refactored to ha...

0dca193747a4c76fb5b29d9d21cb2be5d7e11322 authored over 8 years ago by Tushar Chandra <[email protected]>
Add command line functionality to scoring script.

Also begin commenting score.py script.

19d4f73d6b97b39453645d49ea02e17a9def8323 authored over 8 years ago by Joon Park <[email protected]>
Add script to test profile page scoring.

score.py takes the query 'ithaca' and shows the expected search results
alongside their custom c...

15cd6775313652ac03fa78494b1f485e38557d7f authored over 8 years ago by Joon Park <[email protected]>
Fix syntax errors in metadata table creation script.

Also added minor clarification in full-text guide file and refactored a
few lines in api.py

bab44af7a819f77d03d4e90bd7674fa1e6da0390 authored over 8 years ago by Joon Park <[email protected]>
Create combined metadata table and point search queries to it.

The metadata_profile_script and metadata_table_script files were
updated with additional documen...

9c4f564f5ce1e3bbc9b3c8c3df4e090fafbb35c4 authored over 8 years ago by Tushar Chandra <[email protected]>
Merge branch 'master' of https://github.com/censusreporter/census-api

88fb4d4c6800a6adaeb307dbfa394c03fea8d864 authored over 8 years ago by Joon Park <[email protected]>
Add additional instruction to install help file.

Add the line "add2virtualenv ." to the install instructions. Otherwise
locally running environme...

5a568cf8ecd41f5d77f1ebb0df7abbdc48f8a273 authored over 8 years ago by Joon Park <[email protected]>
Allow app to be run without S3 connection.

For local testing purposes, the app.S3 connection initialization block
is now inside a try excep...

c2fef1c03ea6b60e907222fc517d635e4c2d1606 authored over 8 years ago by Joon Park <[email protected]>
Write initial version of full-text search route functions.

The route is 2.1/geo/search. The query output still needs to be verified
to match the previous v...

98986e78c84b1c65d8ce131eb555265983155dae authored over 8 years ago by Joon Park <[email protected]>
some stuff to knit all of the sitemap stuff together

c13df9080fba50aa821a98a4e15073ff324ce908 authored over 8 years ago by Joe Germuska <[email protected]>
a few tweaks for general production use. Some inappropriately hardcoded stuff, but this won't get used that often, so deal with it.

8ea081f26ce03f14d7b22fa2a89f88e332e715af authored over 8 years ago by Joe Germuska <[email protected]>
Add geo full-text search route and metadata table create script.

The route is under /2.1/geo/search and the create script may be
incomplete in the information it...

61d586fa9b2fa22633fef80ced37a73488f0d1c5 authored over 8 years ago by Joon Park <[email protected]>
Add index on document column.

387810513e607f0eaf694d6eff5b029e4473d908 authored over 8 years ago by Tushar Chandra <[email protected]>
Add change owner instruction to table creation script.

Added a "alter table" statement to change the owner of
table_search_metadata to census.

44f29413dfa5d6565c8c53876aaeb2268bc5250d authored over 8 years ago by Tushar Chandra <[email protected]>
Finish full-text search route function for table data.

/2.1/table/search now functions as a basic route for the full-text
search tabulation data.

776f01afdd6255ca87b54d7b728d6cbe31811332 authored over 8 years ago by Joon Park <[email protected]>
Merge branch 'full-text-search' of https://github.com/censusreporter/census-api into full-text-search

4c49bc47dc6ae04a34dfb841f97ea7569fadc264 authored over 8 years ago by Joon Park <[email protected]>
Add script for creation of table search metadata table.

Wrote a script that creates a table, table_search_metadata,
which stores information about the a...

ee9f1b062564c4e01f1da17639e7876c45449104 authored over 8 years ago by Tushar Chandra <[email protected]>
Merge branch 'master' into full-text-search

58fbfafdc5bf94c118ff76c288df6fa8ee7b8bef authored over 8 years ago by Joon Park <[email protected]>
Support running without S3 credentials.

The INSTALL.md was also edited to better reflect an OS X pylibmc
install.

898a0919618c3cf4f29a66392af7e98a0691246b authored over 8 years ago by Joon Park <[email protected]>
Merge branch 'full-text-search' of https://github.com/censusreporter/census-api into full-text-search

658e5df060b80e0a58fe5fe4370250a52a874058 authored over 8 years ago by Joon Park <[email protected]>