github.com/OpenDataScotland/the_od_bods commits

Fix incorrect date parsing for Aberdeenshire data

If a number or year appeared in the dataset name then it would be
considered as an incomplete da...

9067cdbf91d9b70535b7aaab809f90aad1d63e58 authored over 2 years ago by Kyle Mckay <[email protected]>

Merge branch 'OpenDataScotland:main' into BR-Adding_tests_for_api_scrapers

bdf93abbbc0d13e4415a16b904f2b07905fb8045 authored over 2 years ago by Gavin Barnett <[email protected]>

replaced 5x more deprecated pd.DataFrame.append()

cd8ade7b846930e86f4cc72daf1762cb3f586d50 authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #117 from shieldo/replace_dataframe_append_with_concat

replace deprecated pd.DataFrame.append() method calls

3abbf05f7bc010bd48f9c7131f5097ba6df4ca4f authored over 2 years ago by Karen Jewell <[email protected]>

Merge branch 'main' into BR-Adding_tests_for_api_scrapers

69cf7888f2112236c579d0320e9a25bd5ff067ab authored over 2 years ago by Gavin Barnett <[email protected]>

Merge branch 'main' into replace_dataframe_append_with_concat

5fac4a2d3ea05d85a674e4fc2cd187e9db42c29a authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #119 from shieldo/use_black

format python files using black

c6cc1e7a8eda1d5a032c782f94609bf211f5f15c authored over 2 years ago by Karen Jewell <[email protected]>

Adding more detail to how_to_test.py

Testing:
None - no code changes

2366fbc2b815867d589b84ffdca82305de0e91ef authored over 2 years ago by Gavin Barnett <[email protected]>

renaming test folder to tests

"tests" is the more conventional name for a pytest folder. so just following convention.

Testin...

e5edc12ec4816cb31cc19c6d23417701e28782f1 authored over 2 years ago by Gavin Barnett <[email protected]>

Merge branch 'OpenDataScotland:main' into main

e21c1fa8558dcf6ba21282dfe29011ff4d6ceb51 authored over 2 years ago by Gavin Barnett <[email protected]>

Fix - bug if test/output folder didn't exist

Now if test/output/ doesn't exits it will be added.

Testing:
ran pytest both with and without "...

91635e8c187cb96582e6a505bb051f05c36c9a1f authored over 2 years ago by Gavin Barnett <[email protected]>

Added arcgis_test.py to test arcgis scraper

minor changes to arcgis to allow for testing.

Testing:
ran pytest 100% pass
24 passed in 1.71s

d2492dafd8754b0c019de5a872851c518e4e59dd authored over 2 years ago by Gavin Barnett <[email protected]>

moving helper test function to conftest.py

in order to allow arrest errors to propagate correctly they must be inside conftest.py (or impor...

7d808cff3ebb0d352af8e0f3aba18abf7bd104cc authored over 2 years ago by Gavin Barnett <[email protected]>

Refactoring - common csv validator code moved to separate file

dcat_test & usmart_test both use the same csv validator test - so it makes sense to group this c...

4f74274b2fa568edaa2c50ce1cad99c8ceaccaa2 authored over 2 years ago by Gavin Barnett <[email protected]>

Fix - usmart.py replaces url spaces with "%20"

To pass url validator spaces should be changed for %20

Testing:
ran pytest 100% pass
23 passed ...

364238d93c89b27d969c068be5d56acda9d31054 authored over 2 years ago by Gavin Barnett <[email protected]>

Added umsart test - bug detected with usmart.py

Test added for umsmart.py (similar to dcat).

usmart modified slightly to allow for testing.
Thi...

16a27e2bc595ca3d3f4067f1614d76ab324b0c3e authored over 2 years ago by Gavin Barnett <[email protected]>

Fix - removing unused import

import incorrectly added by my editor. removed.

9676eb3bcb5bdf2fe15ffb1b178108e320a38b8e authored over 2 years ago by Gavin Barnett <[email protected]>

Tiidying gitignore for pytest output

ignoring the whole mockdata/output folder.

Testing:
Ran pytest - ensured files generate in corr...

31a4438214e20fc74e25d996d33813159dc6e48a authored over 2 years ago by Gavin Barnett <[email protected]>

Added test for exact csv match

this test ensures the csv output is the same as expected - (read as the same output as when this...

50aaa2680458101efca86c52ddd072bfd42dcd50 authored over 2 years ago by Gavin Barnett <[email protected]>

Adding - Basic tests for dcat

This is a fairly simple set of tests for dcat.py. It doesn't not cover everything in detail but ...

249d1af5b76c66e075188e259e3a51c29d53cc97 authored over 2 years ago by Gavin Barnett <[email protected]>

Update auto_assign_to_project.yml

06957547e422231fcabfd7a8247013298e080a6e authored over 2 years ago by Jack Gilmore <[email protected]>

#112 auto-assign issues to project

c37b2665915fe6daf4e3ba67609fd85949c4ef16 authored over 2 years ago by Jack Gilmore <[email protected]>

Create auto_assign_to_project.yml

1638960e7acf560a83c9b6ecac1980a989b5c5a0 authored over 2 years ago by Jack Gilmore <[email protected]>

Improving write_csv test by removing csv file

previous code may have allowed write_csv to pass without writing a file if the file previously e...

5ac3446a4840ebca80f129e2f0d0e60eb258f793 authored over 2 years ago by Gavin Barnett <[email protected]>

Fix - extra new lines in CSVs on windows

On windows the previous code add "\r\r\n" when writing lines to the csv. specifying the newline ...

7678f7df7ddfb7aa448057df37cb13c9dda0057d authored over 2 years ago by Gavin Barnett <[email protected]>

Adding test for write_csv()

This is a fairly simple test but is highlighting a windows based newline error.
On windows the c...

009f26bb7a84b22cfc75f2825f39c5f5785fd692 authored over 2 years ago by Gavin Barnett <[email protected]>

replacing "sudo" with "mock" as a better naming convention

sudo was used incorrectly to name mock items. the word mock is better for this.

find/replace al...

816fff436b2cc05299a45396bf32e9f56c5fe769 authored over 2 years ago by Gavin Barnett <[email protected]>

Create processor_test.py

This tests the processor.py script. most functions tested, but not great coverage.
notably - wri...

8c789489a9879aa2f53ae3e9e882a1f3dd844aaf authored over 2 years ago by Gavin Barnett <[email protected]>

Removing sample test

Sample test will lead to confusion if left, removing

Testing:
None - literally just removed the...

fa2fa245a7f2b15b6b6747ac2dd99c991f041e09 authored over 2 years ago by Gavin Barnett <[email protected]>

Adding __init__.py to root of directory

__init__.py required to allow pytest to search for python files.

Testing:
None - nothing to tes...

a4ffa7e8b3311fed4a6a90e0b2e761d26c5b732b authored over 2 years ago by Gavin Barnett <[email protected]>

Fix init py file name

init.py files should have 2 underscores before and after.

Testing:
None - nothing to test

7f32f64e74c5d96a517f0858142108d1ea2f5aa2 authored over 2 years ago by Gavin Barnett <[email protected]>

Adding test folder for pytest scripts

pytest will allow this project to be tested.
This is an important step before refectoring.
Addin...

1963fd4d1f68d8d7c077a437dd09cfe2ed8b15de authored over 2 years ago by Gavin Barnett <[email protected]>

format python files using black

c62dccd3fe84392dd5e14d5e5c887477fd860e20 authored over 2 years ago by Douglas Greenshields <[email protected]>

replace deprecated pd.DataFrame.append() method calls

72461f12232cfe6b5022f120e1ce12d6a009ebf7 authored over 2 years ago by Douglas Greenshields <[email protected]>

Merge pull request #107 from nutcracker22/alternative_nls_scraper

Alternative NLS scraper

95a2923969de55fade429ed0ef2add239df3581c authored over 2 years ago by Karen Jewell <[email protected]>

Comment out file size

9807685b99ddb2a0dd0bed5b58234b7cba9386ed authored over 2 years ago by Jack Gilmore <[email protected]>

Temp fix for file request issues

2b84d43d30d315b33505dec3617d5071586d8d96 authored over 2 years ago by Jack Gilmore <[email protected]>

updated known_licences in tidy_licence function

d6da346ded2f97e6fb39e41748b133d96934abeb authored over 2 years ago by nutcracker22 <[email protected]>

disabled print statements used for debugging

710e7e1281c585f6db0fa3ca5b3afc172502b986 authored over 2 years ago by nutcracker22 <[email protected]>

Fix: implement @ormiret's comments, separate fetch_data_type and fetch_num_recs functions, make list_of_types a list containing only unique values

2bc1f05c6534640c123b6b1e9fdc40ed78a64257 authored over 2 years ago by nutcracker22 <[email protected]>

fixed file types

577f73595b4a6bc916af2b21128427d1b629cdbf authored over 2 years ago by nutcracker22 <[email protected]>

fixed licences

d103512dd024db1f3f8a9894c4a219d61734ecdc authored over 2 years ago by nutcracker22 <[email protected]>

error handling included, scraper runs without errors

591386a12904e11b646b1c24ab3250ced8a94e9d authored over 2 years ago by nutcracker22 <[email protected]>

num_recs and data_type working

eb353f73e290cf3a3ed7cd6034f182272246381d authored over 2 years ago by nutcracker22 <[email protected]>

final commit for the day, missing parts see comments in nls-scraper

76b549ab59ee2f2e86a8e02d3d4688c5dd9f048f authored over 2 years ago by nutcracker22 <[email protected]>

almost done with alternative approach for scraper

f5b78189bafaf38959b9d61dd6e09afa3a40d765 authored over 2 years ago by nutcracker22 <[email protected]>

Add some logging to NLS scraper

86f81c7c399879aa83c23a364e04bd7b1fc56ee7 authored over 2 years ago by Jack Gilmore <[email protected]>

Merge pull request #101 from Lux-Ferre/main

NLS Scraper

629c34c6bf1408693df772280258bcb63df3a550 authored over 2 years ago by Jack Gilmore <[email protected]>

Fix USMART issues?

6abe39a7f11d252086672f5616dec422521bbbf6 authored over 2 years ago by Jack Gilmore <[email protected]>

Merge branch 'main' of https://github.com/OpenDataScotland/the_od_bods

a9cd1f72f9c9c3a7646e5f124a0d659de203ef44 authored over 2 years ago by Jack Gilmore <[email protected]>

Fix USMART script issues

0707b5f86e454018a97a41a48f51d208200fe861 authored over 2 years ago by Jack Gilmore <[email protected]>

Merge branch 'OpenDataScotland:main' into main

c99395507a7655df80e4ec4a5a0453a7dfde7a43 authored over 2 years ago by Chris Milne <[email protected]>

Update README.md

5e419cb669a1aa6557c8e171c1187c6a372be3f4 authored over 2 years ago by Jack Gilmore <[email protected]>

Current output csv for NLS scraper

ec5ce3911c030c3aa1020a0bf3df43a4b4333a66 authored over 2 years ago by lux-ferre <[email protected]>

Web scraper for NLS as per issue #62

6a6b70ecdddbe50a40f001859afba1c798f39f7c authored over 2 years ago by lux-ferre <[email protected]>

Add sorting for categories to tidy up the changelog for syncs

ad3b27587372675bd0efece53567014c1dff818c authored over 2 years ago by Jack Gilmore <[email protected]>

Merge pull request #98 from OpenDataScotland/license_issue_arcgis

Issue#6 License info issue fix for ArcGIS API

3fec382577f8f4a72e7d35f37f01c672204c1afa authored over 2 years ago by Jack Gilmore <[email protected]>

Issue#6 License info issue fix for Arcgis data

276c014c2d3351f242bb479915db57baa10d16f6 authored over 2 years ago by heymanpreet <[email protected]>

update Aberdeenshire scraper to generate categories

e86f8cfed1a956d8f887bc58b0a12733b74654a8 authored over 2 years ago by watty62 <[email protected]>

Merge remote-tracking branch 'origin/main' into main

# Conflicts:
# data/merged_output.csv

07a25ca35a1c4fa65ba6a36c23ec92eba26fdfea authored over 2 years ago by watty62 <[email protected]>

combine food environment

26ee4b1dbb51551994840fb85d42e63a37675024 authored over 2 years ago by watty62 <[email protected]>

Put cron on the correct script

ccb4e97a4d43a01cfa46f1377810d7f968de349b authored over 2 years ago by Jack Gilmore <[email protected]>

Fix paths

dfdb1241af4fa80832b1d6c2252966adc1e01a75 authored over 2 years ago by Jack Gilmore <[email protected]>

Merge branch 'main' of https://github.com/OpenDataScotland/the_od_bods

7a4dfc8351ffc08ce5981cf52036f165f36f94d7 authored over 2 years ago by Karen Jewell <[email protected]>

added web scrapped data to merge_data.py

9b3a3538435b728dd3d5798f1043bd82f6f2dcbc authored over 2 years ago by Karen Jewell <[email protected]>

categories public lavvies

2fce375bb022c974c79063a194ef950c9b435957 authored over 2 years ago by watty62 <[email protected]>

Merge branch 'main' of https://github.com/OpenDataScotland/the_od_bods

f3cee62a8910890aa3430b2db7a419a3d2397c66 authored over 2 years ago by Karen Jewell <[email protected]>

removed renaming of organisations in export2jkan.py since covered in merged_data.py

7c96b2a00a005688b96453b02998d4d70ff832d6 authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #93 from OpenDataScotland/web-scrapers

Web scrapers

cb1959cb6af67d4a38870d4b6cac9413533dd243 authored over 2 years ago by Karen Jewell <[email protected]>

Merge remote-tracking branch 'origin/main' into main

# Conflicts:
# analytics/uncategorised_tags.ipynb

2dc2990b396654a508e37fa9a0fd537c3ae36e1f authored over 2 years ago by watty62 <[email protected]>

add more terms

e268389431a1860171b9bc38f4bf61ab1e65119b authored over 2 years ago by watty62 <[email protected]>

add more termns

8dcd8cf5121c25969ccb304c0628f03eb499ec06 authored over 2 years ago by watty62 <[email protected]>

added dir correction to uncategorised_tags + count uncat datasets

0a03ab431d0da4c7b373f2c4dc4edc2940883872 authored over 2 years ago by Karen Jewell <[email protected]>

sources fixed but analytics.ipynb is broken

714fafaccbb24ad82978ce7c2f058d191969670e authored over 2 years ago by Karen Jewell <[email protected]>

Merge branch 'main' of https://github.com/OpenDataScotland/the_od_bods

8c2d7ff1b0039f0fe2f83d927fc5acb54196f271 authored over 2 years ago by Karen Jewell <[email protected]>

update uncategorised_tags.ipynb to rerun merge_data.py

91c9e5a4ec1cc918f207c18ac24a5f02e120a790 authored over 2 years ago by Karen Jewell <[email protected]>

add health terms

6c0298979c329b58ed79e03c151db741fe33593a authored over 2 years ago by watty62 <[email protected]>

Remove broken link and schedule

9f23d889af20de32c5cc6db6ce35b662f1e1635b authored over 2 years ago by Jack Gilmore <[email protected]>

Test

9dc646797c59c07e39bf0aa1fe79880161899a23 authored over 2 years ago by Jack Gilmore <[email protected]>

fix syntax

f525bf68f0e5daff0c91b562a1c6c03aa1a559e8 authored over 2 years ago by Jack Gilmore <[email protected]>

Try env variables again

bd2ae7947a444b8f8db7147ab4fb2e1e5176ec4a authored over 2 years ago by Jack Gilmore <[email protected]>

Fix env variables

489314b81a7b33e384b4db9f893d06ea3e6f5a87 authored over 2 years ago by Jack Gilmore <[email protected]>

Import git py lib

7d8b6d8cd8de7c295ed5d04c8767cc0633499f13 authored over 2 years ago by Jack Gilmore <[email protected]>

alive.py workflow

6746d3d8ace154be75922551d93d94ae4f329330 authored over 2 years ago by Jack Gilmore <[email protected]>

Added uncategorised_tags analysis

f6d53f923cd720be3b82aeac6a54b0c06e460103 authored over 2 years ago by Karen Jewell <[email protected]>

updated data store

54c1ba324764242c34231ef87c9bd12dd1b74f45 authored over 2 years ago by Karen Jewell <[email protected]>

Update usmart.py

Type changed to USMART

7733c61f8e6930821aa25d88f4220a9a727a20a4 authored over 2 years ago by Andrew Sage <[email protected]>

Update sources.csv

CKAN sources added

7a88e812a667e796c9da28933d34ac551d843823 authored over 2 years ago by Andrew Sage <[email protected]>

Merge branch 'main' of https://github.com/OpenDataScotland/the_od_bods

735d4f0db5b52d84cda219c3430272f41551de6a authored over 2 years ago by Karen Jewell <[email protected]>

removed /data/archive since history can be retrieved from VC

31024c4cee380c3cae3da54abe2f4f46e0af0184 authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #94 from OpenDataScotland/90-convert-usmart-processor-into-python

#90 usmart processor rewritten to use Processor class.

88daca9bbcfd87a3154adb908e222ac6f6f9ba8c authored over 2 years ago by Andrew Sage <[email protected]>

change zoning to development

a1cd177b7317d138ac8cd6e827b575b60332958d authored over 2 years ago by watty62 <[email protected]>

#90 usmart processor rewritten to use Processor class.

83912f0d331ea07e9c656192a91f1cce6bf8ac1e authored over 2 years ago by Andrew Sage <[email protected]>

update business and economy

f716ff9b054be6971ffdf30674c14f7b815568f1 authored over 2 years ago by watty62 <[email protected]>

add sport and leisure

bc18a0134cb2720373f56c76ee877647633a01c1 authored over 2 years ago by watty62 <[email protected]>

added csv to data folder

371f043c65e74d8f0cb53911b4c1471a07835b44 authored over 2 years ago by Ludovico <[email protected]>

deleted csv within scrapers folder

cc1d4a9146d9095d7f4acd8ba01871968889b4e6 authored over 2 years ago by Ludovico <[email protected]>

changed output csv location

4556975919b01e796cae5a81e1aa1c3181217c95 authored over 2 years ago by Ludovico <[email protected]>

Merge branch 'web-scrapers' of https://github.com/OpenDataScotland/the_od_bods into web-scrapers

e6a851944e246d8641355fc2e1c1b138246edc3a authored over 2 years ago by Ludovico <[email protected]>

changed output location

bb05ec3b7ec8afb887771784f09ba4e86d90fdea authored over 2 years ago by Hariss Ali Gills <[email protected]>

Ecosyste.ms: OpenCollective

github.com/OpenDataScotland/the_od_bods