github.com/OpenDataScotland/the_od_bods commits

163 replace csv storage formats (#227)

* feat: Replace merged_output csvs with jsons

Replaced any creation and use of merged_output....

08441e4febbaf3aba899fd6cfd636e3e6b8cfd5d authored almost 2 years ago by Karen Jewell <[email protected]>

fix: changed and added treatment of org names to "Perth & Kinross" (#218)

part fixes issue #217

1ae64946df936bd65e782bc2136384278041fd40 authored almost 2 years ago by Karen Jewell <[email protected]>

Dataset Sync

9472f68cb56d2ac3f58128d780b268c41bd11d93 authored almost 2 years ago by github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Categories (#216)

* Adjust category keywords

* feat: added category healthchecker

* feat: add cleaning of OD...

eb0cfd94db24d577162ff3195d2423fbc51d628e authored almost 2 years ago by Karen Jewell <[email protected]>

Dataset Sync

2e6d3fa1a0e555f19b603c6459f7149806313b5c authored almost 2 years ago by github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Dataset Sync

671f8d46dbcb4532d007525aed4eaeec73e72aec authored almost 2 years ago by github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Dataset Sync

049472e686464ac157087290d13021cc2ec1bc5f authored almost 2 years ago by github-actions[bot] <github-actions[bot]@users.noreply.github.com>

SQA_scraper (issue #136) (#161)

* Revert "refactored according to discussion in PR #128"

This reverts commit 4b19338ac00ba01a...

8d3973295b68ba699a6b2f9fc8cbe76e24e28138 authored almost 2 years ago by Steffen <[email protected]>

Dataset Sync

02d9b66394380fd39d74d6c8439e6a0fef7bbbba authored almost 2 years ago by github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Fix keyword parsing error in dcat

0372ab7f04713e090b4095a12aa5f603c0a5a79b authored almost 2 years ago by Jack Gilmore <[email protected]>

Dataset Sync

38964451c0792ce1d8e26e277847ad3508b9ccdc authored almost 2 years ago by GitHub Actions <>

211 Extract category keywords from dataset title and description (#212)

* feat: Change to use dataset Title and Description to set dataset Category

Change to use com...

7cd93ac44a7a93134589c72023809625450a8cb6 authored about 2 years ago by Karen Jewell <[email protected]>

updated README to link to new docs.opendata.scot

c456df51b6b6cb9c69d6d7c1226553bd99c4ef9d authored about 2 years ago by Karen Jewell <[email protected]>

Add new issue auto-assigner

81cb287eda0b4b4d59fbb3419917c149d4e15ab8 authored about 2 years ago by Jack Gilmore <[email protected]>

Fix NLS scraper

98715c38d3873ffcd3854d6b658b743878e4ff8d authored about 2 years ago by Jack Gilmore <[email protected]>

fix bug #204 in Aberdeenshire scraper (#205)

* fix multiple or conditions

* remove comment

* took categorisation out of aberdeenshire_c...

c5ba74d27e402559d11b7db3e72ba0005205f1ae authored about 2 years ago by Steffen <[email protected]>

Add script to report unknown licence strings

432b752041ef928dbb2130763303fed7d720063f authored about 2 years ago by Robert McWilliam <[email protected]>

Add http versions of https links to known licenses and some cleanup of old code in export2jkan.py

ddd07823dc29e28bdc594b2a1f10a979cd2776ad authored about 2 years ago by Robert McWilliam <[email protected]>

Fix log path confusion

a573f8714f02ca0cd5b05afaef7b9a3e589cda16 authored about 2 years ago by Jack Gilmore <[email protected]>

Add name conversion for publisher "Na h-Eileanan an Iar" to "Comhairle nan Eilean Siar" #164 (#202)

* fix: change name to Comhairle nan Eilean Siar

* fix: council name

d776387781a89c150f808af29e9a1e1b5868c661 authored about 2 years ago by Tim <[email protected]>

reinstated Dundee City Council to sources.csv

Now there is error handling in pipeline for faulty URLs, thanks to issue #193 and PR #200, manua...

71d84304dbf237dbf203352aeae7902d9335339a authored about 2 years ago by Karen Jewell <[email protected]>

handle broken links in pipeline #193 (#200)

* restored merge_data.py to the current version of OpenDataScotland:main

* resolved previousl...

b807cc92f8303032ad61b98da3b5081fe8389fe4 authored about 2 years ago by Steffen <[email protected]>

Restore RDS now SSL is fixed

6399f0739fbba88704b66b074a76d3354fa088e2 authored about 2 years ago by Jack Gilmore <[email protected]>

fix: Remove Research Data Scotland as SSL expired Issue #195

b28c9e234ce77c22da3af6a1a7990a7214e6010d authored about 2 years ago by Karen Jewell <[email protected]>

55 enhance info for file links (#183)

* feat: Change export2jkan.py to display filename

Replaces the display of file extension in J...

6285a75339aa554c3ea34899fd237fb9989b6f55 authored about 2 years ago by Karen Jewell <[email protected]>

Undo commit stupidness

7bfd9ae87fac7006d17e3af773cec06a38cd2a24 authored about 2 years ago by Jack Gilmore <[email protected]>

Merge branch 'main' of https://github.com/OpenDataScotland/the_od_bods

6b660798f7aecf599f61f9b805b6fecfc99698f0 authored about 2 years ago by Jack Gilmore <[email protected]>

Change all licence links to https v2

b6bdb93ffda423b86ecb546d191d04052bae35a1 authored about 2 years ago by Jack Gilmore <[email protected]>

Change all licence links to https

74095bfd32eb23441865cc87b6afdbe29d84cdef authored about 2 years ago by Jack Gilmore <[email protected]>

Merge pull request #196 from OpenDataScotland/155-improve-dependency-management-and-documentation

docs: Add requirements.txt

52cba34993daad0573701fc5cdff408e3809fbda authored about 2 years ago by Jack Gilmore <[email protected]>

Merge pull request #199 from OpenDataScotland/add-main-sh

Added main.sh

885d5eab9353b2f29bb0d9c7f108622592e28146 authored about 2 years ago by Jack Gilmore <[email protected]>

feat: added shell script for entire pipeline

3e3eb6da322f93f9bc58b2731374291155e50072 authored about 2 years ago by Karen Jewell <[email protected]>

National Library Scotland Multiple Data Downloads #130 (#134)

* tidied licences

* refactoring of nls-scraper.py

* tidied licences

* refactoring of nl...

b03cf0cf3366d9377db06e04e9238fef724d5ceb authored about 2 years ago by Steffen <[email protected]>

update tidy_licence function (#154)

* Revert "refactored according to discussion in PR #128"

This reverts commit 4b19338ac00ba01a...

6b7a4971e6d23ed8e8c5e24eceb8e0d4d9023c8c authored about 2 years ago by Steffen <[email protected]>

docs: Add requirements.txt

562a172368d49a7cdcd80d22708406157308a016 authored about 2 years ago by Karen Jewell <[email protected]>

Remove Dundee CC as website is down

0cbb806514f507e228b00b59135df208d842a3b7 authored about 2 years ago by Jack Gilmore <[email protected]>

Merge pull request #188 from OpenDataScotland/187_scottish_forestry

Add Scottish Forestry name mapping

8b7b993fb088b54fe4cc11301cffc69e2d7f610c authored about 2 years ago by Karen Jewell <[email protected]>

Add Scottish Forestry name mapping

fcfc5109f2e3c8c99b2fbe7030d73176143d4867 authored about 2 years ago by Jack Gilmore <[email protected]>

Mirror issue templates from .github repo

I thought these worked on a hierarchical system but apparently not

ea3020723bf805c2928230036ddd6cd15b3ef82b authored about 2 years ago by Jack Gilmore <[email protected]>

Move issue template to correct folder 🤦‍♂️

cf76179cb4f177a2f8cfeeda8b4f65b41ce7b6f0 authored about 2 years ago by Jack Gilmore <[email protected]>

Add suggested changes to issue template

7684fa61f03af749429971159f119bd14099258d authored about 2 years ago by Jack Gilmore <[email protected]>

Merge pull request #185 from OpenDataScotland/103_new_data_source_issue_template

Add issue template for suggesting a new data source

29b36bad895d8a2f1b52b0b6087cc779b7ff11f7 authored about 2 years ago by Karen Jewell <[email protected]>

fix: inconsistent datetime objects for DateUpdated and DateCreated in merge_data.py

some datetime objects were coming in with timezone information which caused errors in appending ...

e1f72000a0aa8482c23e9abb3c092200b3959f66 authored about 2 years ago by Karen Jewell <[email protected]>

Merge pull request #160 from OpenDataScotland/fix_ckan_issues

CKAN and statistics.gov.scot bug fixes

833e2968f290903ec4d13a6e563776754dda8eda authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #186 from marccodess/180-add-scottish-forestry-as-source

#180 added scottish forestry as a data source

81bb94006d2cc9ebca006ddc03368cbfa7b2de57 authored over 2 years ago by Karen Jewell <[email protected]>

added scottish forestry as a data source

3cd710f35f6e46b72dd14a9ed5fa08c46fafe2b7 authored over 2 years ago by Marc Matterson <[email protected]>

Owner and description patches

Patch PHS owner to not use the CKAN specified owners. Patch description to escape unicode for PH...

70dbfff6bfcf77cb486254ae7f134012b927056b authored over 2 years ago by Jack Gilmore <[email protected]>

Add issue template for suggesting a new data source

8c6320eaf1b87053c9981a2705a05869f798bae6 authored over 2 years ago by Jack Gilmore <[email protected]>

Issue 105: Tidy filetypes for datasets (#131)

* updated tidy_data_type function in merge_data.py

* remove tidy_data_types function from nls...

7211b815fe818ba237360e52ed21b1727f135f94 authored over 2 years ago by Steffen <[email protected]>

Moved contributing file to be a global org file

See OpenDataScotland/.github@d90e75de82cea0d140c71ade8e01b61d2d221d5b

bbdf9504d3ea7c82d13475eadbc2eac86aba9d86 authored over 2 years ago by Jack Gilmore <[email protected]>

Fix dcat date issues

b6255db0445b62d12774292f9388a5303e3da7e5 authored over 2 years ago by Jack Gilmore <[email protected]>

Fix some date parsing bugs during merge and export process

d8b60c9a4233ac351a94e629983bf896a99918f3 authored over 2 years ago by Jack Gilmore <[email protected]>

Add fallback values for owner in statistics.gov.scot

13ad464f332c8e3c5e0258c503735f62bc5d42f3 authored over 2 years ago by Jack Gilmore <[email protected]>

Fix missing data for files and description encoding for CKAN

e0f596535ad6a15b08c9d38b651e3c7dbb9c167d authored over 2 years ago by Jack Gilmore <[email protected]>

Merge pull request #158 from gavbarnett/BR-fixing-tests-post-CTC

BR fixing tests post ctc (issue #151)

Looks great to me and runs fine on my local.

b852e2557a68cbbeb2c448b551e05d0812a5775d authored over 2 years ago by Karen Jewell <[email protected]>

Updating mock_data

Updating mock_data due to csv heading ordering being changed in processor.py, thus making old mo...

e70f6064ded51f5a1f449ab6ca75c8c02e6adc78 authored over 2 years ago by Gavin Barnett <[email protected]>

FIX generate_mock_data.py to ignore new datasources types

This is temporary fix until tests are added for new sources/scrapers

Testing:
locally generate ...

7441ae6cfeec487acc16d2dc71559366dba87f24 authored over 2 years ago by Gavin Barnett <[email protected]>

Adding filename check to contest.py

New "FileName" header needs correctly handled in contest.py csv_checker().

Testing:
Tests fail ...

45002eac89a1c909587815de9d48b2ba756ce8b4 authored over 2 years ago by Gavin Barnett <[email protected]>

FIX BUG dcat array numbering

after processor.py titles got updated to add a new column dact.py had an off-by-one error.

This...

7683ec1c8b6b903cebcb38776467701c5aeecf4b authored over 2 years ago by Gavin Barnett <[email protected]>

Merge pull request #157 from OpenDataScotland/KarenJewell/issue156

Changed CKAN PageURL from /package to /dataset

86b7c28d1205d7622ace79da2a760a5637e17309 authored over 2 years ago by Karen Jewell <[email protected]>

Changed CKAN PageURL from /package to /dataset. Added handling for missing trailing forward-slashes in future. Also removed redundant redacted code.

95be86afa985602ce22164ee807e93231572beba authored over 2 years ago by Karen Jewell <[email protected]>

Update sources.csv

d5e8644b539f0523d0f4c5a8b0014b15f1c50398 authored over 2 years ago by Jack Gilmore <[email protected]>

Issue 133 (#153)

Looks good to me - thanks @nutcracker22!

Squashed commits:
* Revert "refactored according to...

f8e79e5b8a9d2517ab78e087bca4c6ade80a9642 authored over 2 years ago by Steffen <[email protected]>

Add contributor guidelines

c1a651c24f2fecaf691da4184a3bda02dbb931d0 authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #147 from OpenDataScotland/license_issue_arcgis

Updated Source and License info ScotGov Sparkql datasets

64e4a5a4e022db2104fc958211e72c30fafb95b5 authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #150 from OpenDataScotland/144-fix-dataset-owners-in-multi-org-portals-ckan

takes owner if provided otherwise takes portal owner

65da459c065434589f557cf85b8197d6dfd44236 authored over 2 years ago by Karen Jewell <[email protected]>

Merge branch 'main' into 144-fix-dataset-owners-in-multi-org-portals-ckan

d877dfdc10e5ba263d46d4a2b4ba142c77ea174c authored over 2 years ago by Karen Jewell <[email protected]>

takes owner if provided otherwise takes portal owner

ec0f4b0febdd53741e14c95cb562088ca2f61fb7 authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #149 from OpenDataScotland:148-add-filename-field-to-processorpy

Added filename placeholder - TESTS WILL FAIL

6717d6bd321db4a64e1e7799c032d50dba0fcc0e authored over 2 years ago by Karen Jewell <[email protected]>

Added filename placeholder

666f514c49044f2bad1b202a4b3da751f154db0f authored over 2 years ago by Karen Jewell <[email protected]>

Updated Source and License info ScotGov Sparkql datasets

34f705ee78c6a1af11060ce5468992ff5126841c authored over 2 years ago by heymanpreet <[email protected]>

Merge pull request #143 from OpenDataScotland:97-add-spatial-hub-as-a-data-source

added spatial hub as source

f85a7f48973db943a993bc5b4d3f73b4bc626d60 authored over 2 years ago by Karen Jewell <[email protected]>

added spatial hub link to ckan sources list

e56bd987ba41f58b4b8146dc462f0dcb230a6d9c authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #142 from OpenDataScotland/license_issue_arcgis

Removing print statements and test csv file in Sparkql Statistics script

24a7252fb9ee5d24a515c7a6bb25528c90e104d2 authored over 2 years ago by ormiret <[email protected]>

Merge pull request #141 from OpenDataScotland/132-add-research-data-scotland-as-a-source

added Research Data Scotland as source

56316a0f26417d1059e7f76a9c9dedd85bc99cc3 authored over 2 years ago by Karen Jewell <[email protected]>

Merge branch 'main' into 132-add-research-data-scotland-as-a-source

302fc0f087fe776c7650e132148d37713b957f4c authored over 2 years ago by Karen Jewell <[email protected]>

Removing print and test csv file

369a8e05a50fc1aec9672ebc2b9d892386ef75db authored over 2 years ago by heymanpreet <[email protected]>

added RDS link to CKAN sources list

09aab7125845d7e0cd8381d4e513a613d4d7cf8b authored over 2 years ago by Karen Jewell <[email protected]>

Merge pull request #140 from OpenDataScotland/license_issue_arcgis

Sparkql dataset python implementation

b462a300a23e439f5ec9c0c90fa103eebbbcc307 authored over 2 years ago by ormiret <[email protected]>

Fix line terminator issue

11b16dfd484e8eb0b7b27e5cf12e108ecc5a1cc1 authored over 2 years ago by Jack Gilmore <[email protected]>

CKAN fixes

f28b7d8398d695921fd963e34198c14dc2a3de30 authored over 2 years ago by Jack Gilmore <[email protected]>

Sparkql dataset python implementation

3e2a4a8072a80d625b92875458d8a23ee6bd004c authored over 2 years ago by heymanpreet <[email protected]>

CKAN stubs

6b73e0a4112aaf54ffac3fbeb7809077ead69677 authored over 2 years ago by Jack Gilmore <[email protected]>

CKAN updates

b6e63c1f05f52a3e635ad964cf987c4f1ebec8f4 authored over 2 years ago by Jack Gilmore <[email protected]>

Merge branch 'main' of https://github.com/OpenDataScotland/the_od_bods

d3dd1deeb39e997a4e9f014195fadfc5322306bb authored over 2 years ago by Jack Gilmore <[email protected]>

CKAN first draft

fb5bf9cb71b5ae181c8d00de7e579d82cb814929 authored over 2 years ago by Jack Gilmore <[email protected]>

Add dateutil dependency for tests

e8386d022794322b97a664d4078d6fbb466d5ff8 authored over 2 years ago by Robert McWilliam <[email protected]>

Fix typo

214a0659f300e72a931412e723ff322b432e9d1c authored over 2 years ago by Robert McWilliam <[email protected]>

Switch to running tests on push.

4447b26171c923c068690a2b71746bb4ea110174 authored over 2 years ago by Robert McWilliam <[email protected]>

Add pytest github action

7a3eba195cc3a287c239513984362f0f78f6dc4b authored over 2 years ago by Robert McWilliam <[email protected]>

Merge pull request #127 from gavbarnett/BR-Adding_tests_for_api_scrapers

adding tests for api scrapers

5d485b9dbd18551dac4d31800b76d2dcadac8583 authored over 2 years ago by ormiret <[email protected]>

mock data for arcgis discards next url

This avoid link list style urls which were messing up the mock data and pytest.

genererate_new_...

40878ade920b94f8c1141848b998c3c00686784a authored over 2 years ago by Gavin Barnett <[email protected]>

Updating api scraper test to use all datasets

Testing:
32 passed in 11.24s

ca99d6a06cda462a581eba5e91366c4d15562662 authored over 2 years ago by Gavin Barnett <[email protected]>

Added Ability to easily generate new test data

generate_new_mock_data.py will get all the data it can from sources.csv and generate the json an...

9cfbc5a6877b2358996719b59d2e94d6f6341baa authored over 2 years ago by Gavin Barnett <[email protected]>

enforcing line endings for csvs - test data not updated

Due to differences between unix & windows system the csv line endings need to be enforced.

Test...

5faacd22835a80a1bf420d7fd8d17f91052bb556 authored over 2 years ago by Gavin Barnett <[email protected]>

enforcing utf-8 encoding in api scrapers & test - test data not updated

Due to issues found between differences in unix and windows systems the file encoding needs to b...

36983ef9881da80f9623757ab3fae5c73ae97763 authored over 2 years ago by Gavin Barnett <[email protected]>

Removed tidy_license() from NLS scraper as is handled in merge_data.py. Added fix for file extension extraction.

56c70c9b63b68fc00324be633f7a086af862cda0 authored over 2 years ago by Karen Jewell <[email protected]>

Changed NLS scraper to use url file extension rather than zip contents

dfd867175bd695d515e0ad7f53b302d824b3934b authored over 2 years ago by Karen Jewell <[email protected]>

Fix NLS scraper

291e5b96b89febeef3c3bb80b80e960689e38606 authored over 2 years ago by JackGilmore <[email protected]>

Merge pull request #129 from kymckay/fix-dates

Fix incorrect date parsing for Aberdeenshire data

92da3862f4e85f87a1b3c0f9b67b2b4207d28178 authored over 2 years ago by Karen Jewell <[email protected]>

Ecosyste.ms: OpenCollective

github.com/OpenDataScotland/the_od_bods