Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/ironArray/Caterva2

REST and on-demand access to local/remote Blosc2 data repositories
https://github.com/ironArray/Caterva2

Playing with drawing sizes

de2e5778c0b87ad398a60067ef179a969d959109 authored 11 months ago
More work on drawings

5e9c0a60ceecd99a8cb345fda15f59cd7c447927 authored 11 months ago
New drawings

17d34198b1adb30fcf4c01b442eb171883bfca8d authored 11 months ago
Merge remote-tracking branch 'origin/pubroot-classes'

This fixes a reference to a global variable instead of self.

cb6d319fa20cbeac81e0e6e1c502a3e08bae1c9a authored 11 months ago
Fix directory root watcher using global variable instead of self

It happened to work because the published did have a global variable holding
that instance.

0ccf4d28c21d9917fd15a3eafa193f6a9d1a2e01 authored 11 months ago
Merge remote-tracking branch 'origin/pubroot-classes'

This defines some exceptions to be used in publisher root classes so as to
have consistent behav...

069b820ec7a4b35258902d91fdc7aca2376c893e authored 11 months ago
Use consistent exception for missing chunk in dataset

0032a284ca8542d169d8e3817142158822c35071 authored 11 months ago
Use consistent exception for missing dataset in publisher root

69d47f4029d2a9b10201f0b8ca47aec0e8ec9565 authored 11 months ago
Merge remote-tracking branch 'origin/pubroot-classes'

This adds publisher root class registration and some docstrings.

beba98628aaa9637965b6b3a66df5e36d1cb26f4 authored 11 months ago
Add docstring to publisher root module

7adf457f78259f596f572bf967e8373fa7d89bbe authored 11 months ago
Add (and use) func to use proper class to instantiate publisher root

By trying an ad hoc method of each registered root publisher class.

a4d74e05db2961ff549b430e7e24545479acf516 authored 11 months ago
Use custom function to register publisher directory classes

So that the collection of such classes can be queried.

0cacfc8956f68ff4a82fbe97e3426aba4a33df43 authored 11 months ago
Remove ellipses from abstract methods in publisher root

The docstring renders them unnecessary.

5f4878f6a0af83042b3763e916c3d84d81f835be authored 11 months ago
Merge pull request #27 from Blosc/pubroot-classes

Use publisher root classes

ff354352701481f144b2f060b655ce64fe5b8723 authored 11 months ago
Some fixes for Qodana warnings

f3e77ff7a5f972fb42fcb89e59c6ac7fb5bff735 authored 11 months ago
Add docstrings to the publisher root module

853e1dae3210c6c6d4b8963e30fe04b82f809cc7 authored 11 months ago
Add (and use) method to read the raw dataset as a file (if possible)

9c9d29807d4964323658d247cb13c861f8d1aa32 authored 11 months ago
Implement (and use) getting dataset chunk from dir publisher root

With no checks other than those in the original code (e.g. specifying a plain
file will fail str...

970630644d174137ad717193e13c314b5f2a3f21 authored 11 months ago
Use HTTP helper to check for dataset existence in publisher root

Consistent with the existing helper that only checks for filesystem paths.

1ff5156d369f7eb665d6400d929ee049ca35e8b5 authored 11 months ago
Implement (and use) getting dataset metadata from dir publisher root

50e120ad11b1974efaf1a712f54c013e17eebf47 authored 11 months ago
Replace absolute w/ relative dataset paths when the latter works

The relative path is converted to the root's representation (also for
sane/canonical representat...

007700972f1183353bc40b39c6644f6545885de2 authored 11 months ago
Make watching for changes in dsets an async method of publisher root

Define it for the dir publisher root to pull watching files out of main
publisher code.

1c21c14e2da67f864093e79f9ad0eb5536c93c87 authored 12 months ago
Dataset queue now holds paths relative to the root

For non-directory publisher roots, it may make no sense to have a path that
crosses from the rea...

cebd0355b5b26881df91b0d13a72c506c089d059 authored 12 months ago
Replace occurrences of root path with `proot.abspath` in publisher

However, uses of that may be an indication that more changes are needed there
to decouple code f...

a03985cb51be2e4a61f22e4319962cd64a5e008b authored 12 months ago
Add (and use) dataset etag method to dir publisher root

a16b36915e43761c08628ff32712e59a41832c4e authored 12 months ago
Add (and use) dataset existence check method to dir publisher root

Including internal check for valid dataset relative path.

0c5ed850167d318fd1d70899e9b3e874a72e3563 authored 12 months ago
Use more compact and consistent names for publisher root methods

c217116a62959a4c007387eb7b31e3ad70e7bdda authored 12 months ago
Start directory-based publisher root class, with walking datasets

And start using that at the publisher.

d23b991ff963000d4c3184dd43db84c861300098 authored 12 months ago
Parse publisher root directory option as a path

21d89d032026ad522042ed9288bbe73c087b13a0 authored 12 months ago
Add abstract publisher root class (`PubRoot`)

With some potential abstract methods to define its interface.

169b2804f7cd701c8503dec7037d6f11fcb86abe authored 12 months ago
Merge pull request #26 from Blosc/import-export-keep-b2params

Carry compression params on import/export.

This updates `cat2export` and (especially) `cat2im...

9b762544af6431e8c5493a5adfa38bbbf1b46bcb authored 12 months ago
cat2_to_hdf5: Handle frames with typesizes other than 1

81a76060be34ca8d0450c431316bb24017cee350 authored 12 months ago
cat2_to_hdf5: Refactor iteration over chunks for arrays/frames

600613b375001ac8e558bca100422b69d36205f0 authored 12 months ago
cat2_to_hdf5: Do iterate over chunks when exporting frames

Using the same technique as when exporting arrays.

f81ae815efb852de391765426eb4be692c4ada31 authored 12 months ago
cat2_to_hdf5: Add missing annotation for returned chunk iterator

2c23c53ef82629669454cefc38393cee0aac1978 authored 12 months ago
cat2_to_hdf5: Do iterate over chunks when exporting arrays

The technique to keep memory usage low and not create an array each time is
similar to import co...

e493ce7b9b46591465de31e5a33b1d12ef0e4919 authored 12 months ago
hdf5_to_cat2: Fix annotation for chunk iterators

They are `Iterator` (they can be iterated directly), not `Iterable`.

2fff6ecdc6a00e540dfeb304bfcccaadf456f3d2 authored 12 months ago
cat2_to_hdf5: Rename variable for consistency

a9a23d8734535b4afcf9dc66046e8b88586d534a authored 12 months ago
cat2_to_hdf5: Prepare dataset export function to iterate over chunks

After creating an empty dataset.

730dedf64d25821379cbef3d6dd36215624c1fc6 authored 12 months ago
hdf5_to_cat2: Take attribute exporting out of try block

No need for it to be that nested.

Also for similarity with export code.

5984cf035abe7cbfb35ddfbfff14e560c133b570 authored 12 months ago
cat2_to_hdf5: Pass shape & dtype arguments to HDF5 dataset creation

They are redundant now, but they will be needed when an empty dataset is
created, so include the...

d1164fe098c4389dd9d3bc0c8a8178319160e6e1 authored 12 months ago
cat2_to_hdf5: Rename function and variable for consistency

c5c7a9bf3c7766e83db1f48ad6eadba12435ac2b authored 12 months ago
cat2_to_hdf5: Get function to create HDF5 dataset creator

For similarity with import code. Getting the creator does not perform any
changes, but it still...

fffdeffd886244c02475d39ec4968b4b991ef183 authored 12 months ago
cat2_to_hdf5: Merge reader funcs into one w/ leaf type determination

For similarity with import code.

b8e45ec3c0e3a0c0d5fdb7ff35dc8739a6e81267 authored 12 months ago
cat2_to_hdf5: Turn around determination of leaf type

For similarity with import code.

2764ac0d373dbc62efb76fab956c082e723113ef authored 12 months ago
cat2_to_hdf5: Note on the reliability of HDF5 filter arguments

e504fa43c8879bc4b7d768dc46757ad49d5a0d65 authored 12 months ago
Add a symlink to drawings

1bf694e671263c6a8b858db70d4500c360590579 authored 12 months ago
Make the logo more visible

f2e1f3b4c9614d45f7b8215e2966cbf925d314e8 authored 12 months ago
sub: use tabs for dataset info

c9fd31b3ba3021f8a38fca0668b462905785914d authored 12 months ago
sub: sort list of roots

04e1c9929b1fdaf998c09c42182ab421ec861ff1 authored 12 months ago
sub: don't display trailing slash

5e504e55141ad40e5356930ee6ee49b439b42984 authored 12 months ago
cat2_to_hdf5: Add todos to remind about not slurp & re-compress

6a6d77eda228235e0aff8853e22ce537aed3a0e7 authored 12 months ago
cat2_to_hdf5: Rename HDF5 compression args from default to file

Since they are only used for that. Also, remove related, obsolete todo
comment.

3343951f78ed7a2f35598c5bd5711c93f7f1f48c authored 12 months ago
sub: don't reset search on root click

df4b3f658dd038e2493d8bd3f806313b7f3c8fcb authored 12 months ago
Remove unused template

9c493bd88986fd702c4ad246aef70daa5b3e6c40 authored 12 months ago
cat2_to_hdf5: Rename default HDF5 compression args variable

For consistency.

35153a93450593afa8c85968e4a14510530dcbd3 authored 12 months ago
sub: helper function to make urls

f32f544442c7282c82d158ef68b141f88daa0c5d authored 12 months ago
cat2_to_hdf5: Avoid HDF5 compression args for scalar Blosc2 arrays

00696bf91e32e904b7d59507a58e506ef64edbc2 authored 12 months ago
hdf5_to_cat2: Always set `blocks` argument on Blosc2 node import

Especially for the case of a 1D array where the blockshape was set, since no
chunkshape is store...

4cb1fa625f7a748ea5dd778a1957cadf160913bd authored 12 months ago
roots multiple selection

c3b13f4272de47a7e3506b3c762b268e97399ea4 authored 12 months ago
Tools: Factor out Blosc2 HDF5 filter identifier to common module

243c7c0d9ab2000dcd462ef2e046d0adce7473b8 authored 12 months ago
cat2_to_hdf5: Derive HDF5 compression args from Blosc2 cparams

For Blosc2 arrays, frames and compressed files (plain files still use the
default).

527e1a664a53a471315c35fc331e0a9736e01b02 authored 12 months ago
hdf5_to_cat2: Only copy Blosc2 chunks directly if sole HDF5 filter

If other filters (like HDF5 shuffle) are enabled, the schunk may be
scrambled (and there is no w...

c44a2cae915ae8fe93cbc27d267cbdeef60cb236 authored 12 months ago
hdf5_to_cat2: Define Blosc2 HDF5 filter identifier as integer

f20b68a70b9714a1f541441d7ad57ffa33731626 authored 12 months ago
roots multiple selection (wip)

e2a6c4c058ea98afb119d5e4d6d479b47eef6cae authored 12 months ago
hdf5_to_cat2: Cut comment for PEP8 compliance

b81d054772971fb028aa7afc3f010cc9907cc872 authored 12 months ago
hdf5_to_cat2: Handle translation exceptions in separate block

Since nothing has been created yet on disc.

841c44a851376addd037d5bb85a0a4a68c48dee3 authored 12 months ago
hdf5_to_cat2: Simplify some return statements

ebaeccf222b39b4d83188e888905cee282bb20ba authored 12 months ago
hdf5_to_cat2: Add docstring to array maker/chunk iterator function

3d520b478585acb4d116c1a9d70887cb4769c843 authored 12 months ago
hdf5_to_cat2: Comments on data kept in memory for chunk conversion

fa3e27455a5bf19f23c41cce8ad87ec1fffdd332 authored 12 months ago
hdf5_to_cat2: Convert chunks from arbitrary datasets to Blosc2

c85d5c3e61237b69485aafef3289872cc1f92908 authored 12 months ago
hdf5_to_cat2: Be more consistent with chunk iterator function names

2a581e8f343cdc59dee670d8504fa8c11d3837d1 authored 12 months ago
hdf5_to_cat2: Only return one result from opening HDF5 chunk

Otherwise, getting `(array, schunk)` was still prone to segmentation faults if
the GC decided to...

87f3df0cca81821cb96efaaa6635196903d1fe64 authored 12 months ago
hdf5_to_cat2: Get empty array maker & chunk iterator via single call

This hides Blosc2 arguments from the caller, and storage issues from Blosc2
conversion machinery...

c187477394ca89bebdfd052cbc851dbb64e0b172 authored 12 months ago
hdf5_to_cat2: Drop unnecessary index from Blosc2 chunk iterator

a96a1c971f7be91ab2adaeacec6355ef6044b11a authored 12 months ago
hdf5_to_cat2: Factor out reading Blosc2 schunk from HDF5 chunk

9fbcf776d8fea702134ca503111a6b7dffb5dce6 authored 12 months ago
hdf5_to_cat2: Get blockshape & c/dparams from first Blosc2 schunk

9c89764e3489af86158786c436ac9ab319f8bfdb authored 12 months ago
hdf5_to_cat2: Put Blosc2 HDF5 filter identifier in variable

ea002f088f17277ff6ef03b083c852094a4fd87f authored 12 months ago
hdf5_to_cat2: Fix way of copying chunks to empty Blosc2 array

The empty array already had chunks which were pushed back by the insertion
operation, so they we...

c57abe22f70e55eb8d1281f44f22fb4924e39706 authored 12 months ago
hdf5_to_cat2: First attempt at passing Blosc2 chunks as they are

c8b07c2a6510978bd3c82f41549e2bc9f22270df authored 12 months ago
hdf5_to_cat2: Minor changes to loop variables and error messages

3b0ca92fab7ce8ee4233e3a37b10002fa2bd8b60 authored 12 months ago
hdf5_to_cat2: Create empty Blosc2 array, insert data chunk by chunk

Source data is still slurped and re-compressed into memory to create the
chunks to be stored.

ca71e15ddb75a1b2d0b01a07cb5b8a8cca4ec83e authored 12 months ago
hdf5_to_cat2: Propagate HDF5 chunk shape to Blosc2 array

738266cb2ae2cddb7aad6d7f59b5cd57d2c775ac authored 12 months ago
cat2_to_hdf5: Avoid applying compression to scalar datasets

7b6bc0fcfcbb62ed41fbf6285efb618f18eec9d8 authored 12 months ago
Specs: Fix typo in example dataset creation code

e74fd489349eef9a91ba57ee6fca90081b7b7732 authored 12 months ago
Merge pull request #24 from Blosc/add-tool-cat2hdf5

Add `cat2export tool` (Caterva to HDF5).

This adds the `caterva2.tools.cat_to_hdf5` a.k.a. `c...

f1af7e49fbf17db2731dac153984bd4dbdc450d1 authored 12 months ago
Docs: Add text for the `cat2export` example

6c5dc89a6816d5bd22b5d42fb4d2e5e801f8406f authored 12 months ago
Docs: Add text for the section on `cat2export`

Missing example subsection.

6ea647cbbaeb457900bcd6ec83de4cbf9e5ad8c4 authored 12 months ago
Docs: Fix typo in `cat2import` introduction

3a25a3dda7b4a01f8aa13a019a7333d33d3b76c4 authored 12 months ago
Docs: Add skeleton for `cat2export` document

936519d22a0ba487e18159238106634c4f69b67d authored 12 months ago
Merge pull request #25 from Blosc/urlpath-fix

Undo root being part of dataset path in clients

9960caca52849eee111cae723deb11095238be79 authored 12 months ago
cat2_to_hdf5: Use some default Blosc2 compression parameters

So that exported dataset enjoy some compression instead of being stored
completely decompressed....

007ee1de523d8f9e46ce607f1c5416cd0d446a05 authored 12 months ago
Undo root being part of dataset path in clients

785200d4bccb65b8c01016a579ad75236f76199e authored 12 months ago
Small changes in import/export tools for PEP8 compliance

6f251fadd11c18302f1b91746ecf0efce9377bd4 authored 12 months ago
Small changes in import/export tools to make Qodana happy

acc31d70c0793b82c1bfdab93496022c4f341743 authored 12 months ago
Warning about data being slurped into memory in import/export tools

aac88f4a0364bcf467cc9d2df1b5c4adaf57e423 authored 12 months ago
cat2_to_hdf5: Expand module docstring / help message

f23bcf5d7a30ccd6ff4a4b2a87657618224d50b4 authored 12 months ago
cat2_to_hdf5: Allow exporting plain files (new reader)

176f2bcceeb7979bd5f96c76ee709dc019978963 authored 12 months ago
cat2_to_hdf5: Have reader function provide HDF5 dataset name

Since it depends on the kind of dataset exported.

47adf5f2f11378f2b1eee380ca2c9ce1e58dc182 authored 12 months ago
Update data on demand figure

2b2ce1f7904ab372ca813c593e34b98d9f18947e authored 12 months ago