github.com/ironArray/Caterva2 commits | Ecosyste.ms: OpenCollective

Support clients reading settings from a config file

Currently just to get the subscriber endpoint.

02fdb3356be33cd1c3edeaf16428d205f8e1e0c7 authored about 1 year ago

Move configuration file support to main utilities module

As clients may also be interested in getting settings like the publisher
endpoint.

As a result,...

773c7cb9317c2cc11bb527fc1a20e73a1d263d3b authored about 1 year ago

Improve search

dfc0dcaba8e7b7a4c104f67de439f3fef8f530f6 authored about 1 year ago

Update meta onclick

a7046a97fab58824fe6ff523f670726ee484b186 authored about 1 year ago

More refinements in docs

ecabc34e999b0d401694928cb0e5a45d98e090f9 authored about 1 year ago

Just rearrange arguments and output of get parser helper function

It gets arguments from more generic to more specific, and outputs option help
the other way around.

afde21a21ec7394ebd8f3bd87808f06554642fcc authored about 1 year ago

Move parsing of `statedir` option into get parser helper function

This avoids boilerplate in service scripts (and the function already contained
support for other...

9215ce97b2f67c68af1ce20bdd28d9895e1ffa7e authored about 1 year ago

Support subscriber reading its settings from a config file

a1d1117ead9d69dfaaf69e53a4f6b6e14ae85664 authored about 1 year ago

Add directory needed to tests to the wheel

35869fb69a1326f5a05434dcd0de879e09aa2613 authored about 1 year ago

Use shortcuts for scripts in docs

6703a3901850e53309c47a03ee83e4ee5526a861 authored about 1 year ago

Move parsing of ID option into get parser helper function

This will be used by several scripts.

5058af842942eb02a348858456186320a2c2dc06 authored about 1 year ago

Always set service ID if check requested, even if empty

To distinguish configuration objects when it was requested but not provided,
from those where it...

33c9730590805a789d8722b1d896951b7cbb02c8 authored about 1 year ago

Allow to specify an ID for the publisher service

4b3b72eb41d3dde5129ad98b006b3d491359a867 authored about 1 year ago

Have service ID explicitly passed to configuration object

So that it can be retrieved later on.

22550a65a10012ec3d0265a522cc7d18f242ffa5 authored about 1 year ago

Check that service ID does not contain invalid characters

947d1d6cbacab011ddbd10c06bc905f8bc7e3988 authored about 1 year ago

Merge pull request #12 from Blosc/new-fetch-api

New fetch API for better separation of concerns

f278d82e53a6cfec5a58142bbd30c79d3f67e147 authored about 1 year ago

Allow to look for ID in CLI arguments, then use it in config prefix

For distinguishing services of the same category in the configuration file.

45e0ee8cb10c8ae13674c2649c72fb3af79ab646 authored about 1 year ago

as_schunk -> prefer_schunk

ec28258bf4c426c0f801f691d292a2a5f04f7e5b authored about 1 year ago

Some tweaking in perf; not a big deal though.

0e6c7f5de20129045edc844d22d3f2375ad2d8b1 authored about 1 year ago

Add a new File.fetch method, mainly for testing, but could also be useful when tweaking performance

84cf5daf2908a63376600babeb80eaf888c07abd authored about 1 year ago

Allow publisher to get broker endpoint if present in the config file

32910656063037f17547e26e21daa6b19f1a39f9 authored about 1 year ago

Support publisher reading its settings from a config file

Still need to support separate IDs for multiple publisher configurations in
the same file, as we...

3a9e13977aa00daafc268966ba5bb0c74a5d75bb authored about 1 year ago

Specs: Fix TOML string syntax in configuration file example

09b01c436aae9ddda2e02374122e078c30b8c203 authored about 1 year ago

Allow a prefix to avoid repetition when retrieving config items

And have the broker use that feature.

fec011cd447d7f92283d6ec96ddf071c0c648c0f authored about 1 year ago

Allow getting loglevel from configuration file

And have the broker use that feature.

8991ee8df48970d91d56295d8d651421398bfaf8 authored about 1 year ago

Use configuration file in broker

By using the loaded configuration in the argument parser defaults, so that
command line options ...

2d95205971a3a702f26e31de82e332ab93727456 authored about 1 year ago

Add simple mechanism to read the TOML configuration file

It uses either the builtin `tomllib` or the external `tomli` according to the
Python version, so...

e0f8047640d5622a3b3e6df890958ad8545ab340 authored about 1 year ago

Fix assignments to global variables in scripts

9446b41223da08cf191cb396df3560e286af0ca8 authored about 1 year ago

First version

dc0a88fa16c743b89fab062891556d6a0e35d94e authored about 1 year ago

Structure data as 3 columns

d2619b7e69843cb48fb4b05a9ca76179aa7c0925 authored about 1 year ago

Add search

91e6c2f23e2df5d8e45eb0d7567dffe4a5f5fc3f authored about 1 year ago

Add project configuration for tool hdf5_to_cat2 as script cat2import

0e96674b49b665d42e69cb67fba2163300e45f5c authored about 1 year ago

Fix existing scripts by defining a main function

Since project configuration points to that function.

2d8473e1a7ff5f77edecab06243ccced31e91e65 authored about 1 year ago

Merge pull request #11 from Blosc/no-partial-downloads

This removes the partial downloads feature. Partial downloads can have different bad effects:

...

9e1fb72113feb00bd1bb91272eb9d2dcaf502cf5 authored about 1 year ago

Unpack regular files if blosc2 is there

f8fa52b2513149dbc492c43d6d7b5fcfafe11caa authored about 1 year ago

Make it explicit that downloads do not support slices anymore

28b278810ab97ce3a5cd182f39ccbc0dab937ad0 authored about 1 year ago

No partial downloads are allowed anymore. Also, downloads/ is not needed either.

a97ee0df9d6fdbd5bcfcb8cc0a597acbf7fcf760 authored about 1 year ago

Fix docstring

d1e009be8309aff75dc575dfface787cb55efd4d authored about 1 year ago

Preliminary commit

f21be7e2d20c36a6525131a24f59611f1e7b7f41 authored about 1 year ago

Readme: Add note about new `hdf5_to_cat2` tool

d987d6c57fe8eae7e127d59caa3194c977f54b95 authored about 1 year ago

hdf5_to_cat2: Module docstring note about default compression params

7fa36a40169362a6440401d283aa4f127d76a953 authored about 1 year ago

Merge branch 'add-tool-hdf5cat2'.

This adds the `caterva2.tools.hdf5_to_cat2` script to help export an existing
HDF5 file to a new...

2a6fdb37721bdb94944c553876f01b46c8777fc5 authored about 1 year ago

hdf5_to_cat2: Define helper function for printing help messages

42a5cf5ca13060a66ec51f2b8f4c4858554064d7 authored about 1 year ago

hdf5_to_cat2: Expand module docstring, print it as help output

b9af5dae007acc2c66a53c9fc45fac089b1f5272 authored about 1 year ago

WIP

4e6de67f76f27633f21b1d45d6dbe72bb355949a authored about 1 year ago

Merge pull request #9 from Blosc/decouple-deps

Decouple deps

ab29f2817296137908d65313610084114d5f575e authored about 1 year ago

Fix for function call in api_utils

ce32899f833136c748e223850ebd76b50be70443 authored about 1 year ago

More PEP8 style fixes

b0b66c703ba06fd7037ac2ceff77eb5351785698 authored about 1 year ago

Small improvements

8f5df8a7bff3ae7dba829d88922dff555150a97a authored about 1 year ago

More PEP8 style fixes

b3a6c3ee601c7998bc519b10f1b67e0d1aed45ad authored about 1 year ago

Fix subscription with ND datasets consisting of strings

The old code failed to get a proper dtype from metadata to build the
uninitialized dataset.

All...

1647a5adf206ae58140dddf1f1b9e18dccc70750 authored about 1 year ago

Fixes for some PEP8 style suggestions

1bb0a4cd9377f2ce4c51542d09c7e30307b8dc93 authored about 1 year ago

Remove duplicated code

8f687ee516c3b6425f100f927a8a4c16a62fd906 authored about 1 year ago

Fixes for some PEP8 style suggestions

45763fdcb4f0777332e98c1074824b0b94cfa11b authored about 1 year ago

Documented HTTP API for sub; code beautification.

4edab935cf5c52f6a546ff42d1e9baf4ebdb13ed authored about 1 year ago

hdf5_to_cat2: Split overlong lines for PEP8 compliance

fc5bcdfbad4cf1a92893c9f5cd8432fa1255bed1 authored about 1 year ago

hdf5_to_cat2: Add brief description after usage output line

96737d16aa23c8cbce3a4cbdea8a3147f9c7099d authored about 1 year ago

hdf5_to_cat2: Manually encode attributes using msgpack

But without strict type checking, so that we can handle subclasses. Then set
the packed result ...

bf9cf477ef2c5fc26ff87f0665e94c0fbc3113d0 authored about 1 year ago

Use names that don't collide with Python names

0fe8b00bbb17675abfa6a9f7fe8428844b5101b5 authored about 1 year ago

Add new API functions and use them in cli

778561e5de6430d1e4f67963e99c41649bd99cc2 authored about 1 year ago

hdf5_to_cat2: Do not force string decoding of bytes attributes

There may be attributes of other types which Blosc2 should be able to
serialize with msgpack int...

07986fcd4c5de8dd391030caead208095aeaff12 authored about 1 year ago

cli show is using the API now

5e8d6497358da7ce547211c8b8cc6a08add436d8 authored about 1 year ago

__getitem__() goes to File, and more tests

fe86dd5c555afcf6a70cbae0de6677724633cc50 authored about 1 year ago

Refine downloads *and* slicing

f1e2be82c2de18e78db9223fc52d209785bec8c9 authored about 1 year ago

Add docstrings to API

31576a7b9f811fbaadfc7ed7ac76617c3268566a authored about 1 year ago

b2_utils.py merged into srv_utils.py

22d0b5985ce25c6ec7ab82556069933135ee706c authored about 1 year ago

Make sure intermediate dirs are created

2ad5bfe9b980e0c74c7dc02735570017f433cbaf authored about 1 year ago

Introduced a 2-step download. 1) get url, 2) download from url

1cb6517f2c9770e240d58c200caec3edcb219c1d authored about 1 year ago

Mount sub cache files in /files

5db802b3e9d9f58bb12101c363ef1912366f3e0e authored about 1 year ago

No more dependency of blosc2 (and other) in clients

8e1343d6df466c149a2d4c89f5c99f7f0e8390fe authored about 1 year ago

hdf5_to_cat2: Use "dataset" instead of "node" in dataset messages

1a589e6006d9950bf0ea9d8c1844711ae1d14a1d authored about 1 year ago

hdf5_to_cat2: Initial attempt at exporting dataset attributes

Only bytes values are supported for the moment.

e5f28cb0c6b107bc74af93edd50564a68cd8e125 authored about 1 year ago

hdf5_to_cat2: Report actual destination path of exported nodes

1ef9b7da2affe7f0caccd37cc8092854d9b2a3dd authored about 1 year ago

hdf5_to_cat2: Report exporting nodes after succeeding

2be61d7e21b2f9036ffe2dfcfae91f2a7121a5ba authored about 1 year ago

hdf5_to_cat2: Initial attempt at exporting dataset to B2ND file

Add dependency on blosc2 for the tools extra.

ccbe67fa95065aa1d922d166152325503529246d authored about 1 year ago

hdf5_to_cat2: Import hdf5plugin to support reading Blosc2 datasets

And depend on it for the tools extra.

f3719a7162b0b73fe42a3900378cda5c9809e505 authored about 1 year ago

hdf5_to_cat2: Implement exporting group by creating directory

82afaf474c03e8060d61c13f9192e09c498e0c4a authored about 1 year ago

hdf5_to_cat2: Invoke (dummy) functions to export individual nodes

c309bae2db01aaa488c0c776525ccacc6829419f authored about 1 year ago

hdf5_to_cat2: Warn when exported node has attributes (unsupported)

3db2187ab8788ae4fef5808be94a5f61de9c2761 authored about 1 year ago

hdf5_to_cat2: Skip nodes with invalid names or unsupported classes

ce07f9b7661d0b302843527f7ea3c27b0592d533 authored about 1 year ago

hdf5_to_cat2: Use the logging module for messages

9c8eacebbb4a85dda0fc5c8fdf22ba63f8aa9246 authored about 1 year ago

hdf5_to_cat2: Add some docstrings

a84f5b3e07817bd21e845c478713afe6ff3e93f2 authored about 1 year ago

hdf5_to_cat2: Visit HDF5 file nodes, call dummy export function

b505009d22ddd0156ab51b5c69dd666384c6a525 authored about 1 year ago

hdf5_to_cat2: Open source & destination arguments

Accessing the HDF5 file depends on h5py, added tool dependency.

eaf7aa8b6e0601efacd13b7d3036d15f30af538d authored about 1 year ago

Add skeleton for hdf5_to_cat2 tool

243c2272894065abaa4fbe7cfc6bfe2935d8d947 authored about 1 year ago

Notes on possible outcome of moving dependencies to services/clients

4bd9c7b702519cc2166d4a097caa7a256857a5f9 authored about 1 year ago

Fix invocation of download function from file API call

Although this may be completely dropped later on.

b2ad877a008ab9b9747b88f1449e0258aa8a5337 authored about 1 year ago

Move dependency on tqdm to clients extra

df70fa069976f41a20dd86c1826cc5b85e1150ac authored about 1 year ago

Have download function caller pass progress report function in

So as to decouple reporting from the function itself.

And define tqdm-based reporter right in c...

ef528aad1391a1f89f892fc924aabf775c6d9199 authored about 1 year ago

Move download function into client utilities

Clients are the only users, and it avoids some extra dependencies in services.

27967e48301ac8fc200d84842d3ae7a8d10d77e0 authored about 1 year ago

Move Blosc2-related utilities into their own module

As many will be used both by services and clients.

91c6370c66cfc10c653f98af1f2e0adb36f0d2e9 authored about 1 year ago

Move some dependencies into the services extra

6c5f6c6b061650720e5fcd03e151cb364dc50417 authored about 1 year ago

Move FastAPI client-related functions to service-specific utilities

f63278d09b44cc94984d6f816ad85a406eee2c92 authored about 1 year ago

Move FastAPI server-related functions to service-specific utilities

51a37b188bd5ba6a98e922062e8a59c6694aa15d authored about 1 year ago

Move pending Blosc2 functions to service-specific utilities

a9cc2fedc2ca62c39b5fbb0fd8754d473006cb38 authored about 1 year ago

Move read_metadata function to service-specific utilities

79d2fe65cd9ba45bbf670e638352c2c7413eca9f authored about 1 year ago

Avoid client dependency on models/pydantic

By explicitly checking for boolean in JSON object without going through schema
validation.

9716acaab55656df88f853d2ed98ede1fedeab2f authored about 1 year ago

Add undeclared dependency on pydantic

Which should be moved to services.

5331b90142fd981d3eb988d37f0e44894a95d033 authored about 1 year ago

Fix a bug in start, stop calculation

d00e5055fd12c90f3fd1719030a4e0df6756579f authored about 1 year ago

Avoid unneeded second slicing operation

09c787fe28b61946f1cacdfc86e065739c39a828 authored about 1 year ago