github.com/openfoodfacts/off-category-classification commits

display information about % of category automatically processable

237715b8ba07dbdc6377cead43d6b397bfcd5df5 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

update run_ablation_study.py to support models with image as input

7b03efbeb1d4bac27fb3026dcac90dda3be4ef1d authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix fix_image_embeddings_mask by creating a non supervised version

b4edda2007455348c23c29e109520e64a9d54d63 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix weight name schema

f1a56ac9f76e4ca834b2f97f099212a3892eef54 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Fix NaN bug during training with image embeddings

The current implementation relied on the transformation of NaN into 0
by a custom layer. This tr...

e351bf0bed474c0766df9e10a3d171e25aabfdde authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Improve generate_analysis_df.py script

246382df6961c825087f0799e26d9a7a93557838 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add generate_analysis_df.py script

7135e099d04f72a97b0e6643242182c665cbd845 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix run_ablation_study.py script

76cc0d52c90dc02426cfdeace8b71289003e89ef authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Add ablation study script

3b447448b3d375452360c0981535f67560bd0362 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

bug fix in train.py

b84339eddd360e6a18d55bfed7730286c769bdee authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

bug fix in train.py

d2f328a352878371937109fc16737d06ba8a99dd authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix computation of val and test metrics on end of training

9065f0d7082da7e5e3c0cf4e9d37c14e76c516d6 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix split checking

268a9f97f22a98629a37ad5ffca93049416922ba authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

remove unused code

787a3cacf95e1f520768e19fa8d6e115225b785d authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Remove legacy files

502219817443c89c3ded059323672eef06a139b0 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

remove items without product name during dataset generation

Only 0.8% of products in dataset don't have a product name, and this
way we're sure that there i...

aaf42d8c26daeed31e234608d4840b2736908dc7 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

update checksums

d0d9ca6b21d00e5f1eedd1fb672bb7f8b9522dcb authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add cleaning scripts

20740195ddc08ad6f2f5461ad80e3ae442ec4641 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Create new version of the dataset

241f4cb304ab55c3696851ec789b414780071c5f authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

save test and val predictions after model training

5da3bbd6967529894e037d5924d4285d687865ee authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix model serialization

ea81fa536a61169f3d8887a7d2f907cf14e75d91 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

bug fix in train.py

57ec715adeb33f9f6f45f0d8ba16f0107ab0c417 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

improve training script

- add dry mode
- compute and save metrics for validation and test sets at the end of
training
...

fd5ba4ae0739e7e51710ec99590e3e2a92538113 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Disable dataset shuffling during build

It resulted in different split depending on training. The v4 version
is now pre-shuffled

47607e95f8a6af0a2c79bce8c504743995d38ccf authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

name important layer and variables in TF graph

70e31f2cc9df04cd1340d411ace9034b2d499eb9 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix bug in transform_ingredients_ocr_tags func

a95e28d97af8cd843b69eb52dad8a905ec2e8335 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add path to image embedding HDF5 file

2833ed8ffaf8a5cbcce53f308111007d62d1ecaa authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add image embeddings as input to training script

afca31a039ee7328a87ae862298969d47a73d643 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Add ingredient OCR input as default (baseline 2)

158240cffc314b794442e0476b6e469128b2811e authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add image embeddings in tfds

fc5328ccfbd169e6c925cabc104b042086585f0e authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix small bug in gen_image_embedding.py

8a1dddb2955303303cb22098088957e4aea3efb9 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

update gen_image_embedding.py

37140c53d7e49efb693c91a3b778f508a55b113c authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Add a script to generate image embeddings

42a8d33e86c951c2010be76cf819899fc155a852 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Update default model dir name

04d7b5ec20ebeb65a9be9efb2ff3cd7bff428659 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix: normalize ingredient OCR name

71d72a3c3e3023295fb8b8ca6dba95b9435ad25b authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix: add missing field to Config

bde63ccd623e18100c28773a132fb0279ae36772 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Remove references typing.Literal

Should be compatible with Python 3.7

dee20f5abf9386826ca01571d5723b6bf7173d3d authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Add OCR ingredient feature in train script

53bdd32f8c581404ab1752fcd210bd29f1649275 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add OCR ingredient feature

9cd94ed4e2613babf2bc121d476b5704655c323c authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add ingredients extracted from OCR to features

fb35ecb7cad1999caf7306eb38f32d8f9c0d0ed2 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Make OffCategories TFDS compatible with multiple file download

f22f330dba95326cb15090b7362d2967a12a6389 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Simplify off_categories build

92c57adffff259cbe658fdbfc005775cb22d1be4 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

pass model as parameter in serving_func

TF raised a warning

6824a9c8dc72174058662ab6eb43d85f54a1a2f0 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add possibility to add notes and tags to W&B through CLI

f7bf4885decca335be6b8ad18b0cbc32cb490583 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

allow to name an experiment

d368adf2c828b7defc5dd5de19b8cab64f7c57ba authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix: fix emission tracker output file

d9a6bf47ff2d559ae09256d9d7bb916a6d8dd1f9 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix: don't add ingredients_tags to output CSV

yield TF conversion error

bdee45b005e719a25deb52e31b1b853a670de3c7 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Load best model before performing prediction on test set

69b618026bdbe6f4d02b1a1776d8fe085dacb46f authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

and Weight & Biases integration

63a41e7bb8c8e4b256b28d4923bede7484593314 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

use "max" mode for checkpoint saving

as we monitor f1_score

80849cdb7ea17eefc6dbaa25c94021d778059449 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix checkpoint saving

an incorrect metric name was provided

1b4b015a72488d17d3fb91c5200ea725b1b07ba6 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add lr cosine scheduler

b7002abf48923912bb4d7d5c0562466c91796207 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

update .gitignore

44419a2dacf0c1f68422690f21fdbcb9425bed71 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add isort ad dev requirement

6087c68c8b531baa3fd496e9522962d2fff5353d authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix train.py

6d4e1f6e50f5d2eb73cfe2cc41ff6c302efa434f authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

move train.py in base directory

a7e6ec400297e1ba2c5e280e28affa5d64a73fb8 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Make the model compatible with a single input feature

460765c4e22dcd47619a015f3c045cb9906d505a authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix small bugs in train.py

e6e50d07dec86afd38800e34fd852e79253a4039 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Don't '_' instead of '-' in all input names

As they cannot be serialized in SavedModel format otherwise

9c27cfccffd5131c455f2522060a367f71091cfc authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix serving_func tracing by providing all parameters

2c0c3ced1a5d03d106f77c8c3a58583cf8c30967 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

update default batch size

b771596dcf8a923ff3f0ec3053568393915664b0 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

don't include optimizer in saved model

8cbdde72e489115a452c00931aaf13ca2c09aa3e authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Save emission tracker results in a single CSV file

efbb62678f7470b85f39bb853df98e7d527d88e5 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Make train.py compatible with python 3.7

1ffc381409d9e591a09441fb78dd73ca7694b2e1 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Add notebook to analyze feature distribution

35c35e157c6c1f7eaf3f97ba41bb08c290e38368 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Save emission file in model dir

913a2de99301737d25a8da579d4353d2dff65cbd authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Save test prediction in TSV file

57ccb5c56ffb320a8ed1431e64a69f9c1f059f92 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Update requirements.txt

cf17c3b4a62a088ae1f5c3dcd2578689b7b038d9 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

create a training script train.py

367055ed246372a3eb2d0c29b7e453ca970606e9 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Make metrics keras serializable

63b171cf5e57ac1d13eb5d466fa5239c6bc4f861 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Fix energy-kcal preprocessing

2cd910315292c08075c1bfcd5d5ca123c0e74ce7 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Move field preprocessing functions to lib.preprocessing

26f6870afa06db074c97c6af7ba40200e6378637 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add nutriment as input in TFDS

2e64ad8fac0ba503bd2611daf3ad0a96d0e51d31 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

update Train.ipynb

839982d098fd04671273136d46f8cbc31b76dabf authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix get_vocabulary function

85c760996b9d5998cbfb8ffe36fcda7a2506047d authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

use StringLookup layer for ingredients_tags input

decbcbf00309200b08555f881e3b632654d181fc authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Remove temporarily product_name_embed in TFDS

69491e6be61ada1d299ae982e87802b3d1a2347b authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Do not only provide [CLS] embedding for product name

f0b8edb19e6880e0e05d1c905059420ecf431be9 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

use git pull instead of git fetch

f394ed8b6a39418d032a0022db464c8b0c6690ef authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix: force git checkout of latest version in colab

b5414d6ab04ab7d68beede2af51a9035dde7253f authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

switch back to classic product embedding & and LSTM over ingredient list

39c78b8271bc8558b1732423b7969a09f0af2fd4 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix: small bug fixes

558a8e4e0a5ac76db47166ffdb7339a3b41b7c56 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

fix: give unique ID to each sample

373354b87c8268021cb03b792f10122228642797 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Adding transformers and sentencepiece as requirement

6e5e6d1813985748e16a389f63904f0afd869ab7 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

add new notebook for training model

92bfed54132193c8e04b6a04014e3b44eae47ecb authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

save product_name_embed field in TFDS dataset

a69d3496d02e77817865741f08212890e22b8887 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Rename Train.ipynb into Train_baseline.ipynb

59d19c9168b19b22197a945401bbe972a54be298 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Save Tensorboard logs in model dir

6d2d4b91a29149f50e671589547b8b286b9171a2 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Remove cache dir

It's not used anymore

73e9177ce139ddb21141bceea5077f6a57984f6b authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Add tf2onnx as requirement

9e4c445b638ca6ea50c513c962e9b3007c1a86d1 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

format Train.ipynb with black

ddafdfb1c8907dccc240c117b883c74017fb280e authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Update Train.ipynb

69b750b35a4a71a2d0f35b05e58d2c5f30d270a2 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Allow to provide kwargs in keras load_model func

224398542b70387ccada4e56a6714684a085b105 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

update name of metrics

b4274a5fcc247259c1c96325807528dbf2f28c5e authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Use micro F1-score as monitoring metrics for best model

e11a28aee11517653f29470be2207580d49845f3 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Reimplement Recall and Precision metrics to get micro/macro average

8e619b645c72b5e31e8e176fada9afa2a9f3e675 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

remove output of jupyter notebook

f7d52deb8ece2fe37a2484ec02efec0554e24fa0 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Don't add again excluded categories

5a1edb483b01ce01200ce2ed2b0aa1ad347d6e4f authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Fix dataset build

3d18eb0628add11162c4bc35dd1f166ac30de2e0 authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

update requirements

to be compatible with Python 3.7

c834e4c141954f56d70fcd1fc175e8f646d7100c authored almost 2 years ago by Raphaël Bournhonesque <[email protected]>

Ecosyste.ms: OpenCollective

github.com/openfoodfacts/off-category-classification