github.com/chaoss/grimoirelab-elk commits | Ecosyste.ms: OpenCollective

Removed projects download from gerrit. Download the full gerrit.

Several fixes to gerrit stack.

757169fd4cf8ae4ecc8dcd2841277379c7da2265 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added break in infinite loop in bulk api.

761d374ec4022c023ba7c2f007fcb6411e54e149 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented incremental annotations for all backends.

352a139c92dc5d139fe647638e1b087c5d0d5cf5 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved bulk sync method in elastic class to work in incremental updates.

Improved logging for bulk uploads.

cc5fabd06f43415a8b0743ea5da28a2fbbbac193 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Bugzill backend must use the Elastic server for State:

bugzilla.set_elastic(elastic_state)

d882e06f7a6784212f99d0f18e6daa345cf2c0af authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented new cache system for CSV (list detail) in Bugzilla.

c504dbdedfbd68cd598dffcb71f7842e25064722 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed Bugzilla stack to work with one type per index, two indexes: index and index_state.

91b78def6c4254ff0650f9908838cda69d8a4a1a authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed datetime diff method.

1cb2bb1e82507d3f2eaf1f4e23d52f082300df21 authored about 9 years ago by Alvaro del Castillo <[email protected]>

When restoring the cache, cp cache_recover, no rename it.

4c9b7865457d02d8e58182e906489d0452e25124 authored about 9 years ago by Alvaro del Castillo <[email protected]>

General refactoring for using different elastic config for State and Kibana.

Improved how types in index are used. Just one type per index.

ba9fef3c1f6dfc80370bea3753ad4f5b330ccc91 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Name refactoring and cleaning in github elk stack.

2ccc44f8e875f005b92520f8bfaf0cd95c7d0b11 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented cache recovering. Show howto manage it in utils/github2el.py.

da9ed7729bb432e3dc70ba93e51a4fc6980fbe7b authored about 9 years ago by Alvaro del Castillo <[email protected]>

Store the cache in a specific directory in perceval storage dir.

6541e61d6fb3acb12aaf622c4af2da16fae22ecc authored about 9 years ago by Alvaro del Castillo <[email protected]>

Removed not needed log

0de938001e53166e6252ecaf22892f9406181e02 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved var names for understandability.

eaa507de500a5f23e42b83a7c3b14b3beff9a664 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed wait until bulk operation is completed and searches include it.

01164e1c5d22477b4241ba3d6d1b319db181e935 authored about 9 years ago by Alvaro del Castillo <[email protected]>

[gerrit elk] Fixed dates in review before using them.

3b6f7eba8b603a0db16240553e9a81f3e7e6df02 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed issues with github geolocations upload to ES.

ada20dbb7a60c09d79294f6a688a20a632c3996f authored about 9 years ago by Alvaro del Castillo <[email protected]>

Wait after a bulk operation until all items are returned in searches.

9697d960b92009e67f49b161d973225bc44a24a2 authored about 9 years ago by Alvaro del Castillo <[email protected]>

All backends now are iterators and get items from state in Elastic.

5e7b381257bdcd8e3846966aea1581585f47aaf5 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented Backend iterator using Elastic State.

22b17b6ae0e1aa2975461175baf371681d9d2bf5 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Removed dump state concept from all code.

57ee35594db4cebfeffc968bd51d96fd118143fe authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved use of Elastic _bulk API to use a limited number of items.

6e05209fa17b03ea43912343c0d8839487b2a639 authored about 9 years ago by Alvaro del Castillo <[email protected]>

backend get_field_unique_id is now public. Used in ELK stack.

Improved _bulk API use limiting the max number of items added per iteration.

f907b1fea6bb303d9be5f794a052c1f567c51798 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Use the same item id in all places.

1d787cc31fc3a2ced419d4b16028e0fd62b10416 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Removed not needed code.

bf2b884369dbdcef1b0409bd5f7a6161409c4276 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed error in no_incremental param.

085b5ebfc84c480429f8708ce07f920f4c0eaa75 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Converted to CacheItems

29d661768eff2714a646cea71c1ea9ac163d11e5 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed cache removal code.

e7b3789c7d468ec430b739990924e8005ba0de18 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Converted gerrit to CacheItems.

764771beeb2dbc87412a733185e5bc5561054b38 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fix time measure units in log message.

6a39569d8eb5719fb7f0cdba82cb36dcf5126537 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added CacheItems class which implements the cache for data sources in perceval.

Updated backend and bugzilla to use CacheItems.

077e944a03f174e2af6b095d80054a3ffa4cd759 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented cache item for gerrit.

bee863d61dc68ef9cf3915f34ba0d089400f3b68 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Use bulk API for uploading reviews to Elastic.

b1788e60109e1298fd275094318cf3878f89d188 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Move get_issues_from_cache to a class method.

56fa6e99c84e46dd15379ba323923dd74284ea92 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved formatting.

21498df34c2fb5c3bca897d45880316c5406f436 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented github cache using one cache file per pull request.

ab783e172d66bce95180b39a85fd0001cce32139 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added ETA for github download.

Convert github backend for using last State from Elastic.
Make more robust data enrich conversion.

8fec71173d84e3760fe2b12613ccaa42bf4b0e6b authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented in bugzilla the State storage in ES.

b9ee00cc1d7980b4b4dd1d1a1de80063ab7362c5 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented cache using one cache file per issue.

33179ec9f5f6ea483460e481f848d84e1197aef7 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Use bulk API for uploading issues to ES.

bfb5cfde035f42f9618cfa1d0bdd91a94fbc03ca authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed remove_last_char_from_file location now in utils lib.

35798a1b3e2c610fffbae3cf267cae0d0c6ed848 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Renamed history param to incremental.

Use "state" to refer to last gathering state for a data source.

ab769bf84699d58816ef49d6e3938dc1e83bde1a authored about 9 years ago by Alvaro del Castillo <[email protected]>

Removed getPullRequests. Ussing issue API, not pull request API.

Renamed history to incremental.

60afe5c69af3885dc093297f4320c70c87ddf458 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented command line parsing from data sources params.

Share init logic for cache and history.
Defined data source internal API for cache and history.

12f1cf6b3acedbef75f93e8b8e5b771c2bee0bea authored about 9 years ago by Alvaro del Castillo <[email protected]>

Support getting all reviews without using a project.

Fixed issue uploading reviews to ES.

8eb2bb4b86cf5c0317f6c10310c0f4d5b45c0f60 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented incremental analysis using ES as history reviews storage.

e6bb4182d0dc0b48a1ebba41267416a1614836ae authored about 9 years ago by Alvaro del Castillo <[email protected]>

get_last_date: support timestamp fields and add optional filtering for searching the dates

59f86b1e113f6d47940ebf47e4e97c23a3089ba1 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added ETA for gerrit.

0e273f086bf4bcd4b427103852249a7ae3590e41 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Cache working in gerrit backend.

Limited max number of reviews to control memory usage.
Publishing in ES working.

26560e676a7d48e7d71545584f24d49b3f43e3ff authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented gerrit backend project list and reviews gathering.

996b0c083025e4809cec469cc4420455514acab9 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Raise exception if _get_name() method is not implemented.

b04e0c77e7a7ebd1191c27f40d70ba185ce75b28 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed backend name implementation.

d8a22603f6bc3cc3007362e275d2634a52df111b authored about 9 years ago by Alvaro del Castillo <[email protected]>

Complete refactoring of gerrit to elastic tool.

Created Perceval backend.
Created helper classes for GrimorieELK for SortingHat, Projects, workin...

9bb3144d8e2c60601ecec75aba84cd6613aaacf4 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added .gitignore

7b782300f8f6ff731de8e0c96af36d98533e79a7 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Removed wrong comment.

e6c5da64fbdc0bc0970ea1faeb44d68db80f1f9b authored about 9 years ago by Alvaro del Castillo <[email protected]>

First complete version of github2es using Perceval and Grimoire ELK.

2ce672031eb34ff8022dca64ccd79bc264150a1c authored about 9 years ago by Alvaro del Castillo <[email protected]>

Track only pull requests from issues.

75caf5aa10baa6a11c39ef2a4e1291827304b6e8 authored about 9 years ago by Alvaro del Castillo <[email protected]>

get_time_diff_days to control if params are None.

6aea4408b7a48c8c00740fb2e262c5b15448eec9 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added loading from cache feature.

085c543b974ffaf2cad704d77dcded91fc8beb54 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added debug param.

ffe48f178c71f2ba8531aa8703e8017ac2e3a5f8 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Created new backend github for Perceval.

3d08c1ae3d74deb02cfb211a1d1e482c9ac0a081 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Move ElasticSearch and BugzillaElastic classes to grimoire.elk package.

c4adba5b5191e3b306f79fe5bfcd75aad9ac169d authored about 9 years ago by Alvaro del Castillo <[email protected]>

First version of grimoire.elk borning library.

a419c20071d78864c1a9fc486f8cdbcbfa910f7d authored about 9 years ago by Alvaro del Castillo <[email protected]>

If --cache then activate automatically --no_history.

8f88ba2f8ef08046a0d142616a4d3eea4922cb59 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved login for debug mode.

484dea39949ac8cadf886e428f1be73fc0003c1f authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added option --debug to increase logging.

5a9dad20a71abbddf8a9929b02317de35cc441d7 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved logging.

6e9c4e6dce76150b709c94f62657a76c7b463ed5 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved cache management logic.

5bc7a91131ffbdeec82fa610ab39252c0200e0a0 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fix dates in bugzilla JSON so Kibana detects them correctly.

8bfb38d7b1b40aa31c14bec12c412143fb7cef05 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Just use cache to retrieve changes when use_cache is True.

26701139c22d25f215391980b1dea83893678c58 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Don't modify date fields in Perceval. It will be done later in the client using Perceval.

af3e57bfb7ae5819a0ed3ce792f666436eaa0d6f authored about 9 years ago by Alvaro del Castillo <[email protected]>

Use Elastic Search

78db02fcfd29a25faa069e171bfc56e1db836f5c authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added cache and renamed delete to no_history in params.

e64868e95d562342b16465c9b5917491418c4468 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Completed support for loading changes from cache.

Added more params to Bugzilla constructor to control history and cache usage.

876153dc044cb97585897556df73445cf3680a3e authored about 9 years ago by Alvaro del Castillo <[email protected]>

Cache write now works correcly for CSV, XML and HTML in JSON files.

df668b76d4c177e5743c91d707e2ef37b8a25190 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Cache is now written to files.

0e5b18c03ff4cd6feb3c5aa99e508b71fef8b6da authored about 9 years ago by Alvaro del Castillo <[email protected]>

Implemented restore/dump state from JSON files.

9e0603671501f19c1d8a6bcebc20ef232ebf583b authored about 9 years ago by Alvaro del Castillo <[email protected]>

Refactored to move closer to perceval architecture.

41af2b5984ec2ce9b6781bfb1b3e133aedba19a6 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Big refactoring to avoid using ES in Perseval backend.

Created a new BugzillaElastic class with processing activities to show data in Kibana.

cf15e4fb84147694cac06dd246996ca6fe340b2d authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved API: get_index -> get_elastic_index_name

4ccb3cebc342a383d24f298de4ca105ef44385d9 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Merge branch 'perceval'

909e79ac3e42d3c5c0a4e784fbc36e7500fafda0 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Refactored all code for moving to Perceval.

5bb00ebd433ce6c580d1ff3dfbab842e66f47e0f authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed when no new issues are in bugzilla.

767619199c34750fcef5354ccf98e25ff37d1b72 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Manage error in KDE bugzilla data: same date for last update for the first 10000 issues.

9d9275e44c4803cf8a97a676d5790f5c6e9143eb authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added ETA using total time analysis and time needed per period of anaylisis time done.

2939a6aff398bbefe90c1b4685575c82d6794127 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added ETA time during execution when analyzing issues.

9e7e5e64d71ebfac62e487100dd3894336de91a6 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Improved ES init logic.

56fd9088cb5ea845bde60525f14f78201835f2d1 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Create mappings so component, product and assignee are not indexed.

Fix issue in issues_list incremental analysis.

e2a7f450f5f1784151ff32132ea0291574a30dc0 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fix in howto count issues processed.

1299e480ea057fed00aa2de0fd934014ca0559db authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed issues in incremental analysis. Now working in all detail levels.

e3ccb5bbf5970bb92b17ae0c132a64a350f2e4dd authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed pep8 style issues.

9d45599416dd2fe3a88189c5befaa613e530829b authored about 9 years ago by Alvaro del Castillo <[email protected]>

Removed not needed print message.

67ac818f49c97297d516278deebecce994878c76 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added incremental support using CSV issues list data.

49b1351f23f5686a0b842a109750ba7ad487e42d authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added support for analyzing just the list of issues, the issues or the changes.

Create analysis index in ES also for list of issues.

23243e20aa0193dc33e1651eca3b265fc4da4f51 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added support for iterting in the list of issues to support >10K tickets sites.

Fixed index name when exists "/" in urls.

2401e5fecb5881d4b7f26cd25d0ad329ac83d34a authored about 9 years ago by Alvaro del Castillo <[email protected]>

Added cache use for comments in bugzilla.

Don't remove raw indexes (cache is preserved between executions).
Some methods renaming.

dadb3011c37c48d8265877afe3fee3eead503c01 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Fixed python3 external string data management. Removed unicode conversions inside python.

48b7bae6e5c73131927abfa217912348fc3acb56 authored about 9 years ago by Alvaro del Castillo <[email protected]>

Moved to Python3.

ElasticSearch small refactoring.

56fcce95778fbdbccd5fa77f9f2cc1c547a0b0ce authored about 9 years ago by Alvaro del Castillo <[email protected]>

Converted to PEP8

ff5487d596b9f46133474dbc77052d837cebd991 authored about 9 years ago by Alvaro del Castillo <[email protected]>