github.com/matomo-org/matomo-log-analytics commits

improve regex for ips

e99908da85c33d21cb1b12b8db9854b1b34ed4ad authored over 7 years ago by sgiehl <[email protected]>

match s3 requester as userid

890f9066752e5bcd58174c675f9e5dcb77d8d0f7 authored over 7 years ago by sgiehl <[email protected]>

Allow wildcards for importing logs

https://github.com/piwik/piwik-log-analytics/issues/104

ae7f49dd3e7b5364eb1128df0641d8db6fa4a64c authored over 7 years ago by Seth G <[email protected]>

Mention tests/run_tests.sh script (#174)

177a2953b3087cc72ca9582de9402ca50b2ced5f authored over 7 years ago by Felix Wolfsteller <[email protected]>

Add gitattributes for LF new lines

fixes https://github.com/piwik/piwik-log-analytics/issues/100

e8c3a60e65d3b7d4800702e7676c5790d8ae77a6 authored over 7 years ago by Matthieu Aubry <[email protected]>

You may also import many log files at once by setting log_file to *.log

refs https://github.com/piwik/piwik-log-analytics/issues/104

f0dafa8a0f39129ee890a35ac100823f7b793cfe authored over 7 years ago by Matthieu Aubry <[email protected]>

rename 'nohost' to more explicit 'no-hostname-found-in-log'

refs https://github.com/piwik/piwik-log-analytics/pull/136/files

dc047c9317601f238d92fbb7be0b208fb9af33cc authored over 7 years ago by Matthieu Aubry <[email protected]>

Fixes #126 (#136)

aa03d9ad52d1597ef21595e450fbd57ced7cae07 authored over 7 years ago by Julian Sternberg <[email protected]>

Clarify that * character must be used in --exclude-path to exclude a path matching a pattern (#176)

0ee9383ae5ff0d53a746d5006a08087887fed34e authored over 7 years ago by Matthieu Aubry <[email protected]>

Proposal to include license information via #172. (#175)

Note that I am NOT the copyright owner or have any affiliation with the
piwik project. This co...

1de6039419e916eb61ac2b769653cc50f3ed4589 authored over 7 years ago by Felix Wolfsteller <[email protected]>

Fix nginx acess_log directive (#159)

* Fix nginx acess_log directive

Tested with nginx/1.11.5

* added working rsyslogd setup to...

f724b5d2ae0de60cd4fec2f0eb340481eda3d213 authored almost 8 years ago by Dmitry Verkhoturov <[email protected]>

Fix NGINX related documentation (#168)

* Remove trailing whitespace in README.md

* Fix NGINX related documentation

When logging m...

bd60a20ae09d8b68b081c746f13eadfbf8e7947d authored almost 8 years ago by Ezra Bühler <[email protected]>

user agents must be lowercase

55a410cf6fcd4249a0b6f820c6b88d56ee7cdf2d authored almost 8 years ago by mattab <[email protected]>

Merge pull request #165 from Kdecherf/pingdom

Fix Pingdom bot exclusion

ab997050199311e881b66b99906dbd982c002da6 authored almost 8 years ago by Thomas Steur <[email protected]>

Fix Pingdom bot exclusion

UA from logs are transformed into lowercase before comparison to
excluded UA. Therefore excluded...

777cc271ae281d5076eb62f81670887205de7b98 authored almost 8 years ago by Kevin Decherf <[email protected]>

Log Analytics is compatible with Python 2.7

fb91694c02db82595a10672cc9ad69e95ae12a70 authored almost 8 years ago by mattab <[email protected]>

Revert "Deprecate JSON2 to use JSON " (#161)

8ec5986b2d4d21af9c9ddaa7c40225e1c8a36180 authored about 8 years ago by Matthieu Aubry <[email protected]>

Fix the system test

4c663b001c510f9e719825eef69935895075130c authored about 8 years ago by mattab <[email protected]>

Deprecate JSON2 to use JSON https://github.com/piwik/piwik/issues/8566 (#160)

26049b008b2ce3cbb9e7179a94033db5475ad698 authored about 8 years ago by Matthieu Aubry <[email protected]>

Add pingdom to EXCLUDED_USER_AGENTS list (#156)

Since pingdom is a popular service I thought it might be worth having on the
bot list. I won't ...

eb0132eec6a05e2c15154cf8b1378787183ae8d7 authored about 8 years ago by Karl Goetz <[email protected]>

OVH compliant

78f09c1202bef826613a2246c3945dbc972c3d8d authored about 8 years ago by Charles-Édouard Coste <[email protected]>

Mention correct nginx access log format (#154)

nginx 1.10.0's default access log format can be parsed with `ncsa_extended`. `common_complete` f...

4bdb19479cae3e796d151a30edd031338f7d8d71 authored about 8 years ago by Christian Weiske <[email protected]>

adding option to provide separate url for API (#129)

* adding option to privide separate url for API

* post review changes

9c4f36e66fc9b82a13643aae7d3310bfbfc731f6 authored about 8 years ago by Jan Wroniszewski <[email protected]>

Mention default nginx access log format (#153)

f5e2492ac82325ecd08003f458c05a1f5fde68bc authored about 8 years ago by Christian Weiske <[email protected]>

Clarify error message https://github.com/piwik/piwik-log-analytics/issues/149

e015a5443083d12b5db33062028219837e26fe7e authored about 8 years ago by Matthieu Aubry <[email protected]>

fix nginx syslog link (#152)

a5d8030fe283cf52029a9f3dd718aa232eedc9e2 authored about 8 years ago by Christian Weiske <[email protected]>

add the following extensions: woff2, flac, json, rtf, webm (#147)

2878508f66c10d998c9e4e0bc276988b3755cde1 authored over 8 years ago by Martin G <[email protected]>

Fix IOError: [Errno 29] Illegal seek (#143)

fixes https://github.com/piwik/piwik-log-analytics/issues/142

3553c4023eefa4b21b35ec42e1e9df7158f92a00 authored over 8 years ago by Matthieu Aubry <[email protected]>

Clarify that token_auth must be an admin or super user token.

ca215c2b850c5429f6a62b96777b5f0367065062 authored over 8 years ago by mattab <[email protected]>

Improve doc as per https://github.com/piwik/piwik-log-analytics/issues/46#issuecomment-138386461

f9f780c8f7aec63582b92e5e8660f517357b5cee authored over 8 years ago by Matthieu Aubry <[email protected]>

Markdown

f3d87ea2cac69549b4361b2141f85a723964759c authored over 8 years ago by Matthieu Aubry <[email protected]>

Adds support for ELB log files (#140)

fff8a1a26553f67752ed111914860ca1ebc20c55 authored over 8 years ago by Stefan Giehl <[email protected]>

Tomcat is supported

d9b4d68d5f08c0db5c80e942a63d19ffd9d16f3c authored over 8 years ago by Matthieu Aubry <[email protected]>

Merge pull request #134 from piwik/8458

Make sure to always request all sites etc as in Piwik 3 there will be the default limit of 100 a...

6fd24921b3e9ce4b46861ae12528ad91e85583b3 authored over 8 years ago by Matthieu Aubry <[email protected]>

Make sure to always request all sites etc as in Piwik 3 there will be the default limit of 100 applied eg to SitesManager.getAllSites

refs https://github.com/piwik/piwik/pull/10009/files
refs piwik/piwik#8458

49fc5ca0b19b740172b37f8b8dd0cc6e9916cf79 authored over 8 years ago by Thomas Steur <[email protected]>

Merge pull request #127 from piwik/shoutcast

Adds support for shoutcast

45b09932404359f98bd44b43b09ea162bd0e3ca5 authored almost 9 years ago by Matthieu Aubry <[email protected]>

improved shoutcast detection; added tests

dbb60f431d13d9feeefbc6f38a18f37d219ef70b authored almost 9 years ago by sgiehl <[email protected]>

adds support for shoutcasst

580a0f5cbc299e48f5058a248fc1df92ac11bf68 authored almost 9 years ago by sgiehl <[email protected]>

Merge pull request #125 from piwik/115_common_user

Parse the user in common log formats & derived formats.

f15f7271439380dbc7366aa117d3c8add358c264 authored almost 9 years ago by Matthieu Aubry <[email protected]>

Parse the user in common log formats & derived formats.

3fb96bc836bb9f5a87c7a83463663ba9f5b6a69c authored almost 9 years ago by diosmosis <[email protected]>

Fixes error NameError: global name 'config' is not defined https://github.com/piwik/piwik-log-analytics/pull/122

aa98189f62f890e19d4105af2b87daf24156f0b0 authored almost 9 years ago by mattab <[email protected]>

Merge pull request #122 from sgjava/master

Added basic auth

dd28985908baff68b6afb26ec41d4e25e3950890 authored almost 9 years ago by Thomas Steur <[email protected]>

Added basic auth

d1662c2cdfee56379ad871fd880db4caae0f6ad4 authored almost 9 years ago by sgjava <[email protected]>

Merge pull request #120 from piwik/16_nginx_hex_escape

Fixes #16, convert hex escapes in malformed JSON to unicode escapes to workaround nginx bug.

a07cbf926f6fbc64575c988c1a92cb42a427eea1 authored almost 9 years ago by Benaka <[email protected]>

Fixes #16, convert hex escapes in malformed JSON to unicode escapes to workaround nginx bug.

5e5d54ca829effb07dfc5f65fc99090fa5e99529 authored almost 9 years ago by diosmosis <[email protected]>

Merge pull request #118 from ipsosante/master

Avoid Python stdin buffering

f0d8d9722962275ff72316904d82a35bc78930cb authored about 9 years ago by Matthieu Aubry <[email protected]>

Avoid Python stdin buffering

810a409fa076286f1d3e8ae6aa64600184856113 authored about 9 years ago by Antonin Blanc <[email protected]>

Merge pull request #116 from piwik/ipv6

Add tests for ipv6 parsing and make sure w3c extended format parses ipv6 addresses correctly.

0780305cd82eb32a60415a79b52d9998f3c93d33 authored about 9 years ago by Matthieu Aubry <[email protected]>

Add tests for ipv6 parsing and make sure w3c extended format parses ipv6 addresses correctly.

8293993203b7a1c98a2f6344e6e7fbb0cfde9e6e authored about 9 years ago by diosmosis <[email protected]>

Merge pull request #110 from gregoire-astruc/nginx-0-time-milli

Issue when generation_time_milli is 0

aa600c9a20049db05443c1ebd69f681908702479 authored about 9 years ago by Matthieu Aubry <[email protected]>

Issue when generation_time_milli is 0

Whenever generation_time_milli is 0 ( or more accurately, `0.0000`), the json returned will be i...

3eae5225b45ba550dfe820ed6d0f1b0f07361958 authored about 9 years ago by Grégoire Astruc <[email protected]>

Fix badge link

11957e29646d348ae8ab238ef84cbe23d728cef6 authored about 9 years ago by mattab <[email protected]>

Merge pull request #109 from piwik/mattab-patch-3

Add Travis build badge to Readme + Supported log formats info + Get involved section

e2f0f0e99e92a9dd43d9df41c896b1284d8525ae authored about 9 years ago by Matthieu Aubry <[email protected]>

Supported log formats, review https://github.com/piwik/piwik-log-analytics/pull/109

400ea837f2d251e501d773e070f94d9b00bef6b7 authored about 9 years ago by Matthieu Aubry <[email protected]>

Add Travis build badge to Readme + Supported log formats info + Get involved section

6f0b0330cca48eef548c678b82aa615a5f733956 authored about 9 years ago by Matthieu Aubry <[email protected]>

Update README.md

12ca9102460a48a2ee7457d8994e8e2fb5f75345 authored about 9 years ago by Matthieu Aubry <[email protected]>

Merge pull request #94 from piwik/report_invalid_line_nos

Refs https://github.com/piwik/piwik/pull/8644, use bulk tracking indices to display the lines of...

6b191b07d01d2766106f9e04df3c3082e1c025de authored over 9 years ago by Benaka <[email protected]>

Default response to empty dict if bulk tracking response is invalid JSON.

a30c10280309ca03d9004d753faa431fc4bb36ee authored over 9 years ago by diosmosis <[email protected]>

Refs https://github.com/piwik/piwik/pull/8644, use bulk tracking indices to display the lines of logs that were considered invalid by the Piwik tracker.

e457b7c5a8e2e7a1089711abc4688bb84e2a255c authored over 9 years ago by diosmosis <[email protected]>

Merge pull request #99 from piwik/97

Fixes #97, make sure to always apply site_id if specified via option, even when --replay-trackin...

363a2c4cf3274a2f957a33c4a2f2bc0fb1435ce3 authored over 9 years ago by Benaka <[email protected]>

as suggested unsetting hit.args.idsite as it is determined by resolver

b7afff42992a7da516df0374e25b3dc543f23e13 authored over 9 years ago by Thomas Steur <[email protected]>

make sure to always apply site_id if specified via option, even when --replay-tracking is used

5f48a9934caba210fa73095566c42ddb259ad6a3 authored over 9 years ago by Thomas Steur <[email protected]>

Merge pull request #95 from piwik/travis_tests

Fixes #48, add .travis.yml for running python tests. Uses container based infrastructure.

64ce810f4d3bf010758e95181e2329fd2b5153d6 authored over 9 years ago by Benaka <[email protected]>

Triggering build.

b5002f7101426e326b31864eb80f75f2584bf684 authored over 9 years ago by diosmosis <[email protected]>

Add initial .travis.yml for running python tests.

4504672d4900902375f22816fd8678b29fd5b18c authored over 9 years ago by diosmosis <[email protected]>

Fixes #67, if a host is found within a log line, prepend it to the path instead of the queried main_url so the page url reports are accurate.

1111f60437d1a7ed1051ec0415e878099f185fd2 authored over 9 years ago by diosmosis <[email protected]>

Fixes #83, add --add-download-extensions option to script so users do not have to replicate default list of download extensions on command line in order to add new ones.

bc334fff60ab9619234400e0b0a40b246525a6a0 authored over 9 years ago by diosmosis <[email protected]>

Fixes #72, add file name and line number to error output when tracker responds w/ 400 HTTP code.

c6200a8c3d8ec40d510b3b6662664f3f9e17e5ee authored over 9 years ago by diosmosis <[email protected]>

Fix python tests (failure due to new option value).

a10da28a92ab8ee05fc825a6e68fa7997d16beea authored over 9 years ago by diosmosis <[email protected]>

Merge pull request #93 from piwik/tracker_debug

Fixes #91, add --debug-tracker and --debug-request-limit options to help users and maintainers d...

e8f709d3c50f59ae25cc4fce2a3c25adf66f4386 authored over 9 years ago by Benaka <[email protected]>

Add debug option --debug-request-limit to limit output when used w/ --debug-tracker.

e08986389001a0885627d73095dd9ac23f21698f authored over 9 years ago by diosmosis <[email protected]>

Add --debug-tracker option to help diagnosing tracker issues that occur w/ the log importer.

ee39efff0c475efc5136db50080ea557a99ed042 authored over 9 years ago by diosmosis <[email protected]>

Merge pull request #92 from piwik/date-format-option2

Add --log-date-format option for date format in logs (second PR)

17f6c14be557acf9b4fec023f665b5762852202f authored over 9 years ago by Benaka <[email protected]>

Adding python test for --log-date-format option.

1d8b954d4c8d456a49790441f93b5ece54c956c0 authored over 9 years ago by diosmosis <[email protected]>

Add some more documentation to --log-date-format option and print out exception reason when an invalid date is found for easier debugging.

32052a3aa64415c569e59868f20d788e051a5a16 authored over 9 years ago by diosmosis <[email protected]>

Add option for date format in logs.

Custom log format can be specified with --log-format-regex option.
But there's no way to also sp...

92eb866f54f0918b918446802c46e894b0801752 authored over 9 years ago by Marcello Nuccio <[email protected]>

Improve documentation for --regex-group-to-visit-cvar --regex-group-to-page-cvar #87

2a6f29858e7a80b23ec1601a979976a5353c4914 authored over 9 years ago by mattab <[email protected]>

Improve documentation for --regex-group-to-visit-cvar --regex-group-to-page-cvar

67d6394809671ec36474fca64f443e75a2460e49 authored over 9 years ago by Matthieu Aubry <[email protected]>

Adding CONTRIBUTING.md

390fb901c92c63111b996b3628f3377434bcbea3 authored over 9 years ago by Matthieu Aubry <[email protected]>

Remove logging statement when queue.get() fails since logging won't work there.

cb1ececdfd87663706a5f178a87f12c159db65d4 authored over 9 years ago by diosmosis <[email protected]>

Requires 2.14.0

e59bf0330a262b64c5cde75f55ca13f08c6ea936 authored over 9 years ago by Matthieu Aubry <[email protected]>

Merge pull request #78 from marcenuc/webp-is-static

Add webp to STATIC_EXTENSIONS.

4b25350a93e10e20a741786cae4b87294e2b4a85 authored over 9 years ago by Matthieu Aubry <[email protected]>

Add webp to STATIC_EXTENSIONS.

Even if it is not a very common image format, mod_pagespeed creates
files in WebP format out of ...

c310cc622debf2e84789a61c06a0fe796d670f5d authored over 9 years ago by Marcello Nuccio <[email protected]>

Fixes #74, ignore error when getting configured timeout when sending requests to Piwik in case the config object is not available yet.

0af7113f0768bbf4ce4a0edf977e06cb67eefa2f authored over 9 years ago by diosmosis <[email protected]>

import_logs.py fetch websites should work independantly of URL set to https or http

Fixes #77 refs https://github.com/piwik/piwik/issues/8070

82bba3cb9c3bc107d05ac90692ca7df459f3d503 authored over 9 years ago by mattab <[email protected]>

Fixes #75, make sure thread doesn't fail if shared queue is destroyed (for example, during interpreter shutdown).

9b26cafd609804e2c683f119eae2070d4fbf7ba2 authored over 9 years ago by diosmosis <[email protected]>

Merge pull request #71 from piwik/better_diagnostics

Improve ability to debug w/ various added diagnostic & debug messages/improvements.

fda83e164389f0a904186ef54856ad911d5728a1 authored over 9 years ago by Benaka <[email protected]>

Merge pull request #69 from RMastop/patch-1

Update import_logs.py

f2dbcd240a99bc1c394cf1b78125b701d20b7665 authored over 9 years ago by Matthieu Aubry <[email protected]>

Do not truncate responses received from server when printing error messages, add response to error message if HTTP error exception is caught, and output redirects in debug logs for easier debugging.

37054ee6380b8f9f3e2a13ee186ff514ae9fdc60 authored over 9 years ago by diosmosis <[email protected]>

Update import_logs.py

added ibooks as download extension

3717f6305e0f0c0ad6dc8f66606ebb02bc095621 authored over 9 years ago by RMastop <[email protected]>

Change the retry attempts to 3 by default refs #60

in case users input a wrong URL (404 or 403) it also retries which can take a long time when wai...

1a8463fc6e2737185234e4697d8565899fd72c52 authored over 9 years ago by mattab <[email protected]>

Fixes #57, issue BulkTracking disabled warning if Piwik returns status code 400.

ef6fbd1bd55d1f924e57e26c82be4240573eaa71 authored over 9 years ago by diosmosis <[email protected]>

Refs #68, revert change to log file, tests are in piwik repo.

76d0d37bc64590d830a3c6ab3e815e5249752d96 authored over 9 years ago by diosmosis <[email protected]>

Fixes #68, during replay tracking process log requests w/ HTTP error codes, since we don't care how the Piwik server responded in the past when replaying logs.

e5fe925867c66522367a4650436dab052878f273 authored over 9 years ago by diosmosis <[email protected]>

Fixes #65, add default request timeout to Piwik requests and allow setting timeout via CLI parameter (which will aid in having a quick system test).

83988d6f18ba028fd16099575c1f8243f902891e authored over 9 years ago by diosmosis <[email protected]>

Fixes #56, add 'Processing your log data' section to summary output to inform users about the need for archiving to run.

f281e7cff2d893eebaaefee28d0ebe7a23684937 authored over 9 years ago by diosmosis <[email protected]>

Refs #59, urldecode user_agent group before sending to Piwik for cloudfront logs. Also, includes extra debug logging statement and added groups to AmazonCloudFrontFormat to automatically use that format instead of W3CExtendedFormat when detecting the format.

ab3fad025b5c5a892e71d5063ed64e56a88598f6 authored over 9 years ago by diosmosis <[email protected]>

Refs #60, rename new options to --retry-max-attempts & --retry-delay so they are better named and better associated.

44e5d97d57fd9f821d2696fd47c627fef8ffd2f6 authored over 9 years ago by diosmosis <[email protected]>

Fixes #60, increase max attempts to 10, delay after failure to 10s, add two options to override the defaults for these values, and log request failures so user is informed.

cb3213171b4e9f04cacee7b55f6fc6c54640a59f authored over 9 years ago by diosmosis <[email protected]>

Fixes #63, fixes #62 allow every unknown field in W3C logs to be quoted string or token and allow client IP field to be empty '-'.

770f7758b3285004d88ea0ff9634f6d410199c33 authored over 9 years ago by diosmosis <[email protected]>

Ecosyste.ms: OpenCollective

github.com/matomo-org/matomo-log-analytics