github.com/matomo-org/matomo-log-analytics commits

Merge pull request #58 from piwik/better_whitespace_handling_w3c

Handle extra whitespace in W3C extended logs, including in between comment lines and in Fields: ...

aa2a5593ca2f398371ef592dc22ee5d6d6d5b302 authored over 9 years ago by Benaka <[email protected]>

Handle extra whitespace in W3C extended logs, including in between comment lines and in Fields: line.

5867e4076cf24afd14c4c90a7cb8f40b1ca32e13 authored over 9 years ago by diosmosis <[email protected]>

Do not fail if result of tracker request looks like GIF beacon.

a4d1eea78e6ea52b1b72e63a494f02e9d6f7223c authored over 9 years ago by diosmosis <[email protected]>

Fixes #55, allow spaces in cs(Cookie) field in W3C extended logs. Includes fixes to tests and removal of no longer relevant TODO.

9ce7e998603af1d6fc293d23a9fa3d1cef29eb59 authored over 9 years ago by diosmosis <[email protected]>

Require Piwik 2.6.0+ refs #54

05dba484f1933a710f171de9b169ee03d2407104 authored almost 10 years ago by Matthieu Aubry <[email protected]>

Add .gitignore file.

5afda62fcd79479e17ea3d81a91030593fbdfc45 authored almost 10 years ago by diosmosis <[email protected]>

Refs #7372 Remove unused option invalidate-dates

147626a8956e6cc26ddfb3b01bb77d57f0cbe248 authored almost 10 years ago by mattab <[email protected]>

Merge pull request #7294 from piwik/7228_multiple_spaces_import_logs

match log lines when multiple spaces are between individual log fields.

4617ac8c11f0f38c22a5bab1d82a469d2c7a8288 authored almost 10 years ago by Matthieu Aubry <[email protected]>

Fixes #7300, add option --replay-tracking-expected-tracker-file to allow users to override expected path suffix for replay tracking requests.

ca3c0078b434212731a4b9e951800200f959376a authored almost 10 years ago by diosmosis <[email protected]>

Fixes #7228, match log lines when multiple spaces are between individual log fields. Includes python tests.

edf0fb86586e3724bff14555df2833513e1b8ba7 authored almost 10 years ago by diosmosis <[email protected]>

Add sanity check to python tests for log importer for testing multiple spaces are parsed.

faa7336233acfc5148b11acf5092e91e8155a959 authored almost 10 years ago by diosmosis <[email protected]>

Refs #7151, add tests for new log importer options, fix small bug where userid not set in hit object, fix ignore group option value spliting.

28988932f79ef808d748b6329616b810c4fd4b1f authored almost 10 years ago by diosmosis <[email protected]>

Merge branch 'master' into api_to_cvar_log_importer

Conflicts:
tests/UI/expected-ui-screenshots

4b979486f75efb5cd85617a9f153e939adf6c6f3 authored almost 10 years ago by diosmosis <[email protected]>

Add new --w3c-field-regex option to log importer which allows specifying regex for any w3c extended log field. Can combine this option w/ --regex-group-to-...-cvar options to track any field that the importer doesn't natively recognize.

e7d19e9079b93dd086269fac72bb021c1fc6bc89 authored almost 10 years ago by diosmosis <[email protected]>

Replace --api-to-cvar params with --regex-group-to-cvar options in log importer for greater flexibility. Added an --ignore-groups parameter to allow ignoring certain parsed groups (like userid). Also added available regex groups to log-format-regex documentation (not added in generic way unfortunately).

472061cf00318f6d84e886d05ed72d2e6740323d authored almost 10 years ago by diosmosis <[email protected]>

Improved readme structure + added call to contributors! linked to #7163

3ac83a39825fb69ab6bf793ce6dc29e518130bd2 authored almost 10 years ago by Matthieu Aubry <[email protected]>

Add option to map API tracking parameter to custom variable in log importer script. Includes change to ImportLogs system test and fix for import log custom variable handling (old behavior was to not set multiple custom variables if they were set already instead of simply adding custom variables).

1b6f346fc99f27f6f3e4dc958c1cd692df9a4f73 authored almost 10 years ago by diosmosis <[email protected]>

Update import_logs.py

9657e9d6f9b54e98cb53e3f99e68961c5bd5faae authored almost 10 years ago by ledahulevogyre <[email protected]>

Update import_logs.py

3e8d0a73676d8c084f1dd937aea86639716171d8 authored almost 10 years ago by ledahulevogyre <[email protected]>

Fixes #6982, check result of bulk tracking request to make sure the request succeeded and fail if it does not succeed. Avoids silent failures in log importer when BulkTracking plugin is disabled.

2cba73e9ab84eb3d532c1a41c63b3617651ed6ff authored almost 10 years ago by diosmosis <[email protected]>

Add --dump-log-regex option to log importer for debugging purposes. Using the option forces the log importer to print out the regex used to parse log lines and exit.

93067d49e14510d1e38e60750d82113db844d0f8 authored almost 10 years ago by diosmosis <[email protected]>

Fixes #7045 Add parameter --title-category-delimiter to log importer

e39cbdf54653ecb090253503b34edca5d223a2d9 authored almost 10 years ago by Timo Besenreuther <[email protected]>

Refs #5248, add support for bw_bytes query parameter in log importer.

9c2c9c434007fe4d34ccae0048e9fc8eec41c4a5 authored almost 10 years ago by diosmosis <[email protected]>

merged conflic resolved

13f867ab6b08516b2a319d21dded9d8d4c691980 authored almost 10 years ago by flora edwards <[email protected]>

merged conflict

b9e5ee82d9d3c0aabd939f5a5d21e032e7393337 authored almost 10 years ago by flora edwards <[email protected]>

Refs #6968, add --w3c-fields option to log importer so log files in W3C extended log file format can be imported from stdin w/o a '#Fields:' line being present.

4f555c920cccc2e8c9572db1bb033ce805bae3a7 authored almost 10 years ago by diosmosis <[email protected]>

Tweak to log importer console help output.

ac81d4419e32d8edad28caeeb203c8b8620e107a authored almost 10 years ago by diosmosis <[email protected]>

Refs #6968, add --w3c-fields option to log importer so log files in W3C extended log file format can be imported from stdin w/o a '#Fields:' line being present.

f24c8bd39fea0cbf0806852ed1b4a3120899a0f7 authored almost 10 years ago by diosmosis <[email protected]>

Fixes #6968, support parsing W3C extended log files from stdin in log importer by refactor W3cExtendedFormat class so it does not seek when creating the regex used when parsing logs. Also make sure to initialize the format class when --log-format-name=w3c_extended is used. Includes fixes to python tests.

f2e264c81a6530bfd771da55c4aa4e04555f7487 authored almost 10 years ago by diosmosis <[email protected]>

Fixes #6919, do not check for host in regex group in log importer if format has no regex. Also add nginx json log to log importer system test.

b4b7159f5321f61c9286f65b59de9d689d649380 authored almost 10 years ago by diosmosis <[email protected]>

Submodules

8ce5b1584c6fac5b86a98de001010b00492c74e7 authored almost 10 years ago by mattab <[email protected]>

When importing visits on a day different from the visits day, invalidate the archived reports

71f63391cb7f9ecf1d8881992fcc242547d1d16e authored about 10 years ago by Thomas Steur <[email protected]>

Refs #6795, remove quotes around referrer & host if found when parsing hits in log importer.

28f53e2b08fbeb10350fd8774f0c028efc577189 authored about 10 years ago by diosmosis <[email protected]>

Remove debugging change in import_logs.py.

639a1732a27def1ae7f22fac5c81f5723a532389 authored about 10 years ago by diosmosis <[email protected]>

Fixing build for merge and move IIS check for --w3c-time-taken-milli to Parser.check_format so only one warning outputted.

f832ba589ccf9068fbf9aba0f35a6ba486e6bb29 authored about 10 years ago by diosmosis <[email protected]>

Adding support for parsing amazon cloudfront log files in log importer & making W3cExtendedFormat treat time-taken field as seconds by default (since that is what the W3C format specifies). Also add support for event tracking and cloudfront RTMP logs (which provide event logs). Also allow multiple spaces in between W3C extended log files. Allow derived classes of W3cExtendedFormat to define new custom field regexes.

4e0c36390b12fc03d99ee4e64cc4a7f9e282294f authored about 10 years ago by diosmosis <[email protected]>

Initial commit, differences in ImportLogsTest.

ee9064d7ec6f3dfe01671816d65bcc1997051bca authored about 10 years ago by diosmosis <[email protected]>

Change IIS log format in log importer script to be more generic and format any log in W3C extended log format. Keeps iis format for backwards compatibility.

4131abe48420230aa9c803f2f97834b00b3ee02d authored about 10 years ago by diosmosis <[email protected]>

Refs #4707, add note in log importer README about netscaler log files.

df52e21e7937d14840d69f58bb1c7af2b1743bf0 authored about 10 years ago by diosmosis <[email protected]>

Refs #6388, map IIS time-taken field to gneration time in log importer.

4bf9d49c3f8bc7870f709497d91340fe3344788d authored about 10 years ago by diosmosis <[email protected]>

Add netscaler logs to python tests for log importer.

3dffccd9a262c3f5c2387cf67909263636076a40 authored about 10 years ago by diosmosis <[email protected]>

Refs #5980, parse cs-username variable in IIS logs and use as userid when present and not anonymous. Includes changes to ImportLogsTest.

8eb6fb8a8cd3c982cb4f9318c8b36847fbc718d3 authored about 10 years ago by diosmosis <[email protected]>

Refs #4707, add test for netscaler logs, support already present due to last commit.

5b6bed51b42717efe704fdc38a253727c9b45a07 authored about 10 years ago by diosmosis <[email protected]>

Refs #6795, add --iis-map-field option to allow users of log importer script to parse log files created using the IIS Advanced Logging module. Includes tests for such a case (including python unit test + PHP System test).

932b1c815dfd635990676643a869b18e018082dc authored about 10 years ago by diosmosis <[email protected]>

More clear output in case one log file could not be found

(cherry picked from commit c824a2e)

74637a9442e789ed2d0c57d739b7b9c6a5aed6e5 authored about 10 years ago by mattab <[email protected]>

refs #4936 on IIS when no query string it shows as '-' so we assume '' instead.

5c6decdbbd7eefa42440a056ede606c96b334a10 authored about 10 years ago by mattab <[email protected]>

Fixes #4936 on IIS when no query string it shows as '-' so we assume '' instead.

54cad39d4c3c584e51454ab3e2159b64727558a9 authored about 10 years ago by mattab <[email protected]>

Fixes #6664, when replaying logs in log importer, use extra information present in Piwik URLs to determine visitor ID when distributing hits among recorders. If IP is not good enough indicator of visitor, this will increase accuracy of replayed logs.

0122671b68caf794b8634166a6ff9d1ad5ef9992 authored about 10 years ago by diosmosis <[email protected]>

Make updatetoken.php script work in single codebase with --piwik-domain

fixes https://github.com/PiwikPRO/plugin-EnterpriseAdmin/issues/4 and #6584

33988643c932a0b4c269dc31ff881dfd7326bb73 authored about 10 years ago by mattab <[email protected]>

Make command to type help message more accurate

976e17516df2b0fa24de6cf30f1ed50a6a0e2282 authored about 10 years ago by mattab <[email protected]>

Small fix to log analytics readme

97700a137cd050d1a2b2345fe3fbaebc92166194 authored about 10 years ago by mattab <[email protected]>

Fix import_logs.py apk downloads (#6276).

81cd3f6068a2ec62f3481a232bc1691f787c9909 authored about 10 years ago by drebs <[email protected]>

Improve parameter comment

11a632c405fe0b6504f1fa4f9a6a98538e472d1a authored over 10 years ago by mattab <[email protected]>

refs #6231 When a line is for a file with a known download extension and the user has specified --download-extension,

then any file with an extension not in the list will see its log line skipped

0a8b62fb57feae2ea800c69ac5e571021638734b authored over 10 years ago by mattab <[email protected]>

Add the ability to override the default download extensions.

93cd3377e7627aae8c17a9befaf8e7140a1c0ce0 authored over 10 years ago by John Kelly <[email protected]>

Update README.md

the right pull request

d0141e109733dc555f494540c384916f41027df8 authored over 10 years ago by ahattouti-canaltp <[email protected]>

Move JSON format bug fixes to new JSON2 format so existing clients will not fail. Add deprecation note for JSON format and use JSON2 in future proof code (angularjs/log importer).

2bd416bc46d7856ae9ba3f744818767c8e3508dc authored over 10 years ago by diosmosis <[email protected]>

Fixing build for Json array rendering bug.

663fb6deb340e3dcc34e7f25bf46f8ea1e98e16d authored over 10 years ago by diosmosis <[email protected]>

Clarify doc for --exclude-path

as suggested in http://forum.piwik.org/read.php?2,118870

adb14ea02b2ab21142f0ac9ac24fb5faa22e627c authored over 10 years ago by mattab <[email protected]>

Revert the date cache feature as it somehow (cause: unknown) causing pageviews to be lost when importing big log files.

This particular log file I'm testing on is for an intranet with thousands times the same IP addre...

b148679421151a03326cb85396a7b6b0c26134ff authored over 10 years ago by mattab <[email protected]>

reduced multiple empty lines to exactly one

e930303de414715c15585907de3b9f177bfe03bf authored over 10 years ago by Christian Raue <[email protected]>

removed lots of trailing whitespace

e9fb70385973f73fd28a72d9eec4edad4199e7ec authored over 10 years ago by Christian Raue <[email protected]>

Refs #5273 Replace other Trac URLs with Github URLs

0e582485b1af0bf429f13032f69dca45add344be authored over 10 years ago by mattab <[email protected]>

Piwik is a free/libre analytics platform. Refs #4455 GNU Package requirement to use free/libre instead of open source

6551caea241b5123b2d1539b14c6567d8f7c2850 authored over 10 years ago by mattab <[email protected]>

tweak message for usability

a865448101e0bd6665bd1286f418b3cea03bb0e6 authored over 10 years ago by mattab <[email protected]>

remove redundant exclusion check

check_methods are called twice for each hit. The first ones are sufficient to
decide if the hit ...

cb9e7da93d8e688cc490cf199f7a0de29590f16e authored over 10 years ago by Cyril Bonté <[email protected]>

add the date in the Hit attributes, forgotten during the merge of a work branch

bb28398a0d80bea07ef380cafbc01d38161831c2 authored over 10 years ago by Cyril Bonté <[email protected]>

preserve compatibility with python 2.6

cbay reported that set comprehension was available only in python 2.7+.
This patch fixes the syn...

a60a041dd1ecebcfc4bdaaa01f21a3c997a168c1 authored over 10 years ago by Cyril Bonté <[email protected]>

silently fail if OrderedDict is not available for python 2.6

42e95a0c8972324e270087284dc4245060303557 authored over 10 years ago by Cyril Bonté <[email protected]>

use a tuple ase the cache key instead of string concatenation

As suggested by cbay, the cache key can be a tuple instead of a string
concatenation.

aeb69e5f15959146d4d72d7e843ccae89fa47669 authored over 10 years ago by Cyril Bonté <[email protected]>

add some documentation about PyPi OrderedDict

0ba944ca0a4274f926b1b7eafb1717d3346a95ab authored over 10 years ago by Cyril Bonté <[email protected]>

disable cache when OrderedDict is not available

Fallback to a non cached dates when OrderedDict is not available.
It can occur with python < 2.7...

0810a6546a749078e47d2027eedcf20734f65625 authored over 10 years ago by Cyril Bonté <[email protected]>

fix date key computation when timezone is missing

4eef419d1827e2b0af6f2616ea6c822d7af40056 authored over 10 years ago by Cyril Bonté <[email protected]>

compute data only if not in dry-run

992e9b624aa409478734af749c0b0f3f3bc19bfa authored over 10 years ago by Cyril Bonté <[email protected]>

use sets data structures to optimize lookups

d9e18fe42b84aec3b09b4f62a863220aaa1835c4 authored over 10 years ago by Cyril Bonté <[email protected]>

add a cache for parsed dates

e91187132a4745f451421a64cfd7999c1951af94 authored over 10 years ago by Cyril Bonté <[email protected]>

retrieve the path extension only once

25ae3f01f85f8b4aae56dee7012229e73d49532f authored over 10 years ago by Cyril Bonté <[email protected]>

Fix typo

b4ef6913f1b064489eefcc1d5103584b6e224fc3 authored over 10 years ago by mattab <[email protected]>

import_logs give a IOError: [Errno 29] Illegal seek when receive log from pipe because

we can't seek stdin in pipe mode

ef5011086c3fa4231ac8dedd7d6a6bc849c3899b authored over 10 years ago by etmatrix <etmatrix@etmatrix>

Refs #4946 Fix typo + tests

8ae85fbc2867a539e15b84bc221b2b473f87b55f authored over 10 years ago by mattab <[email protected]>

Fixes #4946

* Set idaction_name to 0 instead of NULL. This should fix the error of requests not recorded.
*...

436a33ddceaa7f76b823f023ee50190d8b83d1d4 authored over 10 years ago by mattab <[email protected]>

refs #4946 commiting patch from forum post which makes sure to always have an action_name and therefore always have a title. From what I can see so far this does not fix the actual issue as I am still not able to import data from yesterday

1cc8375ae7ae44e45408d61b7fa750a36007c202 authored over 10 years ago by Thomas Steur <[email protected]>

2.1.1-b3

e701db9e6e1298f1a33259a8ff8b7dc9ae134b31 authored over 10 years ago by mattab <[email protected]>

Updating comment to reference to ./console core:archive instead of archive.php

a0bab500b968c6195518ca3bea45ec27c463e03c authored over 10 years ago by mattab <[email protected]>

fixes #4934 should make it work on windows if php.exe is in path

057e3ecbb48f5fbe9a5493135c915248d7f6600c authored over 10 years ago by Thomas Steur <[email protected]>

Slightly better error login in log imports

3d9270da609472f80883e186933fda09eeee09cd authored over 10 years ago by mattab <[email protected]>

Fix variable name in import_logs.py

b38c00deaceef9c211e9d58489fd691e31d8b0a8 authored over 10 years ago by Bartosz Grzyś <[email protected]>

Add log output when line invalid and replaying log

7bda9ee09a43e41821d306051b1cc57bb1e38a46 authored almost 11 years ago by mattab <[email protected]>

Refs #4786 Adding forgotten white space.

Cheers for the follow up!

e96d6dee83c4afb2c02ced85e166eb0cdb6fa9ac authored almost 11 years ago by mattab <[email protected]>

Fixes #4817 Restoring feature to import logs by specifying --login and --password

54d6bcd7aab8ed1590e56d3a4c5a2a2ed279449b authored almost 11 years ago by mattab <[email protected]>

Fixes #4786 Adding ebook formats to piwik.js download tracking and log importer

03e1d419c1198157f2b5ea6a2b372467de3a01ce authored almost 11 years ago by mattab <[email protected]>

Fix import_logs python tests.

3e9a604b9a778465304ec3e56648def511ea1e3a authored almost 11 years ago by diosmosis <[email protected]>

Refs #4670, remove 2.7 header requirement.

675aae035aad18c8118306129545ff4168383b4d authored almost 11 years ago by diosmosis <[email protected]>

Fixes #4670, use popen in import_logs.py to run updatetokenfile.php instead of check_output.

8cbe81d14435b168e729896fcde8d1cc24ba6e63 authored almost 11 years ago by diosmosis <[email protected]>

Adding test log file

61602e403baa6c50490b9aa3662e072b4f108bc3 authored almost 11 years ago by mattab <[email protected]>

Adding log file sample for icecast format refs PR #65

thanks for sending these

bf34249c28da917a9278deb0d87bdde9ffb2434e authored almost 11 years ago by mattab <[email protected]>

Prefixing log file with directory

14c29afed6668e7e5ffb5fbabe6cd24271635df2 authored almost 11 years ago by mattab <[email protected]>

Refs #4670 Document python requirement in the file comment header

5b403593534f6a4213d0800ed48e174f89a85488 authored almost 11 years ago by mattab <[email protected]>

Fixes #4628 --enable-bots now works as expected + our importLogs integration test shows that this wasn't working before.

14e954b4fcb56abc0ff79c8abed380d6e0450b2d authored almost 11 years ago by mattab <[email protected]>

beta8 needed with bug fixes

df06a1e1491b69f832c121f3cd1fc66d21895cd7 authored almost 11 years ago by mattab <[email protected]>

Ecosyste.ms: OpenCollective

github.com/matomo-org/matomo-log-analytics