github.com/ooni/pipeline commits | Ecosyste.ms: OpenCollective

Add binario workflow

4cd191e35f0bbcfaa6ba0d4c730e8439201d8cdb authored over 9 years ago

Import traceback

0de2b2c4b7d02e0b640dfe734c8a2a479275d568 authored over 9 years ago

Adjust parallelism. Print the traceback.

d77c9ca9692da6d269980df49fc3513d6eae7753 authored over 9 years ago

Fix bug in sanitisation procedure

19fe34bc2748494df235835d2d9a0f8fa969012e authored over 9 years ago

Print the traceback..

9196365ac78f2f7e3924fabe78528646693fe59f authored over 9 years ago

Carry the exception along..

85e956d344d79c35847f7f114cdbf6de7b1f0fd2 authored over 9 years ago

Add function to skip over misformatted YAML reports

686a2192622a32e446eeaf43c4b3a8df38f8f506 authored over 9 years ago

Debug various problems with the pipeline by running it on the test cluster.

* Need clj in path
* Set all the dependencies
* Wrong topology
* Fix imports and names
* Add src...

72ef69711dbf6b1e77bd83f20e2a5bd5ddf1b207 authored over 9 years ago

Rename to report (singular)

9ada3384338330103beef11d4aa4978f7815b469 authored over 9 years ago

Update README

15ad5a05584c15c7ec815b3f86d430b67b794751 authored over 9 years ago

Add aggregate task chart

b6e504efb15fc0beb5bbaeaa70ef8c472e380dd3 authored over 9 years ago

Refactor pipeline to use a KafkaSpout and 2 Bolts

18e849be6ab0ec5320d56d885f1906c481172d1b authored over 9 years ago

Fix typos

6f73b236b733c23f94572cdc63b360deba66ad27 authored over 9 years ago

Properly move kafka messages

1869221ffee25861c5df000e398c664cf45172e2 authored over 9 years ago

Update project.clj to use the hortonworks repo

Thanks to @dan-blanchard for the swift reply and fix

8df70710a485365a2657285b4c398d33d69860e2 authored over 9 years ago

Also add footer to report bucket

6efdae7d9e2eba85438600ca50401c36246a5355 authored over 9 years ago

It's called nimbus y0

5d27553dc0e3e59f3681dae0a66b7893cbf69e08 authored over 9 years ago

Ack tuples

072ef3014ab59bdef6bf36e0204f15348268391f authored over 9 years ago

Extra argument was missing..

a0745d0acdb0f6d841b01e24a2971c4625e7ffea authored over 9 years ago

Move flush print into function

77d28cfd068ae593953c0a49f27288bc16877cc8 authored over 9 years ago

Update storm version

075ad0c7082c94cc0be3feca7908d1bb8767dfc1 authored over 9 years ago

Also check for timeouts inside of the consume loop

56783f7e0c3093defec2f35ffaef268605afba5f authored over 9 years ago

Clean dead code

7719cd1c04a852d0758d31a0391ba130ffdb1f82 authored over 9 years ago

Properly call consume_messages()

8648abb4854c4df32c67f2cd65959bc368a54d2b authored over 9 years ago

Set the timeout for the kafka consumer and after the timeout is reached check for timed out buckets

26233ce2b1778a64366564b774d07dd327eb47e2 authored over 9 years ago

Implement TimedOutStringIO and refactor logging

12f6489831ef60d40de75cda9646b7a2f99c0026 authored over 9 years ago

Remove too much verbosity

534ff67a453a74f6b3828a62514774d2576d3bcd authored over 9 years ago

Add some debug lines to kafka-consume

01424e83a1c608b1ba7d19c51276a0ee8baf3547 authored over 9 years ago

Cast everything to string

06e50b7bfda265aa7fb3c3ddd2b690f0d3fa786d authored over 9 years ago

Update config and requirements

9db851c92e66b9d82ecec48bca16c6b9b881d5ab authored over 9 years ago

Actually the bolt probably wants more parallelism for serialization

4f7a980a48a1660a550a32acbb203bfc263cb37b authored over 9 years ago

Specify the parallelism of them bolts and sprouts

f1ecd4efe97dbea6a6b63d6ac77ff2f8f70f0b4c authored over 9 years ago

"" is not ''

e004757188ce8a344511e28bc531c3d6f10bfeeb authored over 9 years ago

Last try

64b2184cfdd0ae33dcaf5c4ed1f9bb396e887d3b authored over 9 years ago

Cast to binary type

7787fd24af134e5110d212bcd8155bc62b69fa3c authored over 9 years ago

No encoding, no fun.

5a03b72256544b16a83287efe6923b4694395bb0 authored over 9 years ago

Fix naming of record types

4b6cfdae7a4db5131e9fceb18bad9c9e0db37d57 authored over 9 years ago

Configure production

98780b96171d420a39779a753659cd5b27aa3da1 authored over 9 years ago

Add report topology requirements.txt

7482351b179e62080e402d1bd73fc624662800ec authored over 9 years ago

They call them tuples, but they actually mean lists...

5d824eff0632f24b5adfbb9c3d4a30b2aaf9dfd0 authored over 9 years ago

Emit a 3 tuple

210f2a19a1060f2befabfeda720adf497b0af5e0 authored over 9 years ago

Implement a basic unittest

797838b1cf53ecd32272fd30002a73de7910d691 authored over 9 years ago

Need tell

8c4600b52bf73e77b62a1c374bc2c13e27f546d7 authored over 9 years ago

Actually it's gzip..

cf6067db55781f063cd70afee3c31356eae902d4 authored over 9 years ago

boto works differently..

c2209e3cdf880e429818e5bbeb818846b0bff510 authored over 9 years ago

Move open into init

03c8e6f9476149ab39950536e4b3872dafcb2038 authored over 9 years ago

boto reports don't support the with construct

f1815b01dda59c034441b34c3f3de4f2e2420424 authored over 9 years ago

Fix calling of parse

85427656b8f7b26b26984d2a44fdb734470c7294 authored over 9 years ago

Use a python file for configuration instead of conifg.json

9e0ff18915336e6cd2a345474c412dfa4a393640 authored over 9 years ago

Almost mistake singular and plural

00982a335061f98eba56987a20307e8f256c6856 authored over 9 years ago

It's actually a KafkaBolt

7621a6cf6b4b0c5d5e76bcae581ec91fc027ab7a authored over 9 years ago

Rename report to reports

dd4d594809f3b54bb82a9e16ffdc2f78a09c6e0f authored over 9 years ago

Refactor pipeline to use streamparse

338749b87155feef631dc0f1597a6833a720c38e authored over 9 years ago

Implement finished method.

Keep track of queue of messages that have been processed and flush them when finished

447d5ba291cb8b52fe5973c3efa759f7199ce5a1 authored over 9 years ago

Add some debugging info

e9afcd725f08952278741718d14e5c4fb4bac04c authored over 9 years ago

Fix path

54a1342b0894aa11e9f9152ed1ff286ba8761af3 authored over 9 years ago

Len is on the date bucket StringIO

9c928ae7d40e92f99251b95fb86f0255a8bff2a2 authored over 9 years ago

Prepend a character to identify the type of message

f263a069ded872cfaad7086baf718d094c2ba68e authored over 9 years ago

Use the bucket manager

8221c409f773fe81d5c4ff8301864f5d083b7da4 authored over 9 years ago

Implement bucket based kafka consumer

a142cb7935b00a1b7d2906e4474da1a26e137c56 authored over 9 years ago

Add the bridge_address key when it's missing

8dd3c8b05a714c334ff81c4ef0702acebcf081d7 authored over 9 years ago

Fix missing imports

a3c15a0f5284e575879fe8f84890dadb471ca64a authored over 9 years ago

Disable output and serialise to error message

6c16e1d1269c5c55b8aec4cf2c892fb40d9a9c92 authored over 9 years ago

Fix calling of close

1def874b863da023d87a106a1fa26020e59fa2d4 authored over 9 years ago

For the moment kafka-python seems like the most stable library

a1df40e1a8bb15e6a7aa66a47b765b5db758a1d6 authored over 9 years ago

Fix bug in sanitisers

ab6f17b463b411b46310a7f0cf7b18c662464d49 authored over 9 years ago

Fix references

dbc8af2cbf89f2db8070b55493d0f7f6c97fde65 authored over 9 years ago

Switch back to pykafka

e9f75a31f0520c5bbac6fa0bfbc6ebbd8f5714c6 authored over 9 years ago

Use older API for kafka-python

4770a00ec187d6ca9964b613d6435259b3f7b113 authored over 9 years ago

Implement consumer using kafka-python

29f2226f6c7636a47c2979c3eb45030f071f8731 authored over 9 years ago

Add simple Kafka consumer and producer

35a587e1715445fbef76062a6932b4e3d2d89715 authored over 9 years ago

Add support for publishing topics to kafka

356d8a1cc91537f49458c218afdca1b9c9662925 authored over 9 years ago

parse quickstart (http://streamparse.readthedocs.org/en/latest/quickstart.html)

a30baa5033557bd72ee26be5c1e57b9fb30582c2 authored over 9 years ago

Refactor util and workflow

9b3d3040ba2bd20ed39b1af775979547d9eaa824 authored over 9 years ago

First commit

c2b28ca272d6a265052b2df0ed9f1de38812a73a authored over 9 years ago