Ecosyste.ms: OpenCollective

An open API service for software projects hosted on Open Collective.

github.com/unigraph-dev/dgraph

Native GraphQL Database with graph backend
https://github.com/unigraph-dev/dgraph

Switch dirtylist with dirtymap, because list causes a merge explosion, when the same posting list gets multiple updates.

baf842c924ff6748e342c9cbd418798c8db23d3b authored about 9 years ago by Manish R Jain <[email protected]>
Add bloom filter to log caches

0e4236711663ecce46376a6f940b204c0b14df60 authored about 9 years ago by Manish R Jain <[email protected]>
Also send header in the log iterator.

0495f7d8384f591ce558c9332614cf472c2e3367 authored about 9 years ago by Manish R Jain <[email protected]>
Clean up periodic cleaning and merging of lists. Now we have a list of dirty posting lists, which can be merged periodically. This is a lot better than iterating over the entire hashmap, and finding dirty lists there. Also, allow for clog to be nil, to make loading fast.

793a515d396ea6e7497eda06680b3dc6127545d0 authored about 9 years ago by Manish R Jain <[email protected]>
Use cache for streaming entries instead of opening files for each posting list. Use iterator instead of channel while streaming entries.

06701f5f89b6c381cc69818158dbd7ee0a85cd12 authored about 9 years ago by Manish R Jain <[email protected]>
Forgot to add the cache file, which supports 1 writer and multiple readers.

7e5ea205e9bdf6fa552d0a62355f62e6eea48218 authored about 9 years ago by Manish R Jain <[email protected]>
Move curfile in it's own struct, and avoid holding global lock in logger as much as possible.

2b6788b919f3f9db7f012923cdb9a82f48b00a06 authored about 9 years ago by Manish R Jain <[email protected]>
Fix the reads being done without mutex locks. Replace with atomics.

0d322c250c4fb73317e1e7eba65e3277f5cdc69f authored about 9 years ago by Manish R Jain <[email protected]>
Add server latency numbers. Modify query to include character names.

f3f7e325d66915dc4517305a6c1625519d7dfb66 authored about 9 years ago by Manish R Jain <[email protected]>
allow setting threshold ram usage via flag. Fix a bug with nil interface conversion.

4c673df4c825d045a569be6123dd625ee1c99e25 authored about 9 years ago by Ubuntu <[email protected]>
Remove hit and miss counters.

e362e550a637bba294139ee761f4a22666093088 authored about 9 years ago by Manish R Jain <[email protected]>
Update README with instructions

d99268ddcdcd1a10c3041c2e13e62f4dcc53050d authored about 9 years ago by Manish R Jain <[email protected]>
Loaded up 21M edges, all of film data + names. And ran a complex query, with a 60ms latency!

ca369823b241dc5c7fc7014b9a320858ed4da504 authored about 9 years ago by Manish R Jain <[email protected]>
Loads of fixes and tweaks to make data loading work. Largely, the system is really fast in inserting data, but memory becomes a problem. So, running continuous memory checks, and if the db exceeds set memory, then stopping in world in batch load mode to deal with it first. In server mode, same code runs but without stopping the world.

879a33290d5ddfdef201d6db4213f5fc9a463d17 authored about 9 years ago by Manish R Jain <[email protected]>
BREAKING changes to map.go

2b6a2e66108cfe4ffc535c92c11f6b1b2b968afc authored about 9 years ago by Manish R Jain <[email protected]>
Using concurrent.map, able to load up 1M edges in under 30 seconds. Memory is the new issue.

471cca3e9dc7e0b6e639cefcbc472571851efe1a authored about 9 years ago by Manish R Jain <[email protected]>
Moved lcache to lmap, a bucket based concurrent hash map. But, the behaviour is still erratic, with 28% of Gets (with inserts) taking over 10us.

3c60a5e67a9e8b15d1c29e781dfa32d2eb697a8b authored about 9 years ago by Manish R Jain <[email protected]>
Use bloom filter in rocksdb. Set counters for posting lists cache hit and miss.

192be4c262c27cd727ae658107c94231abca9689 authored about 9 years ago by Manish R Jain <[email protected]>
Randomize input to avoid contention for RDFs containing the same subject. Don't call assigner clean(), because it hangs the entire process.

6f460e6d61dfb7b1280b7927468ae1f8176bf1bd authored about 9 years ago by Manish R Jain <[email protected]>
Move loader out of server. HandleRDFReader to it's own package.

9fb14d45cba6789ac440ed81344d6800f4e98532 authored about 9 years ago by Manish R Jain <[email protected]>
Handle race condition in UID assignment. Concurrently load rdf data in server.

37039a026bee16de96c70c04571ce42b2baf06d0 authored about 9 years ago by Manish R Jain <[email protected]>
Real world use cases. Allow for dots in names. Handle backspace. Run automatic mutation merging. Load multiple rdf gzips, etc.

e546b0103085ea13def0380741992c86cf9d1c38 authored about 9 years ago by Manish R Jain <[email protected]>
All tests pass with rocksdb store.

f9dddbf97439261283982ee7f7745d75a3aead64 authored about 9 years ago by Manish R Jain <[email protected]>
RocksDB integration done!

a05ecf4aac964f941093f0eb0280ea14cc37879b authored about 9 years ago by Manish R Jain <[email protected]>
Successful import of Rocksdb Cgo, MIT License.

947b1c2d59d0125cc5cb08c8ba6bccf94adbc395 authored about 9 years ago by Manish R Jain <[email protected]>
Remove unnecessary functions.

c0c4e487a8636a3dabd217971b4335087d8f81b2 authored about 9 years ago by Manish R Jain <[email protected]>
Optimized Posting List, yes! AddMutation now largely runs in O(log M + log N), plus potential element shift for mindex. Also reduced complexity.

4f4059e0dd9302bba55f599fbd113f3fb0c1c428 authored about 9 years ago by Manish R Jain <[email protected]>
Move Mutations from store to commit logs.

09c9d98afa083a8bca08dadc46e832c2290aa19a authored about 9 years ago by Manish R Jain <[email protected]>
Ability to stream entries >= min timestamp with a given hash.

24de201f94680aa07679a7e433ad7690e8032ed5 authored about 9 years ago by Manish R Jain <[email protected]>
Log write, rotate and reload works.

ff5374de57608987c5f4b7bafe9cf52aba690c9f authored about 9 years ago by Manish R Jain <[email protected]>
Added periodic sync and file rotation. Appending to existing current file is failing.

74cabf9005fe12a4ad6af89b02b42a76463b4d7c authored about 9 years ago by Manish R Jain <[email protected]>
Add logs, ensure monotonicity, and read back maximum timestamp.

155f82190a41f09cdf6445a8e89da7d252363397 authored about 9 years ago by Manish R Jain <[email protected]>
Start working on an append only commit log.

ef947c7b06aab739cd25c96ed8e6da6faeb398b3 authored about 9 years ago by Manish R Jain <[email protected]>
Add a TODO about language.

d671f4a92c7eb1a19ce08248a036a69737d49517 authored about 9 years ago by Manish R Jain <[email protected]>
Add benchmarks. Change the test data a bit.

9dd2df4cba3a90d0765661e42919ebf1483ba31d authored about 9 years ago by Manish R Jain <[email protected]>
Handle reading rdf data from files. Use underscores for uid and xid, keeping it in sync with internal usage. Output uids as hex.

eb4f0a0c30d88a7a4eda208bf328e09680b870cd authored about 9 years ago by Manish R Jain <[email protected]>
Remove the russian mafia

1addcefab9601c45c268eff9bb5f58b289012878 authored about 9 years ago by Manish R Jain <[email protected]>
It all works!

5c4dba2b70d1983c46c740970fe6729aa9ac13f6 authored about 9 years ago by Manish R Jain <[email protected]>
Welcome to a breathing ToJson. Dgraph has taken it's first small step to mankind.

7ffebae568e488121a4b98db6ac3c70dcfe3f2ce authored about 9 years ago by Manish R Jain <[email protected]>
Make list matrix work, yeah! This would serialize uids for the next query, which would then again return a matrix, so on and so forth.

ecdb53ab359080c5a71ef8c769c09ec311a3b78a authored about 9 years ago by Manish R Jain <[email protected]>
switch uid list to uidmatrix

aeb72365846c5d29fb1e566f82103dc7b26c3ad6 authored about 9 years ago by Manish R Jain <[email protected]>
BREAKING change: convert to JSON. Given we're not tracking uids for each query uid, this will cause problems for the result. So, have to re-structure stuff here.

488b2630af0f429dac162eea63436ef817e830aa authored about 9 years ago by Manish R Jain <[email protected]>
Can add RDFs and run a query.

3749ea4f57f28c76330cc6dc2418b60284baa4d2 authored about 9 years ago by Manish R Jain <[email protected]>
Ability to parse RDF stream. Convert NQuad to DirectedEdge

918a8d912899a5cb0bbd696df01b9d88673aeb3f authored about 9 years ago by Manish R Jain <[email protected]>
Switch naming from Triple to DirectedEdge

f2a1d876a2e638259888cfa116b484633ace4925 authored about 9 years ago by Manish R Jain <[email protected]>
getting started with server

9281dc09b5adaa89f6ee0831bc7a6ed1ad0507b5 authored about 9 years ago by Manish R Jain <[email protected]>
also lex and parse labels

886ff32c78c6f06c7d25d865f6d2f2208b6153d1 authored about 9 years ago by Manish R Jain <[email protected]>
Add tests for parser

00e88fea594441a2e92d26d190765edf3b01f99f authored about 9 years ago by Manish R Jain <[email protected]>
working parser

1d92022e4bfc984ebecc91cc5f60d509e67296fa authored about 9 years ago by Manish R Jain <[email protected]>
Working implementation of nquad lexing.

c1032e50bfa136d91bd446a1c3581c94075550cd authored about 9 years ago by Manish R Jain <[email protected]>
Get cranking on RDF parsing

1dbfa870b9a638ff3a4bba03e337c237f626b9cc authored about 9 years ago by Manish R Jain <[email protected]>
Merge branch 'master' of github.com:dgraph-io/dgraph

3a7419799ceee3f86efcd282be26bdc1c86b3c83 authored about 9 years ago by Manish R Jain <[email protected]>
Move Lexer class out to it's own package so GQL and RDF can share it.

9b1a2f23007121bbbbf480d667179c805f713680 authored about 9 years ago by Manish R Jain <[email protected]>
small typos

7b647f7065bcbcfb88d67823a4ef795526a95aa6 authored about 9 years ago by Manish R Jain <[email protected]>
Set http prefix for url

20f43611d600acd63f6ec179fab0cfa22268d3a9 authored about 9 years ago by Manish R Jain <[email protected]>
Add zerotolaunch.com.au

6f342a33db2f9b5881b280b08c81f198363980a8 authored about 9 years ago by Manish R Jain <[email protected]>
Switch imports to dgraph-io

32c73844d36b703b3962ac70804d06468b19047d authored about 9 years ago by Manish R Jain <[email protected]>
Remove unnecessary stuff

032997f844fcea996e86847808b878ef2f20d12b authored about 9 years ago by Manish R Jain <[email protected]>
small changes

894e9bf5d083cf9e6dfe6df2a3d8a1ea1fe49415 authored about 9 years ago by Manish R Jain <[email protected]>
almost finalized presentation

0b03af34ee45cb0d872eb673edcb6081628c9bff authored about 9 years ago by Manish R Jain <[email protected]>
get started with 5 min presentation

e88fb49c06954b92065377394d8e4221c3551258 authored about 9 years ago by Manish R Jain <[email protected]>
notes about the docs

e25be7e1c31782b59ce20b97a24bf8e79ceb157c authored about 9 years ago by Manish R Jain <[email protected]>
Add parser to generate SubGraph. Should be a joyous moment, don't feel it though :-).

e356b1f4393b814873635254fd9df7b84b9b0001 authored about 9 years ago by Manish R Jain <[email protected]>
Move lexer one level up, we'll use this level for lexer and parser both.

75e6d61edd5961e2d4a507eecfc2523304530694 authored about 9 years ago by Manish R Jain <[email protected]>
Add argument parsing

85adf86a6a74ad89269488e506b69b4e4350b2b0 authored about 9 years ago by Manish R Jain <[email protected]>
fix up left and right curl. Support name, comment and operation types.

1fd88720911c1eb26db036bfad5370066736af2c authored about 9 years ago by Manish R Jain <[email protected]>
Get started on writing a lexer for GraphQL

73b0e41e2254c6b8cb15e47f521066224de69243 authored about 9 years ago by Manish R Jain <[email protected]>
Thorough testing of the SubGraph query system

88a3e1bc60ac7dcdddfefa7375bc686795a8901a authored about 9 years ago by Manish R Jain <[email protected]>
Code for processing SubGraph. Testing needed.

74db725d7452828393d8f493ee149f3d9af2714c authored about 9 years ago by Manish R Jain <[email protected]>
Removed old code from query. Explained how SubGraph approach would work

c9fd7838e95074c878b34195c62ecb5c8ee7aa44 authored about 9 years ago by Manish R Jain <[email protected]>
Test worker thoroughly

b4127a6a6ba44c6f3da7bccf1d8db6fe0fb55796 authored about 9 years ago by Manish R Jain <[email protected]>
Modified posting package to include a way to query multiple posting lists for a given attribute. Introduce a new task.fbs which would be used for query and result encoding.

2f690d21fe267a0dfa2e826a5aaf29c69d9b7ac1 authored about 9 years ago by Manish R Jain <[email protected]>
More thought spent on how query execution would work. Changing things.

9a3472b750bfc03b906ebe19506c56cbdb1eb22d authored about 9 years ago by Manish R Jain <[email protected]>
Target for tomorrow

43145011af57a3c215c0a3a5f31e4e6713cb06b6 authored about 9 years ago by Manish R Jain <[email protected]>
Initial pass at query. Singleton for posting lists.

eea226155109189b971c64995144632cae7f1337 authored about 9 years ago by Manish R Jain <[email protected]>
Make id assignment work, and test it.

64bfe3506fa2c7de2bfe12193e3afd9493442be0 authored about 9 years ago by Manish R Jain <[email protected]>
Allow setting interface{} value to posting list, and test it thoroughly

a7ca241fe88ca0eb3522706ac79e789c46436186 authored about 9 years ago by Manish R Jain <[email protected]>
Add support for interface{} values. Next todo: Add tests.

1ace4574a3890d92dfeae0b9a4124c39eab1244d authored about 9 years ago by Manish R Jain <[email protected]>
remove plist package

80a0a4abee52782158d4c37d9ec567fc2082e5a5 authored about 9 years ago by Manish R Jain <[email protected]>
Works, mutation worksvim list_test.go

39fadeaec283706ce4ef4d9a9abe1d0a927c4db2 authored about 9 years ago by Manish R Jain <[email protected]>
Committing the big example of how mutation layers work over posting list

945d8416ef89b73d3a7b1dcf738db4032e0c75b3 authored about 9 years ago by Manish R Jain <[email protected]>
Overlay mutation chain index over main posting list. TODO: Handle the delete and replace instructions.

44400c9e744b13458664cc7da7d6fcb080ab4573 authored about 9 years ago by Manish R Jain <[email protected]>
Partially add support for external ids. Posting list needs more work

0b9f8b4d192442690189305c9110056de4608e61 authored about 9 years ago by Manish R Jain <[email protected]>
Posting List can store posting lists and mutations in/from different stores.

d47d5c6930a345e825f50eb1629e82c2ae6d3a9c authored about 9 years ago by Manish R Jain <[email protected]>
Add read-write locks to posting list

8469710307ed93e22111693c80c12b64e9300ac1 authored about 9 years ago by Manish R Jain <[email protected]>
Working mutation buffer, with commit back to main buffer

cbf2e445911e12f98d8cd5cfdbd3deabdbe73028 authored about 9 years ago by Manish R Jain <[email protected]>
Going for lunch, working on Posting List buffer regeneration

c859f21281dec8e82024f88232ce282df944401e authored about 9 years ago by Manish R Jain <[email protected]>
PostingList to Flatbuffer and back for AddTriple

0b1ac864fc6acce2228ac66e581a6683b5909ce6 authored about 9 years ago by Manish R Jain <[email protected]>
add store package to allow switch to Rocksdb later. Add license to every file.

9b2fb1245476a1250d1d35733107d9c6fb654073 authored about 9 years ago by Manish R Jain <[email protected]>
Introduce posting lists

508d0e96989933832fe2a4118983f9191c2ef797 authored about 9 years ago by Manish R Jain <[email protected]>
Transform to Flatbuffer + write to and read from leveldb

e12d17e4827a522ca475835de13d5f83ec545453 authored about 9 years ago by Manish R Jain <[email protected]>
spelling

2038c06e7d82351096fc87e130be0cd6e6224d55 authored over 9 years ago by Manish R Jain <[email protected]>
move example to docs

b3f09a3059b8088914fd20d4a80609f02aa025ee authored over 9 years ago by Manish R Jain <[email protected]>
Final draft of MVP before coding begins

741988c98df0fd58c53a41eab0b711ec18749acd authored over 9 years ago by Manish R Jain <[email protected]>
get working on an MVP design doc

67856de013fad75e9093eccb92326674625a8f3b authored over 9 years ago by Manish R Jain <[email protected]>
Understanding Spanner

19808bc67ee237c3c5c295032a6f9c7480dd260a authored over 9 years ago by Manish R Jain <[email protected]>
talk about sharded posting lists

6edf091e6407766ce0e6df664a5dc8bd689eb16d authored over 9 years ago by Manish R Jain <[email protected]>
word about internal repr

e9e1bac0f961c6d5b6b1c089a498a4e36366f481 authored over 9 years ago by Manish R Jain <[email protected]>
continue work on design doc

7b4c4b351c45b87095069b6ed1cdb2ec09296482 authored over 9 years ago by Manish R Jain <[email protected]>
started work on design doc

ed3f074eea617846654bb17709159e0dcf2db6fa authored over 9 years ago by Manish R Jain <[email protected]>