Michael Stephens
54a99d4d05
added basic error stream support
2010-06-30 14:13:31 -04:00
Michael Stephens
c5770668e0
allow first argument of MongoDBEmitter to be a pymongo.database.Database instance
2010-06-30 11:33:58 -04:00
James Turk
559eb725e6
ConditionalPathFilter
2010-06-28 21:03:30 -04:00
James Turk
18a15022a5
Subrecord with non-modifying keys
2010-06-28 15:47:25 -04:00
James Turk
07f50fb6a1
Subrecord improvement and more tests
...
still has a huge issue in that records *must* have side effects
2010-06-28 15:23:33 -04:00
James Turk
cb29e77a05
initial subrecord filter
2010-06-28 10:43:19 -04:00
Michael Stephens
38096b3903
added LoggingEmitter
2010-06-22 14:19:07 -04:00
Michael Stephens
a08771b3b1
call _process_file on the actual file object instead of the path inside JSONSource
2010-06-18 14:10:24 -04:00
Michael Stephens
07d9725ce4
add validator support
2010-06-18 11:07:40 -04:00
Michael Stephens
aeaa7095cf
added UniqueID filter
2010-06-17 17:52:43 -04:00
Michael Stephens
ce55ad0fcf
added FileSource and JSONSource
2010-06-17 17:06:28 -04:00
Michael Stephens
172af19014
skeleton setup.py
2010-06-17 11:26:13 -04:00
Jeremy Carbaugh
dcb0287a79
handle conversion of None to null
2010-03-09 14:33:54 -05:00
Jeremy Carbaugh
f2d36f0d04
StringFilter called super on UnicodeFilter. fixed that
2010-03-09 14:33:33 -05:00
Jeremy Carbaugh
96e8dd7b45
do not quote non-string values
2010-03-09 14:32:43 -05:00
James Turk
6ffce4cf6e
minor cleanups in emitters
2010-02-21 14:21:19 -05:00
James Turk
a6574ecba7
basic emitter tests
2010-02-21 14:20:48 -05:00
James Turk
13d8d1365b
basic source tests
2010-02-21 13:44:39 -05:00
James Turk
de3155cb06
filter test improvements
2010-02-21 12:01:04 -05:00
James Turk
30cb36e17e
cleanup of basic filters from writing tests
2010-02-21 00:35:56 -05:00
James Turk
079857c961
added first batch of tests for many of the standard filters
2010-02-20 23:55:27 -05:00
James Turk
b607441494
filters are no longer callables
...
filters were originally callables, but with the addition of recipe
as a required parameter code no longer looks like
result = filter(filter2(filter3(data)))
--but instead--
result = filter3(filter3(filter3(data, recipe), recipe), recipe)
this looks cleaner when written like:
data2 = filter.attach(data, recipe)
data3 = filter.attach(data2, recipe1)
result = filter.attach(data3, recipe2)
2010-02-20 14:15:07 -05:00
James Turk
c8991869ba
gitignore
2010-02-20 11:26:46 -05:00
James Turk
1e4f536636
get rid of multithreading/multitasking..
2010-02-20 11:25:55 -05:00
James Turk
5421c12acd
some cleanups discovered via test-splitter.py failure
...
* recipe is now a kwarg to Filter.__call__
* recipe.reject_record
* Splitter works with iterables
2010-02-20 11:14:23 -05:00
James Turk
293c7a0a78
remove dotted keys, a terrible idea
2010-02-19 23:10:50 -05:00
Jeremy Carbaugh
2cde3badad
add keep_records parameter on FieldMerger to prevent it from removing merged fields
2009-10-09 17:28:41 -04:00
Jeremy Carbaugh
dcda3db140
add replace option to sqliteemitter and add better kwargs to dictreader on csvsource
2009-09-01 17:45:05 -04:00
Jeremy Carbaugh
1d05d434b9
update FieldAdder filter to update field if it already exists
2009-08-14 16:48:25 -04:00
Jeremy Carbaugh
dc1d3eab14
separate Unicode and String filters and fix bug in utils.Files
2009-08-06 14:29:22 -04:00
Jeremy Carbaugh
20a7beebab
fix unicode encoding of unicode strings, add delimeter option in CSVSource
2009-07-29 16:51:16 -04:00
Jeremy Carbaugh
2e1480d778
update sqlitesource
2009-07-28 16:44:21 -04:00
Jeremy Carbaugh
71fee9897b
add Files to handle opening and iteration over multiple files
2009-07-23 17:44:18 -04:00
Jeremy Carbaugh
d8c3df88ab
add conn init parameter so user can pass in an existing MongoDB connection
2009-07-22 14:26:39 -04:00
Jeremy Carbaugh
7f2619c7df
add MongoDB and sqlite sources
2009-07-22 14:24:21 -04:00
Jeremy Carbaugh
1c996eab29
remove broken, unused done method in MongoDBEmitter
2009-07-09 13:31:01 -04:00
Jeremy Carbaugh
2b1c2d9d26
add ability to drop_collection on MongoDBEmitter
2009-07-02 18:01:53 -04:00
Jeremy Carbaugh
4680cb8e3d
add UnicodeFilter
2009-07-02 18:01:09 -04:00
Jeremy Carbaugh
6e1cdce54d
add MongoDBEmitter
2009-07-02 12:46:07 -04:00
Jeremy Carbaugh
36a1b5ec67
add extra recipe argument to YieldFilter __call__ method
2009-06-30 15:50:40 -04:00
Jeremy Carbaugh
86b819bd11
removed stupid chainging stuff. what is this, jQuery?
2009-06-26 23:57:35 -04:00
Jeremy Carbaugh
68cbbdcb54
add support for chaining and nesting of recipes
2009-06-26 23:38:07 -04:00
Jeremy Carbaugh
6f969ac3d4
add recipe object
2009-06-26 17:54:57 -04:00
James Turk
a0b206e711
small changes for cleanup
2009-02-27 19:49:50 +00:00
James Turk
e8da52e486
fixes for addfield overwriting existant fields\n saucebrush sopr example fixed
2008-11-24 01:27:46 +00:00
James Turk
c114d482da
better import script
2008-11-22 00:49:18 +00:00
James Turk
a585fe95f2
str_or_list for FieldFilters
2008-11-20 15:37:45 +00:00
James Turk
aff9f2295c
DictFlattener
2008-11-17 20:53:04 +00:00
James Turk
8d71088e9a
multithreaded/multiprocess approach to filters
2008-11-14 23:17:43 +00:00
James Turk
163d92d44b
DateCleaner, FieldRenamer, etc.
2008-11-13 21:31:42 +00:00
