Jeremy Carbaugh
f0543842c1
add RemoteFile object for streaming from URL
2011-06-03 12:52:13 -04:00
Jeremy Carbaugh
b47cc70d92
add basic tests for stats filters
2011-06-03 10:56:16 -04:00
Jeremy Carbaugh
2e6723c15e
update stats filters
2011-06-03 10:55:57 -04:00
Jeremy Carbaugh
687679c6e4
stats
2010-08-19 15:18:39 -04:00
Jeremy Carbaugh
2123977895
add CountEmitter and test
2010-07-14 14:54:26 -04:00
Michael Stephens
4c025493b1
improve exceptions
2010-07-01 13:47:22 -04:00
Michael Stephens
e8ce9948de
test run_recipe
2010-06-30 15:31:30 -04:00
Michael Stephens
a0d3010c0c
test for double done() exception
2010-06-30 15:29:05 -04:00
Michael Stephens
636e17d89c
check for run() call on finished recipe
2010-06-30 15:07:21 -04:00
Michael Stephens
e201df8eda
include exception data in error stream output
2010-06-30 14:27:57 -04:00
Michael Stephens
54a99d4d05
added basic error stream support
2010-06-30 14:13:31 -04:00
Michael Stephens
c5770668e0
allow first argument of MongoDBEmitter to be a pymongo.database.Database instance
2010-06-30 11:33:58 -04:00
James Turk
559eb725e6
ConditionalPathFilter
2010-06-28 21:03:30 -04:00
James Turk
18a15022a5
Subrecord with non-modifying keys
2010-06-28 15:47:25 -04:00
James Turk
07f50fb6a1
Subrecord improvement and more tests
...
still has a huge issue in that records *must* have side effects
2010-06-28 15:23:33 -04:00
James Turk
cb29e77a05
initial subrecord filter
2010-06-28 10:43:19 -04:00
Michael Stephens
38096b3903
added LoggingEmitter
2010-06-22 14:19:07 -04:00
Michael Stephens
a08771b3b1
call _process_file on the actual file object instead of the path inside JSONSource
2010-06-18 14:10:24 -04:00
Michael Stephens
07d9725ce4
add validator support
2010-06-18 11:07:40 -04:00
Michael Stephens
aeaa7095cf
added UniqueID filter
2010-06-17 17:52:43 -04:00
Michael Stephens
ce55ad0fcf
added FileSource and JSONSource
2010-06-17 17:06:28 -04:00
Michael Stephens
172af19014
skeleton setup.py
2010-06-17 11:26:13 -04:00
Jeremy Carbaugh
dcb0287a79
handle conversion of None to null
2010-03-09 14:33:54 -05:00
Jeremy Carbaugh
f2d36f0d04
StringFilter called super on UnicodeFilter. fixed that
2010-03-09 14:33:33 -05:00
Jeremy Carbaugh
96e8dd7b45
do not quote non-string values
2010-03-09 14:32:43 -05:00
James Turk
6ffce4cf6e
minor cleanups in emitters
2010-02-21 14:21:19 -05:00
James Turk
a6574ecba7
basic emitter tests
2010-02-21 14:20:48 -05:00
James Turk
13d8d1365b
basic source tests
2010-02-21 13:44:39 -05:00
James Turk
de3155cb06
filter test improvements
2010-02-21 12:01:04 -05:00
James Turk
30cb36e17e
cleanup of basic filters from writing tests
2010-02-21 00:35:56 -05:00
James Turk
079857c961
added first batch of tests for many of the standard filters
2010-02-20 23:55:27 -05:00
James Turk
b607441494
filters are no longer callables
...
filters were originally callables, but with the addition of recipe
as a required parameter code no longer looks like
result = filter(filter2(filter3(data)))
--but instead--
result = filter3(filter3(filter3(data, recipe), recipe), recipe)
this looks cleaner when written like:
data2 = filter.attach(data, recipe)
data3 = filter.attach(data2, recipe1)
result = filter.attach(data3, recipe2)
2010-02-20 14:15:07 -05:00
James Turk
c8991869ba
gitignore
2010-02-20 11:26:46 -05:00
James Turk
1e4f536636
get rid of multithreading/multitasking..
2010-02-20 11:25:55 -05:00
James Turk
5421c12acd
some cleanups discovered via test-splitter.py failure
...
* recipe is now a kwarg to Filter.__call__
* recipe.reject_record
* Splitter works with iterables
2010-02-20 11:14:23 -05:00
James Turk
293c7a0a78
remove dotted keys, a terrible idea
2010-02-19 23:10:50 -05:00
Jeremy Carbaugh
2cde3badad
add keep_records parameter on FieldMerger to prevent it from removing merged fields
2009-10-09 17:28:41 -04:00
Jeremy Carbaugh
dcda3db140
add replace option to sqliteemitter and add better kwargs to dictreader on csvsource
2009-09-01 17:45:05 -04:00
Jeremy Carbaugh
1d05d434b9
update FieldAdder filter to update field if it already exists
2009-08-14 16:48:25 -04:00
Jeremy Carbaugh
dc1d3eab14
separate Unicode and String filters and fix bug in utils.Files
2009-08-06 14:29:22 -04:00
Jeremy Carbaugh
20a7beebab
fix unicode encoding of unicode strings, add delimeter option in CSVSource
2009-07-29 16:51:16 -04:00
Jeremy Carbaugh
2e1480d778
update sqlitesource
2009-07-28 16:44:21 -04:00
Jeremy Carbaugh
71fee9897b
add Files to handle opening and iteration over multiple files
2009-07-23 17:44:18 -04:00
Jeremy Carbaugh
d8c3df88ab
add conn init parameter so user can pass in an existing MongoDB connection
2009-07-22 14:26:39 -04:00
Jeremy Carbaugh
7f2619c7df
add MongoDB and sqlite sources
2009-07-22 14:24:21 -04:00
Jeremy Carbaugh
1c996eab29
remove broken, unused done method in MongoDBEmitter
2009-07-09 13:31:01 -04:00
Jeremy Carbaugh
2b1c2d9d26
add ability to drop_collection on MongoDBEmitter
2009-07-02 18:01:53 -04:00
Jeremy Carbaugh
4680cb8e3d
add UnicodeFilter
2009-07-02 18:01:09 -04:00
Jeremy Carbaugh
6e1cdce54d
add MongoDBEmitter
2009-07-02 12:46:07 -04:00
Jeremy Carbaugh
36a1b5ec67
add extra recipe argument to YieldFilter __call__ method
2009-06-30 15:50:40 -04:00
