James Turk
6ffce4cf6e
minor cleanups in emitters
2010-02-21 14:21:19 -05:00
James Turk
a6574ecba7
basic emitter tests
2010-02-21 14:20:48 -05:00
James Turk
13d8d1365b
basic source tests
2010-02-21 13:44:39 -05:00
James Turk
de3155cb06
filter test improvements
2010-02-21 12:01:04 -05:00
James Turk
30cb36e17e
cleanup of basic filters from writing tests
2010-02-21 00:35:56 -05:00
James Turk
079857c961
added first batch of tests for many of the standard filters
2010-02-20 23:55:27 -05:00
James Turk
b607441494
filters are no longer callables
...
filters were originally callables, but with the addition of recipe
as a required parameter code no longer looks like
result = filter(filter2(filter3(data)))
--but instead--
result = filter3(filter3(filter3(data, recipe), recipe), recipe)
this looks cleaner when written like:
data2 = filter.attach(data, recipe)
data3 = filter.attach(data2, recipe1)
result = filter.attach(data3, recipe2)
2010-02-20 14:15:07 -05:00
James Turk
c8991869ba
gitignore
2010-02-20 11:26:46 -05:00
James Turk
1e4f536636
get rid of multithreading/multitasking..
2010-02-20 11:25:55 -05:00
James Turk
5421c12acd
some cleanups discovered via test-splitter.py failure
...
* recipe is now a kwarg to Filter.__call__
* recipe.reject_record
* Splitter works with iterables
2010-02-20 11:14:23 -05:00
James Turk
293c7a0a78
remove dotted keys, a terrible idea
2010-02-19 23:10:50 -05:00
Jeremy Carbaugh
2cde3badad
add keep_records parameter on FieldMerger to prevent it from removing merged fields
2009-10-09 17:28:41 -04:00
Jeremy Carbaugh
dcda3db140
add replace option to sqliteemitter and add better kwargs to dictreader on csvsource
2009-09-01 17:45:05 -04:00
Jeremy Carbaugh
1d05d434b9
update FieldAdder filter to update field if it already exists
2009-08-14 16:48:25 -04:00
Jeremy Carbaugh
dc1d3eab14
separate Unicode and String filters and fix bug in utils.Files
2009-08-06 14:29:22 -04:00
Jeremy Carbaugh
20a7beebab
fix unicode encoding of unicode strings, add delimeter option in CSVSource
2009-07-29 16:51:16 -04:00
Jeremy Carbaugh
2e1480d778
update sqlitesource
2009-07-28 16:44:21 -04:00
Jeremy Carbaugh
71fee9897b
add Files to handle opening and iteration over multiple files
2009-07-23 17:44:18 -04:00
Jeremy Carbaugh
d8c3df88ab
add conn init parameter so user can pass in an existing MongoDB connection
2009-07-22 14:26:39 -04:00
Jeremy Carbaugh
7f2619c7df
add MongoDB and sqlite sources
2009-07-22 14:24:21 -04:00
Jeremy Carbaugh
1c996eab29
remove broken, unused done method in MongoDBEmitter
2009-07-09 13:31:01 -04:00
Jeremy Carbaugh
2b1c2d9d26
add ability to drop_collection on MongoDBEmitter
2009-07-02 18:01:53 -04:00
Jeremy Carbaugh
4680cb8e3d
add UnicodeFilter
2009-07-02 18:01:09 -04:00
Jeremy Carbaugh
6e1cdce54d
add MongoDBEmitter
2009-07-02 12:46:07 -04:00
Jeremy Carbaugh
36a1b5ec67
add extra recipe argument to YieldFilter __call__ method
2009-06-30 15:50:40 -04:00
Jeremy Carbaugh
86b819bd11
removed stupid chainging stuff. what is this, jQuery?
2009-06-26 23:57:35 -04:00
Jeremy Carbaugh
68cbbdcb54
add support for chaining and nesting of recipes
2009-06-26 23:38:07 -04:00
Jeremy Carbaugh
6f969ac3d4
add recipe object
2009-06-26 17:54:57 -04:00
James Turk
a0b206e711
small changes for cleanup
2009-02-27 19:49:50 +00:00
James Turk
e8da52e486
fixes for addfield overwriting existant fields\n saucebrush sopr example fixed
2008-11-24 01:27:46 +00:00
James Turk
c114d482da
better import script
2008-11-22 00:49:18 +00:00
James Turk
a585fe95f2
str_or_list for FieldFilters
2008-11-20 15:37:45 +00:00
James Turk
aff9f2295c
DictFlattener
2008-11-17 20:53:04 +00:00
James Turk
8d71088e9a
multithreaded/multiprocess approach to filters
2008-11-14 23:17:43 +00:00
James Turk
163d92d44b
DateCleaner, FieldRenamer, etc.
2008-11-13 21:31:42 +00:00
James Turk
07842557fb
things for sopr_lobbyists, including dotted_key_set, flattener, splitter, and name cleaner
2008-11-13 16:41:30 +00:00
James Turk
0bbb1ee4e4
minor pylint cleanups
2008-11-04 21:39:40 +00:00
James Turk
8ead922b4c
added flatten & flattener
2008-11-04 17:20:10 +00:00
James Turk
31ffeaff61
switch filter specification in run_recipe to *args
2008-11-04 17:17:12 +00:00
James Turk
198d5bbd37
kpf in ignore
2008-10-29 20:05:28 +00:00
James Turk
8654f03ba7
saucebrush moving from bzr to svn :(
2008-10-27 19:13:50 +00:00
