Python couchdb script consuming too much memory (getting KILLED) -

-

i'm processing couchdb view has on 10 million rows returned. there way iterate on rows "clean up" memory after i've processed row?

for example, in django can use queryset.iterator() prevent queryset caching helps memory consumption.

this i'm doing:

couch = couchdb.server(url) couch.resource.credentials = (username, password) db = couch[database_name] result = db.view('xxx/xxx', none, stale='update_after', reduce='false') total = 0 row in result.rows:     total += row['value'].num

my actual code more complicated please don't suggest using reduce in couchdb.


Comments

Post a Comment

Popular posts from this blog

java - Suppress Jboss version details from HTTP error response -

-
my jboss verson jboss eap 6.3 when kind of error response sent jboss, adds server information response. following sample row response: http/1.1 400 bad request server: apache-coyote/1.1 content-type: text/html;charset=utf-8 content-length: 2391 date: tue, 19 jul 2016 09:30:46 gmt connection: close jboss web/7.4.8.final-redhat-4 - jbweb000064: error report jbweb000065: http status 400 - org.codehaus.jackson.map.jsonmappingexception: can not construct instance of mybean.type string value 'xyz': value not 1 of declared enum instance names @ [source: org.jboss.resteasy.core.interception.messagebodyreadercontextimpl$inputstreamwrapper@5e421c2b; line: 14, column: 26] (through reference chain: requestargs["request"]) jbweb000309: type jbweb000067: status report jbweb000068: message org.codehaus.jackson.map.jsonmappingexception: can not construct instance of mybean.type string value 'xyz': value not 1 of declared enum instance names @ [source: org.jboss...
Read more

filehandler - java open files not cleaned, even when the process is killed -

-
there java program read files folder, processing , write output files in folder. ran 4 hours , produced few files failed after throwing following error: "too many files" then modified program close (bufferreader , bufferwriter) , started process again. time process failed after starting throwing following error: java.io.filenotfoundexception: <file> (too many open files) then tried running lsof , searched process having no of open files. realized networkmanager. not sure whether should kill process or not. restarted machine. after when ran java program, still got same error. in worst case might possible code still having leaks in terms of not closing everything, still failing after starting process. have feeling lying in system previous run. if start same process in other machine, runs hours before failing. you cannot retain resource between runs of program, if wanted to. leave system without enough resources in theory, though unlikely, restarting...
Read more

gridview - Yii2 DataPorivider $totalSum for a column -

-
i have order , ordersearch models. in order list (actionindex) gridview there filtering , sorting. on of columns in order order_total (sum of products in order). i need implement sum of order_total in gridview. if manually counting activedataprovider->getmodels() array_map spend 3 seconds 3000 orders (localhost). don't want miss time. i see 2 ways make faster: create cache every filter , update lifetime (horrible) the interesting can right in ordersearch->search() method $query method. don't understand how can pass in controller. example of code 2nd way: class ordersearch extends order { public $totalsum; public function search($params) { $query = order::find(); $dataprovider = new activedataprovider([ 'query' => $query, ]); $this->load($params); $this->totalsum = $query->sum('order_total'); // works fast return $dataprovider; } } after i...
Read more