python - Force multiprocessing.Array to be created in memory instead of on disk -

-

i using python 3.4 on linux. want create array in ram shared processes spawned multiprocessing module (i.e. shared memory). according documentation, should possible using multiprocessing.array. when use

array = multiprocessing.array('i', n)

python creates file in /tmp, zeroes it, , uses through mmap.mmap() shared memory (i have verified looking /usr/lib/python3.4/multiprocessing/heap.py). however, array created in memory, not on disk. reason in use case, n large (more 100 gb), have lot of ram low disk capacity. creation of multiprocessing.array() ends ioerror: no space left on device because uses file on disk shared memory rather ram.

i able around mounting directory via tmpfs , setting tempfile.tempdir point directory. however, there easier way? why python creating shared memory in way?

the short answer no. long answer, read this:

enter link description here

you can use os.fork() pass copy of parent's memory child can't share address space.


Comments

Post a Comment

Popular posts from this blog

Sass watch command compiles .scss files before full sftp upload -

-
i use winscp download , edit .scss files , sass on linux (on server) compile them .css . after saving file, use: sass scss/style.scss css/style.css , replaces css file compiled sccs. the problem i want skip part return command line after editing scss file, sake of automation , saving time. but, if use watch command: sass --watch scss/style.scss:css/style.css synchronize 2 files, alerts nonexistent css errors pop up: change detected to: scss/style.scss error scss/style.scss (line 232: invalid css after "...ht: bold; line-": expected "{", "") note that when file uploads quicker usual (sometimes happens) watch command job, no errors. this because, when file upload takes time, sass compiles scss file (on server) before uploaded remote folder. thus, compiles part of file, resulting in css errors. is there way set timer watch command waits few seconds after detects changes , before compiling? any other way overcome is, of course, ...
Read more

filehandler - java open files not cleaned, even when the process is killed -

-
there java program read files folder, processing , write output files in folder. ran 4 hours , produced few files failed after throwing following error: "too many files" then modified program close (bufferreader , bufferwriter) , started process again. time process failed after starting throwing following error: java.io.filenotfoundexception: <file> (too many open files) then tried running lsof , searched process having no of open files. realized networkmanager. not sure whether should kill process or not. restarted machine. after when ran java program, still got same error. in worst case might possible code still having leaks in terms of not closing everything, still failing after starting process. have feeling lying in system previous run. if start same process in other machine, runs hours before failing. you cannot retain resource between runs of program, if wanted to. leave system without enough resources in theory, though unlikely, restarting...
Read more

gridview - Yii2 DataPorivider $totalSum for a column -

-
i have order , ordersearch models. in order list (actionindex) gridview there filtering , sorting. on of columns in order order_total (sum of products in order). i need implement sum of order_total in gridview. if manually counting activedataprovider->getmodels() array_map spend 3 seconds 3000 orders (localhost). don't want miss time. i see 2 ways make faster: create cache every filter , update lifetime (horrible) the interesting can right in ordersearch->search() method $query method. don't understand how can pass in controller. example of code 2nd way: class ordersearch extends order { public $totalsum; public function search($params) { $query = order::find(); $dataprovider = new activedataprovider([ 'query' => $query, ]); $this->load($params); $this->totalsum = $query->sum('order_total'); // works fast return $dataprovider; } } after i...
Read more