Error in parallel run

[nurfee]

Hey,

I've tried running GTG using the parallel version. It works fine for five days run with hourly data set (number of grids is 120 x 751 x 1501). However, I got this error when I ran it for longer time period:

--------------------------------------------------------------------------------
         Starting the MCCSearch Analysis 
--------------------------------------------------------------------------------

 -------------- Reading MERG and TRMM Data ----------

 -------------- findCloudElements ----------
(239, 751, 1501)
Traceback (most recent call last):
  File "mccSearchUI.py", line 302, in <module>
    main()
  File "mccSearchUI.py", line 156, in main
    CEGraph = mccSearch.find_cloud_elements(mergImgs,timeList,DIRS['mainDirStr'], LAT,LON,DIRS['TRMMdirName'])
  File "/Users/nsagitap/script-master/detect_MCS/GTG/gtg_v4/code/mccSearch.py", line 117, in find_cloud_elements
    varsDict['images'] = mergImgs
  File "<string>", line 2, in __setitem__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 758, in _callmethod
    conn.send((self._id, methodname, args, kwds))
SystemError: NULL result without error in PyObject_Call

The size of my mergImgs for this run is (239 x 751 x 1501). The error occurred when the code was trying to transfer mergImgs to the dictionary. I'm suspecting a memory issue here, but I am not really sure. I need to run GTG for one month where the size of time dimension is approximately above 700. I'm running GTG on Mac with OSX Yosemite, processor 3.2 GHz Intel Core i5, and RAM 32 GB.

Do you have any idea about what happen and on how to overcome this?

Thanks in advance.

[kwhitehall]

Hey @nurfee, I was finally able to replicate this problem. I am reading on it here http://stackoverflow.com/questions/22656285/multiprocessing-null-result-without-error-in-pyobject-call
@cgoodale .. interesting limitation of multithreading here!