Python自定义进程池实例分析【生产者、消费者模型问题】
本文实例分析了Python自定义进程池。分享给大家供大家参考,具体如下:
代码说明一切:
#encoding=utf-8 #author:walker #date:2014-05-21 #function:自定义进程池遍历目录下文件 frommultiprocessingimportProcess,Queue,Lock importtime,os #消费者 classConsumer(Process): def__init__(self,queue,ioLock): super(Consumer,self).__init__() self.queue=queue self.ioLock=ioLock defrun(self): whileTrue: task=self.queue.get()#队列中无任务时,会阻塞进程 ifisinstance(task,str)andtask=='quit': break; time.sleep(1)#假定任务处理需要1秒钟 self.ioLock.acquire() print(str(os.getpid())+''+task) self.ioLock.release() self.ioLock.acquire() print'Bye-bye' self.ioLock.release() #生产者 defProducer(): queue=Queue()#这个队列是进程/线程安全的 ioLock=Lock() subNum=4#子进程数量 workers=build_worker_pool(queue,ioLock,subNum) start_time=time.time() forparent,dirnames,filenamesinos.walk(r'D:\test'): forfilenameinfilenames: queue.put(filename) ioLock.acquire() print('qsize:'+str(queue.qsize())) ioLock.release() whilequeue.qsize()>subNum*10:#控制队列中任务数量 time.sleep(1) forworkerinworkers: queue.put('quit') forworkerinworkers: worker.join() ioLock.acquire() print('Done!Timetaken:{}'.format(time.time()-start_time)) ioLock.release() #创建进程池 defbuild_worker_pool(queue,ioLock,size): workers=[] for_inrange(size): worker=Consumer(queue,ioLock) worker.start() workers.append(worker) returnworkers if__name__=='__main__': Producer()
ps:
self.ioLock.acquire() ... self.ioLock.release()
可用:
withself.ioLock: ...
替代。
再来一个好玩的例子:
#encoding=utf-8 #author:walker #date:2016-01-06 #function:一个多进程的好玩例子 importos,sys,time frommultiprocessingimportPool cur_dir_fullpath=os.path.dirname(os.path.abspath(__file__)) g_List=['a'] #修改全局变量g_List defModifyDict_1(): globalg_List g_List.append('b') #修改全局变量g_List defModifyDict_2(): globalg_List g_List.append('c') #处理一个 defProcOne(num): print('ProcOne'+str(num)+',g_List:'+repr(g_List)) #处理所有 defProcAll(): pool=Pool(processes=4) foriinrange(1,20): #ProcOne(i) #pool.apply(ProcOne,(i,)) pool.apply_async(ProcOne,(i,)) pool.close() pool.join() ModifyDict_1()#修改全局变量g_List if__name__=='__main__': ModifyDict_2()#修改全局变量g_List print('Inmaing_List:'+repr(g_List)) ProcAll()
Windows7下运行的结果:
λpython3demo.py Inmaing_List:['a','b','c'] ProcOne1,g_List:['a','b'] ProcOne2,g_List:['a','b'] ProcOne3,g_List:['a','b'] ProcOne4,g_List:['a','b'] ProcOne5,g_List:['a','b'] ProcOne6,g_List:['a','b'] ProcOne7,g_List:['a','b'] ProcOne8,g_List:['a','b'] ProcOne9,g_List:['a','b'] ProcOne10,g_List:['a','b'] ProcOne11,g_List:['a','b'] ProcOne12,g_List:['a','b'] ProcOne13,g_List:['a','b'] ProcOne14,g_List:['a','b'] ProcOne15,g_List:['a','b'] ProcOne16,g_List:['a','b'] ProcOne17,g_List:['a','b'] ProcOne18,g_List:['a','b'] ProcOne19,g_List:['a','b']
Ubuntu14.04下运行的结果:
Inmaing_List:['a','b','c'] ProcOne1,g_List:['a','b','c'] ProcOne2,g_List:['a','b','c'] ProcOne3,g_List:['a','b','c'] ProcOne5,g_List:['a','b','c'] ProcOne4,g_List:['a','b','c'] ProcOne8,g_List:['a','b','c'] ProcOne9,g_List:['a','b','c'] ProcOne7,g_List:['a','b','c'] ProcOne11,g_List:['a','b','c'] ProcOne6,g_List:['a','b','c'] ProcOne12,g_List:['a','b','c'] ProcOne13,g_List:['a','b','c'] ProcOne10,g_List:['a','b','c'] ProcOne14,g_List:['a','b','c'] ProcOne15,g_List:['a','b','c'] ProcOne16,g_List:['a','b','c'] ProcOne17,g_List:['a','b','c'] ProcOne18,g_List:['a','b','c'] ProcOne19,g_List:['a','b','c']
可以看见Windows7下第二次修改没有成功,而Ubuntu下修改成功了。据uliweb作者limodou讲,原因是Windows下是充重启实现的子进程;Linux下是fork实现的。
更多关于Python相关内容感兴趣的读者可查看本站专题:《PythonURL操作技巧总结》、《Python图片操作技巧总结》、《Python数据结构与算法教程》、《PythonSocket编程技巧总结》、《Python函数使用技巧总结》、《Python字符串操作技巧汇总》、《Python入门与进阶经典教程》及《Python文件与目录操作技巧汇总》
希望本文所述对大家Python程序设计有所帮助。