详谈tensorflow gfile文件的用法
一、gfile模块是什么
gfile模块定义在tensorflow/python/platform/gfile.py,但其源代码实现主要位于tensorflow/tensorflow/python/lib/io/file_io.py,那么gfile模块主要功能是什么呢?
google上的定义为:
翻译过来为:
没有线程锁的文件I/O操作包装器
...对于TensorFlow的tf.gfile模块来说是一个特别无用的描述!
tf.gfile模块的主要角色是:
1.提供一个接近Python文件对象的API,以及
2.提供基于TensorFlowC++FileSystemAPI的实现。
C++FileSystemAPI支持多种文件系统实现,包括本地文件,谷歌云存储(以gs://开头)和HDFS(以hdfs:/开头)。TensorFlow将它们导出为tf.gfile,以便我们可以使用这些实现来保存和加载检查点,编写TensorBoardlog以及访问训练数据(以及其他用途)。但是,如果所有文件都是本地文件,则可以使用常规的Python文件API而不会造成任何问题。
以上为google对tf.gfile的说明。
二、gfileAPI介绍
下面将分别介绍每一个gfileAPI!
2-1)tf.gfile.Copy(oldpath,newpath,overwrite=False)
拷贝源文件并创建目标文件,无返回,其形参说明如下:
oldpath:带路径名字的拷贝源文件;
newpath:带路径名字的拷贝目标文件;
overwrite:目标文件已经存在时是否要覆盖,默认为false,如果目标文件已经存在则会报错
2-2)tf.gfile.MkDir(dirname)
创建一个目录,dirname为目录名字,无返回。
2-3)tf.gfile.Remove(filename)
删除文件,filename即文件名,无返回。
2-4)tf.gfile.DeleteRecursively(dirname)
递归删除所有目录及其文件,dirname即目录名,无返回。
2-5)tf.gfile.Exists(filename)
判断目录或文件是否存在,filename可为目录路径或带文件名的路径,有该目录则返回True,否则False。
2-6)tf.gfile.Glob(filename)
查找匹配pattern的文件并以列表的形式返回,filename可以是一个具体的文件名,也可以是包含通配符的正则表达式。
2-7)tf.gfile.IsDirectory(dirname)
判断所给目录是否存在,如果存在则返回True,否则返回False,dirname是目录名。
2-8)tf.gfile.ListDirectory(dirname)
罗列dirname目录下的所有文件并以列表形式返回,dirname必须是目录名。
2-9)tf.gfile.MakeDirs(dirname)
以递归方式建立父目录及其子目录,如果目录已存在且是可覆盖则会创建成功,否则报错,无返回。
2-10)tf.gfile.Rename(oldname,newname,overwrite=False)
重命名或移动一个文件或目录,无返回,其形参说明如下:
oldname:旧目录或旧文件;
newname:新目录或新文件;
overwrite:默认为false,如果新目录或新文件已经存在则会报错,否则重命名或移动成功。
2-11)tf.gfile.Stat(filename)
返回目录的统计数据,该函数会返回FileStatistics数据结构,以dir(tf.gfile.Stat(filename))获取返回数据的属性如下:
2-12)tf.gfile.Walk(top,in_order=True)
递归获取目录信息生成器,top是目录名,in_order默认为True指示顺序遍历目录,否则将无序遍历,每次生成返回如下格式信息(dirname,[subdirname,subdirname,...],[filename,filename,...])。
2-13)tf.gfile.GFile(filename,mode)
获取文本操作句柄,类似于python提供的文本操作open()函数,filename是要打开的文件名,mode是以何种方式去读写,将会返回一个文本操作句柄。
tf.gfile.Open()是该接口的同名,可任意使用其中一个!
2-14)tf.gfile.FastGFile(filename,mode)
该函数与tf.gfile.GFile的差别仅仅在于“无阻塞”,即该函数会无阻赛以较快的方式获取文本操作句柄。
三、API源码
#Copyright2015TheTensorFlowAuthors.AllRightsReserved. # #LicensedundertheApacheLicense,Version2.0(the"License"); #youmaynotusethisfileexceptincompliancewiththeLicense. #YoumayobtainacopyoftheLicenseat # #http://www.apache.org/licenses/LICENSE-2.0 # #Unlessrequiredbyapplicablelaworagreedtoinwriting,software #distributedundertheLicenseisdistributedonan"ASIS"BASIS, #WITHOUTWARRANTIESORCONDITIONSOFANYKIND,eitherexpressorimplied. #SeetheLicenseforthespecificlanguagegoverningpermissionsand #limitationsundertheLicense. #============================================================================== """FileIOmethodsthatwraptheC++FileSystemAPI. TheC++FileSystemAPIisSWIGwrappedinfile_io.i.Thesefunctionscallthose toaccomplishbasicFileIOoperations. """ from__future__importabsolute_import from__future__importdivision from__future__importprint_function importos importuuid importsix fromtensorflow.pythonimportpywrap_tensorflow fromtensorflow.python.frameworkimportc_api_util fromtensorflow.python.frameworkimporterrors fromtensorflow.python.utilimportcompat fromtensorflow.python.utilimportdeprecation fromtensorflow.python.util.tf_exportimporttf_export classFileIO(object): """FileIOclassthatexposesmethodstoread/writeto/fromfiles. Theconstructortakesthefollowingarguments: name:nameofthefile mode:oneof'r','w','a','r+','w+','a+'.Append'b'forbytesmode. Canbeusedasaniteratortoiterateoverlinesinthefile. ThedefaultbuffersizeusedfortheBufferedInputStreamusedforreading thefilelinebylineis1024*512bytes. """ def__init__(self,name,mode): self.__name=name self.__mode=mode self._read_buf=None self._writable_file=None self._binary_mode="b"inmode mode=mode.replace("b","") ifmodenotin("r","w","a","r+","w+","a+"): raiseerrors.InvalidArgumentError( None,None,"modeisnot'r'or'w'or'a'or'r+'or'w+'or'a+'") self._read_check_passed=modein("r","r+","a+","w+") self._write_check_passed=modein("a","w","r+","a+","w+") @property defname(self): """Returnsthefilename.""" returnself.__name @property defmode(self): """Returnsthemodeinwhichthefilewasopened.""" returnself.__mode def_preread_check(self): ifnotself._read_buf: ifnotself._read_check_passed: raiseerrors.PermissionDeniedError(None,None, "Fileisn'topenforreading") witherrors.raise_exception_on_not_ok_status()asstatus: self._read_buf=pywrap_tensorflow.CreateBufferedInputStream( compat.as_bytes(self.__name),1024*512,status) def_prewrite_check(self): ifnotself._writable_file: ifnotself._write_check_passed: raiseerrors.PermissionDeniedError(None,None, "Fileisn'topenforwriting") witherrors.raise_exception_on_not_ok_status()asstatus: self._writable_file=pywrap_tensorflow.CreateWritableFile( compat.as_bytes(self.__name),compat.as_bytes(self.__mode),status) def_prepare_value(self,val): ifself._binary_mode: returncompat.as_bytes(val) else: returncompat.as_str_any(val) defsize(self): """Returnsthesizeofthefile.""" returnstat(self.__name).length defwrite(self,file_content): """Writesfile_contenttothefile.Appendstotheendofthefile.""" self._prewrite_check() witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.AppendToFile( compat.as_bytes(file_content),self._writable_file,status) defread(self,n=-1): """Returnsthecontentsofafileasastring. Startsreadingfromcurrentpositioninfile. Args: n:Read'n'bytesifn!=-1.Ifn=-1,readstoendoffile. Returns: 'n'bytesofthefile(orwholefile)inbytesmodeor'n'bytesofthe stringifinstring(regular)mode. """ self._preread_check() witherrors.raise_exception_on_not_ok_status()asstatus: ifn==-1: length=self.size()-self.tell() else: length=n returnself._prepare_value( pywrap_tensorflow.ReadFromStream(self._read_buf,length,status)) @deprecation.deprecated_args( None, "positionisdeprecatedinfavoroftheoffsetargument.", "position") defseek(self,offset=None,whence=0,position=None): #TODO(jhseu):Deletelater.Usedtoomit`position`fromdocs. #pylint:disable=g-doc-args """Seekstotheoffsetinthefile. Args: offset:Thebytecountrelativetothewhenceargument. whence:Validvaluesforwhenceare: 0:startofthefile(default) 1:relativetothecurrentpositionofthefile 2:relativetotheendoffile.offsetisusuallynegative. """ #pylint:enable=g-doc-args self._preread_check() #Weneededtomakeoffsetakeywordargumentforbackwards-compatibility. #Thischeckexistssothatwecanconvertbacktohavingoffsetbea #positionalargument. #TODO(jhseu):Make`offset`apositionalargumentafter`position`is #deleted. ifoffsetisNoneandpositionisNone: raiseTypeError("seek():offsetargumentrequired") ifoffsetisnotNoneandpositionisnotNone: raiseTypeError("seek():offsetandpositionmaynotbeset" "simultaneously.") ifpositionisnotNone: offset=position witherrors.raise_exception_on_not_ok_status()asstatus: ifwhence==0: pass elifwhence==1: offset+=self.tell() elifwhence==2: offset+=self.size() else: raiseerrors.InvalidArgumentError( None,None, "Invalidwhenceargument:{}.Validvaluesare0,1,or2." .format(whence)) ret_status=self._read_buf.Seek(offset) pywrap_tensorflow.Set_TF_Status_from_Status(status,ret_status) defreadline(self): r"""Readsthenextlinefromthefile.Leavesthe'\n'attheend.""" self._preread_check() returnself._prepare_value(self._read_buf.ReadLineAsString()) defreadlines(self): """Returnsalllinesfromthefileinalist.""" self._preread_check() lines=[] whileTrue: s=self.readline() ifnots: break lines.append(s) returnlines deftell(self): """Returnsthecurrentpositioninthefile.""" self._preread_check() returnself._read_buf.Tell() def__enter__(self): """Makeusablewith"with"statement.""" returnself def__exit__(self,unused_type,unused_value,unused_traceback): """Makeusablewith"with"statement.""" self.close() def__iter__(self): returnself defnext(self): retval=self.readline() ifnotretval: raiseStopIteration() returnretval def__next__(self): returnself.next() defflush(self): """FlushestheWritablefile. Thisonlyensuresthatthedatahasmadeitswayoutoftheprocesswithout anyguaranteesonwhetherit'swrittentodisk.Thismeansthatthe datawouldsurviveanapplicationcrashbutnotnecessarilyanOScrash. """ ifself._writable_file: witherrors.raise_exception_on_not_ok_status()asstatus: ret_status=self._writable_file.Flush() pywrap_tensorflow.Set_TF_Status_from_Status(status,ret_status) defclose(self): """ClosesFileIO.ShouldbecalledfortheWritableFiletobeflushed.""" self._read_buf=None ifself._writable_file: witherrors.raise_exception_on_not_ok_status()asstatus: ret_status=self._writable_file.Close() pywrap_tensorflow.Set_TF_Status_from_Status(status,ret_status) self._writable_file=None @tf_export("gfile.Exists") deffile_exists(filename): """Determineswhetherapathexistsornot. Args: filename:string,apath Returns: Trueifthepathexists,whetheritsafileoradirectory. Falseifthepathdoesnotexistandtherearenofilesystemerrors. Raises: errors.OpError:PropagatesanyerrorsreportedbytheFileSystemAPI. """ try: witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.FileExists(compat.as_bytes(filename),status) excepterrors.NotFoundError: returnFalse returnTrue @tf_export("gfile.Remove") defdelete_file(filename): """Deletesthefilelocatedat'filename'. Args: filename:string,afilename Raises: errors.OpError:PropagatesanyerrorsreportedbytheFileSystemAPI.E.g., NotFoundErrorifthefiledoesnotexist. """ witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.DeleteFile(compat.as_bytes(filename),status) defread_file_to_string(filename,binary_mode=False): """Readstheentirecontentsofafiletoastring. Args: filename:string,pathtoafile binary_mode:whethertoopenthefileinbinarymodeornot.Thischanges thetypeoftheobjectreturned. Returns: contentsofthefileasastringorbytes. Raises: errors.OpError:Raisesvarietyoferrorsthataresubtypese.g. NotFoundErroretc. """ ifbinary_mode: f=FileIO(filename,mode="rb") else: f=FileIO(filename,mode="r") returnf.read() defwrite_string_to_file(filename,file_content): """Writesastringtoagivenfile. Args: filename:string,pathtoafile file_content:string,contentsthatneedtobewrittentothefile Raises: errors.OpError:Ifthereareerrorsduringtheoperation. """ withFileIO(filename,mode="w")asf: f.write(file_content) @tf_export("gfile.Glob") defget_matching_files(filename): """Returnsalistoffilesthatmatchthegivenpattern(s). Args: filename:stringoriterableofstrings.Theglobpattern(s). Returns: Alistofstringscontainingfilenamesthatmatchthegivenpattern(s). Raises: errors.OpError:Iftherearefilesystem/directorylistingerrors. """ witherrors.raise_exception_on_not_ok_status()asstatus: ifisinstance(filename,six.string_types): return[ #Convertthefilenamestostringfrombytes. compat.as_str_any(matching_filename) formatching_filenameinpywrap_tensorflow.GetMatchingFiles( compat.as_bytes(filename),status) ] else: return[ #Convertthefilenamestostringfrombytes. compat.as_str_any(matching_filename) forsingle_filenameinfilename formatching_filenameinpywrap_tensorflow.GetMatchingFiles( compat.as_bytes(single_filename),status) ] @tf_export("gfile.MkDir") defcreate_dir(dirname): """Createsadirectorywiththename'dirname'. Args: dirname:string,nameofthedirectorytobecreated Notes: Theparentdirectoriesneedtoexist.Userecursive_create_dirinsteadif thereisthepossibilitythattheparentdirsdon'texist. Raises: errors.OpError:Iftheoperationfails. """ witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.CreateDir(compat.as_bytes(dirname),status) @tf_export("gfile.MakeDirs") defrecursive_create_dir(dirname): """Createsadirectoryandallparent/intermediatedirectories. Itsucceedsifdirnamealreadyexistsandiswritable. Args: dirname:string,nameofthedirectorytobecreated Raises: errors.OpError:Iftheoperationfails. """ witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.RecursivelyCreateDir(compat.as_bytes(dirname),status) @tf_export("gfile.Copy") defcopy(oldpath,newpath,overwrite=False): """Copiesdatafromoldpathtonewpath. Args: oldpath:string,nameofthefilewho'scontentsneedtobecopied newpath:string,nameofthefiletowhichtocopyto overwrite:boolean,iffalseitsanerrorfornewpathtobeoccupiedbyan existingfile. Raises: errors.OpError:Iftheoperationfails. """ witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.CopyFile( compat.as_bytes(oldpath),compat.as_bytes(newpath),overwrite,status) @tf_export("gfile.Rename") defrename(oldname,newname,overwrite=False): """Renameormoveafile/directory. Args: oldname:string,pathnameforafile newname:string,pathnametowhichthefileneedstobemoved overwrite:boolean,iffalseit'sanerrorfor`newname`tobeoccupiedby anexistingfile. Raises: errors.OpError:Iftheoperationfails. """ witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.RenameFile( compat.as_bytes(oldname),compat.as_bytes(newname),overwrite,status) defatomic_write_string_to_file(filename,contents,overwrite=True): """Writesto`filename`atomically. Thismeansthatwhen`filename`appearsinthefilesystem,itwillcontain allof`contents`.Withwrite_string_to_file,itispossibleforthefile toappearinthefilesystemwith`contents`onlypartiallywritten. Accomplishedbywritingtoatempfileandthenrenamingit. Args: filename:string,pathnameforafile contents:string,contentsthatneedtobewrittentothefile overwrite:boolean,iffalseit'sanerrorfor`filename`tobeoccupiedby anexistingfile. """ temp_pathname=filename+".tmp"+uuid.uuid4().hex write_string_to_file(temp_pathname,contents) try: rename(temp_pathname,filename,overwrite) excepterrors.OpError: delete_file(temp_pathname) raise @tf_export("gfile.DeleteRecursively") defdelete_recursively(dirname): """Deleteseverythingunderdirnamerecursively. Args: dirname:string,apathtoadirectory Raises: errors.OpError:Iftheoperationfails. """ witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.DeleteRecursively(compat.as_bytes(dirname),status) @tf_export("gfile.IsDirectory") defis_directory(dirname): """Returnswhetherthepathisadirectoryornot. Args: dirname:string,pathtoapotentialdirectory Returns: True,ifthepathisadirectory;Falseotherwise """ status=c_api_util.ScopedTFStatus() returnpywrap_tensorflow.IsDirectory(compat.as_bytes(dirname),status) @tf_export("gfile.ListDirectory") deflist_directory(dirname): """Returnsalistofentriescontainedwithinadirectory. Thelistisinarbitraryorder.Itdoesnotcontainthespecialentries"." and"..". Args: dirname:string,pathtoadirectory Returns: [filename1,filename2,...filenameN]asstrings Raises: errors.NotFoundErrorifdirectorydoesn'texist """ ifnotis_directory(dirname): raiseerrors.NotFoundError(None,None,"Couldnotfinddirectory") witherrors.raise_exception_on_not_ok_status()asstatus: #Converteachelementtostring,sincethereturnvaluesofthe #vectorofstringshouldbeinterpretedasstrings,notbytes. return[ compat.as_str_any(filename) forfilenameinpywrap_tensorflow.GetChildren( compat.as_bytes(dirname),status) ] @tf_export("gfile.Walk") defwalk(top,in_order=True): """Recursivedirectorytreegeneratorfordirectories. Args: top:string,aDirectoryname in_order:bool,TraverseinorderifTrue,postorderifFalse. Errorsthathappenwhilelistingdirectoriesareignored. Yields: Eachyieldisa3-tuple:thepathnameofadirectory,followedbylistsof allitssubdirectoriesandleaffiles. (dirname,[subdirname,subdirname,...],[filename,filename,...]) asstrings """ top=compat.as_str_any(top) try: listing=list_directory(top) excepterrors.NotFoundError: return files=[] subdirs=[] foriteminlisting: full_path=os.path.join(top,item) ifis_directory(full_path): subdirs.append(item) else: files.append(item) here=(top,subdirs,files) ifin_order: yieldhere forsubdirinsubdirs: forsubiteminwalk(os.path.join(top,subdir),in_order): yieldsubitem ifnotin_order: yieldhere @tf_export("gfile.Stat") defstat(filename): """Returnsfilestatisticsforagivenpath. Args: filename:string,pathtoafile Returns: FileStatisticsstructthatcontainsinformationaboutthepath Raises: errors.OpError:Iftheoperationfails. """ file_statistics=pywrap_tensorflow.FileStatistics() witherrors.raise_exception_on_not_ok_status()asstatus: pywrap_tensorflow.Stat(compat.as_bytes(filename),file_statistics,status) returnfile_statistics
以上这篇详谈tensorflowgfile文件的用法就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持毛票票。
声明:本文内容来源于网络,版权归原作者所有,内容由互联网用户自发贡献自行上传,本网站不拥有所有权,未作人工编辑处理,也不承担相关法律责任。如果您发现有涉嫌版权的内容,欢迎发送邮件至:czq8825#qq.com(发邮件时,请将#更换为@)进行举报,并提供相关证据,一经查实,本站将立刻删除涉嫌侵权内容。