如何使用Paramiko getfo将文件从SFTP服务器下载到内存以进行处理

问题描述：

我正在尝试使用Paramiko从SFTP下载CSV文件(内存中)，并将其导入到pandas数据框中.

I am trying to download a CSV file (in-memory) from SFTP using Paramiko and import it into a pandas dataframe.

transport = paramiko.Transport((server, 22))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)

with open(file_name, 'wb') as fl:
    sftp.getfo(file_name, fl, callback=printTotals)
    df = pd.read_csv(fl, sep=' ')

下面的代码失败，告诉我:

The code below fails, telling me:

OSError:文件未打开以供读取

OSError: File is not open for reading

我假设我需要某种缓冲区或文件，例如fl的对象，因为open需要一个文件.我对这一切还比较陌生，所以如果有人可以帮忙，我会很高兴.

I assume that I need some kind of buffer or file like object for fl instead, since open needs a file. I am relatively new to all of this, so I would be happy it if someone could help.

答

仍然允许您使用进度回调的简单解决方案是:

A simple solution that still allows you to use progress callback is:

使用 BytesIO类似于文件的对象将下载的文件存储到内存中；
下载文件后，您必须先查找文件指针，使其返回文件开始位置，然后再开始阅读文件.

Use BytesIO file-like object to store a downloaded file to memory;
You have to seek file pointer back to file start after downloading it, before you start reading it.

with io.BytesIO() as fl:
    sftp.getfo(file_name, fl, callback=printTotals)
    fl.seek(0)
    df = pd.read_csv(fl, sep=' ')

尽管使用此解决方案，您最终将文件两次加载到内存中.

Though with this solution, you will end up having the file loaded to memory twice.

更好的解决方案是实现类似文件的对象.它甚至可以让您同时下载和解析文件.

Better solution is to implement a file-like object. It will even allow you to download and parse the file at the same time.

class FileWithProgress:

    def __init__(self, fl):
        self.fl = fl
        self.size = fl.stat().st_size
        self.p = 0

    def read(self, blocksize):
        r = self.fl.read(blocksize)
        self.p += len(r)
        print(str(self.p) + " of " + str(self.size)) 
        return r

并像这样使用它:

with sftp.open(file_name, "rb") as fl:
    fl.prefetch()
    df = pd.read_csv(FileWithProgress(fl), sep=' ')

有关SFTPFile.prefetch调用，请参阅:
读取使用Python Paramiko SFTPClient.open方法打开的文件很慢.

For the SFTPFile.prefetch call, refer to:
Reading file opened with Python Paramiko SFTPClient.open method is slow.

如何使用Paramiko getfo将文件从SFTP服务器下载到内存以进行处理

相关推荐