如何根据给定的标准将一个csv文件拆分为多个csv?

问题描述：

我需要根据给定的时间分割几个csv文件.在这些文件中，时间值以秒为单位，并在时间"列中给出.

I need to split few csv files based on a given time. In these files the time values are in seconds and given in 'Time' column.

例如，如果我想在0.1秒内拆分aaa.csv文件，则需要将时间为0.0到0.1(附件文件中没有1到8)的第一行记录写入aaa1.csv，然后到aaa2.csv的时间大于0.1到0.2(附件文件中的第9到21号)的行，依此类推...(基本上是给定时间的倍数).

For example, if I want to split aaa.csv file in 0.1 seconds, then the first set of rows with time 0.0 to 0.1 (No 1 to 8 in attached file) needs to get written into aaa1.csv, then the rows with time greater than 0.1 to 0.2 (No. 9 to 21 in attached file) to aaa2.csv so on...(basically multiples of the given time).

输出文件的名称必须与输入文件的名称相同，并在末尾加上数字.并且需要将输出文件写入不同的位置/文件夹. 时间值必须是一个变量.因此，一次我可以在0.1秒内拆分文件，另一次我可以在0.7秒内拆分文件，依此类推.

Output files needs to get the same name as input file along with a number at the end. And output files need to get written into a different location/folder. Time value need to be a variable. So at a time I can split in 0.1 sec and at another time I can split the file in 0.7sec so on.

请问我该如何编写python脚本?该文件如下所示(可以从 https://fil.email/vnsZsp7b 下载的整个119K文件) :

How can I write a python script for this please? The file looks like the following (entire 119K file can be downloaded from https://fil.email/vnsZsp7b):

No.,Time,Length
1,0,146
2,0.006752,116
3,0.019767,156
4,0.039635,144
5,0.06009,147
6,0.069165,138
7,0.0797,133
8,0.099397,135
9,0.120142,135
10,0.139721,148
11,0.1401,126
12,0.1401,120
13,0.140101,123
14,0.140101,120
15,0.141294,118
16,0.141295,118
17,0.141295,114
18,0.144909,118
19,0.160639,119
20,0.161214,152
21,0.185625,143
... etc

在@Serafeim回答之后，我尝试了此操作:

import pandas as pd
import numpy as np
import glob
import os

path = '/root/Desktop/TT1/'
mystep = 0.4


for filename in glob(os.path.join(path, '*.csv')):
    df = pd.read_csv(filename)
    def data_splitter(df):
        max_time = df['Time'].max() # get max value of Time for the current csv file (df)
        myrange= np.arange(0, max_time, mystep) # build the threshold range
        for k in range(len(myrange)):
            # build the upper values
            temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
            #temp.to_csv("/root/Desktop/T1/xx_{}.csv".format(k))
            temp.to_csv("/root/Desktop/T1/{}_{}.csv".format(filename, k))

data_splitter(df)

答

您只需使用pandas在数据框上应用逻辑运算即可.✔️

You just need to apply a logical operation on the dataframe using pandas. ✔️

在此答案的结尾，我有一个脚本想法"可以自动执行此操作，但首先让我们一步一步地进行操作:

At the end of this answer I have a "script idea" to do this automatically but first let's go Step by step:

# Load the files using pandas
import pandas as pd

df = pd.read_csv("/Users/serafeim/Downloads/Testfile.csv")

# Get the desired elements based on 'Time' column
mask = df['Time'] < 0.1

# Write the new file
df_1 = df[mask] # or directly use: df_1 = df[df['Time'] < 0.1]

# save it 
df_1.to_csv("Testfile1.csv")

print(df_1)
    No.      Time  Length
0    1  0.000000     146
1    2  0.006752     116
2    3  0.019767     156
3    4  0.039635     144
4    5  0.060090     147
5    6  0.069165     138
6    7  0.079700     133
7    8  0.099397     135

#For 0.1 to 0.2 applying 2 logical conditions
df_2 = df[(df['Time'] > 0.1) & (df['Time'] < 0.2)]

脚本构想:

import pandas as pd
import numpy as np

mystep = 0.2 # the step e.g. 0.2, 0.4, 0.6 

#define the function
def data_splitter(df):
    max_time = df['Time'].max() # get max value of Time for the current csv file (df)
    myrange= np.arange(0, max_time, mystep) # build the threshold range
    for k in range(len(myrange)):
        # build the upper values 
        temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
        temp.to_csv("/Users/serafeim/Downloads/aaa_{}.csv".format(k))

现在，调用函数:

df = pd.read_csv("/Users/serafeim/Downloads/Testfile.csv")
data_splitter(df) # pass the df to the function and call the function

最后，您可以创建一个循环，并在data_splitter()函数中逐个传递每个df.

Finally, you can create a loop and pass each df one by one in the data_splitter() function.

要弄清楚该函数的功能，如下所示:

To make more clear what the function does look this:

for k in range(len(myrange)):
    print myrange[k], myrange[k]+step

此打印:

0.0 0.2
0.2 0.4
0.4 0.6000000000000001
0.6000000000000001 0.8
0.8 1.0

因此它创建了较低的&根据当前.csv文件的Time列的最大值自动设置上限阈值.

So it creates the lower & upper thresholds automatically based on the max value of Time column of the current .csv file.

import glob, os
path = '/Volumes/'

mystep = 0.2 

for filename in glob.glob(os.path.join(path, '*.csv')):
    df = pd.read_csv(filename)
    data_splitter(df)

一起放所有东西:

import pandas as pd
import numpy as np
import glob
import os

path = '/root/Desktop/TT1/'
mystep = 0.4

#define the function
def data_splitter(df, name):
    max_time = df['Time'].max() # get max value of Time for the current csv file (df)
    myrange= np.arange(0, max_time, mystep) # build the threshold range
    for k in range(len(myrange)):
        # build the upper values 
        temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
        temp.to_csv("/root/Desktop/T1/{}_{}.csv".format(name, k))

for filename in glob.glob(os.path.join(path, '*.csv')):
    df = pd.read_csv(filename)
    name = os.path.split(filename)[1] # get the name of the file
    data_splitter(df, name) # call the splitting function

如何根据给定的标准将一个csv文件拆分为多个csv?

一起放所有东西:

相关推荐