将UTF-16转换为UTF-8并删除BOM?

问题描述：

我们有一个数据输入人员，该人员在Windows上使用UTF-16编码，希望拥有utf-8并删除BOM. utf-8转换有效，但BOM仍然存在.我将如何删除?这是我目前拥有的:

We have a data entry person who encoded in UTF-16 on Windows and would like to have utf-8 and remove the BOM. The utf-8 conversion works but BOM is still there. How would I remove this? This is what I currently have:

batch_3={'src':'/Users/jt/src','dest':'/Users/jt/dest/'}
batches=[batch_3]

for b in batches:
  s_files=os.listdir(b['src'])
  for file_name in s_files:
    ff_name = os.path.join(b['src'], file_name)  
    if (os.path.isfile(ff_name) and ff_name.endswith('.json')):
      print ff_name
      target_file_name=os.path.join(b['dest'], file_name)
      BLOCKSIZE = 1048576
      with codecs.open(ff_name, "r", "utf-16-le") as source_file:
        with codecs.open(target_file_name, "w+", "utf-8") as target_file:
          while True:
            contents = source_file.read(BLOCKSIZE)
            if not contents:
              break
            target_file.write(contents)

如果我用十六进制转储-C，我会看到:

If I hexdump -C I see:

Wed Jan 11$ hexdump -C svy-m-317.json 
00000000  ef bb bf 7b 0d 0a 20 20  20 20 22 6e 61 6d 65 22  |...{..    "name"|
00000010  3a 22 53 61 76 6f 72 79  20 4d 61 6c 69 62 75 2d  |:"Savory Malibu-|

在结果文件中.如何删除物料清单?

in the resulting file. How do I remove the BOM?

thx

答

只需使用 str.decode 和 str.encode :

Just use str.decode and str.encode:

with open(ff_name, 'rb') as source_file:
  with open(target_file_name, 'w+b') as dest_file:
    contents = source_file.read()
    dest_file.write(contents.decode('utf-16').encode('utf-8'))

str.decode将为您消除BOM(并推论字节序).

str.decode will get rid of the BOM for you (and deduce the endianness).

将UTF-16转换为UTF-8并删除BOM?

相关推荐