批量写入Firebase Cloud Firestore

问题描述：

我想创建一个新集合，并向其中添加成千上万个大小约为1-2K的文档.我已经在json中保存了数据，所以我认为这很容易.

I want to create a new collection and add thousands of documents sized ~ 1-2K to it. I already have data in json so I thought this would be easy.

我知道该批处理一次可以写入500次，因此将其分成500块，我编写了以下代码.虽然出于测试目的，我以20个块运行它，而我的测试json有72个对象.

I understand that batch can have 500 writes at a time so to break it into chunks of 500 I wrote the following code. Though for testing purpose I am running it with chunks of 20 and my test json has 72 objects.

但是我不断收到以下错误消息

But I keep getting the following error

node_modules\@google-cloud\firestore\src\write-batch.js:148
  throw new Error('Cannot modify a WriteBatch that has been committed.');
  ^

Error: Cannot modify a WriteBatch that has been committed.

我的代码如下

var dataObj = JSON.parse(fs.readFileSync('./bigt.json'))
var tmpdd = dataObj.slice(0, 72)
var batch = db.batch();

console.log(tmpdd.length)

let tc = tmpdd.length
let lc = 0
let upperLimit = 20, dd = null

while(lc<=tc){

    dd = tmpdd.slice(lc, upperLimit )

    console.log(lc, upperLimit)
    dd.map(
    o => batch.set(db.collection('nseStocks').doc(o.Date+o.variable), o)
    )

    batch.commit().then(function () {
        console.log('Written to firestore', lc, lc + upperLimit)
    })
    .catch(
        (err) => console.log('Fail', err)
    )

    lc = upperLimit
    upperLimit = upperLimit + 20 

}

同样奇怪的是，似乎没有在循环的每次迭代中都提交批处理.理想情况下，我会让Firestore确定文档ID，但显然批处理没有添加功能.

Also it's weird that batch doesn't seem to be committed in every iteration of the loop. Ideally I would let Firestore determine document ids but apparently batch does not have add function.

我尝试以循环方式添加文档，而不是批量写入.但是在添加一些文档后，它给了我超时错误.当然，对于大量文档来说，这是不切实际的.

I have tried adding documents in a loop instead of doing batch writes. But it gives me timeout error after adding a few documents. And of course it's not practical for large number of documents.

您可以说我是Firestore的新手，这是我玩第二天.

You could tell I am very new to Firestore and it's my second day playing with it.

请告诉我是否有明显的错误或更好的方法来完成此看似简单的任务.

Please let me know if there are any obvious mistakes or better ways of achieving this seemingly simple task.

谢谢

答

您正在为程序顶层的所有写操作创建一个批处理.对于所有批量写入操作，所有对batch.set()的调用都将重新使用它.

You're creating a single batch for all writes at the top level of your program. It's getting reused for all the calls to batch.set() that you make for all your batch writes.

var batch = db.batch();

相反，您应该为每组写入创建一个新批处理.您可以在while循环的顶部执行此操作:

Instead, you should create a new batch for each set of writes. You can do that at the top of your while loop:

while(lc<=tc) {
    var batch = db.batch();
    // use the new batch here
}

批量写入Firebase Cloud Firestore

相关推荐