Azure表存储事务限制

问题描述:

我正在针对ATS运行性能测试,当针对同一表/存储帐户使用多个虚拟机时,它表现得有些奇怪.

I'm running performance tests against ATS and its behaving a bit weird when using multiple virtual machines against the same table / storage account.

整个管道是非阻塞的(等待/异步),并使用TPL进行并发和并行执行.

The entire pipeline is non blocking (await/async) and using TPL for concurrent and parallel execution.

首先非常奇怪的是,使用这种设置,我只能插入约1200次.它运行在L VM盒上,即4核+ 800mbps.

First of all its very strange that with this setup i'm only getting about 1200 insertions. This is running on a L VM box, that is 4 cores + 800mbps.

我要插入100.000行,这些行具有唯一的PK和唯一的RK,应该利用最终的分布.

I'm inserting 100.000 rows with unique PK and unique RK, that should leverage the ultimate distribution.

以下是更具确定性的行为.

Even more deterministic behavior is the following.

当我运行1个VM时,我每秒获得约1200次插入. 当我运行3个VM时,每秒每次插入可获得大约730.

When I run 1 VM i get about 1200 insertions per second. When I run 3 VM i get about 730 on each insertions per second.

阅读博客文章时很幽默,他们在其中指定了目标. https://azure.microsoft.com/zh-CN/blog/windows-azures-flat-network-storage-and-2012-scalability-targets/

Its quite humors to read the blog post where they are specifying their targets. https://azure.microsoft.com/en-gb/blog/windows-azures-flat-network-storage-and-2012-scalability-targets/

单个表分区–表分区是表中具有相同分区键值的所有实体,通常表具有多个分区.单个表分区的吞吐量目标为:

Single Table Partition– a table partition are all of the entities in a table with the same partition key value, and usually tables have many partitions. The throughput target for a single table partition is:

每秒最多2,000个实体

Up to 2,000 entities per second

注意,这是针对单个分区而不是单个表的.因此,具有良好分区的表最多可以处理20,000个实体/秒,这是上述的总体帐户目标.

Note, this is for a single partition, and not a single table. Therefore, a table with good partitioning, can process up to the 20,000 entities/second, which is the overall account target described above.

我应该怎么做才能每秒使用20k,如何才能在每个VM上执行超过1,2k?

What shall I do to be able to utilize the 20k per second, and how would it be possible to execute more than 1,2k per VM?

-

更新:

我现在还尝试为每个单独的节点使用3个存储帐户,并且仍在获得性能/节流行为.我找不到逻辑上的原因.

I've now also tried using 3 storage accounts for each individual node and is still getting the performance / throttling behavior. Which i can't find a logical reason for.

-

更新2:

我进一步优化了代码,现在可以执行大约1550.

I've optimized the code further and now i'm possible to execute about 1550.

-

更新3:

我现在也在美国西部尝试过.那里的表现较差.降低约33%.

I've now also tried in US West. The performance is worse there. About 33% lower.

-

更新4:

我尝试从XL机器执行代码.这是8核而不是4核,并且内存和带宽增加了一倍,并且性能提高了2%,因此显然这个问题不在我一边.

I tried executing the code from a XL machine. Which is 8 cores instead of 4 and the double amount of memory and bandwidth and got a 2% increase in performance so clearly this problem is not on my side..

一些评论:

  1. 您提到您正在使用唯一的PK/RK来获得终极 分布,但您必须记住PK平衡是 不是立即的.首次创建表格时,整个表格都会 由1个分区服务器提供服务.因此,如果您正在插入 几个不同的PK,它们仍将转到一个分区 服务器,并受到单个扩展目标的瓶颈 分割.分区主机只会开始拆分您的 识别出热点后,在多个分区服务器之间进行分区 分区服务器.在< 2分钟的测试中,您不会看到 多个分区服务器或PK的好处.中的吞吐量 本文针对的是一个分布良好的PK方案, 频繁访问的数据,导致数据在 多个分区服务器.

  1. You mention that you are using unique PK/RK to get ultimate distribution, but you have to keep in mind that the PK balancing is not immediate. When you first create a table, the entire table will be served by 1 partition server. So if you are doing inserts across several different PKs, they will still be going to one partition server and be bottlenecked by the scalability target for a single partition. The partition master will only start splitting your partitions among multiple partition servers after it has identified hot partition servers. In your <2 minute test you will not see the benefit of multiple partiton servers or PKs. The throughput in the article is targeted towards a well distributed PK scheme with frequently accessed data, causing the data to be divided amongst multiple partition servers.

VM的大小不是问题,因为 您不会在CPU,内存或带宽上受阻.你可以实现 小型VM可获得完整的存储性能.

The size of your VM is not the issue as you are not blocked on CPU, Memory, or Bandwidth. You can achieve full storage performance from a small VM size.

退房 http://research.microsoft.com/en-us/downloads/5c8189b9-53aa-4d6a-a086-013d927e15a7/default.aspx . 我刚刚使用WebRole VM中的工具对该工具进行了快速测试 与我的存储帐户相同的数据中心,我一次就实现了 该工具在单个VM上的实例,每秒约上传2800个项目 每秒下载约7300个项目.这正在使用1024字节 实体,10个线程和100个批处理大小.我不知道此工具的效率如何,或者它是否禁用了Nagles算法,因为使用1的批量大小我无法获得出色的结果(我获得了〜1000/秒),但是至少使用100的批量大小,它表明您可以达到较高的项目/秒.这是在美国西部完成的.

Check out http://research.microsoft.com/en-us/downloads/5c8189b9-53aa-4d6a-a086-013d927e15a7/default.aspx. I just now did a quick test using that tool from a WebRole VM in the same datacenter as my storage account and I acheived, from a single instance of the tool on a single VM, ~2800 items per second upload and ~7300 items per second download. This is using 1024 byte entities, 10 threads, and 100 batch size. I don't know how efficient this tool is or if it disables Nagles Algorithm as I was unable to get great results (I got ~1000/second) using a batch size of 1, but at least with the 100 batch size it shows that you can achieve high items/second. This was done in US West.

您是否正在使用存储客户端库1.7(Microsoft.Azure.StorageClient.dll)或2.0(Microsoft.Azure.Storage.dll)? 2.0库具有一些性能改进,应该会产生更好的结果.

Are you using Storage client library 1.7 (Microsoft.Azure.StorageClient.dll) or 2.0 (Microsoft.Azure.Storage.dll)? The 2.0 library has some performance improvements and should yield better results.