AMAZON Kinesis(一)Introduction (Old Note)
AMAZON Kinesis(1)Introduction (Old Note)
AMAZON Kinesis(1)Introduction (Old Note)
1 Introduction
Price:
Send Throughput:
Receive Throughput:
Message Latency:
Message Size: MB/S
Batch Size:
Persist Time: 24 hours
Producers: EC2 Instances, Clients, Mobile Clients, Traditional Servers
Consumers: S3, Amazon Redshift and etc.
The data in the stream are distributed into shards. Each record in the stream has a sequence number. The data blob can be up to 1MB.
The retention period of data can be from 24 hours to 7 days.
Each shard can support up to 5 transactions per second for reads, up to 2 MB per second
1,000 records per second writes, up to 1 MB per second
number_of_shards = max(incoming_write_bandwidth_in_KB/ outgoing_read_bandwidth_in_KB/2000)
Consumers can be auto scaling as well.
Limitation:
1. 50 shards for US East(N. Virginia) US West (Oregon) EU(Ireland). All other regions have a default shard limit of 25.
2. 1 to 7 days.
3. Data blob is up to 1 MB.
http://docs.amazonaws.cn/en_us/kinesis/latest/dev/service-sizes-and-limits.html
Kinesis -> S3 -> Redshift, We can put selected columns into Redshift, we can also add logic in lambda if we want.
http://www.****.net/article/2015-10-09/2825871
Producer —> Kinesis —> Redis Cluster Persist
http://www.****.net/article/2015-01-07/2823464
Kinesis and Spark Streaming
https://github.com/aiyanbo/spark-programming-guide-zh-cn/blob/master/spark-streaming/basic-concepts/kinesis-integration.md
Reference:
http://aws.amazon.com/kinesis/faqs/
search for
"When should I use Amazon Kinesis, and when should I use Amazon SQS"
https://github.com/aiyanbo/spark-programming-guide-zh-cn/blob/master/spark-streaming/basic-concepts/kinesis-integration.md
http://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-create-stream.html
AMAZON Kinesis(1)Introduction (Old Note)
1 Introduction
Price:
Send Throughput:
Receive Throughput:
Message Latency:
Message Size: MB/S
Batch Size:
Persist Time: 24 hours
Producers: EC2 Instances, Clients, Mobile Clients, Traditional Servers
Consumers: S3, Amazon Redshift and etc.
The data in the stream are distributed into shards. Each record in the stream has a sequence number. The data blob can be up to 1MB.
The retention period of data can be from 24 hours to 7 days.
Each shard can support up to 5 transactions per second for reads, up to 2 MB per second
1,000 records per second writes, up to 1 MB per second
number_of_shards = max(incoming_write_bandwidth_in_KB/ outgoing_read_bandwidth_in_KB/2000)
Consumers can be auto scaling as well.
Limitation:
1. 50 shards for US East(N. Virginia) US West (Oregon) EU(Ireland). All other regions have a default shard limit of 25.
2. 1 to 7 days.
3. Data blob is up to 1 MB.
http://docs.amazonaws.cn/en_us/kinesis/latest/dev/service-sizes-and-limits.html
Kinesis -> S3 -> Redshift, We can put selected columns into Redshift, we can also add logic in lambda if we want.
http://www.****.net/article/2015-10-09/2825871
Producer —> Kinesis —> Redis Cluster Persist
http://www.****.net/article/2015-01-07/2823464
Kinesis and Spark Streaming
https://github.com/aiyanbo/spark-programming-guide-zh-cn/blob/master/spark-streaming/basic-concepts/kinesis-integration.md
Reference:
http://aws.amazon.com/kinesis/faqs/
search for
"When should I use Amazon Kinesis, and when should I use Amazon SQS"
https://github.com/aiyanbo/spark-programming-guide-zh-cn/blob/master/spark-streaming/basic-concepts/kinesis-integration.md
http://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-create-stream.html