您的位置: 首页  >  技术问答  >  Google云端存储媒体签名网址

Google云端存储媒体签名网址

分类: 技术问答 • 2022-03-29 12:53:00

Google云端存储媒体签名网址

问题描述：

我已经建立了一个视频网站，该网站为用户提供m3u8和相关的ts文件.我不希望媒体文件免费可用，所以我要做的是:当用户在站点上时，将在mysql中使用其IP和令牌创建一个会话；当他们要求特定媒体子域(mp4.domain.com)中的任何文件时，Nginx身份验证模块使用url和附加的令牌作为javascript设置的请求cookie来查询localhost:8080 ...查询数据库并允许/拒绝根据会话信息访问文件.

I've got a video site set up that serves up m3u8 and associated ts files for users. I don't want the media files freely available so what I've done is: when the user is on the site, a session is created in mysql with their IP and a token; when they ask for any file in the specific media subdomain (mp4.domain.com), Nginx auth module queries localhost:8080 with the url and the attached token as a request cookie set through javascript...queries the database and allows/denies access to the file dependant on session information.

现在这可以正常工作，根据服务器负载，开销在8到20 MS之间，并且在生成m3u8链接时无需处理PHP中的url. OSMF刚得到m3u8并要求文件，javascript添加了令牌cookie，而Bob是你的叔叔.

Now this works fine, overhead is between 8 - 20 MS depending on server load and there's no need to mangle urls in PHP when generating the m3u8 link; OSMF just gets the m3u8 and asks for the files, javascript adds the token cookie and Bob's your uncle.

现在我们正在迁移到Google Cloud Storage，我面临的问题是我真的无法控制其中的任何一个... m3u8的签名URL很容易，但是每一个m3u8都会必须为每个分辨率，缩略图和音频AAC的每个ts文件动态生成带有签名URL的URL(给您一个想法，我选择的随机视频总共有1,043个文件)...大约要生成1,043个不同的签名URL每个6 MS产生6秒钟的总生成时间...这太可怕了.

Now we're migrating to Google Cloud Storage and the issue I'm faced with is that I can't really control any of that...a signed url to the m3u8 is easy enough, but each and every m3u8 would have to be dynamically generated with signed urls for every ts file for every resolution, thumbnail and audio aac (to give you an idea, a random video I chose has 1,043 files total)...that's 1,043 different signed urls to generate, at roughly 6 MS each gives 6 seconds total generation time...which is horrid.

是否有替代方法来管理此问题?我不太热衷于Cloud Storage API，但似乎找不到其他任何东西... ACL对此似乎无济于事，我唯一想到的另一件事就是每天在...上旋转文件位置?...以混淆它们.有没有人遇到过类似的情况，或者对我可以从哪里开始解决这个问题有一个想法?

Is there an alternative way to manage this? I'm not too keen on the Cloud Storage API but I can't seem to find anything else...ACLs seem to be useless for this and the only other thing I can think of is rotating file locations on a ...daily?... basis in order to obfuscate them. Has anyone had a similar situation or have an idea of where I can start playing around to resolve this?

答

经过更多研究，我得出以下结论:

After more research, I've come to the following conclusion:

我的第一个计算是使用Google的gsutil工具完成的，这似乎在计算签名的URL哈希时会带来很多开销，例如:

My first calculations where done using Google's gsutil tool, which seems to introduce a lot of overhead when calculating a signed URL hash, for example:

gsutil代码:

gsutil signurl -d 60m /path/to/google.p12 gs://bucket/file

执行时间:0.73812007904053

Execution time: 0.73812007904053

，但是，使用本机PHP函数创建签名URL的速度要快得多:

,however, using native PHP functions to create a signed URL is much faster:

PHP代码:

function storageURL($bucket,$archivo) {
    $expires = time()+60; 
    $to_sign = ("GET\n\n\n".$expires."\n/".$bucket.'/'.$archivo);
    $fp = fopen('/path/to/google.pem', 'r');
    $priv_key = fread($fp, 8192);
    fclose($fp);
    $pkeyid = openssl_get_privatekey($priv_key);
    if(!openssl_sign($to_sign,$signature,$pkeyid,'sha256')) {
        $signature = 'sinfirma';
    } else {
        $signature = urlencode(base64_encode($signature));
    }
    return ('https://'.$bucket.'.storage.googleapis.com/'.$archivo.'?GoogleAccessId=XXXXXXX@developer.gserviceaccount.com&Expires='.$expires.'&Signature='.$signature);
}

执行时间:0.0007929801940918

Execution time: 0.0007929801940918

这一切都改变了，因为运行2000迭代的PHP代码仍然仅给我1.0643119812012的执行时间，外加用于创建所有m3u8文件的0.0325711573357，外加0.0039050579071045的附加6次迭代来为m3u8s创建签名的URL. ;总执行时间为1.100788196444004秒，其中大部分取决于视频的长度.

That changes everything, as running 2000 iterations of the PHP code still only gives me an execution time of 1.0643119812012 plus an additional 0.0325711573357 for creation of all the m3u8 files plus 0.0039050579071045 for an additional 6 iterations to create the signed URLs for the m3u8s; giving a total execution time of 1.100788196444004 seconds, with the largest part being dependent on the length of the video.

这似乎很好，因为用户习惯于更长的视频的更长的加载"或缓冲"时间，因此，当视频更长的视频时，额外的〜0.5-〜1.5秒不会真正影响可用性.

This actually seems fine, as users are used to longer "loading" or "buffering" times for longer videos so the ~0.5 - ~1.5 additional seconds when the video is longer will not really affect usability that much.

顺便说一句，在当前状态下，服务器上当前有689个视频，总共864,138个相关的.ts和.aac文件，每个视频具有6个m3u8(180,360,480,720,1080，AAC)以及一个附加的m3u8对于主播放列表...因此需要为所有视频生成每小时的url(689 [master m3u8] + 864,138 [assets] + 4134 [qual m3u8])PHP代码的868,961迭代，总运行时间为467.15262699127(〜7分钟)，这是可管理的，但考虑到运行时动态生成每个URL的问题.

As an aside, in the current state, there are currently 689 videos on the server, with a total of 864,138 related .ts and .aac files, each video having 6 m3u8s (180,360,480,720,1080,AAC) plus an additional m3u8 for the master playlist...so generating an hourly url for all videos would require (689 [master m3u8] + 864,138 [assets] + 4134 [qual m3u8]) 868,961 iterations of the PHP code, a total runtime of 467.15262699127 (~ 7 minutes), which is manageable but moot considering the runtime to dynamically generate each URL.

这全部使用的是Google Compute n1-highmem-2实例，但功能不那么强大，因此切换到功能更强大的计算机将使所有这些操作甚至更快.

This is all using a Google Compute n1-highmem-2 instance, which is not that powerful, so switching to a more powerful machine will make all of this even faster.

但是，所有这一切都带来了另一个层面，就像Google(以及其他所有公司一样)对每个存储区中每个PUT操作收取的费用一样，因此必须进行成本计算.查看我们上个月的统计信息，我总共看到447,103个视频播放(嘿，这是一个小站点)，根据建议的方案，每个视频点击都会产生7个PUT操作(6个比特率m3u8 + 1主m3u8)，该月总共增加了3,129,721个PUT，以成本计算(3129721/10000 * 0.01)，我为此付出了3.13美元的额外成本……虽然小，但如果该站点变得更受欢迎，可能会成为一个问题.另一种解决方案(每个人都需要按小时签名的URL)将生成((689 [master m3u8] + 4134 [qual m3u8])* 24 [每天的发电次数] * 30 [每月的天数])3,472,560个额外的PUT ...大约相同，因此我在两种方案之间的收支平衡点(在成本上)接近或接近.我必须使用前几个月的数据在这里做更多的数字，以更好地了解这一点，因为一种方案(每次匹配的网址)取决于用户数量，而另一种方案(全球网址生成)取决于视频数量...并且它们各自以完全不同的方式扩展.

But all of this brings another dimension into the fold, as Google (as all other also do) charges per PUT operation on each bucket, so a cost calculation is in order. Looking at our stats for the last month, I see a total of 447,103 total video plays (hey, it's a small site), which, under the proposed scheme, would have generated 7 PUT operations for each video hit (6 bitrate m3u8 + 1 master m3u8), a total of 3,129,721 additional PUTs that month, calculating for cost (3129721 / 10000 * 0.01) gives me a dollar figure of $3.13 additional cost for this...small but could become an issue if the site becomes more popular. The other solution (hourly signed URLs for everyone) would generate ((689 [master m3u8] + 4134 [qual m3u8]) * 24 [gens per day] * 30 [days per month]) 3,472,560 additional PUTs...which is roughly the same, so I am at or near the break-even point (cost-wise) for choosing between the two schemes. I have to do more numbers here using previous months' data to get a better idea of this since one scheme (URL per hit) depends on amount of users and the other (global URL generation) depends on the amount of videos...and they each scale in totally different ways.

从本质上讲，使用本机代码，问题似乎完全是货币问题，而编码矢量却很小(重写视频播放代码与引入每小时URL生成).在做出最终决定之前，都需要对两者进行研究和比较.

In essence, using native code, the issue seems to be purely monetary with a small coding vector (rewriting the video play code vs. introducing hourly URL generation). Both need to be looked at and compared before making a final decision.

尽管如此，可以绑定到m3u8的Cloud Storage API中的新ACL(例如，以m3u8作为有效负载的媒体部分文件)将使一切工作变得更加顺畅...在某个地方，我可以提出这个建议到Google存储团队?

Although, a new ACL in the Cloud Storage API (say media-part-file with the m3u8 as payload) that can be tied to an m3u8 would make everything smoother to work out...is there someplace I could propose this to the Google Storage team?

-30/10最终解决方案-

-- 30/10 Final Solution --

这是我想出的最终解决方案，到目前为止，它似乎运行良好.

This is the final solution I've come up with, and it seems to be working fine so far.

设置:

Google Cloud Compute实例上的Nginx-m3u8.domain.com

Nginx on Google Cloud Compute Instance - m3u8.domain.com

  • 视频转换器执行以下操作: 1.- ffmpeg将源文件转换为180,360,480,720,1080，AAC子文件 2.- ffmpeg将文件分割为11秒的块(较少的文件，iOS仍然接受) 3.- PHP将所有媒体文件复制到GS存储桶 4.- PHP解析生成的m3u8文件并创建动态m3u8文件 5.- PHP将size.m3u8文件和master.m3u8文件复制到附加硬盘上的正确目录中

  • The video converter does the following: 1.- ffmpeg to convert source files to 180,360,480,720,1080,AAC sub-files 2.- ffmpeg segments files into 11 second chunks (less files, iOS still accepts it) 3.- PHP copies all media files to GS bucket 4.- PHP parses the generated m3u8 files and creates a dynamic m3u8 file 5.- PHP copies size.m3u8 files and master.m3u8 file to proper directory on attached HDD

nginx.conf中的新服务器块将.m3u8文件解析为PHP 1.- OSMF播放器请求主m3u8，JS添加会话令牌 2.- PHP检查会话令牌+ IP以验证用户 3.-如果通过验证，则回显当前视频m3u8 4.-如果未通过验证，则回声m3u8，表示不允许您观看此视频

New server block in nginx.conf that parses .m3u8 files as PHP 1.- OSMF player requests master m3u8, JS adds session token 2.- PHP checks session token + IP to validate user 3.- If validated, echos current video m3u8 4.- If not validated, echos m3u8 saying you are not allowed to see this video

对于2:44:08视频文件，此过程耗时0.7-0.9秒，几乎对用户不可见.对于较短的视频，它的体积很小.

The process, for a 2:44:08 video file takes between 0.7 - 0.9 seconds, almost invisible to users. For shorter videos it is exponentially tiny.

云存储桶(mp4domain)-mp4.domain.com

Cloud Storage Bucket (mp4domain) - mp4.domain.com

存储桶中应用了默认ACL，该默认ACL使所有文件变为私有文件，但可用于生成签名URL的Google ID进行访问.

The bucket has a default ACL applied that makes all files private but accessible to the Google ID used to generate the signed URLs.

因此，单个视频包含以下文件:

So, a single video has the following files:

SERVER/nginx/mp4/uniqid/uniqid.m3u8
SERVER/nginx/mp4/uniqid/180p/stream.m3u8
SERVER/nginx/mp4/uniqid/360p/stream.m3u8
SERVER/nginx/mp4/uniqid/480p/stream.m3u8
SERVER/nginx/mp4/uniqid/720p/stream.m3u8
SERVER/nginx/mp4/uniqid/1080p/stream.m3u8
SERVER/nginx/mp4/uniqid/audio/stream.m3u8

GS/bucketmp4/uniqid/180p/segment##.ts
GS/bucketmp4/uniqid/360p/segment##.ts
GS/bucketmp4/uniqid/480p/segment##.ts
GS/bucketmp4/uniqid/720p/segment##.ts
GS/bucketmp4/uniqid/1080p/segment##.ts
GS/bucketmp4/uniqid/audio/segment##.aac

(SO似乎认为这是代码，不会让我格式化它)

(SO seems to think this is code and won't let me format it otherwise)

那样，对GS的写入只需完成一次，并且由于所有客户端都认为他们正在接收纯m3u8文件，因此无需在客户端进行黑客入侵.

That way, writes to GS are only done once and, since all clients think they're receiving plain m3u8 files, no hacking has to be done client-side.

希望这可以帮助遇到类似问题的人.

Hopefully this can help someone with similar issues.

相关推荐