Play GTA V in your Browser, Sort of (opens in new tab)

This is very reminiscent of two things for me:

- the introduction of replays in Halo 3 (which worked the same way - by saving the data of the entire session, one could freely move the camera around the entire map and observe any part of the game at any point in time)

- Super Meat Boy's level-end combined replays (which replayed all of the user's attempts simultaneously, creating a pretty amusing sort of "heatmap" effect)

I think this will eventually become standard for games where replays would be valuable or fun to watch. But I'm not sure about actually rendering live games server-side until we're at a point where input latency is unnoticeable.

Very creative idea. A similar technique is used by a few games for high score validation. The server runs the player inputs through a local simulation of the game to re-calculate the score, thus preventing blatant cheating. TAS is still possible though.

I think TIS-100 and other Zachtronics games do that as well for score validation.

> it created a text-based record of all objects and their movements on the screen,

Also known as what every single ID Engine / GoldSource / Source[1] / Unreal Engine / many more engines do for records and replays, except they do it in a binary format. It's not exactly a novel idea.

https://developer.valvesoftware.com/wiki/DEM_Format

Gakai isn't what I'm talking about tho - that's just hosting your gaming PC in the cloud.

Server side rendering would exploit the fact that all of your clients are using the same resources to render the game - so you only need to keep 1 instance of mesh x in memory and you can render it for all clients.

As I said below if you have shared instance worlds like MMO you can do much more sharing, you can have shared effects just like you have a shared simulation for clients - the effects can then be optimized for techniques that might not be viable on consumer hardware - for eg. you might have some sort of world space global illumination technique - like radiance transfer. These techniques aren't usually used because you need a lot of memory + processing power - but you can get server GPUs with >10 GB of memory and more in the future you could have one dedicated GPU that would just compute global illumination and then feed view specific data to each client view rendering thread.

It would completely change the way you render games - right now most effects are done screen space because it's faster (eg. deferred shading) but if you could share world scene state you could get a lot more fancy with world space effects. Also the rendering pipeline would need to be a lot more asynchronous to avoid latency (eg. global illumination could be a separate process that would sync occasionally and not every frame to avoid extra latency in a trade-off for lighting latency).

Yep, even on my gigabit internet and sitting smack dab in the middle of the USA (well, pretty far South, but centered) latency is still my achilles heel.

A fun little fact: the name 'Shinra' is a reference to the Shinra Corporation from Final Fantasy VII, one of Square-Enix's most beloved games.

Yes this seems what like the thing I'm talking about - true cloud rendering - not just sticking PC games on to VMs.

Well that's awesome

>You can always add more bandwidth, but latency is bound by the speed of light.

You're always bound by the same latency - your input goes to the server and server returns world state for your PC to render - the only extra latency in this scenario is time required to encode-decode the data which could be offset if the server is faster at rendering than the client PC.

As for economics I already said you can exploit the shared state very much in games if you rework the way rendering works. Right now games focus on camera space rendering because they only care about 1 view output and view space effects are cheapest in this scenario.

If you have shared state rendering suddenly you can process the instance state once per frame for 100s of users just like you simulate the game for 100s of users per instance.

You can have specialized hardware setups (ie. multiple high end GPUs with >10GB of ram) doing dedicated tasks like recomputing lighting/shadows, animation, particle effects, etc. you would need to find a way to make these systems asynchronous to reduce lag so lighting updates might lag a frame or two for eg. behind animation but as usual with rendering there are always clever tricks to fool the eye - and once you have gigabytes of ram at your disposal very different rendering techniques compared to currently used ones become viable for shared state rendering.

Someone already linked a platform that's already doing this - I have no doubt this is the future of VR and gaming at least in some part (maybe you won't be streaming video to the client but some geometric 3D world representation with deltas so that the final rendering can be done client side and you can have low latency rotation for stuff like VR)

It (as with most things) comes down to the implementation. If everything is rendered server-side, and the client is a dumb remote framebuffer, then yes, you'll notice 100ms of input lag. If, on the other hand, environments are composited on the server while the player's avatar and particles are texture-mapped server-side but then motion-tweened and composited into the environment on the client-side (which doesn't require much more work than the dumb remote framebuffer approach), then gameplay feels nearly as snappy as if everything were local.

That is different - they weren't using cloud rendering - they were just stuffing games to some sort of VM and streaming the output.

Cloud rendering would be something like you and me play the same game and the server rendering only has 1 instance of every resource used for rendering to reduce memory overhead.

If you have shared instance worlds (eg. MMO) you can then do shared state effects like animation, advanced lighting, etc. and reuse the calculations for each client.

Here's a dumb idea I just came up with: Attach a bright light that the computer can blink programatically. Write a program which: a) blinks the light, b) Blinks the light again 0.01 seconds after the camera sees the light blink, and c) repeat b a thousand times. d) Measure how long the process took relative to ten seconds, divide by 1000 to get your latency (including latency for sending the signal to the light, but hey, it's at least an upper bound).

Another thought that occurs to me is that people who work with music equipment (especially usb synths and such) care about latency a lot. It maybe worthwhile to hook into that scene and see what kind of insights can be gleaned. Here's a link to get you started: http://www.mathworks.com/help/dsp/examples/measuring-audio-l... Looks like they're basically using a feedback model as well.

Gamer care about input lag. Pro gamer on PC don't use wireless input devices, they prefer cable based ones for exactly that reason. The input lag is also what (afaik) almost killed Gaikai. Sony recently bought it for some older PS3 titles. https://en.wikipedia.org/wiki/Gaikai

Interesting! Could you elaborate? Where do the frames come from?

The API you are looking for is WebRTC. VP8 is the most widely used codec for it, but you could use H.264 baseline if you really wanted.

You will need to include a WebRTC stack in your server though, which is a lot more complicated. But I think you will get better overall performance with it versus TCP - it's what WebRTC was designed for.

Also, you might want to look at ogv.js - if you want to keep going the JS way, it includes a very fast Theora decoder which should still be a lot better than MPEG-1.

Media Source Extensions[1] may be an option. It allows you to feed bytes to a video element, sourcing those bytes from wherever you'd like. There is a demo[2] that literally opens a video and split it in to parts arbitrarily. Chrome, IE, and Safari all have support for MP4 with MSE.

[1] https://en.wikipedia.org/wiki/Media_Source_Extensions [2] http://html5-demos.appspot.com/static/media-source.html

It's essentially thin client gaming, right?

With the roll out of more fiber networks, this could be a reality (assuming you still had enough host locations to reduce latency).

I have never using OnLive, did they support playing via browser? Browser solution is key to this as almost every device support modern browsers capable of ruining OP solution.

Yes I have seen this one, but it wasn't browser based. Somehow browser based solution looks much better.

Are you sure? I thought Sony bought Gaikai, which is basically similar to OnLive - but a competitor service.

For now, yes.

Think 5-10-15 years when everyone has gigabit+ fiber to the home.

Of course for some games it is a problem, but not for every game. Also I hope in the future latency problem will be solved e.g. but placing more DC around the world.

Any idea what the actual performance impact of that is?

I will do just that. Thank you!

There's VP9 too - https://plus.google.com/+JustinUberti/posts/3FLVapCvDaG

Streaming an unbounded file to a <video> tag can have fairly significant delay compared to webRTC. It's really more designed for one to many situations (such as livestreams or VOD) than one to one low latency.

Interesting, I hadn't heard of Teamviewer as an option for this - will check it out. Thanks!

Yep, big time. The latest Rift SDK added depth-aware timewarp though, so if you could stream the depth buffer as well you could get something maybe non-sickening, but full of artifacts (transparent objects aren't even written to the depth buffer; when a nearby object is occluding the background and you peak around it in a warped frame it gets filled in a way that can stand out; specular highlights are all wrong wrong in warped frames; lossy compression of the depth buffer for streaming would cause glitching on the silhouettes of objects).

Actually there so much more you can emulate:

https://github.com/jsmess/jsmess

https://github.com/dreamlayers/em-dosbox

https://bitbucket.org/tsone/em-fceux

and others

https://github.com/kripken/emscripten/wiki/Porting-Examples-...

I agree with your point - as hackers, we are often too quick to dismiss something as "nothing new" because we have seen something similar before ("I don't need Dropbox, I have rsync").

That being said, in this case these three dots have been joined up multiple times already, so there really is nothing new here. The interesting challenge is in reducing latency so that remote games are actually playable when the server is not on the same LAN.

What are you talking about, these pieces have already been put together multiple times.

http://www.pcworld.com/article/2359241/how-steam-in-home-str...

http://shield.nvidia.co.uk/play-pc-games/

http://store.steampowered.com/streaming/

This guy isn't running it from AWS though, like I said. He's running it from a "server" that's right in front of him. By the way, have you ever tried RDCing a remote Windows machine and done some trivial tasks in the GUI?