Possible to get road segmentation?

[benjamintanweihao]

Now it seems like buildings and roads are of the same color.

[dominikdienlin]

It doesn't only seem that way, roads, buildings and mostly anything made of concrete shares the same stencil value. As far as I know the stencil buffer only assigns different values to people, animals, water, sky, vehicles (eg. cars, trucks, bicycles), things made of concrete (eg. buildings, roads), certain light volumes (e.g. from street lamps at night), vegitation (grass, trees, but sometimes also signs on buildings), and some fringe classes that are hard to make out (military entities? Aliens?). Road segmentation, especially lane markings are not possible to extract from the game engine (please correct me if I'm wrong). You can make some educated guesses about the properties of the road and the lanes as done in the lanerewarder class of DeepGTAV

[benjamintanweihao]

Thanks, that makes sense. Is there a way to get patches like these though (without resorting to renderdoc):

[dominikdienlin]

That looks like from the "playing for data" paper by the Visual Inference lab TU Darmstadt. If I remember correctly, they used a directx debugging tool to extract the mesh texture IDs and then mapped those IDs to object classes. However, even those don't seem to contain the lane markings as you can see in the image. It may be possible to do something similar with the technique used here (if you can find the direct X buffer containing those mesh IDs in the rendering pipeline), but I'm no expert when it comes this stuff. Either way youd have to extract the lane markings with some computer vision techniques (some of which are fairly robust I think)

[benjamintanweihao]

Yup it's taken from there :). I don't really need lane markings, just the entire road.

[dominikdienlin]

Then you could have a look at https://github.com/aitorzip/DeepGTAV/blob/master/Rewarders/LaneRewarder.cpp . This gives you the bounds for the road and even lanes (I think), but it's not always accurate. It would probably be best to visualise this information by drawing the corresponding points using the in-game draw functions.