NYU dataset

This tutorial reads and visualizes an RGBDImage from the NYU dataset [Silberman2012].

 5# examples/Python/Basic/rgbd_nyu.py
 6
 7import open3d as o3d
 8import numpy as np
 9import re
10import matplotlib.image as mpimg
11import matplotlib.pyplot as plt
12
13
14# This is special function used for reading NYU pgm format
15# as it is written in big endian byte order.
16def read_nyu_pgm(filename, byteorder='>'):
17    with open(filename, 'rb') as f:
18        buffer = f.read()
19    try:
20        header, width, height, maxval = re.search(
21            b"(^P5\s(?:\s*#.*[\r\n])*"
22            b"(\d+)\s(?:\s*#.*[\r\n])*"
23            b"(\d+)\s(?:\s*#.*[\r\n])*"
24            b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
25    except AttributeError:
26        raise ValueError("Not a raw PGM file: '%s'" % filename)
27    img = np.frombuffer(buffer,
28                        dtype=byteorder + 'u2',
29                        count=int(width) * int(height),
30                        offset=len(header)).reshape((int(height), int(width)))
31    img_out = img.astype('u2')
32    return img_out
33
34
35if __name__ == "__main__":
36    print("Read NYU dataset")
37    # Open3D does not support ppm/pgm file yet. Not using o3d.io.read_image here.
38    # MathplotImage having some ISSUE with NYU pgm file. Not using imread for pgm.
39    color_raw = mpimg.imread("../../TestData/RGBD/other_formats/NYU_color.ppm")
40    depth_raw = read_nyu_pgm("../../TestData/RGBD/other_formats/NYU_depth.pgm")
41    color = o3d.geometry.Image(color_raw)
42    depth = o3d.geometry.Image(depth_raw)
43    rgbd_image = o3d.geometry.RGBDImage.create_from_nyu_format(color, depth)
44    print(rgbd_image)
45    plt.subplot(1, 2, 1)
46    plt.title('NYU grayscale image')
47    plt.imshow(rgbd_image.color)
48    plt.subplot(1, 2, 2)
49    plt.title('NYU depth image')
50    plt.imshow(rgbd_image.depth)
51    plt.show()
52    pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
53        rgbd_image,
54        o3d.camera.PinholeCameraIntrinsic(
55            o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
56    # Flip it, otherwise the pointcloud will be upside down
57    pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
58    o3d.visualization.draw_geometries([pcd])

This tutorial is almost the same as the tutorial processing Redwood dataset, with two differences. First, NYU images are not in standard jpg or png formats. Thus, we use mpimg.imread to read the color image as a numpy array and convert it to an Open3D Image. An additional helper function read_nyu_pgm is called to read depth images from the special big endian pgm format used in the NYU dataset. Second, we use a different conversion function create_rgbd_image_from_nyu_format to parse depth images in the SUN dataset.

Similarly, the RGBDImage can be rendered as numpy arrays:

../../../_images/nyu_rgbd.png

Or a point cloud:

../../../_images/nyu_pcd.png