Not converting CSV to Parquet in Windows

[imrantariq]

Hi,

I tried running this code in Linux and it creates Parquet file but on Windows machine it is not working and I am getting NullPointerException on following line in method createDataset(String resourcePath, String filePath, Class<T> type).

return Datasets.create(format("dataset:file://%s", filePath), descriptor, type);

I think the problem is with file paths in windows. Any suggestions?

[nezihyigitbasi]

I never tried the code on Windows. Can you post the full stacktrace?

[imrantariq]

Exception in thread "main" java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783)
at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.writeDescriptor(FileSystemMetadataProvider.java:475)
at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.create(FileSystemMetadataProvider.java:230)
at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.create(FileSystemDatasetRepository.java:137)
at org.kitesdk.data.Datasets.create(Datasets.java:239)
at org.kitesdk.data.Datasets.create(Datasets.java:307)
at yigitbasi.nezih.ConvertToParquet.createDataset(ConvertToParquet.java:65)
at yigitbasi.nezih.ConvertToParquet.convertBusinessesToParquet(ConvertToParquet.java:70)
at yigitbasi.nezih.ConvertToParquet.main(ConvertToParquet.java:171)

[nezihyigitbasi]

I would try tweaking the paths in the source code (i.e., /tmp/violations) and give proper paths on your windows machine.