sys/gen: generated files are too big

[tarasmadan]

Is your feature request related to a problem? Please describe.
We currently generate <10 go files, ~8.8MB each.

These files are the problem for some analysis tools.

Describe the solution you'd like

1. Generate smaller files?
2. Generate other machine readable format?
3. Syz-sysgen takes only 7 seconds to generate *.go files for all platforms on my machine. Do we need it at all?
   • Reading the descriptions directly from disk/db we'll enable live descriptions modification. How to validate their correctness and store long-term are another questions.

[dvyukov]

Things will get worse after #5545.

[a-nogikh]

4. go:embed the descriptions and parse them at the start of the syzkaller binary.

[tarasmadan]

Thanks. I've learned about embed.FS.

[a-nogikh]

FTR go:embed keeps data uncompressed, but it will likely not be a huge problem for us.

$ find sys/linux -type f \( -iname "*.txt" -o -iname "*.const" \) -exec du -ch {} +
< ... >
3.6M    total

But if we also decide to include the seeds, that's already going to be troublesome:

$ du -sh sys/linux/test/
46M     sys/linux/test/

---

That being said, we will likely still need to parse the descriptions at compile time because we auto-generate parts of syz-executor code in sys-syzgen

syzkaller/sys/syz-sysgen/sysgen.go

Lines 370 to 375 in 5df2386

	writeFile(filepath.Join(*outDir, "executor", "defs.h"), buf.Bytes())
	buf.Reset()
	if err := syscallsTempl.Execute(buf, data); err != nil {
	tool.Failf("failed to execute syscalls template: %v", err)
	}
	writeFile(filepath.Join(*outDir, "executor", "syscalls.h"), buf.Bytes())

syzkaller/sys/syz-sysgen/sysgen.go

Lines 394 to 406 in 5df2386

	var defsTempl = template.Must(template.New("").Parse(`// AUTOGENERATED FILE
	struct call_attrs_t { {{range $attr := $.CallAttrs}}
	uint64_t {{$attr}};{{end}}
	};
	struct call_props_t { {{range $attr := $.CallProps}}
	{{$attr.Type}} {{$attr.Name}};{{end}}
	};
	#define read_call_props_t(var, reader) { \{{range $attr := $.CallProps}}
	(var).{{$attr.Name}} = ({{$attr.Type}})(reader); \{{end}}
	}

[dvyukov]

I would go with go:embed for descriptions+consts for now. Executor stills needs generated files, seeds can be handled as they are now for now.

Theoretically we can make generate step that will produce a compressed archive, and then we embed the archive. But I would start w/o it. syz-manager is quite big already, and it's not causing problems.

To avoid compilation on start we could serialize what's currently in generated files as json, and then simply deserialize it on start. All generated structures use only public fields of prog types, so it should all work out of the box. Instead of having compiler create these global vars, we deserialize them from json and call prog.RegisterTarget.