Skip to content

Commit

Permalink
Merge pull request #12 from cneira/v1.2
Browse files Browse the repository at this point in the history
V1.2
  • Loading branch information
cneira authored May 12, 2021
2 parents 1f40e3c + bb93493 commit ac74b34
Show file tree
Hide file tree
Showing 43 changed files with 308 additions and 7,174 deletions.
101 changes: 87 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Requirements
- KVM enabled in your Linux kernel, and you have read/write access to /dev/kvm.
- tun module loaded
- ip6tables package
- [Container networking plugins](https://github.com/containernetworking/plugins)
- [Container networking plugins (Firecracker branch)](https://github.com/cneira/plugins)
- [tc-redirect-tap](https://github.com/awslabs/tc-redirect-tap)
- [Firecracker binary](https://github.com/firecracker-microvm/firecracker/releases/download/v0.16.0/firecracker-v0.16.0)

Expand Down Expand Up @@ -45,7 +45,7 @@ For more details see the nomad [docs](https://www.nomadproject.io/docs/configura

Container network configuration
---------------------------------------
- Build [cni plugins](https://github.com/containernetworking/plugins) and [tc-redirect-tap](https://github.com/awslabs/tc-redirect-tap) and copy them /opt/cni/bin
- Build [cni plugins (Firecracker branch)](https://github.com/cneira/plugins) and [tc-redirect-tap](https://github.com/awslabs/tc-redirect-tap) and copy them /opt/cni/bin
- Create a network configuration to be used by micro-vms on /etc/cni/conf.d/, for example: default.conflist.


Expand Down Expand Up @@ -114,14 +114,48 @@ Also the filename must match the name of the network, and the suffix .conflist.
Creating a rootfs and kernel image for firecracker
-------------------------------------------------
We need to an ext4 root filesystem to use as disk and an uncompressed vmlinux image, the process on how to generate them is described [here](https://github.com/firecracker-microvm/firecracker/blob/master/docs/rootfs-and-kernel-setup.md).
Prebuilt kernel images and rootfs are provided, default root password for these images is 'toor':

- [Kernel Image Linux-5.4.0-rc5](https://firecracker-kernels.s3-sa-east-1.amazonaws.com/vmlinux-5.4.0-rc5.tar.gz)
- [Rootfs for Ubuntu 16.04](https://firecracker-rootfs.s3-sa-east-1.amazonaws.com/ubuntu16.04.rootfs.tar.gz)
- [Rootfs for Ubuntu 18.04](https://firecracker-rootfs.s3-sa-east-1.amazonaws.com/ubuntu18.04.rootfs.tar.gz)
- [Rootfs for Debian 10](https://firecracker-rootfs.s3-sa-east-1.amazonaws.com/debian10.rootfs.tar.gz)
- [Rootfs for Centos 7](https://firecracker-rootfs.s3-sa-east-1.amazonaws.com/centos-7-x86_64_rootfs.tar.gz)
### Using ZFS zvols to create a rootfs for microvms

Leveraging [ZFS](https://github.com/openzfs/zfs) zvols to expose rootfs to firecracker is really simple, and zfs [has a lot of benefits](https://openzfs.org/wiki/Features).

First download a template image, for example from [OpenVZ](openvz.org)[Centos7](http://download.openvz.org/template/precreated/centos-7-x86_64-minimal.tar.gz)

Now create a ZVOL to host this tarball

```sh
$ zfs create -V 1G zpool/centos7vm
$ mkfs.ext4 /dev/zvol/zpool/centos7vm
$ mount -t ext4 /dev/zvol/zpool/centos7vm /mnt
$ tar xfvz centos-7-x86_64-minimal.tar.gz -C /mnt
$ zfs snapshot zpool/centos7vm@final
```
Now just use your new zvol as your BootDisk For example:

```hcl
job "example3" {
datacenters = ["dc1"]
type = "service"
group "test" {
restart {
attempts = 0
mode = "fail"
}
task "test01" {
driver = "firecracker-task-driver"
config {
Vcpus = 1
Mem = 128
KernelImage= "/home/cneira/kernel-images/vmlinux.bin"
BootDisk = "/dev/zvol/vms/centos7vm"
Network = "default"
}
}
}
}
```


## Firecracker task driver options
Expand Down Expand Up @@ -181,16 +215,19 @@ will contain the following info :

```json
{
"AllocId": "785f9472-52a7-3dbf-8305-d482b1f7dc6f",
"Ip": "192.168.127.77/24",
"Serial": "/dev/pts/9",
"Pid": "14118"
"AllocId": "590983f4-499a-380f-420e-e5be4d5f46d9",
"Ip": "192.168.127.62/24",
"Serial": "/dev/pts/3",
"Pid": "237216",
"Vnic": "veth05fb4547vm"
}

```
- AllocId (given by nomad)
- Ip (Ip address assigned by cni configuration)
- Serial (tty where a serial console is setup for the vm)
- Pid ( Pid for the firecracker process that started the vm)
- Vnic (virtual interface on the host linked to the vm)

## Examples:

Expand Down Expand Up @@ -258,8 +295,6 @@ job "cni-network-configuration-example" {

### Additional Disks configuration
-------------------------------



```hcl
job "neverwinter" {
Expand All @@ -277,6 +312,41 @@ job "neverwinter" {
}
}
}
```
### Accessing the microvm using serial console

The firecracker-task-driver exposes the serial console as this option is handy to troubleshoot network
issues.
Each microvm generates a state file on the /tmp/ directory, named using the job name + allocation id.
For example:

```sh
-rw-r--r--. 1 root root 152 May 12 14:07 /tmp/test01-590983f4-499a-380f-420e-e5be4d5f46d9
```
The contents of the state file should be like the following:

```json
{
"AllocId": "590983f4-499a-380f-420e-e5be4d5f46d9",
"Ip": "192.168.127.62/24",
"Serial": "/dev/pts/3",
"Pid": "237216",
"Vnic": "veth05fb4547vm"
}
```
Using the serial now we know which serial port is expose and it's a matter of connect to it.
You could use SCREEN(1) to connect to the serial console.

```sh
$ sudo screen /dev/pts/3

Started Update UTMP about System Runlevel Changes.

CentOS Linux 7 (Core)
Kernel 4.14.225 on an x86_64

192 login:

```

## Demo
Expand All @@ -287,6 +357,9 @@ job "neverwinter" {
[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/J3J4YM9U)

It's also possible to support the project on [Patreon](https://www.patreon.com/neirac)

I work on this project on my free time and my country is not on the list available for
github sponsors so any help for me continue working on this is appreciated.

## References

Expand Down
13 changes: 12 additions & 1 deletion driver/driver.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import (
"time"

hclog "github.com/hashicorp/go-hclog"
"github.com/hashicorp/nomad/client/stats"
"github.com/hashicorp/nomad/drivers/shared/eventer"
"github.com/hashicorp/nomad/plugins/base"
"github.com/hashicorp/nomad/plugins/drivers"
Expand Down Expand Up @@ -265,6 +266,9 @@ func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {
MachineInstance: m.Machine,
Info: m.Info,
logger: d.logger,
cpuStatsSys: stats.NewCpuStats(),
cpuStatsUser: stats.NewCpuStats(),
cpuStatsTotal: stats.NewCpuStats(),
}

d.tasks.Set(taskState.TaskConfig.ID, h)
Expand Down Expand Up @@ -300,6 +304,9 @@ func (d *Driver) StartTask(cfg *drivers.TaskConfig) (*drivers.TaskHandle, *drive
MachineInstance: m.Machine,
Info: m.Info,
logger: d.logger,
cpuStatsSys: stats.NewCpuStats(),
cpuStatsUser: stats.NewCpuStats(),
cpuStatsTotal: stats.NewCpuStats(),
}

driverState := TaskState{
Expand Down Expand Up @@ -398,13 +405,17 @@ func (d *Driver) InspectTask(taskID string) (*drivers.TaskStatus, error) {
return handle.TaskStatus(), nil
}

// TaskStats will query the driver and return the current usage for the vm
func (d *Driver) TaskStats(ctx context.Context, taskID string, interval time.Duration) (<-chan *drivers.TaskResourceUsage, error) {
handle, ok := d.tasks.Get(taskID)
if !ok {
return nil, drivers.ErrTaskNotFound
}

return handle.stats(ctx, interval)
statsChannel := make(chan *drivers.TaskResourceUsage)
go handle.stats(ctx, statsChannel, interval)

return statsChannel, nil
}

func (d *Driver) TaskEvents(ctx context.Context) (<-chan *drivers.TaskEvent, error) {
Expand Down
33 changes: 19 additions & 14 deletions driver/firevm.go
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
/* Firecracker-task-driver is a task driver for Hashicorp's nomad that allows
* to create microvms using AWS Firecracker vmm
* Copyright (C) 2019 Carlos Neira [email protected]
*
*
* This file is part of Firecracker-task-driver.
*
*
* Foobar is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 2 of the License, or
* (at your option) any later version.
*
*
* Firecracker-task-driver is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
*
* You should have received a copy of the GNU General Public License
* along with Firecracker-task-driver. If not, see <http://www.gnu.org/licenses/>.
*/
Expand Down Expand Up @@ -84,18 +84,19 @@ func taskConfig2FirecrackerOpts(taskConfig TaskConfig, cfg *drivers.TaskConfig)
opts.Debug = true
opts.FcLogLevel = "Debug"
}
if taskConfig.Vcpus > 0 {
opts.FcCPUCount = int64(taskConfig.Vcpus)

if cfg.Resources.NomadResources.Cpu.CpuShares > 100 {
opts.FcCPUCount = cfg.Resources.NomadResources.Cpu.CpuShares / 100
} else {
opts.FcCPUCount = int64(1)
opts.FcCPUCount = 1
}
opts.FcCPUTemplate = taskConfig.Cputype
opts.FcDisableHt = taskConfig.DisableHt

if taskConfig.Mem > 0 {
opts.FcMemSz = int64(taskConfig.Mem)
if cfg.Resources.NomadResources.Memory.MemoryMB > 0 {
opts.FcMemSz = cfg.Resources.NomadResources.Memory.MemoryMB
} else {
opts.FcMemSz = int64(512)
opts.FcMemSz = 300
}
opts.FcBinary = taskConfig.Firecracker

Expand All @@ -112,11 +113,12 @@ type Instance_info struct {
Ip string
Serial string
Pid string
Vnic string
}

func (d *Driver) initializeContainer(ctx context.Context, cfg *drivers.TaskConfig, taskConfig TaskConfig) (*vminfo, error) {
opts, _ := taskConfig2FirecrackerOpts(taskConfig, cfg)
fcCfg, err := opts.getFirecrackerConfig()
fcCfg, err := opts.getFirecrackerConfig(cfg.AllocID)
if err != nil {
log.Errorf("Error: %s", err)
return nil, err
Expand Down Expand Up @@ -196,27 +198,30 @@ func (d *Driver) initializeContainer(ctx context.Context, cfg *drivers.TaskConfi
return nil, fmt.Errorf("Failed getting pid for machine: %v", errpid)
}
var ip string
var vnic string
if len(opts.FcNetworkName) > 0 {
ip = fcCfg.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IPAddr.String()
vnic = fcCfg.NetworkInterfaces[0].CNIConfiguration.IfName + "vm"
} else {
ip = "No network chosen"
vnic = ip
}
info := Instance_info{Serial: ftty, AllocId: cfg.AllocID,
Ip: ip,
Pid: strconv.Itoa(pid)}
Pid: strconv.Itoa(pid), Vnic: vnic}

f, _ := json.MarshalIndent(info, "", " ")

logfile := fmt.Sprintf("/tmp/%s-%s", cfg.Name, cfg.AllocID)

d.logger.Info("Writing to", "driver_initialize_container", hclog.Fmt("%v+", logfile))
log, err := os.OpenFile(logfile,os.O_CREATE|os.O_APPEND|os.O_WRONLY,0644)
log, err := os.OpenFile(logfile, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0644)

if err != nil {
return nil, fmt.Errorf("Failed creating info file=%s err=%v", logfile, err)
}
defer log.Close()
fmt.Fprintf(log,"%s",f)
fmt.Fprintf(log, "%s", f)

return &vminfo{Machine: m, tty: ftty, Info: info}, nil
}
Loading

0 comments on commit ac74b34

Please sign in to comment.