arm64 docker - unexpected error

[cstout1]

When the container boots up I'm able to step through all the welcome screens and even authenticate to UI.com but on the final step I receive:

UNVR-4 Setup Error
An unexpected error occured during setup.

errors.log is filled with:
2021-04-13T17:18:07.574Z - error: UUM does not appear to be running: request to http://127.0.0.1:9080/api/ucore/manifest failed, reason: connect ECONNREFUSED 127.0.0.1:9080

system.log shows:
2021-04-13T16:52:50.294Z - error: Error: ENOENT: no such file or directory, open '/data/unifi-core/5fcf3809-f016-4b18-865b-550012ca4c2d'

may be related to postgresql-cluster:
systemctl status shows:

[email protected] - PostgreSQL initial setup service
   Loaded: loaded (/lib/systemd/system/[email protected]; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/[email protected]
           └─cluster-migrate.conf
   Active: active (exited) since Tue 2021-04-13 17:15:05 UTC; 6min ago
  Process: 75 ExecStartPost=/usr/bin/unifi-protect-db-cluster-cleanup (code=exited, status=0/SUCCESS)
 Main PID: 56 (code=exited, status=0/SUCCESS)

Apr 13 17:14:15 docker-desktop systemd[1]: Starting PostgreSQL initial setup service...
Apr 13 17:15:05 docker-desktop unifi-protect-db-cluster-cleanup[75]: Error: Cluster 9.6 protect does not exist
Apr 13 17:15:05 docker-desktop systemd[1]: Started PostgreSQL initial setup service.

[markdegrootnl]

The no such file or directory error is always there I guess.

The postgress error is the real problem. It's looking for a cluster which does not seem to exist. Which version of the Docker container are you using?

Maybe clear out all the storage directories and restart the container.

[Checksum]

Wonder if the issue I'm seeing is related as well. Running the latest version of docker container on a Pi. Here is what I see in the container:

root@pi:/# systemctl -l status [email protected]
● [email protected] - PostgreSQL Cluster 9.6-main
   Loaded: loaded (/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2021-05-05 10:55:26 UTC; 28s ago
  Process: 189 [email protected] --skip-systemctl-redirect 9.6-main start (code=exited, status=1/FAILURE)

May 05 10:55:26 pi systemd[1]: Starting PostgreSQL Cluster 9.6-main...
May 05 10:55:26 pi [email protected][189]: Error: Config owner (postgres:105) and data owner (root:0) do not match, and config owner is not root
May 05 10:55:26 pi systemd[1]: [email protected]: Control process exited, code=exited status=1
May 05 10:55:26 pi systemd[1]: Failed to start PostgreSQL Cluster 9.6-main.
May 05 10:55:26 pi systemd[1]: [email protected]: Unit entered failed state.
May 05 10:55:26 pi systemd[1]: [email protected]: Failed with result 'exit-code'.

Looks like the Postgres cluster can't start up. Could this possibly be a permission error?

[dividexa]

I ran the same error using RPi3B. Just a thought that this might the fact of lack of memory (1GB) on RPi3B. I suppose this container works with RPi4, so maybe just need to invest buying it. I was unable to find any detailed postgresql logs which could give any hints of the problem. Could there be way to run postgre database on other server, so then there might be change to get this working also with RPi3B?

[jay0112]

i have the same problem.
raspberry Pi 4

when i watch the system.log file, i get an following error:

" error retrieving setup device id ErrorL EACCES: Permission denied, mkdir /persistent/unifi-core"

i'm feeling i am almost there,....but cant figure it out...

[Gert05]

Same problem here. Swetp runs thtough all 8 steps, but gives an error after the final step.
Pi 4 (8GB) with Pi OS lite 64bit. Clean install with docker, docker-compose and the docker-compose.yaml as in the master branch.
Is there a known working configuration (other OS? older version?)?

[Thijxx]

Duplicate of #6

[markdegrootnl]

Please update the docker image to the latest version 1.20.3

[Thijxx]

exactly the same result with version 1.20.3
erased all data on disk, erased the image and the container

[markdegrootnl]

Thanks for the reply. It seems upgrading an existing instance (keeping the storage) does work, but starting a new one doesn't. Will investigate.

[markdegrootnl]

I've updated the docker image. Please (re-)pull 1.20.3 and try. Also add a volume for /persistent

[vPotusFlotus]

Hi all,

For me the root cause of the error message 'An unexpected error occured during setup.' seemed to be due to the Change Hostname from the 'ubnt-systool' that failed because it's now running containerised (and tries to modify /etc/hosts or /etc/hostname). See point 3 below.

I was finally able to get a new instance up and running on my Raspberry Pi 3 B+ (using the ARM version of Raspberry Pi OS).
'uname-a' for reference:

Linux raspberrypi 5.10.63-v8+ #1488 SMP PREEMPT Thu Nov 18 16:16:16 GMT 2021 aarch64 GNU/Linux

The following tweaks were needed to make it work:

1. Initially the container did not seem very responsive, it was up according to Docker PS, however no real logs could be seen in the container because systemd could not be launched in it. This apparently had to do with the CGroup settings of Raspberry Pi OS in combination with the latest version of Docker (moved to cgroupsv2?).
   How to fix it:

Add the following in your '/etc/docker/daemon.json' on your Raspberry Pi:

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "features": { "buildkit": true },
  "experimental": true,
  "cgroup-parent": "docker.slice"
}

Reload systemd & Restart your Docker Service.

sudo systemctl daemon-reload
sudo systemctl restart docker

Next make sure to run your container with the following extra options (use with caution, generally you would like to avoid privileged mode etc. on containers):

--cap-add SYS_ADMIN \
--security-opt seccomp=unconfined \
--cgroup-parent=docker.slice \
--cgroupns private \

The entire command to run the container on a Raspberry Pi looks like this then:

docker run -d --name unifi-protect-arm64  \
    --privileged \
    --cap-add SYS_ADMIN \
    --security-opt seccomp=unconfined \
    --tmpfs /var/run \
    --tmpfs /var/run/lock \
    --tmpfs /tmp \
    -v /storage/srv:/srv \
    -v /storage/data:/data \
    -v /storage/persistent:/persistent \
    --network host \
    --cgroup-parent=docker.slice \
    --cgroupns private \
    markdegroot/unifi-protect-arm64:latest

2. Now the container was responsive and some logs were shown. It was starting to boot! However quickly I was greeted with the following Error Message in the journalctl of the Container itself:

fork rejected by pids controller in /docker.slice/someIDHere/system.slice/unifi-core.service

Basycially the underlying systemd instance of the Raspberry Pi did not allow another Task to be run in the container.

This can be resolved by editing your systemd config file on the Raspberry Pi (so not in the container):

/etc/systemd/system.conf --> Change the line '#DefaultTasksMax=15%' to 'DefaultTasksMax=infinity'

I rebooted the Raspberry Pi afterwards to make sure the changes were taken into effect.

Now the container is successfully booting and does show up the UNVR webpage when browsing to it (it takes around 5 - 8min. on my Pi to startup all the systemd Services etc.).

3. When going through the Setup Wizard on the container's website, I was also greeted with the following Error Message after (or at) step 8:

UNVR-4 Setup Error
An unexpected error occured during setup.

I checked the logs in the container (tail -f /data/unifi-core/logs/system.log) while I was completing the setup. I noticed that the Ubiquiti Application tried to Change the Hostname of the Container (in /etc/hostname & /etc/hosts) by using the Ubiuqiti Tool '/sbin/ubnt-systool 'and throws the following Error:

Unable to modify '/etc/DIHQYPMEJ', resource is busy

I quickly checked the contents of '/sbin/ubnt-systool' with cat and noticed that indeed this Tool tries to set the '/etc/hostname' & '/etc/hosts' file by doing the following:

change_hostname() {
	echo "$1" > /etc/hostname
	echo "$1" > /proc/sys/kernel/hostname
	local HLINE="127.0.1.1	${1}"
	if grep -q 127.0.1.1 /etc/hosts 2>/dev/null; then
		sed -i "s/^127.0.1.1.*/${HLINE}/" /etc/hosts
	else
		echo "$HLINE" >> /etc/hosts
	fi
}

In my (limited) understanding of Docker, this is unfortunately not possible in a Docker Container. You can edit the /etc/hostname or /etc/hosts file but you cannot move it or create a new one (what the 'sed' command above is doing).

So what did I do?

I installed 'vim' on the Container:

sudo apt-get update
sudo apt-get install vim

I'm already happy with my container's hostname and for me it's not needed to change it.

So after the container was running (the setup page wizard shows but I did not start the setup wizard yet) I opened the '/sbin/ubnt-systool' file and edited the 'change_hostname' function to do basically nothing:

change_hostname() {
	echo "no changes required"
}

And made sure to save it (esc + :wq!). You can also alter the 'change_hostname' function to indeed successfully change the hostname by not using 'sed' or by using specific options with the 'sed' command if needed.

(I'm running the container with the --network host option so it shares the /etc/hosts file from the Raspberry Pi)

4. I also encountered an issue with the 'unifi-core' service complaining about not being able resolve the new Hostname that you've entered during the setup.

I resolved this by editing my Raspberry Pi's /etc/hosts file to include the container's hostname and point at localhost before I ran the container. With the option '--network host' the container shares the Network Settings of my Raspberry Pi so also the /etc/hosts file. After walking through the Setup Wizard again the error is gone (or I guess you can just give your UNVR the same name as your raspberry pi to resolve this error).

Next up I started the Setup Wizard and it completed successfully.

However I'm not using the Container for Production Workload, only as a Test to play around with it currently.

But for me the root cause of the error message 'An unexpected error occured during setup.' seemed to be due to the Chang Hostname from the 'ubnt-systool' that failed because it's now running containerised (and tries to modify /etc/hosts or /etc/hostname).

I hope it helps :-)

Resources that helped in troubleshooting this:

• https://serverfault.com/questions/1032747/cgroup-fork-rejected-by-pids-controller
• https://serverfault.com/questions/1053187/systemd-fails-to-run-in-a-docker-container-when-using-cgroupv2-cgroupns-priva
• https://stackoverflow.com/questions/28327458/how-to-add-my-containers-hostname-to-etc-hosts

[paqpaqpaq]

@vPotusFlotus

Thanks for your detailed post.
I have a/the similar issue; I keep receiving the setup error at step 8.

That is: I can't get past this 1 (?) error (system.log):

Error: Command failed: /usr/bin/sudo -n /sbin/ubnt-systool hostname raspberrypi
sed: cannot rename /etc/sedtpLwhh: Device or resource busy
stderr: sed: cannot rename /etc/sedtpLwhh: Device or resource busy

For starters I somehow do not have an existing /sbin/ubnt-systool.
Then I received a copy of an actual system, and pasted that into a new file.

Unfortunately it returns te very same error.

If I run that exact command separately, I get this in return:

/sbin/ubnt-systool: 948: /sbin/ubnt-systool: Syntax error: newline unexpected (expecting ")")

edit:
OK sorry, I needed to look for the ubnt-systool file INSIDE the container. [facepalm]

It seems to be working!
on iOS (latest app)

[networkfusion]

Thanks, this was really helpful. The only thing I would add is how to connect to the docker container terminal (mainly adding for self reference): https://www.ibm.com/docs/en/workload-automation/9.5.0?topic=compose-accessing-docker-containers
And I personally installed nano instead of vim.

Also noted that updating the protect app to 2.0.1 seems to work fine, although have not tried updating the UNVR firmware

Hopefully this will fixed in a future update.

[Gizimo]

this was VERY HELPFUL!!! THANK YOU @vPotusFlotus

[trenb]

Thank you @vPotusFlotus, your instructions were very clear. I have it running on a rpi3+.