Unsolved
This post is more than 5 years old
1 Rookie
•
8 Posts
0
1303
July 6th, 2016 14:00
ECS Single Node Won't Start
VM was hard powered off and now the docker instance won't start. I've verified the data disks are mounted:
/dev/sdc1 on /ecs/uuid-2 type xfs (rw,noatime,attr2,inode64,noquota)
/dev/sdb1 on /ecs/uuid-1 type xfs (rw,noatime,attr2,inode64,noquota)
/dev/sdd1 on /ecs/uuid-3 type xfs (rw,noatime,attr2,inode64,noquota)
/dev/sde1 on /ecs/uuid-4 type xfs (rw,noatime,attr2,inode64,noquota)
Docker service is started:
systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2016-07-06 16:00:08 CDT; 11min ago
Docs: http://docs.docker.com
Main PID: 1216 (docker)
CGroup: /system.slice/docker.service
└─1216 /usr/bin/docker daemon
But when I try to start the docker instance I get this:
docker start ecsstandalone
Error response from daemon: Cannot start container ecsstandalone: Error getting container 6d8831f8f5638c6e183869dea2396703eb3c9847703844e660a2e36c9b169545 from driver devicemapper: Error mounting '/dev/mapper/docker-253:0-137-6d8831f8f5638c6e183869dea2396703eb3c9847703844e660a2e36c9b169545' on '/var/lib/docker/devicemapper/mnt/6d8831f8f5638c6e183869dea2396703eb3c9847703844e660a2e36c9b169545': invalid argument
Error: failed to start containers: [ecsstandalone]
This device shows up in fdisk but not in the mounts. Does it need to? If so, how would I properly mount it?
Aaron_Peterson
1 Rookie
•
26 Posts
0
July 6th, 2016 16:00
fdisk may showing you devices which are physically connected but unmounted; do they show up in df -h ? If not, you may want to try mounting them via:
mount /dev/sdb1 /ecs/uuid-1 -o noatime,seclabel,attr2,inode64,noquota
and so on.
KTice1
1 Rookie
•
8 Posts
0
July 7th, 2016 06:00
Aaron - Thanks for the reply. The disks are mounted:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 37G 14G 24G 37% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 8.5M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sdc1 100G 91G 10G 91% /ecs/uuid-2
/dev/sdb1 100G 91G 10G 91% /ecs/uuid-1
/dev/sdd1 100G 91G 10G 91% /ecs/uuid-3
/dev/sde1 100G 91G 10G 91% /ecs/uuid-4
/dev/sda1 497M 164M 334M 33% /boot
tmpfs 3.2G 0 3.2G 0% /run/user/0
I have them in fstab:
/dev/sdb1 /ecs/uuid-1 xfs rw,noatime,seclabel,attr2,inode64,noquota 0 0
/dev/sdc1 /ecs/uuid-2 xfs rw,noatime,seclabel,attr2,inode64,noquota 0 0
/dev/sdd1 /ecs/uuid-3 xfs rw,noatime,seclabel,attr2,inode64,noquota 0 0
/dev/sde1 /ecs/uuid-4 xfs rw,noatime,seclabel,attr2,inode64,noquota 0 0
Aaron_Peterson
1 Rookie
•
26 Posts
0
July 7th, 2016 09:00
KTice,
I'd try to stop docker gracefully (service docker stop), do a health check of your devicemapper (thin_check --clear-needs-check-flag /var/lib/docker/devicemapper/devicemapper/metadata), and presuming that doesn't return any errors, retry starting the container.
Failing that, after referencing a couple of similar issues, it seems like the issue is likely with the docker devicemapper driver, but can be resolved by switching to their aufs driver. Unfortunately, the suggested action to do so is to wipe out your /var/lib/docker directory, install aufs (
sudo apt-get install linux-image-extra-$(uname -r)), and re-install docker.